Python tricks

Core features

Decoration function

def decorator_func(original_func):
    def wrapper_func(*args, **kwargs):
        # do something before
        result = original_func(*args, **kwargs)
        # do something after
        return result
    return wrapper_func

@decorator_func
def target_func(...):
    # ...

Loop

Loop through a dictionary with `it` `key` `value`

for it, (key, value) in dict:
	# ...

Loop through combinatons of lists

from itertools import product

for i, j in product(ls_1, ls_2):
  # ...

Scripting

Command line arguments

Example:

import argparse

parser = argparse.ArgumentParser(description='Description of your script')
parser.add_argument('--mesh_dir', type=str, required=True, help='Path to the mesh directory')
parser.add_argument('--output', action='store_true', help='Whether to save the output')

args = parser.parse_args()
args.mesh_dir  # Access the mesh directory path
args.output  # Access the output flag (default: False)

Add folder to Python path

Example:

# Add project root to Python path so `from src.*` imports work
# when running this script from any directory
import os
import sys
_PROJECT_ROOT = os.path.join(os.path.dirname(os.path.abspath(__file__)), '..', '..')
sys.path.insert(0, _PROJECT_ROOT)
sys.path.insert(0, os.path.join(_PROJECT_ROOT, 'pkg', 'animer'))

Run script as module

Example:

python -m src.animer_grounding.run --mesh_dir /home/knpob/Documents/Hinton/data/shape-corr/SMAL_r/off/ --out_dir output/smal_r --device cuda:1

The current directory is added to sys.path, making package-level imports resolve properly.

Files

Dotfiles

theskumar/python-dotenv: Reads key-value pairs from a .env file and can set them as environment variables. It helps in developin

Install dependency:

pip install python-dotenv

Create a .env file:

<key1>='<value1>'
<key2>='<value2>'

Load it from Python:

import os
from dotenv import load_dotenv

load_dotenv('.env')
value1 = os.getenv("key1")
value2 = os.getenv("key2")

Don’t forget to ignore .env in .gitignore:

.env

Find all files with certain extension under a folder

from glob import glob
mesh_ls = sorted(glob(str('<path>/*.<ext>')))

Widgets

Progress bar in Python

pip install tqdm

Then in the script:

from tqdm import tqdm

for i in tqdm(range(100), desc="Loading..."):
    # ...

If the description needed to be updated at each iteration:

from tqdm import tqdm

for i in (pdar := tqdm(range(100))):
   pdar.set_description('...')
   # ...

Matplotlib colors

List of named colors — Matplotlib 3.9.2 documentation

My preferred ones:

goldenrod
teal
olive
lightcoral

Environment

Conda environment management

Create a new environment: conda create -n <env_name> python=<version>
Remove an environment: conda remove -n <env_name> --all
Activate environment conda activate <env_name>
Deactivate environment conda deactivate

One-off script running in a conda environment

conda run -n <env_name> python <script.py>

Export conda `environment.yml`

Exporting the environment.yml file | conda 25.3.2.dev62 documentation

conda env export > environment.yml

Creating an environment with commands | conda 25.3.2.dev62 documentation

conda env create -f environment.yml

Or just add packages to an existed environment:

conda env update -f environment.yml

Proxy

![[ubuntu-dev-env#Python with proxy]]

Matplotlib

SciencePlots

GitHub - garrettj403/SciencePlots: Matplotlib styles for scientific plotting · GitHub

pip install SciencePlots

import matplotlib.pyplot as plt
import scienceplots
plt.style.use('ieee')

Axis formatter

from matplotlib.ticker import FuncFormatter
# ...
ax.xaxis.set_major_formatter(FuncFormatter(lambda x, _: f'{int(x/1000)}k'))

Legend

ax.legend(frameon=True, edgecolor='black', framealpha=0.5, fontsize=12, loc='best')

Jupyter notebook

Register a `.venv` kernel

Example:

./pkg/PRIMA/.venv/bin/python -m ipykernel install \
  --user \
  --name prima-venv \
  --display-name "Python (PRIMA .venv)"

Launching Jupyter Lab

Ensure that jupyterlab has been installed:

pip install jupyterlab

Then:

jupyter lab --no-browser --port=8888

Add folder to Python path

import sys
sys.path.append('<path>')

P.S. In Jupyter Notebook, if you’d like to change the executing directory directly, you can use:

%cd <path>

Change IPython/Jupyter notebook working directory - Stack Overflow

Reload package in Jupyter Notebook

By adding this cell to the notebook, package can be automatically reloaded. That’s incredibly important when we are developing & testing a package on the go:

%reload_ext autoreload
%autoreload 2

P.S. It reloads every imported package before running each cell, which may slow down the execution time.

python - How to make VSCode auto-reload external *.py modules? - Stack Overflow

A package can also be reloaded manually:

import importlib
importlib.reload(<pkg>)

e.g.

# import self-defined modules
import importlib
import src.mod as mod

# reload the module everytime the cell is run
importlib.reload(mod)

# load what's actually needed
from src.mod import cls, func

Auto refresh imports (support %autoreload magic) · Issue #4555 · microsoft/vscode-jupyter

Run Jupyter Notebook from terminal

python - How to run an .ipynb Jupyter Notebook from terminal? - Stack Overflow

Running notebooks from command line have two use cases:

Scripting notebook runs.
Avoid having to restarting the kernel when you edit the imported packages.

pip install nbconvert

Then:

jupyter nbconvert --execute --to notebook --inplace <notebook>

To make it easier to type:

alias nbx="jupyter nbconvert --execute --to notebook --inplace"
nbx <notebook>

P.S. It can be accompanied by command line arguments, e.g.:

owner=Knpob nbx convert-alipay.ipynb

Then in the notebooks:

import os

try:
    owner = os.environ['owner']
except:
    pass

Git ignore Jupyter Notebook outputs

How to commit jupyter notebooks without output to git while keeping the notebooks outputs intact locally

Add a filter to git config by running the following command in bash inside the repo:

git config filter.strip-notebook-output.clean 'jupyter nbconvert --ClearOutputPreprocessor.enabled=True --to=notebook --stdin --stdout --log-level=ERROR'

Create a .gitattributes file inside the directory with the notebooks. Add *.ipynb filter=strip-notebook-output to that file:

cd <notebook folder>
touch .gitattributes
echo '*.ipynb filter=strip-notebook-output' > .gitattributes

Example:

cd notebook
touch .gitattributes
echo '*.ipynb filter=strip-notebook-output' > .gitattributes

cd prototype
touch .gitattributes
echo '*.ipynb filter=strip-notebook-output' > .gitattributes

cd ../..

After that, commit to Git as usual. The notebook output will be stripped when the notebook is staged in Git, but the file will remain unchanged locally. P.S. Make sure that jupyterlab is installed in the Python environment.

This gist is based on @dirkjot's answer.

How to clear Jupyter Notebook's output in all cells from the Linux terminal? - Stack Overflow

P.S. In VS Code, the diff of .ipynb file could be selected to ignore outputs/metadata changes in the drop down menu:

SOLVED - Have vscode-jupyter do proper diffing in Git · microsoft/vscode-jupyter · Discussion #10742 · GitHub

PyVista remote rendering

![[ubuntu-dev-env#PyVista remote rendering]]

Release to PyPI

Building and Publishing - Python Packaging User Guide
Publishing package distribution releases using GitHub Actions CI/CD workflows - Python Packaging User Guide

Prepare

pip install twine build

Setup `pyproject.toml`

# local install: pip install -e .
# local build: python -m build

[build-system]
requires = ["setuptools>=61.0"]
build-backend = "setuptools.build_meta"

[project]
name = "<pkg>"
version = "<ver>"
description = "<str>"
readme = { file = "README.md", content-type = "text/markdown" }
requires-python = ">=3.7"
license = { text = "<license>" }
authors = [
  { name = "<name>", email = "<email>" }
]
dependencies = [
  "<pkg1>",
  "<pkg2>",
]

[project.urls]
Repository = "<repo link>"
# You can also add more links, e.g. Homepage, Documentation, Bug Tracker, etc.

[tool.setuptools.packages.find]
where = ["."]
include = ["<pkg folder>*"]

P.S. The pyproject.toml can also be used for local install:

pip install -e .

Build & check

Firstly, to avoid including unnecessary or even sensitive files, e.g. you API keys in .env files, clone the project to other places. In that folder, build the project:

python -m build

Check:

twine check dist/*

Upload

If you want to further confirm the release is OK, firstly upload it to TestPyPi:

twine upload --repository-url https://test.pypi.org/legacy/ dist/*

When you're ready, upload it to PyPI:

twine upload --repository-url dist/*

P.S. You need to signup an account and acquire the API key on both PyPI and TestPyPI, separately.

Cleanup

Clear the dist/ folder:

rm -rf dist/

Hugging Face

Install and login

curl -LsSf https://hf.co/cli/install.sh | bash
hf auth login

Download Hugging Face models to local path

Command Line Interface (CLI)

hf download <user>/<repo> --local-dir <path>

Download Hugging Face dataset to local path

Downloading datasets
Command Line Interface (CLI)

hf download <user>/<repo> --repo-type dataset --local-dir <path>

Upload local folder as Hugging Face dataset

hf upload [repo_id] [local_path] [path_in_repo]
hf upload <repo> . . # upload the current directory at the root of the repo

Every hf upload creates a commit on HF with a timestamp, so you can always roll back to a previous version via the repo's commit history on the web UI.

Python tricks

Core features

Decoration function

Loop

Loop through a dictionary with it key value

Loop through combinatons of lists

Scripting

Command line arguments

Add folder to Python path

Run script as module

Files

Dotfiles

Find all files with certain extension under a folder

Widgets

Progress bar in Python

Matplotlib colors

Environment

Conda environment management

One-off script running in a conda environment

Export conda environment.yml

Proxy

Matplotlib

SciencePlots

Fontsize

Axis formatter

Legend

Jupyter notebook

Register a .venv kernel

Launching Jupyter Lab

Add folder to Python path

Reload package in Jupyter Notebook

Run Jupyter Notebook from terminal

Git ignore Jupyter Notebook outputs

PyVista remote rendering

Release to PyPI

Prepare

Setup pyproject.toml

Build & check

Upload

Cleanup

Hugging Face

Install and login

Download Hugging Face models to local path

Download Hugging Face dataset to local path

Upload local folder as Hugging Face dataset

Loop through a dictionary with `it` `key` `value`

Export conda `environment.yml`

Register a `.venv` kernel

Setup `pyproject.toml`