Python

Python on Pathfinder

Python is heavily used on CADES. Some users require specific versions of Python or packages, which may further depend on numerous other Python packages. Because of all the dependencies that some Python packages require, and all the types of data that exist, it can be quite troublesome to get different Python installations to “play nicely” with each-other, especially on an HPC system where the system environment is complicated. Conda, a package and virtual environment manager from the conda-forge distribution, helps alleviate these issues. Conda allows users to easily install different versions of binary software packages and any required libraries appropriate for their computing platform. To start using the new python-conda environment on CADES, all you need to do is load the module:

module load miniforge3/24.11.3-0

Loading the Python module will put you in a “base” pre-configured conda environment. This option is recommended for users who do not need custom environments, and only require packages that are already installed in the base environment. This option is also recommended for users that just need a Python interpreter or standard packages.

Users can also create their own custom conda environment after loading the Python module. This option is recommended for users that require a different version of Python than the default version available, or for users that want a personal environment to manage specialized packages.

module load miniforge3/24.11.3-0
conda create -p /path/to/my_env python=3.11
source activate /path/to/my_env

Now that you have a fresh conda environment, you can install new package like mpi4py your new environment

pip install mpi4py

When submitting a batch script, $PATH issues are known to occur if not submitting from a fresh login shell, which can result in the wrong environment being detected. To avoid this, you must use the --export=NONE flag during job submission and use unset SLURM_EXPORT_ENV in your job script (before calling srun), which ensures that no previously set environment variables are passed into the batch job, but makes sure that srun can still find your python path:

sbatch --export=NONE submit.sl

This means you will have to load your modules and activate your environment inside the batch script. An example batch script for is provided below:

#!/bin/bash
#SBATCH -p XXXXXXX
#SBATCH -N 2
#SBATCH -n 4
#SBATCH -c 1
#SBATCH -J test-job
#SBATCH --mem=8g
#SBATCH -t 5:00

# Only necessary if submitting like: sbatch --export=NONE ... (recommended)
# Do NOT include this line when submitting without --export=NONE
unset SLURM_EXPORT_ENV

cd $SLURM_SUBMIT_DIR
date

module load miniforge3/24.11.3-0
source activate your_env
python script.py