Customizing Your Environment in SHPC

The SHPC environment may be customized to suit your needs.

For introductory information about customizing your Linux environment, see here.

Project-Specific Environment Variables

Some projects have environment modules that will prepare the environment with the specific needs of the project. To list the available project modules, type module avail. At the top of the output is a section titled /software/tools/modules. The environment modules begin with env/. To load one of these environments:

module load env/cades-bsd

New software stack

CADES is moving its software management to a Spack-based system on Open condos and has the new software stack installed for testing. The following is the instructions to access it. (Note: The following spack stack is not available on Moderate)

You can find spack tutorial sites here: https://spack.readthedocs.io/en/latest/index.html; https://spack-tutorial.readthedocs.io/en/latest/ The new spack environment template can be found here: https://github.com/olcf/spack-environments/blob/main/linux-centos7-broadwell/or-slurm/spack.yaml. This is not a working environment and will need to be changed to be used.

To use the newly installed software stack with LMOD, first we initialize LMOD to replace the environment module system, then set MODULEPATH to the new set of modules. To initialize LMOD:

source /software/cades-open/spack-envs/base/root/linux-centos7-x86_64/gcc-6.3.0/lmod-8.5.6-wdngv4jylfvg2j6jt7xrtugxggh5lpm5/lmod/lmod/init/bash

To get the correct modules into the MODULEPATH:

export MODULEPATH=/software/cades-open/spack-envs/base/modules/site/Core:/software/cades-open/modulefiles/core

A new set of modules should be shown when you enter (this list is subject to change at any time):

$ module avail

 ----- /software/cades-open/spack-envs/base/modules/site/Core -----
   gcc/6.3.0    gcc/11.2.0 (D)    intel/18.0.0

 ----- /software/cades-open/modulefiles/core -----
   python/3.7-anaconda3

The LMOD modulefiles are laid out in a heirarchical tree. First load a compiler, then those modules built with it will be shown. You may need to use module spider to find the necessary modules if you can not find an expected module. Load any of these compilers to access the modules built with them:

$ module load gcc
$ module -t avail

  /software/cades-open/spack-envs/base/modules/spack/linux-centos7-x86_64/gcc/11.2.0:
  boost/
  boost/1.77.0
  cctools/
  cctools/7.2.10-py3
  fftw/
  fftw/3.3.9-omp
  fftw/3.3.9
  hdf5/
  hdf5/1.10.7
  mesa/
  mesa/21.2.1
  metis/
  metis/5.1.0
  nag/
  nag/7.0

  ...

  /software/cades-open/spack-envs/base/modules/site/Core:
  gcc/
  gcc/6.3.0
  gcc/11.2.0
  intel/
  intel/18.0.0

To use the MPI compiler wrappers, first load the appropriate compiler modulefile then load openmpi:

$ module load gcc
$ module load openmpi

  /software/cades-open/spack-envs/base/modules/spack/linux-centos7-x86_64/openmpi/4.0.4-rwt2r5j/gcc/11.2.0:
  boost/
  boost/1.77.0
  fftw/
  fftw/3.3.9-omp
  fftw/3.3.9
  hdf5/
  hdf5/1.10.7
  hypre/
  hypre/2.22.0
  ncl/
  ncl/6.6.2-py3
  ncview/
  ncview/2.1.8
  netcdf-cxx4/
  netcdf-cxx4/4.3.1
  netcdf-fortran/
  netcdf-fortran/4.5.3
  netlib-scalapack/
  netlib-scalapack/2.1.0
  openfoam/
  openfoam/2106
  parallel-netcdf/
  parallel-netcdf/1.12.2
  plumed/
  plumed/2.7.3
  silo/
  silo/4.10.2

NOTE! The openmpi build requires that --mpi=pmi2 be added to the srun command in the launch scripts, ie:

srun --mpi=pmi2 <your usual arguments>

New Python module on CADES

Python is heavily used on CADES. Some users require specific versions of Python or niche scientific packages, which may further depend on numerous other Python packages. Because of all the dependencies that some Python packages require, and all the types of data that exist, it can be quite troublesome to get different Python installations to “play nicely” with each-other, especially on an HPC system where the system environment is complicated. Conda, a package and virtual environment manager from the Anaconda distribution, helps alleviate these issues. Conda allows users to easily install different versions of binary software packages and any required libraries appropriate for their computing platform. To start using the new python-conda environment on CADES, all you need to do is load the module:

module load python/3.7-anaconda3

Loading the Python module will put you in a “base” pre-configured conda environment. This option is recommended for users who do not need custom environments, and only require packages that are already installed in the base environment. This option is also recommended for users that just need a Python interpreter or standard packages.

Users can also create their own custom conda environment after loading the Python module. This option is recommended for users that require a different version of Python than the default version available, or for users that want a personal environment to manage specialized packages.

module load python
conda create -p /path/to/my_env python=3.7
source activate /path/to/my_env

Now that you have a fresh conda environment, you can install new package like mpi4py your new environment

pip install mpi4py

This means you will have to load your modules and activate your environment inside the batch script. An example batch script is provided below:

#!/bin/bash
#SBATCH -A birthright
#SBATCH -p burst
#SBATCH -N 2
#SBATCH -n 4
#SBATCH -c 1
#SBATCH -J test-job
#SBATCH --mem=0
#SBATCH -t 5:00

module load python/3.7-anaconda3
source activate your_env

python script.py