Customizing Your Environment in SHPC
The SHPC environment may be customized to suit your needs.
For introductory information about customizing your Linux environment, see here.
Project-Specific Environment Variables
Some projects have environment modules that will prepare the environment with the specific needs of the project. To list the available project modules, type
module avail. At the top of the output is a section titled
/software/tools/modules. The environment modules begin with
env/. To load one of these environments:
module load env/cades-bsd
New software stack
CADES is moving its software management to a Spack-based system on Open condos and has the new software stack installed for testing. The following is the instructions to access it. (Note: The following spack stack is not available on Moderate)
You can find spack tutorial sites here: https://spack.readthedocs.io/en/latest/index.html; https://spack-tutorial.readthedocs.io/en/latest/ The new spack environment template can be found here: https://github.com/olcf/spack-environments/blob/main/linux-centos7-broadwell/or-slurm/spack.yaml. This is not a working environment and will need to be changed to be used.
To use the newly installed software stack with LMOD, first we initialize LMOD to replace the environment module system, then set MODULEPATH to the new set of modules. To initialize LMOD:
To get the correct modules into the MODULEPATH:
A new set of modules should be shown when you enter (this list is subject to change at any time):
$ module avail ----- /software/cades-open/spack-envs/base/modules/site/Core ----- gcc/6.3.0 gcc/11.2.0 (D) intel/18.0.0 ----- /software/cades-open/modulefiles/core ----- python/3.7-anaconda3
The LMOD modulefiles are laid out in a heirarchical tree. First load a compiler, then those modules built with it will be shown.
You may need to use module spider
$ module load gcc $ module -t avail /software/cades-open/spack-envs/base/modules/spack/linux-centos7-x86_64/gcc/11.2.0: boost/ boost/1.77.0 cctools/ cctools/7.2.10-py3 fftw/ fftw/3.3.9-omp fftw/3.3.9 hdf5/ hdf5/1.10.7 mesa/ mesa/21.2.1 metis/ metis/5.1.0 nag/ nag/7.0 ... /software/cades-open/spack-envs/base/modules/site/Core: gcc/ gcc/6.3.0 gcc/11.2.0 intel/ intel/18.0.0
To use the MPI compiler wrappers, first load the appropriate compiler modulefile then load openmpi:
$ module load gcc $ module load openmpi /software/cades-open/spack-envs/base/modules/spack/linux-centos7-x86_64/openmpi/4.0.4-rwt2r5j/gcc/11.2.0: boost/ boost/1.77.0 fftw/ fftw/3.3.9-omp fftw/3.3.9 hdf5/ hdf5/1.10.7 hypre/ hypre/2.22.0 ncl/ ncl/6.6.2-py3 ncview/ ncview/2.1.8 netcdf-cxx4/ netcdf-cxx4/4.3.1 netcdf-fortran/ netcdf-fortran/4.5.3 netlib-scalapack/ netlib-scalapack/2.1.0 openfoam/ openfoam/2106 parallel-netcdf/ parallel-netcdf/1.12.2 plumed/ plumed/2.7.3 silo/ silo/4.10.2
NOTE! The openmpi build requires that --mpi=pmi2 be added to the srun command in the launch scripts, ie:
srun --mpi=pmi2 <your usual arguments>
New Python module on CADES
Python is heavily used on CADES. Some users require specific versions of Python or niche scientific packages, which may further depend on numerous other Python packages. Because of all the dependencies that some Python packages require, and all the types of data that exist, it can be quite troublesome to get different Python installations to “play nicely” with each-other, especially on an HPC system where the system environment is complicated. Conda, a package and virtual environment manager from the Anaconda distribution, helps alleviate these issues. Conda allows users to easily install different versions of binary software packages and any required libraries appropriate for their computing platform. To start using the new python-conda environment on CADES, all you need to do is load the module:
module load python/3.7-anaconda3
Loading the Python module will put you in a “base” pre-configured conda environment. This option is recommended for users who do not need custom environments, and only require packages that are already installed in the base environment. This option is also recommended for users that just need a Python interpreter or standard packages.
Users can also create their own custom conda environment after loading the Python module. This option is recommended for users that require a different version of Python than the default version available, or for users that want a personal environment to manage specialized packages.
module load python conda create -p /path/to/my_env python=3.7 source activate /path/to/my_env
Now that you have a fresh conda environment, you can install new package like mpi4py your new environment
pip install mpi4py
This means you will have to load your modules and activate your environment inside the batch script. An example batch script is provided below:
#!/bin/bash #SBATCH -A birthright #SBATCH -p burst #SBATCH -N 2 #SBATCH -n 4 #SBATCH -c 1 #SBATCH -J test-job #SBATCH --mem=0 #SBATCH -t 5:00 module load python/3.7-anaconda3 source activate your_env python script.py