Container

Container Usage

Apptainer v1.4.5 is installed on Pathfinder. Apptainer can be used for both building and running containers on Pathfinder. The main user documentation on how to use Apptainer can be found here. This section of our documentation will only cover any additional info that you might need to make containers run correctly on Pathfinder.

Example Build and Run Workflow

This section will demonstrate how to build a container with openMPI and run a simple MPI example

Building and running a container with Rocky and OpenMPI

Create a directory named mpiexample in your directory and cd into it.

Create a file mpiexample.c with the following contents.

#include <stdio.h>
#include <mpi.h>

int main (int argc, char *argv[])
{
int rank, size;
MPI_Comm comm;

comm = MPI_COMM_WORLD;
MPI_Init (&argc, &argv);
MPI_Comm_rank (comm, &rank);
MPI_Comm_size (comm, &size);

printf("Hello from rank %d\n", rank);

MPI_Barrier(comm);
MPI_Finalize();
}

Create a file named rockyopenmpi.def with the following contents

Bootstrap: docker
From: rockylinux:9

%environment
# Point to OMPI binaries, libraries, man pages
export OMPI_DIR=/opt/ompi
export PATH="$OMPI_DIR/bin:$PATH"
export LD_LIBRARY_PATH="$OMPI_DIR/lib:$LD_LIBRARY_PATH"
export MANPATH="$OMPI_DIR/share/man:$MANPATH"

%files
mpiexample.c /opt/mpiexample.c

%post
echo "Installing required packages..."
dnf install -y wget sudo git gzip gcc gcc-g++ gcc-gfortran ucx-devel libibverbs librdmacm-devel pmix-devel
echo "Installing Open MPI"
export OMPI_DIR=/opt/ompi
export OMPI_VERSION=4.0.4
export OMPI_URL="https://download.open-mpi.org/release/open-mpi/v4.0/openmpi-$OMPI_VERSION.tar.gz"
mkdir -p /ompi
mkdir -p /opt
# Download
cd /ompi && wget -O openmpi-$OMPI_VERSION.tar.gz $OMPI_URL && tar -xzf openmpi-$OMPI_VERSION.tar.gz
# Compile and install
cd /ompi/openmpi-$OMPI_VERSION && ./configure --prefix=$OMPI_DIR  --with-ucx --with-pmix
make -j$(nproc) install

# Set env variables so we can compile our application
export PATH=$OMPI_DIR/bin:$PATH
export LD_LIBRARY_PATH=$OMPI_DIR/lib:$LD_LIBRARY_PATH
rm -rf /ompi

cd /opt && mpicc -o mpiexample mpiexample.c

Build the container image:
```
apptainer build rockyopenmpi.sif rockyopenmpi.def
```
- This builds a container with Rocky Linux 9 and installs OpenMPI 4.0.4. It copies the file mpiexample.c into the image in the /app directory in the image and compiles it to an executable named mpiexample in the /app directory.
- Apptainer builds the container image in the SIF file format. Unlike Podman, Apptainer gives you a single file for your image that you can later run as your container.

Running the container in a batch job

Create a file named submit.sl

#!/bin/bash
#SBATCH -t01:20:00
#SBATCH -N4
#SBATCH -J apptainer_test
#SBATCH -o %x_%j.out
#SBATCH -e %x_%j.out
#SBATCH -p XXXXXX

module load openmpi

export APPTAINERENV_LD_LIBRARY_PATH="$LD_LIBRARY_PATH"
export APPTAINER_BIND="/software/baseline/nsp,/opt/mellanox,/var/spool/slurm,/etc/slurm,$(pwd)"
export APPTAINER_CONTAINLIBS="/lib64/libpmi.so.0,/lib64/libpmi2.so.0,/usr/lib64/libslurm.so,/usr/lib64/slurm/libslurm_pmi.so,/usr/lib64/slurm/libslurmfull.so,/lib64/librdmacm.so.1,/lib64/libibverbs.so.1,/lib64/libucp.so.0,/lib64/libuct.so.0,/lib64/libucm.so.0,/lib64/libucs.so.0,/lib64/libpmi2.so.0,/lib64/libpmi.so.0,/usr/lib64/slurm/libslurm_pmi.so,/lib64/libnl-3.so.200,/lib64/libnl-route-3.so.200 "

srun  -N4  -n8 --tasks-per-node 2 apptainer exec --workdir `pwd` rockyopenmpi.sif  /opt/mpiexample

The export lines are required in order to bind the host’s MPI libraries into the container in order for MPI to work correctly.

Submit a batch job with sbatch submit.sl

The output of the job should look something like

...
<several INFO messages that can be ignored>
Hello from rank 7
Hello from rank 6
Hello from rank 2
Hello from rank 0
Hello from rank 1
Hello from rank 3
Hello from rank 5
Hello from rank 4

MPI performance with containers is slower than bare metal MPI. We are working on finding ways to improve performance.