Execute a Job: Makefile Example

The tutorial assumes you have already worked through the Execute a Job Tutorial. Therefore, the instructions here are abbreviated but will follow the same format so you may easily consult the extended tutorial.

On this page, we will use a Makefile to automate the compiling of the C, C++, and Fortran programs. In our PBS script, we will send a command to the Makefile which will compile codes prior to submitting the job to MPI.

Table of Contents

📝 Note: Do not execute jobs on the login nodes; only use the login nodes to access your compute nodes. Processor-intensive, memory-intensive, or otherwise disruptive processes running on login nodes will be killed without warning.

Step 1: Access Your Allocation

If you need to request an allocation, see instructions here.

  1. Open a Bash terminal (or PuTTY for Windows users).

  2. Execute ssh username@or-condo-login.ornl.gov.

  3. Replace "username" with your XCAMS or UCAMS ID.

  4. When prompted, enter your XCAMS or UCAMS password.

Step 2: Create a PBS Script

Example PBS Script

Here is an example PBS script for running a batch job on a SHPC Condo allocation.

#!/bin/bash

#PBS -N mpi_hello_world_make
#PBS -M your_email@ornl.gov
#PBS -l nodes=1:ppn=16
#PBS -l walltime=0:00:6:0
#PBS -W group_list=cades-birthright
#PBS -A birthright
#PBS -l qos=burst
#PBS -V

module purge
module load PE-gnu
module load python
module list
cd $PBS_O_WORKDIR
pwd

make all

echo "=================================="
echo "Output of the MPI and C program"
echo "=================================="

mpirun hello_world_c

echo "=================================="
echo "Output of the MPI and C++ program"
echo "==================================="

mpirun hello_world_cpp

echo "====================================="
echo "Output of the MPI and Fortran program"
echo "====================================="

mpirun hello_world_f

echo "====================================="
echo "Output of the MPI and Python program"
echo "====================================="

mpirun python hello_world.py

echo "============================="
echo "Successful!!!... End of File"
echo "============================="

PBS Procedure

  1. From the login node, change your working directory to the desired file system. We are going to use our Lustre allocation for this example. If Lustre storage is not available, you may complete this tutorial from within your home directory on NFS.
cd /lustre/or-scratch/cades-birthright/username

Replace "username" with your UCAMS/XCAMS user ID.

  1. Use Vi to create and edit your PBS script.
vi hello_world_make.pbs
  1. Create your PBS script within Vi or paste the contents of your PBS script into Vi.
  2. Save your file and return to the Bash shell.

Step 3: Create the Makefile

A makefile contains instructions for Make software to automate the build of programs and source codes.

Makefile Code

This file must be located in the same path as the other source programs.

CC = mpicc
CXX = mpic++
FC = mpifort

OPTFLAGS = -O3
CFLAGS = $(OPTFLAGS) -g
CXXFLAGS = $(OPTFLAGS) -g
FFLAGS = $(OPTFLAGS) -g

all:hello_world_c hello_world_cpp hello_world_f

hello_world_c: hello_world.c
        $(CC) $(CFLAGS) -o hello_world_c hello_world.c

hello_world_cpp: hello_world.cpp
        $(CXX) $(CXXFLAGS) -o hello_world_cpp hello_world.cpp

hello_world_f: hello_world.f90
        $(FC)  $(FFLAGS) -o hello_world_f hello_world.f90

clean:
        rm hello_world_*

📝 Note: Indentations in a Makefile must be a Tab and not spaces.

Makefile Procedure

  1. Ensure that you are still in your working directory (/lustre/or-scratch/cades-birthright/username) using pwd.
  2. Use Vi (vi) to create your Makefile within your working directory.
vi Makefile
  1. Paste the hello world Makefile code into Vi.
  2. Save your file and return to the Bash shell.

If you have been following along the CADES tutorials in order, you will already have compiled programs for C, C++, and Fortran. However, if you did not already have compiled codes, the Makefile we have created here would compile all of them when we run our PBS script. In the hello_world_make.pbs file, there is a line that says make all. This command will run the contents of our Makefile, which will compile the C, C++, and Fortran programs prior to running the job.

Step 4: Run the Job

  1. Before proceeding, ensure that you are still in your working directory (using pwd) and that you still have the PE-gnu and python/3.6.1 modules loaded (using module list).

  2. We need to be in the same path/directory as our PBS script and our Makefile. Use ls -al to confirm their presence.

  3. Use qsub to schedule your batch job in the queue.

qsub hello_world_make.pbs

This command will automatically queue your job using Torque and produce a six-digit job number (shown below).

143295.or-condo-pbs01

You can check the status of your job at any time with the checkjob command.

checkjob 143295

You can also stop your job at any time with the qdel command.

qdel 143295
  1. View your results.
    You can view the contents of these files using the more command followed by the file name.
more mpi_hello_world_make.o143295

Your output should look something like this (the output is truncated.):

Processor or-condo-c229.ornl.gov ID=9  Hello world
Processor or-condo-c229.ornl.gov ID=4  Hello world
Processor or-condo-c229.ornl.gov ID=0  Hello world
Processor or-condo-c229.ornl.gov ID=1  Hello world
Processor or-condo-c229.ornl.gov ID=3  Hello world
Processor or-condo-c229.ornl.gov ID=5  Hello world
Processor or-condo-c229.ornl.gov ID=2  Hello world
Processor or-condo-c229.ornl.gov ID=6  Hello world
Processor or-condo-c229.ornl.gov ID=7  Hello world
Processor or-condo-c229.ornl.gov ID=8  Hello world
.
.
.
  1. Download your results (using the scp command or an SFTP client) or move them to persistent storage. See our moving data section for help.

Additional Examples