SHPC Condo Slurm Resource Queues

This page describes Slurm resources queues in the CADES SHPC Condo environment. The hardware in each queue, queue access policies, and batch directives required to access each queue are described. If your access group is not listed here, please check the Moab resources queues section.

The following tables below list the technical specifications of each resource queue available in the CADES SHPC Condos. For more information on submitting jobs to a resource queue, view the Execute a Slurm Job page. The SBATCH directives required to submit to each queue are listed in the SBATCH directive section near the bottom of the page.

Open Slurm Condo Queues

Name	# Nodes	Cores	Micro arch.	RAM	Max Walltime (DD-HH:MM:SS)	Local Scratch	GPU
testing	2	16	Broadwell	31G	4:00:00	160G	N/A
batch	169, 44	32, 36	Haswell, Broadwell	125G	14-00:00:00	250G, 2T	N/A
ceb_high_mem	4	40	Cascade Lake	375G	14-00:00:00	1T	N/A
skylake	77	36	Skylake	187G	14-00:00:00	2T	N/A
high_mem	98	36	Broadwell	250G	14-00:00:00	2T	N/A
high_mem_cd	26	36	Skylake	375G	14-00:00:00	2T	N/A
high_mem_d	2	32	Haswell	500G	14-00:00:00	2T	N/A
gpu	12	32	Haswell	250G	14-00:00:00	250G	2x K80 (GK210)
gpu_p100	8	36	Broadwell	500G	14-00:00:00	2T	2x P100 (GP100)
eere_gpu	5	128	Zen 2	500G	14-00:00:00	2T	2x V100 (GV100)
pss	15	128	Zen 2	500G	14-00:00:00	1.5T	N/A
fossil	9	128	Zen 2	500G	14-00:00:00	1.5T	N/A
amd_128	8	128	Zen 2	1TB	14-00:00:00	1.8T	N/A

Burst Queues

Name	Max Walltime (DD-HH:MM:SS)	# Nodes	Notes
burst	02-00:00:00	543	Generally available queue which contains all nodes except for the testing queue.
estd_burst	14-00:00:00	69	Queue available to estd users which contains eere_gpu and pss, queue nodes.
btsd_burst	14-00:00:00	24	Queue available to btsd users which contains pss queue nodes.

Nodes by Group

Group	Nodes
amo	eere_gpu:5,
beto	eere_gpu:5,
birthright	gpu:10, high_mem_cd:26
bsd	batch:25, high_mem:12, high_mem_cd:8, gpu:3
ccsd	batch:25, high_mem:20, gpu_p100:8
ccsi	batch:30
ceb	ceb_high_mem:4
chem	high_mem_cd:6, amd_128:8
cnms	batch:38, high_mem:32
mstd	high_mem:2
ntrc	high_mem:32, pss:15
nucphys	high_mem_d:2
sns	batch:50,skylake:5
theory	batch:30,skylake:32
qmc	skylake:40
fe	fossil:9

📝 Note: Jobs run in the burst queue can be preempted at any time.

To set preempted jobs to restart automatically add the following to your batch script:

#SBATCH --requeue 

Queue Access and Policy

Groups which have purchased nodes in a resource queue have priority access to those resources when submitting jobs. Groups which have not purchase resources in a queue can still submit jobs to the queue using the burst directives, but their jobs will be preempted and restarted if a user with priority submits a job that needs the resources. This is why we note that all jobs run in the burst queue can be preempted.

Open Condo Resource queue group access quick reference:

Purchased Queue Resources	Can Submit Jobs	Job Flags
Yes	Yes	Preemptor
No	Yes	Preemptee, Restartable

To submit a job with priority access to your group resources, you need to specify the queue that those resources are in in your job script. The queues that each group has priority access to are listed in the 'Nodes by Group' tables below. You can also determine which queues you have priority access to with the sinfo command.

To submit a job to resources your group does not have priority access to, you need to specify a burst queue. The burst queues allow you to schedule preemptable jobs on any free resources in the cluster. Details on the burst queues are shown in the queues table above.

Node Sharing

Open Condo users do not have exclusive access to their scheduled nodes unless they have requested all of the resources on each node explicitly. In other words, users can share nodes.

Job Priority

• Small jobs have higher priority than large jobs.
• A fairshare policy increases priority for users who have not run jobs recently.
• Jobs gain priority while waiting in the queue at a rate of one point per minute.
• A maximum of 2 jobs per user gain age based priority in the queue at a time.

Moderate Slurm Condo Queues

Name	# Nodes	Cores	Micro arch.	RAM	Max Walltime (DD-HH:MM:SS)	Local Scratch	GPU	GPU Details
testing	6	32	Sandy Bridge	62G	4:00:00	1T	N/A	N/A
ridge	40	128	Zen 2	500G		2T	N/A
batch	155	32	Haswell	125G	2-00:00:00	250G	N/A	N/A
dell_gpu	9, 2	36, 128	Broadwell, Zen 2	250G, 1T	2-00:00:00	N/A	8x K80 (GK210), 2x V100 (GV100)	12G GDDR5 Kepler, 16G HBM2 Volta

Nodes by Group

Group	Nodes
nsed	batch:155
nset	dell_gpu:11
amo	ridge:20
beto	ridge:20

Queue Access and Policy

• Groups which have purchased nodes in a resource queue have access to those resources when submitting jobs.
• The Moderate condo does not have a burst queue.
• Once a node is scheduled by a user, that user has exclusive access to the node.

Job Priority

• Large jobs have higher priority than small jobs.
• A fairshare policy increases priority for users who have not run jobs recently.
• Jobs gain priority while waiting in the queue at a rate of one point per minute.
• A maximum of 2 jobs per user gain age based priority in the queue at a time.

NSED Specific:

• Maximum of 2 running jobs at a time.
• Jobs with a reservation can be released from this limit to get full use of the reserved nodes.

Resource Queue SBATCH Directives

The SBATCH directives required to submit jobs to each resource queue are listed below organized by group. Each group has two lists of directives:

• Standard SBATCH directives
  • These directives are used to submit non-burst jobs.
• Burst SBATCH directives
  • These directives are used to submit burst jobs.

Groups that are still using the Moab/Torque scheduler have their PBS directives listed on the Moab Resource Queues page.

📝 Note: This syntax indicates that you should pick one of the options in brackets. Lines without brackets can be copied without any changes. * [ option_a | option_b | ... ]

All SBATCH Directives by Group:

• CADES Birthright
• Advanced Manufacturing Office (AMO)
• Atmospheric Radiation Measurement (ARM)
• Bioenergy Technologies Office (BETO)
• Biosciences Division (BSD)
• Computing and Computational Sciences Directorate (CCSD)
• Climate Change Science Institute (CCSI)
• Collective Electronic Behaviour (CEB)
• Chemical Sciences Division (CSD)
• Center for Nanophase Materials Sciences (CNMS)
• Materials Science and Technology Division (MSTD)
• National Transportation Research Center (NTRC)
• Nuclear Physics (nucphys)
• Spallation Neutron Source (SNS)
• MSTD First-Principles-Based Materials Theory (MSTD Theory)
• MSTD Quantum Monte Carlo (MSTD QMC)
• Nuclear Science and Engineering Directorate (NSED)
• National Security Emerging Technologies (NSET)

📝 Note: If you do not see your group listed, please contact the CADES team and include: * Help with SHPC Condo Registration * UCAMS ID or XCAMS ID, contact information, reason for requesting an SHPC Condo allocation, and the name of your directorate and division.

CADES Birthright

Standard SBATCH directives:

#SBATCH -A birthright
#SBATCH -p [gpu|high_mem_cd]

Burst SBATCH directives:

#SBATCH -A birthright
#SBATCH -p burst

Advanced Manufacturing Office (AMO)

Standard SBATCH directives:

#SBATCH -A amo
#SBATCH -p [eere_gpu|ridge]

Burst SBATCH directives:

#SBATCH -A amo
#SBATCH -p [burst|ridge_burst]

Note: The Ridge queue has been moved from open to moderate

Atmospheric Radiation Measurement (ARM)

Standard SBATCH directives:

#SBATCH -A arm
#SBATCH -p testing 

Burst SBATCH directives:

#SBATCH -A arm
#SBATCH -p burst

Bioenergy Technologies Office (BETO)

Standard SBATCH directives:

#SBATCH -A beto
#SBATCH -p [eere_gpu|ridge]

Burst SBATCH directives:

#SBATCH -A beto
#SBATCH -p [burst|ridge_burst]

Note: The Ridge queue has been moved from open to moderate

Biosciences Division (BSD)

Standard SBATCH directives:

#SBATCH -A bsd
#SBATCH -p [batch|high_mem|high_mem_cd|gpu]

Burst SBATCH directives:

#SBATCH -A bsd
#SBATCH -p burst

Computing and Computational Sciences Directorate (CCSD)

Standard SBATCH directives:

#SBATCH -A ccsd
#SBATCH -p [batch|high_mem|gpu_p100]

Burst SBATCH directives:

#SBATCH -A ccsd
#SBATCH -p burst

Climate Change Science Institute (CCSI)

Standard SBATCH directives:

#SBATCH -A ccsi
#SBATCH -p batch

Burst SBATCH directives:

#SBATCH -A ccsi
#SBATCH -p burst

Collective Electronic Behaviour (CEB)

Standard SBATCH directives:

#SBATCH -A ceb
#SBATCH -p ceb_high_mem

Burst SBATCH directives:

#SBATCH -A ceb
#SBATCH -p burst

Chemical Sciences Division (CSD)

Standard SBATCH directives:

#SBATCH -A chem
#SBATCH -p high_mem_cd

Burst SBATCH directives:

#SBATCH -A chem
#SBATCH -p burst

Center for Nanophase Materials Sciences (CNMS)

Standard SBATCH directives:

#SBATCH -A cnms
#SBATCH -p [batch|high_mem]

Burst SBATCH directives:

#SBATCH -A cnms
#SBATCH -p burst

Materials Science and Technology Division (MSTD)

Standard SBATCH directives:

#SBATCH -A mstd
#SBATCH -p high_mem

Burst SBATCH directives:

#SBATCH -A mstd
#SBATCH -p burst

National Transportation Research Center (NTRC)

Standard SBATCH directives:

#SBATCH -A ntrc
#SBATCH -p high_mem

Standard SBATCH directives:

#SBATCH -A ntrc
#SBATCH -p pss

Standard SBATCH directives:

#SBATCH -A fe
#SBATCH -p fossil

Burst SBATCH directives:

#SBATCH -A ntrc
#SBATCH -p burst

Nuclear Physics (nucphys)

Standard SBATCH directives:

#SBATCH -A nucphys
#SBATCH -p high_mem_d

Burst SBATCH directives:

#SBATCH -A nucphys
#SBATCH -p burst

Spallation Neutron Source (SNS)

Standard SBATCH directives:

#SBATCH -A sns
#SBATCH -p [batch|skylake]

Burst SBATCH directives:

#SBATCH -A sns
#SBATCH -p burst

MSTD First-Principles-Based Materials Theory (MSTD Theory)

Standard SBATCH directives:

#SBATCH -A theory
#SBATCH -p [batch|skylake]

Burst SBATCH directives:

#SBATCH -A theory
#SBATCH -p burst

MSTD Quantum Monte Carlo (MSTD QMC)

Standard SBATCH directives:

#SBATCH -A qmc
#SBATCH -p skylake

Burst SBATCH directives:

#SBATCH -A qmc
#SBATCH -p burst

Nuclear Science and Engineering Directorate (NSED)

Standard SBATCH directives:

#SBATCH -A nsed
#SBATCH -p batch

National Security Emerging Technologies (NSET)

Standard SBATCH directives:

#SBATCH -A nset
#SBATCH -p dell_gpu