Running programs on Heracles by using SLURM

SLURM

It is an open source, fault-tolerant, and highly scalable cluster management and job scheduling system for large and small Linux clusters.

Slurm User Guide

Download Slurm Job Scripts

Slurm job Sequential code

Slurm job openMP code

Slurm job MPI code

Slurm job MPI + OpenMP code

Slurm job for CUDA/GPU code

Run Sequential Code Using `slurm-seq.sh`

sbatch --jobname=<myjobname> slurm-seq.sh ./sequential_code <arg1 arg2 argn>

<myjobname> : Name of your chice to be the name of job which will be used as the name of the output file (myjobname_out.jobID) and error file (myjobname_err.jobID).

Run OpenMP Code Using `slurm-openmp.sh`

export OMP_PROC_BIND=<true, false, close, spread>

sbatch --job-name=<myjobname> --export=ALL --cpus-per-task=<num_threads> slurm-openmp.sh ./openmp_code <arg1 arg2 argn>

<myjobname> : Name of your chice to be the name of job which will be used as the name of the output file (myjobname_out.jobID) and error file (myjobname_err.jobID).

<num_threads> is a number of your choice in a range from 1 to 48. According to the Heracles architecture each node has 24 hard-cores with hyper-threads in a total of 48.

Run MPI Code Using `slurm-mpi.sh`

Set up Intel compiler environment:

source /opt/intel/oneapi/setvars.sh

sbatch --job-name=<myjobname> --nodes=<num_nodes> --ntasks=<num_tasks> slurm-mpi.sh ./mpi_code <arg1 arg2 argn>

<myjob-name> : Name of your chice to be the name of job which will be used as the name of the output file (myjobname_out.jobID) and error file (myjobname_err.jobID).

<num_nodes> Setup a total of N nodes for the job. MPI tasks will be distributed across these nodes

<num_tasks> Specifies the total of MPI tasks. These tasks will be distributed across the requested nodes. If you want to know home many tasks per node will be launched by slurm, just divide --ntasks by --nodes

Run MPI+openMP Code Using `slurm-mpi-omp.sh`

Set up Intel compiler environment:

source /opt/intel/oneapi/setvars.sh

export OMP_PROC_BIND=true or false or close or spread

sbatch  --export=ALL --job-name=<myjobname> --cpus-per-task=<num_threads> --nodes=<num_nodes> --ntasks=<num_tasks> slurm-mpi-omp.sh. /mpiomp_code <arg1 arg2 argn>

<myjobname> Name of your chice to be the name of job which will be used as the name of the output file (myjobname_out.jobID) and error file (myjobname_err.jobID).

<num_threads> is a number of your choice in a range from 1 to 48. According to the Heracles architecture each node has 24 hard-cores with hyper-threads in a total of 48.

<num_nodes> Setup a total of N nodes for the job. MPI tasks will be distributed across these nodes

Run CUDA/GPU Code Using `slurm-gpu.sh`

Submit the Job

sbatch --job-name=<myjobname> slurm-gpu.sh ./cuda_code <arg1 arg2 argn>

Profile CUDA code

sbatch --job-name=<myjobname> slurm-gpu.sh nsys profile --trace=cuda,nvtx,osrt ./cuda_code <arg1 arg2 argn>

sbatch --job-name=<myjobname> slurm-gpu.sh nsys nvprof --print-gpu-trace ./cuda_code

<myjobname> Name of your chice to be the name of job which will be used as the name of the output file (myjobname_out.jobID) and error file (myjobname_err.jobID).

Monitor Slurm jobs and check the output and error files

Check Your Job on the Queue

The job may not be displayed in the queue because it runs very fast. To check the queue, use:

squeue

Check the Error File

To check the error file, use the following command (replace <jobID> with your actual job ID):

cat codename_err.<jobID>

Check the Output File

To check the output file, use the following command (replace <jobID> with your actual job ID):

cat codename_out.<jobID>

Usefull commands for slurm

sbatch - submit jobs
sinfo - reports the state of partitions and nodes managed by Slurm
squeue - reports the state of jobs or job steps
scancel - is used to cancel a pending or running job or job step

scancel -u username # cancel all jobs of username
slurmd -C print the node's current configuration
sinfo
scontrol show jobid -dd
scontrol show node node18
squeue : list all submited jobs