Running MPI program on Heracles

Home

Syntax
mpirun  --display-map -hostfile <hostfile-name> -np <# processes>  /path/to/myprogram arg1 arg2

if your program is in your home directory, use:

mpirun --display-map -hostfile <hostfile-name> -np <# processes>  ./myprogram arg1 arg2

Options:

--display-map: Display the process map just before launch.

-hostfile : specifies a host file that list the nodes and slots (cores) for running the program.

 Example of a host file:
$ cat hosts
node2 slots=4
node3 slots=4
node4 slots=4

Note1:  In the example above the number of cores for each node is at most 4 cores (slots), so you cannot run an application that requieres more then 12 cores/slots (3x4 slots = 12).

Note2:  Remember  that Heracles has 48 cores for each node, so you can set it to 48 slots for each node in your host file. Heracles also has 16 nodes (node2 to node17) availables for running processes.

-np: run n copies of the program on the given nodes, i.e., distribut the processes among nodes

Note 3: -np cannot be greater than the total of slots in your host file.

Note 4: Matrix size must be multiple of the number of processes in order to MPI distribute the the tasks equally.
Example:

mpirun --display-map --hostfile hosts -np 12 ./mat-mult 48

 Data for JOB [65417,1] offset 0

 ========================   JOB MAP   ========================

 Data for node: node2    Num slots: 4    Max slots: 0    Num procs: 4
     Process OMPI jobid: [65417,1] App: 0 Process rank: 0
     Process OMPI jobid: [65417,1] App: 0 Process rank: 1
     Process OMPI jobid: [65417,1] App: 0 Process rank: 2
     Process OMPI jobid: [65417,1] App: 0 Process rank: 3

 Data for node: node3    Num slots: 4    Max slots: 0    Num procs: 4
     Process OMPI jobid: [65417,1] App: 0 Process rank: 4
     Process OMPI jobid: [65417,1] App: 0 Process rank: 5
     Process OMPI jobid: [65417,1] App: 0 Process rank: 6
     Process OMPI jobid: [65417,1] App: 0 Process rank: 7

 Data for node: node4    Num slots: 4    Max slots: 0    Num procs: 4
     Process OMPI jobid: [65417,1] App: 0 Process rank: 8
     Process OMPI jobid: [65417,1] App: 0 Process rank: 9
     Process OMPI jobid: [65417,1] App: 0 Process rank: 10
     Process OMPI jobid: [65417,1] App: 0 Process rank: 11

 =============================================================
* The command will run mat-mult using matrix size 48x48 and it distributes the processing among 3 nodes (node2, node3 and node4) and will use 12 slots in total, according to the host file configuration.