Skip to content

Latest commit

 

History

History
 
 

ex11

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
code
code
 
 
README.md
README.md
 
 

README.md

title author
Exercise 11 - MPI on JASMIN
Fatima Chami

Exercise 11: MPI on JASMIN

Scenario

Parallel computing is the use of two or more processors to solve a large problem size. There are different forms of parallelism, Shared Memory parallelism (threading) e.g. OpenMP and Distributed Memory parallelism which uses the Message Passing Interface (MPI). This exercise will demonstrate how to compile and test a parallel MPI Fortran code on LOTUS.

The Fortran code generates two vectors X(n) and Y(n) of n=2**10 elements and then calculates a vector Z(n) as Z(i)= a * X(i) + Y(i). The code outputs the maximum value of the vector elements Z(i), maxval(abs(Z))

A very important part of this workflow is that there are 2 separate steps:

  1. Compilation of code:
    • On a single LOTUS host
    • Run interactively
  2. Execution of compiled code:
    • On the required number of LOTUS hosts
    • Run as batch job(s)

There are a limited number of licences available for compilers so please adhere to this 2-step approach.

Objectives

After completing this exercise you will:

  • know about the MPI implementation on JASMIN
  • learn about the Fortran/C compilers and how to use them
  • compile an MPI parallel code on a LOTUS compute node interactively
  • become aware of the special SLURM submission options to request resources for an MPI parallel job

JASMIN resources

  • Scientific analysis servers: sci[1-6,8].jasmin.ac.uk
  • Group workspace: /gws/pw/j05/workshop
  • LOTUS batch queues: 'workshop' (par-single or par-multi outside the event)
  • Fortran MPI source code is (available in the Github repository): /gws/pw/j05/workshop/exercises/ex11/code/axpyMPI.f90
  • Help documentation at https://help.jasmin.ac.uk

Local resources

  • SSH key & passphrase
  • Terminal application or NX client
  • A valid jasmin-login grant associated with your JASMIN account

Your task

This is the outline of what you need to do. The recommended way of doing each step is covered in the "Cheat Sheet" but you may wish to try solving it for yourself first.

  1. Login to a JASMIN scientific analysis server
    • Launch two terminal sessions
    • Access a JASMIN login server on each terminal (see exercise 01)
    • Choose a Sci server with the lowest load
    • Login to the chosen sci server on each terminal
    • Copy the Fortran source code from the exercise directory (shown in the JASMIN resources section) to your current working directory

    NOTE: One terminal will be used for compiling and testing codes on LOTUS while the second terminal will be used for submitting and monitoring batch jobs.

  2. Compile and test a Fortran code interactively on LOTUS
    • On terminal 1, invoke a pseudo-interactive session on LOTUS using the SLURM command srun with two CPU cores allocation: srun --ntasks=2 --partition=workshop --account=workshop --pty /bin/bash
    • What is the compute node allocated and what type of CPU model the node has?
    • On the LOTUS compute node, load the Intel compiler module module load intel/20.0.0 and the OpenMPI library module: module load eb/OpenMPI/intel/3.1.1 and check that the two modules are loaded.
    • Compile the Fortran code using the command mpif90 axpyMPI.f90 -o axpyMPI.exe
    • Execute the binary using 1 core and then using two cores: mpirun -np 2 axpyMPI.exe
    • What is the ouput?
    • On terminal 2, check the job ID associated to this pseudo-interactive session on LOTUS
    • Exit the interactive session on LOTUS exit. The Job should be cleared from SLURM
  3. Prepare a script to submit the parallel MPI code to SLURM
    • On terminal 1, launch a text editor to prepare the job script e.g. jobscriptMPI.sbatch to submit the binary MPI compiled earlier.
    • Specify the number of parallel MPI tasks
    • Submit the job e.g. job1 to SLURM scheduler and note the job ID sbatch axpyMPI.sbatch
    • On terminal 2, monitor the job state using SLURM command squeue
    • What is the name of the compute node the job run on? is it the same node type on which the code was compiled?
    • Check the resources used by the job memory,CPU using 'scontrol show job jobID'
    • Inspect the job output and error file
  4. Estimate and refine an MPI job requirements
    • Specify the memory required per core
    • Specify the node type
    • Define a distribution of cores across nodes e.g. job2 or on a single node e.g. job3
    • Submit the same job script file but pass the new memory and core distribution to the SLURM 'sbatch'
    • What is the job wait time?
    • What is the elapsed time for job2 and job3?
  5. Writing to the scratch area
    *
    *
    *

Questions to test yourself

All too easy? Here are some questions to test your knowledge and understanding. You might find the answers by exploring the JASMIN Documentation

  1. Is there a limit on the number of parallel tasks an MPI job can have?
  2. What is the MPI implementation supported on JASMIN?
  3. Is it possible to run an MPI binary that was compiled on a different system?
  4. What is the difference between an MPI code and a code that uses MPI IO only?
  5. How to find out about the MPI libray that the code was compiled against?
  6. What type of storage is suitable for pararallel MPI IO?
  7. Can I run a Python script in parallel using MPI4py?

Review / alternative approaches / best practice

By completing this exercise you will be able to compile and test a parallel MPI Fortran code interactively on LOTUS. You will be able to use the special submission flags to submit an MPI job to SLURM. You will be able to use compilers via the module environment. MPI message passing interface is a library to facilitate data sharing and communication between CPU cores -often called ranks- as each rank accesses its own data space.

  • LOTUS par-single and par-multi are dedicated queues for MPI and OpenMP parallel codes
  • Use mpirun to execute MPI parallel codes and to ensure that the MPI communications runs over the private MPI network.
  • The OpenMPI library is the only supported MPI library on LOTUS. OpenMPI v3.1.1 and v4.0.0 are provided which are fully MPI3 compliant.
  • MPI I/O features are fully supported only on the LOTUS /work/scratch-pw directory as this uses a Panasas fully parallel file system
  • Run the MPI code on two cores before scaling up to many cores
  • Keep your source code in your home directory which is backed up
  • There is a limited number of licences available for Intel compiler, so please do not submit many jobs to compile the same code.
  • Run the MPI code on the same CPU model used to compile the MPI source code.

Cheat Sheet

  • scontrol show job jobid check the Features=intel Features=ivybridge128G if a specific Intel model is explicitely specified 1. Login to a JASMIN scientific analysis server

  1. Login to a JASMIN scientific analysis server

    • Login to the chosen sci server on each terminal
    $ ssh -A [email protected]
[train049@sci3 ~]$
    • Copy the Fortran source code from the exercise directory (shown in the JASMIN resources section) to your current working directory
    cp /gws/pw/j05/workshop/exercises/ex11/code/axpyMPI.f90 .

    NOTE: One terminal will be used for compiling and testing codes on LOTUS while the second terminal will be used for submitting and monitoring batch jobs.

  2. Compile and test a Fortran code interactively on LOTUS

    • On terminal 1, invoke a pseudo-interactive session on LOTUS using the SLURM command srun with two CPU cores allocation
    $ srun --ntasks=2 --partition=workshop --account=workshop  --pty /bin/bash`
srun: job 64164115 queued and waiting for resources
srun: job 64164115 has been allocated resources
cpu-bind=MASK - host149, task  0  0 [11224]: mask 0x1 set
[train049@host149 ] $
    • What is the compute node allocated and what is its CPU model?
    @host149 ] $ scontrol show node host149 | grep Features
 AvailableFeatures=ivybridge128G,lotus241,lotus2,intel
ActiveFeatures=ivybridge128G,lotus241,lotus2,intel

    or

    @host149 ] $ cat /proc/cpuinfo
...
model name	: Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz
...

    CPU E5-2650 v2 corresponds to hostgroup ivybridge128G see the Table in this helppage https://help.jasmin.ac.uk/article/4932-lotus-cluster-specification

    • On the LOTUS compute node, load the Intel compiler module module load intel/20.0.0 and the OpenMPI library module: module load eb/OpenMPI/intel/3.1.1 and check that the two modules are loaded.
    @host149 ] $ module load intel/20.0.0
@host149 ] $ module load eb/OpenMPI/intel/3.1.1
@host149 ] $ module li
Currently Loaded Modulefiles:
 1) intel/cce/20.0.0         2) intel/fce/20.0.0         3) intel/20.0.0             4) eb/OpenMPI/intel/3.1.1
    • Compile the Fortran code using the command
    @host149 ] $ mpif90 axpyMPI.f90 -o axpyMPI.exe
    • Execute the binary using 1 core and then using two cores:
    @host149 ] $ mpirun -np 2 axpyMPI.exe
  1.07598934124890       0.000000000000000E+000
    • On terminal 2, check the job ID associated to this pseudo-interactive session on LOTUS and the number of cores allocated
     @sci3 ~ ]$squeue -u trai049
          JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
       64164115  workshop     bash   train049 R      24:17      1 host149
@sci3 ~ ]$ scontrol show job 64164115 
...
NumNodes=1 NumCPUs=2 NumTasks=2 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
...
    • Exit the interactive session on LOTUS exit. The Job should be cleared from SLURM
     @host149 ] $ exit

  3. Prepare a script to submit the parallel MPI code to SLURM

    • On terminal 1, launch a text editor to prepare the job script e.g. jobscriptMPI.sbatch to submit the binary MPI compiled earlier.
    • Specify the number of parallel MPI tasks
    • Submit the job e.g. job1 to SLURM scheduler and note the job ID sbatch axpyMPI.sbatch
    • On terminal 2, monitor the job state using SLURM command squeue
    • What is the name of the compute node the job run on? is it the same node type on which the code was compiled?
    • Check the resources used by the job memory,CPU using 'scontrol show job jobID'
    • Inspect the job output and error file

  4. Estimate and refine an MPI job requirements

    • Specify the memory required per core
    • Specify the node type
    • Define a distribution of cores across nodes e.g. job2 or on a single node e.g. job3
    • Submit the same job script file but pass the new memory and core distribution to the SLURM 'sbatch'
    • What is the job wait time?
    • What is the elapsed time for job2 and job3?

Answers to questions

  1. Is there a limit on the number of parallel tasks an MPI job can have?

The limit is 16 cores for the par-single queue and 256 cores for the par-multi queue. https://help.jasmin.ac.uk/article/4881-lotus-queues

  1. What is the MPI implementation supported on JASMIN?

The OpenMPI library is the only supported MPI library on LOTUS. OpenMPI v3.1.1 and v4.0.0 are provided which are fully MPI3 compliant:

eb/OpenMPI/gcc/3.1.1
eb/OpenMPI/gcc/4.0.0   
eb/OpenMPI/intel/3.1.1 
eb/OpenMPI/intel/4.1.0
  1. Is it possible to run an MPI binary that was compiled on a different system?

Recompilation is recommended

  1. What is the difference between an MPI code and a code that uses MPI IO only?

The writing and reading of file is done in parallel. Hence, the storage system has to support parallel IO write operation.

  1. How to find out about the MPI libray that the code was compiled against?

Use the Linux command ldd <name-of-executable>

  1. What type of storage is suitable for pararallel MPI IO?

MPI I/O features are fully supported only on the LOTUS /work/scratch-pw directory as this uses a Panasas fully parallel file system

  1. Can I run a Python script in parallel using MPI4py?

This needs rewriting/converting the Python serila code in parallel using the MPI4py library