OP2 on CX1

This page gives a detailed tutorial on how to build the OP2 libraries on the Imperial College CX1 compute cluster and run the provided example applications.

Building OP2 on cx1

CX1 runs Red Hat Enterprise 6, uses environment modules and PBS.

Environment

Create a file cx1.env in the root of your OP2-Common clone:

#!/bin/bash

module unload cuda
module unload hdf5
module unload intel-suite
module unload libscotch
module unload mpi
module unload parmetis
module unload cmake

module load cmake/2.8.7
module load cuda/4.0.17
module load hdf5/1.8.5
module load intel-suite/11.1
module load libscotch/5.1.12b
module load mpi/intel-3.1
module load parmetis/3.1

export CUDA_TOOLKIT_ROOT_DIR=$CUDA_HOME
export PARMETIS_DIR=$PARMETIS_HOME
export SCOTCH_DIR=$LIBSCOTCH_HOME

export CC=mpicc
export CXX=mpicxx

unset LD_PRELOAD

Sourcing this file (. cx1.env) sets up your bash environment as required to build OP2.

Note: intel/11.1 is crucial when using PTScotch, it will not work with 10.1

Building the OP2 C library

Create the following script as cx1.sh in the op2/c directory of your OP2-Common clone:

#!/bin/bash

. ../../cx1.env

# FindMPI.cmake cannot properly detect MPI on CX1, hence we need to set _both_
# MPI_INCLUDE_PATH and MPI_LIBRARY to disable auto-detection
# Due to a bug in FindMPI.cmake we also need to set MPI_EXTRA_LIBRARY to empty
./cmake.local -DBUILD_SHARED_LIBS=OFF \
  -DMPI_INCLUDE_PATH=$MPI_INCLUDE \
  -DMPI_LIBRARY=$MPI_HOME/lib/libmpi.a \
  -DMPI_EXTRA_LIBRARY=''

Execute the script for a standard developer build of the OP2 C libraries in the current directory.

Building the OP2 C applications

Create the following script as cx1.sh in the apps/c directory of your OP2-Common clone:

#!/bin/bash

. ../../cx1.env

./cmake.local

Execute the script for a standard developer build of the OP2 C applications.