OP2 on CX1

This page gives a detailed tutorial on how to build the OP2 libraries on the Imperial College CX1 compute cluster and run the provided example applications.

Note: for use PyOP2 you need to build the cpp-matrices branch of OP2-Common.

Building OP2 on cx1

CX1 runs Red Hat Enterprise 6, uses environment modules and PBS.

Environment

Create a file cx1.env in the root of your OP2-Common clone:

#!/bin/bash

module unload cuda
module unload hdf5
module unload intel-suite
module unload libscotch
module unload mpi
module unload parmetis
module unload cmake
module unload petsc

module load cmake/2.8.7
module load cuda/4.0.17
module load hdf5/1.8.5
module load intel-suite/11.1
module load libscotch/5.1.12b
module load mpi/intel-3.1
module load parmetis/3.1
module load petsc/3.1-p8-intel-11

export CUDA_TOOLKIT_ROOT_DIR=$CUDA_HOME
export PARMETIS_DIR=$PARMETIS_HOME
export SCOTCH_DIR=$LIBSCOTCH_HOME

export CC=mpicc
export CXX=mpicxx

unset LD_PRELOAD

Sourcing this file (. cx1.env) sets up your bash environment as required to build OP2.

Note: intel/11.1 is crucial when using PTScotch, it will not work with 10.1

Building the OP2 C library

Create the following script as cx1.sh in the op2/c directory of your OP2-Common clone:

#!/bin/bash

. ../../cx1.env

# FindMPI.cmake cannot properly detect MPI on CX1, hence we need to set _both_
# MPI_INCLUDE_PATH and MPI_LIBRARY to disable auto-detection
./cmake.local -DBUILD_SHARED_LIBS=OFF \
  -DMPI_C_INCLUDE_PATH=$MPI_INCLUDE \
  -DMPI_C_LIBRARIES=$MPI_HOME/lib/libmpi.a \
  -DMPI_CXX_INCLUDE_PATH=$MPI_INCLUDE \
  -DMPI_CXX_LIBRARIES=$MPI_HOME/lib/libmpi.a

Execute the script for a standard developer build of the OP2 C libraries in the current directory.

Detailed instructions for customising the build are available in the README.

Building the OP2 C applications

Create the following script as cx1.sh in the apps/c directory of your OP2-Common clone:

#!/bin/bash

. ../../cx1.env

./cmake.local

Execute the script for a standard developer build of the OP2 C applications.

Building the OP2 Fortran library/apps

Note: there is no CMake build available for Fortran yet

Create a file cx1.env in the the op2/fortran directory of your OP2-Common clone:

module unload intel-suite
module unload pgi
module unload cuda

module load intel-suite/11.1
module load pgi/12.1
module load cuda/4.0.17

export OP2=`pwd`/..
export OP2_COMPILER=pgi
export CUDA_INSTALL_PATH=$CUDA_HOME
export LD_PRELOAD=

Sourcing this file (. cx1.env) sets up your bash environment as required to build OP2. Run make to build the libraries.

Source the same file before building any of the OP2 Fortran apps.

Note: Due to license restrictions the PGI compiler is only available on login-0.