LCM with Kokkos CUDA support on Shannon

Building Albany/LCM on Shannon with CUDA

The purpose of this guide is to try and explain and make sense of the build process for Albany/LCM for CUDA on the Shannon GPU cluster. The process is quite complicated and many modules are not updated to compile for GPUs.

Environment Setup

The first thing we will do is set up our environment. All of the following commands can easily be added to a shell script or just executed before compiling. In my scripts I sourced an sh file called "build_common.sh" that I used for environment settings.

First, we need to load the required modules:

module purge
module add cmake/2.8.11.2
module add openmpi/1.8.4/gnu/4.7.2/cuda/7.0.28
module add libmpc/1.0.1
module add nvcc-wrapper/gnu

Trilinos will need Cmake version 2.8.11 in order to compile. Additionally, we want to use OpenMPI here with GCC version 4.7.2. While Shannon has newer versions of GCC including 5.10, the version of CUDA installed on the cluster does not necessarily support those versions of GCC. MPC is a library required by Trilinos that is available on the cluster. Finally, nvcc-wrapper is a utility that wraps compiler calls to forward properly to either GCC or NVCC.

Next we set our install prefix variable:

mkdir -p install
PREFIX=$PWD/install

Next we need to export the compiler environment variables that are needed for building the various dependencies of Albany/LCM. First the compiler search paths:

export CPATH=$CPATH:$PREFIX/include
export LIBRARY_PATH=$LIBRARY_PATH:$PREFIX/lib

Now we need to set which compilers should be used. We want to compile everything with mpicc except for the CUDA code. To do this we will set the NVCC_WRAPPER_DEFAULT_COMPILER environment variable for the nvcc-wrapper script. This determines what compiler will be used for C and C++ files rather than Cuda files.

export CC=mpicc
export CXX=mpicxx
export FC=mpif90
export NVCC_WRAPPER_DEFAULT_COMPILER=mpicc

###Compiling Dependencies

The Trilinos packages we need for Albany require a few dependencies. This document will only show how to obtain the dependencies specifically required to build LCM.

Note: On Shannon you may not be able to use wget or git clone; if this is the case then you should download these packages and transfer them over using sftp or scp.

####Boost

Boost is used throughout Trilinos. It's mostly a header only library so very little will need to be compiled. I've tested this with the latest version of boost at the time of writing, 1.58.

wget http://downloads.sourceforge.net/project/boost/boost/1.58.0/boost_1_58_0.tar.bz2
tar -xjf boost_1_58_0.tar.bz2

Once you have it downloaded, cd into the directory and run bootstrap.sh with the following arguments:

./boostrap.sh --with-libraries=system,program_options --prefix=$PREFIX

This configures boost to only build the specified libraries. You can omit the argument if you want but you will then build the entirety of boost which takes a long time. --prefix sets the installation directory. So if you set $PREFIX to install/ earlier then when you install boost it will place the header files in install/include/, the library files in install/lib/, and so on.

Once configuring is done we can build boost and install it:

./b2 -j 8
./b2 install

This will run boost's bjam build system on 8 processors and install it to $PREFIX.

####zLib

Next we will obtain and build zLib:

wget http://zlib.net/zlib-1.2.8.tar.gz
tar -xzf zlib-1.2.8.tar.gz
pushd zlib-1.2.8
./configure --prefix=$PREFIX
make -j8
make install
popd

####HDF5

HDF5 is built similarly:

wget http://www.hdfgroup.org/ftp/HDF5/current/src/hdf5-1.8.15-patch1.tar.bz2
tar -xjf hdf5-1.8.15-patch1.tar.bz2
pushd hdf5-1.8.15-patch1
./configure --prefix=$PREFIX --enable-parallel
make -j8
make install
popd

####NetCDF

You know the drill...

wget ftp://ftp.unidata.ucar.edu/pub/netcdf/netcdf-4.3.3.1.tar.gz
tar -xzf netcdf-4.3.3.1.tar.gz
pushd netcdf-4.3.3.1
./configure --prefix=$PREFIX
make -j8
make install
popd

####BLAS/LAPACK

Shannon has the Intel MKL package installed, so you can use that for BLAS and LAPACK if you want. I'll show you how to build the default LAPACK implementation from source though. It won't be as fast or as optimized as MKL however.

wget http://www.netlib.org/lapack/lapack-3.5.0.tgz
tar -xzf lapack-3.5.0.tgz
pushd lapack-2.5.0
mkdir -p build
cd build
cmake -DCMAKE_INSTALL_PREFIX=$PREFIX ..
make -j8 install
popd

###Building Trilinos

We are now finally ready to build Trilinos.

First, clone the git repository for the latest version of Trilinos. In my script I also run git pull in order to make sure the current version is the latest.

git clone https://software.sandia.gov/trilinos/repositories/publicTrilinos
pushd publicTrilinos
mkdir -p build
cd build

Now we run cmake on Trilinos similarly to how we did it for LAPACK. Trilinos has a lot of CMake cache variables so it is recommended to run this with a shell script. I will be breaking up the cmake command into several parts. Each line of the command needs the continuation symbol \ at the end, so don't forget that.

Note: I'm separating out the command into multiple parts for readability. The entire cmake command should be run at once.

cmake \
-DTPL_ENABLE_MPI=ON \
-DTPL_ENABLE_CUDA=ON \
-DTPL_ENABLE_Thrust=On \
\

This part of the command enables specific third party libraries, MPI, CUDA, and Thrust that we want to build with. Thrust is necessary for Kokkos if CUDA is enabled. On Shannon, Thrust comes with CUDA so you don't need to install it.

-DTrilinos_ENABLE_ALL_PACKAGES=OFF \
-DKokkos_ENABLE_Cuda_UVM=ON \
-DTrilinos_ENABLE_EXPLICIT_INSTANTIATION=ON \
-DTrilinos_ENABLE_DEVELOPMENT_MODE=OFF \
-DTrilinos_ASSERT_MISSING_PACKAGES=OFF \ 
\

First, we should not enable all packages. This will dramatically reduce compile times. Furthermore, some Trilinos packages conflict or do not work at all so this will probably also break your build if you do not set it.

We also need to enable UVM for CUDA. Kokkos and some Trilinos libraries like Phalanx use UVM to assign data in Kokkos views.

In order to reduce memory use while compiling, avoiding annoying other people on the cluster, we enable explicit template instantiation.

Disabling Trilinos development mode will disable pedantic compiling and (I think) warnings as errors. Trilinos has a lot of warnings when building so I would not recommend enabling this flag since it may break your build.

Finally, it's necessary to disable asserting on missing packages. Trilinos contains references to packages that don't actually exist, so enabling this flag would break your build.

-DBoost_INCLUDE_DIRS=$PREFIX/include \
-DBoost_LIBRARY_DIRS=$PREFIX/lib \ 
-DBoostLib_INCLUDE_DIRS=$PREFIX/include \
-DBoostLib_LIBRARY_DIRS=$PREFIX/lib \
\

These flags just set the location of boost. Shannon already has boost installed but it may be the wrong version. I recommend installing the version of boost I specify in this guide.

-DTrilinos_ENABLE_DEBUG=OFF \
-DTrilinos_ENABLE_Piro=ON \
-DTrilinos_ENABLE_Phalanx=ON \
-DTrilinos_ENABLE_ThyraTpetraAdapters=On \ 
-DTrilinos_ENABLE_STKIO=On \
-DTrilinos_ENABLE_STKMesh=On \
-DTrilinos_ENABLE_STKClassic=Off \
-DTrilinos_ENABLE_STKDoc_tests=Off \
-DTrilinos_ENABLE_Teko=Off \ 
\

First, we want to disable debug mode. If you want to enable it to get asserts you can do that and then also set -DCMAKE_BUILD_TYPE=Debug.

Next we enable and disable specific libraries that Albany/LCM require. Piro, Phalanx, and ThyraTpetraAdapters are all required. The Albany makefile claims STKIO and STKMesh are not required but they actually are. Do not include all of STK. This will break your build. Additionally, explicitly disable STKClassic and STKDoc_tests as they will break your build. Finally, for some reason Teko doesn't compile on Shannon. Disable it.

-DCMAKE_INSTALL_PREFIX=$PREFIX \
..

#Done with CMake command
make -j8 install
popd

We finally set our install location and our source location. I like having my install location in one place; however, this can cause issues if you update Trilinos. If you are pulling from Trilinos make sure you delete the prefix folder. The reason for this is Trilinos copies all its headers in a flat directory. So Trilinos will use the headers in the prefix location when building instead of the headers in the source directory. If you update Trilinos it will use outdated headers and fail to compile unless you delete the install directory.

###Building Albany

Finally, we get to Albany.

Albany also uses CMake in a similar manner to Trilinos. First, we clone the repository:

git clone https://github.com/SNLComputation/Albany.git
pushd Albany
mkdir -p build
cd build

Now for the CMake command.

Note: I'm separating out the command into multiple parts again.

cmake \
-DTrilinos_DIR=$PREFIX \
-DALBANY_TRILINOS_DIR=$PREFIX \
-DCMAKE_CXX_FLAGS="-std=c++11" \
-DENABLE_DEBUGGING=OFF \
\

This sets the trilinos directory for Albany. Also it sets the CXX flags. You can add optimization flags and so on if you would like. However, you need to at least set something because if not Albany copies the Trilinos flags. This has some weird consequences such as NVCC complaining about duplicate flags and failing to compile your code. We also disable debugging but you can turn it on.

-DENABLE_LCM=ON \
-DENABLE_BGL=OFF \
-DENABLE_QCAD=OFF \
-DENABLE_LCM_SPECULATIVE=OFF \
-DENABLE_LAME=OFF \
-DENABLE_HYDRIDE=OFF \
-DENABLE_SCOREC=OFF \
-DENABLE_AMP=OFF \
-DENABLE_SEE=OFF \
-DENABLE_FELIX=OFF \
-DENABLE_MOR=OFF \
-DENABLE_ALBANY_CI=OFF \
-DENABLE_ASCR=OFF \
-DENABLE_SG_MP=OFF \
\

Next we enable LCM and disable other parts of Albany to speed up compile time. You can enable these if you want.

-ENABLE_DEMO_PDES=OFF \
-DENABLE_64BIT_INT=OFF \
-DENABLE_CHECK_FPE=ON \
..

#End of CMake command
make -j8
popd

Make sure you disable Demo PDES. There are some issues compiling those with CUDA on Shannon. I'm not exactly sure if 64 bit ints are used at all, so it may not be necessary to disable this. Finally, if you want floating point exception checking, set ENABLE_CHECK_FPE.

I hope this guide was helpful in getting LCM copmpiled with Kokkos support on Shannon.