ALCF Vesta

Set up your build environment

Please place the following lines in your ~/.soft file:

	@default
	+hpctoolkit
	+cmake
	+mpiwrapper-gcc

Install the required Third Party Libraries (TPLs) for Trilinos

These install directions assume that a base install directory of /home/ghansen/FASTMath/src/ is used

Building ZLib

• Version 1.2.8

    cd /home/ghansen/FASTMath/src/
    wget http://zlib.net/zlib-1.2.8.tar.gz
    tar -xvf zlib-1.2.8.tar.gz
    cd zlib-1.2.8
    CC=mpicc CXX=mpicxx CFLAGS=-O3 CXXFLAGS=-O3 ./configure --64 --static --prefix=/home/ghansen/FASTMath
    make install
  

Building HDF5

• Version 1.8.13

    cd /home/ghansen/FASTMath/src/
    wget http://www.hdfgroup.org/ftp/HDF5/current/src/hdf5-1.8.13.tar.gz
    tar -xvf hdf5-1.8.13.tar.gz
    cd hdf5-1.8.13
    ./configure CC=mpicc FC=mpif90 CXX=mpicxx CXXFLAGS="-fPIC -O3" CFLAGS="-fPIC -O3" FCFLAGS="-fPIC -O3" \
    --enable-parallel --enable-shared=no --with-zlib=/home/ghansen/FASTMath --prefix=/home/ghansen/FASTMath
    make -j 8
    make install
  

Building Netcdf

• Version of the day from the git repo. Note that hdf5-1.8.13 made some changes that are not reflected in the latest release, so we are using the votd for now.

    cd /home/ghansen/FASTMath/src/
    git clone https://github.com/Unidata/netcdf-c.git
    cd netcdf-c
  

• Edit the file include/netcdf.h. Make the following edits near line 228:

    #define NC_MAX_DIMS    65536    /* max dimensions per file */
    #define NC_MAX_ATTRS    8192    
    #define NC_MAX_VARS   524288    /* max variables per file */
    #define NC_MAX_NAME   256 
    #define NC_MAX_VAR_DIMS      8     /* max per variable dimensions */
  

• Now, configure using CMake and build:

    mkdir build
    cd build
    cmake \
    -DCMAKE_C_COMPILER=mpicc \
    -DCMAKE_C_FLAGS="-I/home/ghansen/FASTMath/include -O3" \
    -DCMAKE_EXE_LINKER_FLAGS="-L/home/ghansen/FASTMath/lib -O3" \
    -DCMAKE_INSTALL_PREFIX=/home/ghansen/FASTMath \
    -DENABLE_DAP=OFF -DBUILD_SHARED_LIBS=OFF -DENABLE_FSYNC=OFF -DENABLE_CDMREMOTE=OFF \
    -DENABLE_DOXYGEN=OFF -DENABLE_NETCDF_4=ON -DCMAKE_BUILD_TYPE=RELEASE -DENABLE_EXAMPLES=OFF \
    -DENABLE_TESTS=OFF -DCMAKE_SKIP_INSTALL_RPATH=ON \
    -DZLIB_INCLUDE_DIRS=/home/ghansen/FASTMath/include \
    -DZLIB_LIBRARY=/home/ghansen/FASTMath/lib/libz.a \
    -DBUILD_UTILITIES=OFF \
    ..
  
    make -j 8 
    make install
  

Building boost

• Version 1.55.0

    cd /home/ghansen/FASTMath/src/
    wget http://sourceforge.net/projects/boost/files/boost/1.55.0/boost_1_55_0.tar.gz/download
    tar -xvf boost_1_55_0.tar.gz
    cd boost_1_55_0
  

• Build boost with the commands

    echo "using gcc : 4.4.7 : /bgsys/drivers/ppcfloor/gnu-linux/bin/powerpc64-bgq-linux-g++ ;" \
    >> ./tools/build/v2/user-config.jam
    echo "using mpi : /bgsys/drivers/ppcfloor/comm/bin/gcc/mpicxx ;" >> ./tools/build/v2/user-config.jam
    ./bootstrap.sh \
    --with-libraries=signals,regex,filesystem,system,mpi,serialization,thread,program_options,exception \
    --prefix=/home/ghansen/FASTMath
    ./b2 -j 8
    ./b2 -j 8 install
  

Building ParMetis

• Version 4.0.3

    cd /home/ghansen/FASTMath/src/
    wget http://glaros.dtc.umn.edu/gkhome/fetch/sw/parmetis/parmetis-4.0.3.tar.gz
    tar -xvf parmetis-4.0.3.tar.gz
    cd parmetis-4.0.3
  

• Edit metis.h to use an IDXTYPEWIDTH of 64

    cd metis/include
    vi metis.h
    #define IDXTYPEWIDTH 64
  

• Run CMake to build parmetis

    cd /home/ghansen/FASTMath/src/parmetis-4.0.3
    cd build
    export metis=/home/ghansen/FASTMath/src/parmetis-4.0.3/metis
  
    cmake \
    -DCMAKE_INSTALL_PREFIX=/home/ghansen/FASTMath \
    -DMETIS_PATH=$metis \
    -DGKLIB_PATH=$metis/GKlib \
    -DCMAKE_C_COMPILER=mpicc \
    -DCMAKE_CXX_COMPILER=mpicxx \
    ..
  
    make -j 8
    make install
  
    cp libmetis/libmetis.a /home/ghansen/FASTMath/lib
    cp $metis/include/metis.h /home/ghansen/FASTMath/include
  

SuperLU

• Version 4.3

    cd /home/ghansen/FASTMath/src
    wget http://crd-legacy.lbl.gov/~xiaoye/SuperLU/superlu_4.3.tar.gz
    tar -xvf superlu_4.3.tar.gz
    cd superlu_4.3
  
    mkdir /home/ghansen/FASTMath/SuperLU_4.3
    mkdir /home/ghansen/FASTMath/SuperLU_4.3/include
    mkdir /home/ghansen/FASTMath/SuperLU_4.3/lib
  

• Edit make.inc for your machine and environment

    PLAT = _linux
    SuperLUroot = /home/ghansen/FASTMath/SuperLU_4.3
    SUPERLULIB      = $(SuperLUroot)/lib/libsuperlu_4.3.a
    TMGLIB          = libtmglib.a
    BLASDEF     = -DUSE_VENDOR_BLAS
    BLASLIB     = /gpfs/vesta-fs0/projects/FASTMath/ATPESC-2014/install/blas/19Apr11/linux-rhel_6-ppc64-gcc-4.4.6/lib/blas.a
    LIBS        = $(SUPERLULIB) $(BLASLIB)
    ARCH         = ar
    ARCHFLAGS    = cr
    RANLIB       = ranlib
    CC           = mpicc
    CFLAGS       = -DPRNTlevel=0 -O3
    NOOPTS       =
    FORTRAN      = mpif90
    FFLAGS       = -O3
    LOADER       = $(CC)
    LOADOPTS     =
    CDEFS        = -DNoChange
    MATLAB       =
  

• Build SuperLU

    make -j 4
    cd SRC
    cp *.h /home/ghansen/FASTMath/SuperLU_4.3/include
    ar d home/ghansen/FASTMath/SuperLU_4.3/lib/libsuperlu_4.3.a dlamch.o slamch.o
  

• The last command deletes the dlamch and slamch objects, which already exist in the LAPACK library on Vesta

Build Trilinos

• Download Trilinos

    cd /home/ghansen/FASTMath/src
    git clone https://software.sandia.gov/trilinos/repositories/publicTrilinos
  

• Download SCOREC

    cd /home/ghansen/FASTMath/src/publicTrilinos
    git clone https://github.com/SCOREC/core.git SCOREC
  

• Configure the Trilinos build

    mkdir build
    cd build
    ./do-configure
  

• Example do-configure script executed above

    #!/bin/sh
    #
    export TRILINOS_HOME=/home/ghansen/FASTMath/src/publicTrilinos
    BUILD_DIR=`pwd`
    INSTALL_DIR=/home/ghansen/FASTMath/Trilinos/MPI_REL
    BOOST_DIR=/home/ghansen/FASTMath
    MPI_BASE_DIR=/bgsys/drivers/V1R2M1/ppc64/comm
    NETCDF=/home/ghansen/FASTMath
    HDFDIR=/home/ghansen/FASTMath
    PARMETISDIR=/home/ghansen/FASTMath
    BLAS=/gpfs/vesta-fs0/projects/FASTMath/ATPESC-2014/install/blas/19Apr11/linux-rhel_6-ppc64-gcc-4.4.6
    LAPACK=/gpfs/vesta-fs0/projects/FASTMath/ATPESC-2014/install/lapack/3.5.0/linux-rhel_6-ppc64-gcc-4.4.6
    
    export BOOST_ROOT=$BOOST_DIR
    EXTRA_ARGS=$@
    
    cmake \
    -D Trilinos_EXTRA_REPOSITORIES:STRING=SCOREC \
    -D Trilinos_CONFIGURE_OPTIONS_FILE:FILEPATH=$TRILINOS_HOME/sampleScripts/AlbanySettings.cmake \
    -D CMAKE_BUILD_TYPE:STRING=NONE \
    -D TPL_FIND_SHARED_LIBS:BOOL=OFF \
    \
    -D CMAKE_C_FLAGS:STRING="-O3 -ffast-math -funroll-loops -DCODE_MP -DDISABLE_MPIIO -DNO_FILE_LOCK -DTEMPLATE_FRIENDS_NOT_SUPPORTED -DMPICH_SKIP_MPICXX -DNDEBUG -DHAVE_CONFIG_H" \
    -D CMAKE_CXX_FLAGS:STRING="-O3 -ffast-math -funroll-loops -DCODE_MP -DDISABLE_MPIIO -DNO_FILE_LOCK -DTEMPLATE_FRIENDS_NOT_SUPPORTED -DMPICH_SKIP_MPICXX -DNDEBUG -DHAVE_CONFIG_H" \
    -D CMAKE_Fortran_FLAGS:STRING="-O3 -ffast-math -funroll-loops" \
    \
    -D TPL_ENABLE_MPI:BOOL=ON \
    -D MPI_BASE_DIR:PATH=$MPI_BASE_DIR \
    -D Trilinos_EXTRA_LINK_FLAGS:STRING="-ldl" \
    \
    -D CMAKE_VERBOSE_MAKEFILE:BOOL=OFF \
    -D Trilinos_VERBOSE_CONFIGURE:BOOL=OFF \
    \
    -D TPL_ENABLE_Matio:BOOL=OFF \
    -D SEACAS_ENABLE_SEACASSVDI:BOOL=OFF \
    -D Trilinos_ENABLE_SEACASFastq:BOOL=OFF \
    -D Trilinos_ENABLE_SEACASBlot:BOOL=OFF \
    -D Trilinos_ENABLE_SEACASPLT:BOOL=OFF \
    -D TPL_ENABLE_X11:BOOL=OFF \
    \
    -D Boost_INCLUDE_DIRS:PATH=$BOOST_DIR/include \
    -D Boost_LIBRARY_DIRS:PATH=$BOOST_DIR/lib \
    -D BoostAlbLib_INCLUDE_DIRS:PATH=$BOOST_DIR/include \
    -D BoostAlbLib_LIBRARY_DIRS:PATH=$BOOST_DIR/lib \
    \
    -D Netcdf_INCLUDE_DIRS:PATH="${NETCDF}/include" \
    -D Netcdf_LIBRARY_DIRS:PATH="${NETCDF}/lib" \
    -D TPL_Netcdf_LIBRARIES:STRING="/home/ghansen/FASTMath/lib/libnetcdf.a;/home/ghansen/FASTMath/lib/libhdf5_hl.a;/home/ghansen/FASTMath/lib/libhdf5.a;/home/ghansen/FASTMath/lib/libz.a" \
    -D TPL_ENABLE_HDF5:BOOL=ON \
    -D HDF5_INCLUDE_DIRS:PATH="${HDFDIR}/include" \
    -D HDF5_LIBRARY_DIRS:PATH="${HDFDIR}/lib" \
    \
    -D TPL_ENABLE_Zlib:STRING=ON \
    -D Zlib_INCLUDE_DIRS:PATH="${HDFDIR}/include" \
    -D Zlib_LIBRARY_DIRS:PATH="${HDFDIR}/lib" \
    \
    -D ParMETIS_INCLUDE_DIRS:PATH="${PARMETISDIR}/include" \
    -D ParMETIS_LIBRARY_DIRS:PATH="${PARMETISDIR}/lib" \
    \
    -D TPL_ENABLE_SuperLU:STRING=ON \
    -D SuperLU_INCLUDE_DIRS:PATH="/home/ghansen/FASTMath/SuperLU_4.3/include" \
    -D SuperLU_LIBRARY_DIRS:PATH="/home/ghansen/FASTMath/SuperLU_4.3/lib" \
    \
    -D TPL_ENABLE_BLAS:STRING=ON \
    -D TPL_BLAS_LIBRARIES:FILEPATH="$BLAS/lib/blas.a" \
    -D TPL_ENABLE_LAPACK:STRING=ON \
    -D LAPACK_LIBRARY_DIRS:FILEPATH="$LAPACK/lib" \
    \
    -D Zoltan_ENABLE_ULONG_IDS:BOOL=ON \
    \
    -D Trilinos_ENABLE_SCOREC:BOOL=ON \
    -D SCOREC_DISABLE_STRONG_WARNINGS:BOOL=ON \
    \
    -D Trilinos_ENABLE_ThyraTpetraAdapters:BOOL=ON \
    -D Trilinos_ENABLE_Ifpack2:BOOL=ON \
    -D Trilinos_ENABLE_Amesos2:BOOL=ON \
    -D Trilinos_ENABLE_MueLu:BOOL=ON \
    -D Teuchos_ENABLE_COMPLEX:BOOL=OFF \
    \
    -D CMAKE_INSTALL_PREFIX:PATH=${INSTALL_DIR} \
    \
    $EXTRA_ARGS \
    ${TRILINOS_HOME}
  

• Build Trilinos

    make -j 8
    make install
  

Build Albany

• Clone the repo

    cd /home/ghansen/FASTMath/src
    git clone https://github.com/gahansen/Albany.git
    cd Albany
    git checkout -b tpetra origin/tpetra
  

• Configure the Albany build

    mkdir build
    cd build
    ./do-configure
  

• CMake configure script ./do-configure:

    #!/bin/sh
    #
    export ALB_ENABLE_SCOREC=ON
    export ALB_ENABLE_LCM=ON
    export ALB_ENABLE_MOR=OFF
    export ALB_ENABLE_SPECULATIVE=OFF
    export ALB_ENABLE_LAME=OFF
    export ALB_ENABLE_SG_MP=OFF
    export ALB_ENABLE_INSTALL=ON
    export TRILINOS_INSTALL_DIR=/home/ghansen/FASTMath/Trilinos/MPI_REL
    export ALBANY_INSTALL_DIR=/home/ghansen/FASTMath/Albany
  
    cmake \
    -D ALBANY_TRILINOS_DIR:FILEPATH="$TRILINOS_INSTALL_DIR" \
    -D ENABLE_LCM:BOOL=${ALB_ENABLE_LCM} \
    -D ENABLE_LCM_SPECULATIVE:BOOL=${ALB_ENABLE_SPECULATIVE} \
    -D ENABLE_SCOREC:BOOL=${ALB_ENABLE_SCOREC} \
    -D ENABLE_MOR:BOOL=${ALB_ENABLE_MOR} \
    -D ENABLE_SG_MP:BOOL=${ALB_ENABLE_SG_MP} \
    -D ENABLE_INSTALL:BOOL=${ALB_ENABLE_INSTALL} \
    -D CMAKE_INSTALL_PREFIX:FILEPATH=${ALBANY_INSTALL_DIR} \
    -D ALBANY_MPI_OPTIONS:BOOL=ON \
    -D ALBANY_MPI_EXEC:STRING="/home/ghansen/FASTMath/scripts/run_Albany" \
    -D ALBANY_MPI_EXEC_NUMPROCS_FLAG:STRING="-n" \
    -D ALBANY_MPI_EXEC_MAX_NUMPROCS:STRING="4" \
    -D ALBANY_MPI_LEADING_OPTIONS:STRING="" \
    \
    /home/ghansen/FASTMath/src/Albany
  

• Build Albany

    make -j 8
    make install
  

Running the Albany regression test suite

• Note that the above Albany configure script also sets up ctest to run the regression suite on BG/Q, employing the script /home/ghansen/FASTMath/scripts/run_Albany to dispatch each parallel test to the compute nodes while ctest itself runs on the frontend node. From the build directory on the frontend node, typing

    qsub -A FASTMath -n 4 -t 60 --mode script /home/ghansen/FASTMath/scripts/reserve.sh
    qstat
  

• Get the blockid info from qstat, and now run ctest

    export MPIEXEC_BLOCKID=VST-20220-31331-32
    ctest
  

• will run the regression suite in this manner. Note also that all the usual ctest options apply to this command.