Skip to content

ALCF Vesta

Jump to bottom
Glen Hansen edited this page Jul 30, 2016 · 24 revisions

Instructions for 2016

My .soft file looks like this:

@default
+cmake
+bgqtoolchain-gcc472
+mpiwrapper-gcc

Note that the +bgqtoolchain-gcc472 line does not appear to set the toolchain, so I also placed the line:

soft add +bgqtoolchain-gcc472

in my .bash_profile

Building Trilinos

The Trilinos configure script used for the tutorials this year is committed to the Albany repository at:

doc/do-cmake-trilinos-vesta

Note that this build uses LAPACK 3.6.1. This version has deprecated the calls dggsvd and sggsvd. Please edit the Trilinos file packages/epetra/src/Epetra_LAPACK_wrappers.h and change the lines:

#define DGGSVD_F77  F77_BLAS_MANGLE(dggsvd3,DGGSVD3)
#define SGGSVD_F77  F77_BLAS_MANGLE(sggsvd3,SGGSVD3)

Building Albany

The Albany configure script used for the tutorials this year is committed to the Albany repository at:

doc/do-cmake-albany-vesta

Running the Albany regression test suite

  • Note that the above Albany configure script also sets up ctest to run the regression suite on BG/Q, employing the script /projects/FASTMath/ATPESC-2016/installs/albany/bin/run_Albany to dispatch each parallel test to the compute nodes while ctest itself runs on the frontend node. From the build directory on the frontend node, typing

      qsub -A ATPESC16_Instructors -n 4 -t 60 --mode script /projects/FASTMath/ATPESC-2016/installs/albany/bin/reserve.sh
  qstat

  • Get the blockid info from qstat, and now run ctest

      export MPIEXEC_BLOCKID=VST-02400-13511-32
  ctest

  • will run the regression suite in this manner. Note also that all the usual ctest options apply to this command.

Instructions for 2015

Add this to your .soft file:

+mpiwrapper-bgclang
+cmake-3.0.2

Configure Trilinos as follows. (The block starting with Trilinos_ENABLE_Zoltan is for another user of this Trilinos build and can be omitted in a build meant only for Albany.)

# July 2015.
#   Need cmake:
#     => Use this line in your .soft file:
#          +cmake-3.0.2
#   C++11 requirement in Trilinos, especially in Kokkos:
#     => Use bgclang.
#     => Use this line in your .soft file:
#          +mpiwrapper-bgclang
#   However, (i) bgclang has no Fortran equivalent, and (ii) XL's
# binaries appear not to be compatible with bgclang's:
#     => Use the CMAKE_Fortran_COMPILER flag to point to gcc's Fortran compiler.

# Install next to the build directory.
TRILINSTALLDIR=../install-atpesc

# These TPLs are already built for us.
NETCDFDIR=/projects/FASTMath/install/netcdf/4.3.0/linux-rhel_6-ppc64-gcc-4.4.6
HDF5DIR=/projects/FASTMath/install/hdf5/1.8.11/linux-rhel_6-ppc64-gcc-4.4.6
ZLIBDIR=/projects/FASTMath/install/zlib/1.2.8/linux-rhel_6-ppc64-gcc-4.4.6
BLASDIR=/projects/FASTMath/install/blas/19Apr11/linux-rhel_6-ppc64-gcc-4.4.6
LAPACKDIR=/projects/FASTMath/install/lapack/3.4.2/linux-rhel_6-ppc64-gcc-4.4.6
BOOSTDIR=/projects/FASTMath/install/boost/1.54.0/linux-rhel_6-ppc64-gcc-4.4.6
# These are not.
MYTPLDIR=/projects/FASTMath/ATPESC-2015/install/Trilinos/tpl

# Make Trilinos choose the .a rather than .so files.
BOOST_LIBS="$BOOSTDIR/lib/libboost_program_options.a;$BOOSTDIR/lib/libboost_system.a"

# netcdf and hdf5 ordering are wrong for static linking. Fix with the
# usual hack of appending these as extra link flags. List libz.a
# specifically so libz.so is not used.
EXTRA_LINK_FLAGS="${HDF5DIR}/lib/libhdf5_hl.a;${HDF5DIR}/lib/libhdf5.a;${ZLIBDIR}/lib/libz.a"

# Turn on for a 64-bit local ordinal.
HAVE_LL=OFF

rm -f CMakeCache.txt;

cmake \
    -D Trilinos_DISABLE_ENABLED_FORWARD_DEP_PACKAGES=ON \
    -D CMAKE_INSTALL_PREFIX:PATH=${TRILINSTALLDIR} \
    -D CMAKE_BUILD_TYPE:STRING="RELEASE" \
    -D TPL_ENABLE_MPI:BOOL=ON \
    -D CMAKE_VERBOSE_MAKEFILE:BOOL=OFF \
    -D BUILD_SHARED_LIBS:BOOL=OFF \
    -D Trilinos_ENABLE_SECONDARY_TESTED_CODE:BOOL=OFF \
    \
    -D Trilinos_ENABLE_ALL_PACKAGES:BOOL=OFF \
    -D Trilinos_WARNINGS_AS_ERRORS_FLAGS:STRING="" \
    -D Trilinos_ENABLE_Teuchos:BOOL=ON \
    -D Trilinos_ENABLE_Shards:BOOL=ON \
    -D Trilinos_ENABLE_Sacado:BOOL=ON \
    -D Trilinos_ENABLE_Epetra:BOOL=ON \
    -D Trilinos_ENABLE_EpetraExt:BOOL=ON \
    -D Trilinos_ENABLE_Ifpack:BOOL=ON \
    -D Trilinos_ENABLE_AztecOO:BOOL=ON \
    -D Trilinos_ENABLE_Amesos:BOOL=ON \
    -D Trilinos_ENABLE_Anasazi:BOOL=ON \
    -D Trilinos_ENABLE_Belos:BOOL=ON \
    -D Trilinos_ENABLE_ML:BOOL=ON \
    -D Trilinos_ENABLE_Phalanx:BOOL=ON \
    -D Trilinos_ENABLE_Intrepid:BOOL=ON \
    -D Trilinos_ENABLE_NOX:BOOL=ON \
    -D Trilinos_ENABLE_Stratimikos:BOOL=ON \
    -D Trilinos_ENABLE_Thyra:BOOL=ON \
    -D Trilinos_ENABLE_Rythmos:BOOL=ON \
    -D Trilinos_ENABLE_MOOCHO:BOOL=ON \
    -D Trilinos_ENABLE_Stokhos:BOOL=ON \
    -D Trilinos_ENABLE_Piro:BOOL=ON \
    -D Trilinos_ENABLE_Teko:BOOL=ON \
    -D Trilinos_ENABLE_Moertel:BOOL=ON \
    \
    -D Trilinos_ENABLE_STKIO:BOOL=ON \
    -D Trilinos_ENABLE_STKMesh:BOOL=ON \
    -D TPL_ENABLE_Boost:BOOL=ON \
    -D Boost_INCLUDE_DIRS:FILEPATH="$BOOSTDIR/include" \
    -D TPL_Boost_LIBRARIES=${BOOST_LIBS} \
    -D TPL_ENABLE_BoostLib:BOOL=ON \
    -D BoostLib_INCLUDE_DIRS:FILEPATH="$BOOSTDIR/include" \
    -D TPL_BoostLib_LIBRARIES=${BOOST_LIBS} \
    \
    -D Trilinos_ENABLE_SEACAS:BOOL=ON \
    -D TPL_ENABLE_X11:BOOL=OFF \
    -D TPL_ENABLE_Matio:BOOL=OFF \
    -D Trilinos_ENABLE_SEACASIoss:BOOL=ON \
    -D Trilinos_ENABLE_SEACASExodus:BOOL=ON \
    \
    -D TPL_ENABLE_Netcdf:BOOL=ON \
    -D Netcdf_INCLUDE_DIRS:PATH="$NETCDFDIR/include" \
    -D TPL_Netcdf_LIBRARIES="$NETCDFDIR/lib/libnetcdf.a" \
    -D TPL_ENABLE_HDF5:BOOL=ON \
    -D TPL_HDF5_LIBRARIES="$HDF5DIR/lib/libhdf5_hl.a;$HDF5DIR/lib/libhdf5.a;$ZLIBDIR/lib/libz.a" \
    -D HDF5_INCLUDE_DIRS:PATH="$HDF5DIR/include" \
    \
    -D Trilinos_ENABLE_Tpetra:BOOL=ON \
    -D Trilinos_ENABLE_Kokkos:BOOL=ON \
    -D Trilinos_ENABLE_Ifpack2:BOOL=ON \
    -D Trilinos_ENABLE_Amesos2:BOOL=ON \
    -D Trilinos_ENABLE_Zoltan2:BOOL=ON \
    -D Trilinos_ENABLE_MueLu:BOOL=ON \
    -D Amesos2_ENABLE_KLU2:BOOL=ON \
    -D Amesos2_ENABLE_Basker:BOOL=ON \
    \
    -D Trilinos_ENABLE_SCOREC:BOOL=ON \
    -D SCOREC_DISABLE_STRONG_WARNINGS:BOOL=ON \
    -D Trilinos_ENABLE_EXPORT_MAKEFILES:BOOL=OFF \
    -D Trilinos_ASSERT_MISSING_PACKAGES:BOOL=OFF \
    -D MDS_ID_TYPE:STRING="int" \
    \
    -D TPL_ENABLE_ParMETIS:STRING=ON \
    -D ParMETIS_INCLUDE_DIRS:PATH="${MYTPLDIR}/include" \
    -D ParMETIS_LIBRARY_DIRS:PATH="${MYTPLDIR}/lib" \
    -D TPL_ENABLE_METIS:STRING=ON \
    -D METIS_INCLUDE_DIRS:PATH="${MYTPLDIR}/include" \
    -D METIS_LIBRARY_DIRS:PATH="${MYTPLDIR}/lib" \
    -D Zoltan_ENABLE_ULLONG_IDS:BOOL=ON \
    -D Teuchos_ENABLE_LONG_LONG_INT:BOOL=$HAVE_LL \
    \
    -D Trilinos_ENABLE_EXPLICIT_INSTANTIATION:BOOL=ON \
    -D Tpetra_INST_FLOAT=OFF \
    -D Tpetra_INST_INT_INT=ON \
    -D Tpetra_INST_DOUBLE=ON \
    -D Tpetra_INST_COMPLEX_FLOAT=OFF \
    -D Tpetra_INST_COMPLEX_DOUBLE=OFF \
    -D Tpetra_INST_INT_LONG=OFF \
    -D Tpetra_INST_INT_UNSIGNED=OFF \
    -D Tpetra_INST_INT_LONG_LONG=$HAVE_LL \
    \
    -D Trilinos_ENABLE_Kokkos:BOOL=ON \
    -D Trilinos_ENABLE_KokkosCore:BOOL=ON \
    -D Phalanx_KOKKOS_DEVICE_TYPE:STRING="SERIAL" \
    -D Phalanx_INDEX_SIZE_TYPE:STRING="INT" \
    -D Phalanx_SHOW_DEPRECATED_WARNINGS:BOOL=OFF \
    -D Kokkos_ENABLE_Serial:BOOL=ON \
    -D Kokkos_ENABLE_OpenMP:BOOL=OFF \
    -D Kokkos_ENABLE_Pthread:BOOL=OFF \
    \
    -D Trilinos_ENABLE_Zoltan:BOOL=ON \
    -D Zoltan_ENABLE_TESTS:BOOL=ON \
    -D Zoltan_ENABLE_EXAMPLES:BOOL=ON \
    -D Zoltan_ENABLE_F90INTERFACE:BOOL=ON \
    -D Zoltan_ENABLE_ParMETIS:BOOL=ON \
    -D Zoltan_ENABLE_Scotch:BOOL=ON \
    -D Scotch_LIBRARY_DIRS:FILEPATH="${MYTPLDIR}/lib" \
    -D Scotch_INCLUDE_DIRS:FILEPATH="${MYTPLDIR}/include" \
    \
    -D CMAKE_CXX_FLAGS:STRING="-Wno-deprecated-declarations -Wno-sign-compare" \
    -D TPL_BLAS_LIBRARIES="$BLASDIR/lib/libblas.a" \
    -D TPL_LAPACK_LIBRARIES="$LAPACKDIR/lib/liblapack.a;$LAPACKDIR/lib/libtmglib.a" \
    -D CMAKE_Fortran_COMPILER="/soft/compilers/wrappers/gcc/mpif90" \
    -D Trilinos_EXTRA_LINK_FLAGS:STRING=${EXTRA_LINK_FLAGS} \
    -D F77_FUNC:STRING="(name,NAME) name" \
    -D F77_FUNC_:STRING="(name,NAME) name" \
    \
    ../

Instructions for 2014

What follows are instructions for the 2014 ATPESC build.

Set up your build environment

Please place the following lines in your ~/.soft file:

	@default
	+hpctoolkit
	+cmake
	+mpiwrapper-gcc

Install the required Third Party Libraries (TPLs) for Trilinos

These install directions assume that a base install directory of /home/ghansen/FASTMath/src/ is used

Building ZLib

  • Version 1.2.8

      cd /home/ghansen/FASTMath/src/
  wget http://zlib.net/zlib-1.2.8.tar.gz
  tar -xvf zlib-1.2.8.tar.gz
  cd zlib-1.2.8
  CC=mpicc CXX=mpicxx CFLAGS=-O3 CXXFLAGS=-O3 ./configure --64 --static --prefix=/home/ghansen/FASTMath
  make install

Building HDF5

  • Version 1.8.13

      cd /home/ghansen/FASTMath/src/
  wget http://www.hdfgroup.org/ftp/HDF5/current/src/hdf5-1.8.13.tar.gz
  tar -xvf hdf5-1.8.13.tar.gz
  cd hdf5-1.8.13
  ./configure CC=mpicc FC=mpif90 CXX=mpicxx CXXFLAGS="-fPIC -O3" CFLAGS="-fPIC -O3" FCFLAGS="-fPIC -O3" \
  --enable-parallel --enable-shared=no --with-zlib=/home/ghansen/FASTMath --prefix=/home/ghansen/FASTMath
  make -j 8
  make install

Building Netcdf

  • Version of the day from the git repo. Note that hdf5-1.8.13 made some changes that are not reflected in the latest release, so we are using the votd for now.

      cd /home/ghansen/FASTMath/src/
  git clone https://github.com/Unidata/netcdf-c.git
  cd netcdf-c

  • Edit the file include/netcdf.h. Make the following edits near line 228:

      #define NC_MAX_DIMS    65536    /* max dimensions per file */
  #define NC_MAX_ATTRS    8192    
  #define NC_MAX_VARS   524288    /* max variables per file */
  #define NC_MAX_NAME   256 
  #define NC_MAX_VAR_DIMS      8     /* max per variable dimensions */

  • Now, configure using CMake and build:

      mkdir build
  cd build
  cmake \
  -DCMAKE_C_COMPILER=mpicc \
  -DCMAKE_C_FLAGS="-I/home/ghansen/FASTMath/include -O3" \
  -DCMAKE_EXE_LINKER_FLAGS="-L/home/ghansen/FASTMath/lib -O3" \
  -DCMAKE_INSTALL_PREFIX=/home/ghansen/FASTMath \
  -DENABLE_DAP=OFF -DBUILD_SHARED_LIBS=OFF -DENABLE_FSYNC=OFF -DENABLE_CDMREMOTE=OFF \
  -DENABLE_DOXYGEN=OFF -DENABLE_NETCDF_4=ON -DCMAKE_BUILD_TYPE=RELEASE -DENABLE_EXAMPLES=OFF \
  -DENABLE_TESTS=OFF -DCMAKE_SKIP_INSTALL_RPATH=ON \
  -DZLIB_INCLUDE_DIRS=/home/ghansen/FASTMath/include \
  -DZLIB_LIBRARY=/home/ghansen/FASTMath/lib/libz.a \
  -DBUILD_UTILITIES=OFF \
  ..

  make -j 8 
  make install

Building boost

  • Version 1.55.0

      cd /home/ghansen/FASTMath/src/
  wget http://sourceforge.net/projects/boost/files/boost/1.55.0/boost_1_55_0.tar.gz/download
  tar -xvf boost_1_55_0.tar.gz
  cd boost_1_55_0

  • Build boost with the commands

      echo "using gcc : 4.4.7 : /bgsys/drivers/ppcfloor/gnu-linux/bin/powerpc64-bgq-linux-g++ ;" \
  >> ./tools/build/v2/user-config.jam
  echo "using mpi : /bgsys/drivers/ppcfloor/comm/bin/gcc/mpicxx ;" >> ./tools/build/v2/user-config.jam
  ./bootstrap.sh \
  --with-libraries=signals,regex,filesystem,system,mpi,serialization,thread,program_options,exception \
  --prefix=/home/ghansen/FASTMath
  ./b2 -j 8
  ./b2 -j 8 install

Building ParMetis

  • Version 4.0.3

      cd /home/ghansen/FASTMath/src/
  wget http://glaros.dtc.umn.edu/gkhome/fetch/sw/parmetis/parmetis-4.0.3.tar.gz
  tar -xvf parmetis-4.0.3.tar.gz
  cd parmetis-4.0.3

  • Edit metis.h to use an IDXTYPEWIDTH of 64

      cd metis/include
  vi metis.h
  #define IDXTYPEWIDTH 64

  • Run CMake to build parmetis

      cd /home/ghansen/FASTMath/src/parmetis-4.0.3
  cd build
  export metis=/home/ghansen/FASTMath/src/parmetis-4.0.3/metis

  cmake \
  -DCMAKE_INSTALL_PREFIX=/home/ghansen/FASTMath \
  -DMETIS_PATH=$metis \
  -DGKLIB_PATH=$metis/GKlib \
  -DCMAKE_C_COMPILER=mpicc \
  -DCMAKE_CXX_COMPILER=mpicxx \
  ..

  make -j 8
  make install

  cp libmetis/libmetis.a /home/ghansen/FASTMath/lib
  cp $metis/include/metis.h /home/ghansen/FASTMath/include

SuperLU

  • Version 4.3

      cd /home/ghansen/FASTMath/src
  wget http://crd-legacy.lbl.gov/~xiaoye/SuperLU/superlu_4.3.tar.gz
  tar -xvf superlu_4.3.tar.gz
  cd superlu_4.3

  mkdir /home/ghansen/FASTMath/SuperLU_4.3
  mkdir /home/ghansen/FASTMath/SuperLU_4.3/include
  mkdir /home/ghansen/FASTMath/SuperLU_4.3/lib

  • Edit make.inc for your machine and environment

      PLAT = _linux
  SuperLUroot = /home/ghansen/FASTMath/SuperLU_4.3
  SUPERLULIB      = $(SuperLUroot)/lib/libsuperlu_4.3.a
  TMGLIB          = libtmglib.a
  BLASDEF     = -DUSE_VENDOR_BLAS
  BLASLIB     = /gpfs/vesta-fs0/projects/FASTMath/ATPESC-2014/install/blas/19Apr11/linux-rhel_6-ppc64-gcc-4.4.6/lib/blas.a
  LIBS        = $(SUPERLULIB) $(BLASLIB)
  ARCH         = ar
  ARCHFLAGS    = cr
  RANLIB       = ranlib
  CC           = mpicc
  CFLAGS       = -DPRNTlevel=0 -O3
  NOOPTS       =
  FORTRAN      = mpif90
  FFLAGS       = -O3
  LOADER       = $(CC)
  LOADOPTS     =
  CDEFS        = -DNoChange
  MATLAB       =

  • Build SuperLU

      make -j 4
  cd SRC
  cp *.h /home/ghansen/FASTMath/SuperLU_4.3/include
  ar d home/ghansen/FASTMath/SuperLU_4.3/lib/libsuperlu_4.3.a dlamch.o slamch.o

  • The last command deletes the dlamch and slamch objects, which already exist in the LAPACK library on Vesta

Build Trilinos

  • Download Trilinos

      cd /home/ghansen/FASTMath/src
  git clone https://software.sandia.gov/trilinos/repositories/publicTrilinos

  • Download SCOREC

      cd /home/ghansen/FASTMath/src/publicTrilinos
  git clone https://github.com/SCOREC/core.git SCOREC

  • Configure the Trilinos build

      mkdir build
  cd build
  ./do-configure

  • Example do-configure script executed above

      #!/bin/sh
  #
  export TRILINOS_HOME=/home/ghansen/FASTMath/src/publicTrilinos
  BUILD_DIR=`pwd`
  INSTALL_DIR=/home/ghansen/FASTMath/Trilinos/MPI_REL
  BOOST_DIR=/home/ghansen/FASTMath
  MPI_BASE_DIR=/bgsys/drivers/V1R2M1/ppc64/comm
  NETCDF=/home/ghansen/FASTMath
  HDFDIR=/home/ghansen/FASTMath
  PARMETISDIR=/home/ghansen/FASTMath
  BLAS=/gpfs/vesta-fs0/projects/FASTMath/ATPESC-2014/install/blas/19Apr11/linux-rhel_6-ppc64-gcc-4.4.6
  LAPACK=/gpfs/vesta-fs0/projects/FASTMath/ATPESC-2014/install/lapack/3.5.0/linux-rhel_6-ppc64-gcc-4.4.6
  
  export BOOST_ROOT=$BOOST_DIR
  EXTRA_ARGS=$@
  
  cmake \
  -Wno-dev \
  -D Trilinos_EXTRA_REPOSITORIES:STRING=SCOREC \
  -D Trilinos_ENABLE_SCOREC:BOOL=ON \
  -D SCOREC_DISABLE_STRONG_WARNINGS:BOOL=ON \
  \
  -D Trilinos_CONFIGURE_OPTIONS_FILE:FILEPATH=$TRILINOS_HOME/sampleScripts/AlbanySettings.cmake \
  -D CMAKE_BUILD_TYPE:STRING=NONE \
  -D TPL_FIND_SHARED_LIBS:BOOL=OFF \
  \
  -D CMAKE_C_FLAGS:STRING="-O3 -ffast-math -funroll-loops -DCODE_MP -DDISABLE_MPIIO -DNO_FILE_LOCK -DTEMPLATE_FRIENDS_NOT_SUPPORTED -DMPICH_SKIP_MPICXX -DNDEBUG -DHAVE_CONFIG_H" \
  -D CMAKE_CXX_FLAGS:STRING="-O3 -ffast-math -funroll-loops -DCODE_MP -DDISABLE_MPIIO -DNO_FILE_LOCK -DTEMPLATE_FRIENDS_NOT_SUPPORTED -DMPICH_SKIP_MPICXX -DNDEBUG -DHAVE_CONFIG_H" \
  -D CMAKE_Fortran_FLAGS:STRING="-O3 -ffast-math -funroll-loops" \
  \
  -D TPL_ENABLE_MPI:BOOL=ON \
  -D MPI_BASE_DIR:PATH=$MPI_BASE_DIR \
  -D Trilinos_EXTRA_LINK_FLAGS:STRING="-ldl" \
  \
  -D SEACAS_ENABLE_SEACASSVDI:BOOL=OFF \
  -D Trilinos_ENABLE_SEACASFastq:BOOL=OFF \
  -D Trilinos_ENABLE_SEACASBlot:BOOL=OFF \
  -D Trilinos_ENABLE_SEACASPLT:BOOL=OFF \
  -D TPL_ENABLE_X11:BOOL=OFF \
  \
  -D Boost_INCLUDE_DIRS:PATH=$BOOST_DIR/include \
  -D Boost_LIBRARY_DIRS:PATH=$BOOST_DIR/lib \
  -D BoostLib_INCLUDE_DIRS:PATH=$BOOST_DIR/include \
  -D BoostLib_LIBRARY_DIRS:PATH=$BOOST_DIR/lib \
  -D BoostAlbLib_INCLUDE_DIRS:PATH=$BOOST_DIR/include \
  -D BoostAlbLib_LIBRARY_DIRS:PATH=$BOOST_DIR/lib \
  \
  -D TPL_ENABLE_Netcdf:BOOL=ON \
  -D Netcdf_INCLUDE_DIRS:PATH="${NETCDF}/include" \
  -D Netcdf_LIBRARY_DIRS:PATH="${NETCDF}/lib" \
  -D TPL_Netcdf_LIBRARIES:STRING="/home/ghansen/FASTMath/lib/libnetcdf.a;/home/ghansen/FASTMath/lib/libhdf5_hl.a;/home/ghansen/FASTMath/lib/libhdf5.a;/home/ghansen/FASTMath/lib/libz.a" \
  \
  -D TPL_ENABLE_HDF5:BOOL=ON \
  -D HDF5_INCLUDE_DIRS:PATH="${HDFDIR}/include" \
  -D HDF5_LIBRARY_DIRS:PATH="${HDFDIR}/lib" \
  \
  -D TPL_ENABLE_Zlib:STRING=ON \
  -D Zlib_INCLUDE_DIRS:PATH="${HDFDIR}/include" \
  -D Zlib_LIBRARY_DIRS:PATH="${HDFDIR}/lib" \
  \
  -D TPL_ENABLE_ParMETIS:STRING=ON \
  -D ParMETIS_INCLUDE_DIRS:PATH="${PARMETISDIR}/include" \
  -D ParMETIS_LIBRARY_DIRS:PATH="${PARMETISDIR}/lib" \
  \
  -D TPL_ENABLE_SuperLU:STRING=ON \
  -D SuperLU_INCLUDE_DIRS:PATH="/home/ghansen/FASTMath/SuperLU_4.3/include" \
  -D SuperLU_LIBRARY_DIRS:PATH="/home/ghansen/FASTMath/SuperLU_4.3/lib" \
  \
  -D TPL_ENABLE_BLAS:STRING=ON \
  -D TPL_BLAS_LIBRARIES:FILEPATH="$BLAS/lib/blas.a" \
  -D TPL_ENABLE_LAPACK:STRING=ON \
  -D LAPACK_LIBRARY_DIRS:FILEPATH="$LAPACK/lib" \
  \
  -D CMAKE_INSTALL_PREFIX:PATH=${INSTALL_DIR} \
  \
  $EXTRA_ARGS \
  ${TRILINOS_HOME}

  • Build Trilinos

      make -j 8
  make install

Build Albany

  • Clone the repo

      cd /home/ghansen/FASTMath/src
  git clone https://github.com/gahansen/Albany.git
  cd Albany
  git checkout -b tpetra origin/tpetra

  • Configure the Albany build

      mkdir build
  cd build
  ./do-configure

  • CMake configure script ./do-configure:

      #!/bin/sh
  #
  export ALB_ENABLE_SCOREC=ON
  export ALB_ENABLE_LCM=ON
  export ALB_ENABLE_MOR=OFF
  export ALB_ENABLE_SPECULATIVE=OFF
  export ALB_ENABLE_LAME=OFF
  export ALB_ENABLE_SG_MP=OFF
  export ALB_ENABLE_INSTALL=ON
  export TRILINOS_INSTALL_DIR=/home/ghansen/FASTMath/Trilinos/MPI_REL
  export ALBANY_INSTALL_DIR=/home/ghansen/FASTMath/Albany

  cmake \
  -D ALBANY_TRILINOS_DIR:FILEPATH="$TRILINOS_INSTALL_DIR" \
  -D ENABLE_LCM:BOOL=${ALB_ENABLE_LCM} \
  -D ENABLE_LCM_SPECULATIVE:BOOL=${ALB_ENABLE_SPECULATIVE} \
  -D ENABLE_SCOREC:BOOL=${ALB_ENABLE_SCOREC} \
  -D ENABLE_MOR:BOOL=${ALB_ENABLE_MOR} \
  -D ENABLE_SG_MP:BOOL=${ALB_ENABLE_SG_MP} \
  -D ENABLE_INSTALL:BOOL=${ALB_ENABLE_INSTALL} \
  -D CMAKE_INSTALL_PREFIX:FILEPATH=${ALBANY_INSTALL_DIR} \
  -D ALBANY_MPI_OPTIONS:BOOL=ON \
  -D ALBANY_MPI_EXEC:STRING="/home/ghansen/FASTMath/scripts/run_Albany" \
  -D ALBANY_MPI_EXEC_NUMPROCS_FLAG:STRING="-n" \
  -D ALBANY_MPI_EXEC_MAX_NUMPROCS:STRING="4" \
  -D ALBANY_MPI_LEADING_OPTIONS:STRING="" \
  \
  /home/ghansen/FASTMath/src/Albany

  • Build Albany

      make -j 8
  make install

Running the Albany regression test suite

  • Note that the above Albany configure script also sets up ctest to run the regression suite on BG/Q, employing the script /home/ghansen/FASTMath/scripts/run_Albany to dispatch each parallel test to the compute nodes while ctest itself runs on the frontend node. From the build directory on the frontend node, typing

      qsub -A FASTMath -n 4 -t 60 --mode script /home/ghansen/FASTMath/scripts/reserve.sh
  qstat

  • Get the blockid info from qstat, and now run ctest

      export MPIEXEC_BLOCKID=VST-20220-31331-32
  ctest

  • will run the regression suite in this manner. Note also that all the usual ctest options apply to this command.

Running a typical Albany analysis

Using Albany to perform an analysis is very similar to using any typical finite element analysis code. Two preparation steps are required: a) define the problem spatial domain and nature by meshing the parts and assemblies that participate in the analysis, and b) define the materials, properties, boundary and initial conditions and specify numerical solver parameters within the Albany input (driver) file.

  1. Given the mesh, one decomposes the mesh into a set of subdomains for parallel execution, with one subdomain per processing element (or MPI rank).
  1. One then constructs an xml input file for Albany that specifies how to perform the simulation on the supplied decomposed mesh files.

  • Reserve a run allocation, this example is for 128 processors for 30 minutes

      qsub -A FASTMath -n 128 -t 30 --mode script /home/ghansen/FASTMath/scripts/reserve.sh
  qstat

  • Get the blockid info from qstat and set the environment variable

      export MPIEXEC_BLOCKID=VST-20220-31331-32

  • Use the script /home/ghansen/FASTMath/scripts/run_Albany to dispatch the parallel job to the compute nodes.

      export PATH=/home/ghansen/FASTMath/scripts:/home/ghansen/FASTMath/bin
  run_Albany -np 128 AlbanyT input.xml

Docker Image

SPACK Package

ToDo List

Albany FAQs

Capabilities

Sub-Projects

Compiling

Nightly Tests

Code HowTo

User Group Meetings

PAALS Tutorials

Wiki HowTo

Visualization Utilities

Good Commits

Python and Albany

Clone this wiki locally