Skip to content

Compton

Jump to bottom
Irina edited this page Jan 20, 2016 · 7 revisions

Building instructions for Compton (tpetra_kokkos branch)

Getting Albany repository

In purpose to clone Albnay form the github you need to set up proxy:

 git config --global http.proxy bc-proxy-5.sandia.gov:80

Then you run:

 git clone https://github.com/gahansen/Albany.git

Once you have cloned Albany, checkout and switch to tpetra_kokkos:

 git checkout -b tpetra origin/tpetra

Building for Sandy Bridge

  1. Load modules for SB:

    module load intelmpi/intel64/4.1.3.045

    module load intel/intel/compiler/15.2.164

    module load cmake

  2. Build Trilinos (panzer_kokkos branch) with next configure script:

    `BOOSTDIR=/home/ipdemes/install/boost_1_54`

`NETCDF=/home/ipdemes/install/netcdf`

`HDFDIR=/home/ipdemes/install/hdf5`

`EXTRA_ARGS=$@`

`cmake \`

 `-D Trilinos_DISABLE_ENABLED_FORWARD_DEP_PACKAGES=ON \`
 `-D CMAKE_INSTALL_PREFIX:PATH="path_to_install_dir"  \`
 `-D CMAKE_BUILD_TYPE:STRING="NONE"  \`
 `-D Trilinos_ENABLE_ALL_PACKAGES:BOOL=OFF \`
 `-D Trilinos_WARNINGS_AS_ERRORS_FLAGS:STRING="" \`
 `-D Trilinos_ENABLE_Teuchos:BOOL=ON \`
 `-D Trilinos_ENABLE_Shards:BOOL=ON \`
 `-D TPL_ENABLE_GLM=OFF \`
 `-D Trilinos_ENABLE_Sacado:BOOL=ON \`
 `-D Trilinos_ENABLE_Epetra:BOOL=ON \`
 `-D Trilinos_ENABLE_Tpetra:BOOL=ON \`
 `-D Tpetra_ENABLE_Kokkos_Refactor:BOOL=ON \`
 `-D Trilinos_ENABLE_EpetraExt:BOOL=ON \`
 `-D Trilinos_ENABLE_Ifpack:BOOL=ON \`
 `-D Trilinos_ENABLE_AztecOO:BOOL=ON \`
 `-D Trilinos_ENABLE_Amesos:BOOL=ON \`
 `-D Trilinos_ENABLE_Anasazi:BOOL=ON \`
 `-D Trilinos_ENABLE_Belos:BOOL=ON \`
 `-D Trilinos_ENABLE_ML:BOOL=ON \`
 `-D Trilinos_ENABLE_Phalanx:BOOL=ON \`
 `-D Phalanx_ENABLE_COMPILETIME_ARRAY_CHECK:BOOL=ON\`
 `-D Phalanx_KOKKOS_DEVICE_TYPE:STRING="OPENMP" \`
 `-D Phalanx_INDEX_SIZE_TYPE="UINT" \`
 `-D Trilinos_ENABLE_Intrepid:BOOL=ON \`
 `-D HAVE_INTREPID_KOKKOSCORE:BOOL=ON \`
 `-D Trilinos_ENABLE_NOX:BOOL=ON \`
 `-D Trilinos_ENABLE_Stratimikos:BOOL=ON \`
 `-D Trilinos_ENABLE_Thyra:BOOL=ON \`
 `-D Trilinos_ENABLE_ThyraTpetraAdapters:BOOL=ON \`
 `-D Trilinos_ENABLE_Rythmos:BOOL=ON \`
 `-D Trilinos_ENABLE_MOOCHO:BOOL=ON \`
 `-D Trilinos_ENABLE_Stokhos:BOOL=ON \`
 `-D Trilinos_ENABLE_STKIO:BOOL=ON \`
 `-D Trilinos_ENABLE_STKMesh:BOOL=ON \`
 `-D Trilinos_ENABLE_Teko:BOOL=ON \`
 `-D Trilinos_ENABLE_SEACASIoss:BOOL=ON \`
 `-D Trilinos_ENABLE_SEACASExodus:BOOL=ON \`
 `-D SEACASExodus_PARALLEL_AWARE:BOOL=OFF \`
 `-D Trilinos_ENABLE_TriKota:BOOL=OFF \`
 `-D Trilinos_Enable_Kokkos:BOOL=ON \`
 `-D Trilinos_ENABLE_KokkosCore:BOOL=ON \`
 `-D Trilinos_ENABLE_KokkosClassic:BOOL=ON \`
 `-D Trilinos_ENABLE_KokkosContainers:BOOL=ON \`
 `-D Trilinos_ENABLE_KokkosCompat:BOOL=ON \`
 `-D Trilinos_ENABLE_KokkosTPL:BOOL=ON \`
 `-D Trilinos_ENABLE_KokkosLinAlg:BOOL=ON \`
 `-D Trilinos_ENABLE_TeuchosKokkosCompat:BOOL=ON \`
 `-D Trilinos_ENABLE_TpetraKernels:BOOL=ON \`
 `-D Trilinos_ENABLE_TeuchosKokkosComm:BOOL=ON \`
 `-D KokkosClassic_DefaultNode:STRING="Kokkos::Compat::KokkosOpenMPWrapperNode" \`
 `-D Trilinos_ENABLE_OpenMP:BOOL=ON \`
 `-D Kokkos_ENABLE_OpenMP:BOOL=ON \`
 `-D TPL_ENABLE_Boost:BOOL=ON \`
 `-D Boost_INCLUDE_DIRS:FILEPATH="$BOOSTDIR/include" \`
 `-D Boost_LIBRARY_DIRS:FILEPATH="$BOOSTDIR/lib" \`
 `-D TPL_ENABLE_BoostLib:BOOL=ON \`
 `-D BoostLib_INCLUDE_DIRS:FILEPATH="$BOOSTDIR/include" \`
 `-D BoostLib_LIBRARY_DIRS:FILEPATH="$BOOSTDIR/lib" \`
 `-D TPL_ENABLE_Netcdf:BOOL=ON \`
 `-D Netcdf_INCLUDE_DIRS:PATH="$NETCDF/include" \`
 `-D Netcdf_LIBRARY_DIRS:PATH="$NETCDF/lib" \`
 `-D TPL_ENABLE_HDF5:BOOL=ON \`
 `-D HDF5_INCLUDE_DIRS:PATH="$HDFDIR/include" \`
 `-D HDF5_LIBRARY_DIRS:PATH="$HDFDIR/lib" \`
 `-D Trilinos_ENABLE_Mesquite:BOOL=OFF \`
 `-D Trilinos_ENABLE_EXAMPLES:BOOL=OFF \`
 `-D Trilinos_ENABLE_TESTS:BOOL=OFF \`
 `-D TPL_ENABLE_BinUtils:BOOL=OFF \`
 `-D TPL_ENABLE_MPI:BOOL=ON \`
 `-D CMAKE_C_COMPILER="mpiicc" \`
 `-D CMAKE_CXX_COMPILER="mpiicpc" \`
 `-D CMAKE_Fortran_COMPILER="mpiifort" \`
 `-D SEACASExodus_PARALLEL_AWARE:BOOL=OFF \`
 `-D CMAKE_VERBOSE_MAKEFILE:BOOL=OFF \`
 `-D CMAKE_CXX_FLAGS:STRING="-g -DMPICH_IGNORE_CXX_SEEK -DMPICH_SKIP_MPICXX -I./install/include/" \`
 `-D CMAKE_C_FLAGS:STRING="-g -DMPICH_IGNORE_CXX_SEEK -DMPICH_SKIP_MPICXX" \`
 `-D Trilinos_ENABLE_Fortran:BOOL=OFF \`
 `-D TPL_ENABLE_Pthread=ON \`
 `-D TPL_Pthread_LIBRARIES="pthread" \`
 `-D TPL_ENABLE_HWLOC=ON \`
 `-D HWLOC_INCLUDE_DIRS=/home/ipdemes/install/hwloc/include \`
 `-D HWLOC_LIBRARY_DIRS=/home/ipdemes/install/hwloc/lib \`
 `-D HAVE_GCC_ABI_DEMANGLE_EXITCODE=0 \`
 `-D HAVE_TEUCHOS_BLASFLOAT_EXITCODE=0 \`
 `-D LAPACK_SLAPY2_WORKS_EXITCODE=0 \`
 `-D BLAS_LIBRARY_DIRS:FILEPATH="${MKLROOT}/lib/intel64" \`
 `-D BLAS_LIBRARY_NAMES="mkl_intel_lp64;mkl_sequential;mkl_core;pthread;m" \`
 `-D LAPACK_LIBRARY_NAMES="" \`
 `-D TPL_LAPACK_LIBRARIES="" \`
 `-D Trilinos_ENABLE_CXX11:BOOL=ON \`
 `-D Kokkos_ENABLE_CXX11:BOOL=ON \`
 `-D Amesos2_ENABLE_KLU2:BOOL=ON \`
 `\`
 `\`
 `$EXTRA_ARGS \`
 `${TRILINOS_HOME}`

  3. Build Albany

  4. In purpose to run on Compton on SandyBridge you need to do next: salloc -N 1 mpirun ...

Building for MIC

  1. Building supporting packages

    Instructions on how to build hdf5 for MIC can be found here: http://linux4hippos.blogspot.com/2013/05/cross-compile-hdf5-for-intel-xeon-phi.html

  2. load next modules:

    1. Load modules for SB:

    module load intelmpi/mic/4.1.1.036

    module load intel/compiler/15.2.164

    module load cmake

  3. Building Trilinos

Please use next configure script as a reference:

     export TRILINOS_HOME=/home/ipdemes/TrilinosDir/Trilinos_github/Trilinos
     BUILD_DIR=`pwd`
     INSTALL_DIR=your_install_dir
     BOOSTDIR=/home/ipdemes/install/boost_1_54_mic_3
     NETCDF=/home/projects/x86-64/netcdf/mic/4.3.3.1/intel/15.2.164/intelmpi/4.1.1.036
     HDFDIR=/home/projects/x86-64/hdf5/mic/1.8.14/intel/15.2.164/intelmpi/4.1.1.036
     ZLIB_DIR=/home/gahanse/mic
     EXTRA_ARGS=$@

     rm -f CMakeCache.txt

     cmake \
    -D Trilinos_DISABLE_ENABLED_FORWARD_DEP_PACKAGES=ON \
    -D CMAKE_INSTALL_PREFIX:PATH="/home/ipdemes/TrilinosDir/BuildTrilinos_Albany_MIC/install"  \
    -D CMAKE_BUILD_TYPE:STRING="NONE"  \
    -D Trilinos_ENABLE_ALL_PACKAGES:BOOL=OFF \
    -D TPL_FIND_SHARED_LIBS:BOOL=OFF \
    -D Trilinos_WARNINGS_AS_ERRORS_FLAGS:STRING="" \
    -D Trilinos_ENABLE_Teuchos:BOOL=ON \
    -D Trilinos_ENABLE_Shards:BOOL=ON \
    -D TPL_ENABLE_GLM=OFF \
    -D Trilinos_ENABLE_Sacado:BOOL=ON \
    -D Trilinos_ENABLE_Epetra:BOOL=ON \
    -D Trilinos_ENABLE_Tpetra:BOOL=ON \
    -D Tpetra_ENABLE_Kokkos_Refactor:BOOL=ON \
    -D Trilinos_ENABLE_EpetraExt:BOOL=ON \
    -D Trilinos_ENABLE_Ifpack:BOOL=ON \
    -D Trilinos_ENABLE_AztecOO:BOOL=ON \
    -D Trilinos_ENABLE_Amesos:BOOL=ON \
    -D Trilinos_ENABLE_Anasazi:BOOL=ON \
    -D Trilinos_ENABLE_Belos:BOOL=ON \
    -D Trilinos_ENABLE_ML:BOOL=ON \
    -D Trilinos_ENABLE_Phalanx:BOOL=ON \
    -D Phalanx_ENABLE_COMPILETIME_ARRAY_CHECK:BOOL=ON\
    -D Phalanx_KOKKOS_DEVICE_TYPE:STRING="OPENMP" \
    -D Phalanx_INDEX_SIZE_TYPE="UINT" \
    -D Trilinos_ENABLE_Intrepid:BOOL=ON \
    -D HAVE_INTREPID_KOKKOSCORE:BOOL=ON \
    -D Trilinos_ENABLE_STKIO:BOOL=ON \
    -D Trilinos_ENABLE_STKMesh:BOOL=ON \
    -D Trilinos_ENABLE_NOX:BOOL=ON \
    -D Trilinos_ENABLE_Stratimikos:BOOL=ON \
    -D Trilinos_ENABLE_Thyra:BOOL=ON \
    -D Trilinos_ENABLE_ThyraTpetraAdapters:BOOL=ON \
    -D Trilinos_ENABLE_Rythmos:BOOL=ON \
    -D Trilinos_ENABLE_MOOCHO:BOOL=ON \
    -D Trilinos_ENABLE_Stokhos:BOOL=ON \
    -D Trilinos_ENABLE_Teko:BOOL=ON \
    -D Trilinos_ENABLE_SEACASIoss:BOOL=ON \
    -D Trilinos_ENABLE_SEACASExodus:BOOL=ON \
    -D SEACASExodus_PARALLEL_AWARE:BOOL=OFF \
    -D Trilinos_ENABLE_TriKota:BOOL=OFF \
    -D Trilinos_Enable_Kokkos:BOOL=ON \
    -D Trilinos_ENABLE_KokkosCore:BOOL=ON \
    -D Trilinos_ENABLE_KokkosClassic:BOOL=ON \
    -D Trilinos_ENABLE_KokkosContainers:BOOL=ON \
    -D Trilinos_ENABLE_KokkosCompat:BOOL=ON \
    -D Trilinos_ENABLE_KokkosTPL:BOOL=ON \
    -D Trilinos_ENABLE_KokkosLinAlg:BOOL=ON \
    -D Trilinos_ENABLE_TeuchosKokkosCompat:BOOL=ON \
    -D Trilinos_ENABLE_TpetraKernels:BOOL=ON \
    -D Trilinos_ENABLE_TeuchosKokkosComm:BOOL=ON \
    -D Teuchos_ENABLE_LONG_LONG_INT:BOOL=ON \
    -D Trilinos_ENABLE_OpenMP:BOOL=ON \
    -D Kokkos_ENABLE_OpenMP:BOOL=ON \
    -D TPL_ENABLE_Boost:BOOL=ON \
    -D Boost_INCLUDE_DIRS:FILEPATH="$BOOSTDIR/include" \
    -D Boost_LIBRARY_DIRS:FILEPATH="$BOOSTDIR/lib" \
    -D TPL_ENABLE_BoostLib:BOOL=ON \
    -D BoostLib_INCLUDE_DIRS:FILEPATH="$BOOSTDIR/include" \
    -D BoostLib_LIBRARY_DIRS:FILEPATH="$BOOSTDIR/lib" \
    -D TPL_ENABLE_Netcdf:BOOL=ON \
    -D Netcdf_INCLUDE_DIRS:PATH="$NETCDF/include" \
    -D Netcdf_LIBRARY_DIRS:PATH="$NETCDF/lib" \
    -D TPL_ENABLE_HDF5:BOOL=ON \
    -D HDF5_INCLUDE_DIRS:PATH="$HDFDIR/include" \
    -D HDF5_LIBRARY_DIRS:PATH="$HDFDIR/lib" \
    \
    -D TPL_ENABLE_Zlib:STRING=ON \
    -D Zlib_INCLUDE_DIRS:PATH="${ZLIB_DIR}/include" \
    -D Zlib_LIBRARY_DIRS:PATH="${ZLIB_DIR}/lib" \
    -D Trilinos_ENABLE_Mesquite:BOOL=OFF \
    -D Trilinos_ENABLE_EXAMPLES:BOOL=OFF \
    -D Trilinos_ENABLE_TESTS:BOOL=OFF \
    -D TPL_ENABLE_BinUtils:BOOL=OFF \
    -D TPL_ENABLE_MPI:BOOL=ON \
    -D CMAKE_C_COMPILER="mpiicc" \
    -D CMAKE_CXX_COMPILER="mpiicpc" \
    -D CMAKE_Fortran_COMPILER="mpiifort" \
    -D SEACASExodus_PARALLEL_AWARE:BOOL=OFF \
    -D CMAKE_VERBOSE_MAKEFILE:BOOL=OFF \
     -D CMAKE_CXX_FLAGS:STRING="-O3 -mmic -mkl -DMPICH_IGNORE_CXX_SEEK -DMPICH_SKIP_MPICXX -DPREC_TIMER -restrict -vec_report -fasm-blocks -DDEVICE=1wq  -fopenmp  -lhwloc -lrt -I./install/include/ -L/home/projects/libraries/mic/hwloc/1.6.2/lib -L/home/ipdemes/install/zlib_mic/lib -L/home/projects/x86-64/hdf5/mic/1.8.14/intel/15.2.164/intelmpi/4.1.1.036/lib -lhdf5_hl -I/home/projects/x86-64/hdf5/mic/1.8.14/intel/15.2.164/intelmpi/4.1.1.036/include -lhdf5 -L/home/projects/x86-64/netcdf/mic/4.3.3.1/intel/15.2.164/intelmpi/4.1.1.036/lib -lnetcdf  " \
    -D CMAKE_C_FLAGS:STRING="-O3 -mmic -mkl -DMPICH_IGNORE_CXX_SEEK -DMPICH_SKIP_MPICXX -DPREC_TIMER -restrict -vec_report -fasm-blocks -DDEVICE=1wq  -fopenmp  -lhwloc -lrt -I./install/include/ -L/home/projects/libraries/mic/hwloc/1.6.2/lib -L/home/ipdemes/install/zlib_mic/lib -L/home/projects/x86-64/hdf5/mic/1.8.14/intel/15.2.164/intelmpi/4.1.1.036/lib -lhdf5_hl  -I/home/projects/x86-64/hdf5/mic/1.8.14/intel/15.2.164/intelmpi/4.1.1.036/include -lhdf5 -L/home/projects/x86-64/netcdf/mic/4.3.3.1/intel/15.2.164/intelmpi/4.1.1.036/lib -lnetcdf  " \
    -D Trilinos_ENABLE_Fortran:BOOL=OFF \
    -D TPL_ENABLE_Pthread=ON \
    -D TPL_Pthread_LIBRARIES="pthread" \
    -D TPL_ENABLE_HWLOC=OFF \
    -D HWLOC_INCLUDE_DIRS="/home/projects/libraries/mic/hwloc/1.6.2/include" \
    -D HWLOC_LIBRARY_DIRS="/home/projects/libraries/mic/hwloc/1.6.2/lib" \
    -D HAVE_GCC_ABI_DEMANGLE_EXITCODE=0 \
    -D HAVE_TEUCHOS_BLASFLOAT_EXITCODE=0 \
    -D LAPACK_SLAPY2_WORKS_EXITCODE=0 \
    -D BLAS_LIBRARY_DIRS:FILEPATH="${MKLROOT}/lib/mic" \
    -D BLAS_LIBRARY_NAMES="mkl_intel_lp64;mkl_sequential;mkl_core;pthread;m" \
    -D LAPACK_LIBRARY_NAMES="" \
    -D TPL_LAPACK_LIBRARIES="" \
    -D Trilinos_ENABLE_CXX11:BOOL=ON \
    -D Amesos2_ENABLE_KLU2:BOOL=ON \
    -D Trilinos_ENABLE_Piro:BOOL=ON \
    -D Trilinos_ENABLE_Teko:BOOL=ON \
    \
    \
    $EXTRA_ARGS \
    ${TRILINOS_HOME}

  1. Building Albany

    Please use next example as a reference:

rm CMakeCache.txt

   cmake \
   -D ALBANY_TRILINOS_DIR:FILEPATH=/home/ipdemes/TrilinosDir/BuildTrilinos_Albany_MIC/install \
   -D CMAKE_VERBOSE_MAKEFILE:BOOL=OFF \
   -D ENABLE_DEMO_PDES=OFF -D ENABLE_SG_MP=OFF -D ENABLE_MOR=OFF -DENABLE_QCAD=OFF  -DENABLE_ASCR=OFF \
   -D ENABLE_FELIX=OFF \
   -D ENABLE_AERAS=ON \
   -D ENABLE_FAST_FELIX:BOOL=ON \
   -D CMAKE_CXX_FLAGS:STRING="-O3 -mmic -mkl -DMPICH_IGNORE_CXX_SEEK -DMPICH_SKIP_MPICXX -DPREC_TIMER -restrict -vec_report -fasm-blocks -DDEVICE=1wq  -fopenmp  -lhwloc -lrt -I./install/include/ -L/home/projects/libraries/mic/hwloc/1.6.2/lib -L/home/ipdemes/install/zlib_mic/lib -L/home/projects/x86-64/hdf5/mic/1.8.14/intel/15.2.164/intelmpi/4.1.1.036/lib -lhdf5_hl -I/home/projects/x86-64/hdf5/mic/1.8.14/intel/15.2.164/intelmpi/4.1.1.036/include -lhdf5 -L/home/projects/x86-64/netcdf/mic/4.3.3.1/intel/15.2.164/intelmpi/4.1.1.036/lib -lnetcdf  " \
   -D CMAKE_C_FLAGS:STRING="-O3 -mmic -mkl -DMPICH_IGNORE_CXX_SEEK -DMPICH_SKIP_MPICXX -DPREC_TIMER -restrict -vec_report -fasm-blocks -DDEVICE=1wq  -fopenmp  -lhwloc -lrt -I./install/include/ -L/home/projects/libraries/mic/hwloc/1.6.2/lib -L/home/ipdemes/install/zlib_mic/lib -L/home/projects/x86-64/hdf5/mic/1.8.14/intel/15.2.164/intelmpi/4.1.1.036/lib -lhdf5_hl  -I/home/projects/x86-64/hdf5/mic/1.8.14/intel/15.2.164/intelmpi/4.1.1.036/include -lhdf5     -L/home/projects/x86-64/netcdf/mic/4.3.3.1/intel/15.2.164/intelmpi/4.1.1.036/lib -lnetcdf  " \
   -D ENABLE_KOKKOS_UNDER_DEVELOPMENT:BOOL=ON \
   -D ALBANY_ENABLE_FORTRAN:BOOL=OFF \
   -D ENABLE_SLFAD:BOOL=ON \
   -D SLFAD_SIZE=48 \
   ..

  1. In purpose to run on Compton on MIC you need to do next: salloc -N 1 ssh hostname-mic0 mpiexec your executable

    you might want to add next flags to your executable for better performance: mpiexec -np 4 -env OMP_NUM_THREADS=56 -env KMP_AFFINITY=compact Albany input.xml --kokkos-threads=56

Docker Image

SPACK Package

ToDo List

Albany FAQs

Capabilities

Sub-Projects

Compiling

Nightly Tests

Code HowTo

User Group Meetings

PAALS Tutorials

Wiki HowTo

Visualization Utilities

Good Commits

Python and Albany

Clone this wiki locally