-
Notifications
You must be signed in to change notification settings - Fork 90
Shannon
Please email [email protected] with comments or questions.
Shannon is integrated by Appro
- Current configuration installed 9/2014
- 22 compute nodes
- Each node has two 8 core Sandy Bridge Xeon E5-2670 processors @ 2.6 GHz
- Each node has 128 GB - DDR3 memory 1600 MHz
- 6 nodes have 2x NVIDIA K20x (Kepler), 12 have 4x K20x, 4 have 4x K40m, and 9 have 2x K80s
- 2688 732MHz cores
- Mellanox QDR Infiniband
Here is the module load sequence used
module purge
module load openmpi/1.10.1/gnu/4.7.2/cuda/7.5.7
export CUDA_LAUNCH_BLOCKING=1
These install directions assume that a base install directory of /home/gahanse/gcc-4.7.2/ompi-1.10.1
is used
Note that "shannon" has prebuilt versions of zlib, hfd5, and netcdf. These are actually used, but we show build instructions on how one would build these libraries if they do not exist. We mark these as (optional).
-
Version 1.2.8
cd /home/gahanse/src wget http://zlib.net/zlib-1.2.8.tar.gz tar -xvf zlib-1.2.8.tar.gz cd zlib-1.2.8 CC=mpicc CFLAGS="-O3" ./configure --prefix=/home/gahanse/gcc-4.7.2/ompi-1.10.1 make install
-
Version 1.9
./configure --prefix=/home/gahanse/gcc-4.7.2/ompi-1.10.1 CC=mpicc CXX=mpicxx
-
Version 1.8.13
cd /home/gahanse/src/ wget http://www.hdfgroup.org/ftp/HDF5/current/src/hdf5-1.8.13.tar.bz2 tar -xvf hdf5-1.8.13.tar.gz cd hdf5-1.8.13 ./configure CC=mpicc FC=mpif90 CXX=mpicxx CXXFLAGS="-fPIC -O3" CFLAGS="-fPIC -O3" FCFLAGS="-fPIC -O3" \ --enable-parallel --with-zlib=/home/gahanse/gcc-4.7.2/ompi-1.10.1 --prefix=/home/gahanse/gcc-4.7.2/ompi-1.10.1 make -j 8 make install
-
Version 4.3.3.1
cd /home/gahanse/src/ wget ftp://ftp.unidata.ucar.edu/pub/netcdf/netcdf-4.3.3.1.tar.gz tar -xvf netcdf-4.3.3.1.tar.gz cd netcdf-4.3.3.1
-
Edit the file
include/netcdf.h
. Make the following edits near line 228:#define NC_MAX_DIMS 65536 /* max dimensions per file */ #define NC_MAX_ATTRS 8192 #define NC_MAX_VARS 524288 /* max variables per file */ #define NC_MAX_NAME 256 #define NC_MAX_VAR_DIMS 8 /* max per variable dimensions */
-
Now, configure build:
./configure --prefix=/home/gahanse/gcc-4.7.2/ompi-1.10.1 CC=mpicc FC=mpifort CXX=mpicxx \ CFLAGS="-I/home/gahanse/gcc-4.7.2/ompi-1.10.1/include -fpic -O3" \ CXXFLAGS="-I/home/gahanse/gcc-4.7.2/ompi-1.10.1/include -fPIC -O3" \ LDFLAGS=-L/home/gahanse/gcc-4.7.2/ompi-1.10.1/lib \ FCFLAGS="-fPIC -O3" --disable-fsync --disable-cdmremote --disable-dap --disable-doxygen --enable-netcdf-4 make -j 8 make install
-
Version 1.59.0
cd /home/gahanse/src/ wget http://sourceforge.net/projects/boost/files/boost/1.59.0/boost_1_59_0.tar.gz/download tar -xvf boost_1_59_0.tar.gz cd boost_1_59_0
-
Build boost with the commands
echo "using gcc : 4.7.2 : /home/projects/gcc/4.7.2/bin/g++ ;" > ~/user-config.jam echo "using mpi : /home/projects/x86-64-sandybridge-nvidia/openmpi/1.10.1/gnu/4.7.2/cuda/7.5.7/bin/mpicxx ;" >> ~/user-config.jam ./bootstrap.sh \ --with-libraries=signals,regex,filesystem,system,mpi,serialization,thread,program_options,exception \ --prefix=/home/gahanse/gcc-4.7.2/ompi-1.10.1 ./b2 -j 8 ./b2 -j 8 install
-
Version 4.0.3
cd /home/gahanse/src/ wget http://glaros.dtc.umn.edu/gkhome/fetch/sw/parmetis/parmetis-4.0.3.tar.gz tar -xvf parmetis-4.0.3.tar.gz cd parmetis-4.0.3
-
Edit
metis.h
to use anIDXTYPEWIDTH
of 64cd metis/include vi metis.h #define IDXTYPEWIDTH 64 #define REALTYPEWIDTH 64
-
Run CMake to build
parmetis
cd /home/gahanse/src/parmetis-4.0.3 cd build export metis=/home/gahanse/src/parmetis-4.0.3/metis cmake \ -DCMAKE_INSTALL_PREFIX=/home/gahanse/gcc-4.7.2/ompi-1.10.1 \ -DMETIS_PATH=$metis \ -DGKLIB_PATH=$metis/GKlib \ -DCMAKE_C_COMPILER=mpicc \ -DCMAKE_CXX_COMPILER=mpicxx \ -DMPI_INCLUDE_PATH=/home/gahanse/gcc-4.7.2/ompi-1.10.1/include \ -DCMAKE_C_FLAGS="-fPIC -O3" \ -DCMAKE_CXX_FLAGS="-fPIC -O3" \ .. make -j 8 make install cp libmetis/libmetis.a /home/gahanse/gcc-4.7.2/ompi-1.10.1/lib cp $metis/include/metis.h /home/gahanse/gcc-4.7.2/ompi-1.10.1/include
-
Download Trilinos
cd /home/gahanse/src git clone https://github.com/trilinos/Trilinos.git
-
Download SCOREC
cd /home/gahanse/src/Trilinos git clone [email protected]:SCOREC/core.git SCOREC
-
Configure the Trilinos build
mkdir build cd build ./do-configure
-
Example
do-configure
script executed above\#!/bin/sh \# export TRILINOS_HOME=/home/gahanse/Codes/Trilinos BUILD_DIR=`pwd` INSTALL_DIR=/home/gahanse/trilinos/MPI_NGCC BOOST_DIR=/home/gahanse/gcc-4.9.0/mpich-3.1.4 MPI_BASE_DIR=/home/gahanse/gcc-4.9.0/mpich-3.1.4 NETCDF=/home/gahanse/gcc-4.9.0/mpich-3.1.4 HDFDIR=/home/gahanse/gcc-4.9.0/mpich-3.1.4 PARMETISDIR=/home/gahanse/gcc-4.9.0/mpich-3.1.4 MKL_PATH=/opt/intel/mkl/lib/intel64 HWLOC_PATH=/home/gahanse/gcc-4.9.0/mpich-3.1.4 LABLAS_LIBRARIES="-L$MKL_PATH/lib/intel64 -Wl,--start-group $MKL_PATH/mkl/lib/intel64/libmkl_intel_lp64.a $MKL_PATH/mkl/lib/intel64/ cmake \ -Wno-dev \ -D Trilinos_CONFIGURE_OPTIONS_FILE:FILEPATH=$TRILINOS_HOME/sampleScripts/AlbanySettings.cmake \ -D Trilinos_ENABLE_SCOREC:BOOL=ON \ -D SCOREC_DISABLE_STRONG_WARNINGS:BOOL=ON \ -D CMAKE_BUILD_TYPE:STRING=NONE \ -D CMAKE_CXX_COMPILER:FILEPATH="/home/gahanse/Codes/Trilinos/build/nvcc_wrapper_gh" \ -D CMAKE_C_COMPILER:FILEPATH="mpicc" \ -D CMAKE_Fortran_COMPILER:FILEPATH="mpifort" \ -D CMAKE_CXX_FLAGS:STRING="-DNDEBUG" \ -D Trilinos_CXX11_FLAGS:STRING="-std=c++11 --expt-extended-lambda --expt-relaxed-constexpr -Wno-unused-local-typedefs -Wno-sign- -D CMAKE_C_FLAGS:STRING="-O3 -DNDEBUG" \ -D CMAKE_Fortran_FLAGS:STRING="-O3 -DNDEBUG" \ -D Trilinos_ENABLE_EXPLICIT_INSTANTIATION:BOOL=ON \ -D Tpetra_INST_INT_LONG_LONG:BOOL=OFF \ -D Tpetra_INST_INT_INT:BOOL=ON \ -D Tpetra_INST_DOUBLE:BOOL=ON \ -D Tpetra_INST_FLOAT:BOOL=OFF \ -D Tpetra_INST_COMPLEX_FLOAT:BOOL=OFF \ -D Tpetra_INST_COMPLEX_DOUBLE:BOOL=OFF \ -D Tpetra_INST_INT_LONG:BOOL=OFF \ -D Tpetra_INST_INT_UNSIGNED:BOOL=OFF \ -D Zoltan_ENABLE_ULONG_IDS:BOOL=OFF \ -D Teuchos_ENABLE_LONG_LONG_INT:BOOL=ON \ -D Rythmos_ENABLE_DEBUG:BOOL=ON \ -D Trilinos_ENABLE_Kokkos:BOOL=ON \ -D Phalanx_KOKKOS_DEVICE_TYPE:STRING="CUDA" \ -D Phalanx_INDEX_SIZE_TYPE:STRING="INT" \ -D Phalanx_SHOW_DEPRECATED_WARNINGS:BOOL=OFF \ -D Kokkos_ENABLE_Serial:BOOL=ON \ -D Kokkos_ENABLE_OpenMP:BOOL=OFF \ -D Kokkos_ENABLE_Pthread:BOOL=OFF \ -D Kokkos_ENABLE_Cuda:BOOL=ON \ -D TPL_ENABLE_CUDA:BOOL=ON \ -D Kokkos_ENABLE_Cuda_UVM:BOOL=ON \ -D Kokkos_ENABLE_Cuda_LAMBDA:BOOL=ON \ -D TPL_ENABLE_CUSPARSE:BOOL=ON \ \ -D TPL_ENABLE_MPI:BOOL=ON \ -D MPI_BASE_DIR:PATH=$MPI_BASE_DIR \ \ -D TPL_ENABLE_Pthread:BOOL=OFF \ \ -D TPL_ENABLE_Boost:BOOL=ON \ -D TPL_ENABLE_BoostLib:BOOL=ON \ -D TPL_ENABLE_BoostAlbLib:BOOL=ON \ -D Boost_INCLUDE_DIRS:PATH=$BOOST_DIR/include \ -D Boost_LIBRARY_DIRS:PATH=$BOOST_DIR/lib \ -D BoostLib_INCLUDE_DIRS:PATH=$BOOST_DIR/include \ -D BoostLib_LIBRARY_DIRS:PATH=$BOOST_DIR/lib \ -D BoostAlbLib_INCLUDE_DIRS:PATH=$BOOST_DIR/include \ -D BoostAlbLib_LIBRARY_DIRS:PATH=$BOOST_DIR/lib \ \ -D TPL_ENABLE_Netcdf:STRING=ON \ -D Netcdf_INCLUDE_DIRS:PATH=$NETCDF/include \ -D Netcdf_LIBRARY_DIRS:PATH=$NETCDF/lib \ \ -D TPL_ENABLE_HDF5:STRING=ON \ -D HDF5_INCLUDE_DIRS:PATH=$HDFDIR/include \ -D HDF5_LIBRARY_DIRS:PATH=$HDFDIR/lib \ \ -D TPL_ENABLE_Zlib:STRING=ON \ -D Zlib_INCLUDE_DIRS:PATH=$HDFDIR/include \ -D Zlib_LIBRARY_DIRS:PATH=$HDFDIR/lib \ \ -D TPL_ENABLE_BLAS:STRING=ON \ -D TPL_ENABLE_LAPACK:STRING=ON \ -D BLAS_LIBRARY_DIRS:FILEPATH=${MKL_PATH} \ -D BLAS_LIBRARY_NAMES="mkl_intel_lp64;mkl_sequential;mkl_core;m" \ -D LAPACK_LIBRARY_NAMES="" \ \ -D TPL_ENABLE_ParMETIS:STRING=ON \ -D ParMETIS_INCLUDE_DIRS:PATH="${PARMETISDIR}/include" \ -D ParMETIS_LIBRARY_DIRS:PATH="${PARMETISDIR}/lib" \ \ -D CMAKE_INSTALL_PREFIX:PATH=$INSTALL_DIR \ -D Trilinos_EXTRA_LINK_FLAGS:STRING="-L${HDFDIR}/lib -lnetcdf -lhdf5_hl -lhdf5 -lz" \ \ -D Trilinos_ENABLE_Moertel:BOOL=OFF \ -D Trilinos_ENABLE_TriKota:BOOL=OFF \ -D Trilinos_ENABLE_SEACASSVDI:BOOL=OFF \ -D Trilinos_ENABLE_SEACASPLT:BOOL=OFF \ -D Trilinos_ENABLE_SEACASBlot:BOOL=OFF \ -D Trilinos_ENABLE_STK:BOOL=ON \ -D Trilinos_ENABLE_STKUtil:BOOL=ON \ -D Trilinos_ENABLE_STKTopology:BOOL=ON \ -D Trilinos_ENABLE_STKMesh:BOOL=ON \ -D Trilinos_ENABLE_STKIO:BOOL=ON \ -D Trilinos_ENABLE_STKTransfer:BOOL=ON \ -D Trilinos_ENABLE_STKUnit_tests:BOOL=OFF \ -D Trilinos_ENABLE_STKDoc_tests:BOOL=OFF \ -D TPL_ENABLE_X11:BOOL=OFF \ -D Trilinos_ENABLE_ThreadPool:BOOL=OFF \ -D Trilinos_ENABLE_Teko:BOOL=OFF \ -D Trilinos_ENABLE_MueLu:BOOL=ON \ -D Trilinos_ENABLE_Amesos2:BOOL=ON \ -D Amesos2_ENABLE_KLU2:BOOL=ON \ -D BUILD_SHARED_LIBS:BOOL=OFF \ -D Trilinos_ENABLE_OptiPack:BOOL=OFF \ -D Trilinos_ENABLE_GlobiPack:BOOL=OFF \ \ $EXTRA_ARGS \ ${TRILINOS_HOME} exit 0
-
Build Trilinos
make -j 8 make install
-
Clone the repo
cd /home/gahanse/src git clone [email protected]:gahansen/Albany.git cd Albany
-
Configure the Albany build
mkdir build cd build ./do-configure
-
CMake configure script
./do-configure
:\#!/bin/sh \# export TRILINOS_HOME=/home/gahanse/trilinos/MPI_CUVM cmake \ -DALBANY_TRILINOS_DIR:PATH=${TRILINOS_HOME} \ -DENABLE_LCM:BOOL=ON \ -DENABLE_LCM_SPECULATIVE:BOOL=OFF \ -DENABLE_HYDRIDE:BOOL=OFF \ -DENABLE_SCOREC:BOOL=ON \ -DENABLE_SG_MP:BOOL=OFF \ -DENABLE_FELIX:BOOL=ON \ -DENABLE_AERAS:BOOL=ON \ -DENABLE_QCAD:BOOL=OFF \ -DENABLE_MOR:BOOL=OFF \ -DENABLE_ATO:BOOL=OFF \ -DENABLE_ASCR:BOOL=OFF \ -DENABLE_CHECK_FPE:BOOL=OFF \ -DENABLE_ALBANY_EPETRA_EXE:BOOL=ON \ -DENABLE_KOKKOS_UNDER_DEVELOPMENT:BOOL=ON \ \ .. exit 0
-
Build Albany
make -j 8 make install
-
Run Albany tests
salloc -n 4 -N 4 -p stella ctest