-
Notifications
You must be signed in to change notification settings - Fork 90
ALCF Vesta
dalabre edited this page Nov 4, 2014
·
24 revisions
Please place the following lines in your ~/.soft file:
@default
+hpctoolkit
+cmake
+mpiwrapper-gcc
These install directions assume that a base install directory of /home/ghansen/FASTMath/src/ is used
-
Version 1.2.8
cd /home/ghansen/FASTMath/src/ wget http://zlib.net/zlib-1.2.8.tar.gz tar -xvf zlib-1.2.8.tar.gz cd zlib-1.2.8 CC=mpicc CXX=mpicxx CFLAGS=-O3 CXXFLAGS=-O3 ./configure --64 --static --prefix=/home/ghansen/FASTMath make install
-
Version 1.8.13
cd /home/ghansen/FASTMath/src/ wget http://www.hdfgroup.org/ftp/HDF5/current/src/hdf5-1.8.13.tar.gz tar -xvf hdf5-1.8.13.tar.gz cd hdf5-1.8.13 ./configure CC=mpicc FC=mpif90 CXX=mpicxx CXXFLAGS="-fPIC -O3" CFLAGS="-fPIC -O3" FCFLAGS="-fPIC -O3" \ --enable-parallel --enable-shared=no --with-zlib=/home/ghansen/FASTMath --prefix=/home/ghansen/FASTMath make -j 8 make install
-
Version of the day from the git repo. Note that hdf5-1.8.13 made some changes that are not reflected in the latest release, so we are using the votd for now.
cd /home/ghansen/FASTMath/src/ git clone https://github.com/Unidata/netcdf-c.git cd netcdf-c -
Edit the file
include/netcdf.h. Make the following edits near line 228:#define NC_MAX_DIMS 65536 /* max dimensions per file */ #define NC_MAX_ATTRS 8192 #define NC_MAX_VARS 524288 /* max variables per file */ #define NC_MAX_NAME 256 #define NC_MAX_VAR_DIMS 8 /* max per variable dimensions */ -
Now, configure using CMake and build:
mkdir build cd build cmake \ -DCMAKE_C_COMPILER=mpicc \ -DCMAKE_C_FLAGS="-I/home/ghansen/FASTMath/include -O3" \ -DCMAKE_EXE_LINKER_FLAGS="-L/home/ghansen/FASTMath/lib -O3" \ -DCMAKE_INSTALL_PREFIX=/home/ghansen/FASTMath \ -DENABLE_DAP=OFF -DBUILD_SHARED_LIBS=OFF -DENABLE_FSYNC=OFF -DENABLE_CDMREMOTE=OFF \ -DENABLE_DOXYGEN=OFF -DENABLE_NETCDF_4=ON -DCMAKE_BUILD_TYPE=RELEASE -DENABLE_EXAMPLES=OFF \ -DENABLE_TESTS=OFF -DCMAKE_SKIP_INSTALL_RPATH=ON \ -DZLIB_INCLUDE_DIRS=/home/ghansen/FASTMath/include \ -DZLIB_LIBRARY=/home/ghansen/FASTMath/lib/libz.a \ -DBUILD_UTILITIES=OFF \ .. make -j 8 make install
-
Version 1.55.0
cd /home/ghansen/FASTMath/src/ wget http://sourceforge.net/projects/boost/files/boost/1.55.0/boost_1_55_0.tar.gz/download tar -xvf boost_1_55_0.tar.gz cd boost_1_55_0 -
Build boost with the commands
echo "using gcc : 4.4.7 : /bgsys/drivers/ppcfloor/gnu-linux/bin/powerpc64-bgq-linux-g++ ;" \ >> ./tools/build/v2/user-config.jam echo "using mpi : /bgsys/drivers/ppcfloor/comm/bin/gcc/mpicxx ;" >> ./tools/build/v2/user-config.jam ./bootstrap.sh \ --with-libraries=signals,regex,filesystem,system,mpi,serialization,thread,program_options,exception \ --prefix=/home/ghansen/FASTMath ./b2 -j 8 ./b2 -j 8 install
-
Version 4.0.3
cd /home/ghansen/FASTMath/src/ wget http://glaros.dtc.umn.edu/gkhome/fetch/sw/parmetis/parmetis-4.0.3.tar.gz tar -xvf parmetis-4.0.3.tar.gz cd parmetis-4.0.3 -
Edit
metis.hto use anIDXTYPEWIDTHof 64cd metis/include vi metis.h #define IDXTYPEWIDTH 64 -
Run CMake to build
parmetiscd /home/ghansen/FASTMath/src/parmetis-4.0.3 cd build export metis=/home/ghansen/FASTMath/src/parmetis-4.0.3/metis cmake \ -DCMAKE_INSTALL_PREFIX=/home/ghansen/FASTMath \ -DMETIS_PATH=$metis \ -DGKLIB_PATH=$metis/GKlib \ -DCMAKE_C_COMPILER=mpicc \ -DCMAKE_CXX_COMPILER=mpicxx \ .. make -j 8 make install cp libmetis/libmetis.a /home/ghansen/FASTMath/lib cp $metis/include/metis.h /home/ghansen/FASTMath/include
-
Version 4.3
cd /home/ghansen/FASTMath/src wget http://crd-legacy.lbl.gov/~xiaoye/SuperLU/superlu_4.3.tar.gz tar -xvf superlu_4.3.tar.gz cd superlu_4.3 mkdir /home/ghansen/FASTMath/SuperLU_4.3 mkdir /home/ghansen/FASTMath/SuperLU_4.3/include mkdir /home/ghansen/FASTMath/SuperLU_4.3/lib -
Edit
make.incfor your machine and environmentPLAT = _linux SuperLUroot = /home/ghansen/FASTMath/SuperLU_4.3 SUPERLULIB = $(SuperLUroot)/lib/libsuperlu_4.3.a TMGLIB = libtmglib.a BLASDEF = -DUSE_VENDOR_BLAS BLASLIB = /gpfs/vesta-fs0/projects/FASTMath/ATPESC-2014/install/blas/19Apr11/linux-rhel_6-ppc64-gcc-4.4.6/lib/blas.a LIBS = $(SUPERLULIB) $(BLASLIB) ARCH = ar ARCHFLAGS = cr RANLIB = ranlib CC = mpicc CFLAGS = -DPRNTlevel=0 -O3 NOOPTS = FORTRAN = mpif90 FFLAGS = -O3 LOADER = $(CC) LOADOPTS = CDEFS = -DNoChange MATLAB = -
Build SuperLU
make -j 4 cd SRC cp *.h /home/ghansen/FASTMath/SuperLU_4.3/include ar d home/ghansen/FASTMath/SuperLU_4.3/lib/libsuperlu_4.3.a dlamch.o slamch.o -
The last command deletes the
dlamchandslamchobjects, which already exist in the LAPACK library on Vesta
-
Download Trilinos
cd /home/ghansen/FASTMath/src git clone https://software.sandia.gov/trilinos/repositories/publicTrilinos -
Download SCOREC
cd /home/ghansen/FASTMath/src/publicTrilinos git clone https://github.com/SCOREC/core.git SCOREC -
Configure the Trilinos build
mkdir build cd build ./do-configure -
Example
do-configurescript executed above#!/bin/sh # export TRILINOS_HOME=/home/ghansen/FASTMath/src/publicTrilinos BUILD_DIR=`pwd` INSTALL_DIR=/home/ghansen/FASTMath/Trilinos/MPI_REL BOOST_DIR=/home/ghansen/FASTMath MPI_BASE_DIR=/bgsys/drivers/V1R2M1/ppc64/comm NETCDF=/home/ghansen/FASTMath HDFDIR=/home/ghansen/FASTMath PARMETISDIR=/home/ghansen/FASTMath BLAS=/gpfs/vesta-fs0/projects/FASTMath/ATPESC-2014/install/blas/19Apr11/linux-rhel_6-ppc64-gcc-4.4.6 LAPACK=/gpfs/vesta-fs0/projects/FASTMath/ATPESC-2014/install/lapack/3.5.0/linux-rhel_6-ppc64-gcc-4.4.6 export BOOST_ROOT=$BOOST_DIR EXTRA_ARGS=$@ cmake \ -D Trilinos_EXTRA_REPOSITORIES:STRING=SCOREC \ -D Trilinos_CONFIGURE_OPTIONS_FILE:FILEPATH=$TRILINOS_HOME/sampleScripts/AlbanySettings.cmake \ -D CMAKE_BUILD_TYPE:STRING=NONE \ -D TPL_FIND_SHARED_LIBS:BOOL=OFF \ \ -D CMAKE_C_FLAGS:STRING="-O3 -ffast-math -funroll-loops -DCODE_MP -DDISABLE_MPIIO -DNO_FILE_LOCK -DTEMPLATE_FRIENDS_NOT_SUPPORTED -DMPICH_SKIP_MPICXX -DNDEBUG -DHAVE_CONFIG_H" \ -D CMAKE_CXX_FLAGS:STRING="-O3 -ffast-math -funroll-loops -DCODE_MP -DDISABLE_MPIIO -DNO_FILE_LOCK -DTEMPLATE_FRIENDS_NOT_SUPPORTED -DMPICH_SKIP_MPICXX -DNDEBUG -DHAVE_CONFIG_H" \ -D CMAKE_Fortran_FLAGS:STRING="-O3 -ffast-math -funroll-loops" \ \ -D TPL_ENABLE_MPI:BOOL=ON \ -D MPI_BASE_DIR:PATH=$MPI_BASE_DIR \ -D Trilinos_EXTRA_LINK_FLAGS:STRING="-ldl" \ \ -D CMAKE_VERBOSE_MAKEFILE:BOOL=OFF \ -D Trilinos_VERBOSE_CONFIGURE:BOOL=OFF \ \ -D TPL_ENABLE_Matio:BOOL=OFF \ -D SEACAS_ENABLE_SEACASSVDI:BOOL=OFF \ -D Trilinos_ENABLE_SEACASFastq:BOOL=OFF \ -D Trilinos_ENABLE_SEACASBlot:BOOL=OFF \ -D Trilinos_ENABLE_SEACASPLT:BOOL=OFF \ -D TPL_ENABLE_X11:BOOL=OFF \ \ -D Boost_INCLUDE_DIRS:PATH=$BOOST_DIR/include \ -D Boost_LIBRARY_DIRS:PATH=$BOOST_DIR/lib \ -D BoostAlbLib_INCLUDE_DIRS:PATH=$BOOST_DIR/include \ -D BoostAlbLib_LIBRARY_DIRS:PATH=$BOOST_DIR/lib \ \ -D Netcdf_INCLUDE_DIRS:PATH="${NETCDF}/include" \ -D Netcdf_LIBRARY_DIRS:PATH="${NETCDF}/lib" \ -D TPL_Netcdf_LIBRARIES:STRING="/home/ghansen/FASTMath/lib/libnetcdf.a;/home/ghansen/FASTMath/lib/libhdf5_hl.a;/home/ghansen/FASTMath/lib/libhdf5.a;/home/ghansen/FASTMath/lib/libz.a" \ -D TPL_ENABLE_HDF5:BOOL=ON \ -D HDF5_INCLUDE_DIRS:PATH="${HDFDIR}/include" \ -D HDF5_LIBRARY_DIRS:PATH="${HDFDIR}/lib" \ \ -D TPL_ENABLE_Zlib:STRING=ON \ -D Zlib_INCLUDE_DIRS:PATH="${HDFDIR}/include" \ -D Zlib_LIBRARY_DIRS:PATH="${HDFDIR}/lib" \ \ -D ParMETIS_INCLUDE_DIRS:PATH="${PARMETISDIR}/include" \ -D ParMETIS_LIBRARY_DIRS:PATH="${PARMETISDIR}/lib" \ \ -D TPL_ENABLE_SuperLU:STRING=ON \ -D SuperLU_INCLUDE_DIRS:PATH="/home/ghansen/FASTMath/SuperLU_4.3/include" \ -D SuperLU_LIBRARY_DIRS:PATH="/home/ghansen/FASTMath/SuperLU_4.3/lib" \ \ -D TPL_ENABLE_BLAS:STRING=ON \ -D TPL_BLAS_LIBRARIES:FILEPATH="$BLAS/lib/blas.a" \ -D TPL_ENABLE_LAPACK:STRING=ON \ -D LAPACK_LIBRARY_DIRS:FILEPATH="$LAPACK/lib" \ \ -D Zoltan_ENABLE_ULONG_IDS:BOOL=ON \ \ -D Trilinos_ENABLE_SCOREC:BOOL=ON \ -D SCOREC_DISABLE_STRONG_WARNINGS:BOOL=ON \ \ -D Trilinos_ENABLE_ThyraTpetraAdapters:BOOL=ON \ -D Trilinos_ENABLE_Ifpack2:BOOL=ON \ -D Trilinos_ENABLE_Amesos2:BOOL=ON \ -D Trilinos_ENABLE_MueLu:BOOL=ON \ -D Teuchos_ENABLE_COMPLEX:BOOL=OFF \ \ -D Trilinos_ASSERT_MISSING_PACKAGES=OFF \ -D CMAKE_INSTALL_PREFIX:PATH=${INSTALL_DIR} \ \ $EXTRA_ARGS \ ${TRILINOS_HOME} -
Build Trilinos
make -j 8 make install
-
Clone the repo
cd /home/ghansen/FASTMath/src git clone https://github.com/gahansen/Albany.git cd Albany git checkout -b tpetra origin/tpetra -
Configure the Albany build
mkdir build cd build ./do-configure -
CMake configure script
./do-configure:#!/bin/sh # export ALB_ENABLE_SCOREC=ON export ALB_ENABLE_LCM=ON export ALB_ENABLE_MOR=OFF export ALB_ENABLE_SPECULATIVE=OFF export ALB_ENABLE_LAME=OFF export ALB_ENABLE_SG_MP=OFF export ALB_ENABLE_INSTALL=ON export TRILINOS_INSTALL_DIR=/home/ghansen/FASTMath/Trilinos/MPI_REL export ALBANY_INSTALL_DIR=/home/ghansen/FASTMath/Albany cmake \ -D ALBANY_TRILINOS_DIR:FILEPATH="$TRILINOS_INSTALL_DIR" \ -D ENABLE_LCM:BOOL=${ALB_ENABLE_LCM} \ -D ENABLE_LCM_SPECULATIVE:BOOL=${ALB_ENABLE_SPECULATIVE} \ -D ENABLE_SCOREC:BOOL=${ALB_ENABLE_SCOREC} \ -D ENABLE_MOR:BOOL=${ALB_ENABLE_MOR} \ -D ENABLE_SG_MP:BOOL=${ALB_ENABLE_SG_MP} \ -D ENABLE_INSTALL:BOOL=${ALB_ENABLE_INSTALL} \ -D CMAKE_INSTALL_PREFIX:FILEPATH=${ALBANY_INSTALL_DIR} \ -D ALBANY_MPI_OPTIONS:BOOL=ON \ -D ALBANY_MPI_EXEC:STRING="/home/ghansen/FASTMath/scripts/run_Albany" \ -D ALBANY_MPI_EXEC_NUMPROCS_FLAG:STRING="-n" \ -D ALBANY_MPI_EXEC_MAX_NUMPROCS:STRING="4" \ -D ALBANY_MPI_LEADING_OPTIONS:STRING="" \ \ /home/ghansen/FASTMath/src/Albany -
Build Albany
make -j 8 make install
-
Note that the above Albany configure script also sets up ctest to run the regression suite on BG/Q, employing the script
/home/ghansen/FASTMath/scripts/run_Albanyto dispatch each parallel test to the compute nodes while ctest itself runs on the frontend node. From the build directory on the frontend node, typingqsub -A FASTMath -n 4 -t 60 --mode script /home/ghansen/FASTMath/scripts/reserve.sh qstat -
Get the blockid info from qstat, and now run ctest
export MPIEXEC_BLOCKID=VST-20220-31331-32 ctest -
will run the regression suite in this manner. Note also that all the usual ctest options apply to this command.
Using Albany to perform an analysis is very similar to using any typical finite element analysis code. Two preparation steps are required: a) define the problem spatial domain and nature by meshing the parts and assemblies that participate in the analysis, and b) define the materials, properties, boundary and initial conditions and specify numerical solver parameters within the Albany input (driver) file.
- Given the mesh, one decomposes the mesh into a set of subdomains for parallel execution, with one subdomain per processing element (or MPI rank).
- One then constructs an xml input file for Albany that specifies how to perform the simulation on the supplied decomposed mesh files.
-
Reserve a run allocation, this example is for 128 processors for 30 minutes
qsub -A FASTMath -n 128 -t 30 --mode script /home/ghansen/FASTMath/scripts/reserve.sh qstat -
Get the blockid info from qstat and set the environment variable
export MPIEXEC_BLOCKID=VST-20220-31331-32 -
Use the script
/home/ghansen/FASTMath/scripts/run_Albanyto dispatch the parallel job to the compute nodes.export PATH=/home/ghansen/FASTMath/scripts:/home/ghansen/FASTMath/bin run_Albany -np 128 AlbanyT input.xml