-
Notifications
You must be signed in to change notification settings - Fork 90
Rhel6 using Gcc
Glen Hansen edited this page Feb 11, 2019
·
6 revisions
This set of build instructions is for the Albany suite on Rhel6 linux.
The compilers are gcc 5.2.0, OpenMPI 1.8.5, and RHEL 6 linux.
Modules on CEE:
module load sierra-devel/gcc-5.2.0-openmpi-1.8.8
module load sierra-mkl/16.0-2016.2.181
Trilinos requires a set of "third party libraries" (TPLs) to support Albany. The actual libraries needed depends on the final Albany configuration desired. These instructions build a superset, all the libraries needed to support any Albany configuration.
Notes: In this example
- The TPLs are installed in
/projects/albany
-
Version 1.2.8
CC=mpicc CFLAGS=-O3 ./configure --64 --archs="-arch x86_64" --prefix=/projects/albany make -j 4 make install
-
Version 1.8.17
./configure CC=mpicc FC=mpifort CXX=mpicxx CXXFLAGS="-O3 -fPIC -march=native" \ CFLAGS="-O3 -fPIC -march=native" \ FCFLAGS="-O3 -fPIC -march=native" --enable-parallel \ --with-zlib=/projects/albany --prefix=/projects/albany make -j 4 make install
-
Version 1.7.0
-
Configure and build:
./configure --prefix=/projects/albany MPICC=mpicc MPIF77=mpifort MPIF90=mpifort MPICXX=mpicxx \ CXXFLAGS="-fPIC -O3 -march=native" CFLAGS="-fPIC -O3 -march=native" \ F77FLAGS="-fPIC -O3 -march=native" F90FLAGS="-fPIC -O3 -march=native" make -j 8 make install
-
Version 4.4.0
-
Edit the file
include/netcdf.h. Make the following edits near line 265:#define NC_MAX_DIMS 65536 /* max dimensions per file */ #define NC_MAX_ATTRS 8192 #define NC_MAX_VARS 524288 /* max variables per file */ #define NC_MAX_NAME 256 #define NC_MAX_VAR_DIMS 8 /* max per variable dimensions */ -
Now, configure using CMake and build:
./configure CC=mpicc FC=mpifort CXX=mpicxx \ CXXFLAGS="-fPIC -I/projects/albany/include -O3 -march=native" \ CFLAGS="-fPIC -I/projects/albany/include -O3 -march=native" \ LDFLAGS="-fPIC -L/projects/albany/lib -O3 -march=native" \ FCFLAGS="-fPIC -I/projects/albany/include -O3 -march=native" \ --prefix=/projects/albany --disable-doxygen --enable-netcdf4 --enable-pnetcdf make -j 8 make install
-
Version 1.55.0
-
Build boost with the commands
echo "using mpi : /sierra/sntools/SDK/mpi/openmpi/1.8.5-gcc-5.2.0-RHEL6/bin/mpicxx ;" >> ./tools/build/v2/user-config.jam echo "using gcc : /sierra/sntools/SDK/compilers/gcc/5.2.0-RHEL6/bin/g++ ;" >> ./tools/build/v2/user-config.jam ./bootstrap.sh --with-libraries=signals,regex,filesystem,system,mpi,serialization,thread,program_options,exception --prefix=/projects/albany ./b2 -j 4 ./b2 -j 4 install
-
Version 4.0.3
wget http://glaros.dtc.umn.edu/gkhome/fetch/sw/parmetis/parmetis-4.0.3.tar.gz tar -xvf parmetis-4.0.3.tar.gz -
Edit
metis.hto use anIDXTYPEWIDTHof 64cd parmetis-4.0.3/metis/include vi metis.h #define IDXTYPEWIDTH 64 #define REALTYPEWIDTH 64 -
Run CMake to build
parmetiscd build export metis=/projects/albany/src/parmetis-4.0.3/metis/ cmake \ -DCMAKE_INSTALL_PREFIX=/projects/albany/parmetis-4.0.3 \ -DMETIS_PATH=$metis \ -DGKLIB_PATH=$metis/GKlib \ -DCMAKE_C_COMPILER=mpicc \ -DCMAKE_CXX_COMPILER=mpicxx \ -DCMAKE_C_FLAGS=-O3 \ -DCMAKE_CXX_FLAGS=-O3 \ .. make -j 4 make install cp libmetis/libmetis.a /projects/albany/parmetis-4.0.3/lib cp $metis/include/metis.h /projects/albany/parmetis-4.0.3/include
-
Version 4.3
-
Build paths
mkdir /projects/albany/SuperLU_4.3 mkdir /projects/albany/SuperLU_4.3/include mkdir /projects/albany/SuperLU_4.3/lib -
Edit
make.incfor your machine and environmentPLAT = _linux SuperLUroot = /projects/albany/SuperLU_4.3 SUPERLULIB = $(SuperLUroot)/lib/libsuperlu_4.3.a TMGLIB = libtmglib.a BLASDEF = -DUSE_VENDOR_BLAS BLASLIB = -L/sierra/sntools/SDK/compilers/intel/composer_xe_2016.3.210/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core LIBS = $(SUPERLULIB) $(BLASLIB) ARCH = ar ARCHFLAGS = cr RANLIB = ranlib CC = gcc CFLAGS = -O3 NOOPTS = FORTRAN = gfortran FFLAGS = -O3 LOADER = $(CC) LOADOPTS = CDEFS = -DAdd_ MATLAB = /usr/sww/matlab -
Build SuperLU
make -j 4 cd SRC cp *.h /projects/albany/SuperLU_4.3/include
-
HWLOC is an optional package that can be used for thread pinning by Kokkos. If you are not building Kokkos, or if you don't care about thread pinning, please skip.
-
Version 1.8
./configure CC=mpicc CXX=mpicxx --prefix=/projects/albany make -j 4 make install
Notes: In this example
-
The Trilinos source is installed in
/home/gahanse/Codes/Trilinos, using a command likecd /home/gahanse/Codes git clone [email protected]:trilinos/Trilinos.git -
The SCOREC tools are installed as a subdirectory of Trilinos, and are build automatically by the Trilinos build process. The commands to do this are
cd /home/gahanse/Codes/Trilinos git clone https://github.com/SCOREC/core.git SCOREC -
do-configure script
#!/bin/sh # export TRILINOS_HOME=/projects/albany/src/Trilinos BUILD_DIR=`pwd` INSTALL_DIR=/projects/albany/trilinos/MPI_REL BOOST_DIR=/projects/albany LIB_DIR=/projects/albany MPI_BASE_DIR=/sierra/sntools/SDK/mpi/openmpi/1.8.8-gcc-5.2.0-RHEL6 NETCDF=/projects/albany HDFDIR=/projects/albany PARMETISDIR=/projects/albany MKL_PATH=/sierra/sntools/SDK/compilers/intel/composer_xe_2016.3.210 HWLOC_PATH=/projects/albany LABLAS_LIBRARIES="-L$MKL_PATH/lib/intel64 -Wl,--start-group $MKL_PATH/mkl/lib/intel64/libmkl_intel_lp64.a $MKL_PATH/mkl/lib/intel64/libmkl_core.a $MKL_PATH/mkl/lib/intel64/libmkl_sequential.a -Wl,--end-group" cmake \ -Wno-dev \ -D Trilinos_CONFIGURE_OPTIONS_FILE:FILEPATH=$TRILINOS_HOME/sampleScripts/AlbanySettings.cmake \ -D Trilinos_ENABLE_SCOREC:BOOL=ON \ -D SCOREC_DISABLE_STRONG_WARNINGS:BOOL=ON \ -D CMAKE_BUILD_TYPE:STRING=NONE \ -D CMAKE_CXX_FLAGS:STRING="-O3 -march=native -DNDEBUG" \ -D CMAKE_C_FLAGS:STRING="-O3 -march=native -DNDEBUG" \ -D CMAKE_Fortran_FLAGS:STRING="-O3 -march=native -DNDEBUG" \ -D Rythmos_ENABLE_DEBUG:BOOL=ON \ -D KokkosClassic_DefaultNode:STRING="Kokkos::Compat::KokkosSerialWrapperNode" \ -D Trilinos_ENABLE_EXPLICIT_INSTANTIATION:BOOL=ON \ -D Tpetra_INST_INT_LONG_LONG:BOOL=ON \ -D Tpetra_INST_INT_INT:BOOL=ON \ -D Tpetra_INST_DOUBLE:BOOL=ON \ -D Tpetra_INST_FLOAT:BOOL=OFF \ -D Tpetra_INST_COMPLEX_FLOAT:BOOL=OFF \ -D Tpetra_INST_COMPLEX_DOUBLE:BOOL=OFF \ -D Tpetra_INST_INT_LONG:BOOL=OFF \ -D Tpetra_INST_INT_UNSIGNED:BOOL=OFF \ -D Trilinos_ENABLE_Kokkos:BOOL=ON \ -D Trilinos_ENABLE_KokkosCore:BOOL=ON \ -D Phalanx_KOKKOS_DEVICE_TYPE:STRING="SERIAL" \ -D Phalanx_INDEX_SIZE_TYPE:STRING="INT" \ -D Phalanx_SHOW_DEPRECATED_WARNINGS:BOOL=OFF \ -D Kokkos_ENABLE_Serial:BOOL=ON \ -D Kokkos_ENABLE_OpenMP:BOOL=OFF \ -D Kokkos_ENABLE_Pthread:BOOL=OFF \ \ -D TPL_ENABLE_MPI:BOOL=ON \ -D MPI_BASE_DIR:PATH=$MPI_BASE_DIR \ \ -D TPL_ENABLE_Pthread:BOOL=OFF \ \ -D TPL_ENABLE_Boost:BOOL=ON \ -D TPL_ENABLE_BoostLib:BOOL=ON \ -D TPL_ENABLE_BoostAlbLib:BOOL=ON \ -D Boost_INCLUDE_DIRS:PATH=$BOOST_DIR/include \ -D Boost_LIBRARY_DIRS:PATH=$BOOST_DIR/lib \ -D BoostLib_INCLUDE_DIRS:PATH=$BOOST_DIR/include \ -D BoostLib_LIBRARY_DIRS:PATH=$BOOST_DIR/lib \ -D BoostAlbLib_INCLUDE_DIRS:PATH=$BOOST_DIR/include \ -D BoostAlbLib_LIBRARY_DIRS:PATH=$BOOST_DIR/lib \ \ -D TPL_ENABLE_Netcdf:STRING=ON \ -D Netcdf_INCLUDE_DIRS:PATH=$NETCDF/include \ -D Netcdf_LIBRARY_DIRS:PATH=$NETCDF/lib \ \ -D TPL_ENABLE_HDF5:STRING=ON \ -D HDF5_INCLUDE_DIRS:PATH=$HDFDIR/include \ -D HDF5_LIBRARY_DIRS:PATH=$HDFDIR/lib \ \ -D TPL_ENABLE_Zlib:STRING=ON \ -D Zlib_INCLUDE_DIRS:PATH=$HDFDIR/include \ -D Zlib_LIBRARY_DIRS:PATH=$HDFDIR/lib \ \ -D TPL_ENABLE_BLAS:STRING=ON \ -D TPL_ENABLE_LAPACK:STRING=ON \ -D TPL_BLAS_LIBRARIES:STRING="$LABLAS_LIBRARIES" \ -D TPL_LAPACK_LIBRARIES:STRING="$LABLAS_LIBRARIES" \ \ -D TPL_ENABLE_ParMETIS:STRING=ON \ -D ParMETIS_INCLUDE_DIRS:PATH="${PARMETISDIR}/include" \ -D ParMETIS_LIBRARY_DIRS:PATH="${PARMETISDIR}/lib" \ \ -D CMAKE_INSTALL_PREFIX:PATH=$INSTALL_DIR \ -D Trilinos_EXTRA_LINK_FLAGS:STRING="-L${HDFDIR}/lib -lnetcdf -lhdf5_hl -lhdf5 -lz -lgfortran -Wl,-rpath,/projects/albany/lib" \ \ -D Trilinos_ENABLE_Moertel:BOOL=ON \ -D Moertel_ENABLE_EXAMPLES:BOOL=ON \ -D Moertel_ENABLE_Morkon:BOOL=ON \ -D Trilinos_ENABLE_TriKota:BOOL=OFF \ -D HAVE_INTREPID_KOKKOSCORE:BOOL=ON \ -D Trilinos_ENABLE_SECONDARY_TESTED_CODE=OFF \ -D Trilinos_ENABLE_Amesos2:BOOL=ON \ -D Amesos2_ENABLE_KLU2:BOOL=ON \ \ $EXTRA_ARGS \ ${TRILINOS_HOME}
To build Trilinos, use an out-of-source build. I usually make a build directory that matches the configuration of Trilinos that I am building in the Trilinos root directory:
cd /home/gahanse/Codes/Trilinos
mkdir build
cd build
mkdir MPI_REL
-
Place the above do-configure file into the MPI_REL directory, change directories into MPI_REL, and perform the configure, build, and install:
cd MPI_REL ./do-configure make -j 4 make -j 4 install
-
CMake configure script:
#!/bin/sh export ALB_64BIT_INT=OFF export ALB_EPETRA=ON export ALB_ENABLE_SCOREC=ON export ALB_ENABLE_LCM=ON export ALB_ENABLE_LAME=OFF export ALB_ENABLE_HYDRIDE=OFF export ALB_ENABLE_QCAD=OFF export ALB_ENABLE_MOR=OFF export ALB_ENABLE_ASCR=OFF export ALB_ENABLE_AERAS=ON export ALB_ENABLE_FELIX=ON export ALB_ENABLE_AlbanyCI=OFF export ALB_DEBUG=OFF export ALB_ENABLE_CHECK_FPE=OFF export ALB_ENABLE_SPECULATIVE=OFF export ALB_ENABLE_SG_MP=OFF export ALB_ENABLE_PERF_TESTS=ON export TRILINOS_INSTALL_DIR=/projects/albany/trilinos/MPI_REL export ALB_VERBOSE=OFF cmake \ -D ALBANY_TRILINOS_DIR:FILEPATH="$TRILINOS_INSTALL_DIR" \ -D ENABLE_LCM:BOOL=${ALB_ENABLE_LCM} \ -D ENABLE_AERAS:BOOL=${ALB_ENABLE_AERAS} \ -D ENABLE_QCAD:BOOL=${ALB_ENABLE_QCAD} \ -D ENABLE_HYDRIDE:BOOL=${ALB_ENABLE_HYDRIDE} \ -D ENABLE_LCM_SPECULATIVE:BOOL=${ALB_ENABLE_SPECULATIVE} \ -D CMAKE_VERBOSE_MAKEFILE:BOOL=${ALB_VERBOSE} \ -D ENABLE_DEBUGGING:BOOL=${ALB_DEBUG} \ -D ENABLE_CHECK_FPE:BOOL=${ALB_ENABLE_CHECK_FPE} \ -D ENABLE_SCOREC:BOOL=${ALB_ENABLE_SCOREC} \ -D ENABLE_FELIX:BOOL=${ALB_ENABLE_FELIX} \ -D ENABLE_MOR:BOOL=${ALB_ENABLE_MOR} \ -D ENABLE_ALBANY_CI:BOOL=${ALB_ENABLE_AlbanyCI} \ -D ENABLE_ASCR:BOOL=${ALB_ENABLE_ASCR} \ -D ENABLE_SG_MP:BOOL=${ALB_ENABLE_SG_MP} \ -D ENABLE_PERFORMANCE_TESTS:BOOL=${ALB_ENABLE_PERF_TESTS} \ -D ALBANY_CTEST_TIMEOUT:INTEGER=70 \ -D ENABLE_64BIT_INT:BOOL=${ALB_64BIT_INT} \ -D ENABLE_ALBANY_EPETRA_EXE:BOOL=${ALB_EPETRA} \ \ ..
Clone the Albany repo in a suitable place:
cd /home/ghansen/Codes
git clone https://github.com/SNLComputation/Albany.git
-
Create an out-of-source build directory
cd Albany mkdir build -
Place the above configure script into the build directory, then configure, build, and test
cd build ./buildAlbany make -j 4 ctest