Skip to content

TAU Integration Example with MPI, ADIOS support

Jump to bottom
Kevin Huck edited this page Dec 20, 2017 · 6 revisions

Overview

The TAU Performance Measurement System has integrated with SOS so that TAU data is automatically aggregated over SOS on demand, periodically, or at the end of execution for single applications, pipelines, coupled applications or workflows. All TAU metadata, timers and counters are aggregated over SOS for the purposes of integrated analysis, runtime monitoring and/or visualization. For more information, see the paper by Wood et. al, A Scalable Observation System for Introspection and In Situ Analytics.

Building TAU with SOS support

The latest TAU release is available for download at http://tau.uoregon.edu/tau.tgz. (For newer versions, please contact the TAU team at the University of Oregon).

Steps for building TAU with SOS support:

Download TAU

wget http://tau.uoregon.edu/tau.tgz
tar -xzf tau.tgz
# the version number may vary
cd tau-2.26.3

Configure TAU

GNU compilers

Add the SOS installation path to the TAU configuration like so:

./configure -sos=/path/to/sos/installation ...

...adding whatever other TAU configuration options you require, such as -mpi, -pthread, -ompt (for OpenMP support), -adios (for ADIOS support), etc. For example, here is a configuration that includes many of the commonly used features in TAU (on a Cray XK7):

    ./configure \
    -arch=craycnl \
    -iowrapper -mpi -pthread -bfd=download -unwind=download \
    -pdt=${pdtdir} \
    -pdt_c++=g++ \
    -prefix=${instdir}/tau \
    -sos=${sosdir} \
    -adios=${ADIOS_DIR} \
    -ompt=download \
    -cuda=${CRAY_CUDATOOLKIT_DIR} \
    -cupti=${CRAY_CUDATOOLKIT_DIR}/extras/CUPTI \
    -papi=${PAPI_PATH}
    make -j4 install

These and other TAU configuration options are explained with ./configure -help, or see the documentation at the TAU website.

PGI compilers

Add the SOS installation path to the TAU configuration like so:

./configure -sos=/path/to/sos/installation ...

...adding whatever other TAU configuration options you require, such as -mpi, -pthread, -ompt (for OpenMP support), -adios (for ADIOS support), etc. For example, here is a configuration that includes many of the commonly used features in TAU (on a Cray XK7):

    ./configure \
    -arch=craycnl \
    -iowrapper -mpi -pthread -bfd=download -unwind=download \
    -pdt=${pdtdir} \
    -pdt_c++=pgc++ \
    -prefix=${instdir}/tau \
    -sos=${sosdir} \
    -adios=${ADIOS_DIR} \
    -openmp \
    -cuda=${CRAY_CUDATOOLKIT_DIR} \
    -cupti=${CRAY_CUDATOOLKIT_DIR}/extras/CUPTI \
    -papi=${PAPI_PATH}
    make -j4 install

These and other TAU configuration options are explained with ./configure -help, or see the documentation at the TAU website.

Adding TAU to an application (link only)

For linking SOS/TAU into your application(s), it is straightforward.

GNU compilers:

For example, on Titan, assume that there is a build of TAU in /ccs/proj/csc143/CODAR_Demo/titan.gnu/tau. Also assume that TAU was built with GNU compilers and with MPI, pthread, SOS and ADIOS support. To link TAU into your example, set the following environment variables before linking your application:

TAU_ROOT=/ccs/proj/csc143/CODAR_Demo/titan.gnu/tau
TAU_ARCH=craycnl
TAU_CONFIG=tau-gnu-mpi-pthread-pdt-sos-adios
export TAU_MAKEFILE=${TAU_ROOT}/${TAU_ARCH}/lib/Makefile.${TAU_CONFIG}
export PATH=${TAU_ROOT}/${TAU_ARCH}/bin:${PATH}

If you just want to link TAU into your application, add the output from tau_cc.sh -tau:showlibs to your list of libraries (your output may be different):

-L/opt/cray/mpt/7.5.2/gni/mpich-gnu/4.9/lib \
-L/ccs/proj/csc143/CODAR_Demo/titan.gnu/tau/craycnl/lib \
-lTauMpi-gnu-mpi-pthread-pdt-sos-adios -lpthread -lrt -lmpich -lrt \
-Wl,-rpath,/opt/cray/mpt/7.5.2/gni/mpich-gnu/4.9/lib \
-L/ccs/proj/csc143/CODAR_Demo/titan.gnu/tau/craycnl/lib \
-ltau-gnu-mpi-pthread-pdt-sos-adios -lpthread -ldl \
-L/ccs/proj/csc143/CODAR_Demo/titan.gnu/tau/craycnl/binutils-2.23.2/lib \
-L/ccs/proj/csc143/CODAR_Demo/titan.gnu/tau/craycnl/binutils-2.23.2/lib64 \
-Wl,-rpath,/ccs/proj/csc143/CODAR_Demo/titan.gnu/tau/craycnl/binutils-2.23.2/lib \
-Wl,-rpath,/ccs/proj/csc143/CODAR_Demo/titan.gnu/tau/craycnl/binutils-2.23.2/lib64 \
-lbfd -liberty -lz -lrt \
-L/ccs/proj/csc143/CODAR_Demo/titan.gnu/tau/craycnl/libunwind-1.1-cc-GNU/lib \
-lunwind \
-Wl,-rpath,/ccs/proj/csc143/CODAR_Demo/titan.gnu/tau/craycnl/libunwind-1.1-cc-GNU/lib \
-L/ccs/proj/csc143/CODAR_Demo/titan.gnu/sos_flow/lib \
-Wl,-rpath,/ccs/proj/csc143/CODAR_Demo/titan.gnu/sos_flow/lib \
-lsos -pthread -fPIC \
-L/ccs/proj/csc143/CODAR_Demo/titan.gnu/ADIOS/ADIOS-gcc/lib \
-Wl,-rpath,/ccs/proj/csc143/CODAR_Demo/titan.gnu/ADIOS/ADIOS-gcc/lib \
-pthread -fPIC \
-L/opt/cray/mpt/7.5.2/gni/mpich-gnu/4.9/lib \
-lpthread -lrt -lmpich -lrt \
-Wl,-rpath,/opt/cray/mpt/7.5.2/gni/mpich-gnu/4.9/lib \
-ldl -lm -L/lib -lstdc++ -lgcc_eh

PGI compilers:

For example, on Titan, assume that there is a build of TAU in /ccs/proj/csc143/tau_mona/titan.pgi/tau. Also assume that TAU was built with PGI compilers and with MPI, pthread, SOS and ADIOS support. To link TAU into your example, set the following environment variables before linking your application:

TAU_ROOT=/ccs/proj/csc143/tau_mona/titan.pgi/tau
TAU_ARCH=craycnl
TAU_CONFIG=tau-mpi-pthread-pdt-sos-adios-pgi
export TAU_MAKEFILE=${TAU_ROOT}/${TAU_ARCH}/lib/Makefile.${TAU_CONFIG}
export PATH=${TAU_ROOT}/${TAU_ARCH}/bin:${PATH}

If you just want to link TAU into your application, add the output from tau_cc.sh -tau:showlibs to your list of libraries (your output may be different):

-L/opt/cray/mpt/7.5.2/gni/mpich-pgi/15.3/lib \
-L/ccs/proj/csc143/tau_mona/titan.pgi/tau/craycnl/lib \
-lTauMpi-mpi-pthread-pdt-sos-adios-pgi -lpthread -lrt -lmpichcxx_pgi -lmpich_pgi -lrt \
-Wl,-rpath,/opt/cray/mpt/7.5.2/gni/mpich-pgi/15.3/lib \
-L/ccs/proj/csc143/tau_mona/titan.pgi/tau/craycnl/lib \
-ltau-mpi-pthread-pdt-sos-adios-pgi -lpthread -ldl \
-L/ccs/proj/csc143/tau_mona/titan.pgi/tau/craycnl/binutils-2.23.2/lib \
-L/ccs/proj/csc143/tau_mona/titan.pgi/tau/craycnl/binutils-2.23.2/lib64 \
-Wl,-rpath,/ccs/proj/csc143/tau_mona/titan.pgi/tau/craycnl/binutils-2.23.2/lib \
-Wl,-rpath,/ccs/proj/csc143/tau_mona/titan.pgi/tau/craycnl/binutils-2.23.2/lib64 \
-lbfd -liberty -lz -lrt \
-L/ccs/proj/csc143/tau_mona/titan.pgi/tau/craycnl/libunwind-1.1-cc-PGI/lib -lunwind \
-Wl,-rpath,/ccs/proj/csc143/tau_mona/titan.pgi/tau/craycnl/libunwind-1.1-cc-PGI/lib \
-L/ccs/proj/csc143/tau_mona/titan.pgi/sos_flow/lib \
-Wl,-rpath,/ccs/proj/csc143/tau_mona/titan.pgi/sos_flow/lib -lsos -fPIC \
-L/ccs/proj/csc143/tau_mona/titan.pgi/ADIOS/ADIOS-pgi/lib \
-Wl,-rpath,/ccs/proj/csc143/tau_mona/titan.pgi/ADIOS/ADIOS-pgi/lib \
-fPIC -ldl -lm -L/opt/pgi/17.7.0/linux86-64/17.7/bin/../lib -lstdc++ -lstdc++

Or, in your Makefile you can add the following to the list of libraries:

TAU_LIBRARIES=$(shell tau_cc.sh -tau:showlibs)

If you are using ADIOS, change this line in your Makefile/configuration from this:

ADIOS_LIBS=$(shell ${ADIOSDIR}/bin/adios_config -l -f)

to this:

ADIOS_LIBS=$(shell tau_cc.sh -tau:showlibs) $(shell ${ADIOSDIR}/bin/adios_config -l -f)

Making sure that the TAU libraries are before the ADIOS libraries (due to the fact that ADIOS is a static library, and TAU needs to overwrite one weak symbol in the ADIOS library). The weak symbol overwriting happens if the strong definition is resolved in the TAU library before the weak one in the ADIOS library. For examples, see this Makefile example and/or this shell script example.

For more information on how to use TAU, see the TAU website.

Clone this wiki locally