Build

Dependencies

Dependency	Version	Required	Runtime
Kokkos	4.3.00	✔️	✔️
MPI	2		✔️
CMake	3.16	✔️

Build and installation

The build instructions for the Kokkos library can be found here. ArborX requires Kokkos CMake build to have Kokkos_ENABLE_CUDA_LAMBDA=ON if Kokkos_ENABLE_CUDA=ON.

Example: OLCF Frontier

To build Kokkos on Frontier (AMD EPYC 3rd gen CPU and AMD 250X) with three backends (Serial, OpenMP and HIP), configure it with

OPTIONS=(
    -D CMAKE_INSTALL_PREFIX="${KOKKOS_INSTALL_DIR}"
    -D CMAKE_CXX_COMPILER=hipcc
    -D Kokkos_ENABLE_SERIAL=ON
    -D Kokkos_ENABLE_OPENMP=ON
    -D Kokkos_ENABLE_HIP=ON
        -D Kokkos_ENABLE_HIP_MULTIPLE_KERNEL_INSTANTIATIONS=ON
    -D Kokkos_ARCH_ZEN2=ON
    -D Kokkos_ARCH_VEGA90A=ON
    )
cmake "${OPTIONS[@]}" "${KOKKOS_SOURCE_DIR:-../}"

Example: ALCF Aurora

To build Kokkos on Aurora (Intel Xeon Saphire Rapids with Intel Pointe Vecchio) with two backends (OpenMP and SYCL), configure it with

OPTIONS=(
    -D CMAKE_INSTALL_PREFIX="${KOKKOS_INSTALL_DIR}"
    -D CMAKE_CXX_COMPILER=icpx
    -D CMAKE_CXX_FLAGS="-ffp-model=precise -fsycl-device-code-split=per_kernel"
    -D CMAKE_EXE_LINKER_FLAGS="-ffp-model=precise"
    -D Kokkos_ENABLE_OPENMP=ON
    -D Kokkos_ENABLE_SYCL=ON
    -D Kokkos_ARCH_INTEL_PVC=ON
    )
cmake "${OPTIONS[@]}" "${KOKKOS_SOURCE_DIR:-../}"

Example: NERSC Perlmutter

To build Kokkos on Perlmutter (AMD EPYC 7763 CPU with Nvidia A100 GPU) with two backends (OpenMP and CUDA), configure it with

OPTIONS=(
    -D CMAKE_INSTALL_PREFIX="${KOKKOS_INSTALL_DIR}"
    -D CMAKE_CXX_EXTENSIONS=OFF
    -D CMAKE_CXX_COMPILER="${KOKKOS_SOURCE_DIR}/bin/nvcc_wrapper"
    -D Kokkos_ENABLE_OPENMP=ON
    -D Kokkos_ENABLE_CUDA=ON
        -D Kokkos_ENABLE_CUDA_LAMBDA=ON
    -D Kokkos_ARCH_ZEN2=ON
    -D Kokkos_ARCH_AMPERE80=ON
    )
cmake "${OPTIONS[@]}" "${KOKKOS_SOURCE_DIR:-../}"

Note: the actual architecture is Zen3, however the compiler support at the moment is spotty. The current successful build uses gcc/9.3.0 with cuda/11.3.0.

CMake

Assuming that Kokkos library is installed in $KOKKOS_INSTALL_DIR, configure ArborX as

OPTIONS=(
    -D CMAKE_INSTALL_PREFIX="${ARBORX_INSTALL_DIR}"
    -D ARBORX_ENABLE_MPI=ON
    -D Kokkos_ROOT="${KOKKOS_INSTALL_DIR}"
    -D CMAKE_CXX_COMPILER="${KOKKOS_INSTALL_DIR}/bin/nvcc_wrapper" # only for CUDA-enabled Kokkos
    -D CMAKE_CXX_COMPILER=hipcc # for HIP-enabled Kokkos
    -D CMAKE_CXX_COMIPLER=icpx  # for SYCL-enabled Kokkos
    -D CMAKE_CXX_EXTENSIONS=OFF # required by Kokkos
    )
cmake "${OPTIONS[@]}" "${ARBORX_DIR:-../}"

then run make install. ArborX will then be installed in $ARBORX_INSTALL_DIR.

To disable MPI, configure with

    -D ARBORX_ENABLE_MPI=OFF

For GPU-aware MPI, configure with

    -D ARBORX_ENABLE_GPU_AWARE_MPI=ON

Note

On Frontier, this requires additionally loading craype-accel-amd-gfx90a and adding MPICH_GPU_SUPPORT_ENABLED=1 environment variable

To enable ArborX examples, configure with

    -D ARBORX_ENABLE_EXAMPLES=ON

To validate the ArborX build, install Boost (versions 1.67, 1.68, or 1.75+) ($BOOST_INSTALL_DIR), configure with

    -D Kokkos_ROOT="${KOKKOS_INSTALL_DIR}"
    -D Boost_ROOT="${BOOST_INSTALL_DIR}"
    -D ARBORX_ENABLE_TESTS=ON

and run ctest after completing the build.

To enable ArborX benchmarks, install Google Benchmark version 1.5.4 or later ($BENCHMARK_INSTALL_DIR), Boost version 1.67 or later ($BOOST_INSTALL_DIR), configure with

    -D Kokkos_ROOT="${KOKKOS_INSTALL_DIR}"
    -D Boost_ROOT="${BOOST_INSTALL_DIR}"
    -D benchmark_ROOT="${BENCHMARK_INSTALL_DIR}"
    -D ARBORX_ENABLE_BENCHMARKS=ON

The individual benchmarks can then be run from benchmarks directory.

Note

If benchmark is not not found, but ARBORX_ENABLE_BENCHMARKS=ON, it will be downloaded and built automatically.

ArborX also supports building against Kokkos built as part of Trilinos. This requires Trilinos hash de15ca5352 or later. In this case, $KOKKOS_INSTALL_DIR should point to the Trilinos installation directory.

Spack

ArborX can also be installed using the Spack package manager. A basic installation can be done as:

spack install arborx

Spack allows options and compilers to be turned on in the install command:

spack install [email protected] %[email protected] ~mpi+cuda cuda_arch=70

This example illustrates the most common parameters:

• Variants are specified with ~ or + (e.g. +cuda) and enable or disable certain options
• Version (@version) immediately follows arborx and can specify a specific ArborX version
• Compiler (%compiler) can be set if an exact compiler is desired; otherwise, a default compiler is chosen

Note

Building with CUDA (+cuda) requires cuda_arch speicification. Building with ROCm (+rocm) requires amdgpu_target specification.

For a complete list of available ArborX options, run:

spack info arborx

Building against ArborX

For projects using CMake, add to CMakeLists.txt

find_package(ArborX REQUIRED)

For any targets requiring ArborX, add

target_link_libraries(<target> ArborX::ArborX)

Note that because of the way CMake deals with dependencies, users of ArborX will need to make sure that Kokkos installation directory is visible to CMake. The easiest way to address it is to add both to the configuration options:

    -D CMAKE_PREFIX_PATH="$ARBORX_INSTALL_DIR;$KOKKOS_INSTALL_DIR"