Reproducing PR Testing Errors

Note: these steps assume that the system you will be running on has been setup with the proper modules, meets the hardware requirements, and has access to the various git repositories needed for configuration.

Steps to reproduce a Trilinos_PR_* Pull Request Build

Gather the necessary information

1. Identify the first build which failed by navigating to the most recent CDash link posted by the AutoTester:

2. On CDash, identify the build of interest and navigate to the files uploaded within the build by clicking the tan box:

Important: If you do not have access to CDash, please ask someone with access to gather the information you need.

3. Download packageEnables.cmake and copy it to a suitable machine.

   Note: there is a hostname listed at "Site: " at the top of the cdash viewFiles.php page.

Set up the build environment

For the remaining steps, $TRILINOS_BUILD denotes the absolute path to your local Trilinos build directory, and $TRILINOS_SRC denotes the absolute path to your local Trilinos source directory.

4. mkdir $TRILINOS_BUILD

5. Copy packageEnables.cmake from Step 3 to $TRILINOS_BUILD.

6. Checkout the correct commit of Trilinos:

   cd $TRILINOS_SRC
   git checkout <BRANCH>
   

   where <BRANCH> is the source branch of your GitHub PR.

   Important: Ensure that your source branch is up-to-date via git pull.

7. Merge the tip of develop into your source branch

   git merge upstream/develop
   

8. Fetch GenConfig and ini files.

   Important: Ensure your SSH public key is added to gitlab-ex.sandia.gov and cee-gitlab.sandia.gov before running get_dependencies.sh.

   cd /path/to/Trilinos
   module load git
   git --version
   git version 2.29.0
   

   • To use SRN ini files in $TRILINOS_SRC/packages/framework/srn-ini-files:

     ./packages/framework/get_dependencies.sh --srn
     

   • To use SON ini files in $TRILINOS_SRC/packages/framework/son-ini-files:

     ./packages/framework/get_dependencies.sh --son
     

   • To use public ini files in $TRILINOS_SRC/packages/framework/ini-files:

     ./packages/framework/get_dependencies.sh
     

9. Setup the environment and configuration:

   cd $TRILINOS_BUILD
   source $TRILINOS_SRC/packages/framework/GenConfig/gen-config.sh --cmake-fragment gen-config.cmake <BUILD_NAME> $TRILINOS_SRC
   

   where <BUILD_NAME> is the build name from https://github.com/trilinos/Trilinos/wiki/Pull-Request-Testing-Interface.

   Important: If you're encountering a configuration failure, please go to step 11 below.

10. Enable the correct Trilinos packages:

    cd $TRILINOS_BUILD
    rm -rf CMake* && cmake -G Ninja -C packageEnables.cmake -C gen-config.cmake $TRILINOS_SRC
    

    NOTE: See step 3 above for how to acquire packageEnables.cmake

Build the binaries and reproduce the test failures

11. Build and test Trilinos.

    Important: If appropriate, follow the system instructions (typically in /opt/*_INTRO) for acquiring a compute node before running make and ctest.

    make NP=<X>
    ctest -R <TEST_NAME>
    

    where <X> is the number of threads and <TEST_NAME> is a regular expression capturing the tests you're interested in reproducing.

    Note: please respect multi-tenant systems and ensure X does not oversubscribe the system.

12. Verify that the test is failing the same way as reported via CDash.

Update the Trilinos source code

13. Update the Trilinos source code and repeat step 10 (or step 9 if you're encountering a configuration failure) until you're satisfied with the results.
14. Commit your changes and push them to <BRANCH> so the AutoTester can confirm that you have fixed the pull request build.