MALI Development Overview

The MPAS-Albany Land Ice (MALI) ice sheet model is 3d higher-order ice sheet model developed by the US Department of Energy at Los Alamos National Laboratory and Sandia National Laboratories. The model makes use of the MPAS and Albany modeling frameworks. MALI is the ice-sheet component of the Energy Exascale Earth System Model (E3SM), and as of May 2021, MALI source code is developed within the E3SM repository. The MALI-Dev GitHub organization is the primary location for MALI development and uses a fork of the primary E3SM repository. This document provide guidelines for working with MALI in the development fork.

Code Organization

• MALI code is located within the E3SM repository at components/mpas-albany-landice/src.
• MALI can be compiled as a standalone executable independent of E3SM by using the Makefile located at components/mpas-albany-landice/Makefile. Including CORE=landice is no longer required/supported. Example using gcc in DEBUG mode and including Albany: make gfortran ALBANY=true DEBUG=true
• Directions for how to compile on machines currently supported can be found within the COMPASS documentation.
• Prior to May 2021, MALI lived with the MPAS-Dev/MPAS-Model repository. Old Pull Requests can be found there by using the Land Ice PR label.

Development Workflow and Opening a Pull Request

• The primary branch used for collecting features branches in MALI-Dev/E3SM is develop. The MALI team will do our development here and then periodically update MALI in E3SM by opening Pull Requests for merging MALI-Dev/E3SM back into master on E3SM-Project/E3SM.
• MALI development team members can push branches directly to the MALI-Dev/E3SM repository. You can also choose to work in a fork if you prefer. (Note that forks of this repo are also forks of E3SM-Project/E3SM.)
• Branches for new features should always be made off of develop.
• Name your branch $user/mali/$feature where $user is your Github username and $feature is a short phrase describing your feature. Example: matthewhoffman/mali/hydro_channel_CFL. This naming convention is required by E3SM, and we have to follow it in our fork of E3SM to ensure uniformity when we merge MALI developments back to the primary E3SM repo.
• Important: When doing your MALI development, your code changes should only touch the components/mpas-albany-landice directory. If you find you need to modify other files, your branch needs to be coordinated with E3SM development and a Pull Request to the primary E3SM repo is likely the more appropriate workflow.
• When you work is done or ready for discussion open a new Pull Request with the green button in the upper right here: https://github.com/MALI-Dev/E3SM/pulls. Make sure you set the following:
  • Base Repository: MALI-Dev/E3SM (Note: depending on how you cloned your local repository, this may default to the E3SM-Project/E3SM version of the repo, which is not what you want!)
  • Base Branch: develop
  • Head Repository: If you are working in a fork, it's your fork. If you are working in the shared repository then it's MALI-Dev/E3SM.
  • Compare: your branch name
• After opening a PR assign a reviewer and tags as appropriate. If your branch is not ready for review, but you want to open the Pull Request to facilitate discussion, add the "in progress" label.
• The reviewer should get a notification that the PR is awaiting review, but it is best to touch base with them via Slack, email, etc., about it. However try to keep most discussion of the PR itself to the PR page comments to save them for future reference.

COMPASS Automated Testing System

Configuration Of Model for Prediction Across Scales Setups (COMPASS) is an automated system to set up test cases for MALI and MPAS-Ocean. COMPASS is currently undergoing a transition to conda package. Detailed instructions are available here. Nightly testing of MALI is performed on Cori three times per week. The test suite should also be run manually before merging any PRs to develop. Instructions for this will be added here once the COMPASS workflow has stabilized.

Testing MALI with COMPASS

Setting up or updating COMPASS

Note: Eventually we will do day to day testing using a COMPASS release, but for now we use the head of the COMPASS development branch. Using a compass release would replace these steps. See COMPASS Quick Start for Developers for details of developing with COMPASS. For MALI we currently use the option for "only the compass environment", because we use our own pre-compiled dependency libraries for MPAS, but eventually we could use the standard COMPASS env approach that loads or compiles things like PIO for you.

1. Optionally manually install Mambaforge into your home directory somewhere. Mambaforge is a convenient installation of both the conda and mamba tools for package and environment management. Below, we will assume the path where you installed mambaforge is $CONDADIR.
2. Clone COMPASS if you haven't already: git clone [email protected]:MPAS-Dev/compass.git
3. Update COMPASS repo if you haven't already (from within COMPASS dir): git fetch --all; git checkout origin/main Note: If you are performing development of COMPASS itself, you would obviously check out your branch.
4. Create the COMPASS conda environment, if you have not already: ./conda/configure_compass_env.py --conda $CONDADIR --env_only If you do not have a conda/mambaforge installation from step 1, compass will automatically install it for you at $CONDADIR.
5. Load the COMPASS conda environment created in the previous step, if you have not already: source ./load_dev_compass_<version>.sh

Establish a COMPASS baseline against which your branch can be compared

Note: Eventually we will have maintained baselines on each machine you could use instead.

1. Compile a baseline executable of MALI. This should generally be MALI-Dev/develop. COMPASS includes a submodule for MALI in the MALI-Dev directory. It points to our copy of E3SM: https://github.com/MALI-Dev/E3SM. COMPASS supports using a baseline exe here, or you can use a separate MALI repo. To follow the standard approach of using this location and if you are on Perlmutter in your compass directory, the steps are: source load_dev_compass_1.2.0-alpha.9_pm-cpu_gnu_mpich_albany.sh; cd MALI-Dev; git fetch --all -p; git checkout origin/develop; cd components/mpas-albany-landice/; git log to make sure you are the head of MALI-Dev/develop before compiling; make gnu-cray DEBUG=true ALBANY=true. Note: For COMPASS testing it is recommended to always compile in DEBUG mode, because it can catch some errors that optimized mode does not.
2. Use COMPASS to set up the integration test suite in your scratch space, e.g. on Perlmutter: compass suite -s -c landice -t full_integration -w $PSCRATCH/COMPASS_REGRESSION/BASELINE Note: if you did not build the baseline exe within the MALI submodule included in COMPASS, you need to specify the path to the directory containing your executable using the -p option. Do not specify the name of the executable itself.
3. Go to the directory where the test suite was set up, e.g.: cd $PSCRATCH/COMPASS_REGRESSION/BASELINE
4. Edit the automatically generated job script, job_script.full_integration.sh . full_integration will generally run in a debug job, but the job script defaults to an hour-long job in the regular queue. Submit the job script with sbatch job_script.full_integration.sh. If you'd rather run the suite on an interactive node, you can get that node using salloc -N 1 -q interactive -t 01:00:00, then load the COMPASS env using source load_compass_env.sh, and then run the suite with compass run.
5. Information will be written to a log file compass_full_integration.o${jobid} (or to the screen if using an interactive node) about each test. More verbose log files for each test can be found in the case_outputs subdirectory in the working directory. The entire test suite currently takes about 10–20 min on Perlmutter. I think there is a lot of unnecessary and slow I/O that could be improved to reduce that number significantly. Note that occasionally the Spack environment will take a long time to load (sometimes takes up most of a debug job). As of March 2024, we don't know the reason for this, but the issue usually goes away on its own within a few hours.

Running the test suite on your branch

Note: For integrating a branch, it is best to run the test on a test merge of your branch intoMALI-Dev/develop. For initial testing, you can run on the head of your branch.

1. Compile a testing executable of MALI from your branch. Follow step 1 in the previous section or use your own independent clone of the repository
2. Set up the testing version of the integration suite. This follows step 2 in the previous section, but the working directory changes and you include an additional argument to the path of the baseline result that was run in the previous section, e.g.: compass suite -s -c landice -t full_integration -b $PSCRATCH/COMPASS_REGRESSION/BASELINE -w $PSCRATCH/COMPASS_REGRESSION/TESTING
3. Submit the batch script or run using an interactive compute node, as above.
4. Check output once the suite is complete. Note that the test results report now includes a "baseline comparison" line that looks for answer changes relative to the baseline. If all tests pass, the branch is clear to be merged. If any failures come up, they should be resolved before merging.