Skip to content

Building GEOS with latest modifications to GOCART for the Carbon group

Jump to bottom
Sourish Basu edited this page Aug 27, 2024 · 26 revisions

Checking out and building the model

Setting up GitHub credentials

Checking out GEOS

Much of this is already documented here, only the essential steps are replicated here for convenience.

Setting up the environment

First, you need to have the correct modules to check out the code. If you are expert user, make sure you have git and mepo in your path. If you're not, execute the following to get them (as well as anything else you might need).

module purge
module use -a /discover/swdev/gmao_SIteam/modulefiles-SLES15
module load GEOSenv

Getting the code

Then, decide where you want to set up the model code. While you can check it out in your home directory, we recommend checking it out on a scratch space because it can get pretty big, especially when you are testing multiple different model versions. On NCCS, you typically want to use /discover/nobackup/${USER}. Somewhere in that folder, check out the model with

git clone -b v11.5.2 [email protected]:GEOS-ESM/GEOSgcm.git GEOSgcm-v11.5.2

Note that 11.5.2 simply happens to be the latest released tag at the time this is being written. There is nothing sacred about that tag. If you want a later tag, you can find all release versions here.

The model consists of code in several sub-repositories, by default none of which are checked out. There is a file called components.yaml in the source tree you just checked out, which contains the tags for each repository that will be checked out. Save a copy of this file somewhere, say as components.yaml.orig. Then add the following block to check out RRG:

RRG:
  local: ./src/Components/@GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/@GEOSchem_GridComp/@RRG
  remote: ../RRG.git
  branch: main
  develop: develop

In addition, both GEOSchem_GridComp and GOCART will need to be changed to branches that contain the latest GOCART code. In components.yaml, change the tag line following GOCART to branch: feature/sbasu1/gocart+11.5.2, and the tag line following GEOSchem_GridComp to branch: feature/sbasu1/gocart+11.5.2. Note that these are different repositories, although the branch names are identical.

Now issue mepo clone at the command line to check out all the repositories at the branches/tags in components.yaml.

Building the code

It is best to build the code on a compute node of the same architecture as the ones you will be running the model on. For this example we will be building and running on AMD Milan nodes, so get a terminal on such a compute node with

salloc --nodes=1 --constraint=mil -t 60 -A s1460 --qos debug

This gets you a terminal on a Milan node under the debug queue, which is pretty fast but has a wall clock limit of 1 hour. You could, alternatively, issue this command first thing in the morning with -t 480 and get a node for 8 hours. You will need to wait longer to get a node, but once you do, you're set for a day's worth of building and debugging.

Once you get on a compute node, go to the folder where you checked out the source tree, and just to be safe create a clean environment as follows:

module purge
cd @env
source g5_modules.sh
cd ..

Now you're ready to build. Since you're already on a compute node, no need to submit a parallel build job. Instead, issue the following commands in order:

mkdir build
cd build
cmake .. -DBASEDIR=$BASEDIR/Linux -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=../install
make -j install

This builds the model into ../install, specifically the GEOS GCM executable is ../install/bin/GEOSgcm.x.

Setting up a run

Use gcm_setup to install the model somewhere

Go into install/bin and execute ./gcm_setup.

  • Experiment ID is any name you want to give the run. It's a good idea to include the model version and something about which tracers you are running in a short name. Mostly something that you will remember. If you call it (say) Apple it's pretty much guaranteed that you won't remember what it is for two years down the line. I'm calling mine GCM-11.5.2-methane-c180.
  • Experiment Description is a short description to help you remember.
  • CLONE is the ability of a model to copy over someone else's run folder. This is a very useful ability, but for now let's choose NO.
  • Atmospheric Horizontal Resolution depends on what you want to run. I'm choosing c180. It's perfectly fine to choose c90 for model development.
  • Default Vertical Resolution is fine
  • Default Microphysics is fine
  • Default Hydrostatic Atmosphere is fine
  • Use IOSERVER if you're running c180 or higher.
  • Default processor type of mil is fine
  • Default COUPLED Ocean/Sea-Ice Model is fine
  • Default Data_Ocean Horizontal Resolution 360x180 is fine
  • I have no idea what to choose for Land Surface Boundary Conditions, let's just go with the default of Icarus-NLv3 and not Icarus
  • Default Land Surface Model of Catchment is fine
  • Feel free to run GOCART with Climatological aerosols
  • Use default GOCART Emission Files, you will change this later anyway
  • For c180, a HEARTBEAT_DT of 450 is fine
  • Don't worry about the HISTORY template, you are going to change the history file anyway
  • The HOME Directory is where the run folder will be created. Just make sure it's created somewhere inside /discover/nobackup/projects/gmao/geos_carb/${USER}
  • In theory EXPERIMENT Directory can be different from HOME Directory, but no one has ever tried it. Either set it to be the same, or try at your own risk and don't expect any sympathy if you break something.
  • The Build directory should already be correct
  • Our GROUP ID is s1460

Create restart files

This is a dark art. Remembering Robert the Bruce before embarking on this endeavor would be well advised.

GEOS restart files are called *_rst, even though they're really netcdf files.

Set up emission files

Set up output history collection

Errors that will keep coming back like bad pennies

Incorrect number of tiles in water restarts

Every so often, when you try to run a fresh model setup, you'll get an error such as

Error! Found 339967 tiles in openwater. Expect to find 359523 tiles.
Your restarts are probably for a different ocean.

This is probably because the water restarts you are using come from a run with a different choice of Land Surface Boundary Conditions. Look in your gcm_run.j, specifically setenv BCSDIR. Set this to whatever is in the run you copied the water restarts from. There is a specific combination of BCSDIR in gcm_run.j and the water restarts that will work. Unfortunately, gcm_setup is not your friend here; it will not tell you which folder to copy the restarts from given your choice of land boundary conditions.

Clone this wiki locally