From eaa4552f85023adb27a7b7b6949cac269cca79d2 Mon Sep 17 00:00:00 2001 From: Ian Date: Thu, 25 Jul 2024 13:22:29 -0400 Subject: [PATCH 1/4] Update fre/pp/README.md --- fre/pp/README.md | 101 ++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 100 insertions(+), 1 deletion(-) diff --git a/fre/pp/README.md b/fre/pp/README.md index 5ee90688..d956a7a7 100644 --- a/fre/pp/README.md +++ b/fre/pp/README.md @@ -1,3 +1,9 @@ + + + + + + + +## **Quickstart instructions to postprocess FMS history output on PP/AN or gaea** + +1. Checkout postprocessing workflow template +This will clone the postprocessing repository into `/home/$USER/cylc-src/EXPNAME__PLATFORM__TARGET`. +``` +module load fre/canopy +fre pp checkout -e EXPNAME -p PLATFORM -t TARGET +``` + +2. Configure pp template with either XML or pp.yaml + +``` +fre pp configure-xml -e EXPNAME -p PLATFORM -t TARGET -x XML +``` +or +``` +fre pp configure-yaml -e EXPNAME -p PLATFORM -t TARGET -y YAML + +``` + +3. (OPTIONAL BUT RECCOMENDED) Create `history-manifest` for config validation + +Create a `history-manifest` of a single tar file archive first for use in the validation. +This list represents the available source files within the history tar archives, and enables the +validation procedure to catch a wider variety of potential errors. This can be done like so- +``` +tar -tf /archive/$USER/path/to/history/files/YYYYMMDD.nc.tar | grep -v tile[2-6] | sort > /home/$USER/cylc-src/EXPNAME__PLATFORM__TARGET/history-manifest +``` + +4. Validate the configuration +``` +fre pp validate -e EXPNAME -p PLATFORM -t TARGET +``` + +Warnings related to directories are probably valid and should be fixed in `rose-suite.conf`, or created as necessary via `mkdir`. + +If you are running postprocessing gaea, you'll need to change the `SITE` variable in `rose-suite.conf` from `ppan` to `gaea`. + +5. Install the workflow + +``` +fre pp install -e EXPNAME -p PLATFORM -t TARGET +``` + +If you are attempting this on gaea, you'll need to make two one-time changes before installing. +- Currently, `cylc`, `rose`, and `isodatetime` must be in your PATH for new shells. One approach to do this is +to symlink the fms-user-installed fre-cli cylc/rose/isodatetime scripts into your local `~/bin` directory, +and then add that `~/bin` directory to your PATH in your `.bashrc` or `.cshrc`. (If you don't do this, Cylc tasks +will fail complaining those 3 tools are not available.) + +``` +cd ~/bin +ln -s /ncrc/home2/Flexible.Modeling.System/conda/envs/fre-cli/bin/{cylc,rose,isodatetime} . +echo 'setenv PATH ${PATH}:~/bin' >> ~/.cshrc +``` +- Currently, the cylc available on gaea (through `module load cylc` or the `PATH` trick above) does not +include any global configuration, so you'll need to create a file `~/.cylc/flow/global.cylc` that contains the following. +If you don't do this, Cylc will use your home directory for the scratch space and rapidly fill your quota.) + +``` +[install] + [[symlink dirs]] + [[[localhost]]] + run = /gpfs/f5/scratch/gfdl_f/$USER +``` + +6. Run the workflow + +``` +fre pp run -e EXPNAME -p PLATFORM -t TARGET +``` + +7. Report status of workflow progress + +``` +fre pp status -e EXPNAME -p PLATFORM -t TARGET +``` + +8. Launch GUI + +``` +TODO: fre pp gui? + +The full GUI can be launched on jhan or jhanbigmem (an107 or an201). + +cylc gui --ip=`hostname -f` --port=`jhp 1` --no-browser +``` + +## **PP Wrapper Usage** + +Here + ### **PP Wrapper Decision Tree** ![pp_wrapper_decsiontree](https://github.com/NOAA-GFDL/fre-cli/assets/98476720/d3eaa237-1e29-4922-9d83-8d9d11925c54) From 7930e0602768ac2a9b6acd839c90c57fc793dc9e Mon Sep 17 00:00:00 2001 From: Ian Date: Thu, 25 Jul 2024 13:47:21 -0400 Subject: [PATCH 2/4] Update subtools.rst add space to `fre ` in subtools doc list --- docs/subtools.rst | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/subtools.rst b/docs/subtools.rst index 83a0fce2..3e919ae5 100644 --- a/docs/subtools.rst +++ b/docs/subtools.rst @@ -1,20 +1,20 @@ Subtools ======== -frecheck +fre check -------- -frelist +fre list ------- -fremake +fre make ------- -frepp +fre pp ----- -frerun +fre run ------ -fretest +fre test ------- From 7683e11f6b4607e17c4a9bfafa5c2e3d3ce790a8 Mon Sep 17 00:00:00 2001 From: Ian Date: Thu, 25 Jul 2024 13:56:13 -0400 Subject: [PATCH 3/4] Update README.md tweaks to main readme tool list --- README.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index a785d72a..e1ac6129 100644 --- a/README.md +++ b/README.md @@ -9,7 +9,7 @@ FMS Runtime Environment (FRE) CLI developed using Python's Click package ## **Background** -As part of fre/canopy, MSD wanted to develop a modern, user-friendly CLI that will allow users to call upon FRE commands using a **_fre_** **tool** _subtool_ syntax. Developed with Click, a Python package easily installable through PyPI and Conda, the main goal of this is to allow users access to most, if not all of MSD-managed tools and workflows from one packaged, centralized CLI. +`fre-cli` is a modern, user-friendly CLI that will allow users to call FRE commands using a **_fre_** **tool** _subtool_ syntax. Leveraging Click, an easily installable Python package available via PyPI and/or Conda, `fre-cli` gives users intuitive and easy-to-understand access to many FRE tools and workflows from one packaged, centralized CLI. ![Screenshot from 2024-04-18 13-42-04](https://github.com/NOAA-GFDL/fre-cli/assets/98476720/43c028a6-4e6a-42fe-8bec-008b6758ea9b) @@ -57,12 +57,14 @@ Following the instructions above, the user will be able to run `fre` from any di ## **Checklist: Currently Implemented Tools** To be developed: -- [ ] **fre check** + - [x] **fre app** - [x] **fre catalog** +- [ ] **fre check** +- [x] **fre cmor** - [ ] **fre list** - [x] **fre make** -- [x] **fre cmor** +- [x] **fre pp** - [ ] **fre run** - [ ] **fre test** - [ ] **fre yamltools** From ef818528206d9dd7f74281d0c427fcf71a917320 Mon Sep 17 00:00:00 2001 From: Ian Date: Thu, 25 Jul 2024 14:01:53 -0400 Subject: [PATCH 4/4] Update subtools.rst --- docs/subtools.rst | 30 +++++++++++++++++++++--------- 1 file changed, 21 insertions(+), 9 deletions(-) diff --git a/docs/subtools.rst b/docs/subtools.rst index 3e919ae5..900ea0e8 100644 --- a/docs/subtools.rst +++ b/docs/subtools.rst @@ -1,20 +1,32 @@ Subtools ======== -fre check +fre app -------- -fre list -------- +fre catalog +-------- + +fre check (not yet implemented) +-------- + +fre cmor +-------- + +fre list (not yet implemented) +-------- fre make -------- +-------- fre pp ------ +-------- + +fre run (not yet implemented) +-------- -fre run ------- +fre test (not yet implemented) +-------- -fre test -------- +fre yamltools (not yet implemented) +--------