Chemical renaming (#59)

* Use config instead of local variables

* Add logging for whole chain

* GitHub Action: Apply Pep8-formatting

* Add logging for jobs

* GitHub Action: Apply Pep8-formatting

* Small fix

* GitHub Action: Apply Pep8-formatting

* Format duration

* Remove --wait from submit function

* Let geosp be a separate job

* GitHub Action: Apply Pep8-formatting

* Add run_chain argument to process specified chunk

* GitHub Action: Apply Pep8-formatting

* Fix datetime comparison in int2lm

* Submit all jobs as sbatch

* GitHub Action: Apply Pep8-formatting

* Use existing submit() function and only return job_id

* GitHub Action: Apply Pep8-formatting

* Fix prepare_data

* Disable logging for individual jobs

* GitHub Action: Apply Pep8-formatting

* Write casename into logfile

* Fix timers for jobs

* GitHub Action: Apply Pep8-formatting

* Use UTC times

* GitHub Action: Apply Pep8-formatting

* More async jobs and workflows

* GitHub Action: Apply Pep8-formatting

* Add logging to new jobs

* GitHub Action: Apply Pep8-formatting

* Remove unused imports

* Remove global icon-art from prepare_data

* GitHub Action: Apply Pep8-formatting

* Replace os.path with Pathlib

* GitHub Action: Apply Pep8-formatting

* Fix for PosixPaths

* ADD: convenience function for slurm job info

* REF: move get_job_info method to the end of the class definition

* GitHub Action: Apply Pep8-formatting

* Add some docstrings

* Fix quotes for f-strings

* Add imports for art-global job

* Add dependencies for all icon workflows

* Fix pathname

* GitHub Action: Apply Pep8-formatting

* Fix chunk calculation

* Add icontools job

* GitHub Action: Apply Pep8-formatting

* Add icontools to dependencies

* Modify docstrings

* Some code cleanup

* fix(icontools): remove unsused packages + fix dependency logic

- The `icontools` job already depends on `prepapre_data`, no need to
  add a `copy_id` in the dependencies (which was anyways undefinied
  in this scope).

- The name of the first argument of `cfg.submit()` must be 'icontools'
  so that all the sub job ids are associated with the icontools job
  and are taken into account for other jobs depending on it (icon).
  Otherwise, icon will find no dependency in
  `cfg.job_ids['current']['icontools']`.

- `cfg.get_job_info()` should be used later in `run_chain.py` to
  monitor time for all async jobs. Currently the reported time is
  only the submission time.

* Make prepare_art_global dependent on previous icon

* Fix import

* Explicitly set async variable

* Add logfile for icontools

* Set variables for icontools

* Revert "Explicitly set async variable"

This reverts commit 99b96cf.

* Define sequential workflow in icon-seq-test

* Make cosmo-ghg workflow async

* Add info about (a)sync mode

* fix: also launch waiting job when some workflow jobs failed

* add: other keys to default job info dict

* fix: empty current job ids before each chunk

* GitHub Action: Apply Pep8-formatting

* Split prepare_data into cosmo and icon

* Remove unused imports

* GitHub Action: Apply Pep8-formatting

* Rename prepare_data job in workflows

* Rename prepare_data in icon-seq case

* Set additional cfg variables in prepare_icon

* GitHub Action: Apply Pep8-formatting

* Time logging for all jobs

* Merge geosp into prepare_art

* GitHub Action: Apply Pep8-formatting

* Include oem in icon-art-oem workflow

* Fix icon job

* Move geosp to icontools job

* GitHub Action: Apply Pep8-formatting

* add: placeholder for monitoring async jobs

* Run geosp after icontools

* Remove oem job from icon-art-oem case

* Change to current logfile within jobs

* GitHub Action: Apply Pep8-formatting

* Add missing log inits

* Configure root logger

* set logger

* Fix logger

* Format logging output

* Introduce BASIC_PYTHON_JOB option to call jobs directly in async mode

* GitHub Action: Apply Pep8-formatting

* add: only submit basic python jobs through a nested run_chain

* GitHub Action: Apply Pep8-formatting

* fix: loop over jobs in run_chunk

* ref: job_id becomes chunk_id

reflects reality and avoids confusion with actual jobs id

* GitHub Action: Apply Pep8-formatting

* fix: leftover `job_id` -> `chunk_id`

* GitHub Action: Apply Pep8-formatting

* add(untested): Slurm monitoring

* GitHub Action: Apply Pep8-formatting

* Add BASIC_PYHTON_JOB to missing jobs

* Fix function arguments

* Small fix and hint to KeyError

* GitHub Action: Apply Pep8-formatting

* ref(slurm summary): streamline code a bit

* GitHub Action: Apply Pep8-formatting

* Some settings for cosmo-ghg case

* afterany -> afterok for wait job

* Comment some function calls

* add:ref: remove unused `info_requests` + print failing jobs

* fix: job summary for previous chunk, not current

* fix: reactivate empty current job ids at beginning of chunk

* fix: only wait and monitor if not very first chunk

* GitHub Action: Apply Pep8-formatting

* ref: move icon-art error handling in slurm job itself

* fix: remove BASIC_PYTHON_JOB workaround for icon

* fix(icon.py): escape curly brackets for string formatting

* fix(icon): escape curly brackets in python way

* Add walltime and remove conda activation

* GitHub Action: Apply Pep8-formatting

* Set walltime correctly

* Fix call to handle_error function

* Set walltimes for icon workflows

* Debug

* Remove job_ids override

* ref(config): clean up

* Add smaller walltime to wait job

* Remove prints

* Some fixes for cosmo case

* Remove model check

* Don't pass logfile anymore

* GitHub Action: Apply Pep8-formatting

* Submit int2lm and cosmo jobs correctly

* GitHub Action: Apply Pep8-formatting

* Small fixes for int2lm and cosmo

* Store job scripts in separate directory

* Add BASIC_PYTHON_JOB to int2lm

* GitHub Action: Apply Pep8-formatting

* Add post_cosmo dependency

* Bugfixes for cosmo jobs

* Further fixes

* Make post_cosmo a submit job

* Fix post_cosmo

* Fix post_cosmo

* ref: refactor cycling and monitoring

- Regroup waiting, monitoring nd cycling in a single `Config.cycle()`
  method.
- write chunk monitoring info into chain log file

* GitHub Action: Apply Pep8-formatting

* Fix config variables

* Fix log file output

* Formatting

* Remove time logging from jobs

* GitHub Action: Apply Pep8-formatting

* Fix for icon job

* GitHub Action: Apply Pep8-formatting

* ref: exception handling in Config.submit()

* Change table cell widths

* Remove old way of logging

* Cleanup

* NNodes -> N

* GitHub Action: Apply Pep8-formatting

* Fix for N/NNodes

* GitHub Action: Apply Pep8-formatting

* Yet another fix for N/NNodes

* Simplify job names

* Change missing job names

* Fix syntax error from commit 8810dcf

* GitHub Action: Apply Pep8-formatting

* Just jobname for jobs

* Cleaner console output

* GitHub Action: Apply Pep8-formatting

* Unify restart and spinup runs

* GitHub Action: Apply Pep8-formatting

* Complete config file for spinup test

* Fix for spinup

* GitHub Action: Apply Pep8-formatting

* Fix 2 for spinup

* Fancy formatting

* Define separate spinup workflow

* Fix formatting (hopefully)

* Aligning case + workflow

* Custom workflow_name and improve check

* Remove workflow_name check in jobs

* GitHub Action: Apply Pep8-formatting

* Bugfix in cosmo job

* Remove restart info

* Compute chunks separately

* GitHub Action: Apply Pep8-formatting

* Add function to get previous chunk ID

* GitHub Action: Apply Pep8-formatting

* Directly get previous chunk id

* Further cleanup and refactoring

* GitHub Action: Apply Pep8-formatting

* Fix for chunk_id_prev

* Fix

* Save total chunk list

* Fix restart variables

* Don't print chunk list twice

* Fix for cosmo_restart_out

* Remove sequential part

* GitHub Action: Apply Pep8-formatting

* Remove sequential case

* Add dependencies to all workflows

* Remove is_async config variable

* Include renaming scheme for chemicals for OEM

* GitHub Action: Apply Pep8-formatting

* re-order in and output tracers

* Shortened variable name

* GitHub Action: Apply Pep8-formatting

---------

Co-authored-by: Michael Jähn <[email protected]>
Co-authored-by: github-actions <[email protected]>
Co-authored-by: Matthieu <[email protected]>

cases/icon-art-oem-test/config.yaml

@@ -41,6 +41,8 @@ chem:
     nameformat: '%Y%m%d_%H'
     suffix: .grb
     inc: 3
+    remap_tracers:
+        CH4_BG: TRCH4_chemtr
 
 icontools_runjobs:
     - icontools_remap_ic_runjob.cfg

jobs/prepare_art_oem.py

@@ -74,7 +74,10 @@ def main(cfg):
             ds_chem['PS'] = ds_chem['LNPS']
             ds_chem['PS'].attrs = ds_chem['LNPS'].attrs
             ds_chem['PS'].attrs["long_name"] = 'surface pressure'
-            ds_chem['TRCH4_chemtr'] = ds_chem['CH4_BG']
+            # Remapping chemical tracer names
+            if "remap_tracers" in cfg.chem:
+                for chem_in, chem_out in cfg.chem['remap_tracers'].items():
+                    ds_chem[chem_out] = ds_chem[chem_in]
             # merge:
             ds_merged = xr.merge([ds_meteo, ds_chem], compat="override")
             #ds_merged.attrs = ds.attrs