Refactoring (part 1) (#18)

* Add src folder for model code * Use devbuildcosmo and environment for cosmo/icon builds * Fix cosmo and icon builds * Add further jenkins scripts * Fix devbuildcosmo * Update gitignore * Use c2sm branch for cosmo-ghg * Update submodules for icon * Adapt spack in Jenkinsfile * devbuilds for int2lm and cosmo * GitHub Action: Apply Pep8-formatting * Add int2lm to pipeline * Add global icon-art case * Set own initial file in icon job * GitHub Action: Apply Pep8-formatting * Add new tool scripts * First work on prepare_data job for global simulations * Fix cosmo executable * Add checks for jenkins script * Fix icon exe * Add check for cosmo case * Install icontools with spack * Fix icontools scripts (new spack) * Properly initialize conda within jenkins script * Add dask package to environment * Add namelist parameter to fix ICON case * Only clone last commit for int2lm/cosmo/icon * Convert variable names to lowercase * Change target and fix errors * GitHub Action: Apply Pep8-formatting * Add everything to prepare_data job * GitHub Action: Apply Pep8-formatting * Adapt icon runjob * Move data download script to jenkins folder * Adapt Jenkinsfile * Add cdsapi package * Replace subprocess.call with .run + remove unused imports * Update year in license * Add default jobs * Move model info to .yaml file * target -> model; subtarget -> variant * Fix error message * Fix Jenkinsfile * Replace (sub)target with model/variant in config files * GitHub Action: Apply Pep8-formatting * Modify variants and add restart * Fix run_chain * GitHub Action: Apply Pep8-formatting * run_chain refactoring * GitHub Action: Apply Pep8-formatting * Adapt some jobs * GitHub Action: Apply Pep8-formatting * Pass model_cfg to job functions * Fix cosmo job * Fix run_chain for icon * GitHub Action: Apply Pep8-formatting * Generalize copying data in prepare_data * Fix prepare_data job * Add lateral grid info to config file * Fix icon job * remove logging and sys from config files * Remove unnecessary time variables * GitHub Action: Apply Pep8-formatting * Start working on icon-art-test case * GitHub Action: Apply Pep8-formatting * Fix grid folder for input data and reduce dataset on ftp server * Use gpu nodes and normal queue for test cases * Load icontools for pre-processing scripts * Fix prepare_data for icon-art-test * Further work on icon-art input data * Further work on icon-art test case * GitHub Action: Apply Pep8-formatting * Only keep test cases * Add template for icon-art-global-test * Modify config for icon-art-global * GitHub Action: Apply Pep8-formatting * Fix and unify scripts for icon-art-global * Add icbc folder for prepare_data job * Add new tests for Jenkins * GitHub Action: Apply Pep8-formatting * Add actual test scripts * Fix check for input data size * Add Q field to initial file * GitHub Action: Apply Pep8-formatting * Add PS field to meteo file * GitHub Action: Apply Pep8-formatting * unify config files * Add icon-art-global-test to Jenkinsfile * GitHub Action: Apply Pep8-formatting * Change input data path * Fix modifying initial file * Rename icon-art-test to icon-art-oem-test * Small fixes and error handling * Fixes for cams chem data * Fix icontools runjobs * Adapt jenkins test for icon-art-oem * Use latest spack-c2sm version * Fix ICON-ART build and tests * Replace oae with oem * Replace oae with oem in job * Rename job * Update models.yaml * Further renaming * Update README.md * Change permissions for jenkins script * Add string2char function again for emissions * Remove ICON part for GHG namelist script * Add argparse for jenkins main script --------- Co-authored-by: github-actions <[email protected]>
C2SM · Sep 5, 2023 · c02c541 · c02c541
1 parent bc6fbc3
commit c02c541
Show file tree

Hide file tree

Showing 144 changed files with 844 additions and 8,682 deletions.
diff --git a/LICENSE b/LICENSE
@@ -1,6 +1,6 @@
 MIT License
 
-Copyright (c) 2021-2022 C2SM and other Processing Chain Developers.
+Copyright (c) 2021-2023 C2SM and other Processing Chain Developers.
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

diff --git a/README.md b/README.md
@@ -16,28 +16,29 @@ instruction for the installation.
 
 ### 1\. Install Miniconda
 
-Install as user specific miniconda, e.g. on `/scratch` (enter `cd
-$SCRATCH` and `pwd` at the command line to get to your personal scratch
-directory on Daint). When the command prompt asks for installation
-location, provide the path to your scratch and append `/miniconda3`.
+Install as user specific Miniconda, e.g. on your `$HOME` directory,
+which is the default location.
 
-> **Note**: The default location would be on your `$HOME` directory, which
-> may lead to memory issues.
+> **Note**: Only conda itself should be installed in your `$HOME`.
+> All environments should be stored in your `$PROJECT` directory,
+> otherwise you risk filling up your `$HOME` directory. See below for instructions.
 
-To install the latest miniconda, type:
+To install the latest Miniconda, type:
 
     wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
     bash Miniconda3-latest-Linux-x86_64.sh
 
-Then, export `$PATH` to your conda installation:
-
-    export PATH="$SCRATCH/miniconda3/bin:$PATH"
+Further deails on Miniconda can be found on the [Miniconda documentation page](https://docs.conda.io/en/latest/miniconda.html).
 
 ### 2\. Create the Conda Environment
 
 Create a conda environment `proc-chain` with and install requirements:
 
-    conda env create -f env/environment.yml
+    conda env create --prefix $PROJECT/envs/proc-chain -f env/environment.yml
+
+To be able to activate your conda environment by simply using `conda activate proc-chain` instead of the full path, add the following to your `.bashrc`:
+
+    export CONDA_ENVS_PATH=$PROJECT/envs
 
 Activate the environment (use "source activate" in case "conda activate"
 does not work):
@@ -60,7 +61,7 @@ these files within your home directory:
 
 Once everything has been set up correctly according to the above steps,
 you just need to execute the following command to activate your
-environment:
+environment (if not done already):
 
     conda activate proc-chain
 
@@ -69,7 +70,7 @@ line help to see the available arguments for the main script:
 
     python run_chain.py -h
 
-To run the example cases with their standard jobs, please ensure
+To run the test cases with their standard jobs, please ensure
 that you clone the Processing Chain to `$SCRATCH`, as input and
 output data are stored in subdirectories of the chain itself.
 
@@ -78,62 +79,30 @@ output data are stored in subdirectories of the chain itself.
 > adapt the configuration file `config.py` in your case folder in
 > a way that output files are written to a specified folder on `$SCRATCH`.
 
-For these test cases, the necessary input data can be obtained via
-the following script (this may take some time):
-
-    ./get_data.sh
-
-Furthermore, executables for COSMO-GHG and ICON are needed. For both COSMO-GHG and ICON,
-a spack instance needs to be initialized first. Then, COSMO-GHG can be 
-installed:
+For these test cases, you can use the Jenkins script
 
-    source /project/g110/spack/user/daint/spack/share/spack/setup-env.sh
-    spack installcosmo cosmo@empa-ghg%nvhpc cosmo_target=gpu +cppdycore
-
-For ICON, type:
+    ./jenkins/scripts/jenkins.sh
 
-    spack install icon@c2sm-master%[email protected] icon_target=cpu +eccodes +ocean
-
-> **Note**: For further information about building ICON or building software
-> with Spack in general, consider the 
-> [C2SM wiki page](https://wiki.c2sm.ethz.ch/MODELS/ICONIconModel) and the official 
-> [C2SM Spack Documentation](https://c2sm.github.io/spack-c2sm/QuickStart.html).
+This script calls other scripts that are located in `jenkins/scripts/`. 
+They will
+- activate the conda environment (if not done already)
+- setup spack-c2sm
+- download input data to `input/`
+- build `int2lm`, `cosmo-ghg`, `icon` and `icon-art`
+- test `cosmo-ghg`, `icon`, `icon-art-oem`, `icon-art-global`
 
-Alternatively, you can use the following scripts:
-
-    ./jenkins/scripts/build_cosmo-ghg.sh
-
-or
-
-    ./jenkins/scripts/build_icon.sh
-
-Finally, to run the COSMO-GHG test case, type:
+To run the test cases manually, type:
 
+```bash
+    # COSMO-GHG
     python run_chain.py cosmo-ghg-11km-test 2015-01-01 0 24
-
-> **Note**: Be sure to have spack initialized via 
-> `source /project/g110/spack/user/daint/spack/share/spack/setup-env.sh`
-> before you run COSMO or ICON jobs.
-
-For ICON, type:
-
+    # ICON
     python run_chain.py icon-test 2018-01-01 0 24 -j prepare_data icon
-
-Empa users can perform additional tests:
-
-    python run_chain.py cosmo-art-mother-test cosmo-art-nested-test 2015-06-26 0 24
-
-or:
-
-    python run_chain.py icon-art-test 2018-01-01 0 24 -j prepare_data icon
-
-or:
-
+    # ICON-ART (OEM)
     python run_chain.py icon-art-oem-test 2018-01-01 0 24 -j prepare_data oae icon
-
-or:
-
-    python run_chain.py icon-art-oem-ensembles-test 2018-01-01 0 24 -j prepare_data oae icon
+    # ICON-ART (global)
+    python run_chain.py icon-art-global-test 2018-01-01 0 24 -j prepare_data oae icon
+```
 
 ## Documentation
 

diff --git a/cases/carbosense/config.py b/cases/carbosense/config.py
diff --git a/cases/carbosense/cosmo_INPUT_AF.cfg b/cases/carbosense/cosmo_INPUT_AF.cfg
diff --git a/cases/carbosense/cosmo_INPUT_ASS.cfg b/cases/carbosense/cosmo_INPUT_ASS.cfg
diff --git a/cases/carbosense/cosmo_INPUT_DIA.cfg b/cases/carbosense/cosmo_INPUT_DIA.cfg