Deploy preview for PR 43 🛫

pr-preview/pr-43/.buildinfo

@@ -0,0 +1,4 @@
+# Sphinx build info version 1
+# This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done.
+config: 16d3d200e4cd62c836b81b30e404c784
+tags: 645f666f9bcd5a90fca523b33c5a78b7

pr-preview/pr-43/_sources/file-structure.rst.txt

@@ -0,0 +1,32 @@
+.. _file-structure-section:
+
+File Structure
+--------------
+
+::
+
+    + README.rst
+    + run_script.py                 # main script
+    + jobs/
+    |    + *.py                     # jobs-files
+    |    \ tools/                   # tools-files
+    + cases/                        # example test cases
+    |    + cosmo-ghg-11km-test/     # COSMO-GHG example
+    |    |    + config.py           # user-configuration
+    |    |    \ *.cfg               # templates for namelists & batch-jobs
+    |    + cosmo-art-mother-test/   # COSMO-ART example (mother domain)
+    |    |    + config.py
+    |    |    \ *.cfg
+    |    + cosmo-art-nested-test/   # COSMO-ART example (nested domain)
+    |    |    + config.py
+    |    |    \ *.cfg
+    |    \ icon-test                # ICON example
+    |    |    + config.py
+    |    |    \ *.cfg
+    |    \ icon-oem-test            # ICON-OEM example
+    |    |    + config.py
+    |    |    \ *.cfg
+    + docs/
+         + source/                  # *.rst documentation files
+         \ Makefile                 # Makefile for generating docs
+

pr-preview/pr-43/_sources/howtorun.rst.txt

@@ -0,0 +1,133 @@
+.. _script-section:
+
+How to Run
+==========
+
+The Python file ``run_chain.py`` in the root directory is the main script of the
+Processing Chain.
+It reads the user's input from the command line and from the ``config.py`` file of the
+respective case.
+Then it will start the Processing Chain.
+
+Starting the Chain
+------------------
+
+The chain has to be run with the following command: ::
+
+    $ python run_chain.py <casename> <startdate> <hstart> <hstop> -j [jobs]
+
+``<casename>`` is the name of a directory in the ``cases/``-directory where
+there is a ``config.py``-file specifying the configurations, as well as templates
+for the necessary namelist files for **int2lm**, **COSMO** or **ICON**.
+
+If you don't supply a joblist, the default joblist will be executed.
+
+For **COSMO**, that is ``prepare_data`` ``emissions`` ``biofluxes`` ``int2lm``
+``post_int2lm`` ``cosmo`` ``post_cosmo``,
+
+For **COSMOART** it is ``prepare_data`` ``emissions`` ``obs_nudging``
+``photo_rate`` ``int2lm`` ``cosmo`` ``post_cosmo``.
+
+For **ICON** it is ``prepare_data`` ``icon``.
+
+For **ICONART** it is ``prepare_data`` ``icon``.
+
+For **ICONARTOEM** it is ``prepare_data`` ``oae`` ``icon``.
+
+The model type can be chosen by setting the variable ``target`` in the ``config.py``-file.
+
+To run the **COSMO-GHG** example test case, run::
+
+    $ python run_chain.py cosmo-ghg-11km-test 2015-01-01 0 24 -j prepare_data emissions biofluxes int2lm post_int2lm cosmo post_cosmo
+
+To run the **COSMO-ART** example case, run::
+
+    $ python run_chain.py cosmo-art-mother-test cosmo-art-nested-test 2015-06-26 0 24 -j prepare_data emissions obs_nudging photo_rate int2lm cosmo post_cosmo
+
+To run the **ICON** or **ICON-ART** example cases, run::
+
+    $ python run_chain.py icon-test 2018-01-01 0 24 -j prepare_data icon
+
+or::
+
+    $ python run_chain.py icon-art-test 2018-01-01 0 24 -j prepare_data icon
+
+To run the **ICON-OEM** example cases, run::
+
+    $ python run_chain.py icon-oem-test 2018-01-01 0 24 -j prepare_data oae icon
+
+or::
+
+    $ python run_chain.py icon-oem-ensembles-test 2018-01-01 0 24 -j prepare_data oae icon
+        
+What it Does
+------------
+
+The script ``run_chain.py`` reads the command-line arguments and the config-file.
+It then calls the function :func:`run_chain.restart_runs` which divides the
+simuation time according to the specified restart steps. Then it calls
+:func:`run_chain.run_chain` for each sub-run. This function sets up the directory
+structure of the chain and then starts the specified :ref:`jobs<jobs-section>`
+sequentially.
+
+The directory structure generated by the Processing Chain for a **COSMO** run
+looks like this:::
+
+  cfg.work_root/
+    + output/cfg.output_root/
+    \ <casename>/cfg.chain_root/
+                   + cfg.int2lm_base/
+		   |   + cfg.int2lm_input/
+		   |   + cfg.int2lm_work/
+		   |   \ cfg.int2lm_output/
+		   + cfg.cosmo_base/
+		   |   + cfg.cosmo_work/
+		   |   + cfg.cosmo_output/
+		   |   \ cfg.cosmo_restart_out/
+		   \ checkpoints/
+		       + cfg.log_working_dir/
+		       \ cfg.log_finished_dir/
+                   
+Running the ``cosmo-ghg-11km-test``-case therefore produces the following directories:::
+
+  $SCRATCH/processing_chain/
+             + output/cosmo-ghg-11km-test
+	     \ cosmo-ghg-11km-test/2015010100_0_24/
+	                 + int2lm/
+			 |   + input/
+			 |   |  + emissions/
+			 |   |  + extpart/
+			 |   |  + prepare_data/
+			 |   |  \ vprm/
+			 |   + run/
+			 |   |  + int2lm   # executable
+			 |   |  + INPUT
+			 |   |  \ run.job
+			 |   \ output/
+			 + cosmo/
+			 |   + run/
+			 |   |  + cosmo    # executable
+			 |   |  + INPUT_*
+			 |   |  \ run.job
+			 |   + output/
+			 |   \ restart/
+			 \ checkpoints
+			     + working/    # 1 logfile per started job
+			     \ finished    # 1 logfile per finished job
+
+
+Functions in ``run_chain.py``
+-----------------------------
+
+.. autofunction:: run_chain.restart_runs
+
+.. autofunction:: run_chain.restart_runs_spinup
+
+.. autofunction:: run_chain.run_chain
+
+.. autofunction:: run_chain.load_config_file
+
+.. autofunction:: run_chain.set_simulation_type
+
+.. autofunction:: run_chain.parse_arguments
+

pr-preview/pr-43/_sources/index.rst.txt

@@ -0,0 +1,27 @@
+.. Processing Chain documentation master file, created by
+   sphinx-quickstart on Thu Sep 27 14:11:04 2018.
+   You can adapt this file completely to your liking, but it should at least
+   contain the root `toctree` directive.
+
+.. toctree::
+    :maxdepth: 2
+    :caption: Getting Started
+
+    file-structure
+    howtorun
+
+.. toctree::
+    :maxdepth: 3
+    :caption: Configuration
+
+    config
+    namelists
+
+
+.. toctree::
+    :maxdepth: 3
+    :caption: Jobs
+
+    jobs
+    tools
+

pr-preview/pr-43/_sources/jobs.rst.txt

@@ -0,0 +1,148 @@
+.. _jobs-section:
+
+Overview
+--------
+
+The jobs described here are available for use in the processing chain.
+For every target, you can choose from a list of available jobs.
+As some jobs depend on the result of others, the order indicated here
+has to be respected.
+
+**COSMO**:
+
+1.  :func:`jobs.biofluxes.main` | :func:`jobs.prepare_data.main` | 
+    :func:`jobs.emissions.main` | :func:`jobs.online_vprm.main` | 
+    :func:`jobs.oae.main` | :func:`jobs.obs_nudging.main`
+2.  :func:`jobs.int2lm.main`
+3.  :func:`jobs.post_int2lm.main`
+4.  :func:`jobs.octe.main`
+5.  :func:`jobs.cosmo.main`
+6.  :func:`jobs.check_output.main`
+7.  :func:`jobs.reduce_output.main` 
+8.  :func:`jobs.post_cosmo.main` | :func:`jobs.verify_chain.main`
+
+**COSMOART**:
+
+1. :func:`jobs.emissions.main` | :func:`jobs.prepare_data.main` |
+   :func:`jobs.photo_rate.main` | :func:`jobs.obs_nudging.main` |
+   :func:`jobs.online_vprm.main` | :func:`jobs.oae.main`
+2. :func:`jobs.int2lm.main`
+3. :func:`jobs.cosmo.main`
+4. :func:`jobs.check_output.main`
+5. :func:`jobs.reduce_output.main`
+6. :func:`jobs.post_cosmo.main` | :func:`jobs.verify_chain.main`
+
+**ICON**:
+
+1.  :func:`jobs.prepare_data.main`
+2.  :func:`jobs.icon.main`
+
+**ICONART**:
+
+1.  :func:`jobs.prepare_data.main`
+2.  :func:`jobs.icon.main`
+
+
+**ICONARTOEM**:
+
+1.  :func:`jobs.prepare_data.main` 
+2.  :func:`jobs.oae.main`
+3.  :func:`jobs.icon.main`
+
+
+Adding new jobs
+---------------
+
+Adding a new job to the chain is simple:
+
+1. In the directory ``jobs/``, create a file called ``<jobname>.py`` containing
+   a function called ``main`` which takes the same arguments as every other job.
+   Make sure the function is documented with a docstring.
+2. Import it in ``jobs/__init__.py`` to make it accessible to ``run_chain.py``.
+3. Add the function to the documentation. You find the file describing this page
+   at ``doc/source/jobs.rst``.
+
+List of available jobs
+----------------------
+
+* :func:`jobs.biofluxes.main`
+* :func:`jobs.check_output.main`
+* :func:`jobs.cosmo.main`
+* :func:`jobs.emissions.main`
+* :func:`jobs.icon.main`
+* :func:`jobs.int2lm.main`
+* :func:`jobs.prepare_data.main`
+* :func:`jobs.oae.main`
+* :func:`jobs.obs_nudging.main`
+* :func:`jobs.octe.main`
+* :func:`jobs.online_vprm.main`  
+* :func:`jobs.photo_rate.main`
+* :func:`jobs.post_cosmo.main`
+* :func:`jobs.post_int2lm.main`
+* :func:`jobs.reduce_output.main`
+* :func:`jobs.verify_chain.main`
+
+-------------------------------------------
+
+.. autofunction:: jobs.biofluxes.main
+
+-------------------------------------------
+
+.. autofunction:: jobs.check_output.main
+
+-------------------------------------------
+
+.. autofunction:: jobs.cosmo.main
+
+-------------------------------------------
+
+.. autofunction:: jobs.emissions.main
+
+-------------------------------------------
+
+.. autofunction:: jobs.icon.main
+
+-------------------------------------------
+
+.. autofunction:: jobs.int2lm.main
+
+-------------------------------------------
+
+.. autofunction:: jobs.prepare_data.main
+
+-------------------------------------------
+
+.. autofunction:: jobs.oae.main
+
+-------------------------------------------
+
+.. autofunction:: jobs.obs_nudging.main
+
+-------------------------------------------
+
+.. autofunction:: jobs.octe.main
+
+-------------------------------------------
+
+.. autofunction:: jobs.online_vprm.main
+
+-------------------------------------------
+
+.. autofunction:: jobs.photo_rate.main
+
+-------------------------------------------
+
+.. autofunction:: jobs.post_cosmo.main
+
+-------------------------------------------
+
+.. autofunction:: jobs.post_int2lm.main
+
+-------------------------------------------
+
+.. autofunction:: jobs.reduce_output.main
+
+-------------------------------------------
+
+.. autofunction:: jobs.verify_chain.main
+

pr-preview/pr-43/_sources/namelists.rst.txt

@@ -0,0 +1,25 @@
+.. _namelists:
+
+Namelist Templates
+------------------
+
+Namelists for **int2lm** and **COSMO** are generated using templates which are also located in
+the cases-directory. These templates are essentially textfiles containing "normal" namelist
+parameters and python-variables in curly braces.
+
+These files get read by their respective job.
+The resulting string is formatted using python's ``.format()``-function which replaces the
+python-variables with their value. The formatted strings are then saved as namelist-files in the
+run-directory of their respective jobs and then read by the executable. ::
+
+  cases/example/example_namelist.cfg -> [read file] ->
+  "exvar = '{cfg.prefix}{cfg.suffix}'" -> ["".format(cfg)] ->
+  "exvar = 'pref_suff.nc'" -> [write to disk] ->
+  int2lm/run/example_namelist
+
+The same procedure is done for the slurm-runscripts for **int2lm** and **COSMO**.
+
+A special case is ``INPUT_ART`` for **int2lm** and ``INPUT_GHG`` for **COSMO** . These namelists are
+generated by :func:`jobs.tools.write_int2lm_input_art.main` and :func:`jobs.tools.write_cosmo_input_ghg.main`
+from ``.csv``-files containing all necessary information.
+