Overhaul ci (#31)

Replace `e2e` tests with a new set of tests that is independent of ICON
data. These tests basically tests the command-line interfaces of the
click-subcommand. So far `cdo-table`, `tolerance`, `perturb`,
`performance` and `stats` are covered.

### Pytest fixtures
In order to handle data requirements of the tests a set of fixtures is
used.
This allows to organize the data requirements of the tests in a central
place.

### Execute all tests using Pytest
In order to start transition to pytests, collect and execute all tests
using pytest

### Reference files in `tests/data`
The tests compare its results to precomputed values located in the repo.
This ensures consistency over time. In case new reference data is
required, pytests automatically stores
the new references for each test-session in a folder in `/tmp`. The
location is printed to terminal
when running pytest.

### cdo-table overrides function at runtime
See #32,
a quick and dirty fix has been introduce in the PR.

---------

Co-authored-by: Jonas Jucker <[email protected]>

.gitignore

@@ -1,18 +1,15 @@
 .idea*
 .vscode/*
-!.vscode/launch.json
 !.vscode/settings.json
 *pycache*
 *.swp
 *.pdf
 *.png
 *.log
 *.nc
-tests/util/icon/test/*
-tests/tmp/*
-probtest_env
 
-mock_output
-env
-probtest_data
-reference_data
+# output from unittest
+test_stats.csv
+test_stats_csv.csv
+test_stats_csv.dat
+random_tolerance.csv

README.md

@@ -152,8 +152,22 @@ python probtest.py check --input-file-ref stats_ref.csv --input-file-cur stats_c
 Note that the reference `--input-file-ref` and test stats files `--input-file-cur` need to be set by command line arguments. This is because the default stored in the `ICON.jinja` template is pointing to two files from the ensemble as a sanity check.
 
 ## Developing in probtest
+#### Testing with [pytest](https://docs.pytest.org/en/8.2.x/)
 
-The tool has been developed using Visual Studio Code. To ease development there is a `.vscode/launch.json` file containing the configurations to run all of the probtest commands discussed in [Commands](#commands) based on data generated by ICON and probtest. All input data needed to run are generated by a [Jenkins plan](https://jenkins-mch.cscs.ch/job/Probtest/job/generate_probtest_testdata_github) and stored on daint:/project/g110/probtest_testdata. If you plan to make relevant changes in probtest, you need to run the Jenkins plan and update the hashes in the files `test_reference` and `.env` accordingly.
+Our tests are executed using `pytest`, ensuring a consistent and efficient testing process. Each test dynamically generates its necessary test data, allowing for flexible and isolated testing scenarios.
+
+Simply run
+```console
+pytest -s -v tests/*
+```
+in order to run all tests.
+
+To run only a subset of test run
+```console
+pytest -s -v path/to/your/test.py
+```
+
+Reference data, crucial for validating the outcomes of our tests and detecting any deviations in `probtests` results, is maintained in the [tests/data](tests/data) directory. This approach guarantees that our tests are both comprehensive and reliable, safeguarding the integrity of our codebase.
 
 ### Code formatting
 
@@ -170,56 +184,3 @@ pre-commit run --all-files
 ```
 
 If you are using VSCode with the settings provided by this repository in `.vscode/settings.json` formatting is already enabled on save.
-
-### Debugging individual commands
-
-For testing, probtest has data stored in two different places.
-
-1. Probtest input data (generated by ICON/externals/probtest) is saved on Piz Daint in /project/g110/probtest_testdata/
-
-2. Probtest performance data. This is very little data and is saved inside this repo in tests/data/
-
-To execute all tests, you first need to download the probtest input data, then run the initialization script to generate the associated config file from the template:
-
-    cd /local/path/to/probtest
-    REFERENCE_DATA=./reference_data # Directory to read reference data
-    PROBTEST_CUR_DATA=./probtest_data # Directory to write probtest output
-    test_reference=$(cat test_reference)
-    scp -r daint:/project/g110/probtest_testdata/$test_reference $REFERENCE_DATA
-    python probtest.py init --codebase-install $REFERENCE_DATA/icon_data --reference $PROBTEST_CUR_DATA --template-name templates/testdata.jinja --experiment-name atm_amip_les_test --config testdata.json --file-id NetCDF "*atm_3d_ml*" --file-id NetCDF "*atm_3d_hl*" --member-num 2,5
-
-Now you can run and debug any probtest command from the _Run_ tab in VS code. (Note that the template `testdata.jinja` treats `codebase-install` differently than the default `ICON.jinja`.)
-
-### Executing end to end tests
-
-The reference data bundle not only includes data produced by ICON but also reference output generated by probtest. To assert that this output is not affected by local changes to probtest, one can execute the following steps:
-
-Generate a set of probtest output with the local probtest version:
-
-    ICON_DATA=$REFERENCE_DATA/icon_data PROBTEST_DATA=./probtest_data ./scripts/generate_icon_testdata.sh
-
-Adjust the `.env` file to reflect the directory structure:
-
-`PROBTEST_REF_DATA`: path to the `probtest_data` subdirectory of the reference data.
-`PROBTEST_CUR_DATA`: path to the output directory you chose in the last step.
-`PROBTEST_TEST_EXPERIMENT`: full sets of reference data are generated for the `mch_opr_r04b07` experiment.
-
-Finally, you can execute the end to end tests in one of two ways:
-
-#### VSCode Testing
-
-In VSCode, the python interpreter checks for the `.env` file and reads the variables from it into the run environment. The settings needed to make VSCode aware of pythons `unittest`s are already in the `.vscode/settings.json` file. Therefore, the tests can simply be executed by switching to the "Testing" view and running all tests.
-
-#### Command line on Piz Daint
-
-The tests can be run from the command line as well. The variables in the `.env` file can be used for testing on Piz Daint. Because the `.env` file does not export any variables, we need a little work-around. Execute
-
-     . .env && export $(cut -d= -f1 .env)
-
-in your shell to export the variables set in the `.env` file. Next, execute the end to end tests as described above. The \<icon-commit\> and \<probtest-commit> in the command below are only the first four digits of the respective commit hashes. The hashes used for current testing, can be found in the `test_reference` file.
-
-    ICON_DATA=/project/g110/probtest_testdata/i-<icon-commit>_p-<probtest-commit>/icon_data PROBTEST_DATA=./probtest_data ./scripts/generate_icon_testdata.sh
-
-If the test data is generated without any error, you can execute the `unittest`s with:
-
-    python3 -m unittest

engine/cdo_table.py

@@ -125,6 +125,8 @@ def cdo_table(
     file_specification = file_specification[0]  # can't store dicts as defaults in click
     assert isinstance(file_specification, dict), "must be dict"
 
+    # save original method and restore at the end of this module
+    dataframe_from_ncfile_orig = model_output_parser.dataframe_from_ncfile
     # modify netcdf parse method:
     model_output_parser.dataframe_from_ncfile = rel_diff_stats
 
@@ -193,3 +195,4 @@ def cdo_table(
 
         Path(cdo_table_file).parent.mkdir(parents=True, exist_ok=True)
         df.to_csv(cdo_table_file)
+        model_output_parser.dataframe_from_ncfile = dataframe_from_ncfile_orig

jenkins/mergerequest

@@ -6,17 +6,14 @@ pipeline {
     }
 
     stages {
-        stage('generate testdata') {
+        stage('Pytest') {
             steps {
-                sh './jenkins/scripts/step_generate_testdata.sh'
-            }
-        }
-
-        stage('run unittests mch') {
-            steps {
-                sh './jenkins/scripts/step_execute_unittest_mch.sh'
+                sh'''
+                source /project/g110/icon/probtest/conda/miniconda/bin/activate
+                conda activate probtest
+                pytest -v tests/*
+                '''
             }
         }
     }
-
 }