Merge pull request #34 from MeteoSwiss-APN/shape-file-feat

Shape files, pre-commits, scaling function and ReadME

README.md

@@ -1,18 +1,18 @@
 # PyFlexPlot
-PyFlexPlot is a Python-based tool to visualize FLEXPART dispersion simulation results stored in NetCDF format.
+PyFlexPlot is a Python-based tool to visualize/post-process FLEXPART dispersion simulation results stored in NetCDF format.
 ## Table of Contents
 - [Prerequisites and Cloning the Repository](#prerequisites-and-cloning-the-repository)
-- [Quick Start](#quick-start)
+- [Getting Started](#getting-started)
 - [Usage](#usage)
   - [Usage Example](#usage-example)
-- [The Developer's Guide through the Project](#the-developers-guide-through-the-project)
-  - [Getting Started](#getting-started)
+- [Developer Notes](#developer-notes)
   - [Implemented Debugging Features](#implemented-debugging-features)
   - [Roadmap to your first Contribution](#roadmap-to-your-first-contribution)
   - [Testing and Coding Standards](#testing-and-coding-standards)
     - [Pre-commit on GitHub Actions](#pre-commit-on-github-actions)
     - [Jenkins](#jenkins)
-- [Features](#features)
+    - [Updating the Test References](#updating-the-test-references)
+- [Features](#key-features)
 - [Credits](#credits)
 - [External Links](#external-links)
 - [License](#license)
@@ -23,15 +23,45 @@ To get a local copy of this repository, run following commands and naviate into
 git clone https://github.com/MeteoSwiss-APN/pyflexplot <custom_directory_name>
 cd <custom_directory_name>
 ```
-## Quick Start
-For a quick setup to use pyflexplot, run the following commands within the root folder:
-1. Install pinned environment: ```tools/setup_env.sh```
-2. Activate the environment and build the package:
+
+## Getting Started
+
+Once you created or cloned this repository, make sure the installation is running properly. Install the package dependencies with the provided script `setup_env.sh`.
+Check available options with
 ```bash
-conda activate pyflexplot
-pip install --no-deps .
+tools/setup_env.sh -h
 ```
-3. To check if the tool runs properly you can run the tests by running ```pytest tests```
+We distinguish pinned installations based on exported (reproducible) environments and free installations where the installation
+is based on top-level dependencies listed in `requirements/requirements.yml`. If you start developing, you might want to do an unpinned installation and export the environment:
+
+```bash
+tools/setup_env.sh -u -e -n <custom_environment_name>
+```
+*Hint*: If you are the package administrator, it is a good idea to understand what this script does, you can do everything manually with `conda` instructions.
+
+*Hint*: Use the flag `-m` to speed up the installation using mamba. Of course you will have to install mamba first (we recommend to install mamba into your base
+environment `conda install -c conda-forge mamba`. If you install mamba in another (maybe dedicated) environment, environments installed with mamba will be located
+in `<miniconda_root_dir>/envs/mamba/envs`, which is not very practical.
+
+The package itself is installed with `pip`. For development, install in editable mode:
+
+```bash
+conda activate <custom_environment_name>
+pip install --editable .
+```
+
+*Warning:* Make sure you use the right pip, i.e. the one from the installed conda environment (`which pip` should point to something like `path/to/miniconda/envs/<custom_environment_name>/bin/pip`).
+
+Once your package is installed, navigate to the root directory and run the tests by typing:
+
+```bash
+cd <custom_directory_name>
+conda activate <custom_environment_name>
+pytest
+```
+
+If the tests pass, you are good to go. If not, contact the package administrator Stefan Ruedisuehli. Make sure to update your requirement files and export your environments after installation
+every time you add new imports while developing. Check the next section to find some guidance on the development process if you are new to Python and/or SEN.
 
 ## Usage
 To utilize pyflexplot, first ensure you are in the root directory of the project and have activated the necessary conda environment:
@@ -55,12 +85,13 @@ To utilize all available CPUs for the command, use the option:
 ### Usage Example
 After you've set up pyflexplot ([Prerequisites and cloning the repository](#prerequisites-and-cloning-the-repository) and [Getting started](#getting-started)),
 you'll need to specify a configuration file and an output directory.
-Define an output directory:
+Create an output directory using:
 ```bash
 dest=test_output/
 ```
-Note: The directory will be created on run time if it doesn't already exist.
+Note: The directory will be automatically created if it doesn't already exist.
 
+There are several default config files available under ```src/pyflexplot/data/presets/opr```.
 Furthermore, there are already several default config files available in the directory ```src/pyflexplot/data/presets/opr```.
 To run the program for all presets in the PDF graphics format with the default input data, use:
  ```bash
@@ -81,46 +112,7 @@ After selecting a preset, run pyflexplot interactively:
 pyflexplot --preset "$preset" --merge-pdfs --dest=$dest
 ```
 
-## The Developer's Guide through the Project
-
-### Getting Started
-
-Once you created or cloned this repository, make sure the installation is running properly. Install the package dependencies with the provided script `setup_env.sh`.
-Check available options with
-```bash
-tools/setup_env.sh -h
-```
-We distinguish pinned installations based on exported (reproducible) environments and free installations where the installation
-is based on top-level dependencies listed in `requirements/requirements.yml`. If you start developing, you might want to do an unpinned installation and export the environment:
-
-```bash
-tools/setup_env.sh -u -e -n <custom_environment_name>
-```
-*Hint*: If you are the package administrator, it is a good idea to understand what this script does, you can do everything manually with `conda` instructions.
-
-*Hint*: Use the flag `-m` to speed up the installation using mamba. Of course you will have to install mamba first (we recommend to install mamba into your base
-environment `conda install -c conda-forge mamba`. If you install mamba in another (maybe dedicated) environment, environments installed with mamba will be located
-in `<miniconda_root_dir>/envs/mamba/envs`, which is not very practical.
-
-The package itself is installed with `pip`. For development, install in editable mode:
-
-```bash
-conda activate <custom_environment_name>
-pip install --editable .
-```
-
-*Warning:* Make sure you use the right pip, i.e. the one from the installed conda environment (`which pip` should point to something like `path/to/miniconda/envs/<custom_environment_name>/bin/pip`).
-
-Once your package is installed, navigate to the root directory and run the tests by typing:
-
-```bash
-cd <custom_directory_name>
-conda activate <custom_environment_name>
-pytest
-```
-
-If the tests pass, you are good to go. Make sure to update your requirement files and export your environments after installation
-every time you add new imports while developing. Check the next section to find some guidance on the development process if you are new to Python and/or SEN.
+## Developer Notes
 
 As this package was created with the SEN Python blueprint, it comes with a stack of development tools, which are described in more detail on the [Website](https://meteoswiss-apn.github.io/mch-python-blueprint/). Here, we give a brief overview on what is implemented.
 
@@ -139,8 +131,8 @@ Here are some of the key debugging features:
 
 ### Roadmap to your first Contribution
 
-Generally, the source code of your library is located in `src/<library_name>`. `cli.py` thereby serves as an entry
-point for functionalities you want to execute from the command line and it is based on the Click library. If you do not need interactions with the command line, you should remove `cli.py`. Moreover, of course there exist other options for command line interfaces,
+Generally, the source code of your library is located in `src/<library_name>`. The blueprint will generate some example code in `mutable_number.py`, `utils.py` and `cli.py`. `cli.py` thereby serves as an entry
+point for functionalities you want to execute from the command line, it is based on the Click library. If you do not need interactions with the command line, you should remove `cli.py`. Moreover, of course there exist other options for command line interfaces,
 a good overview may be found [here](https://realpython.com/comparing-python-command-line-parsing-libraries-argparse-docopt-click/), we recommend however to use click. The provided example
 code should provide some guidance on how the individual source code files interact within the library. In addition to the example code in `src/<library_name>`, there are examples for
 unit tests in `tests/<library_name>/`, which can be triggered with `pytest` from the command line. Once you implemented a feature (and of course you also
@@ -186,10 +178,25 @@ both commits on branches and PRs. Your jenkins pipeline will not be set up
 automatically. If you need to run your tests on CSCS machines, contact DevOps to help you with the setup of the pipelines. Otherwise, you can ignore the jenkinsfiles
 and exclusively run your tests and checks on GitHub actions.
 
-## Features
-
-- TODO
-
+### Updating the Test References
+Pyflexplot includes a set of functionality tests that compare generated output against predefined reference data.
+These reference files, which are in the .py format, are derived from and stored alongside the original data in the tests/data directory.
+To update these references, uncomment the lines of code in the test file you wish to update and run the test.
+
+## Key Features
+
+### PDF and PNG Files
+Pyflexplot allows to visualize data on a map plot and save the output in either PDF or PNG format. To utilize this feature, simply adjust the outfile variable with the appropriate file extension.
+![Example Image Output](img/integrated_concentration_site-Goesgen_species-1_domain-full_lang-de_ts-20200217T0900.png)
+### Shape File Generation
+Furthermore, Pyflexplot provides the functionality to export data into shape files (.shp) to utilize them in GIS programs such as QGIS 3. The output is a ZIP archive containing the essential components of a shapefile: .shp, .dbf, .shx, .prj, and .shp.xml.
+Key aspects of this feature include:
+* __Filtering Zero Values__: The tool initially removes zero values from fields (e.g., concentration) before processing.
+* __Logarithmic Transformation__: Field values undergo a log_10 transformation to optimize the visualization of data ranges.
+* __Precision Handling__: The transformed field values are recorded with 15 decimal places, accommodating the precision limitations of some GIS software.
+* __Metadata Storage__: Information, such as details about released materials, are stored within a .shp.xml file as metadata.
+### Scaling the field values
+Another feature is to manipulate the field values by scaling with an arbitrary factor. This factor can be set in the preset with the variable `multiplier`.
 ## Credits
 
 This package was created with [`copier`](https://github.com/copier-org/copier) and the [`MeteoSwiss-APN/mch-python-blueprint`](https://github.com/MeteoSwiss-APN/mch-python-blueprint) project template.
@@ -198,6 +205,7 @@ This package was created with [`copier`](https://github.com/copier-org/copier) a
 
 * [copier](https://github.com/copier-org/copier) - Based library and CLI app for rendering project templates.
 * [mch-python-blueprint](https://github.com/MeteoSwiss-APN/mch-python-blueprint) - Project template embedded in this project based on copier.
+* [pyshp](https://github.com/GeospatialPython/pyshp) - Python module to generate Shapefiles
 
 
 ## License

README.meteoswiss.md

@@ -286,7 +286,7 @@ and without committing it as new tag (--no-commit --no-tag)
     bumpversion --verbose --allow-dirty --no-commit --no-tag --new-version=1.0.6-pre-1 dummy
 
 
-Save environment specificatons to allow for an exact replication of
+Save environment specifications to allow for an exact replication of
 environment with make install
 
     make install-dev

VERSION

@@ -1 +1 @@
-1.0.10
+1.1.0

jenkins/Jenkinsfile

@@ -28,7 +28,7 @@ pipeline {
                 '''
             }
         }
-        stage('build package') {
+        stage('build unpinned') {
             steps {
                 sh '''#!/usr/bin/env bash
                 set -e
@@ -41,7 +41,7 @@ pipeline {
                 '''
             }
         }
-        stage('test') {
+        stage('test unpinned') {
             steps {
                 sh '''#!/usr/bin/env bash
                 source $WORKSPACE/miniconda/etc/profile.d/conda.sh
@@ -51,69 +51,25 @@ pipeline {
                 '''
             }
         }
-        stage('dev-environment') {
+        stage('build unpinned') {
             steps {
                 sh '''#!/usr/bin/env bash
                 set -e
-                source $WORKSPACE/miniconda/etc/profile.d/conda.sh
-                conda init bash --no-user --install --system
-                conda activate
-                bash tools/setup_env.sh -n dev-pyflexplot -u
-                conda activate dev-pyflexplot
-                pip install --no-deps --editable .
-                '''
-            }
-        }
-        stage('dev-test') {
-            steps {
-                sh '''#!/usr/bin/env bash
-                source $WORKSPACE/miniconda/etc/profile.d/conda.sh
-                conda init bash --no-user --install --system
-                conda activate dev-pyflexplot
-                pytest tests
-                '''
-            }
-        }
-        stage('pinned-environment') {
-            steps {
-                sh '''#!/usr/bin/env bash
-                set -e
-                source $WORKSPACE/miniconda/etc/profile.d/conda.sh
+                source ${WORKSPACE}/miniconda/etc/profile.d/conda.sh
                 conda init bash --no-user --install --system
                 conda activate
-                bash tools/setup_env.sh -n pinned-pyflexplot
-                conda activate pinned-pyflexplot
+                bash tools/setup_env.sh -n pyflexplot-pinned
+                conda activate pyflexplot-pinned
                 pip install --no-deps .
                 '''
             }
         }
-        stage('pinned-test') {
+        stage('test pinned') {
             steps {
                 sh '''#!/usr/bin/env bash
                 source $WORKSPACE/miniconda/etc/profile.d/conda.sh
-                conda activate pinned-pyflexplot
-                pytest tests
-                '''
-            }
-        }
-        stage('pinned-dev-environment') {
-            steps {
-                sh '''#!/usr/bin/env bash
-                set -e
-                source $WORKSPACE/miniconda/etc/profile.d/conda.sh
                 conda init bash --no-user --install --system
-                conda activate
-                bash tools/setup_env.sh -n pinned-dev-pyflexplot
-                conda activate pinned-dev-pyflexplot
-                pip install --no-deps --editable .
-                '''
-            }
-        }
-        stage('pinned-dev-test') {
-            steps {
-                sh '''#!/usr/bin/env bash
-                source $WORKSPACE/miniconda/etc/profile.d/conda.sh
-                conda activate pinned-dev-pyflexplot
+                conda activate pyflexplot-pinned
                 pytest tests
                 '''
             }

pyproject.toml

@@ -7,7 +7,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "pyflexplot"
-version = "1.0.12"
+version = "1.1.0"
 description = "Stefan Ruedisuehli's PyFlexPlot"
 readme = "README.md"
 keywords = [
@@ -195,4 +195,4 @@ ignore_directives = [
 
 [tool.codespell]
 ignore-words-list = "ende,ue,fo,activ,nd,feld,nd"
-skip = "requirements/*,*.pdf"
+skip = "requirements/*,*.pdf,environment.yml"

requirements/requirements.yml

@@ -12,7 +12,7 @@ dependencies:
   - netcdf4
   - numpy >= 1.20
   - pillow >= 8.4.0
-  - pypdf2 == 1.*
+  - pypdf >= 3.10.0
   - scipy
   - toml
   - typing-extensions

src/pyflexplot/cli/cli.py

@@ -79,7 +79,7 @@
     ),
     metavar="DEST_DIR",
     type=click.Path(exists=False),
-    default=None,
+    default="output/",
 )
 @click.option(
     "--merge-pdfs/--no-merge-pdfs",

src/pyflexplot/cli/click.py

@@ -76,7 +76,7 @@ def click_set_verbosity(ctx, param, value) -> None:
 
 
 def wrap_callback(fct):
-    """Wrapp click callback functions to conditionally drop into ipdb."""
+    """Wrap click callback functions to conditionally drop into ipdb."""
 
     def wrapper(ctx, param, value):
         fct_loc = wrap_pdb(fct) if (ctx.obj or {}).get("pdb") else fct