This repo is a collection of python scripts to visualise ICON-simulations on the unstructered grid. The different folders contain example code for various types of plots. Example datasets for testing can be downloaded following the instructions in the data folder. Example plots for each folder are shown below. More detailed descriptions for each plot are in the README files of the different folders. The routines are mainly based on the python library psyplot. The C2SM/iconarray python package was developed together with icon-vis, to contain the modules used in this repository. For visualizing data along a transect, psy-transect is currently under development.
If you have any feature requests, feel free to raise an issue or contact us by email. We are also happy if you want so share your own plotting scripts.
We recommend to use a conda environment for the usage of the provided scripts. Please follow the instruction for the installation.
Instructions
-
Look up most recent Miniconda version for Linux 64-bit on the Miniconda documentation pages
-
Install an user specific miniconda. When the command prompt asks for the installation location, provide the path to your scratch and append the name of your miniconda version
$SCRATCH/miniconda3(the default location would be on your home directory, which may lead to memory issues) and don't runconda init.wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh bash Miniconda3-latest-Linux-x86_64.sh -
Export path to your conda installation (if using daint/euler/tsa: install miniconda on scratch to avoid memory issues).
export PATH="$SCRATCH/miniconda3/bin:$PATH"
Create a conda environment psyplot with python[version>=3.7,<3.10] (psy-view requirement) and install requirements:
conda env create -n psyplot -f env/environment.yml
Activate environment (use "source activate" in case "conda activate" does not work):
conda activate psyplot
If you already have the environment but want to update it:
conda env update --file env/environment.yml --prune
If you are using the conda setup and want to use GRIB data, you will need to set the GRIB_DEFINITION_PATH. This can be done on Tsa/Daint by sourcing the script setup-conda-env.sh. It only needs to be run a single time, as it will save the GRIB_DEFINITION_PATH environment variable to the conda environment. You will need to deactivate and reactivate the conda environment after doing this. You can check it has been correctly set by conda env config vars list. This script also sets the Fieldextra path, which is used for data interpolation. See FAQs if you get an error running this.
source env/setup-conda-env.sh
After creating the virtual environment and installing the requirements, the environment only needs to be activated for future usage. Make sure that the path is exported to ~/miniconda3/bin.
If you have jupyter notebook installed, you can run the ipython scripts (.ipynb) by opening jupyter notebook after sourcing your psyplot environment. For Piz Daint please follow the instructions below.
Instructions for Piz Daint
For running the ipython scripts on Piz Daint, you need to follow the instructions on JupyterLab on CSCS, which are summarized here for icon-vis:
Load the modules daint-gpu and jupyter-utils (before activating the conda environment!)
module load daint-gpu jupyter-utils
Then, activate your psyplot environment
conda activate psyplot
Create psyplot-kernel:
kernel-create -n psyplot-kernel
It may be necessary to add a CONDA_PREFIX to the launcher file to work with cartopy. Therefore open your psyplot-kernel launcher file:
vim $HOME/.local/share/jupyter/kernels/psyplot-kernel/launcher
and add the following line after the first line:
export CONDA_PREFIX=$SCRATCH/miniconda3/envs/psyplot
You can now start JupyterLab with https://jupyter.cscs.ch and open the psyplot-kernel notebook.
In case you need to reinstall the kernel, you can delete it with
rm -rf $HOME/.local/share/jupyter/kernels/psyplot-kernel/
See the mapplot folder for details on how this plot was made.
See the vectorplot folder for details on how these plots were made.
See the difference_map folder for details on how this plot was made.
See the timeseries folder for details on how this plot was made.
See the vertical_profile folder for details on how this plot was made.
See the transect folder for details on how these plots were made.
See the combinedplot folder for details on how this plot was made.
The edgeplot folder provides examples of plotting variables defined on the edge of ICON grid cells, as opposed to the cell center. See the edgeplot folder for details on how the below plots were made.
Within this repository there are both Jupyter Notebooks and Python scripts for various examples of plots. The Python scripts can be used with your input data as parameters, or as guidance for creating your own script which is tailored to your data or visualization needs. The scripts and notebooks often use Python modules from iconarray, as well as custom formatoptions which can be used very easily while plotting with psyplot.
The notebooks and example plots in this repository use data which is stored on an FTP server. This data can be downloaded by running the get_data.py script. cd icon_vis/icon_vis/modules and then run:
python get_data.py
Or you can use the function get_example_data in your notebooks. More information on the data downloaded can be found in the data folder README.
There are a number of modules from the C2SM/iconarray package installed by conda (see env/environment.yml), which you can import like a normal python package into your scripts. To work with the modules and formatoptions from within icon-vis or elsewhere, you can add this code block to the start of your script / notebook. You will see many examples of the modules being used within the scripts in this repo.
import iconarray as iconvis # import iconarray modules
from iconarray.plot import formatoptions # import plotting formatoptions (for use with psyplot)Then you can use the functions or modules as needed, e.g.:
iconvis.get_example_data()Some of the most useful modules for plotting from iconarray are described here:
grid - grid.py
combine_grid_information() This adds the required grid information from a provided grid file to your dataset if not present. It also adds coordinates encoding to each variable, which is needed to plot using psyplot.
check_grid_information() Checks whether or not the grid data needs to be added, e.g.:
if check_grid_information(nc_file):
print('The grid information is available')
data = psy.open_dataset(nc_file)
else:
print('The grid information is not available')
data = combine_grid_information(nc_file,grid_file)utils - utilities.py
ind_from_latlon() Returns the nearest neighbouring index of lat-lon within given lats-lons.
add_coordinates() Returns the position of given coordinates on the plot (useful to add markers at a fixed location).
get_stats() Returns the mean of two given variables, the difference of the mean and the p values.
wilks() Returns a value for which differences are significant when data point dependencies are accounted for (based on Wilks 2016).
show_data_vars() Returns a table with variables in your data. The first column shows the variable name psyplot will need to plot that variable.
This is useful if you plot GRIB data, because if GRIB_cfVarName is defined, cfgrib will set this as the variable name, as opposed to GRIB_shortName which you might expect.
interpolate.py - interpolate.py
The functions in interpolate.py are used to facilitate the interpolation of ICON vector data to a regular grid, or a coarser ICON grid, for the purpose of vectorplots, e.g., wind plots. For psyplot we recommend to plot wind data on the regular grid as you can then scale the density of arrows in a vector plot as desired.
remap_ICON_to_ICON() This calls the create_ICON_to_ICON_remap_namelist() function to create a fieldextra namelist with your datafile, and subsequently runs fieldextra with this namelist. The output file along with a LOG and the namelist are saved in a tmp folder. The function returns the file location of the output file.
remap_ICON_to_regulargrid() This calls the create_ICON_to_Regulargrid_remap_nl() function to create a fieldextra namelist with your datafile, and subsequently runs fieldextra with this namelist. The output file along with a LOG and the namelist are saved in a tmp folder. The function returns the file location of the output file.
Descriptions of the formatoption modules and data modules can be found in Example Data and Formatoptions sections.
Psyplot has a large number of ‘formatoptions’ which can be used to customize the look of visualizations. For example, the descriptions of the formatoptions associated with the MapPlotter class of psyplot can be found in the psyplot documentation. The documentation for using formatoptions is also all on the psyplot documentation, or seen in the examples.
Psyplot is designed in a way that is very modular and extensible, allowing users to easily create custom formatoptions and register them to plotters. Instructions for doing so are here.
The iconarray repository includes various custom formatoptions, that are not included in psyplot. For example:
- Borders - Adds internal land borders to mapplot, vectorplots, and combinedplots.
- Rivers - Adds rivers to mapplot, vectorplots, and combinedplots.
- Lakes - Adds lakes to mapplot, vectorplots, and combinedplots.
- Standard Title - Adds a descriptive title based on your data to your mapplot.
- Mean Max Wind - Work In Progress.
- Custom Text - Work In Progress.
We encourage you to create your own formatoptions and contribute to this repository if they would be useful for others.
Once registered to a plotter class, the formatoptions can be used as seen in many of the scripts, for example in mapplot.py.
If you want to plot derived variables, psyplot requires that the new variable has the correct coordinate encoding. These need to be set by you. For example, if you create a variable delta_t on your dataset ds, based on temperature calculated on the cell center, then you must set:
ds.delta_t.encoding['coordinates'] = 'clat clon'Whereas if your derived variable is an edge variable, for example derived from the tangential and normal components of the wind on the edges (VN, VT), then the coordinates encoding should be set as:
ds.derived_edge_var.encoding['coordinates'] = 'elat elon'You should also ensure that you have the cell or edge data required from the grid merged in the dataset. For variables on the cell center, your dataset will need not only clat, clon, but the bounds clon_bnds, clat_bnds, and the relationship must be defined between them, e.g.:
ds.clon.attrs['bounds'] = 'clon_bnds'
ds.clat.attrs['bounds'] = 'clat_bnds'Likewise for edge variables, your dataset will require elat, elon, as well as:
ds.elon.attrs['bounds'] = 'elon_bnds'
ds.elat.attrs['bounds'] = 'elat_bnds'The function combine_grid_information in the grid.py module of iconarray sets the bounds attributes (among others) while merging the required grid data with the dataset.
NETCDF data often has everything you need to plot the data using psyplot, but sometimes it doesn't. For example the data could be missing the grid data, which is required for plotting. In this case the grid information can be added using the combine_grid_information function in the grid.py module. You just need to provide the location to the corresponding grid file. If you still have trouble plotting, check that the encoding coordinates for the variable you want to plot are set correctly - see Plotting Derived Variables for more information.
To open GRIB data using psyplot or xarray, you will need to use the cfgrib engine. eg:
ds = psy.open_dataset(icon_grib_file, engine='cfgrib', backend_kwargs={'indexpath': '', 'errors': 'ignore'})GRIB data does not contain the grid information. This needs to be merged, and can be done using the combine_grid_information function in the grid.py module. You can provide either the file locations or xarray datasets to this function. This also sets the encoding coordinates as required.
The cfgrib engine relies on an eccodes installation. The easiest way to set up your environment with the required dependencies for cfgrib is to use the Conda setup.
You can specify the vertical level (height/altitude/pressure levels) at which you are plotting data by specifying the z formatoption. This specifies the index of the vertical level array.
myplot = ds.psy.plot.mapvector(time=0, name=[['U', 'V']], z=8)You can see which vertical dimension and value this corresponds to by printing the axes of the plot.
print(myplot.axes)
# OrderedDict([(<GeoAxesSubplot:title={'center':'Vector Plot after interpolating ICON data to Regular Grid'}>,
# psyplot.project.Project([ arr11: 3-dim DataArray of U, V, with (variable, y_1, x_1)=(2, 101, 101),
# time=2021-11-23, grid_mapping_1=b'', z_1=1.05e+04]))])Alternatively you can specify the vertical level using the dimension name. E.g., if the name of the vertical dimension is generalVerticalLayer:
myplot = ds.psy.plot.mapvector(time=0, name=[['U', 'V']], generalVerticalLayer=8)If you have specific question about plotting, you can write that into the discussion section or check if you find the answer there already: icon-vis disucssion
-
Problems setting conda environment variables via
source env/setup-conda-env.sh.init() got an unexpected keyword argument 'capture_output'
Check for outdated spack commands in your $HOME/.bashrc (should align with instructione in C2SM spack Documentation, and if using VS Code/Remote-SSH you might also need to uncheck Remote.SSH: Use Local Server in your VS Code Remote-SSH settings, to force a new connection upon reconnecting.
-
Value error on
psy.open_dataset(f_grib2, engine="cfgrib", ...)ValueError: conflicting sizes for dimension 'values': length 1567452 on 'VN' and length 1043968 on {'generalVerticalLayer': 'generalVerticalLayer', 'values': 'P'}
Solution: You might be trying to open a heterogeneous GRIB file with multiple hypercubes. Try
cfgrib.open_datasets(open_datasets with an s!) which automates the selection of appropriate filter_by_keys and returns a list of all valid xarray.Dataset's in the GRIB file (see cfgrib Documentation).
import cfgrib
cfgrib.open_datasets(f_grib2, engine="cfgrib", backend_kwargs={'indexpath': '', 'squeeze':False}) -
The psyplot library needs the boundary variables (clon_bnds, clat_bnds). If they are not in the nc file, the information needs to be added with a grid file. The error is likely to be:
ValueError: Can only plot 2-dimensional data!
Solutions: Add the path to a grid file in the config under the section 'var' with the option 'grid_file'. Equally you could use the function combine_grid_information in the grid module if you do not use the config file.
-
ValueError: numpy.ndarray size changed, may indicate binary incompatibility.
Can be solved by reinstalling numpy:
pip uninstall numpy pip install numpy -
ImportError: libproj.so.22: cannot open shared object file: No such file or directory
For some reason the LD_LIBRARY_PATH is set wrong (probably a daint issue). Can be solved by setting the path to the lib folder of your environment:
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:~/miniconda3/envs/your_env/libMore information on this issue: conda-forge/cartopy-feedstock#93
-
AttributeError: 'MapTransectMapPlot2D' object has no attribute 'convert_coordinate'
That's a psyplot 1.4.1 error and should be resolved by installing the newest version of psyplot.
Note: make sure there are no psyplot packages installed on the local user, e.g., under /users/username/.local/lib/python3.9/site-packages/. If there are, they need to be uninstalled and installed again.
-
Random error in a python package pointing to /users/username/.local/lib/python3.9/site-packages/ instead of your environment
Deactivate your environment and uninstall the package causing the error with
pip uninstall. Now activate your environment again and the package should now point to the right location. Update your conda environment if not. -
TypeError: an integer is required
This error points at an incompatibility between the installed library versions and the base python version. While the cause for this error is not fully understood, loading the python module corresponding to your system from
env/setup-conda-env.shbefore installing conda helped in the past. -
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding Python runtime state: core initialized LookupError: no codec search functions registered: can't find encoding
The content of your miniconda repo might have been deleted (happens regularly on scratch). Follow the instructions to reinstall miniconda.
- Something like:
-bash: export: `QUERY:=': not a valid identifier
-bash: export: `[email protected]%[email protected]/COSMODEFINITIONS/DEFINITIONS/:==>': not a valid identifier
-bash: export: `[email protected]%[email protected]=': not a valid identifier
-bash: export: `~AEC/SHARE/ECCODES/DEFINITIONS/=': not a valid identifier
This error is due to same changes on Daint on 10.9.2022. To solve this issue, you need to delete your local conda version and install miniconda again. Don't forget to pull the newest version of icon-vis before installing the psyplot environment again.
This repo has been developed by:
- Annika Lauber (C2SM) - [email protected]
- Victoria Cherkas (MeteoSwiss) - [email protected]
Whenever using psyplot for a publication it should be cited https://psyplot.github.io/psyplot/#how-to-cite-psyplot.