The cesmep_simu_finder

Data access and finding the data is one of the main tasks we do in climate model outputs analysis.

The cesmep_simu_finder builds on the data access functionality of CliMAF to:

• help you find your data (use it as a browser) with keywords
• check that your datasets specified in the datasets_setup of a comparison points at existing files

Use the cesmep_simu_finder as a data browser

The cesmep_simu_finder provides a way to browse your existing files. The user simply has to fill in the header. For instance, this request will return the list of all the historical simulations produced by p86denv with the IPSLCM6 model:

# --------------------------------------------------------------------------
# -- USER INTERFACE
# -- Specify your simulation/dataset request here
# -- This tool helps you to browse the existing results using '*'
# -- and refine your request

# -- Dictionary of your requested dataset -> same dictionary as in the models list in datasets_setup.py
dat_dict = dict(project = 'IGCM_OUT',
                login   = 'p86denv',
                model   = 'IPSLCM6',
                experiment = 'historical',
                simulation = '*',
                period     = '0001_9998',
                frequency  = 'monthly',
                )

# -- Variable dictionary -> specifications that might be project-dependant to reach one file
var_dict = dict(variable = 'tas',
                DIR      = 'ATM')
# --------------------------------------------------------------------------

We have to specify a variable name to complete the request. It is possible to set it to '*' but with the risk of having too many listed files as output. Note that in the example, 'DIR' is optional. But if the variable name alone points at multiple files (ATM and SBG directories, Amon and Lmon tables...), you can specify it here to check that your final request points at one simulation (potentially with multiple files covering the period). You can then add those additional specifications to the variable (in a dictionary) in the params_[...].py file if you use the Atlas Explorer mechanism (see the Atlas Explorer page).

Then you execute the cesmep_simu_finder like this:

python cesmep_simu_finder.py

Note: if you encounter this error:

Traceback (most recent call last):
  File "cesmep_simu_finder.py", line 31, in <module>
    from CM_atlas import *
ImportError: No module named CM_atlas

it simply means that your environment does not point at the CliMAF install we use for the C-ESM-EP. You just have to do in your shell:

source setenv_C-ESM-EP.sh

and re-try.

The cesmep_simu_finder typically returns a message like this if successful:

 CliMAF found those files with your request  {'period': '0001_9998', 'simulation': '*', 'project': 'IGCM_OUT', 'experiment': 'historical', 'frequency': 'monthly', 'variable': 'tas', 'login': 'p86denv', 'root': '[...]', 'model': 'IPSLCM6'}
-->
[...]/p86denv/IPSLCM6/DEVT/historical/CM6014-historical-01/ATM/Analyse/TS_MO/CM6014-historical-01_18500101_20091231_1M_t2m.nc
[...]/p86denv/IPSLCM6/PROD/historical/CM61-LR-hist-03.1990/ATM/Analyse/TS_MO/CM61-LR-hist-03.1990_18500101_19591231_1M_t2m.nc
[...]/p86denv/IPSLCM6/PROD/historical/CM61-LR-hist-03.1990/SRF/Analyse/TS_MO/CM61-LR-hist-03.1990_18500101_19591231_1M_t2m.nc

 -- Check that your request points to only one simulation, variable and frequency (and table for MIP simulations)

and this message if it didn't find any file (error on the login):

No File found for your request --  {'period': '0001_9998', 'simulation': '*', 'project': 'IGCM_OUT', 'experiment': 'historical', 'frequency': 'monthly', 'variable': 'tas', 'login': 'p86denvb', 'root': '[...]', 'model': 'IPSLCM6'}

If your request didn't find what you expected, you can check the patterns that CliMAF tried:

debug    : Globbing 0 files for varname on [...]/p86denvb/IGCM_OUT/IPSLCM6/*/historical/*/*/Analyse/*/*_????????_????????_1M_t2m.nc :
debug    : Globbing 0 files for filenamevar on [...]/p86denvb/IGCM_OUT/IPSLCM6/*/historical/*/*/Analyse/*/*_????????_????????_1M_histmth.nc:
debug    : Globbing 0 files for varname on [...]/p86denvb/IPSLCM6/*/historical/*/*/Analyse/*/*_????????_????????_1M_t2m.nc :
debug    : Globbing 0 files for filenamevar on [...]/p86denvb/IPSLCM6/*/historical/*/*/Analyse/*/*_????????_????????_1M_histmth.nc:
debug    : Globbing 0 files for varname on [...]/p86denvb/IGCM_OUT/IPSLCM6/*/historical/*/*/Analyse/1M*/*_1M_????_????_1M_t2m.nc :
debug    : Globbing 0 files for filenamevar on [...]/p86denvb/IGCM_OUT/IPSLCM6/*/historical/*/*/Analyse/1M*/*_1M_????_????_1M_histmth.nc:
debug    : Globbing 0 files for varname on [...]/p86denvb/IPSLCM6/*/historical/*/*/Analyse/1M*/*_1M_????_????_1M_t2m.nc :
debug    : Globbing 0 files for filenamevar on [...]/p86denvb/IPSLCM6/*/historical/*/*/Analyse/1M*/*_1M_????_????_1M_histmth.nc:

Set clog('debug') in cesmep_simu_finder to output this information, or set it to 'critical' if you don't need such verbosity.

You can then correct your request (or your CliMAF data project definition) accordingly.

Use the cesmep_simu_finder to check your data specification in the datasets_setup of a comparison

Most of the errors encountered by the C-ESM-EP users are coming from a bad data specification. The cesmep_simu_finder can help you check that your data specifications in the datasets_setup.py of a given comparison is correct (i.e. that they actually point at existing files).

You simply have to pass your comparison name as an argument to the cesmep_simu_finder:

python cesmep_simu_finder.py my_comparison [n]

You can pass an optional argument n to focus on the nth dataset in your models list (in datasets_setup) and not check all of them.