Propose scalable approach to storing irregular gridded data

[agstephens]

Have a look at:

ncdump -h /badc/cmip6/data/CMIP6/CMIP/MOHC/HadGEM3-GC31-MM/historical/r1i1p1f3/SImon/sidmasslat/gn/v20191207/sidmasslat_SImon_HadGEM3-GC31-MM_historical_r1i1p1f3_gn_195001-196912.nc

and

ncdump -h  /badc/cordex/data/CORDEX/output/EUR-44/MOHC/ECMWF-ERAINT/evaluation/r1i1p1/MOHC-HadGEM3-RA/v1/6hr/mrro/v20150701/mrro_EUR-44_ECMWF-ERAINT_evaluation_r1i1p1_MOHC-HadGEM3-RA_v1_6hr_2010010103-2010123121.nc

[agstephens]

Need to also consider capturing the bounds of the axes.

[ellesmith88]

@agstephens
Update on what I’ve found so far:

Looking at the two examples above I wrote some quick code to add to the character scanner to get the metadata about the grid when a grid_mapping variable exists:

    try:
        grid_mapping = ds[da.grid_mapping].attrs
 
    except AttributeError:
        grid_mapping = None

    return {
        'grid_mapping': grid_mapping,
    }

For the cordex example above this returns

"grid_metadata": {
    "grid_mapping": {
        "grid_mapping_name": "rotated_latitude_longitude",
        "grid_north_pole_latitude": 39.25,
        "grid_north_pole_longitude": -162.0,
        "north_pole_grid_longitude": 0.0
    }

I then looked at another example: /group_workspaces/jasmin2/cp4cds1/vol1/data/c3s-cordex/output/EUR-11/IPSL/MOHC-HadGEM2-ES/rcp85/r1i1p1/IPSL-WRF381P/v1/day/psl/v20190212/psl_EUR-11_MOHC-HadGEM2-ES_rcp85_r1i1p1_IPSL-WRF381P_v1_day_20060101-20101231.nc

This has a variable for its grid (rotated_pole), but doesn’t specify grid_mapping_name outside of this, only:

:WEST-EAST_GRID_DIMENSION = 504 ;
:SOUTH-NORTH_GRID_DIMENSION = 492 ;
:BOTTOM-TOP_GRID_DIMENSION = 32 ;
:GRIDTYPE = "C" ;
:GRID_FDDA = 0 ;
:GRID_SFDDA = 0 ;
:GRID_ID = 1 ;
:PARENT_GRID_RATIO = 1 ;

in the global attributes, so would need to be done in a different way.

I've also looked at extracting the bounds/vertices and the dimensions of each coordinate (using ds.coords instead of ds.coords.dims to get all coordinates) - an example json file including this is here: https://github.com/roocs/dachar/blob/multidim_grids/outputs/register/CORDEX/output/EUR-44/MOHC/ECMWF-ERAINT/evaluation/r1i1p1/MOHC-HadGEM3-RA/v1.6hr.mrro.v20150701.json

[ellesmith88]

I also thought about the problem of extracting the max and min of the data when the datasets are too large. My thoughts were to identify when this was the case and only run in quick mode. Maybe identify this by the shape of the data array?

[agstephens]

I also thought about the problem of extracting the max and min of the data when the datasets are too large. My thoughts were to identify when this was the case and only run in quick mode. Maybe identify this by the shape of the data array?

@ellesmith88 please put this in a new/separate issue.

[agstephens]

@ellesmith88: thanks for looking into this. Let's chat about it tomorrow.

It might be worth having a read around the ESMF regridding tool and how it classifies grids, i.e.:

https://www.earthsystemcog.org/projects/esmf/regridding

https://www.earthsystemcog.org/projects/esmf/regridding_7_1_0r

[agstephens]

CF conventions on grid mapppings:

http://cfconventions.org/Data/cf-conventions/cf-conventions-1.8/cf-conventions.html#grid-mappings-and-projections