IFS open-data support

[joelfiddes]

All (most) fields required for TopoPyScale now available for ECMWF forecast open data:

https://github.com/ecmwf/ecmwf-opendata
https://www.ecmwf.int/en/forecasts/datasets/open-data

I have started implementation and made commits of first working version. Task/challenges included:

1. Renaming dimension names (lat, lon, plev -> latitude, longitude, level)
2. renaming variable names (2t, 2d -> t2m, d2m)
3. Conversion gh on pressure levels to z
4. create z on surface from msl, sp and t2m
5. Deaccumulation. Forecast data is accumulaed from beginning of forecast ie step=0, era 5 was from previous step, here we convert to era5 mode
6. timestep 3h at step 0-144 and 6h steps 144-244 - some implementation needed to allow steps other than era5 1h
7. New module fetch_ifs_forecast (needs to be neatened up still) used in place of module era5 - specified in config (expansion of class)
8. grib format and geo subset of fc data
9. 15 May 2024: pressure level units converted from Pa to hPa
10. 15 May 2024: pressure level order reverse, starts at top of atmosphere.

Workflow

1. run normal TopoPyscale
2. build archive (T-10d) of forecast data. This means we can cover the missing 5-10 days in immediate past and run 10 days into future.
3. Run TopoPyscale forecast (hindcast 3h data)
4. Run TopoPyscale forcast (steps 0-144 3h data)
5. Run TopoPyscale forecast (steps 144-244 6h data)
6. Interpolate all to 1h timestep (see TopoClIM)
7. Merge datasets
8. Run FSM

todo

• [ SOLVED May 14 PA -> hPA conversion of pressure levels ] 1. currently runs fine on branch "joel_testJune2023" NOT on main. some warnings on branch run that maybe clue, probably an issue with "z" (missing parameter) computations (see code output below). 30000hPa would be 2000m below sea level, right. Should be 300hPA so 2 orders magnitude out. Check units of gh on plev!
• 2. Handle automatisation of hindcast, forecast1, forcast2
• 3. Handle disagg to 1H step and merge of era5, hindcast, forecast1 and forecast2

---> Storing point 87 to outputs/tmp/
---> WARNING: Point 99 is -6806.0 m lower than the 30000.0 hPa geopotential
=> Values sampled from Psurf and lowest Plevel. No vertical interpolation
---> Storing point 96 to outputs/tmp/
Downscaling t,q,p,tp,ws, wd for point: 88
Downscaling t,q,p,tp,ws, wd for point: 97
---> WARNING: Point 99 is -6599.0 m lower than the 30000.0 hPa geopotential
=> Values sampled from Psurf and lowest Plevel. No vertical interpolation
---> Storing point 88 to outputs/tmp/
---> WARNING: Point 99 is -6692.0 m lower than the 30000.0 hPa geopotential
=> Values sampled from Psurf and lowest Plevel. No vertical interpolation
---> Storing point 97 to outputs/tmp/
Downscaling t,q,p,tp,ws, wd for point: 89
Downscaling t,q,p,tp,ws, wd for point: 98
---> WARNING: Point 99 is -8328.0 m lower than the 30000.0 hPa geopotential
=> Values sampled from Psurf and lowest Plevel. No vertical interpolation
---> Storing point 89 to outputs/tmp/
---> WARNING: Point 99 is -8579.0 m lower than the 30000.0 hPa geopotential
=> Values sampled from Psurf and lowest Plevel. No vertical interpolation
---> Storing point 98 to outputs/tmp/

Dummy results GST (only 3 day forecast data run! 100 clusters) look reasonable: