Download historical GRIB files instead of CSV's

[diegormsouza]

Hi Alberto,

Would it possible to download GRIB files instead of CSV's using a similar methodology?

Thanks,
Diego

[albertotb]

Writing arbitrarily GRIB files is not something that can be done with xarray (yet). See ecmwf/cfgrib#18 for a discussion. However, it is fairly easy to download files from GFS using the OpenDAP protocol and writing them to a NetCDF file (instead of GRIB2). Note that you can always download the whole GRIB2 file directly. The advantage of OpenDAP is that you can subset the file on the server, so you do not have to download all the variables/coordinates/time steps. I have create two example scripts in a new branch https://github.com/albertotb/get-gfs/tree/xarray (still a proof of concept), one for the operational server and another one for the historical server. You can see the catalogs with the available dates here:

• Operational server: https://nomads.ncep.noaa.gov/dods/gfs_0p25_1hr
• Historical server: https://www.ncei.noaa.gov/thredds/catalog/model-gfs-004-files-old/catalog.html

As for the available variables, you can just navigate to any file in the previous links. To download the data, just checkout the branch and run:

./get_gfs_xarray.py --date 2021-07-24 --run 0 --hour 0 

or

./get_gfs_hist_xarray.py --date 2020-05-15 --run 6 --time 0

These scripts are still WIP and currently only download a couple of test variable for all lat/lons, but it would not be hard to improve them so that:

• Lats and lons are subsettable
• Variables are passed as args to the script

If this is something useful to you just let me know and I will try to make those changes.

[diegormsouza]

Thanks a lot Alberto,

With this sentence: "Note that you can always download the whole GRIB2 file directly.", I realized I could use the "requests" library to download data directly from the historical servers (both analysis and forecasts):

https://www.ncei.noaa.gov/thredds/catalog/model-gfs-004-files/catalog.html
https://www.ncei.noaa.gov/thredds/catalog/model-gfs-004-files-old/catalog.html
https://www.ncei.noaa.gov/thredds/catalog/model-gfs-g4-anl-files/catalog.html
https://www.ncei.noaa.gov/thredds/catalog/model-gfs-g4-anl-files-old/catalog.html
https://www.ncei.noaa.gov/thredds/catalog/model-gfs-003-files/catalog.html
https://www.ncei.noaa.gov/thredds/catalog/model-gfs-003-files-old/catalog.html
https://www.ncei.noaa.gov/thredds/catalog/model-gfs-g3-anl-files/catalog.html
https://www.ncei.noaa.gov/thredds/catalog/model-gfs-g3-anl-files-old/catalog.html

... as I was doing with the NOMADS server. Right now I'm OK with downloading the full GRIB files from the historical servers.

I'll test the example scripts you mentioned and check the results! Thanks Alberto!

[zxdawn]

@diegormsouza Thanks for your updates!
I'm trying to download historical GFS forecast data from 2018-2020.
According to your links, the gfs-003 is the option. However, the old one ends at 202005 and the new one begins at 202107.
Do you know why there're missing data between them?
Thanks,
Xin

[albertotb]

In this website: https://www.ncei.noaa.gov/products/weather-climate-models/global-forecast It seems that you can access archived data but with 0.5º resolution: https://www.ncei.noaa.gov/thredds/catalog/model-gfs-004-files/catalog.html I don't know if you can get the same data with finer resolution Best regards Alberto

…

On Wed, 26 Jan 2022 at 09:46, Xin Zhang ***@***.***> wrote: @diegormsouza <https://github.com/diegormsouza> Thanks for your updates! I'm trying to download historical GFS forecast data from 2018-2020. According to your links, the gfs-003 is the option. However, the old one <https://www.ncei.noaa.gov/thredds/catalog/model-gfs-003-files-old/catalog.html> ends at 202005 and the new one <https://www.ncei.noaa.gov/thredds/catalog/model-gfs-003-files/catalog.html> begins at 202107. Do you know why there're missing data between them? Thanks, Xin — Reply to this email directly, view it on GitHub <#9 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABRWQAXM7DQM3Z5UWCTRNR3UX6YETANCNFSM457LMSLA> . You are receiving this because you commented.Message ID: ***@***.***>

[zxdawn]

@albertotb Thanks for your reply. Here's what I found:

• https://www.ncei.noaa.gov/thredds/catalog/model-gfs-003-files-old/catalog.html
  201508-202005, 1 degree resolution
• https://www.ncei.noaa.gov/thredds/catalog/model-gfs-003-files/catalog.html
  202107-202201, 1 degree resolution
• https://www.ncei.noaa.gov/thredds/catalog/model-gfs-004-files-old/catalog.html
  201908-202005, 0.5 degree resolution
• https://www.ncei.noaa.gov/thredds/catalog/model-gfs-004-files/catalog.html
  202005-202201, 0.5 degree resolution