Add GNSSRO bufr2ioda and json files #1448

XuanliLi-NOAA · 2025-01-16T19:36:41Z

Description

Add JSON and bufr2ioda python scripts for GNSSRO data. Each GNSSRO mission has a separate ioda obs file.

Companion PRs

N/A

Issues

Refs #750

Automated CI tests to run in Global Workflow

CoryMartin-NOAA · 2025-01-17T15:29:57Z

Should these go in here or in spoc? @emilyhcliu what are your thoughts?

emilyhcliu · 2025-01-17T15:49:20Z

@CoryMartin-NOAA
The json and Python scripts in this PR are written in bufr-query low-level Python API.
These are good to use with the current setup in GDASApp. So, they should be checked in here in GDASApp so Xuanli can keep track of changes and repeat her tests.

In the meanwhile, @nicholasesposito will create these low-level Pythons into one Python script with high-level API and mapping files. These will be checked into SPOC.

Once we have most of the Python Script, mapping files in SPOC, we will work on GDASApp to use them from SPOC.

CoryMartin-NOAA · 2025-01-17T15:50:26Z

ok sounds good thanks @emilyhcliu I just wanted to double check

CoryMartin-NOAA

a few minor comments, and I didn't closely look at the converters but assume they have been thoroughly tested

parm/atm/jcb-prototype_3dvar.yaml.j2

CoryMartin-NOAA · 2025-01-17T15:58:18Z

ush/ioda/bufr2ioda/bufr2ioda_gnssro_cosmic2.py

@@ -0,0 +1,805 @@
+#!/usr/bin/env python3
+# (C) Copyright 2023 NOAA/NWS/NCEP/EMC


if we are putting dates in all of these, they should probably be 2025 not 2023. We can also just not put dates in it at all, we don't need to put a license string in every file like the JCSDA does

The date line has been removed from the python scripts.

emilyhcliu · 2025-01-17T16:00:04Z

@XuanliLi-NOAA one question.
Is it possible to write one Python to convert all platforms?
GPSRO data is in one subset. It should be possible to write one Python to handle all platforms.
Are these platforms using the same BUFR table?

No worries, what you have in this PR is OK. But. we should try to create one Python script with high-level Python API for GPSRO.
@nicholasesposito will work with you on this.

Co-authored-by: Cory Martin <[email protected]>

…C/GDASApp into feature/gnssro_bufr2ioda

XuanliLi-NOAA · 2025-01-17T16:28:18Z

@emilyhcliu , it would be great to have one Python script to convert all platforms. Yes, let's do that with the with high-level Python API.

emilyhcliu · 2025-01-17T16:29:56Z

@emilyhcliu , it would be great to have one Python script to convert all platforms. Yes, let's do that with the with high-level Python API.

Do all GPSRO platform use the same bufr table?

XuanliLi-NOAA · 2025-01-17T16:46:15Z

@emilyhcliu, yes, for the test date 00z 2024-02-19, they use the same bufr table. But after June 2024, we have new EUMETSAT Spire data, this has an additional BUFR mnemonic "SASBID". Should we consider that in the high level python API?

@XuanliLi-NOAA Yes, we should include the new one. It is good that all platforms share the same bufr table mnemonics. We can have one mapping file to handle all of them. So, the one Python script with high-level API should be doable for GPSRO.
Thanks for the information.

@emilyhcliu Sounds good, thank you for confirming that.

nicholasesposito

Recommend changing the global attributes, but @emilyhcliu should comment on that. The rest looked fine with the exception of I think changing the data_format description.

nicholasesposito · 2025-01-22T18:09:29Z

parm/ioda/bufr2ioda/bufr2ioda_gnssro_cosmic2.json

+  "ioda_directory"   : "{{ COM_OBS }}",
+  "subsets"          : [ "NC003010" ],
+  "data_description" : "Satellite radio occultation data", 
+  "data_provider"    : "U.S. NOAA",


For the data_provider, I believe we are using "U.S. NOAA NCEP". I won't comment on every single one.

nicholasesposito · 2025-01-22T18:16:29Z

ush/ioda/bufr2ioda/bufr2ioda_gnssro_cosmic2.py

+
+    # Create Global attributes
+    logger.debug(f" ... ... Create global attributes")
+    obsspace.write_attr('source_file', bufrfile)


I think the global attributes need to be closer to the standards.
At a minimum,
data_format -> dataOriginalFormatSpec
subsets -> platforms
data_type should be deleted altogether since it's redundant with "source"
data_provider -> dataProviderOrigin

I'm unsure of the others. but I would look at the global attributes tab of this document to help figure it out.
https://docs.google.com/spreadsheets/d/1fHb2QIHOwgp4d6plyn9Dj9HZFZsDRxcPaUV6hdTb5BU/edit?gid=46200857#gid=46200857

@emilyhcliu do you agree?

I won't mark each script.

I think fixing it to the minimum as @nicholasesposito is good.
@XuanliLi-NOAA could you modify three global attribute name based on nick's suggestions.
Thanks!

XuanliLi-NOAA · 2025-01-24T19:35:59Z

@emilyhcliu and @nicholasesposito data_provider has been corrected in the json files to reflect the actual originating center. data_format has been changed to dataOriginalFormatSpec in the python scripts. I'm not sure if we should replace subsets with platforms, as subsets is a bufr message for the name of the subset.

nicholasesposito

I think just clean up some of the comments unless they are needed. Otherwise no glaring issues from me. Looks very complicated so thank you for taking the time to do all this!

nicholasesposito · 2025-01-31T21:44:17Z

ush/ioda/bufr2ioda/bufr2ioda_gnssro_cosmic2.py

+    impp1 = r.get('impactParameterRO_roseq2repl1', 'latitude')
+    impp2 = r.get('impactParameterRO_roseq2repl2', 'latitude')
+    impp3 = r.get('impactParameterRO_roseq2repl3', 'latitude')
+#    impp1 = r.get('impactParameterRO_roseq2repl1', 'latitude', type='double')


Do we need these comments?

nicholasesposito · 2025-01-31T21:51:00Z

ush/ioda/bufr2ioda/bufr2ioda_gnssro_cosmic2.py

+        print(' ... dim = ', nobs)
+
+        # iodafile = f"{cycle_type}.t{hh}z.{data_type}.tm00.{data_format}.nc"
+        # iodafile = f"{cycle_type}.t{hh}z.{ioda_data_type}.{mission}.tm00.{data_format}.nc"


Unneeded comments I think

XuanliLi-NOAA · 2025-01-31T23:31:54Z

@nicholasesposito @emilyhcliu Thank you so much, Nick. I cleaned up the python scripts and also uncommented mefr1 to maintain consistency with the other variables.

Add GNSSRO bufr2ioda and json files

4dca3a9

XuanliLi-NOAA self-assigned this Jan 16, 2025

XuanliLi-NOAA added 3 commits January 16, 2025 14:04

update bufr2ioda scripts

89f0e43

update bufr2ioda python scripts

ba3fb3e

update bufr2ioda scripts

a396f11

XuanliLi-NOAA requested review from emilyhcliu, ADCollard, jianjunj, CoryMartin-NOAA and nicholasesposito January 16, 2025 20:16

CoryMartin-NOAA reviewed Jan 17, 2025

View reviewed changes

XuanliLi-NOAA and others added 3 commits January 17, 2025 11:13

Update parm/atm/jcb-prototype_3dvar.yaml.j2

fd59b1c

Co-authored-by: Cory Martin <[email protected]>

remove the date line

c8e37bf

Merge branch 'feature/gnssro_bufr2ioda' of https://github.com/NOAA-EM…

2024752

…C/GDASApp into feature/gnssro_bufr2ioda

nicholasesposito reviewed Jan 22, 2025

View reviewed changes

modified variable names in json and python scripts

9fdbe6a

nicholasesposito previously approved these changes Jan 31, 2025

View reviewed changes

clean up the python script

9f952b0

XuanliLi-NOAA dismissed nicholasesposito’s stale review via 9f952b0 January 31, 2025 23:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add GNSSRO bufr2ioda and json files #1448

Add GNSSRO bufr2ioda and json files #1448

XuanliLi-NOAA commented Jan 16, 2025

CoryMartin-NOAA commented Jan 17, 2025

emilyhcliu commented Jan 17, 2025

CoryMartin-NOAA commented Jan 17, 2025

CoryMartin-NOAA left a comment

CoryMartin-NOAA Jan 17, 2025

XuanliLi-NOAA Jan 17, 2025

emilyhcliu commented Jan 17, 2025

XuanliLi-NOAA commented Jan 17, 2025

emilyhcliu commented Jan 17, 2025

XuanliLi-NOAA commented Jan 17, 2025 •

edited

Loading

nicholasesposito left a comment

nicholasesposito Jan 22, 2025

nicholasesposito Jan 22, 2025

emilyhcliu Jan 24, 2025 •

edited

Loading

XuanliLi-NOAA commented Jan 24, 2025 •

edited

Loading

nicholasesposito left a comment

nicholasesposito Jan 31, 2025

nicholasesposito Jan 31, 2025

XuanliLi-NOAA commented Jan 31, 2025 •

edited

Loading

		@@ -0,0 +1,805 @@
		#!/usr/bin/env python3
		# (C) Copyright 2023 NOAA/NWS/NCEP/EMC

Add GNSSRO bufr2ioda and json files #1448

Are you sure you want to change the base?

Add GNSSRO bufr2ioda and json files #1448

Conversation

XuanliLi-NOAA commented Jan 16, 2025

Description

Companion PRs

Issues

Automated CI tests to run in Global Workflow

CoryMartin-NOAA commented Jan 17, 2025

emilyhcliu commented Jan 17, 2025

CoryMartin-NOAA commented Jan 17, 2025

CoryMartin-NOAA left a comment

Choose a reason for hiding this comment

CoryMartin-NOAA Jan 17, 2025

Choose a reason for hiding this comment

XuanliLi-NOAA Jan 17, 2025

Choose a reason for hiding this comment

emilyhcliu commented Jan 17, 2025

XuanliLi-NOAA commented Jan 17, 2025

emilyhcliu commented Jan 17, 2025

XuanliLi-NOAA commented Jan 17, 2025 • edited Loading

nicholasesposito left a comment

Choose a reason for hiding this comment

nicholasesposito Jan 22, 2025

Choose a reason for hiding this comment

nicholasesposito Jan 22, 2025

Choose a reason for hiding this comment

emilyhcliu Jan 24, 2025 • edited Loading

Choose a reason for hiding this comment

XuanliLi-NOAA commented Jan 24, 2025 • edited Loading

nicholasesposito left a comment

Choose a reason for hiding this comment

nicholasesposito Jan 31, 2025

Choose a reason for hiding this comment

nicholasesposito Jan 31, 2025

Choose a reason for hiding this comment

XuanliLi-NOAA commented Jan 31, 2025 • edited Loading

XuanliLi-NOAA commented Jan 17, 2025 •

edited

Loading

emilyhcliu Jan 24, 2025 •

edited

Loading

XuanliLi-NOAA commented Jan 24, 2025 •

edited

Loading

XuanliLi-NOAA commented Jan 31, 2025 •

edited

Loading