Metadata for New Processing Campaign

[cmarshak]

Goal

Update the GUNW metadata to:

1. Make searching for (newly generated) products from coming processing campaign easier
   • Specifically, being able to uniquely identify a GUNW on CMR using information that will be had a job-submission to avoid re-submitting GUNWs for generation (de-duplication)
   • Being able to uniquely identify GUNWs would make de-duplication faster and more efficiently
2. Update metadata with fixed-frame id
3. Include / update start-time to include reference and secondary scene.
4. Ensure some backward compatibility with previous products / hyp3 job submission.

Current Approach

For the GUNWs, they are uniquely identified by input reference and secondary scenes. We download entire corpus due to this data being uniquely specific to echo10 using Andrew Johnston's queries (see "Checking CMR" here).

Worth investigating for current GUNWs

Finding ways to query unique products that we want to generate is highly valuable because we can then multithread these requests and quickly de-duplicate. In an ideal situation, we would just request GUNWs that have specific reference/secondary scenes. However currently this is not possible.

We do generate a 4 digit hash at the end of a GUNW id, i.e. S1-GUNW-D-R-144-tops-20230125_20221220-140020-00121W_00036N-PP-28c2-v2_0_6 has hash 28c2. This is generated using the string form the sorted reference and secondary scenes as indicated here. We would then assume "exact" GUNW match and not containment of reference and secondary scenes. This is ok.

(From Andrew) The CMR Search API supports searching against keywords using wildcards, see the "pattern" keyword under Parameter Options. Performance might not be great for these searches, but you can try it out and see if it meets your needs. For example, to find all GUNWs with "28c2" in their name: https://cmr.earthdata.nasa.gov/search/granules.umm_json?short_name=SENTINEL-1_INTERFEROGRAMS&granule_ur=*28c2*&options[granule_ur][pattern]=true. But even though there are $(10 + 26)^4 = 1.7e6$ possibilities, there are 14 hits for different GUNWs. This will require some more filtering, i.e. they occur across unique tracks and that is easily accessible from the IDs or CMR. But this is work. An example to get a unique hit would be to use track and date in addition to hash as in https://cmr.earthdata.nasa.gov/search/granules.umm_json?short_name=SENTINEL-1_INTERFEROGRAMS&granule_ur=*065*20170220*28c2*&options[granule_ur][pattern]=true.
 

Frame IDs

For the frames, the unique identification is:

1. Frame ID (assuming it does not change)
2. Reference Date
3. Secondary Date

This will not be compatible for de-duplicating old products since 2 GUNWs could potentially share 2 of the same SLCs and span slightly different areas because are using the fixed frame's bounding box to direct ISCE2 processing.

It is also clear we need to get Secondary Date into CMR correctly as we are only using start/stop times from the reference date as indicated here.

Will also have to add frame ID

Questions / Comments

1. Prioritize frame de-duplication and metadata updates.
2. Will likely be minor/patch dataset increment making any changes distinctive

[cmarshak]

Here is a working PR: #102

As noted - we will want to coordinate with ASF and make sure the schema in the tests matches any updated ingest schema.

[cmarshak]

So, from what I am gleaning from David/Piyush, it will be best to:

1. Keep start/stop times as is (i.e. for reference granule)
2. Add temporal_baseline_days (or something like that) into metadata
3. Add frame_id for frames