Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle reprocessing complexity for historical granules in the OPERA_L3_DSWx-HLS_V1 collection #53

Open
torimcd opened this issue Feb 11, 2025 · 0 comments

Comments

@torimcd
Copy link
Contributor

torimcd commented Feb 11, 2025

This is a placeholder issue to make sure we can track the following use case:
The collection OPERA_L3_DSWx-HLS_V1 was previously named OPERA_L3_DSWx-HLS_PROVISIONAL_V1. This means that in Cumulus, there are data files for this collection under two prefixes. Data received prior to the shortname change (~May 2023) lives under the OPERA_L3_DSWx-HLS_PROVISIONAL_V1 prefix, and all data ingested into this collection after the shortname change lives under the OPERA_L3_DSWx-HLS_V1 prefix.

(A) Upstream, the HLS mission is planning to reprocess a number of old data granules that were acquired prior to the shortname change. These reprocessed granules will flow downstream to the OPERA project where they will generate new DSWx-HLS granules, which will be sent to PO.DAAC. These new granules, for old acquisition dates, will go into the OPERA_L3_DSWx-HLS_V1 prefix in the bucket.

(B) Separately, work is underway for a feature to allow granules that were previously ingested to have browse images generated without having to re-ingest the granules. Some of the granules that need browse images generated are under the older OPERA_L3_DSWx-HLS_PROVISIONAL_V1 prefix.

If the HLS reprocessing (A) occurs before the historical browse generation (B), we need to ensure we don't generate browse images for the old granules from OPERA_L3_DSWx-HLS_PROVISIONAL_V1, or else they will overwrite the newer images from the HLS reprocessing downstream in GITC/Worldview.

The easiest way to handle this may be to delete the older granules from the OPERA_L3_DSWx-HLS_PROVISIONAL_V1 bucket when new granules during the HLS reprocessing are received. Otherwise add handling to check both prefixes for a granule's acquisition time, and if it's duplicated, prefer the one with the more recent processing date.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant