[New Feature]: Dynamic Compressed CSLC checking, blocking, and running for DISP-S1 workflow forward and reprocessing modes

[philipjyoon]

Checked for duplicates

Yes - I've already checked

Alternatives considered

Yes - and alternatives don't suffice

Related problems

No response

Describe the feature request

DISP-S1 SCIFLO can run if and only if all m compressed cslcs have been satisfied. Compressed cslcs are generated also by the DISP-S1 SCIFLO jobs. CSLC Download jobs currently perform the compressed cslc satiety check and fail if it is not met.

Instead of failing such jobs, we want to block (pause, etc) the download jobs if compressed cslcs have not been satisfied. And when they are satisfied then we can trigger the download jobs.

https://github.com/nasa/opera-sds-pcm/blob/develop/tools/run_disp_s1_historical_processing.py does something very similar so we should refactor that and integrate it into this design.

Quick design:

• Store in GRQ ES information about CSLC download jobs that do not yet have all c-cslcs created.
• In the query job, go through those first, and if any of them have c-cslcs satisfied, submit those download jobs.
  • We could also create a separate job and queue started by lambda that does this. That would be similar to ionosphere download job design in R2