Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: Unknown CSLC Query behavior when a frame has no sensing_datetimes in DISP-S1 DB #977

Open
philipjyoon opened this issue Sep 4, 2024 · 0 comments
Assignees
Labels
bug Something isn't working needs triage Issue that requires triage

Comments

@philipjyoon
Copy link
Contributor

Checked for duplicates

Yes - I've already checked

Describe the bug

Traceback (most recent call last): File "/home/ops/verdi/ops/opera-pcm/data_subscriber/daac_data_subscriber.py", line 319, in <module> main() File "/home/ops/verdi/ops/opera-pcm/util/exec_util.py", line 35, in wrapper status = func(*args, **kwargs) File "/home/ops/verdi/ops/opera-pcm/data_subscriber/daac_data_subscriber.py", line 56, in main run(sys.argv) File "/home/ops/verdi/ops/opera-pcm/data_subscriber/daac_data_subscriber.py", line 99, in run results["query"] = run_query(args, token, es_conn, cmr, job_id, settings) File "/home/ops/verdi/ops/opera-pcm/data_subscriber/daac_data_subscriber.py", line 137, in run_query return cmr_query.run_query(args, token, es_conn, cmr, job_id, settings) File "/home/ops/verdi/ops/opera-pcm/data_subscriber/query.py", line 50, in run_query granules = self.query_cmr(args, token, cmr, settings, query_timerange, now) File "/home/ops/verdi/ops/opera-pcm/data_subscriber/cslc/cslc_query.py", line 429, in query_cmr self.extend_additional_records(all_granules) File "/home/ops/verdi/ops/opera-pcm/data_subscriber/cslc/cslc_query.py", line 80, in extend_additional_records parse_cslc_native_id(granule_id, self.burst_to_frames, self.disp_burst_map_hist)) File "/home/ops/verdi/ops/opera-pcm/data_subscriber/cslc_utils.py", line 348, in parse_cslc_native_id acquisition_cycles[frame_id] = determine_acquisition_cycle_cslc(acquisition_dts, frame_id, frame_to_bursts) File "/home/ops/verdi/ops/opera-pcm/data_subscriber/cslc_utils.py", line 172, in determine_acquisition_cycle_cslc day_index, seconds = sensing_time_day_index(acquisition_dts, frame_number, frame_to_bursts) File "/home/ops/verdi/ops/opera-pcm/data_subscriber/cslc_utils.py", line 85, in sensing_time_day_index return (_calculate_sensing_time_day_index(sensing_time, frame.sensing_datetimes[0])) IndexError: list index out of range

This error happened because when the current version of DISP-S1 database json was created, the update tool did not find all bursts for frame 28471 that the original claimed it had and so it didn't mark any sensing times for that frame.
This is the issue that ADT (Scott S) and I've been trying to resolve. This current version of database json was created by a tool that I wrote; but going forward ADT will take back that responsibility (decided last Friday) providing we give them the entire list of CSLC files from CMR (which I'm working on but running into CMR issues, you've been CCed)

So once we have an updated DISP-S1 database json, this will be a non-issue.

What did you expect?

Actually not sure. A frame in the in DISP-S1 database having no sensing_datetime is an impossibility: if it were true, that frame does not belong in the database at all. Probably best to write out a meaningful error to the user but no behavior change from current.

Reproducible steps

.py', 'query', '--collection-shortname=OPERA_L2_CSLC-S1_V1', '--endpoint=OPS', '--start-date=2024-09-03T08:01:09Z', '--end-date=2024-09-03T09:01:09Z', '--release-version=3.1.0-rc.5.0', '--job-queue=opera-job_worker-cslc_data_download', '--chunk-size=1', '--k=5', '--m=1', '--grace-mins=120', '--max-revision=1000', '--temporal-start-date=2024-08-04T09:01:09Z', '--transfer-protocol=auto' using the database file created mid-August 2024

This error can be reduced to the following:
```burst_id, acquisition_dts, acquisition_cycles, frame_ids = \
        cslc_utils.parse_cslc_native_id(
            "OPERA_L2_CSLC-S1_T107-227769-IW3_20240902T003138Z_20240903T073341Z_S1A_VV_v1.1", burst_to_frames,
            disp_burst_map_hist)```

Environment

- Version of this software [e.g. vX.Y.Z]
- Operating System: [e.g. MacOSX with Docker Desktop vX.Y]
...
@philipjyoon philipjyoon added bug Something isn't working needs triage Issue that requires triage labels Sep 4, 2024
@philipjyoon philipjyoon self-assigned this Sep 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working needs triage Issue that requires triage
Projects
None yet
Development

No branches or pull requests

1 participant