Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Don't crop when there is no data #254

Merged
merged 11 commits into from
May 7, 2024
Merged

Don't crop when there is no data #254

merged 11 commits into from
May 7, 2024

Conversation

jhkennedy
Copy link
Contributor

@jhkennedy jhkennedy commented May 4, 2024

4612/4636 AutoRIFT job failures in the last two weeks are because they are attempting to crop a data file with no valid data, resulting in an error that looks like this:

Traceback (most recent call last):
  File "/opt/conda/envs/hyp3-autorift/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/conda/envs/hyp3-autorift/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/opt/conda/envs/hyp3-autorift/lib/python3.9/site-packages/hyp3_autorift/__main__.py", line 47, in <module>
    main()
  File "/opt/conda/envs/hyp3-autorift/lib/python3.9/site-packages/hyp3_autorift/__main__.py", line 42, in main
    process_entry_point.load()()
  File "/opt/conda/envs/hyp3-autorift/lib/python3.9/site-packages/hyp3_autorift/process.py", line 595, in main
    product_file, browse_file = process(g1, g2, parameter_file=args.parameter_file, naming_scheme=args.naming_scheme)
  File "/opt/conda/envs/hyp3-autorift/lib/python3.9/site-packages/hyp3_autorift/process.py", line 543, in process
    cropped_file = crop_netcdf_product(netcdf_file)
  File "/opt/conda/envs/hyp3-autorift/lib/python3.9/site-packages/hyp3_autorift/crop.py", line 68, in crop_netcdf_product
    grid_x_min, grid_x_max = x_values.min(), x_values.max()
  File "/opt/conda/envs/hyp3-autorift/lib/python3.9/site-packages/numpy/core/_methods.py", line 44, in _amin
    return umr_minimum(a, axis, None, out, keepdims, initial, where)
ValueError: zero-size array to reduction operation minimum which has no identity

Scenes with no valid data will end with the suffix P000.nc, where P000 represents the % valid pixels in the scene.

You can reproduce this failure by running this from the development branch:

hyp3_autorift LC09_L1GT_209113_20240325_20240325_02_T2 LC09_L1GT_209113_20230915_20230915_02_T2

and show it's resolved by re-running from this branch.

This PR

This PR looks at the file name for the uncropped file and only crops if there is a non-zero % of valid pixels. Otherwise it uploads the file as-is

(ITS_LIVE had been uploading them before we added cropping to mark "successfully processed" so those scenes wouldn't get re-processed).

This PR also:

  • drops the ASF naming scheme because we've never used it, resulting in simpler logic
  • moves everything in .io to .utils to prevent shadowing the builtin io and make flake8 happy
  • moves thumbnail creation inside process next to the browse image creation

@jhkennedy jhkennedy added the bug Something isn't working label May 4, 2024
@jhkennedy jhkennedy marked this pull request as ready for review May 7, 2024 20:14
@jhkennedy jhkennedy requested a review from a team May 7, 2024 20:14
@jhkennedy jhkennedy merged commit 326680d into develop May 7, 2024
7 checks passed
@jhkennedy jhkennedy deleted the fix-cropping-failues branch May 7, 2024 21:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant