Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix various problems with S3 related code #3090

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

DennisHeimbigner
Copy link
Collaborator

re: PR #3068

Part of splitting PR 3068.

General goal is to fix minor bugs and issues involving S3 code outside of libnczarr.

  1. libsrc/s3io.c:
    • Fix handling of error output of NC_s3sdkinfo.
    • Implement delete arg to s3io_close
  2. Add support for Zarr-Over-HTTP (ZOH) protocol, but leave disabled until given the go-ahead from Manuel Reis.
  3. Change the ncs3sdk API to light of eventual addition of Zarr V3 support.
  4. Modify s3cleanup.in to catch and remove some previously overlooked entries in the Unidata S3 test bucket.
  5. Modify s3gc.in to catch and properly remove Unidata S3 test bucket entries with old UIDs.
  6. Modify ds3util.c:
    • Support use of a non-standard port -- H/T Manuel Reis
    • Clean up some memory leaks
    • Add disabled ZOH support
    • Better doc on how URLs are interpreted.
    • Document how .aws/config and .aws/credentials files are parsed.
  7. Repair bugs in the internal S3 reader/writer module.
    • Allow API functions to return an http code (see also nch5s3comms.h).
    • Fix some casting warnings.
  8. Fixes to ncs3sdk_h5.c, the dispatch wrapper for the internal S3 reader/writer
    • Move httptonc to this file.
    • Make conform to API modifications
    • Refactor the key search code.
    • Rename getkeys to list and searchkeys to listall.
  9. Fixes to nczarr_test/s3util.c
    • Reflect ds3util.c API changes: e.g search->listall.
  10. Add s3util.c to v3_nczarr_directory
  11. Extend unit_test/test_s3dk.c to reflect S3 API changes.
  12. Extend unit_test/run_s3sdk.sh to add new tests involving S3 API changes.

re: PR Unidata#3068

Part of splitting PR 3068.

General goal is to fix minor bugs and issues involving
S3 code outside of libnczarr.

1. libsrc/s3io.c:
    - Fix handling of error output of NC_s3sdkinfo.
    - Implement delete arg to s3io_close
2. Add support for Zarr-Over-HTTP (ZOH) protocol, but leave disabled until given the go-ahead from Manuel Reis.
3. Change the ncs3sdk API to light of eventual addition of Zarr V3 support.
4. Modify s3cleanup.in to catch and remove some previously overlooked entries in the Unidata S3 test bucket.
5. Modify s3gc.in to catch and properly remove Unidata S3 test bucket entries with old UIDs.
6. Modify ds3util.c:
    - Support use of a non-standard port -- H/T Manuel Reis
    - Clean up some memory leaks
    - Add disabled ZOH support
    - Better doc on how URLs are interpreted.
    - Document how .aws/config and .aws/credentials files are parsed.
7. Repair bugs in the internal S3 reader/writer module.
    - Allow API functions to return an http code (see also nch5s3comms.h).
    - Fix some casting warnings.
8. Fixes to ncs3sdk_h5.c, the dispatch wrapper for the internal S3 reader/writer
    - Move httptonc to this file.
    - Make conform to API modifications
    - Refactor the key search code.
    - Rename getkeys to list and searchkeys to listall.
9. Fixes to nczarr_test/s3util.c
    - Reflect ds3util.c API changes: e.g search->listall.
10. Add s3util.c to v3_nczarr_directory
11. Extend unit_test/test_s3dk.c to reflect S3 API changes.
12. Extend unit_test/run_s3sdk.sh to add new tests involving S3 API changes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant