New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Preliminary Zarr Implementation + Tests #60

Draft

MrFok wants to merge 8 commits into apache:develop from MrFok:dev

MrFok commented Aug 19, 2022

Generated separate files for Zarr implementation (without disturbing existing pipeline)
Local test generated for Lazy Load Zarr Aggregation (LLZA)
Minor dependency changes, bug fixes (needs checking), commenting
Preliminary methods for multiple parts in collection-manager and granule-ingester

Changes in Proposed Pipeline

Dataset metadata push for new, YAML format (needs dynamic public bucket boolean implementation)
Streamlined processors for xarray Dataset instead of nexus processing
Direct Lazy Load Zarr Aggregation write into S3 bucket instead of tile storage in Cassandra

NOTE: All changes have NOT been completely tested in a full run of the entire pipeline.

drfok added 8 commits

July 28, 2022 15:56


          Loose Local Implementation of LZA

e26d04a


          Adding Missing Files Loose Implementation


          Initial Ingester Processor | New Ingestion History Generated

8d6eca5


          CM and GI Zarr Work, minor bug fixes, dependncy updates

6933b4e


          Local store test comment out + comments

d6dfd7b


          remove ZarrPipeline (inadequete implementation)

57f9355


          Zarr collection processor dependency fix

aceb345


          README updates

fba7481

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet