ODD PI 25.2 Objective 5: Deliver Virtual Zarr Stores for NASA Datasets Using Icechunk

[abarciauskas-bgse]

Motivation

Enable virtual zarr access to GPM IMERG and MUR SST datasets.

Description

Collaboration on virtual dataset development has begun in the GitHub repository earth-mover/icechunk-nasa, using the VEDA hub for initial work. The team is coordinating with GES DISC and PODAAC to ensure dataset validity and utility while addressing bugs and limitations in libraries (e.g., Zarr, xarray, VirtualiZarr, icechunk) and dataset structures. Continuous adaptation is required due to updates in Zarr-Python 3.0 and related tools, but this effort is critical to enable broader dataset adoption.

Acceptance Criteria

• Develop datasets with design docs, usage documentation, performance demonstration, and data validation.
• Stretch goal: automated updates.

Sub-tasks

Tasks

Preview Give feedback

1. Finish virtual MUR SST dataset to 2023-09-04 #38
   +0
   
2. Finish documentation for MUR SST dataset up to 2023-09-04 (validation, design doc and notebooks) #50
   +0
   
3. Share MUR SST dataset and docs with PO.DAAC team #51
   +0
   
4. Virtual MUR SST dataset from 2023-09-05 to date #52
   +0
   
5. [Stretch] Setup infrastructure for ongoing appending to this virtual dataset #53
   +0
6. [Stretch] Determine plan for sharing virtual mur sst data store with more people #55
   +0
7. Support GPM IMERG development + coordination with GES DISC #56
   +0
   
8. Outline + schedule for end of February icechunk demo #54
   +0