Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add zarr-python v3 spec tests cases #52

Draft
wants to merge 6 commits into
base: main
Choose a base branch
from

Conversation

grlee77
Copy link
Contributor

@grlee77 grlee77 commented Dec 17, 2021

marked as a draft since this is using a branch corresponding to unmerged upstream PRs

Note that the environment file here installs from the branch corresponding to zarr-developers/zarr-python#898

@grlee77
Copy link
Contributor Author

grlee77 commented Dec 17, 2021

@joshmoore, could you take a look at the first three commits here in terms of the API for writing the files? In the first commit, I was trying the reuse the same high-level open function as for the v2 case, but for v3 it requires specifying a path ('astronaut' was used in that first commit). However, this makes an additional 'astronaut' subfolder within /meta/root and /data/root which is not present in the data generated from zarrita or xtensor-zarr.

The forms in commits 2 and 3 are equivalent, just using zarr.open_array or zarr.array directly.

The main issue is that in the current v3 implementation, one cannot create a Group without some for the group. i.e. the group cannot be at just /data/root/ but needs to be /data/root/<name>. In zarrita and xtensor-zarr there is a Hierarchy class that represents this top root level. The hierarchy just contains nodes corresponding to either groups or arrays. We can potentially also add this in zarr-python if it is helpful (specifically should zarr.open return a Hierarchy object in some cases where it currently returns a Group?)

Copy link
Member

@joshmoore joshmoore left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, @grlee77. Yeah, I imagine anything you struggle with like this also points to issues that users will have, so we will need to engage with it seriously. Not that I have a solution at the moment, so the changes look within reason for the interim.

I've also hit on the desire for a hierarchy class a couple of times. I don't know if now is the time to do it or not (and perhaps just for V3). @constantinpape said something similar when comparing the API to Z5.

@@ -33,14 +33,15 @@ def generate_zarr_format(compressors=['gzip', 'blosc', 'zlib', None]):

nested_str = '_nested' if nested else '_flat'
path = f'data/zarr_{StoreClass.__name__}{nested_str}.zr'
zarr_version = StoreClass._store_version
if zarr_version == 3:
path = path.replace('.zr', '.zr3')
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could imagine that we even offer "default file ending" somewhere as a constant.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants