Add ZEP 9 (extension naming) draft #65
Conversation
| 🛠️ We propose defining two categories of names for immediate use by extensions: | ||
| raw names and URI-based. |
There was a problem hiding this comment.
this would be a good place to add an explanation for why exactly we want to use URIs instead of some other kind of prefixing scheme. I understand that URIs are mentioned in the spec, but the spec doesn't provide any reason for their use.
There was a problem hiding this comment.
The aspect of URLs that I like is that it delegates the name registration problem to DNS. But I also am unclear on the advantages of using fake https URLs compared to something slightly less verbose like <domain>/<suffix> which less strongly implies a real resource. Someone is surely going to try to fetch the documents, find them missing, and think that the URLs are just stale. Then also there is the question of using http or https --- or are other schemes like ftp, etc. allowed?
Aside from that, the added https:// part just adds extra verbosity. Tensorstore relies on just the json representation itself for specifying certain metadata, and making it more verbose just makes it harder to read and write.
There was a problem hiding this comment.
Reading the proposal a bit more, I see that the URLs are recommended to resolve to some description of the extension, that is just not a requirement. I was thrown off a bit by the later examples showing ttps://zarr.dev/array/data_type which didn't seem like a plausible actual URL, since for one thing it does not specify zarr v3 at all, and also having a separate document for each field in the zarr v3 metadata was also rather different from the current documentation.
There was a problem hiding this comment.
this would be a good place to add an explanation for why exactly we want to use URIs
Thanks, will do.
I see that the URLs are recommended to resolve to some description of the extension, that is just not a requirement
Is this generally a positive? A request that it's a requirement?
I was thrown off a bit by the later examples showing
ttps://zarr.dev/array/data_typewhich didn't seem like a plausible actual URL
Sorry, perhaps chosen too quickly. I'll review the example for plausibility.
|
Josh thanks so much for this important ZEP which addresses some critical issues around V3. 🙏 |
|
Regarding "raw" name registration: Can you clarify a bit more how you are proposing this to work?
I think it would be better to leave delegation to the URL-based names, or at least names that are clearly prefixed by the delegate name, in order to make the nature of the name clearer. |
|
Regarding "@context": my expectation is that the nature of the zarr metadata is such that each name is typically only specified once per a given array metadata, and furthermore other than the standard names in the zarr specification itself, a given URL prefix is also unlikely to be used more than once. Consequently, I'm concerned that adding an additional "@context" mechanism (similar to xmlns) would not really help with verbosity, but would make it more complicated for computers to parse, and harder for humans to read the format easily, since they would now need to keep track of this additional name indirection mechanism. |
In general, I would say we're trying to define how things get technically added more than the decision making but a few point from my perspetive:
I'd probably tend to less, say, schema reviewing and more an evaluation of whether or not the name can be safely given out. Will it lead to confusion, name squatting, etc. But open for discussion and perhaps a matter for additional governance document.
I think that depends on which versioning scheme is chosen by the implementation. If versioning is "within extension" then it's just one PR. If it's "versioning by name", then it would be multiple, but that would also put the burden on the reviewers to be comfortable with accepting the multiple names.
Agreed.
Phase 3 definitely needs multiple proposals with weighted pros & cons after phase 1 and phase 2 are complete. I think it's useful though to take a full specification that doesn't need designing and see which modifications (if any) are needed to make it useful. For example, is a single flag in the metadata which states which version of the context one is currently using sufficient? This is a bit like RDFas "initial contexts": Were this elevated to a zarr "extension point" this could then also take a URI (at the risk of verbosity). P.S. apologies to @context. I suggest we write |
|
Pushed the minor clarifications. I would work towards merging this as a draft ZEP. Questions and comments would continue to be welcome in issues, PRs, Zulip, or most other places. The bulk of feedback on phase 1, however, is likely better placed on the PR with spec changes in zarr-developers/zarr-specs#330. Once those clarifications have been released (e.g., tagged as Timeline:
|
I added a section to the ZEP about the registration process that we envision for the raw names in 3538de5. We hope to keep the process very lightweight. The review is only supposed to avoid confusing names and prevent malicious activities. It is not intended to review the contents of the specification to design decisions. If changes to an extension specification go into the zarr-extensions repo, which I would encourage, there needs to be another review for updates as well. |
|
All the (initial) TODOs have been handled from my side. Follow on from @rabernat's comment, I'd be for getting this in as a "draft ZEP" knowing that a) we can still modify it and b) feedback can be taken via zarr-developers/zarr-specs#330 cc: @zarr-developers/steering-council @zarr-developers/implementation-council @zarr-developers/python-core-devs |
This ZEP drafted with with @normanrz and reviewed by the rest of the @zarr-developers/steering-council attempts to unblock lingering spec-related issues like:
TODOs:
Clarify stores are not extensions (not written in metadata)(This is never asserted, but we should agree that they are not extensions)Please see https://zarr.dev/zeps/draft/ZEP0009.html 🎉 Comments welcome as issues or on Zulip.