Allow setting fill_value on Zarr format 3 arrays

[dcherian]

cc @aldenks

• Closes Zarr v3 fill_value #10064
• Tests added
• User visible changes (including notable bug fixes) are documented in whats-new.rst

[aldenks]

Thank you!

[d-v-b]

does this assume that np.nan is serializable to the attributes field using the same JSON encoding that we use for the fill_value field? Because the spec doesn't cover how scalars should be encoded in attributes.

[dcherian]

Yes. I think so:

xarray/xarray/backends/zarr.py

Lines 1150 to 1155 in cdebbf5

	fill_value = None
	if "_FillValue" in attrs:
	# replace with encoded fill value
	fv = attrs.pop("_FillValue")
	if fv is not None:
	attrs["_FillValue"] = FillValueCoder.encode(fv, dtype)

[d-v-b]

since the zarr spec only defines a special JSON encoding for nan in the specific context of the fill_value field, I would recommend handling the JSON serialization explicitly here.

Also, In zarr-developers/zarr-python#2874 I remove the behavior where all numpy scalars get special serialization to JSON (instead, the dtype object is responsible for that). But it seems like we will break xarray if we move forward with that change.

[d-v-b]

I checked what zarr v3 does today -- although we apply special JSON encoding to any nan, +infinity, -inifinity when writing JSON (e.g., {"foo": np.nan} -> {"foo": "NaN"}), we only apply special decoding when reading the fill value. Special nan values in attributes will not get special decoding. Is xarray basically handling that special decoding here? If so, we don't have anything to worry about w.r.t. the new dtype stuff.

[dcherian]

Yes looks like Ryan added base64 encode/decode.

[rabernat]

It would be great if we could take that custom attribute encoding decoding stuff out of Xarray and replace it with @d-v-b's new ZarrDType machinery.

[d-v-b]

I have stand-alone functions that handle this, but they are currently not public API. Would be happy to make them public though!

[d-v-b]

("this" meaning "convert special floats to / from JSON")

[rabernat]

Thanks for working on this Deepak.

This is a loose end that we left dangling when we implemented Zarr V3 support.