Test XArray in combination with cattrs and dask #62
Conversation
|
so we don't have to worry about numpy/xarray arrays going through cattrs to the de/serializer? they will not be duplicated by default? |
We will need to create unstructure hooks so that we just return the original object, already on the DataTree level. I did a test that I didn't save, where I saw that I received a different DataTree id. I will add it so it's clear what is also not possible in the current setup. |
This is the simplest way to keep DataTree object the original. Otherwise it would return a copy of some sort.
There was a problem hiding this comment.
I guess as we chatted about last week we have to decide whether to fully adopt xarray for array/list data variables, or keep data variables as top level attrs and expose xarray views of them.. if the former we will send xarray objects thru cattrs, but in the latter case we would not, is that right?
Are there any more questions we need to ask and try to answer (with research and/or prototyping) to inform the decision? I'll think more on this.
I guess this question is not quite so pressing since you established that cattrs will not be necessary/advisable for mf6 format io, where we can send objects directly to the serializer, and may just be useful for cases like dict-based initialization and/or dict-like serialization formats like json/yaml
Added some simple test prototypes that show how we could keep a low memory profile with chunks, and how to unstructure data that contains XArray objects.