From 0217fe37acdea556c431d765bb70bc3f1330888a Mon Sep 17 00:00:00 2001 From: Thomas Nicholas Date: Mon, 12 Jun 2023 15:01:57 -0400 Subject: [PATCH 01/38] draft updates --- doc/user-guide/duckarrays.rst | 179 +++++++++++++++++++++++++++++++--- 1 file changed, 165 insertions(+), 14 deletions(-) diff --git a/doc/user-guide/duckarrays.rst b/doc/user-guide/duckarrays.rst index 78c7d1e572a..5b2b0330a54 100644 --- a/doc/user-guide/duckarrays.rst +++ b/doc/user-guide/duckarrays.rst @@ -3,28 +3,171 @@ Working with numpy-like arrays ============================== +NumPy-like arrays (often known as :term:`duck array`s) are drop-in replacements for the :py:class:`numpy.ndarray` +class but with different features, such as propagating physical units or a different layout in memory. +Xarray can often wrap these array types, allowing you to use labelled dimensions and indexes whilst benefiting from the +additional features of these array libraries. + .. warning:: - This feature should be considered experimental. Please report any bug you may find on + This feature should be considered somewhat experimental. Please report any bugs you find on xarray’s github repository. -NumPy-like arrays (:term:`duck array`) extend the :py:class:`numpy.ndarray` with -additional features, like propagating physical units or a different layout in memory. +.. note:: + + For information on wrapping dask arrays see :ref:`dask`. Whilst xarray wraps dask arrays in a similar way to that + described on this page, chunked array types like `dask.array.Array` implement additional methods that require + slightly different user code (e.g. calling ``.chunk`` or ``.compute``). + +What is a numpy-like array? +--------------------------- + +A "numpy-like array" (also known as a "duck array") is a class that contains array-like data, and implements key +numpy-like functionality such as indexing, broadcasting, and computation methods. + +For example, the ``sparse`` library provides a sparse array type which is useful for representing ``sparse matrices`` +in a memory-efficient manner. We can create a sparse array object (of the ``sparse.COO`` type) from a numpy array like this: + +.. ipython:: python + + import sparse + + x = np.eye(4, dtype=np.uint8) + s = COO.from_numpy(x) + s + +This sparse object does not attempt to explicitly store every element in the array, only the non-zero elements. +This approach is much more efficient for large arrays with only a few non-zero elements (such as tri-diagonal matrices). +It does mean that in order to clearly see what is stored in our sparse array object we have to convert it back to a +"dense" array using ``.todense``: + +.. ipython:: python + + s.todense() + +Just like `numpy.ndarray` objects, `sparse.COO` arrays support indexing + +.. ipython:: python + + s[2, 3] = 5 + s -:py:class:`DataArray` and :py:class:`Dataset` objects can wrap these duck arrays, as -long as they satisfy certain conditions (see :ref:`internals.duck_arrays`). +broadcasting, + +.. ipython:: python + + x3 = np.zeros((4, 1), dtype=np.uint8) + x3[2, 0] = 1 + s3 = COO.from_numpy(x3) + (s * s3).todense() + +and various computation methods + +.. ipython:: python + + s.sum(axis=1).todense() + +This numpy-like array also supports calling so-called numpy ufuncs (link to numpy docs) on it directly: + +.. ipython:: python + + np.sum(s, axis=1).todense() + + +Notice that in each case the API for calling the operation on the sparse array is identical to that of calling it on the +equivalent numpy array - this is the sense in which the sparse array is "numpy-like". + +Why is it also called a "duck" array, you might ask? This comes from a common statement in object-oriented programming - +"If it walks like a duck, and quacks like a duck, treat it like a duck". In other words, a library like xarray that +is capable of using multiple different types of arrays does not have to explicitly check that each one it encounters is +permitted (e.g. `if dask`, `if numpy`, `if sparse` etc.). Instead xarray can take the more permissive approach of simply +treating the wrapped array as valid, attempting to call the relevant methods (e.g. `.mean()`) and only raising an +error if a problem occurs (e.g. the method is not found on the wrapped class). This is much more flexible, and allows +objects and classes from different libraries to work together more easily. .. note:: - For ``dask`` support see :ref:`dask`. + For discussion on exactly which methods a class needs to implement to be considered "numpy-like", see :ref:`internals.duck_arrays`. + +Wrapping numpy-like arrays in xarray +------------------------------------ + +:py:class:`DataArray` and :py:class:`Dataset` (and :py:class:`Variable`) objects can wrap these numpy-like arrays. + +Constructing xarray objects which wrap numpy-like arrays +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The primary way to create an xarray object which wraps a numpy-like array is to pass that numpy-like array instance directly +to the constructor of the xarray class. The page on xarray data structures shows how :py:class:`DataArray` and :py:class:`Dataset` +both accept data in various forms through their ``data`` argument, but in fact this data can also be any wrappable numpy-like array. + +For example, we can wrap the sparse array we created earlier inside a new DataArray object: + +.. ipython:: python + + s_da = xr.DataArray(s2, dims=["x", "y"]) + s_da + +We can see what's inside - the printable representation of our xarray object (the repr) automatically uses the printable +representation of the underlying wrapped array. + +Of course our sparse array object is still there underneath - it's stored under the `.data` attribute of the dataarray: + +.. ipython:: python + + s_da.data + +Array methods +~~~~~~~~~~~~~ + +We saw above that numpy-like arrays provide numpy methods. Xarray automatically uses these when you call the corresponding xarray method: + +.. ipython:: python + + s_da.sum(dim="y") + +Numpy ufuncs +~~~~~~~~~~~~ + +Xarray objects support calling numpy functions direction on the xarray objects, e.g. ``np.func(da)``. +This also works when wrapping numpy-like arrays: + +.. ipython:: python + + np.sum(s_da, axis=1) + +Converting wrapped types +~~~~~~~~~~~~~~~~~~~~~~~~ + +If you want to change the type inside your xarray object you can use :py:meth:`DataArray.as_numpy`: + +.. ipython:: python + + s_da.as_numpy() + +This returns a new :py:class:`DataArray` object, but now wrapping a normal numpy array. + +If instead you want to convert to numpy and return that numpy array you can use either :py:meth:`DataArray.to_numpy` or +:py:meth:`DataArray.values` (what is the difference here?). + +This illustrates the difference between `.values` and `.data`, which is sometimes a point of confusion for new xarray users. +:py:meth:`DataArray.data` returns the underlying numpy-like array, regardless of type, whereas :py:meth:`DataArray.values` +converts the underlying array to a numpy array before returning it. + +Conversion to numpy as a fallback +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If a wrapped array does not implement the corresponding array method then xarray will often attempt to convert the +underlying array to a numpy array so that the operation can be performed. You may want to watch out for this behavior, +and report any instances in which it causes problems. Missing features ---------------- -Most of the API does support :term:`duck array` objects, but there are a few areas where -the code will still cast to ``numpy`` arrays: -- dimension coordinates, and thus all indexing operations: +Most of xarray's API does support using :term:`duck array` objects, but there are a few areas where +the code will still convert to ``numpy`` arrays: + +- Dimension coordinates, and thus all indexing operations: * :py:meth:`Dataset.sel` and :py:meth:`DataArray.sel` * :py:meth:`Dataset.loc` and :py:meth:`DataArray.loc` @@ -33,7 +176,7 @@ the code will still cast to ``numpy`` arrays: :py:meth:`DataArray.reindex` and :py:meth:`DataArray.reindex_like`: duck arrays in data variables and non-dimension coordinates won't be casted -- functions and methods that depend on external libraries or features of ``numpy`` not +- Functions and methods that depend on external libraries or features of ``numpy`` not covered by ``__array_function__`` / ``__array_ufunc__``: * :py:meth:`Dataset.ffill` and :py:meth:`DataArray.ffill` (uses ``bottleneck``) @@ -49,17 +192,25 @@ the code will still cast to ``numpy`` arrays: :py:class:`numpy.vectorize`) * :py:func:`apply_ufunc` with ``vectorize=True`` (uses :py:class:`numpy.vectorize`) -- incompatibilities between different :term:`duck array` libraries: +- Incompatibilities between different :term:`duck array` libraries: * :py:meth:`Dataset.chunk` and :py:meth:`DataArray.chunk`: this fails if the data was not already chunked and the :term:`duck array` (e.g. a ``pint`` quantity) should wrap the new ``dask`` array; changing the chunk sizes works. - Extensions using duck arrays ---------------------------- -Here's a list of libraries extending ``xarray`` to make working with wrapped duck arrays -easier: + +Whilst the features above allow many numpy-like array libraries to be used pretty seamlessly with xarray, it often also +makes sense to use an interfacing package to make certain tasks easier. + +For example the ``pint-xarray`` package offers a custom `.pint` accessor (link to accessors docs) which provides +convenient access to information stored within the wrapped array (e.g. `.units` and `.magnitude`), and makes makes +creating wrapped pint arrays (and especially xarray-wrapping-pint-wrapping-dask arrays) simpler for the user. + +We maintain a list of libraries extending ``xarray`` to make working with particular wrapped duck arrays +easier. If you know of more that aren't on this list please raise an issue to add them! - `pint-xarray `_ - `cupy-xarray `_ +- `cubed-xarray `_ From 5a221bbd2b793457c1fde88d778b4553ce113187 Mon Sep 17 00:00:00 2001 From: Thomas Nicholas Date: Mon, 12 Jun 2023 17:34:33 -0400 Subject: [PATCH 02/38] discuss array API standard --- doc/internals/duck-arrays-integration.rst | 30 +++++++++++++++++++---- 1 file changed, 25 insertions(+), 5 deletions(-) diff --git a/doc/internals/duck-arrays-integration.rst b/doc/internals/duck-arrays-integration.rst index d403328aa2f..39fd2b33943 100644 --- a/doc/internals/duck-arrays-integration.rst +++ b/doc/internals/duck-arrays-integration.rst @@ -6,18 +6,38 @@ Integrating with duck arrays .. warning:: - This is a experimental feature. + This is a experimental feature. Please report any bugs or other difficulties on xarray's issue tracker. -Xarray can wrap custom :term:`duck array` objects as long as they define numpy's -``shape``, ``dtype`` and ``ndim`` properties and the ``__array__``, -``__array_ufunc__`` and ``__array_function__`` methods. +Xarray can wrap custom numpy-like arrays (":term:`duck array`s") - see the user guide documentation. + +Duck array requirements +~~~~~~~~~~~~~~~~~~~~~~~ + +Xarray does not explicitly check that that required methods are defined by the underlying duck array object before +attempting to wrap the given array. However, a wrapped array type should at a minimum support numpy's ``shape``, +``dtype`` and ``ndim`` properties, as well as the ``__array__``, ``__array_ufunc__`` and ``__array_function__`` methods. +The array ``shape`` property needs to obey numpy's broadcasting rules. + +Python Array API standard support +================================= + +As an integration library xarray benefits greatly from the standardization of duck-array libraries' APIs, and so is a +big supporter of the python Array API Standard (link). In fact the crystallization of different array libraries' APIs towards +the standard has already helped xarray remove a lot of internal adapter code. + +As such, we aim to support any array librarie that follows the standard out-of-the-box. However, xarray does occasionally +call some numpy functions which are not (yet) part of the standard (e.g. :py:class:`DataArray.pad` calls `np.pad`, +). (link to issue) + +Custom inline reprs +~~~~~~~~~~~~~~~~~~~ In certain situations (e.g. when printing the collapsed preview of variables of a ``Dataset``), xarray will display the repr of a :term:`duck array` in a single line, truncating it to a certain number of characters. If that would drop too much information, the :term:`duck array` may define a ``_repr_inline_`` method that takes ``max_width`` (number of characters) as an -argument: +argument .. code:: python From 1971da4a052eb1041ab8b8d7e2a3801806c8fd24 Mon Sep 17 00:00:00 2001 From: Thomas Nicholas Date: Tue, 13 Jun 2023 12:49:25 -0400 Subject: [PATCH 03/38] fix sparse examples so they run --- doc/user-guide/duckarrays.rst | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/doc/user-guide/duckarrays.rst b/doc/user-guide/duckarrays.rst index 5b2b0330a54..85d96eacebd 100644 --- a/doc/user-guide/duckarrays.rst +++ b/doc/user-guide/duckarrays.rst @@ -30,9 +30,9 @@ in a memory-efficient manner. We can create a sparse array object (of the ``spar .. ipython:: python - import sparse + from sparse import COO - x = np.eye(4, dtype=np.uint8) + x = np.eye(4, dtype=np.uint8) # create diagonal identity matrix s = COO.from_numpy(x) s @@ -49,17 +49,18 @@ Just like `numpy.ndarray` objects, `sparse.COO` arrays support indexing .. ipython:: python - s[2, 3] = 5 - s + s[1, 1] # diagonal elements should be ones + s[2, 3] # off-diagonal elements should be zero broadcasting, .. ipython:: python - x3 = np.zeros((4, 1), dtype=np.uint8) - x3[2, 0] = 1 - s3 = COO.from_numpy(x3) - (s * s3).todense() + x2 = np.zeros( + (4, 1), dtype=np.uint8 + ) # create second sparse array of different shape + s2 = COO.from_numpy(x2) + (s * s2).todense() # multiplication requires broadcasting and various computation methods @@ -105,7 +106,7 @@ For example, we can wrap the sparse array we created earlier inside a new DataAr .. ipython:: python - s_da = xr.DataArray(s2, dims=["x", "y"]) + s_da = xr.DataArray(s, dims=["i", "j"]) s_da We can see what's inside - the printable representation of our xarray object (the repr) automatically uses the printable @@ -124,7 +125,7 @@ We saw above that numpy-like arrays provide numpy methods. Xarray automatically .. ipython:: python - s_da.sum(dim="y") + s_da.sum(dim="j") Numpy ufuncs ~~~~~~~~~~~~ From fa58fff9d13010e4cb829543eefdd7411c637035 Mon Sep 17 00:00:00 2001 From: Tom Nicholas Date: Wed, 14 Jun 2023 12:10:59 -0400 Subject: [PATCH 04/38] Deepak's suggestions Co-authored-by: Deepak Cherian --- doc/internals/duck-arrays-integration.rst | 4 ++-- doc/user-guide/duckarrays.rst | 6 +++--- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/doc/internals/duck-arrays-integration.rst b/doc/internals/duck-arrays-integration.rst index 39fd2b33943..aa2193ba2c7 100644 --- a/doc/internals/duck-arrays-integration.rst +++ b/doc/internals/duck-arrays-integration.rst @@ -6,7 +6,7 @@ Integrating with duck arrays .. warning:: - This is a experimental feature. Please report any bugs or other difficulties on xarray's issue tracker. + This is an experimental feature. Please report any bugs or other difficulties on `xarray's issue tracker `_. Xarray can wrap custom numpy-like arrays (":term:`duck array`s") - see the user guide documentation. @@ -25,7 +25,7 @@ As an integration library xarray benefits greatly from the standardization of du big supporter of the python Array API Standard (link). In fact the crystallization of different array libraries' APIs towards the standard has already helped xarray remove a lot of internal adapter code. -As such, we aim to support any array librarie that follows the standard out-of-the-box. However, xarray does occasionally +We aim to support any array libraries that follows the standard out-of-the-box. However, xarray does occasionally call some numpy functions which are not (yet) part of the standard (e.g. :py:class:`DataArray.pad` calls `np.pad`, ). (link to issue) diff --git a/doc/user-guide/duckarrays.rst b/doc/user-guide/duckarrays.rst index 85d96eacebd..ea03bbae23a 100644 --- a/doc/user-guide/duckarrays.rst +++ b/doc/user-guide/duckarrays.rst @@ -16,7 +16,7 @@ additional features of these array libraries. .. note:: For information on wrapping dask arrays see :ref:`dask`. Whilst xarray wraps dask arrays in a similar way to that - described on this page, chunked array types like `dask.array.Array` implement additional methods that require + described on this page, chunked array types like :py:class:`dask.array.Array` implement additional methods that require slightly different user code (e.g. calling ``.chunk`` or ``.compute``). What is a numpy-like array? @@ -25,7 +25,7 @@ What is a numpy-like array? A "numpy-like array" (also known as a "duck array") is a class that contains array-like data, and implements key numpy-like functionality such as indexing, broadcasting, and computation methods. -For example, the ``sparse`` library provides a sparse array type which is useful for representing ``sparse matrices`` +For example, the `sparse `_ library provides a sparse array type which is useful for representing nD array objects like sparse matrices in a memory-efficient manner. We can create a sparse array object (of the ``sparse.COO`` type) from a numpy array like this: .. ipython:: python @@ -205,7 +205,7 @@ Extensions using duck arrays Whilst the features above allow many numpy-like array libraries to be used pretty seamlessly with xarray, it often also makes sense to use an interfacing package to make certain tasks easier. -For example the ``pint-xarray`` package offers a custom `.pint` accessor (link to accessors docs) which provides +For example the ``pint-xarray`` package offers a custom ``.pint`` accessor (link to accessors docs) which provides convenient access to information stored within the wrapped array (e.g. `.units` and `.magnitude`), and makes makes creating wrapped pint arrays (and especially xarray-wrapping-pint-wrapping-dask arrays) simpler for the user. From 258dd54ecacf091f6a90b1831e7f52f8f48579b9 Mon Sep 17 00:00:00 2001 From: Thomas Nicholas Date: Wed, 14 Jun 2023 15:46:32 -0400 Subject: [PATCH 05/38] link to duck arrays user guide from internals page --- doc/internals/duck-arrays-integration.rst | 9 +++++---- doc/user-guide/duckarrays.rst | 2 ++ 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/doc/internals/duck-arrays-integration.rst b/doc/internals/duck-arrays-integration.rst index aa2193ba2c7..cd3d24281ef 100644 --- a/doc/internals/duck-arrays-integration.rst +++ b/doc/internals/duck-arrays-integration.rst @@ -1,5 +1,5 @@ -.. _internals.duck_arrays: +.. _internals.duckarrays: Integrating with duck arrays ============================= @@ -8,7 +8,8 @@ Integrating with duck arrays This is an experimental feature. Please report any bugs or other difficulties on `xarray's issue tracker `_. -Xarray can wrap custom numpy-like arrays (":term:`duck array`s") - see the user guide documentation. +Xarray can wrap custom numpy-like arrays (":term:`duck array`\s") - see the :ref:`user guide documentation `. +This page is intended for developers who are interested in wrapping a custom array type with xarray. Duck array requirements ~~~~~~~~~~~~~~~~~~~~~~~ @@ -19,13 +20,13 @@ attempting to wrap the given array. However, a wrapped array type should at a mi The array ``shape`` property needs to obey numpy's broadcasting rules. Python Array API standard support -================================= +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ As an integration library xarray benefits greatly from the standardization of duck-array libraries' APIs, and so is a big supporter of the python Array API Standard (link). In fact the crystallization of different array libraries' APIs towards the standard has already helped xarray remove a lot of internal adapter code. -We aim to support any array libraries that follows the standard out-of-the-box. However, xarray does occasionally +We aim to support any array libraries that follow the standard out-of-the-box. However, xarray does occasionally call some numpy functions which are not (yet) part of the standard (e.g. :py:class:`DataArray.pad` calls `np.pad`, ). (link to issue) diff --git a/doc/user-guide/duckarrays.rst b/doc/user-guide/duckarrays.rst index ea03bbae23a..fdc625d18f1 100644 --- a/doc/user-guide/duckarrays.rst +++ b/doc/user-guide/duckarrays.rst @@ -1,5 +1,7 @@ .. currentmodule:: xarray +.. _userguide.duckarrays: + Working with numpy-like arrays ============================== From b26e7ac5af040a7ad4e27d2a8061c3f4874779b2 Mon Sep 17 00:00:00 2001 From: Thomas Nicholas Date: Thu, 15 Jun 2023 09:18:09 -0400 Subject: [PATCH 06/38] fix various links --- doc/internals/duck-arrays-integration.rst | 4 ++-- doc/internals/extending-xarray.rst | 2 ++ doc/user-guide/duckarrays.rst | 10 +++++----- 3 files changed, 9 insertions(+), 7 deletions(-) diff --git a/doc/internals/duck-arrays-integration.rst b/doc/internals/duck-arrays-integration.rst index cd3d24281ef..5dd99ed2ef7 100644 --- a/doc/internals/duck-arrays-integration.rst +++ b/doc/internals/duck-arrays-integration.rst @@ -9,7 +9,7 @@ Integrating with duck arrays This is an experimental feature. Please report any bugs or other difficulties on `xarray's issue tracker `_. Xarray can wrap custom numpy-like arrays (":term:`duck array`\s") - see the :ref:`user guide documentation `. -This page is intended for developers who are interested in wrapping a custom array type with xarray. +This page is intended for developers who are interested in wrapping a new custom array type with xarray. Duck array requirements ~~~~~~~~~~~~~~~~~~~~~~~ @@ -23,7 +23,7 @@ Python Array API standard support ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ As an integration library xarray benefits greatly from the standardization of duck-array libraries' APIs, and so is a -big supporter of the python Array API Standard (link). In fact the crystallization of different array libraries' APIs towards +big supporter of the `Python Array API Standard `_. In fact the crystallization of different array libraries' APIs towards the standard has already helped xarray remove a lot of internal adapter code. We aim to support any array libraries that follow the standard out-of-the-box. However, xarray does occasionally diff --git a/doc/internals/extending-xarray.rst b/doc/internals/extending-xarray.rst index 56aeb8fa462..a180b85044f 100644 --- a/doc/internals/extending-xarray.rst +++ b/doc/internals/extending-xarray.rst @@ -1,4 +1,6 @@ +.. _internals.accessors: + Extending xarray using accessors ================================ diff --git a/doc/user-guide/duckarrays.rst b/doc/user-guide/duckarrays.rst index fdc625d18f1..5dce80d3072 100644 --- a/doc/user-guide/duckarrays.rst +++ b/doc/user-guide/duckarrays.rst @@ -5,7 +5,7 @@ Working with numpy-like arrays ============================== -NumPy-like arrays (often known as :term:`duck array`s) are drop-in replacements for the :py:class:`numpy.ndarray` +NumPy-like arrays (often known as :term:`duck array`\s) are drop-in replacements for the :py:class:`numpy.ndarray` class but with different features, such as propagating physical units or a different layout in memory. Xarray can often wrap these array types, allowing you to use labelled dimensions and indexes whilst benefiting from the additional features of these array libraries. @@ -90,7 +90,7 @@ objects and classes from different libraries to work together more easily. .. note:: - For discussion on exactly which methods a class needs to implement to be considered "numpy-like", see :ref:`internals.duck_arrays`. + For discussion on exactly which methods a class needs to implement to be considered "numpy-like", see :ref:`internals.duckarrays`. Wrapping numpy-like arrays in xarray ------------------------------------ @@ -101,7 +101,7 @@ Constructing xarray objects which wrap numpy-like arrays ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The primary way to create an xarray object which wraps a numpy-like array is to pass that numpy-like array instance directly -to the constructor of the xarray class. The page on xarray data structures shows how :py:class:`DataArray` and :py:class:`Dataset` +to the constructor of the xarray class. The :ref:`page on xarray data structures ` shows how :py:class:`DataArray` and :py:class:`Dataset` both accept data in various forms through their ``data`` argument, but in fact this data can also be any wrappable numpy-like array. For example, we can wrap the sparse array we created earlier inside a new DataArray object: @@ -207,8 +207,8 @@ Extensions using duck arrays Whilst the features above allow many numpy-like array libraries to be used pretty seamlessly with xarray, it often also makes sense to use an interfacing package to make certain tasks easier. -For example the ``pint-xarray`` package offers a custom ``.pint`` accessor (link to accessors docs) which provides -convenient access to information stored within the wrapped array (e.g. `.units` and `.magnitude`), and makes makes +For example the `pint-xarray package `_ offers a custom ``.pint`` accessor (see :ref:`internals.accessors`) which provides +convenient access to information stored within the wrapped array (e.g. ``.units`` and ``.magnitude``), and makes makes creating wrapped pint arrays (and especially xarray-wrapping-pint-wrapping-dask arrays) simpler for the user. We maintain a list of libraries extending ``xarray`` to make working with particular wrapped duck arrays From ad818115e847ac81bf840563deefdef26e580b5a Mon Sep 17 00:00:00 2001 From: Thomas Nicholas Date: Thu, 15 Jun 2023 09:34:52 -0400 Subject: [PATCH 07/38] itemized list --- doc/internals/duck-arrays-integration.rst | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/doc/internals/duck-arrays-integration.rst b/doc/internals/duck-arrays-integration.rst index 5dd99ed2ef7..6ee7d3c22b4 100644 --- a/doc/internals/duck-arrays-integration.rst +++ b/doc/internals/duck-arrays-integration.rst @@ -15,9 +15,19 @@ Duck array requirements ~~~~~~~~~~~~~~~~~~~~~~~ Xarray does not explicitly check that that required methods are defined by the underlying duck array object before -attempting to wrap the given array. However, a wrapped array type should at a minimum support numpy's ``shape``, -``dtype`` and ``ndim`` properties, as well as the ``__array__``, ``__array_ufunc__`` and ``__array_function__`` methods. -The array ``shape`` property needs to obey numpy's broadcasting rules. +attempting to wrap the given array. However, a wrapped array type should at a minimum define these attributes: + +* ``shape`` property, +* ``dtype`` property, +* ``ndim`` property, +* ``__array__`` method, +* ``__array_ufunc__`` method, +* ``__array_function__`` method. + +These need to be defined consistently with numpy :py:class:`numpy.ndarray`, for example the array ``shape`` +property needs to obey `numpy's broadcasting rules `_ +(see also the `Python Array API standard's explanation `_ +of these same rules). Python Array API standard support ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ From 99394a3417f302459f9d2151bdf6660eb2db2c81 Mon Sep 17 00:00:00 2001 From: Thomas Nicholas Date: Thu, 15 Jun 2023 09:44:49 -0400 Subject: [PATCH 08/38] mention dispatching on functions not in the array API standard --- doc/internals/duck-arrays-integration.rst | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/doc/internals/duck-arrays-integration.rst b/doc/internals/duck-arrays-integration.rst index 6ee7d3c22b4..f7c4e9a2c04 100644 --- a/doc/internals/duck-arrays-integration.rst +++ b/doc/internals/duck-arrays-integration.rst @@ -37,8 +37,10 @@ big supporter of the `Python Array API Standard `_ for a list of such functions. We can still support dispatching on these functions through +the array protocols above, it just means that if you exclusively implement the methods in the Python Array API standard +then some features in xarray will not work. Custom inline reprs ~~~~~~~~~~~~~~~~~~~ From c93f14368f1c4cc5b4cd20f107476fb94ec908f4 Mon Sep 17 00:00:00 2001 From: Thomas Nicholas Date: Wed, 21 Jun 2023 13:13:56 -0400 Subject: [PATCH 09/38] examples of duckarrays --- doc/user-guide/duckarrays.rst | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/doc/user-guide/duckarrays.rst b/doc/user-guide/duckarrays.rst index 5dce80d3072..a6143e4b675 100644 --- a/doc/user-guide/duckarrays.rst +++ b/doc/user-guide/duckarrays.rst @@ -10,10 +10,16 @@ class but with different features, such as propagating physical units or a diffe Xarray can often wrap these array types, allowing you to use labelled dimensions and indexes whilst benefiting from the additional features of these array libraries. +Some numpy-like array types that xarray already has some support for: + +* `Cupy `_ - GPU support, +* `Sparse `_ - for performant arrays with many zero elements, +* `Pint `_ - for tracking the physical units of your data. + .. warning:: This feature should be considered somewhat experimental. Please report any bugs you find on - xarray’s github repository. + `xarray’s issue tracker `_. .. note:: From b6279fdb6091d3b9b220435644e3bf6e853aeb69 Mon Sep 17 00:00:00 2001 From: Thomas Nicholas Date: Wed, 21 Jun 2023 13:15:09 -0400 Subject: [PATCH 10/38] add intended audience to xarray internals section --- doc/internals/index.rst | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/doc/internals/index.rst b/doc/internals/index.rst index e4ca9779dd7..25671877297 100644 --- a/doc/internals/index.rst +++ b/doc/internals/index.rst @@ -8,6 +8,11 @@ stack, NumPy and pandas. It is written in pure Python (no C or Cython extensions), which makes it easy to develop and extend. Instead, we push compiled code to :ref:`optional dependencies`. +The pages in this section are intended for: +- Contributors to xarray who wish to better understand some of the internals, +- Developers who wish to extend xarray with domain-specific logic, perhaps to support a new scientific community of users, +- Developers who wish to interface xarray with their existing tooling, e.g. by creating a plugin for reading a new file format, or wrapping a custom array type. + .. toctree:: :maxdepth: 2 From 0eea00bf5e52a8915781f673ef5b1a469362070f Mon Sep 17 00:00:00 2001 From: Thomas Nicholas Date: Tue, 27 Jun 2023 14:35:21 -0400 Subject: [PATCH 11/38] move paragraph on why its called a duck array upwards --- doc/user-guide/duckarrays.rst | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/doc/user-guide/duckarrays.rst b/doc/user-guide/duckarrays.rst index a6143e4b675..c3d354cd98d 100644 --- a/doc/user-guide/duckarrays.rst +++ b/doc/user-guide/duckarrays.rst @@ -27,6 +27,17 @@ Some numpy-like array types that xarray already has some support for: described on this page, chunked array types like :py:class:`dask.array.Array` implement additional methods that require slightly different user code (e.g. calling ``.chunk`` or ``.compute``). +Why "duck"? +----------- + +Why is it also called a "duck" array? This comes from a common statement of object-oriented programming - +"If it walks like a duck, and quacks like a duck, treat it like a duck". In other words, a library like xarray that +is capable of using multiple different types of arrays does not have to explicitly check that each one it encounters is +permitted (e.g. ``if dask``, ``if numpy``, ``if sparse`` etc.). Instead xarray can take the more permissive approach of simply +treating the wrapped array as valid, attempting to call the relevant methods (e.g. ``.mean()``) and only raising an +error if a problem occurs (e.g. the method is not found on the wrapped class). This is much more flexible, and allows +objects and classes from different libraries to work together more easily. + What is a numpy-like array? --------------------------- @@ -86,14 +97,6 @@ This numpy-like array also supports calling so-called numpy ufuncs (link to nump Notice that in each case the API for calling the operation on the sparse array is identical to that of calling it on the equivalent numpy array - this is the sense in which the sparse array is "numpy-like". -Why is it also called a "duck" array, you might ask? This comes from a common statement in object-oriented programming - -"If it walks like a duck, and quacks like a duck, treat it like a duck". In other words, a library like xarray that -is capable of using multiple different types of arrays does not have to explicitly check that each one it encounters is -permitted (e.g. `if dask`, `if numpy`, `if sparse` etc.). Instead xarray can take the more permissive approach of simply -treating the wrapped array as valid, attempting to call the relevant methods (e.g. `.mean()`) and only raising an -error if a problem occurs (e.g. the method is not found on the wrapped class). This is much more flexible, and allows -objects and classes from different libraries to work together more easily. - .. note:: For discussion on exactly which methods a class needs to implement to be considered "numpy-like", see :ref:`internals.duckarrays`. From cc4fac03adfa387cf61b4fbcf48936dfab502fa0 Mon Sep 17 00:00:00 2001 From: Thomas Nicholas Date: Tue, 27 Jun 2023 14:36:03 -0400 Subject: [PATCH 12/38] delete section on numpy ufuncs --- doc/user-guide/duckarrays.rst | 10 ---------- 1 file changed, 10 deletions(-) diff --git a/doc/user-guide/duckarrays.rst b/doc/user-guide/duckarrays.rst index c3d354cd98d..ba15eda163c 100644 --- a/doc/user-guide/duckarrays.rst +++ b/doc/user-guide/duckarrays.rst @@ -138,16 +138,6 @@ We saw above that numpy-like arrays provide numpy methods. Xarray automatically s_da.sum(dim="j") -Numpy ufuncs -~~~~~~~~~~~~ - -Xarray objects support calling numpy functions direction on the xarray objects, e.g. ``np.func(da)``. -This also works when wrapping numpy-like arrays: - -.. ipython:: python - - np.sum(s_da, axis=1) - Converting wrapped types ~~~~~~~~~~~~~~~~~~~~~~~~ From 5e8015f6373a77e0cefea8c90e2592436fd8aaf1 Mon Sep 17 00:00:00 2001 From: Thomas Nicholas Date: Tue, 27 Jun 2023 14:39:48 -0400 Subject: [PATCH 13/38] explain difference between .values and to_numpy --- doc/user-guide/duckarrays.rst | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/doc/user-guide/duckarrays.rst b/doc/user-guide/duckarrays.rst index ba15eda163c..0b7a5baa46c 100644 --- a/doc/user-guide/duckarrays.rst +++ b/doc/user-guide/duckarrays.rst @@ -150,11 +150,13 @@ If you want to change the type inside your xarray object you can use :py:meth:`D This returns a new :py:class:`DataArray` object, but now wrapping a normal numpy array. If instead you want to convert to numpy and return that numpy array you can use either :py:meth:`DataArray.to_numpy` or -:py:meth:`DataArray.values` (what is the difference here?). +:py:meth:`DataArray.values`, where the former is preferred. (The difference is in the way they coerce to numpy - `.values` +always uses `np.asarray` which will fail for some array types (e.g. ``cupy``, whereas `to_numpy` uses the correct method +depending on the array type.) This illustrates the difference between `.values` and `.data`, which is sometimes a point of confusion for new xarray users. -:py:meth:`DataArray.data` returns the underlying numpy-like array, regardless of type, whereas :py:meth:`DataArray.values` -converts the underlying array to a numpy array before returning it. +Explicitly: :py:meth:`DataArray.data` returns the underlying numpy-like array, regardless of type, whereas +:py:meth:`DataArray.values` converts the underlying array to a numpy array before returning it. Conversion to numpy as a fallback ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ From 70bfda5d423af093a786dd6d0c7ccd311ced0d81 Mon Sep 17 00:00:00 2001 From: Thomas Nicholas Date: Tue, 27 Jun 2023 14:40:48 -0400 Subject: [PATCH 14/38] strongly prefer to_numpy over values --- doc/user-guide/duckarrays.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/user-guide/duckarrays.rst b/doc/user-guide/duckarrays.rst index 0b7a5baa46c..cd24a348c57 100644 --- a/doc/user-guide/duckarrays.rst +++ b/doc/user-guide/duckarrays.rst @@ -150,13 +150,13 @@ If you want to change the type inside your xarray object you can use :py:meth:`D This returns a new :py:class:`DataArray` object, but now wrapping a normal numpy array. If instead you want to convert to numpy and return that numpy array you can use either :py:meth:`DataArray.to_numpy` or -:py:meth:`DataArray.values`, where the former is preferred. (The difference is in the way they coerce to numpy - `.values` +:py:meth:`DataArray.values`, where the former is strongly preferred. (The difference is in the way they coerce to numpy - `.values` always uses `np.asarray` which will fail for some array types (e.g. ``cupy``, whereas `to_numpy` uses the correct method depending on the array type.) This illustrates the difference between `.values` and `.data`, which is sometimes a point of confusion for new xarray users. Explicitly: :py:meth:`DataArray.data` returns the underlying numpy-like array, regardless of type, whereas -:py:meth:`DataArray.values` converts the underlying array to a numpy array before returning it. +:py:meth:`DataArray.to_numpy` converts the underlying array to a numpy array before returning it. Conversion to numpy as a fallback ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ From 5fdb7e3263d0180d87d59b26278dd04f5570a248 Mon Sep 17 00:00:00 2001 From: Thomas Nicholas Date: Tue, 27 Jun 2023 14:42:17 -0400 Subject: [PATCH 15/38] recommend to_numpy instead of values in the how do I? page --- doc/howdoi.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/howdoi.rst b/doc/howdoi.rst index b6374cc5100..8cc4e9939f2 100644 --- a/doc/howdoi.rst +++ b/doc/howdoi.rst @@ -42,7 +42,7 @@ How do I ... * - extract the underlying array (e.g. NumPy or Dask arrays) - :py:attr:`DataArray.data` * - convert to and extract the underlying NumPy array - - :py:attr:`DataArray.values` + - :py:attr:`DataArray.to_numpy` * - convert to a pandas DataFrame - :py:attr:`Dataset.to_dataframe` * - sort values From 68315f84d372d4c1c5a716db786ced454d4f560c Mon Sep 17 00:00:00 2001 From: Thomas Nicholas Date: Tue, 27 Jun 2023 14:45:16 -0400 Subject: [PATCH 16/38] clearer about using to_numpy --- doc/user-guide/duckarrays.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/doc/user-guide/duckarrays.rst b/doc/user-guide/duckarrays.rst index cd24a348c57..78be8907cb8 100644 --- a/doc/user-guide/duckarrays.rst +++ b/doc/user-guide/duckarrays.rst @@ -154,9 +154,10 @@ If instead you want to convert to numpy and return that numpy array you can use always uses `np.asarray` which will fail for some array types (e.g. ``cupy``, whereas `to_numpy` uses the correct method depending on the array type.) -This illustrates the difference between `.values` and `.data`, which is sometimes a point of confusion for new xarray users. +This illustrates the difference between ``.data`` and ``.values``, which is sometimes a point of confusion for new xarray users. Explicitly: :py:meth:`DataArray.data` returns the underlying numpy-like array, regardless of type, whereas -:py:meth:`DataArray.to_numpy` converts the underlying array to a numpy array before returning it. +:py:meth:`DataArray.values` converts the underlying array to a numpy array before returning it. +(This is another reason to use ``.to_numpy`` over ``.values`` - the intention is clearer.) Conversion to numpy as a fallback ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ From 2931b86d33b83b3808964e45fedc47d53c98ab58 Mon Sep 17 00:00:00 2001 From: Thomas Nicholas Date: Tue, 27 Jun 2023 14:47:48 -0400 Subject: [PATCH 17/38] merge section on missing features --- doc/user-guide/duckarrays.rst | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/doc/user-guide/duckarrays.rst b/doc/user-guide/duckarrays.rst index 78be8907cb8..33d52fe66ee 100644 --- a/doc/user-guide/duckarrays.rst +++ b/doc/user-guide/duckarrays.rst @@ -166,9 +166,6 @@ If a wrapped array does not implement the corresponding array method then xarray underlying array to a numpy array so that the operation can be performed. You may want to watch out for this behavior, and report any instances in which it causes problems. -Missing features ----------------- - Most of xarray's API does support using :term:`duck array` objects, but there are a few areas where the code will still convert to ``numpy`` arrays: @@ -201,7 +198,7 @@ the code will still convert to ``numpy`` arrays: * :py:meth:`Dataset.chunk` and :py:meth:`DataArray.chunk`: this fails if the data was not already chunked and the :term:`duck array` (e.g. a ``pint`` quantity) should - wrap the new ``dask`` array; changing the chunk sizes works. + wrap the new ``dask`` array; changing the chunk sizes works however. Extensions using duck arrays ---------------------------- From 9f21b00598216269d7494607edb053ba24016e27 Mon Sep 17 00:00:00 2001 From: Thomas Nicholas Date: Tue, 27 Jun 2023 14:51:18 -0400 Subject: [PATCH 18/38] remove todense from examples --- doc/user-guide/duckarrays.rst | 13 ++++--------- 1 file changed, 4 insertions(+), 9 deletions(-) diff --git a/doc/user-guide/duckarrays.rst b/doc/user-guide/duckarrays.rst index 33d52fe66ee..03fa82cdaf3 100644 --- a/doc/user-guide/duckarrays.rst +++ b/doc/user-guide/duckarrays.rst @@ -57,12 +57,7 @@ in a memory-efficient manner. We can create a sparse array object (of the ``spar This sparse object does not attempt to explicitly store every element in the array, only the non-zero elements. This approach is much more efficient for large arrays with only a few non-zero elements (such as tri-diagonal matrices). -It does mean that in order to clearly see what is stored in our sparse array object we have to convert it back to a -"dense" array using ``.todense``: - -.. ipython:: python - - s.todense() +Sparse array objects can be converted back to a "dense" numpy array by calling ``.todense``. Just like `numpy.ndarray` objects, `sparse.COO` arrays support indexing @@ -79,19 +74,19 @@ broadcasting, (4, 1), dtype=np.uint8 ) # create second sparse array of different shape s2 = COO.from_numpy(x2) - (s * s2).todense() # multiplication requires broadcasting + (s * s2) # multiplication requires broadcasting and various computation methods .. ipython:: python - s.sum(axis=1).todense() + s.sum(axis=1) This numpy-like array also supports calling so-called numpy ufuncs (link to numpy docs) on it directly: .. ipython:: python - np.sum(s, axis=1).todense() + np.sum(s, axis=1) Notice that in each case the API for calling the operation on the sparse array is identical to that of calling it on the From 2bb65d584bc7fbd709056089bc1428de1253e8d7 Mon Sep 17 00:00:00 2001 From: Thomas Nicholas Date: Tue, 27 Jun 2023 14:53:04 -0400 Subject: [PATCH 19/38] whatsnew --- doc/whats-new.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/doc/whats-new.rst b/doc/whats-new.rst index 5c0d3c3c843..21f381e1901 100644 --- a/doc/whats-new.rst +++ b/doc/whats-new.rst @@ -50,6 +50,8 @@ Bug fixes Documentation ~~~~~~~~~~~~~ +- Expanded the page on wrapping numpy-like "duck" arrays. + (:pull:`7911`) By `Tom Nicholas `_. Internal Changes ~~~~~~~~~~~~~~~~ From 0b405a14cd4bd09a3fc7f7dc6bcd5a1a39fdd84e Mon Sep 17 00:00:00 2001 From: Tom Nicholas Date: Wed, 28 Jun 2023 17:13:53 -0400 Subject: [PATCH 20/38] double that Co-authored-by: Deepak Cherian --- doc/internals/duck-arrays-integration.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/internals/duck-arrays-integration.rst b/doc/internals/duck-arrays-integration.rst index f7c4e9a2c04..95f31bc3922 100644 --- a/doc/internals/duck-arrays-integration.rst +++ b/doc/internals/duck-arrays-integration.rst @@ -14,7 +14,7 @@ This page is intended for developers who are interested in wrapping a new custom Duck array requirements ~~~~~~~~~~~~~~~~~~~~~~~ -Xarray does not explicitly check that that required methods are defined by the underlying duck array object before +Xarray does not explicitly check that required methods are defined by the underlying duck array object before attempting to wrap the given array. However, a wrapped array type should at a minimum define these attributes: * ``shape`` property, From ed6195c4a87307e8a4ebd14fa316bc62a2255cc8 Mon Sep 17 00:00:00 2001 From: Tom Nicholas Date: Wed, 28 Jun 2023 17:14:17 -0400 Subject: [PATCH 21/38] numpy array class clarification Co-authored-by: Deepak Cherian --- doc/internals/duck-arrays-integration.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/internals/duck-arrays-integration.rst b/doc/internals/duck-arrays-integration.rst index 95f31bc3922..2b7ca0644b7 100644 --- a/doc/internals/duck-arrays-integration.rst +++ b/doc/internals/duck-arrays-integration.rst @@ -24,7 +24,7 @@ attempting to wrap the given array. However, a wrapped array type should at a mi * ``__array_ufunc__`` method, * ``__array_function__`` method. -These need to be defined consistently with numpy :py:class:`numpy.ndarray`, for example the array ``shape`` +These need to be defined consistently with :py:class:`numpy.ndarray`, for example the array ``shape`` property needs to obey `numpy's broadcasting rules `_ (see also the `Python Array API standard's explanation `_ of these same rules). From 40eb53b95adb67710e37a060f4e562a19bd3e8ac Mon Sep 17 00:00:00 2001 From: Tom Nicholas Date: Wed, 28 Jun 2023 17:14:45 -0400 Subject: [PATCH 22/38] Remove sentence about xarray's internals Co-authored-by: Deepak Cherian --- doc/internals/duck-arrays-integration.rst | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/doc/internals/duck-arrays-integration.rst b/doc/internals/duck-arrays-integration.rst index 2b7ca0644b7..9a34090a870 100644 --- a/doc/internals/duck-arrays-integration.rst +++ b/doc/internals/duck-arrays-integration.rst @@ -33,8 +33,7 @@ Python Array API standard support ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ As an integration library xarray benefits greatly from the standardization of duck-array libraries' APIs, and so is a -big supporter of the `Python Array API Standard `_. In fact the crystallization of different array libraries' APIs towards -the standard has already helped xarray remove a lot of internal adapter code. +big supporter of the `Python Array API Standard `_. . We aim to support any array libraries that follow the standard out-of-the-box. However, xarray does occasionally call some numpy functions which are not (yet) part of the standard (e.g. :py:meth:`xarray.DataArray.pad` calls :py:func:`numpy.pad`). From a567aa4bb8bb04192c5ee6f0d73f400d11c3751f Mon Sep 17 00:00:00 2001 From: Tom Nicholas Date: Wed, 28 Jun 2023 17:14:57 -0400 Subject: [PATCH 23/38] array API standard Co-authored-by: Deepak Cherian --- doc/internals/duck-arrays-integration.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/internals/duck-arrays-integration.rst b/doc/internals/duck-arrays-integration.rst index 9a34090a870..3b6313dbf2f 100644 --- a/doc/internals/duck-arrays-integration.rst +++ b/doc/internals/duck-arrays-integration.rst @@ -35,7 +35,7 @@ Python Array API standard support As an integration library xarray benefits greatly from the standardization of duck-array libraries' APIs, and so is a big supporter of the `Python Array API Standard `_. . -We aim to support any array libraries that follow the standard out-of-the-box. However, xarray does occasionally +We aim to support any array libraries that follow the Array API standard out-of-the-box. However, xarray does occasionally call some numpy functions which are not (yet) part of the standard (e.g. :py:meth:`xarray.DataArray.pad` calls :py:func:`numpy.pad`). See `xarray issue #7848 `_ for a list of such functions. We can still support dispatching on these functions through the array protocols above, it just means that if you exclusively implement the methods in the Python Array API standard From 76237a9f52f9c71e977e1f4ffdcd13e097bc19be Mon Sep 17 00:00:00 2001 From: Tom Nicholas Date: Wed, 28 Jun 2023 17:15:43 -0400 Subject: [PATCH 24/38] proper link for sparse.COO type Co-authored-by: Deepak Cherian --- doc/user-guide/duckarrays.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/user-guide/duckarrays.rst b/doc/user-guide/duckarrays.rst index 03fa82cdaf3..73c2cdc57d9 100644 --- a/doc/user-guide/duckarrays.rst +++ b/doc/user-guide/duckarrays.rst @@ -45,7 +45,7 @@ A "numpy-like array" (also known as a "duck array") is a class that contains arr numpy-like functionality such as indexing, broadcasting, and computation methods. For example, the `sparse `_ library provides a sparse array type which is useful for representing nD array objects like sparse matrices -in a memory-efficient manner. We can create a sparse array object (of the ``sparse.COO`` type) from a numpy array like this: +in a memory-efficient manner. We can create a sparse array object (of the :py:class:`sparse.COO` type) from a numpy array like this: .. ipython:: python From 1923d4b20e97cb0f173e821cf0edc459f0602916 Mon Sep 17 00:00:00 2001 From: Tom Nicholas Date: Wed, 28 Jun 2023 17:16:19 -0400 Subject: [PATCH 25/38] links to docstrings of array types Co-authored-by: Deepak Cherian --- doc/user-guide/duckarrays.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/user-guide/duckarrays.rst b/doc/user-guide/duckarrays.rst index 73c2cdc57d9..734633f36c1 100644 --- a/doc/user-guide/duckarrays.rst +++ b/doc/user-guide/duckarrays.rst @@ -59,7 +59,7 @@ This sparse object does not attempt to explicitly store every element in the arr This approach is much more efficient for large arrays with only a few non-zero elements (such as tri-diagonal matrices). Sparse array objects can be converted back to a "dense" numpy array by calling ``.todense``. -Just like `numpy.ndarray` objects, `sparse.COO` arrays support indexing +Just like :py:class:`numpy.ndarray` objects, :py:class:`sparse.COO` arrays support indexing .. ipython:: python From b26cbd83ceb3748c06c392933f03e23bf59eb6b8 Mon Sep 17 00:00:00 2001 From: Tom Nicholas Date: Wed, 28 Jun 2023 17:16:41 -0400 Subject: [PATCH 26/38] don't put variable in parentheses Co-authored-by: Deepak Cherian --- doc/user-guide/duckarrays.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/user-guide/duckarrays.rst b/doc/user-guide/duckarrays.rst index 734633f36c1..288c4e897ef 100644 --- a/doc/user-guide/duckarrays.rst +++ b/doc/user-guide/duckarrays.rst @@ -99,7 +99,7 @@ equivalent numpy array - this is the sense in which the sparse array is "numpy-l Wrapping numpy-like arrays in xarray ------------------------------------ -:py:class:`DataArray` and :py:class:`Dataset` (and :py:class:`Variable`) objects can wrap these numpy-like arrays. +:py:class:`DataArray`, :py:class:`Dataset`, and :py:class:`Variable` objects can wrap these numpy-like arrays. Constructing xarray objects which wrap numpy-like arrays ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ From f62b4a9ac60e998cd3f8f7862768c05d6943bd7f Mon Sep 17 00:00:00 2001 From: Tom Nicholas Date: Wed, 28 Jun 2023 17:17:18 -0400 Subject: [PATCH 27/38] double backquote formatting Co-authored-by: Deepak Cherian --- doc/user-guide/duckarrays.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/user-guide/duckarrays.rst b/doc/user-guide/duckarrays.rst index 288c4e897ef..261bc91dc0e 100644 --- a/doc/user-guide/duckarrays.rst +++ b/doc/user-guide/duckarrays.rst @@ -118,7 +118,7 @@ For example, we can wrap the sparse array we created earlier inside a new DataAr We can see what's inside - the printable representation of our xarray object (the repr) automatically uses the printable representation of the underlying wrapped array. -Of course our sparse array object is still there underneath - it's stored under the `.data` attribute of the dataarray: +Of course our sparse array object is still there underneath - it's stored under the ``.data`` attribute of the dataarray: .. ipython:: python From 8d4bd3fd5ecfb584d69b77f43dc8ca0f3505d132 Mon Sep 17 00:00:00 2001 From: Tom Nicholas Date: Wed, 28 Jun 2023 17:17:38 -0400 Subject: [PATCH 28/38] better bracketing Co-authored-by: Deepak Cherian --- doc/user-guide/duckarrays.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/doc/user-guide/duckarrays.rst b/doc/user-guide/duckarrays.rst index 261bc91dc0e..7c470750bb4 100644 --- a/doc/user-guide/duckarrays.rst +++ b/doc/user-guide/duckarrays.rst @@ -145,9 +145,9 @@ If you want to change the type inside your xarray object you can use :py:meth:`D This returns a new :py:class:`DataArray` object, but now wrapping a normal numpy array. If instead you want to convert to numpy and return that numpy array you can use either :py:meth:`DataArray.to_numpy` or -:py:meth:`DataArray.values`, where the former is strongly preferred. (The difference is in the way they coerce to numpy - `.values` -always uses `np.asarray` which will fail for some array types (e.g. ``cupy``, whereas `to_numpy` uses the correct method -depending on the array type.) +:py:meth:`DataArray.values`, where the former is strongly preferred. The difference is in the way they coerce to numpy - `.values` +always uses `np.asarray` which will fail for some array types (e.g. ``cupy``), whereas `to_numpy` uses the correct method +depending on the array type. This illustrates the difference between ``.data`` and ``.values``, which is sometimes a point of confusion for new xarray users. Explicitly: :py:meth:`DataArray.data` returns the underlying numpy-like array, regardless of type, whereas From e9287dec19dd12c90cb81ca72c3c31ad16e7d5d6 Mon Sep 17 00:00:00 2001 From: Thomas Nicholas Date: Wed, 28 Jun 2023 17:24:04 -0400 Subject: [PATCH 29/38] fix list formatting --- doc/internals/index.rst | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/doc/internals/index.rst b/doc/internals/index.rst index 25671877297..132f6c40ede 100644 --- a/doc/internals/index.rst +++ b/doc/internals/index.rst @@ -9,9 +9,10 @@ extensions), which makes it easy to develop and extend. Instead, we push compiled code to :ref:`optional dependencies`. The pages in this section are intended for: -- Contributors to xarray who wish to better understand some of the internals, -- Developers who wish to extend xarray with domain-specific logic, perhaps to support a new scientific community of users, -- Developers who wish to interface xarray with their existing tooling, e.g. by creating a plugin for reading a new file format, or wrapping a custom array type. + +* Contributors to xarray who wish to better understand some of the internals, +* Developers who wish to extend xarray with domain-specific logic, perhaps to support a new scientific community of users, +* Developers who wish to interface xarray with their existing tooling, e.g. by creating a plugin for reading a new file format, or wrapping a custom array type. .. toctree:: From d1e9b8fb0873c0ceefcbef3c6da10a777f7ae0fa Mon Sep 17 00:00:00 2001 From: Thomas Nicholas Date: Wed, 28 Jun 2023 17:46:32 -0400 Subject: [PATCH 30/38] add links to glue packages, dask, and cubed --- doc/user-guide/duckarrays.rst | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/doc/user-guide/duckarrays.rst b/doc/user-guide/duckarrays.rst index 03fa82cdaf3..587e0e2b88c 100644 --- a/doc/user-guide/duckarrays.rst +++ b/doc/user-guide/duckarrays.rst @@ -12,9 +12,11 @@ additional features of these array libraries. Some numpy-like array types that xarray already has some support for: -* `Cupy `_ - GPU support, +* `Cupy `_ - GPU support (see `cupy-xarray `_), * `Sparse `_ - for performant arrays with many zero elements, -* `Pint `_ - for tracking the physical units of your data. +* `Pint `_ - for tracking the physical units of your data (see `pint-xarray `_), +* `Dask `_ - parallel computing on larger-than-memory arrays (see :ref:`using dask with xarray `), +* `Cubed `_ - another parallel computing framework that emphasises reliability (see `cubed-xarray `_). .. warning:: From 1ea207827fe2b6c78126377992c5d50afd0cf45a Mon Sep 17 00:00:00 2001 From: Tom Nicholas Date: Wed, 28 Jun 2023 17:47:56 -0400 Subject: [PATCH 31/38] link to todense method Co-authored-by: Deepak Cherian --- doc/user-guide/duckarrays.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/user-guide/duckarrays.rst b/doc/user-guide/duckarrays.rst index 7be7e96d3eb..3b2d94170a3 100644 --- a/doc/user-guide/duckarrays.rst +++ b/doc/user-guide/duckarrays.rst @@ -59,7 +59,7 @@ in a memory-efficient manner. We can create a sparse array object (of the :py:cl This sparse object does not attempt to explicitly store every element in the array, only the non-zero elements. This approach is much more efficient for large arrays with only a few non-zero elements (such as tri-diagonal matrices). -Sparse array objects can be converted back to a "dense" numpy array by calling ``.todense``. +Sparse array objects can be converted back to a "dense" numpy array by calling :py:meth:`sparse.COO.todense`. Just like :py:class:`numpy.ndarray` objects, :py:class:`sparse.COO` arrays support indexing From be919b684354e564b62b985612e676c9dc3bdbe8 Mon Sep 17 00:00:00 2001 From: Thomas Nicholas Date: Wed, 28 Jun 2023 17:51:24 -0400 Subject: [PATCH 32/38] link to numpy-like arrays page --- doc/user-guide/data-structures.rst | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/doc/user-guide/data-structures.rst b/doc/user-guide/data-structures.rst index e0fd4bd0d25..64e7b3625ac 100644 --- a/doc/user-guide/data-structures.rst +++ b/doc/user-guide/data-structures.rst @@ -19,7 +19,8 @@ DataArray :py:class:`xarray.DataArray` is xarray's implementation of a labeled, multi-dimensional array. It has several key properties: -- ``values``: a :py:class:`numpy.ndarray` holding the array's values +- ``values``: a :py:class:`numpy.ndarray` or + :ref:`numpy-like array ` holding the array's values - ``dims``: dimension names for each axis (e.g., ``('x', 'y', 'z')``) - ``coords``: a dict-like container of arrays (*coordinates*) that label each point (e.g., 1-dimensional arrays of numbers, datetime objects or @@ -46,7 +47,8 @@ Creating a DataArray The :py:class:`~xarray.DataArray` constructor takes: - ``data``: a multi-dimensional array of values (e.g., a numpy ndarray, - :py:class:`~pandas.Series`, :py:class:`~pandas.DataFrame` or ``pandas.Panel``) + a :ref:`numpy-like array `, :py:class:`~pandas.Series`, + :py:class:`~pandas.DataFrame` or ``pandas.Panel``) - ``coords``: a list or dictionary of coordinates. If a list, it should be a list of tuples where the first element is the dimension name and the second element is the corresponding coordinate array_like object. From d03e12597c9a3490d0b6d5ee9093d88eab1e692d Mon Sep 17 00:00:00 2001 From: Thomas Nicholas Date: Wed, 28 Jun 2023 17:52:44 -0400 Subject: [PATCH 33/38] link to numpy ufunc docs --- doc/user-guide/duckarrays.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/doc/user-guide/duckarrays.rst b/doc/user-guide/duckarrays.rst index 3b2d94170a3..0ec1b6f80cd 100644 --- a/doc/user-guide/duckarrays.rst +++ b/doc/user-guide/duckarrays.rst @@ -84,7 +84,8 @@ and various computation methods s.sum(axis=1) -This numpy-like array also supports calling so-called numpy ufuncs (link to numpy docs) on it directly: +This numpy-like array also supports calling so-called `numpy ufuncs `_ +("universal functions") on it directly: .. ipython:: python From 90a8bcb57da9441c8c96d4df0b2d74f6f89f58b1 Mon Sep 17 00:00:00 2001 From: Thomas Nicholas Date: Wed, 28 Jun 2023 18:39:24 -0400 Subject: [PATCH 34/38] add example of using .to_numpy --- doc/user-guide/duckarrays.rst | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/doc/user-guide/duckarrays.rst b/doc/user-guide/duckarrays.rst index 0ec1b6f80cd..90ee8ee63aa 100644 --- a/doc/user-guide/duckarrays.rst +++ b/doc/user-guide/duckarrays.rst @@ -148,14 +148,21 @@ If you want to change the type inside your xarray object you can use :py:meth:`D This returns a new :py:class:`DataArray` object, but now wrapping a normal numpy array. If instead you want to convert to numpy and return that numpy array you can use either :py:meth:`DataArray.to_numpy` or -:py:meth:`DataArray.values`, where the former is strongly preferred. The difference is in the way they coerce to numpy - `.values` -always uses `np.asarray` which will fail for some array types (e.g. ``cupy``), whereas `to_numpy` uses the correct method -depending on the array type. +:py:meth:`DataArray.values`, where the former is strongly preferred. The difference is in the way they coerce to numpy - :py:meth:`~DataArray.values` +always uses :py:func:`numpy.asarray` which will fail for some array types (e.g. ``cupy``), whereas :py:meth:`~DataArray.to_numpy` +uses the correct method depending on the array type. -This illustrates the difference between ``.data`` and ``.values``, which is sometimes a point of confusion for new xarray users. +.. ipython:: python + + s_da.to_numpy() + + s_da.values + +This illustrates the difference between :py:meth:`~DataArray.data` and :py:meth:`~DataArray.values`, +which is sometimes a point of confusion for new xarray users. Explicitly: :py:meth:`DataArray.data` returns the underlying numpy-like array, regardless of type, whereas :py:meth:`DataArray.values` converts the underlying array to a numpy array before returning it. -(This is another reason to use ``.to_numpy`` over ``.values`` - the intention is clearer.) +(This is another reason to use :py:meth:`~DataArray.to_numpy` over :py:meth:`~DataArray.values` - the intention is clearer.) Conversion to numpy as a fallback ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ From 14057b95ff7adb6faf5a6bd14e9000873c74efdb Mon Sep 17 00:00:00 2001 From: Thomas Nicholas Date: Wed, 28 Jun 2023 18:57:25 -0400 Subject: [PATCH 35/38] show example of .values failing --- doc/user-guide/duckarrays.rst | 3 +++ 1 file changed, 3 insertions(+) diff --git a/doc/user-guide/duckarrays.rst b/doc/user-guide/duckarrays.rst index 90ee8ee63aa..dc1d2d1cb8a 100644 --- a/doc/user-guide/duckarrays.rst +++ b/doc/user-guide/duckarrays.rst @@ -156,6 +156,9 @@ uses the correct method depending on the array type. s_da.to_numpy() +.. ipython:: python + :okexcept: + s_da.values This illustrates the difference between :py:meth:`~DataArray.data` and :py:meth:`~DataArray.values`, From 45000e426eafd53c5a2d541089bfba253a1f1c03 Mon Sep 17 00:00:00 2001 From: Thomas Nicholas Date: Wed, 28 Jun 2023 19:02:51 -0400 Subject: [PATCH 36/38] move whatsnew entry to unreleased version --- doc/whats-new.rst | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/doc/whats-new.rst b/doc/whats-new.rst index 3c206e298bc..9e0ae04f775 100644 --- a/doc/whats-new.rst +++ b/doc/whats-new.rst @@ -38,6 +38,8 @@ Bug fixes Documentation ~~~~~~~~~~~~~ +- Expanded the page on wrapping numpy-like "duck" arrays. + (:pull:`7911`) By `Tom Nicholas `_. Internal Changes ~~~~~~~~~~~~~~~~ @@ -96,10 +98,7 @@ Bug fixes By `Juniper Tyree `_. Documentation -~~~~~~~~~~~~~ - -- Expanded the page on wrapping numpy-like "duck" arrays. - (:pull:`7911`) By `Tom Nicholas `_. +~~~~~~~~~~~~ Internal Changes ~~~~~~~~~~~~~~~~ From da8719d6add5c5de68cb5d098b6c4d8df6c4d170 Mon Sep 17 00:00:00 2001 From: Thomas Nicholas Date: Wed, 28 Jun 2023 19:28:42 -0400 Subject: [PATCH 37/38] fix warning in docs build --- doc/whats-new.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/whats-new.rst b/doc/whats-new.rst index 9e0ae04f775..ce2c0a698ac 100644 --- a/doc/whats-new.rst +++ b/doc/whats-new.rst @@ -98,7 +98,7 @@ Bug fixes By `Juniper Tyree `_. Documentation -~~~~~~~~~~~~ +~~~~~~~~~~~~~ Internal Changes ~~~~~~~~~~~~~~~~ From 08c0f8476e6817a098a0cb262c04cfe7f7b09790 Mon Sep 17 00:00:00 2001 From: Thomas Nicholas Date: Thu, 29 Jun 2023 09:06:33 -0400 Subject: [PATCH 38/38] trigger CI