diff --git a/doc/user-guide/interpolation.md b/doc/user-guide/interpolation.md index ff9e778547d..db711b4ec74 100644 --- a/doc/user-guide/interpolation.md +++ b/doc/user-guide/interpolation.md @@ -295,6 +295,25 @@ filled.interp(x=[0.5, 1.5, 2.5], method="cubic") For the details of {py:meth}`~xarray.DataArray.interpolate_na`, see {ref}`Missing values `. +(interp.dask)= + +## Interpolating Dask arrays + +{py:meth}`~xarray.DataArray.interp` works lazily on {ref}`Dask arrays `. +Each task needs the full extent of the dimensions being interpolated along, +so the chunks along these dimensions are merged into a single chunk. + +With `method="linear"` or `"nearest"`, the data is first subset to the range +of the new coordinates, so only the chunks that are needed are read. This +requires the new coordinates to be in memory: if they are chunked, xarray does +not compute them in advance, so all chunks along the interpolated dimensions +are read and merged. If the new coordinates are cheap to compute, loading them +first can reduce the memory use considerably: + +```python +da.interp(x=new_x.compute()) +``` + ## Example Let's see how {py:meth}`~xarray.DataArray.interp` works on real data. diff --git a/doc/whats-new.rst b/doc/whats-new.rst index 3abeff78a77..6fdf85ff01b 100644 --- a/doc/whats-new.rst +++ b/doc/whats-new.rst @@ -349,6 +349,11 @@ Bug Fixes Documentation ~~~~~~~~~~~~~ +- Document how :py:meth:`Dataset.interp` and :py:meth:`DataArray.interp` handle + chunks of Dask arrays, and that loading chunked new coordinates first can + reduce the memory use for ``method="linear"`` and ``"nearest"`` + (:pull:`11786`). + By `Michael Niklas `_. - Add hidden intersphinx inventory entries for methods, properties and attributes at the path where they are defined, e.g. ``xarray.core.dataarray.DataArray.sel``. This lets tools such as diff --git a/xarray/core/dataarray.py b/xarray/core/dataarray.py index 202924b5b8a..d4d6a4d7fce 100644 --- a/xarray/core/dataarray.py +++ b/xarray/core/dataarray.py @@ -2436,6 +2436,12 @@ def interp( - When interpolating along multiple dimensions with methods `linear` and `nearest`, the process attempts to decompose the interpolation into independent interpolations along one dimension at a time. + - For Dask arrays, the chunks along the interpolated dimensions are merged + into a single chunk. With methods `linear` and `nearest`, the data is + first subset to the range of the new coordinates, so only the chunks that + are needed are read. This only happens if the new coordinates are in + memory, so load chunked new coordinates first (e.g. with ``.compute()``) + if they are cheap to compute. See :ref:`interp.dask`. - The specific interpolation method and dimensionality determine which interpolant is used: diff --git a/xarray/core/dataset.py b/xarray/core/dataset.py index 7976a8b5f62..18246ca61ac 100644 --- a/xarray/core/dataset.py +++ b/xarray/core/dataset.py @@ -3902,6 +3902,12 @@ def interp( - When interpolating along multiple dimensions with methods `linear` and `nearest`, the process attempts to decompose the interpolation into independent interpolations along one dimension at a time. + - For Dask arrays, the chunks along the interpolated dimensions are merged + into a single chunk. With methods `linear` and `nearest`, the data is + first subset to the range of the new coordinates, so only the chunks that + are needed are read. This only happens if the new coordinates are in + memory, so load chunked new coordinates first (e.g. with ``.compute()``) + if they are cheap to compute. See :ref:`interp.dask`. - The specific interpolation method and dimensionality determine which interpolant is used: