Repository navigation
Fix rolling(center=True) splitting the rolling dimension into two chunks - #11759
inchang-ing wants to merge 3 commits into
Conversation
headtr1ck
left a comment
There was a problem hiding this comment.
Thanks for looking into this! I think this fix is too narrow. It targets one symptom in one code path, but the misaligned chunks come up in more places than the issue shows. Here are the output chunks along s for da.rolling(s=5, center=...).mean() (input chunked with t=1 plus the listed chunks along s):
| input chunks | center | bottleneck | main | this PR |
|---|---|---|---|---|
| (20,) | True | False | (19, 1) | (20,) ✅ |
| (20,) | True | True | (17, 3) | (17, 3) ❌ |
| (10, 10) | False | False | (14, 6) | (14, 6) ❌ |
| (10, 10) | True | False | (12, 7, 1) | (12, 8) ❌ |
| (10, 10) | True | True | (8, 9, 3) | (8, 9, 3) ❌ |
| (5, 5, 5, 5) | False | False | (9, 5, 5, 1) | (9, 5, 5, 1) ❌ |
| (5, 5, 5, 5) | True | True | (3, 5, 5, 7) | (3, 5, 5, 7) ❌ |
import numpy as np
import xarray as xr
da = xr.DataArray(np.ones((4, 20)), dims=("t", "s")).chunk(t=1, s=10)
with xr.set_options(use_bottleneck=False, use_numbagg=False):
print(da.rolling(s=5).mean().chunksizes["s"]) # (14, 6)So:
- The default bottleneck path (
_bottleneck_reduce, which has its own dask padding workaround) is not covered. The new test disables bottleneck/numbagg, which hides that. center=Trueis not really the cause. Withcenter=False, an input that is already chunked along the rolling dimension also comes out with different chunks.- The fix depends on how
dask.padandsliding_window_viewhappen to chunk the padding. It is also dask-only (is_duck_dask_arrayand.rechunk), so other chunked array types are not covered.
A more general approach might be to rechunk the result along the rolled dimensions back to the input chunks at the Rolling level, through the ChunkManager (e.g. Variable.chunk), so it covers every reduction path, both center settings and any chunked array type. The test could then simply assert result.chunksizes == da.chunksizes with bottleneck/numbagg on and off. The cost is an extra rechunk in the graph, so it would be good to hear from others whether that's acceptable, or whether this should rather be fixed upstream in dask's sliding_window_view(automatic_rechunk=True).
cc @dcherian @max-sixty, who have worked on the dask/numbagg rolling paths.
[This is Claude Code on behalf of Michael Niklas]
When a dask-backed array is rolled and then reduced, the rolling dimension of the result could come out with a different chunk structure than the input: padding appends its padding as separate chunks and dask's sliding_window_view rechunks so that every window chunk is complete. center=True split the dimension into an extra tiny chunk (e.g. (N-1, 1)), but center=False with an already chunked rolling dimension also came out misaligned, and the bottleneck path produced yet another chunking. Rechunk the result along the rolled dimensions back to the input chunks at the Rolling level, which covers every reduction path (the numpy, bottleneck and mean/count variants), both center settings and any chunked array type via the ChunkManager. This only changes chunking, not values. Fixes pydata#11689
1d40b26 to
a629c39
Compare
|
Thanks for the thorough review — you're right that the first attempt was too narrow, and your matrix was exactly what I needed. I've reworked the PR along the lines you suggested: New approach (
This covers every reduction path, both On Test: On the open question you raised (fix here vs. upstream in dask's |
ruff flags as PLR0124 (name compared with itself).
Description
Fixes #11689.
When a dask-backed array is rolled and then reduced (e.g.
.median()), the rollingdimension of the result could come out with a different chunk structure than the
input. The extra tiny chunk breaks downstream operations that use the rolling
dimension as a core dimension (e.g. ufuncs).
Root cause
Variable._rolling_windowpads the array before callingsliding_window_view.dask.array.padappends the right-hand padding as separate chunks, and dask'ssliding_window_viewrechunks so that every window chunk is complete. Both changethe chunking along the rolling dimension of the result:
center=Truesplit the dimension into an extra tiny chunk (e.g.(N-1, 1)).center=Falsewith an already chunked rolling dimension also came outmisaligned (e.g.
(10, 10)→(14, 6))._bottleneck_reduce, the default for many reductions)produced yet another chunking (e.g.
(17, 3)), because it does its own daskpadding workaround.
Fix
Rechunk the result along the rolled dimensions back to the input chunks at the
Rollinglevel (Rolling._restore_input_chunks), applied to:Rolling.reduce(covers the numpy path),DataArrayRolling._bottleneck_reduce(covers the bottleneck path),Rolling.count(and therebymean, which issum / count).This covers every reduction path, both
centersettings, and any chunked arraytype, since it goes through the ChunkManager (
obj.chunk(...)). The cost is oneextra rechunk in the graph. It only changes chunking, not values.
construct()is deliberately not touched: its chunking is controlled by theautomatic_rechunkargument ofsliding_window_view(seetest_rolling_construct_automatic_rechunk), and rechunking a windowed resultback to the input chunks would be quadratic in the number of chunks.
Checklist
doc/whats-new.rst)xarray/tests/test_rolling.pyxarray/tests/test_rolling.pypasses (3152 passed, 691 skipped)Testing notes
The new test (
test_rolling_dask_result_keeps_input_chunks) assertsresult.chunksizes == da.chunksizesacross a matrix of input chunkings(
(20,),(10, 10),(5, 5, 5, 5)),centeron/off, bottleneck on/off,numbagg on/off, and
mean/sum/count/median/std/max(144 cases).Before the fix, only the single-chunk
center=Truenon-bottleneck case wasaligned.