[pull] main from pydata:main - #1080
Merged
Merged
Conversation
Co-authored-by: Deepak Cherian <dcherian@users.noreply.github.com> Co-authored-by: Michael Niklas <mick.niklas@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
) * Support reading Zarr V3 rectilinear (variable-sized) chunks Extracts the read-only half of #11279 (by Max Jones) into its own change. The read path just needs to catch the NotImplementedError that zarr_array.chunks raises for a rectilinear grid and fall back to write_chunk_sizes for the per-dimension chunk tuples. Write support (#11279) still has open correctness issues flagged in review -- align_chunks/safe_chunks validation reuses uniform-grid logic that doesn't handle partial chunks or region-start alignment -- so it's left for a separate follow-up PR rather than gating this on resolving those. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Use read_chunk_sizes instead of write_chunk_sizes for rectilinear reads Addresses review feedback on #11279 (same line of code, carried over in the split): write_chunk_sizes gives outer/shard sizes, but .chunks (used in the regular-grid branch just above) gives the inner chunk shape when sharding is enabled. read_chunk_sizes keeps the two branches consistent for sharded rectilinear arrays. #11279 (comment) #11279 (comment) Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Add tests for mixed regular/rectilinear dims and dask-backed reads Mirrors coverage from #11279 that was missing here: a dimension mix of regular and variable-length chunks, and reading into dask arrays (checking the dask array's own .chunks, not just Dataset.chunks). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Organize rectilinear chunk read tests into a parametrized class Groups the four tests into TestZarrRectilinearChunksRead, sharing the zarr-array-creation helper and parametrizing the two shape/chunk cases (pure rectilinear, mixed regular+rectilinear) across a plain read test and a dask-backed read test. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Fix rectilinear test expectations for zarr's actual regular-grid check CI failed on test-py314-no-dask: my local zarr (3.2.1) let .chunks return a compact per-dim value (2, (5, 10, 5)) for a mixed regular/rectilinear array without raising, but zarr 3.3.0 (what CI installs) correctly raises NotImplementedError there per the docstring, so xarray falls back to read_chunk_sizes and reports the fully-expanded form (2, 2, 2), (5, 10, 5)) instead. Verified against zarr 3.3.0 locally and updated the expected values to match the documented/correct behavior rather than the older version's quirk. Also collapsed the parametrize down to a single expected_chunks field, since it's the same fully-expanded tuple in both the encoding check and the dask-chunks check. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Handle rectilinear shard grids when reading (not just rectilinear chunks) zarr_array.shards raises NotImplementedError for a rectilinear shard grid the same way zarr_array.chunks does for a rectilinear chunk grid, but only the latter was handled. Regular inner chunks with rectilinear shards (e.g. chunks=(1,), shards=((1, 2),)) crashed xr.open_zarr with an uncaught NotImplementedError. Falls back to write_chunk_sizes (the outer/storage-level sizes, matching what "shards" means) the same way the chunks fallback uses read_chunk_sizes (the inner/read-level sizes). Reported at #11592 (comment) Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Address review feedback: PR reference, error message, region-write docs 1. whats-new.rst cited #11279 (the write-side PR) instead of this PR (#11592); fixed to reference both. 2/3. Trying to write a Dataset read from a rectilinear-chunked store (via round-trip to_zarr, or a region write) hit a generic "must be an int or a tuple of ints" TypeError that gave no indication this was about rectilinear chunks or how to work around it. Both cases go through the same _determine_zarr_chunks code path, so a single fix covers both: the error now names rectilinear chunks and states the two-step workaround (clear the chunk encoding *and* rechunk to a uniform size -- verified both are actually needed). Documented the region-write limitation in the rectilinear chunks docs note. 4. CI coverage: confirmed this is an upstream constraint, not something to fix here -- zarr>=3.2 requires Python>=3.12 (pypi.org/pypi/zarr/3.2.0/json), so xarray's Python-3.11-pinned environments can never resolve a zarr new enough to exercise this feature and correctly skip via requires_zarr_rectilinear_chunks. #11592 (comment) (Ian Hunt-Isaak) Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Validate encoding['shards'] on write; it bypassed the chunks guard Investigated the claim that reading a rectilinear-shard store and writing it back "succeeds cleanly" with valid rectilinear output. Traced zarr-python's ArrayV3Metadata.chunks/.shards properties: both gate on the identical `isinstance(self.chunk_grid, RegularChunkGridMetadata)` check, so for a store actually read through our open_store_variable fallback, encoding["chunks"] and encoding["shards"] always agree -- if the shard grid is genuinely rectilinear, both raise NotImplementedError together, and our existing encoding["chunks"] guard already catches it (verified: a round-trip from a real rectilinear-sharded store still raises TypeError, not a silent write). The part that *is* real: encoding["shards"] is passed straight through to zarr_group.create() in _create_new_array with no validation at all, unlike encoding["chunks"] (_determine_zarr_chunks). Confirmed this is pre-existing on main, unrelated to anything in this PR: hand-setting ds['var'].encoding = {"chunks": (10,), "shards": ((20, 10, 30),)} and calling to_zarr() writes a genuine rectilinear shard grid on main today, with no rectilinear reading involved at all. Added the same style of validation for encoding["shards"] that encoding["chunks"] already has, plus a regression test reproducing the hand-set-encoding case. #11592 (comment) Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Fix regression: int encoding['shards'] crashed the new write guard any(not isinstance(x, int) for x in shards) iterated shards without checking it was iterable first. A bare int shards value (applies to every dimension, same convention as encoding['chunks']) crashed with TypeError: 'int' object is not iterable instead of writing normally. Guard it the same way _determine_zarr_chunks already guards chunks: skip validation when shards is an int. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Validate encoding before resizing an existing array on append extract_zarr_variable_encoding ran after zarr_array.resize() in set_variables, so if it raised (as it now does for rectilinear chunk encodings, which xarray can't write yet) the store was left with a grown array whose new region was never written. Move the encoding extraction ahead of the resize; it only depends on the variable, not on the region or the existing array. Co-authored-by: Claude <noreply@anthropic.com> * Only flag encoding['shards'] as rectilinear when it nests sequences The new shards guard iterated whatever it was given and rejected any non-int element. zarr also accepts shards="auto" (which worked before this branch) and a config dict, both of which iterate to non-int values and so hit a misleading "looks like a rectilinear shard grid" error. Restrict the check to a list/tuple containing lists/tuples, mirroring the chunks guard, and cover "auto" in the regression test. Co-authored-by: Claude <noreply@anthropic.com> * Normalise rectilinear chunk encodings across zarr-python versions What zarr-python reports for an array whose chunk grid is regular along only some dimensions, or for regular inner chunks inside rectilinear shards, differs by version: 3.2 returns a mixed tuple like (2, (5, 10, 5)) from Array.chunks, 3.3 raises and read_chunk_sizes expands the regular dimension to (2, 2, 2), and 3.4 returns the compact inner chunk shape for sharded arrays where 3.3 raises. The tests were pinned to 3.3 and failed on 3.2.x (the locally installed version) and 3.4.0 (what unpinned CI environments resolve to). Compact zarr's per-chunk size listings back to a single int along every dimension that describes a regular grid, so encoding['chunks'] and encoding['shards'] come out the same regardless of zarr version: an int per regular dimension, a tuple of sizes per rectilinear one. Test expectations updated to that representation, with the fully expanded form kept for the dask-chunks assertion. zarr-python 3.2.x also crashes on a bare-int shards spec inside its own metadata parsing, so that test case is skipped below 3.3. Co-authored-by: Claude <noreply@anthropic.com> * Clarify rectilinear write errors for region and append writes The messages told users to clear the variable's chunk/shard encoding, but for writes into an existing array (region= or append_dim=) the store's own encoding is re-applied to the variable, so that advice cannot work. Say explicitly that such writes are unsupported and that the workaround only applies when writing to a new array. Co-authored-by: Claude <noreply@anthropic.com> * Expand a bare int encoding['shards'] to a per-dimension tuple zarr-python 3.2.x crashes on an int shard spec ("'int' object is not iterable" from its metadata parsing), so expand it to one size per dimension in extract_zarr_variable_encoding, as zarr itself does for chunks. This keeps the documented zarr >= 3.2 minimum working without a version skip in the tests. As a side effect the dask/shard alignment checks in set_variables, which only run for a tuple spec, now also cover int shard specs. Co-authored-by: Claude <noreply@anthropic.com> * Shorten comments and test docstrings Co-authored-by: Claude <noreply@anthropic.com> * Share one helper for the rectilinear chunks/shards write errors Co-authored-by: Claude <noreply@anthropic.com> * Fix opening a rectilinear array with a zero-length dimension into dask zarr's read_chunk_sizes lists no chunks at all along a zero-length dimension, and passing that empty tuple through as the preferred chunk made dask raise "Empty tuples are not allowed in chunks" on zarr >= 3.3. Report it as (0,) instead, which is what dask itself uses. Co-authored-by: Claude <noreply@anthropic.com> * Clarify that rectilinear reads don't raise the minimum zarr version Co-authored-by: Claude <noreply@anthropic.com> * Mark zero-length rectilinear test as requiring dask Co-authored-by: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
`pixi run release-contributors fails` with "the task 'release-contributors' is ambiguous" because the task is defined in two pixi environments, release and build-package. You have to name one.
* Add linecollection * Update dataarray_plot.py * add more lines * Update dataarray_plot.py * Remove old code * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update accessor.py * keep similar format as scatter * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update dataarray_plot.py * keep same format as scatter * Fix lines plot * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update dataarray_plot.py * Update utils.py * Attempt at hanlding coordinates with multiple dimensions * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update test_plot.py * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update test_plot.py * merge tests * Use similar format as scatter * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update utils.py * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update utils.py * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update utils.py * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update utils.py * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update utils.py * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update utils.py * Update utils.py * s can be float, no __len__ on those * Update dataarray_plot.py * Update dataarray_plot.py * Update dataarray_plot.py * Update dataarray_plot.py * add Hashable vs None ignores * Update utils.py * clean up * Fix color="k" * cleanup * Add tests for color and linestyle * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update test_plot.py * Update test_plot.py * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * assert not needed * Update utils.py * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update utils.py * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update utils.py * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update utils.py * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update utils.py * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add some docs examples * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update plotting.rst * Update plotting.rst * Update plotting.rst * Update plotting.rst * Update plotting.rst * Update plotting.rst * Update plotting.rst * Update api.rst * improve docs * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update __init__.py * fix legend values * Update api-hidden.rst * Update utils.py * Add more legend labels tests * Try without fix * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add fix * mypy fixes * Make sure lines legend is correct * Update test_plot.py * Update test_plot.py * Update test_plot.py * Update test_plot.py * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update facetgrid.py * Update facetgrid.py * Update dataarray_plot.py * remove commented code * Update xarray/plot/utils.py * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update whats-new.rst * Add more docs * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix bad merge * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * make mypy happy again * Update whats-new.rst * Update whats-new.rst * Update plotting.rst * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Apply suggestions from code review * align with ax.scatter * align more with ax.scatter * Update utils.py * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update plotting.rst * hide warning with :stderr: * extra linebreak helps? * 2 stderr to catch the 2 warnings? * nope didn't work * test all * remove the unneccessary stderr * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update whats-new.rst * Update whats-new.rst * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * make pre-commit happy * Fix mypy error with newer matplotlib stubs Co-authored-by: Claude <noreply@anthropic.com> --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Michael Niklas <mick.niklas@gmail.com> Co-authored-by: Claude <noreply@anthropic.com>
* Update whats-new for v2026.09.0 release - Rename the unreleased section to v2026.09.0 and add the release summary and contributor list - Move entries for #11348, #11292, #11290 and #11239 that were merged into older release sections - Add missing entries for #11552, #11521, #11486 and #11513 - Add missing PR links, fix the #11547 link and other markup - Fix the documented default of the use_bottleneck option - Add contributor names to the typos allow-list * Mention plot.lines and last Python 3.11 release in whats-new
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )