Skip to content

chunkmirage materialize writes exactly the box, as the browser engine does - #42

Merged
davidackerman merged 1 commit into
mainfrom
feature/materialize-box-python
Oct 16, 2026
Merged

davidackerman merged 1 commit into
mainfrom
feature/materialize-box-python

Conversation

@davidackerman

Copy link
Copy Markdown
Member

The Python side of #41: chunkmirage materialize --box (and chunkmirage.materialize(box=)) writes exactly the box by default, as the browser engine does.

--extent box (the default): each level is an array of the box's shape on it (the box rounded out to whole voxels of that level), its translation moved by the box's corner so it lands where the box is, chunked as the dataset is, counted from that corner. The frontends write it unchanged: the box is a cropped view of the pipeline (_Cropped, with info, num_levels, chunk and source, all a frontend asks of a pipeline), so the metadata and chunk encoding are the server's own, and an edge chunk is stored whole with zeros past the box as zarr's always are.

Each of the dataset's chunks computed once. A box chunk off the chunk edges reads up to eight of the dataset's chunks. ChunkedSource.read already stitches them; when the last stage keeps no chunks of its own (no cache: true), the read goes through a cache of up to 1 GiB (BOX_BUFFER_BYTES), and the box's chunks are walked as #41 walks them: the axes the corner is on chunk edges along first, then the one with the most chunks outermost, so a slab across the others is open at a time. While that slab fits, each chunk is computed once (22 for the test's box, 84 with no cache); past it, chunks are recomputed rather than held. A stage that caches is read as it is.

--extent level keeps #40's write: the box's chunks, rounded out to whole chunks, in arrays of the levels' whole shapes, for several boxes in one store. With no box the two are the same store.

Tests (tests/test_materialize.py, 2 new): a box off the chunk edges on every axis written as zarr v3 and v2, read back at the box's shape and translation with the served values on both levels, each of the dataset's chunks the box reaches computed exactly once through an uncached counting op; the command-line test keeps --extent level.

Docs: concepts/formats.md (the Python section: both extents, the cache and walk order), reference/cli.md (--extent, and --box's rounding), reference/api.md, CHANGELOG.md (both materialize bullets), where they said the Python side wrote the level's extent only. Checks: pytest (324 passed), ruff, mkdocs build --strict; chunkmirage schema unchanged; web/ untouched.

…rowser engine does: --extent box (chunkmirage.materialize(extent="box")) writes each level as an array of the box's shape on it, its translation moved by the box's corner so it lands where the box is, in the dataset's chunk shape counted from that corner, each chunk read from the dataset's chunks it overlaps; when the last stage keeps no chunks (no cache: true) those pass through a cache of up to 1 GiB, walked with the axes the corner is on chunk edges along first and then the one with the most chunks, so each is computed once while a slab fits (22 computed for the test's box, 84 without the cache); --extent level keeps #40's arrays of the levels' whole shapes, for several boxes in one store; tests write an off-grid box as zarr v3 and v2 and read it back in place, counting each chunk computed once; docs (formats, CLI, API) and the CHANGELOG say so, where they said the Python side wrote the level's extent only
@davidackerman
davidackerman merged commit 4e054cb into main Oct 16, 2026
2 checks passed
@davidackerman
davidackerman deleted the feature/materialize-box-python branch October 16, 2026 20:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant