Repository navigation
chunkmirage materialize writes exactly the box, as the browser engine does - #42
Merged
Merged
Conversation
…rowser engine does: --extent box (chunkmirage.materialize(extent="box")) writes each level as an array of the box's shape on it, its translation moved by the box's corner so it lands where the box is, in the dataset's chunk shape counted from that corner, each chunk read from the dataset's chunks it overlaps; when the last stage keeps no chunks (no cache: true) those pass through a cache of up to 1 GiB, walked with the axes the corner is on chunk edges along first and then the one with the most chunks, so each is computed once while a slab fits (22 computed for the test's box, 84 without the cache); --extent level keeps #40's arrays of the levels' whole shapes, for several boxes in one store; tests write an off-grid box as zarr v3 and v2 and read it back in place, counting each chunk computed once; docs (formats, CLI, API) and the CHANGELOG say so, where they said the Python side wrote the level's extent only
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Python side of #41:
chunkmirage materialize --box(andchunkmirage.materialize(box=)) writes exactly the box by default, as the browser engine does.--extent box(the default): each level is an array of the box's shape on it (the box rounded out to whole voxels of that level), its translation moved by the box's corner so it lands where the box is, chunked as the dataset is, counted from that corner. The frontends write it unchanged: the box is a cropped view of the pipeline (_Cropped, withinfo,num_levels,chunkandsource, all a frontend asks of a pipeline), so the metadata and chunk encoding are the server's own, and an edge chunk is stored whole with zeros past the box as zarr's always are.Each of the dataset's chunks computed once. A box chunk off the chunk edges reads up to eight of the dataset's chunks.
ChunkedSource.readalready stitches them; when the last stage keeps no chunks of its own (nocache: true), the read goes through a cache of up to 1 GiB (BOX_BUFFER_BYTES), and the box's chunks are walked as #41 walks them: the axes the corner is on chunk edges along first, then the one with the most chunks outermost, so a slab across the others is open at a time. While that slab fits, each chunk is computed once (22 for the test's box, 84 with no cache); past it, chunks are recomputed rather than held. A stage that caches is read as it is.--extent levelkeeps #40's write: the box's chunks, rounded out to whole chunks, in arrays of the levels' whole shapes, for several boxes in one store. With no box the two are the same store.Tests (
tests/test_materialize.py, 2 new): a box off the chunk edges on every axis written as zarr v3 and v2, read back at the box's shape and translation with the served values on both levels, each of the dataset's chunks the box reaches computed exactly once through an uncached counting op; the command-line test keeps--extent level.Docs:
concepts/formats.md(the Python section: both extents, the cache and walk order),reference/cli.md(--extent, and--box's rounding),reference/api.md,CHANGELOG.md(both materialize bullets), where they said the Python side wrote the level's extent only. Checks:pytest(324 passed),ruff,mkdocs build --strict;chunkmirage schemaunchanged;web/untouched.