Repository navigation
Engine.materializePlan: what materialize will write, and its peak memory, before it starts - #43
Merged
Merged
Conversation
…ed, what materialize will write and the most bytes of the box's chunks it will hold, as the chunkmirage session asked in its second review of #41, so a page can warn, or offer to put the box on the chunk grid along an axis, before it starts: plan(region(...)) in materialize.ts gives the array's shape and chunk shape (the channel axes the ops add first), its chunks, the served chunks it computes, the axes it splits and the walk's order, and peakChunks with its bytes, peakBytes (chunkBytes each, besides the served chunks on their way); peakChunks is exactly what the walk holds, the documented bound (a slab across the split axes inside the outermost, n2 × n3 + n3 + 1 along three, n2 + 1 along two, 1 along one, none along none) where each split axis has three chunks or more, and where one has fewer (one, or two the last of which stays within one served chunk), every axis inside it has two at most and they are counted one by one, so it is never more than the walk holds (the bound alone overcounted 20 of 3000 random boxes), and it takes a tenth of a millisecond for a 10k³ box; the Region keeps the channel axes it adds (added); Engine.materialize and materializePlan share the region's making (regionOf); the tests check the plan of a box with channels and of the level's extent, that the projected peak is what the walk holds for each alignment (three, two, one and no split axes), for the wide box and for 400 random boxes; formats, design and the CHANGELOG say so, and that a smaller peak, should it ever matter, comes from computing the plane of served chunks two slabs share twice along the outermost split axis, not from compressing chunks while they wait or refusing a box; two over-long lines the commit before left in caching.md and design.md are rewrapped
davidackerman
force-pushed
the
feature/materialize-box
branch
from
October 16, 2026 20:12
821f22f to
f2fa984
Compare
davidackerman
marked this pull request as ready for review
October 16, 2026 20:28
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
Engine.materializePlan(view, options), which reports whatmaterializewill write and the most memory it will hold, without computing anything. This is the follow-up the second review of #41 asked for. #41 merged before this commit was pushed, so it comes as its own PR.With it, a page can warn before a large box starts, or offer to snap the box to the chunk grid along one axis.
MaterializePlanhas these fields:shapeandchunk: the array's shape and chunk shape, channel axes first.total: the array's chunks.needed: the served chunks it computes, each once.split: the axes the box splits.walk: the walk's axis order, outermost first.chunkBytes.peakChunks: the most box chunks held at once.peakBytes:peakChunks × chunkBytes. This leaves out the up toconcurrencyserved chunks in flight, as the docs say.The same plan is available without an Engine, as
plan(region(...)).materializeandmaterializePlanshare how the region is built (regionOf).peakChunksis the walk's exact count, not just the documented bound.Tests. The projected peak equals the walk's measured count:
There are also two plan checks: the full plan of a 2-channel float32 box, and that the level's extent plans a peak of 0.
Docs.
materializePlannext to the bound. It also records the review's preferred fix, should a smaller peak ever matter: compute the plane of served chunks that two slabs share twice, along the outermost split axis, rather than compressing waiting chunks or refusing the box.Checks:
npm run checkin web/ passes (typecheck, 71/71 tests, build), as domkdocs build --strict,pytest -q tests/test_docs.pyand ruff.