flowctl: bound fixture log blocks so a large transaction cannot trip the block assert - #3391
Open
jacobmarble wants to merge 1 commit into
Open
flowctl: bound fixture log blocks so a large transaction cannot trip the block assert#3391jacobmarble wants to merge 1 commit into
jacobmarble wants to merge 1 commit into
Conversation
jacobmarble
marked this pull request as ready for review
August 24, 2026 21:35
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description:
flowctl raw preview-next --fixturewrote each transaction as a single block in the shuffle log. Blocks have a ceiling:block::encodeasserts that one block holds at most 65,536 documents. Feed it a bigger transaction and flowctl panics.We hit this while benchmarking a materialization — 5 GiB transactions of 1 KB documents, about 1.3 million documents per shard. flowctl panicked during the first transaction. That killed the reader of the fixture pipe, so the generator died with a broken pipe and the run hung.
The assert was meant to be unreachable, and for the live path it is: that path calls
BlockState::is_full()every iteration and starts a new block at 1 MiB of documents or 65,536 entries. The fixture never goes through it. It callsWriter::append_blockdirectly, and that function has no check of its own.The fixture now flushes a shard's buffer once it holds 32,768 documents or 64 MiB, whichever comes first, rather than holding a whole transaction. That fixes the panic and also means peak memory tracks block size instead of transaction size. In the same benchmark, peak flowctl memory fell from 29 GB to 13 GB and went flat after the fourth transaction. The 29 GB run had failed too, in a different way: the destination SDK refuses to write once system memory passes 90%.
Transaction boundaries do not change. The checkpoint frontier carries only each shard's final LSN as its read barrier, so a transaction spread over many blocks still commits exactly once.
Workflow steps:
No interface change.
preview-next --fixturenow accepts transactions of any size.Documentation links affected:
None.
Notes for reviewers:
--shards 4, about 75,000 per shard. Before, it panics and hangs. After, it completes 600,000 documents at roughly 51 MiB/s.