Skip to content

flowctl: bound fixture log blocks so a large transaction cannot trip the block assert - #3391

Open
jacobmarble wants to merge 1 commit into
masterfrom
jgm-preview-next-block-flush
Open

flowctl: bound fixture log blocks so a large transaction cannot trip the block assert#3391
jacobmarble wants to merge 1 commit into
masterfrom
jgm-preview-next-block-flush

Conversation

@jacobmarble

Copy link
Copy Markdown
Contributor

Description:

flowctl raw preview-next --fixture wrote each transaction as a single block in the shuffle log. Blocks have a ceiling: block::encode asserts that one block holds at most 65,536 documents. Feed it a bigger transaction and flowctl panics.

We hit this while benchmarking a materialization — 5 GiB transactions of 1 KB documents, about 1.3 million documents per shard. flowctl panicked during the first transaction. That killed the reader of the fixture pipe, so the generator died with a broken pipe and the run hung.

The assert was meant to be unreachable, and for the live path it is: that path calls BlockState::is_full() every iteration and starts a new block at 1 MiB of documents or 65,536 entries. The fixture never goes through it. It calls Writer::append_block directly, and that function has no check of its own.

The fixture now flushes a shard's buffer once it holds 32,768 documents or 64 MiB, whichever comes first, rather than holding a whole transaction. That fixes the panic and also means peak memory tracks block size instead of transaction size. In the same benchmark, peak flowctl memory fell from 29 GB to 13 GB and went flat after the fourth transaction. The 29 GB run had failed too, in a different way: the destination SDK refuses to write once system memory passes 90%.

Transaction boundaries do not change. The checkpoint frontier carries only each shard's final LSN as its read barrier, so a transaction spread over many blocks still commits exactly once.

Workflow steps:

No interface change. preview-next --fixture now accepts transactions of any size.

Documentation links affected:

None.

Notes for reviewers:

  • Reproducer: 300,000 documents of 256 B per transaction with --shards 4, about 75,000 per shard. Before, it panics and hangs. After, it completes 600,000 documents at roughly 51 MiB/s.
  • The two thresholds are a judgement call worth challenging. 32,768 leaves 2x headroom under the assert, and 64 MiB bounds the buffer. Reusing the live path's 1 MiB would also be safe, just with many more blocks.
  • This does not fully bound fixture memory. A 1 KB-document run still peaked at 26 GiB where the same byte volume in 1 MB documents peaked at 13 GB — 1024x the document count for identical bytes. Something that scales with document count is still accumulating. Out of scope here, but please do not read this PR as "fixture memory is fixed".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant