Repository navigation
perf(query): keep compaction, rollups and gates ahead of sustained ingest - #312
Merged
Merged
Conversation
…gest On a 32-vCPU host at 478k items/s, Parquet files grew from 3,038 to 17,251 in five minutes, DuckDB native memory swung up to 31 GiB with no read load, and version-rollup and anomaly writes timed out on the write gate. - The edge rollup built its parent hash table from every span within an hour of the affected minute. It now semi-joins parents against the distinct traces of the affected children, keeping the one-hour range and identical call edges, so the build side no longer grows with ingest rate. - Compaction starts its next cycle after 100 ms while cycles make progress and returns to the 10-second cadence when idle. - Span compaction builds trace.fidx from the ordered merge stream instead of rereading the output; the index bytes are identical. A 128-input compaction fell from 5.9-6.2 s to 5.1-5.3 s. - Version-marker transfer and retirement use set-based statements, cutting their time under the write gate from about 30 ms to 3 ms for 128 inputs. - OTLP conversion preallocates row slices and hoists per-resource work: 1,000 spans convert in 1.9 ms instead of 2.1 ms with 2.3 MB instead of 3.4 MB allocated. Locally under sustained mixed ingest, the end-of-run file count fell 37%, the service rollup's gate hold fell from 3.9 s to 2.3 s, and the anomaly writer's gate wait halved. Claude-Session: https://claude.ai/code/session_01XA52nWyQ3R5f5kK8kjpAQW
The compaction and maintenance callbacks append to slices that the test body reads; under the race detector that is a data race even inside a synctest bubble. Guard both with a mutex and read copies. Claude-Session: https://claude.ai/code/session_01XA52nWyQ3R5f5kK8kjpAQW
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
On a 32-vCPU host at 478k items/s (fanout-bench publish preset, generators on a separate host), Parquet files grew from 3,038 to 17,251 in five minutes, DuckDB native memory swung up to 31 GiB during ingest-only load while the Go heap stayed under 2 GiB, and version-rollup and anomaly writes timed out on the write gate.
trace.fidxfrom the ordered merge stream instead of rereading the output, with byte-identical index files. A 128-input compaction fell from 5.9–6.2 s to 5.1–5.3 s.No configuration, format, physical-schema, durability or timeout changes.
Verification
just check(pre-push hook)just test-raceTestEdgeRollupParentTraceScope, plan test), byte-identical streamed index (TestCompactionStreamingIndexMatchesReread), cancelled merge cleanup, backlog-driven schedule, bulk marker transfer, conversion fidelity (internal/ingest/testdata/conversion_rows.json)https://claude.ai/code/session_01XA52nWyQ3R5f5kK8kjpAQW