Skip to content

perf: keep low-concurrency exporters fast and edge parent lookups near - #314

Merged
vishr merged 1 commit into
mainfrom
perf/saturation-2
Oct 15, 2026
Merged

vishr merged 1 commit into
mainfrom
perf/saturation-2

Conversation

@vishr

@vishr vishr commented Oct 15, 2026

Copy link
Copy Markdown
Member

Summary

An 8-vCPU benchmark of v2026.10.4-alpha (Fanout and telemetrygen on separate ccx33s) passed all gates and kept compaction ahead. But traces fell from 194k to 76k spans/s across repetitions, and sustained mixed ingest reached only 246k items/s. This PR fixes the two causes.

Writer:

  • While compaction debt is high, a group commit now closes after 20 ms without a new request. The 250 ms deadline and the row, request and byte bounds still apply.
  • Before, every group waited the full 250 ms. A telemetrygen trace process keeps one export in flight, so that capped each process at about four exports per second, which matches the observed 76k spans/s with 16 processes.

Edge rollup:

  • Each pass first looks up parents within five minutes of the affected buckets.
  • The one-hour lookup now runs only for traces with a child whose parent was not found nearby, and is skipped when there are none. Before, every pass scanned two hours of span keys.
  • Output is unchanged.

Local measurements:

  • Closed-loop submitters keep 94% (8 submitters) and 99% (16 submitters) of their unpressured throughput under pressure, up from about 20%.
  • A paced stream still writes about 13 times fewer batches under pressure.
  • With 60 minutes of history at 1M spans/minute, an edge pass takes 349 ms instead of 726 ms.
  • With 1% permanently missing parents, a pass takes 7.5% longer; with 5-30 minutes of history, 6-26% longer.

Test plan

  • just check
  • just test-race
  • Writer regressions for the idle close, deadline and bounds under pressure.
  • Edge regressions:
    • a parent 30 minutes before its child (one-hour lookup);
    • no unmatched children (lookup skipped);
    • existing late-arrival, messaging and plan tests.
  • 8-vCPU benchmark after release

https://claude.ai/code/session_01XA52nWyQ3R5f5kK8kjpAQW

An 8-vCPU benchmark of v2026.10.4-alpha showed traces falling from
194k to 76k spans/s across repetitions, and sustained mixed ingest
reaching only 246k items/s.

- While compaction debt is high, a group commit now closes after 20 ms
  without a new request, as well as at the 250 ms deadline or its
  row, request and byte bounds. A trace exporter keeps one export in
  flight, so a fixed 250 ms wait capped it at four exports per second.
  Closed-loop submitters now keep 94-99% of their unpressured
  throughput. A paced stream still writes about 13 times fewer batches
  under pressure.
- Edge passes look up parents within five minutes of the affected
  buckets first. The one-hour lookup runs only for traces with a child
  whose parent was not found nearby, and is skipped when there are none.
  With 60 minutes of dense history, a pass takes 349 ms instead of
  726 ms. Output is unchanged.

Claude-Session: https://claude.ai/code/session_01XA52nWyQ3R5f5kK8kjpAQW
@vishr
vishr merged commit 1c4863a into main Oct 15, 2026
14 checks passed
@vishr
vishr deleted the perf/saturation-2 branch October 15, 2026 16:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant