Skip to content

perf(query): resume edge drains without rediscovering the whole window - #317

Merged
vishr merged 1 commit into
mainfrom
perf/edge-resume
Oct 15, 2026
Merged

vishr merged 1 commit into
mainfrom
perf/edge-resume

Conversation

@vishr

@vishr vishr commented Oct 15, 2026

Copy link
Copy Markdown
Member

Summary

A resumed edge pass rebuilt its affected buckets over the entire frozen ingestion window before applying the event-time cursor. Draining an N-minute backlog therefore repeated discovery N times (audit F2). On Hetzner (v2026.10.6-alpha, 8 vCPU), edge work arrived in 60-72 s bursts, which matches these drains.

  • The first pass stores the window's per-bucket affected-span counts in a private cache table (read_edge_buckets). Later passes read those counts instead of rescanning Parquet.
  • Summaries, output and cursor commit in one transaction. They are cleared with the cursor on completion, skip-to-latest and restart rewind.
  • The read-cache version moves to 5, so private read tables rebuild once on upgrade.
  • The publication horizon, five-minute late-arrival repair, complete bucket output, namespace matching and yield behaviour are unchanged. The new table is rejected at the public SQL boundary.

Local measurement at 250k spans/minute:

Backlog Before After
5 min 988 ms 672 ms
15 min 5,659 ms 2,126 ms
30 min 25,581 ms 5,357 ms

Over a 30-minute backlog, incoming discovery falls from 225M to 7.5M logical rows.

It also fixes a flaky browser test. The system-overview test read every panel-query response body in a listener, so a body discarded by its first navigation could fail it (Network.getResponseBody: No resource with given identifier found). The test now collects only the final pinned document's responses and reads them while that document is alive. Every assertion is kept.

Test plan

  • just check
  • just test-race
  • Regression tests:
    • setup work stays linear in the backlog length, across a restart;
    • a late parent arriving during a paused drain;
    • failure injection on initial and resumed passes;
    • cache-version mismatch;
    • SQL boundary rejection.
  • Codex adversarial review: approved

https://claude.ai/code/session_01XA52nWyQ3R5f5kK8kjpAQW

A resumed edge pass rebuilt its affected buckets over the entire frozen
ingestion window before applying the event-time cursor, so draining an
N-minute backlog repeated discovery N times. The first pass now stores
the window's per-bucket affected-span counts in a private cache table,
and later passes read them instead of rescanning Parquet. Summaries,
output and cursor commit in one transaction and are cleared with the
cursor on completion, skip-to-latest and restart rewind. The read-cache
version moves to 5, so private read tables rebuild once.

A 30-minute drain at 250k spans/minute went from 25.6 s to 5.4 s
locally, with incoming discovery falling from 225M to 7.5M logical rows.

The system-overview browser test now collects only responses for the
final pinned document and reads their bodies while that document is
alive, instead of reading discarded first-document bodies in a listener.

Claude-Session: https://claude.ai/code/session_01XA52nWyQ3R5f5kK8kjpAQW
@vishr
vishr merged commit 44ba421 into main Oct 15, 2026
14 checks passed
@vishr
vishr deleted the perf/edge-resume branch October 15, 2026 23:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant