Repository navigation
perf(query): resume edge drains without rediscovering the whole window - #317
Merged
Merged
Conversation
A resumed edge pass rebuilt its affected buckets over the entire frozen ingestion window before applying the event-time cursor, so draining an N-minute backlog repeated discovery N times. The first pass now stores the window's per-bucket affected-span counts in a private cache table, and later passes read them instead of rescanning Parquet. Summaries, output and cursor commit in one transaction and are cleared with the cursor on completion, skip-to-latest and restart rewind. The read-cache version moves to 5, so private read tables rebuild once. A 30-minute drain at 250k spans/minute went from 25.6 s to 5.4 s locally, with incoming discovery falling from 225M to 7.5M logical rows. The system-overview browser test now collects only responses for the final pinned document and reads their bodies while that document is alive, instead of reading discarded first-document bodies in a listener. Claude-Session: https://claude.ai/code/session_01XA52nWyQ3R5f5kK8kjpAQW
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A resumed edge pass rebuilt its affected buckets over the entire frozen ingestion window before applying the event-time cursor. Draining an N-minute backlog therefore repeated discovery N times (audit F2). On Hetzner (v2026.10.6-alpha, 8 vCPU), edge work arrived in 60-72 s bursts, which matches these drains.
read_edge_buckets). Later passes read those counts instead of rescanning Parquet.Local measurement at 250k spans/minute:
Over a 30-minute backlog, incoming discovery falls from 225M to 7.5M logical rows.
It also fixes a flaky browser test. The system-overview test read every panel-query response body in a listener, so a body discarded by its first navigation could fail it (
Network.getResponseBody: No resource with given identifier found). The test now collects only the final pinned document's responses and reads them while that document is alive. Every assertion is kept.Test plan
just checkjust test-racehttps://claude.ai/code/session_01XA52nWyQ3R5f5kK8kjpAQW