perf: per-IATA activity tables for channels/traces, plus scope + routes query fixes#87
Closed
MrAlders0n wants to merge 6 commits into
Closed
Conversation
MrAlders0n
marked this pull request as draft
July 24, 2026 01:37
MrAlders0n
marked this pull request as ready for review
July 24, 2026 02:19
The IATA filter ran a correlated EXISTS over packets/observations with ILIKE, which skipped the iata index and took ~7s live. Keep a small channel_iatas table at ingest (like node_iatas) and filter against it. Also honor the iatas= param so multi-site regions stop getting the global list. Filter now ages out with packet retention rather than matching any retained packet.
Refresh last_heard on duplicate observations too, capped at hourly, so steady traffic can't age a channel out of the filter while its packets stay retained (dedup key has no heard_at). Guard the seed join against the docker /dev/shm cap like the trace one, and drop the unused last_heard index column so the upserts stay HOT.
The trace list joined every trace packet to ~50M observations to apply the IATA filter; the parallel hash join overran docker's 64MB /dev/shm and the filtered list errored. Keep a small trace_iatas table at ingest (like channel_iatas) and group packets by tag instead. Filtered stats are now per tag rather than per matching packet.
Refresh last_heard on duplicate observations too, capped at hourly, so steady traffic can't age a trace out of the filter (dedup key has no heard_at). Drop the unused last_heard index column so upserts stay HOT, and share one retention cutoff across the cleanup deletes.
GetScopeStats left-joined packets, observer_scopes and nodes at once, multiplying rows into the millions before COUNT(DISTINCT) deduped them (~10s per call). Count each table on its own and index packets(scope_id).
The routes list orders by last_seen with no index, so every page seq-scanned and sorted ~150k rows. Same fix as the packets list got in 008.
MrAlders0n
force-pushed
the
perf/iata-activity-and-query-tuning
branch
from
July 24, 2026 02:25
4f8eac3 to
b870ba1
Compare
MrAlders0n
marked this pull request as draft
July 24, 2026 12:41
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rebased on the latest
dev, so this is only the six commits that aren't upstream yet — the packet index / bpchar filter / Go bump work that landed today already dropped out.These are the perf fixes for the per-IATA pages and a couple of query hot-spots I hit while poking at a busy region. Grouped them into one PR since they share the same shape (track activity in a small side table at ingest instead of scanning packets at query time) and touch overlapping files.
What's here
channel_iatas) — the IATA filter ran a correlatedEXISTSover packets/observations withILIKE, skipped the iata index, ~7s live. Now there's a tiny table kept up to date at ingest (same idea asnode_iatas) and the filter hits that. Also actually honors theiatas=param so multi-site regions stop getting the global list.trace_iatas) — same problem, worse: the filter joined every trace packet to ~50M observations and the parallel hash join overran docker's 64MB/dev/shm, so the filtered list just errored out. Same side-table fix.GetScopeStatswas left-joining packets + observer_scopes + nodes in one shot, multiplying into the millions beforeCOUNT(DISTINCT)deduped (~10s/call). Count each table on its own and indexpackets(scope_id).last_seenwith no index, so every page seq-scanned + sorted ~150k rows (2-4s). Added the index; same fix the packets list already got in 008.The two
fix(...)commits are review follow-ups on the channel/trace tables: refreshlast_heardon duplicate observations too (capped hourly) so steady traffic can't age an active channel/trace out of the filter, guard the one-time seed join against the/dev/shmcap, and drop an unused index column so the upserts stay HOT.Migrations
Four new ones,
011–014.011/012create the side tables and seed them from retained packets (parallelism off for the seed so the hash join spills to disk instead of/dev/shm);013/014are the two indexes. Renumbered to sit after the clock-drift/telemetry migrations that went in today.Notes
go build ./...,go vet, andgo test ./...all green (12 pkgs, 0 fail).011/012will take a moment on a full packet table — worth an eye during deploy, but they're one-time.