perf: per-IATA activity tables + analytics query/matview tuning - #92
Merged
446564 merged 10 commits intoJul 24, 2026
Merged
Conversation
The IATA filter ran a correlated EXISTS over packets/observations with ILIKE, which skipped the iata index and took ~7s live. Keep a small channel_iatas table at ingest (like node_iatas) and filter against it. Also honor the iatas= param so multi-site regions stop getting the global list. Filter now ages out with packet retention rather than matching any retained packet.
Refresh last_heard on duplicate observations too, capped at hourly, so steady traffic can't age a channel out of the filter while its packets stay retained (dedup key has no heard_at). Guard the seed join against the docker /dev/shm cap like the trace one, and drop the unused last_heard index column so the upserts stay HOT.
The trace list joined every trace packet to ~50M observations to apply the IATA filter; the parallel hash join overran docker's 64MB /dev/shm and the filtered list errored. Keep a small trace_iatas table at ingest (like channel_iatas) and group packets by tag instead. Filtered stats are now per tag rather than per matching packet.
Refresh last_heard on duplicate observations too, capped at hourly, so steady traffic can't age a trace out of the filter (dedup key has no heard_at). Drop the unused last_heard index column so upserts stay HOT, and share one retention cutoff across the cleanup deletes.
GetScopeStats left-joined packets, observer_scopes and nodes at once, multiplying rows into the millions before COUNT(DISTINCT) deduped them (~10s per call). Count each table on its own and index packets(scope_id).
The routes list orders by last_seen with no index, so every page seq-scanned and sorted ~150k rows. Same fix as the packets list got in 008.
…queries The payload-breakdown and top-observers matviews stored a single 7-day total per group, so the 24h/7d/30d selector did nothing: payload ignored the window entirely (the store discarded `since`) and top-observers only varied which rows cleared the last_heard cutoff, never the counts. Bucket both views by hour over a 30-day horizon and sum the buckets in range, mirroring mv_hourly_iata_stats. The stats endpoints now honour the requested window while still reading a small precomputed table.
Member
Author
|
Take a peak at https://analyzer.meshmapper.net/?tab=Packets&types=4&iata=SEA And click around :) |
MrAlders0n
marked this pull request as ready for review
July 24, 2026 17:23
446564
approved these changes
Jul 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This rolls the work from #87 and #88 into one branch, plus a couple of follow-up fixes I found while testing. They were stacked on each other anyway, so reviewing them together is easier. Supersedes #87 and #88.
The common thread: the stats/analytics side was doing big live scans and cross-joins over
packet_observations(tens of millions of rows on a busy instance). This moves that work onto per-IATA activity tables and precomputed matviews so the read path stays cheap.Per-IATA activity tables
channel_iatastable kept current on ingest.trace_iatas); the old query spilled the hash join on large data.Query + index fixes
COUNT(DISTINCT)(~10s). Each table is now counted on its own.Analytics matviews
payload_typeonto observations and serves the payload breakdown from a precomputed view instead of scanning a week of data per request.Follow-ups found while testing
mv_hourly_iata_stats) so the read just sums the buckets in range. The windows work now and the read stays small.channel_messagestopacket_observations(each message heard by ~15 observers), exploding to ~5.9M rows and taking ~29s at 30d on our data. Both now read hourly-bucketed matviews like the rest, refreshed in the background.Testing
Deployed and running on a large instance (the meshmapper analyzer, ~55M observation rows). Verified all the panels across 24h/7d/30d.
One heads-up for anyone with a big table: migration
015(addingpayload_typeand backfilling the last 7 days) ran for ~1h43m on that instance — a 6.8 GBpacket_observationstable, single-statementUPDATEwith parallelism off, holding an exclusive lock the whole time, so the app can't finish startup until it's done. The matview builds after it were quick (seconds). If you're deploying against a large DB, expect that migration to be the long pole and maybe run it in a maintenance window.