perf: fix the payload-breakdown 500 and slow top-observers on the analytics tab - #88
Closed
MrAlders0n wants to merge 8 commits into
Closed
perf: fix the payload-breakdown 500 and slow top-observers on the analytics tab#88MrAlders0n wants to merge 8 commits into
MrAlders0n wants to merge 8 commits into
Conversation
The IATA filter ran a correlated EXISTS over packets/observations with ILIKE, which skipped the iata index and took ~7s live. Keep a small channel_iatas table at ingest (like node_iatas) and filter against it. Also honor the iatas= param so multi-site regions stop getting the global list. Filter now ages out with packet retention rather than matching any retained packet.
Refresh last_heard on duplicate observations too, capped at hourly, so steady traffic can't age a channel out of the filter while its packets stay retained (dedup key has no heard_at). Guard the seed join against the docker /dev/shm cap like the trace one, and drop the unused last_heard index column so the upserts stay HOT.
The trace list joined every trace packet to ~50M observations to apply the IATA filter; the parallel hash join overran docker's 64MB /dev/shm and the filtered list errored. Keep a small trace_iatas table at ingest (like channel_iatas) and group packets by tag instead. Filtered stats are now per tag rather than per matching packet.
Refresh last_heard on duplicate observations too, capped at hourly, so steady traffic can't age a trace out of the filter (dedup key has no heard_at). Drop the unused last_heard index column so upserts stay HOT, and share one retention cutoff across the cleanup deletes.
GetScopeStats left-joined packets, observer_scopes and nodes at once, multiplying rows into the millions before COUNT(DISTINCT) deduped them (~10s per call). Count each table on its own and index packets(scope_id).
The routes list orders by last_seen with no index, so every page seq-scanned and sorted ~150k rows. Same fix as the packets list got in 008.
MrAlders0n
force-pushed
the
perf/analytics-stats-tuning
branch
from
July 24, 2026 02:25
0a29469 to
db40e6b
Compare
Member
Author
|
Deployed this to the meshmapper analyzer instance to test and wanted to flag the cost of migration 015. On that box packet_observations is ~55M rows / 6.8 GB, and the payload_type backfill ran for 1h43m. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the two slow bits on the Analytics tab. Both follow the same pattern already used for top-nodes/hourly stats: precompute into a materialized view refreshed on the existing timer, so the request just reads a small table.
Payload Types was 500ing
GetStatsPayloadBreakdownjoined every observation in the 7-day window back topacketsjust to readpayload_type. On a busy region that parallel hash over the whole packets table overran the db container's 64MB/dev/shmand returnedcould not resize shared memory segment ... (SQLSTATE 53100)— so the panel intermittently 500'd, and when it did succeed it took 4–9s.payload_typeonto the observation at ingest, the same waysource_brokerand the radio fields already are. No more join.mv_payload_breakdown_by_iataso the request is instant and the (now join-free) aggregate runs on the refresh timer.Top Observers was ~2s every load
It scanned 7 days of observations and grouped by observer on every request, and the bitmap went lossy under the 4MB
work_mem. Now served frommv_top_observers_by_iata, same shape asmv_top_nodes_by_iata.Notes
view_refreshtask; each does the ~2s aggregate in the background instead of on the request.go build ./...,go vet,go test ./...all green (12 pkgs).