[bashgeek]

Summary

A database review of a large production install (59 GB) found that the slowness comes almost entirely from spans. Spans are 57 GB of it, and 33 GB of that is indexes. Three of those indexes (23 GB) were never scanned in 9 days, but every span insert and every retention delete still had to update them. The server read about 1.4 TB of span index pages from disk in that time.

This PR shrinks the span indexes, makes inserts cheaper, lets retention keep up with heavy ingest, and removes three queries that read whole tables.

Type of change

• Bug fix
• New feature
• Refactor / cleanup
• Docs / config only

Changes

Span indexes (migration 0037)

• New partial indexes, one per span page category: spans_db_start, spans_cache_start and spans_job_start. Each span lands in at most one of them.
• They replace spans_proj_start_op, spans_op_start and spans_start_op.
• Also dropped:
  • transactions_id_covering: its only reader was the foreign key check on each span insert, which transactions_pkey serves just as well.
  • transactions_project_start: never scanned.
• Span samples add the filter for their op's category, so the planner can prove the partial index predicate. With only op = $3 it can't.
• Queries across all projects list the project ids explicitly. That keeps the leading project_id column usable on Postgres before 18, which has no skip scan.
• An unknown span category now matches nothing instead of scanning every span. No caller sends one.

Ingest

• Spans, logs and events get UUIDv7 ids, generated in Go as transactions already were. This keeps primary-key inserts at the right edge of the index, and works before Postgres 18.

Retention

• Time budget: each table gets a one-minute budget per pass instead of a 200k row cap, and a pass that is still behind is followed after 30 s instead of 5 minutes. The old cap held age retention to about 2.4M rows an hour per table.
• Adaptive batches: batch sizes adapt toward one-second statements. Transactions batch at most 1000, since each one cascades to its spans.
• Issue counts: event purges subtract the rows they removed from each issue's count, instead of recounting every event of the issue while holding its row lock. That recount stalled the grouper on large issues. Whether an issue is empty is now decided by an existence probe.

Queries

• /api/environments: walks each (project_id, environment) index one distinct value at a time.
• Unfiltered hourly transaction counts (dashboard density widget): read the telemetry_usage counters. The partial hours at the window edges are still counted from raw rows.
• Uptime stats: only read the 30 days they report on.

Behaviour changes

• Retention no longer repairs a drifted event_count. The one exception is an issue left with no events, whose count is reset to 0.
• Without filters, the density widget buckets transactions by the hour they arrived, not the hour they started.

Upgrade notes

On a large install, migration 0037 builds three indexes, and each build reads the whole spans table twice. Tindra waits for migrations at startup. Before upgrading, build the indexes by hand. That doesn't block ingest, and the migration then skips them:

CREATE INDEX CONCURRENTLY IF NOT EXISTS spans_db_start ON spans (project_id, start_timestamp DESC, op text_pattern_ops) WHERE op LIKE 'db%';
CREATE INDEX CONCURRENTLY IF NOT EXISTS spans_cache_start ON spans (project_id, start_timestamp DESC, op text_pattern_ops) WHERE op LIKE 'cache%';
CREATE INDEX CONCURRENTLY IF NOT EXISTS spans_job_start ON spans (project_id, start_timestamp DESC, op text_pattern_ops) WHERE op LIKE 'task%' OR op LIKE 'queue%' OR op LIKE 'celery%' OR op LIKE 'job%';

Test plan

• Span index plans: TestSpanPageQueriesUseCategoryIndexes records the SQL each span function sends and EXPLAINs it with sequential scans disabled. It requires the matching category index for summaries, timeseries and samples, both with and without a project filter. With the samples category filter removed, the test fails as it should.

• Benchmark, Postgres 18, 3M spans over 7 days (68% view, 10% each db and cache), old indexes vs new:

  Query	Old	New
  db summaries, one project, 24h	33 ms	31 ms
  db summaries, all projects, 24h	44 ms	45 ms
  db summaries, one project, 7d	225 ms	214 ms
  job summaries	0.4 ms	0.5 ms
  timeseries	9 ms	6 ms
  samples, one project	6 ms	6 ms
  samples, all projects	3 ms	9 ms

  • Index size: 13 MB against about 150 MB for the three old ones.
  • Inserting 300k spans: 16% faster with 26% less WAL. This was with every index in memory; installs where the old indexes don't fit gain more.

• Ingest: TestIngestWritesTimeOrderedRowIDs checks that events, logs, transactions and spans get version 7 ids.

• Retention:

  • New tests cover per-table budgets and fairness, adaptive batch sizing, and draining past the old 200k cap.
  • Decrements are checked across batches.
  • Empty resolved or ignored issues are deleted, while open ones are kept at 0, including when the stored count has drifted.
  • The opt-in performance tests pass.

• Queries:

  • Environments are compared against the old DISTINCT query across projects, tables, NULL and empty values, and token scope. On PG16 with 2M transactions: 767 ms became 0.4 ms.
  • Counts: the usage path and the raw path give the same buckets, including at window edges. Every filter falls back to the raw query. 452 ms became 3 ms.
  • Uptime: checks older than 30 days don't change the stats.

• go vet, golangci-lint (0 issues), and go test -race ./... (all packages) pass.

Checklist

• Tests added or updated
• No new dependencies introduced (or justified in the PR description)
• I assign copyright in this contribution to Blendbyte GmbH, as required by the
  Contributor License Agreement

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!