fix(warehouse): route alerts to the minutely rollup and sweep the MV set

[Makisuo]

Started as "which of our 40 materialized views are actually useful?" Two of the findings turned out to be bugs rather than bloat, so this is mostly a fix.

Everything below is measured — production storage from datasources_storage, and a week of read traffic from our own query.context spans. Read time is ~97 CPU-hours/week, of which ~91 is background automation; interactive UI traffic is under 5%. So MV value is decided by what the alerting and error-tick loops do.

Alert evaluation could never reach the tier built for it

canUseAnnualServiceOverview required allMetrics === true. computeAlertBuckets never sets it, so all 372k alert evaluations/week fell through to a flat scan of the 325M-row per-span service_overview_spans — 165k scans per 3 days — while the 2.2M-row service_overview_minutely, built for exactly that shape, sat unreachable.

allMetrics is not a capability limit. It only picks which aggregates the raw paths bother computing (metricSelectExprs); the rollup tiers read pre-aggregated columns and produce all five MetricNeeds unconditionally, so every TracesMetric is serveable. The flag was encoding "this is the dashboard caller."

Hour-multiple buckets are deliberately yielded back to traces_aggregates_hourly: it stores sample-weighted quantile states where these tiers store unweighted ones, so claiming them would have silently downgraded accuracy under heterogeneous sampling. Only the sub-hour case is claimed, where the alternative is the per-span scan.

span_metrics_calls_hourly held 0 rows for its entire existence

I was about to delete this as dead. A code comment citing a measured ~7s p95 didn't fit that story, so I checked: the metric is live as traces.span.metrics.calls — ~880k rows / 2 days across 30 services, monotonic. The MV filtered MetricName IN ('span.metrics.calls', 'calls'), because spanmetricsconnector output is namespaced by its pipeline.

The write filter and the read fast-path missed it identically, so they agreed with each other and disagreed with reality. Fixed on both sides rather than deleted.

Sweep (migration 0019 / local schema v9)

• Drop error_spans + its MV. No query ever called from(ErrorSpans), yet warehouse-catalog.ts still described it to LLM agents as "use this instead of traces WHERE StatusCode = 'Error'" — so run_sql agents were actively steered onto a table no product code reads.
• Bound attribute_values_hourly cardinality. With AttributeValue != '' as its only filter it had reached 1.59B rows / 12.3 GB to serve an autocomplete dropdown read ~217 times a week. The rules come from measuring the table: a length cap and an id-key denylist — the two intuitive mechanisms — would both have missed idle_ns and busy_ns, short numeric measurements that were ~70% of the rows. Verified against production: 93.6% of rows dropped, zero values lost for http.response.status_code, http.method, db.system, service.name, deployment.environment, http.route, server.address and every other canonical OTel key. attribute_keys_hourly is untouched, so all keys stay discoverable and filterable.
• Drop the three unread Events* columns from trace_detail_spans (1.19 GB). Confirmed unused by all nine readers, including the two AI-session queries in query-engine-integrations.
• Left alone deliberately: metrics_exponential_histogram is empty but is written by ingest — it's an unused capability, not dead code. Recorded in the doc so it isn't re-flagged.

Alert evaluation cache

buildEvaluateCacheKey snapped to 15s while endMs = now and the tick runs every 60s, and the TTL was 30s — shorter than the tick that created the entry. Windows now snap to the tick boundary, which also excludes the still-filling current minute, and the TTL clears one tick.

Reviewer note: this adds up to one tick of lag to alert evaluation, bounded by the tick period the scheduler already imposes. It's the one deliberate behaviour trade-off here. It does not collapse the per-service fan-out (each service compiles a distinct plan); it lets rules over one identical signal — e.g. warn-at-100 and page-at-500 — share an entry.

Verification

• Migration replay against real ClickHouse 26.2 — proves the DROP VIEW → DROP TABLE ordering (the inverse wedges ingest with Code: 60).
• SQL catalog analyzer sweep, 157 shapes, and the raw-vs-rollup parity suite.
• Production parity check of the old vs new alert route: 73 buckets, 0 mismatches on count and spanCount, 1,686,684 spans matched exactly.
• 2,272 unit tests across domain / query-engine / cli, plus the affected API suites; schema drift gates, manifest gate, lint, oxfmt.

One test worth flagging: ch.test.ts asserted toContain("FROM service_overview_spans") for this route. The tiered union reads that table too, as its raw edge — so the assertion passed on both routes and could never have caught this. It now asserts the tier that distinguishes them.

Scope

alertRawQuery — 173,000 s/week, now the largest remaining consumer by far — is untouched. It's user-authored SQL against raw tables, so no MV can see it and no routing guard applies. It and the other open items (trace_detail_spans TTL, the service_map_spans amplification) are recorded in docs/warehouse-rollups.md.

That doc is the durable half of this: the tier ladder, the three shapes that justify an MV, and the four rules whose absence produced each finding above.

🤖 Generated with Claude Code

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}

[github-actions]

🍁 Maple PR preview

Note

Preview resources were removed when this pull request closed.

Final commit 8a09917 · View workflow run