perf: keep ingest, rollups and compaction healthy under saturation

[vishr]

Summary

A benchmark of v2026.10.3-alpha on an 8-vCPU host (Fanout alone, generators on
a separate VM) exposed three problems under sustained overload:

• Live Parquet files grew to about 50,000 because compaction fell behind.
• Throughput fell across repetitions.
• Version rollup, anomaly persistence and detector queries timed out on the
  analytical write gate.

A local profile traced most of it to the rollups' five-minute safety lag. Every
service and edge pass re-aggregated the last five minutes of ingest and held
the write gate for tens of seconds.

Rollups

• Rollups follow an exact publication horizon instead of a five-minute lag.
  Each commit attempt gets a strictly increasing ingestion stamp, and the
  repository tracks attempts still in flight. Service and edge passes
  process disjoint windows up to that horizon, with no SQL scan for the
  newest or oldest stamp.
• Edge passes repair call and messaging relationships whose spans arrive up
  to five minutes apart. The affected buckets are computed once per pass in a
  transaction-local temporary table.

Detectors

• Error-rate, latency and volume detectors read completed service-rollup
  minutes instead of scanning 30 minutes of raw spans. Latency now uses the
  rollup's served-span p95; log patterns still read logs.
• Detectors persist the window they evaluated. They skip a check until a new
  minute completes and every query succeeds, and annotation writes that time
  out retry on their own.

Background work

• Version history drains on its own loop with a larger per-pass budget, so its
  waits no longer delay the read cache.
• Background readers (read cache, rollups, stats) do not enter the Parquet
  snapshot while a publication waits. Dashboard queries keep their 30-second
  grace period.

Ingest backpressure

• Ingest waits when reclaimable generation-0 compaction debt reaches 1,024
  batches.
• From 512, the writer's admission window widens from 10 ms to 250 ms, and
  compaction takes generation-0 groups first.
• New metrics: fanout_compaction_backlog_batches and
  fanout_ingest_backpressure_seconds_total.

No configuration, Parquet format, physical schema, or durability change.
Acknowledgement still follows durable publication.

Test plan

• just check
• just test-race
• Regression tests for:
  • publication horizon: in-flight, retried and adopted commits, restart and clock skew, first-publication bound;
  • edge late arrivals, in both orders and past the five-minute gap;
  • messaging keys;
  • temporary-table lifetime;
  • detector windows, stalled watermark and failed queries;
  • debt accounting and deadlock freedom;
  • writer pressure window;
  • background admission;
  • version budget and anomaly contention.
• Local end-to-end runs at a fixed mixed telemetrygen load:
  • rollup lag fell from 300 s to 0;
  • live files held near 2,000;
  • no write-gate timeouts or ingest backpressure in the final run.
• 8-vCPU publication benchmark after release

https://claude.ai/code/session_01XA52nWyQ3R5f5kK8kjpAQW