perf: merge Parquet by column, bound pressured merges, and free rollup gates

[vishr]

Summary

An 8-vCPU benchmark of v2026.10.5-alpha (Fanout and telemetrygen on separate ccx33s) failed its gates. A compaction merge of 128 log batches ran on one core at about 52k rows/s, while logs arrived at 250-380k/s. Generation-0 debt reached the hard limit, ingest stopped for about 100 s, and generators timed out on their export deadline. The sustained mixed phase spent 152 of 300 s in backpressure.

• Column-wise merge. Compaction merges each physical column with one shared row schedule. Span sort keys are merged once and drive it. Up to eight columns decode, encode and compress concurrently, with no wide-row conversion.
  • Logical values, physical schema, row groups, footer bounds and the trace index are unchanged. The index is byte-identical.
• Bounded merges under pressure. While reclaimable debt is at or above half the limit, a generation-0 merge takes at most 4M rows, so debt retires in steps of seconds. Inputs above half that ceiling no longer count toward blocking debt, which keeps the bound deadlock-free.
• No gate parking. Service, edge and version passes release the analytical gate when background Parquet admission is unavailable, and retry on their cadence. Before, they held it for up to 15 s while anomaly persistence has a 10 s budget.
• Service rollup bounds. The service rollup materialises its affected keys once and binds exact native event-time bounds instead of scalar subqueries (audit F1).

Local measurements on 6.4M-row realistic merges, idle:

Signal	Before	After
Logs	575k rows/s	2.36M rows/s
Metrics	586k rows/s	2.42M rows/s
Spans	233k rows/s	794k rows/s

Other local measurements:

• Pressured 4M-row merges finish in 2.8-7.4 s with six cores busy.
• A rewrite model of 1B rows changes rewritten rows by at most 1.2%.
• On a dense 15M-row minute, the service rollup went from 2.9 s to 0.5 s, and peak native memory from 954 MiB to 52 MiB.
• An anomaly write that failed at 50 ms while publication was pending now succeeds in 10 ms.

No configuration, format, physical-schema or durability change.

Test plan

• just check
• just test-race
• Native verification of full-scale merge outputs:
  • all columns decoded;
  • span ordering and index ranges;
  • column parity and footer-bound parity with the previous merge.
• Tests for:
  • the pressured bound, oversized inputs and deadlock freedom;
  • nonblocking admission skips and retries;
  • service bounds with late arrivals, signed bounds and empty sets.
• Codex adversarial review: approved.
• 8-vCPU benchmark after release

https://claude.ai/code/session_01XA52nWyQ3R5f5kK8kjpAQW