perf(telemetry): preempt long merges for generation-0 debt and cut span merge allocation

[vishr]

Summary

An 8-vCPU benchmark of v2026.10.6-alpha (Fanout and telemetrygen on separate ccx33s) improved on v2026.10.5:

Workload	v2026.10.5	v2026.10.6
Traces	196k/s	275k/s
Mixed peak	401k/s	454k/s
Sustained	193k/s	314k/s

It still failed one gate. A higher-generation merge that started below pressure could not be interrupted. Two such merges took 132 s and 74 s, generation-0 debt reached the hard limit, and ingest stopped long enough for generators to time out.

Preemption:

• When reclaimable generation-0 debt reaches the hard limit, an unprepared higher-generation merge is cancelled and generation-0 work runs immediately.
• Cancellation stops before PrepareReplacement and the completion marker, so recovery, retired rollback sets and publication are unchanged.
• Each preemption halves the next higher-generation attempt. A minimum pair is never cancelled for debt, which guarantees higher-generation progress. A successful higher-generation publication restores the normal ceiling.
• Codex compared this against bounding higher-generation rows (about 3x more files over 30 days) and a second compaction lane (two recovery transactions). Cancellation keeps one lane and the existing file-count policy.

Span merge allocation:

• Span merges read their three sort keys through narrow column cursors that borrow page bytes, instead of rebuilding rows.
• Outputs and trace indexes are byte-identical, checked by SHA-256 across signals and generations.

Local measurements for a 128 x 50k span merge:

• 43% fewer allocated bytes and 87% fewer objects.
• 28% less CPU.
• 2.35x faster when idle and 1.68x faster with six cores busy.

The forced worst-case stall went from 42.8 s to 21.9 s with six cores busy. That reproducer makes debt cross the limit during a 24M-row pair. At 300k rows/s, debt needs about 85 s to climb from 512 to 1,024 batches.

Test plan

• just check
• just test-race
• Tests for:
  • repeated debt preemption that still publishes higher generations while debt retires;
  • cancellation, the watcher join and generation-0 isolation;
  • native recovery and reopen with no row loss.
• Native verification of 19.2M- and 72M-row outputs.
• Codex adversarial review: one starvation finding, fixed here and re-reviewed as approved.
• 8-vCPU benchmark after release

https://claude.ai/code/session_01XA52nWyQ3R5f5kK8kjpAQW