Make UK build outcomes honest in staging, telemetry and the Logbook

[juaristi22]

Stacked on #1099 (always-on telemetry emitter); retarget to main once it merges. Stacked PRs get no CI here, so CI runs after the retarget.

Readers must accept staging contract version 3 first: PolicyEngine/calibration-diagnostics#206 has to be deployed and its collector qualified (a blocked event returns 2xx) before this merges. An old collector answers blocked with 422, and #1099's delivery queue retries it indefinitely.

Why

After #1115, a review of how UK builds show up in the calibration dashboard found eight gaps:

1. Every telemetry failure carried error_code: BUILD_FAILED, and failure_class was never written.
2. Gate-blocked builds closed their staging run as completed; the dashboard inferred "blocked" from counts.
3. The spine build never recorded its output H5's sha256.
4. A graph-built dense candidate could not be assembled: its *.local_gates.json held the graph's unsigned 26-gate document, with no release_id or shippable.
5. Killed builds stayed running with no Logbook row. There was no SIGTERM handler, and the spine build ignored Ctrl-C.
6. A build refused at the preflight gates returned 1 with no gate event, so the dashboard showed it as passed.
7. The Logbook disposition and the telemetry status disagreed on every outcome that wasn't a pass.
8. Resumed builds weren't linked to the attempts whose store they reused, and reuse wasn't recorded.

What changes (one commit per area)

A. Staging contract v3 and one outcome classification (gaps 1, 2, 6, 7).

• Staging documents move to schema_version 3. A run can now end blocked, with block {phase, blocking_failure_count, blocking_gate_ids}. Failures gain failure_class.
• The delivery summary keeps contract_version 2, which publish_cli, both assemblers and the dashboard pin.
• Version 2 documents stay readable. The v2 fixtures are frozen beside a new v3 set (completed, calibration, failed with a class, blocked).
• microcosm.build.run_outcome classifies a build's end once, and the staging bundle, the hosted emitter and the Logbook disposition all take it from there:
  • a gate block anywhere in the cause chain is blocked;
  • Ctrl-C is INTERRUPTED, SIGTERM TERMINATED, MemoryError OUT_OF_MEMORY, a graph-node failure GRAPH_NODE_FAILED, and anything else BUILD_FAILED.
• The Logbook keeps its vocabulary:
  • blocked or failed → failed;
  • interrupted, terminated, or a rung abort → discarded.
• A dense build refused at the preflight gates now records:
  • a preflight_gates stage;
  • Logbook verdicts whose receipts resolve in the preflight gate document;
  • a blocked run at phase preflight.
• The hosted emitter gains a dedicated blocked run event and error_code on failures.
• With Add always-on build telemetry delivery #1099's _run_with_telemetry, each attempt's own close-out closes the hosted emitter with this classification first, so the wrapper's generic complete or fail only runs when nothing closed it. An error raised before the attempt takes over (its preflight digest, say) is classified there too. Add always-on build telemetry delivery #1099's test that expected a gate-blocked dense run to read failed now expects blocked.

B. Termination signals (gap 5).

• raise_on_sigterm() turns the first SIGTERM into BuildTerminatedError. It is a KeyboardInterrupt subclass, so the existing interrupt arms record it and kernels' except Exception can't swallow it.
• The spine, dense and national entry points install it. After the close-out they exit 143, so supervisors still see a termination, not a crash.
• The spine build gains an interrupt arm. A Ctrl-C or SIGTERM closes the run failed with its own class and writes a discarded row.

C. Sign the dense local gate report (gap 4).

• The driver projects the six local outcomes from the graph's terminal document. Nothing is re-evaluated, and the projection checks that the declarations match.
• It replays them through the gate battery under MARKS_ARTIFACT, grafts the scoped-report trio, and signs the result with the Logbook build id as release_id.
• The report is written before any refusal, so a blocked candidate leaves one too. Without the key or an attempt it stays unsigned and is never shippable.
• The full document keeps its evidence name, uk.full.gates.calibrated.gate_report.json. The package binds it, and it is recorded as outputs.full_gate_report.
• Gate receipts now resolve: the six local gates point into the signed report, and the rest point into the full document.
• --release-candidate refuses to start without a 32-byte signing key, or with --target-geographies that select no local targets.
• A filtered build writes no local report: outputs.local_gate_report is null, with the reason recorded. It never fills the missing local gates in as not applicable.
• The release preflight now requires exactly 32 bytes of key.
• The assembler and preflight tests now consume the build's own report. No test signs a report by hand.

D. Spine output checksum (gap 3).

• The spine's H5 digest is measured after the last write, smoke marking included. It is recorded in:
  • the sidecar (output);
  • a <h5>.sha256 file;
  • the spine_h5_creation stage event;
  • the Logbook pipeline verdict.
• A full build refuses an input H5 whose digest differs from the sidecar's output.sha256. Older sidecars still bind by content identity.

E. Resume lineage (gap 8).

• Each attempt's request.json names its Logbook build id and its staging run id.
• The rowwise manifest gains an execution block, outside the run parameters, so the identity is unchanged. It records:
  • the graph store and the attempt directory;
  • the nodes reused versus computed, where the first run to reach a node decides;
  • the earlier attempts on the same store.
• The counts also reach the staging run as a graph_execution stage event.

Known limits

• After the first SIGTERM, settlement re-hashes the sources and can outlast a supervisor's grace period. A second SIGTERM then kills the process mid-settlement.
• SIGKILL and out-of-memory kills can't be caught. Add always-on build telemetry delivery #1099's heartbeat and unexpected_process_exit event cover them, and they still write no Logbook row. Area E lets the next attempt on the same store name the killed one.

Testing

• Unit tests:
  • classify_failure, including the cause chain, wrapped spine blocks and every class;
  • logbook_disposition;
  • the SIGTERM handler: fires once, restores the previous handler, does nothing off the main thread.
• Both close-outs, tested for:
  • a pass;
  • a terminal block;
  • a preflight refusal;
  • a raised error;
  • Ctrl-C;
  • SIGTERM (exit 143, TERMINATED, discarded) on the spine, dense and national lines.
• The dense replay:
  • signed with a key, and accepted by _check_uk_dense_gate_report;
  • unsigned without a key, and unsigned without an attempt;
  • a filtered build;
  • a posture mismatch.
• The graph-built dense candidate passes the release preflight and the assembler on its own signed report.
• The spine smoke run checks its digest everywhere it is recorded. The checkpoint binding refuses a mismatched digest.
• Resume tests: a cold attempt reports zero reuse, and a --resume require replay reports reuse and names the cold attempt. Request ids and the stage event are tested too.
• tools/generate_staging_contract_fixtures.py --check passes, the v2 SHA256SUMS is unchanged, and tools/ci_test_plan.py verify passes.
• On the rebased branch (Add always-on build telemetry delivery #1099 head 72a969d), the UK and shared engine-free suites plus the data publish guard and dense release contract tests ran with 6554 passed and 3 failed:
  • one is fixed since: Add always-on build telemetry delivery #1099's early-emitter test fake now accepts the classification;
  • test_engine_is_reported_unavailable_without_the_uk_extra is environmental: the local venv has the UK extra;
  • test_subprocess_exchanges_ambient_token_and_delivers_events failed only under -n 6 and passes serially.
• The UK staging integration tests pass (2), and every commit imports and lints on its own.

🤖 Generated with Claude Code