Record process telemetry for US release builds

[PavelMakarchuk]

Handoff (Pavel, 2026-10-01). A proposal for @MaxGhenis to decide: interim process telemetry for Route A, or wait for the graph line and emit staging events from the graph executor's observer hook (#951) instead, so every graph build reports itself. Schema questions (v2 resources, identity and work fields) go to @anth-volk. No CI has run yet because the PR is stacked on #1082; it runs when this retargets to main. The dashboard side is already on calibration-diagnostics main (PolicyEngine/calibration-diagnostics#198) and works with or without it.

Stacked on #1082. GitHub retargets it to main once #1082 merges.

Why

Staging telemetry says which stage a US build is in, but not how it is running or why it stopped:

• US target compilation emits nothing for about 2.8 h.
• A build killed for running out of memory stays running until the dashboard's 6-hour stall rule.
• No stage records CPU or memory.
• Failures are a free-text message with no class.

Without these, the staging dashboard cannot tell a slow build from a dead one, find the bottleneck, or count failures by kind.

What changes

All of this is opt-in on StagingTelemetry, and the fiscal-refresh release turns it on. Other callers and the version 1 contract fixtures are byte-identical.

• Resources (record_resources): every stage event and the progress document carry resources, meaning CPU user and system seconds (finished child processes included) plus current and peak RSS. Cores per stage then come from CPU over wall time.
• Heartbeat (heartbeat_seconds, 60 s for the release): a worker refreshes heartbeat_at and resources in the progress document. A run that dies without a final event is visible within minutes. Heartbeat uploads go through Keep US staging telemetry on for every build #1082's background uploader.
• Work progress (work_progress(), with _WorkCounter in the release): the progress document carries work, meaning units done, total, unit and elapsed seconds, so one snapshot gives a rate. Work reports add no events.
  • Target compilation: one base pass plus one income-tax pass per requested JCT family, over the same household batches. Cache hits advance a whole pass.
  • Post-export scoring: each sweep reports its own batches.
• Classed outcomes (record_outcome): a failed event records failure_class, failed_during and elapsed_seconds, and a complete event records elapsed_seconds. The classes are gate_refused, terminated, interrupted, out_of_memory, refused and error.
• Identity (record_identity()): the run manifest records the git commit, runtime versions, platform, CPU count, memory and thread-pool default. It records the command line as option names plus a SHA-256 only, because staging documents are served publicly and argument values are local paths.
• The progress document is now written only under the write lock, because the heartbeat thread writes it too.

Testing

• New tests:
  • resources on events and progress;
  • work progress with elapsed time, cleared on the next stage, and adding no events;
  • a heartbeat that keeps a silent stage alive and stops on completion;
  • classed failures, and the classifier itself;
  • outcome fields absent by default;
  • identity in the manifest;
  • the release's settings and identity, with no argument values leaked;
  • the work counter, with and without telemetry.
• Run locally: test_staging.py, test_us_exact_k_ladder_launcher.py, test_us_fiscal_refresh_builder.py and test_publish_guard.py give 398 passed, 8 skipped. ruff check . and ruff format --check pass.
• End to end: a script drove this StagingTelemetry through a simulated US release, local only with no uploads. The staging dashboard's Build progress tab (Show process telemetry on the build progress tab calibration-diagnostics#198) showed the measured batch rate driving the remaining time, cores and memory per stage, the heartbeat age, the host and commit, and a gate refusal with its class and failure lines.

🤖 Generated with Claude Code