Assemble the certified UK national candidate into a publishable release (#806)

changelog.d/806-national-contract-coverage.changed.md

@@ -0,0 +1,3 @@
+Extended the release contract to apply UK diagnostics and PolicyEngine UK
+model-package checks to `microcosm-uk-2024-25-national`, while accepting its
+non-empty per-cut artifact revisions and retaining the June legacy exclusion.

changelog.d/806-publish-tag-guards.changed.md

@@ -0,0 +1,2 @@
+Hardened publication so every artifact-pinned release must create a tag and
+all declared artifact revisions must uniformly match the tag being created.

changelog.d/806-release-integrity-fail-closed.changed.md

@@ -0,0 +1,9 @@
+The national release-integrity paths fail closed (#812 review round 2):
+publication refuses to move `latest.json` for any per-cut tag; the contract
+requires every certification-signed evidence file to exist in the release
+directory with exactly the signed bytes, and refuses a manifest pinning more
+than one revision; the assembler derives release identity (attempt id, spine
+digest, runtime pins) only from the signed diagnostics build block, requires
+the build record to agree, stages into a private directory, and atomically
+renames only an already-validated release into empty destinations. The
+printed publish command is rendered shell-safe.

changelog.d/806-seam-runtime-provenance.added.md

@@ -0,0 +1,6 @@
+The UK calibration seam records the calibrating environment's package
+versions (`python`, `policyengine-core`, `policyengine-uk`, and the microcosm
+shards) into the signed diagnostics build block. The release assembler pins
+manifests only from that authenticated provenance — a `--runtime-version`
+override may re-assert a signed value but never replace it — so a release
+can no longer describe an invented assembly environment.

changelog.d/806-uk-release-assembler.added.md

@@ -0,0 +1,4 @@
+Added `tools/assemble_uk_release_dir.py` to verify the certified UK national
+identity join, mint the calibration weight NPZ, assemble and self-validate the
+publishable release directory, and print the immutable inspect-lane publish
+command.

docs/uk-national-release-assembly-runbook-806.md

@@ -0,0 +1,160 @@
+# UK national release assembly runbook (#806)
+
+This runbook turns a green UK national calibration candidate into an
+inspectable release without promoting it to `latest.json`. The release id is
+constant across cuts (`microcosm-uk-2024-25-national`); each cut gets an
+immutable tag derived from the calibration attempt id.
+
+Do not run this sequence in PR CI. Calibration, certification, assembly, and
+publication consume licensed data and operator credentials.
+
+## Prerequisites
+
+Before starting, export:
+
+- `MICROCOSM_UK_TERMINAL_GATE_SIGNING_KEY`, containing the stable release key
+  as base64-encoded 32 bytes. Certification signs with it and assembly uses it
+  to re-verify the copied certification.
+- `HF_TOKEN`, authenticated for write access to
+  `policyengine/populace-uk-private`.
+- `SLACK_WEBHOOK_POPULACE_UK`, for the eventual promoted-release alert. The
+  inspect publication below uses `--no-latest`, so it does not announce a new
+  latest release.
+
+Keep the spine H5, its sibling `.build.json` and `.spine_gates.json`, the
+Ledger consumer artifact and manifest digest, and the licensed input-mass
+reference together through the run.
+
+## 1. Calibrate the national candidate
+
+Use the national calibration driver and record the input digest rather than
+relying on a mutable path:
+
+```bash
+uv run --no-sync python tools/calibrate_uk_national_dataset.py \
+  --input-h5 <spine-h5> \
+  --input-sha256 <spine-h5-sha256> \
+  --ledger-facts <ledger-consumer-facts> \
+  --ledger-facts-sha256 <ledger-facts-sha256> \
+  --ledger-manifest-sha256 <ledger-manifest-sha256> \
+  --staging-h5 <candidate-dir>/microcosm_uk_2024.h5 \
+  --diagnostics-json <candidate-dir>/calibration_diagnostics.json \
+  --build-record-json <candidate-dir>/build_record.json \
+  --terminal-gate-json <candidate-dir>/terminal_gates.json \
+  --release-id dev-uk-national-calibration
+```
+
+Use only the campaign doctrine overrides that were separately adjudicated.
+The build record's id has the form
+`uk-frs-calibration-attempt-<YYYYMMDDTHHMMSSZ>-<uuid8>`; assembly derives the
+per-cut tag from that suffix.
+
+## 2. Score and certify the cut
+
+First create the rule-1 score receipt against the pinned incumbent, following
+the scoring section of
+`docs/uk-national-calibration-runbook-623.md`. Then run the release-cut battery
+and compose the signed certification:
+
+```bash
+uv run --no-sync python tools/certify_uk_release_cut.py \
+  --candidate-h5 <candidate-dir>/microcosm_uk_2024.h5 \
+  --candidate-sha256 <candidate-sha256-from-build-record> \
+  --candidate-name microcosm_uk_2024 \
+  --spine-h5 <spine-h5> \
+  --diagnostics-json <candidate-dir>/calibration_diagnostics.json \
+  --build-record-json <candidate-dir>/build_record.json \
+  --seam-gate-report <candidate-dir>/terminal_gates.json \
+  --ledger-facts <ledger-consumer-facts> \
+  --ledger-facts-sha256 <ledger-facts-sha256> \
+  --ledger-manifest-sha256 <ledger-manifest-sha256> \
+  --input-mass-reference <licensed-input-mass-reference> \
+  --score-receipt <candidate-dir>/score_vs_enhanced_frs.json \
+  --release-id microcosm-uk-2024-25-national
+```
+
+With the default paths, this writes
+`microcosm_uk_2024.release_cut_gates.json` and
+`microcosm_uk_2024.release_certification.json` next to the candidate. Continue
+only when the certification says `shippable: true`.
+
+## 3. Assemble the release directory
+
+Assembly verifies the complete hash join before writing, mints the calibration
+NPZ from the candidate and spine weights, copies signed evidence byte-for-byte,
+and validates the finished directory:
+
+```bash
+uv run --no-sync python tools/assemble_uk_release_dir.py \
+  --candidate-h5 <candidate-dir>/microcosm_uk_2024.h5 \
+  --spine-h5 <spine-h5> \
+  --certification-json <candidate-dir>/microcosm_uk_2024.release_certification.json \
+  --build-record-json <candidate-dir>/build_record.json \
+  --diagnostics-json <candidate-dir>/calibration_diagnostics.json \
+  --seam-gate-report <candidate-dir>/terminal_gates.json \
+  --release-cut-gate-json <candidate-dir>/microcosm_uk_2024.release_cut_gates.json \
+  --score-receipt <candidate-dir>/score_vs_enhanced_frs.json \
+  --out-dir releases
+```
+
+The output is
+`releases/microcosm-uk-2024-25-national/`. The JSON summary records every
+digest, the derived cut tag, and the exact publication command. Use
+`--cut-tag microcosm-uk-2024-25-national-<YYYYMMDDTHHMMSSZ>-<uuid8>` only to
+override the derived tag deliberately; the override must keep that grammar,
+which the contract validates on every artifact revision.
+
+Assembly stages into a private directory, validates there, and atomically
+renames into empty destinations: re-assembling a cut requires removing the
+previous `releases/microcosm-uk-2024-25-national/` directory and the
+previously minted calibration NPZ first. Release identity — the attempt id,
+spine digest, and every runtime pin — comes only from the signed diagnostics
+build block; `--runtime-version PACKAGE=VERSION` may re-assert a signed value
+as an operator cross-check but refuses to replace one.
+
+## 4. Publish for inspection
+
+Run the command printed by the assembler. Its shape is:
+
+```bash
+uv run python -m microcosm.data.publish_cli \
+  releases/microcosm-uk-2024-25-national \
+  --repo-id policyengine/populace-uk-private \
+  --artifact-root <candidate-dir> \
+  --no-latest \
+  --tag-name microcosm-uk-2024-25-national-<timestamp>-<uuid8>
+```
+
+Do not omit `--tag-name`, and do not pass `--no-create-tag`: every artifact in
+the manifest is pinned to that immutable per-cut tag. `--no-latest` is
+mandatory for this inspect lane — and enforced: publication refuses to move
+`latest.json` for any tag that is not the release id itself, so omitting the
+flag fails closed instead of promoting an inspect cut.
+
+If tag creation returns HTTP 409 after the staging commit, publication can
+leave the constant branch
+`release-staging/microcosm-uk-2024-25-national` behind. Delete that branch
+manually in the private Hugging Face repository before retrying the same cut.
+Do not delete the immutable cut tag.
+
+## 5. Inspect on the dashboard
+
+Open the calibration-diagnostics dashboard with:
+
+```text
+?country=uk&release=microcosm-uk-2024-25-national
+```
+
+Adjudicate the release using the copied certification, scoped gate reports,
+calibration diagnostics, and score receipt. The release remains inspect-only
+until that review is complete.
+
+## Promotion is a separate change
+
+Do not write `latest.json` for this line yet. The current pointer cannot name a
+per-cut tag, while the certified loader expects the artifact revision to equal
+the release id and fetches the manifest from a tag named by that id. Promotion
+needs the loader/pointer design tracked in microcosm#823 before a reviewed cut
+can become the default; publication enforces this by refusing a pointer move
+for any per-cut tag. Until then, publish every national cut with `--no-latest`
+and its explicit per-cut tag.

packages/microcosm-build/src/microcosm/build/uk_runtime/calibration_run.py

@@ -8,11 +8,13 @@
 import hmac
 import json
 import os
+import platform
 import time
 import uuid
 from collections.abc import Mapping
 from dataclasses import dataclass
 from datetime import UTC, datetime
+from importlib import metadata
 from pathlib import Path
 from types import SimpleNamespace
 from typing import Any
@@ -438,6 +440,10 @@ def _run_uk_calibration_attempt(
     build_block = {
         "build_id": state.build_id,
         "code_pin": code_pin,
+        # Captured at solve time and signed with the diagnostics bytes, so a
+        # release assembler can only ever pin the environment that actually
+        # calibrated the candidate — never an invented one.
+        "runtime": _runtime_provenance(),
         "source_pins": dict(source_pins),
         "ledger": run_config["ledger"],
         "input_posture": {
@@ -894,6 +900,31 @@ def _ledger_provenance(artifact: Any) -> dict[str, object]:
     return provenance
 
 
+#: Packages whose versions the seam pins into the signed diagnostics build
+#: block. The release assembler treats these as the only legitimate source
+#: for manifest runtime/compatibility pins.
+_RUNTIME_PROVENANCE_PACKAGES = (
+    "policyengine-core",
+    "policyengine-uk",
+    "microcosm-frame",
+    "microcosm-calibrate",
+    "microcosm-build",
+    "microcosm-data",
+)
+
+
+def _runtime_provenance() -> dict[str, str]:
+    """The calibrating environment's package versions, for the build block."""
+
+    runtime = {"python": platform.python_version()}
+    for package in _RUNTIME_PROVENANCE_PACKAGES:
+        try:
+            runtime[package] = metadata.version(package)
+        except metadata.PackageNotFoundError:
+            runtime[package] = "unavailable"
+    return runtime
+
+
 def _register_census(
     registry: TargetRegistry, exclusions: Mapping[str, Mapping[str, str]]
 ) -> dict[str, object]: