Swap the microcosm-built UK spine into the national build and retire the incumbent feed

Parent: #145 (workstream E of the UK-migration epic #665). Successor to #686. Reconciled 2026-08-25 with the #623 calibration-seam plan after the first armed calibration campaign (9 runs on the uk-first-calibrated-run-623-686 build branch).

Scope, and what moved out

This issue owns the spine lane after #747 merges: the defect batch the calibration campaign surfaced (A), the per-step gating regime for the spine build (B), and the W9–W11 retirement plus dead code/markdown cleanup (C).

Moved out to #623 (PR #743): the calibration-only driver. The new production calibration seam (tools/calibrate_uk_national_dataset.py + calibration_seam.py) is the "national driver becomes calibration-only" step — it takes whichever h5 it is given, materializes measures, solves weights under doctrine, and never modifies a data variable. What stays here is the other half: retiring the legacy June path (tools/build_uk_national_dataset.py, with its SPI re-derivation and imputation stages) once the seam is proven, which is spine-lane cleanup per the 2026-08-24 ruling.

Gate: nothing here starts until #747 is reviewed and merged. Every item either modifies the spine build (invalidating #747's licensed evidence) or retires a fallback the comparison depends on.

State at the point of handover

leg	state
L0 build + determinism	complete — 3 rungs, twins payload-identical, record identity exact at 52,846
L1 identity receipts	e4–e8 green on both the post-E8 and pre-E8 spines
L2 whole-spine parity	`signed_parity` — 26 beyond-band divergences all signed, 0 unsigned, `--strict` clean
L3 baselines	measured — 177 input-mass totals, 47 QRF tail grids
First armed calibration	loss 0.079 · 378/388 bindable · candidate beats incumbent 293–48 on the frozen register · fiscal totals (UC, income tax) still 10–18% low

A. Spine defect remainder

Two of the five landed in #747 rather than waiting for a rebuild here — the review made the case that fixing the spine before the swap is cheaper than certifying it twice:

~~12 structural-NaN SPI person columns~~ fixed in Whole-spine parity for the UK: re-pinned incumbent, comparison instruments, signed register, and the E10 evidence (#686) #747: the SPI stage now writes the adjudicated stage-time zero on the FRS channel, so the artifact loads and the calibration seam's NaN fence can stay fail-loud without refusing every spine artifact. This no longer gates the first production calibration.
~~85+ age-tail disaggregation~~ fixed in Whole-spine parity for the UK: re-pinned incumbent, comparison instruments, signed register, and the E10 evidence (#686) #747 as the age_tail declarative source stage, with the six ONS band populations committed as a resource whose every cell records the register target id it must agree with, and the campaign's person_source_id clone-twin keying preserved.

Still open here:

Spine sidecar self-pin. spine-c.build.json carries the full stage census, artifact/resource pins, seeds and rules-engine version — and no reader joins it today. Downstream consumers should inherit provenance from the sidecar rather than re-deriving it; wire the join and pin the h5 to its sidecar.
Support-channel gaps: slc.repayments.england_postgraduate has zero support on the spine; the top-income (£1m+) channel holds 2 records on both datasets. Each needs a measured support channel or an explicit adjudication that the gap is accepted.
Investigation, not yet a code fix: fiscal totals 10–18% low on the calibrated candidate increasingly looks like spine income distribution rather than target defects. Assessment lane; findings feed back here or into a successor.

B. Per-step gating regime for the spine build

The 2026-08-24 ruling: no imputation or spine gating in the calibration path — the seam runs a calibration-scoped battery only. That is only coherent if the spine build polices itself, stage by stage, the way the US builds do (#327 doctrine):

Per-stage imputation-health gates. Each imputation stage gets a gate consuming its own receipts (support bounds, incidence vs donor, conservation) so a regression is caught at the stage that caused it, not at a whole-spine diff. The signed-differences register stays the swap-decision record; the per-step gates are the standing health checks.
Wire the dormant CGT donor support bounds — or decide they cannot be terminal. cgt_band_donor_support_bounds.json is declared in country_package.json and read by nothing; the other four support-bounds resources (was_wealth, lcfs_consumption, etb_vat, etb_services) are each listed in the uk_support gate's support_bounds_resources and in battery_bindings.py. The CGT one is not an oversight — the resource records why in its own semantic_fit_note: its intervals describe donor-stage initial support, and E8's hmrc_cgt_gains_spine Table-3 redraw moves amounts onto a different band surface afterwards, so "terminal support-gate applicability requires reviewer confirmation".

That makes this the sharpest argument for per-step gating rather than a stray wiring chore. The other four work as terminal gates because nothing rewrites their columns after they clip to donor; CGT is the one stage where a terminal gate provably cannot express the constraint. Two ways out, and the choice is the adjudication: gate the donor stage at stage time, before the redraw, on the bounds as they stand; or derive post-redraw bounds from the Table-3 surface and gate terminally like the others. Either way the resource stops being declared-but-dead. (This also closes the third of the three sign-offs left open on Port the CGT structure, salary sacrifice, and student loan stages onto the UK spine at FRS 2024-25 #740.)
Re-home the family-coverage contract. uk_release_family_build_stages requires 12 required_at_build families; the spine driver builds 10, the June driver 2, so neither driver alone can pass — the W9 seam. Fix by pointing the guard at the spine plan that actually builds the families (the spine's hmrc_spi_income_spine / hmrc_cgt_gains_spine supersede the June driver's two). Never by relaxing the gate, and never by sidecar plumbing — family_coverage status='required_at_build' is an executable contract, not documentation.
Arm the thresholds against the spine's own line (US Export input-mass-parity gate references the raw base, flagging correct calibration gains on PUF-imputed columns #327: a parity gate compares a release to its own previous certified line, not a foreign artifact): flip uk_input_mass_parity's reference to the spine's first certified release; arm uk_qrf_tail_concentration from the Assemble the microcosm-built UK spine: whole-spine parity and swap acceptance #686 L3 baselines (47 QRF tail grids, 12 household); RC battery + macro_realism on the swap build. Every armed threshold records which run measured it.
Release-cut certification. The microcosm-data release-verification contract expects the full 21-entry battery on a published artifact — that expectation maps to the monolith. Combining spine-side per-step gating + the seam's calibration-scoped battery into a publishable certification is contract evolution; until it lands, seam-built records say shippable: false with the reason.

C. W9–W11 retirement, dead code, markdowns

Retire the legacy June path once the seam reproduces the campaign result (the v9-equivalent live proof in the Calibrate the UK national build from Ledger-backed targets #623 plan): tools/build_uk_national_dataset.py's SPI re-derivation, replace_uk_spi_support_tables's zero-weight-channel refusal, and the CGT redraw all encode the June convention the spine has superseded.
W8 — certified-identity registry entry for the spine artifact. W11 — retire the CGT injection path the incumbent needed.
UK_CANDIDATE_DATASET_NAME → microcosm_uk_2024 (uk_runtime/terminal_gates.py:72, currently populace_uk_2023). Ruling 2026-08-25: the candidate carries the microcosm name, not the legacy populace one — superseding the earlier populace_uk_2024 note; the repo itself has moved to PolicyEngine/microcosm, and the parity-instrument tests already use microcosm_uk_2024.h5 as the canonical candidate filename. Sweep the stray populace_uk_2024* fixture names in test_uk_rowwise_build_driver.py in the same pass. Check UK_REFERENCE_DATASET_NAME alongside: once the spine is the build, "reference" means the previous certified microcosm line, not the enhanced-FRS artifact.
Un-pin the incumbent from microcosm-data/contract.py:219 and uk_runtime/weighted_integrity.py:241 once nothing reads it as an input. The parity reference itself stays — it is the historical comparison instrument and keeping it is how the swap remains auditable.
Dead code: UKPolicyEngineAdapter (zero production callers — the harness rebuilt its capability outboard); the assessment harness tools/calibrate_uk_national_spine.py retires once the seam driver covers it.
Markdown dispositions — a committed .md is a claim nobody re-checks; promote to a gate/test, make regenerable, or archive with a date:

file	disposition
`UK_COVERAGE_PROGRESS.md`	stale and misleading — baselined on the enhanced-FRS 2023-24 artifact; live content already lives in `release_input_coverage_manifest.json`. Delete.
`experiments/686-uk-spine-comparison-ledger.md`	the adjudication packet; make regenerable so it cannot drift from the register beside it
`experiments/686-uk-spine-swap-receipts.md`	R0–R5 run receipts. Archive as dated history once the swap lands
`experiments/630-uk-gate-adjudication-receipts.md`	check each threshold it justifies is pinned in `gates.json` with provenance, then archive
`experiments/612-uk-carrier-payload-receipt.md`	the "should be a test" case — confirm `compare_uk_h5_payload.py` covers it, then delete
`CONTRACT_FINDINGS.md`	#462 findings; the fix is merged and gated. Delete

Sweep the same pattern outside the UK lane (COVERAGE_PROGRESS.md, PROGRESS_BUILD*.md, _698-*.md, _*LANE-NOTES.md, FINAL_REPORT.md) as a separate cross-lane PR.

D. Acceptance

A: the spine self-pins its sidecar and resolves or adjudicates both support-channel gaps. (The NaN and age-tail fixes landed in Whole-spine parity for the UK: re-pinned incumbent, comparison instruments, signed register, and the E10 evidence (#686) #747; its licensed evidence and parity receipt are re-measured there.)
B: every imputation stage gated on its own receipts, including the CGT donor stage, whose support bounds are either wired at stage time or re-derived post-redraw rather than left declared-but-dead; the family-coverage guard passes against the spine plan unchanged; uk_input_mass_parity references the spine's own certified line and uk_qrf_tail_concentration is armed from L3, each with run provenance.
C: the national build runs from the microcosm spine with no policyengine-uk-data artifact as an input; the June path and the CGT injection are retired; the candidate is microcosm_uk_2024; no committed Markdown asserts an invariant not also enforced by a test or gate.
The whole-spine parity instrument still runs green against the frozen reference, as the historical record of what the swap changed.

Carried forward, not blocking

owned_land reviewed exclusion expires 2026-09-20 — calendar-critical, independent of this issue.
would_claim_uc frozen at 0.55 (U8; uk-data#452) — the pre-registered lever if an armed run's UC fit is strained.
The incumbent's ETB per-head division — a second upstream defect of the UK: banded CGT gains targets (by size of gain) to constrain the distribution, not just count + total #467 family, observed at Assemble the microcosm-built UK spine: whole-spine parity and swap acceptance #686, not filed. Needs a decision.
Calibration-lane follow-ups (welfare-cap UC fact, multi-family concept gap, UC nil-award/top-band cells, degenerate composition bindings) live with Calibrate the UK national build from Ledger-backed targets #623, not here.

@MaxGhenis

Shippability: what the contract measures today

Findings from the #743 review round (@MaxGhenis's audit, @vahid-ahmadi's second pass) that land on B.10. Measured by running a seam build under --release-candidate with a canonical release id through _check_uk_gate_battery_report:

Gate report shippable: true, over 6 of 21 declared entries
Build-record shippable: false — a literal the seam writes; nothing in production reads it
Promotion verifier refusals: 9

Two fields, one verdict

The gate report's shippable is computed (gate_battery.py:968): release_candidate and blocked_at_phase is None and blocking_ok and signing_key is not None. A run with release_candidate=False reports false whatever the gates found — a posture statement, not an assessment.

The build record's shippable is a hardcoded literal the seam invented. tools/build_uk_national_dataset.py writes no such field, and grep finds no production reader.

What B.10 has to change

#	Verifier check	What a scoped report carries
1	exact payload key set (`contract.py:2222`)	adds `posture`, `scope_exclusions`, `aggregate_admin_measurement`
2	`phases == [preflight, terminal]`	`[terminal]`
3	`phases_evaluated` covers both phases	terminal only
4	`policy_sha256` pin	appends `; calibration_seam_scope`
5	`gates_manifest_sha256` pin	filtered manifest, different canonical digest
6	`spec_fingerprint` pin	derives from that digest
7	entry-id set covers all 21	6
8	`target_fit.details.targets_checked` matches diagnostics	mismatch
9	signature authenticates under the release key	signs the scoped payload

The contract accepts one report, from one run, over one unfiltered manifest. Two reports covering disjoint scopes cannot satisfy it. B.10 picks the shape: a declared multi-report certification whose parts must union to all 21 entries with no gap and no overlap, each signed by its producer; or a compose step that merges signed parts and re-signs. Either way the phase and digest checks move from per-report to per-certification.

Sequencing consequence: the seam's shippable: false cannot block a real candidate. A candidate's verdict comes from the release-cut producer's full-battery run, and the nine refusals hold unconditionally, so the seam's report can never be that artifact.

Two seam-side adjudications

Refuse --release-candidate on the seam. The flag has two effects: it makes evidence_absent blocking, and it permits shippable: true. All six in-scope gates read evidence the same run produces, and absent evidence already refuses upstream, so the first cannot fire. The second signs a claim covering 6 of 21 entries. Refusing costs nothing measurable; the hand-written build-record literal goes with it.

Add expiry to the measure-exclusion register.

	`weighted_integrity.py:259`	`calibration_measure_exclusions.json`
Fields	`reason`, `approved_by`, `adjudication`, `approved_on`, `expires_on`	`name`, `reason`, `tracking`
Dates	canonical ISO, `expires_on > approved_on`	none
Lifetime	window enforced at gate evaluation; outside it the gate fails with a correct-or-renew message	never lapses
Tamper-evidence	full record sealed into the policy digest	effect sealed through `register_sha256`; justification unsealed

Pruning specs moves register_sha256, which reaches the identity digest and the Logbook row, so the effect already travels. Expiry is the gap: nothing forces a re-look. Adding the four fields and enforcing the window at apply time covers it; a second policy digest duplicates what register_sha256 holds. All five committed entries carry one salary-sacrifice adjudication, so this is a single migration needing an approver and a date.

Three carried defects

The June driver signs a declared diagnostics digest before the file exists. --calibration-diagnostics-sha256 enters the battery's release evidence at build_uk_national_dataset.py:939; the driver writes diagnostics at :980; nothing compares the two. The seam writes, hashes the bytes, then signs. Retires with C.11.
The parity trio's target half is not incumbent-sourced. EfrsParityReference carries source identity and input-column shares, no target surface, so uk_target_surface compares the solve's realized diagnostics against the declared register. The gate note now says so. An incumbent comparison needs an instrument that carries a target surface — B.9 territory.
The rule-1 score is not cross-pinned into signed run evidence. The scorer verifies both artifacts against required digests and emits a per-target drift table, but score_vs_enhanced_frs stays null in both calibration paths. Sealing it belongs to the same certification shape as B.10.

Name the scored candidate from the artifact actually written

Where the swap stands, and what the follow-up PR owns

PR #787 is the swap-machinery increment (11 commits, review-ready): the spine builds under its own 14-gate battery at the assembled/transferred checkpoints (first fully-green licensed build: 25 stages, 14/14, no phase blocked), the calibration seam refuses any input whose build was not gated, the June path is retired whole (−8,434 lines, no incumbent byte feeds any build), the candidate is microcosm_uk_2024, the #747 post-merge audit (#770–#775) is closed with licensed receipts, and four targets are re-bound to the facts they publish — measured on the seam: obr.ni −76.7% → −2.9%, the ONS lone-person categories now exact, couple_no_children +63.7% → +34.2% pending the exclusions below.

The follow-up PR completes the issue's acceptance — every remaining item, with why it sequences after the swap merges:

B4 — threshold arming against the spine's own line

Flip uk_input_mass_parity from the efrs-post-calibration incumbent descriptor to the spine's first certified line, then retire the descriptor (C4 completion). Cannot precede the cut that creates the line (US Export input-mass-parity gate references the raw base, flagging correct calibration gains on PUF-imputed columns #327 doctrine).
Arm uk_qrf_tail_concentration from the Assemble the microcosm-built UK spine: whole-spine parity and swap acceptance #686 L3 baselines (47 QRF tail grids), each armed threshold recording the run that measured it.

B5 — the release-cut certification producer

The executable home of the 16 declared national preflight/terminal gates, whose June runner retired in Swap the microcosm spine into the UK national build: self-policing build, retired June path, hardened parity instruments (#757) #787. No live path lost coverage in the interim — everything that runs today is gated by the spine battery, the seam battery, and the cross-run refusals — but the release cut needs its producer.
The multi-part certification: spine + seam + release-cut reports must union to the full declared entry set, no gap, no overlap (the three-way ownership partition in Swap the microcosm spine into the UK national build: self-policing build, retired June path, hardened parity instruments (#757) #787 already enforces the precondition; uk_aggregate_admin is the one declared shared id). Resolves the nine verifier refusal points from the 2026-08-25 audit (comment 5413502559): phase/digest checks move from per-report to per-certification.
The seam refuses --release-candidate and the hand-written build-record shippable literal goes, replaced by a pointer to the certification.

The `uk_target_fit` dispositions (adjudicated partition, 2026-08-26)

The post-rebind diagnostics (calibration_diagnostics_rebind.json, 403 targets, loss 0.1362) are the working packet:

~33 signed exclusions on the schema-2 register (approver + window, enforced at apply time): the 14 zero-support channels (11 top-income £1m+, 3 SLC), the 16 UC payment-distribution cells + welfare-cap pair, multi_family + unrelated_adult + lone_parent_non_dependent (pending a relationship-to-head frame column — the ONS split is unmeasurable without it), the 6 sparse HMRC band cells, and obr.fuel_duties (binding verified correct; the gap is universe scope — all-road-users receipts vs a household frame — plus LCFS diary under-capture at 37% of implied litres).
The would_claim_uc lever run (U8, uk-data#452) for the 7 UC caseload/two-child-limit targets — a spine rebuild at the raised rate plus seam re-run, with its take-up contract change signed.
The fuel-duties ledger retarget question (household-incidence fact vs total receipts) — chronicle-side adjudication.

Phase 3 — the first certified cut

End-to-end under the full regime: spine build (battery green) → seam calibration → B5 certification → the first certified microcosm_uk_2024 line, identity ladder + strict parity + scorer receipts committed, uk_input_mass_parity armed against the just-cut line with run provenance, and the #731 scorecard as the acceptance probe.

Carried, with dates

owned_land input-mass exclusion re-signed 2026-08-26 on a fresh 25-stage stability receipt (swing persists: 53.8% national); expires 2026-09-26 — the winsorised-donor / separate-land-imputation revisit registered on Port UK compact source stages and 2024-25 refresh from uk-data #145 falls due inside the follow-up window.
Measure-exclusion register window (5 salary-sacrifice entries) expires 2026-11-25.
UK_REFERENCE_DATASET_NAME re-semantics once a second certified line exists; the cross-lane markdown sweep as its own PR; Derive NHS allocation (and audit age-conditioned imputations) from the disaggregated age surface #785 (age-first NHS derivation) and policyengine-uk-data#469 (ETB per-head division) filed as successors.

The issue stays open until the certified cut lands; #787 closes #770–#775 on merge.

🤖 Posted with Claude Code

The follow-up PR is up as #793 (stacked on #787): the adjudicated exclusion pass, the U8 lever run — measured and re-frozen at 0.55 on the receipts (the raise leaves every UC cell unchanged while destroying the legacy housing-benefit surface; the real lever is capital-test support, microcosm#750) — the release-cut certification producer and its contract, and the QRF tail re-arm from the L3 baselines. The first certified cut blocks, correctly, at uk_target_fit on 13 characterized cells; the adjudication queue is in the PR body. Two successors filed: #791 (relationship-to-head frame column) and #792 (exclusion-set-dependent measure materialization — certification-blocking defect class).

🤖 Posted with Claude Code

Merge pull request #787 from PolicyEngine/uk-spine-swap-757

Merge pull request #793 from PolicyEngine/uk-release-certification-757

Rebase onto the certified-cut main and give the local gates their runner

Re-pin the UK spec digest for the registered adjudication register

Rebase onto the certified-cut main and give the local gates their runner

Bind the OBR cars fuel duty, ORR rail support, ONS energy spend and d…

Navigation Menu

Repository navigation

Swap the microcosm-built UK spine into the national build and retire the incumbent feed #757

Scope, and what moved out

State at the point of handover

A. Spine defect remainder

B. Per-step gating regime for the spine build

C. W9–W11 retirement, dead code, markdowns

D. Acceptance

Carried forward, not blocking

Shippability: what the contract measures today

Two fields, one verdict

What B.10 has to change

Two seam-side adjudications

Three carried defects

Where the swap stands, and what the follow-up PR owns

B4 — threshold arming against the spine's own line

B5 — the release-cut certification producer

The `uk_target_fit` dispositions (adjudicated partition, 2026-08-26)

Phase 3 — the first certified cut

Carried, with dates

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Swap the microcosm-built UK spine into the national build and retire the incumbent feed #757

Description

Scope, and what moved out

State at the point of handover

A. Spine defect remainder

B. Per-step gating regime for the spine build

C. W9–W11 retirement, dead code, markdowns

D. Acceptance

Carried forward, not blocking

Activity

juaristi22 commented on Aug 25, 2026

Shippability: what the contract measures today

Two fields, one verdict

What B.10 has to change

Two seam-side adjudications

Three carried defects

juaristi22 commented on Aug 26, 2026

Where the swap stands, and what the follow-up PR owns

B4 — threshold arming against the spine's own line

B5 — the release-cut certification producer

The uk_target_fit dispositions (adjudicated partition, 2026-08-26)

Phase 3 — the first certified cut

Carried, with dates

juaristi22 commented on Aug 26, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

The `uk_target_fit` dispositions (adjudicated partition, 2026-08-26)