UK: supply the engine's local_authority input from the ladder's April 2023 codes (#953)

[juaristi22]

Closes the input half of #953: every household row of the UK rowwise build now carries the engine's local_authority enum input, resolved from the ladder's April 2023 ONS local_authority_code, so council tax and the five bespoke Council Tax Reduction schemes (Merton, Kingston upon Thames, Newham, Oxford, Westminster) stop seeing the MAIDSTONE default on Microcosm builds.

What changes

• Names source. tools/generate_uk_local_authority_names.py downloads (or reads from build/uk/) the ONS "Local Authority Districts (April 2023) Names and Codes in the United Kingdom" lookup (item e8b361ba…, sha256 6c2d811f…, 361 rows: E 296, W 22, S 32, N 11) and writes the committed resource uk/local_authority_names.json (code → ONS name → engine key), registered in the country package. geography_sources.LAD23_NAMES_SHA256 pins the lookup bytes: the loader verifies the download, the generator refuses a source whose digest differs (a re-published lookup is a reviewed re-pin), and the resource loader asserts the recorded digest and URL. One file covers the crosswalk's three roster vintages, because Scottish council-area and Northern Irish district codes are unchanged since ca_2019 / lgd_2014. geography_sources.load_lad23_names_lookup is the pinned loader.
• Key rule. uk_runtime/local_authority_input.py: upper-case, non-alphanumeric runs to one underscore, strip the ends, reproduces the LocalAuthority member name for every roster authority except King's Lynn and West Norfolk (the engine drops the apostrophe), which is the single declared alias. The resource loader refuses any row whose stored key is not the rule's output.
• Write point. assign_uk_geography_ladder writes local_authority right after local_authority_code; resolve_local_authority_engine_keys raises on any code the roster does not carry (never a default). local_authority joins UK_GEOGRAPHY_LADDER_COLUMNS and the spec's derive: list, so the non-blank validator, the pre-assigned-geography refusal and the assignment summary pick it up.
• Gate. uk_geography_ladder_gate reports blank inputs, off-roster codes and any row whose local_authority disagrees with its code. Because uk_local_geography_ladder_post_calibration reruns this gate on the calibrated candidate, the local battery carries the verdict with no driver change.
• Engine domain. The rowwise clone verifies the assigned member names against the installed engine's enum (verify_local_authority_engine_domain); test_uk_local_authority_input.py asserts the whole roster.
• Export declaration. household.local_authority is added to UK_ALLOWED_EXTRA_EXPORT_COLUMNS and its gates.json mirror, following the Align UK UC claim, child and period definitions #882 precedent for engine inputs the incumbent never carried.

Two corrections to the issue text

1. The issue asked to add local_authority to release_input_coverage_manifest.json as required. That manifest is locked set-equal to the licensed enhanced-FRS populated surface (assert_uk_release_input_coverage_manifest_current), so a net-new input cannot go there; the allow-list is the declared path.
2. No enum_domain battery gate is declared: that family needs a rules_engine evidence artifact the rowwise local battery never supplies, and the existing post-calibration ladder gate already reruns the consistency check.

Engine dependency and sequencing

The engine at 2.98.0 lacked the six April 2023 unitaries (North Northamptonshire, West Northamptonshire, Cumberland, Westmorland and Furness, North Yorkshire, Somerset), exactly the six roster codes that did not resolve. PolicyEngine/policyengine-uk#1870 added them (merged 2026-09-22, released as 2.100.0), and this PR now pins that release: uv.lock moves policyengine-uk 2.98.0 → 2.100.0 and microcosm-build[uk] floors at >=2.100.0, so the roster-wide membership test is green at the pin and no build can run against an engine without the members. Between 2.98.0 and 2.100.0 the engine also added three person-level capital-gains inputs (2.99.0) and changed the relative-poverty line to the median over individuals (2.99.2); the engine-sensitive UK tests (release input coverage, measure simulation, terminal gates, battery bindings, FRS spine, take-up gate, calibration run, the frame adapter) pass on 2.100.0.

Expected measured effect at 2.100.0

council_tax_benefit = where(council_tax_reduction_scheme_supported, simulated, reported). At 2.100.0 council_tax_reduction_scheme_supported is the England-pensioner scheme, the Scotland scheme, the Wales scheme, or one of the five working-age bespoke schemes (Merton, Kingston upon Thames, Newham, Westminster, Oxford); only those five terms read local_authority, and each also requires England and no pensioner. So supplying the input changes council_tax_benefit only for working-age England households drawn into those five authorities, and HBAI-net-income deltas are confined to them. Nothing else moves until policyengine-uk#1855 lands; the issue's Band D receipts (distinct council_tax_band_d_amount per nation = 296/22/32, the England simulated_council_tax aggregate) are taken on a pin at or past #1855.

Verification

• Focused set (the new test file plus every touched test, country spec, battery bindings, gate-battery contract pins, the US worker-identity and spec-bundle tests): green on 2.100.0; the roster-wide membership test passes at the pin.
• All 71 engine-marked UK tests (-m requires_uk, build + frame) pass on 2.100.0; the engine-sensitive UK files (release input coverage, measure simulation, terminal gates, battery bindings, FRS spine, take-up gate, calibration run, frame adapter) pass.
• tools/ci_test_groups.py --verify green; the new test file lands in spine-uk and uk:build. Generator --check green against the pinned ONS download.
• Engine round trip: test_cloned_local_authority_survives_the_engine_loader writes a cloned frame to H5, reads it back, loads it through Microsimulation and gets the written member names back decoded.
• The engine bump moves uv.lock, so us_runtime/worker_identity.py's APPROVED_UV_LOCK_SHA256 is re-pinned to this branch's lock (ee85373f…), as Align UK UC claim, child and period definitions #882's floor bump did; the sealed legacy-campaign digest is a separate identity and stays. If main's lock moves before this merges, that constant re-pins again at rebase. The three _UK_GATE_BATTERY_* digests in contract.py conflict with UK CGT: interpolate the sub-exempt remainder, anchor the clone incidence, and fence projected entrants (#970) #979; whichever lands second re-pins them.

🤖 Generated with Claude Code

[vahid-ahmadi]

Review at 71c25434 (worktree checkout of the three commits on 18c6d39a, engine 2.100.0 installed via uv sync --all-extras). Local: ruff clean, ci_test_groups.py --verify ok, coverage-manifest --check current, generator --check green against a fresh ONS download (sha256 6c2d811f… matches), pytest over the new file plus every touched test file 751 passed / 1 skipped / 0 failed. Merge-tree against main clean.

The UK side is sound. One blocker, three body corrections.

Blocking

1. APPROVED_UV_LOCK_SHA256 is stale. us_runtime/worker_identity.py:30-32 pins eb328a5c… (main's uv.lock); this branch's lock hashes to ee85373f…. _approved_uv_lock_sha256 (:264-274) hashes the repo lock when it finds a root and raises on mismatch, and the session-prime fixture in tests/conftest.py:232 calls the identity path with no lock argument, so every US test that primes identity fails: locally test_us_stacked_spine.py -k "worker_identity or semantic_identity or uv_lock" gives 10 failed / 6 errors, plus 2 errors in test_us_spec_bundle.py. CI agrees: fast (3.13, rest) is red, and the spine-uk / us-qs lanes carry the same files. The last engine-floor bump (Align UK UC claim, child and period definitions #882, c6763880) re-pinned this constant in the same commit; this one needs the same, to ee85373fdc0a53cd299bf2c3e93e9c5f09ef3bb67919db66865a5b24651321f4 (and check whether the legacy-campaign doctrine wants the old value retained).

What I verified

• Resource: 361 areas (E296/W22/S32/N11), the six unitaries at E06000061–66, King's Lynn alias at E07000146; stored keys reproduce from the rule, loader refuses otherwise (local_authority_input.py:114-118); roster test green at 2.100.0 (engine has 385 members, the 24 extra are abolished pre-2023 districts).
• Vintages: crosswalk ladder_vintage is ew:2023_april_lad;scotland:2019_council_area;ni:2014_lgd and E/W codes come from the OA→LAD April 2023 best-fit, so pre-2023 English codes can't reach the write point; the off-roster raise is tested (test_uk_geography_ladder.py:393-404).
• Write point, ladder columns, spec derive:, non-blank validator, pre-assigned refusal, gate (blank/off-roster/disagree) and the post-calibration rerun via battery_bindings.py:1524 all as described.
• Export: allow-list and gates.json mirror asserted by two tests; the coverage manifest is indeed locked set-equal to the eFRS surface, so the allow-list is the right path. I wrote a toy result through write_uk_rowwise_dataset, read the H5 back (string column of member names), loaded it through Microsimulation and got the same names with decode_enums=True. The in-repo round-trip test is in-memory only; a file round-trip would be the stronger test.
• 2.100.0 exists on PyPI; no other engine-version literal needed re-pinning. Conflicts with UK CGT: interpolate the sub-exempt remainder, anchor the clone incidence, and fence projected entrants (#970) #979 on the three _UK_GATE_BATTERY_* digests in contract.py; clean against UK national line: the signed v20 deferral, six A16 retirements on measured evidence, and the scoring-path fix #967.

Should fix

2. "sha-pinned" overclaims. The generator records the hash of whatever it downloaded (tools/generate_uk_local_authority_names.py:111), the loader only requires the field to be truthy (local_authority_input.py:91-92), and geography_sources.py:112 pins the item id but no digest. The only pin is a test literal. Add a LAD23_NAMES_SHA256 next to the item id that generator and loader both assert, as the crosswalk sources do.
3. Engine logic misstated. council_tax_reduction_scheme_supported at 2.100.0 (config.py:64-74) is england_pensioner | scotland | wales | merton | kingston | newham | westminster | oxford, not "true only for England, non-pensioner households in the five bespoke authorities". The conclusion (deltas confined to the five LAs) still holds since only those five terms read local_authority; fix the sentence.
4. Stale verification line. "1 deselected (the roster test above, red as designed)" contradicts "green at the pin" and the changelog.

Nits. test_uk_ladder_rowwise_clone.py:579-588 has a dead first disjunct; test_uk_local_authority_input.py:123-130 doesn't compute the bare rule it describes (.strip() runs before the alias lookup); unrelated format hunks in two test files; assign_uk_geography_ladder checks only the constituency layer's vintage, so a future ladder rebuild surfaces as the off-roster raise rather than a vintage message.

Not a formal review while it's a draft; happy to approve once the digest is re-pinned and CI is green.

[juaristi22]

Thanks, all addressed in 22e25fe.

1 (blocking). APPROVED_UV_LOCK_SHA256 re-pinned to this branch's lock, ee85373fdc0a53cd299bf2c3e93e9c5f09ef3bb67919db66865a5b24651321f4, the way #882's floor bump did. LEGACY_CAMPAIGN_UV_LOCK_SHA256 is a separately sealed identity (the legacy worker attestation), so it stays as is. test_us_stacked_spine.py -k "worker_identity or semantic_identity or uv_lock" and test_us_spec_bundle.py pass locally; if main's lock moves before this merges, the constant re-pins again at rebase.

2. geography_sources.LAD23_NAMES_SHA256 now pins the lookup bytes next to the item id. load_lad23_names_lookup verifies the download before parsing, the generator refuses a source whose digest differs (naming both digests, so a re-published lookup is a reviewed re-pin), and load_uk_local_authority_names_resource asserts the resource's recorded sha256 and URL against the constants; tests cover all three refusals, and the resource pin test asserts the constant against the literal so a re-pin shows in that diff too.

3. Body fixed: council_tax_reduction_scheme_supported is the England-pensioner, Scotland and Wales schemes or one of the five working-age bespoke schemes; only those five terms read local_authority, so the deltas stay confined to working-age England households in those authorities.

4. Stale verification line replaced with the 2.100.0 results.

Nits. Dead disjunct gone: the round-trip test asserts the exact assignment (London row from the two London OAs, then Anglesey, Aberdeen City, Antrim and Newtownabbey), and it now goes through write_uk_rowwise_dataset → H5 → pandas → Microsimulation, as you suggested. The bare-rule test computes the bare rule explicitly (and asserts it yields KING_S_LYNN_AND_WEST_NORFOLK). The unrelated format hunks in test_uk_rowwise_build_driver.py and test_uk_ladder_rowwise_clone.py are reverted (those files are not format-clean on main, so I formatted only my hunks). assign_uk_geography_ladder now refuses a ladder whose local_authority layer vintage is not 2023_april_lad, naming the names resource to regenerate, with a test; the clone fixture declares that vintage (its codes are real April 2023 codes).

Noted the #979 conflict on the three _UK_GATE_BATTERY_* digests; whichever lands second re-pins them.

[juaristi22]

Rebased onto main (e66d6f6c) and fixed CI at 0a25312:

• The three whole-manifest and two release_cut gate digests are re-pinned on top of main's re-pin (the UK CGT: interpolate the sub-exempt remainder, anchor the clone incidence, and fence projected entrants (#970) #979 conflict Vahid flagged); the approved uv.lock digest is unchanged because main's lock bytes did not move.
• The engine-uk and spine-uk failures on the previous head were my new ladder vintage refusal meeting test_uk_rowwise_candidate.py's toy ladder, which declared a synthetic local-authority vintage; it now declares 2023_april_lad (its codes are real London codes).
• Folded in, on María's ask: @pytest.mark.requires_us on the eight test_us_multispine_pool_tool.py tests that reach the policyengine-us metadata index, so a UK-only venv skips them instead of failing on the missing distribution.

Locally on the rebased tree: 1,342 tests across the touched files, the data shard, the contract pins and the US identity tests, 0 failures.

[vahid-ahmadi]

Second pass at 0a25312d (rebased onto e66d6f6c; worktree with engine 2.100.0 and policyengine-us installed). Local: ruff check clean on the 19 changed files (the two test files that ruff format flags are already unclean on main, and lint runs ruff check only); ci_test_groups.py --verify ok; coverage-manifest --check current; generator --check current against a fresh ONS download; pytest over the new and touched tests, contract pins and US identity tests 831 passed / 1 skipped (unmounted CSV) / 0 failed. Merge-tree against main b78f5a88 clean, and the merged lock still hashes to the new pin.

All round-one items closed:

1. APPROVED_UV_LOCK_SHA256 (worker_identity.py:30-32) equals this branch's uv.lock digest; the identity tests pass (25 + 2). Legacy constant untouched, as it should be.
2. LAD23_NAMES_SHA256 (geography_sources.py:121) is asserted by the lookup loader before parsing, by the generator (naming both digests), and by the resource loader against the recorded sha256 and URL; each refusal has a test, and the pin test ties the constant to the literal.
3. Body's scheme-supported sentence and verification paragraph corrected.
4. Nits: exact-assignment round trip now goes through write_uk_rowwise_dataset → H5 → Microsimulation; bare-rule test computes the rule and asserts KING_S_LYNN_AND_WEST_NORFOLK; the two test files carry only your hunks; assign_uk_geography_ladder (geography_ladder.py:447-455) refuses a non-2023_april_lad LA layer naming the resource to regenerate, with a test; both toy ladders declare the vintage.

Rebase commit: the three whole-manifest and two release-cut digests are re-pinned and the pin tests pass; requires_us is a registered marker with skip logic in the root conftest, and the pool-tool tests still run in the us-am engine lane, so the fold-in is mechanical.

Two non-blocking notes: the loader's URL refusal (local_authority_input.py:101-102) has no test, only the digest branch does; and the generator records whatever --url it was given while the loader asserts the constant, so a --url regeneration yields a resource the loader rejects. Self-consistent, but worth a line in the tool's help.

CI: 20 lanes green, including three of the six that were red on the previous head (engine-uk 3.13, fast spine-uk 3.13, wheels 3.14); their 3.14/3.13 twins are still running at post time and run the same tests. Approving on that and the local run; merge on green.