UK: certify a Microcosm national dataset as the UK default, with separate national and local-area datasets and a corrected local-authority list #553

Summary

We plan to replace the UK default, enhanced FRS 1.56.16, with a Microcosm UK national release built on the policyengine-uk version this repo will ship. The certification tooling already handles a Microcosm UK release: a dry run on main passes every metadata check. What is missing is everything below the national level:

• a local-area dataset beside the national one;
• a correct local-authority list;
• nation scoping that works on any dataset.

This issue tracks the changes here and what the Microcosm release has to provide.

Dry run

This used a scratch clone of main (6a9c878); nothing was pushed. It ran on the published release hf://dataset/policyengine/populace-uk-private@microcosm-uk-2024-25-national-20261002T230158Z-5c6b3f68/releases/microcosm-uk-2024-25-national/release_manifest.json.

• The model check refused the pinned engine. certify-data --country uk --data-producer populace refused policyengine-uk 2.102.3: the release was built on 2.100.0 and claims only ==2.100.0. It passed for 2.100.0 on the built_with_model_package basis. The release we certify will be built on the policyengine-uk version this repo ships, so the basis will be exact.
• The metadata checks pass. With the pin moved and uv.lock refreshed, bundle.py check, generate --include-tros and check --include-tros --strict-tros all pass. The certification itself needs no code change.
• 13 of the 234 tests run fail, all on hard-coded enhanced-FRS values (policyengine-uk-data, enhanced_frs_2024_25, 2.102.3, the 1.56.16 URI). The full suite was not run.
  • 3 in test_bundle.py
  • 1 in test_bundle_metadata.py
  • 4 in test_release_manifests.py
  • 1 in test_uk_regions.py
  • 3 in test_dataset_runtime.py
  • 1 in test_models.py
• The enhanced FRS leaves the bundle. Certification rewrites data_releases.uk, which removes every enhanced-FRS entry.

1. Local-authority list

The UK area lists load from the policyengine-uk-data-private GCS bucket with Google credentials (data/uk_geography_assets.py). Without credentials, policyengine-core asks for a service-account key path in an interactive session; otherwise the constituency and local-authority lists come back empty.

• The list's codes are newer than its name. It is local_authorities_2021.csv, but its rows are on the April 2023 LAD frame. Its copy in policyengine-uk-data, identical to policyengine-api's, has 360 codes. Cumberland, Westmorland and Furness, North Yorkshire, Somerset and ten Northern Ireland districts are in it; none of the 17 districts abolished in 2023 are.
• It lacks Ards and North Down (N09000011). Microcosm's frame has all 361. With the aliases from UK local-area exports: add the constituency_code_oa, la_code_oa and region_code_oa columns that policyengine.py and the simulation API read, and record the area-code frames in the release manifest microcosm#1114, every other code matches.
• The enhanced FRS is out of step with it today. The enhanced FRS's household codes are on the pre-2023 frame (363 codes in Great Britain). Runs for the four new unitaries and the ten Northern Ireland districts find no households. Households in the 17 abolished districts belong to no listed authority.

Proposed:

• Ship both lists in the package and read them from package data rather than GCS:
  • local_authorities_2023.csv: 361 rows from the ONS "Local Authority Districts (April 2023) Names and Codes in the United Kingdom" file, plus the x/y hex positions.
  • constituencies_2024.csv: 650 rows including Northern Ireland.
• Add N09000011 with a hex position.
• The bucket copy could not be read without GCS credentials. Confirm it matches the policyengine-uk-data copy before relying on the counts above.

2. Separate national and local-area datasets

The US already works this way. The national file is certified, the local-area file is a named overlay (#472), and #552 routes state and district regions to it. #552 adds regional_dataset_defaults.{country}, which maps a region type to a named dataset. That block survives certification and cannot change the national default; the PR wires US regions only.

• Region hooks: in countries/uk/regions.py, give constituency and local-authority regions dataset_path=resolve_region_dataset_path("uk", "constituency" | "local_authority"). Without a regional default this returns nothing, and the regions keep filtering the national dataset as now.
• Registering the local release: once Microcosm publishes one, add it as dataset_overlays.uk.<name> and set regional_dataset_defaults.uk to {"constituency": "<name>", "local_authority": "<name>"}.
• Until then: with only the national file certified, constituency and local-authority runs raise Variable 'constituency_code_oa' not found in household data. The national file carries region and no area codes.
• Report breakdowns: the constituency and local-authority breakdowns of a national report (compute_uk_constituency_impacts, compute_uk_local_authority_impacts) need household outputs from a run on the local file (decision 2 below).

3. Nation scoping (existing bug, both datasets)

• The bug: country/england, country/scotland, country/wales and country/northern_ireland filter the input household table on country. Neither dataset stores that column: policyengine-uk derives country from region in a formula, and .py filters before the engine runs. On both files the filter raises Variable 'country' not found in household data. The simulation API resolves UK nation regions through this registry.
• Proposed: scope nations by region with a RegionGroupStrategy of region row filters. England is the nine English regions; Scotland, Wales and Northern Ireland are one region each.
• Checked on the published Microcosm file: these groups split all 63,827 households among the four nations. England has 24.26m weighted households, Scotland 2.63m, Wales 1.38m and Northern Ireland 0.76m.
• Optional: add the twelve regions as a region type, which the national file supports.

4. The certification PR

Once the Microcosm national release exists:

• Certify:

  python scripts/bundle.py certify-data --country uk --data-producer populace --manifest-uri "hf://dataset/policyengine/populace-uk-private@<cut-tag>/releases/microcosm-uk-2024-25-national/release_manifest.json" --model-version <build version>

  • Pin the per-cut tag, never main: each cut rewrites the line's release directory.
  • The command moves the policyengine-uk pin to the build version in the same PR: 2.102.3 here, while PyPI is at 2.122.2.
  • Bumping policyengine-uk in a separate PR first would need another compatibility claim on the enhanced-FRS release, which lists ==2.89.2, ==2.90.2 and ==2.102.3.
• Core version: certification does not check built_with_core_package. Build on the policyengine-core pinned here (3.32.10; PyPI 3.32.19), or bump it in the same PR with bundle.py update-packages --core.
• Checks:
  • Re-lock uv.lock.
  • Run generate --include-tros, check --published-spm --include-tros --strict-tros and release_lock.py, then the full suite.
  • Refresh the UK household snapshots for the policyengine-uk bump with PE_UPDATE_SNAPSHOTS=1.
• Tests: update the 13 tests listed above.
• Keep the enhanced FRS loadable by name: add enhanced_frs_2024_25 and frs_2024_25 (1.56.16 pins) under dataset_overlays.uk, next to the existing enhanced_frs_2023_24, frs_2023_24 and populace_uk_2023.
• Docs and examples:
  • Update those that name the enhanced FRS as the UK default: README.md (see README population quickstart still teaches enhanced_frs_2023_24 #473), docs/bundles.md, docs/countries.md, docs/microsim.md, docs/release-bundles.md, examples/income_bands_uk.py, examples/paper_repro_uk.py and examples/policy_change_uk.py.
  • In docs/engineering/skills/data-certification.md, the UK template @<tag>/releases/<tag>/ does not fit Microcosm's layout, @<cut-tag>/releases/<release-id>/.
• Access: CI's HUGGING_FACE_TOKEN and every deployment that loads the default need read access to policyengine/populace-uk-private.

What the Microcosm release provides

• Engine: built on the policyengine-uk version this repo will ship (planned for the next build).
• Area codes: the aliases on local releases (UK local-area exports: add the constituency_code_oa, la_code_oa and region_code_oa columns that policyengine.py and the simulation API read, and record the area-code frames in the release manifest microcosm#1114).
• A cut-specific build.build_id:
  • The release records the line name, microcosm-uk-2024-25-national, as build_id for every cut.
  • This repo stores it as data_releases.uk.version and names the changelog fragment after it, so two cuts look identical here.
  • The cut tag is in build.cut_tag, which the schema here ignores. Either the release sets build_id to the cut tag, or this repo reads cut_tag.

Other repositories

These need their own issues before the API runs a release with a Microcosm UK default:

• policyengine-sim-api:
  • .py's default registry has no constituency or local-authority regions, so the API builds them itself as WeightReplacementStrategy regions over the enhanced-FRS weight matrices (_build_uk_weight_replacement_region in simulation_runtime.py).
  • Those matrices have one column per enhanced-FRS household, so on any other dataset they raise Weight matrix row length … does not match household count.
  • Its geographic reports also read constituency_code_oa and la_code_oa.
• policyengine-api: uk_constituency_breakdown downloads the enhanced-FRS constituency weight matrix.

Decisions needed

1. What serves constituency and local-authority runs between the national switch and the Microcosm local release:
   • nothing (those runs raise);
   • the enhanced FRS through regional_dataset_defaults, which mixes data sources and leaves 14 listed authorities empty;
   • or hold the switch until the local release exists.
2. Breakdowns in national reports: a second run on the local file, or none until one file carries area codes.
3. Whether constituency and local-authority regions join the default UK registry once the lists ship here. If they do, the API stops reaching its weight-matrix fallback and filters rows instead, which changes its constituency and local-authority numbers.
4. Acceptance evidence for the switch:
   • Bump to policyengine-uk 2.90.2 and revert the UK certified default to enhanced FRS 2024-25 #491's floor: UC caseload and spend within 10% of OBR/DWP.
   • Whether Promotion certification suite: pre-registered out-of-sample scorecard at default-flip time #462's pre-registered scorecard is required.

Related: #408, #462, #473, #490, #491, #495, #551, #552.

🤖 Generated with Claude Code