Remove the remaining Enhanced FRS and policyengine-uk-data references; pin and test on Microcosm UK #2682

PolicyEngine UK no longer uses the Enhanced FRS (EFRS) or policyengine-uk-data. The current UK dataset is Microcosm UK (microcosm_uk_2024_25.h5, in the Hugging Face dataset repository policyengine/populace-uk-private). This repository still pins, documents and tests against the EFRS. As a result, contributors and coding agents who follow the testing guidance treat the EFRS as the dataset to use.

References below are to main at fba0c0a53.

What still points at the EFRS or policyengine-uk-data

Test dataset pin and the guidance that names it

• policyengine_uk/tests/certified_dataset.yaml pins enhanced_frs_2024_25.h5:
  • data_package: policyengine-uk-data
  • data_version: 1.58.1
  • repo_id: policyengine/policyengine-uk-data-private
  • revision 2526edbb
• docs/engineering/skills/testing.md, "Microsimulation Data", says microsimulation tests run on the dataset pinned in that file.

Agent and contributor guidance

• docs/engineering/skills/dataset-sources.md:237 (uk-data#448) and :268 (an audit run on enhanced_frs_2024_25).
• .github/CONTRIBUTING.md:65-66, which defers dataset semantics to policyengine-uk-data.
• AGENTS.md doesn't say which dataset is current.

Tests and fixtures

• policyengine_uk/tests/microsimulation/reforms_config.yaml: the dataset: block and the header describe the EFRS. Keep the existing receipts as history.
• policyengine_uk/utils/aggregate_impact/standard_reforms.yaml: the reform responsiveness minimums were calibrated on EFRS 1.58.1.
• .github/test_outcome_allowlist.toml:78-80 and policyengine_uk/tests/microsimulation/test_annual_allowance_against_hmrc.py: a strict xfail tied to policyengine-uk-data#606, the EFRS employer-contribution imputation. It may pass on Microcosm UK, which strict mode reports as a failure.
• policyengine_uk/tests/test_latest_data_smoke.py loads an unpinned EFRS file.
• EFRS or policyengine-uk-data example values:
  • test_aggregate_impact_compare.py
  • test_aggregate_impact_certified.py
  • test_simulation_dataset_sources.py
  • test_deterministic_variables.py
  • test_uprating_inputs_reach_data.py
• Comments in:
  • test_household_income_decile.py
  • test_projection_horizon.py
  • test_current_education_order.py
  • test_child_and_adult_definitions_properties.py
  • four policy YAML tests
• test_build_hook.py and test_build_metadata.py use a directory named policyengine-uk-data as an example outer repository. Renaming it is cosmetic.

Loading code

• hf:// references are downloaded as Hugging Face model repositories, through policyengine-core's download path. Microcosm UK is published in a dataset repository, so two things need dataset-repository support:
  • Simulation URL loading;
  • the certified-dataset download in policyengine_uk/utils/aggregate_impact/dataset.py.
• policyengine_uk/utils/aggregate_impact/compare.py exempts a locally installed policyengine-uk-data. Its README also names the EFRS.

User documentation

• EFRS hf:// and gs:// examples appear in:
  • docs/book/index.md
  • docs/book/usage/getting-started.md
  • docs/book/usage/scenarios.md
  • docs/book/usage/simulations.md
• docs/book/assumptions/mp30k-uk-switch-plan.md and reform-impacts-dashboard-plan.md describe the EFRS pin as current.

Variable and parameter metadata

• 23 files under policyengine_uk/variables, parameters and reforms describe how the EFRS built an input or cite policyengine-uk-data issues. Each needs rewording against Microcosm UK's build of the same column.

Records of past analysis (keep)

• Docs and comments that report measured EFRS results should stay as dated records, rewording only text that presents the EFRS as current:
  • docs/engineering/vat-coverage.md
  • docs/engineering/pension-annual-allowance.md
  • policyengine_uk/data/economic_assumptions.py:688
  • policyengine_uk/dynamics/participation.py:500
  • several book pages

Decisions and access needed first

1. Which certified Microcosm UK national build to pin.
   • microcosm-uk-2024-25-national-20261019T083323Z-6cba5d71 is in the policyengine.py bundle manifest from 6.9.0 to 6.11.0.
   • microcosm-uk-2024-25-national-20261025T091414Z-a6299504 is being certified in Certify the Microcosm UK pair on policyengine-uk 2.151.0 (a6299504 national, c9df4b86 local area) policyengine.py#720.
2. CI read access. The HUGGING_FACE_TOKEN Actions secret, which test-suite.yaml and aggregate-impact.yaml use, must be able to read policyengine/populace-uk-private. This is unverified. Read the CI test dataset through a short-lived OIDC token, not the org-wide Hugging Face secret #2409 would replace that secret with a token described as unable to read private repositories, so the two changes need to be reconciled.
3. Recompute the reform fixtures on the new pin, as testing.md requires: add new values under receipts, and recalibrate the responsiveness minimums.

Done when

• certified_dataset.yaml pins a certified Microcosm UK build by commit, SHA-256 and size.
• The loader downloads from the dataset repository.
• AGENTS.md, CONTRIBUTING.md, dataset-sources.md and testing.md say Microcosm UK is the dataset and the EFRS and policyengine-uk-data are not used.
• The reform fixtures, responsiveness minimums and allowlist entries are recomputed or re-checked on the new pin. The data tests pass with POLICYENGINE_UK_REQUIRE_TEST_DATA=1.
• User documentation examples load Microcosm UK.
• Remaining EFRS references are either removed or kept as dated records of past analysis.

Related: #1692 (a switch plan that keeps a way back to the EFRS; this issue removes that path).