Weight UK BRMA assignment by census private-rented households

[MaxGhenis]

What changes

The UK frs_brma stage drew each benefit unit's Broad Rental Market Area in proportion to the row counts of the 2019-20 LHA list of rents (brma_rent_counts.json). That list's Scottish, Welsh and Northern Ireland lists are byte-identical copies of English BRMAs' lists, each matched to the English list whose 2019 30th percentile is nearest the nation's April 2019 rate, so those nations' draws ignored their rental markets. PolicyEngine/policyengine-uk-data#515 has the evidence. For example, Greater Glasgow had 2% of the Scottish entries; the genuine Scottish list gives it 18%, and Scotland's Census 2022 16% of the nation's private renters.

This PR replaces the resource with brma_private_rented_households.json: census private-rented households by region and BRMA, using the bedroom band that matches each LHA category.

• Category mapping: A and B use one bedroom, C two, D three, E four or more.
• Sources: England and Wales Census 2021, Scotland's Census 2022 and Northern Ireland Census 2021.
• Northern Ireland has no bedrooms question, so every category gets the same NI weights. They come from NISRA households by postcode district, mapped to NIHE's BRMAs and scaled by NI's private-rented share. The private-rented split by area would need ONS Postcode Directory NI records, whose LPS licence doesn't clearly allow publishing derived figures.
• The JSON keeps the existing cells[region][LHA_category][brma] shape, so frs_brma.py's sampler is unchanged.
• tools/build_uk_brma_household_table.py replaces build_uk_brma_count_table.py. It pins the households CSV by sha256 (fd40dae0…) and row count (936). Weight FRS BRMA draws by census private-rented households policyengine-uk-data#516 adds the same CSV, with sources, method, validation and a byte-for-byte rebuild pipeline.

The stage's source text and every resource reference (country package, source stages, spec, tests, graph parity fixture) are renamed to match.

Validation of the weights

The check is the within-region correlation of BRMA shares with DWP's Universal Credit households assessed under LHA, averaged over April 2019 to November 2020:

	England	Scotland	Wales
Census private renters (new)	0.94	0.82	0.95
List-of-rents counts (old)	0.82	0.21	0.23

Invariants

Tested in test_uk_brma_resource.py:

1. The resource is the generator's output for the pinned CSV. This regeneration test runs wherever the CSV is staged; I ran it locally and it passed.
2. There are 12 regions × 5 categories = 60 cells and 200 BRMAs. Every count is positive, and every cell totals at least 18,000 households.
3. The cells are pinned in CI by a digest of their canonical JSON and by each nation's census total: England 4,795,158, Wales 228,601, Scotland 323,001, Northern Ireland 132,449.
4. Categories A and B have identical cells (both use one-bedroom homes).
5. Northern Ireland's five category cells are identical and hold all 8 BRMAs.
6. Lothian and Greater Glasgow are Scotland's two largest BRMAs (Census 2022: 19.3% and 15.9%). This guards against the copied counts, which put Greater Glasgow last.
7. A benefit unit whose region × category cell is missing makes assign_brma_by_cell raise KeyError.

sample_categorical_from_counts and the identity-keyed draws are unchanged.

Not in this PR

• efrs_parity_reference.json compares against an incumbent build. It needs regenerating once the incumbent dataset is rebuilt with the same weights, which happens outside PR CI.
• The BRMA→local-authority bridge that cross_grain_private_rent_scotland_brma is waiting for could come from the same Scottish output-area crosswalk. That is a separate change.

Review

An independent Opus review (round 1) requested changes, and all of them are addressed:

• NI licensing: the NI weights now come only from NISRA and NIHE sources.
• Cells pinned in CI: a digest of the cells and each nation's total.
• Fail-closed test: now exercises the code.
• Generator: one default staged-CSV path, and the error message names the band actually used.

The UK spine graph is re-pinned for the new resource. Round 2 approved after rerunning the resource tests. A one-count mutation of the JSON fails the CI digest test.

Commands run

• pytest on engine_free/uk/test_uk_brma_resource.py, test_uk_take_up_contract.py, test_uk_frs_spine.py, engine_free/shared/test_country_spec.py, test_spec_engine_country_bundles.py and test_spec_engine_identity_contracts.py: all passed. The regeneration test ran with UK_BRMA_HOUSEHOLDS_CSV set. engine/uk/test_uk_release_input_coverage.py and microcosm-graph engine/uk/test_acceptance_h_parity.py pass after regenerating the coverage manifest and the pinned spine graph.
• ruff check and ruff format: clean.
• python3 tools/ci_test_plan.py verify: ok.

Update 2026-10-05: main merged, parity with uk-data#516, review

• Merged origin/main (fcbda41, no rebase), then formatting and a changelog fragment (766df94). Two files had conflicts: release_input_coverage_manifest.json and the uk_spine.json parity fixture. Both keep main's content; only the source-manifest hashes and the frs_brma contract hash change, and the review rebuilt the latter independently. All 15 of this PR's changes survive, and every file that changed only on main keeps main's contents and modes.
• Parity with policyengine-uk-data#516 at 29e1bd7a: exact.
  • The same 200 BRMAs, 12 regions and 240 region–BRMA pairs.
  • All 1,200 region/category/BRMA counts match, with no missing or extra keys.
  • The same category mapping: A and B on one bedroom, C on two, D on three, E on four or more.
  • The same NI treatment.
  • The same source SHA-256 (fd40dae0…).
• Review: GPT-6.1 Sol gave APPROVE WITH NITS at 766df94. Its one P3 nit, a stale docstring in tools/emit_uk_brma_distribution.py, is fixed in d23bd3e; that docstring is the only change.
  • Targeted tests: 152 passed and 1 skipped. Another 137 errored at setup, and every one of those errors was the review sandbox having no temporary directory; no test body failed. CI covers them.
• Merge timing (decision d830): this merges right after the uk-data batch release that carries QRF donor contains aggregate-scale PUF rows (single-record home_mortgage_interest up to $236B); support_clip donor_realized is vacuous against them #516, on its own gates.

🤖 Generated with Claude Code