Weight FRS BRMA draws by census private-rented households

[MaxGhenis]

Fixes #515.

What changes

Each FRS benefit unit's Broad Rental Market Area (BRMA) is now drawn within its region in proportion to census private-rented households by BRMA and bedrooms, instead of the row counts of lha_list_of_rents.csv.gz.

• The household then takes one of its benefit units' BRMAs at random, as before.
• LHA categories map to bedroom bands: A and B use one bedroom, C two, D three and E four or more.
• Northern Ireland's census has no bedrooms question, so its weights are the same for every category.

Why. The old file's Scottish, Welsh and NI lists are byte-identical copies of English BRMAs' lists, matched on the April 2019 rate; #515 has the evidence. In the current enhanced FRS, Scotland's private renters are imputed as follows:

BRMA	Imputed share (current)	Census 2022 share
Aberdeen and Shire	25.7%	9.9%
East Dunbartonshire	14.2%	0.8%
Greater Glasgow	10.2%	15.9%
Lothian	2.9%	19.3%

The new table

storage/brma_private_rented_households.csv (936 rows: region, brma, bedrooms, households). Sources, method and validation are in storage/BRMA_DATA_SOURCES.md.

• England and Wales, Census 2021.
  • TS054 private-rented households per LSOA, split by each LSOA's bedroom mix for "private rented or lives rent free". That is the only tenure × bedrooms split ONS releases for every LSOA, and rent-free households are 0.3-1.2% of it.
  • LSOAs are assigned by population-weighted centroid in the VOA (May 2020) and Rent Officers Wales BRMA boundaries.
• Scotland, Census 2022. Tenure × bedrooms by ward (the finest unsuppressed level), split by output-area centroids in the Scottish Government's BRMA polygons, with corrections from Rent Service Scotland's postcode lookup.
• Northern Ireland, Census 2021. NISRA households by postcode district, mapped to NIHE's postcode-district BRMAs and scaled by NI's private-rented share. Splitting NI's private renting by area would need ONS Postcode Directory NI records, whose LPS licence doesn't clearly allow publishing derived figures. Using all-tenure households instead moves NI BRMA shares by 1.5 points on average.
• BRMAs that cross a region boundary get one row per region. 37 BRMAs have households in more than one region; a household is drawn from the BRMA parts in its own region.
• Reconciliation. National totals are within 0.03% of the published private-rented counts.
• Rebuild. tools/brma_households/ downloads every input, checks it against a pinned sha256 and rebuilds the CSV byte for byte. The one exception is the Scotland's Census table-builder export, which needs a browser session and has step-by-step instructions.

Validation. Within each region, I compared each weight's BRMA shares with DWP's Universal Credit households assessed under LHA, averaged over April 2019 to November 2020 (UC statistics supplementary table 3.2):

Pearson r	England	Scotland	Wales
Census private renters (new)	0.94	0.82	0.95
List-of-rents row counts (old)	0.82	0.21	0.23

Against the genuine lists, the census bedroom bands correlate with each LHA category's share as follows:

• England (VOA): 0.82-0.94 for categories B-E.
• Scotland (Scottish Government FOI 202200303624): 0.95-0.98.
• Category A (shared rooms) has no census counterpart: 0.51 in England and 0.95 in Scotland, using one-bedroom homes.

Invariants

These hold for every weights table and every set of benefit units:

1. Support. A benefit unit's BRMA has a positive weight in its region × LHA category cell.
2. Determinism. The same generator state gives the same BRMAs.
3. Fail closed. A benefit unit in a cell with no positive weight raises ValueError; it never gets None.
4. Proportionality. Draws converge on the cell's weights. 400,000 Scottish category C draws stay within 0.4 percentage points of every BRMA's weight.
5. Table. It covers all 200 BRMAName members and all 12 regions. Every non-NI region has bands 1, 2, 3 and 4+, and NI has only all. Each nation's total is within 0.1% of its published census total.
6. Built dataset. Every household's (region, BRMA) pair has a positive weight.

Invariants 1-3 are Hypothesis property tests (200 and 100 derandomised examples), and 4-6 are example tests. hypothesis is added to the dev extra.

Differential. microcosm's frs_brma stage reads the same table, pinned by sha256 in its generator (PolicyEngine/microcosm PR to follow). The two implementations draw differently (seeded generator here, identity-keyed uniforms there) but from the same weights.

Impact

Every figure below is for 2026, from policyengine-uk runs on seeded enhanced FRS 2024-25 builds, using the published LHA rates from policyengine-uk#2022 (head aef9c2427). Running on policyengine-uk main instead gives smaller Scottish changes, because main's Scottish rates come from the copied lists. Scripts and outputs are in impact/ of the working folder.

1. Expected effect of the new weights, with dataset weights held fixed. On the baseline build, BRMAs were re-drawn 10 times with the census weights and 10 times with the list-of-rents weights. The effect is the difference in means, with its standard error. "One draw, SD" is the spread between draws with the old weights.

	Scotland	Wales	NI	England	UK	One draw, SD (UK)
Mean weekly LHA, LHA-eligible benefit units	+£17.29 (3.97)	+£4.42 (1.52)	−£3.60 (1.08)	+£1.40 (1.20)	+£2.36 (1.12)	£2.76
Universal Credit paid	+£43m (11m)	+£3m (7m)	−£6m (3m)	+£52m (73m)	+£92m (73m)	£161m
Housing Benefit	†	†	£0m	+£72m (66m)	+£71m (66m)	£179m
People in AHC poverty	†	†	†	−9k (25k)	−20k (25k)	46k

† Suppressed, because fewer than 10 records changed in at least one draw.

Scotland's private renters move to where Scotland's private renters actually live, so they now face Lothian's and Glasgow's higher LHA rates. Averaged over draws, Scottish private renters' imputed BRMA shares now match the census: Lothian 20.6% (census 19.3%) and Greater Glasgow 15.4% (15.9%), against 12.3% and 3.1% under the old weights. England's effect is within its noise.

2. Full rebuild. I built this branch and main with the same seed; two builds of main are output-identical. The branch build is commit e0d4b24, from before the Northern Ireland rows switched to all-tenure households; NI is 2.4% of UK private renters. Under published rates, the branch moves:

	UK	Scotland	England
UC	−£268m	+£420m	−£687m
Housing Benefit	+£151m
People in AHC poverty	+160k

3. The rebuild difference is mostly the draw, not the weights. I also rebuilt main with only the BRMA seed changed and the old weights kept. It moves:

	UK	Scotland	England
UC	−£321m	+£216m	−£513m
Housing Benefit	−£71m
People in AHC poverty	+150k

With one clone per household, recalibration amplifies a single BRMA draw into aggregate changes of this size. A single-build comparison therefore cannot separate this PR's effect from that noise, and (1) is the estimate to use. Reducing this draw noise is a separate follow-up.

Review

An independent Opus review (round 1) requested changes, and all of them are addressed:

• NI licensing: the ONS Postcode Directory is removed.
• Household pick: now pick_household_brmas, with a property test.
• Bedroom bands: a test pins each category's band, with spot values.
• Property tests: strategies give each region's BRMAs distinct names, and the fail-closed test always removes a cell.
• Enum regions: decoded before the draw.
• Bedroom headers: Scotland's are asserted exactly.
• Docs: Cardiff Bay and Balloch overrides, centroid wording, and what the validation correlates.

Round 2 approved after rerunning the tests and mutation checks. It also re-verified the copied-list claim from the raw file: 480 of 480 blocks match an English block from the same year. Its one remaining note, a stale NI description in sources.yaml, is fixed. A round-3 delta review approved head 29e1bd7: the history audit, ignore protection, manifest check and tests all passed (24 passed, 1 skipped).

Merge timing: per Max's ruling d833, uk-data fixes land together as one data release on his go. This PR is approved and ready, and should merge with that batch alongside PolicyEngine/microcosm#1087.

Other notes

• lha_list_of_rents.csv.gz is removed; nothing else in this repo reads it.
• policyengine-uk still ships its own copy. #2022 there replaces the rates computed from it with published rates, and I've raised the copied Scottish, Welsh and NI lists with that PR's author.
• Every input is under the Open Government Licence except NIHE's BRMA postcode-district list (no stated licence; only its district lists are used). No ONS Postcode Directory records are used.

Commands run

• pytest policyengine_uk_data/tests/test_brma_assignment.py test_legacy_benefit_proxies.py: 25 passed, including the built-dataset test. CI's previous run passed all 9 BRMA tests on its own built dataset.
• tools/brma_households/build.py against a cache of the retained originals verifies all 87 inputs and rebuilds the CSV byte for byte (sha256 fd40dae0…).
• ruff format and ruff check on changed files: clean.

🤖 Generated with Claude Code