Draw private renters' BRMAs given their rent and bedrooms

[MaxGhenis]

Stacked on #516 (base brma-private-rented-weights); retarget to main after #516 merges.

What changes

#516 draws each FRS household's Broad Rental Market Area (BRMA) within its region in proportion to census private-rented households. The draw ignores the household's rent, so a high-rent household is as likely as a low-rent one to land in its region's cheapest BRMA. That breaks the link between a renter's rent and its Local Housing Allowance (LHA) rate, which decides whether rent is above the cap and how much UC and HB pay towards it.

Private renters who report a rent (ptentyp2 3 or 4, hhrent > 0) are now redrawn:

P(BRMA | region, bedrooms, rent) ∝ census private-rented households(BRMA, bedroom band) × density of the rent on the BRMA's list of rents for homes of that size.

• Bedrooms describe the home, not the entitlement. The census bands and the list categories B-E both count the home's bedrooms, so both use FRS bedroom6. The LHA category describes entitlement: an under-35 single person renting a one-bedroom flat is category A, but pays a one-bedroom rent.
• Reported rents are not list rents. The model behind the density is fitted by maximum likelihood to the survey's own private renters at build time (fit_reported_rent_model). On FRS 2024-25:
  • reported rents sit 4% to 20% below the list median, with a separate shift per region;
  • 84% of households follow the list's spread plus 0.13 log points of noise;
  • 16% pay far less, 0.64 log points lower with wide noise. Their rent says little about where they live, so they keep roughly the census shares.
• Everyone else keeps the Weight FRS BRMA draws by census private-rented households #516 draw, byte for byte: the redraw runs after it and overrides only the households above, with the same generator.

Fitting at build time takes a few seconds.

The rents table

storage/brma_private_rents.csv: 1,000 rows (200 BRMAs × categories A-E) giving the median weekly rent and the standard deviation of log rents, in 2024-25 prices. tools/brma_rents/ downloads every source, checks it against a pinned sha256 and rebuilds the file. All sources are public (OGL); no survey data goes into the table.

Nation	Median	Spread
England	VOA lists of rents for April 2025 and April 2026, pooled (rents collected October 2023 to September 2025)	Same lists: interquartile range of log rents ÷ 1.349
Wales	Rent Officers Wales's lists for April 2024 and April 2025, pooled (Welsh Government FOI ATISN 25142)	Same lists
Scotland	Published 30th percentiles, April 2025 and 2026, raised to a median assuming log-normal rents	Rent Service Scotland's lists, years to September 2019-21 (FOI 202200303624)
Northern Ireland	As Scotland, from April 2024, the latest 30th percentiles in policyengine-uk's published-rates table (#2022)	Median spread of the English, Welsh and Scottish cells of the same category

Every list and percentile is uprated to 2024-25 with the ONS Price Index of Private Rents for its region or nation. Each region has its own fitted shift, which absorbs a region-wide error in that uprating; what drives the draw is a BRMA's rents relative to the rest of its region.

Wales. The Welsh lists reproduce the published April 2024 and April 2025 30th percentiles: median difference 0, with 94% and 98% of cells within 1%.

Northern Ireland publishes no lists, so it uses the Scottish construction with a borrowed spread. To test that treatment, I applied it to the three nations whose lists exist. An FRS renter's BRMA probabilities move by 4.5 points of total variation in England, 7.4 in Wales and 6.6 in Scotland. Conditioning on rent at all moves them by 20.5, 21.3 and 32.8. So the percentile treatment keeps most of the information.

Invariants

These hold for every census table, rents table, model and set of households with data for their cells. Hypothesis property tests are in tests/test_brma_rent_conditioning.py:

1. Support. Each household's probabilities sum to 1 and are zero outside its region × bedroom-band cell; a draw never leaves that support.
2. Determinism. The same generator state gives the same BRMAs.
3. Fail closed. ValueError is raised for a non-positive rent, a home with no bedrooms, a missing region × band cell, a BRMA in the cell with no list of rents, or a region with no fitted shift. A household is never left without a BRMA.
4. Level invariance. Scaling a region's rents, or its BRMAs' list medians, by a constant while moving the region's shift to match leaves the probabilities unchanged.
5. No information, no change. If a cell's BRMAs share one list, the probabilities are the census shares.
6. Reference agreement. Results match a scalar re-implementation of the formula (differential test).
7. Dearer rents, dearer BRMAs. Without the below-market component, the probability of the dearer of two otherwise equal BRMAs never falls as rent rises. With it, very low rents revert towards the census shares; this is intended.
8. Calibration. Households simulated from the census and the lists have probabilities that average to where they really live, within 6 points of total variation per region. The fit recovers the true model parameters, and the drawn BRMAs' rent levels track rents as the true ones do.

Three example tests run on a built FRS dataset:

• Census shares within each region. The built private renters' weighted mean BRMA probabilities are within 5 points of total variation of the census shares for their homes' bedrooms, in every region.
• Draws follow the probabilities. The log score of the drawn BRMAs, standardised with its exact mean and variance for independent draws, is within ±4.
• Rent tracks BRMA. Within region and bedroom band, log rent and the drawn BRMA's log list median correlate above 0.15.

Results on three builds:

Build	Census shares	Log score z	Rent-BRMA correlation
This branch: CI-equivalent TESTING=1 build of 6184534	pass	−0.00	pass
A #516 build with this PR's redraw applied (2,079 of 2,288 renters change BRMA)	pass	+0.29	pass
A #516 build (census-only draw)	pass	−19.3 (fail)	0.007 (fail)

Validation

• Out of sample. I took 150,000 April 2026 list entries (categories B-E) as households with a known BRMA. The prior was each region's April 2026 entry counts; medians and spreads came from the April 2025 list only.

  • The mean probability given to the true BRMA rises from 0.085 (census shares) to 0.129.

  • The top choice is right for 21.8% of entries, against 16.1%.

  • Stated probabilities match observed frequencies:

    Stated probability (bin)	Mean stated	Observed frequency
    20-30%	24.2%	23.8%
    50-60%	54.2%	53.4%
    90%+	93.3%	91.3%

• Scotland's medians. The FOI lists for the year to September 2021, uprated with each Scottish BRMA's own ONS index, correlate at 0.98 with the medians used here (mean gap 1.8%, standard deviation 8.4%). The ONS indices grew by 10% to 36% across Scottish BRMAs over that time. That is why current 30th percentiles are used rather than uprating old lists.
• Census shares on raw FRS. Within every region, FRS 2024-25 private renters' weighted mean probabilities are within 3 points of total variation of the census shares.

Impact

Method. Each figure is the exact expected change over the BRMA draw, with no draw noise and dataset weights held fixed. The calculation:

• Every household in a seeded enhanced-FRS build of Weight FRS BRMA draws by census private-rented households #516's head was simulated under every BRMA of its region (up to 38).
• Those outcomes were then weighted by this PR's probabilities and by Weight FRS BRMA draws by census private-rented households #516's.
• As a check, reading each household's outcomes at a drawn BRMA reproduces a full simulation of that draw to the penny.
• The recalibration a rebuild would add is not included. Weight FRS BRMA draws by census private-rented households #516 showed that recalibration amplifies any single draw into noise of about £0.3bn in UC.

Results are shown for policyengine-uk main and for #2022 head 23a31d5, which has the published LHA rates. Main's Scottish, Welsh and NI rates come from the copied lists (#515), so #2022 is the one to read.

#2022 head, UK: change from the census-only draw

Year	UC	UC housing element of UC recipients	Housing Benefit	Household net income	People in AHC poverty	Mean weekly LHA, eligible renters	UC recipients with rent above LHA
2024	+£0m	−£81m	−£39m	−£39m	−8.7k	+£1.26	+7.6 pts
2025	−£83m	−£158m	−£62m	−£146m	−8.5k	+£1.26	+7.9 pts
2026	−£100m	−£210m	−£73m	−£173m	−3.7k	+£1.26	+8.2 pts

#2022 head, 2026, by nation

	England	Scotland	Wales	NI
UC housing element of UC recipients	−£233m	+£31m	−£16m	+£7m
Housing Benefit	−£72m	+£1m	−£2m	£0m
Mean weekly LHA, LHA-eligible renters	−£0.36	+£23.82	+£0.99	−£0.67
LHA-eligible renters with rent above LHA	+5.3 pts	+8.4 pts	+5.2 pts	+5.3 pts
UC recipients with rent above LHA	+8.0 pts	+11.7 pts	+9.2 pts	+6.0 pts

Mean weekly LHA changes by the same amount in every year because #2022's rates are the same in 2024, 2025 and 2026: the April 2024 determination, then frozen.

policyengine-uk main, UK

Year	UC	UC housing element of UC recipients	Housing Benefit	Household net income	People in AHC poverty	Mean weekly LHA	UC recipients with rent above LHA
2024	+£43m	−£47m	−£32m	+£12m	−17.5k	+£0.55	+6.2 pts
2025	−£28m	−£114m	−£51m	−£79m	−14.2k	+£0.55	+6.1 pts
2026	−£43m	−£159m	−£59m	−£102m	−11.1k	+£0.55	+6.1 pts

What moves.

• In Scotland, high-rent renters now land in Lothian and Greater Glasgow, so their LHA rates rise (+£23.82 a week on average under published rates).
• Elsewhere the mean rate barely moves, but each renter's rate now tracks its rent. Within LHA-eligible renters, the correlation between log rent and log LHA rate rises from 0.50 to 0.64 (#2022, 2024).
• The mean of log(rent ÷ LHA) hardly changes (0.247 to 0.240), but its spread falls from 0.49 to 0.42.
• Because most renters' rent is above their rate, a tighter spread puts more of them above it. More rents are then capped at the LHA rate, so overall UC and HB fall slightly; Scotland's rise.

Against DWP. CIH UK Housing Review 2026 table 111b (sourced to DWP Stat-Xplore) gives the share of private-rented UC households with a housing element whose LHA does not cover their rent, each May. The model counterpart is UC recipients with a housing element whose rent exceeds their UC LHA rate. The definitions differ: FRS-reported rent against DWP's verified rent, and benefit units against claim households.

Area	DWP, May 2024	Census-only draw	This PR	DWP, May 2025	Census-only draw	This PR
England	45.5%	59.8%	67.3%	52.1%	65.7%	73.4%
Wales	57.8%	73.7%	82.6%	63.0%	78.6%	87.6%
Scotland	34.1%	57.3%	68.2%	39.9%	58.5%	70.0%

(#2022 head; model years 2024 and 2025.)

Across the 11 areas CIH publishes (9 English regions, Wales and Scotland):

	2024, census-only	2024, this PR	2025, census-only	2025, this PR
Mean gap to DWP	+14.2 pts	+22.9 pts	+13.4 pts	+22.3 pts
Correlation with DWP across areas	0.62	0.68	0.63	0.72
Root mean square error after removing the mean gap	6.7 pts	5.5 pts	7.3 pts	5.3 pts

So the model already overstates rent above LHA by about 14 points with the census-only draw. This PR gets the pattern across areas closer to DWP's but widens the level gap. The level gap predates this PR, and its cause is not established here. Candidates to check:

• whether FRS rents of UC recipients exceed their verified rents;
• the LHA category assignment;
• benefit units against claim households.

It is a separate investigation. On main, whose Scottish and Welsh rates are wrong, the correlation is 0.32-0.39 under either draw.

Draw noise. Rent conditioning lowers the noise of a single BRMA draw only a little. Single-draw standard deviations, exact, #2022 head, 2026:

	Census-only draw	This PR
UC	£201m	£177m
Housing Benefit	£261m	£239m
UC housing element of UC recipients	£398m	£314m
People in AHC poverty	56.5k	57.0k

A handful of heavily weighted households carry most of the variance. Lower-variance sampling is separate work and composes with this PR: its samplers take these probabilities in place of the census shares.

Not in this PR

• FRS bedrooms are read in the BRMA step only. num_bedrooms is not saved to the dataset, because the wealth imputation lists it as a predictor and, with the column absent, receives the default of zero for every household. That is a separate fix.
• Shared accommodation (category A) is never used to place a household, because the FRS does not identify rooms in shared houses.
• Why the model overstates rent above LHA compared with DWP, before this PR (see Impact).

Merge

uk-data fixes are approved on gates but land together in one data release on Max's go. This PR stays a draft until #516 merges.

It also needs a methodology call. The draw is more faithful: its probabilities are calibrated out of sample, census shares are kept, and the pattern across areas moves closer to DWP's. But it widens the model's existing overstatement of rent above LHA, by about 8 points UK-wide.

Commands run

• A CI-equivalent build of 6184534 (TESTING=1 python policyengine_uk_data/datasets/create_datasets.py), then pytest on test_brma_rent_conditioning.py, test_brma_assignment.py, test_frs_survey_year.py and test_legacy_benefit_proxies.py against it: 47 passed. (Stacked PRs get only the deploy check in CI, so the dataset build and tests were run locally.)
• python tools/brma_rents/build.py <cache> --check verifies all 6 sources by sha256 and reproduces the committed table.
• ruff format --check and ruff check on changed files: clean.

Aggregates only: no survey records appear in this PR or its tables.

🤖 Generated with Claude Code

[MaxGhenis]

PolicyEngine/policyengine-uk#2022 (published LHA rates) is merged into policyengine-uk main at c004e5318 (PR head 088addbb3).

What this means for this PR's impact section, which reads #2022 at 23a31d5:

• The published-rates table is unchanged. lha_published_rates.csv.gz is identical between 23a31d5 and the merged head, so the Northern Ireland 30th percentiles cited above stand.
• Re-run the "#2022 head" results on policyengine-uk main at c004e5318 or later. Main now has the published rates, so the main-vs-#2022 split can go. Between 23a31d5 and the merge, policyengine-uk took three merges of main. Among them is #2006, which gives Housing Benefit its own category of dwelling: housing_benefit_LHA_rate now reads the published table through housing_benefit_LHA_category, and BRMA_LHA_rate reads it through the UC category. HB amounts on main can therefore differ from the 23a31d5 runs even where rates per category do not.