Measure Scotland's BRMA × council overlap; keep BRMA rents out of council targets

[MaxGhenis]

Records the decision in #1090 and ships the table it rests on. #1090 stays open for the two routes below.

The question

ONS's Price Index of Private Rents (PIPR) publishes Scottish rents for the 18 Broad Rental Market Areas, not the 32 council areas, so all 32 Scottish ons.rent.private_rent cells are signed deferred. The parity concern cross_grain_private_rent_scotland_brma said to keep them deferred "until a source-backed BRMA-to-LA bridge with an explicit aggregation rule exists". A census crosswalk now exists (policyengine-uk-data#516, #1087). Can a rule such as "household-weighted mean of BRMA rents per council" lift the deferral?

No. The table shows why, and the cells stay deferred.

What changes

• uk/scotland_brma_la_overlap.json: households in each of the 63 BRMA × council intersections.
  • private_rented_households comes from Census 2022 tenure by ward (the finest level with this split), with a ward that crosses a BRMA boundary split by its share of output-area households. 36 of 355 wards are split; they hold 22,821 of the 322,995 private-rented households.
  • all_households sums output-area household counts directly, with no within-ward assumption.
  • Output areas are placed in BRMAs by population-weighted centroid in the Scottish Government's BRMA polygons, checked against Rent Service Scotland's postcode lookup (FOI 202300368850).
• tools/build_uk_scotland_brma_la_overlap.py rebuilds the resource byte for byte from five sha-pinned official originals. It needs geopandas, so it runs under uvx, not the workspace.
• uk_runtime/scotland_brma_la_overlap.py loads and validates the table and answers the structural questions: shares on each side, connected blocks, coincident councils.
• The concern and the deferral are restated with the measured overlap. The concern stays routed, now to Scottish private-rent targets: PIPR is BRMA-grain, so council cells cannot bind by translation #1090. No target value changes; the 32 cells were deferred before and are deferred after.
• docs/uk-scotland-private-rent-brma-1090.md is the decision note.

Why the weighted mean is not a translation

• It is the allocation the fence forbids. 14 councils sit wholly inside one BRMA, so the rule returns that BRMA's average for them unchanged; 11 of those share their BRMA with another such council and would get identical rents.
• The geographies do not nest. 18 councils span more than one BRMA, 16 BRMAs span more than one council, and 29 councils and 17 BRMAs form one connected block.
• The erased differences are large. In Rent Service Scotland's market evidence for the year to September 2021 (FOI 202200303624, 31,833 lettings), the rule's value is up to 30.9% from a council's own mean rent: Midlothian +30.9%, Clackmannanshire +26.8%, East Lothian +23.1%, Falkirk +20.5%, Stirling −15.2%. Of 31 councils with at least 30 records, 9 are over 10% out. Records are placed in councils by postcode district, so this measures the size of the differences, not council rents.
• CrossGrainBridge cannot carry it. It equates additive totals over a partition of legs. A BRMA rent is a mean, BRMA is not a grain of UK_CROSS_GRAIN_RULE, and leg_of_area gives each council one leg.

What would lift the deferral (#1090)

• BRMA target rows: keep PIPR's geography on the target side and aggregate the model side with the overlap shares. This needs a third area type on the local surface.
• A signed exception for four councils that coincide with one BRMA at 99% of private renters both ways (Dumfries and Galloway, Fife, West Lothian, Scottish Borders). The rule is within 0.1% for them in the market evidence, but binding them lets a BRMA-frame fact into a local-authority cell, which the boundary-frame gate refuses by design. That is a maintainer's ruling, so this PR does not take it.

Invariants

Stated in test_uk_scotland_brma_la_overlap.py:

1. Coverage. The cells cover all 18 BRMAs and exactly the crosswalk's 32 Scottish local authorities.
2. Conservation. Private-rented cells sum to the census ward total and all-household cells to the output-area total; shares sum to one per council and per BRMA.
3. Bounds. No cell or council has more private renters than households.
4. Differential. The two weights occupy the same cells, give the same blocks and put every council in the same main BRMA.
5. Allocation (property-based). For any wards, councils and output-area assignment, splitting ward counts across BRMAs conserves each council's total and the grand total, and gives each BRMA exactly its wards' renters when no ward is split.
6. Blocks (property-based). At any sliver threshold the blocks partition both rosters, and a higher threshold only refines them.
7. The rejected rule (property-based). For any BRMA rents, the weighted mean conserves the rent bill, stays within the range of BRMA rents, and gives a council wholly inside one BRMA exactly that BRMA's rent.

The cells are pinned by digest. A rebuild test runs wherever the originals are staged (UK_SCOTLAND_BRMA_LA_CACHE). A tripwire test fails if a BRMA grain, area type or rent bridge is added, so the concern is revisited then.

Limits

• The deferral rationale in local_target_reference_membership.json was edited in place, because the licensed Chronicle feed was not staged to rerun the generator. A test asserts the committed text equals the generator's, and only that text differs.
• One centroid places a whole output area. The polygons date from 2015, the postcode lookup from 2023 and the census from 2022.

Commands run

• uvx --with geopandas --with pandas --with openpyxl --with pyogrio python tools/build_uk_scotland_brma_la_overlap.py --cache <originals> --check: current. The cells also match an earlier, separately written build of the same sources (all-household cells identical, private-rented within rounding).
• pytest on engine_free/uk/test_uk_scotland_brma_la_overlap.py (with UK_SCOTLAND_BRMA_LA_CACHE set, so the rebuild test ran), test_uk_pipr_deferrals.py, test_uk_data_target_parity.py, and engine_free/shared/test_country_spec.py, test_spec_engine_country_bundles.py, test_spec_engine_identity_contracts.py: all passed. The one skip is the existing uk-data tree re-hash test.
• tools/census_uk_data_target_parity.py --check: current.
• ruff check .: clean. python3 tools/ci_test_plan.py verify: ok.
• Not run locally: the full engine-free suite and the country engine jobs; CI runs them.

🤖 Generated with Claude Code