diff --git a/changelog.d/1090-scotland-brma-la-overlap.added.md b/changelog.d/1090-scotland-brma-la-overlap.added.md new file mode 100644 index 000000000..af0a4395d --- /dev/null +++ b/changelog.d/1090-scotland-brma-la-overlap.added.md @@ -0,0 +1 @@ +Measure how Scotland's 18 Broad Rental Market Areas overlap its 32 council areas, and settle whether ONS's BRMA-grain private rents can become council targets (#1090). `uk/scotland_brma_la_overlap.json` holds Census 2022 private-rented and all households for the 63 BRMA × council intersections, rebuilt from five sha-pinned official originals by `tools/build_uk_scotland_brma_la_overlap.py` and read through `uk_runtime/scotland_brma_la_overlap.py`. The table shows they cannot: 18 councils span more than one BRMA, 29 councils and 17 BRMAs form one connected block, and a household-weighted mean of BRMA rents would give every council inside a BRMA that BRMA's average (up to 30.9% from the council's own mean in Rent Service Scotland's 2020-21 market evidence, `experiments/1090-scotland-brma-la-rent-dispersion.json`). The 32 Scottish `ons.rent.private_rent` cells stay signed deferred with the reason restated, and the `cross_grain_private_rent_scotland_brma` parity concern stays routed, now to #1090: bind the rents at BRMA grain. No target value changes. diff --git a/docs/uk-scotland-private-rent-brma-1090.md b/docs/uk-scotland-private-rent-brma-1090.md new file mode 100644 index 000000000..d5cd99633 --- /dev/null +++ b/docs/uk-scotland-private-rent-brma-1090.md @@ -0,0 +1,127 @@ +# Scottish private rents: BRMA facts and council targets (#1090) + +ONS's Price Index of Private Rents (PIPR) publishes Scottish rent levels for the +18 Broad Rental Market Areas (area codes `S33000001` to `S33000018`). The +`ons.rent.private_rent` target is bound at local-authority grain, where each +cell is 12 × the area's mean monthly rent × its private-renter households +(microcosm#355). Scotland has no council-grain rent facts, so its 32 cells are +signed deferred under `private_rent_pipr_scotland_brma_grain`. + +This note records the measured overlap between the two geographies and the +decision it supports: BRMA rents are not translated into council targets. + +## The overlap table + +`uk/scotland_brma_la_overlap.json` holds households for every BRMA × council +intersection. Each 2022 census output area is placed in one BRMA by its +population-weighted centroid in the Scottish Government's BRMA polygons. Rent +Service Scotland's postcode lookup (FOI 202300368850) supplies the BRMA of nine +centroids outside every polygon and corrects 61 output areas around Balloch +where the newer lookup contradicts the 2015 polygons. Output areas nest in 2022 +electoral wards, and wards nest in council areas. + +The table carries two weights and says which is which: + +- `private_rented_households` comes from Census 2022 tenure by ward, the finest + level at which the census releases the private-rented split. A ward that + crosses a BRMA boundary is split by its share of output-area households, so + within those 36 wards private renters are assumed to be spread like all + households. They hold 22,821 of the 322,995 private-rented households. +- `all_households` sums output-area household counts directly and needs no + such assumption. Both weights occupy the same 63 cells and put every council + in the same main BRMA. + +Rebuild it from the five sha-pinned originals (the ward table is a manual +export; the tool's `SOURCES` gives each URL): + +```bash +uvx --with geopandas --with pandas --with openpyxl --with pyogrio \ + python tools/build_uk_scotland_brma_la_overlap.py --cache --check +``` + +`UK_SCOTLAND_BRMA_LA_CACHE=` runs the same comparison as a test wherever +the originals are staged. PR CI pins the cells by digest and checks the +invariants listed in `test_uk_scotland_brma_la_overlap.py`. + +Limits: one centroid places a whole output area; the polygons date from 2015, +the postcode lookup from 2023 and the census from 2022; census cells are +perturbed for disclosure control. + +## What the table shows + +- 18 of the 32 councils fall in more than one BRMA, and 16 of the 18 BRMAs + cover more than one council. +- 29 councils and 17 BRMAs form one connected block. The three Ayrshire + councils and the Ayrshires BRMA are the only group that separates, so no + grouping finer than that makes BRMAs a union of whole councils. +- Much of the joining is slivers. Dropping an intersection that is under 1% of + both its council and its BRMA leaves 11 blocks, but five BRMAs and seven + councils still sit in one of them. +- Four councils coincide with one BRMA at 99% of private-rented households in + both directions: Dumfries and Galloway, Fife, West Lothian and Scottish + Borders. +- 14 councils sit wholly inside one BRMA; 11 of them share that BRMA with + another such council. + +## Decision: no translation into council targets + +The candidate rule was the mean of BRMA rents weighted by each council's +private-rented households in each BRMA. It does not satisfy the parity +concern's fence, "preserve source geography without allocating BRMA averages +over local authorities", for three reasons. + +1. **It is the allocation the fence names.** A council wholly inside one BRMA + gets that BRMA's average unchanged, and a split council gets a blend of + averages. Councils sharing a BRMA therefore get the same rent, whatever + their own rents are. The parity register already rules out the same move at + constituency grain (`local_devolved_constituency_rent_anchors`: a signed + absence is preferred to an allocated local target). +2. **The differences it erases are large.** Rent Service Scotland's market + evidence for the year to September 2021 (FOI 202200303624: 31,833 lettings + with a BRMA, a rent and a postcode district) gives each council's own mean + rent. The rule's value is up to 30.9% away from it: + + | Council | BRMA | Rule ÷ council's own mean | + |---|---|---| + | Midlothian | Lothian | +30.9% | + | Clackmannanshire | Forth Valley | +26.8% | + | East Lothian | Lothian | +23.1% | + | Falkirk | Forth Valley | +20.5% | + | Stirling | Forth Valley | −15.2% | + | Orkney Islands | Highland and Islands | +13.6% | + | Na h-Eileanan Siar | Highland and Islands | +13.3% | + | East Renfrewshire | Greater Glasgow, Renfrewshire/Inverclyde | −12.5% | + | Angus | Dundee and Angus | +11.1% | + + Across the 31 councils with at least 30 records, the median absolute gap is + 3.3%; 12 exceed 5% and 9 exceed 10%. The four coincident councils are + within 0.1%. `experiments/1090-scotland-brma-la-rent-dispersion.py` rebuilds + the receipt. Records are placed in councils by postcode district, which is + approximate, and they are rent officers' market evidence rather than PIPR, + so the receipt measures the size of the differences, not council rents. +3. **The cross-grain operator cannot carry it.** `CrossGrainBridge` declares + that a set of higher-grain contract targets is the same additive total as a + lower side, and `UK_CROSS_GRAIN_RULE` rescales lower rows onto that control + over legs that partition the areas (`country > region > constituency > la`). + A BRMA rent is a mean, not a total; BRMA is not a grain of the rule; and + `leg_of_area` gives each council one leg, which 18 councils do not have. + +The one thing the rule does preserve is the rent bill: renters × rent summed +over councils equals the same sum over BRMAs. That is a property of the totals +at BRMA grain, which is where the facts should bind. + +## What would lift the deferral + +- **BRMA target rows.** Keep PIPR's geography on the target side (12 × BRMA + mean × BRMA private renters) and aggregate the model side with the overlap + shares. This uses all 18 published rents and allocates nothing. The local + surface has two area types today (`constituency`, `la`), so it needs a third + through the rowwise matrix, the support checks and the receipts. +- **A signed exception for the four coincident councils.** For these the BRMA + rent is the council's own rent up to the 1% on each side that lies elsewhere. + Binding them lets a fact on the BRMA boundary set reach a local-authority + cell, which the boundary-frame gate added in #795 refuses by design, so it + needs a maintainer's ruling rather than a translation rule. + +`test_the_surface_still_has_no_brma_grain` fails when either route lands, so +the concern's reason and fence are revisited then. diff --git a/experiments/1090-scotland-brma-la-rent-dispersion.json b/experiments/1090-scotland-brma-la-rent-dispersion.json new file mode 100644 index 000000000..428a38b91 --- /dev/null +++ b/experiments/1090-scotland-brma-la-rent-dispersion.json @@ -0,0 +1,438 @@ +{ + "issue": "PolicyEngine/microcosm#1090", + "question": "What would a council target built as the household-weighted mean of BRMA rents get wrong?", + "evidence": { + "filename": "scot_foi_brma_2016_2021.xlsx", + "sha256": "dba86fa1cce920c8f646460bb13e1d99e16927d57fdf290e874f9631e804ed06", + "landing_page": "https://www.gov.scot/publications/foi-202200303624/", + "sheet": "2021" + }, + "pipr": { + "filename": "pipr_monthly_price_statistics_july2026_edition.xlsx", + "sha256": "d429e553ade454903b731f6c257609f14d0c2171822791e09c21a9d53d5e24a9", + "landing_page": "https://www.ons.gov.uk/economy/inflationandpriceindices/datasets/priceindexofprivaterentsukmonthlypricestatistics", + "sheet": "Table 1" + }, + "overlap_resource": "packages/microcosm-build/src/microcosm/build/uk/scotland_brma_la_overlap.json", + "records": 31833, + "records_placed_by_district_only": 32, + "min_records_per_council": 30, + "brma_means": { + "ABERDEEN_AND_SHIRE": { + "records_mean_monthly": 638.84, + "records": 3841, + "pipr_2025_calendar_mean_monthly": 847.75 + }, + "ARGYLL_AND_BUTE": { + "records_mean_monthly": 561.1, + "records": 431, + "pipr_2025_calendar_mean_monthly": 832.42 + }, + "AYRSHIRES": { + "records_mean_monthly": 508.31, + "records": 1897, + "pipr_2025_calendar_mean_monthly": 627.92 + }, + "DUMFRIES_AND_GALLOWAY": { + "records_mean_monthly": 475.96, + "records": 810, + "pipr_2025_calendar_mean_monthly": 526.58 + }, + "DUNDEE_AND_ANGUS": { + "records_mean_monthly": 585.37, + "records": 1985, + "pipr_2025_calendar_mean_monthly": 828.92 + }, + "EAST_DUNBARTONSHIRE": { + "records_mean_monthly": 776.72, + "records": 303, + "pipr_2025_calendar_mean_monthly": 1110.5 + }, + "FIFE": { + "records_mean_monthly": 650.77, + "records": 1770, + "pipr_2025_calendar_mean_monthly": 812.42 + }, + "FORTH_VALLEY": { + "records_mean_monthly": 669.08, + "records": 1239, + "pipr_2025_calendar_mean_monthly": 879.25 + }, + "GREATER_GLASGOW": { + "records_mean_monthly": 785.37, + "records": 5668, + "pipr_2025_calendar_mean_monthly": 1237.75 + }, + "HIGHLAND_AND_ISLANDS": { + "records_mean_monthly": 599.88, + "records": 1712, + "pipr_2025_calendar_mean_monthly": 704.83 + }, + "LOTHIAN": { + "records_mean_monthly": 994.03, + "records": 5881, + "pipr_2025_calendar_mean_monthly": 1408.58 + }, + "NORTH_LANARKSHIRE": { + "records_mean_monthly": 526.48, + "records": 1056, + "pipr_2025_calendar_mean_monthly": 818.92 + }, + "PERTH_AND_KINROSS": { + "records_mean_monthly": 594.25, + "records": 884, + "pipr_2025_calendar_mean_monthly": 746.58 + }, + "RENFREWSHIRE_AND_INVERCLYDE": { + "records_mean_monthly": 507.65, + "records": 1538, + "pipr_2025_calendar_mean_monthly": 811.83 + }, + "SCOTTISH_BORDERS": { + "records_mean_monthly": 533.04, + "records": 660, + "pipr_2025_calendar_mean_monthly": 704.58 + }, + "SOUTH_LANARKSHIRE": { + "records_mean_monthly": 586.21, + "records": 1129, + "pipr_2025_calendar_mean_monthly": 872.08 + }, + "WEST_DUNBARTONSHIRE": { + "records_mean_monthly": 521.2, + "records": 361, + "pipr_2025_calendar_mean_monthly": 904.17 + }, + "WEST_LOTHIAN": { + "records_mean_monthly": 666.44, + "records": 668, + "pipr_2025_calendar_mean_monthly": 838.92 + } + }, + "summary": { + "councils_compared": 31, + "median_abs_gap_pct": 3.3, + "max_abs_gap_pct": 30.9, + "councils_over_5_pct": 12, + "councils_over_10_pct": 9 + }, + "councils": [ + { + "local_authority": "S12000005", + "main_brma": "FORTH_VALLEY", + "share_of_council_in_main_brma": 0.9982, + "placed_records": 166.7, + "rule_value_pipr_2025_monthly": 879.02, + "records_mean_monthly": 527.69, + "rule_value_records_monthly": 668.94, + "rule_over_records_mean_pct": 26.8 + }, + { + "local_authority": "S12000006", + "main_brma": "DUMFRIES_AND_GALLOWAY", + "share_of_council_in_main_brma": 0.9989, + "placed_records": 810.8, + "rule_value_pipr_2025_monthly": 526.97, + "records_mean_monthly": 476.12, + "rule_value_records_monthly": 476.08, + "rule_over_records_mean_pct": -0.0 + }, + { + "local_authority": "S12000008", + "main_brma": "AYRSHIRES", + "share_of_council_in_main_brma": 1.0, + "placed_records": 685.1, + "rule_value_pipr_2025_monthly": 627.92, + "records_mean_monthly": 486.6, + "rule_value_records_monthly": 508.31, + "rule_over_records_mean_pct": 4.5 + }, + { + "local_authority": "S12000010", + "main_brma": "LOTHIAN", + "share_of_council_in_main_brma": 1.0, + "placed_records": 409.8, + "rule_value_pipr_2025_monthly": 1408.58, + "records_mean_monthly": 807.53, + "rule_value_records_monthly": 994.03, + "rule_over_records_mean_pct": 23.1 + }, + { + "local_authority": "S12000011", + "main_brma": "GREATER_GLASGOW", + "share_of_council_in_main_brma": 0.6791, + "placed_records": 224.2, + "rule_value_pipr_2025_monthly": 1101.07, + "records_mean_monthly": 795.98, + "rule_value_records_monthly": 696.24, + "rule_over_records_mean_pct": -12.5 + }, + { + "local_authority": "S12000013", + "main_brma": "HIGHLAND_AND_ISLANDS", + "share_of_council_in_main_brma": 1.0, + "placed_records": 45.0, + "rule_value_pipr_2025_monthly": 704.83, + "records_mean_monthly": 529.33, + "rule_value_records_monthly": 599.88, + "rule_over_records_mean_pct": 13.3 + }, + { + "local_authority": "S12000014", + "main_brma": "FORTH_VALLEY", + "share_of_council_in_main_brma": 0.9938, + "placed_records": 460.8, + "rule_value_pipr_2025_monthly": 879.0, + "records_mean_monthly": 555.32, + "rule_value_records_monthly": 669.06, + "rule_over_records_mean_pct": 20.5 + }, + { + "local_authority": "S12000017", + "main_brma": "HIGHLAND_AND_ISLANDS", + "share_of_council_in_main_brma": 0.999, + "placed_records": 1104.7, + "rule_value_pipr_2025_monthly": 704.97, + "records_mean_monthly": 626.13, + "rule_value_records_monthly": 599.84, + "rule_over_records_mean_pct": -4.2 + }, + { + "local_authority": "S12000018", + "main_brma": "RENFREWSHIRE_AND_INVERCLYDE", + "share_of_council_in_main_brma": 1.0, + "placed_records": 397.8, + "rule_value_pipr_2025_monthly": 811.83, + "records_mean_monthly": 478.81, + "rule_value_records_monthly": 507.65, + "rule_over_records_mean_pct": 6.0 + }, + { + "local_authority": "S12000019", + "main_brma": "LOTHIAN", + "share_of_council_in_main_brma": 1.0, + "placed_records": 337.0, + "rule_value_pipr_2025_monthly": 1408.58, + "records_mean_monthly": 759.57, + "rule_value_records_monthly": 994.03, + "rule_over_records_mean_pct": 30.9 + }, + { + "local_authority": "S12000020", + "main_brma": "HIGHLAND_AND_ISLANDS", + "share_of_council_in_main_brma": 0.9957, + "placed_records": 497.6, + "rule_value_pipr_2025_monthly": 705.45, + "records_mean_monthly": 550.63, + "rule_value_records_monthly": 600.05, + "rule_over_records_mean_pct": 9.0 + }, + { + "local_authority": "S12000021", + "main_brma": "AYRSHIRES", + "share_of_council_in_main_brma": 1.0, + "placed_records": 544.3, + "rule_value_pipr_2025_monthly": 627.92, + "records_mean_monthly": 492.25, + "rule_value_records_monthly": 508.31, + "rule_over_records_mean_pct": 3.3 + }, + { + "local_authority": "S12000023", + "main_brma": "HIGHLAND_AND_ISLANDS", + "share_of_council_in_main_brma": 1.0, + "placed_records": 47.0, + "rule_value_pipr_2025_monthly": 704.83, + "records_mean_monthly": 527.91, + "rule_value_records_monthly": 599.88, + "rule_over_records_mean_pct": 13.6 + }, + { + "local_authority": "S12000026", + "main_brma": "SCOTTISH_BORDERS", + "share_of_council_in_main_brma": 0.99, + "placed_records": 664.7, + "rule_value_pipr_2025_monthly": 706.89, + "records_mean_monthly": 533.59, + "rule_value_records_monthly": 534.05, + "rule_over_records_mean_pct": 0.1 + }, + { + "local_authority": "S12000027", + "main_brma": "HIGHLAND_AND_ISLANDS", + "share_of_council_in_main_brma": 1.0, + "placed_records": 19.0, + "rule_value_pipr_2025_monthly": 704.83 + }, + { + "local_authority": "S12000028", + "main_brma": "AYRSHIRES", + "share_of_council_in_main_brma": 1.0, + "placed_records": 667.6, + "rule_value_pipr_2025_monthly": 627.92, + "records_mean_monthly": 543.69, + "rule_value_records_monthly": 508.31, + "rule_over_records_mean_pct": -6.5 + }, + { + "local_authority": "S12000029", + "main_brma": "SOUTH_LANARKSHIRE", + "share_of_council_in_main_brma": 0.9977, + "placed_records": 1082.7, + "rule_value_pipr_2025_monthly": 872.5, + "records_mean_monthly": 579.95, + "rule_value_records_monthly": 586.46, + "rule_over_records_mean_pct": 1.1 + }, + { + "local_authority": "S12000030", + "main_brma": "FORTH_VALLEY", + "share_of_council_in_main_brma": 0.9987, + "placed_records": 628.0, + "rule_value_pipr_2025_monthly": 879.56, + "records_mean_monthly": 789.04, + "rule_value_records_monthly": 669.22, + "rule_over_records_mean_pct": -15.2 + }, + { + "local_authority": "S12000033", + "main_brma": "ABERDEEN_AND_SHIRE", + "share_of_council_in_main_brma": 1.0, + "placed_records": 3142.6, + "rule_value_pipr_2025_monthly": 847.75, + "records_mean_monthly": 633.06, + "rule_value_records_monthly": 638.84, + "rule_over_records_mean_pct": 0.9 + }, + { + "local_authority": "S12000034", + "main_brma": "ABERDEEN_AND_SHIRE", + "share_of_council_in_main_brma": 0.9741, + "placed_records": 724.2, + "rule_value_pipr_2025_monthly": 847.2, + "records_mean_monthly": 655.97, + "rule_value_records_monthly": 637.47, + "rule_over_records_mean_pct": -2.8 + }, + { + "local_authority": "S12000035", + "main_brma": "ARGYLL_AND_BUTE", + "share_of_council_in_main_brma": 1.0, + "placed_records": 421.2, + "rule_value_pipr_2025_monthly": 832.42, + "records_mean_monthly": 557.1, + "rule_value_records_monthly": 561.1, + "rule_over_records_mean_pct": 0.7 + }, + { + "local_authority": "S12000036", + "main_brma": "LOTHIAN", + "share_of_council_in_main_brma": 1.0, + "placed_records": 5133.5, + "rule_value_pipr_2025_monthly": 1408.58, + "records_mean_monthly": 1024.34, + "rule_value_records_monthly": 994.03, + "rule_over_records_mean_pct": -3.0 + }, + { + "local_authority": "S12000038", + "main_brma": "RENFREWSHIRE_AND_INVERCLYDE", + "share_of_council_in_main_brma": 0.9991, + "placed_records": 1113.2, + "rule_value_pipr_2025_monthly": 812.21, + "records_mean_monthly": 516.97, + "rule_value_records_monthly": 507.9, + "rule_over_records_mean_pct": -1.8 + }, + { + "local_authority": "S12000039", + "main_brma": "WEST_DUNBARTONSHIRE", + "share_of_council_in_main_brma": 0.9574, + "placed_records": 371.6, + "rule_value_pipr_2025_monthly": 901.38, + "records_mean_monthly": 527.04, + "rule_value_records_monthly": 523.05, + "rule_over_records_mean_pct": -0.8 + }, + { + "local_authority": "S12000040", + "main_brma": "WEST_LOTHIAN", + "share_of_council_in_main_brma": 0.9955, + "placed_records": 651.2, + "rule_value_pipr_2025_monthly": 838.85, + "records_mean_monthly": 666.36, + "rule_value_records_monthly": 665.87, + "rule_over_records_mean_pct": -0.1 + }, + { + "local_authority": "S12000041", + "main_brma": "DUNDEE_AND_ANGUS", + "share_of_council_in_main_brma": 1.0, + "placed_records": 564.3, + "rule_value_pipr_2025_monthly": 828.92, + "records_mean_monthly": 526.74, + "rule_value_records_monthly": 585.37, + "rule_over_records_mean_pct": 11.1 + }, + { + "local_authority": "S12000042", + "main_brma": "DUNDEE_AND_ANGUS", + "share_of_council_in_main_brma": 1.0, + "placed_records": 1350.4, + "rule_value_pipr_2025_monthly": 828.92, + "records_mean_monthly": 612.87, + "rule_value_records_monthly": 585.37, + "rule_over_records_mean_pct": -4.5 + }, + { + "local_authority": "S12000045", + "main_brma": "EAST_DUNBARTONSHIRE", + "share_of_council_in_main_brma": 0.9843, + "placed_records": 303.5, + "rule_value_pipr_2025_monthly": 1105.93, + "records_mean_monthly": 771.54, + "rule_value_records_monthly": 772.8, + "rule_over_records_mean_pct": 0.2 + }, + { + "local_authority": "S12000047", + "main_brma": "FIFE", + "share_of_council_in_main_brma": 0.9998, + "placed_records": 1766.8, + "rule_value_pipr_2025_monthly": 812.43, + "records_mean_monthly": 650.76, + "rule_value_records_monthly": 650.77, + "rule_over_records_mean_pct": 0.0 + }, + { + "local_authority": "S12000048", + "main_brma": "PERTH_AND_KINROSS", + "share_of_council_in_main_brma": 0.9093, + "placed_records": 932.9, + "rule_value_pipr_2025_monthly": 754.48, + "records_mean_monthly": 595.05, + "rule_value_records_monthly": 594.42, + "rule_over_records_mean_pct": -0.1 + }, + { + "local_authority": "S12000049", + "main_brma": "GREATER_GLASGOW", + "share_of_council_in_main_brma": 0.9982, + "placed_records": 5458.3, + "rule_value_pipr_2025_monthly": 1237.23, + "records_mean_monthly": 784.2, + "rule_value_records_monthly": 785.11, + "rule_over_records_mean_pct": 0.1 + }, + { + "local_authority": "S12000050", + "main_brma": "NORTH_LANARKSHIRE", + "share_of_council_in_main_brma": 0.9494, + "placed_records": 1106.5, + "rule_value_pipr_2025_monthly": 828.07, + "records_mean_monthly": 536.88, + "rule_value_records_monthly": 533.14, + "rule_over_records_mean_pct": -0.7 + } + ] +} diff --git a/experiments/1090-scotland-brma-la-rent-dispersion.py b/experiments/1090-scotland-brma-la-rent-dispersion.py new file mode 100644 index 000000000..ea89bebf6 --- /dev/null +++ b/experiments/1090-scotland-brma-la-rent-dispersion.py @@ -0,0 +1,247 @@ +#!/usr/bin/env python +"""How far a BRMA's mean rent sits from its councils' mean rents (#1090). + +ONS's Price Index of Private Rents gives one rent level per Scottish Broad +Rental Market Area. This script asks what a council-grain target built from +those levels would get wrong, using the only Scottish rent records that carry +both a BRMA and a finer location: Rent Service Scotland's market evidence for +the year to September 2021 (FOI 202200303624), one row per letting with its +BRMA, weekly rent and postcode district. + +For each council it compares + +- the candidate rule: the mean of the BRMA means, weighted by the council's + private-rented households in each BRMA (``scotland_brma_la_overlap.json``); +- the council's own mean rent in the same records. + +A record is placed in councils by the household shares of its (postcode +district, BRMA) pair among 2022 output areas; a pair no output area carries +falls back to the district's shares. That placement is approximate, so the +receipt reports magnitudes, not council rent statistics. + + uvx --with geopandas --with pandas --with openpyxl --with pyogrio \\ + python experiments/1090-scotland-brma-la-rent-dispersion.py --cache + +``--cache`` holds the originals the overlap tool pins, plus the two below. +""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import sys +from pathlib import Path + +REPO = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(REPO)) + +from tools.build_uk_scotland_brma_la_overlap import ( # noqa: E402 + PIPR_BRMAS, + RESOURCE, + _brma_key, + output_area_assignments, + verify_cache, +) + +RECEIPT = Path(__file__).with_suffix(".json") +EVIDENCE = { + "filename": "scot_foi_brma_2016_2021.xlsx", + "sha256": "dba86fa1cce920c8f646460bb13e1d99e16927d57fdf290e874f9631e804ed06", + "landing_page": "https://www.gov.scot/publications/foi-202200303624/", + "sheet": "2021", +} +PIPR = { + "filename": "pipr_monthly_price_statistics_july2026_edition.xlsx", + "sha256": "d429e553ade454903b731f6c257609f14d0c2171822791e09c21a9d53d5e24a9", + "landing_page": ( + "https://www.ons.gov.uk/economy/inflationandpriceindices/datasets/" + "priceindexofprivaterentsukmonthlypricestatistics" + ), + "sheet": "Table 1", +} +#: Councils with fewer placed records than this are left out of the gap table. +MIN_RECORDS = 30 +WEEKS_PER_MONTH = 52 / 12 + + +def main(argv: list[str] | None = None) -> int: + import pandas as pd + + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--cache", type=Path, required=True) + parser.add_argument("--out", type=Path, default=RECEIPT) + args = parser.parse_args(argv) + + verify_cache(args.cache) + for extra in (EVIDENCE, PIPR): + path = args.cache / extra["filename"] + if hashlib.sha256(path.read_bytes()).hexdigest() != extra["sha256"]: + raise SystemExit(f"{path} does not match its pin") + + overlap = json.loads((REPO / RESOURCE).read_text(encoding="utf-8")) + cells = pd.DataFrame(overlap["cells"]) + council_total = cells.groupby("local_authority")[ + "private_rented_households" + ].transform("sum") + cells["share_of_council"] = cells["private_rented_households"] / council_total + + records = _market_evidence(args.cache) + placed, district_only = _place_records(records, args.cache) + brma_mean = records.groupby("brma")["monthly_rent"].mean() + council = ( + placed.assign(weighted=placed["monthly_rent"] * placed["weight"]) + .groupby("council")[["weighted", "weight"]] + .sum() + ) + council["records_mean"] = council["weighted"] / council["weight"] + + cells["brma_records_mean"] = cells["brma"].map(brma_mean) + rule = ( + (cells["share_of_council"] * cells["brma_records_mean"]) + .groupby(cells["local_authority"]) + .sum() + ) + pipr = _pipr_2025_means(args.cache) + cells["brma_pipr_2025"] = cells["brma"].map(pipr) + rule_pipr = ( + (cells["share_of_council"] * cells["brma_pipr_2025"]) + .groupby(cells["local_authority"]) + .sum() + ) + main_brma = cells.loc[ + cells.groupby("local_authority")["share_of_council"].idxmax() + ].set_index("local_authority") + + rows = [] + for code in sorted(rule.index): + n = float(council["weight"].get(code, 0.0)) + own = float(council["records_mean"].get(code, float("nan"))) + row = { + "local_authority": code, + "main_brma": str(main_brma.at[code, "brma"]), + "share_of_council_in_main_brma": round( + float(main_brma.at[code, "share_of_council"]), 4 + ), + "placed_records": round(n, 1), + "rule_value_pipr_2025_monthly": round(float(rule_pipr[code]), 2), + } + if n >= MIN_RECORDS: + row["records_mean_monthly"] = round(own, 2) + row["rule_value_records_monthly"] = round(float(rule[code]), 2) + row["rule_over_records_mean_pct"] = round( + 100 * (float(rule[code]) / own - 1), 1 + ) + rows.append(row) + + gaps = sorted( + abs(row["rule_over_records_mean_pct"]) + for row in rows + if "rule_over_records_mean_pct" in row + ) + receipt = { + "issue": "PolicyEngine/microcosm#1090", + "question": ( + "What would a council target built as the household-weighted mean " + "of BRMA rents get wrong?" + ), + "evidence": EVIDENCE, + "pipr": PIPR, + "overlap_resource": str(RESOURCE), + "records": int(len(records)), + "records_placed_by_district_only": district_only, + "min_records_per_council": MIN_RECORDS, + "brma_means": { + brma: { + "records_mean_monthly": round(float(brma_mean[brma]), 2), + "records": int((records["brma"] == brma).sum()), + "pipr_2025_calendar_mean_monthly": round(float(pipr[brma]), 2), + } + for brma in sorted(PIPR_BRMAS) + }, + "summary": { + "councils_compared": len(gaps), + "median_abs_gap_pct": gaps[len(gaps) // 2], + "max_abs_gap_pct": gaps[-1], + "councils_over_5_pct": sum(gap > 5 for gap in gaps), + "councils_over_10_pct": sum(gap > 10 for gap in gaps), + }, + "councils": rows, + } + args.out.write_text(json.dumps(receipt, indent=2) + "\n", encoding="utf-8") + print(json.dumps(receipt["summary"], indent=2)) + return 0 + + +def _market_evidence(cache: Path): + import pandas as pd + + frame = pd.read_excel( + cache / EVIDENCE["filename"], sheet_name=EVIDENCE["sheet"], dtype=str + ) + if set(frame["FREQUENCY"]) != {"Weekly"}: + raise SystemExit("expected weekly rents only") + frame["monthly_rent"] = frame["NET RENT"].astype(float) * WEEKS_PER_MONTH + frame["brma"] = frame["BRMA"].map(_brma_key) + if set(frame["brma"]) != set(PIPR_BRMAS): + raise SystemExit("unexpected BRMA names in the market evidence") + frame["district"] = frame["POST CODE"].str.strip().str.upper() + return frame[["brma", "district", "monthly_rent"]] + + +def _place_records(records, cache: Path): + import geopandas as gpd + import pandas as pd + + areas = output_area_assignments(cache) + centroids = cache / "scot_oa2022_population_weighted_centroids.zip" + postcodes = gpd.read_file( + f"zip://{centroids}!OutputArea2022_PWC/OutputArea2022_PWC.shp", + ignore_geometry=True, + )[["code", "masterpc"]] + areas = areas.merge( + postcodes, left_on="output_area", right_on="code", validate="one_to_one" + ) + areas["district"] = areas["masterpc"].str.split().str[0].str.upper() + + def shares(keys: list[str]): + grouped = areas.groupby([*keys, "council"])["households"].sum() + total = grouped.groupby(level=list(range(len(keys)))).transform("sum") + return (grouped / total).rename("weight").reset_index() + + exact = records.reset_index().merge( + shares(["district", "brma"]), on=["district", "brma"], how="left" + ) + missing = exact["weight"].isna() + fallback = exact.loc[missing, ["index", "brma", "district", "monthly_rent"]].merge( + shares(["district"]), on="district", how="left" + ) + if fallback["weight"].isna().any(): + raise SystemExit("a record's postcode district has no output area") + placed = pd.concat([exact.loc[~missing], fallback], ignore_index=True) + if abs(placed["weight"].sum() - len(records)) > 1e-6: + raise SystemExit("record placement lost or duplicated records") + return placed, int(missing.sum()) + + +def _pipr_2025_means(cache: Path): + import pandas as pd + + frame = pd.read_excel( + cache / PIPR["filename"], + sheet_name=PIPR["sheet"], + header=2, + usecols=["Time period", "Area code", "Rental price"], + ) + codes = {code: brma for brma, (code, _name) in PIPR_BRMAS.items()} + frame = frame[frame["Area code"].isin(codes)] + frame = frame[pd.to_datetime(frame["Time period"]).dt.year == 2025] + counts = frame.groupby("Area code")["Rental price"].count() + if len(counts) != len(codes) or not counts.eq(12).all(): + raise SystemExit("expected twelve 2025 months for all 18 BRMAs") + means = frame.groupby("Area code")["Rental price"].mean() + return means.rename(index=codes) + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/packages/microcosm-build/src/microcosm/build/uk/country_package.json b/packages/microcosm-build/src/microcosm/build/uk/country_package.json index 60e08be56..3efc4858f 100644 --- a/packages/microcosm-build/src/microcosm/build/uk/country_package.json +++ b/packages/microcosm-build/src/microcosm/build/uk/country_package.json @@ -322,6 +322,11 @@ "kind": "legacy_json", "schema_id": "legacy_json" }, + { + "path": "scotland_brma_la_overlap.json", + "kind": "legacy_json", + "schema_id": "legacy_json" + }, { "path": "uk_atomic_area_supports.provenance.json", "kind": "legacy_json", diff --git a/packages/microcosm-build/src/microcosm/build/uk/local_target_reference_membership.json b/packages/microcosm-build/src/microcosm/build/uk/local_target_reference_membership.json index 39f479b8f..ee149b6c5 100644 --- a/packages/microcosm-build/src/microcosm/build/uk/local_target_reference_membership.json +++ b/packages/microcosm-build/src/microcosm/build/uk/local_target_reference_membership.json @@ -5586,7 +5586,7 @@ "target_id": "ons.rent.private_rent", "geography_level": "local_authority", "reason_id": "private_rent_pipr_scotland_brma_grain", - "rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. CrossGrainBridge declares target identity but cannot translate overlapping BRMA geographies to LAs, and no signed BRMA-to-LA crosswalk is present, so allocation is forbidden and all 32 cells remain deferred.", + "rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. The BRMAs overlap the councils (uk/scotland_brma_la_overlap.json: 63 intersections, 18 councils in more than one BRMA), so CrossGrainBridge, which equates totals over a partition, cannot translate them, and a household-weighted mean of BRMA rents would give every council inside a BRMA that BRMA's average. Allocation is forbidden and all 32 cells remain deferred until PIPR binds at BRMA grain (microcosm#1090).", "area_ids": [ "S12000005", "S12000006", @@ -412222,7 +412222,7 @@ "matched_fact_count_at_or_before_period": 0, "error": "ledger_reference_compile_status_no_fact_for_area", "signed_reason_id": "private_rent_pipr_scotland_brma_grain", - "signed_rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. CrossGrainBridge declares target identity but cannot translate overlapping BRMA geographies to LAs, and no signed BRMA-to-LA crosswalk is present, so allocation is forbidden and all 32 cells remain deferred." + "signed_rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. The BRMAs overlap the councils (uk/scotland_brma_la_overlap.json: 63 intersections, 18 councils in more than one BRMA), so CrossGrainBridge, which equates totals over a partition, cannot translate them, and a household-weighted mean of BRMA rents would give every council inside a BRMA that BRMA's average. Allocation is forbidden and all 32 cells remain deferred until PIPR binds at BRMA grain (microcosm#1090)." }, { "name": "ons.rent.private_rent@S12000006", @@ -412234,7 +412234,7 @@ "matched_fact_count_at_or_before_period": 0, "error": "ledger_reference_compile_status_no_fact_for_area", "signed_reason_id": "private_rent_pipr_scotland_brma_grain", - "signed_rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. CrossGrainBridge declares target identity but cannot translate overlapping BRMA geographies to LAs, and no signed BRMA-to-LA crosswalk is present, so allocation is forbidden and all 32 cells remain deferred." + "signed_rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. The BRMAs overlap the councils (uk/scotland_brma_la_overlap.json: 63 intersections, 18 councils in more than one BRMA), so CrossGrainBridge, which equates totals over a partition, cannot translate them, and a household-weighted mean of BRMA rents would give every council inside a BRMA that BRMA's average. Allocation is forbidden and all 32 cells remain deferred until PIPR binds at BRMA grain (microcosm#1090)." }, { "name": "ons.rent.private_rent@S12000008", @@ -412246,7 +412246,7 @@ "matched_fact_count_at_or_before_period": 0, "error": "ledger_reference_compile_status_no_fact_for_area", "signed_reason_id": "private_rent_pipr_scotland_brma_grain", - "signed_rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. CrossGrainBridge declares target identity but cannot translate overlapping BRMA geographies to LAs, and no signed BRMA-to-LA crosswalk is present, so allocation is forbidden and all 32 cells remain deferred." + "signed_rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. The BRMAs overlap the councils (uk/scotland_brma_la_overlap.json: 63 intersections, 18 councils in more than one BRMA), so CrossGrainBridge, which equates totals over a partition, cannot translate them, and a household-weighted mean of BRMA rents would give every council inside a BRMA that BRMA's average. Allocation is forbidden and all 32 cells remain deferred until PIPR binds at BRMA grain (microcosm#1090)." }, { "name": "ons.rent.private_rent@S12000010", @@ -412258,7 +412258,7 @@ "matched_fact_count_at_or_before_period": 0, "error": "ledger_reference_compile_status_no_fact_for_area", "signed_reason_id": "private_rent_pipr_scotland_brma_grain", - "signed_rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. CrossGrainBridge declares target identity but cannot translate overlapping BRMA geographies to LAs, and no signed BRMA-to-LA crosswalk is present, so allocation is forbidden and all 32 cells remain deferred." + "signed_rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. The BRMAs overlap the councils (uk/scotland_brma_la_overlap.json: 63 intersections, 18 councils in more than one BRMA), so CrossGrainBridge, which equates totals over a partition, cannot translate them, and a household-weighted mean of BRMA rents would give every council inside a BRMA that BRMA's average. Allocation is forbidden and all 32 cells remain deferred until PIPR binds at BRMA grain (microcosm#1090)." }, { "name": "ons.rent.private_rent@S12000011", @@ -412270,7 +412270,7 @@ "matched_fact_count_at_or_before_period": 0, "error": "ledger_reference_compile_status_no_fact_for_area", "signed_reason_id": "private_rent_pipr_scotland_brma_grain", - "signed_rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. CrossGrainBridge declares target identity but cannot translate overlapping BRMA geographies to LAs, and no signed BRMA-to-LA crosswalk is present, so allocation is forbidden and all 32 cells remain deferred." + "signed_rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. The BRMAs overlap the councils (uk/scotland_brma_la_overlap.json: 63 intersections, 18 councils in more than one BRMA), so CrossGrainBridge, which equates totals over a partition, cannot translate them, and a household-weighted mean of BRMA rents would give every council inside a BRMA that BRMA's average. Allocation is forbidden and all 32 cells remain deferred until PIPR binds at BRMA grain (microcosm#1090)." }, { "name": "ons.rent.private_rent@S12000013", @@ -412282,7 +412282,7 @@ "matched_fact_count_at_or_before_period": 0, "error": "ledger_reference_compile_status_no_fact_for_area", "signed_reason_id": "private_rent_pipr_scotland_brma_grain", - "signed_rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. CrossGrainBridge declares target identity but cannot translate overlapping BRMA geographies to LAs, and no signed BRMA-to-LA crosswalk is present, so allocation is forbidden and all 32 cells remain deferred." + "signed_rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. The BRMAs overlap the councils (uk/scotland_brma_la_overlap.json: 63 intersections, 18 councils in more than one BRMA), so CrossGrainBridge, which equates totals over a partition, cannot translate them, and a household-weighted mean of BRMA rents would give every council inside a BRMA that BRMA's average. Allocation is forbidden and all 32 cells remain deferred until PIPR binds at BRMA grain (microcosm#1090)." }, { "name": "ons.rent.private_rent@S12000014", @@ -412294,7 +412294,7 @@ "matched_fact_count_at_or_before_period": 0, "error": "ledger_reference_compile_status_no_fact_for_area", "signed_reason_id": "private_rent_pipr_scotland_brma_grain", - "signed_rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. CrossGrainBridge declares target identity but cannot translate overlapping BRMA geographies to LAs, and no signed BRMA-to-LA crosswalk is present, so allocation is forbidden and all 32 cells remain deferred." + "signed_rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. The BRMAs overlap the councils (uk/scotland_brma_la_overlap.json: 63 intersections, 18 councils in more than one BRMA), so CrossGrainBridge, which equates totals over a partition, cannot translate them, and a household-weighted mean of BRMA rents would give every council inside a BRMA that BRMA's average. Allocation is forbidden and all 32 cells remain deferred until PIPR binds at BRMA grain (microcosm#1090)." }, { "name": "ons.rent.private_rent@S12000017", @@ -412306,7 +412306,7 @@ "matched_fact_count_at_or_before_period": 0, "error": "ledger_reference_compile_status_no_fact_for_area", "signed_reason_id": "private_rent_pipr_scotland_brma_grain", - "signed_rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. CrossGrainBridge declares target identity but cannot translate overlapping BRMA geographies to LAs, and no signed BRMA-to-LA crosswalk is present, so allocation is forbidden and all 32 cells remain deferred." + "signed_rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. The BRMAs overlap the councils (uk/scotland_brma_la_overlap.json: 63 intersections, 18 councils in more than one BRMA), so CrossGrainBridge, which equates totals over a partition, cannot translate them, and a household-weighted mean of BRMA rents would give every council inside a BRMA that BRMA's average. Allocation is forbidden and all 32 cells remain deferred until PIPR binds at BRMA grain (microcosm#1090)." }, { "name": "ons.rent.private_rent@S12000018", @@ -412318,7 +412318,7 @@ "matched_fact_count_at_or_before_period": 0, "error": "ledger_reference_compile_status_no_fact_for_area", "signed_reason_id": "private_rent_pipr_scotland_brma_grain", - "signed_rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. CrossGrainBridge declares target identity but cannot translate overlapping BRMA geographies to LAs, and no signed BRMA-to-LA crosswalk is present, so allocation is forbidden and all 32 cells remain deferred." + "signed_rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. The BRMAs overlap the councils (uk/scotland_brma_la_overlap.json: 63 intersections, 18 councils in more than one BRMA), so CrossGrainBridge, which equates totals over a partition, cannot translate them, and a household-weighted mean of BRMA rents would give every council inside a BRMA that BRMA's average. Allocation is forbidden and all 32 cells remain deferred until PIPR binds at BRMA grain (microcosm#1090)." }, { "name": "ons.rent.private_rent@S12000019", @@ -412330,7 +412330,7 @@ "matched_fact_count_at_or_before_period": 0, "error": "ledger_reference_compile_status_no_fact_for_area", "signed_reason_id": "private_rent_pipr_scotland_brma_grain", - "signed_rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. CrossGrainBridge declares target identity but cannot translate overlapping BRMA geographies to LAs, and no signed BRMA-to-LA crosswalk is present, so allocation is forbidden and all 32 cells remain deferred." + "signed_rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. The BRMAs overlap the councils (uk/scotland_brma_la_overlap.json: 63 intersections, 18 councils in more than one BRMA), so CrossGrainBridge, which equates totals over a partition, cannot translate them, and a household-weighted mean of BRMA rents would give every council inside a BRMA that BRMA's average. Allocation is forbidden and all 32 cells remain deferred until PIPR binds at BRMA grain (microcosm#1090)." }, { "name": "ons.rent.private_rent@S12000020", @@ -412342,7 +412342,7 @@ "matched_fact_count_at_or_before_period": 0, "error": "ledger_reference_compile_status_no_fact_for_area", "signed_reason_id": "private_rent_pipr_scotland_brma_grain", - "signed_rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. CrossGrainBridge declares target identity but cannot translate overlapping BRMA geographies to LAs, and no signed BRMA-to-LA crosswalk is present, so allocation is forbidden and all 32 cells remain deferred." + "signed_rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. The BRMAs overlap the councils (uk/scotland_brma_la_overlap.json: 63 intersections, 18 councils in more than one BRMA), so CrossGrainBridge, which equates totals over a partition, cannot translate them, and a household-weighted mean of BRMA rents would give every council inside a BRMA that BRMA's average. Allocation is forbidden and all 32 cells remain deferred until PIPR binds at BRMA grain (microcosm#1090)." }, { "name": "ons.rent.private_rent@S12000021", @@ -412354,7 +412354,7 @@ "matched_fact_count_at_or_before_period": 0, "error": "ledger_reference_compile_status_no_fact_for_area", "signed_reason_id": "private_rent_pipr_scotland_brma_grain", - "signed_rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. CrossGrainBridge declares target identity but cannot translate overlapping BRMA geographies to LAs, and no signed BRMA-to-LA crosswalk is present, so allocation is forbidden and all 32 cells remain deferred." + "signed_rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. The BRMAs overlap the councils (uk/scotland_brma_la_overlap.json: 63 intersections, 18 councils in more than one BRMA), so CrossGrainBridge, which equates totals over a partition, cannot translate them, and a household-weighted mean of BRMA rents would give every council inside a BRMA that BRMA's average. Allocation is forbidden and all 32 cells remain deferred until PIPR binds at BRMA grain (microcosm#1090)." }, { "name": "ons.rent.private_rent@S12000023", @@ -412366,7 +412366,7 @@ "matched_fact_count_at_or_before_period": 0, "error": "ledger_reference_compile_status_no_fact_for_area", "signed_reason_id": "private_rent_pipr_scotland_brma_grain", - "signed_rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. CrossGrainBridge declares target identity but cannot translate overlapping BRMA geographies to LAs, and no signed BRMA-to-LA crosswalk is present, so allocation is forbidden and all 32 cells remain deferred." + "signed_rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. The BRMAs overlap the councils (uk/scotland_brma_la_overlap.json: 63 intersections, 18 councils in more than one BRMA), so CrossGrainBridge, which equates totals over a partition, cannot translate them, and a household-weighted mean of BRMA rents would give every council inside a BRMA that BRMA's average. Allocation is forbidden and all 32 cells remain deferred until PIPR binds at BRMA grain (microcosm#1090)." }, { "name": "ons.rent.private_rent@S12000026", @@ -412378,7 +412378,7 @@ "matched_fact_count_at_or_before_period": 0, "error": "ledger_reference_compile_status_no_fact_for_area", "signed_reason_id": "private_rent_pipr_scotland_brma_grain", - "signed_rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. CrossGrainBridge declares target identity but cannot translate overlapping BRMA geographies to LAs, and no signed BRMA-to-LA crosswalk is present, so allocation is forbidden and all 32 cells remain deferred." + "signed_rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. The BRMAs overlap the councils (uk/scotland_brma_la_overlap.json: 63 intersections, 18 councils in more than one BRMA), so CrossGrainBridge, which equates totals over a partition, cannot translate them, and a household-weighted mean of BRMA rents would give every council inside a BRMA that BRMA's average. Allocation is forbidden and all 32 cells remain deferred until PIPR binds at BRMA grain (microcosm#1090)." }, { "name": "ons.rent.private_rent@S12000027", @@ -412390,7 +412390,7 @@ "matched_fact_count_at_or_before_period": 0, "error": "ledger_reference_compile_status_no_fact_for_area", "signed_reason_id": "private_rent_pipr_scotland_brma_grain", - "signed_rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. CrossGrainBridge declares target identity but cannot translate overlapping BRMA geographies to LAs, and no signed BRMA-to-LA crosswalk is present, so allocation is forbidden and all 32 cells remain deferred." + "signed_rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. The BRMAs overlap the councils (uk/scotland_brma_la_overlap.json: 63 intersections, 18 councils in more than one BRMA), so CrossGrainBridge, which equates totals over a partition, cannot translate them, and a household-weighted mean of BRMA rents would give every council inside a BRMA that BRMA's average. Allocation is forbidden and all 32 cells remain deferred until PIPR binds at BRMA grain (microcosm#1090)." }, { "name": "ons.rent.private_rent@S12000028", @@ -412402,7 +412402,7 @@ "matched_fact_count_at_or_before_period": 0, "error": "ledger_reference_compile_status_no_fact_for_area", "signed_reason_id": "private_rent_pipr_scotland_brma_grain", - "signed_rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. CrossGrainBridge declares target identity but cannot translate overlapping BRMA geographies to LAs, and no signed BRMA-to-LA crosswalk is present, so allocation is forbidden and all 32 cells remain deferred." + "signed_rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. The BRMAs overlap the councils (uk/scotland_brma_la_overlap.json: 63 intersections, 18 councils in more than one BRMA), so CrossGrainBridge, which equates totals over a partition, cannot translate them, and a household-weighted mean of BRMA rents would give every council inside a BRMA that BRMA's average. Allocation is forbidden and all 32 cells remain deferred until PIPR binds at BRMA grain (microcosm#1090)." }, { "name": "ons.rent.private_rent@S12000029", @@ -412414,7 +412414,7 @@ "matched_fact_count_at_or_before_period": 0, "error": "ledger_reference_compile_status_no_fact_for_area", "signed_reason_id": "private_rent_pipr_scotland_brma_grain", - "signed_rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. CrossGrainBridge declares target identity but cannot translate overlapping BRMA geographies to LAs, and no signed BRMA-to-LA crosswalk is present, so allocation is forbidden and all 32 cells remain deferred." + "signed_rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. The BRMAs overlap the councils (uk/scotland_brma_la_overlap.json: 63 intersections, 18 councils in more than one BRMA), so CrossGrainBridge, which equates totals over a partition, cannot translate them, and a household-weighted mean of BRMA rents would give every council inside a BRMA that BRMA's average. Allocation is forbidden and all 32 cells remain deferred until PIPR binds at BRMA grain (microcosm#1090)." }, { "name": "ons.rent.private_rent@S12000030", @@ -412426,7 +412426,7 @@ "matched_fact_count_at_or_before_period": 0, "error": "ledger_reference_compile_status_no_fact_for_area", "signed_reason_id": "private_rent_pipr_scotland_brma_grain", - "signed_rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. CrossGrainBridge declares target identity but cannot translate overlapping BRMA geographies to LAs, and no signed BRMA-to-LA crosswalk is present, so allocation is forbidden and all 32 cells remain deferred." + "signed_rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. The BRMAs overlap the councils (uk/scotland_brma_la_overlap.json: 63 intersections, 18 councils in more than one BRMA), so CrossGrainBridge, which equates totals over a partition, cannot translate them, and a household-weighted mean of BRMA rents would give every council inside a BRMA that BRMA's average. Allocation is forbidden and all 32 cells remain deferred until PIPR binds at BRMA grain (microcosm#1090)." }, { "name": "ons.rent.private_rent@S12000033", @@ -412438,7 +412438,7 @@ "matched_fact_count_at_or_before_period": 0, "error": "ledger_reference_compile_status_no_fact_for_area", "signed_reason_id": "private_rent_pipr_scotland_brma_grain", - "signed_rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. CrossGrainBridge declares target identity but cannot translate overlapping BRMA geographies to LAs, and no signed BRMA-to-LA crosswalk is present, so allocation is forbidden and all 32 cells remain deferred." + "signed_rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. The BRMAs overlap the councils (uk/scotland_brma_la_overlap.json: 63 intersections, 18 councils in more than one BRMA), so CrossGrainBridge, which equates totals over a partition, cannot translate them, and a household-weighted mean of BRMA rents would give every council inside a BRMA that BRMA's average. Allocation is forbidden and all 32 cells remain deferred until PIPR binds at BRMA grain (microcosm#1090)." }, { "name": "ons.rent.private_rent@S12000034", @@ -412450,7 +412450,7 @@ "matched_fact_count_at_or_before_period": 0, "error": "ledger_reference_compile_status_no_fact_for_area", "signed_reason_id": "private_rent_pipr_scotland_brma_grain", - "signed_rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. CrossGrainBridge declares target identity but cannot translate overlapping BRMA geographies to LAs, and no signed BRMA-to-LA crosswalk is present, so allocation is forbidden and all 32 cells remain deferred." + "signed_rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. The BRMAs overlap the councils (uk/scotland_brma_la_overlap.json: 63 intersections, 18 councils in more than one BRMA), so CrossGrainBridge, which equates totals over a partition, cannot translate them, and a household-weighted mean of BRMA rents would give every council inside a BRMA that BRMA's average. Allocation is forbidden and all 32 cells remain deferred until PIPR binds at BRMA grain (microcosm#1090)." }, { "name": "ons.rent.private_rent@S12000035", @@ -412462,7 +412462,7 @@ "matched_fact_count_at_or_before_period": 0, "error": "ledger_reference_compile_status_no_fact_for_area", "signed_reason_id": "private_rent_pipr_scotland_brma_grain", - "signed_rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. CrossGrainBridge declares target identity but cannot translate overlapping BRMA geographies to LAs, and no signed BRMA-to-LA crosswalk is present, so allocation is forbidden and all 32 cells remain deferred." + "signed_rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. The BRMAs overlap the councils (uk/scotland_brma_la_overlap.json: 63 intersections, 18 councils in more than one BRMA), so CrossGrainBridge, which equates totals over a partition, cannot translate them, and a household-weighted mean of BRMA rents would give every council inside a BRMA that BRMA's average. Allocation is forbidden and all 32 cells remain deferred until PIPR binds at BRMA grain (microcosm#1090)." }, { "name": "ons.rent.private_rent@S12000036", @@ -412474,7 +412474,7 @@ "matched_fact_count_at_or_before_period": 0, "error": "ledger_reference_compile_status_no_fact_for_area", "signed_reason_id": "private_rent_pipr_scotland_brma_grain", - "signed_rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. CrossGrainBridge declares target identity but cannot translate overlapping BRMA geographies to LAs, and no signed BRMA-to-LA crosswalk is present, so allocation is forbidden and all 32 cells remain deferred." + "signed_rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. The BRMAs overlap the councils (uk/scotland_brma_la_overlap.json: 63 intersections, 18 councils in more than one BRMA), so CrossGrainBridge, which equates totals over a partition, cannot translate them, and a household-weighted mean of BRMA rents would give every council inside a BRMA that BRMA's average. Allocation is forbidden and all 32 cells remain deferred until PIPR binds at BRMA grain (microcosm#1090)." }, { "name": "ons.rent.private_rent@S12000038", @@ -412486,7 +412486,7 @@ "matched_fact_count_at_or_before_period": 0, "error": "ledger_reference_compile_status_no_fact_for_area", "signed_reason_id": "private_rent_pipr_scotland_brma_grain", - "signed_rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. CrossGrainBridge declares target identity but cannot translate overlapping BRMA geographies to LAs, and no signed BRMA-to-LA crosswalk is present, so allocation is forbidden and all 32 cells remain deferred." + "signed_rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. The BRMAs overlap the councils (uk/scotland_brma_la_overlap.json: 63 intersections, 18 councils in more than one BRMA), so CrossGrainBridge, which equates totals over a partition, cannot translate them, and a household-weighted mean of BRMA rents would give every council inside a BRMA that BRMA's average. Allocation is forbidden and all 32 cells remain deferred until PIPR binds at BRMA grain (microcosm#1090)." }, { "name": "ons.rent.private_rent@S12000039", @@ -412498,7 +412498,7 @@ "matched_fact_count_at_or_before_period": 0, "error": "ledger_reference_compile_status_no_fact_for_area", "signed_reason_id": "private_rent_pipr_scotland_brma_grain", - "signed_rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. CrossGrainBridge declares target identity but cannot translate overlapping BRMA geographies to LAs, and no signed BRMA-to-LA crosswalk is present, so allocation is forbidden and all 32 cells remain deferred." + "signed_rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. The BRMAs overlap the councils (uk/scotland_brma_la_overlap.json: 63 intersections, 18 councils in more than one BRMA), so CrossGrainBridge, which equates totals over a partition, cannot translate them, and a household-weighted mean of BRMA rents would give every council inside a BRMA that BRMA's average. Allocation is forbidden and all 32 cells remain deferred until PIPR binds at BRMA grain (microcosm#1090)." }, { "name": "ons.rent.private_rent@S12000040", @@ -412510,7 +412510,7 @@ "matched_fact_count_at_or_before_period": 0, "error": "ledger_reference_compile_status_no_fact_for_area", "signed_reason_id": "private_rent_pipr_scotland_brma_grain", - "signed_rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. CrossGrainBridge declares target identity but cannot translate overlapping BRMA geographies to LAs, and no signed BRMA-to-LA crosswalk is present, so allocation is forbidden and all 32 cells remain deferred." + "signed_rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. The BRMAs overlap the councils (uk/scotland_brma_la_overlap.json: 63 intersections, 18 councils in more than one BRMA), so CrossGrainBridge, which equates totals over a partition, cannot translate them, and a household-weighted mean of BRMA rents would give every council inside a BRMA that BRMA's average. Allocation is forbidden and all 32 cells remain deferred until PIPR binds at BRMA grain (microcosm#1090)." }, { "name": "ons.rent.private_rent@S12000041", @@ -412522,7 +412522,7 @@ "matched_fact_count_at_or_before_period": 0, "error": "ledger_reference_compile_status_no_fact_for_area", "signed_reason_id": "private_rent_pipr_scotland_brma_grain", - "signed_rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. CrossGrainBridge declares target identity but cannot translate overlapping BRMA geographies to LAs, and no signed BRMA-to-LA crosswalk is present, so allocation is forbidden and all 32 cells remain deferred." + "signed_rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. The BRMAs overlap the councils (uk/scotland_brma_la_overlap.json: 63 intersections, 18 councils in more than one BRMA), so CrossGrainBridge, which equates totals over a partition, cannot translate them, and a household-weighted mean of BRMA rents would give every council inside a BRMA that BRMA's average. Allocation is forbidden and all 32 cells remain deferred until PIPR binds at BRMA grain (microcosm#1090)." }, { "name": "ons.rent.private_rent@S12000042", @@ -412534,7 +412534,7 @@ "matched_fact_count_at_or_before_period": 0, "error": "ledger_reference_compile_status_no_fact_for_area", "signed_reason_id": "private_rent_pipr_scotland_brma_grain", - "signed_rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. CrossGrainBridge declares target identity but cannot translate overlapping BRMA geographies to LAs, and no signed BRMA-to-LA crosswalk is present, so allocation is forbidden and all 32 cells remain deferred." + "signed_rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. The BRMAs overlap the councils (uk/scotland_brma_la_overlap.json: 63 intersections, 18 councils in more than one BRMA), so CrossGrainBridge, which equates totals over a partition, cannot translate them, and a household-weighted mean of BRMA rents would give every council inside a BRMA that BRMA's average. Allocation is forbidden and all 32 cells remain deferred until PIPR binds at BRMA grain (microcosm#1090)." }, { "name": "ons.rent.private_rent@S12000045", @@ -412546,7 +412546,7 @@ "matched_fact_count_at_or_before_period": 0, "error": "ledger_reference_compile_status_no_fact_for_area", "signed_reason_id": "private_rent_pipr_scotland_brma_grain", - "signed_rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. CrossGrainBridge declares target identity but cannot translate overlapping BRMA geographies to LAs, and no signed BRMA-to-LA crosswalk is present, so allocation is forbidden and all 32 cells remain deferred." + "signed_rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. The BRMAs overlap the councils (uk/scotland_brma_la_overlap.json: 63 intersections, 18 councils in more than one BRMA), so CrossGrainBridge, which equates totals over a partition, cannot translate them, and a household-weighted mean of BRMA rents would give every council inside a BRMA that BRMA's average. Allocation is forbidden and all 32 cells remain deferred until PIPR binds at BRMA grain (microcosm#1090)." }, { "name": "ons.rent.private_rent@S12000047", @@ -412558,7 +412558,7 @@ "matched_fact_count_at_or_before_period": 0, "error": "ledger_reference_compile_status_no_fact_for_area", "signed_reason_id": "private_rent_pipr_scotland_brma_grain", - "signed_rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. CrossGrainBridge declares target identity but cannot translate overlapping BRMA geographies to LAs, and no signed BRMA-to-LA crosswalk is present, so allocation is forbidden and all 32 cells remain deferred." + "signed_rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. The BRMAs overlap the councils (uk/scotland_brma_la_overlap.json: 63 intersections, 18 councils in more than one BRMA), so CrossGrainBridge, which equates totals over a partition, cannot translate them, and a household-weighted mean of BRMA rents would give every council inside a BRMA that BRMA's average. Allocation is forbidden and all 32 cells remain deferred until PIPR binds at BRMA grain (microcosm#1090)." }, { "name": "ons.rent.private_rent@S12000048", @@ -412570,7 +412570,7 @@ "matched_fact_count_at_or_before_period": 0, "error": "ledger_reference_compile_status_no_fact_for_area", "signed_reason_id": "private_rent_pipr_scotland_brma_grain", - "signed_rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. CrossGrainBridge declares target identity but cannot translate overlapping BRMA geographies to LAs, and no signed BRMA-to-LA crosswalk is present, so allocation is forbidden and all 32 cells remain deferred." + "signed_rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. The BRMAs overlap the councils (uk/scotland_brma_la_overlap.json: 63 intersections, 18 councils in more than one BRMA), so CrossGrainBridge, which equates totals over a partition, cannot translate them, and a household-weighted mean of BRMA rents would give every council inside a BRMA that BRMA's average. Allocation is forbidden and all 32 cells remain deferred until PIPR binds at BRMA grain (microcosm#1090)." }, { "name": "ons.rent.private_rent@S12000049", @@ -412582,7 +412582,7 @@ "matched_fact_count_at_or_before_period": 0, "error": "ledger_reference_compile_status_no_fact_for_area", "signed_reason_id": "private_rent_pipr_scotland_brma_grain", - "signed_rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. CrossGrainBridge declares target identity but cannot translate overlapping BRMA geographies to LAs, and no signed BRMA-to-LA crosswalk is present, so allocation is forbidden and all 32 cells remain deferred." + "signed_rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. The BRMAs overlap the councils (uk/scotland_brma_la_overlap.json: 63 intersections, 18 councils in more than one BRMA), so CrossGrainBridge, which equates totals over a partition, cannot translate them, and a household-weighted mean of BRMA rents would give every council inside a BRMA that BRMA's average. Allocation is forbidden and all 32 cells remain deferred until PIPR binds at BRMA grain (microcosm#1090)." }, { "name": "ons.rent.private_rent@S12000050", @@ -412594,7 +412594,7 @@ "matched_fact_count_at_or_before_period": 0, "error": "ledger_reference_compile_status_no_fact_for_area", "signed_reason_id": "private_rent_pipr_scotland_brma_grain", - "signed_rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. CrossGrainBridge declares target identity but cannot translate overlapping BRMA geographies to LAs, and no signed BRMA-to-LA crosswalk is present, so allocation is forbidden and all 32 cells remain deferred." + "signed_rationale": "The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. The BRMAs overlap the councils (uk/scotland_brma_la_overlap.json: 63 intersections, 18 councils in more than one BRMA), so CrossGrainBridge, which equates totals over a partition, cannot translate them, and a household-weighted mean of BRMA rents would give every council inside a BRMA that BRMA's average. Allocation is forbidden and all 32 cells remain deferred until PIPR binds at BRMA grain (microcosm#1090)." }, { "name": "ons.rent.private_rent@W06000001", diff --git a/packages/microcosm-build/src/microcosm/build/uk/scotland_brma_la_overlap.json b/packages/microcosm-build/src/microcosm/build/uk/scotland_brma_la_overlap.json new file mode 100644 index 000000000..b6c2d3af3 --- /dev/null +++ b/packages/microcosm-build/src/microcosm/build/uk/scotland_brma_la_overlap.json @@ -0,0 +1,533 @@ +{ + "schema_version": 1, + "resource": "scotland_brma_la_overlap", + "description": "Households in each Scottish Broad Rental Market Area x council area intersection, from Scotland's Census 2022 and the official BRMA geography. Rebuilt by tools/build_uk_scotland_brma_la_overlap.py.", + "weights": { + "private_rented_households": "Census 2022 ward households renting from a private landlord or letting agency or other private renting, summed over the published bedroom bands and split across BRMAs by each ward's share of output-area households. The ward is the finest level with this tenure split, so within a ward that crosses a BRMA boundary private renters are assumed to be spread like all households.", + "all_households": "NRS 2022 output-area household counts summed per intersection; each output area has one council and one BRMA." + }, + "method": { + "output_area_to_brma": "Population-weighted centroid within the Scottish Government BRMA polygon. A centroid outside every polygon takes its master postcode's single BRMA in Rent Service Scotland's postcode lookup. Where the lookup contradicts the polygon the newer lookup wins when it names one BRMA; the two output areas it leaves ambiguous are listed under overrides.", + "output_area_to_council": "NRS Census 2022 index (OA_TO_HIGHER_AREAS.csv): 2022 electoral ward and 2019 council area of every output area.", + "overrides": { + "S00179382": "WEST_DUNBARTONSHIRE", + "S00179383": "WEST_DUNBARTONSHIRE" + }, + "private_rented_decimals": 4 + }, + "sources": [ + { + "id": "scot_brma_boundaries_2009", + "publisher": "Scottish Government", + "title": "Broad Rental Market Areas (2009 boundaries, revised 28 July 2015)", + "url": "https://maps.gov.scot/ATOM/shapefiles/SG_BroadRentalMarketAreas_2009.zip", + "landing_page": "https://www.data.gov.uk/dataset/39d08296-874a-4604-9ca4-6fe807aee4a8/broad-rental-market-areas", + "sha256": "4d121236a35ce896edeb663a7c2c390053c0deebe0cc149c84696bbfd4bddec0", + "licence": "Open Government Licence v3.0", + "filename": "scot_brma_boundaries_2009.zip" + }, + { + "id": "scot_oa2022_population_weighted_centroids", + "publisher": "National Records of Scotland", + "title": "2022 Census Output Area Population Weighted Centroids", + "url": "https://www.nrscotland.gov.uk/media/gm4dvdsv/output-area-2022-pwc.zip", + "landing_page": "https://www.nrscotland.gov.uk/publications/2022-census-geography-products/", + "sha256": "509d7d937c8cc1477259dcd8e4e0e61240f5cc782721a79e9c279bd589e0ec0c", + "licence": "Open Government Licence v3.0", + "filename": "scot_oa2022_population_weighted_centroids.zip" + }, + { + "id": "scot_census2022_geography_index", + "publisher": "National Records of Scotland", + "title": "2022 Census Index: Postcode to OA and OA to higher areas", + "url": "https://www.nrscotland.gov.uk/media/utrbt5ze/census_2022_index.zip", + "landing_page": "https://www.nrscotland.gov.uk/publications/2022-census-geography-products/", + "sha256": "0e4096a321cba2318dc5d2e35c197bf815ac333aebdf4cb0ddce82112b57a261", + "licence": "Open Government Licence v3.0", + "filename": "scot_census2022_geography_index.zip" + }, + { + "id": "scot_foi_postcodes_brma_2023", + "publisher": "Scottish Government / Rent Service Scotland", + "title": "FOI 202300368850: postcode to BRMA list held in LHA Direct", + "url": "https://www.gov.scot/binaries/content/documents/govscot/publications/foi-eir-release/2023/11/foi-202300368850/documents/foi-202300368850---information-released---postcodes/foi-202300368850---information-released---postcodes/govscot%3Adocument/FOI%2B202300368850%2B-%2BInformation%2BReleased%2B-%2BPostcodes.xlsx", + "landing_page": "https://www.gov.scot/publications/foi-202300368850/", + "sha256": "50366fea2b87bd6b697e7aa8736fbc870838b5e327458213294f31c3de7ede5e", + "licence": "Open Government Licence v3.0", + "filename": "scot_foi_postcodes_brma_2023.xlsx" + }, + { + "id": "nrs_census_2022_ward2022_tenure_bedrooms", + "publisher": "National Records of Scotland", + "title": "Scotland's Census 2022: household tenure by electoral ward 2022 by number of bedrooms (dataset household2022ftb)", + "url": "https://www.scotlandscensus.gov.uk/webapi/downloadTable?type=csv", + "landing_page": "https://www.scotlandscensus.gov.uk/webapi/home", + "sha256": "03d6157d34bd3df44e79ded1c03510954dee55e9e05d3c276b26ad463a565677", + "licence": "Open Government Licence v3.0", + "filename": "census_2022_ward2022_tenure_bedrooms.csv", + "manual": "Guest session in the census table builder: all 355 Electoral Ward 2022 codes as rows, Number of bedrooms (One to Five or more, and Total) as columns, Household tenure (all eight values and Total) as wafers, Counting: Households; download as CSV unmodified." + } + ], + "brmas": { + "ABERDEEN_AND_SHIRE": { + "pipr_area_code": "S33000001", + "pipr_area_name": "Aberdeen and Shire" + }, + "ARGYLL_AND_BUTE": { + "pipr_area_code": "S33000002", + "pipr_area_name": "Argyll and Bute" + }, + "AYRSHIRES": { + "pipr_area_code": "S33000003", + "pipr_area_name": "Ayrshires" + }, + "DUMFRIES_AND_GALLOWAY": { + "pipr_area_code": "S33000004", + "pipr_area_name": "Dumfries and Galloway" + }, + "DUNDEE_AND_ANGUS": { + "pipr_area_code": "S33000005", + "pipr_area_name": "Dundee and Angus" + }, + "EAST_DUNBARTONSHIRE": { + "pipr_area_code": "S33000006", + "pipr_area_name": "East Dunbartonshire" + }, + "FIFE": { + "pipr_area_code": "S33000007", + "pipr_area_name": "Fife" + }, + "FORTH_VALLEY": { + "pipr_area_code": "S33000008", + "pipr_area_name": "Forth Valley" + }, + "GREATER_GLASGOW": { + "pipr_area_code": "S33000009", + "pipr_area_name": "Greater Glasgow" + }, + "HIGHLAND_AND_ISLANDS": { + "pipr_area_code": "S33000010", + "pipr_area_name": "Highland and Islands" + }, + "LOTHIAN": { + "pipr_area_code": "S33000011", + "pipr_area_name": "Lothian" + }, + "NORTH_LANARKSHIRE": { + "pipr_area_code": "S33000012", + "pipr_area_name": "North Lanarkshire" + }, + "PERTH_AND_KINROSS": { + "pipr_area_code": "S33000013", + "pipr_area_name": "Perth and Kinross" + }, + "RENFREWSHIRE_AND_INVERCLYDE": { + "pipr_area_code": "S33000014", + "pipr_area_name": "Renfrewshire/Inverclyde" + }, + "SCOTTISH_BORDERS": { + "pipr_area_code": "S33000015", + "pipr_area_name": "Scottish Borders" + }, + "SOUTH_LANARKSHIRE": { + "pipr_area_code": "S33000016", + "pipr_area_name": "South Lanarkshire" + }, + "WEST_DUNBARTONSHIRE": { + "pipr_area_code": "S33000017", + "pipr_area_name": "West Dunbartonshire" + }, + "WEST_LOTHIAN": { + "pipr_area_code": "S33000018", + "pipr_area_name": "West Lothian" + } + }, + "totals": { + "cells": 63, + "output_areas": 46363, + "wards": 355, + "wards_split_across_brmas": 36, + "private_rented_households": 322995, + "all_households": 2508542 + }, + "cells": [ + { + "brma": "FORTH_VALLEY", + "local_authority": "S12000005", + "private_rented_households": 2111.2797, + "all_households": 23989 + }, + { + "brma": "PERTH_AND_KINROSS", + "local_authority": "S12000005", + "private_rented_households": 3.7203, + "all_households": 56 + }, + { + "brma": "DUMFRIES_AND_GALLOWAY", + "local_authority": "S12000006", + "private_rented_households": 9488.4589, + "all_households": 70583 + }, + { + "brma": "SOUTH_LANARKSHIRE", + "local_authority": "S12000006", + "private_rented_households": 10.5411, + "all_households": 89 + }, + { + "brma": "AYRSHIRES", + "local_authority": "S12000008", + "private_rented_households": 5299.0, + "all_households": 55395 + }, + { + "brma": "LOTHIAN", + "local_authority": "S12000010", + "private_rented_households": 4169.0, + "all_households": 48770 + }, + { + "brma": "GREATER_GLASGOW", + "local_authority": "S12000011", + "private_rented_households": 1548.3122, + "all_households": 28640 + }, + { + "brma": "RENFREWSHIRE_AND_INVERCLYDE", + "local_authority": "S12000011", + "private_rented_households": 731.6878, + "all_households": 11061 + }, + { + "brma": "HIGHLAND_AND_ISLANDS", + "local_authority": "S12000013", + "private_rented_households": 714.0, + "all_households": 12735 + }, + { + "brma": "FORTH_VALLEY", + "local_authority": "S12000014", + "private_rented_households": 5931.9244, + "all_households": 71804 + }, + { + "brma": "WEST_LOTHIAN", + "local_authority": "S12000014", + "private_rented_households": 37.0756, + "all_households": 695 + }, + { + "brma": "ARGYLL_AND_BUTE", + "local_authority": "S12000017", + "private_rented_households": 12.4183, + "all_households": 129 + }, + { + "brma": "HIGHLAND_AND_ISLANDS", + "local_authority": "S12000017", + "private_rented_households": 11896.5817, + "all_households": 111594 + }, + { + "brma": "RENFREWSHIRE_AND_INVERCLYDE", + "local_authority": "S12000018", + "private_rented_households": 3961.0, + "all_households": 37367 + }, + { + "brma": "LOTHIAN", + "local_authority": "S12000019", + "private_rented_households": 3138.0, + "all_households": 41060 + }, + { + "brma": "ABERDEEN_AND_SHIRE", + "local_authority": "S12000020", + "private_rented_households": 22.2089, + "all_households": 269 + }, + { + "brma": "HIGHLAND_AND_ISLANDS", + "local_authority": "S12000020", + "private_rented_households": 5125.7911, + "all_households": 42848 + }, + { + "brma": "AYRSHIRES", + "local_authority": "S12000021", + "private_rented_households": 6356.0, + "all_households": 64214 + }, + { + "brma": "HIGHLAND_AND_ISLANDS", + "local_authority": "S12000023", + "private_rented_households": 1023.0, + "all_households": 10622 + }, + { + "brma": "LOTHIAN", + "local_authority": "S12000026", + "private_rented_households": 9.0211, + "all_households": 67 + }, + { + "brma": "SCOTTISH_BORDERS", + "local_authority": "S12000026", + "private_rented_households": 7555.9266, + "all_households": 55031 + }, + { + "brma": "SOUTH_LANARKSHIRE", + "local_authority": "S12000026", + "private_rented_households": 67.0523, + "all_households": 498 + }, + { + "brma": "HIGHLAND_AND_ISLANDS", + "local_authority": "S12000027", + "private_rented_households": 743.0, + "all_households": 10594 + }, + { + "brma": "AYRSHIRES", + "local_authority": "S12000028", + "private_rented_households": 5558.0, + "all_households": 52473 + }, + { + "brma": "GREATER_GLASGOW", + "local_authority": "S12000029", + "private_rented_households": 21.0753, + "all_households": 268 + }, + { + "brma": "SCOTTISH_BORDERS", + "local_authority": "S12000029", + "private_rented_households": 10.725, + "all_households": 106 + }, + { + "brma": "SOUTH_LANARKSHIRE", + "local_authority": "S12000029", + "private_rented_households": 14071.1997, + "all_households": 147452 + }, + { + "brma": "EAST_DUNBARTONSHIRE", + "local_authority": "S12000030", + "private_rented_households": 6.8003, + "all_households": 78 + }, + { + "brma": "FORTH_VALLEY", + "local_authority": "S12000030", + "private_rented_households": 5136.1997, + "all_households": 40232 + }, + { + "brma": "ABERDEEN_AND_SHIRE", + "local_authority": "S12000033", + "private_rented_households": 21718.0, + "all_households": 109471 + }, + { + "brma": "ABERDEEN_AND_SHIRE", + "local_authority": "S12000034", + "private_rented_households": 10223.2495, + "all_households": 112658 + }, + { + "brma": "DUNDEE_AND_ANGUS", + "local_authority": "S12000034", + "private_rented_households": 266.5841, + "all_households": 2903 + }, + { + "brma": "HIGHLAND_AND_ISLANDS", + "local_authority": "S12000034", + "private_rented_households": 5.1664, + "all_households": 64 + }, + { + "brma": "ARGYLL_AND_BUTE", + "local_authority": "S12000035", + "private_rented_households": 4838.0, + "all_households": 41824 + }, + { + "brma": "LOTHIAN", + "local_authority": "S12000036", + "private_rented_households": 55141.0, + "all_households": 238899 + }, + { + "brma": "GREATER_GLASGOW", + "local_authority": "S12000038", + "private_rented_households": 8.461, + "all_households": 63 + }, + { + "brma": "RENFREWSHIRE_AND_INVERCLYDE", + "local_authority": "S12000038", + "private_rented_households": 9550.539, + "all_households": 86650 + }, + { + "brma": "ARGYLL_AND_BUTE", + "local_authority": "S12000039", + "private_rented_households": 145.3676, + "all_households": 1354 + }, + { + "brma": "GREATER_GLASGOW", + "local_authority": "S12000039", + "private_rented_households": 2.2858, + "all_households": 28 + }, + { + "brma": "WEST_DUNBARTONSHIRE", + "local_authority": "S12000039", + "private_rented_households": 3320.3466, + "all_households": 41192 + }, + { + "brma": "FORTH_VALLEY", + "local_authority": "S12000040", + "private_rented_households": 3.1814, + "all_households": 47 + }, + { + "brma": "NORTH_LANARKSHIRE", + "local_authority": "S12000040", + "private_rented_households": 30.1726, + "all_households": 416 + }, + { + "brma": "WEST_LOTHIAN", + "local_authority": "S12000040", + "private_rented_households": 7353.646, + "all_households": 79361 + }, + { + "brma": "DUNDEE_AND_ANGUS", + "local_authority": "S12000041", + "private_rented_households": 6404.0, + "all_households": 54223 + }, + { + "brma": "DUNDEE_AND_ANGUS", + "local_authority": "S12000042", + "private_rented_households": 14717.0, + "all_households": 69850 + }, + { + "brma": "EAST_DUNBARTONSHIRE", + "local_authority": "S12000045", + "private_rented_households": 2636.9705, + "all_households": 45869 + }, + { + "brma": "NORTH_LANARKSHIRE", + "local_authority": "S12000045", + "private_rented_households": 42.0295, + "all_households": 576 + }, + { + "brma": "FIFE", + "local_authority": "S12000047", + "private_rented_households": 19619.5638, + "all_households": 169439 + }, + { + "brma": "FORTH_VALLEY", + "local_authority": "S12000047", + "private_rented_households": 3.4362, + "all_households": 46 + }, + { + "brma": "DUNDEE_AND_ANGUS", + "local_authority": "S12000048", + "private_rented_households": 755.6551, + "all_households": 6196 + }, + { + "brma": "FIFE", + "local_authority": "S12000048", + "private_rented_households": 27.4199, + "all_households": 294 + }, + { + "brma": "FORTH_VALLEY", + "local_authority": "S12000048", + "private_rented_households": 90.8303, + "all_households": 841 + }, + { + "brma": "PERTH_AND_KINROSS", + "local_authority": "S12000048", + "private_rented_households": 8756.0947, + "all_households": 62570 + }, + { + "brma": "EAST_DUNBARTONSHIRE", + "local_authority": "S12000049", + "private_rented_households": 39.2972, + "all_households": 408 + }, + { + "brma": "GREATER_GLASGOW", + "local_authority": "S12000049", + "private_rented_households": 49476.341, + "all_households": 292431 + }, + { + "brma": "NORTH_LANARKSHIRE", + "local_authority": "S12000049", + "private_rented_households": 23.4445, + "all_households": 329 + }, + { + "brma": "RENFREWSHIRE_AND_INVERCLYDE", + "local_authority": "S12000049", + "private_rented_households": 4.6928, + "all_households": 77 + }, + { + "brma": "SOUTH_LANARKSHIRE", + "local_authority": "S12000049", + "private_rented_households": 24.2244, + "all_households": 456 + }, + { + "brma": "EAST_DUNBARTONSHIRE", + "local_authority": "S12000050", + "private_rented_households": 22.5863, + "all_households": 430 + }, + { + "brma": "FORTH_VALLEY", + "local_authority": "S12000050", + "private_rented_households": 3.359, + "all_households": 25 + }, + { + "brma": "GREATER_GLASGOW", + "local_authority": "S12000050", + "private_rented_households": 215.1557, + "all_households": 3826 + }, + { + "brma": "NORTH_LANARKSHIRE", + "local_authority": "S12000050", + "private_rented_households": 12350.703, + "all_households": 140650 + }, + { + "brma": "SOUTH_LANARKSHIRE", + "local_authority": "S12000050", + "private_rented_households": 417.196, + "all_households": 6357 + } + ] +} diff --git a/packages/microcosm-build/src/microcosm/build/uk/uk_data_target_parity.json b/packages/microcosm-build/src/microcosm/build/uk/uk_data_target_parity.json index cb4efa40e..31f3b7726 100644 --- a/packages/microcosm-build/src/microcosm/build/uk/uk_data_target_parity.json +++ b/packages/microcosm-build/src/microcosm/build/uk/uk_data_target_parity.json @@ -174,13 +174,13 @@ "covers": [ "policyengine_uk_data.targets.sources.local_la_extras" ], - "evidence": "Pinned feed has 18 Scottish statistical_scope BRMA rows at 2026-06 and zero Scottish LA rows; the local crosswalk has 32 Scottish LAs.", + "evidence": "Pinned feed has 18 Scottish statistical_scope BRMA rows at 2026-06 and zero Scottish LA rows; the local crosswalk has 32 Scottish LAs. uk/scotland_brma_la_overlap.json (Scotland's Census 2022 in the official BRMA geography) holds 322,995 private-rented households in 63 BRMA x council cells: 18 councils span more than one BRMA, 16 BRMAs span more than one council, 29 councils and 17 BRMAs form one connected block, and 4 councils coincide with one BRMA at 99% both ways. In Rent Service Scotland's 2020-21 market evidence the household-weighted mean of BRMA rents is up to 30.9% from a council's own mean rent, and over 10% for 9 of 31 councils (experiments/1090-scotland-brma-la-rent-dispersion.json).", "fence": { "origin": "PIPR publishes Scottish rent levels at BRMA grain.", "purpose": "Preserve source geography without allocating BRMA averages over local authorities.", - "verdict_basis": "Keep all 32 Scottish LA cells signed deferred until a source-backed BRMA-to-LA bridge with an explicit aggregation rule exists." + "verdict_basis": "Keep all 32 Scottish LA cells signed deferred. The source-backed overlap now exists and shows the cells cannot bind by translation: a household-weighted mean of BRMA rents is the allocation this fence forbids. Bind PIPR at its own grain instead (BRMA target rows whose model side uses the overlap shares), or sign a boundary-frame exception for the four coincident councils (microcosm#1090)." }, - "reason": "CrossGrainBridge cannot express an overlapping BRMA-to-LA geography translation and no signed crosswalk is available.", + "reason": "PIPR's Scottish rents are means over 18 BRMAs that overlap the 32 councils, and no rule turns them into council targets without spreading a BRMA average over the councils inside it. CrossGrainBridge equates additive totals over a partition of legs; a mean over overlapping areas is neither. Routed to microcosm#1090: bind the rents at BRMA grain.", "status": "routed" }, { diff --git a/packages/microcosm-build/src/microcosm/build/uk_runtime/data_target_parity.py b/packages/microcosm-build/src/microcosm/build/uk_runtime/data_target_parity.py index 554400887..a3859a003 100644 --- a/packages/microcosm-build/src/microcosm/build/uk_runtime/data_target_parity.py +++ b/packages/microcosm-build/src/microcosm/build/uk_runtime/data_target_parity.py @@ -372,12 +372,12 @@ def _fence(origin: str, purpose: str, verdict_basis: str) -> dict[str, str]: "concern_id": "cross_grain_private_rent_scotland_brma", "status": "routed", "classification": "cross_grain_grain_gap", - "reason": "CrossGrainBridge cannot express an overlapping BRMA-to-LA geography translation and no signed crosswalk is available.", - "evidence": "Pinned feed has 18 Scottish statistical_scope BRMA rows at 2026-06 and zero Scottish LA rows; the local crosswalk has 32 Scottish LAs.", + "reason": "PIPR's Scottish rents are means over 18 BRMAs that overlap the 32 councils, and no rule turns them into council targets without spreading a BRMA average over the councils inside it. CrossGrainBridge equates additive totals over a partition of legs; a mean over overlapping areas is neither. Routed to microcosm#1090: bind the rents at BRMA grain.", + "evidence": "Pinned feed has 18 Scottish statistical_scope BRMA rows at 2026-06 and zero Scottish LA rows; the local crosswalk has 32 Scottish LAs. uk/scotland_brma_la_overlap.json (Scotland's Census 2022 in the official BRMA geography) holds 322,995 private-rented households in 63 BRMA x council cells: 18 councils span more than one BRMA, 16 BRMAs span more than one council, 29 councils and 17 BRMAs form one connected block, and 4 councils coincide with one BRMA at 99% both ways. In Rent Service Scotland's 2020-21 market evidence the household-weighted mean of BRMA rents is up to 30.9% from a council's own mean rent, and over 10% for 9 of 31 councils (experiments/1090-scotland-brma-la-rent-dispersion.json).", "fence": _fence( "PIPR publishes Scottish rent levels at BRMA grain.", "Preserve source geography without allocating BRMA averages over local authorities.", - "Keep all 32 Scottish LA cells signed deferred until a source-backed BRMA-to-LA bridge with an explicit aggregation rule exists.", + "Keep all 32 Scottish LA cells signed deferred. The source-backed overlap now exists and shows the cells cannot bind by translation: a household-weighted mean of BRMA rents is the allocation this fence forbids. Bind PIPR at its own grain instead (BRMA target rows whose model side uses the overlap shares), or sign a boundary-frame exception for the four coincident councils (microcosm#1090).", ), }, { diff --git a/packages/microcosm-build/src/microcosm/build/uk_runtime/scotland_brma_la_overlap.py b/packages/microcosm-build/src/microcosm/build/uk_runtime/scotland_brma_la_overlap.py new file mode 100644 index 000000000..8e2b3bce8 --- /dev/null +++ b/packages/microcosm-build/src/microcosm/build/uk_runtime/scotland_brma_la_overlap.py @@ -0,0 +1,261 @@ +"""How Scotland's Broad Rental Market Areas overlap its council areas. + +ONS's Price Index of Private Rents publishes Scottish rent levels for the 18 +BRMAs; the local target surface is bound at council (local-authority) grain. +``scotland_brma_la_overlap.json`` measures the intersections in households +from Scotland's Census 2022 and the official BRMA geography, and this module +reads it and answers the structural questions the private-rent parity concern +rests on (microcosm#1090): which councils a BRMA covers, whether any grouping +makes BRMAs a union of whole councils, and which council is the same place as +one BRMA. + +The resource is a measurement, not a translation: nothing here turns a BRMA +rent into a council target. +""" + +from __future__ import annotations + +import json +import math +from collections.abc import Mapping +from dataclasses import dataclass +from functools import cache +from importlib import resources as importlib_resources +from typing import Any + +UK_SCOTLAND_BRMA_LA_OVERLAP_RESOURCE = "scotland_brma_la_overlap.json" +UK_SCOTLAND_BRMA_LA_OVERLAP_SCHEMA_VERSION = 1 +SCOTLAND_BRMA_COUNT = 18 +#: ``private_rented_households`` is the measure PIPR's rents describe; +#: ``all_households`` needs no within-ward assumption and is the cross-check. +OVERLAP_WEIGHTS = ("private_rented_households", "all_households") +#: A council and a BRMA coincide when each holds at least this share of the +#: other's private-rented households. +COINCIDENT_MIN_SHARE = 0.99 +#: Relative tolerance of the cells against the census total they are rounded +#: from (four decimals over 63 cells). +_TOTAL_RTOL = 1e-6 + + +@dataclass(frozen=True) +class BrmaCouncilOverlap: + """One BRMA x council intersection and its share of each side.""" + + brma: str + local_authority: str + households: float + share_of_council: float + share_of_brma: float + + +def load_uk_scotland_brma_la_overlap() -> Mapping[str, Any]: + """Load and validate the packaged overlap resource.""" + + return _load() + + +@cache +def _load() -> Mapping[str, Any]: + payload = json.loads( + importlib_resources.files("microcosm.build.uk") + .joinpath(UK_SCOTLAND_BRMA_LA_OVERLAP_RESOURCE) + .read_text() + ) + validate_uk_scotland_brma_la_overlap(payload) + return payload + + +def validate_uk_scotland_brma_la_overlap(payload: Mapping[str, Any]) -> None: + """Refuse an overlap table that does not cover both rosters or add up.""" + + label = UK_SCOTLAND_BRMA_LA_OVERLAP_RESOURCE + if payload.get("schema_version") != UK_SCOTLAND_BRMA_LA_OVERLAP_SCHEMA_VERSION: + raise ValueError(f"{label}: schema_version does not match the runtime.") + brmas = payload.get("brmas") + cells = payload.get("cells") + totals = payload.get("totals") + if not isinstance(brmas, Mapping) or len(brmas) != SCOTLAND_BRMA_COUNT: + raise ValueError(f"{label}: expected {SCOTLAND_BRMA_COUNT} BRMAs.") + codes = [str(entry.get("pipr_area_code", "")) for entry in brmas.values()] + if len(set(codes)) != len(codes) or not all( + code.startswith("S33") for code in codes + ): + raise ValueError(f"{label}: BRMAs need distinct S33 PIPR area codes.") + if not isinstance(cells, list) or not cells or not isinstance(totals, Mapping): + raise ValueError(f"{label}: cells and totals are required.") + + councils = _scottish_council_roster() + seen: set[tuple[str, str]] = set() + for cell in cells: + key = (str(cell.get("brma", "")), str(cell.get("local_authority", ""))) + if key in seen: + raise ValueError(f"{label}: duplicate cell {key}.") + seen.add(key) + if key[0] not in brmas: + raise ValueError(f"{label}: cell names unknown BRMA {key[0]!r}.") + if key[1] not in councils: + raise ValueError( + f"{label}: cell names {key[1]!r}, which is not a Scottish " + "local authority in the local-area crosswalk." + ) + for weight in OVERLAP_WEIGHTS: + value = cell.get(weight) + if ( + isinstance(value, bool) + or not isinstance(value, (int, float)) + or not math.isfinite(value) + or value < 0 + ): + raise ValueError(f"{label}: cell {key} has invalid {weight}.") + if cell["all_households"] <= 0: + # A cell exists because an output area's households sit in it. + raise ValueError(f"{label}: cell {key} holds no households.") + if {brma for brma, _ in seen} != set(brmas): + raise ValueError(f"{label}: a BRMA has no cell.") + if {council for _, council in seen} != councils: + raise ValueError(f"{label}: a Scottish local authority has no cell.") + if totals.get("cells") != len(cells): + raise ValueError(f"{label}: totals.cells does not match the cells.") + for weight in OVERLAP_WEIGHTS: + total = totals.get(weight) + summed = math.fsum(float(cell[weight]) for cell in cells) + if ( + isinstance(total, bool) + or not isinstance(total, int) + or total <= 0 + or not math.isclose(summed, total, rel_tol=_TOTAL_RTOL, abs_tol=0.0) + ): + raise ValueError( + f"{label}: {weight} cells sum to {summed!r}, not the declared " + f"total {total!r}." + ) + + +def _scottish_council_roster() -> frozenset[str]: + crosswalk = json.loads( + importlib_resources.files("microcosm.build.uk") + .joinpath("local_area_crosswalk.json") + .read_text() + ) + return frozenset( + str(area_id) + for area_id in crosswalk["levels"]["local_authority"]["area_ids"] + if str(area_id).startswith("S") + ) + + +def scotland_brma_council_overlaps( + weight: str = "private_rented_households", +) -> tuple[BrmaCouncilOverlap, ...]: + """Every intersection holding ``weight``, with its share of each side.""" + + if weight not in OVERLAP_WEIGHTS: + raise ValueError(f"unknown overlap weight {weight!r}; use {OVERLAP_WEIGHTS}.") + cells = [ + cell + for cell in load_uk_scotland_brma_la_overlap()["cells"] + if float(cell[weight]) > 0 + ] + by_council: dict[str, float] = {} + by_brma: dict[str, float] = {} + for cell in cells: + value = float(cell[weight]) + council, brma = str(cell["local_authority"]), str(cell["brma"]) + by_council[council] = by_council.get(council, 0.0) + value + by_brma[brma] = by_brma.get(brma, 0.0) + value + return tuple( + BrmaCouncilOverlap( + brma=str(cell["brma"]), + local_authority=str(cell["local_authority"]), + households=float(cell[weight]), + share_of_council=float(cell[weight]) + / by_council[str(cell["local_authority"])], + share_of_brma=float(cell[weight]) / by_brma[str(cell["brma"])], + ) + for cell in cells + ) + + +def scotland_brma_council_blocks( + weight: str = "private_rented_households", + *, + min_share: float = 0.0, +) -> tuple[tuple[frozenset[str], frozenset[str]], ...]: + """Smallest groups in which a set of BRMAs equals a set of whole councils. + + Each block is ``(brmas, councils)``: the connected pieces of the overlap. + A rent total is additive over a block and over nothing finer, so a block + is the finest area at which BRMA rents and council rows describe the same + households. ``min_share`` drops an intersection that is below that share + of both its council and its BRMA before grouping, to show how much of the + joining is slivers; the default drops nothing. + """ + + if not 0.0 <= min_share <= 1.0: + raise ValueError("min_share must lie in [0, 1].") + links: dict[str, set[str]] = {} + overlaps = scotland_brma_council_overlaps(weight) + for overlap in overlaps: + brma, council = f"brma:{overlap.brma}", f"la:{overlap.local_authority}" + links.setdefault(brma, set()) + links.setdefault(council, set()) + if max(overlap.share_of_council, overlap.share_of_brma) >= min_share: + links[brma].add(council) + links[council].add(brma) + blocks: list[tuple[frozenset[str], frozenset[str]]] = [] + seen: set[str] = set() + for start in sorted(links): + if start in seen: + continue + block: set[str] = set() + pending = [start] + while pending: + node = pending.pop() + if node in block: + continue + block.add(node) + pending.extend(links[node] - block) + seen |= block + blocks.append( + ( + frozenset(n.removeprefix("brma:") for n in block if n[:5] == "brma:"), + frozenset(n.removeprefix("la:") for n in block if n[:3] == "la:"), + ) + ) + return tuple(blocks) + + +def scotland_coincident_councils( + *, + min_share: float = COINCIDENT_MIN_SHARE, +) -> Mapping[str, str]: + """Councils that are one BRMA's private renters, and it theirs. + + ``council -> brma`` for each pair where the intersection holds at least + ``min_share`` of the council's and of the BRMA's private-rented + households. For such a pair the BRMA's published rent describes the + council's renters up to the remaining share on each side. + """ + + if not 0.5 < min_share <= 1.0: + raise ValueError("min_share must lie in (0.5, 1].") + return { + overlap.local_authority: overlap.brma + for overlap in scotland_brma_council_overlaps() + if overlap.share_of_council >= min_share and overlap.share_of_brma >= min_share + } + + +__all__ = [ + "COINCIDENT_MIN_SHARE", + "OVERLAP_WEIGHTS", + "SCOTLAND_BRMA_COUNT", + "UK_SCOTLAND_BRMA_LA_OVERLAP_RESOURCE", + "UK_SCOTLAND_BRMA_LA_OVERLAP_SCHEMA_VERSION", + "BrmaCouncilOverlap", + "load_uk_scotland_brma_la_overlap", + "scotland_brma_council_blocks", + "scotland_brma_council_overlaps", + "scotland_coincident_councils", + "validate_uk_scotland_brma_la_overlap", +] diff --git a/packages/microcosm-build/tests/engine_free/shared/test_country_spec.py b/packages/microcosm-build/tests/engine_free/shared/test_country_spec.py index 8119fac95..401990772 100644 --- a/packages/microcosm-build/tests/engine_free/shared/test_country_spec.py +++ b/packages/microcosm-build/tests/engine_free/shared/test_country_spec.py @@ -977,6 +977,7 @@ def test_spi_spine_adds_no_country_package_resources(self) -> None: "uk_firms_targets.json", "local_area_crosswalk.json", "local_authority_names.json", + "scotland_brma_la_overlap.json", "uk_atomic_area_supports.provenance.json", "target_references.json", "target_reference_membership.json", @@ -1096,6 +1097,7 @@ def test_uk_package_loads(self) -> None: "uk_firms_targets.json", "local_area_crosswalk.json", "local_authority_names.json", + "scotland_brma_la_overlap.json", "uk_atomic_area_supports.provenance.json", "target_references.json", "target_reference_membership.json", diff --git a/packages/microcosm-build/tests/engine_free/uk/test_uk_scotland_brma_la_overlap.py b/packages/microcosm-build/tests/engine_free/uk/test_uk_scotland_brma_la_overlap.py new file mode 100644 index 000000000..1264fc635 --- /dev/null +++ b/packages/microcosm-build/tests/engine_free/uk/test_uk_scotland_brma_la_overlap.py @@ -0,0 +1,605 @@ +"""Scotland's BRMA x council overlap and the private-rent decision it backs. + +Invariants of the resource (hold for the committed table): + +1. Coverage: the cells cover all 18 BRMAs and exactly the crosswalk's 32 + Scottish local authorities, once each per intersection. +2. Conservation: private-rented cells sum to the census ward total and + all-household cells to the output-area total; per-council and per-BRMA + shares each sum to one. +3. Bounds: every cell holds households, and no cell or council has more + private renters than households. +4. Differential: the private-rented and all-household weights place every + council in the same main BRMA and occupy the same cells. + +Invariants of the allocation in ``tools/build_uk_scotland_brma_la_overlap.py`` +(hold for every input, property-tested): splitting ward counts across BRMAs +conserves each ward's total, each council's total and the grand total. + +Properties of the rejected rule (hold for every set of BRMA rents): the +household-weighted mean of BRMA rents conserves the total rent bill, and +gives each council wholly inside one BRMA exactly that BRMA's rent, so it +cannot express any rent difference between councils that share a BRMA. +""" + +from __future__ import annotations + +import hashlib +import json +import math +import os +import shutil +import subprocess +from collections import Counter +from importlib import resources as importlib_resources +from pathlib import Path + +import pandas as pd +import pytest +from hypothesis import given, settings +from hypothesis import strategies as st + +from microcosm.build.uk_runtime import data_target_parity +from microcosm.build.uk_runtime import scotland_brma_la_overlap as overlap_module +from microcosm.build.uk_runtime.ledger_targets import ( + UK_CROSS_GRAIN_BRIDGES, + UK_CROSS_GRAIN_GRAIN_PRECEDENCE, +) +from microcosm.build.uk_runtime.local_targets import ( + AREA_TYPES, + load_uk_population_contract, +) +from microcosm.build.uk_runtime.scotland_brma_la_overlap import ( + COINCIDENT_MIN_SHARE, + OVERLAP_WEIGHTS, + load_uk_scotland_brma_la_overlap, + scotland_brma_council_blocks, + scotland_brma_council_overlaps, + scotland_coincident_councils, +) +from test_support.paths import paths_for +from tools import build_uk_scotland_brma_la_overlap as tool +from tools.generate_uk_local_target_references import _area_signed_deferrals + +_REPOSITORY = paths_for("microcosm-build").repository +_CELLS_SHA256 = "f9c6c79c7eba5630fdfca97dd12bc326711584a0236f4d9017521e637bfbeab5" +_AYRSHIRE_COUNCILS = frozenset({"S12000008", "S12000021", "S12000028"}) +_COINCIDENT = { + "S12000006": "DUMFRIES_AND_GALLOWAY", + "S12000026": "SCOTTISH_BORDERS", + "S12000040": "WEST_LOTHIAN", + "S12000047": "FIFE", +} + + +def _resource() -> dict: + return dict(load_uk_scotland_brma_la_overlap()) + + +def _packaged(name: str) -> dict: + return json.loads( + importlib_resources.files("microcosm.build.uk").joinpath(name).read_text() + ) + + +def _scottish_roster() -> set[str]: + crosswalk = _packaged("local_area_crosswalk.json") + return { + area_id + for area_id in crosswalk["levels"]["local_authority"]["area_ids"] + if area_id.startswith("S") + } + + +def _weighted_mean_rule(brma_rents: dict[str, float]) -> dict[str, float]: + """The candidate rule: a council's BRMA rents, weighted by its renters.""" + + values: dict[str, float] = {} + for item in scotland_brma_council_overlaps(): + values[item.local_authority] = ( + values.get(item.local_authority, 0.0) + + item.share_of_council * brma_rents[item.brma] + ) + return values + + +# --- the resource --------------------------------------------------------- + + +def test_cells_are_pinned_and_cover_both_rosters() -> None: + resource = _resource() + cells = resource["cells"] + canonical = json.dumps(cells, sort_keys=True, separators=(",", ":")) + assert hashlib.sha256(canonical.encode()).hexdigest() == _CELLS_SHA256 + assert len(cells) == resource["totals"]["cells"] == 63 + assert {cell["brma"] for cell in cells} == set(resource["brmas"]) + assert len(resource["brmas"]) == 18 + assert {cell["local_authority"] for cell in cells} == _scottish_roster() + assert len(_scottish_roster()) == 32 + keys = [(cell["brma"], cell["local_authority"]) for cell in cells] + assert len(keys) == len(set(keys)) + assert keys == sorted(keys, key=lambda key: (key[1], key[0])) + + +def test_cells_conserve_the_census_totals() -> None: + resource = _resource() + cells, totals = resource["cells"], resource["totals"] + assert totals["private_rented_households"] == 322_995 + assert totals["all_households"] == 2_508_542 + assert totals["output_areas"] == 46_363 + assert totals["wards"] == 355 + assert totals["wards_split_across_brmas"] == 36 + private = math.fsum(cell["private_rented_households"] for cell in cells) + # Each cell is rounded to four decimals, so 63 cells stay within 63 * 5e-5. + assert abs(private - totals["private_rented_households"]) < 63 * 5e-5 + assert sum(cell["all_households"] for cell in cells) == totals["all_households"] + + +def test_private_renters_never_exceed_households() -> None: + cells = _resource()["cells"] + by_council: Counter[str] = Counter() + households: Counter[str] = Counter() + for cell in cells: + assert cell["all_households"] > 0 + assert 0 <= cell["private_rented_households"] <= cell["all_households"] + by_council[cell["local_authority"]] += cell["private_rented_households"] + households[cell["local_authority"]] += cell["all_households"] + assert all(0 < by_council[code] < households[code] for code in households) + + +@pytest.mark.parametrize("weight", OVERLAP_WEIGHTS) +def test_shares_sum_to_one_on_each_side(weight: str) -> None: + by_council: dict[str, float] = {} + by_brma: dict[str, float] = {} + for item in scotland_brma_council_overlaps(weight): + assert 0 < item.share_of_council <= 1 + assert 0 < item.share_of_brma <= 1 + by_council[item.local_authority] = ( + by_council.get(item.local_authority, 0.0) + item.share_of_council + ) + by_brma[item.brma] = by_brma.get(item.brma, 0.0) + item.share_of_brma + assert len(by_council) == 32 and len(by_brma) == 18 + assert all(math.isclose(total, 1.0, abs_tol=1e-12) for total in by_council.values()) + assert all(math.isclose(total, 1.0, abs_tol=1e-12) for total in by_brma.values()) + + +def test_both_weights_describe_the_same_geography() -> None: + def main_brma(weight: str) -> dict[str, str]: + best: dict[str, tuple[float, str]] = {} + for item in scotland_brma_council_overlaps(weight): + if item.share_of_council > best.get(item.local_authority, (0.0, ""))[0]: + best[item.local_authority] = (item.share_of_council, item.brma) + return {council: brma for council, (_, brma) in best.items()} + + private, everyone = ( + { + (item.brma, item.local_authority) + for item in scotland_brma_council_overlaps(w) + } + for w in OVERLAP_WEIGHTS + ) + assert private == everyone + assert main_brma("private_rented_households") == main_brma("all_households") + assert scotland_brma_council_blocks( + "private_rented_households" + ) == scotland_brma_council_blocks("all_households") + + +def test_resource_sources_are_the_tool_pins() -> None: + resource = _resource() + assert resource["sources"] == [dict(source) for source in tool.SOURCES] + assert len(resource["sources"]) == 5 + for source in resource["sources"]: + assert len(source["sha256"]) == 64 + assert int(source["sha256"], 16) >= 0 + assert source["url"].startswith("https://") + assert resource["brmas"] == { + brma: {"pipr_area_code": code, "pipr_area_name": name} + for brma, (code, name) in sorted(tool.PIPR_BRMAS.items()) + } + assert sorted(entry["pipr_area_code"] for entry in resource["brmas"].values()) == [ + f"S330000{index:02d}" for index in range(1, 19) + ] + assert resource["method"]["overrides"] == tool.OUTPUT_AREA_OVERRIDES + + +def _mutated(change) -> dict: + payload = json.loads(json.dumps(_resource())) + change(payload) + return payload + + +@pytest.mark.parametrize( + ("change", "message"), + [ + ( + lambda p: p["cells"][0].update(local_authority="E06000001"), + "not a Scottish local authority", + ), + (lambda p: p["cells"].append(dict(p["cells"][0])), "duplicate cell"), + (lambda p: p["cells"][0].update(brma="CENTRAL_LONDON"), "unknown BRMA"), + ( + lambda p: p["cells"][0].update(private_rented_households=-1.0), + "invalid private_rented_households", + ), + (lambda p: p["cells"][0].update(all_households=0), "holds no households"), + ( + lambda p: p["cells"][0].update(private_rented_households=float("nan")), + "invalid private_rented_households", + ), + ( + lambda p: p["totals"].update(private_rented_households=322_000), + "not the declared total", + ), + (lambda p: p.update(schema_version=2), "schema_version"), + ( + lambda p: p.update( + cells=[c for c in p["cells"] if c["local_authority"] != "S12000027"] + ), + "has no cell", + ), + ], +) +def test_validation_refuses_a_broken_table(change, message: str) -> None: + with pytest.raises(ValueError, match=message): + overlap_module.validate_uk_scotland_brma_la_overlap(_mutated(change)) + + +def test_rebuild_from_the_pinned_originals_matches() -> None: + cache = os.environ.get("UK_SCOTLAND_BRMA_LA_CACHE") + uvx = shutil.which("uvx") + if not cache or uvx is None: + pytest.skip("UK_SCOTLAND_BRMA_LA_CACHE names no staged originals") + # geopandas is not a workspace dependency, so the tool runs under uvx. + extras = ("geopandas", "pandas", "openpyxl", "pyogrio") + command = [uvx, *(arg for extra in extras for arg in ("--with", extra))] + command += ["python", str(_REPOSITORY / "tools" / Path(tool.__file__).name)] + result = subprocess.run( + [*command, "--cache", cache, "--check"], + cwd=_REPOSITORY, + capture_output=True, + text=True, + check=False, + ) + assert result.returncode == 0, result.stderr + + +# --- the allocation in the tool ------------------------------------------- + + +@st.composite +def _geographies(draw): + wards = draw(st.integers(min_value=1, max_value=6)) + rows = [] + counts = {} + for index in range(wards): + ward = f"S13{index:06d}" + council = f"S12{draw(st.integers(min_value=0, max_value=2)):06d}" + counts[ward] = draw(st.integers(min_value=0, max_value=5_000)) + for area in range(draw(st.integers(min_value=1, max_value=5))): + rows.append( + { + "output_area": f"S00{index:03d}{area:03d}", + "households": draw(st.integers(min_value=1, max_value=400)), + "ward": ward, + "council": council, + "brma": draw(st.sampled_from(["FIFE", "LOTHIAN", "AYRSHIRES"])), + } + ) + return pd.DataFrame(rows), counts + + +@settings(max_examples=200, deadline=None) +@given(_geographies()) +def test_allocation_conserves_ward_council_and_total(case) -> None: + output_areas, wards = case + cells, totals = tool.overlap_cells( + output_areas, wards, expected_councils=output_areas["council"].nunique() + ) + + def private(**match: str) -> float: + return sum( + cell["private_rented_households"] + for cell in cells + if all(cell[key] == value for key, value in match.items()) + ) + + tolerance = len(cells) * 10**-tool.PRIVATE_RENTED_DECIMALS + assert totals["private_rented_households"] == sum(wards.values()) + assert abs(private() - sum(wards.values())) <= tolerance + assert sum(cell["all_households"] for cell in cells) == int( + output_areas["households"].sum() + ) + assert all(cell["private_rented_households"] >= 0 for cell in cells) + ward_council = output_areas.groupby("ward")["council"].first() + for council in set(output_areas["council"]): + expected = sum( + count for ward, count in wards.items() if ward_council[ward] == council + ) + assert abs(private(local_authority=council) - expected) <= tolerance + # When no ward crosses a BRMA boundary, each BRMA gets its wards' renters + # and no within-ward assumption is in play. + if output_areas.groupby("ward")["brma"].nunique().eq(1).all(): + ward_brma = output_areas.groupby("ward")["brma"].first() + for brma in set(output_areas["brma"]): + expected = sum( + count for ward, count in wards.items() if ward_brma[ward] == brma + ) + assert abs(private(brma=brma) - expected) <= tolerance + + +def test_allocation_refuses_a_ward_in_two_councils() -> None: + output_areas = pd.DataFrame( + [ + { + "output_area": "a", + "households": 5, + "ward": "w", + "council": "x", + "brma": "FIFE", + }, + { + "output_area": "b", + "households": 5, + "ward": "w", + "council": "y", + "brma": "FIFE", + }, + ] + ) + with pytest.raises(SystemExit, match="spans two council areas"): + tool.overlap_cells(output_areas, {"w": 3}) + + +def _ward_export(rows: list[list[str]]) -> str: + header = ["Number of bedrooms", *tool.BEDROOM_BANDS, "Total", ""] + lines = [["SuperWEB2(tm)"], ["Counting: Households"]] + for tenure in (" Owned: Owned outright", *tool.PRIVATE_RENTED_TENURES): + lines += [[tenure], header, ["Electoral Ward 2022", ""]] + lines += ( + rows + if tenure in tool.PRIVATE_RENTED_TENURES + else [[row[0], *["1000"] * 6, ""] for row in rows] + ) + return "\n".join(",".join(f'"{cell}"' for cell in line) for line in lines) + "\n" + + +def test_ward_parser_sums_only_the_private_rented_wafers(tmp_path) -> None: + rows = [ + [f"S13{index:06d}", "1", "2", "3", "4", "5", "999", ""] + for index in range(tool.EXPECTED_WARDS) + ] + (tmp_path / "census_2022_ward2022_tenure_bedrooms.csv").write_text( + _ward_export(rows), encoding="utf-8" + ) + wards = tool.ward_private_rented_households(tmp_path) + # Two private-rented wafers of 1+2+3+4+5; the owner wafer and the + # separately perturbed Total column stay out. + assert set(wards.values()) == {30} + assert len(wards) == tool.EXPECTED_WARDS + + +def test_ward_parser_refuses_a_suppressed_export(tmp_path) -> None: + (tmp_path / "census_2022_ward2022_tenure_bedrooms.csv").write_text( + '"Counting: Households"\n"ERROR","table too sparse"\n', encoding="utf-8" + ) + with pytest.raises(SystemExit, match="suppression"): + tool.ward_private_rented_households(tmp_path) + + +# --- the structure the decision rests on ---------------------------------- + + +def test_brmas_and_councils_do_not_nest() -> None: + overlaps = scotland_brma_council_overlaps() + councils_per_brma = Counter(item.brma for item in overlaps) + brmas_per_council = Counter(item.local_authority for item in overlaps) + assert sum(count > 1 for count in brmas_per_council.values()) == 18 + assert sum(count > 1 for count in councils_per_brma.values()) == 16 + blocks = scotland_brma_council_blocks() + assert sorted((len(brmas), len(councils)) for brmas, councils in blocks) == [ + (1, 3), + (17, 29), + ] + ayrshire = next(block for block in blocks if len(block[1]) == 3) + assert ayrshire == (frozenset({"AYRSHIRES"}), _AYRSHIRE_COUNCILS) + + +def test_four_councils_coincide_with_one_brma() -> None: + assert COINCIDENT_MIN_SHARE == 0.99 + assert dict(scotland_coincident_councils()) == _COINCIDENT + # Loosening the bar adds councils; it never removes one. + looser = scotland_coincident_councils(min_share=0.95) + assert set(_COINCIDENT) < set(looser) + with pytest.raises(ValueError, match="min_share"): + scotland_coincident_councils(min_share=0.5) + + +@settings(max_examples=100, deadline=None) +@given( + st.floats(min_value=0.0, max_value=1.0), + st.floats(min_value=0.0, max_value=1.0), +) +def test_blocks_partition_and_refine_as_slivers_are_dropped(a: float, b: float) -> None: + low, high = sorted((a, b)) + coarse = scotland_brma_council_blocks(min_share=low) + fine = scotland_brma_council_blocks(min_share=high) + for blocks in (coarse, fine): + brmas = [brma for block in blocks for brma in block[0]] + councils = [council for block in blocks for council in block[1]] + assert len(brmas) == len(set(brmas)) == 18 + assert len(councils) == len(set(councils)) == 32 + assert all(block[0] or block[1] for block in blocks) + assert len(fine) >= len(coarse) + for brmas, councils in fine: + assert any(brmas <= big[0] and councils <= big[1] for big in coarse) + + +# --- the rejected rule ---------------------------------------------------- + +_BRMAS = sorted(tool.PIPR_BRMAS) + + +@settings(max_examples=200, deadline=None) +@given( + st.lists( + st.floats(min_value=100.0, max_value=5_000.0), + min_size=len(_BRMAS), + max_size=len(_BRMAS), + ) +) +def test_weighted_mean_rule_gives_councils_their_brma_average(rents) -> None: + brma_rents = dict(zip(_BRMAS, rents, strict=True)) + values = _weighted_mean_rule(brma_rents) + overlaps = scotland_brma_council_overlaps() + + # A council's value is a blend of BRMA averages and nothing else. + assert all(min(rents) - 1e-9 <= v <= max(rents) + 1e-9 for v in values.values()) + # It conserves the rent bill: households x rent agrees on both sides. + council_households: dict[str, float] = {} + bill = 0.0 + for item in overlaps: + council_households[item.local_authority] = ( + council_households.get(item.local_authority, 0.0) + item.households + ) + bill += item.households * brma_rents[item.brma] + assert math.isclose( + sum(council_households[code] * values[code] for code in values), + bill, + rel_tol=1e-9, + ) + # A council wholly inside one BRMA gets that BRMA's rent, unchanged, so + # councils sharing a BRMA cannot differ. + whole = { + item.local_authority: item.brma + for item in overlaps + if item.share_of_council == 1.0 + } + assert len(whole) == 14 + for council, brma in whole.items(): + assert values[council] == brma_rents[brma] + shared = Counter(whole.values()) + assert sum(count for count in shared.values() if count > 1) == 11 + + +def test_rent_dispersion_receipt_backs_the_parity_evidence() -> None: + receipt = json.loads( + ( + _REPOSITORY / "experiments/1090-scotland-brma-la-rent-dispersion.json" + ).read_text(encoding="utf-8") + ) + summary = receipt["summary"] + gaps = [ + abs(row["rule_over_records_mean_pct"]) + for row in receipt["councils"] + if "rule_over_records_mean_pct" in row + ] + assert summary == { + "councils_compared": len(gaps), + "median_abs_gap_pct": sorted(gaps)[len(gaps) // 2], + "max_abs_gap_pct": max(gaps), + "councils_over_5_pct": sum(gap > 5 for gap in gaps), + "councils_over_10_pct": sum(gap > 10 for gap in gaps), + } + assert (summary["councils_compared"], summary["max_abs_gap_pct"]) == (31, 30.9) + assert summary["councils_over_10_pct"] == 9 + by_code = {row["local_authority"]: row for row in receipt["councils"]} + # The coincident councils are the ones the rule gets right. + for council in _COINCIDENT: + assert abs(by_code[council]["rule_over_records_mean_pct"]) <= 0.1 + assert len(receipt["councils"]) == 32 + assert receipt["overlap_resource"] == str(tool.RESOURCE) + + +# --- the concern, the deferral and the fence ------------------------------ + + +def test_parity_concern_states_the_measured_overlap() -> None: + rows = { + row["concern_id"]: row + for row in data_target_parity.build_uk_data_target_parity()["concerns"] + } + concern = rows["cross_grain_private_rent_scotland_brma"] + assert concern["status"] == "routed" + assert "microcosm#1090" in concern["reason"] + totals = _resource()["totals"] + overlaps = scotland_brma_council_overlaps() + big = max(scotland_brma_council_blocks(), key=lambda block: len(block[1])) + for claim in ( + f"{totals['private_rented_households']:,} private-rented households in " + f"{totals['cells']} BRMA x council cells", + f"{sum(c > 1 for c in Counter(i.local_authority for i in overlaps).values())} " + "councils span more than one BRMA", + f"{sum(c > 1 for c in Counter(i.brma for i in overlaps).values())} " + "BRMAs span more than one council", + f"{len(big[1])} councils and {len(big[0])} BRMAs form one connected block", + f"{len(scotland_coincident_councils())} councils coincide with one BRMA at 99%", + "up to 30.9%", + "9 of 31 councils", + ): + assert claim in concern["evidence"], claim + assert ( + "Keep all 32 Scottish LA cells signed deferred" + in (concern["fence"]["verdict_basis"]) + ) + + +def test_committed_deferral_is_the_generator_output() -> None: + membership = _packaged("local_target_reference_membership.json") + generated = { + deferral.reason_id: deferral + for deferral in _area_signed_deferrals( + load_uk_population_contract(), _packaged("local_area_crosswalk.json") + ) + if deferral.target_id == "ons.rent.private_rent" + } + deferral = generated["private_rent_pipr_scotland_brma_grain"] + committed = next( + row + for row in membership["signed_deferrals"] + if row["reason_id"] == deferral.reason_id + ) + assert committed["rationale"] == deferral.rationale + assert ( + committed["area_ids"] == list(deferral.area_ids) == sorted(_scottish_roster()) + ) + candidates = membership["targets"]["ons.rent.private_rent"]["geography_levels"][ + "local_authority" + ]["candidates"] + scottish = [row for row in candidates if row["geography_id"].startswith("S")] + assert len(scottish) == 32 + assert {row["status"] for row in scottish} == {"no_fact_for_area"} + assert {row["signed_rationale"] for row in scottish} == {deferral.rationale} + overlaps = scotland_brma_council_overlaps() + split = sum(c > 1 for c in Counter(i.local_authority for i in overlaps).values()) + assert f"{len(overlaps)} intersections, {split} councils in more than one BRMA" in ( + deferral.rationale + ) + + +def test_the_surface_still_has_no_brma_grain() -> None: + """Tripwire: the concern says BRMA rents have no grain to bind at. + + If a BRMA grain, area type or bridge is ever added, this fails and the + concern's reason and fence need re-adjudicating rather than silently + going stale. + """ + + assert UK_CROSS_GRAIN_GRAIN_PRECEDENCE == ( + "country", + "region", + "constituency", + "la", + ) + assert AREA_TYPES == ("constituency", "la") + assert not any( + "ons.rent." in side + for bridge in UK_CROSS_GRAIN_BRIDGES + for side in (*bridge.higher_target_ids, bridge.lower_side) + ) + target = next( + row + for row in load_uk_population_contract()["targets"] + if row["target_id"] == "ons.rent.private_rent" + ) + assert target["geography_levels"] == ["local_authority"] diff --git a/tools/build_uk_scotland_brma_la_overlap.py b/tools/build_uk_scotland_brma_la_overlap.py new file mode 100644 index 000000000..f2710c91c --- /dev/null +++ b/tools/build_uk_scotland_brma_la_overlap.py @@ -0,0 +1,485 @@ +#!/usr/bin/env python +"""Rebuild the Scottish BRMA x council-area overlap from official originals. + +ONS's Price Index of Private Rents publishes Scottish rent levels for the 18 +Broad Rental Market Areas, not for the 32 council areas. This tool measures +how the two geographies overlap, in households, so the question "can a BRMA +rent become a council target?" is answered from sources rather than assumed +(microcosm#1090). + +Every 2022 census output area is placed in one BRMA by its population-weighted +centroid in the Scottish Government's BRMA polygons, checked against Rent +Service Scotland's postcode lookup. Output areas nest in 2022 electoral wards, +and wards nest in council areas. Two weights come out per (BRMA, council): + +- ``private_rented_households``: Census 2022 ward private-rented households + (private landlord or letting agency, plus other private rented), each ward + split across BRMAs by the share of its output-area households in each. The + ward is the finest level at which the census releases this tenure split, so + inside a split ward private renters are assumed to be spread like all + households. +- ``all_households``: output-area household counts summed directly, with no + within-ward assumption. + +The five originals are pinned by sha256 and read from ``--cache``; nothing is +downloaded. Four have stable URLs (``SOURCES``); the ward table is a manual +export from the census table builder. The geography step needs geopandas, +which is not a Microcosm dependency: + + uvx --with geopandas --with pandas --with openpyxl --with pyogrio \\ + python tools/build_uk_scotland_brma_la_overlap.py --cache + +``--check`` compares the rebuild with the committed resource instead of +writing it. +""" + +from __future__ import annotations + +import argparse +import csv +import hashlib +import json +import re +import sys +import zipfile +from pathlib import Path +from typing import Any + +RESOURCE = Path( + "packages/microcosm-build/src/microcosm/build/uk/scotland_brma_la_overlap.json" +) +SCHEMA_VERSION = 1 + +SOURCES: tuple[dict[str, Any], ...] = ( + { + "id": "scot_brma_boundaries_2009", + "publisher": "Scottish Government", + "title": "Broad Rental Market Areas (2009 boundaries, revised 28 July 2015)", + "url": "https://maps.gov.scot/ATOM/shapefiles/SG_BroadRentalMarketAreas_2009.zip", + "landing_page": ( + "https://www.data.gov.uk/dataset/39d08296-874a-4604-9ca4-6fe807aee4a8/" + "broad-rental-market-areas" + ), + "sha256": "4d121236a35ce896edeb663a7c2c390053c0deebe0cc149c84696bbfd4bddec0", + "licence": "Open Government Licence v3.0", + "filename": "scot_brma_boundaries_2009.zip", + }, + { + "id": "scot_oa2022_population_weighted_centroids", + "publisher": "National Records of Scotland", + "title": "2022 Census Output Area Population Weighted Centroids", + "url": "https://www.nrscotland.gov.uk/media/gm4dvdsv/output-area-2022-pwc.zip", + "landing_page": ( + "https://www.nrscotland.gov.uk/publications/2022-census-geography-products/" + ), + "sha256": "509d7d937c8cc1477259dcd8e4e0e61240f5cc782721a79e9c279bd589e0ec0c", + "licence": "Open Government Licence v3.0", + "filename": "scot_oa2022_population_weighted_centroids.zip", + }, + { + "id": "scot_census2022_geography_index", + "publisher": "National Records of Scotland", + "title": "2022 Census Index: Postcode to OA and OA to higher areas", + "url": "https://www.nrscotland.gov.uk/media/utrbt5ze/census_2022_index.zip", + "landing_page": ( + "https://www.nrscotland.gov.uk/publications/2022-census-geography-products/" + ), + "sha256": "0e4096a321cba2318dc5d2e35c197bf815ac333aebdf4cb0ddce82112b57a261", + "licence": "Open Government Licence v3.0", + "filename": "scot_census2022_geography_index.zip", + }, + { + "id": "scot_foi_postcodes_brma_2023", + "publisher": "Scottish Government / Rent Service Scotland", + "title": "FOI 202300368850: postcode to BRMA list held in LHA Direct", + "url": ( + "https://www.gov.scot/binaries/content/documents/govscot/publications/" + "foi-eir-release/2023/11/foi-202300368850/documents/" + "foi-202300368850---information-released---postcodes/" + "foi-202300368850---information-released---postcodes/" + "govscot%3Adocument/" + "FOI%2B202300368850%2B-%2BInformation%2BReleased%2B-%2BPostcodes.xlsx" + ), + "landing_page": "https://www.gov.scot/publications/foi-202300368850/", + "sha256": "50366fea2b87bd6b697e7aa8736fbc870838b5e327458213294f31c3de7ede5e", + "licence": "Open Government Licence v3.0", + "filename": "scot_foi_postcodes_brma_2023.xlsx", + }, + { + "id": "nrs_census_2022_ward2022_tenure_bedrooms", + "publisher": "National Records of Scotland", + "title": ( + "Scotland's Census 2022: household tenure by electoral ward 2022 by " + "number of bedrooms (dataset household2022ftb)" + ), + "url": "https://www.scotlandscensus.gov.uk/webapi/downloadTable?type=csv", + "landing_page": "https://www.scotlandscensus.gov.uk/webapi/home", + "sha256": "03d6157d34bd3df44e79ded1c03510954dee55e9e05d3c276b26ad463a565677", + "licence": "Open Government Licence v3.0", + "filename": "census_2022_ward2022_tenure_bedrooms.csv", + "manual": ( + "Guest session in the census table builder: all 355 Electoral Ward " + "2022 codes as rows, Number of bedrooms (One to Five or more, and " + "Total) as columns, Household tenure (all eight values and Total) " + "as wafers, Counting: Households; download as CSV unmodified." + ), + }, +) + +# The Price Index of Private Rents' own codes and names for the 18 BRMAs +# (ONS monthly price statistics, July 2026 edition, Table 1). +PIPR_BRMAS: dict[str, tuple[str, str]] = { + "ABERDEEN_AND_SHIRE": ("S33000001", "Aberdeen and Shire"), + "ARGYLL_AND_BUTE": ("S33000002", "Argyll and Bute"), + "AYRSHIRES": ("S33000003", "Ayrshires"), + "DUMFRIES_AND_GALLOWAY": ("S33000004", "Dumfries and Galloway"), + "DUNDEE_AND_ANGUS": ("S33000005", "Dundee and Angus"), + "EAST_DUNBARTONSHIRE": ("S33000006", "East Dunbartonshire"), + "FIFE": ("S33000007", "Fife"), + "FORTH_VALLEY": ("S33000008", "Forth Valley"), + "GREATER_GLASGOW": ("S33000009", "Greater Glasgow"), + "HIGHLAND_AND_ISLANDS": ("S33000010", "Highland and Islands"), + "LOTHIAN": ("S33000011", "Lothian"), + "NORTH_LANARKSHIRE": ("S33000012", "North Lanarkshire"), + "PERTH_AND_KINROSS": ("S33000013", "Perth and Kinross"), + "RENFREWSHIRE_AND_INVERCLYDE": ("S33000014", "Renfrewshire/Inverclyde"), + "SCOTTISH_BORDERS": ("S33000015", "Scottish Borders"), + "SOUTH_LANARKSHIRE": ("S33000016", "South Lanarkshire"), + "WEST_DUNBARTONSHIRE": ("S33000017", "West Dunbartonshire"), + "WEST_LOTHIAN": ("S33000018", "West Lothian"), +} + +# Two Balloch output areas whose master postcodes the lookup lists under two +# BRMAs, neither of them the older polygon's answer. LHA Direct's search for +# West Dunbartonshire council returned the Argyll and Bute and West +# Dunbartonshire BRMAs on 2026-10-02, and each postcode's candidates meet that +# pair in West Dunbartonshire only. The reply page is evidence, not an input. +OUTPUT_AREA_OVERRIDES = { + "S00179382": "WEST_DUNBARTONSHIRE", # G83 8NQ + "S00179383": "WEST_DUNBARTONSHIRE", # G83 8SD +} +OVERRIDE_COUNCIL = "S12000039" + +PRIVATE_RENTED_TENURES = ( + "Private rented: Private landlord or letting agency", + "Private rented: Other", +) +BEDROOM_BANDS = ( + "One bedroom", + "Two bedrooms", + "Three bedrooms", + "Four bedrooms", + "Five or more bedrooms", +) +EXPECTED_OUTPUT_AREAS = 46_363 +EXPECTED_WARDS = 355 +EXPECTED_COUNCILS = 32 +#: Decimal places kept for the fractional private-rented cells: enough to +#: reconcile to the census total, few enough to be stable across platforms. +PRIVATE_RENTED_DECIMALS = 4 + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--cache", type=Path, required=True) + parser.add_argument("--out", type=Path, default=RESOURCE) + parser.add_argument("--check", action="store_true") + args = parser.parse_args(argv) + + verify_cache(args.cache) + payload = build_resource(args.cache) + text = json.dumps(payload, indent=2) + "\n" + if args.check: + if args.out.read_text(encoding="utf-8") != text: + print(f"stale: {args.out} differs from the rebuild", file=sys.stderr) + return 1 + print(f"current: {args.out}") + return 0 + args.out.write_text(text, encoding="utf-8") + print(f"wrote {args.out}: {len(payload['cells'])} cells") + return 0 + + +def verify_cache(cache: Path) -> None: + for source in SOURCES: + path = cache / source["filename"] + if not path.is_file(): + raise SystemExit(f"missing original {path} ({source['url']})") + digest = hashlib.sha256(path.read_bytes()).hexdigest() + if digest != source["sha256"]: + raise SystemExit( + f"{path} has sha256 {digest}; the pin is {source['sha256']}." + ) + + +def build_resource(cache: Path) -> dict[str, Any]: + output_areas = output_area_assignments(cache) + wards = ward_private_rented_households(cache) + cells, totals = overlap_cells(output_areas, wards) + return { + "schema_version": SCHEMA_VERSION, + "resource": "scotland_brma_la_overlap", + "description": ( + "Households in each Scottish Broad Rental Market Area x council " + "area intersection, from Scotland's Census 2022 and the official " + "BRMA geography. Rebuilt by tools/build_uk_scotland_brma_la_overlap.py." + ), + "weights": { + "private_rented_households": ( + "Census 2022 ward households renting from a private landlord " + "or letting agency or other private renting, summed over the " + "published bedroom bands and split across BRMAs by each " + "ward's share of output-area households. The ward is the " + "finest level with this tenure split, so within a ward that " + "crosses a BRMA boundary private renters are assumed to be " + "spread like all households." + ), + "all_households": ( + "NRS 2022 output-area household counts summed per " + "intersection; each output area has one council and one BRMA." + ), + }, + "method": { + "output_area_to_brma": ( + "Population-weighted centroid within the Scottish Government " + "BRMA polygon. A centroid outside every polygon takes its " + "master postcode's single BRMA in Rent Service Scotland's " + "postcode lookup. Where the lookup contradicts the polygon " + "the newer lookup wins when it names one BRMA; the two output " + "areas it leaves ambiguous are listed under overrides." + ), + "output_area_to_council": ( + "NRS Census 2022 index (OA_TO_HIGHER_AREAS.csv): 2022 " + "electoral ward and 2019 council area of every output area." + ), + "overrides": dict(sorted(OUTPUT_AREA_OVERRIDES.items())), + "private_rented_decimals": PRIVATE_RENTED_DECIMALS, + }, + "sources": [dict(source) for source in SOURCES], + "brmas": { + brma: {"pipr_area_code": code, "pipr_area_name": name} + for brma, (code, name) in sorted(PIPR_BRMAS.items()) + }, + "totals": totals, + "cells": cells, + } + + +def output_area_assignments(cache: Path): + """One row per 2022 output area: households, ward, council and BRMA.""" + + import geopandas as gpd + import pandas as pd + + brmas = gpd.read_file(f"zip://{cache / 'scot_brma_boundaries_2009.zip'}") + if len(brmas) != len(PIPR_BRMAS) or not brmas.geometry.is_valid.all(): + raise SystemExit("expected 18 valid BRMA polygons") + brmas["brma"] = brmas["BRMAName"].map(_brma_key) + if set(brmas["brma"]) != set(PIPR_BRMAS): + raise SystemExit(f"unexpected BRMA names: {sorted(set(brmas['brma']))}") + + centroids = cache / "scot_oa2022_population_weighted_centroids.zip" + oas = gpd.read_file( + f"zip://{centroids}!OutputArea2022_PWC/OutputArea2022_PWC.shp" + ).rename(columns={"code": "output_area", "HHcount": "households"}) + oas = oas.to_crs(brmas.crs) + if len(oas) != EXPECTED_OUTPUT_AREAS or oas["output_area"].duplicated().any(): + raise SystemExit("expected 46,363 unique output areas") + if oas["households"].isna().any() or (oas["households"] < 0).any(): + raise SystemExit("invalid output-area household counts") + + polygons = brmas[["brma", "geometry"]] + matches = gpd.sjoin(oas, polygons, how="left", predicate="within") + outside = matches.loc[matches["brma"].isna(), "output_area"] + boundary = gpd.sjoin( + oas[oas["output_area"].isin(outside)], + polygons, + how="left", + predicate="intersects", + ) + matches = pd.concat( + [matches[~matches["output_area"].isin(outside)], boundary], ignore_index=True + ) + hits = matches.groupby("output_area")["brma"].count() + single = matches[matches["output_area"].map(hits).eq(1)] + oas = oas.drop(columns="geometry").merge( + single[["output_area", "brma"]], + on="output_area", + how="left", + validate="one_to_one", + ) + polygon_brma = oas["brma"].copy() + + lookup = pd.read_excel( + cache / "scot_foi_postcodes_brma_2023.xlsx", usecols=[0, 1], dtype=str + ) + lookup.columns = ["brma", "postcode"] + lookup["postcode"] = lookup["postcode"].map(_postcode_key) + lookup["brma"] = lookup["brma"].map(_brma_key) + candidates = lookup.groupby("postcode")["brma"].agg(set) + oas["candidates"] = oas["masterpc"].map(_postcode_key).map(candidates) + unique = oas["candidates"].map( + lambda found: ( + next(iter(found)) if isinstance(found, set) and len(found) == 1 else None + ) + ) + oas["brma"] = oas["brma"].fillna(unique) + if oas["brma"].isna().any(): + raise SystemExit("an output area outside the polygons has no single BRMA") + + with zipfile.ZipFile(cache / "scot_census2022_geography_index.zip") as archive: + with archive.open("Census_2022_Index/OA_TO_HIGHER_AREAS.csv") as stream: + higher = pd.read_csv( + stream, + encoding="utf-8-sig", + usecols=["OA2022", "EW2022", "CA2019"], + ) + oas = oas.merge( + higher, left_on="output_area", right_on="OA2022", validate="one_to_one" + ) + if len(oas) != EXPECTED_OUTPUT_AREAS: + raise SystemExit("the census index does not cover every output area") + + conflict = pd.Series( + [ + isinstance(found, set) and pd.notna(polygon) and polygon not in found + for polygon, found in zip(polygon_brma, oas["candidates"], strict=True) + ], + index=oas.index, + ) + corrected = conflict & unique.notna() + oas.loc[corrected, "brma"] = unique[corrected] + unresolved = conflict & unique.isna() + if set(oas.loc[unresolved, "output_area"]) != set(OUTPUT_AREA_OVERRIDES): + raise SystemExit("the set of ambiguous output areas changed") + if not oas.loc[unresolved, "CA2019"].eq(OVERRIDE_COUNCIL).all(): + raise SystemExit("an overridden output area left West Dunbartonshire") + oas.loc[unresolved, "brma"] = oas.loc[unresolved, "output_area"].map( + OUTPUT_AREA_OVERRIDES + ) + disagree = [ + area + for area, brma, found in zip( + oas["output_area"], oas["brma"], oas["candidates"], strict=True + ) + if isinstance(found, set) and brma not in found + ] + if disagree: + raise SystemExit(f"assignments contradict the postcode lookup: {disagree}") + return oas[["output_area", "households", "EW2022", "CA2019", "brma"]].rename( + columns={"EW2022": "ward", "CA2019": "council"} + ) + + +def ward_private_rented_households(cache: Path) -> dict[str, int]: + """Ward private-rented households from the unmodified census export.""" + + path = cache / "census_2022_ward2022_tenure_bedrooms.csv" + with path.open(encoding="utf-8-sig", newline="") as stream: + rows = list(csv.reader(stream)) + if ["Counting: Households"] not in rows: + raise SystemExit("the ward export does not count households") + if any(row and row[0].strip() == "ERROR" for row in rows): + raise SystemExit("the ward export carries suppression errors") + tenure = None + headers: list[str] | None = None + wards: dict[str, int] = {} + seen: dict[str, set[str]] = {} + for row in rows: + if not row: + continue + label = row[0].strip() + if len(row) == 1: + # Each wafer opens with a one-cell tenure label; resetting on + # every one keeps other tenures out of the private-rented sum. + tenure = label if label in PRIVATE_RENTED_TENURES else None + elif label == "Number of bedrooms": + headers = [header.strip() for header in row[1:]] + elif tenure is not None and re.fullmatch(r"S13\d{6}", label): + if headers is None: + raise SystemExit("ward rows precede the bedroom header") + values = dict(zip(headers, row[1:], strict=True)) + if label in seen.setdefault(tenure, set()): + raise SystemExit(f"ward {label} repeats under {tenure!r}") + seen[tenure].add(label) + wards[label] = wards.get(label, 0) + sum( + int(values[band]) for band in BEDROOM_BANDS + ) + if set(seen) != set(PRIVATE_RENTED_TENURES) or any( + len(found) != EXPECTED_WARDS for found in seen.values() + ): + raise SystemExit("expected both private-rented wafers for all 355 wards") + return wards + + +def overlap_cells( + output_areas, + wards: dict[str, int], + *, + expected_councils: int = EXPECTED_COUNCILS, +): + """Split each ward's private renters across BRMAs and sum by council.""" + + councils_per_ward = output_areas.groupby("ward")["council"].nunique() + if not councils_per_ward.eq(1).all(): + raise SystemExit("a 2022 ward spans two council areas") + if set(output_areas["ward"]) != set(wards): + raise SystemExit("the census wards and the geography wards differ") + if output_areas["council"].nunique() != expected_councils: + raise SystemExit(f"expected {expected_councils} council areas") + + ward_households = output_areas.groupby("ward")["households"].sum() + if not ward_households.gt(0).all(): + raise SystemExit("a ward has no output-area households") + by_ward = output_areas.groupby(["ward", "council", "brma"], as_index=False)[ + "households" + ].sum() + by_ward["share"] = by_ward["households"] / by_ward["ward"].map(ward_households) + by_ward["private_rented"] = by_ward["share"] * by_ward["ward"].map(wards) + private = by_ward.groupby(["council", "brma"])["private_rented"].sum() + everyone = output_areas.groupby(["council", "brma"])["households"].sum() + + private_total = int(sum(wards.values())) + if abs(float(private.sum()) - private_total) > 1e-6: + raise SystemExit("the BRMA split changed the census private-rented total") + + cells = [ + { + "brma": brma, + "local_authority": council, + "private_rented_households": round( + float(private[(council, brma)]), PRIVATE_RENTED_DECIMALS + ), + "all_households": int(everyone[(council, brma)]), + } + for council, brma in sorted(everyone.index) + ] + totals = { + "cells": len(cells), + "output_areas": int(len(output_areas)), + "wards": int(len(wards)), + "wards_split_across_brmas": int( + by_ward.groupby("ward")["brma"].nunique().gt(1).sum() + ), + "private_rented_households": private_total, + "all_households": int(output_areas["households"].sum()), + } + return cells, totals + + +def _brma_key(name: str) -> str: + key = re.sub(r"\s*/\s*", " AND ", str(name).strip()).upper().replace(" ", "_") + return "AYRSHIRES" if key == "AYRSHIRE" else key + + +def _postcode_key(value: str) -> str: + # NRS suffixes a split postcode's records with A or B; the lookup does not. + key = re.sub(r"\s+", "", str(value).upper()) + match = re.fullmatch(r"([A-Z]{1,2}[0-9][0-9A-Z]?[0-9][A-Z]{2})[A-Z]?", key) + if not match: + raise SystemExit(f"unrecognised postcode {value!r}") + return match.group(1) + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tools/generate_uk_local_target_references.py b/tools/generate_uk_local_target_references.py index e264c9775..49d1d2708 100644 --- a/tools/generate_uk_local_target_references.py +++ b/tools/generate_uk_local_target_references.py @@ -454,7 +454,7 @@ def add( target_id="ons.rent.private_rent", geography_level="local_authority", reason_id="private_rent_pipr_scotland_brma_grain", - rationale="The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. CrossGrainBridge declares target identity but cannot translate overlapping BRMA geographies to LAs, and no signed BRMA-to-LA crosswalk is present, so allocation is forbidden and all 32 cells remain deferred.", + rationale="The pinned PIPR monthly series carries 18 Scottish BRMA rows at statistical_scope grain and no Scottish LA rows for the 32-authority crosswalk. The BRMAs overlap the councils (uk/scotland_brma_la_overlap.json: 63 intersections, 18 councils in more than one BRMA), so CrossGrainBridge, which equates totals over a partition, cannot translate them, and a household-weighted mean of BRMA rents would give every council inside a BRMA that BRMA's average. Allocation is forbidden and all 32 cells remain deferred until PIPR binds at BRMA grain (microcosm#1090).", area_ids=scottish_local_authorities, ) add(