Use Rent Service Scotland's own lists of rents for Scottish percentile ratios

[MaxGhenis]

Fixes #2036

Stacked on #2022 (lha-published-rates, head aef9c24). Retarget to main after #2022 merges.

What changes

lha_list_of_rents.csv.gz had Scottish lists that were copies of English BRMAs' lists (#2036). This replaces them with Rent Service Scotland's own lists, released by the Scottish Government under FOI 202200303624.

• Which lists. The list for the April Y determination is the workbook's sheet for the twelve months to September Y-1. The file's year 2019 is sheet 2018 and year 2020 is sheet 2019.
• The ratios use April 2020's list. _percentile_ratios reads the file's latest year, 2020, for England and now for Scotland.
• Wales and Northern Ireland keep the median English ratio from aef9c24, because their lists are still copies. The English median no longer includes Scottish BRMAs.
• What's kept. Each rent with its BRMA and LHA category. Postcode districts and property types are dropped, along with two rents at the end of sheet 2019 that name no BRMA.
• Other rows are byte-for-byte unchanged in content: 1,124,082 English, Welsh and NI rows.

How well the FOI lists reproduce the published 30th percentile

I applied statutory_percentile (Sch 3B para 2(8)) with half-up penny rounding to each FOI sheet, and compared the result with the published 30th percentile for the following April.

Determination	Sheet	Exact to the penny	Within 1%	Largest gap
April 2017	2016	88 of 90	90	0.02%
April 2018	2017	89 of 90	90	0.6%
April 2019	2018	87 of 90	89	1.6% (Greater Glasgow, shared)
April 2020	2019	86 of 90	88	3.7% (Dumfries and Galloway, 4-bed, 45 rents)
April 2021	2020	88 of 90	89	1.5%
April 2022	2021	84 of 90	88	5.0%

Over all six years, 522 of 540 cells match to the penny. I have not established why the other 18 cells differ.

Effect

The baseline is unchanged, because a ratio is 1 at the 30th percentile. A percentile reform now scales Scottish rates by Scotland's own rent distributions. At the 50th percentile:

	Ratio (A, B, C, D, E)
English median (aef9c24), every Scottish BRMA	1.085, 1.074, 1.080, 1.087, 1.111
Lothian, own list	1.092, 1.051, 1.091, 1.182, 1.168
Greater Glasgow, own list	1.143, 1.162, 1.154, 1.267, 1.143

Across all 90 Scottish cells, the ratio ranges from 1.010 to 1.267, with a mean of 1.111.

Invariants and tests

1. Not a copy. No Scottish (year, BRMA, category) list is identical to any English list, and there are 2 × 18 × 5 of them. Test: test_scottish_lists_are_rent_service_scotlands_own.
2. Reproduces the published percentile. At least 170 of the 180 Scottish cells reproduce the published 30th percentile to the penny; it is 173 today. Same test.
3. Scottish ratios come from Scotland's own lists. Test: test_scottish_percentile_ratios_use_scotlands_own_lists.
4. Welsh and NI BRMAs get the English median, and every ratio is at least 1 at the 50th percentile. Test: test_welsh_and_ni_percentile_ratios_use_the_english_median, which replaces the devolved test from aef9c24.
5. The build is reproducible. scripts/build_scottish_list_of_rents.py fetches the workbook, checks its pinned sha256 and rebuilds the CSV. Starting from aef9c24's file, it reproduces the committed file exactly (sha256 3a083649…).

Review

An independent Opus review (review/REVIEW_2037_r1.md) approved the PR after reading the code; it could not run anything. I applied its findings and ran its four checks:

• Latest year per region. _sorted_list_of_rents takes the latest year per region, so a newer sheet for one nation can't drop the others.
• One shared constant. NON_ENGLISH_REGIONS defines the English set for the code and tests alike.
• Test median. The Welsh/NI test computes the English median from the lists, the same way the code does.
• Pinned ratios. The Scottish ratio test pins Greater Glasgow D (1.267) and Lothian C (1.091). Both Scottish tests fail on aef9c24's file and pass on this one.
• Missing BRMAs. The build asserts the number of BRMA-less rents per sheet.
• Rounding. The FOI rents are already whole pennies (largest difference from 2 dp is 0), so rounding plays no part in the mismatches.
• Reproducibility. Rebuilding from aef9c24's original file reproduces the committed file exactly (sha256 3a083649…), with the non-Scottish rows identical.

Commands run

• uv run --with openpyxl python scripts/build_scottish_list_of_rents.py --cache ~/.cache/lha, starting from aef9c24's file: the output is identical to the committed file.
• pytest policyengine_uk/tests/test_lha_published_rates.py: 75 passed.
• ruff format and ruff check on the changed files: clean.

🤖 Generated with Claude Code