diff --git a/changelog.d/brma-rent-conditioned.changed.md b/changelog.d/brma-rent-conditioned.changed.md new file mode 100644 index 00000000..c62935e1 --- /dev/null +++ b/changelog.d/brma-rent-conditioned.changed.md @@ -0,0 +1 @@ +- Draw private renters' Broad Rental Market Areas given their reported rent and bedrooms, using the LHA lists of rents (England and Wales), Scotland's lists with published 30th percentiles (Scotland) and published 30th percentiles (Northern Ireland), so that rents and Local Housing Allowance rates move together. Other households keep the census-weighted draw. New table `brma_private_rents.csv`, rebuilt by `tools/brma_rents/`. diff --git a/docs/imputations.md b/docs/imputations.md index 75fd1a56..377add2c 100644 --- a/docs/imputations.md +++ b/docs/imputations.md @@ -213,14 +213,25 @@ Assigns student loan plan type based on age and reported repayments. ## Broad Rental Market Area Assignment -**Source:** Census private-rented households by BRMA and bedrooms (not QRF; applied when the base FRS dataset is built) +**Source:** Census private-rented households by BRMA and bedrooms, and the LHA lists of rents (not QRF; applied when the base FRS dataset is built) The FRS identifies only the region, but Local Housing Allowance rates vary by Broad Rental Market Area (BRMA). `datasets/brma.py` draws each benefit unit's BRMA within its region, in proportion to the private-rented households in each BRMA: - shared-accommodation and one-bedroom LHA categories use one-bedroom homes; - the two-, three- and four-or-more-bedroom categories use homes with that many bedrooms; - Northern Ireland's census has no bedrooms, so its weights are the same for every category. -A household takes one of its benefit units' BRMAs, chosen at random. Sources, method and validation are in `storage/BRMA_DATA_SOURCES.md`. +A household takes one of its benefit units' BRMAs, chosen at random. + +Private renters who report a rent are then redrawn so that their rent and their BRMA's rents move together: + +P(BRMA | region, bedrooms, rent) ∝ census private-rented households(BRMA, bedrooms) × density of the rent on the BRMA's list of rents for homes with that many bedrooms. + +- The lists of rents are summarised in `storage/brma_private_rents.csv` as a median and a spread of log rents per BRMA and LHA category, in the prices of the survey year. +- The home's bedrooms (FRS `bedroom6`) select both the census band and the list category, because both describe the home. The household's LHA category describes its entitlement, which can be smaller than its home. +- Reported rents are not list rents. A model fitted to the survey's own private renters allows for a shift in each region, extra noise, and a share of households paying well below the market. A household with such a rent keeps roughly the census shares. +- Other households keep the draw above, unchanged. + +Sources, method and validation are in `storage/BRMA_DATA_SOURCES.md`. --- diff --git a/policyengine_uk_data/datasets/brma.py b/policyengine_uk_data/datasets/brma.py index 9aec9bc6..1919920e 100644 --- a/policyengine_uk_data/datasets/brma.py +++ b/policyengine_uk_data/datasets/brma.py @@ -14,8 +14,18 @@ These weights replace the row counts of the 2019-20 LHA list of rents. Its Scottish, Welsh and Northern Ireland rows were copies of English BRMAs' lists, so their counts said nothing about those nations' rental markets. + +Private renters who report a rent are then redrawn given that rent +(``assign_private_renter_brmas``). The probability of each BRMA is the census +share of private-rented homes with the household's number of bedrooms, times +the density of its rent on the BRMA's list of rents for homes of that size +(``storage/brma_private_rents.csv``). Reported rents differ from the rents on +the rent officers' lists: ``ReportedRentModel`` measures that difference from +the survey itself. Without this step a +household's rent says nothing about its Local Housing Allowance rate. """ +from dataclasses import dataclass from pathlib import Path import numpy as np @@ -104,3 +114,280 @@ def pick_household_brmas( return units.groupby("household_id").brma.aggregate( lambda x: x.sample(n=1, random_state=rng).iloc[0] ) + + +BRMA_RENTS_PATH = STORAGE_FOLDER / "brma_private_rents.csv" + +# Census bedroom band of a home, and the list-of-rents category that covers +# self-contained homes of that size. +BEDROOM_BAND_CATEGORY = {"1": "B", "2": "C", "3": "D", "4+": "E"} + + +def bedroom_band(bedrooms: np.ndarray) -> np.ndarray: + """Census bedroom band (``1``, ``2``, ``3``, ``4+``) for a number of bedrooms.""" + bedrooms = np.asarray(bedrooms, dtype=float) + if not (np.isfinite(bedrooms) & (bedrooms >= 1)).all(): + raise ValueError("Every home needs at least one bedroom.") + return np.where( + bedrooms >= 4, "4+", np.minimum(bedrooms, 3).astype(int).astype(str) + ) + + +@dataclass(frozen=True) +class ReportedRentModel: + """How households' reported rents relate to their BRMA's list of rents. + + The log of a reported rent is the log of a rent on the list for the + household's BRMA and bedrooms, plus ``region_shift`` for its region, plus + noise. A share ``below_market_share`` of households pay + ``below_market_discount`` log points less than that, with noise standard + deviation ``below_market_noise``; the rest have noise standard deviation + ``noise``. + """ + + region_shift: dict[str, float] + noise: float + below_market_share: float + below_market_discount: float + below_market_noise: float + + +def _rent_cells( + region, bedrooms, households: pd.DataFrame | None, rents: pd.DataFrame | None +): + """Each household's BRMA prior, log median rent and log rent spread. + + Returns the BRMA names and three arrays of shape (households, BRMAs). The + prior is the share of its region's private-rented households with its + number of bedrooms that live in each BRMA. Outside the prior's support the + median is 0 and the spread 1; those cells never contribute. + """ + if households is None: + households = pd.read_csv(BRMA_HOUSEHOLDS_PATH, dtype={"bedrooms": str}) + if rents is None: + rents = pd.read_csv(BRMA_RENTS_PATH) + bands = pd.Series(list(BEDROOM_BAND_CATEGORY), name="bedrooms") + any_band = households[households.bedrooms == "all"].drop(columns="bedrooms") + counts = pd.concat( + [households[households.bedrooms != "all"], any_band.merge(bands, how="cross")] + ).pivot_table( + index=["region", "bedrooms"], + columns="brma", + values="households", + aggfunc="sum", + fill_value=0, + ) + if hasattr(region, "decode_to_str"): # EnumArray of region codes + region = region.decode_to_str() + region = np.asarray(region).astype(str) + band = bedroom_band(bedrooms) + cell = counts.index.get_indexer(pd.MultiIndex.from_arrays([region, band])) + if (cell < 0).any(): + missing = sorted(set(zip(region[cell < 0], band[cell < 0]))) + raise ValueError(f"No BRMA weights for region × bedrooms cells {missing}.") + category = counts.index.get_level_values("bedrooms").map(BEDROOM_BAND_CATEGORY) + by_category = rents.set_index(["lha_category", "brma"]) + log_median = np.log( + by_category.median_weekly_rent.unstack().reindex( + category, columns=counts.columns + ) + ).to_numpy() + log_sd = ( + by_category.log_sd.unstack() + .reindex(category, columns=counts.columns) + .to_numpy() + ) + supported = counts.to_numpy() > 0 + unlisted = supported & ~(np.isfinite(log_median) & (log_sd > 0)) + if unlisted[cell].any(): + rows, columns = np.nonzero( + unlisted & np.isin(np.arange(len(counts)), cell)[:, None] + ) + missing = sorted(set(zip(category[rows], counts.columns[columns]))) + raise ValueError(f"No list of rents for LHA category × BRMA cells {missing}.") + prior = counts.to_numpy() / counts.to_numpy().sum(axis=1, keepdims=True) + return ( + counts.columns.to_numpy(), + prior[cell], + np.where(supported, log_median, 0.0)[cell], + np.where(supported, log_sd, 1.0)[cell], + ) + + +def _log_rent_density(log_rent, shift, log_median, log_sd, shape) -> np.ndarray: + """Log density of each reported rent in each BRMA, up to a constant.""" + noise, share, discount, wide_noise = shape + error = log_rent[:, None] - shift[:, None] - log_median + + def log_normal(error, variance): + return -0.5 * error**2 / variance - 0.5 * np.log(variance) + + with np.errstate(divide="ignore"): + return np.logaddexp( + np.log1p(-share) + log_normal(error, log_sd**2 + noise**2), + np.log(share) + log_normal(error + discount, log_sd**2 + wide_noise**2), + ) + + +def _shifts(region, model: ReportedRentModel) -> np.ndarray: + if hasattr(region, "decode_to_str"): + region = region.decode_to_str() + region = np.asarray(region).astype(str) + missing = sorted(set(region) - set(model.region_shift)) + if missing: + raise ValueError(f"No reported-rent shift for regions {missing}.") + return np.array([model.region_shift[r] for r in region], dtype=float) + + +def brma_probabilities( + region: np.ndarray, + bedrooms: np.ndarray, + weekly_rent: np.ndarray, + model: ReportedRentModel, + households: pd.DataFrame | None = None, + rents: pd.DataFrame | None = None, +) -> tuple[np.ndarray, np.ndarray]: + """Each private-renting household's probability of living in each BRMA. + + P(BRMA | region, bedrooms, rent) is proportional to the census + private-rented households in the BRMA with that many bedrooms, times the + density of the household's rent under ``model`` and the BRMA's list of + rents for homes of that size. + + Args: + region: Region name of each household. + bedrooms: Number of bedrooms in each household's home (at least 1). + weekly_rent: Weekly rent each household reports (positive). + model: Output of ``fit_reported_rent_model``. + households: Census table (``BRMA_HOUSEHOLDS_PATH``) if omitted. + rents: List-of-rents table (``BRMA_RENTS_PATH``) if omitted. + + Returns: + BRMA names, and probabilities of shape (households, BRMAs). Each row + sums to 1 and is zero outside the household's region × bedrooms cell. + + Raises: + ValueError: If a rent is not positive, or a household's region, + region × bedrooms cell or a BRMA in that cell has no data. + """ + weekly_rent = np.asarray(weekly_rent, dtype=float) + if not (np.isfinite(weekly_rent) & (weekly_rent > 0)).all(): + raise ValueError("Every household needs a positive weekly rent.") + brmas, prior, log_median, log_sd = _rent_cells(region, bedrooms, households, rents) + shape = ( + model.noise, + model.below_market_share, + model.below_market_discount, + model.below_market_noise, + ) + with np.errstate(divide="ignore"): + log_p = np.log(prior) + _log_rent_density( + np.log(weekly_rent), _shifts(region, model), log_median, log_sd, shape + ) + p = np.exp(log_p - log_p.max(axis=1, keepdims=True)) + return brmas, p / p.sum(axis=1, keepdims=True) + + +def fit_reported_rent_model( + region: np.ndarray, + bedrooms: np.ndarray, + weekly_rent: np.ndarray, + weight: np.ndarray, + households: pd.DataFrame | None = None, + rents: pd.DataFrame | None = None, +) -> ReportedRentModel: + """Fit ``ReportedRentModel`` to private renters' reported rents. + + Maximises the weighted likelihood of the rents, each household's BRMA + being unobserved with the census prior. The fit has one shift per region + and four shared shape parameters, each rounded to three decimal places so + that the BRMA probabilities do not depend on the optimiser's last digits. + + Raises: + RuntimeError: If the optimiser does not converge. + """ + from scipy.optimize import minimize + + if hasattr(region, "decode_to_str"): + region = region.decode_to_str() + region = np.asarray(region).astype(str) + weight = np.asarray(weight, dtype=float) + weekly_rent = np.asarray(weekly_rent, dtype=float) + positive = np.isfinite(weekly_rent) & (weekly_rent > 0) + if not (positive.all() and (weight >= 0).all() and weight.sum() > 0): + raise ValueError("Rents must be positive and weights non-negative.") + log_rent = np.log(weekly_rent) + _, prior, log_median, log_sd = _rent_cells(region, bedrooms, households, rents) + # Keep each household's supported BRMAs only: far fewer than all 200. + widest = (prior > 0).sum(axis=1).max() + keep = np.argsort(-prior, axis=1, kind="stable")[:, :widest] + prior, log_median, log_sd = ( + np.take_along_axis(a, keep, axis=1) for a in (prior, log_median, log_sd) + ) + names, index = np.unique(region, return_inverse=True) + with np.errstate(divide="ignore"): + log_prior = np.log(prior) + + def loss(x): + shape = (x[-4], x[-3], x[-2], x[-4] + x[-1]) + log_p = log_prior + _log_rent_density( + log_rent, x[index], log_median, log_sd, shape + ) + peak = log_p.max(axis=1) + likelihood = peak + np.log(np.exp(log_p - peak[:, None]).sum(axis=1)) + return -(weight * likelihood).sum() / weight.sum() + + # Start each region's shift at its median gap to the prior-mean list median. + gap = log_rent - (prior * log_median).sum(axis=1) + start = [np.median(gap[index == i]) for i in range(len(names))] + result = minimize( + loss, + np.array(start + [0.2, 0.15, 0.5, 0.5]), + method="L-BFGS-B", + bounds=[(-3, 3)] * len(names) + [(0.01, 2), (0.001, 0.5), (0, 3), (0.05, 3)], + ) + if not result.success: + raise RuntimeError(f"Reported-rent model did not converge: {result.message}") + x = np.round(result.x, 3) + return ReportedRentModel( + region_shift=dict(zip(names.tolist(), x[: len(names)].tolist())), + noise=float(x[-4]), + below_market_share=float(x[-3]), + below_market_discount=float(x[-2]), + below_market_noise=float(round(x[-4] + x[-1], 3)), + ) + + +def draw_brmas( + brmas: np.ndarray, probabilities: np.ndarray, rng: np.random.Generator +) -> np.ndarray: + """Draw one BRMA per row of ``probabilities``; never one with probability 0.""" + cumulative = np.cumsum(probabilities, axis=1) + cumulative /= cumulative[:, -1:] + drawn = (cumulative <= rng.random(len(probabilities))[:, None]).sum(axis=1) + return np.asarray(brmas, dtype=object)[drawn] + + +def assign_private_renter_brmas( + region: np.ndarray, + bedrooms: np.ndarray, + weekly_rent: np.ndarray, + weight: np.ndarray, + rng: np.random.Generator, + households: pd.DataFrame | None = None, + rents: pd.DataFrame | None = None, +) -> np.ndarray: + """Draw each private-renting household's BRMA given its rent and bedrooms. + + Fits ``ReportedRentModel`` to these households, then draws each one's BRMA + from ``brma_probabilities``. A household that pays more than is usual for + its region and number of bedrooms is more likely to land in a dearer BRMA, + so its rent and its Local Housing Allowance rate move together. + """ + model = fit_reported_rent_model( + region, bedrooms, weekly_rent, weight, households, rents + ) + brmas, p = brma_probabilities( + region, bedrooms, weekly_rent, model, households, rents + ) + return draw_brmas(brmas, p, rng) diff --git a/policyengine_uk_data/datasets/frs.py b/policyengine_uk_data/datasets/frs.py index 96b4f453..fa87b3bc 100644 --- a/policyengine_uk_data/datasets/frs.py +++ b/policyengine_uk_data/datasets/frs.py @@ -19,7 +19,11 @@ from policyengine_uk.variables.household.income.employment_status import ( EmploymentStatus, ) -from policyengine_uk_data.datasets.brma import assign_brmas, pick_household_brmas +from policyengine_uk_data.datasets.brma import ( + assign_brmas, + assign_private_renter_brmas, + pick_household_brmas, +) from policyengine_uk_data.datasets.disability_benefits import ( add_disability_benefit_categories_from_reported_amounts, add_disability_benefit_flags_from_reported_amounts, @@ -1425,6 +1429,21 @@ def determine_education_level(fted_val, typeed2_val, age_val): ) pe_household["brma"] = household_brma[sim.calculate("household_id")].values + # Private renters who report a rent are redrawn given that rent and their + # home's bedrooms, so that dearer rents fall in dearer BRMAs. Everyone + # else keeps the draw above. + reports_rent = (pe_household.tenure_type.values == "RENT_PRIVATELY") & ( + household.hhrent.values > 0 + ) + if reports_rent.any(): + pe_household.loc[reports_rent, "brma"] = assign_private_renter_brmas( + pe_household.region.values[reports_rent].astype(str), + household.bedroom6.values[reports_rent], + household.hhrent.values[reports_rent], + pe_household.household_weight.values[reports_rent], + brma_rng, + ) + pe_person = add_disability_benefit_flags_from_reported_amounts( pe_person, year, diff --git a/policyengine_uk_data/storage/BRMA_DATA_SOURCES.md b/policyengine_uk_data/storage/BRMA_DATA_SOURCES.md index 2c01e058..95e3e384 100644 --- a/policyengine_uk_data/storage/BRMA_DATA_SOURCES.md +++ b/policyengine_uk_data/storage/BRMA_DATA_SOURCES.md @@ -47,3 +47,46 @@ Against the genuine list-of-rents category counts, the census bedroom bands corr ## Rebuilding `tools/brma_households/` (run from the repository root) downloads every source, checks each one against a pinned sha256 and rebuilds this file byte for byte; see its `README.md`. + +# BRMA private rents + +File: `brma_private_rents.csv`, with columns `brma, lha_category, median_weekly_rent, log_sd, rents, basis`. For each BRMA and LHA category it holds the median weekly rent on the rent officers' list of rents, in 2024-25 prices, and the standard deviation of log rents on the list. `rents` is the number of list entries behind the spread, and `basis` says how the row was made. + +`datasets/brma.py` uses it to draw a private renter's BRMA given its rent: the probability of each BRMA in the household's region is proportional to the census private-rented households with the home's number of bedrooms (the file above) times the density of the household's rent on the BRMA's list for homes of that size (log-normal, with this file's median and spread). + +## Sources + +| Nation | Median | Spread | `basis` | +|---|---|---|---| +| England | VOA lists of rents for April 2025 and April 2026 (rents collected October 2023 to September 2025), pooled. | The same lists: interquartile range of log rents ÷ 1.349. | `list` | +| Scotland | Published 30th percentiles for April 2025 and April 2026, raised to a median assuming log-normal rents. | Rent Service Scotland's lists for the years to September 2019-2021 (Scottish Government FOI 202200303624), the latest released. | `30th percentile and list spread` | +| Wales | Rent Officers Wales's lists for April 2024 and April 2025 (rents collected October 2022 to September 2024; Welsh Government FOI ATISN 25142), pooled. They reproduce the published April 2024 and April 2025 30th percentiles (median difference 0; 94% and 98% of cells within 1%). | The same lists. | `list` | +| Northern Ireland | As Scotland, from the Housing Executive's 30th percentiles for April 2024, the latest in policyengine-uk's table. | The median spread of the English, Welsh and Scottish cells of the same category; no Northern Ireland list is published. | `30th percentile and typical spread` | + +The published 30th percentiles are those compiled in policyengine-uk's `lha_published_rates.csv.gz`. Every list and percentile is moved to 2024-25 with the ONS Price Index of Private Rents for its region or nation. + +## Why reported rents need a model + +The lists are the rent officers' lists of rents that set LHA rates. Private renters in the FRS report lower and more varied rents. Against FRS 2024-25, a model fitted by maximum likelihood (`fit_reported_rent_model`) finds: +- reported rents are 4% to 20% below the list median for the same region and bedrooms (a separate shift per region); +- about 84% of households follow the list's distribution plus a little noise (0.13 log points); +- about 16% pay far less (0.64 log points lower on average, with wide noise). Their rent says little about where they live, so they keep roughly the census shares. + +Because every region has its own shift, a region-wide error in the level of the lists (for example, in uprating) changes nothing: only a BRMA's rents relative to others in its region matter. + +## Validation + +- **The rent kernel is calibrated out of sample.** Taking 150,000 April 2026 list entries in categories B-E as households whose BRMA is known, with each region's April 2026 entry counts as the prior and medians and spreads from the April 2025 list only, the mean probability given to the true BRMA rises from 0.085 (census shares) to 0.129 with rent, and the most likely BRMA is the true one for 21.8% of entries against 16.1%. Stated probabilities match observed frequencies: BRMAs given 20-30% are right 23.8% of the time (mean stated 24.2%), those given 50-60% are right 53.4% (54.2%), and those given 90%+ are right 91.3% (93.3%). +- **Published percentiles stand in for lists.** Treating England, Wales and Scotland as Northern Ireland is treated (30th percentile and typical spread) changes an FRS private renter's BRMA probabilities by 4.5 points of total variation on average in England, 7.4 in Wales and 6.6 in Scotland. Conditioning on rent at all moves them by 20.5, 21.3 and 32.8 points. Measured against the April 2024 LHA rate for the home's size (not the household's LHA category), the mean weekly rent within that rate is, with the lists, the percentile treatment and census shares only: England £197.24, £197.42 and £190.59; Wales £127.25, £128.17 and £121.53; Scotland £146.91, £148.16 and £133.63. +- **Scotland's medians agree with its own lists.** The 2021 list medians, uprated with each Scottish BRMA's own ONS index, correlate at 0.98 with the medians used here (mean gap 1.8%, standard deviation 8.4%). The ONS indices grew by between 10% and 36% across Scottish BRMAs, which is why the current 30th percentiles are used rather than one national uprating. +- **The census shares survive.** Within every region, FRS private renters' weighted mean BRMA probabilities stay within 3 points of total variation of the census shares for their homes' bedrooms. A test on the built dataset enforces 5 points. + +## Limits + +- The census band and the list category are chosen by the home's bedrooms. Shared accommodation (category A) is never used to place a household, because the FRS does not identify rooms in shared houses. +- A household's BRMA is still not tied to the output area or local authority given to it later in the build. +- Northern Ireland borrows the spread of rents, and its 30th percentiles are for April 2024, uprated. NIHE publishes mean advertised rents by BRMA and bedrooms, which could pin its spreads; they are not used. + +## Rebuilding + +`tools/brma_rents/` downloads every source, checks each against a pinned sha256 and rebuilds this file; see its `README.md`. diff --git a/policyengine_uk_data/storage/brma_private_rents.csv b/policyengine_uk_data/storage/brma_private_rents.csv new file mode 100644 index 00000000..ef058118 --- /dev/null +++ b/policyengine_uk_data/storage/brma_private_rents.csv @@ -0,0 +1,1001 @@ +brma,lha_category,median_weekly_rent,log_sd,rents,basis +ABERDEEN_AND_SHIRE,A,90.15,0.2047,742,30th percentile and list spread +ABERDEEN_AND_SHIRE,B,131.78,0.1466,2888,30th percentile and list spread +ABERDEEN_AND_SHIRE,C,174.66,0.1788,5275,30th percentile and list spread +ABERDEEN_AND_SHIRE,D,238.43,0.2323,1582,30th percentile and list spread +ABERDEEN_AND_SHIRE,E,369.62,0.2866,699,30th percentile and list spread +ARGYLL_AND_BUTE,A,97.44,0.1599,111,30th percentile and list spread +ARGYLL_AND_BUTE,B,123.98,0.1811,362,30th percentile and list spread +ARGYLL_AND_BUTE,C,167.14,0.1447,570,30th percentile and list spread +ARGYLL_AND_BUTE,D,207.23,0.2397,248,30th percentile and list spread +ARGYLL_AND_BUTE,E,373.08,0.2696,51,30th percentile and list spread +ASHFORD,A,108.37,0.1279,374,list +ASHFORD,B,212.43,0.1046,325,list +ASHFORD,C,256.57,0.1551,943,list +ASHFORD,D,310.27,0.1323,747,list +ASHFORD,E,393.81,0.164,183,list +AYLESBURY,A,119.65,0.1348,350,list +AYLESBURY,B,214.8,0.1305,819,list +AYLESBURY,C,273.93,0.1742,1639,list +AYLESBURY,D,357.79,0.1266,828,list +AYLESBURY,E,429.61,0.2026,206,list +AYRSHIRES,A,108.28,0.2499,331,30th percentile and list spread +AYRSHIRES,B,99.7,0.1447,1326,30th percentile and list spread +AYRSHIRES,C,128.37,0.1521,2322,30th percentile and list spread +AYRSHIRES,D,156.24,0.1866,1277,30th percentile and list spread +AYRSHIRES,E,230.1,0.3083,285,30th percentile and list spread +BARNSLEY,A,84.96,0.181,566,list +BARNSLEY,B,118.04,0.1771,447,list +BARNSLEY,C,134.49,0.1903,3433,list +BARNSLEY,D,159.34,0.2219,2158,list +BARNSLEY,E,203.43,0.2287,216,list +BARROW_IN_FURNESS,A,116.78,0.1573,207,list +BARROW_IN_FURNESS,B,133.08,0.1826,244,list +BARROW_IN_FURNESS,C,150.59,0.1788,1046,list +BARROW_IN_FURNESS,D,185.2,0.192,529,list +BARROW_IN_FURNESS,E,243.98,0.2421,106,list +BASINGSTOKE,A,119.65,0.2371,134,list +BASINGSTOKE,B,211.22,0.1274,1109,list +BASINGSTOKE,C,262.54,0.1842,2083,list +BASINGSTOKE,D,322.2,0.1933,1287,list +BASINGSTOKE,E,413.69,0.1917,344,list +BATH,A,150.62,0.1142,531,list +BATH,B,237.07,0.224,1202,list +BATH,C,300.98,0.2825,1855,list +BATH,D,359.39,0.2826,1241,list +BATH,E,545.54,0.1666,844,list +BEDFORD,A,116.92,0.1812,282,list +BEDFORD,B,179.58,0.1552,1169,list +BEDFORD,C,222.14,0.1817,2081,list +BEDFORD,D,287.32,0.2048,1664,list +BEDFORD,E,366.53,0.2004,393,list +BELFAST,A,94.77,0.1816,0,30th percentile and typical spread +BELFAST,B,173.78,0.1757,0,30th percentile and typical spread +BELFAST,C,195.72,0.1898,0,30th percentile and typical spread +BELFAST,D,220.87,0.2194,0,30th percentile and typical spread +BELFAST,E,284.82,0.249,0,30th percentile and typical spread +BIRMINGHAM,A,112.55,0.2186,170,list +BIRMINGHAM,B,197.29,0.1605,5043,list +BIRMINGHAM,C,218.04,0.2241,7933,list +BIRMINGHAM,D,246.0,0.2171,6217,list +BIRMINGHAM,E,329.34,0.221,816,list +BLACKWATER_VALLEY,A,135.28,0.1224,322,list +BLACKWATER_VALLEY,B,223.62,0.1488,1519,list +BLACKWATER_VALLEY,C,288.08,0.1458,2144,list +BLACKWATER_VALLEY,D,358.01,0.1801,1289,list +BLACKWATER_VALLEY,E,491.95,0.1749,417,list +BLACK_COUNTRY,A,101.64,0.2051,191,list +BLACK_COUNTRY,B,143.48,0.1864,3223,list +BLACK_COUNTRY,C,174.57,0.1821,6722,list +BLACK_COUNTRY,D,212.45,0.231,6116,list +BLACK_COUNTRY,E,279.54,0.2605,762,list +BLAENAU_GWENT,A,85.05,0.2619,158,list +BLAENAU_GWENT,B,101.84,0.2435,93,list +BLAENAU_GWENT,C,127.8,0.1949,439,list +BLAENAU_GWENT,D,149.97,0.2325,378,list +BLAENAU_GWENT,E,197.68,0.315,126,list +BOLTON_AND_BURY,A,118.66,0.1951,179,list +BOLTON_AND_BURY,B,150.59,0.2019,449,list +BOLTON_AND_BURY,C,177.44,0.1821,2914,list +BOLTON_AND_BURY,D,216.84,0.2332,1652,list +BOLTON_AND_BURY,E,310.52,0.245,246,list +BOURNEMOUTH,A,125.53,0.2635,673,list +BOURNEMOUTH,B,197.93,0.1499,2592,list +BOURNEMOUTH,C,260.91,0.1945,4951,list +BOURNEMOUTH,D,332.06,0.1826,2472,list +BOURNEMOUTH,E,426.77,0.2155,769,list +BRADFORD_SOUTH_DALES,A,89.12,0.1824,505,list +BRADFORD_SOUTH_DALES,B,123.93,0.2132,1768,list +BRADFORD_SOUTH_DALES,C,152.57,0.2363,4128,list +BRADFORD_SOUTH_DALES,D,179.7,0.207,2920,list +BRADFORD_SOUTH_DALES,E,214.73,0.3328,535,list +BRECON_AND_RADNOR,A,80.69,0.0628,53,list +BRECON_AND_RADNOR,B,95.84,0.2279,226,list +BRECON_AND_RADNOR,C,123.89,0.2681,285,list +BRECON_AND_RADNOR,D,156.12,0.2927,292,list +BRECON_AND_RADNOR,E,210.86,0.3233,47,list +BRIDGEND,A,83.87,0.2619,148,list +BRIDGEND,B,130.15,0.2378,291,list +BRIDGEND,C,149.77,0.2126,1237,list +BRIDGEND,D,167.74,0.2582,1369,list +BRIDGEND,E,221.69,0.2435,165,list +BRIGHTON_AND_HOVE,A,151.5,0.1889,264,list +BRIGHTON_AND_HOVE,B,256.57,0.1945,2949,list +BRIGHTON_AND_HOVE,C,324.24,0.1878,4066,list +BRIGHTON_AND_HOVE,D,396.69,0.2523,2061,list +BRIGHTON_AND_HOVE,E,537.01,0.1654,794,list +BRISTOL,A,143.82,0.1859,681,list +BRISTOL,B,249.04,0.2133,4513,list +BRISTOL,C,303.23,0.1938,8124,list +BRISTOL,D,355.79,0.2165,4568,list +BRISTOL,E,509.95,0.2939,1518,list +BURY_ST_EDMUNDS,A,120.03,0.1439,76,list +BURY_ST_EDMUNDS,B,187.49,0.1554,597,list +BURY_ST_EDMUNDS,C,239.44,0.2051,2238,list +BURY_ST_EDMUNDS,D,287.32,0.1465,2422,list +BURY_ST_EDMUNDS,E,355.42,0.1863,766,list +CAERPHILLY,A,100.02,0.1169,123,list +CAERPHILLY,B,117.36,0.2897,268,list +CAERPHILLY,C,155.18,0.2603,809,list +CAERPHILLY,D,162.94,0.2494,1067,list +CAERPHILLY,E,221.65,0.3052,93,list +CAMBRIDGE,A,142.42,0.1331,723,list +CAMBRIDGE,B,266.56,0.2776,2239,list +CAMBRIDGE,C,301.69,0.2855,4723,list +CAMBRIDGE,D,343.6,0.2178,2887,list +CAMBRIDGE,E,430.99,0.3006,1076,list +CANTERBURY,A,108.46,0.1237,563,list +CANTERBURY,B,190.94,0.1916,890,list +CANTERBURY,C,250.6,0.2099,2043,list +CANTERBURY,D,298.33,0.1941,1358,list +CANTERBURY,E,357.79,0.1962,466,list +CARDIFF,A,115.38,0.1945,232,list +CARDIFF,B,195.61,0.1863,1859,list +CARDIFF,C,234.73,0.2234,2944,list +CARDIFF,D,286.89,0.2663,1280,list +CARDIFF,E,378.18,0.2361,354,list +CARMARTHENSHIRE,A,83.24,0.0632,44,list +CARMARTHENSHIRE,B,110.85,0.1462,415,list +CARMARTHENSHIRE,C,143.45,0.2193,1228,list +CARMARTHENSHIRE,D,156.48,0.2519,1770,list +CARMARTHENSHIRE,E,195.61,0.3189,276,list +CENTRAL_GREATER_MANCHESTER,A,115.16,0.129,511,list +CENTRAL_GREATER_MANCHESTER,B,221.8,0.2026,4696,list +CENTRAL_GREATER_MANCHESTER,C,266.15,0.2414,10467,list +CENTRAL_GREATER_MANCHESTER,D,295.15,0.2542,3894,list +CENTRAL_GREATER_MANCHESTER,E,375.85,0.2384,714,list +CENTRAL_LANCS,A,77.05,0.1822,240,list +CENTRAL_LANCS,B,126.49,0.1945,818,list +CENTRAL_LANCS,C,156.61,0.183,3012,list +CENTRAL_LANCS,D,188.53,0.2133,2232,list +CENTRAL_LANCS,E,277.25,0.292,442,list +CENTRAL_LONDON,A,223.34,0.169,201,list +CENTRAL_LONDON,B,576.51,0.253,2605,list +CENTRAL_LONDON,C,774.16,0.2746,2780,list +CENTRAL_LONDON,D,1225.1,0.3979,936,list +CENTRAL_LONDON,E,1824.32,0.508,308,list +CENTRAL_NORFOLK_NORWICH,A,114.05,0.1814,277,list +CENTRAL_NORFOLK_NORWICH,B,166.41,0.1989,1987,list +CENTRAL_NORFOLK_NORWICH,C,194.66,0.1806,5635,list +CENTRAL_NORFOLK_NORWICH,D,233.24,0.2373,4765,list +CENTRAL_NORFOLK_NORWICH,E,323.24,0.2425,1245,list +CEREDIGION,A,87.83,0.2006,326,list +CEREDIGION,B,122.58,0.2378,609,list +CEREDIGION,C,143.77,0.2293,956,list +CEREDIGION,D,156.48,0.178,950,list +CEREDIGION,E,182.57,0.2199,207,list +CHELMSFORD,A,126.03,0.1966,437,list +CHELMSFORD,B,221.48,0.2086,1612,list +CHELMSFORD,C,277.67,0.2123,3187,list +CHELMSFORD,D,332.09,0.1767,1745,list +CHELMSFORD,E,422.06,0.1982,449,list +CHELTENHAM,A,114.18,0.1149,64,list +CHELTENHAM,B,179.69,0.2111,1389,list +CHELTENHAM,C,224.62,0.2396,2417,list +CHELTENHAM,D,292.0,0.2574,1597,list +CHELTENHAM,E,403.22,0.2788,414,list +CHERWELL_VALLEY,A,137.55,0.22,197,list +CHERWELL_VALLEY,B,206.85,0.1307,698,list +CHERWELL_VALLEY,C,257.16,0.1651,1423,list +CHERWELL_VALLEY,D,311.94,0.1886,992,list +CHERWELL_VALLEY,E,391.33,0.2668,247,list +CHESTERFIELD,A,96.62,0.2229,130,list +CHESTERFIELD,B,130.18,0.1583,390,list +CHESTERFIELD,C,155.76,0.2014,1912,list +CHESTERFIELD,D,179.72,0.2067,1275,list +CHESTERFIELD,E,227.65,0.3105,174,list +CHICHESTER,A,126.49,0.0567,76,list +CHICHESTER,B,206.85,0.1176,579,list +CHICHESTER,C,273.93,0.1654,1311,list +CHICHESTER,D,334.86,0.1766,946,list +CHICHESTER,E,427.22,0.213,238,list +CHILTERNS,A,140.06,0.1235,70,list +CHILTERNS,B,245.98,0.1707,822,list +CHILTERNS,C,313.06,0.2208,1203,list +CHILTERNS,D,405.73,0.2556,668,list +CHILTERNS,E,584.74,0.2909,255,list +COLCHESTER,A,118.48,0.1885,148,list +COLCHESTER,B,179.58,0.1565,1332,list +COLCHESTER,C,227.47,0.1911,3694,list +COLCHESTER,D,293.22,0.2088,2067,list +COLCHESTER,E,359.16,0.1856,550,list +COVENTRY,A,118.61,0.1381,362,list +COVENTRY,B,166.21,0.1801,818,list +COVENTRY,C,200.15,0.1555,2584,list +COVENTRY,D,243.92,0.2031,2028,list +COVENTRY,E,324.27,0.1864,305,list +CRAWLEY_REIGATE,A,162.42,0.182,333,list +CRAWLEY_REIGATE,B,238.67,0.165,2699,list +CRAWLEY_REIGATE,C,305.24,0.1592,4605,list +CRAWLEY_REIGATE,D,374.56,0.1683,2175,list +CRAWLEY_REIGATE,E,476.14,0.2034,625,list +DARLINGTON,A,84.14,0.2025,101,list +DARLINGTON,B,104.55,0.2201,416,list +DARLINGTON,C,131.95,0.1931,2035,list +DARLINGTON,D,155.93,0.2521,1133,list +DARLINGTON,E,214.59,0.342,253,list +DERBY,A,94.61,0.2493,445,list +DERBY,B,136.59,0.2032,1439,list +DERBY,C,173.73,0.1808,5028,list +DERBY,D,206.08,0.2106,3169,list +DERBY,E,287.55,0.256,604,list +DONCASTER,A,88.73,0.1434,1131,list +DONCASTER,B,117.54,0.2087,946,list +DONCASTER,C,140.46,0.1811,3306,list +DONCASTER,D,158.23,0.2213,3637,list +DONCASTER,E,214.73,0.342,385,list +DOVER_SHEPWAY,A,117.14,0.157,208,list +DOVER_SHEPWAY,B,151.81,0.2514,958,list +DOVER_SHEPWAY,C,208.84,0.2517,1639,list +DOVER_SHEPWAY,D,257.16,0.2576,899,list +DOVER_SHEPWAY,E,334.13,0.2361,221,list +DUMFRIES_AND_GALLOWAY,A,103.27,0.1333,350,30th percentile and list spread +DUMFRIES_AND_GALLOWAY,B,101.99,0.0764,382,30th percentile and list spread +DUMFRIES_AND_GALLOWAY,C,115.28,0.094,846,30th percentile and list spread +DUMFRIES_AND_GALLOWAY,D,134.22,0.1537,462,30th percentile and list spread +DUMFRIES_AND_GALLOWAY,E,193.7,0.1972,135,30th percentile and list spread +DUNDEE_AND_ANGUS,A,100.4,0.1919,579,30th percentile and list spread +DUNDEE_AND_ANGUS,B,118.82,0.1693,1378,30th percentile and list spread +DUNDEE_AND_ANGUS,C,176.36,0.2332,2568,30th percentile and list spread +DUNDEE_AND_ANGUS,D,239.97,0.2991,777,30th percentile and list spread +DUNDEE_AND_ANGUS,E,349.45,0.2537,316,30th percentile and list spread +DURHAM,A,124.56,0.2912,82,list +DURHAM,B,99.04,0.2219,377,list +DURHAM,C,118.75,0.2221,3701,list +DURHAM,D,143.06,0.2413,2672,list +DURHAM,E,203.91,0.3729,432,list +EASTBOURNE,A,115.31,0.1001,292,list +EASTBOURNE,B,190.08,0.171,1067,list +EASTBOURNE,C,245.98,0.1952,2375,list +EASTBOURNE,D,298.33,0.1944,1366,list +EASTBOURNE,E,381.87,0.1959,290,list +EASTERN_STAFFORDSHIRE,A,93.33,0.1947,156,list +EASTERN_STAFFORDSHIRE,B,134.18,0.1293,602,list +EASTERN_STAFFORDSHIRE,C,167.72,0.1492,1336,list +EASTERN_STAFFORDSHIRE,D,201.27,0.1752,990,list +EASTERN_STAFFORDSHIRE,E,286.96,0.2697,229,list +EAST_CHESHIRE,A,121.92,0.142,71,list +EAST_CHESHIRE,B,166.35,0.2018,168,list +EAST_CHESHIRE,C,205.17,0.3006,968,list +EAST_CHESHIRE,D,299.42,0.3006,466,list +EAST_CHESHIRE,E,543.41,0.3783,148,list +EAST_DUNBARTONSHIRE,A,116.01,0.2065,90,30th percentile and list spread +EAST_DUNBARTONSHIRE,B,158.05,0.2086,203,30th percentile and list spread +EAST_DUNBARTONSHIRE,C,207.66,0.2255,465,30th percentile and list spread +EAST_DUNBARTONSHIRE,D,282.34,0.22,215,30th percentile and list spread +EAST_DUNBARTONSHIRE,E,419.1,0.2378,94,30th percentile and list spread +EAST_LANCS,A,79.99,0.1465,93,list +EAST_LANCS,B,120.47,0.1519,652,list +EAST_LANCS,C,138.63,0.1654,3045,list +EAST_LANCS,D,166.35,0.2133,1618,list +EAST_LANCS,E,266.15,0.3597,297,list +EAST_THAMES_VALLEY,A,153.31,0.1464,106,list +EAST_THAMES_VALLEY,B,267.02,0.1674,1096,list +EAST_THAMES_VALLEY,C,343.01,0.1506,1946,list +EAST_THAMES_VALLEY,D,424.87,0.1977,922,list +EAST_THAMES_VALLEY,E,592.36,0.2482,310,list +EXETER,A,136.38,0.1446,356,list +EXETER,B,177.89,0.1767,874,list +EXETER,C,224.62,0.2315,1954,list +EXETER,D,291.87,0.2267,1426,list +EXETER,E,379.5,0.3826,462,list +FIFE,A,108.64,0.2249,387,30th percentile and list spread +FIFE,B,119.72,0.15,887,30th percentile and list spread +FIFE,C,153.1,0.2028,2524,30th percentile and list spread +FIFE,D,202.19,0.2482,1002,30th percentile and list spread +FIFE,E,370.6,0.3997,330,30th percentile and list spread +FLINTSHIRE,A,98.91,0.1518,70,list +FLINTSHIRE,B,123.89,0.1766,117,list +FLINTSHIRE,C,166.54,0.1399,346,list +FLINTSHIRE,D,191.69,0.1863,397,list +FLINTSHIRE,E,275.3,0.2955,58,list +FORTH_VALLEY,A,117.49,0.1376,177,30th percentile and list spread +FORTH_VALLEY,B,133.2,0.1877,660,30th percentile and list spread +FORTH_VALLEY,C,185.98,0.2373,1979,30th percentile and list spread +FORTH_VALLEY,D,241.28,0.3624,624,30th percentile and list spread +FORTH_VALLEY,E,385.09,0.3648,185,30th percentile and list spread +FYLDE_COAST,A,94.09,0.1571,196,list +FYLDE_COAST,B,109.79,0.1796,1662,list +FYLDE_COAST,C,144.56,0.1716,3200,list +FYLDE_COAST,D,174.68,0.1821,2466,list +FYLDE_COAST,E,217.1,0.3721,458,list +GLOUCESTER,A,107.05,0.1181,59,list +GLOUCESTER,B,164.85,0.1608,1105,list +GLOUCESTER,C,202.15,0.1486,2046,list +GLOUCESTER,D,260.91,0.2605,1395,list +GLOUCESTER,E,332.06,0.2133,414,list +GRANTHAM_NEWARK,A,104.56,0.1808,285,list +GRANTHAM_NEWARK,B,122.4,0.1758,563,list +GRANTHAM_NEWARK,C,155.77,0.1783,1689,list +GRANTHAM_NEWARK,D,186.36,0.2004,1444,list +GRANTHAM_NEWARK,E,278.16,0.2965,281,list +GREATER_GLASGOW,A,126.94,0.2261,1016,30th percentile and list spread +GREATER_GLASGOW,B,194.66,0.2606,4131,30th percentile and list spread +GREATER_GLASGOW,C,235.3,0.2616,8344,30th percentile and list spread +GREATER_GLASGOW,D,305.92,0.4419,2110,30th percentile and list spread +GREATER_GLASGOW,E,488.28,0.2774,616,30th percentile and list spread +GREATER_LIVERPOOL,A,95.01,0.111,1419,list +GREATER_LIVERPOOL,B,155.26,0.2423,1875,list +GREATER_LIVERPOOL,C,180.7,0.2267,5078,list +GREATER_LIVERPOOL,D,199.62,0.2251,4409,list +GREATER_LIVERPOOL,E,265.03,0.3462,478,list +GRIMSBY,A,90.27,0.2471,528,list +GRIMSBY,B,101.71,0.1655,480,list +GRIMSBY,C,124.32,0.2019,1311,list +GRIMSBY,D,140.46,0.2323,1960,list +GRIMSBY,E,214.73,0.3099,169,list +GUILDFORD,A,147.93,0.1676,164,list +GUILDFORD,B,262.54,0.1539,1573,list +GUILDFORD,C,335.43,0.1507,2245,list +GUILDFORD,D,405.73,0.1945,1007,list +GUILDFORD,E,559.04,0.1984,461,list +HALIFAX,A,95.4,0.1119,139,list +HALIFAX,B,124.13,0.2071,612,list +HALIFAX,C,146.92,0.1924,1835,list +HALIFAX,D,177.05,0.2381,895,list +HALIFAX,E,224.91,0.2783,142,list +HARLOW_STORTFORD,A,126.03,0.1622,90,list +HARLOW_STORTFORD,B,221.48,0.1355,1296,list +HARLOW_STORTFORD,C,287.32,0.1718,1979,list +HARLOW_STORTFORD,D,344.31,0.1654,1236,list +HARLOW_STORTFORD,E,442.96,0.3083,418,list +HARROGATE,A,118.97,0.1492,127,list +HARROGATE,B,155.96,0.1292,638,list +HARROGATE,C,200.66,0.1811,1594,list +HARROGATE,D,248.64,0.2698,934,list +HARROGATE,E,382.19,0.3597,320,list +HEREFORDSHIRE,A,101.95,0.1904,129,list +HEREFORDSHIRE,B,134.18,0.1852,504,list +HEREFORDSHIRE,C,173.38,0.1648,977,list +HEREFORDSHIRE,D,215.22,0.2063,593,list +HEREFORDSHIRE,E,313.09,0.2281,184,list +HIGHLAND_AND_ISLANDS,A,107.62,0.1835,625,30th percentile and list spread +HIGHLAND_AND_ISLANDS,B,127.31,0.1653,787,30th percentile and list spread +HIGHLAND_AND_ISLANDS,C,161.94,0.1787,2217,30th percentile and list spread +HIGHLAND_AND_ISLANDS,D,194.66,0.2194,971,30th percentile and list spread +HIGHLAND_AND_ISLANDS,E,245.21,0.239,223,30th percentile and list spread +HIGH_WEALD,A,125.26,0.218,139,list +HIGH_WEALD,B,219.36,0.165,1156,list +HIGH_WEALD,C,305.24,0.1816,2023,list +HIGH_WEALD,D,368.98,0.2243,1166,list +HIGH_WEALD,E,537.01,0.2801,348,list +HULL_EAST_RIDING,A,98.99,0.2343,960,list +HULL_EAST_RIDING,B,107.37,0.2204,901,list +HULL_EAST_RIDING,C,131.1,0.1846,2685,list +HULL_EAST_RIDING,D,159.34,0.2243,1658,list +HULL_EAST_RIDING,E,218.37,0.3166,271,list +HUNTINGDON,A,124.83,0.2234,111,list +HUNTINGDON,B,176.04,0.137,722,list +HUNTINGDON,C,211.03,0.1343,1562,list +HUNTINGDON,D,255.46,0.2034,1096,list +HUNTINGDON,E,355.42,0.235,344,list +INNER_EAST_LONDON,A,204.18,0.2321,179,list +INNER_EAST_LONDON,B,423.37,0.1816,1660,list +INNER_EAST_LONDON,C,526.18,0.2082,2061,list +INNER_EAST_LONDON,D,653.19,0.2192,734,list +INNER_EAST_LONDON,E,846.59,0.1732,171,list +INNER_NORTH_LONDON,A,195.78,0.1813,162,list +INNER_NORTH_LONDON,B,423.37,0.2083,1716,list +INNER_NORTH_LONDON,C,523.55,0.2351,1990,list +INNER_NORTH_LONDON,D,665.29,0.2599,855,list +INNER_NORTH_LONDON,E,891.15,0.2771,249,list +INNER_SOUTH_EAST_LONDON,A,199.35,0.2038,646,list +INNER_SOUTH_EAST_LONDON,B,363.88,0.2081,4845,list +INNER_SOUTH_EAST_LONDON,C,446.01,0.2361,6521,list +INNER_SOUTH_EAST_LONDON,D,575.78,0.2447,2259,list +INNER_SOUTH_EAST_LONDON,E,752.63,0.2275,784,list +INNER_SOUTH_WEST_LONDON,A,263.31,0.3439,131,list +INNER_SOUTH_WEST_LONDON,B,401.59,0.1705,1579,list +INNER_SOUTH_WEST_LONDON,C,508.04,0.2154,2502,list +INNER_SOUTH_WEST_LONDON,D,665.29,0.2703,1197,list +INNER_SOUTH_WEST_LONDON,E,883.02,0.2399,384,list +INNER_WEST_LONDON,A,200.8,0.231,246,list +INNER_WEST_LONDON,B,387.08,0.1719,2080,list +INNER_WEST_LONDON,C,471.75,0.2027,2757,list +INNER_WEST_LONDON,D,639.88,0.2831,828,list +INNER_WEST_LONDON,E,909.7,0.2846,363,list +IPSWICH,A,110.1,0.2119,205,list +IPSWICH,B,166.41,0.1631,1368,list +IPSWICH,C,197.53,0.1339,3307,list +IPSWICH,D,238.25,0.2209,2638,list +IPSWICH,E,312.1,0.2186,425,list +ISLE_OF_WIGHT,A,80.89,0.0862,100,list +ISLE_OF_WIGHT,B,143.2,0.191,1064,list +ISLE_OF_WIGHT,C,184.48,0.1894,2350,list +ISLE_OF_WIGHT,D,223.62,0.2053,1332,list +ISLE_OF_WIGHT,E,300.72,0.2782,303,list +KENDAL,A,93.49,0.2478,95,list +KENDAL,B,149.71,0.0904,367,list +KENDAL,C,180.7,0.1481,727,list +KENDAL,D,215.63,0.1747,430,list +KENDAL,E,289.12,0.1813,118,list +KERNOW_WEST,A,121.89,0.1817,164,list +KERNOW_WEST,B,154.17,0.212,1325,list +KERNOW_WEST,C,195.68,0.2156,3263,list +KERNOW_WEST,D,237.19,0.2746,2503,list +KERNOW_WEST,E,349.85,0.3242,591,list +KINGS_LYNN,A,114.89,0.2239,108,list +KINGS_LYNN,B,152.16,0.1188,700,list +KINGS_LYNN,C,179.58,0.1426,1962,list +KINGS_LYNN,D,215.49,0.1654,1642,list +KINGS_LYNN,E,310.99,0.2448,366,list +KIRKLEES,A,87.89,0.2183,270,list +KIRKLEES,B,124.32,0.2053,1108,list +KIRKLEES,C,148.73,0.2236,2980,list +KIRKLEES,D,177.05,0.2661,1798,list +KIRKLEES,E,236.07,0.311,419,list +LANCASTER,A,101.03,0.1085,188,list +LANCASTER,B,138.63,0.1852,535,list +LANCASTER,C,162.63,0.1767,1571,list +LANCASTER,D,188.53,0.1875,781,list +LANCASTER,E,221.8,0.3032,233,list +LEEDS,A,115.28,0.2599,174,list +LEEDS,B,180.83,0.1814,2864,list +LEEDS,C,206.56,0.2256,6624,list +LEEDS,D,236.07,0.2477,2744,list +LEEDS,E,339.06,0.2849,866,list +LEICESTER,A,111.38,0.2272,466,list +LEICESTER,B,155.76,0.1458,2073,list +LEICESTER,C,189.15,0.1715,4467,list +LEICESTER,D,222.86,0.1817,3753,list +LEICESTER,E,311.52,0.2229,919,list +LINCOLN,A,93.43,0.1833,300,list +LINCOLN,B,149.77,0.2463,534,list +LINCOLN,C,176.91,0.2126,2643,list +LINCOLN,D,200.27,0.1986,2263,list +LINCOLN,E,275.58,0.2838,506,list +LINCOLNSHIRE_FENS,A,102.54,0.2951,198,list +LINCOLNSHIRE_FENS,B,137.79,0.1787,503,list +LINCOLNSHIRE_FENS,C,166.89,0.1216,1757,list +LINCOLNSHIRE_FENS,D,200.27,0.1352,1430,list +LINCOLNSHIRE_FENS,E,255.91,0.2356,151,list +LOTHIAN,A,138.29,0.1967,1156,30th percentile and list spread +LOTHIAN,B,226.36,0.1281,3969,30th percentile and list spread +LOTHIAN,C,309.94,0.21,7772,30th percentile and list spread +LOTHIAN,D,433.95,0.3103,2470,30th percentile and list spread +LOTHIAN,E,609.0,0.2492,1214,30th percentile and list spread +LOUGH_NEAGH_LOWER,A,104.45,0.1816,0,30th percentile and typical spread +LOUGH_NEAGH_LOWER,B,121.4,0.1757,0,30th percentile and typical spread +LOUGH_NEAGH_LOWER,C,152.22,0.1898,0,30th percentile and typical spread +LOUGH_NEAGH_LOWER,D,175.23,0.2194,0,30th percentile and typical spread +LOUGH_NEAGH_LOWER,E,194.73,0.249,0,30th percentile and typical spread +LOUGH_NEAGH_UPPER,A,88.92,0.1816,0,30th percentile and typical spread +LOUGH_NEAGH_UPPER,B,120.87,0.1757,0,30th percentile and typical spread +LOUGH_NEAGH_UPPER,C,148.97,0.1898,0,30th percentile and typical spread +LOUGH_NEAGH_UPPER,D,170.7,0.2194,0,30th percentile and typical spread +LOUGH_NEAGH_UPPER,E,183.55,0.249,0,30th percentile and typical spread +LOWESTOFT_GREAT_YARMOUTH,A,107.53,0.1616,174,list +LOWESTOFT_GREAT_YARMOUTH,B,122.18,0.1605,851,list +LOWESTOFT_GREAT_YARMOUTH,C,166.6,0.186,1502,list +LOWESTOFT_GREAT_YARMOUTH,D,188.82,0.217,1664,list +LOWESTOFT_GREAT_YARMOUTH,E,266.56,0.2726,209,list +LUTON,A,117.64,0.1174,297,list +LUTON,B,194.37,0.1476,2050,list +LUTON,C,251.41,0.1929,2655,list +LUTON,D,311.27,0.1718,1822,list +LUTON,E,383.1,0.1945,277,list +MAIDSTONE,A,124.07,0.0779,325,list +MAIDSTONE,B,212.43,0.1274,607,list +MAIDSTONE,C,268.33,0.1462,1396,list +MAIDSTONE,D,335.43,0.1602,784,list +MAIDSTONE,E,424.87,0.1579,248,list +MEDWAY_SWALE,A,125.62,0.165,175,list +MEDWAY_SWALE,B,201.25,0.2445,1400,list +MEDWAY_SWALE,C,257.16,0.2243,2586,list +MEDWAY_SWALE,D,293.38,0.2157,2171,list +MEDWAY_SWALE,E,380.14,0.2085,363,list +MENDIP,A,126.77,0.2021,225,list +MENDIP,B,157.24,0.1654,544,list +MENDIP,C,202.15,0.145,1244,list +MENDIP,D,269.54,0.2325,854,list +MENDIP,E,359.39,0.1928,279,list +MERTHYR_CYNON,A,83.87,0.2619,149,list +MERTHYR_CYNON,B,129.1,0.1416,133,list +MERTHYR_CYNON,C,143.77,0.2034,368,list +MERTHYR_CYNON,D,163.57,0.2416,542,list +MERTHYR_CYNON,E,221.69,0.2649,68,list +MID_EAST_DEVON,A,121.62,0.1747,494,list +MID_EAST_DEVON,B,140.39,0.1676,531,list +MID_EAST_DEVON,C,177.89,0.1666,1364,list +MID_EAST_DEVON,D,224.62,0.2108,944,list +MID_EAST_DEVON,E,314.46,0.2398,223,list +MID_STAFFS,A,100.49,0.1531,177,list +MID_STAFFS,B,149.47,0.1828,981,list +MID_STAFFS,C,184.5,0.1676,3053,list +MID_STAFFS,D,218.04,0.224,2843,list +MID_STAFFS,E,311.43,0.2448,482,list +MID_WEST_DORSET,A,118.6,0.3051,386,list +MID_WEST_DORSET,B,166.04,0.1595,941,list +MID_WEST_DORSET,C,207.54,0.1895,2033,list +MID_WEST_DORSET,D,269.54,0.2349,1372,list +MID_WEST_DORSET,E,336.93,0.2309,309,list +MILTON_KEYNES,A,134.47,0.1772,186,list +MILTON_KEYNES,B,218.02,0.2327,1957,list +MILTON_KEYNES,C,262.54,0.1983,3891,list +MILTON_KEYNES,D,321.27,0.1801,2488,list +MILTON_KEYNES,E,405.73,0.158,847,list +MONMOUTHSHIRE,A,100.02,0.1495,121,list +MONMOUTHSHIRE,B,161.74,0.2393,113,list +MONMOUTHSHIRE,C,195.61,0.1352,368,list +MONMOUTHSHIRE,D,234.77,0.2297,480,list +MONMOUTHSHIRE,E,335.46,0.2299,133,list +NEATH_PORT_TALBOT,A,104.22,0.2495,194,list +NEATH_PORT_TALBOT,B,114.76,0.1921,263,list +NEATH_PORT_TALBOT,C,130.41,0.1998,969,list +NEATH_PORT_TALBOT,D,143.77,0.2016,1414,list +NEATH_PORT_TALBOT,E,176.05,0.3016,112,list +NEWBURY,A,136.36,0.2345,687,list +NEWBURY,B,202.87,0.1485,562,list +NEWBURY,C,250.6,0.201,1010,list +NEWBURY,D,324.24,0.1654,415,list +NEWBURY,E,429.61,0.2386,159,list +NEWPORT,A,100.02,0.1492,116,list +NEWPORT,B,143.77,0.2246,395,list +NEWPORT,C,185.71,0.1954,839,list +NEWPORT,D,203.68,0.2616,731,list +NEWPORT,E,287.21,0.2705,124,list +NORTHAMPTON,A,116.18,0.1451,982,list +NORTHAMPTON,B,173.73,0.1091,1443,list +NORTHAMPTON,C,206.08,0.1373,3820,list +NORTHAMPTON,D,255.91,0.1974,2363,list +NORTHAMPTON,E,333.8,0.1989,469,list +NORTHANTS_CENTRAL,A,104.56,0.1687,264,list +NORTHANTS_CENTRAL,B,154.66,0.1421,650,list +NORTHANTS_CENTRAL,C,191.7,0.1202,2222,list +NORTHANTS_CENTRAL,D,221.66,0.1569,2154,list +NORTHANTS_CENTRAL,E,311.52,0.2328,386,list +NORTHUMBERLAND,A,99.68,0.1655,138,list +NORTHUMBERLAND,B,107.95,0.1842,159,list +NORTHUMBERLAND,C,121.06,0.2071,1224,list +NORTHUMBERLAND,D,152.86,0.2427,734,list +NORTHUMBERLAND,E,214.7,0.2167,109,list +NORTH_CHESHIRE,A,97.93,0.2372,350,list +NORTH_CHESHIRE,B,138.63,0.2057,879,list +NORTH_CHESHIRE,C,168.66,0.2086,3579,list +NORTH_CHESHIRE,D,210.71,0.2982,1997,list +NORTH_CHESHIRE,E,312.61,0.3116,328,list +NORTH_CLWYD,A,93.21,0.1574,112,list +NORTH_CLWYD,B,119.81,0.2229,543,list +NORTH_CLWYD,C,161.74,0.2133,1242,list +NORTH_CLWYD,D,185.71,0.2393,755,list +NORTH_CLWYD,E,260.79,0.3408,158,list +NORTH_CORNWALL_DEVON_BORDERS,A,113.86,0.3206,167,list +NORTH_CORNWALL_DEVON_BORDERS,B,141.13,0.1921,233,list +NORTH_CORNWALL_DEVON_BORDERS,C,183.83,0.1709,587,list +NORTH_CORNWALL_DEVON_BORDERS,D,224.62,0.2315,573,list +NORTH_CORNWALL_DEVON_BORDERS,E,313.34,0.1981,127,list +NORTH_CUMBRIA,A,88.41,0.2128,443,list +NORTH_CUMBRIA,B,110.83,0.1658,515,list +NORTH_CUMBRIA,C,138.63,0.192,2147,list +NORTH_CUMBRIA,D,173.47,0.2209,1367,list +NORTH_CUMBRIA,E,220.69,0.2667,318,list +NORTH_DEVON,A,130.46,0.214,152,list +NORTH_DEVON,B,129.26,0.1616,449,list +NORTH_DEVON,C,173.05,0.1492,1130,list +NORTH_DEVON,D,213.39,0.1654,839,list +NORTH_DEVON,E,284.94,0.2432,212,list +NORTH_LANARKSHIRE,A,95.67,0.1866,164,30th percentile and list spread +NORTH_LANARKSHIRE,B,118.18,0.1413,740,30th percentile and list spread +NORTH_LANARKSHIRE,C,155.46,0.1397,1626,30th percentile and list spread +NORTH_LANARKSHIRE,D,186.56,0.2109,805,30th percentile and list spread +NORTH_LANARKSHIRE,E,254.16,0.2282,172,30th percentile and list spread +NORTH_NI,A,92.85,0.1816,0,30th percentile and typical spread +NORTH_NI,B,149.4,0.1757,0,30th percentile and typical spread +NORTH_NI,C,154.95,0.1898,0,30th percentile and typical spread +NORTH_NI,D,170.34,0.2194,0,30th percentile and typical spread +NORTH_NI,E,191.32,0.249,0,30th percentile and typical spread +NORTH_NOTTINGHAM,A,93.05,0.192,355,list +NORTH_NOTTINGHAM,B,125.8,0.2133,829,list +NORTH_NOTTINGHAM,C,155.76,0.167,3484,list +NORTH_NOTTINGHAM,D,173.73,0.2202,4445,list +NORTH_NOTTINGHAM,E,239.22,0.3022,525,list +NORTH_POWYS,A,87.83,0.2018,330,list +NORTH_POWYS,B,98.25,0.2863,87,list +NORTH_POWYS,C,125.8,0.237,139,list +NORTH_POWYS,D,155.75,0.2416,125,list +NORTH_POWYS,E,182.57,0.2064,47,list +NORTH_WEST_KENT,A,132.23,0.1554,153,list +NORTH_WEST_KENT,B,223.62,0.2012,917,list +NORTH_WEST_KENT,C,298.33,0.2133,1829,list +NORTH_WEST_KENT,D,357.79,0.2133,1246,list +NORTH_WEST_KENT,E,447.23,0.2555,228,list +NORTH_WEST_LONDON,A,163.62,0.1326,162,list +NORTH_WEST_LONDON,B,338.69,0.2336,1895,list +NORTH_WEST_LONDON,C,399.18,0.1998,3120,list +NORTH_WEST_LONDON,D,501.27,0.1658,1376,list +NORTH_WEST_LONDON,E,629.0,0.1712,431,list +NORTH_WEST_NI,A,110.35,0.1816,0,30th percentile and typical spread +NORTH_WEST_NI,B,123.76,0.1757,0,30th percentile and typical spread +NORTH_WEST_NI,C,148.51,0.1898,0,30th percentile and typical spread +NORTH_WEST_NI,D,170.16,0.2194,0,30th percentile and typical spread +NORTH_WEST_NI,E,199.2,0.249,0,30th percentile and typical spread +NORTH_WEST_WALES,A,89.02,0.096,160,list +NORTH_WEST_WALES,B,131.79,0.2858,285,list +NORTH_WEST_WALES,C,160.54,0.2295,729,list +NORTH_WEST_WALES,D,188.81,0.2377,631,list +NORTH_WEST_WALES,E,221.69,0.3032,112,list +NOTTINGHAM,A,100.44,0.1749,1204,list +NOTTINGHAM,B,166.89,0.2095,3116,list +NOTTINGHAM,C,194.1,0.2058,9128,list +NOTTINGHAM,D,221.42,0.2109,6913,list +NOTTINGHAM,E,300.41,0.2848,957,list +OLDHAM_ROCHDALE,A,104.21,0.1186,176,list +OLDHAM_ROCHDALE,B,138.54,0.1778,326,list +OLDHAM_ROCHDALE,C,168.66,0.2225,1892,list +OLDHAM_ROCHDALE,D,212.16,0.2361,1082,list +OLDHAM_ROCHDALE,E,301.17,0.297,160,list +OUTER_EAST_LONDON,A,172.95,0.1895,190,list +OUTER_EAST_LONDON,B,345.32,0.1684,2677,list +OUTER_EAST_LONDON,C,411.27,0.19,3156,list +OUTER_EAST_LONDON,D,483.84,0.2026,1354,list +OUTER_EAST_LONDON,E,561.43,0.1863,297,list +OUTER_NORTH_EAST_LONDON,A,163.62,0.2411,119,list +OUTER_NORTH_EAST_LONDON,B,290.3,0.1437,1897,list +OUTER_NORTH_EAST_LONDON,C,362.89,0.1603,3320,list +OUTER_NORTH_EAST_LONDON,D,435.47,0.1711,2035,list +OUTER_NORTH_EAST_LONDON,E,554.74,0.1801,431,list +OUTER_NORTH_LONDON,A,166.04,0.1898,176,list +OUTER_NORTH_LONDON,B,320.55,0.1522,1903,list +OUTER_NORTH_LONDON,C,389.88,0.1603,3272,list +OUTER_NORTH_LONDON,D,490.13,0.1994,1428,list +OUTER_NORTH_LONDON,E,653.19,0.2441,492,list +OUTER_SOUTH_EAST_LONDON,A,178.23,0.1596,183,list +OUTER_SOUTH_EAST_LONDON,B,300.76,0.162,2554,list +OUTER_SOUTH_EAST_LONDON,C,367.6,0.1539,3816,list +OUTER_SOUTH_EAST_LONDON,D,445.57,0.1811,1945,list +OUTER_SOUTH_EAST_LONDON,E,544.33,0.1953,520,list +OUTER_SOUTH_LONDON,A,163.89,0.2052,150,list +OUTER_SOUTH_LONDON,B,278.48,0.1558,2379,list +OUTER_SOUTH_LONDON,C,345.32,0.1484,4185,list +OUTER_SOUTH_LONDON,D,445.57,0.1916,1679,list +OUTER_SOUTH_LONDON,E,556.96,0.1849,442,list +OUTER_SOUTH_WEST_LONDON,A,178.23,0.1589,112,list +OUTER_SOUTH_WEST_LONDON,B,334.18,0.1766,1302,list +OUTER_SOUTH_WEST_LONDON,C,418.53,0.2226,2135,list +OUTER_SOUTH_WEST_LONDON,D,556.96,0.2705,1030,list +OUTER_SOUTH_WEST_LONDON,E,725.77,0.2726,400,list +OUTER_WEST_LONDON,A,163.62,0.2065,215,list +OUTER_WEST_LONDON,B,300.76,0.1484,1095,list +OUTER_WEST_LONDON,C,367.6,0.1352,1432,list +OUTER_WEST_LONDON,D,432.88,0.233,971,list +OUTER_WEST_LONDON,E,534.69,0.2054,203,list +OXFORD,A,156.83,0.1627,282,list +OXFORD,B,245.98,0.2502,1896,list +OXFORD,C,313.06,0.197,4171,list +OXFORD,D,363.37,0.2135,2567,list +OXFORD,E,501.21,0.2935,807,list +PEAKS_DALES,A,87.86,0.207,222,list +PEAKS_DALES,B,131.8,0.1934,369,list +PEAKS_DALES,C,161.75,0.2265,950,list +PEAKS_DALES,D,190.51,0.232,747,list +PEAKS_DALES,E,287.55,0.334,165,list +PEMBROKESHIRE,A,85.9,0.1605,43,list +PEMBROKESHIRE,B,107.83,0.2231,466,list +PEMBROKESHIRE,C,143.45,0.2647,929,list +PEMBROKESHIRE,D,168.23,0.2299,815,list +PEMBROKESHIRE,E,215.17,0.2514,188,list +PERTH_AND_KINROSS,A,101.19,0.1826,180,30th percentile and list spread +PERTH_AND_KINROSS,B,108.09,0.099,734,30th percentile and list spread +PERTH_AND_KINROSS,C,148.42,0.1762,1248,30th percentile and list spread +PERTH_AND_KINROSS,D,198.5,0.2011,378,30th percentile and list spread +PERTH_AND_KINROSS,E,350.68,0.3021,135,30th percentile and list spread +PETERBOROUGH,A,98.7,0.2304,246,list +PETERBOROUGH,B,161.05,0.1403,1671,list +PETERBOROUGH,C,194.37,0.1397,4569,list +PETERBOROUGH,D,227.47,0.1877,4035,list +PETERBOROUGH,E,311.27,0.186,806,list +PLYMOUTH,A,118.6,0.1444,1095,list +PLYMOUTH,B,154.17,0.1807,2088,list +PLYMOUTH,C,185.31,0.1715,4328,list +PLYMOUTH,D,225.33,0.2165,2457,list +PLYMOUTH,E,303.23,0.2213,501,list +PORTSMOUTH,A,117.7,0.1906,570,list +PORTSMOUTH,B,190.94,0.1521,2832,list +PORTSMOUTH,C,238.67,0.1899,5997,list +PORTSMOUTH,D,300.72,0.1664,3715,list +PORTSMOUTH,E,369.93,0.1801,754,list +READING,A,145.27,0.2222,352,list +READING,B,250.6,0.1753,2808,list +READING,C,313.06,0.1583,4123,list +READING,D,380.14,0.1642,2015,list +READING,E,489.27,0.1952,731,list +RENFREWSHIRE_AND_INVERCLYDE,A,98.68,0.2083,206,30th percentile and list spread +RENFREWSHIRE_AND_INVERCLYDE,B,115.35,0.1439,1454,30th percentile and list spread +RENFREWSHIRE_AND_INVERCLYDE,C,153.19,0.1896,2121,30th percentile and list spread +RENFREWSHIRE_AND_INVERCLYDE,D,189.21,0.2344,762,30th percentile and list spread +RENFREWSHIRE_AND_INVERCLYDE,E,327.6,0.2836,136,30th percentile and list spread +RICHMOND_HAMBLETON,A,118.66,0.2249,84,list +RICHMOND_HAMBLETON,B,123.93,0.1103,211,list +RICHMOND_HAMBLETON,C,153.45,0.1654,694,list +RICHMOND_HAMBLETON,D,182.96,0.2264,465,list +RICHMOND_HAMBLETON,E,239.64,0.2396,116,list +ROTHERHAM,A,94.66,0.1252,94,list +ROTHERHAM,B,118.67,0.1811,354,list +ROTHERHAM,C,146.36,0.2057,1929,list +ROTHERHAM,D,146.92,0.2364,2945,list +ROTHERHAM,E,230.92,0.3337,210,list +RUGBY_EAST,A,120.35,0.1895,82,list +RUGBY_EAST,B,156.55,0.1061,413,list +RUGBY_EAST,C,195.68,0.1256,842,list +RUGBY_EAST,D,251.09,0.1991,570,list +RUGBY_EAST,E,335.46,0.1641,166,list +SALISBURY,A,125.22,0.1894,322,list +SALISBURY,B,177.89,0.1429,577,list +SALISBURY,C,213.39,0.1617,1480,list +SALISBURY,D,260.91,0.2334,1331,list +SALISBURY,E,358.94,0.2212,348,list +SCARBOROUGH,A,92.44,0.1853,142,list +SCARBOROUGH,B,115.28,0.1512,415,list +SCARBOROUGH,C,153.45,0.1898,654,list +SCARBOROUGH,D,192.13,0.2212,365,list +SCARBOROUGH,E,213.61,0.2501,101,list +SCOTTISH_BORDERS,A,98.41,0.2215,143,30th percentile and list spread +SCOTTISH_BORDERS,B,99.65,0.1413,455,30th percentile and list spread +SCOTTISH_BORDERS,C,134.9,0.1909,735,30th percentile and list spread +SCOTTISH_BORDERS,D,171.25,0.2493,449,30th percentile and list spread +SCOTTISH_BORDERS,E,336.74,0.3698,116,30th percentile and list spread +SCUNTHORPE,A,80.05,0.1756,143,list +SCUNTHORPE,B,107.39,0.1441,306,list +SCUNTHORPE,C,129.97,0.1583,729,list +SCUNTHORPE,D,159.34,0.1716,850,list +SCUNTHORPE,E,200.66,0.2423,132,list +SHEFFIELD,A,94.71,0.199,272,list +SHEFFIELD,B,159.34,0.1673,2423,list +SHEFFIELD,C,181.78,0.2066,4331,list +SHEFFIELD,D,202.3,0.2259,2971,list +SHEFFIELD,E,282.54,0.3069,522,list +SHROPSHIRE,A,100.27,0.1801,398,list +SHROPSHIRE,B,137.51,0.1945,1159,list +SHROPSHIRE,C,167.72,0.1642,3232,list +SHROPSHIRE,D,203.27,0.1771,2598,list +SHROPSHIRE,E,286.96,0.2546,562,list +SOLIHULL,A,113.91,0.1562,361,list +SOLIHULL,B,203.5,0.1755,361,list +SOLIHULL,C,234.81,0.1844,1267,list +SOLIHULL,D,300.12,0.1802,881,list +SOLIHULL,E,423.79,0.2527,283,list +SOUTHAMPTON,A,123.28,0.2618,652,list +SOUTHAMPTON,B,195.71,0.1274,4473,list +SOUTHAMPTON,C,247.02,0.1872,7557,list +SOUTHAMPTON,D,311.26,0.1886,4184,list +SOUTHAMPTON,E,393.81,0.1774,997,list +SOUTHEND,A,112.85,0.1529,851,list +SOUTHEND,B,188.82,0.1752,3334,list +SOUTHEND,C,244.35,0.2181,5200,list +SOUTHEND,D,311.27,0.2083,2858,list +SOUTHEND,E,395.08,0.2088,626,list +SOUTHERN_GREATER_MANCHESTER,A,115.16,0.1668,416,list +SOUTHERN_GREATER_MANCHESTER,B,188.53,0.1542,717,list +SOUTHERN_GREATER_MANCHESTER,C,228.89,0.1943,2861,list +SOUTHERN_GREATER_MANCHESTER,D,289.12,0.245,2020,list +SOUTHERN_GREATER_MANCHESTER,E,431.28,0.2395,408,list +SOUTHPORT,A,116.74,0.0737,132,list +SOUTHPORT,B,132.52,0.1603,404,list +SOUTHPORT,C,174.68,0.1603,940,list +SOUTHPORT,D,221.8,0.2341,553,list +SOUTHPORT,E,325.65,0.3029,106,list +SOUTH_CHESHIRE,A,94.34,0.1927,108,list +SOUTH_CHESHIRE,B,132.52,0.1426,401,list +SOUTH_CHESHIRE,C,168.66,0.1476,1526,list +SOUTH_CHESHIRE,D,205.17,0.1659,1280,list +SOUTH_CHESHIRE,E,301.17,0.2193,276,list +SOUTH_DEVON,A,99.33,0.0818,199,list +SOUTH_DEVON,B,140.39,0.19,1570,list +SOUTH_DEVON,C,185.31,0.1932,2712,list +SOUTH_DEVON,D,235.84,0.2778,1780,list +SOUTH_DEVON,E,308.35,0.2299,374,list +SOUTH_EAST_HERTS,A,128.02,0.1879,71,list +SOUTH_EAST_HERTS,B,233.45,0.1643,1558,list +SOUTH_EAST_HERTS,C,299.88,0.1629,2712,list +SOUTH_EAST_HERTS,D,371.13,0.1674,1437,list +SOUTH_EAST_HERTS,E,444.27,0.197,464,list +SOUTH_EAST_NI,A,99.71,0.1816,0,30th percentile and typical spread +SOUTH_EAST_NI,B,136.72,0.1757,0,30th percentile and typical spread +SOUTH_EAST_NI,C,181.22,0.1898,0,30th percentile and typical spread +SOUTH_EAST_NI,D,207.35,0.2194,0,30th percentile and typical spread +SOUTH_EAST_NI,E,258.07,0.249,0,30th percentile and typical spread +SOUTH_GWYNEDD,A,89.02,0.096,160,list +SOUTH_GWYNEDD,B,96.33,0.216,95,list +SOUTH_GWYNEDD,C,127.8,0.2563,181,list +SOUTH_GWYNEDD,D,153.88,0.2349,119,list +SOUTH_GWYNEDD,E,186.91,0.2104,61,list +SOUTH_LANARKSHIRE,A,107.98,0.2128,170,30th percentile and list spread +SOUTH_LANARKSHIRE,B,121.39,0.1334,912,30th percentile and list spread +SOUTH_LANARKSHIRE,C,163.46,0.1891,1858,30th percentile and list spread +SOUTH_LANARKSHIRE,D,208.33,0.1981,789,30th percentile and list spread +SOUTH_LANARKSHIRE,E,338.75,0.3032,224,30th percentile and list spread +SOUTH_NI,A,81.93,0.1816,0,30th percentile and typical spread +SOUTH_NI,B,137.29,0.1757,0,30th percentile and typical spread +SOUTH_NI,C,156.57,0.1898,0,30th percentile and typical spread +SOUTH_NI,D,176.69,0.2194,0,30th percentile and typical spread +SOUTH_NI,E,213.6,0.249,0,30th percentile and typical spread +SOUTH_WEST_ESSEX,A,149.97,0.1336,173,list +SOUTH_WEST_ESSEX,B,215.49,0.2062,2665,list +SOUTH_WEST_ESSEX,C,277.67,0.2235,5107,list +SOUTH_WEST_ESSEX,D,335.21,0.1868,2982,list +SOUTH_WEST_ESSEX,E,420.21,0.2251,697,list +SOUTH_WEST_HERTS,A,143.57,0.2266,163,list +SOUTH_WEST_HERTS,B,263.38,0.1548,2977,list +SOUTH_WEST_HERTS,C,335.21,0.1673,4482,list +SOUTH_WEST_HERTS,D,420.95,0.189,1896,list +SOUTH_WEST_HERTS,E,555.33,0.2488,611,list +SOUTH_WEST_NI,A,89.17,0.1816,0,30th percentile and typical spread +SOUTH_WEST_NI,B,110.6,0.1757,0,30th percentile and typical spread +SOUTH_WEST_NI,C,132.13,0.1898,0,30th percentile and typical spread +SOUTH_WEST_NI,D,144.76,0.2194,0,30th percentile and typical spread +SOUTH_WEST_NI,E,172.72,0.249,0,30th percentile and typical spread +STAFFORDSHIRE_NORTH,A,93.77,0.1479,331,list +STAFFORDSHIRE_NORTH,B,117.41,0.1584,876,list +STAFFORDSHIRE_NORTH,C,143.48,0.2247,4272,list +STAFFORDSHIRE_NORTH,D,177.79,0.2279,2466,list +STAFFORDSHIRE_NORTH,E,239.14,0.3275,352,list +STEVENAGE_NORTH_HERTS,A,117.64,0.1571,373,list +STEVENAGE_NORTH_HERTS,B,217.69,0.1453,999,list +STEVENAGE_NORTH_HERTS,C,288.78,0.1465,1495,list +STEVENAGE_NORTH_HERTS,D,355.42,0.126,870,list +STEVENAGE_NORTH_HERTS,E,419.02,0.1734,190,list +ST_HELENS,A,86.89,0.1655,164,list +ST_HELENS,B,121.99,0.1817,215,list +ST_HELENS,C,155.26,0.1458,1094,list +ST_HELENS,D,194.91,0.2282,807,list +ST_HELENS,E,262.89,0.2795,84,list +SUNDERLAND,A,86.13,0.1332,87,list +SUNDERLAND,B,118.75,0.1494,294,list +SUNDERLAND,C,136.14,0.1856,1700,list +SUNDERLAND,D,149.94,0.2092,1069,list +SUNDERLAND,E,198.08,0.338,151,list +SUSSEX_EAST,A,101.39,0.2248,109,list +SUSSEX_EAST,B,173.04,0.2022,1139,list +SUSSEX_EAST,C,214.8,0.218,2149,list +SUSSEX_EAST,D,268.33,0.2526,1134,list +SUSSEX_EAST,E,337.76,0.2703,226,list +SWANSEA,A,102.05,0.2443,187,list +SWANSEA,B,155.75,0.2363,1134,list +SWANSEA,C,179.71,0.2803,2255,list +SWANSEA,D,185.44,0.2898,1720,list +SWANSEA,E,263.58,0.3308,243,list +SWINDON,A,123.58,0.1969,290,list +SWINDON,B,179.69,0.1092,1400,list +SWINDON,C,213.39,0.1244,3185,list +SWINDON,D,275.16,0.1931,1723,list +SWINDON,E,348.15,0.1892,416,list +TAFF_RHONDDA,A,78.24,0.2619,145,list +TAFF_RHONDDA,B,130.41,0.315,93,list +TAFF_RHONDDA,C,151.27,0.3276,617,list +TAFF_RHONDDA,D,156.48,0.3155,885,list +TAFF_RHONDDA,E,203.68,0.4097,102,list +TAMESIDE_GLOSSOP,A,109.75,0.239,146,list +TAMESIDE_GLOSSOP,B,156.61,0.1394,231,list +TAMESIDE_GLOSSOP,C,188.53,0.1492,1644,list +TAMESIDE_GLOSSOP,D,226.77,0.2133,772,list +TAMESIDE_GLOSSOP,E,310.52,0.1614,110,list +TAUNTON_WEST_SOMERSET,A,123.84,0.1765,459,list +TAUNTON_WEST_SOMERSET,B,154.17,0.1969,858,list +TAUNTON_WEST_SOMERSET,C,201.61,0.1943,2402,list +TAUNTON_WEST_SOMERSET,D,237.19,0.2556,1769,list +TAUNTON_WEST_SOMERSET,E,314.46,0.2819,402,list +TEESSIDE,A,79.74,0.1312,241,list +TEESSIDE,B,114.19,0.1817,632,list +TEESSIDE,C,131.95,0.208,2780,list +TEESSIDE,D,154.07,0.2293,2756,list +TEESSIDE,E,229.49,0.3149,414,list +THANET,A,108.75,0.1875,445,list +THANET,B,167.07,0.1502,863,list +THANET,C,214.8,0.1681,1434,list +THANET,D,279.52,0.1958,605,list +THANET,E,324.24,0.2108,152,list +TORFAEN,A,78.24,0.2619,145,list +TORFAEN,B,133.58,0.1948,98,list +TORFAEN,C,163.01,0.2031,317,list +TORFAEN,D,199.25,0.2068,336,list +TORFAEN,E,221.69,0.3158,118,list +TYNESIDE,A,99.54,0.1468,272,list +TYNESIDE,B,154.07,0.3416,1341,list +TYNESIDE,C,166.73,0.2898,6432,list +TYNESIDE,D,197.91,0.3201,3007,list +TYNESIDE,E,287.87,0.3787,546,list +VALE_OF_GLAMORGAN,A,78.24,0.2619,145,list +VALE_OF_GLAMORGAN,B,160.44,0.2733,338,list +VALE_OF_GLAMORGAN,C,203.68,0.2469,816,list +VALE_OF_GLAMORGAN,D,215.65,0.2509,615,list +VALE_OF_GLAMORGAN,E,347.45,0.2944,95,list +WAKEFIELD,A,98.73,0.1616,256,list +WAKEFIELD,B,129.97,0.1945,709,list +WAKEFIELD,C,158.23,0.2032,3143,list +WAKEFIELD,D,188.86,0.2212,1859,list +WAKEFIELD,E,258.5,0.2746,315,list +WALTON,A,162.42,0.1786,352,list +WALTON,B,268.33,0.1541,1439,list +WALTON,C,335.43,0.1807,2340,list +WALTON,D,429.61,0.2499,1125,list +WALTON,E,596.67,0.3554,454,list +WARWICKSHIRE_SOUTH,A,113.91,0.1562,358,list +WARWICKSHIRE_SOUTH,B,191.31,0.1742,1195,list +WARWICKSHIRE_SOUTH,C,227.64,0.1929,3504,list +WARWICKSHIRE_SOUTH,D,298.92,0.1955,1685,list +WARWICKSHIRE_SOUTH,E,375.71,0.2126,509,list +WESTON_S_MARE,A,129.45,0.1275,150,list +WESTON_S_MARE,B,166.04,0.1708,613,list +WESTON_S_MARE,C,202.15,0.2067,1301,list +WESTON_S_MARE,D,272.77,0.2224,714,list +WESTON_S_MARE,E,351.95,0.2412,178,list +WEST_CHESHIRE,A,105.65,0.1241,533,list +WEST_CHESHIRE,B,160.81,0.2284,947,list +WEST_CHESHIRE,C,198.77,0.1961,3457,list +WEST_CHESHIRE,D,228.89,0.2695,2412,list +WEST_CHESHIRE,E,354.88,0.2769,578,list +WEST_CUMBRIA,A,89.64,0.2233,465,list +WEST_CUMBRIA,B,110.83,0.2149,498,list +WEST_CUMBRIA,C,133.08,0.2197,1415,list +WEST_CUMBRIA,D,165.04,0.2499,865,list +WEST_CUMBRIA,E,210.71,0.3329,167,list +WEST_DUNBARTONSHIRE,A,98.97,0.1898,117,30th percentile and list spread +WEST_DUNBARTONSHIRE,B,122.72,0.1223,279,30th percentile and list spread +WEST_DUNBARTONSHIRE,C,168.59,0.1542,546,30th percentile and list spread +WEST_DUNBARTONSHIRE,D,202.76,0.2385,173,30th percentile and list spread +WEST_DUNBARTONSHIRE,E,294.93,0.2258,28,30th percentile and list spread +WEST_LOTHIAN,A,115.69,0.1802,173,30th percentile and list spread +WEST_LOTHIAN,B,131.1,0.0833,296,30th percentile and list spread +WEST_LOTHIAN,C,168.63,0.0908,1038,30th percentile and list spread +WEST_LOTHIAN,D,217.52,0.1648,412,30th percentile and list spread +WEST_LOTHIAN,E,314.51,0.1839,129,30th percentile and list spread +WEST_PENNINE,A,85.32,0.1654,361,list +WEST_PENNINE,B,114.34,0.1756,288,list +WEST_PENNINE,C,128.64,0.1513,1881,list +WEST_PENNINE,D,150.59,0.2195,739,list +WEST_PENNINE,E,225.07,0.3437,79,list +WEST_WILTSHIRE,A,125.71,0.1817,295,list +WEST_WILTSHIRE,B,162.85,0.1677,1054,list +WEST_WILTSHIRE,C,201.03,0.1692,2569,list +WEST_WILTSHIRE,D,258.31,0.2322,1822,list +WEST_WILTSHIRE,E,343.93,0.2203,449,list +WIGAN,A,99.81,0.198,207,list +WIGAN,B,119.26,0.2264,736,list +WIGAN,C,154.16,0.2067,3899,list +WIGAN,D,176.33,0.25,2773,list +WIGAN,E,243.98,0.2899,312,list +WINCHESTER,A,125.74,0.1209,257,list +WINCHESTER,B,232.7,0.1817,527,list +WINCHESTER,C,301.88,0.1816,1172,list +WINCHESTER,D,356.67,0.2066,645,list +WINCHESTER,E,464.17,0.2376,340,list +WIRRAL,A,89.46,0.1302,203,list +WIRRAL,B,119.26,0.1819,544,list +WIRRAL,C,156.61,0.2394,2222,list +WIRRAL,D,191.54,0.2133,1910,list +WIRRAL,E,252.98,0.3345,319,list +WOLDS_AND_COAST,A,82.99,0.0911,213,list +WOLDS_AND_COAST,B,110.15,0.2211,441,list +WOLDS_AND_COAST,C,144.64,0.1627,1117,list +WOLDS_AND_COAST,D,173.73,0.1945,640,list +WOLDS_AND_COAST,E,221.97,0.3108,112,list +WORCESTER_NORTH,A,97.24,0.2356,85,list +WORCESTER_NORTH,B,145.87,0.1796,1070,list +WORCESTER_NORTH,C,179.35,0.1709,2029,list +WORCESTER_NORTH,D,212.45,0.2106,1873,list +WORCESTER_NORTH,E,301.9,0.2542,250,list +WORCESTER_SOUTH,A,100.94,0.132,220,list +WORCESTER_SOUTH,B,155.44,0.1317,902,list +WORCESTER_SOUTH,C,197.29,0.136,1886,list +WORCESTER_SOUTH,D,239.14,0.2063,1100,list +WORCESTER_SOUTH,E,324.27,0.23,248,list +WORTHING,A,125.62,0.1726,171,list +WORTHING,B,199.46,0.1462,1800,list +WORTHING,C,267.22,0.1899,2602,list +WORTHING,D,322.2,0.1681,1629,list +WORTHING,E,393.81,0.178,317,list +WREXHAM,A,108.58,0.1034,123,list +WREXHAM,B,131.79,0.1563,323,list +WREXHAM,C,149.97,0.1897,1043,list +WREXHAM,D,179.71,0.2195,581,list +WREXHAM,E,239.62,0.345,120,list +YEOVIL,A,130.46,0.222,288,list +YEOVIL,B,146.0,0.1592,945,list +YEOVIL,C,190.93,0.1752,1820,list +YEOVIL,D,241.47,0.2342,1445,list +YEOVIL,E,348.15,0.2009,305,list +YORK,A,124.29,0.172,148,list +YORK,B,188.86,0.1916,1053,list +YORK,C,216.99,0.1911,3105,list +YORK,D,245.35,0.2859,1414,list +YORK,E,341.58,0.2587,372,list diff --git a/policyengine_uk_data/tests/test_brma_rent_conditioning.py b/policyengine_uk_data/tests/test_brma_rent_conditioning.py new file mode 100644 index 00000000..56e986e8 --- /dev/null +++ b/policyengine_uk_data/tests/test_brma_rent_conditioning.py @@ -0,0 +1,519 @@ +"""Rent-conditioned BRMA assignment for private renters. + +Invariants of ``brma_probabilities`` and ``draw_brmas``, for any census table, +list-of-rents table, model and households with data for their cells: + +1. Support: each row of probabilities sums to 1 and is zero outside the + household's region × bedrooms cell, and a draw never leaves that support. +2. Determinism: the same generator state gives the same BRMAs. +3. Fail closed: a household without a positive rent, a known region × bedrooms + cell, a list of rents for every BRMA in that cell, or a region shift raises. +4. Level invariance: scaling a region's rents (or its BRMAs' list medians) by + a constant and moving the region's shift to match changes nothing, so the + list of rents needs no further uprating once each region has its own shift. +5. No information, no change: if a cell's BRMAs have the same list of rents, + the probabilities are the census shares. +6. Agreement with a scalar reference implementation of the same formula. +7. Dearer rents, dearer BRMAs: without the below-market component, the + probability of the dearer of two otherwise equal BRMAs rises with rent. + (With it, very low rents revert towards the census shares: intended.) +8. Calibration: when rents come from the model, households' probabilities + average to the census shares, and the fit recovers the model. +""" + +import math + +import numpy as np +import pandas as pd +import pytest +from hypothesis import assume, given, settings +from hypothesis import strategies as st +from policyengine_uk.variables.household.demographic.locations import BRMAName + +from policyengine_uk.data import UKSingleYearDataset + +from policyengine_uk_data.datasets.brma import ( + BEDROOM_BAND_CATEGORY, + BRMA_HOUSEHOLDS_PATH, + BRMA_RENTS_PATH, + ReportedRentModel, + assign_private_renter_brmas, + bedroom_band, + brma_probabilities, + draw_brmas, + fit_reported_rent_model, +) +from policyengine_uk_data.datasets.frs_release import CURRENT_FRS_RELEASE +from policyengine_uk_data.storage import STORAGE_FOLDER + +REGIONS = ["LONDON", "WALES", "SCOTLAND", "NORTHERN_IRELAND"] +BANDS = list(BEDROOM_BAND_CATEGORY) +CATEGORIES = ["A", "B", "C", "D", "E"] + + +@pytest.fixture(scope="module") +def households(): + return pd.read_csv(BRMA_HOUSEHOLDS_PATH, dtype={"bedrooms": str}) + + +@pytest.fixture(scope="module") +def rents(): + return pd.read_csv(BRMA_RENTS_PATH) + + +def test_rents_table_covers_every_brma_and_category(rents): + assert list(rents.columns)[:4] == [ + "brma", + "lha_category", + "median_weekly_rent", + "log_sd", + ] + assert not rents.duplicated(["brma", "lha_category"]).any() + assert set(rents.brma) == set(BRMAName.__members__) + assert (rents.groupby("brma").lha_category.agg(set) == set(CATEGORIES)).all() + assert rents.median_weekly_rent.between(40, 2_000).all() + assert rents.log_sd.between(0.03, 0.6).all() + + +def test_each_nation_uses_its_best_source(households, rents): + by_region = households.groupby(["brma", "region"]).households.sum().reset_index() + region = by_region.sort_values("households").groupby("brma").region.last() + nation = region.where( + region.isin(["WALES", "SCOTLAND", "NORTHERN_IRELAND"]), "ENGLAND" + ) + basis = rents.groupby("brma").basis.agg(set).map(lambda b: b.pop()) + assert basis.groupby(nation).agg(set).to_dict() == { + "ENGLAND": {"list"}, + "WALES": {"list"}, + "SCOTLAND": {"30th percentile and list spread"}, + "NORTHERN_IRELAND": {"30th percentile and typical spread"}, + } + # Every cell with its own spread has enough rents to estimate it. + assert ( + rents[rents.basis != "30th percentile and typical spread"].rents >= 25 + ).all() + + +def test_larger_homes_cost_more_in_nearly_every_brma(rents): + median = rents.pivot( + index="brma", columns="lha_category", values="median_weekly_rent" + ) + for small, large in zip("ABCD", "BCDE"): + assert (median[large] > median[small]).mean() > 0.97, (small, large) + + +def test_bedroom_bands(): + assert ( + bedroom_band([1, 2, 3, 4, 5, 6]).tolist() == ["1", "2", "3", "4+"] + ["4+"] * 2 + ) + for bad in (0, -1, np.nan): + with pytest.raises(ValueError, match="at least one bedroom"): + bedroom_band([2, bad]) + + +@st.composite +def tables(draw): + """A census table and a list-of-rents table for a few made-up BRMAs.""" + regions = draw( + st.lists(st.sampled_from(REGIONS), min_size=1, max_size=3, unique=True) + ) + count = draw(st.integers(1, 4)) + census, lists = [], [] + for region in regions: + # Northern Ireland's census has one band, ``all``. + for band in ["all"] if region == "NORTHERN_IRELAND" else BANDS: + sizes = [draw(st.sampled_from([0, 1, 7, 1000])) for _ in range(count)] + sizes[draw(st.integers(0, count - 1))] = 5 # a non-empty cell + census += [ + (region, f"{region}_{i}", band, size) + for i, size in enumerate(sizes) + if size + ] + for i in range(count): + for category in BEDROOM_BAND_CATEGORY.values(): + median = draw(st.floats(40, 2_000, allow_nan=False)) + lists.append( + (f"{region}_{i}", category, median, draw(st.floats(0.05, 0.6))) + ) + return ( + pd.DataFrame(census, columns=["region", "brma", "bedrooms", "households"]), + pd.DataFrame( + lists, columns=["brma", "lha_category", "median_weekly_rent", "log_sd"] + ), + ) + + +@st.composite +def cases(draw, below_market=True): + census, lists = draw(tables()) + regions = sorted(census.region.unique()) + n = draw(st.integers(1, 25)) + region = np.array([draw(st.sampled_from(regions)) for _ in range(n)]) + bedrooms = np.array([draw(st.integers(1, 6)) for _ in range(n)]) + rent = np.array([draw(st.floats(1, 5_000, allow_nan=False)) for _ in range(n)]) + noise = draw(st.floats(0.01, 1)) + model = ReportedRentModel( + region_shift={r: draw(st.floats(-1, 1)) for r in regions}, + noise=noise, + below_market_share=draw(st.floats(0.001, 0.5)) if below_market else 0.0, + below_market_discount=draw(st.floats(0, 2)), + below_market_noise=noise + draw(st.floats(0.05, 2)), + ) + return census, lists, region, bedrooms, rent, model + + +def supported(census, region, band, brma): + cell = census[ + (census.region == region) + & census.bedrooms.isin([band, "all"]) + & (census.households > 0) + ] + return brma in set(cell.brma) + + +@settings(max_examples=200, deadline=None, derandomize=True) +@given(cases(), st.integers(0, 2**32 - 1)) +def test_probabilities_and_draws_stay_on_the_census_support(case, seed): + census, lists, region, bedrooms, rent, model = case + brmas, p = brma_probabilities(region, bedrooms, rent, model, census, lists) + assert p.shape == (len(region), len(brmas)) + assert np.isfinite(p).all() and (p >= 0).all() + assert np.allclose(p.sum(axis=1), 1) + band = bedroom_band(bedrooms) + for i, j in zip(*np.nonzero(p)): + assert supported(census, region[i], band[i], brmas[j]) + first = draw_brmas(brmas, p, np.random.default_rng(seed)) + second = draw_brmas(brmas, p, np.random.default_rng(seed)) + assert list(first) == list(second) + for i, brma in enumerate(first): + assert p[i, list(brmas).index(brma)] > 0 + assert supported(census, region[i], band[i], brma) + + +@settings(max_examples=100, deadline=None, derandomize=True) +@given(cases(), st.sampled_from(["rent", "cell", "list", "shift", "bedrooms"])) +def test_missing_inputs_fail_closed(case, what): + census, lists, region, bedrooms, rent, model = case + if what == "rent": + rent = rent.copy() + rent[0] = [0.0, -5.0, np.nan][len(rent) % 3] + match = "positive weekly rent" + elif what == "cell": + census = census[census.region != region[0]] + assume(len(census)) + match = "No BRMA weights" + elif what == "list": + band = bedroom_band(bedrooms)[0] + cell = census[ + (census.region == region[0]) & census.bedrooms.isin([band, "all"]) + ] + lists = lists[ + ~( + (lists.brma == cell.brma.iloc[0]) + & (lists.lha_category == BEDROOM_BAND_CATEGORY[band]) + ) + ] + match = "No list of rents" + elif what == "shift": + shifts = {r: s for r, s in model.region_shift.items() if r != region[0]} + model = ReportedRentModel(shifts, *list(model.__dict__.values())[1:]) + match = "No reported-rent shift" + else: + bedrooms = bedrooms.astype(float) + bedrooms[0] = 0 + match = "at least one bedroom" + with pytest.raises(ValueError, match=match): + brma_probabilities(region, bedrooms, rent, model, census, lists) + + +@settings(max_examples=100, deadline=None, derandomize=True) +@given(cases(), st.floats(0.25, 4)) +def test_a_regions_rent_level_is_absorbed_by_its_shift(case, factor): + census, lists, region, bedrooms, rent, model = case + _, base = brma_probabilities(region, bedrooms, rent, model, census, lists) + target = region[0] + shifted = ReportedRentModel( + { + r: s + (math.log(factor) if r == target else 0) + for r, s in model.region_shift.items() + }, + *list(model.__dict__.values())[1:], + ) + scaled_rent = np.where(region == target, rent * factor, rent) + _, a = brma_probabilities(region, bedrooms, scaled_rent, shifted, census, lists) + assert np.allclose(a, base, atol=1e-9) + # Equivalently, uprate the region's lists and lower its shift. + uprated = lists.copy() + in_target = uprated.brma.str.startswith(target) + uprated.loc[in_target, "median_weekly_rent"] *= factor + lowered = ReportedRentModel( + { + r: s - (math.log(factor) if r == target else 0) + for r, s in model.region_shift.items() + }, + *list(model.__dict__.values())[1:], + ) + _, b = brma_probabilities(region, bedrooms, rent, lowered, census, uprated) + assert np.allclose(b, base, atol=1e-9) + + +@settings(max_examples=100, deadline=None, derandomize=True) +@given(cases(), st.floats(40, 2_000), st.floats(0.05, 0.6)) +def test_identical_lists_leave_the_census_shares(case, median, log_sd): + census, lists, region, bedrooms, rent, model = case + lists = lists.assign(median_weekly_rent=median, log_sd=log_sd) + brmas, p = brma_probabilities(region, bedrooms, rent, model, census, lists) + band = bedroom_band(bedrooms) + for i in range(len(region)): + cell = census[ + (census.region == region[i]) & census.bedrooms.isin([band[i], "all"]) + ] + share = cell.set_index("brma").households / cell.households.sum() + assert np.allclose(p[i], share.reindex(brmas, fill_value=0).to_numpy()) + + +def reference_probabilities(census, lists, region, band, rent, model): + """The same formula for one household, written out with scalars.""" + cell = census[(census.region == region) & census.bedrooms.isin([band, "all"])] + result = {} + for _, row in cell.iterrows(): + entry = lists[ + (lists.brma == row.brma) + & (lists.lha_category == BEDROOM_BAND_CATEGORY[band]) + ].iloc[0] + error = ( + math.log(rent) + - model.region_shift[region] + - math.log(entry.median_weekly_rent) + ) + + def normal(x, variance): + return math.exp(-0.5 * x * x / variance) / math.sqrt(variance) + + density = (1 - model.below_market_share) * normal( + error, entry.log_sd**2 + model.noise**2 + ) + model.below_market_share * normal( + error + model.below_market_discount, + entry.log_sd**2 + model.below_market_noise**2, + ) + result[row.brma] = row.households * density + total = sum(result.values()) + return {brma: value / total for brma, value in result.items()} + + +@settings(max_examples=100, deadline=None, derandomize=True) +@given(cases()) +def test_probabilities_match_the_scalar_reference(case): + census, lists, region, bedrooms, rent, model = case + # Keep rents where the scalar densities do not underflow. + rent = np.clip(rent, 30, 3_000) + brmas, p = brma_probabilities(region, bedrooms, rent, model, census, lists) + band = bedroom_band(bedrooms) + for i in range(len(region)): + want = reference_probabilities( + census, lists, region[i], band[i], rent[i], model + ) + assume(all(math.isfinite(v) for v in want.values())) + got = dict(zip(brmas, p[i])) + for brma, value in want.items(): + assert got[brma] == pytest.approx(value, rel=1e-6, abs=1e-12) + + +@settings(max_examples=100, deadline=None, derandomize=True) +@given( + st.floats(60, 400), + st.floats(1.05, 3), + st.floats(0.05, 0.6), + st.floats(0.01, 1), + st.lists(st.floats(20, 3_000), min_size=2, max_size=12), +) +def test_higher_rents_favour_the_dearer_brma(median, premium, log_sd, noise, rent): + census = pd.DataFrame( + [("WALES", "CHEAP", "2", 300), ("WALES", "DEAR", "2", 100)], + columns=["region", "brma", "bedrooms", "households"], + ) + lists = pd.DataFrame( + [("CHEAP", "C", median, log_sd), ("DEAR", "C", median * premium, log_sd)], + columns=["brma", "lha_category", "median_weekly_rent", "log_sd"], + ) + model = ReportedRentModel({"WALES": 0.0}, noise, 0.0, 0.5, noise + 0.5) + rent = np.sort(np.array(rent)) + brmas, p = brma_probabilities( + np.full(len(rent), "WALES"), np.full(len(rent), 2), rent, model, census, lists + ) + dear = p[:, list(brmas).index("DEAR")] + assert (np.diff(dear) >= -1e-12).all() + + +def simulate(households, rents, model, n, seed): + """Households drawn from the census, with rents drawn from ``model``.""" + rng = np.random.default_rng(seed) + banded = households[households.bedrooms != "all"] + cells = banded.sample(n, weights="households", replace=True, random_state=rng) + bedrooms = cells.bedrooms.map({"1": 1, "2": 2, "3": 3, "4+": 5}).to_numpy() + lists = rents.set_index(["brma", "lha_category"]) + key = list(zip(cells.brma, cells.bedrooms.map(BEDROOM_BAND_CATEGORY))) + below = rng.random(n) < model.below_market_share + log_rent = ( + np.log(lists.median_weekly_rent.loc[key].to_numpy()) + + lists.log_sd.loc[key].to_numpy() * rng.standard_normal(n) + + cells.region.map(model.region_shift).to_numpy() + + np.where( + below, + -model.below_market_discount + + model.below_market_noise * rng.standard_normal(n), + model.noise * rng.standard_normal(n), + ) + ) + return cells.region.to_numpy(), bedrooms, np.exp(log_rent), cells.brma.to_numpy() + + +def test_model_draws_average_to_the_census_and_the_fit_recovers_the_model( + households, rents +): + regions = sorted(set(households.region) - {"NORTHERN_IRELAND"}) + truth = ReportedRentModel( + {r: -0.05 - 0.01 * i for i, r in enumerate(regions)}, 0.13, 0.16, 0.6, 0.7 + ) + region, bedrooms, rent, true_brma = simulate(households, rents, truth, 40_000, 3) + weight = np.ones(len(rent)) + fitted = fit_reported_rent_model(region, bedrooms, rent, weight) + assert fitted.noise == pytest.approx(truth.noise, abs=0.03) + assert fitted.below_market_share == pytest.approx( + truth.below_market_share, abs=0.03 + ) + assert fitted.below_market_discount == pytest.approx(0.6, abs=0.1) + assert fitted.below_market_noise == pytest.approx(0.7, abs=0.1) + for r in regions: + assert fitted.region_shift[r] == pytest.approx(truth.region_shift[r], abs=0.03) + brmas, p = brma_probabilities(region, bedrooms, rent, truth) + drawn = draw_brmas(brmas, p, np.random.default_rng(4)) + medians = rents[rents.lha_category == "C"].set_index("brma").median_weekly_rent + for r in regions: + rows = region == r + expected = p[rows].mean(axis=0) + actual = pd.Series(true_brma[rows]).value_counts(normalize=True) + actual = actual.reindex(brmas, fill_value=0).to_numpy() + # Probabilities average to where the simulated households really live. + assert 0.5 * np.abs(expected - actual).sum() < 0.06, r + # The drawn BRMAs' rent levels track rents as the true BRMAs' do. + want = np.corrcoef(np.log(rent[rows]), np.log(medians[true_brma[rows]]))[0, 1] + got = np.corrcoef(np.log(rent[rows]), np.log(medians[drawn[rows]]))[0, 1] + assert got == pytest.approx(want, abs=0.06), r + + +def test_assignment_is_deterministic_and_fit_needs_valid_inputs(households, rents): + truth = ReportedRentModel( + {r: -0.1 for r in households.region.unique()}, 0.15, 0.1, 0.5, 0.6 + ) + region, bedrooms, rent, _ = simulate(households, rents, truth, 3_000, 5) + weight = np.linspace(1, 2, len(rent)) + first = assign_private_renter_brmas( + region, bedrooms, rent, weight, np.random.default_rng(0) + ) + second = assign_private_renter_brmas( + region, bedrooms, rent, weight, np.random.default_rng(0) + ) + assert list(first) == list(second) + with pytest.raises(ValueError, match="positive and weights non-negative"): + fit_reported_rent_model(region, bedrooms, rent, -weight) + with pytest.raises(ValueError, match="positive and weights non-negative"): + fit_reported_rent_model(region, bedrooms, np.zeros(len(rent)), weight) + + +@pytest.fixture(scope="module") +def built_renters(): + """Private renters who report a rent, from the raw survey and the built FRS. + + The dataset does not keep bedrooms, so they are read from the raw household + table, as the build reads them. + """ + raw_path = STORAGE_FOLDER / CURRENT_FRS_RELEASE.name / "househol.tab" + built_path = STORAGE_FOLDER / CURRENT_FRS_RELEASE.base_dataset_file + if not (raw_path.exists() and built_path.exists()): + pytest.skip("Raw FRS household table or built FRS dataset not available") + raw = pd.read_csv(raw_path, sep="\t", low_memory=False) + raw.columns = raw.columns.str.lower() + raw = raw[["sernum", "ptentyp2", "hhrent", "bedroom6", "gross4"]] + raw = raw.apply(pd.to_numeric, errors="coerce") + raw = raw[raw.ptentyp2.isin([3, 4]) & (raw.hhrent > 0)] + built = UKSingleYearDataset(built_path).household.set_index("household_id") + built = built.loc[raw.sernum] + assert (built.tenure_type.astype(str) == "RENT_PRIVATELY").all() + renters = pd.DataFrame( + { + "region": built.region.astype(str).to_numpy(), + "bedrooms": raw.bedroom6.to_numpy(), + "weekly_rent": raw.hhrent.to_numpy(), + "weight": raw.gross4.to_numpy(), + "brma": built.brma.astype(str).to_numpy(), + } + ) + args = (renters.region, renters.bedrooms, renters.weekly_rent) + model = fit_reported_rent_model(*args, renters.weight) + brmas, p = brma_probabilities(*args, model) + flat = ReportedRentModel(model.region_shift, 50.0, 0.0, 0.0, 51.0) + _, census = brma_probabilities(*args, flat) + return renters, brmas, p, census + + +def test_built_private_renters_match_the_census_within_each_region(built_renters): + """Within each region, the built private renters' weighted mean probability + of each BRMA stays within 5 percentage points (total variation) of the + census shares for their homes' bedroom bands.""" + renters, brmas, p, census = built_renters + for region in renters.region.unique(): + rows = (renters.region == region).to_numpy() + w = renters.weight[rows].to_numpy() + w = w / w.sum() + gap = 0.5 * np.abs(w @ p[rows] - w @ census[rows]).sum() + assert gap < 0.05, (region, gap) + + +def test_built_private_renters_brmas_are_draws_from_their_probabilities( + built_renters, +): + """The BRMAs in the built dataset look like independent draws from the + probabilities, scored by their log probability. + + For independent draws the log score's mean and variance are exact sums + over households, so the standardised score is close to normal. A draw + from the census shares instead (uk-data#516's) scores about -19. + """ + renters, brmas, p, _ = built_renters + drawn = pd.Index(brmas).get_indexer(renters.brma) + rows = np.arange(len(renters)) + assert (p[rows, drawn] > 0).all() + log_p = np.log(np.where(p > 0, p, 1)) + score = np.log(p[rows, drawn]).sum() + expected = (p * log_p).sum() + variance = ((p * log_p**2).sum(axis=1) - (p * log_p).sum(axis=1) ** 2).sum() + assert abs(score - expected) / np.sqrt(variance) < 4 + + +def test_built_private_renters_rents_track_their_brmas_rent_levels( + built_renters, rents +): + """Dearer rents sit in dearer BRMAs, within region and bedroom band.""" + renters, *_ = built_renters + category = pd.Series(bedroom_band(renters.bedrooms)).map(BEDROOM_BAND_CATEGORY) + listed = rents.set_index(["brma", "lha_category"]).median_weekly_rent + frame = pd.DataFrame( + { + "cell": renters.region + category.to_numpy(), + "rent": np.log(renters.weekly_rent), + "level": np.log(listed.loc[list(zip(renters.brma, category))].to_numpy()), + "w": renters.weight, + } + ) + for column in ("rent", "level"): + mean = frame.groupby("cell").apply( + lambda g: np.average(g[column], weights=g.w), include_groups=False + ) + frame[column] -= frame.cell.map(mean) + covariance = np.average(frame.rent * frame.level, weights=frame.w) + correlation = covariance / np.sqrt( + np.average(frame.rent**2, weights=frame.w) + * np.average(frame.level**2, weights=frame.w) + ) + assert correlation > 0.15, correlation diff --git a/pyproject.toml b/pyproject.toml index 7973343c..5d7e3493 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -31,6 +31,7 @@ dependencies = [ "openpyxl", "pydantic>=2.0", "pyyaml", + "scipy", "sqlmodel>=0.0.16", ] diff --git a/tools/brma_rents/README.md b/tools/brma_rents/README.md new file mode 100644 index 00000000..8b522ceb --- /dev/null +++ b/tools/brma_rents/README.md @@ -0,0 +1,30 @@ +# BRMA private rents + +Rebuilds `policyengine_uk_data/storage/brma_private_rents.csv`: for each Broad Rental Market Area (BRMA) and LHA category, the median weekly rent on the rent officers' list of rents and the spread of log rents, in the prices of the dataset year. `datasets/brma.py` uses it to place private renters given their rent. + +## Run + +From the repository root, in an environment with `openpyxl` and `xlrd` (the Welsh lists are `.xls`), for example `uv run --no-project --python .venv/bin/python --with xlrd`: + +``` +python tools/brma_rents/build.py # download, verify, write the CSV +python tools/brma_rents/build.py --check # compare the rebuild with the CSV +``` + +Every source in `sources.yaml` is downloaded once to the cache and checked against its pinned sha256 before anything is read. Fetching reuses `tools/brma_households/fetch.py`. + +## Method + +`--year` (default 2024) is the dataset year, April to March. The list of rents behind an April determination holds rents collected from October two years earlier to September of the year before. + +1. **Uprating.** Each list is moved to the dataset year with the ONS Price Index of Private Rents for its region or nation: the mean index over the dataset year divided by the mean index over the list's collection months. A BRMA that crosses a region boundary uses the region holding most of its private renters. +2. **England (`basis = list`).** The two VOA lists whose collection months overlap the dataset year (for 2024: April 2025 and April 2026) are uprated and pooled. The median is the median of log rents; the spread is their interquartile range divided by 1.349, the interquartile range of a standard normal. +3. **Scotland (`30th percentile and list spread`).** The spread comes from Rent Service Scotland's lists released under FOI 202200303624 (years to September 2019, 2020 and 2021, the latest released), averaged over sheets by their number of rents. The median comes from the 30th percentiles published for the same two determinations as England's lists, uprated, assuming log-normal rents: median = 30th percentile × exp(0.5244 × spread). +4. **Wales (`basis = list`).** As England, from Rent Officers Wales's lists for April 2024 and April 2025 (Welsh Government FOI ATISN 25142), the latest released. +5. **Northern Ireland (`30th percentile and typical spread`).** No list is published. The median comes from the 30th percentiles as for Scotland, for the latest determination with 30th percentiles in the published-rates file (April 2024); the spread is the median spread of the English, Welsh and Scottish cells of the same category. + +`rents` is the number of list entries behind a cell's spread (0 where the spread is borrowed). + +## When the dataset year changes + +Add the newer VOA lists, the newer published-rates file and a newer ONS index to `sources.yaml`, run with the new `--year`, and commit the CSV. A region-wide error in the level of rents does not change any household's BRMA probabilities, because the model fitted in `datasets/brma.py` has a free shift per region; only rents of BRMAs relative to others in their region matter. diff --git a/tools/brma_rents/build.py b/tools/brma_rents/build.py new file mode 100644 index 00000000..61717c3a --- /dev/null +++ b/tools/brma_rents/build.py @@ -0,0 +1,301 @@ +#!/usr/bin/env python3 +"""Rebuild ``storage/brma_private_rents.csv`` from pinned public sources. + +One row per BRMA and LHA category: the median weekly rent on the BRMA's list +of rents, in the prices of the dataset year, and the standard deviation of +log rents on the list. See ``README.md`` for the method. + +Usage (from the repository root, with openpyxl and xlrd installed): + python tools/brma_rents/build.py [--year 2024] [--check] +""" + +import argparse +import sys +from pathlib import Path +from statistics import NormalDist + +import numpy as np +import pandas as pd +import yaml + +HERE = Path(__file__).resolve().parent +STORAGE = HERE.parents[1] / "policyengine_uk_data" / "storage" +OUTPUT = STORAGE / "brma_private_rents.csv" +sys.path.insert(0, str(HERE.parent / "brma_households")) +from fetch import fetch # noqa: E402 + +CATEGORIES = list("ABCDE") +Z30 = NormalDist().inv_cdf(0.3) +IQR_SDS = NormalDist().inv_cdf(0.75) - NormalDist().inv_cdf(0.25) +SCOTTISH_SHEETS = (2019, 2020, 2021) # years to September; the latest three released +WELSH_SHEETS = {"Table 1": 2024, "Table 2": 2025} # sheet: April determination +PIPR_AREA = { + "North East": "NORTH_EAST", + "North West": "NORTH_WEST", + "Yorkshire and The Humber": "YORKSHIRE", + "East Midlands": "EAST_MIDLANDS", + "West Midlands": "WEST_MIDLANDS", + "East of England": "EAST_OF_ENGLAND", + "London": "LONDON", + "South East": "SOUTH_EAST", + "South West": "SOUTH_WEST", + "Wales": "WALES", + "Scotland": "SCOTLAND", + "Northern Ireland": "NORTHERN_IRELAND", +} + + +def brma_name(published: pd.Series) -> pd.Series: + """policyengine-uk's ``BRMAName`` for a published BRMA name.""" + name = published.str.upper().str.replace(r"[^A-Z0-9]+", "_", regex=True) + return name.str.strip("_").replace( + { + "HIGHLANDS_AND_ISLANDS": "HIGHLAND_AND_ISLANDS", + "RENFREWSHIRE_INVERCLYDE": "RENFREWSHIRE_AND_INVERCLYDE", + } + ) + + +def spread(log_rent: pd.Series) -> float: + """Standard deviation of log rents implied by their interquartile range.""" + return (log_rent.quantile(0.75) - log_rent.quantile(0.25)) / IQR_SDS + + +def uprating(cache: Path, year: int) -> pd.DataFrame: + """Log of each region's rent index in the dataset year over each list window. + + The list for April Y holds rents collected from October Y-2 to September + Y-1. Columns are April determination years; the dataset year runs from + April to March. + """ + table = pd.read_excel( + cache / "ons_price_index_of_private_rents_2026_09.xlsx", + sheet_name="Table 1", + header=2, + usecols=["Time period", "Area name", "Index"], + ) + table = table[table["Area name"].isin(PIPR_AREA)] + index = table.pivot(index="Time period", columns="Area name", values="Index") + # Months ONS has not yet published for an area are marked ``[x]``. + index = index.rename(columns=PIPR_AREA).apply(pd.to_numeric, errors="coerce") + months = pd.date_range(f"{year}-04-01", periods=12, freq="MS") + assert months.isin(index.index).all() and index.loc[months].notna().all().all() + target = index.loc[months].mean() + windows = {} + for determination in range(2017, int(index.index.max().year) + 1): + window = pd.date_range(f"{determination - 2}-10-01", periods=12, freq="MS") + if window.isin(index.index).all(): + windows[determination] = np.log( + target / index.loc[window].mean(skipna=False) + ) + return pd.DataFrame(windows) + + +def english_lists(cache: Path, determinations) -> pd.DataFrame: + """England's VOA lists: one row per rent, with its April determination.""" + lists = [] + for determination in determinations: + rows = pd.read_csv( + cache / f"voa_list_of_rents_{determination}.csv", + encoding="cp1252", + dtype=str, + ).dropna(how="all") + assert (rows.PERIOD == "Week").all(), determination + lists.append( + pd.DataFrame( + { + "brma": brma_name(rows.BRMA), + "lha_category": rows.LHA_TYPE.str.removeprefix("Cat "), + "weekly_rent": rows.NET_RENT.str.replace("[£,]", "", regex=True) + .astype(float) + .to_numpy(), + "determination": determination, + } + ) + ) + return pd.concat(lists) + + +def welsh_lists(cache: Path) -> pd.DataFrame: + """Rent Officers Wales's lists for April 2024 and April 2025 (FOI ATISN 25142).""" + lists = [] + for sheet, determination in WELSH_SHEETS.items(): + rows = pd.read_excel( + cache / "welsh_lists_of_rents_atisn25142.xls", sheet_name=sheet, header=2 + ).dropna(how="all") + lists.append( + pd.DataFrame( + { + "brma": brma_name(rows["BRMA"].astype(str).str.strip()), + "lha_category": rows["Category"].astype(str).str.strip(), + "weekly_rent": rows["Weekly Rent"].astype(float).to_numpy(), + "determination": determination, + } + ) + ) + return pd.concat(lists) + + +def list_cells(rents: pd.DataFrame, factor: pd.DataFrame, region: pd.Series): + """Median and spread of uprated log rents, pooled over determinations.""" + assert (rents.weekly_rent > 0).all() + uprate = [factor.loc[region[b], y] for b, y in zip(rents.brma, rents.determination)] + log_rent = pd.Series(np.log(rents.weekly_rent.to_numpy()) + uprate) + cell = log_rent.groupby([rents.brma.to_numpy(), rents.lha_category.to_numpy()]) + cells = pd.DataFrame( + { + "median_weekly_rent": np.exp(cell.median()), + "log_sd": cell.apply(spread), + "rents": cell.size(), + "basis": "list", + } + ) + cells.index.names = ["brma", "lha_category"] + return cells.reset_index() + + +def scottish_spreads(cache: Path) -> pd.DataFrame: + """Spread of log rents on Scotland's lists, averaged over the latest sheets.""" + sheets = [] + for year in SCOTTISH_SHEETS: + sheet = pd.read_excel( + cache / "scottish_list_of_rents_foi_2016_2021.xlsx", sheet_name=str(year) + ).dropna(how="all") + sheet = sheet[sheet["BRMA"].notna() & (sheet["NET RENT"] > 0)] + assert (sheet["FREQUENCY"].str.strip() == "Weekly").all(), year + sheets.append( + pd.DataFrame( + { + "sheet": year, + "brma": brma_name(sheet["BRMA"].str.strip()), + "lha_category": sheet["LHA TYPE"] + .str.strip() + .str.removeprefix("Cat "), + "log_rent": np.log(sheet["NET RENT"].astype(float)), + } + ) + ) + by_sheet = pd.concat(sheets).groupby(["brma", "lha_category", "sheet"]).log_rent + table = pd.DataFrame({"log_sd": by_sheet.apply(spread), "rents": by_sheet.size()}) + cell = table.reset_index().groupby(["brma", "lha_category"]) + return pd.DataFrame( + { + "log_sd": cell.apply( + lambda c: np.average(c.log_sd, weights=c.rents), include_groups=False + ), + "rents": cell.rents.sum(), + } + ).reset_index() + + +def percentile_cells(published, factor, region, brmas, determinations, spreads, basis): + """Median from published 30th percentiles, assuming log-normal rents. + + Each 30th percentile is uprated from its list's window to the dataset + year; the median is the 30th percentile raised by ``-Z30`` spreads. + """ + rows = published[ + published.brma.isin(brmas) + & published.year.isin(determinations) + & published.percentile_30.notna() + ] + log_p30 = np.log(rows.percentile_30) + [ + factor.loc[region[b], y] for b, y in zip(rows.brma, rows.year) + ] + cells = log_p30.groupby([rows.brma, rows.lha_category]).mean().rename("log_p30") + cells = cells.reset_index().merge( + spreads, on=spreads.columns.intersection(["brma", "lha_category"]).tolist() + ) + cells["median_weekly_rent"] = np.exp(cells.log_p30 - Z30 * cells.log_sd) + return cells.assign(basis=basis).drop(columns="log_p30") + + +def build(cache: Path, year: int) -> pd.DataFrame: + households = pd.read_csv(STORAGE / "brma_private_rented_households.csv") + # A BRMA that crosses a region boundary is uprated with the region that + # holds most of its private renters. + by_region = households.groupby(["brma", "region"]).households.sum() + region = ( + by_region.reset_index().sort_values("households").groupby("brma").region.last() + ) + factor = uprating(cache, year) # rows: region; columns: determination + published = pd.read_csv(cache / "lha_published_rates.csv.gzip", compression="gzip") + + def in_nation(nation): + return set(region.index[region == nation]) + + english = list_cells(english_lists(cache, [year + 1, year + 2]), factor, region) + welsh = list_cells(welsh_lists(cache), factor, region) + scottish = scottish_spreads(cache) + # No Northern Ireland list is used: take the typical spread of the other + # nations' lists for the category. + typical = ( + pd.concat([english, welsh, scottish]) + .groupby("lha_category") + .log_sd.median() + .reset_index() + ) + recent = [year + 1, year + 2] + northern_irish = published[ + published.brma.isin(in_nation("NORTHERN_IRELAND")) + & published.percentile_30.notna() + ].year + latest = [y for y in recent if y in set(northern_irish)] or [northern_irish.max()] + table = pd.concat( + [ + english, + welsh, + percentile_cells( + published, + factor, + region, + in_nation("SCOTLAND"), + recent, + scottish, + "30th percentile and list spread", + ), + percentile_cells( + published, + factor, + region, + in_nation("NORTHERN_IRELAND"), + latest, + typical.assign(rents=0), + "30th percentile and typical spread", + ), + ] + ) + table = table[ + ["brma", "lha_category", "median_weekly_rent", "log_sd", "rents", "basis"] + ] + table = table.sort_values(["brma", "lha_category"]).reset_index(drop=True) + expected = pd.MultiIndex.from_product([sorted(region.index), CATEGORIES]) + assert table.set_index(["brma", "lha_category"]).index.equals(expected) + assert table[["median_weekly_rent", "log_sd"]].notna().all().all() + table["median_weekly_rent"] = table.median_weekly_rent.round(2) + table["log_sd"] = table.log_sd.round(4) + return table + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("cache", type=Path) + parser.add_argument( + "--year", type=int, default=2024, help="dataset year (April start)" + ) + parser.add_argument( + "--check", action="store_true", help="compare with storage, do not write" + ) + args = parser.parse_args() + fetch(args.cache, yaml.safe_load((HERE / "sources.yaml").read_text())) + table = build(args.cache, args.year) + if args.check: + pd.testing.assert_frame_equal(table, pd.read_csv(OUTPUT)) + print(f"{OUTPUT.name} matches the rebuild ({len(table)} rows).") + else: + table.to_csv(OUTPUT, index=False) + print(f"Wrote {OUTPUT} ({len(table)} rows).") + + +if __name__ == "__main__": + main() diff --git a/tools/brma_rents/sources.yaml b/tools/brma_rents/sources.yaml new file mode 100644 index 00000000..8034831b --- /dev/null +++ b/tools/brma_rents/sources.yaml @@ -0,0 +1,54 @@ +- id: voa_list_of_rents_2025 + publisher: Valuation Office Agency + title: 'Shadow List of Rents April 2025: collated 1st October 2023 - 30th September 2024' + url: https://assets.publishing.service.gov.uk/media/67ea8a4dea9f8afd81056265/Shadow_List_of_Rents_April_2025.csv + landing_page: https://www.gov.uk/government/publications/shadow-list-of-rents-april-2025-collated-1st-october-2023-30th-september-2024 + licence: Open Government Licence v3.0 + notes: One row per rent on England's lists (weekly net rent, BRMA, LHA category). + filename: voa_list_of_rents_2025.csv + sha256: f9257d4685da972fb0b0bdcf38c574502e1a521981031fbd87a373f57082201e +- id: voa_list_of_rents_2026 + publisher: HM Revenue & Customs + title: 'Shadow List of Rents April 2026: collated 1st October 2024 - 30th September 2025' + url: https://assets.publishing.service.gov.uk/media/69c136e27e02b81c0d1c7652/Shadow_List_of_Rents_April_2026.csv + landing_page: https://www.gov.uk/government/publications/shadow-list-of-rents-april-2026-collated-1st-october-2024-30th-september-2025 + licence: Open Government Licence v3.0 + notes: One row per rent on England's lists (weekly net rent, BRMA, LHA category). + filename: voa_list_of_rents_2026.csv + sha256: 4ed4e12b2307a339545271dea3f4d5805cf25cdcb6f757d2f9dfa3c3fe22edbc +- id: scottish_list_of_rents_foi + publisher: Scottish Government + title: 'FOI 202200303624: information released, BRMA 2016-2021' + url: https://www.gov.scot/binaries/content/documents/govscot/publications/foi-eir-release/2023/08/foi-202200303624/documents/foi-202200303624---information-released---brma-2016-2021/foi-202200303624---information-released---brma-2016-2021/govscot%3Adocument/FOI%2B202200303624%2B-%2BInformation%2BReleased%2B-%2BBRMA%2B2016-2021.xlsx?download=true + landing_page: https://www.gov.scot/publications/foi-202200303624/ + licence: Open Government Licence v3.0 + notes: Rent Service Scotland's list-of-rents records, one sheet per year to September (2016-2021). Used for the spread of rents only. + filename: scottish_list_of_rents_foi_2016_2021.xlsx + sha256: dba86fa1cce920c8f646460bb13e1d99e16927d57fdf290e874f9631e804ed06 +- id: welsh_lists_of_rents + publisher: Welsh Government (Rent Officers Wales) + title: 'ATISN 25142: lists of rents collected October 2022 - September 2023 (Table 1) and shadow list October 2023 - September 2024 (Table 2)' + url: https://www.gov.wales/sites/default/files/publications/2025-10/atisn25142doc1.xls + landing_page: https://www.gov.wales/atisn25142 + licence: Open Government Licence v3.0 + notes: One row per rent (category, BRMA, weekly rent), released 6 October 2025. Table 1 is the list for April 2024 and Table 2 the shadow list for April 2025. + filename: welsh_lists_of_rents_atisn25142.xls + sha256: 8ab047dba4b8a6808cadbeb33f81e076a527930d68cb9cb4830be9d25c61c9c2 +- id: lha_published_rates + publisher: PolicyEngine + title: Published LHA rates and 30th percentiles by BRMA, category and April determination (policyengine-uk#2022) + url: https://raw.githubusercontent.com/PolicyEngine/policyengine-uk/23a31d5ec35af08ae0900093a08e81e4f0c0fc28/policyengine_uk/parameters/gov/dwp/LHA/lha_published_rates.csv.gz + landing_page: https://github.com/PolicyEngine/policyengine-uk/pull/2022 + licence: Open Government Licence v3.0 (compiled from Scottish Government, Rent Officers Wales, NIHE, VOA and DWP tables) + notes: Built by policyengine-uk's scripts/build_lha_published_rates.py from the publishers' tables, each pinned by sha256 in lha_published_rates_sources.csv. Supplies the published 30th percentiles for Scotland, Wales and Northern Ireland. + filename: lha_published_rates.csv.gzip + sha256: 221ff310567970043d329c9cbaf837d7160a8bde560bfe6896195a76263d51d0 +- id: ons_price_index_of_private_rents + publisher: Office for National Statistics + title: 'Price Index of Private Rents, UK: monthly price statistics, 16 September 2026' + url: https://www.ons.gov.uk/file?uri=/economy/inflationandpriceindices/datasets/priceindexofprivaterentsukmonthlypricestatistics/16september2026/priceindexofprivaterentsukmonthlypricestatistics.xlsx + landing_page: https://www.ons.gov.uk/economy/inflationandpriceindices/datasets/priceindexofprivaterentsukmonthlypricestatistics + licence: Open Government Licence v3.0 + notes: Table 1, monthly index (January 2023 = 100) by country and English region, January 2015 to August 2026. Northern Ireland's latest two months are ONS estimates. + filename: ons_price_index_of_private_rents_2026_09.xlsx + sha256: ab1fdfbb27738b20cd1762521a19489c9274dedae1e39926e734ac1cf867f2de diff --git a/uv.lock b/uv.lock index 672737ea..9c713086 100644 --- a/uv.lock +++ b/uv.lock @@ -1452,6 +1452,7 @@ dependencies = [ { name = "requests" }, { name = "rich" }, { name = "ruff" }, + { name = "scipy" }, { name = "sqlmodel" }, { name = "tabulate" }, { name = "tqdm" }, @@ -1499,6 +1500,7 @@ requires-dist = [ { name = "rich", specifier = ">=13.0.0" }, { name = "ruff", specifier = ">=0.9.0" }, { name = "ruff", marker = "extra == 'dev'", specifier = ">=0.9.0" }, + { name = "scipy" }, { name = "sqlmodel", specifier = ">=0.0.16" }, { name = "tables", marker = "extra == 'dev'" }, { name = "tabulate" },