Release the 10/8 uk-data batch (15 PRs, relock to policyengine-uk 2.122.2)

.gitignore

@@ -19,3 +19,10 @@
 **/_build
 !policyengine_uk_data/storage/*.csv
 **/version.json
+# Build output: household weights by local area, derived from FRS microdata
+policyengine_uk_data/storage/local_geography_weights.csv.gz
+
+.brma-cache/
+
+# Hypothesis example database (property-based tests)
+.hypothesis/

changelog.d/brma-census-weights.fixed.md

@@ -0,0 +1 @@
+- Draw FRS households' Broad Rental Market Areas in proportion to census private-rented households by BRMA and bedrooms, instead of the row counts of the 2019-20 LHA list of rents, whose Scottish, Welsh and Northern Ireland lists were copies of English BRMAs' lists (uk-data#515). Remove `lha_list_of_rents.csv.gz`.

changelog.d/frs-claimant-or-partner.added.md

@@ -0,0 +1 @@
+Supply `is_claimant_or_partner`, each benefit unit's single adult or couple, from FRS adult-table membership, so policyengine-uk reads the survey's roles instead of inferring them from ages. SPI records and the public transfer dataset (`enhanced_cps_2025.h5`, column added in place) carry it too, and stacking refuses to combine a person table that has it with one that does not.

changelog.d/frs-cvpay-not-property-income.fixed.md

@@ -0,0 +1 @@
+Stop counting the rent a boarder or lodger pays (FRS CVPAY) as that boarder's or lodger's own property income.

changelog.d/frs-empstati-other-inactive.fixed.md

@@ -0,0 +1 @@
+Map FRS EMPSTATI code 11 ("Other inactive") to `employment_status` OTHER_INACTIVE; it had fallen through to LONG_TERM_DISABLED, which also put other-inactive adults into the ESA health-condition and support-group proxies. An adult EMPSTATI code the mapping does not know now fails the build instead of defaulting.

changelog.d/frs-pc-reported-capital.changed.md

@@ -0,0 +1 @@
+Carry each benefit unit's FRS total capital (TOTCAPB4, DWP's current benefit-unit savings and investments measure; TOTCAPB3 for earlier survey years) into policyengine-uk's `pension_credit_reported_capital`, so Pension Credit's capital test uses the survey's own benefit-unit capital instead of imputed household wealth.

changelog.d/frs-property-losses-subrent.fixed.md

@@ -0,0 +1 @@
+Stop counting FRS property losses (ROYYR1 with RENTPROF = 2) as property income, and count rent from sub-letting part of the home (SUBRENT) for every tenure, not only owner-occupiers.

changelog.d/frs-uc-gainful-self-employment.fixed.md

@@ -0,0 +1 @@
+Set policyengine-uk's Universal Credit input uc_is_in_gainful_self_employment from the FRS, as a survey proxy for UC Regs 2013 reg 64 that overrides the model's income-based default: true for every adult whose main job (EMPSTATI) is self-employment, including traders who break even or make a loss, and for anyone whose self-employment profit is above their employment income (ADM H4034); false otherwise. The SPI copy re-derives it from its own imputed incomes.

changelog.d/hb-dwp-age-targets.changed.md

@@ -0,0 +1 @@
+Calibrate Housing Benefit to DWP's Great Britain spending and claims by age group, replacing the OBR Housing Benefit target that the model compared with UK-wide Housing Benefit. Pension-age Housing Benefit is calibrated to DWP's figures over Pension Credit qualifying age. Working-age Housing Benefit is calibrated to DWP's figures under that age less all supported and temporary accommodation Housing Benefit, for which policyengine-uk has no rules (£585m and 107k claims in 2025-26, against £5.78bn and 460k in total); treating the model's working-age Housing Benefit as general needs is an approximation. Modelled GB Housing Benefit for 2025-26 falls from about £11.7bn to £8.5bn (DWP: £12.9bn), because pension-age Housing Benefit was about 1.6 times DWP's figure while about £5.2bn of supported and temporary accommodation Housing Benefit remains unmodelled as such. Stop setting `would_claim_uc` for benefit units whose adults have all reached State Pension age, and build with policyengine-uk 2.102.5 so pension-age families can make new Housing Benefit claims.

changelog.d/hmrc-salary-sacrifice-relief-targets.fixed.md

@@ -0,0 +1 @@
+Restore the HMRC salary sacrifice income tax and NICs relief calibration targets. They had been lost since the July 2026 private pension statistics release, when the old HMRC CSV was withdrawn (410 Gone). The targets now come from the 2024-25 Tables 6.1 and 6.2, mapped to year 2024. A committed copy is used when the download fails, and a table for another tax year or without the expected rows fails the build. Income tax relief is the rise in tax on pay under the person's rUK or Scottish rates, relieved at each rate the sacrifice straddles; it had been assigned whole to one band by comparing adjusted net income with taxable-income thresholds. The NICs targets follow the Class 1 rates, including the employer rate rise to 15% from April 2025. The OBR-labelled NICs relief targets, which repeated HMRC's 2023-24 figures, and the separately rounded income tax relief total are removed.

changelog.d/oa-region-constrained.fixed.md

@@ -0,0 +1 @@
+Draw each cloned household's Output Area from its own FRS region rather than from anywhere in its country, so `region_code_oa`, `la_code_oa` and `constituency_code_oa` no longer contradict `region` (88.5% of English households in release 1.57.4 carried an OA from another region).

changelog.d/obr-fallback-non-workbook.fixed.md

@@ -0,0 +1 @@
+Fall back to the committed OBR EFO workbooks when obr.uk answers 200 with a body that is not a workbook (such as its HTML "No Access" page), not only when the request fails. The parse error used to escape the fallback, and `get_targets()` dropped every target parsed from that workbook (the nine OBR receipts and NICs targets in the case observed), logging the error while the build carried on. A body openpyxl cannot read is now treated as a permanent failure: it is not retried, and a warning names the URL, the status and the parse error.

changelog.d/obr-uc-welfare-cap-targets.fixed.md

@@ -0,0 +1 @@
+Calibrate universal credit to one OBR total for Great Britain, the sum of EFO table 4.9's rows inside and outside the welfare cap. The outside-the-cap row is DWP's UC equivalent of JSA (the Intensive Work Search group), not UC for households the benefit cap leaves alone: computed that way, it asked for £12.9bn of nearly all UC (estimate £72bn, +460%), and the inside-the-cap row was compared with all UK UC. Also count the open top band of the DWP UC payment distribution ("£2500.01 or over" a month), which parsed to missing bounds so its four targets (1.4k to 83k households) could never be met, and make the bands meet without gaps, so awards of exactly a band's top, or a penny a month after deductions, land in their Stat-Xplore band.

changelog.d/pc-takeup-over-eligible.fixed.md

@@ -0,0 +1 @@
+Solve Pension Credit take-up over entitled benefit units in Great Britain, at DWP's FYE 2024 caseload take-up of 62%, after the imputations so entitlement reflects imputed capital. Entitled non-reporters, including in Northern Ireland, are drawn at the solved probability. Units with no entitlement in the calibration year are drawn at DWP's Savings Credit-only caseload take-up (37% in FYE 2024), the rate at which units a reform or a later year newly entitles claim. Reports on SPI-synthetic households, which are imputed from SPI incomes rather than observed, no longer anchor take-up. Calibrate Pension Credit to DWP's Great Britain spending and caseload, replacing the OBR target that the model compared with UK-wide Pension Credit.

changelog.d/spi-income-earnings-groups.fixed.md

@@ -0,0 +1 @@
+Draw SPI incomes for the enhanced FRS's SPI-synthetic rows within earnings groups (employee, self-employed, both, neither) set by each row's FRS employment status, so employees draw pay, the self-employed draw a trade, people out of work draw no earnings, and children keep their own (zero) incomes (#504). Calibrate to the ONS LFS counts of employees and the self-employed, and report rather than train on HMRC's local counts of taxpayers with employment income, which are annual and include part-year earners the FRS records as out of work.

changelog.d/spi-synthetic-reported-benefits.fixed.md

@@ -0,0 +1 @@
+SPI-synthetic rows no longer report income-related (except council tax reduction, which keeps its imputed value for now), out-of-work or Child Benefit receipt, take their industrial injuries, armed forces compensation and bereavement support from the FRS donor, and get UC, Pension Credit and `receives_benefits_in_own_right` flags from their own reports instead of the donor's.

changelog.d/uk-data-batch-relock.changed.md

@@ -0,0 +1 @@
+Lock policyengine-uk 2.122.2 (from 2.93.0) and policyengine-core 3.32.13 (from 3.31.1); #525 needs policyengine-uk 2.122.0 or later.

docs/imputations.md

@@ -211,6 +211,19 @@ Assigns student loan plan type based on age and reported repayments.
 
 ---
 
+## Broad Rental Market Area Assignment
+
+**Source:** Census private-rented households by BRMA and bedrooms (not QRF; applied when the base FRS dataset is built)
+
+The FRS identifies only the region, but Local Housing Allowance rates vary by Broad Rental Market Area (BRMA). `datasets/brma.py` draws each benefit unit's BRMA within its region, in proportion to the private-rented households in each BRMA:
+- shared-accommodation and one-bedroom LHA categories use one-bedroom homes;
+- the two-, three- and four-or-more-bedroom categories use homes with that many bedrooms;
+- Northern Ireland's census has no bedrooms, so its weights are the same for every category.
+
+A household takes one of its benefit units' BRMAs, chosen at random. Sources, method and validation are in `storage/BRMA_DATA_SOURCES.md`.
+
+---
+
 ## Calibration Targets
 
 After imputation, household weights are calibrated to match aggregate statistics from:

docs/oa_calibration_pipeline.md

@@ -23,7 +23,7 @@ Build the OA crosswalk and population-weighted assignment function.
 **Deliverables:**
 - `policyengine_uk_data/calibration/oa_crosswalk.py` — downloads/builds the OA → LSOA → MSOA → LA → constituency → region → country crosswalk
 - `policyengine_uk_data/storage/oa_crosswalk.csv.gz` — compressed crosswalk file
-- `policyengine_uk_data/calibration/oa_assignment.py` — assigns cloned records to OAs (population-weighted, country-constrained)
+- `policyengine_uk_data/calibration/oa_assignment.py` — assigns cloned records to OAs (population-weighted within the household's FRS region)
 - Tests validating crosswalk completeness and assignment correctness
 
 **Data sources:**
@@ -42,13 +42,14 @@ Clone each FRS household N times and assign each clone a different OA.
 
 **Deliverables:**
 - `policyengine_uk_data/calibration/clone_and_assign.py` — clones all three entity tables (household, person, benunit), remaps IDs, divides weights by N, attaches OA geography columns
-- `datasets/create_datasets.py` — clone step inserted after imputations, before uprating/calibration (N=10 production, N=2 testing)
+- `datasets/create_datasets.py` — clone step inserted after imputations, before uprating/calibration (N defaults to 10, or 2 with `TESTING=1`; `PE_UK_DATA_OA_CLONES` overrides it)
 - `tests/test_clone_and_assign.py` — 14 tests covering dimensions, weight preservation, ID uniqueness, FK integrity, country constraints, data preservation
 
 **Key design:**
-- N=10 clones in production, N=2 in testing mode
+- N defaults to 10 clones (2 with `TESTING=1`); the `PE_UK_DATA_OA_CLONES` environment variable overrides it, and the release (`push.yaml`) and pull request workflows set it to 1
 - Constituency collision avoidance: each clone gets a different constituency where possible
-- Country constraint preserved: English households → English OAs only
+- Region constraint: each clone's OA is drawn from the household's own FRS region (Wales and Scotland are one region each), so `region_code_oa`, `la_code_oa` and `constituency_code_oa` never contradict `region`. A household with no region below the country falls back to its country. No LA or constituency straddles a region, so every OA stays reachable
+- Collision avoidance draws from the same region; the smallest (North East) has 27 constituencies, more than the default 10 clones
 - Weights divided by N so population totals are preserved
 - Pure pandas/numpy operations — no simulation required, fast execution
 
@@ -133,7 +134,7 @@ Generate per-area H5 files from sparse L0-calibrated weights.
 **Deliverables:**
 - `policyengine_uk_data/calibration/publish_local_h5s.py` — extracts per-area H5 subsets from the sparse weight vector; each H5 contains only active households (non-zero weight) with their calibrated weights, plus the linked person and benunit rows
 - `policyengine_uk_data/calibration/long_geography.py` — exports matrix-free local geography weights as an OA-first long table, with constituency and LA rows derived from assigned OA geography
-- `datasets/create_datasets.py` — publish step wired in after calibration, before downrating
+- `datasets/create_datasets.py` — exports `local_geography_weights.csv.gz` after calibration; `publish_local_h5s()` is not called by the build
 - `tests/test_publish_local_h5s.py` — 13 tests covering area-household mapping, H5 structure, pruned-household exclusion, weight correctness, person/benunit FK integrity, full publish cycle, summary statistics, and validation
 
 **Key design:**

policyengine_uk_data/calibration/clone_and_assign.py

@@ -1,9 +1,9 @@
 """Clone-and-assign: duplicate FRS households and assign OA geography.
 
 Each FRS household is cloned N times. Each clone gets a different
-Output Area (population-weighted, country-constrained, with
-constituency collision avoidance). Weights are divided by N so
-population totals are preserved.
+Output Area (population-weighted, drawn from the household's FRS
+region, with constituency collision avoidance). Weights are divided
+by N so population totals are preserved.
 
 This is the UK equivalent of policyengine-us-data's clone-and-assign
 approach (PRs #457, #531).
@@ -19,37 +19,35 @@
 from policyengine_uk.data import UKSingleYearDataset
 
 from policyengine_uk_data.calibration.oa_assignment import (
+    FRS_COUNTRY_MAP,
+    _REGION_CODE_PREFIX_TO_COUNTRY,
+    _normalise_region,
     assign_random_geography,
 )
 
 logger = logging.getLogger(__name__)
 
-# FRS region values that map to each country
-_REGION_TO_COUNTRY_CODE = {
-    "NORTH_EAST": 1,
-    "NORTH_WEST": 1,
-    "YORKSHIRE": 1,
-    "EAST_MIDLANDS": 1,
-    "WEST_MIDLANDS": 1,
-    "EAST_OF_ENGLAND": 1,
-    "LONDON": 1,
-    "SOUTH_EAST": 1,
-    "SOUTH_WEST": 1,
-    "WALES": 2,
-    "SCOTLAND": 3,
-    "NORTHERN_IRELAND": 4,
-    "UNKNOWN": 1,  # Default to England
-}
+_COUNTRY_TO_FRS_CODE = {name: code for code, name in FRS_COUNTRY_MAP.items()}
 
 
 def _household_country_codes(dataset: UKSingleYearDataset) -> np.ndarray:
-    """Extract FRS country codes (1-4) from household region."""
-    regions = dataset.household["region"].values
-    codes = np.array(
-        [_REGION_TO_COUNTRY_CODE.get(str(r), 1) for r in regions],
-        dtype=np.int32,
-    )
-    return codes
+    """Extract FRS country codes (1-4) from household region.
+
+    Uses the same region normalisation as the OA sampler, so both read
+    a region the same way. A region with nothing below the country
+    (missing or ``UNKNOWN``) counts as England; an unrecognised value
+    raises.
+    """
+    codes = []
+    for region in dataset.household["region"].values:
+        region_code = _normalise_region(region)
+        country = (
+            "England"
+            if region_code is None
+            else _REGION_CODE_PREFIX_TO_COUNTRY[region_code[0]]
+        )
+        codes.append(_COUNTRY_TO_FRS_CODE[country])
+    return np.array(codes, dtype=np.int32)
 
 
 def _remap_ids(
@@ -76,9 +74,11 @@ def clone_and_assign(
 ) -> UKSingleYearDataset:
     """Clone each FRS household N times and assign OA geography.
 
-    Each clone gets a population-weighted random Output Area,
-    constrained to its country, with constituency collision
-    avoidance across clones.
+    Each clone gets a population-weighted random Output Area in
+    its household's FRS region (Wales and Scotland are one region
+    each), with constituency collision avoidance across clones.
+    The OA's region, LA and constituency therefore never
+    contradict the household's ``region``.
 
     Household weights are divided by n_clones so aggregate
     population totals are preserved.
@@ -112,7 +112,6 @@ def clone_and_assign(
     # NI is excluded until NISRA updates their download URLs.
     from policyengine_uk_data.calibration.oa_assignment import (
         _load_country_distributions,
-        FRS_COUNTRY_MAP,
     )
 
     available_distributions = _load_country_distributions(
@@ -145,6 +144,7 @@ def clone_and_assign(
             n_clones=n_clones,
             seed=seed,
             crosswalk_path=crosswalk_path,
+            household_regions=hh["region"].values[has_oa],
         )
     else:
         geography = None