Calibrate DWP and OBR welfare targets on the countries their source covers

[MaxGhenis]

What and why

Several national calibration targets are Great Britain statistics, but their loss-matrix columns summed over every UK household. Northern Ireland runs its own social security, so its households inflated the model side of every DWP target and every DWP line of OBR EFO table 4.9. For the benefits whose executive competence passed to the Scottish Government, DWP's figures cover England and Wales only, so Scottish households inflated those too.

This PR:

• Uses the countries field, GREAT_BRITAIN and restrict_to_countries now on main through Calibrate Housing Benefit to DWP's GB figures by age group #490, Solve Pension Credit take-up over entitled benefit units #510 and Calibrate GB universal credit to one OBR total; reach the UC payment top band #530. Adds ENGLAND_AND_WALES next to GREAT_BRITAIN, while retaining main's carry_forward field and _resolve_value behaviour.
• Sets countries on every target in targets/sources/dwp.py: Great Britain, except England and Wales for the two PIP claimant counts. dwp/scotland_uc_households_child_under_1 gets ("SCOTLAND",); its compute already filters to Scotland, so its column doesn't change.
• Sets countries on the OBR table 4.9 welfare targets in _parse_welfare. Each line in a "DWP social security" block defaults to Great Britain. The map names only the exceptions: England and Wales for pip (the "Disability living allowance and personal independence payments" line), attendance_allowance, carers_allowance and winter_fuel_allowance, and UK-wide for child_benefit (HMRC). The surviving combined obr/universal_credit target from Calibrate GB universal credit to one OBR total; reach the UC payment top band #530 remains Great Britain.
• Uses main's existing hypothesis>=6.168.3 dev dependency for the property tests. The duplicate unpinned dependency was removed during the refresh. Main's uv.lock was retained, then uv lock --check required one metadata update: the editable project version from 1.57.4 to 1.58.0. No dependency resolution changed (uv lock --check passes; only the editable project version changed). The locked engines remain policyengine-uk 2.122.2 and policyengine-core 3.32.13.

This is the follow-up #530's owner filed (GB scope for the other DWP-GB targets).

Evidence

OBR EFO March 2026, detailed table 4.9 (committed at storage/obr_efo/efo_expenditure.xlsx):

• Each section opens a block with a "DWP social security" total and closes it with an "Other DWP" line: rows 7–19 inside the welfare cap and 38–44 outside it.
• Northern Ireland has its own rows: "NI social security in welfare cap" (34) and "NI social security outside welfare cap" (45).
• Child benefit (32), personal tax credits (31) and tax-free childcare (33) are HMRC lines outside the blocks.

DWP benefit expenditure and caseload tables, Spring Forecast 2026:

• The "UK welfare" sheet labels these lines "DWP Social Security (GB)": £308.31bn in 2025-26. Table 4.9's two DWP blocks give £148.87bn + £159.34bn = £308.21bn. "Northern Ireland Social Security" is £10.659bn, and table 4.9's NI rows give £6.244bn + £4.414bn.

• The line-by-line figures for 2025-26 agree:

  Line	DWP tables	Table 4.9
  Attendance Allowance	£8,497m	£8.492bn
  Carer's Allowance	£4,580m	£4.569bn
  Pension Credit	£6,144m	£6.146bn
  DLA + PIP	£36,832m	£36.910bn

• Notes, geographical coverage: "Figures relate to Great Britain, or people resident overseas who are receiving United Kingdom benefits, except for Over 75 TV Licences, War Pensions and Financial Assistance Scheme payments which also cover Northern Ireland. All other benefit expenditure on residents of Northern Ireland is the responsibility of the Northern Ireland Executive. Great Britain figures include expenditure for Scotland, with the exception of benefits which are given as England and Wales spending only, from the point that executive competence was transferred to Scottish Government."

• Note K: Carer's Allowance is England and Wales only from 2019/20, with Scotland transferred on 3 September 2018.

• Note L: Disability Living Allowance, Personal Independence Payment, Attendance Allowance, Severe Disablement Allowance and Industrial Injuries Disablement Benefit are England and Wales only from 2020/21.

• Note N: Winter Fuel Payment is England and Wales only from 2024/25.

• Table 4.9 starts at 2024-25, so all of its years fall after these dates.

DWP statistics (each release's own coverage statement):

Targets	Source	Coverage statement
dwp/esa_*, dwp/jsa_claimants	DWP benefit statistics, February 2026	"This release covers DWP benefits in Great Britain."
dwp/pip_dl_*	PIP statistics to January 2026	"Coverage: England and Wales". It also says "PIP statistics do not include anybody who is claiming ADP and residing in Scotland."
dwp/benefit_cap*	Benefit cap, households capped to November 2025	"covers data for Great Britain"
dwp/uc/households	UC deductions statistics, March 2025 to February 2026	"All figures are for Great Britain only."
dwp/uc/two_child_limit/*	UC two-child limit statistics, April 2025 (data tables)	"Coverage: Great Britain". Also: "UC data is for Great Britain only as DWP does not administer Universal Credit in Northern Ireland."
dwp/uc/claimants_*, dwp/uc_payment_dist/*, dwp/scotland_uc_*	Stat-Xplore Universal Credit	"for Great Britain (England, Scotland, and Wales)". The methodology note says "all claims registered at a Northern Ireland office are removed to create a GB dataset only".

Historical diagnostic model side (policyengine-uk 2.93.0, locked when the pre-batch diagnostic below was run; the refreshed branch now locks 2.122.2):

• winter_fuel_allowance is zero in Scotland in every year.
• carers_allowance is zero in Scotland from 2025, when Carer Support Payment replaces it.
• pip and attendance_allowance have no Scotland condition, so Scottish households' disability benefits counted towards DWP's England and Wales totals.

Historical diagnostic: how much of each column lay outside the target's countries, on pre-batch main's weights before any recalibration:

• Dataset: pre-batch main's seeded build (b45c373, seed 0, the same file Calibrate GB universal credit to one OBR total; reach the UC payment top band #530 used), uprated to 2025. The table retains the target names and definitions used for that diagnostic: Calibrate Housing Benefit to DWP's GB figures by age group #490 and Solve Pension Credit take-up over entitled benefit units #510 subsequently replaced OBR HB and PC, and Calibrate GB universal credit to one OBR total; reach the UC payment top band #530 combined the two UC targets. It is not evidence of impact against the 1.58.0 baseline.
• Method: scored on this PR's target definitions with fit.py, which records each column before and after restrict_to_countries.
• Suppression: only targets with at least 10 records outside their countries are shown. 56 of the 135 scoped targets in the 2025 matrix have 1–9 such records and are suppressed, and 33 have none.

Target	Covers	Target	Model, UK-wide column (main)	Model, scoped column (this PR)	Outside the target's countries
obr/pip	E&W	£36.91bn	£27.85bn (−24.5%)	£25.68bn (−30.4%)	7.8%
obr/attendance_allowance	E&W	£8.49bn	£8.76bn (+3.1%)	£7.89bn (−7.0%)	9.9%
obr/carers_allowance	E&W	£4.57bn	£4.69bn (+2.6%)	£4.40bn (−3.7%)	6.1%
obr/winter_fuel_allowance	E&W	£1.91bn	£1.73bn (−9.5%)	£1.70bn (−11.1%)	1.8%
obr/state_pension	GB	£146.19bn	£124.36bn (−14.9%)	£120.91bn (−17.3%)	2.8%
obr/pension_credit	GB	£6.15bn	£6.36bn (+3.5%)	£6.10bn (−0.7%)	4.1%
obr/esa	GB	£7.21bn	£7.25bn (+0.5%)	£6.51bn (−9.7%)	10.2%
obr/housing_benefit	GB	£12.10bn	£12.27bn (+1.3%)	£10.96bn (−9.4%)	10.6%
obr/statutory_maternity_pay	GB	£3.19bn	£3.36bn (+5.1%)	£3.32bn (+3.9%)	1.1%
obr/universal_credit_in_cap	GB	£66.41bn	£74.45bn (+12.1%)	£72.80bn (+9.6%)	2.2%
obr/universal_credit_outside_cap	GB	£12.88bn	£72.10bn (+460%)	£70.49bn (+447%)	2.2%
dwp/pip_dl_standard_claimants	E&W	1,283k	1,419k (+10.6%)	1,311k (+2.2%)	7.6%
dwp/pip_dl_enhanced_claimants	E&W	1,608k	2,011k (+25.1%)	1,849k (+15.0%)	8.0%
dwp/esa_claimants	GB	999k	916k (−8.3%)	836k (−16.4%)	8.8%
dwp/esa_contrib_claimants	GB	620k	729k (+17.6%)	669k (+7.9%)	8.3%
dwp/esa_income_claimants	GB	180k	190k (+5.8%)	171k (−5.2%)	10.4%
dwp/uc/households	GB	6,700k	6,405k (−4.4%)	6,256k (−6.6%)	2.3%

On main's weights:

• The eight dwp/uc/claimants_* targets shown have 1.9%–3.5% of their column outside Great Britain. Their mean absolute error falls from 7.5% to 5.1%.
• For dwp/uc_payment_dist/*, 21 of 104 targets clear the threshold; their median share outside Great Britain is 1.4%.
• The mean calibration loss term over all 637 national targets barely moves, from 0.05094 to 0.05091. Recalibration is what changes the weights: see the impact section.

The historical diagnostic also exposed target-definition issues. #530 has since replaced the split UC targets; the surviving PIP and overseas State Pension issues are listed below.

Invariants

These hold for every input:

1. Locality: restrict_to_countries leaves each household in the target's countries unchanged and sets every other household to zero.
2. Conservation: for any set of countries C, restrict(col, C) + restrict(col, all countries not in C) == col. A GB column plus its Northern Ireland and unknown-country remainder is the UK column, so nothing is lost or double counted.
3. Composition: restrict(restrict(col, A), B) == restrict(col, A ∩ B). Restriction is idempotent, so the Scotland-only target, whose compute already filters Scotland, is unchanged.
4. Identity: with countries=None, restrict_to_countries returns the column object itself. Every UK-wide target's column is the same object main produced.
5. Source structure:
   • Every table 4.9 target whose values come from a line in a "DWP social security" block sets countries: England and Wales for the four devolved benefits, Great Britain for the rest.
   • Every target from outside those blocks is UK-wide.
   • Each block's lines sum to its total in every year, and both NI rows lie outside the blocks.
6. Completeness: every target from a dwp* source module declares countries, and each is a subset of Great Britain. This includes Calibrate Housing Benefit to DWP's GB figures by age group #490's dwp_housing_benefit.py and Solve Pension Credit take-up over entitled benefit units #510's dwp_pension_credit.py, which already set GREAT_BRITAIN.
7. Constants: GREAT_BRITAIN and ENGLAND_AND_WALES hold values the live country variable returns.
8. Batch coverage: Calibrate Housing Benefit to DWP's GB figures by age group #490 HB, Solve Pension Credit take-up over entitled benefit units #510 PC and Calibrate GB universal credit to one OBR total; reach the UC payment top band #530 combined UC remain GB; Draw SPI incomes within earnings groups set by FRS employment status #529's LFS employee and self-employed targets remain UK-wide; Restore HMRC salary sacrifice relief calibration targets #533's HMRC salary-sacrifice targets retain main's scope. Removed OBR targets are absent. The extended country test exercises each surviving batch target through the loss-matrix country mask.

Tests

Before the refresh, policyengine_uk_data/tests/test_target_countries.py had 10 tests, all passing locally:

• Hypothesis property tests for invariants 1–4. Each runs 100 to 300 generated cases, with household columns of up to 30 households and arbitrary country sets.
• create_target_matrix end to end on a real policyengine-uk simulation with one household in each country (invariant 7). Each household holds a distinct power of two, so each column's total identifies exactly which countries it counted: UK 15, GB 7, E&W 5, NI 8.
• Table 4.9 against the committed workbook (invariant 5). Each target's source rows are found by matching its values (one row, or two rows summed, which covers Calibrate GB universal credit to one OBR total; reach the UC payment top band #530's obr/universal_credit), not by the parser's label lookup.
• Every DWP target declares its countries (invariant 6), and the PIP claimant targets cover England and Wales.
• Batch invariants assert exact GB scope for all HB/PC targets (including HB diagnostics), one combined GB UC target and all UC payment bands; LFS stays UK-wide. Every HMRC salary-sacrifice target retains main's unrestricted scope: the committed table does not explicitly state coverage, so no new geographical assumption is made. Obsolete OBR HB/PC/split-UC/NI-relief targets stay absent.

Pre-refresh mutation check: nine single-line mutations, each caught by at least one test. The outside-the-cap UC mutation below refers to the old split target, since replaced by #530.

1. restrict_to_countries as the identity.
2. Inverted scope.
3. The loss matrix ignoring countries.
4. pip defaulting to GB.
5. Child benefit defaulting to GB.
6. Outside-the-cap UC left UK-wide.
7. One DWP target with no countries.
8. The PIP claimant counts set to GB.
9. ENGLAND_AND_WALES spelt as display names.

At the old head f0a5dbc (pre-batch main), test_target_countries.py, test_obr_efo_fallback.py, test_target_registry.py, test_esa_jsa_targets.py, test_target_db.py, test_obr_receipts_sheet.py, test_obr_nics.py and test_obr_council_tax.py give 60 passed.

A pre-refresh trial batch was reported as 106 passed, 2 skipped and 11 fallback-test failures. Those obsolete fallback assertions have since been fixed on main: the offline OBR target-set test now compares the offline set with the served set. The trial counts are historical, and the refresh results below are the current evidence.

Impact

Pending a real rebuild of the refreshed head against post-batch main 4cbedbe (1.58.0). No dataset was rebuilt for this refresh. The table above is a historical diagnostic on pre-batch weights, not an impact estimate. The hub runs the real seeded comparison after the push, under its build schedule.

The hub should build both sides with the same seed, authorised input donor and HTTP cache, then score both using this PR's refreshed target definitions. The commands below pin the baseline and proposed commits and use fresh labels so run_build.sh cannot reuse an older build. Run outside this restricted refresh workspace, with more than 60 GB free and no other build running.

The old setup script's uk-data-cvpay donor no longer exists. uk-data-combined-1008 currently has the required raw-input symlinks and seven trained-model files. Use it for both sides; this does not claim parity with the old diagnostic's donor. The command verifies the prerequisites, records the donor and model hashes, and makes a setup copy with that donor before building.

set -e
R=~/reviews/gb-scope-2026-10-05
SETUP_TEMPLATE=~/reviews/brma-imputation-weights-2026-10-02/data-build/setup_worktree.sh
CACHE=~/reviews/brma-imputation-weights-2026-10-02/data-build/http_cache
SOURCE_STORAGE=~/PolicyEngine/_worktrees/uk-data-combined-1008/policyengine_uk_data/storage
BASE_SHA=4cbedbecca352f52dffe47e849553c2759a00711
PR_SHA=ec312d6fcc7d9b8222445c44bd051397dbaf8e75
BASE_LABEL=batchmain_${BASE_SHA:0:10}_s0
PR_LABEL=gbscope_${PR_SHA:0:10}_s0
[[ ! -e "$R/data-build/efrs_$BASE_LABEL.h5" && ! -e "$R/data-build/efrs_$PR_LABEL.h5" ]] || { print 'Build labels already exist; choose fresh labels'; exit 1; }
BASE_WT=~/PolicyEngine/_worktrees/uk-data-gbscope-main-${BASE_SHA:0:10}-build
PR_WT=~/PolicyEngine/_worktrees/uk-data-gbscope-${PR_SHA:0:10}-build

for input in etb_1977_24 frs_2024_25 lcfs_2023_24 was_2006_22 spi_2022_23/put2223uk.tab; do
  [[ -L "$SOURCE_STORAGE/$input" && -e "$SOURCE_STORAGE/$input" ]] || { print "Missing input: $input"; exit 1; }
done
models=("$SOURCE_STORAGE"/*.pkl(N))
(( ${#models} > 0 )) || { print 'No trained models in input donor'; exit 1; }
mkdir -p "$R/data-build" "$R/impact"
printf 'input_source=%s\n' "$SOURCE_STORAGE" > "$R/data-build/postbatch_inputs.meta"
shasum -a 256 "${models[@]}" >> "$R/data-build/postbatch_inputs.meta"
SETUP="$R/setup_worktree_postbatch.sh"
sed "s|^SRC=.*|SRC=\"$SOURCE_STORAGE\"|" "$SETUP_TEMPLATE" > "$SETUP"
chmod +x "$SETUP"
"$SETUP" "$BASE_WT" "$BASE_SHA"
"$SETUP" "$PR_WT" "$PR_SHA"
TESTING=0 "$R/locked.sh" "build_$BASE_LABEL" "$R/run_build.sh" "$BASE_LABEL" "$BASE_WT" 0
TESTING=0 "$R/locked.sh" "build_$PR_LABEL" "$R/run_build.sh" "$PR_LABEL" "$PR_WT" 0
cd "$PR_WT"
.venv/bin/python "$R/fit.py" "$R/data-build/efrs_$BASE_LABEL.h5" "$CACHE" "$R/impact/$BASE_LABEL"
.venv/bin/python "$R/fit.py" "$R/data-build/efrs_$PR_LABEL.h5" "$CACHE" "$R/impact/$PR_LABEL"

setup_worktree.sh uses the pinned commit's frozen Python 3.13 environment and copies the same trained models into each worktree. The runner still matches main's frs_2024_25 filenames and builds with 512 epochs and PE_UK_DATA_OA_CLONES=1. fit.py still matches the loss-matrix API, uses the refreshed branch's _resolve_value/carry_forward rules and scores calibration year 2025. The HTTP cache pins target downloads; a new target URL is fetched live during a build and logged in build_<label>.http. No command in this block was executed during the refresh.

Merge interactions

The former 10/8 batch landed on 10/7 as uk-data 1.58.0. This PR was held out and remains for the next batch, on Max's go (d833). The refreshed branch merges post-batch main 4cbedbe and retains its newer target set and download fallback.

• Calibrate Housing Benefit to DWP's GB figures by age group #490, Solve Pension Credit take-up over entitled benefit units #510 and Calibrate GB universal credit to one OBR total; reach the UC payment top band #530: retain main's countries, GREAT_BRITAIN and restrict_to_countries, and Calibrate Housing Benefit to DWP's GB figures by age group #490's carry_forward field and _resolve_value logic. schema.py now includes both carry_forward and this PR's ENGLAND_AND_WALES. build_loss_matrix.py and dwp.py auto-merged; their resulting semantics were checked against main and the original PR diff.
• Calibrate GB universal credit to one OBR total; reach the UC payment top band #530: retain one GB obr/universal_credit for the summed inside/outside welfare-cap expenditure. No split UC target was reintroduced. The coverage test matches its two workbook rows independently.
• Calibrate Housing Benefit to DWP's GB figures by age group #490 and Solve Pension Credit take-up over entitled benefit units #510: retain the DWP HB and PC spend/caseload targets, each GB. Removed obr/housing_benefit and obr/pension_credit targets stay removed; the country map has no stale entries for them.
• Fall back to the committed OBR workbooks when obr.uk answers 200 with a non-workbook page #536 and the batch F1 fix: preserve the committed-workbook fallback and the offline-versus-served target-set test. The landed main retains obr/ni; previous trial-batch counts and descriptions of its removal do not describe this baseline.
• Restore HMRC salary sacrifice relief calibration targets #533: keep HMRC's salary-sacrifice employee and employer NI relief targets and their existing source coverage; the removed OBR equivalents stay removed.
• Draw SPI incomes within earnings groups set by FRS employment status #529: keep UK-wide LFS employee and self-employed counts unchanged. The extended country invariant covers them alongside the DWP and OBR replacements.
• Dependencies: remove the auto-merged duplicate hypothesis; pyproject.toml matches main's dependency set. Main's lock check failed because its editable project still said 1.57.4. uv lock changed only that project version to 1.58.0; every dependency entry, including policyengine-uk 2.122.2 and core 3.32.13, is unchanged.

Post-batch refresh (10/7)

• New head: ec312d6fcc7d9b8222445c44bd051397dbaf8e75, a fast-forward descendant of f0a5dbc66f2562f6f8bd6076142c1ac4efcbde3b. The merge commit 5f6c55ba5c7dc0f8499ad07d422c828883e591bb has the old head and main 4cbedbecca352f52dffe47e849553c2759a00711 as its parents.
• Resolved schema.py by keeping main's carry_forward and the PR's ENGLAND_AND_WALES; resolved both obr.py hunks by combining the country-constant imports and preserving main's surviving targets and workbook fallback while applying this PR's source country scopes. Used main's uv.lock, removed the duplicate hypothesis from pyproject.toml, and after the lock check failed refreshed only the editable project version to 1.58.0.
• Checked all merge-touched files against main and the original PR diff, including Calibrate Housing Benefit to DWP's GB figures by age group #490/Solve Pension Credit take-up over entitled benefit units #510 HB/PC, Calibrate GB universal credit to one OBR total; reach the UC payment top band #530 UC, Restore HMRC salary sacrifice relief calibration targets #533 HMRC salary sacrifice, Draw SPI incomes within earnings groups set by FRS employment status #529 LFS employment, Fall back to the committed OBR workbooks when obr.uk answers 200 with a non-workbook page #536 fallback and the batch F1 offline target-set fix. Extended test_target_countries.py for the batch targets and unchanged source coverage.
• Pre-refresh independent review: 60 targeted tests passed at the old head; its only requested change was real-build impact evidence. Refresh validation: 170 passed, 4 skipped, 0 failed across all 15 requested files. The original review's eight files now give 99 passed, versus its reported 60 at the old head; the original old-head tests were not rerun during this refresh. 4 enhanced-FRS integration cases skipped because that built dataset is absent. Files ran sequentially, one test file per pytest process, with -p no:cacheprovider and no xdist. The first fourteen files ran on Python 3.13.9. For test_calibrate_validation_targets.py, two startup attempts were interrupted and a diagnostic retry exited with a native SIGSEGV during Torch lazy Dynamo/SymPy imports. Both tests then passed on Python 3.13.15 with the identical locked dependencies, a real CPU Adam preloaded before pytest, and OMP/MKL/OpenBLAS threads limited to 2. Test sources and assertions were unchanged; failed startup diagnostics were retained.
• uv lock --check passes; only the editable project version changed. Installed policyengine-uk 2.122.2 and policyengine-core 3.32.13.
• Impact remains pending the hub's real seeded comparison against the post-batch baseline. No dataset build was run.

Test file	Passed	Skipped	Failed
test_target_countries.py	17	0	0
test_obr_efo_fallback.py	36	0	0
test_target_registry.py	17	0	0
test_esa_jsa_targets.py	2	0	0
test_target_db.py	12	0	0
test_obr_receipts_sheet.py	7	0	0
test_obr_nics.py	5	0	0
test_obr_council_tax.py	3	0	0
test_obr_universal_credit_target.py	10	0	0
test_dwp_housing_benefit_targets.py	12	1	0
test_pension_credit_takeup.py	10	1	0
test_lfs_employment_targets.py	5	1	0
test_hmrc_salary_sacrifice_targets.py	22	1	0
test_uc_payment_distribution_targets.py	10	0	0
test_calibrate_validation_targets.py	2	0	0

Follow-ups (not in this PR)

• obr/pip compares the "Disability living allowance and personal independence payments" line (£36.91bn in 2025-26) with the model's pip alone. There is no custom compute, so it falls through to _compute_simple_gbp on pip. DWP's tables put DLA at £8.29bn of that. Without dla the model side can't reach the line, and calibration pushes PIP up to make up the gap. Scoping to England and Wales doesn't change this, but it widens the pre-recalibration gap from −24.5% to −30.4%.
• obr/state_pension includes State Pension paid to people living abroad, which no UK household in the model receives. DWP's tables put it at £5.63bn in 2025-26.
• Other national targets whose source covers less than the UK weren't audited here. For example, council tax exists only in Great Britain. I didn't check whether policyengine-uk gives Northern Ireland households a council tax liability.

🤖 Generated with Claude Code