Impute UC deductions and private school attendance in constituency and local-authority runs

[MaxGhenis]

Fixes #1902

Builds on #1899 (merged), which adds Simulation.built_from_dataset. The branch has main merged in at 2fad4c0.

Summary

Three variables chose between microdata imputation and household defaults by testing whether total weight was below 1e6:

Variable	Old test	Default below 1e6
uc_deduction_random_draw	benefit-unit weight	draw of 1.0, so no UC deductions
uc_deduction_type_random_draw	benefit-unit weight	1.0
attends_private_school	household weight projected to each person (person.household(...)), so in effect people	no private school attendance

policyengine.py builds constituency and local-authority simulations by filtering rows from the national data. src/policyengine/countries/uk/regions.py uses RowFilterStrategy on constituency_code_oa / la_code_oa. filter_dataset_by_household_ids in src/policyengine/utils/entity_utils.py keeps rows without rescaling weights. run() in src/policyengine/tax_benefit_models/uk/model.py wraps them in UKSingleYearDataset and calls Microsimulation. So every constituency and local authority fell below the threshold and got household defaults: no UC deductions, and no private school attendance even under a private school VAT reform.

Each variable now gates on built_from_data(simulation) (new, policyengine_uk/utils/data_source.py), which is true for any simulation built from data, however little weight it carries. It reads Simulation.built_from_dataset and falls back to policyengine-core's is_over_dataset, so a core Simulation built over the UK system from data counts too. #1899's months_since_last_birthday now uses the same helper.

The builders now set built_from_dataset: build_from_situation clears it and build_from_ids, which every data path goes through, sets it. Previously only __init__ set it, so a simulation rebuilt in place kept a stale value. The builders also drop core's simulation-level caches (_fast_cache, _user_input_keys), so a calculated clone rebuilt in place never reads arrays sized for its old population. attends_private_school also loses a dead hasattr(person.simulation, "dataset") check (Simulation.dataset is a class attribute, so it always held).

filter_dataset now carries each person's attends_private_school into the household it extracts, as #1899 does for months_since_last_birthday. A household alone ranks at the 100th income percentile (rate 0.47 × 0.85), so without this about 40% of children in an extract would be assigned to private school. UC draws need no carrying: they hash benunit_id, which the extract keeps.

attends_private_school also no longer raises when no household has weight. The old gate returned before ranking, and MicroSeries cannot rank zero total weight. Households without weight stay at percentile 0, as before.

The attends_private_school YAML cases were vacuous under the new gate (situations always return False), so they are replaced with situation cases: a household with 1e9 of weight and the top income attends no private school unless set, and a set value is kept.

clone() and get_branch() copy the instance __dict__, so the flag survives into baseline and branch simulations.

Invariants (stated and tested)

policyengine_uk/tests/test_data_built_imputations.py:

1. Construction decides, not weight. In a data-built simulation, scaling every weight by 2^k for any k in [-14, 30] (totals from about a hundred to about 10^15) leaves the draws, uc_has_deduction, uc_deduction_combination, uc_deductions and attends_private_school unchanged (Hypothesis). Powers of two scale exactly in floating point, so the property holds exactly and can't flake on percentile boundaries.
2. Row-filter invariance for UC deductions (differential test). A region filtered from a national-scale data-built simulation (about 1.6m of household weight; the region about 0.4m, or 0.8m of people, under the old threshold for both), with rows kept as RowFilterStrategy keeps them, gives every benefit unit the draws, deduction flag, combination and amount it has in the full simulation.
3. Hashed draws in small data. A data-built simulation with under a million units of weight gets splitmix64_uniform(benunit_id) draws, some deductions and some private school attendance.
4. Extracts keep their imputations. A filter_dataset extract reproduces the full simulation's UC deductions and private school attendance, and the test asserts both sets are non-empty.
5. Data without weight. A data-built simulation where every household has zero weight attends no private school and still gets hashed draws.
6. Situations get defaults whatever their weight. A household situation with 1e9 of weight gets draws of 1.0, no deductions and no private school attendance.
7. The flag follows the builder. A calculated clone of a data simulation, rebuilt in place from a situation, gets the household defaults. A calculated clone of a situation, rebuilt from data, gets the imputations, sized for the new population.
8. Core simulations over data. A policyengine-core Simulation(tax_benefit_system=system, dataset=<DataFrame>) gets hashed draws and private school attendance.

All eight fail on the pre-PR code (325d585), and the first new YAML case fails there too. Targeted checks, each run with one fix removed:

• The extract test fails without the filter_dataset carry.
• The rebuild test fails with the flag set only in __init__, and with the cache reset removed (it reproduces the broadcast error).
• The core test fails without the is_over_dataset fallback.
• The region test fails on its private school assertion under the old gate alone.

Intended exception: private school attendance is not row-filter invariant. It ranks incomes within the simulated population, so a constituency ranks against itself (see caveats).

Constituency and local-authority runs, before and after

Real runs, following policyengine.py's path: filter the national tables by constituency_code_oa / la_code_oa / region, build UKSingleYearDataset + Microsimulation, and calculate 2026.

• Data: enhanced FRS 2024-25, policyengine-uk-data-private 1.56.16, the policyengine.py uk-6.2.0 bundle.
• Before: Set State Pension age from date of birth, including the rise to 67 #1899's head, c46893e. After: 40aaab0. The later review commit changes only the zero-weight path, tests and docs.
• National run: the unfiltered simulation (identical before and after on every metric), grouped by the same geography.
• Runs: 632 GB constituencies (Northern Ireland records carry no constituency or LA code), 363 GB local authorities and Northern Ireland, each run under both versions: 1,992 filtered simulations plus 2 national runs.

632 constituencies (sums over constituency runs)

	Before	After	National run
Constituencies with any UC deductions	0	596	597
UC benefit units with deductions (k)	0	2,938	2,940
UC deductions (£m)	0	1,901	1,902
UC paid (£m)	79,609	77,708	77,618
Private school pupils (k)	0	811	865
Constituencies with private school pupils	0	578	532

363 local authorities

	Before	After	National run
LAs with any UC deductions	0	346	346
UC benefit units with deductions (k)	0	2,939	2,940
UC deductions (£m)	0	1,901	1,902
Private school pupils (k)	0	809	865

• Household net income falls by £1,901m summed over constituencies, equal to the deductions.
• UC deductions match the national run to within float32 rounding (relative gap ≤ 1e-6) in 619 of 632 constituencies and 350 of 363 LAs.
• The rest differ because UC entitlement differs between the filtered and national runs, not the draws. In the one case examined (E14001101, one differing benefit unit), a 66-year-old is under State Pension age in the constituency run and over it nationally. Set State Pension age from date of birth, including the rise to 67 #1899's months_since_last_birthday spreads birthdays within the simulated population. UC paid differs by more than 0.1% in 168 constituencies. I did not trace every case.
• Northern Ireland as a filtered region is unchanged: in 2026 it carries 1,006,938 of benefit-unit weight and about 2.0m people, both above the old threshold.

Caveats and follow-ups

• Private school attendance ranks locally. Constituency totals come to 811k against 865k from the national run. Per constituency, the correlation with the national run is 0.61. The ratio has a median of 1.09 and a 90th percentile of 24, as poorer areas give their top local earners top-percentile rates. The previous behaviour was zero everywhere. Ranking by national income needs the percentile, or attendance itself, carried in the data (follow-up task).
• Separate issue found while measuring, not changed here. Several incidence variables spread a national total over the simulated population by each household's share of a weighted sum, e.g. shareholding and corporate_land_value. In a filtered constituency this puts the whole national total on the constituency. For E14001063 (87 records), corporate_tax_incidence is £34,863m in the constituency run vs £28m for the same households nationally, and business_rates is £31,733m vs £25.6m. Household net income is −£35,634m vs £2,328m. This affects every constituency run in policyengine.py (follow-up task). The net-income figures above are differences, in which it cancels.
• Ordinary household situations (well under a million of weight) are unchanged: they got the defaults before and still do. A situation given over a million of weight used to get hashed UC draws and imputed private school attendance; it now gets the defaults (intended; tested).

Tests run

• test_data_built_imputations.py: 8 passed.
• test_state_pension_age.py + test_uc_deductions.py at the latest head: 41 passed. Labour, is_SP_age and state_pension_age YAML: 45 passed.
• test_uc_deductions.py + test_state_pension_age.py: 37 passed.
• contrib/labour/attends_private_school.yaml + private_school_vat.yaml: 5 passed.
• Independent reviews:
  • An Opus 5.5 peer approved 40aaab0 with nits. I executed and fixed the zero-weight crash it found, and addressed its test nits.
  • A GPT-6.1 Sol review of 2fad4c0 requested changes, with three executed findings: the stale flag on rebuild, core simulations losing the imputations, and a region assertion that passed under the old gate. All three are fixed in 77141d9, with tests that fail without each fix.
  • The re-review of 77141d9 confirmed those fixes and found two more. It executed a crash: a calculated clone rebuilt from data read stale cached arrays. It also flagged an overstated line in this description. Both are fixed in the next commit; the rebuild test reproduces the crash without the cache reset.
• ruff format / ruff check clean.
• The full suite was not run locally (the targeted tests cover the changed variables); CI runs it.

axiom: n/a: microsimulation imputation

🤖 Generated with Claude Code