eFRS source_household_id doesn't name the survey household: SPI copies and capital-gains clones count as separate households #591

What's wrong

The disclosure layer in policyengine.py (PolicyEngine/policyengine.py#605) counts support by the survey household each row came from, source_household_id, and refuses a dataset without it (src/policyengine/outputs/uk_geography_disclosure.py:115-122). On the enhanced FRS that id does not name the survey household:

• impute_income adds a copy of a subsample of FRS households with SPI incomes (household_is_spi_synthetic, datasets/imputations/income.py:501-505), and impute_capital_gains doubles the dataset (household_is_capital_gains_clone, datasets/imputations/capital_gains.py:144-154). Both go through stack_datasets, which shifts the copy's ids by max(first half) + 1 (utils/stack.py:17-22) and records no link back.
• clone_and_assign then sets source_household_id to the id after stacking (calibration/clone_and_assign.py:192). The SPI copy and the capital-gains clone of one FRS household therefore get different source ids.

So, depending on whether the published file keeps that column, the layer either refuses the eFRS, or accepts it and counts up to four rows of one survey household as separate households. The second case overstates support for exactly the measures that sit on high incomes: the dividend, savings and property rate rises, the higher-rate threshold and capital gains.

This matters before Budget day because the eFRS constituency weights are the named fallback for seat results (PolicyEngine/microcosm#1131 and PolicyEngine/microcosm#1241). And if the regional results planned through the layer for about 20 October (PolicyEngine/uk-autumn-budget-dashboard-2026#24) use the bundle's default UK dataset, which is still the eFRS (1.58.0), they meet the same problem.

Evidence

Code references above, on main (the same clone_and_assign line is in the 1.58.0 tag). Not checked against the 1.58.0 H5 itself, which needs private data access.

Proposed fix

• In stack_datasets, carry source_household_id through: set it on both halves before the offset (defaulting to household_id on the first stack) and leave it unchanged on the copy.
• In clone_and_assign, keep an existing source_household_id instead of overwriting it with the pre-clone id.
• Add a test: every row's source_household_id is the household_id of a row with neither copy flag set.
• Until a release carries it, callers can rebuild the id by undoing the two offsets (SPI copy, then capital-gains clone) and checking every result is a raw FRS household. A private script for UK local: rerun the 2025 Autumn Budget constituency analysis on the local candidate to find data gaps and per-seat support before Budget day microcosm#1241 does this and falls back to a coarse demographic key if the check fails.

Priority

High if the regional release on about 20 October or any seat release uses the eFRS; otherwise needed before the eFRS is used under the layer at all.

Related: PolicyEngine/policyengine.py#589, PolicyEngine/policyengine.py#605, #558, PolicyEngine/microcosm#1241.

🤖 Generated with Claude Code