Stored household ISA totals fall to zero after 2032 #2740

On main (2d437eb), a dataset that stores the household cash_isa and stocks_and_shares_isa totals loses both after 2032. In 2033 and later years they equal zero, and Universal Credit capital drops by the full ISA value.

Reproduction

import pandas as pd
from policyengine_uk import Microsimulation
from policyengine_uk.data.dataset_schema import UKSingleYearDataset

person = pd.DataFrame({"person_id": [1, 2], "person_benunit_id": [1, 1], "person_household_id": [1, 1], "age": [30, 28]})
household = pd.DataFrame({"household_id": [1], "household_weight": [1.0], "region": ["LONDON"], "tenure_type": ["RENT_PRIVATELY"],
    "council_tax": [1_500.0], "rent": [12_000.0], "savings": [1_000.0], "cash_isa": [3_000.0], "stocks_and_shares_isa": [2_000.0]})
ds = UKSingleYearDataset(person=person, benunit=pd.DataFrame({"benunit_id": [1]}), household=household, fiscal_year=2025)
sim = Microsimulation(dataset=ds)
for y in (2025, 2031, 2032, 2033, 2034):
    print(y, {v: round(float(sim.calculate(v, y).values[0]), 2) for v in ("savings", "cash_isa", "stocks_and_shares_isa", "uc_assessable_capital")})

Output on 2d437eb:

savings keeps growing after 2032; the two ISA totals drop to zero.

Cause

#2712 (issue #2710, squash commit 3d5e7cc) added the person inputs cash_isa_balance and stocks_and_shares_isa_balance. It replaced the household variables' uprating = "gov.economic_assumptions.indices.obr.per_capita.gdp" with adds = ["cash_isa_balance"] and adds = ["stocks_and_shares_isa_balance"]. It also listed both totals in uprating_indices.yaml, so extend_single_year_dataset grows a stored total in every dataset year up to PROJECTION_END_YEAR (2032). Its commit message intends that datasets storing only the totals "keep them, projected as before".

The comment above PROJECTION_END_YEAR in policyengine_uk/data/economic_assumptions.py describes what happens next: past that year, policyengine-core serves each stored input from the last projected year, except that it calculates an input whose variable has a formula. The adds attribute gives both totals a formula, so from 2033 the model sums the person balances. A dataset that stores only the household totals holds no person balances, so the sums equal zero. Before #2712 the totals had no formula and carried the GDP-per-capita uprating attribute, so Core scaled them by that index after 2032.

This contradicts both variables' documentation, which says a dataset that stores the column "keeps the stored value, and projects it with savings" (cash_isa) or "with corporate_wealth" (stocks_and_shares_isa).

Which datasets it hits

• Microcosm UK exports: the was_wealth stage writes both household columns (UK_WAS_WEALTH_OUTPUT_COLUMNS in packages/microcosm-build/src/microcosm/build/uk_runtime/was_wealth.py, PolicyEngine/microcosm at fa1bdb18e). #2729 also notes that all current Microcosm exports store only the household totals.
• policyengine-uk-data's Enhanced FRS: unaffected. Its source (e1acdf96) writes neither column and folds the WAS investment-ISA value into corporate_wealth.
• Any user dataset or test fixture that stores the household totals.

Every year up to 2032 gives the right totals. Runs for 2033 and later understate ISA holdings, and with them means-tested capital for UC, Housing Benefit, Pension Credit and the legacy benefits that read these totals.

Fix

Past the horizon, a stored cash_isa or stocks_and_shares_isa should grow from its 2032 value the way years up to 2032 grew it, and the person-balance sum should apply only where the dataset stores no total. Two routes:

• make the past-horizon fallback serve and uprate a column that the dataset stores even when its variable has adds, for these two variables or generally; or
• keep the stored household totals in separate input variables, and give cash_isa and stocks_and_shares_isa a formula that prefers them and otherwise sums the person balances.

A regression test can store only household totals and check that cash_isa, stocks_and_shares_isa and uc_assessable_capital in 2033 equal their 2032 values grown by the same index, with a companion case where only person balances are stored.

Found while reviewing #1969; that PR does not change this behaviour.