diff --git a/changelog.d/savings-income-obr-uprating.changed.md b/changelog.d/savings-income-obr-uprating.changed.md new file mode 100644 index 0000000000..006a6106c6 --- /dev/null +++ b/changelog.d/savings-income-obr-uprating.changed.md @@ -0,0 +1 @@ +Datasets can declare that their savings income was calibrated to grow by the OBR's savings income forecast per person from 2022-23 to 2025-26 (October 2026 EFO, Table 4.5), in the H5 root attribute `policyengine_uk.savings_income_projection`. Dataset projection then grows savings interest, other investment income, interest on cash held in non-cash ISAs and, in datasets on the additive ISA contract, ISA interest by that series in those years and by GDP per head in other years. Datasets on the legacy gross ISA contract grow gross interest and hold its ISA part flat, as before. Datasets that declare nothing project exactly as before, by GDP per head in every year. diff --git a/docs/book/assumptions/growthfactors.md b/docs/book/assumptions/growthfactors.md index 4a4c717902..e472dcca18 100644 --- a/docs/book/assumptions/growthfactors.md +++ b/docs/book/assumptions/growthfactors.md @@ -103,7 +103,22 @@ We derive these rates as calendar-year nominal GDP growth from the OBR's October |-------------|------|------|------|------|------|------|------|------|------|------| | Growth rate | 10.0% | 5.2% | 4.0% | 4.6% | 4.0% | 3.1% | 3.4% | 3.6% | 3.4% | 3.3% | -We use per capita GDP for these variables: `benunit_reported_capital`, `capital_gains`, `capital_gains_badr`, `capital_gains_before_response`, `capital_gains_carried_interest`, `capital_gains_residential_property`, `cash_isa`, `cash_isa_balance`, `corporate_wealth`, `directly_held_shares`, `dividend_income`, `gross_financial_wealth`, `household_lifetime_isa_balance`, `lifetime_isa_balance`, `lump_sum_income`, `main_residence_value`, `maintenance_income`, `miscellaneous_income`, `mortgage_capital_repayment`, `net_financial_wealth`, `non_residential_property_value`, `other_investment_income`, `other_residential_property_value`, `owned_land`, `pension_credit_reported_capital`, `pension_income`, `private_pension_wealth`, `private_transfer_income`, `property_income`, `property_rental_income`, `savings`, `savings_interest_income`, `spousal_maintenance_income`, `stocks_and_shares_isa`, `stocks_and_shares_isa_balance`, `sublet_income`, `uc_reported_capital`, `unit_and_investment_trusts` +We use per capita GDP for these variables: `benunit_reported_capital`, `capital_gains`, `capital_gains_badr`, `capital_gains_before_response`, `capital_gains_carried_interest`, `capital_gains_residential_property`, `cash_isa`, `cash_isa_balance`, `corporate_wealth`, `directly_held_shares`, `dividend_income`, `gross_financial_wealth`, `household_lifetime_isa_balance`, `lifetime_isa_balance`, `lump_sum_income`, `main_residence_value`, `maintenance_income`, `miscellaneous_income`, `mortgage_capital_repayment`, `net_financial_wealth`, `non_residential_property_value`, `other_residential_property_value`, `owned_land`, `pension_credit_reported_capital`, `pension_income`, `private_pension_wealth`, `private_transfer_income`, `property_income`, `property_rental_income`, `savings`, `spousal_maintenance_income`, `stocks_and_shares_isa`, `stocks_and_shares_isa_balance`, `sublet_income`, `uc_reported_capital`, `unit_and_investment_trusts` + +## Savings income + +When we project a dataset to later years (up to 2032-33), we grow savings income by the path the dataset says its savings income was calibrated to. A dataset that declares the `obr_savings_income_2022_2025` projection grows it by the OBR's savings income forecast per person from 2022-23 to 2025-26 and by per capita GDP in other years. A dataset that declares nothing grows it by per capita GDP in every year, as before. The OBR's series is the "Savings income" row of its receipts table "Self-employed income streams: growth rates" (October 2026 EFO, Table 4.5, from 2024-25; earlier years from the March 2026 and October 2024 EFOs), divided by population growth. + +| Fiscal year | 2022 | 2023 | 2024 | 2025 | +|-------------|------|------|------|------| +| OBR savings income | 50.4% | 129.9% | 35.0% | 0.6% | +| Per person | 48.8% | 126.8% | 33.7% | 0.3% | + +From 2023-24 to 2025-26 the OBR's savings income rises 35.8%, where per capita GDP and population growth would give 10.2%. After 2025-26 these variables follow per capita GDP under either rule. + +The declaration matters because a dataset's savings income is fitted to targets its builder moved forward along some path. Growing it by a different path would miss those targets. Datasets store the declaration in the H5 root attribute `policyengine_uk.savings_income_projection`. + +We apply this to `non_cash_isa_cash_interest`, `other_investment_income` and `savings_interest_income`, and to ISA interest in datasets whose ISA income contract projects it additively. Datasets with the legacy gross contract grow gross interest, ISA interest included, and hold the ISA part flat. Inputs given directly to a calculation, and years after the dataset projection, use the index each variable declares: household interest income (ONS D.41) for interest and per capita GDP for other investment income. ## Council tax diff --git a/docs/engineering/skills/dataset-sources.md b/docs/engineering/skills/dataset-sources.md index 57efacef55..4a4c199baa 100644 --- a/docs/engineering/skills/dataset-sources.md +++ b/docs/engineering/skills/dataset-sources.md @@ -69,8 +69,9 @@ and `additive`/`legacy_gross` for normalized legacy data. Representation and projection origin are separate: legacy projection grows the original gross amount before subtracting the effective exemption. It must not grow the already-normalized ordinary amount instead. New additive dataset projections -grow ISA interest alongside ordinary interest by the dataset's GDP-per-capita -rule; outside the stored horizon new inputs use their variable's own index. +grow ISA interest alongside ordinary interest by the dataset's savings income +projection rule (see "Savings income projection" below); outside the stored +horizon new inputs use their variable's own index. Legacy-origin fallback retains the old gross-index calculation and unindexed ISA carry-over, while explicit inputs and applicable ISA detail formulas take priority. Legacy files containing only an aggregate exemption still have that @@ -100,6 +101,29 @@ discard H5 attributes must explicitly preserve this declaration before their output can be reloaded safely. Test routing and migration on synthetic frames and small H5 fixtures, without downloading private datasets. +## Savings income projection + +A dataset's savings income is fitted to targets that its builder moved to the +dataset year and later along some growth path, so dataset projection must grow +it along the same path. The dataset declares the path as a plain string in the +H5 root attribute `policyengine_uk.savings_income_projection` +(`data/savings_income_projection.py`): + +- `gdp_per_head`: GDP per capita in every year. Datasets without the attribute + get this rule, which is how savings income was projected before. +- `obr_savings_income_2022_2025`: the OBR's savings income per person from + 2022 to 2025 and GDP per capita in other years. + +`growth_rate` in `data/economic_assumptions.py` applies the rule to +`savings_interest_income`, `non_cash_isa_cash_interest`, +`other_investment_income`, gross interest under the legacy contract and ISA +interest under the additive contract. `UKSingleYearDataset` takes +`savings_income_projection` and refuses a value that contradicts its file. +Multi-year datasets hold one rule for every year and refuse mixed rules. +Copies, filtered households, simulation exports and input frames +(`frame.attrs["policyengine_uk.savings_income_projection"]`) keep the rule. A +rule's name fixes its years, so a later window needs a new rule. + ## Apprenticeship status and tenure Dataset producers should export `is_apprentice` and diff --git a/policyengine_uk/data/dataset_schema.py b/policyengine_uk/data/dataset_schema.py index 23561e0317..708d510877 100644 --- a/policyengine_uk/data/dataset_schema.py +++ b/policyengine_uk/data/dataset_schema.py @@ -17,6 +17,13 @@ read_energy_price_basis, write_energy_price_basis, ) +from policyengine_uk.data.savings_income_projection import ( + SAVINGS_INCOME_PROJECTION_KEY, + read_savings_income_projection, + resolve_savings_income_projection, + uniform_savings_income_projection, + validate_savings_income_projection, +) from policyengine_uk.data.isa_income import ( ISA_INCOME_CONTRACT_KEY, MISSING_CONTRACT, @@ -78,6 +85,7 @@ def __init__( fiscal_year: int = None, energy_price_basis: dict[str, str | int] | EnergyPriceBasis | None = None, isa_income_contract: dict | None = None, + savings_income_projection: str | None = None, ): basis = parse_energy_price_basis(energy_price_basis) file_path = str(file_path) if file_path else None @@ -126,6 +134,9 @@ def __init__( read_h5_contract(file_path), [int(self.time_period[:4])], )[int(self.time_period[:4])] + self.savings_income_projection = resolve_savings_income_projection( + savings_income_projection, read_savings_income_projection(file_path) + ) self.data_format = "arrays" # tables tuple points at the raw (int16) DataFrames used by the # simulation engine internals - do not decode these. @@ -198,6 +209,9 @@ def save(self, file_path: str): write_energy_price_basis(file_path, self._energy_price_basis) with h5py.File(file_path, "a") as f: f.attrs[ISA_INCOME_CONTRACT_KEY] = json.dumps(contract) + f.attrs[SAVINGS_INCOME_PROJECTION_KEY] = validate_savings_income_projection( + self.savings_income_projection + ) def load(self): data = {} @@ -215,6 +229,7 @@ def copy(self): fiscal_year=self.time_period, energy_price_basis=self._energy_price_basis, isa_income_contract=validate_contract(self.isa_income_contract), + savings_income_projection=self.savings_income_projection, ) def validate(self): @@ -229,6 +244,7 @@ def from_simulation( simulation: "Microsimulation", fiscal_year: int = 2025, energy_price_basis: dict[str, str | int] | EnergyPriceBasis | None = None, + savings_income_projection: str | None = None, ): entity_dfs = {} @@ -253,6 +269,11 @@ def from_simulation( fiscal_year=fiscal_year, energy_price_basis=energy_price_basis, isa_income_contract=simulation_contract(simulation, fiscal_year), + savings_income_projection=( + getattr(simulation, "_savings_income_projection", None) + if savings_income_projection is None + else savings_income_projection + ), ) @@ -272,10 +293,15 @@ def __init__( ) year = int(dataset.time_period[:4]) self.datasets[year] = dataset + # Projection grows every year by one declared rule. + uniform_savings_income_projection( + dataset.savings_income_projection for dataset in datasets + ) if file_path is not None: UKMultiYearDataset.validate_file_path(file_path) stored_contract = read_h5_contract(file_path) + stored_savings_rule = read_savings_income_projection(file_path) with pd.HDFStore(file_path, mode="r") as f: self.datasets = {} years = [ @@ -296,6 +322,7 @@ def __init__( household=household_df, fiscal_year=fiscal_year, isa_income_contract=contracts[fiscal_year], + savings_income_projection=stored_savings_rule, ) for year, dataset in self.datasets.items(): dataset.energy_price_basis = read_energy_price_basis( @@ -324,11 +351,18 @@ def isa_income_contract(self): } ) + @property + def savings_income_projection(self) -> str: + return uniform_savings_income_projection( + dataset.savings_income_projection for dataset in self.datasets.values() + ) + def __getitem__(self, fiscal_year: int): return self.get_year(fiscal_year) def save(self, file_path: str): contracts = self.isa_income_contract + savings_rule = self.savings_income_projection Path(file_path).unlink(missing_ok=True) # Remove existing file if it exists with pd.HDFStore(file_path) as f: for year, dataset in self.datasets.items(): @@ -364,6 +398,7 @@ def save(self, file_path: str): with h5py.File(file_path, "a") as f: f.attrs[ISA_INCOME_CONTRACT_KEY] = json.dumps(contracts) + f.attrs[SAVINGS_INCOME_PROJECTION_KEY] = savings_rule def copy(self): new_datasets = {year: dataset.copy() for year, dataset in self.datasets.items()} diff --git a/policyengine_uk/data/economic_assumptions.py b/policyengine_uk/data/economic_assumptions.py index 6db07617f5..815a8d31d1 100644 --- a/policyengine_uk/data/economic_assumptions.py +++ b/policyengine_uk/data/economic_assumptions.py @@ -3,6 +3,13 @@ CGT_DATASET_INPUTS, CGT_INDEX_PATH, ) +from policyengine_uk.data.savings_income_projection import ( + DEFAULT_SAVINGS_INCOME_PROJECTION, + OBR_SAVINGS_INCOME_2022_2025, + SAVINGS_INCOME_SERIES_YEARS, + uniform_savings_income_projection, + validate_savings_income_projection, +) from policyengine_uk.data.isa_income import ( ISA_INTEREST, SAVINGS_INTEREST, @@ -101,6 +108,45 @@ # them a second time after LEGACY_PROJECTION_END_YEAR. COUNTRY_UPRATED_INPUTS = frozenset({"water_and_sewerage_charges"}) +# Savings income (savings interest, ISA interest and other investment income) +# grows in dataset projection by the path the dataset declares it was +# calibrated to (savings_income_projection.py). Under the +# obr_savings_income_2022_2025 rule that is the OBR's savings income forecast +# per person from 2022-23 to 2025-26, and GDP per head in other years, like +# dividends and the other capital incomes. The OBR's series rises 35.0% in +# 2024-25 and 0.6% in 2025-26, where GDP per head rises 4.0% and 4.6%. It +# starts in 2022-23, so earlier years keep GDP per head rather than the +# parameter's backdated first value. A dataset that declares no rule grows +# savings income by GDP per head in every year, as before. +SAVINGS_INCOME_GROWTH = ( + "gov.economic_assumptions.yoy_growth.obr.per_capita.savings_income" +) +GDP_PER_HEAD_GROWTH = "gov.economic_assumptions.yoy_growth.obr.per_capita.gdp" +SAVINGS_INCOME_GROWTH_FIRST_YEAR, SAVINGS_INCOME_GROWTH_LAST_YEAR = ( + SAVINGS_INCOME_SERIES_YEARS[OBR_SAVINGS_INCOME_2022_2025] +) + + +def growth_rate( + parameters: ParameterNode, + index_name: str, + year, + savings_income_projection: str = DEFAULT_SAVINGS_INCOME_PROJECTION, +) -> float: + """Growth into ``year`` by the uprating_indices.yaml series ``index_name``. + + The savings income series applies only in the years the dataset's + ``savings_income_projection`` rule gives it; GDP per head serves the + other years, and every year under the default rule. + """ + if index_name == SAVINGS_INCOME_GROWTH: + years = SAVINGS_INCOME_SERIES_YEARS[ + validate_savings_income_projection(savings_income_projection) + ] + if years is None or not years[0] <= int(str(year)[:4]) <= years[1]: + index_name = GDP_PER_HEAD_GROWTH + return parameters.get_child(index_name)(year) + # The last year a single-year dataset is projected to: 2032, fiscal year # 2032-33. An OBR forecast runs five years past the fiscal year it is made in, @@ -350,9 +396,17 @@ def apply_single_year_uprating( # First, apply standard variable-YoY growth based uprating. + savings_rule = uniform_savings_income_projection( + [ + previous_year.savings_income_projection, + current_year.savings_income_projection, + ] + ) uprating = _uprating_index_map() for index_name, variables in uprating.items(): - index_rel_change = parameters.get_child(index_name)(current_year.time_period) + index_rel_change = growth_rate( + parameters, index_name, current_year.time_period, savings_rule + ) for variable in variables: for table_name, df in zip(current_year.table_names, current_year.tables): if variable in df.columns: @@ -374,23 +428,23 @@ def apply_single_year_uprating( current_year.isa_income_contract["projection"] == "legacy_gross" and SAVINGS_INTEREST in current_year.person ): - growth = parameters.get_child( - "gov.economic_assumptions.yoy_growth.obr.per_capita.gdp" - )(current_year.time_period) + growth = growth_rate( + parameters, SAVINGS_INCOME_GROWTH, current_year.time_period, savings_rule + ) values = gross_interest(previous_year) * (1 + growth) if current_year.isa_income_contract["representation"] == "additive": values = values - effective_tax_free(current_year.person) current_year.person[SAVINGS_INTEREST] = values - # New additive ISA receipts use the same dataset GDP rule as ordinary + # New additive ISA receipts use the same dataset rule as ordinary # interest. Legacy receipts retain their former unindexed treatment. if ( current_year.isa_income_contract["projection"] == "additive" and ISA_INTEREST in current_year.person ): - growth = parameters.get_child( - "gov.economic_assumptions.yoy_growth.obr.per_capita.gdp" - )(current_year.time_period) + growth = growth_rate( + parameters, SAVINGS_INCOME_GROWTH, current_year.time_period, savings_rule + ) current_year.person[ISA_INTEREST] = previous_year.person[ISA_INTEREST] * ( 1 + growth ) diff --git a/policyengine_uk/data/filter_dataset.py b/policyengine_uk/data/filter_dataset.py index 9d80dcc189..09eac75ec3 100644 --- a/policyengine_uk/data/filter_dataset.py +++ b/policyengine_uk/data/filter_dataset.py @@ -71,4 +71,5 @@ def filter_dataset( fiscal_year=year, energy_price_basis=new_dataset.energy_price_basis, isa_income_contract=new_dataset.isa_income_contract, + savings_income_projection=new_dataset.savings_income_projection, ) diff --git a/policyengine_uk/data/input_dataframe.py b/policyengine_uk/data/input_dataframe.py index 8248503fb1..6f2c964b8a 100644 --- a/policyengine_uk/data/input_dataframe.py +++ b/policyengine_uk/data/input_dataframe.py @@ -38,6 +38,9 @@ UKMultiYearDataset, UKSingleYearDataset, ) +from policyengine_uk.data.savings_income_projection import ( + SAVINGS_INCOME_PROJECTION_KEY, +) from policyengine_uk.data.isa_income import ( ISA_INCOME_CONTRACT_KEY, MISSING_CONTRACT, @@ -447,6 +450,7 @@ def no_tables() -> Dict[str, Dict[str, np.ndarray]]: ), fiscal_year=fiscal_year, isa_income_contract=contracts[fiscal_year], + savings_income_projection=df.attrs.get(SAVINGS_INCOME_PROJECTION_KEY), ) for fiscal_year, tables in sorted(years.items()) ] diff --git a/policyengine_uk/data/savings_income_projection.py b/policyengine_uk/data/savings_income_projection.py new file mode 100644 index 0000000000..8065f57e55 --- /dev/null +++ b/policyengine_uk/data/savings_income_projection.py @@ -0,0 +1,100 @@ +"""Which series a UK dataset's savings income was calibrated to grow by. + +A dataset's savings amounts are fitted to targets that its builder moved to +the dataset year and beyond with some growth path. Dataset projection must +grow them by the same path, or the projected totals miss the targets. So the +dataset declares the path, in an H5 root attribute, and projection follows +the declaration. + +- ``gdp_per_head``: GDP per head in every year. Datasets that declare nothing + get this rule, which is how savings income was projected before the OBR + series was added. +- ``obr_savings_income_2022_2025``: the OBR's savings income forecast per + person from 2022-23 to 2025-26, and GDP per head in other years. + +A rule's name fixes its years, so data built under one rule keeps its meaning +if a later rule covers other years. +""" + +from pathlib import Path + +import h5py + +SAVINGS_INCOME_PROJECTION_KEY = "policyengine_uk.savings_income_projection" +GDP_PER_HEAD = "gdp_per_head" +OBR_SAVINGS_INCOME_2022_2025 = "obr_savings_income_2022_2025" +DEFAULT_SAVINGS_INCOME_PROJECTION = GDP_PER_HEAD + +# The fiscal years (by starting year) in which each rule grows savings income +# by the OBR's savings income series per person, or None for no years. +SAVINGS_INCOME_SERIES_YEARS = { + GDP_PER_HEAD: None, + OBR_SAVINGS_INCOME_2022_2025: (2022, 2025), +} + + +def _error(message): + return ValueError("Savings income projection: " + message) + + +def validate_savings_income_projection(rule) -> str: + if not isinstance(rule, str) or rule not in SAVINGS_INCOME_SERIES_YEARS: + raise _error( + f"unknown rule {rule!r}; expected one of " + f"{sorted(SAVINGS_INCOME_SERIES_YEARS)}." + ) + return rule + + +def read_savings_income_projection(path) -> str | None: + """The rule an H5 file declares, or None if it declares none.""" + if path is None or not Path(path).is_file() or Path(path).suffix != ".h5": + return None + with h5py.File(path, "r") as file: + if SAVINGS_INCOME_PROJECTION_KEY not in file.attrs: + return None + rule = file.attrs[SAVINGS_INCOME_PROJECTION_KEY] + if isinstance(rule, bytes): + try: + rule = rule.decode("utf-8") + except UnicodeError as error: + raise _error("malformed H5 metadata.") from error + return validate_savings_income_projection(rule) + + +def write_savings_income_projection(path, rule) -> None: + with h5py.File(path, "a") as file: + file.attrs[SAVINGS_INCOME_PROJECTION_KEY] = validate_savings_income_projection( + rule + ) + + +def resolve_savings_income_projection(explicit, stored) -> str: + """An absent declaration is the default, not a contradictory one.""" + explicit = ( + None if explicit is None else validate_savings_income_projection(explicit) + ) + stored = None if stored is None else validate_savings_income_projection(stored) + if explicit is not None and stored is not None and explicit != stored: + raise _error("explicit object and file metadata disagree.") + return explicit or stored or DEFAULT_SAVINGS_INCOME_PROJECTION + + +def uniform_savings_income_projection(rules) -> str: + """The one rule a multi-year dataset's years share.""" + rules = {validate_savings_income_projection(rule) for rule in rules} + if len(rules) > 1: + raise _error(f"a dataset's years declare different rules: {sorted(rules)}.") + return rules.pop() if rules else DEFAULT_SAVINGS_INCOME_PROJECTION + + +def core_dataset_savings_income_projection(dataset) -> str: + """Read a Core dataset's declaration without its dynamic variable lookup.""" + try: + explicit = object.__getattribute__(dataset, "savings_income_projection") + except AttributeError: + explicit = None + return resolve_savings_income_projection( + explicit, + read_savings_income_projection(getattr(dataset, "file_path", None)), + ) diff --git a/policyengine_uk/data/uprating_indices.yaml b/policyengine_uk/data/uprating_indices.yaml index ee591c95de..921f97903a 100644 --- a/policyengine_uk/data/uprating_indices.yaml +++ b/policyengine_uk/data/uprating_indices.yaml @@ -124,7 +124,6 @@ gov.economic_assumptions.yoy_growth.obr.per_capita.gdp: - net_financial_wealth - non_residential_property_value - non_residential_property_secured_debt -- other_investment_income - other_residential_property_secured_debt # Non-residential purchase prices and land retain GDP-per-capita growth. - non_residential_property_purchased @@ -138,8 +137,6 @@ gov.economic_assumptions.yoy_growth.obr.per_capita.gdp: - property_income - property_rental_income - savings -- savings_interest_income -- non_cash_isa_cash_interest - spousal_maintenance_income - stocks_and_shares_isa - stocks_and_shares_isa_balance @@ -155,6 +152,12 @@ gov.economic_assumptions.yoy_growth.obr.per_capita.gdp: # financial input; its default no longer derives from that input. - uc_reported_capital - unit_and_investment_trusts +# Savings income: the OBR's savings income forecast per person to 2025-26, +# then GDP per head (growth_rate in economic_assumptions.py). +gov.economic_assumptions.yoy_growth.obr.per_capita.savings_income: +- non_cash_isa_cash_interest +- other_investment_income +- savings_interest_income gov.economic_assumptions.yoy_growth.obr.per_capita.mixed_income: - self_employment_income - self_employment_gross_receipts diff --git a/policyengine_uk/parameters/gov/economic_assumptions/yoy_growth.yaml b/policyengine_uk/parameters/gov/economic_assumptions/yoy_growth.yaml index f654017784..bd5cb74d6c 100644 --- a/policyengine_uk/parameters/gov/economic_assumptions/yoy_growth.yaml +++ b/policyengine_uk/parameters/gov/economic_assumptions/yoy_growth.yaml @@ -635,6 +635,43 @@ obr: - title: OBR EFO October 2026 (detailed forecast tables, economy, Table 1.12) href: https://obr.uk/efo/economic-and-fiscal-outlook-october-2026/ + savings_income: + description: Savings income year-on-year growth, from the OBR's receipts forecast. + # The "Savings income" row of the OBR's "Self-employed income streams: + # growth rates" table, percentage change on a year earlier, pre-measures, + # rounded to 4 decimals. The fiscal year starting in year t is stored at + # t-01-01, as for ons.population. Each year is from the latest table that + # has it: + # - 2024-2031: EFO October 2026, receipts Table 4.5 (2024-25 34.961%, + # 2025-26 0.650%). The table heads 2024-25 onwards "Forecast". + # - 2023: EFO March 2026, receipts Table 3.5 (129.854%, the same in the + # November 2025 table). + # - 2022: EFO October 2024, receipts Table 3.5 (50.358%). It is the first + # listed year, so it is the base of the generated index (index 1 at + # 2022) and its own growth is not in the index (#2692). + # The table ends at 2031-32; a parameter keeps its last value after that. + values: + 2022-01-01: 0.5036 + 2023-01-01: 1.2985 + 2024-01-01: 0.3496 + 2025-01-01: 0.0065 + 2026-01-01: 0.0067 + 2027-01-01: 0.0614 + 2028-01-01: 0.0514 + 2029-01-01: 0.0494 + 2030-01-01: 0.0556 + 2031-01-01: 0.0589 + metadata: + unit: /1 + label: Savings income growth + reference: + - title: OBR, Economic and fiscal outlook October 2026, detailed forecast tables, receipts, Table 4.5 (sha256 d115705716d77b11c190385d41cb1924f30080c9331a8a386c04d4e65a368e95) + href: https://obr.uk/efo/economic-and-fiscal-outlook-october-2026/ + - title: OBR, Economic and fiscal outlook March 2026, detailed forecast tables, receipts, Table 3.5 (sha256 f65b6cb7f96931d4abd85940794eb60f6f65afca8bbf16c74a4548bf92766a28) + href: https://obr.uk/docs/d055fbf02d5b3g6jq8l2/efo-march-2026-detailed-forecast-tables-receipts.xlsx + - title: OBR, Economic and fiscal outlook October 2024, detailed forecast tables, receipts, Table 3.5 (sha256 16d69c0981f45e5b7fe3fa35d08bfe09e6170440231d546f374a9f4d48191991) + href: https://obr.uk/docs/dlm_uploads/Receipts_Detailed_forecast_tables_October_2024.xlsx + # Growth in each country's council tax receipts, net of council tax # reduction. Receipts are the Band D level times the tax base (Band D # equivalent dwellings), so they grow with the number of homes as well as @@ -943,6 +980,31 @@ obr: - title: OBR EFO October 2026 (detailed forecast tables, economy, Table 1.12) href: https://obr.uk/efo/economic-and-fiscal-outlook-october-2026/ + savings_income: + description: Per capita savings income year-on-year growth. + values: + # obr.savings_income divided by one plus ons.population, both for + # the fiscal year starting in year t: (1 + growth) / (1 + population) + # - 1, rounded to 4 decimals. Dataset projection grows household + # weights by population, so each person's savings income grows by + # this from 2022 to 2025 (data/economic_assumptions.py, growth_rate). + 2022-01-01: 0.4880 + 2023-01-01: 1.2681 + 2024-01-01: 0.3366 + 2025-01-01: 0.0032 + 2026-01-01: 0.0047 + 2027-01-01: 0.0600 + 2028-01-01: 0.0496 + 2029-01-01: 0.0472 + 2030-01-01: 0.0530 + 2031-01-01: 0.0564 + metadata: + unit: /1 + label: Per capita savings income growth + reference: + - title: OBR, Economic and fiscal outlook October 2026, detailed forecast tables, receipts, Table 4.5, divided by ons.population growth + href: https://obr.uk/efo/economic-and-fiscal-outlook-october-2026/ + house_prices: description: House prices year-on-year growth. values: diff --git a/policyengine_uk/simulation.py b/policyengine_uk/simulation.py index dd1e49c53a..e38e15bba0 100644 --- a/policyengine_uk/simulation.py +++ b/policyengine_uk/simulation.py @@ -44,6 +44,11 @@ ) from policyengine_uk.data.dataset_sources import materialize_gcs_dataset_url from policyengine_uk.data.energy_price_basis import core_dataset_energy_price_basis +from policyengine_uk.data.savings_income_projection import ( + SAVINGS_INCOME_PROJECTION_KEY, + core_dataset_savings_income_projection, + resolve_savings_income_projection, +) from policyengine_uk.data.input_dataframe import ( dataset_from_input_dataframe, hold_identifiers_in, @@ -650,6 +655,9 @@ def clone(self, *args, **kwargs) -> "Simulation": clone._isa_income_contracts = copy.deepcopy( getattr(self, "_isa_income_contracts", {}) ) + clone._savings_income_projection = getattr( + self, "_savings_income_projection", None + ) return clone def _isa_latest_input(self, name, period, *, earlier=False): @@ -1252,6 +1260,9 @@ def to_input_dataframe( frame.attrs[ISA_INCOME_CONTRACT_KEY] = contract_envelope( {year: simulation_contract(self, year) for year in years} ) + frame.attrs[SAVINGS_INCOME_PROJECTION_KEY] = resolve_savings_income_projection( + getattr(self, "_savings_income_projection", None), None + ) return frame def build_from_dataset(self, dataset: Dataset) -> None: @@ -1261,6 +1272,7 @@ def build_from_dataset(self, dataset: Dataset) -> None: dataset: PolicyEngine Dataset object containing simulation data """ energy_price_basis = core_dataset_energy_price_basis(dataset) + savings_income_projection = core_dataset_savings_income_projection(dataset) data: Dict[str, Dict[str, Union[float, int, str]]] = dataset.load_dataset() first_variable = data[list(data.keys())[0]] @@ -1290,6 +1302,7 @@ def get_first_array(variable_name: str) -> np.ndarray: read_h5_contract(getattr(dataset, "file_path", None)), [period_(first_time_period).start.year], ) + self._savings_income_projection = savings_income_projection # Populations use sorted group IDs; arrays in a Core dataset follow # the order of that period's supplied identifier array instead. @@ -1321,7 +1334,10 @@ def get_first_array(variable_name: str) -> np.ndarray: self.dataset = dataset dataset = UKSingleYearDataset.from_simulation( - self, fiscal_year=first_time_period, energy_price_basis=energy_price_basis + self, + fiscal_year=first_time_period, + energy_price_basis=energy_price_basis, + savings_income_projection=savings_income_projection, ) multi_year_dataset = self._extend_dataset_with_cgt_provenance(dataset) @@ -1380,6 +1396,7 @@ def build_from_multi_year_dataset(self, dataset: UKMultiYearDataset) -> None: year: copy.deepcopy(dataset[year].isa_income_contract) for year in dataset.years } + self._savings_income_projection = dataset.savings_income_projection # Load variable values for all years for year in dataset.years: @@ -1428,6 +1445,7 @@ def _start_new_population(self, built_from_dataset: bool) -> None: self._cgt_operation_inputs = set() self._cgt_warned_unused = False self._isa_income_contracts = {} + self._savings_income_projection = None def build_from_ids( self, @@ -1547,6 +1565,7 @@ def calculate_all(self, year: int) -> None: benunit=benunit, household=household, fiscal_year=year, + savings_income_projection=getattr(self, "_savings_income_projection", None), ) def _calculate(self, variable_name: str, period: Period = None): diff --git a/policyengine_uk/tests/code_health/test_uprating_declarations.py b/policyengine_uk/tests/code_health/test_uprating_declarations.py index 24c1c898dd..8c2fa65dac 100644 --- a/policyengine_uk/tests/code_health/test_uprating_declarations.py +++ b/policyengine_uk/tests/code_health/test_uprating_declarations.py @@ -42,13 +42,15 @@ # Variables the YAML deliberately uprates by an index other than the one they # declare, with the reason. KNOWN_INDEX_OVERRIDES = { - # Declares ons.household_interest_income; the YAML applies per_capita.gdp. - # Flagged in #1862 for someone with the context to say which is intended. + # The savings income inputs. Dataset projection grows them by the OBR's + # savings income forecast per person to 2025-26 and by GDP per head after + # it (growth_rate in data/economic_assumptions.py), which no single + # declared index expresses. The first two declare + # ons.household_interest_income and the third obr.per_capita.gdp, which + # Core uses for direct inputs. Aligning those surfaces is #1862. "savings_interest_income", - # Like ordinary savings interest, noncash ISA interest declares the ONS - # household index but follows the existing GDP dataset rule. Broader - # alignment of those two surfaces remains separate work in #2638. "non_cash_isa_cash_interest", + "other_investment_income", } diff --git a/policyengine_uk/tests/test_projection_horizon.py b/policyengine_uk/tests/test_projection_horizon.py index 6883f132e3..6b2ff922c0 100644 --- a/policyengine_uk/tests/test_projection_horizon.py +++ b/policyengine_uk/tests/test_projection_horizon.py @@ -32,10 +32,19 @@ extend_single_year_dataset, reset_growthfactor_uprating, ) +from policyengine_uk.data.savings_income_projection import ( + OBR_SAVINGS_INCOME_2022_2025, +) from policyengine_uk.system import system BASE_YEAR = 2024 GROWTH = parameters.gov.economic_assumptions.yoy_growth +# The fixture declares the OBR savings income projection, under which the +# savings income series grows its inputs only from 2022 to 2025; GDP per head +# serves the other years. Undeclared data is checked in +# test_savings_income_uprating.py. +SAVINGS_INCOME = "gov.economic_assumptions.yoy_growth.obr.per_capita.savings_income" +GDP_PER_HEAD = "gov.economic_assumptions.yoy_growth.obr.per_capita.gdp" UPRATING_INDICES = yaml.safe_load( (Path(economic_assumptions.__file__).parent / "uprating_indices.yaml").read_text() ) @@ -91,6 +100,15 @@ } +def _growth(index, year, savings_rule=OBR_SAVINGS_INCOME_2022_2025): + """The growth uprating_indices.yaml's ``index`` gives into ``year``.""" + if index == SAVINGS_INCOME and not ( + savings_rule == OBR_SAVINGS_INCOME_2022_2025 and 2022 <= year <= 2025 + ): + index = GDP_PER_HEAD + return parameters.get_child(index)(year) + + def _dataset(amount=1_000.0, base_year=BASE_YEAR): """One person per household, with every indexed input and excluded energy input set to ``amount`` (or one amount per household), plus rent and @@ -131,6 +149,7 @@ def _dataset(amount=1_000.0, base_year=BASE_YEAR): benunit=pd.DataFrame(tables["benunit"]), household=pd.DataFrame(tables["household"]), fiscal_year=base_year, + savings_income_projection=OBR_SAVINGS_INCOME_2022_2025, ) @@ -195,7 +214,8 @@ def test_each_singly_indexed_input_grows_by_its_index_every_year(projected): """For every variable uprating_indices.yaml lists and every projected year, check its index: residential stocks use the original observation and cumulative HPI. This fixture stores the completed bereavement split, - so BSP stays flat; other inputs grow from the previous year's raw YoY.""" + so BSP stays flat; other inputs grow from the previous year's raw YoY, + with savings income on GDP per head outside 2022 to 2025.""" observed = _dataset() for year in range(BASE_YEAR + 1, PROJECTION_END_YEAR + 1): for variable, index in INDEX_OF.items(): @@ -205,7 +225,7 @@ def test_each_singly_indexed_input_grows_by_its_index_every_year(projected): expected = _residential_stock_level(original, variable, year) else: previous = _table(projected[year - 1], variable)[variable].to_numpy() - growth = parameters.get_child(index)(year) + growth = _growth(index, year) expected = ( previous if variable == "bsp_reported" else previous * (1 + growth) ) @@ -320,7 +340,7 @@ def test_growth_ratio_is_one_plus_the_index(amounts, year, variable): elif variable == "bsp_reported": expected_ratio = np.ones(len(amounts)) else: - index = parameters.get_child(INDEX_OF[variable])(year) + index = _growth(INDEX_OF[variable], year) expected_ratio = np.full(len(amounts), 1 + index) assert ratio == pytest.approx(expected_ratio) diff --git a/policyengine_uk/tests/test_savings_income_uprating.py b/policyengine_uk/tests/test_savings_income_uprating.py new file mode 100644 index 0000000000..643fd45e1e --- /dev/null +++ b/policyengine_uk/tests/test_savings_income_uprating.py @@ -0,0 +1,583 @@ +"""Savings income grows by the OBR's savings income forecast to 2025-26. + +obr.savings_income is the "Savings income" row of the OBR's receipts table +"Self-employed income streams: growth rates" (percentage change on a year +earlier, pre-measures), fiscal year t-(t+1) stored at t-01-01, rounded to 4 +decimals. Each year comes from the latest table that has it: + +- 2024-2031: EFO October 2026, detailed forecast tables, receipts, Table 4.5 + (sha256 d115705716d77b11c190385d41cb1924f30080c9331a8a386c04d4e65a368e95); +- 2023: EFO March 2026, receipts Table 3.5 (sha256 f65b6cb7...766a28); +- 2022: EFO October 2024, receipts Table 3.5 (sha256 16d69c09...191991). + +obr.per_capita.savings_income is that growth net of ons.population growth. +A dataset that declares the obr_savings_income_2022_2025 projection (an H5 +root attribute, savings_income_projection.py) grows savings_interest_income, +non_cash_isa_cash_interest, other_investment_income and additive ISA interest +by the per-person series from 2022 to 2025 and by obr.per_capita.gdp in other +years: after 2025, and before 2022, where the series has no source and the +parameter only repeats its first value. A dataset that declares nothing, as +every dataset built before the declaration existed, grows them by GDP per +head in every year, as before. +""" + +from copy import deepcopy +from types import SimpleNamespace + +import h5py +import numpy as np +import pandas as pd +import pytest +from hypothesis import given, settings +from hypothesis import strategies as st + +from policyengine_core.data import Dataset + +from policyengine_uk import Microsimulation +from policyengine_uk.data.dataset_schema import UKMultiYearDataset, UKSingleYearDataset +from policyengine_uk.data.filter_dataset import filter_dataset +from policyengine_uk.data.savings_income_projection import ( + DEFAULT_SAVINGS_INCOME_PROJECTION, + GDP_PER_HEAD, + OBR_SAVINGS_INCOME_2022_2025, + SAVINGS_INCOME_PROJECTION_KEY, + core_dataset_savings_income_projection, + read_savings_income_projection, + write_savings_income_projection, +) +from policyengine_uk.data.economic_assumptions import ( + GDP_PER_HEAD_GROWTH, + SAVINGS_INCOME_GROWTH, + SAVINGS_INCOME_GROWTH_FIRST_YEAR, + SAVINGS_INCOME_GROWTH_LAST_YEAR, + _uprating_index_map, + apply_single_year_uprating, + extend_single_year_dataset, + growth_rate, +) +from policyengine_uk.system import system + +PARAMETERS = system.parameters +YOY = PARAMETERS.gov.economic_assumptions.yoy_growth +INDICES = PARAMETERS.gov.economic_assumptions.indices +LEGACY = {"version": 1, "representation": "legacy_gross", "projection": "legacy_gross"} +ADDITIVE = {"version": 1, "representation": "additive", "projection": "additive"} + +# The OBR's "Savings income" growth, per cent, as the workbooks store it. +OBR_SAVINGS_INCOME_GROWTH = { + 2022: 50.358, # EFO October 2024, Table 3.5 (shown to 3 decimals) + 2023: 129.854, # EFO March 2026, Table 3.5 (shown to 3 decimals) + 2024: 34.96108270919673, # EFO October 2026, Table 4.5, from here on + 2025: 0.6496896187592638, + 2026: 0.6683540761534434, + 2027: 6.138376573221471, + 2028: 5.1434341752103485, + 2029: 4.936559409911245, + 2030: 5.559480586702126, + 2031: 5.893489202164259, +} +SAVINGS_INPUTS = ( + "savings_interest_income", + "non_cash_isa_cash_interest", + "other_investment_income", +) + + +def at(parameter, year): + return parameter(f"{year}-01-01") + + +@pytest.mark.parametrize("year", sorted(OBR_SAVINGS_INCOME_GROWTH)) +def test_savings_income_is_the_obr_row(year): + expected = round(OBR_SAVINGS_INCOME_GROWTH[year] / 100, 4) + assert at(YOY.obr.savings_income, year) == pytest.approx(expected, abs=1e-12) + + +@pytest.mark.parametrize("year", sorted(OBR_SAVINGS_INCOME_GROWTH)) +def test_per_capita_savings_income_divides_by_population(year): + expected = round( + (1 + at(YOY.obr.savings_income, year)) / (1 + at(YOY.ons.population, year)) - 1, + 4, + ) + assert at(YOY.obr.per_capita.savings_income, year) == pytest.approx( + expected, abs=1e-12 + ) + + +def test_aggregate_index_carries_2023_24_to_2025_26_on_the_obr_path(): + """The level a 2023-24 figure reaches by 2025-26, as the OBR has it.""" + obr = (1 + OBR_SAVINGS_INCOME_GROWTH[2024] / 100) * ( + 1 + OBR_SAVINGS_INCOME_GROWTH[2025] / 100 + ) + index = INDICES.obr.savings_income + assert obr == pytest.approx(1.358379, abs=1e-6) + assert at(index, 2025) / at(index, 2023) == pytest.approx(obr, abs=1e-4) + assert at(index, 2024) / at(index, 2023) == pytest.approx(1.3496, abs=1e-5) + assert at(index, 2023) / at(index, 2022) == pytest.approx(2.2985, abs=1e-5) + + +def test_the_savings_window_runs_from_the_first_sourced_year_to_2025_26(): + assert (SAVINGS_INCOME_GROWTH_FIRST_YEAR, SAVINGS_INCOME_GROWTH_LAST_YEAR) == ( + min(OBR_SAVINGS_INCOME_GROWTH), + 2025, + ) + + +@pytest.mark.parametrize("rule", [OBR_SAVINGS_INCOME_2022_2025, GDP_PER_HEAD]) +@pytest.mark.parametrize("year", range(2015, 2041)) +def test_growth_rate_uses_gdp_per_head_outside_2022_23_to_2025_26(year, rule): + series = ( + YOY.obr.per_capita.savings_income + if rule == OBR_SAVINGS_INCOME_2022_2025 and 2022 <= year <= 2025 + else YOY.obr.per_capita.gdp + ) + assert growth_rate(PARAMETERS, SAVINGS_INCOME_GROWTH, str(year), rule) == at( + series, year + ) + # Every other series uprating_indices.yaml maps reads its own values. + others = set(_uprating_index_map()) - {SAVINGS_INCOME_GROWTH} + assert { + GDP_PER_HEAD_GROWTH, + "gov.economic_assumptions.yoy_growth.ons.population", + } <= (others) + for name in sorted(others): + assert growth_rate(PARAMETERS, name, str(year), rule) == PARAMETERS.get_child( + name + )(str(year)) + + +@pytest.mark.parametrize("year", range(2015, 2041)) +def test_growth_rate_defaults_to_gdp_per_head_in_every_year(year): + assert DEFAULT_SAVINGS_INCOME_PROJECTION == GDP_PER_HEAD + assert growth_rate(PARAMETERS, SAVINGS_INCOME_GROWTH, str(year)) == at( + YOY.obr.per_capita.gdp, year + ) + + +def test_growth_rate_refuses_an_unknown_rule(): + with pytest.raises(ValueError, match="unknown rule"): + growth_rate(PARAMETERS, SAVINGS_INCOME_GROWTH, "2024", "cpi") + + +def dataset(year, contract, people, rule=OBR_SAVINGS_INCOME_2022_2025): + """One household per person, each holding every savings input, declaring + ``rule`` (None declares nothing).""" + n = len(people) + person = pd.DataFrame( + { + "person_id": np.arange(1, n + 1), + "person_benunit_id": np.arange(1, n + 1), + "person_household_id": np.arange(1, n + 1), + "age": [45] * n, + "employment_income": [30_000.0] * n, + **{ + name: [float(amounts[name]) for amounts in people] + for name in ( + *SAVINGS_INPUTS, + "individual_savings_account_interest_income", + ) + }, + } + ) + return UKSingleYearDataset( + person=person, + benunit=pd.DataFrame({"benunit_id": np.arange(1, n + 1)}), + household=pd.DataFrame( + { + "household_id": np.arange(1, n + 1), + "region": ["LONDON"] * n, + "tenure_type": ["RENT_PRIVATELY"] * n, + "council_tax": [1_500.0] * n, + "rent": [12_000.0] * n, + "household_weight": [1_000.0] * n, + } + ), + fiscal_year=year, + isa_income_contract=deepcopy(contract), + savings_income_projection=rule, + ) + + +PEOPLE = [ + { + "savings_interest_income": 2_000, + "non_cash_isa_cash_interest": 100, + "other_investment_income": 300, + "individual_savings_account_interest_income": 500, + }, + { + "savings_interest_income": 40_000, + "non_cash_isa_cash_interest": 0, + "other_investment_income": 9_000, + "individual_savings_account_interest_income": 1_200, + }, +] + + +@pytest.mark.parametrize("contract", [LEGACY, ADDITIVE], ids=["legacy", "additive"]) +def test_projection_from_2023_24_reaches_the_obr_level_in_2025_26(contract): + """Weighted savings income in 2025-26 over 2023-24 is the OBR's growth. + + Weights grow by population and each person's income by the per-person + series, so the total grows by the OBR's aggregate up to the 4-decimal + rounding of the stored series. Under the legacy contract the stored + amount is gross of ISA interest, which is what grows. + """ + projected = extend_single_year_dataset( + dataset(2023, contract, PEOPLE), PARAMETERS, end_year=2027 + ) + obr = (1 + OBR_SAVINGS_INCOME_GROWTH[2024] / 100) * ( + 1 + OBR_SAVINGS_INCOME_GROWTH[2025] / 100 + ) + + def total(year, name): + data = projected.datasets[year] + weight = data.household.household_weight.values + return float((data.person[name].values * weight).sum()) + + for name in SAVINGS_INPUTS: + assert total(2025, name) / total(2023, name) == pytest.approx(obr, rel=2e-4) + if contract == ADDITIVE: + isa = "individual_savings_account_interest_income" + assert total(2025, isa) / total(2023, isa) == pytest.approx(obr, rel=2e-4) + # After 2025-26 the projection follows GDP per head, as before. + for year in (2026, 2027): + for name in SAVINGS_INPUTS: + before = projected.datasets[year - 1].person[name].values + after = projected.datasets[year].person[name].values + np.testing.assert_allclose( + after, before * (1 + at(YOY.obr.per_capita.gdp, year)), rtol=1e-12 + ) + + +@pytest.mark.parametrize("contract", [LEGACY, ADDITIVE], ids=["legacy", "additive"]) +def test_projection_before_2022_23_follows_gdp_per_head(contract): + """The series starts in 2022-23, so a 2020-21 dataset grows by GDP per + head into 2021-22, not by a backdated 2022-23 rate, then by the series.""" + gdp_2021 = at(YOY.obr.per_capita.gdp, 2021) + savings_2022 = at(YOY.obr.per_capita.savings_income, 2022) + assert gdp_2021 != pytest.approx(savings_2022) + projected = extend_single_year_dataset( + dataset(2020, contract, PEOPLE), PARAMETERS, end_year=2022 + ) + for year, growth in ((2021, gdp_2021), (2022, savings_2022)): + for name in SAVINGS_INPUTS: + before = projected.datasets[year - 1].person[name].values + after = projected.datasets[year].person[name].values + np.testing.assert_allclose(after, before * (1 + growth), rtol=1e-12) + + +@pytest.mark.parametrize("contract", [LEGACY, ADDITIVE], ids=["legacy", "additive"]) +def test_undeclared_data_grows_savings_income_by_gdp_per_head_every_year(contract): + """Data built before the declaration existed projects as it did before + the OBR series was added: by GDP per head in every year, 2023-24 to + 2025-26 included. Legacy ISA interest stays flat.""" + undeclared = dataset(2020, contract, PEOPLE, rule=None) + assert undeclared.savings_income_projection == GDP_PER_HEAD + projected = extend_single_year_dataset(undeclared, PARAMETERS, end_year=2027) + assert projected.savings_income_projection == GDP_PER_HEAD + names = list(SAVINGS_INPUTS) + if contract == ADDITIVE: + names.append("individual_savings_account_interest_income") + for year in range(2021, 2028): + growth = at(YOY.obr.per_capita.gdp, year) + for name in names: + before = projected.datasets[year - 1].person[name].values + after = projected.datasets[year].person[name].values + np.testing.assert_allclose(after, before * (1 + growth), rtol=1e-12) + + +@settings(max_examples=60, deadline=None) +@given( + year=st.integers(min_value=2015, max_value=2031), + rule=st.sampled_from([None, GDP_PER_HEAD, OBR_SAVINGS_INCOME_2022_2025]), + additive=st.booleans(), + amounts=st.lists( + st.tuples( + st.floats(min_value=0, max_value=5e6, allow_nan=False), + st.floats(min_value=0, max_value=5e5, allow_nan=False), + st.floats(min_value=0, max_value=5e6, allow_nan=False), + st.floats(min_value=0, max_value=5e4, allow_nan=False), + ), + min_size=1, + max_size=4, + ), +) +def test_each_year_grows_every_savings_input_by_the_same_rate( + year, rule, additive, amounts +): + """One year of projection multiplies each savings input by 1 + growth. + + The rate is growth_rate's for the year projected into under the declared + rule (GDP per head when none is declared), the same for the + YAML-mapped inputs and the code paths for gross legacy interest and + additive ISA interest (a differential check between them). Amounts stay + non-negative, and legacy ISA interest stays flat. + """ + contract = ADDITIVE if additive else LEGACY + people = [ + { + "savings_interest_income": gross + isa if not additive else gross, + "non_cash_isa_cash_interest": min(noncash, isa), + "other_investment_income": other, + "individual_savings_account_interest_income": isa, + } + for gross, isa, other, noncash in amounts + ] + previous = dataset(year, contract, people, rule=rule) + current = previous.copy() + current.time_period = str(year + 1) + apply_single_year_uprating(current, previous, PARAMETERS) + factor = 1 + growth_rate( + PARAMETERS, + SAVINGS_INCOME_GROWTH, + str(year + 1), + rule or DEFAULT_SAVINGS_INCOME_PROJECTION, + ) + names = list(SAVINGS_INPUTS) + if additive: + names.append("individual_savings_account_interest_income") + for name in names: + before = previous.person[name].values + after = current.person[name].values + np.testing.assert_allclose(after, before * factor, rtol=1e-12, atol=1e-9) + assert (after >= 0).all() + if not additive: + isa = "individual_savings_account_interest_income" + np.testing.assert_array_equal( + current.person[isa].values, previous.person[isa].values + ) + + +# The declaration travels with the data, and data that comes back from a file +# or a copy projects by it (assert_projects_by, below). + + +@pytest.mark.parametrize("rule", [GDP_PER_HEAD, OBR_SAVINGS_INCOME_2022_2025]) +def test_a_single_year_file_keeps_its_rule(tmp_path, rule): + path = tmp_path / "single.h5" + dataset(2023, ADDITIVE, PEOPLE, rule=rule).save(str(path)) + with h5py.File(path, "r") as file: + assert file.attrs[SAVINGS_INCOME_PROJECTION_KEY] == rule + assert read_savings_income_projection(path) == rule + assert_projects_by(UKSingleYearDataset(str(path)), rule) + + +def test_a_file_without_the_attribute_reads_as_gdp_per_head(tmp_path): + path = tmp_path / "old.h5" + dataset(2023, ADDITIVE, PEOPLE).save(str(path)) + with h5py.File(path, "a") as file: + del file.attrs[SAVINGS_INCOME_PROJECTION_KEY] + assert read_savings_income_projection(path) is None + assert UKSingleYearDataset(str(path)).savings_income_projection == GDP_PER_HEAD + write_savings_income_projection(path, OBR_SAVINGS_INCOME_2022_2025) + assert read_savings_income_projection(path) == OBR_SAVINGS_INCOME_2022_2025 + + +def test_a_declaration_that_contradicts_the_file_is_refused(tmp_path): + path = tmp_path / "obr.h5" + dataset(2023, ADDITIVE, PEOPLE).save(str(path)) + with pytest.raises(ValueError, match="disagree"): + UKSingleYearDataset(str(path), savings_income_projection=GDP_PER_HEAD) + # Repeating the file's own rule is not a contradiction. + assert ( + UKSingleYearDataset( + str(path), savings_income_projection=OBR_SAVINGS_INCOME_2022_2025 + ).savings_income_projection + == OBR_SAVINGS_INCOME_2022_2025 + ) + + +def test_an_unknown_rule_is_refused(tmp_path): + for rule in ("cpi", "", False, 0): + with pytest.raises(ValueError, match="unknown rule"): + dataset(2023, ADDITIVE, PEOPLE, rule=rule) + path = tmp_path / "bad.h5" + dataset(2023, ADDITIVE, PEOPLE).save(str(path)) + with h5py.File(path, "a") as file: + file.attrs[SAVINGS_INCOME_PROJECTION_KEY] = "cpi" + with pytest.raises(ValueError, match="unknown rule"): + UKSingleYearDataset(str(path)) + + +@pytest.mark.parametrize("rule", [GDP_PER_HEAD, OBR_SAVINGS_INCOME_2022_2025]) +def test_projection_copies_and_multi_year_files_keep_the_rule(tmp_path, rule): + projected = extend_single_year_dataset( + dataset(2023, ADDITIVE, PEOPLE, rule=rule), PARAMETERS, end_year=2026 + ) + assert {projected[year].savings_income_projection for year in projected.years} == { + rule + } + assert projected.copy().savings_income_projection == rule + assert_projects_by(projected.copy()[2023], rule) + path = tmp_path / "multi.h5" + projected.save(str(path)) + with h5py.File(path, "r") as file: + assert file.attrs[SAVINGS_INCOME_PROJECTION_KEY] == rule + reloaded = UKMultiYearDataset(str(path)) + assert {reloaded[year].savings_income_projection for year in reloaded.years} == { + rule + } + assert_projects_by(reloaded[2023], rule) + + +def test_a_multi_year_dataset_refuses_mixed_rules(): + first = dataset(2023, ADDITIVE, PEOPLE, rule=GDP_PER_HEAD) + second = dataset(2024, ADDITIVE, PEOPLE, rule=OBR_SAVINGS_INCOME_2022_2025) + with pytest.raises(ValueError, match="different rules"): + UKMultiYearDataset(datasets=[first, second]) + + +def test_core_datasets_declare_through_an_attribute_or_their_file(tmp_path): + assert ( + core_dataset_savings_income_projection( + SimpleNamespace(savings_income_projection=OBR_SAVINGS_INCOME_2022_2025) + ) + == OBR_SAVINGS_INCOME_2022_2025 + ) + assert core_dataset_savings_income_projection(SimpleNamespace()) == GDP_PER_HEAD + path = tmp_path / "core.h5" + dataset(2023, ADDITIVE, PEOPLE).save(str(path)) + assert ( + core_dataset_savings_income_projection(SimpleNamespace(file_path=str(path))) + == OBR_SAVINGS_INCOME_2022_2025 + ) + + +# Data that goes into a simulation and comes back out declares the rule it +# went in with, and projects by it. + + +def projected_factor(rule, start, end): + """What projection multiplies each person's savings income by from + ``start`` to ``end`` under ``rule``.""" + factor = 1.0 + for year in range(start + 1, end + 1): + factor *= 1 + growth_rate(PARAMETERS, SAVINGS_INCOME_GROWTH, str(year), rule) + return factor + + +def test_the_rules_project_2023_24_to_2025_26_differently(): + """So each round trip below tells a kept rule from a dropped one.""" + assert projected_factor(OBR_SAVINGS_INCOME_2022_2025, 2023, 2025) != pytest.approx( + projected_factor(GDP_PER_HEAD, 2023, 2025), rel=0.01 + ) + + +def assert_grows_by(projected, rule, start=2023, end=2025): + # Data that has been through a simulation holds float32 amounts. + for name in ("non_cash_isa_cash_interest", "other_investment_income"): + np.testing.assert_allclose( + projected[end].person[name].values, + projected[start].person[name].values * projected_factor(rule, start, end), + rtol=1e-6, + ) + + +def assert_projects_by(single_year, rule): + """A single-year dataset declares ``rule`` and projects by it.""" + assert single_year.savings_income_projection == rule + projected = extend_single_year_dataset(single_year, PARAMETERS, end_year=2025) + assert projected.savings_income_projection == rule + assert_grows_by(projected, rule, start=int(single_year.time_period)) + + +@pytest.mark.parametrize("rule", [None, OBR_SAVINGS_INCOME_2022_2025]) +def test_a_simulation_projects_by_and_exports_the_rule(rule): + """A simulation built from the data projects by its rule. Its input + frames declare the same rule, and datasets exported from it or from its + clones declare it and project by it.""" + sim = Microsimulation(dataset=dataset(2023, ADDITIVE, PEOPLE, rule=rule)) + expected = rule or GDP_PER_HEAD + assert sim._savings_income_projection == expected + assert_grows_by(sim.dataset, expected) + assert sim.to_input_dataframe().attrs[SAVINGS_INCOME_PROJECTION_KEY] == expected + for source in (sim, sim.clone()): + assert_projects_by( + UKSingleYearDataset.from_simulation(source, fiscal_year=2023), expected + ) + + +@pytest.mark.parametrize("rule", ["", False, 0, "cpi"]) +def test_an_invalid_rule_given_to_an_export_is_refused_not_ignored(rule): + sim = Microsimulation(dataset=dataset(2023, ADDITIVE, PEOPLE)) + with pytest.raises(ValueError, match="unknown rule"): + UKSingleYearDataset.from_simulation( + sim, fiscal_year=2023, savings_income_projection=rule + ) + + +@pytest.mark.parametrize("rule", [None, OBR_SAVINGS_INCOME_2022_2025]) +def test_an_input_frame_reloads_with_its_rule(rule): + exported = Microsimulation( + dataset=dataset(2023, ADDITIVE, PEOPLE, rule=rule) + ).to_input_dataframe() + reloaded = Microsimulation(dataset=exported) + expected = rule or GDP_PER_HEAD + assert reloaded._savings_income_projection == expected + assert_projects_by(reloaded.dataset[2023], expected) + + +@pytest.mark.parametrize("rule", [None, OBR_SAVINGS_INCOME_2022_2025]) +def test_a_filtered_household_keeps_its_rule(rule): + sim = Microsimulation(dataset=dataset(2023, ADDITIVE, PEOPLE, rule=rule)) + extract = filter_dataset(sim, household_id=1, year=2023) + expected = rule or GDP_PER_HEAD + assert_projects_by(extract, expected) + assert Microsimulation(dataset=extract)._savings_income_projection == expected + + +@pytest.mark.parametrize("rule", [None, OBR_SAVINGS_INCOME_2022_2025]) +def test_the_full_variable_export_keeps_the_rule(rule): + sim = Microsimulation(dataset=dataset(2023, ADDITIVE, PEOPLE, rule=rule)) + assert_projects_by(sim.calculate_all(2023), rule or GDP_PER_HEAD) + + +def core_arrays(source): + """A dataset's arrays as a policyengine-core Dataset holds them.""" + arrays = {} + for table in source.tables: + for name in table.columns: + values = table[name].to_numpy() + possible_values = getattr(system.variables[name], "possible_values", None) + if possible_values is not None: + values = np.asarray(possible_values.encode(values)) + arrays[name] = {source.time_period: values} + return arrays + + +class InMemoryCore(Dataset): + """A fixture-scale policyengine-core Dataset.""" + + def __init__(self, arrays): + self.arrays = arrays + + def load_dataset(self): + return self.arrays + + +@pytest.mark.parametrize("rule", [None, OBR_SAVINGS_INCOME_2022_2025]) +@pytest.mark.parametrize("source_kind", ["core-h5", "core"]) +def test_a_core_dataset_projects_by_its_rule(source_kind, rule, tmp_path): + """A Core-format file declares the rule in the same H5 root attribute, + and a Core Dataset object through a savings_income_projection + attribute.""" + arrays = core_arrays(dataset(2023, LEGACY, PEOPLE, rule=None)) + if source_kind == "core-h5": + source = tmp_path / "core.h5" + with h5py.File(source, "w") as file: + for name, periods in arrays.items(): + for period, values in periods.items(): + file.create_dataset(f"{name}/{period}", data=values) + if rule is not None: + write_savings_income_projection(source, rule) + source = str(source) + else: + source = InMemoryCore(arrays) + if rule is not None: + source.savings_income_projection = rule + sim = Microsimulation(dataset=source) + expected = rule or GDP_PER_HEAD + assert sim._savings_income_projection == expected + assert sim.dataset.savings_income_projection == expected + assert_grows_by(sim.dataset, expected)