diff --git a/changelog.d/triple-lock-statutory-inputs-compatibility.changed.md b/changelog.d/triple-lock-statutory-inputs-compatibility.changed.md new file mode 100644 index 0000000000..8bca1a60f4 --- /dev/null +++ b/changelog.d/triple-lock-statutory-inputs-compatibility.changed.md @@ -0,0 +1 @@ +Changed `yoy_growth.triple_lock` to use the previous year's September CPI and May-July AWE inputs rounded to 0.1 percentage points before taking the maximum of the included elements and configured floor, including CPI forecast gaps for observation years 2026–2030 and earnings forecast gaps for 2027–2030, with April 2027 determined by the provisional 3.9% earnings input; `triple_lock.outturn` is null from 2012, the generated uprating series runs through April 2074 and follows rounded lagged calendar-year earnings from April 2035 under the stored baseline, and basic and new State Pension levels are about 0.48% higher in 2027–2034 than under the previous baseline, with a growing gap thereafter and the 2027-onward changes provisional until the October 2026 labour market release. diff --git a/changelog.d/triple-lock-statutory-inputs.added.md b/changelog.d/triple-lock-statutory-inputs.added.md new file mode 100644 index 0000000000..49c80cd4cd --- /dev/null +++ b/changelog.d/triple-lock-statutory-inputs.added.md @@ -0,0 +1 @@ +Add an optional State Pension earnings-path guarantee (`gov.dwp.state_pension.triple_lock.earnings_path_guarantee`), so reforms that keep the pension in line with earnings over time, such as one reading of the plan announced in September 2026, can be modelled as parameter changes. Off under current law. diff --git a/changelog.d/triple-lock-statutory-inputs.changed.md b/changelog.d/triple-lock-statutory-inputs.changed.md new file mode 100644 index 0000000000..c3849ce615 --- /dev/null +++ b/changelog.d/triple-lock-statutory-inputs.changed.md @@ -0,0 +1 @@ +Computed the State Pension triple lock from September CPI and May-July AWE total pay growth used in each uprating review, followed by OBR September CPI and Q2 earnings forecasts and then calendar-year growth, with macro scenarios moving those forecasts and `active: false` leaving non-negative earnings growth; April 2027's 3.9% rise and projected £250.71 new State Pension weekly rate use the 15 September 2026 first earnings estimate and stay provisional until the October labour market release, the vintage the review uses. diff --git a/docs/book/assumptions/nowcasting-comparison.md b/docs/book/assumptions/nowcasting-comparison.md index 7b257cfad5..f86d67c466 100644 --- a/docs/book/assumptions/nowcasting-comparison.md +++ b/docs/book/assumptions/nowcasting-comparison.md @@ -48,7 +48,7 @@ earnings index. | Aspect | Resolution Foundation | PolicyEngine UK | |-------|------------------------|-----------------| | Working-age benefits | Uses statutory uprating with explicit overrides for announced policy (e.g. CoL Payments, benefit freezes). | Same approach. Parameters under `gov/dwp/` and `gov/hmrc/child_benefit/` track legislated rates; ad hoc payments live under `gov/treasury/cost_of_living_support`. | -| State Pension | Models the triple lock explicitly, using its own internal earnings/CPI forecasts. | Models the triple lock via `gov/dwp/state_pension/triple_lock/*.yaml`; the uprating value tracks announced DWP rates with a fallback to the maximum of earnings, CPI and the 2.5% floor. | +| State Pension | Models the triple lock explicitly, using its own internal earnings/CPI forecasts. | Models the triple lock from its statutory inputs (September CPI, May-July AWE total pay growth, 2.5% floor) in `gov/economic_assumptions/statutory_uprating_inputs/` and `gov/dwp/state_pension/triple_lock/`, forecast from the OBR's September CPI and Q2 earnings growth to the end of the EFO and calendar-year growth after; see the State Pension page. | ### Take-up diff --git a/docs/book/programs/gov/dwp/state-pension.md b/docs/book/programs/gov/dwp/state-pension.md index 24513d3d7c..e144b6809b 100644 --- a/docs/book/programs/gov/dwp/state-pension.md +++ b/docs/book/programs/gov/dwp/state-pension.md @@ -36,19 +36,152 @@ before or on/after 6 April 2016. State Pension flat-rate parameters live in `gov/dwp/state_pension/basic_state_pension/amount.yaml` and -`gov/dwp/state_pension/new_state_pension/amount.yaml`. Both are uprated -under the **triple lock**: the maximum of earnings growth, CPI -inflation, or the 2.5% floor parameterised at -`gov/dwp/state_pension/triple_lock/minimum_rate.yaml`. Active components -of the triple lock are controlled by: - -- `triple_lock/active.yaml` — top-level toggle. -- `triple_lock/include_earnings.yaml` — whether the earnings limb is - active. -- `triple_lock/include_inflation.yaml` — whether the CPI limb is active. - -These flags exist so that policy reforms can disable individual limbs -(e.g. "double lock" scenarios that drop the earnings or inflation limb). +`gov/dwp/state_pension/new_state_pension/amount.yaml`. Published rates run +to 2026-27; later years are uprated by +`gov.economic_assumptions.indices.triple_lock`, built from the yearly rates +in `gov.economic_assumptions.yoy_growth.triple_lock`. + +### The triple lock + +The rise each April is the highest of: + +- **earnings growth**: average weekly earnings, total pay, whole economy, in + May to July of the previous year on a year earlier (ONS KAC3); +- **CPI inflation**: the 12-month rate in September of the previous year + (ONS D7G7); +- **2.5%**: `triple_lock/minimum_rate.yaml`. + +`create_triple_lock.py` computes each element from these statutory inputs, +rounded to the one decimal place the ONS publishes. From them it reproduces +every published rise from April 2012 to April 2026. April 2011 is the one +override (`triple_lock/outturn.yaml`): the basic State Pension rose by +September 2010 RPI (4.6%) during the switch to CPI. The April 2022 +suspension of the earnings element is `include_earnings` set to false for +that year. + +The yearly rates run from April 2011 to one year past the end of the +economic-assumption series. + +### The statutory inputs + +`gov.economic_assumptions.statutory_uprating_inputs` holds: + +- `cpi_september`: September CPI, keyed to 1 September; +- `awe_total_pay_may_july`: May-July earnings growth, keyed to 1 July, as + used in the uprating review. + +Each holds published figures and then a null. From the null onwards, +`create_statutory_uprating_inputs.py` fills in a forecast: calendar-year +growth in the matching OBR series (`yoy_growth.obr.consumer_price_index` or +`yoy_growth.obr.average_earnings`) plus `forecast_gap`. The gap is the OBR's +statutory-basis forecast minus the calendar-year growth stored in +`yoy_growth.yaml`, so that in the baseline the input equals the OBR's figure: + +- September CPI: the OBR's September CPI forecast (receipts Table 3.19, the + CPI used to uprate tax thresholds), or Q3 CPI (economy Table 1.7) in years + that table does not cover; +- May-July earnings: Q2 average earnings growth (economy Table 1.6). The + OBR does not forecast the ONS AWE series; its measure is wages and + salaries per employee, and Q2 is the quarter nearest May to July. + +After the EFO horizon the gap is zero, so the inputs follow calendar-year +growth. Smooth forecasts pay the higher of earnings, CPI and 2.5% each year, +so the baseline has none of the extra cost the triple lock builds up when +September CPI and May-July earnings take turns to spike; the OBR's long-run +projections add 0.56 percentage points a year over earnings for it (Fiscal +risks and sustainability, July 2026). To capture it, supply simulated paths +of the statutory inputs, as below. + +`policyengine_uk/utils/import_obr_forecasts.py` regenerates the gaps with the +calendar-year series, from the EFO economy and receipts tables +(`--receipts-file` or `--receipts-url`). After editing `yoy_growth.yaml` by +hand, run it with `--gaps-only`. + +### Scenarios + +A macro scenario applied before the data load that edits calendar-year +growth moves the forecast inputs one for one, and so the triple lock. +Calendar-year series are keyed to 1 January, so key the change +`year:YYYY-01-01:1`: a bare year names the fiscal year from 6 April, which +lands in the next calendar-year value and so moves the following year's +inputs. + +```python +from policyengine_uk.model_api import Scenario + +scenario = Scenario( + parameter_changes={ + "gov.economic_assumptions.yoy_growth.obr.average_earnings": { + "year:2027-01-01:1": 0.05, + }, + }, + applied_before_data_load=True, +) +``` + +A scenario can also set the statutory inputs directly, for example from a +model of the monthly series, since the triple lock pays out on how September +CPI and May-July earnings differ from each other. A value replaces the +forecast for that year only; a bare year names the fiscal year from 6 April, +which contains both observation dates: + +```python +Scenario( + parameter_changes={ + "gov.economic_assumptions.statutory_uprating_inputs.cpi_september": { + "2027": 0.031, + }, + "gov.economic_assumptions.statutory_uprating_inputs.awe_total_pay_may_july": { + "2027": 0.024, + }, + }, + applied_before_data_load=True, +) +``` + +These changes, and the reform levers below, must go through +`Scenario(parameter_changes=...)`. The rates are built when parameters are +processed, and a `reform=` dictionary edits parameters after that, so it +changes the parameter but not the uprating. + +### Reform levers + +- `triple_lock/minimum_rate.yaml` sets the floor. A double lock, the higher + of earnings and CPI, is the floor set to 0 (or below 0 to allow cash + cuts). +- `triple_lock/include_earnings.yaml` and + `triple_lock/include_inflation.yaml` drop an element but keep the floor. + A CPI link is `include_earnings` false with the floor at or below 0. +- `triple_lock/active.yaml` switches the triple lock off. The pension then + rises by the statutory minimum from the review under the Social Security + Administration Act 1992, section 150A: earnings growth, and nothing when + earnings fall. The floor and the include flags no longer apply, and + neither do the one-year changes to section 150A for April 2021 and April + 2022. +- `triple_lock/earnings_path_guarantee.yaml` (off under current law) keeps + the pension on or above an earnings path started from its level in the + year before the guarantee first applies. With `include_earnings` false, + it models a plan that drops the earnings element of the triple lock but + keeps the pension in line with earnings over time. The plan announced on + 29 September 2026 gave no formula; one reading, from April 2030, is: + +```python +Scenario( + parameter_changes={ + "gov.dwp.state_pension.triple_lock.include_earnings": { + "year:2030-01-01:100": False, + }, + "gov.dwp.state_pension.triple_lock.earnings_path_guarantee": { + "year:2030-01-01:100": True, + }, + }, + applied_before_data_load=True, +) +``` + +Each guaranteed year, the rise is the highest of the triple lock elements +still included and the rise needed to reach the earnings path, rounded up to +0.1 percentage points. ## Known aggregate gap (#1632) diff --git a/policyengine_uk/parameters/gov/dwp/state_pension/basic_state_pension/amount.yaml b/policyengine_uk/parameters/gov/dwp/state_pension/basic_state_pension/amount.yaml index 5e88ff3f61..2fe6e3b7b1 100644 --- a/policyengine_uk/parameters/gov/dwp/state_pension/basic_state_pension/amount.yaml +++ b/policyengine_uk/parameters/gov/dwp/state_pension/basic_state_pension/amount.yaml @@ -29,6 +29,7 @@ metadata: unit: currency-GBP label: Basic State Pension amount uprating: gov.economic_assumptions.indices.triple_lock + documentation: Projected weekly State Pension rates are not rounded to 5p; SSAA 1992 s.150A(4) permits discretionary rounding up or down (https://www.legislation.gov.uk/ukpga/1992/5/section/150A/4). reference: - title: House of Commons Library - href: https://researchbriefings.files.parliament.uk/documents/SN05649/SN05649.pdf \ No newline at end of file + href: https://researchbriefings.files.parliament.uk/documents/SN05649/SN05649.pdf diff --git a/policyengine_uk/parameters/gov/dwp/state_pension/new_state_pension/amount.yaml b/policyengine_uk/parameters/gov/dwp/state_pension/new_state_pension/amount.yaml index 3dab192729..6271ec4868 100644 --- a/policyengine_uk/parameters/gov/dwp/state_pension/new_state_pension/amount.yaml +++ b/policyengine_uk/parameters/gov/dwp/state_pension/new_state_pension/amount.yaml @@ -15,6 +15,7 @@ metadata: unit: currency-GBP label: New State Pension amount uprating: gov.economic_assumptions.indices.triple_lock + documentation: Projected weekly State Pension rates are not rounded to 5p; SSAA 1992 s.150A(4) permits discretionary rounding up or down (https://www.legislation.gov.uk/ukpga/1992/5/section/150A/4). reference: - title: House of Commons Library - href: https://researchbriefings.files.parliament.uk/documents/SN05649/SN05649.pdf \ No newline at end of file + href: https://researchbriefings.files.parliament.uk/documents/SN05649/SN05649.pdf diff --git a/policyengine_uk/parameters/gov/dwp/state_pension/triple_lock/active.yaml b/policyengine_uk/parameters/gov/dwp/state_pension/triple_lock/active.yaml index cec29c6f10..cdf75f7d8f 100644 --- a/policyengine_uk/parameters/gov/dwp/state_pension/triple_lock/active.yaml +++ b/policyengine_uk/parameters/gov/dwp/state_pension/triple_lock/active.yaml @@ -1,6 +1,10 @@ -description: The triple lock is active if this value is true. +description: Whether the triple lock applies to the April uprating of the State Pension; without it, the pension rises by the statutory minimum, earnings growth, and not at all when earnings fall. values: 2010-01-01: true metadata: + documentation: Changes take effect through Scenario(parameter_changes=...), which rebuilds the uprating rates; a reform dictionary changes this parameter but not the rates. unit: bool - label: Triple lock \ No newline at end of file + label: Triple lock + reference: + - title: Social Security Administration Act 1992, section 150A + href: https://www.legislation.gov.uk/ukpga/1992/5/section/150A diff --git a/policyengine_uk/parameters/gov/dwp/state_pension/triple_lock/create_triple_lock.py b/policyengine_uk/parameters/gov/dwp/state_pension/triple_lock/create_triple_lock.py index fc884db555..b977aca217 100644 --- a/policyengine_uk/parameters/gov/dwp/state_pension/triple_lock/create_triple_lock.py +++ b/policyengine_uk/parameters/gov/dwp/state_pension/triple_lock/create_triple_lock.py @@ -1,47 +1,219 @@ -from policyengine_core.parameters import ( - ParameterNode, - Parameter, - get_parameter, +"""Build the State Pension uprating series from the statutory inputs. + +Each April the basic and new State Pension rise by the highest of: + +- earnings growth: average weekly earnings, total pay, whole economy, the + three months May to July of the previous year on a year earlier; +- CPI inflation: the 12-month rate in September of the previous year; +- the minimum rate (2.5%). + +The inputs live in ``gov.economic_assumptions.statutory_uprating_inputs``, +which ``create_statutory_uprating_inputs`` completes with forecasts before +this module runs. Each input is rounded to the one decimal place the ONS +publishes, as the uprating review uses the published figure. + +With ``active`` false there is no triple lock, only the statutory minimum +from the review under Social Security Administration Act 1992 s150A as it +stands: the pension rises by earnings growth, and not at all when earnings +fall. The one-year modifications of s150A for April 2021 and April 2022 are +not applied, and ``include_earnings``, ``include_inflation`` and +``minimum_rate`` are ignored. + +Years with a published rate that the rule cannot reproduce are overridden in +``triple_lock/outturn.yaml``. Only April 2011, when the basic State Pension +rose by RPI during the switch to CPI, needs one; the April 2022 suspension of +the earnings element is expressed through ``include_earnings``. + +``earnings_path_guarantee`` adds an optional floor for modelling reforms: in +each year it applies, the pension also rises at least enough to stay on an +earnings path started from its level in the year before the guarantee first +applied. Current law leaves it off. + +The output parameter ``gov.economic_assumptions.yoy_growth.triple_lock`` holds +the rate taking effect in April of each year, keyed to 1 January as the other +growth series are, and runs to one year past the last year of the +economic-assumption series. +""" + +from dataclasses import dataclass +from decimal import ROUND_CEILING, ROUND_HALF_UP, Decimal +from typing import Dict, Iterable, Optional + +from policyengine_core.parameters import Parameter, ParameterNode + +from policyengine_uk.parameters.gov.economic_assumptions.create_statutory_uprating_inputs import ( + AWE_OBSERVATION_MONTH_DAY, + CPI_OBSERVATION_MONTH_DAY, + last_input_year, ) -YEARS = list(range(2022, 2035)) +FIRST_UPRATING_YEAR = 2011 +# Published statistics are quoted to one decimal place of a percentage. +PUBLISHED_PRECISION = Decimal("0.001") +# Guards the round-up of a guaranteed minimum against floating-point noise, +# so a ratio of 1.025000000000001 is not taken up to 2.6%. +RATIO_PRECISION = Decimal("0.000000001") -def add_triple_lock(parameters: ParameterNode) -> ParameterNode: - obr = parameters.gov.economic_assumptions.yoy_growth.obr - average_earnings = obr.average_earnings - cpi = obr.consumer_price_index - triple_lock = parameters.gov.dwp.state_pension.triple_lock - min_rate = triple_lock.minimum_rate - # Years with a published outturn or DWP-announced rate to use directly - # instead of the formula (the formula relies on OBR calendar-year averages, - # which don't match the May-July AWE window DWP actually uses). - outturn_years = {int(v.instant_str[:4]) for v in triple_lock.outturn.values_list} +def uprating_instant(year: int) -> str: + """Date at which policy parameters are read for the April ``year`` uprating. + + 30 April is the date fiscal-year conversion samples. A YAML value from + 1 January, or a parameter change keyed to the bare year (the fiscal year + from 6 April) or to ``year:YYYY-01-01:N``, takes effect in the uprating + for that April. A change keyed to a single day covers only that day. + """ + return f"{year}-04-30" + + +def round_to_published_precision(rate: float) -> float: + """Round a growth rate to 0.1 percentage points, halves away from zero.""" + return float( + Decimal(repr(float(rate))).quantize(PUBLISHED_PRECISION, ROUND_HALF_UP) + ) + + +def round_up_to_published_precision(rate: float) -> float: + """Round a guaranteed minimum rate up to the next 0.1 percentage points.""" + cleaned = Decimal(repr(float(rate))).quantize(RATIO_PRECISION, ROUND_HALF_UP) + return float(cleaned.quantize(PUBLISHED_PRECISION, ROUND_CEILING)) + - values = {} +@dataclass(frozen=True) +class UpratingYear: + """Inputs to the April uprating in one year.""" - for year in YEARS: - if year in outturn_years: - triple_lock_increase = triple_lock.outturn(year) + earnings: float + cpi: float + minimum_rate: float + active: bool = True + include_earnings: bool = True + include_inflation: bool = True + earnings_path_guarantee: bool = False + outturn: Optional[float] = None + + +def triple_lock_rate( + earnings: float, + cpi: float, + minimum_rate: float, + include_earnings: bool = True, + include_inflation: bool = True, +) -> float: + """The highest of the included elements and the minimum rate.""" + candidates = [minimum_rate] + if include_earnings: + candidates.append(earnings) + if include_inflation: + candidates.append(cpi) + return max(candidates) + + +def rule_rate(inputs: UpratingYear) -> float: + """The year's rate before any earnings-path top-up or override.""" + if not inputs.active: + # s150A: at least the rise in earnings; no rise when earnings fall. + return max(inputs.earnings, 0.0) + return triple_lock_rate( + inputs.earnings, + inputs.cpi, + inputs.minimum_rate, + inputs.include_earnings, + inputs.include_inflation, + ) + + +def uprating_rates(years: Dict[int, UpratingYear]) -> Dict[int, float]: + """Uprating rate for each year, in year order. + + The earnings-path guarantee makes the rule path dependent. When it first + applies, it anchors an earnings path at the pension's level in the year + before; each guaranteed year the path grows by that year's earnings input, + and the rate is raised, rounding up, to keep the pension on or above it. + The path is dropped in any year the guarantee is off and re-anchored if it + returns. + """ + rates = {} + level = 1.0 + earnings_path = None + for year in sorted(years): + inputs = years[year] + rate = inputs.outturn if inputs.outturn is not None else rule_rate(inputs) + if inputs.earnings_path_guarantee: + if earnings_path is None: + earnings_path = level + earnings_path *= 1 + inputs.earnings + if inputs.outturn is None: + rate = max( + rate, + round_up_to_published_precision(earnings_path / level - 1), + ) else: - earnings_increase = average_earnings(year - 1) - cpi_increase = cpi(year - 1) - min_rate_y = min_rate(year) - if triple_lock.include_earnings(year) and triple_lock.include_inflation( - year - ): - triple_lock_increase = max(earnings_increase, cpi_increase, min_rate_y) - elif triple_lock.include_earnings(year): - triple_lock_increase = max(earnings_increase, min_rate_y) - elif triple_lock.include_inflation(year): - triple_lock_increase = max(cpi_increase, min_rate_y) - else: - triple_lock_increase = min_rate_y - values[f"{year}-01-01"] = round(triple_lock_increase, 3) + earnings_path = None + rates[year] = rate + level *= 1 + rate + return rates + +def _flag(parameter: Parameter, instant: str) -> bool: + value = parameter(instant) + return bool(value) if value is not None else False + + +def uprating_years(parameters: ParameterNode) -> Iterable[int]: + """Uprating years covered: from April 2011 to one year past the inputs.""" + return range(FIRST_UPRATING_YEAR, last_input_year(parameters) + 2) + + +def read_uprating_years(parameters: ParameterNode) -> Dict[int, UpratingYear]: + """Collect each year's rule inputs from a parameter tree.""" + inputs = parameters.gov.economic_assumptions.statutory_uprating_inputs + triple_lock = parameters.gov.dwp.state_pension.triple_lock + + years = {} + for year in uprating_years(parameters): + review_year = year - 1 + instant = uprating_instant(year) + earnings = inputs.awe_total_pay_may_july( + f"{review_year}-{AWE_OBSERVATION_MONTH_DAY}" + ) + cpi = inputs.cpi_september(f"{review_year}-{CPI_OBSERVATION_MONTH_DAY}") + # Null outside the overridden years. + outturn = triple_lock.outturn(instant) + if outturn is None and (earnings is None or cpi is None): + raise ValueError( + f"Missing statutory uprating input for April {year}: " + f"May-July {review_year} earnings = {earnings}, " + f"September {review_year} CPI = {cpi}." + ) + years[year] = UpratingYear( + earnings=round_to_published_precision(earnings or 0), + cpi=round_to_published_precision(cpi or 0), + minimum_rate=triple_lock.minimum_rate(instant), + active=_flag(triple_lock.active, instant), + include_earnings=_flag(triple_lock.include_earnings, instant), + include_inflation=_flag(triple_lock.include_inflation, instant), + earnings_path_guarantee=_flag(triple_lock.earnings_path_guarantee, instant), + outturn=outturn, + ) + return years + + +def add_triple_lock(parameters: ParameterNode) -> ParameterNode: + """Add ``gov.economic_assumptions.yoy_growth.triple_lock``.""" + rates = uprating_rates(read_uprating_years(parameters)) new_parameter = Parameter( "gov.economic_assumptions.yoy_growth.triple_lock", - data={"values": values, "metadata": {"unit": "/1"}}, + data={ + "description": ( + "State Pension uprating taking effect in April of each year." + ), + "values": {f"{year}-01-01": rate for year, rate in rates.items()}, + "metadata": { + "unit": "/1", + "label": "State Pension uprating rate", + }, + }, ) parameters.gov.economic_assumptions.yoy_growth.add_child( "triple_lock", new_parameter diff --git a/policyengine_uk/parameters/gov/dwp/state_pension/triple_lock/earnings_path_guarantee.yaml b/policyengine_uk/parameters/gov/dwp/state_pension/triple_lock/earnings_path_guarantee.yaml new file mode 100644 index 0000000000..ce8546216c --- /dev/null +++ b/policyengine_uk/parameters/gov/dwp/state_pension/triple_lock/earnings_path_guarantee.yaml @@ -0,0 +1,23 @@ +description: Whether the State Pension also rises at least enough to stay on an earnings path, started from its level in the year before this guarantee first applies and uprated each year by the May-July earnings input. +values: + 2010-01-01: false +metadata: + unit: bool + label: State Pension earnings path guarantee + documentation: | + Not part of current law. It lets reforms that replace the triple lock + with a lower annual floor, but promise that the pension keeps pace with + earnings over time, be modelled as parameter changes. The plan announced + on 29 September 2026 drops the annual earnings link from April 2030 but + says the pension will hold its value relative to earnings over time. The + speech gave no formula; one reading (rises of at least CPI or 2.5%, and + the pension never below an earnings link from its 2029-30 level) is + include_earnings false and this guarantee true from 2030. + Each guaranteed year, the rise is the higher of the triple lock + elements still included and the rise needed to reach the earnings path, + rounded up to 0.1 percentage points. Changes take effect through + Scenario(parameter_changes=...), which rebuilds the uprating rates; a + reform dictionary changes this parameter but not the rates. + reference: + - title: BBC News live, Labour conference speech, 29 September 2026 + href: https://www.bbc.co.uk/news/live/c6x2zrv774gvt diff --git a/policyengine_uk/parameters/gov/dwp/state_pension/triple_lock/include_earnings.yaml b/policyengine_uk/parameters/gov/dwp/state_pension/triple_lock/include_earnings.yaml index f8f0e838cd..b66a2fe614 100644 --- a/policyengine_uk/parameters/gov/dwp/state_pension/triple_lock/include_earnings.yaml +++ b/policyengine_uk/parameters/gov/dwp/state_pension/triple_lock/include_earnings.yaml @@ -1,8 +1,15 @@ description: Whether the triple lock should include average earnings. values: - 2012-01-01: true + 2011-01-01: true + # April 2022: the review for 2022-23 used prices instead of earnings, with + # a 2.5% minimum, under the Social Security (Up-rating of Benefits) Act + # 2021. 2022-01-01: false 2023-01-01: true metadata: + documentation: Changes take effect through Scenario(parameter_changes=...), which rebuilds the uprating rates; a reform dictionary changes this parameter but not the rates. unit: bool - label: Include average earnings in triple lock \ No newline at end of file + label: Include average earnings in triple lock + reference: + - title: Social Security (Up-rating of Benefits) Act 2021, section 1 + href: https://www.legislation.gov.uk/ukpga/2021/32/section/1/enacted diff --git a/policyengine_uk/parameters/gov/dwp/state_pension/triple_lock/include_inflation.yaml b/policyengine_uk/parameters/gov/dwp/state_pension/triple_lock/include_inflation.yaml index 84a2d176e9..ed5bd0b566 100644 --- a/policyengine_uk/parameters/gov/dwp/state_pension/triple_lock/include_inflation.yaml +++ b/policyengine_uk/parameters/gov/dwp/state_pension/triple_lock/include_inflation.yaml @@ -1,6 +1,7 @@ description: Whether the triple lock should include inflation. values: - 2012-01-01: true + 2011-01-01: true metadata: + documentation: Changes take effect through Scenario(parameter_changes=...), which rebuilds the uprating rates; a reform dictionary changes this parameter but not the rates. unit: bool label: Include inflation in triple lock \ No newline at end of file diff --git a/policyengine_uk/parameters/gov/dwp/state_pension/triple_lock/minimum_rate.yaml b/policyengine_uk/parameters/gov/dwp/state_pension/triple_lock/minimum_rate.yaml index e9e4db4ffa..b89790d9ef 100644 --- a/policyengine_uk/parameters/gov/dwp/state_pension/triple_lock/minimum_rate.yaml +++ b/policyengine_uk/parameters/gov/dwp/state_pension/triple_lock/minimum_rate.yaml @@ -12,3 +12,5 @@ metadata: The triple lock is a policy commitment introduced by the Coalition Government in 2010, not enshrined in primary legislation. The 2.5% minimum is one of three components ensuring pensions increase by the highest of earnings, inflation, or 2.5%. + Changes take effect through Scenario(parameter_changes=...), which rebuilds the + uprating rates; a reform dictionary changes this parameter but not the rates. diff --git a/policyengine_uk/parameters/gov/dwp/state_pension/triple_lock/outturn.yaml b/policyengine_uk/parameters/gov/dwp/state_pension/triple_lock/outturn.yaml index 315a55e5dc..ef7e13aaed 100644 --- a/policyengine_uk/parameters/gov/dwp/state_pension/triple_lock/outturn.yaml +++ b/policyengine_uk/parameters/gov/dwp/state_pension/triple_lock/outturn.yaml @@ -1,21 +1,14 @@ -description: Published triple lock uprating rates (outturn or DWP-announced). Used in preference to the max-of(earnings, CPI, minimum) formula because the formula relies on calendar-year OBR averages, while the actual uprating uses May-July AWE growth. +description: Published State Pension upratings that the rule cannot reproduce from the statutory inputs, used in place of the rule in those years. values: - 2022-01-01: 0.031 - 2023-01-01: 0.101 - 2024-01-01: 0.085 - 2025-01-01: 0.041 - 2026-01-01: 0.048 + # April 2011: the basic State Pension rose by September 2010 RPI (4.6%), + # above CPI (3.1%) and earnings (1.3%), a one-off guarantee during the + # switch from RPI to CPI. + 2011-01-01: 0.046 + # No other year departs from the rule. + 2012-01-01: null metadata: unit: /1 label: Triple lock outturn rate reference: - - title: DWP benefit and pension rates 2022 to 2023 - href: https://www.gov.uk/government/publications/benefit-and-pension-rates-2022-to-2023 - - title: DWP benefit and pension rates 2023 to 2024 - href: https://www.gov.uk/government/publications/benefit-and-pension-rates-2023-to-2024 - - title: DWP benefit and pension rates 2024 to 2025 - href: https://www.gov.uk/government/publications/benefit-and-pension-rates-2024-to-2025 - - title: DWP benefit and pension rates 2025 to 2026 - href: https://www.gov.uk/government/publications/benefit-and-pension-rates-2025-to-2026 - - title: DWP benefit and pension rates 2026 to 2027 - href: https://www.gov.uk/government/news/over-12-million-pensioners-to-receive-575-state-pension-boost + - title: Explanatory Memorandum to the Social Security Benefits Up-rating Order 2011 (SI 2011/821), paragraph 7.4 + href: https://www.legislation.gov.uk/uksi/2011/821/contents/made diff --git a/policyengine_uk/parameters/gov/economic_assumptions/README.md b/policyengine_uk/parameters/gov/economic_assumptions/README.md index 5418feddc2..74a070a776 100644 --- a/policyengine_uk/parameters/gov/economic_assumptions/README.md +++ b/policyengine_uk/parameters/gov/economic_assumptions/README.md @@ -58,6 +58,29 @@ new Fiscal Risks and Sustainability report. The convergence path itself is conservative interpolation and is not load-bearing for short-horizon analysis. +## Statutory uprating inputs + +[`statutory_uprating_inputs/`](./statutory_uprating_inputs/) holds the two +published statistics that set the April uprating of the State Pension: +September CPI (ONS D7G7) and May-July average weekly earnings total pay +growth (ONS KAC3). They are keyed to their observation month, not to +1 January, and carry `preserve_calendar_dates`. + +Each series holds published figures and then a null. From there, +[`create_statutory_uprating_inputs.py`](./create_statutory_uprating_inputs.py) +fills in calendar-year growth from `yoy_growth.obr` plus `forecast_gap`, the +OBR's statutory-basis forecast minus the calendar-year growth stored here, so +the baseline input equals the OBR's figure. +The gap is zero after the EFO horizon, so the forecasts fall back to +calendar-year growth. A scenario that edits calendar-year growth therefore +moves them, and a scenario can also set them directly. + +After each release, add the new published figure in place of the null and +move the null one year on: September CPI in October, May-July earnings in +September. Then replace May-July earnings with the October release's figure, +which the review uses: the April 2025 and April 2026 rises used October's +4.1% and 4.8%, where September's first estimates were 4.0% and 4.7%. + ## Refreshing after a new EFO 1. Replace the 2025-2030 block in each series with values from the new EFO @@ -68,5 +91,10 @@ analysis. to regenerate the cumulative `indices/` parameters that uprating depends on. 4. Update the EFO reference in each series' `metadata.reference`. +5. Regenerate `statutory_uprating_inputs/forecast_gap/` from the same EFO. + `policyengine_uk/utils/import_obr_forecasts.py` does this when it updates + `yoy_growth.yaml`; pass the receipts tables with `--receipts-file` or + `--receipts-url` so September CPI uses the OBR's September forecast. If + you edit `yoy_growth.yaml` by hand instead, rerun it with `--gaps-only`. [rpi-cpi]: https://obr.uk/box/the-long-run-difference-between-rpi-and-cpi-inflation/ diff --git a/policyengine_uk/parameters/gov/economic_assumptions/create_statutory_uprating_inputs.py b/policyengine_uk/parameters/gov/economic_assumptions/create_statutory_uprating_inputs.py new file mode 100644 index 0000000000..fe1fdac8c7 --- /dev/null +++ b/policyengine_uk/parameters/gov/economic_assumptions/create_statutory_uprating_inputs.py @@ -0,0 +1,80 @@ +"""Complete the statutory uprating inputs with forecasts. + +Two published statistics set the April uprating of the State Pension: + +- ``cpi_september``: the CPI 12-month rate in September (ONS D7G7). +- ``awe_total_pay_may_july``: average weekly earnings, total pay, whole + economy, the three months May to July on a year earlier (ONS KAC3), as + used in each year's uprating review: the October labour market release, + with the latest year holding September's first estimate until then. + +Each is keyed to its observation month (1 September, 1 July) and carries +``preserve_calendar_dates`` so fiscal-year conversion leaves the dates alone. +The YAML holds the published figures and then a null: from there on, values +are forecasts. + +For a year without a published figure, the forecast is calendar-year growth +in the matching series of ``gov.economic_assumptions.yoy_growth.obr`` plus +``forecast_gap``: the OBR's statutory-basis forecast (its September CPI, or +the quarter nearest the statutory period) minus the stored calendar-year +growth, so the baseline reproduces the OBR's figure. The gap is zero after +the EFO horizon, so the forecast falls back to calendar-year growth. A macro +scenario that edits calendar-year growth moves the inputs one for one; a +scenario that sets an input directly replaces the forecast for the years it +covers. +""" + +from policyengine_core.parameters import Parameter, ParameterNode + +CPI_OBSERVATION_MONTH_DAY = "09-01" +AWE_OBSERVATION_MONTH_DAY = "07-01" + +# Statutory input: (calendar-year series in yoy_growth.obr, observation date). +STATUTORY_INPUTS = { + "cpi_september": ("consumer_price_index", CPI_OBSERVATION_MONTH_DAY), + "awe_total_pay_may_july": ("average_earnings", AWE_OBSERVATION_MONTH_DAY), +} + + +def _last_year(parameter: Parameter) -> int: + return max( + int(value.instant_str[:4]) + for value in parameter.values_list + if value.value is not None + ) + + +def _first_year(parameter: Parameter) -> int: + return min(int(value.instant_str[:4]) for value in parameter.values_list) + + +def last_input_year(parameters: ParameterNode) -> int: + """Last year covered by the calendar-year series behind the inputs.""" + obr = parameters.gov.economic_assumptions.yoy_growth.obr + return min( + _last_year(getattr(obr, series)) for series, _ in STATUTORY_INPUTS.values() + ) + + +def add_statutory_uprating_inputs(parameters: ParameterNode) -> ParameterNode: + """Fill each input's unpublished years with its forecast. + + Runs after any scenario has edited the raw parameters and before the + uprating rules read the inputs. + """ + inputs = parameters.gov.economic_assumptions.statutory_uprating_inputs + obr = parameters.gov.economic_assumptions.yoy_growth.obr + end_year = last_input_year(parameters) + for name, (series, month_day) in STATUTORY_INPUTS.items(): + parameter = getattr(inputs, name) + gap = getattr(inputs.forecast_gap, name) + calendar_growth = getattr(obr, series) + for year in range(_first_year(parameter), end_year + 1): + observed = f"{year}-{month_day}" + if parameter(observed) is not None: + continue + forecast = float(calendar_growth(f"{year}-01-01")) + float( + gap(observed) or 0 + ) + parameter.update(period=f"year:{observed}:1", value=forecast) + return parameters diff --git a/policyengine_uk/parameters/gov/economic_assumptions/statutory_uprating_inputs/awe_total_pay_may_july.yaml b/policyengine_uk/parameters/gov/economic_assumptions/statutory_uprating_inputs/awe_total_pay_may_july.yaml new file mode 100644 index 0000000000..8da101a37e --- /dev/null +++ b/policyengine_uk/parameters/gov/economic_assumptions/statutory_uprating_inputs/awe_total_pay_may_july.yaml @@ -0,0 +1,50 @@ +description: Growth in average weekly earnings (total pay, whole economy) in May to July on a year earlier, the figure the Secretary of State uses in the review that sets the April uprating of the State Pension the following year. +values: + # Outturn: the figure each year's review used, as stated in the + # Explanatory Memorandum to the following April's Social Security Benefits + # Up-rating Order (for 2021, the Government Actuary's report on the 2022 + # Order). Reviews use the latest estimate before the autumn uprating + # statement: the October labour market release's figure, unchanged in + # November in every year checked, which the explanatory notes to the 2021 + # Up-rating of Benefits Act call the final figure. It can differ from the + # September first estimate (May-July 2024: 4.0% in September, 4.1% in + # October and used) and from the latest ONS vintage of KAC3 (4.4% for + # May-July 2024). + 2010-07-01: 0.013 # SI 2011/821 EM, footnote 7 + 2011-07-01: 0.028 # SI 2012/780 EM + 2012-07-01: 0.016 # SI 2013/574 EM + 2013-07-01: 0.012 # SI 2014/516 EM + 2014-07-01: 0.006 # SI 2015/457 EM + 2015-07-01: 0.029 # SI 2016/230 EM + 2016-07-01: 0.024 # SI 2017/260 EM + 2017-07-01: 0.022 # SI 2018/281 EM + 2018-07-01: 0.026 # SI 2019/480 EM + 2019-07-01: 0.039 # SI 2020/234 EM + 2020-07-01: -0.010 # SI 2021/162 EM + 2021-07-01: 0.083 # GAD report on the 2022 Order (the earnings element was suspended) + 2022-07-01: 0.055 # SI 2023/316 EM + 2023-07-01: 0.085 # SI 2024/242 EM + 2024-07-01: 0.041 # SI 2025/295 EM + 2025-07-01: 0.048 # SI 2026/148 EM + # Provisional: 3.9% is the 15 September 2026 first estimate and stays + # provisional until the October labour market release (due 20 October), + # the vintage the uprating review uses; replace it once published. + 2026-07-01: 0.039 + # Not yet published. From here the value is calendar-year earnings growth + # (yoy_growth.obr.average_earnings) plus forecast_gap.awe_total_pay_may_july. + 2027-07-01: null +metadata: + unit: /1 + label: May-July earnings growth + preserve_calendar_dates: true + reference: + - title: ONS KAC3, AWE whole economy total pay, three-month average growth on a year earlier + href: https://www.ons.gov.uk/employmentandlabourmarket/peopleinwork/earningsandworkinghours/timeseries/kac3/lms + - title: Explanatory Memorandum to the Social Security Benefits Up-rating Order 2026 (SI 2026/148) + href: https://www.legislation.gov.uk/uksi/2026/148/contents/made + - title: Explanatory Memorandum to the Social Security Benefits Up-rating Order 2025 (SI 2025/295) + href: https://www.legislation.gov.uk/uksi/2025/295/contents/made + - title: Explanatory Notes to the Social Security (Up-rating of Benefits) Act 2021, paragraph 4 + href: https://www.legislation.gov.uk/ukpga/2021/32/pdfs/ukpgaen_20210032_en.pdf + - title: Government Actuary's Department, report on the Social Security Benefits Up-rating Order 2022 + href: https://assets.publishing.service.gov.uk/media/61e59a0a8fa8f5058bc04a51/E02705826_Un_Act_GAD_NIF_Uprating_Report_Jan22_Web_Accessible_-_UPLOADED_GOV_UK.pdf diff --git a/policyengine_uk/parameters/gov/economic_assumptions/statutory_uprating_inputs/cpi_september.yaml b/policyengine_uk/parameters/gov/economic_assumptions/statutory_uprating_inputs/cpi_september.yaml new file mode 100644 index 0000000000..9925e29396 --- /dev/null +++ b/policyengine_uk/parameters/gov/economic_assumptions/statutory_uprating_inputs/cpi_september.yaml @@ -0,0 +1,34 @@ +description: CPI 12-month rate in September, the price input to the April uprating of the State Pension and most benefits the following year. +values: + # Outturn: ONS D7G7, as published. ONS does not amend published CPI + # (its last revisions were in 2000 and 2006), so these are the figures + # each review used. Store the published rate: rates derived from the + # 1dp index (D7BT) round differently in some years. + 2010-09-01: 0.031 + 2011-09-01: 0.052 + 2012-09-01: 0.022 + 2013-09-01: 0.027 + 2014-09-01: 0.012 + 2015-09-01: -0.001 + 2016-09-01: 0.010 + 2017-09-01: 0.030 + 2018-09-01: 0.024 + 2019-09-01: 0.017 + 2020-09-01: 0.005 + 2021-09-01: 0.031 + 2022-09-01: 0.101 + 2023-09-01: 0.067 + 2024-09-01: 0.017 + 2025-09-01: 0.038 + # Not yet published. From here the value is calendar-year CPI growth + # (yoy_growth.obr.consumer_price_index) plus forecast_gap.cpi_september. + 2026-09-01: null +metadata: + unit: /1 + label: September CPI inflation + preserve_calendar_dates: true + reference: + - title: ONS D7G7, CPI annual rate, all items + href: https://www.ons.gov.uk/economy/inflationandpriceindices/timeseries/d7g7/mm23 + - title: ONS, Revisions and correction of errors policies for consumer price inflation statistics (December 2025) + href: https://www.ons.gov.uk/economy/inflationandpriceindices/articles/revisionsandcorrectionoferrorspoliciesforconsumerpriceinflationstatistics/december2025 diff --git a/policyengine_uk/parameters/gov/economic_assumptions/statutory_uprating_inputs/forecast_gap/awe_total_pay_may_july.yaml b/policyengine_uk/parameters/gov/economic_assumptions/statutory_uprating_inputs/forecast_gap/awe_total_pay_may_july.yaml new file mode 100644 index 0000000000..126d1ffcaf --- /dev/null +++ b/policyengine_uk/parameters/gov/economic_assumptions/statutory_uprating_inputs/forecast_gap/awe_total_pay_may_july.yaml @@ -0,0 +1,32 @@ +description: Forecast difference between May-July earnings growth and calendar-year earnings growth, added to calendar-year growth in years without a published May-July figure. +values: + # No gap applies to published years. + 2010-07-01: 0 + # OBR EFO March 2026: Q2 average earnings growth on a year earlier (economy Table 1.6) minus calendar-year growth in yoy_growth.yaml. + 2025-07-01: 0.00259 + 2026-07-01: 0.00274 + 2027-07-01: 0.00027 + 2028-07-01: -0.00021 + 2029-07-01: -0.00015 + 2030-07-01: -0.00011 + # After the EFO horizon: no gap, so the forecast is calendar-year growth. + 2031-07-01: 0 +metadata: + unit: /1 + label: May-July earnings forecast gap to calendar-year earnings + preserve_calendar_dates: true + documentation: | + The OBR does not forecast the ONS AWE series. Its average earnings + measure is wages and salaries per employee from the national accounts, + forecast by quarter; the gap is its Q2 (April to June) growth, the + quarter nearest May to July, minus the calendar-year growth stored in + yoy_growth.yaml, so that stored growth plus the gap is the OBR's Q2 + figure. In the EFOs from November 2023 to March 2026, every OBR + triple-lock forecast equals the highest of this Q2 growth, Q3 CPI and + 2.5%. policyengine_uk/utils/import_obr_forecasts.py regenerates the gaps + with each EFO (or with --gaps-only after editing yoy_growth.yaml). The + gap is zero after the EFO horizon, where the May-July forecast falls back + to calendar-year growth. + reference: + - title: OBR EFO March 2026 (detailed forecast tables, economy, Table 1.6) + href: https://obr.uk/efo/economic-and-fiscal-outlook-march-2026/ diff --git a/policyengine_uk/parameters/gov/economic_assumptions/statutory_uprating_inputs/forecast_gap/cpi_september.yaml b/policyengine_uk/parameters/gov/economic_assumptions/statutory_uprating_inputs/forecast_gap/cpi_september.yaml new file mode 100644 index 0000000000..37f30d0198 --- /dev/null +++ b/policyengine_uk/parameters/gov/economic_assumptions/statutory_uprating_inputs/forecast_gap/cpi_september.yaml @@ -0,0 +1,33 @@ +description: Forecast difference between September CPI inflation and calendar-year CPI inflation, added to calendar-year growth in years without a published September figure. +values: + # No gap applies to published years. + 2010-09-01: 0 + # OBR EFO March 2026: September CPI (receipts Table 3.19) minus calendar-year CPI growth in yoy_growth.yaml. + 2025-09-01: 0.00411 + 2026-09-01: -0.00184 + 2027-09-01: 0.00072 + 2028-09-01: 0.0003 + 2029-09-01: 0.00048 + # OBR EFO March 2026: Q3 CPI 12-month rate (economy Table 1.7) minus calendar-year growth in yoy_growth.yaml. + 2030-09-01: 0 + # After the EFO horizon: no gap, so the forecast is calendar-year growth. + 2031-09-01: 0 +metadata: + unit: /1 + label: September CPI forecast gap to calendar-year CPI + preserve_calendar_dates: true + documentation: | + The gap is the OBR's September CPI forecast minus the calendar-year CPI + growth stored in yoy_growth.yaml, so that stored growth plus the gap is + the OBR's figure. The OBR publishes September CPI in its receipts tables, + as the CPI used to uprate tax thresholds. Where that row stops before the + end of the forecast, the gap uses the quarter containing September (Q3 + CPI, economy Table 1.7). policyengine_uk/utils/import_obr_forecasts.py + regenerates the gaps with each EFO (or with --gaps-only after editing + yoy_growth.yaml). The gap is zero after the EFO horizon, where the + September forecast falls back to calendar-year CPI. + reference: + - title: OBR EFO March 2026 (detailed forecast tables, economy, Table 1.7) + href: https://obr.uk/efo/economic-and-fiscal-outlook-march-2026/ + - title: OBR EFO March 2026 (detailed forecast tables, receipts, Table 3.19) + href: https://obr.uk/efo/economic-and-fiscal-outlook-march-2026/ diff --git a/policyengine_uk/scenarios/no_economic_assumptions.py b/policyengine_uk/scenarios/no_economic_assumptions.py index 55a60abbe3..61623d7e43 100644 --- a/policyengine_uk/scenarios/no_economic_assumptions.py +++ b/policyengine_uk/scenarios/no_economic_assumptions.py @@ -9,16 +9,22 @@ def remove_economic_assumptions(simulation: Simulation): simulation.tax_benefit_system.reset_parameters() cutoff = f"{simulation.default_input_period}-01-01" - yoy_growth = ( - simulation.tax_benefit_system.parameters.gov.economic_assumptions.yoy_growth + economic_assumptions = ( + simulation.tax_benefit_system.parameters.gov.economic_assumptions ) - for parameter in yoy_growth.get_descendants(): - if not isinstance(parameter, Parameter): - continue - for value_at_instant in parameter.values_list: - if value_at_instant.instant_str >= cutoff: - value_at_instant.value = 0 + # Growth series and the statutory uprating inputs derived from growth + # (September CPI and May-July earnings, including their forecast gaps). + for node in ( + economic_assumptions.yoy_growth, + economic_assumptions.statutory_uprating_inputs, + ): + for parameter in node.get_descendants(): + if not isinstance(parameter, Parameter): + continue + for value_at_instant in parameter.values_list: + if value_at_instant.instant_str >= cutoff: + value_at_instant.value = 0 simulation.tax_benefit_system.process_parameters() diff --git a/policyengine_uk/tax_benefit_system.py b/policyengine_uk/tax_benefit_system.py index e9339b7453..d632be9dd9 100644 --- a/policyengine_uk/tax_benefit_system.py +++ b/policyengine_uk/tax_benefit_system.py @@ -24,6 +24,9 @@ from policyengine_uk.parameters.gov.economic_assumptions.create_economic_assumption_indices import ( create_economic_assumption_indices, ) +from policyengine_uk.parameters.gov.economic_assumptions.create_statutory_uprating_inputs import ( + add_statutory_uprating_inputs, +) from policyengine_uk.parameters.gov.economic_assumptions.lag_average_earnings import ( add_lagged_earnings, ) @@ -84,6 +87,7 @@ def process_parameters(self) -> None: Applies various parameter transformations including: - Private pension uprating factors - Lagged earnings and CPI indices + - Statutory uprating inputs (September CPI, May-July earnings) - Triple lock calculations for state pensions - Economic assumption indices - Parameter uprating and backdating @@ -94,6 +98,7 @@ def process_parameters(self) -> None: self.parameters = add_private_pension_uprating_factor(self.parameters) self.parameters = add_lagged_earnings(self.parameters) self.parameters = add_lagged_cpi(self.parameters) + self.parameters = add_statutory_uprating_inputs(self.parameters) self.parameters = add_triple_lock(self.parameters) self.parameters = create_economic_assumption_indices(self.parameters) self.parameters = add_lsr_deprecation_aliases(self.parameters) diff --git a/policyengine_uk/tests/test_import_obr_forecasts.py b/policyengine_uk/tests/test_import_obr_forecasts.py index 541e0d655d..9e14ed94b9 100644 --- a/policyengine_uk/tests/test_import_obr_forecasts.py +++ b/policyengine_uk/tests/test_import_obr_forecasts.py @@ -1,11 +1,28 @@ +import os +from datetime import date +from pathlib import Path from io import BytesIO from zipfile import ZipFile +import pytest +import yaml + +from policyengine_uk.utils import import_obr_forecasts + from policyengine_uk.utils.import_obr_forecasts import ( + STATUTORY_GAP_SPECS, + ForecastGap, + format_gap, + main, + read_calendar_values, + render_forecast_gap_yaml, build_efo_href, + compute_statutory_forecast_gaps, extract_annual_series_from_xlsx, + extract_september_cpi_from_receipts, infer_forecast_start_year, infer_release, + update_forecast_gap_yaml, update_yoy_growth_yaml, ) @@ -31,7 +48,7 @@ def make_sheet(rows: dict[int, list[str]]) -> bytes: return xml.encode() -def make_test_xlsx() -> bytes: +def make_test_xlsx(earnings_quarter: bool = True) -> bytes: workbook_xml = """ @@ -61,6 +78,16 @@ def make_test_xlsx() -> bytes: make_inline_cell("B98", "2026"), make_number_cell("Q98", 3.33), ], + **( + { + 99: [ + make_inline_cell("B99", "2026Q2"), + make_number_cell("Q99", 3.67), + ] + } + if earnings_quarter + else {} + ), } ) sheet_17 = make_sheet( @@ -94,6 +121,14 @@ def make_test_xlsx() -> bytes: make_number_cell("H99", 7.97), make_number_cell("I99", 3.34), ], + 100: [ + make_inline_cell("B100", "2026Q3"), + make_number_cell("C100", 3.2), + make_number_cell("E100", 2.08), + make_number_cell("F100", 2.3), + make_number_cell("H100", 8.0), + make_number_cell("I100", 3.3), + ], } ) sheet_116 = make_sheet( @@ -328,3 +363,361 @@ def test_update_yoy_growth_yaml_keeps_existing_values_when_obr_has_blank_years( assert "2025-01-01: 0.0952" in content assert "2026-01-01: 0.0797" in content assert "2027-01-01: 0.0553" in content + + +def make_receipts_xlsx() -> bytes: + workbook_xml = """ + + + + + + +""" + rels_xml = """ + + + + +""" + sheet_318 = make_sheet({2: [make_inline_cell("B2", "Other table")]}) + sheet_319 = make_sheet( + { + # An earlier table on the same sheet, with other years. + 2: [ + make_inline_cell("C2", "2020-21"), + make_inline_cell("D2", "2021-22"), + ], + 4: [ + make_inline_cell("C4", "2026-27"), + make_inline_cell("D4", "2027-28"), + ], + 21: [ + make_inline_cell("B21", "Memo: CPI used to uprate thresholds"), + make_number_cell("C21", 3.81), + make_number_cell("D21", 2.12), + ], + } + ) + buffer = BytesIO() + with ZipFile(buffer, "w") as archive: + archive.writestr("xl/workbook.xml", workbook_xml) + archive.writestr("xl/_rels/workbook.xml.rels", rels_xml) + archive.writestr("xl/worksheets/sheet1.xml", sheet_318) + archive.writestr("xl/worksheets/sheet2.xml", sheet_319) + return buffer.getvalue() + + +def test_september_cpi_is_keyed_to_the_september_before_the_uprating_year(): + table, rates = extract_september_cpi_from_receipts(make_receipts_xlsx()) + + assert table == "3.19" + assert rates == {2025: 0.0381, 2026: 0.0212} + + +def test_statutory_forecast_gaps_use_the_statutory_quarter(): + gaps = compute_statutory_forecast_gaps(make_test_xlsx(), 2025, 2) + + # Q3 CPI 2.08% minus calendar-year 2.48%; no 2025 quarter in the fixture. + assert {y: g.value for y, g in gaps["cpi_september"].items()} == {2026: -0.004} + # Q2 earnings 3.67% minus calendar-year 3.33%. + assert {y: g.value for y, g in gaps["awe_total_pay_may_july"].items()} == { + 2026: 0.0034 + } + + +def test_statutory_forecast_gaps_prefer_published_september_cpi(): + gaps = compute_statutory_forecast_gaps( + make_test_xlsx(), 2025, 2, receipts_xlsx_bytes=make_receipts_xlsx() + ) + + # September 2025 (3.81%) minus calendar-year 2025 CPI (3.45%), and + # September 2026 (2.12%) minus calendar-year 2026 CPI (2.48%). + cpi = gaps["cpi_september"] + assert {y: g.value for y, g in cpi.items()} == {2025: 0.0036, 2026: -0.0036} + assert "receipts Table 3.19" in cpi[2026].source + + +def test_update_forecast_gap_yaml_rewrites_values_and_reference(tmp_path): + yaml_path = tmp_path / "cpi_september.yaml" + yaml_path.write_text("""description: Gap. +values: + 2010-09-01: 0 + 2025-09-01: 0.001 + 2031-09-01: 0 +metadata: + unit: /1 + documentation: | + Kept. + reference: + - title: OBR EFO November 2025 (detailed forecast tables, economy, Table 1.7) + href: https://obr.uk/efo/economic-and-fiscal-outlook-november-2025/ +""") + spec = next(s for s in STATUTORY_GAP_SPECS if s.key == "cpi_september") + gaps = compute_statutory_forecast_gaps( + make_test_xlsx(), 2025, 2, receipts_xlsx_bytes=make_receipts_xlsx() + )["cpi_september"] + + update_forecast_gap_yaml(yaml_path, spec, gaps, "March", 2026) + + content = yaml_path.read_text() + assert "2025-09-01: 0.0036" in content + assert "2026-09-01: -0.0036" in content + assert "2027-09-01: 0" in content + assert "0.001" not in content + assert "Kept." in content + assert "November 2025" not in content + assert ( + "OBR EFO March 2026 (detailed forecast tables, economy, Table 1.7)" in content + ) + assert ( + "OBR EFO March 2026 (detailed forecast tables, receipts, Table 3.19)" in content + ) + assert "https://obr.uk/efo/economic-and-fiscal-outlook-march-2026/" in content + + +def test_gaps_are_measured_from_the_stored_calendar_growth(): + """Stored growth plus the gap must equal the OBR figure, so the gap is + taken from whatever yoy_growth.yaml holds, not the unrounded workbook.""" + gaps = compute_statutory_forecast_gaps( + make_test_xlsx(), + 2026, + 1, + calendar_values={ + "consumer_price_index": {2026: 0.025}, + "average_earnings": {2026: 0.033}, + }, + ) + assert gaps["cpi_september"][2026].value == pytest.approx(0.0208 - 0.025) + assert gaps["awe_total_pay_may_july"][2026].value == pytest.approx(0.0367 - 0.033) + + +def test_read_calendar_values(tmp_path): + path = tmp_path / "yoy_growth.yaml" + path.write_text(YOY_GROWTH_FIXTURE) + values = read_calendar_values(path) + assert values["consumer_price_index"] == { + 2024: 0.02, + 2025: 0.0, + 2026: 0.0, + 2031: 0.02, + } + + +@pytest.mark.parametrize( + "value, text", + [ + (3e-05, "0.00003"), + (-4e-05, "-0.00004"), + (0.0003, "0.0003"), + (0.0, "0"), + (-0.00184, "-0.00184"), + ], +) +def test_small_gaps_are_written_in_fixed_point(value, text): + """YAML 1.1 reads 3e-05 as a string, which breaks parameter loading.""" + assert format_gap(value) == text + spec = next(s for s in STATUTORY_GAP_SPECS if s.key == "cpi_september") + content = render_forecast_gap_yaml( + GAP_FIXTURE, spec, {2027: ForecastGap(value, "test", "test")}, "March", 2026 + ) + assert isinstance(yaml.safe_load(content)["values"][date(2027, 9, 1)], (int, float)) + + +SERIES_KEYS = [ + "rpi", + "average_earnings", + "consumer_price_index", + "cpih", + "house_prices", + "mortgage_interest", + "rent", +] +YOY_GROWTH_FIXTURE = "obr:\n" + "".join( + f""" {key}: + values: + 2024-01-01: 0.0200 + 2025-01-01: 0.0000 + 2026-01-01: 0.0000 + 2031-01-01: 0.0200 + metadata: + reference: + - title: Old + href: https://example.com/old +""" + for key in SERIES_KEYS +) +GAP_FIXTURE = """description: Gap. +values: + 2010-09-01: 0 +metadata: + unit: /1 + reference: + - title: Old + href: https://example.com/old +""" + + +def write_tree(tmp_path): + """A yoy_growth.yaml with the forecast_gap files next to it.""" + yoy = tmp_path / "economic_assumptions" / "yoy_growth.yaml" + gap_dir = yoy.parent / "statutory_uprating_inputs" / "forecast_gap" + gap_dir.mkdir(parents=True) + yoy.write_text(YOY_GROWTH_FIXTURE) + for spec in STATUTORY_GAP_SPECS: + (gap_dir / f"{spec.key}.yaml").write_text( + GAP_FIXTURE.replace("09-01", spec.month_day) + ) + return yoy, gap_dir + + +def run_main(tmp_path, workbook, *extra): + economy = tmp_path / "economy_march_2026.xlsx" + economy.write_bytes(workbook) + receipts = tmp_path / "receipts.xlsx" + receipts.write_bytes(make_receipts_xlsx()) + return main( + [ + "--file", + str(economy), + "--forecast-start-year", + "2025", + "--forecast-years", + "2", + "--release-month", + "March", + "--release-year", + "2026", + "--receipts-file", + str(receipts), + *extra, + ] + ) + + +def test_main_writes_growth_and_gaps_next_to_the_yaml_path(tmp_path): + yoy, gap_dir = write_tree(tmp_path) + assert run_main(tmp_path, make_test_xlsx(), "--yaml-path", str(yoy)) == 0 + + assert "2026-01-01: 0.0248" in yoy.read_text() + cpi = yaml.safe_load((gap_dir / "cpi_september.yaml").read_text())["values"] + # September 2026 (2.12%) minus the 2026 CPI just written (2.48%). + assert cpi[date(2026, 9, 1)] == pytest.approx(-0.0036) + awe = yaml.safe_load((gap_dir / "awe_total_pay_may_july.yaml").read_text()) + assert awe["values"][date(2026, 7, 1)] == pytest.approx(0.0034) + + +def test_main_writes_nothing_when_a_gap_cannot_be_built(tmp_path): + """Without the Q2 earnings row there is no earnings gap: the run fails + before any file changes, so growth and gaps never mix two forecasts.""" + yoy, gap_dir = write_tree(tmp_path) + before = {path: path.read_text() for path in [yoy, *gap_dir.iterdir()]} + + with pytest.raises(ValueError, match="No forecast gaps"): + run_main( + tmp_path, make_test_xlsx(earnings_quarter=False), "--yaml-path", str(yoy) + ) + + assert {path: path.read_text() for path in before} == before + + +def test_main_requires_the_receipts_tables_for_the_gaps(tmp_path): + yoy, _ = write_tree(tmp_path) + economy = tmp_path / "economy_march_2026.xlsx" + economy.write_bytes(make_test_xlsx()) + with pytest.raises(ValueError, match="receipts"): + main(["--file", str(economy), "--yaml-path", str(yoy)]) + + +def test_gaps_only_leaves_growth_alone(tmp_path): + yoy, gap_dir = write_tree(tmp_path) + growth = yoy.read_text() + assert ( + run_main(tmp_path, make_test_xlsx(), "--yaml-path", str(yoy), "--gaps-only") + == 0 + ) + + assert yoy.read_text() == growth + cpi = yaml.safe_load((gap_dir / "cpi_september.yaml").read_text())["values"] + # September 2026 (2.12%) minus the stored 2026 CPI (0%). + assert cpi[date(2026, 9, 1)] == pytest.approx(0.0212) + + +@pytest.mark.parametrize("failing_call", [1, 2, 3]) +def test_main_restores_every_file_when_a_write_fails( + tmp_path, monkeypatch, failing_call +): + """A failure while moving the second or third file into place (or the + first) leaves growth and both gap files exactly as they were.""" + yoy, gap_dir = write_tree(tmp_path) + before = {path: path.read_text() for path in [yoy, *gap_dir.iterdir()]} + real_replace = os.replace + calls = [] + + def flaky_replace(source, target): + calls.append(target) + if len(calls) == failing_call: + raise OSError("disk full") + real_replace(source, target) + + monkeypatch.setattr(import_obr_forecasts.os, "replace", flaky_replace) + with pytest.raises(OSError, match="disk full"): + run_main(tmp_path, make_test_xlsx(), "--yaml-path", str(yoy)) + + assert {path: path.read_text() for path in before} == before + leftovers = [*tmp_path.rglob("*.staged"), *tmp_path.rglob("*.backup")] + assert leftovers == [] + + +def test_main_leaves_files_alone_when_staging_fails(tmp_path, monkeypatch): + yoy, gap_dir = write_tree(tmp_path) + before = {path: path.read_text() for path in [yoy, *gap_dir.iterdir()]} + real_write_text = Path.write_text + staged = [] + + def flaky_write_text(self, *args, **kwargs): + if self.name.endswith(".staged"): + staged.append(self) + if len(staged) == 2: + raise OSError("disk full") + return real_write_text(self, *args, **kwargs) + + monkeypatch.setattr(Path, "write_text", flaky_write_text) + with pytest.raises(OSError, match="disk full"): + run_main(tmp_path, make_test_xlsx(), "--yaml-path", str(yoy)) + + assert {path: path.read_text() for path in before} == before + assert [*tmp_path.rglob("*.staged"), *tmp_path.rglob("*.backup")] == [] + + +def test_a_failed_restore_keeps_the_original_and_never_empties_a_file( + tmp_path, monkeypatch +): + """The third move fails and so does the first restore. No file may be + left empty: the unrestored file keeps the new text, its original is kept + in a backup named in the error, and the other file is restored.""" + yoy, gap_dir = write_tree(tmp_path) + before = {path: path.read_text() for path in [yoy, *gap_dir.iterdir()]} + real_replace = os.replace + calls = [] + + def failing_replace(source, target): + calls.append(target) + if len(calls) in (3, 4): + raise OSError("I/O error") + real_replace(source, target) + + monkeypatch.setattr(import_obr_forecasts.os, "replace", failing_replace) + with pytest.raises(OSError, match="original text is kept in") as error: + run_main(tmp_path, make_test_xlsx(), "--yaml-path", str(yoy)) + + unrestored = Path(calls[3]) + backup = unrestored.with_name(f".{unrestored.name}.backup") + assert str(backup) in str(error.value) + assert backup.read_text() == before[unrestored] + for path, original in before.items(): + assert path.read_text(), path + if path not in (unrestored, Path(calls[2])): + assert path.read_text() == original, path + assert list(tmp_path.rglob("*.staged")) == [] diff --git a/policyengine_uk/tests/test_triple_lock_outturn.py b/policyengine_uk/tests/test_triple_lock_outturn.py index cb20e43ac7..588157b62e 100644 --- a/policyengine_uk/tests/test_triple_lock_outturn.py +++ b/policyengine_uk/tests/test_triple_lock_outturn.py @@ -1,26 +1,70 @@ -"""Verify the triple lock and downstream state pension amounts match published -DWP / OBR uprating outturn for years where the actual rate is known. +"""Differential test: the triple lock computed from the statutory inputs +against every published State Pension uprating from April 2011. -Issue #953: previously the triple lock was computed from OBR calendar-year -average earnings growth, which differs from the May-July AWE window DWP uses -for the actual triple lock. That left every uprating year ~0.5-1.2pp off. +The inputs are the September CPI rate and the May-July earnings growth +figure each review used (from the Explanatory Memorandum to the following +April's Up-rating Order). The rule reproduces every published rise from +April 2012 to April 2026, and every published weekly rate once rounded to +the nearest 5p. One year needs an override: in April 2011 the basic State +Pension rose by September 2010 RPI (4.6%), a one-off guarantee during the +switch from RPI to CPI, where the rule gives CPI (3.1%). The April 2022 +suspension of the earnings element under the Social Security (Up-rating of +Benefits) Act 2021 is ``include_earnings`` false, not an override. + +Issue #953: the rule previously used OBR calendar-year growth, which left +every uprating year 0.5-1.2pp off. """ +from decimal import ROUND_HALF_UP, Decimal + import pytest +from policyengine_uk.parameters.gov.dwp.state_pension.triple_lock.create_triple_lock import ( + read_uprating_years, + rule_rate, +) from policyengine_uk.system import system - -# (year, expected uprating rate). Sources are linked in -# parameters/gov/dwp/state_pension/triple_lock/outturn.yaml. -OUTTURN_RATES = [ - (2022, 0.031), - (2023, 0.101), - (2024, 0.085), - (2025, 0.041), - (2026, 0.048), +# (April, rise, element that set it, Up-rating Order). Rates from the +# Explanatory Memorandum to each Order. +PUBLISHED_UPRATINGS = [ + (2011, 0.046, "rpi", "SI 2011/821"), + (2012, 0.052, "cpi", "SI 2012/780"), + (2013, 0.025, "floor", "SI 2013/574"), + (2014, 0.027, "cpi", "SI 2014/516"), + (2015, 0.025, "floor", "SI 2015/457"), + (2016, 0.029, "earnings", "SI 2016/230"), + (2017, 0.025, "floor", "SI 2017/260"), + (2018, 0.030, "cpi", "SI 2018/281"), + (2019, 0.026, "earnings", "SI 2019/480"), + (2020, 0.039, "earnings", "SI 2020/234"), + (2021, 0.025, "floor", "SI 2021/162"), + (2022, 0.031, "cpi", "SI 2022/292"), + (2023, 0.101, "cpi", "SI 2023/316"), + (2024, 0.085, "earnings", "SI 2024/242"), + (2025, 0.041, "earnings", "SI 2025/295"), + (2026, 0.048, "earnings", "SI 2026/148"), ] +# The inputs each review used, transcribed independently of the parameter +# files, so a typo in a year where the input did not set the rate is still +# caught. September CPI: ONS D7G7. May-July earnings: DWP, Abstract of DWP +# benefit rate statistics 2025, table 5 ("Earnings (KAC3)"), except 2017, +# where the table shows a later revision (2.3%) and the Explanatory +# Memorandum to SI 2018/281 gives the 2.2% the review used. +SEPTEMBER_CPI_USED = { + 2010: 0.031, 2011: 0.052, 2012: 0.022, 2013: 0.027, 2014: 0.012, + 2015: -0.001, 2016: 0.010, 2017: 0.030, 2018: 0.024, 2019: 0.017, + 2020: 0.005, 2021: 0.031, 2022: 0.101, 2023: 0.067, 2024: 0.017, + 2025: 0.038, +} # fmt: skip +MAY_JULY_EARNINGS_USED = { + 2010: 0.013, 2011: 0.028, 2012: 0.016, 2013: 0.012, 2014: 0.006, + 2015: 0.029, 2016: 0.024, 2017: 0.022, 2018: 0.026, 2019: 0.039, + 2020: -0.010, 2021: 0.083, 2022: 0.055, 2023: 0.085, 2024: 0.041, + 2025: 0.048, +} # fmt: skip + # (year, expected weekly £). Cross-referenced against gov.uk benefit and # pension rates publications. BASIC_STATE_PENSION_WEEKLY = [ @@ -35,28 +79,105 @@ (2026, 241.30), ] +parameters = system.parameters +state_pension = parameters.gov.dwp.state_pension +UPRATING_YEARS = read_uprating_years(parameters) + + +def uprating(year): + return parameters.gov.economic_assumptions.yoy_growth.triple_lock(f"{year}-01-01") -@pytest.mark.parametrize("year, expected", OUTTURN_RATES) -def test_triple_lock_yoy_matches_published_outturn(year, expected): - """The generated triple lock yoy parameter should equal the DWP-announced - rate for years where outturn is known.""" - yoy = system.parameters.gov.economic_assumptions.yoy_growth.triple_lock( - f"{year}-01-01" + +def nearest_5p(amount): + return float( + (Decimal(repr(amount)) / Decimal("0.05")).quantize(Decimal(1), ROUND_HALF_UP) + * Decimal("0.05") ) - assert yoy == pytest.approx(expected, abs=1e-3) + + +def test_inputs_are_the_figures_each_review_used(): + inputs = parameters.gov.economic_assumptions.statutory_uprating_inputs + for year, value in SEPTEMBER_CPI_USED.items(): + assert inputs.cpi_september(f"{year}-09-01") == pytest.approx(value), year + for year, value in MAY_JULY_EARNINGS_USED.items(): + assert inputs.awe_total_pay_may_july(f"{year}-07-01") == pytest.approx(value), ( + year + ) + + +@pytest.mark.parametrize("year, rate, element, order", PUBLISHED_UPRATINGS) +def test_uprating_matches_published_rate(year, rate, element, order): + assert uprating(year) == pytest.approx(rate, abs=1e-9), order + + +@pytest.mark.parametrize( + "year, rate, element, order", + [row for row in PUBLISHED_UPRATINGS if row[2] != "rpi"], +) +def test_rule_reproduces_the_rate_from_the_statutory_inputs(year, rate, element, order): + """No override: the rule alone gives the published rate, set by the + published element.""" + inputs = UPRATING_YEARS[year] + assert inputs.outturn is None + assert rule_rate(inputs) == pytest.approx(rate, abs=1e-9) + value_by_element = { + "earnings": inputs.earnings if inputs.include_earnings else None, + "cpi": inputs.cpi if inputs.include_inflation else None, + "floor": inputs.minimum_rate, + } + assert value_by_element[element] == pytest.approx(rate, abs=1e-9) + + +def test_only_april_2011_needs_an_override(): + overridden = [year for year, inputs in UPRATING_YEARS.items() if inputs.outturn] + assert overridden == [2011] + april_2011 = UPRATING_YEARS[2011] + # The rule would pay September 2010 CPI; the basic State Pension rose by RPI. + assert rule_rate(april_2011) == pytest.approx(0.031) + assert april_2011.outturn == pytest.approx(0.046) + + +@pytest.mark.parametrize( + "amount, first_year", + [ + (state_pension.basic_state_pension.amount, 2011), + # The new State Pension started in April 2016; the triple lock has + # applied to it since April 2017. + (state_pension.new_state_pension.amount, 2017), + ], + ids=["basic", "new"], +) +def test_published_weekly_rates_are_the_uprated_rate_to_the_nearest_5p( + amount, first_year +): + for year in range(first_year, 2027): + previous = amount(f"{year - 1}-06-01") + published = amount(f"{year}-06-01") + assert nearest_5p(previous * (1 + uprating(year))) == pytest.approx( + published, abs=1e-9 + ), year + + +def test_triple_lock_is_never_below_the_statutory_minimum(): + """Social Security Administration Act 1992 s150A requires a rise of at + least earnings growth when earnings rise. For April 2022 the Social + Security (Up-rating of Benefits) Act 2021 required at least the higher of + CPI and 2.5% instead.""" + for year, inputs in UPRATING_YEARS.items(): + if year == 2022: + statutory_minimum = max(inputs.cpi, 0.025) + else: + statutory_minimum = max(inputs.earnings, 0.0) + assert uprating(year) >= statutory_minimum - 1e-12, year @pytest.mark.parametrize("year, expected", BASIC_STATE_PENSION_WEEKLY) def test_basic_state_pension_matches_published_rate(year, expected): - weekly = system.parameters.gov.dwp.state_pension.basic_state_pension.amount( - f"{year}-04-01" - ) + weekly = state_pension.basic_state_pension.amount(f"{year}-04-01") assert weekly == pytest.approx(expected, abs=0.01) @pytest.mark.parametrize("year, expected", NEW_STATE_PENSION_WEEKLY) def test_new_state_pension_matches_published_rate(year, expected): - weekly = system.parameters.gov.dwp.state_pension.new_state_pension.amount( - f"{year}-04-01" - ) + weekly = state_pension.new_state_pension.amount(f"{year}-04-01") assert weekly == pytest.approx(expected, abs=0.01) diff --git a/policyengine_uk/tests/test_triple_lock_properties.py b/policyengine_uk/tests/test_triple_lock_properties.py new file mode 100644 index 0000000000..1abe452122 --- /dev/null +++ b/policyengine_uk/tests/test_triple_lock_properties.py @@ -0,0 +1,290 @@ +"""Property tests for the State Pension uprating rule. + +Invariants, for every input path: + +1. The triple lock rate is at least each included element and the minimum + rate, and equals one of them. Without the triple lock, the rate is + earnings growth, or zero when earnings fall. +2. It never falls when an element rises. +3. Under the earnings-path guarantee, the pension level is never below the + earnings path, and each year's rise is at least the rule's own rate + (for one reading of the plan announced in September 2026, max(CPI, + 2.5%)), so the level + is never below the path compounded at those rates either. +4. The guarantee's top-up is the smallest 0.1 percentage point step that + meets the earnings path. +5. With the guarantee off, the path is the year-by-year rule. +6. The path agrees with an independent transcription of + PolicyEngine/uk-triple-lock's ``rules.rates_matrix`` (the reference + implementation of that reading of the September 2026 plan). +""" + +import math + +import pytest +from hypothesis import HealthCheck, given, settings +from hypothesis import strategies as st + +from policyengine_uk.parameters.gov.dwp.state_pension.triple_lock.create_triple_lock import ( + UpratingYear, + round_to_published_precision, + round_up_to_published_precision, + rule_rate, + triple_lock_rate, + uprating_rates, +) + +# Published figures are to 0.1 percentage points; draw from the range seen +# since 1990 and beyond (deflation to double-digit inflation). +published_rate = st.integers(min_value=-60, max_value=150).map(lambda n: n / 1000) +minimum_rate = st.sampled_from([0.0, 0.02, 0.025, 0.03]) +LEVEL_TOLERANCE = 1e-9 +# Generation is cheap, but a loaded CI runner can trip the speed check. +PROPERTIES = settings( + max_examples=300, deadline=None, suppress_health_check=[HealthCheck.too_slow] +) + + +def uprating_year(earnings_path_guarantee=st.booleans()): + return st.builds( + UpratingYear, + earnings=published_rate, + cpi=published_rate, + minimum_rate=minimum_rate, + active=st.booleans(), + include_earnings=st.booleans(), + include_inflation=st.booleans(), + earnings_path_guarantee=earnings_path_guarantee, + ) + + +def paths(earnings_path_guarantee=st.booleans(), min_size=1, max_size=20): + return st.lists( + uprating_year(earnings_path_guarantee), min_size=min_size, max_size=max_size + ).map(lambda years: {2030 + i: year for i, year in enumerate(years)}) + + +@PROPERTIES +@given( + earnings=published_rate, + cpi=published_rate, + floor=minimum_rate, + include_earnings=st.booleans(), + include_inflation=st.booleans(), +) +def test_triple_lock_is_the_highest_included_element( + earnings, cpi, floor, include_earnings, include_inflation +): + rate = triple_lock_rate(earnings, cpi, floor, include_earnings, include_inflation) + assert rate >= floor + if include_earnings: + assert rate >= earnings + if include_inflation: + assert rate >= cpi + included = [floor] + if include_earnings: + included.append(earnings) + if include_inflation: + included.append(cpi) + assert rate in included + + +@PROPERTIES +@given(inputs=uprating_year()) +def test_rule_rate_is_the_triple_lock_or_the_statutory_earnings_link(inputs): + rate = rule_rate(inputs) + if inputs.active: + assert rate == triple_lock_rate( + inputs.earnings, + inputs.cpi, + inputs.minimum_rate, + inputs.include_earnings, + inputs.include_inflation, + ) + else: + assert rate == max(inputs.earnings, 0.0) + assert rate >= 0 + + +@PROPERTIES +@given( + earnings=published_rate, + cpi=published_rate, + floor=minimum_rate, + rise=st.integers(min_value=1, max_value=50).map(lambda n: n / 1000), +) +def test_triple_lock_never_falls_when_an_element_rises(earnings, cpi, floor, rise): + rate = triple_lock_rate(earnings, cpi, floor) + assert triple_lock_rate(earnings + rise, cpi, floor) >= rate + assert triple_lock_rate(earnings, cpi + rise, floor) >= rate + assert triple_lock_rate(earnings, cpi, floor + rise) >= rate + + +@PROPERTIES +@given(years=paths(earnings_path_guarantee=st.just(False))) +def test_without_guarantee_the_path_is_the_yearly_rule(years): + rates = uprating_rates(years) + assert rates == {year: rule_rate(inputs) for year, inputs in years.items()} + + +@PROPERTIES +@given(years=paths()) +def test_guarantee_keeps_the_pension_on_or_above_its_earnings_path(years): + rates = uprating_rates(years) + level = 1.0 + earnings_path = None + rule_path = None + for year in sorted(years): + inputs = years[year] + # Each year's rise is at least the rule's own rate. + assert rates[year] >= rule_rate(inputs) + if inputs.earnings_path_guarantee: + if earnings_path is None: + earnings_path = rule_path = level + earnings_path *= 1 + inputs.earnings + rule_path *= 1 + rule_rate(inputs) + else: + earnings_path = rule_path = None + level *= 1 + rates[year] + if earnings_path is not None: + assert level >= earnings_path * (1 - LEVEL_TOLERANCE) + assert level >= rule_path * (1 - LEVEL_TOLERANCE) + + +@PROPERTIES +@given(years=paths()) +def test_guarantee_top_up_is_the_smallest_step_that_reaches_the_path(years): + rates = uprating_rates(years) + level = 1.0 + earnings_path = None + for year in sorted(years): + inputs = years[year] + if inputs.earnings_path_guarantee: + if earnings_path is None: + earnings_path = level + earnings_path *= 1 + inputs.earnings + if rates[year] > rule_rate(inputs): + one_step_less = level * (1 + rates[year] - 0.001) + assert one_step_less < earnings_path * (1 + LEVEL_TOLERANCE) + else: + earnings_path = None + level *= 1 + rates[year] + + +@PROPERTIES +@given(years=paths()) +def test_rates_stay_on_the_published_grid(years): + for rate in uprating_rates(years).values(): + assert rate * 1000 == pytest.approx(round(rate * 1000), abs=1e-6) + + +def reference_plan_rates(cpi, earnings, switch_index, floor=0.025, decimals=3): + """Transcription of uk-triple-lock ``rules.rates_matrix`` for the + ``burnham_2030`` policy, one draw, without numpy.""" + scale = 10**decimals + + def rnd(r): + return round(r, decimals) + + def rnd_up(r): + return math.ceil(round(r * scale, 9)) / scale + + rates, level, anchor = [], 1.0, 1.0 + for j, (c, e) in enumerate(zip(cpi, earnings)): + if j < switch_index: + rate = rnd(max(max(c, e), floor)) + level *= 1 + rate + anchor = level + else: + anchor *= 1 + e + floor_rate = rnd(max(max(c, floor), 0.0)) + rate = max(floor_rate, rnd_up(anchor / level - 1)) + level *= 1 + rate + rates.append(rate) + return rates + + +@PROPERTIES +@given( + data=st.lists(st.tuples(published_rate, published_rate), min_size=1, max_size=15), + switch_index=st.integers(min_value=0, max_value=15), +) +def test_plan_matches_the_uk_triple_lock_reference(data, switch_index): + cpi = [c for c, _ in data] + earnings = [e for _, e in data] + years = { + 2020 + j: UpratingYear( + earnings=e, + cpi=c, + minimum_rate=0.025, + include_earnings=j < switch_index, + earnings_path_guarantee=j >= switch_index, + ) + for j, (c, e) in enumerate(data) + } + rates = uprating_rates(years) + expected = reference_plan_rates(cpi, earnings, switch_index) + assert [rates[year] for year in sorted(rates)] == pytest.approx(expected, abs=1e-12) + + +@PROPERTIES +@given(rate=st.floats(min_value=-0.5, max_value=0.5, allow_nan=False)) +def test_rounding_to_published_precision(rate): + rounded = round_to_published_precision(rate) + assert abs(rounded - rate) <= 0.0005 + 1e-12 + assert round_to_published_precision(rounded) == rounded + rounded_up = round_up_to_published_precision(rate) + assert rate - 1e-9 <= rounded_up < rate + 0.001 + 1e-12 + + +def test_rounding_takes_halves_away_from_zero(): + assert round_to_published_precision(0.0255) == 0.026 + # Round-half-even would give 0.024 and -0.024. + assert round_to_published_precision(0.0245) == 0.025 + assert round_to_published_precision(-0.0245) == -0.025 + assert round_to_published_precision(-0.0095) == -0.010 + assert round_up_to_published_precision(0.025000000000000355) == 0.025 + assert round_up_to_published_precision(0.02500001) == 0.026 + + +def test_plan_example_by_hand(): + """Two years under the plan: CPI 2%, earnings 4%, then CPI 2%, earnings 1%. + + Year 1 anchors the earnings path at 1 and moves it to 1.04; the floor + pays 2.5%, so the top-up to 4.0% binds. Year 2: the path is + 1.04 x 1.01 = 1.0504 against a level of 1.04, a 1.0% rise, below the + 2.5% floor, so the floor pays. + """ + plan = dict( + minimum_rate=0.025, include_earnings=False, earnings_path_guarantee=True + ) + rates = uprating_rates( + { + 2030: UpratingYear(earnings=0.04, cpi=0.02, **plan), + 2031: UpratingYear(earnings=0.01, cpi=0.02, **plan), + } + ) + assert rates == {2030: 0.04, 2031: 0.025} + + +def test_overrides_are_paid_as_published_even_when_zero(): + rates = uprating_rates( + {2030: UpratingYear(earnings=0.03, cpi=0.02, minimum_rate=0.025, outturn=0.0)} + ) + assert rates == {2030: 0.0} + + +def test_an_override_under_the_guarantee_is_paid_as_is_and_the_path_carries_on(): + """An override sets the rate with no top-up, but the earnings path still + grows that year: 1.05 against a level of 1.01, so the next year needs + 1.05 / 1.01 - 1 = 3.96%, rounded up to 4.0%.""" + plan = dict( + minimum_rate=0.025, include_earnings=False, earnings_path_guarantee=True + ) + rates = uprating_rates( + { + 2030: UpratingYear(earnings=0.05, cpi=0.0, outturn=0.01, **plan), + 2031: UpratingYear(earnings=0.0, cpi=0.0, **plan), + } + ) + assert rates == {2030: 0.01, 2031: 0.04} diff --git a/policyengine_uk/tests/test_triple_lock_statutory_inputs.py b/policyengine_uk/tests/test_triple_lock_statutory_inputs.py new file mode 100644 index 0000000000..a3101905d0 --- /dev/null +++ b/policyengine_uk/tests/test_triple_lock_statutory_inputs.py @@ -0,0 +1,452 @@ +"""The statutory uprating inputs, their forecasts, and how scenarios move them. + +The April uprating of the State Pension uses September CPI and May-July +earnings growth from the previous year. These tests check that forecasts of +those inputs follow the OBR where it publishes them, fall back to +calendar-year growth after, respond to macro scenarios applied before the +data load, and can be set directly; and that the earnings-path guarantee +models one reading of the plan announced in September 2026 as a +parameter reform. +""" + +import numpy as np +import pandas as pd +import pytest + +from policyengine_uk import Simulation +from policyengine_uk.data.dataset_schema import UKSingleYearDataset +from policyengine_uk.model_api import Scenario +from policyengine_uk.scenarios import no_economic_assumptions +from policyengine_uk.system import system + +INPUTS = "gov.economic_assumptions.statutory_uprating_inputs" +OBR = "gov.economic_assumptions.yoy_growth.obr" +TRIPLE_LOCK = "gov.dwp.state_pension.triple_lock" +WEEKS_IN_YEAR = 52 + +PENSIONER = { + "people": {"person": {"age": {2026: 70}}}, + "benunits": {"benunit": {"members": ["person"]}}, + "households": {"household": {"members": ["person"]}}, +} + + +def parameters_under(changes): + simulation = Simulation( + situation=PENSIONER, + scenario=Scenario(parameter_changes=changes, applied_before_data_load=True), + ) + return simulation.tax_benefit_system.parameters + + +def uprating(parameters, year): + return parameters.gov.economic_assumptions.yoy_growth.triple_lock(f"{year}-01-01") + + +def new_state_pension_weekly(parameters, year): + return parameters.gov.dwp.state_pension.new_state_pension.amount(f"{year}-06-01") + + +def statutory_inputs(parameters): + return parameters.gov.economic_assumptions.statutory_uprating_inputs + + +def test_april_2027_uses_published_may_july_2026_earnings(): + """May-July 2026 total pay growth was 3.9% (first estimate), above the + model's September 2026 CPI forecast (the OBR's 2.1%; August 2026 CPI was + 3.1%). Unless September CPI exceeds 3.9%, the new State Pension rises from + £241.30 to £250.71 (before rounding to 5p).""" + parameters = system.parameters + assert uprating(parameters, 2027) == pytest.approx(0.039) + assert new_state_pension_weekly(parameters, 2027) == pytest.approx( + 241.30 * 1.039, abs=0.005 + ) + + +def test_every_published_year_has_its_own_value(): + """A missing year would silently carry the previous year's figure.""" + inputs = statutory_inputs(system.parameters) + for name, month_day, last_published in [ + ("cpi_september", "09-01", 2025), + ("awe_total_pay_may_july", "07-01", 2026), + ]: + parameter = getattr(inputs, name) + explicit = {value.instant_str for value in parameter.values_list} + for year in range(2010, last_published + 1): + assert f"{year}-{month_day}" in explicit, (name, year) + + +@pytest.mark.parametrize( + "name, series, month_day, first_forecast_year", + [ + ("cpi_september", "consumer_price_index", "09-01", 2026), + ("awe_total_pay_may_july", "average_earnings", "07-01", 2027), + ], +) +def test_forecasts_are_calendar_growth_plus_the_obr_gap( + name, series, month_day, first_forecast_year +): + parameters = system.parameters + statutory = getattr(statutory_inputs(parameters), name) + gap = getattr(statutory_inputs(parameters).forecast_gap, name) + calendar = getattr(parameters.gov.economic_assumptions.yoy_growth.obr, series) + last_year = max(int(v.instant_str[:4]) for v in calendar.values_list) + assert last_year >= 2073 + for year in range(first_forecast_year, last_year + 1): + observed = f"{year}-{month_day}" + assert statutory(observed) == pytest.approx( + calendar(f"{year}-01-01") + gap(observed), abs=1e-12 + ), year + if year > 2030: + # After the EFO horizon the forecast is calendar-year growth. + assert gap(observed) == 0 + assert statutory(f"{last_year + 1}-{month_day}") is None + + +# OBR March 2026 EFO: September CPI from receipts Table 3.19 ("CPI used to +# uprate thresholds"), Q3 CPI from economy Table 1.7 for 2030, and Q2 average +# earnings growth on a year earlier from economy Table 1.6. +OBR_SEPTEMBER_CPI = { + 2026: 0.021157, + 2027: 0.020719, + 2028: 0.020299, + 2029: 0.020480, + 2030: 0.019996, +} +OBR_Q2_EARNINGS = {2027: 0.024266, 2028: 0.020795, 2029: 0.021852, 2030: 0.023890} + + +def test_baseline_forecasts_reproduce_the_obr_statutory_forecasts(): + """The gaps are measured from the calendar-year growth stored in + yoy_growth.yaml, so stored growth plus the gap is the OBR's figure, to + the 1e-5 the gaps are stored to. This fails if yoy_growth.yaml is + refreshed without regenerating the gaps.""" + inputs = statutory_inputs(system.parameters) + for year, value in OBR_SEPTEMBER_CPI.items(): + observed = inputs.cpi_september(f"{year}-09-01") + assert observed == pytest.approx(value, abs=1e-5), year + for year, value in OBR_Q2_EARNINGS.items(): + observed = inputs.awe_total_pay_may_july(f"{year}-07-01") + assert observed == pytest.approx(value, abs=1e-5), year + + +def test_rule_reproduces_the_obr_triple_lock_forecast_from_its_own_inputs(): + """OBR Long-term economic determinants (March 2026 EFO), 'Triple Lock' + row: 3.7% in April 2027 and 2.5% in April 2028-2031. With the published + May-July 2026 figure removed, so earnings follow the OBR forecast, the + rule gives the same rates.""" + + def drop_may_july_2026(simulation): + system_ = simulation.tax_benefit_system + system_.reset_parameters() + statutory_inputs(system_.parameters).awe_total_pay_may_july.update( + period="year:2026-07-01:1", value=None + ) + system_.process_parameters() + + simulation = Simulation( + situation=PENSIONER, + scenario=Scenario( + simulation_modifier=drop_may_july_2026, applied_before_data_load=True + ), + ) + parameters = simulation.tax_benefit_system.parameters + assert statutory_inputs(parameters).awe_total_pay_may_july( + "2026-07-01" + ) == pytest.approx(0.036736, abs=1e-5) + assert [uprating(parameters, year) for year in range(2027, 2032)] == ( + pytest.approx([0.037, 0.025, 0.025, 0.025, 0.025]) + ) + + +def test_horizon_runs_to_the_end_of_the_economic_assumptions(): + triple_lock = system.parameters.gov.economic_assumptions.yoy_growth.triple_lock + years = sorted(int(value.instant_str[:4]) for value in triple_lock.values_list) + assert years[0] == 2011 + assert years[-1] == 2074 + assert years == list(range(2011, 2075)) + # 2035 onwards follows the long-run earnings path, not a value carried + # from the last year of the old horizon (2034). + assert uprating(system.parameters, 2040) == pytest.approx(0.038) + + +def test_macro_scenario_on_calendar_growth_moves_the_uprating(): + """Raising 2027 earnings growth from 2.4% to 5% moves May-July 2027 + earnings with it (plus the OBR gap, 0.027pp) and so the April 2028 rise. + + Calendar-year series are keyed to 1 January, so the change is keyed + year:2027-01-01:1; a bare "2027" names the fiscal year from 6 April 2027 + and would change the 2028 calendar-year value instead.""" + parameters = parameters_under( + {f"{OBR}.average_earnings": {"year:2027-01-01:1": 0.05}} + ) + assert statutory_inputs(parameters).awe_total_pay_may_july( + "2027-07-01" + ) == pytest.approx(0.05 + 0.00027) + assert uprating(parameters, 2028) == pytest.approx(0.05) + assert uprating(system.parameters, 2028) == pytest.approx(0.025) + assert new_state_pension_weekly(parameters, 2028) > new_state_pension_weekly( + system.parameters, 2028 + ) + # Only that year moves. + assert uprating(parameters, 2029) == uprating(system.parameters, 2029) + + +def test_macro_scenario_moves_the_state_pension_in_a_microsimulation(): + """The rate reaches the benefit: a full-rate new State Pension recipient + in 2026 data is paid the uprated full rate in 2028.""" + person = pd.DataFrame( + { + "person_id": [1], + "person_benunit_id": [1], + "person_household_id": [1], + "age": [70], + "state_pension_reported": [241.30 * WEEKS_IN_YEAR], + } + ) + dataset = UKSingleYearDataset( + person=person, + benunit=pd.DataFrame({"benunit_id": [1]}), + household=pd.DataFrame( + { + "household_id": [1], + "region": ["LONDON"], + "tenure_type": ["OWNED_OUTRIGHT"], + "council_tax": [2_000.0], + "rent": [0.0], + } + ), + fiscal_year=2026, + ) + scenario = Scenario( + parameter_changes={f"{OBR}.average_earnings": {"year:2027-01-01:1": 0.05}}, + applied_before_data_load=True, + ) + baseline = Simulation(dataset=dataset) + reformed = Simulation(dataset=dataset, scenario=scenario) + + baseline_pension = float(baseline.calculate("new_state_pension", 2028)[0]) + reformed_pension = float(reformed.calculate("new_state_pension", 2028)[0]) + full_rate_2027 = 241.30 * 1.039 + # Uprating indices are stored to 5 decimal places. + assert baseline_pension == pytest.approx( + full_rate_2027 * 1.025 * WEEKS_IN_YEAR, rel=1e-5 + ) + assert reformed_pension == pytest.approx( + full_rate_2027 * 1.05 * WEEKS_IN_YEAR, rel=1e-5 + ) + + +def test_setting_a_statutory_input_directly_overrides_its_forecast(): + """A scenario can supply September CPI and May-July earnings itself, for + example from a model of the monthly series. The value replaces the + forecast for that year only, whatever calendar-year growth says.""" + parameters = parameters_under( + { + f"{INPUTS}.awe_total_pay_may_july": {"2027": 0.07}, + f"{INPUTS}.cpi_september": {"2028": 0.061}, + f"{OBR}.average_earnings": {"year:2027-01-01:1": 0.01}, + } + ) + assert uprating(parameters, 2028) == pytest.approx(0.07) + assert uprating(parameters, 2029) == pytest.approx(0.061) + assert uprating(parameters, 2030) == uprating(system.parameters, 2030) + inputs = statutory_inputs(parameters) + assert inputs.awe_total_pay_may_july("2028-07-01") == pytest.approx( + statutory_inputs(system.parameters).awe_total_pay_may_july("2028-07-01") + ) + + +PLAN = { + f"{TRIPLE_LOCK}.include_earnings": {"year:2030-01-01:100": False}, + f"{TRIPLE_LOCK}.earnings_path_guarantee": {"year:2030-01-01:100": True}, +} + + +def test_plan_is_a_parameter_reform(): + """From April 2030: rises of at least max(CPI, 2.5%), and the pension + never below an earnings link from its 2029-30 level. + + Earnings inputs of 1.0% (May-July 2029 and 2030) then 6.0% (2031) and + 3.0% (2032), CPI 2.0% throughout: + + - April 2030 and 2031: the floor, 2.5%, is above earnings (the triple + lock would also pay 2.5%). + - April 2032: the earnings path has grown 1.01 x 1.01 x 1.06 = 1.0813 + from the 2029-30 level against 1.025 x 1.025 = 1.0506 for the pension, + so it rises by 1.0813 / 1.0506 - 1 = 2.92%, rounded up to 3.0%, where + the triple lock pays 6.0%. + - April 2033: the path grows 3.0% to 1.1137 and the pension, at + 1.0821, again needs 2.92%, rounded up to 3.0%, as the triple lock pays. + """ + growth = { + f"{INPUTS}.awe_total_pay_may_july": { + "2029": 0.01, + "2030": 0.01, + "2031": 0.06, + "2032": 0.03, + }, + f"{INPUTS}.cpi_september": {str(year): 0.02 for year in range(2029, 2033)}, + } + triple_lock = parameters_under(growth) + plan = parameters_under({**growth, **PLAN}) + + assert [uprating(triple_lock, year) for year in range(2030, 2034)] == ( + pytest.approx([0.025, 0.025, 0.06, 0.03]) + ) + assert [uprating(plan, year) for year in range(2030, 2034)] == ( + pytest.approx([0.025, 0.025, 0.030, 0.030]) + ) + # Unchanged before the switch. + for year in range(2027, 2030): + assert uprating(plan, year) == uprating(triple_lock, year) + + base = new_state_pension_weekly(plan, 2029) + assert base == new_state_pension_weekly(triple_lock, 2029) + earnings_path = base + for year in range(2030, 2034): + earnings_path *= 1 + [0.01, 0.01, 0.06, 0.03][year - 2030] + # Uprating indices are stored to 5 decimal places. + assert new_state_pension_weekly(plan, year) >= earnings_path * (1 - 1e-5) + assert new_state_pension_weekly(plan, year) <= new_state_pension_weekly( + triple_lock, year + ) + + +def test_plan_is_off_under_current_law(): + triple_lock = system.parameters.gov.dwp.state_pension.triple_lock + for year in range(2011, 2075): + assert not triple_lock.earnings_path_guarantee(f"{year}-04-30") + + +def test_without_the_triple_lock_the_pension_follows_earnings(): + """With the triple lock off, the statutory review (SSAA 1992 s150A) + requires a rise of at least earnings growth, and none when earnings + fall: 1.2% and then 0%, where the triple lock pays its 2.5% floor.""" + growth = { + f"{INPUTS}.awe_total_pay_may_july": {"2027": 0.012, "2028": -0.004}, + f"{INPUTS}.cpi_september": {"2027": 0.02, "2028": 0.02}, + } + triple_lock = parameters_under(growth) + no_triple_lock = parameters_under( + {**growth, f"{TRIPLE_LOCK}.active": {"year:2028-01-01:100": False}} + ) + assert [uprating(triple_lock, year) for year in (2028, 2029)] == ( + pytest.approx([0.025, 0.025]) + ) + assert [uprating(no_triple_lock, year) for year in (2028, 2029)] == ( + pytest.approx([0.012, 0.0]) + ) + assert uprating(no_triple_lock, 2027) == uprating(triple_lock, 2027) + + +def test_no_economic_assumptions_leaves_only_the_floor(): + simulation = Simulation(situation=PENSIONER, scenario=no_economic_assumptions) + parameters = simulation.tax_benefit_system.parameters + cutoff_year = int(simulation.default_input_period) + for year in range(cutoff_year + 2, 2075): + assert uprating(parameters, year) == pytest.approx(0.025), year + + +def test_numpy_inputs_are_accepted(): + """Draws from a time-series model often arrive as numpy scalars.""" + simulation = Simulation(situation=PENSIONER) + simulation.apply_parameter_changes( + { + f"{INPUTS}.cpi_september": {"2027": np.float64(0.045)}, + f"{OBR}.average_earnings": {"year:2028-01-01:1": np.float64(0.031)}, + } + ) + parameters = simulation.tax_benefit_system.parameters + assert uprating(parameters, 2028) == pytest.approx(0.045) + assert uprating(parameters, 2029) == pytest.approx(0.031) + + +def test_a_zero_input_is_kept_not_forecast(): + """Zero is a value, not a missing figure: with the triple lock off, the + rise is the 0% earnings input, not the 2.4% forecast.""" + parameters = parameters_under( + { + f"{INPUTS}.awe_total_pay_may_july": {"2027": 0.0}, + f"{INPUTS}.cpi_september": {"2027": 0.0}, + f"{TRIPLE_LOCK}.active": {"year:2028-01-01:1": False}, + } + ) + inputs = statutory_inputs(parameters) + assert inputs.awe_total_pay_may_july("2027-07-01") == 0 + assert inputs.cpi_september("2027-09-01") == 0 + assert uprating(parameters, 2028) == 0 + + +def test_no_economic_assumptions_zeroes_the_forecast_inputs(): + simulation = Simulation(situation=PENSIONER, scenario=no_economic_assumptions) + inputs = statutory_inputs(simulation.tax_benefit_system.parameters) + cutoff_year = int(simulation.default_input_period) + for year in range(max(cutoff_year, 2027), 2074): + assert inputs.cpi_september(f"{year}-09-01") == 0, year + assert inputs.awe_total_pay_may_july(f"{year}-07-01") == 0, year + assert inputs.forecast_gap.cpi_september(f"{year}-09-01") == 0, year + + +def test_policy_levers_apply_from_the_april_of_the_year_they_name(): + """A floor of 3% keyed to the bare year 2028 (the fiscal year from + 6 April 2028) sets the April 2028 rise, read on 30 April, and no other + year's; so do the include flags.""" + low_growth = { + f"{INPUTS}.awe_total_pay_may_july": {"2027": 0.01, "2028": 0.01}, + f"{INPUTS}.cpi_september": {"2027": 0.05, "2028": 0.01}, + } + floor = parameters_under( + {**low_growth, f"{TRIPLE_LOCK}.minimum_rate": {"2028": 0.03}} + ) + assert [uprating(floor, year) for year in (2028, 2029)] == pytest.approx( + [0.05, 0.025] + ) + floor_binds = parameters_under( + { + **low_growth, + f"{INPUTS}.cpi_september": {"2027": 0.01, "2028": 0.01}, + f"{TRIPLE_LOCK}.minimum_rate": {"2028": 0.03}, + } + ) + assert [uprating(floor_binds, year) for year in (2027, 2028, 2029)] == ( + pytest.approx([0.039, 0.03, 0.025]) + ) + no_cpi = parameters_under( + {**low_growth, f"{TRIPLE_LOCK}.include_inflation": {"2028": False}} + ) + assert uprating(no_cpi, 2028) == pytest.approx(0.025) + + +def test_inputs_are_rounded_to_published_precision(): + """The review uses the published one-decimal figure: 3.04% CPI pays + 3.0%.""" + parameters = parameters_under( + { + f"{INPUTS}.cpi_september": {"2027": 0.0304}, + f"{INPUTS}.awe_total_pay_may_july": {"2027": 0.0296}, + } + ) + assert uprating(parameters, 2028) == pytest.approx(0.030) + + +def test_horizon_ends_with_the_shorter_calendar_series(): + parameters = parameters_under( + {f"{OBR}.consumer_price_index": {"year:2075-01-01:1": 0.02}} + ) + triple_lock = parameters.gov.economic_assumptions.yoy_growth.triple_lock + assert max(int(v.instant_str[:4]) for v in triple_lock.values_list) == 2074 + + +def test_the_rule_needs_the_forecasts_filled_in_first(): + """Read before the forecasts are filled in, September 2026 CPI is still + null, and the rule refuses rather than guessing.""" + from policyengine_uk.parameters.gov.dwp.state_pension.triple_lock.create_triple_lock import ( + read_uprating_years, + ) + from policyengine_uk.tax_benefit_system import CountryTaxBenefitSystem + + raw = CountryTaxBenefitSystem() + raw.reset_parameters() + with pytest.raises(ValueError, match="September 2026 CPI"): + read_uprating_years(raw.parameters) diff --git a/policyengine_uk/utils/create_triple_lock.py b/policyengine_uk/utils/create_triple_lock.py deleted file mode 100644 index b4af9b8c66..0000000000 --- a/policyengine_uk/utils/create_triple_lock.py +++ /dev/null @@ -1,22 +0,0 @@ -from policyengine_uk.system import system - -# Run this script to generate the triple lock parameter for updated CPI and average earnings forecasts from the OBR. - -cpi = system.parameters.gov.economic_assumptions.indices.obr.consumer_price_index -average_earnings = ( - system.parameters.gov.economic_assumptions.indices.obr.average_earnings -) - -START_YEAR = 2021 - -triple_lock = system.parameters.gov.dwp.state_pension.triple_lock -lock_value = triple_lock(START_YEAR - 1) - -for year in range(START_YEAR, 2029): - earnings_increase = average_earnings(year - 1) / average_earnings(year - 2) - cpi_increase = cpi(year - 1) / cpi(year - 2) - triple_lock_increase = max(earnings_increase, cpi_increase, 1.025) - lock_value *= triple_lock_increase - print( - f" {year}-01-01: {lock_value:.3f} # Earnings increase FY{year - 1}/{year - 2} = {earnings_increase - 1:.1%}, CPI increase FY{year - 1}/{year - 2} = {cpi_increase - 1:.1%}" - ) diff --git a/policyengine_uk/utils/import_obr_forecasts.py b/policyengine_uk/utils/import_obr_forecasts.py index 3d7e6c8eed..cba8bce113 100644 --- a/policyengine_uk/utils/import_obr_forecasts.py +++ b/policyengine_uk/utils/import_obr_forecasts.py @@ -1,7 +1,9 @@ from __future__ import annotations import argparse +import os import re +import shutil from dataclasses import dataclass from io import BytesIO from pathlib import Path @@ -92,6 +94,45 @@ class SeriesSpec: ) +QUARTER_RE = re.compile(r"^(\d{4})\s*Q([1-4])$") + + +@dataclass(frozen=True) +class StatutoryGapSpec: + """Forecast gap between a statutory uprating input and calendar-year growth. + + The OBR publishes September CPI only as the receipts tables' memo row of + the CPI used to uprate tax thresholds, and does not forecast May-July AWE + at all. Where the September row is not supplied or does not cover a year, + the statutory figure is the quarter containing (or nearest) the statutory + period, from the same economy-table column as the calendar-year figure. + """ + + key: str + series_key: str + quarter: int + month_day: str + description: str + + +STATUTORY_GAP_SPECS = ( + StatutoryGapSpec( + key="cpi_september", + series_key="consumer_price_index", + quarter=3, + month_day="09-01", + description="Q3 CPI 12-month rate", + ), + StatutoryGapSpec( + key="awe_total_pay_may_july", + series_key="average_earnings", + quarter=2, + month_day="07-01", + description="Q2 average earnings growth on a year earlier", + ), +) + + def get_repo_root() -> Path: current = Path(__file__).resolve() while current != current.parent: @@ -108,6 +149,12 @@ def get_yoy_growth_path() -> Path: ) +def get_forecast_gap_dir(yoy_growth_path: Path | None = None) -> Path: + """The forecast_gap directory next to a yoy_growth.yaml.""" + yoy_growth_path = yoy_growth_path or get_yoy_growth_path() + return yoy_growth_path.parent / "statutory_uprating_inputs" / "forecast_gap" + + def normalise_label(value: str | None) -> str: if not value: return "" @@ -270,6 +317,248 @@ def extract_annual_series_from_xlsx( return result +def extract_series_by_period_from_xlsx( + xlsx_bytes: bytes, series_keys: set[str] +) -> dict[str, dict[str, float]]: + """Full-precision values (as fractions) keyed by the row label in column B, + e.g. "2026" or "2026Q3".""" + specs = [spec for spec in SERIES_SPECS if spec.key in series_keys] + rows_by_sheet = { + sheet: read_sheet_rows(xlsx_bytes, sheet) for sheet in {s.sheet for s in specs} + } + result: dict[str, dict[str, float]] = {} + for spec in specs: + rows = rows_by_sheet[spec.sheet] + column = find_series_column(rows, spec) + values: dict[str, float] = {} + for row in rows: + label = str(row.get("B") or "").strip() + raw_value = row.get(column) + if not label or raw_value in (None, ""): + continue + if YEAR_RE.match(label) or QUARTER_RE.match(label): + values[label.replace(" ", "")] = float(raw_value) / 100 + result[spec.key] = values + return result + + +SEPTEMBER_CPI_LABEL = "cpi used to uprate thresholds" +FISCAL_YEAR_RE = re.compile(r"^(\d{4})-(\d{2})$") + + +@dataclass(frozen=True) +class ForecastGap: + value: float + source: str + reference_title: str + + +def extract_september_cpi_from_receipts( + xlsx_bytes: bytes, +) -> tuple[str, dict[int, float]]: + """September CPI 12-month rates from the receipts tables. + + The OBR publishes them as the memo row "CPI used to uprate thresholds", + one column per fiscal year of uprating: 2027-28 holds September 2026. + Returns the table number and the rates keyed by September's year. + """ + with ZipFile(BytesIO(xlsx_bytes)) as archive: + sheet_names = list(_sheet_paths(archive)) + for sheet in sheet_names: + rows = read_sheet_rows(xlsx_bytes, sheet) + for index, row in enumerate(rows): + if SEPTEMBER_CPI_LABEL not in normalise_label(row.get("B")): + continue + columns = {} + for header in reversed(rows[:index]): + columns = { + column: int(match.group(1)) + for column, value in header.items() + if value and (match := FISCAL_YEAR_RE.match(str(value).strip())) + } + if columns: + break + rates = { + fiscal_year - 1: float(row[column]) / 100 + for column, fiscal_year in columns.items() + if row.get(column) not in (None, "") + } + if rates: + return sheet, rates + raise ValueError("Could not find September CPI in the receipts workbook") + + +def compute_statutory_forecast_gaps( + xlsx_bytes: bytes, + forecast_start_year: int, + forecast_years: int, + receipts_xlsx_bytes: bytes | None = None, + calendar_values: dict[str, dict[int, float]] | None = None, +) -> dict[str, dict[int, ForecastGap]]: + """Gap for each statutory input and forecast year, rounded to 1e-5. + + The gap is the OBR's statutory-basis figure minus the calendar-year growth + PolicyEngine stores, so that stored growth plus the gap reproduces the + OBR figure. ``calendar_values`` defaults to the values this importer + writes to yoy_growth.yaml. September CPI comes from the receipts tables + where they are supplied and cover the year; otherwise each input uses its + quarter from the economy tables. + """ + series = extract_series_by_period_from_xlsx( + xlsx_bytes, {spec.series_key for spec in STATUTORY_GAP_SPECS} + ) + if calendar_values is None: + calendar_values = extract_annual_series_from_xlsx(xlsx_bytes) + september_table, september_cpi = ( + extract_september_cpi_from_receipts(receipts_xlsx_bytes) + if receipts_xlsx_bytes is not None + else (None, {}) + ) + gaps: dict[str, dict[int, ForecastGap]] = {} + for spec in STATUTORY_GAP_SPECS: + values = series[spec.series_key] + calendar = calendar_values.get(spec.series_key, {}) + table = next(s.table for s in SERIES_SPECS if s.key == spec.series_key) + gaps[spec.key] = {} + for year in range(forecast_start_year, forecast_start_year + forecast_years): + annual = calendar.get(year) + if annual is None: + continue + if spec.key == "cpi_september" and year in september_cpi: + statutory = september_cpi[year] + source = ( + f"September CPI (receipts Table {september_table}) minus " + "calendar-year CPI growth in yoy_growth.yaml" + ) + reference = ( + f"detailed forecast tables, receipts, Table {september_table}" + ) + else: + statutory = values.get(f"{year}Q{spec.quarter}") + if statutory is None: + continue + source = ( + f"{spec.description} (economy Table {table}) minus " + "calendar-year growth in yoy_growth.yaml" + ) + reference = f"detailed forecast tables, economy, Table {table}" + gaps[spec.key][year] = ForecastGap( + value=round(statutory - annual, 5) + 0.0, + source=source, + reference_title=reference, + ) + return gaps + + +def read_calendar_values(yaml_path: Path) -> dict[str, dict[int, float]]: + """Calendar-year growth by year for each series in a yoy_growth.yaml.""" + obr = yaml.safe_load(yaml_path.read_text())["obr"] + return { + key: { + int(str(instant)[:4]): float(value) + for instant, value in series["values"].items() + } + for key, series in obr.items() + if isinstance(series, dict) and "values" in series + } + + +def format_gap(value: float) -> str: + """Fixed-point text: YAML 1.1 reads exponent notation such as 3e-05 as a + string.""" + if value == 0: + return "0" + text = f"{value:.5f}".rstrip("0") + return text + "0" if text.endswith(".") else text + + +def render_forecast_gap_values( + spec: StatutoryGapSpec, gaps: dict[int, ForecastGap], month: str, year: int +) -> str: + """The values block of a forecast_gap YAML file.""" + if not gaps: + raise ValueError(f"No forecast gaps for {spec.key}") + lines = [ + "values:", + " # No gap applies to published years.", + f" 2010-{spec.month_day}: 0", + ] + source = None + for gap_year in sorted(gaps): + if gaps[gap_year].source != source: + source = gaps[gap_year].source + lines.append(f" # OBR EFO {month} {year}: {source}.") + lines.append( + f" {gap_year}-{spec.month_day}: {format_gap(gaps[gap_year].value)}" + ) + lines += [ + " # After the EFO horizon: no gap, so the forecast is calendar-year growth.", + f" {max(gaps) + 1}-{spec.month_day}: 0", + ] + return "\n".join(lines) + "\n" + + +def render_forecast_gap_references( + spec: StatutoryGapSpec, gaps: dict[int, ForecastGap], month: str, year: int +) -> str: + table = next(s.table for s in SERIES_SPECS if s.key == spec.series_key) + titles = [f"detailed forecast tables, economy, Table {table}"] + for gap in gaps.values(): + if gap.reference_title not in titles: + titles.append(gap.reference_title) + lines = [" reference:"] + for title in titles: + lines += [ + f" - title: OBR EFO {month} {year} ({title})", + f" href: {build_efo_href(month, year)}", + ] + return "\n".join(lines) + "\n" + + +def render_forecast_gap_yaml( + content: str, + spec: StatutoryGapSpec, + gaps: dict[int, ForecastGap], + month: str, + year: int, +) -> str: + """Rewrite the values block and the reference list (the last metadata key), + and check every value loads as a number.""" + values_pattern = re.compile( + r"^values:\n.*?(?=^metadata:)", re.MULTILINE | re.DOTALL + ) + content, values_count = values_pattern.subn( + lambda _: render_forecast_gap_values(spec, gaps, month, year), + content, + count=1, + ) + reference_pattern = re.compile(r"^ reference:\n.*\Z", re.MULTILINE | re.DOTALL) + content, reference_count = reference_pattern.subn( + lambda _: render_forecast_gap_references(spec, gaps, month, year), + content, + count=1, + ) + if not (values_count and reference_count): + raise ValueError(f"Could not find the values and reference for {spec.key}") + loaded = yaml.safe_load(content)["values"] + bad = {k: v for k, v in loaded.items() if not isinstance(v, (int, float))} + if bad: + raise ValueError(f"Non-numeric forecast gaps for {spec.key}: {bad}") + return content + + +def update_forecast_gap_yaml( + yaml_path: Path, + spec: StatutoryGapSpec, + gaps: dict[int, ForecastGap], + month: str, + year: int, +) -> None: + yaml_path.write_text( + render_forecast_gap_yaml(yaml_path.read_text(), spec, gaps, month, year) + ) + + def infer_release(source_name: str) -> tuple[str, int]: match = MONTH_RE.search(source_name) if not match: @@ -341,7 +630,26 @@ def update_yoy_growth_yaml( forecast_start_year: int, forecast_years: int, ) -> None: - content = yaml_path.read_text() + yaml_path.write_text( + render_yoy_growth_yaml( + yaml_path.read_text(), + series_values, + month, + year, + forecast_start_year, + forecast_years, + ) + ) + + +def render_yoy_growth_yaml( + content: str, + series_values: dict[str, dict[int, float]], + month: str, + year: int, + forecast_start_year: int, + forecast_years: int, +) -> str: yaml.safe_load(content) forecast_end_year = forecast_start_year + forecast_years - 1 href = build_efo_href(month, year) @@ -378,7 +686,8 @@ def update_yoy_growth_yaml( ) content = replace_series_section(content, spec.key, section) - yaml_path.write_text(content) + yaml.safe_load(content) + return content def print_summary( @@ -439,6 +748,30 @@ def build_arg_parser() -> argparse.ArgumentParser: default=6, help="Number of forecast years to update (default: 6)", ) + parser.add_argument( + "--receipts-url", + help=( + "OBR receipts detailed forecast tables (XLSX), for September CPI; " + "without it the September CPI gap uses Q3 CPI" + ), + ) + parser.add_argument("--receipts-file", help="Local receipts XLSX file path") + parser.add_argument( + "--gaps-only", + action="store_true", + help=( + "Leave yoy_growth.yaml as it is and regenerate the forecast gaps " + "against the calendar-year growth it holds" + ), + ) + parser.add_argument( + "--skip-statutory-gaps", + action="store_true", + help=( + "Do not update the statutory uprating input forecast gaps " + "(September CPI, May-July earnings)" + ), + ) parser.add_argument( "--dry-run", action="store_true", @@ -447,6 +780,61 @@ def build_arg_parser() -> argparse.ArgumentParser: return parser +def write_all_or_none(outputs: dict[Path, str]) -> None: + """Write every file or leave all of them as they were. + + Each new file is staged beside its target and each existing target is + copied to a backup beside it before anything moves. Targets are then + replaced by atomic moves. On any failure, each replaced target is put + back by an atomic move from its backup, which never truncates it, so a + second failure cannot leave a file empty. If a restore fails, its backup + is kept and named in the error. Growth and gaps never mix forecasts + silently. + """ + staged: dict[Path, Path] = {} + backups: dict[Path, Path] = {} + replaced: list[Path] = [] + keep: set[Path] = set() + try: + for path, content in outputs.items(): + staging = path.with_name(f".{path.name}.staged") + staging.write_text(content) + staged[path] = staging + for path in outputs: + if path.exists(): + backup = path.with_name(f".{path.name}.backup") + shutil.copy2(path, backup) + backups[path] = backup + for path, staging in staged.items(): + os.replace(staging, path) + replaced.append(path) + except BaseException as error: + unrestored = [] + for path in replaced: + try: + if path in backups: + os.replace(backups[path], path) + else: + path.unlink() + except OSError: + unrestored.append(path) + if unrestored: + keep = {backups[path] for path in unrestored if path in backups} + raise OSError( + "Could not restore " + + ", ".join(str(path) for path in unrestored) + + "; the original text is kept in " + + ", ".join(str(backup) for backup in sorted(keep)) + ) from error + raise + finally: + for staging in staged.values(): + staging.unlink(missing_ok=True) + for backup in backups.values(): + if backup not in keep: + backup.unlink(missing_ok=True) + + def main(argv: list[str] | None = None) -> int: args = build_arg_parser().parse_args(argv) @@ -467,24 +855,64 @@ def main(argv: list[str] | None = None) -> int: forecast_start_year = args.forecast_start_year or infer_forecast_start_year( month, year ) + source_yaml_path = args.yaml_path or get_yoy_growth_path() + write_gaps = not args.skip_statutory_gaps and args.output is None + if args.gaps_only and not write_gaps: + raise ValueError("--gaps-only cannot be combined with --output") + if write_gaps and not (args.receipts_url or args.receipts_file): + message = ( + "Pass --receipts-file or --receipts-url so September CPI uses the " + "OBR's September forecast, or --skip-statutory-gaps" + ) + if not args.dry_run: + raise ValueError(message) + print(f"Warning: {message}") + print_summary(series_values, forecast_start_year, args.forecast_years) + receipts_bytes = None + if args.receipts_url or args.receipts_file: + _, receipts_bytes = load_source_bytes(args.receipts_url, args.receipts_file) + calendar_values = ( + read_calendar_values(source_yaml_path) if args.gaps_only else series_values + ) + statutory_gaps = compute_statutory_forecast_gaps( + workbook_bytes, + forecast_start_year, + args.forecast_years, + receipts_bytes, + calendar_values, + ) + for key, gaps in statutory_gaps.items(): + window = ", ".join( + f"{y}: {format_gap(g.value)}" for y, g in sorted(gaps.items()) + ) + print(f"- forecast gap {key}: {window}") + + # Render and check every file before writing any, so a failure cannot + # leave growth and gaps from different forecasts. + outputs: dict[Path, str] = {} + if not args.gaps_only: + outputs[args.output or source_yaml_path] = render_yoy_growth_yaml( + source_yaml_path.read_text(), + series_values, + month, + year, + forecast_start_year, + args.forecast_years, + ) + if write_gaps: + gap_dir = get_forecast_gap_dir(source_yaml_path) + for spec in STATUTORY_GAP_SPECS: + gap_path = gap_dir / f"{spec.key}.yaml" + outputs[gap_path] = render_forecast_gap_yaml( + gap_path.read_text(), spec, statutory_gaps[spec.key], month, year + ) if args.dry_run: return 0 - - source_yaml_path = args.yaml_path or get_yoy_growth_path() - target_yaml_path = args.output or source_yaml_path - if target_yaml_path != source_yaml_path: - target_yaml_path.write_text(source_yaml_path.read_text()) - update_yoy_growth_yaml( - yaml_path=target_yaml_path, - series_values=series_values, - month=month, - year=year, - forecast_start_year=forecast_start_year, - forecast_years=args.forecast_years, - ) - print(f"Updated {target_yaml_path}") + write_all_or_none(outputs) + for path in outputs: + print(f"Updated {path}") return 0