From 0e0217483bffafb28d5f2dc3b0f55988cb2b5e82 Mon Sep 17 00:00:00 2001 From: vahid-ahmadi Date: Mon, 21 Sep 2026 15:20:13 +0100 Subject: [PATCH 1/2] Stop CPI-uprating energy, which the data build already prices MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Reverses the direction of #1860, which I wrote. Fixes #1867. policyengine-uk-data calibrates the energy variables to NEED mean kWh by income band and converts to pounds at Ofgem Q2 2026 unit rates. Its imputations/consumption.py says so outright: "the stored £/yr values represent FY26/27 price levels. Energy variables are not CPI-uprated, so this avoids the need for price-level adjustment at simulation time." uprating_indices.yaml uprates from the dataset's DATA year, 2024. So applying CPI to these variables re-applies 2024-2026 price changes the build had already priced in — about 5.8pp of a 7.9% uplift. The cap parameter falls over this period, which makes the stored price level testable. On enhanced_frs_2024_25 the modelled mean of electricity + gas is £1,584, against cap parameters of £2,389 (2024), £1,849 (2025) and £1,641 (2026). A 2024-priced series would have to sit near the 2024 cap; it sits just below the 2026 one, which is the April 2026 level the docstring describes. #1859 read the symptom backwards and #1860 followed it. The flat pair was right — accidentally, being absent from the YAML rather than deliberately excluded — and domestic_energy_consumption, on the CPI list since before 2.90.2, was carrying the double count. #1860 restored the documented identity by propagating the defect to all three. Removing all three restores the identity AND holds the level: electricity £895 + gas £689 = £1,584 = domestic, in every year from 2024 to 2027. Other consumption variables come from the LCFS at data-year prices and do need uprating, so only these three are excluded. The rationale is recorded at the top of the YAML, and a code-health test asserts they appear under no index. Verified that test fails when one is re-added rather than only passing today. Downstream: modelled energy spending in projected years falls about 8%, back to the build's own price level. Anyone with pinned regression values on domestic_energy_consumption will see them move; that movement is the correction. Co-Authored-By: Claude Opus 5 (1M context) --- changelog.d/energy-not-cpi-uprated.md | 1 + policyengine_uk/data/uprating_indices.yaml | 17 ++++++-- .../code_health/test_energy_price_level.py | 42 +++++++++++++++++++ 3 files changed, 57 insertions(+), 3 deletions(-) create mode 100644 changelog.d/energy-not-cpi-uprated.md create mode 100644 policyengine_uk/tests/code_health/test_energy_price_level.py diff --git a/changelog.d/energy-not-cpi-uprated.md b/changelog.d/energy-not-cpi-uprated.md new file mode 100644 index 0000000000..b1090e4835 --- /dev/null +++ b/changelog.d/energy-not-cpi-uprated.md @@ -0,0 +1 @@ +- Stop CPI-uprating `domestic_energy_consumption`, `electricity_consumption` and `gas_consumption`. The data build calibrates them to NEED mean kWh at Ofgem Q2 2026 unit rates, so the stored values already carry FY26/27 price levels and uprating from the data year re-applied price changes that were already included. This reverses the direction of the fix in #1860, which resolved the #1859 inconsistency by uprating electricity and gas to match the aggregate rather than by removing all three. Modelled energy spending in projected years falls by about 8%. diff --git a/policyengine_uk/data/uprating_indices.yaml b/policyengine_uk/data/uprating_indices.yaml index ff0352ccdb..373eef9f49 100644 --- a/policyengine_uk/data/uprating_indices.yaml +++ b/policyengine_uk/data/uprating_indices.yaml @@ -1,3 +1,17 @@ +# Energy variables are deliberately ABSENT from every index below. +# +# policyengine-uk-data calibrates domestic_energy_consumption, +# electricity_consumption and gas_consumption to NEED mean kWh by income band +# and converts to pounds at Ofgem Q2 2026 unit rates, so the stored values +# already carry FY26/27 price levels. Its imputations/consumption.py says so +# outright: "the stored £/yr values represent FY26/27 price levels. Energy +# variables are not CPI-uprated, so this avoids the need for price-level +# adjustment at simulation time." +# +# Uprating them from the dataset's DATA year (2024) would therefore re-apply +# 2024-2026 price changes the build has already priced in (#1867). Other +# consumption variables come from the LCFS at data-year prices and do need +# uprating, which is why only these three are excluded. gov.economic_assumptions.yoy_growth.obr.average_earnings: - employee_pension_contributions - employer_pension_contributions @@ -16,16 +30,13 @@ gov.economic_assumptions.yoy_growth.obr.consumer_price_index: - childcare_expenses - clothing_and_footwear_consumption - communication_consumption -- domestic_energy_consumption - education_consumption -- electricity_consumption - esa_contrib_reported - esa_income_reported - food_and_non_alcoholic_beverages_consumption - free_school_fruit_veg - free_school_meals - free_school_milk -- gas_consumption - health_consumption - household_furnishings_consumption - housing_benefit_reported diff --git a/policyengine_uk/tests/code_health/test_energy_price_level.py b/policyengine_uk/tests/code_health/test_energy_price_level.py new file mode 100644 index 0000000000..6464c1d1ea --- /dev/null +++ b/policyengine_uk/tests/code_health/test_energy_price_level.py @@ -0,0 +1,42 @@ +"""Energy variables must not be CPI-uprated. + +policyengine-uk-data calibrates the energy variables to NEED mean kWh and +converts to pounds at Ofgem Q2 2026 unit rates, so the stored values already +carry FY26/27 price levels. Its imputations/consumption.py states that they +"are not CPI-uprated, so this avoids the need for price-level adjustment at +simulation time". + +Uprating them from the dataset's data year re-applies price changes the build +has already applied. #1859 reported the resulting inconsistency and #1860 +resolved it the wrong way round — by uprating electricity and gas to match +domestic_energy_consumption, rather than by removing all three (#1867). + +This guard stops that from happening again. +""" + +from pathlib import Path + +import yaml + +UPRATING_INDICES = Path(__file__).parents[2] / "data" / "uprating_indices.yaml" + +# Calibrated to NEED at Ofgem Q2 2026 rates, so already at FY26/27 prices. +ENERGY_VARIABLES_AT_BUILD_PRICE_LEVEL = { + "domestic_energy_consumption", + "electricity_consumption", + "gas_consumption", +} + + +def test_energy_variables_are_not_uprated_by_any_index(): + listed = yaml.safe_load(UPRATING_INDICES.read_text()) + offenders = sorted( + f"{variable} (under {index})" + for index, variables in listed.items() + for variable in variables + if variable in ENERGY_VARIABLES_AT_BUILD_PRICE_LEVEL + ) + assert not offenders, ( + "these variables are stored at the data build's own FY26/27 price " + f"level and must not be uprated: {offenders}. See #1867." + ) From c6bcc2deaf2b902365d7097ca9beb8d1af22391f Mon Sep 17 00:00:00 2001 From: vahid-ahmadi Date: Tue, 22 Sep 2026 10:01:51 +0100 Subject: [PATCH 2/2] Scope the energy exclusion to dataset projection, and fix the fragment MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Addresses the three findings on this PR. C1: the changelog fragment had no type component, so towncrier omitted it entirely — the draft rendered "No significant changes". Renamed to energy-not-cpi-uprated.fixed.md, per changelog.d/..md in CONTRIBUTING, and the leading bullet removed so towncrier does not double it. Verified the draft now renders it under Fixed. A1: the wording claimed more than the change delivers, and the reviewer is right. uprating_indices.yaml governs dataset projection. It does not govern the class-level `uprating` attribute, which policyengine-core still uses to carry a direct situation input forward: a 2024 electricity_consumption input queried at 2027 returns 1078.93 against 1000, which I confirmed directly. That path is deliberately left alone. A value a user supplies for 2024 may be their own contemporaneous bill rather than this build's artifact, so it has no claim on the build's price basis. The YAML header, the test docstring and the test name now say "projection" rather than making a global claim. Worth recording because #1860 and #1862 both described the class-level attribute as dead metadata that nothing reads. That is true of dataset projection and false of the direct-input path; core reads it there. A2: docs/book/assumptions/growthfactors.md still listed domestic_energy_consumption under "all consumption categories" get CPI. The list now excludes the three energy variables and states why. Full suite: 266 passed, 45 skipped. Co-Authored-By: Claude Opus 5 (1M context) --- changelog.d/energy-not-cpi-uprated.fixed.md | 1 + changelog.d/energy-not-cpi-uprated.md | 1 - docs/book/assumptions/growthfactors.md | 5 +++- policyengine_uk/data/uprating_indices.yaml | 17 +++++++++--- .../code_health/test_energy_price_level.py | 27 ++++++++++++------- 5 files changed, 36 insertions(+), 15 deletions(-) create mode 100644 changelog.d/energy-not-cpi-uprated.fixed.md delete mode 100644 changelog.d/energy-not-cpi-uprated.md diff --git a/changelog.d/energy-not-cpi-uprated.fixed.md b/changelog.d/energy-not-cpi-uprated.fixed.md new file mode 100644 index 0000000000..263ecfb1a2 --- /dev/null +++ b/changelog.d/energy-not-cpi-uprated.fixed.md @@ -0,0 +1 @@ +Stop projecting `domestic_energy_consumption`, `electricity_consumption` and `gas_consumption`. The data build calibrates them to NEED mean kWh at Ofgem Q2 2026 unit rates, so the stored values already carry FY26/27 price levels and projecting from the data year re-applied price changes that were already included. This reverses the direction of the fix in #1860, which resolved the #1859 inconsistency by uprating electricity and gas to match the aggregate rather than by removing all three. Modelled energy spending in projected years falls by about 8%. diff --git a/changelog.d/energy-not-cpi-uprated.md b/changelog.d/energy-not-cpi-uprated.md deleted file mode 100644 index b1090e4835..0000000000 --- a/changelog.d/energy-not-cpi-uprated.md +++ /dev/null @@ -1 +0,0 @@ -- Stop CPI-uprating `domestic_energy_consumption`, `electricity_consumption` and `gas_consumption`. The data build calibrates them to NEED mean kWh at Ofgem Q2 2026 unit rates, so the stored values already carry FY26/27 price levels and uprating from the data year re-applied price changes that were already included. This reverses the direction of the fix in #1860, which resolved the #1859 inconsistency by uprating electricity and gas to match the aggregate rather than by removing all three. Modelled energy spending in projected years falls by about 8%. diff --git a/docs/book/assumptions/growthfactors.md b/docs/book/assumptions/growthfactors.md index a6994f31ed..1b093f7b61 100644 --- a/docs/book/assumptions/growthfactors.md +++ b/docs/book/assumptions/growthfactors.md @@ -16,7 +16,10 @@ We use the OBR's CPI projections to drive most benefit and consumption variables We apply CPI to these variables: - All reported benefits– `carers_allowance_reported`, `child_benefit_reported`, `child_tax_credit_reported`, `esa_contrib_reported`, `esa_income_reported`, `housing_benefit_reported`, `iidb_reported`, `incapacity_benefit_reported`, `income_support_reported`, `jsa_contrib_reported`, `jsa_income_reported`, `maternity_allowance_reported`, `pension_credit_reported`, `sda_reported`, `state_pension_reported`, `universal_credit_reported`, `winter_fuel_allowance_reported`, `working_tax_credit_reported` -- All consumption categories– `alcohol_and_tobacco_consumption`, `clothing_and_footwear_consumption`, `communication_consumption`, `domestic_energy_consumption`, `education_consumption`, `food_and_non_alcoholic_beverages_consumption`, `health_consumption`, `household_furnishings_consumption`, `housing_water_and_electricity_consumption`, `miscellaneous_consumption`, `recreation_consumption`, `restaurants_and_hotels_consumption`, `transport_consumption` +- All consumption categories EXCEPT energy– `alcohol_and_tobacco_consumption`, `clothing_and_footwear_consumption`, `communication_consumption`, `education_consumption`, `food_and_non_alcoholic_beverages_consumption`, `health_consumption`, `household_furnishings_consumption`, `housing_water_and_electricity_consumption`, `miscellaneous_consumption`, `recreation_consumption`, `restaurants_and_hotels_consumption`, `transport_consumption` + +`domestic_energy_consumption`, `electricity_consumption` and `gas_consumption` are deliberately excluded. The data build calibrates them to NEED mean kWh and converts to pounds at Ofgem Q2 2026 unit rates, so the stored values already carry FY26/27 price levels; uprating them from the dataset's data year would re-apply price changes that are already included. See PolicyEngine/policyengine-uk#1867. + - Other variables– `afcs_reported`, `bsp_reported`, `childcare_expenses`, `free_school_fruit_veg`, `free_school_meals`, `free_school_milk`, `maintenance_expenses`, `statutory_maternity_pay`, `statutory_paternity_pay`, `statutory_sick_pay`, `state_pension` ## Road fuel volume diff --git a/policyengine_uk/data/uprating_indices.yaml b/policyengine_uk/data/uprating_indices.yaml index 373eef9f49..4fbc8bc819 100644 --- a/policyengine_uk/data/uprating_indices.yaml +++ b/policyengine_uk/data/uprating_indices.yaml @@ -1,5 +1,14 @@ # Energy variables are deliberately ABSENT from every index below. # +# SCOPE: this file governs DATASET PROJECTION only — the uprating applied to +# a dataset's stored values when it is projected to later years. It does not +# govern the class-level `uprating` attribute on the Variable, which +# policyengine-core still uses to carry a direct situation input forward from +# an earlier period. That path is unchanged and out of scope here: a 2024 +# value a user supplies directly may be their own contemporaneous bill rather +# than this data build's artifact, so it has no claim on the build's price +# basis (#1868 review A1). +# # policyengine-uk-data calibrates domestic_energy_consumption, # electricity_consumption and gas_consumption to NEED mean kWh by income band # and converts to pounds at Ofgem Q2 2026 unit rates, so the stored values @@ -8,10 +17,10 @@ # variables are not CPI-uprated, so this avoids the need for price-level # adjustment at simulation time." # -# Uprating them from the dataset's DATA year (2024) would therefore re-apply -# 2024-2026 price changes the build has already priced in (#1867). Other -# consumption variables come from the LCFS at data-year prices and do need -# uprating, which is why only these three are excluded. +# Projecting them from the dataset's DATA year (2024) would therefore +# re-apply 2024-2026 price changes the build has already priced in (#1867). +# Other consumption variables come from the LCFS at data-year prices and do +# need uprating, which is why only these three are excluded. gov.economic_assumptions.yoy_growth.obr.average_earnings: - employee_pension_contributions - employer_pension_contributions diff --git a/policyengine_uk/tests/code_health/test_energy_price_level.py b/policyengine_uk/tests/code_health/test_energy_price_level.py index 6464c1d1ea..3ff5a714b1 100644 --- a/policyengine_uk/tests/code_health/test_energy_price_level.py +++ b/policyengine_uk/tests/code_health/test_energy_price_level.py @@ -1,4 +1,4 @@ -"""Energy variables must not be CPI-uprated. +"""Energy variables must not be uprated by DATASET PROJECTION. policyengine-uk-data calibrates the energy variables to NEED mean kWh and converts to pounds at Ofgem Q2 2026 unit rates, so the stored values already @@ -6,12 +6,20 @@ "are not CPI-uprated, so this avoids the need for price-level adjustment at simulation time". -Uprating them from the dataset's data year re-applies price changes the build -has already applied. #1859 reported the resulting inconsistency and #1860 -resolved it the wrong way round — by uprating electricity and gas to match -domestic_energy_consumption, rather than by removing all three (#1867). - -This guard stops that from happening again. +Projecting them from the dataset's data year re-applies price changes the +build has already applied. #1859 reported the resulting inconsistency and +#1860 resolved it the wrong way round — by uprating electricity and gas to +match domestic_energy_consumption, rather than by removing all three (#1867). + +SCOPE. This guard covers uprating_indices.yaml, which governs dataset +projection. It deliberately does NOT cover the class-level `uprating` +attribute, which policyengine-core still uses to carry a direct situation +input forward from an earlier period. That path is unchanged: a value a user +supplies directly for 2024 may be their own contemporaneous bill rather than +this build's artifact, so it has no claim on the build's price basis. Worth +noting because #1860 and #1862 described that attribute as dead metadata — +it is inert for dataset projection, but core does read it for direct inputs +(#1868 review A1). """ from pathlib import Path @@ -28,7 +36,7 @@ } -def test_energy_variables_are_not_uprated_by_any_index(): +def test_energy_variables_are_not_projected_by_any_index(): listed = yaml.safe_load(UPRATING_INDICES.read_text()) offenders = sorted( f"{variable} (under {index})" @@ -38,5 +46,6 @@ def test_energy_variables_are_not_uprated_by_any_index(): ) assert not offenders, ( "these variables are stored at the data build's own FY26/27 price " - f"level and must not be uprated: {offenders}. See #1867." + f"level and must not be projected by an index: {offenders}. " + "See #1867." )