Project household energy spending with retail energy price indices, not CPI

.gitignore

@@ -56,6 +56,7 @@ policyengine_uk/calibration/*.h5
 !docs/**/*.ipynb
 !uprating_growth_factors.csv
 !policyengine_uk/parameters/gov/dwp/LHA/lha_published_rates_sources.csv
+!policyengine_uk/tests/fixtures/energy_prices/*.csv
 
 # Virtual environments
 .venv/

changelog.d/energy-retail-price-index.changed.md

@@ -0,0 +1 @@
+Project household electricity and gas spending with retail energy price indices instead of headline CPI. The indices follow the ONS CPI electricity, gas and solid fuel indices to the latest month and the OBR's utilities inflation forecast after it, and leave out the electricity VAT cut and its extension, so a VAT reform no longer moves projected spending.

docs/book/assumptions/growthfactors.md

@@ -18,7 +18,7 @@ We apply CPI to these variables:
 - All reported benefits– `carers_allowance_reported`, `child_benefit_reported`, `child_tax_credit_reported`, `esa_contrib_reported`, `esa_income_reported`, `housing_benefit_reported`, `iidb_reported`, `incapacity_benefit_reported`, `income_support_reported`, `jsa_contrib_reported`, `jsa_income_reported`, `maternity_allowance_reported`, `pension_credit_reported`, `sda_reported`, `state_pension_reported`, `universal_credit_reported`, `winter_fuel_allowance_reported`, `working_tax_credit_reported`
 - All consumption categories EXCEPT energy– `alcohol_and_tobacco_consumption`, `clothing_and_footwear_consumption`, `communication_consumption`, `education_consumption`, `food_and_non_alcoholic_beverages_consumption`, `health_consumption`, `household_furnishings_consumption`, `housing_water_and_electricity_consumption`, `miscellaneous_consumption`, `recreation_consumption`, `restaurants_and_hotels_consumption`, `transport_consumption`
 
-`domestic_energy_consumption`, `electricity_consumption` and `gas_consumption` are deliberately excluded. The data build calibrates them to NEED mean kWh and converts to pounds at Ofgem Q2 2026 unit rates, so the stored values already carry FY26/27 price levels; uprating them from the dataset's data year would re-apply price changes that are already included. See PolicyEngine/policyengine-uk#1867. `liquid_fuel_consumption` and `solid_fuel_consumption` (heating oil, LPG, coal and wood) are not covered by Ofgem's price cap, so they are CPI-uprated from the data year like other LCFS spending.
+`domestic_energy_consumption`, `electricity_consumption` and `gas_consumption` are deliberately excluded. The data build calibrates them to NEED mean kWh and converts to pounds at Ofgem Q2 2026 unit rates, so the stored values already carry FY26/27 price levels; uprating them from the dataset's data year would re-apply price changes that are already included. See PolicyEngine/policyengine-uk#1867. A dataset that declares its energy price year instead projects them with household electricity and gas price indices (`gov.economic_assumptions.energy_prices`): ONS CPI prices to the latest outturn, then the OBR's utilities inflation forecast, holding VAT on electricity at 5% so the VAT formulas apply any change in the rate. See the economic assumptions README. `liquid_fuel_consumption` and `solid_fuel_consumption` (heating oil, LPG, coal and wood) are not covered by Ofgem's price cap, so they are CPI-uprated from the data year like other LCFS spending.
 
 - Other variables– `afcs_reported`, `bsp_reported`, `childcare_expenses`, `free_school_fruit_veg`, `free_school_meals`, `free_school_milk`, `maintenance_expenses`, `statutory_maternity_pay`, `statutory_paternity_pay`, `statutory_sick_pay`, `state_pension`, `war_widows_pension_reported`, `widowed_parents_allowance_reported`
 

docs/engineering/skills/dataset-sources.md

@@ -142,8 +142,10 @@ Do not infer its price year from the dataset year, filename, URL or file format.
 Legacy builds can hold FY2026-27 energy prices in a dataset labelled 2024.
 
 An explicit declaration opts only `domestic_energy_consumption`,
-`electricity_consumption` and `gas_consumption` into the existing generic annual
-OBR consumer-price-index assumption as an interim price proxy. Other generic
+`electricity_consumption` and `gas_consumption` into the household energy
+price indices, `gov.economic_assumptions.energy_prices`: ONS CPI electricity
+and gas prices to the latest outturn, then the OBR's utilities forecast,
+holding VAT at the rate the inputs assume. Other generic
 uprating exclusions do not join this declaration or its reset group. For an
 input whose producer receipt declares calendar-2024 prices, a caller can label
 a local UK single-year H5 before constructing a simulation:
@@ -164,22 +166,23 @@ financial year. Both keys are required. The year must be an integer from 1
 through 9999; malformed or conflicting caller/file declarations raise a clear
 error. A caller's `None` leaves an existing file declaration intact.
 
-The model retains the period kind as provenance but uses the January 1 index
-level of the starting year for both calendar and fiscal price periods.
-Projection multiplies the supplied amount by `I[target year] / I[price year]`.
-This shared start-year convention approximates calendar/fiscal alignment;
-generic CPI does not establish constant-consumption energy-price outturns or
-an energy-specific forecast. Open [issue #2526](https://github.com/PolicyEngine/policyengine-uk/issues/2526)
-tracks energy-specific index alignment and a sourced future assumption. The
-original observation remains unchanged; a projected child's declaration records
-its effective fiscal-year price basis so later extension does not apply the
-original ratio twice.
+The period kind selects the index: a `calendar_year` basis reads the January to
+December level of its year, a `fiscal_year` basis the April to March level
+starting in its year. The target is always the fiscal year level, since model
+year Y is fiscal year Y–(Y+1). Projection multiplies the supplied amount by
+`I[target year] / I[price year]`, holding consumption fixed. The economic
+assumptions
+[README](../../../policyengine_uk/parameters/gov/economic_assumptions/README.md#household-energy-prices)
+describes the indices, their sources and the VAT adjustment.
+The original observation remains unchanged; a projected child's declaration
+records its effective fiscal-year price basis so later extension does not apply
+the original ratio twice.
 
 When a declaration projects stored energy inputs, both January 1 dates must
-fall on or after the selected index's earliest non-null value. An earlier
-basis or target, including January 1 before a midyear start, raises a clear
-error rather than using Core's backward first-value fallback. An index with
-no usable history also raises an error for that projection. Unlabelled
+fall on or after the selected index's first stated value (1 January 2015). An
+earlier basis or target raises a clear error rather than using Core's backward
+first-value fallback. An index with no values also raises an error for that
+projection. Unlabelled
 observations and declarations with no relevant stored inputs retain their
 existing behaviour without requiring index history.
 

policyengine_uk/data/economic_assumptions.py

@@ -166,7 +166,13 @@ def _unmapped_uprating_indices(uprating):
 SOCIAL_RENT_GROWTH = "gov.economic_assumptions.yoy_growth.obr.social_rent"
 PRIVATE_RENT_GROWTH = "gov.economic_assumptions.yoy_growth.ons.private_rental_prices"
 RESIDENTIAL_PROPERTY_INDEX = "gov.economic_assumptions.indices.obr.house_prices"
-ENERGY_PRICE_INDEX = "gov.economic_assumptions.indices.obr.consumer_price_index"
+ENERGY_PRICES = "gov.economic_assumptions.energy_prices"
+# The annual energy price index that projects each declared input.
+ENERGY_PRICE_SERIES = {
+    "electricity_consumption": "electricity",
+    "gas_consumption": "gas",
+    "domestic_energy_consumption": "electricity_gas_and_solid_fuels",
+}
 RESIDENTIAL_PROPERTY_INPUTS = (
     "main_residence_value",
     "other_residential_property_value",
@@ -184,7 +190,7 @@ def uprating_input_paths() -> list:
     ``apply_single_year_uprating`` and the functions it calls read these, and
     no other parameters: the growth rates in ``uprating_indices.yaml``, the
     council tax and rent series, the cumulative residential property index,
-    the conditional energy CPI index, the indices of the input variables that
+    the annual energy price indices, the indices of the input variables that
     file leaves out (after ``LEGACY_PROJECTION_END_YEAR``) and the capital
     gains projection's metadata (``CGT_INDEX_PATH``). A path may name a node
     (the regional private rent series), which covers its parameters.
@@ -201,7 +207,8 @@ def uprating_input_paths() -> list:
             PRIVATE_RENT_GROWTH,
             CGT_INDEX_PATH,
             RESIDENTIAL_PROPERTY_INDEX,
-            ENERGY_PRICE_INDEX,
+            f"{ENERGY_PRICES}.calendar_year",
+            f"{ENERGY_PRICES}.fiscal_year",
         }
         | set(_unmapped_uprating_indices(uprating).values())
     )
@@ -460,52 +467,90 @@ def uprate_energy(
     observed_year: UKSingleYearDataset,
     parameters: ParameterNode,
 ) -> UKSingleYearDataset:
-    """Project declared energy prices with the existing annual CPI assumption."""
+    """Project declared energy spending with the energy price indices.
+
+    Each input moves by its fuel's index (``ENERGY_PRICE_SERIES``) from the
+    period its prices describe (the calendar or fiscal year index, by the
+    declared basis) to the projected fiscal year. The indices hold VAT at the
+    rate the inputs assume, so the spending carries no VAT change, and the
+    VAT formulas apply the rate in force. An undeclared basis holds the
+    amounts flat.
+    """
     basis = observed_year.energy_price_basis
     variables = [
         name
         for name in DECLARED_ENERGY_INPUTS
         if name in observed_year.household.columns
         and name in current_year.household.columns
     ]
-    ratio = 1
-    projected = False
-    if basis is not None and variables:
-        index = parameters.get_child(ENERGY_PRICE_INDEX)
-        source_instant = f"{basis['period_value']:04d}-01-01"
-        target_instant = f"{int(current_year.time_period):04d}-01-01"
-        supported_instants = [
-            value.instant_str
-            for value in getattr(index, "values_list", [])
-            if value.value is not None
+    if basis is None or not variables:
+        for variable in variables:
+            current_year.household[variable] = observed_year.household[
+                variable
+            ].to_numpy()
+        current_year.energy_price_basis = basis
+        return current_year
+    source_instant = f"{basis['period_value']:04d}-01-01"
+    target_instant = f"{int(current_year.time_period):04d}-01-01"
+    # Check every ratio before changing any amount.
+    ratios = {
+        variable: _energy_price_ratio(
+            parameters,
+            f"{ENERGY_PRICES}.{basis['period_type']}.{ENERGY_PRICE_SERIES[variable]}",
+            source_instant,
+            f"{ENERGY_PRICES}.fiscal_year.{ENERGY_PRICE_SERIES[variable]}",
+            target_instant,
+        )
+        for variable in variables
+    }
+    for variable, ratio in ratios.items():
+        current_year.household[variable] = (
+            observed_year.household[variable].to_numpy() * ratio
+        )
+    current_year.energy_price_basis = {
+        "period_type": "fiscal_year",
+        "period_value": int(current_year.time_period),
+    }
+    return current_year
+
+
+def _energy_price_ratio(
+    parameters: ParameterNode,
+    source_path: str,
+    source_instant: str,
+    target_path: str,
+    target_instant: str,
+) -> float:
+    """Target index level over source level.
+
+    Each instant must be on or after its index's first stated value; after the
+    last, the last value holds, as in Core's variable uprating.
+    """
+    levels = []
+    for path, instant in ((source_path, source_instant), (target_path, target_instant)):
+        index = parameters.get_child(path)
+        stated = [
+            entry.instant_str
+            for entry in getattr(index, "values_list", [])
+            if entry.value is not None
         ]
-        if not supported_instants:
+        if not stated:
             raise ValueError(
-                f"Energy index {ENERGY_PRICE_INDEX!r} has no non-null history "
-                f"for declared basis {source_instant} and target {target_instant}."
+                f"Energy price index {path!r} has no values; the projection "
+                f"needs {instant}."
             )
-        first_supported = min(supported_instants)
-        if source_instant < first_supported or target_instant < first_supported:
+        if instant < min(stated):
             raise ValueError(
-                f"Energy index {ENERGY_PRICE_INDEX!r} first supports "
-                f"{first_supported}; declared basis {source_instant} and target "
-                f"{target_instant} must be on or after that instant."
+                f"Energy price index {path!r} starts at {min(stated)}; the "
+                f"projection needs {instant}."
             )
-        source_index = _uprating_index_value(index, source_instant)
-        target_index = _uprating_index_value(index, target_instant)
-        projected = source_index not in (None, 0) and target_index is not None
-        if projected:
-            ratio = target_index / source_index
-    for variable in variables:
-        current_year.household[variable] = (
-            observed_year.household[variable].to_numpy() * ratio
+        levels.append(_uprating_index_value(index, instant))
+    source, target = levels
+    if not source > 0:
+        raise ValueError(
+            f"Energy price index {source_path!r} is {source} at {source_instant}."
         )
-    current_year.energy_price_basis = (
-        {"period_type": "fiscal_year", "period_value": int(current_year.time_period)}
-        if projected
-        else basis
-    )
-    return current_year
+    return target / source
 
 
 def uprate_residential_property(

policyengine_uk/data/uprating_indices.yaml

@@ -1,9 +1,9 @@
 # Energy variables use the declaration-aware projector in economic_assumptions.py,
 # rather than an unconditional mapping here. Unlabelled dataset inputs remain
 # flat: legacy builds can already carry FY2026-27 prices despite a 2024 data year.
-# An explicit energy_price_basis opts into the variables' existing annual OBR-CPI
-# assumption from the declared price year. This does not supply an energy-specific
-# forecast or automatically label existing private Microcosm artifacts.
+# An explicit energy_price_basis opts into the household energy price indices
+# (gov.economic_assumptions.energy_prices) from the declared price year. This does
+# not automatically label existing private Microcosm artifacts.
 # liquid_fuel_consumption and solid_fuel_consumption (heating oil, LPG, coal
 # and wood) are not in that group: Ofgem's price cap does not cover them, so
 # they are uprated by CPI from the data year below, like other LCFS spending.