Uprate electricity_consumption and gas_consumption

[vahid-ahmadi]

Fixes #1859.

Root cause

The class-level uprating = "..." attribute on a Variable is dead metadata: nothing in policyengine-uk reads it, and policyengine-core only inspects it to refuse combining it with its own @uprated decorator. data/uprating_indices.yaml is the only mechanism that actually applies uprating (apply_single_year_uprating / reset_growthfactor_uprating in data/economic_assumptions.py).

So a variable can declare an index and silently never be uprated. That is exactly what happened here.

The bug

electricity_consumption and gas_consumption both declare

uprating = "gov.economic_assumptions.indices.obr.consumer_price_index"

in variables/input/consumption/energy.py, but they were missing from data/uprating_indices.yaml — the list reset_growthfactor_uprating actually reads. So they returned identical values in every year while every other consumption variable on the same index rose with CPI.

Effect, on enhanced_frs_2024_25

variable	2024	2027 before	2027 after
electricity_consumption	838.29	838.29 (+0.0%)	904.47 (+7.9%)
gas_consumption	658.36	658.36 (+0.0%)	710.33 (+7.9%)
domestic_energy_consumption	1496.65	1614.79 (+7.9%)	1614.79 (unchanged)

It also fixes the second symptom in the issue

domestic_energy_consumption is documented as "Combined gas and electric bills". In the raw dataset it equals electricity_consumption + gas_consumption exactly — I verified this directly against the HDF5. But after uprating they diverged by 7.9% from 2025 onward, because only the combined variable was in the list.

With this change they agree in every year: elec+gas = 1614.79 = domestic in 2027. So the two symptoms had one cause, and there is no separate calibration difference to reconcile.

Why it matters

2024 was near the tail of the energy crisis — gov.ofgem.energy_price_cap falls from £2,389 (2024) to £1,641 (2026 onward) — so a frozen 2024 energy base is materially mis-levelled against later years. Any downstream study using these variables carried a 2024-priced energy baseline labelled as its target year. We hit this in a cost-of-living study where the energy channel is the largest single component.

Verification

• 219 passed, 44 skipped — full policyengine_uk/tests suite, no failures.
• Before/after measured in a clean virtualenv on the certified dataset.
• The two entries are inserted alphabetically within the CPI block, matching the surrounding style. The list as a whole was already not fully sorted, so nothing else is reordered.

CI

The red Test job is not this PR: all failures are PermissionError: file /Users/runner/.cache/huggingface/hub/blobs/... exists but it can not be written at Microsimulation() construction — the macOS runner's HF cache. I counted 56 such errors in the failed log and zero mentions of electricity_consumption, gas_consumption or uprating_indices. Re-run requested.

Same gap elsewhere

Four more variables declare uprating but are missing from the YAML, and measure flat 2024 → 2027: water_and_sewerage_charges, housing_service_charges, pension_contributions_via_salary_sacrifice, rail_usage (plus an index mismatch on savings_interest_income). Filed separately as #1862 with a suggested code-health guard, to keep this PR to the energy fix.

Note on scope

This changes modelled energy spending in every year after the base year, so downstream results that use these variables will move by roughly CPI. That is the intended correction, but it is worth flagging for anyone with pinned regression values.

🤖 Generated with Claude Code

[juaristi22]

Verified locally on the cached enhanced_frs_2024_25 (household means, 2024 → 2027):

• before: electricity_consumption 838.45 → 838.45 (0.0 %), gas_consumption 658.40 → 658.40 (0.0 %)
• after: 838.45 → 904.64 (+7.9 %), 658.40 → 710.38 (+7.9 %); electricity + gas = domestic_energy_consumption = 1615.01 in 2027

So the diff does what it says and the documented identity is restored.

CI. The red Test job is not this PR. All 28 failures (3 failed + 25 errors) are the same PermissionError: file /Users/runner/.cache/huggingface/hub/blobs/... exists but it can not be written at Microsimulation() construction, i.e. the macOS runner's HF cache. main's last run (2026-09-16) and every other PR branch this month were green on the same job, so a re-run should clear it.

Root cause, worth stating in the PR body. The class-level uprating = "..." attribute is dead metadata: nothing in policyengine-uk reads it, and policyengine-core only looks at it to refuse combining it with its own @uprated decorator. data/uprating_indices.yaml is the only mechanism (apply_single_year_uprating / reset_growthfactor_uprating in data/economic_assumptions.py). A declaration without a YAML entry therefore does nothing, which is exactly what #1859 observed.

Same gap elsewhere — see the inline comment on the YAML. Happy for that to be a follow-up issue rather than scope creep here.

Downstream note: the populace UK build levels LCFS energy to the FY2024 base year and binds the NEED/DESNZ targets there, so base-year targets don't move; projected-year energy outputs from populace datasets will rise with CPI once this lands, which is the intended correction.

[juaristi22]

Four more dataset columns declare uprating but are missing from this list, and I measured them flat 2024 → 2027 on the same dataset:

• water_and_sewerage_charges (declares indices.ofwat.water_bills) — 375.34 in both years
• housing_service_charges (indices.obr.lagged_average_earnings) — 91.89
• pension_contributions_via_salary_sacrifice (indices.obr.average_earnings) — 417.07
• rail_usage (gov.dft.rail.ridership_index) — 483.10

Also savings_interest_income declares indices.ons.household_interest_income, but this file applies per_capita.gdp to it.

Since the class attribute is never read, the durable fix is a code-health test (natural home: policyengine_uk/tests/code_health/) asserting that every variable declaring uprating appears in this file under the matching index. Two of the four above point at indices that may have no yoy_growth parameter yet, so they probably need their own PR — but the guard would make the gap visible instead of silent.

[vahid-ahmadi]

Thanks — the root-cause framing is better than mine and I've moved it into the PR body: the class-level uprating attribute is dead metadata, and uprating_indices.yaml is the only mechanism, so a declaration without a YAML entry does nothing. That is the generalisable point, and it explains why the declaration in energy.py looked correct.

CI. I verified your diagnosis independently before accepting it: 56 PermissionError hits on the runner's HF cache in the failed log, and zero mentions of electricity_consumption, gas_consumption or uprating_indices. Re-run requested.

The other four. Reproduced all of them on enhanced_frs_2024_25 — water_and_sewerage_charges 375.51, housing_service_charges 91.81, pension_contributions_via_salary_sacrifice 418.69, rail_usage 515.59, each identical in 2024 and 2027. Filed as #1862 with the savings_interest_income index mismatch and your code-health-test suggestion, rather than widening this PR. You're right that two of them point at indices which may have no yoy_growth parameter yet, so they need their own change.

I'd agree the code-health guard is the durable fix: it would have caught this at the point it was introduced, rather than in a downstream study three data releases later.

On the downstream note — that is exactly our situation. Our cost-of-living study has the energy channel as its largest component, and the baseline it multiplies was carrying 2024 prices labelled 2027-28. Once this lands we regenerate, and the headline moves by roughly CPI.

[vahid-ahmadi]

Test job: three re-runs, three identical failures — it is the runner's Hugging Face cache, not this PR. I am not going to re-run it a fourth time; flagging for whoever owns the workflow instead.

Evidence:

• 56 errors, all the same message, and all pointing at a single blob: PermissionError: file /Users/runner/.cache/huggingface/hub/blobs/19/19cb8246dc50358976b2344dfdc981b24a3be2d01f3c69fb2e34f3c86b737e7e exists but it can not be written. One distinct hash across all 56.
• Zero occurrences of electricity_consumption, gas_consumption or uprating_indices anywhere in the failed logs.
• The failures are in test_reform_impacts.py, test_uc_deductions_aggregates.py and test_behavioral_responses.py — all failing at Microsimulation() construction, before any variable this PR touches is evaluated.
• It is not specific to this branch. cgt-schedules-1858 hit the same failure at 10:27 today and passed on re-run at 10:55 (pr_code_changes.yaml). So the cache does recover; my three attempts happened to land while it was broken.
• The affected tests pass locally on this branch: 219 passed, 44 skipped.

It is a write-permission failure on an already-present cached blob, so an HF token would not change anything — the fix is on the runner side, most likely a restored cache with bad ownership, or not restoring the HF cache across runs.

Happy to rebase on main if that would pick up a cache-key change, but I do not think there is anything to fix in the diff.

[juaristi22]

Correction to my earlier review: this is not a runner flake, and re-runs won't help.

huggingface_hub 1.32.0 (released 2026-09-17) introduced a shared blob store that keeps downloads read-only under hub/blobs/<prefix>/<hash> — the exact path in every error here — and dataset_schema.py opened HDF5 files in pandas' default append mode. The last green PR run on 2026-09-16 installed hub 1.31.0; every run today installed 1.32.0. So the failure is deterministic on any branch without the loader fix.

The fix (mode="r" in dataset_schema.py) reached main today via #1861 (b1e5305). The cgt-schedules-1858 branch went green at 10:55 on that commit, not from cache recovery — its 10:27 failure was on the commit before it. This PR is based on 709a9b6, before the fix, which is why three re-runs failed identically.

Verified the post-rebase state locally. Cherry-picking this PR onto current main applies cleanly (#1861's edit to uprating_indices.yaml is in the per_capita.gdp block). On that tree the full policyengine_uk/tests suite passes with zero failures, and the five modules that were red here — test_reform_impacts, test_uc_deductions_aggregates, test_salary_sacrifice_cap_reform, test_state_pension_reform, test_behavioral_responses — pass 37/37 with the dataset token set.

Approving on the condition of a rebase onto main. The diff is correct and needs no change; please rebase (or merge main), let the run go green, and merge then. #1862 stands for the other four columns.

[vahid-ahmadi]

Rebased onto main (c71d5ea) — thank you for the correction, and I was wrong.

I had concluded "flaky runner cache" from the fact that cgt-schedules-1858 failed at 10:27 and passed at 10:55. You found the actual cause: that branch went green because it contained #1861's mode="r" fix, not because a cache recovered. huggingface_hub 1.32.0 landed on 2026-09-17 with a read-only shared blob store, and dataset_schema.py was opening HDF5 files in pandas' default append mode — so it was deterministic on any branch without the fix, and my three re-runs were never going to help.

Verified your explanation before acting on it: mode="r" is on main at lines 56 and 198 of dataset_schema.py, via 06e81c0, and my branch was based on 709a9b6 — before it.

Post-rebase, as you predicted:

• Both changes survive: electricity_consumption and gas_consumption in the CPI block, Charge BADR, residential property and carried interest gains at their own CGT schedules #1861's capital_gains_badr / capital_gains_residential_property entries intact in the per_capita.gdp block.
• Full suite 262 passed, 45 skipped — up from 219 passed here, since the modules that were erroring now run.

The lesson I'll take: "it passed on re-run elsewhere" is weak evidence for a flake when the two runs sat on different base commits. I should have diffed the bases before concluding infrastructure.

Waiting on the run to go green, then this is ready to merge. #1862 still stands for the other four columns and the code-health guard.