Uprate three more columns, and guard the gap that hid them

[vahid-ahmadi]

Follow-up to #1860, closing most of #1862. Thanks @juaristi22 for finding these and for suggesting the guard.

The three fixes

Each declared an uprating index but was absent from uprating_indices.yaml, so was never uprated (on main since #2376's projection fallback, it stays flat through 2030 and grows only after). On enhanced_frs_2024_25, 2024 → 2027:

variable	2024	2027	index compounded
water_and_sewerage_charges	375.51	525.77 (+40.0%)	ofwat water bills +40.0%
housing_service_charges	91.81	105.06 (+14.4%)	OBR lagged avg earnings +14.4%
pension_contributions_via_salary_sacrifice	418.69	466.81 (+11.5%)	OBR avg earnings +11.5%

The water row uses the published 2025-26 and 2026-27 bill increases (see Provenance below); it was 522.12 (+39.0%) with the old 26% and 5.5%. Each matches its own index exactly, which is the check that they are now uprated by the right thing rather than merely uprated.

All three indices already had the yoy_growth twin the YAML needs — your caution about missing parameters turned out not to apply to these — so this is a YAML-only change.

rail_usage is deliberately excluded

Its declared index is a cumulative level, not a year-on-year rate:

gov.dft.rail.ridership_index — "Cumulative index of rail ridership (passenger journeys). Base year 2020 = 1.0."

Values run 0.95 (2024) → 1.005 (2027). Putting it in a yoy_growth block compounds the level and uprates the variable by ~684% over 2024–27. I had it in the YAML at first and caught it on the before/after check. Uprating it properly needs a year-on-year series, which is more than a YAML entry.

The guard

policyengine_uk/tests/code_health/test_uprating_declarations.py asserts every variable declaring uprating appears in the YAML under the matching index — the durable fix you suggested, which would have caught #1859 at the point it was introduced.

I verified it fails when an entry is removed, rather than only passing today. (That check matters: a guard I wrote on the dashboard side turned out to pass regardless of the thing it claimed to prevent, so I no longer trust one I have not seen fail.)

Three exceptions are listed with reasons rather than silently tolerated:

• rail_usage — cumulative index, as above
• employee_pension_contributions_reported — reported counterpart feeding imputation
• savings_interest_income — the index mismatch you flagged; declares ons.household_interest_income while the YAML applies per_capita.gdp. Left as-is for someone with the context to say which is intended — I did not want to change an uprating basis on a guess.

A third test keeps those lists honest: each entry must still exist and still declare uprating, so a stale exception fails rather than lingering.

Since review (rebased onto main, 7 Oct)

• ruff format on the guard test; the changelog fragment is typed .fixed.md, so it renders under "Fixed".
• After the rebase the guard found 10 more variables added to main since this PR opened:
  • 7 real gaps, now uprated by the index each declares. Per-capita GDP: cash_isa, stocks_and_shares_isa, mortgage_debt, consumer_debt, rent_paid_as_boarder, rent_paid_as_lodger. Per-capita mixed income: self_employment_gross_receipts.
  • 3 deliberate: electricity_consumption, gas_consumption and domestic_energy_consumption are listed in KNOWN_UNUPRATED, because the data build prices energy spend (Stop CPI-uprating energy, which the data build already prices #1868).
• This goes slightly beyond the formatting and fragment fixes. Without it, the guard test fails on today's main.
• Checks at 8300b0a96: guard tests 3 passed (and fail with cash_isa removed); tests/code_health 1,953 passed; the YAML suite 2,605 passed; ruff format --check . and ruff check . clean.

Verification

265 passed, 45 skipped (262 before, plus the three new tests). Changelog fragment included.

Checks at 7ab0afa33: guard tests 8 passed; tests/code_health, the water test and the top-level uprating test files 2,314 passed; the YAML suite 3,058 passed; ruff format --check . and ruff check . clean; .github/check_fix_pr.py PASS against this description. The new water test fails with the old 0.26 and 0.055.

#1862 stays open for rail_usage and the savings_interest_income basis question.

Merge with main (14 Oct)

Merged origin/main (232 commits behind). Three things changed beyond the conflict:

• uprating_indices.yaml conflicted on obr.per_capita.mixed_income: main added trading_loss, this PR self_employment_gross_receipts. Both kept.
• The guard caught four inputs main added for UC reg 77 (owned_company_capital, owned_company_holding_value, owned_company_trade_assets under per-capita GDP, owned_company_income_share under per-capita mixed income). Listed under the index each declares.
• Main's projection now reads the uprating attribute to grow unlisted columns after 2030 (_unmapped_uprating_indices). The guard docstring no longer calls that attribute dead metadata, and test_unmapped_inputs_preserve_legacy_years_and_grow_after_2030 now asserts only the columns still unlisted (rail_usage, employee_pension_contributions_reported) are fallbacks, since the columns this PR lists grow every year instead.

Provenance of the series these columns now use (since review, 12 Oct)

Water bills (ofwat.water_bills). Values are charging years from April, stored at 1 January (2025 holds 2025-26).

• 2025-26: 27%, from Ofwat's statement of 29 January 2026. It replaces the 26% forecast in Ofwat's 30 January 2025 statement.
• 2026-27: 5.4%, from Water UK, 28 January 2026: national average £639, up £33. It replaces a 5.5% projection.
• 2027-2029 (4.6%, 4.5%, 4.5%): kept. They come from utils/water/forecast_water_bills.py (Improve water forecasts #1239). The script takes the customer-weighted mean of the CCW per-company table and adds a hardcoded CPIH path. The CCW page says that table shows Ofwat's PR24 draft determinations. The weights were in ofwat_increases.csv, which is gitignored and was never committed, so these values cannot be reproduced. Filed as Water bill projections for 2027-29 cannot be reproduced from the repo #2384.
• The update is two values in one parameter. It is in this PR because its only runtime consumer is water_and_sewerage_charges, which this PR starts uprating. create_ahc_deflator.py reads it offline and was not rerun. Effect: the 2026 and later water factors rise by 0.70% (1.27 × 1.054 against 1.26 × 1.055), about £3.50 a household a year on the mean above.

Population (ons.population), the denominator for per_capita.gdp and per_capita.mixed_income.

• 2022-2073 is the "Population growth" row of the OBR's Long-term economic determinants, March 2025 EFO (published 19 June 2025, brought in by Add OBR long-term economic determinants #1305). Fiscal year t/t+1 is stored at t-01-01, rounded to four decimals, and every year matches. Per the OBR's footnote, the row is the OBR's March 2025 forecast to 2029-30 and the ONS 2022-based migration category variant after that. The 2021 value has no recorded source.
• So the gap with the March 2026 workbook comes from vintage. The per-capita series are March 2026 nominal growth minus this March 2025 population. Calendar-year growth implied by the March 2026 economy tables (Table 1.1 real GDP over Table 1.5 real GDP per person) is 0.37% for 2025 against 0.72% stored, and 0.30-0.36% against 0.37-0.45% for 2026-2030. The ONS has since published 2024-based projections (28 April 2026).
• Bound. Re-deriving with the March 2026 implied population would raise the per-capita GDP and mixed income factors from 2024 by about 0.35% by 2025, 0.46% by 2027 and 0.73% by 2030. That is the most the seven newly uprated per-capita columns could move. household_weight is uprated by the same population series, so in the uprating step aggregates still grow with nominal GDP or mixed income. The vintage mainly shifts the split between amounts per household and household counts.
• Not changed here. ons.population drives household_weight and every per_capita.* series, so a refresh has to re-derive them together. Filed as Refresh ons.population and re-derive the per-capita series from one vintage #2383. The YAML now cites the dated release, the mapping and the vintage instead of the ONS homepage.

Reproducer: policyengine_uk/tests/code_health/test_uprating_declarations.py::test_declared_uprating_variables_are_in_the_yaml, policyengine_uk/tests/code_health/test_uprating_declarations.py::test_the_yaml_uprates_by_the_declared_index, policyengine_uk/tests/code_health/test_uprating_declarations.py::test_every_known_exception_still_holds, policyengine_uk/tests/code_health/test_uprating_declarations.py::test_the_guard_passes_exceptions_that_still_hold, policyengine_uk/tests/code_health/test_uprating_declarations.py::test_the_guard_flags_an_unuprated_exception_added_to_the_yaml, policyengine_uk/tests/code_health/test_uprating_declarations.py::test_the_guard_flags_an_override_that_now_matches, policyengine_uk/tests/code_health/test_uprating_declarations.py::test_the_guard_flags_an_override_dropped_from_the_yaml, policyengine_uk/tests/code_health/test_uprating_declarations.py::test_the_guard_flags_missing_or_undeclared_exceptions, policyengine_uk/tests/test_water_bills_growth.py::test_published_water_bill_growth, policyengine_uk/tests/test_water_bills_growth.py::test_water_index_compounds_the_published_years

axiom: n/a: uprating of survey input columns and economic-assumption provenance; no tax or benefit rule changes

[juaristi22]

Changes requested: clear the formatting failure and register the changelog fragment. Reviewed head: 747a1d1381cb4ac6c742fdd486f9233ec124ee79.

The uprating additions remain useful. Main 3c48247eb92b221312565bff3a1dbf0b87306e22 still omits pension_contributions_via_salary_sacrifice, housing_service_charges and water_and_sewerage_charges from its uprating map. This PR wires those columns to their already-declared indices. The three new guard tests pass, so the two concrete blockers below should be repairable without changing policy values or test expectations.

1. Format the new test file. The UPRATING_INDICES assignment fails the configured formatter. The executed check returned Would reformat: policyengine_uk/tests/code_health/test_uprating_declarations.py and 1 file would be reformatted, matching the failing Lint job. Apply uv run ruff format policyengine_uk/tests/code_health/test_uprating_declarations.py; retain all three tests, declared-index checks and documented exception sets.
2. Give the fragment a recognised type. uprate-four-remaining-columns.md omits the type required by Towncrier's configuration. The executed draft rendered “No significant changes.” Rename it to uprate-four-remaining-columns.fixed.md, preserving the repair description. The draft must then show that text in its Fixed section. Leave the generated CHANGELOG.md to the release workflow.

Acceptance commands from the repaired checkout:

uv run pytest policyengine_uk/tests/code_health/test_uprating_declarations.py -q
uv run ruff format --check policyengine_uk/tests/code_health/test_uprating_declarations.py
uv run ruff check policyengine_uk/tests/code_health/test_uprating_declarations.py
uv run towncrier build --draft --version 0.0.0

Require all three tests and both Ruff checks to pass, verify that Towncrier includes the fragment, and obtain green CI on the updated head. Keep this repair scoped to formatting and the fragment filename; changing the economic indices or removing exceptions requires separate evidence.

Validation performed: exact-snapshot guard tests 3 passed in 0.91s; formatter failure and ignored fragment independently reproduced. Execution used the existing Python 3.13.14/core 3.32.9 environment. The map was compared with current main and existing variable declarations. The guard confirms consistency with those declarations; it does not independently establish the economic choice of indices. No private-data projection or rebased integration run was performed, and the proposed repair has not been applied or tested.

Live posting check (2026-10-02T15:11:38.782329+00:00): same reviewed commit; GitHub reports MERGEABLE. Failing checks: Lint. These checks do not replace the remaining validation listed above.

[juaristi22]

Re-review of 8300b0a: the two requested blockers are addressed in the diff. The guard test is formatted, the changelog fragment now has the .fixed.md type, and the added mappings are placed under the variables’ declared indices. Lint and the four install/smoke jobs pass; the current Test job is still running, so this is a comment rather than an approval.

Should address: test_every_known_exception_still_declares_uprating checks only that an exception variable exists and still declares uprating. It does not check that a KNOWN_UNUPRATED entry remains absent from the YAML or that a KNOWN_INDEX_OVERRIDES entry still has an index mismatch. If an exception becomes stale, it can mask a later gap. Please check that each exception’s reason still holds, or remove it when resolved. This is a guard-coverage concern; I found no demonstrated current mapping error.

I could not independently run the guard, Towncrier draft, or private-dataset projections in this workspace. Please wait for the Test job before treating this re-review as complete.

[juaristi22]

Review of #1863 — Uprate three more columns, and guard the gap that hid them

Base repository: PolicyEngine/policyengine-uk
PR number: 1863
Reviewed head SHA: 155f50b
Merge base SHA: e146253
Mode: incremental from 155f50b
Scope: changed behavior and affected dependencies
Source evidence: Official sources are linked below; local source copies and validation artifacts were retained.
Review status: PARTIAL

Source Documents

No source documents registered; see scope and validation.

Critical

None.

Should Address

None.

Suggestions

None.

Evidence Gaps

• The numerical provenance of the newly activated OBR earnings, lagged earnings, per-capita GDP/mixed-income series and Ofwat bill-growth series could not be independently checked against the original tables: OBR and Ofwat source routes returned HTTP403 within the acquisition budget. The mapping/declaration consistency is established, but that does not certify the economic index choice or baseline series.
• Role checks incomplete: current-pr-1863-review-policy.md; see recovered report.

Notes

• All ten newly listed variables agree with their declared index and entity. A focused dataset diagnostic actually extends each £1,000 input from2025 to2026 and verifies the exact declared YoY factor (water1055, servicecharges1053, salarysacrifice1034, six GDP inputs1029.2, grossreceipts1030.6); reset_growthfactor_uprating returns all ten to£1,000. The new guard detects missing, wrong-index and stale exception declarations. No duplicate new YAML memberships were found.
• Optional Hypothesis regression test_rent_uprating.py NOT RUN: Hypothesis absent. Runnable changed/regression batch passed after excluding that path.
• Evidence reused after confirming identical reviewed head and merge base. The fetched main branch has advanced; the review remains scoped to this unchanged head. Current CI was captured separately. Local tests were not rerun.
• Captured base branch: main. Actual fetched base SHA: f1a9a3c. Incremental scope contains no code changes.

Validation Summary

Reused exact-head validation (not rerun in this batch): 12 passed, 1 warning in 6.35s; snapshot import confirmed; raw command/paths in pr-1863-code-tests-config.json and pr-1863-code-tests.log; Diagnostic: ten-column uprating script (pr-1863-code-uprating-diagnostic.py; retained locally), output (pr-1863-code-uprating-diagnostic.log; retained locally), PASS; bounded elapsed2.88s.

Reused exact-head validation (not rerun in this batch): 0 official originals; static source/policy review; no duplicate model execution. Manifest: pr-1863-review-sources.json; bundled check_source_manifest.py passed under existing Python3.13; cache result pr-1863-review-source-cache-check.json

CI at 155f50b: Install + smoke-import (py3.11)=SUCCESS, Install + smoke-import (py3.12)=SUCCESS, Install + smoke-import (py3.13)=SUCCESS, Install + smoke-import (py3.14)=SUCCESS, Lint=SUCCESS, Test=SUCCESS.

Detailed code, policy, source and reproduction artifacts were retained locally.

Timing

elapsed seconds: 1151.70s; setup seconds: 271.12s

Review Severity

COMMENT. Open findings: 0 critical, 0 should address, 0 suggestions.

[MaxGhenis]

Suggested Reproducer: line for the new Fix evidence check; add it to the description, or narrow it to the test that reproduces the bug. It lists the tests this PR adds or changes, and the check (.github/check_fix_pr.py) accepts it against this PR's diff.

Reproducer: `policyengine_uk/tests/code_health/test_uprating_declarations.py::test_declared_uprating_variables_are_in_the_yaml`, `policyengine_uk/tests/code_health/test_uprating_declarations.py::test_the_yaml_uprates_by_the_declared_index`, `policyengine_uk/tests/code_health/test_uprating_declarations.py::test_every_known_exception_still_holds`, `policyengine_uk/tests/code_health/test_uprating_declarations.py::test_the_guard_passes_exceptions_that_still_hold`, `policyengine_uk/tests/code_health/test_uprating_declarations.py::test_the_guard_flags_an_unuprated_exception_added_to_the_yaml`, `policyengine_uk/tests/code_health/test_uprating_declarations.py::test_the_guard_flags_an_override_that_now_matches`

The check isn't required yet (decision d1368); it fails fix PRs whose description names no reproducer.

🤖 Generated with Claude Code

[vahid-ahmadi]

Rebased onto main (d150116). The one conflict was in uprating_indices.yaml, where main had added uc_reported_capital next to the six new per-capita GDP entries, so I kept both; the guard finds no new gaps.
At the new head, ruff check and format are clean, the guard tests pass (8), as do the code health tests (2,277), 21 related uprating test files (1,731 passed, 66 skipped) and the full YAML policy suite (3,058), with the same counts on main.

[juaristi22]

Re-reviewed d150116d0d08c9b9db2234f348dd38c84ea816e1. The rebase retains both main’s uc_reported_capital entry and this PR’s uprating entries. The stale-exception guard now fails as intended; all eight focused current-head guard tests pass. I found no new YAML membership defect.

I checked the official OBR March 2026 economy workbook: its earnings, nominal-GDP and mixed-income figures support the PR’s 11.5% and 14.4% earnings examples and the numerator series. The remaining source gap is the dated ONS population projection/vintage used to turn GDP and mixed income into per-capita factors. The YAML points only to the ONS homepage, and its 2025 population growth differs from the rate implicit in the OBR workbook.

For water bills, Ofwat’s published 2025–26 26% figure was a forecast; its later release reports 27% for that year and 5.4% for 2026–27, versus the model’s 26% and 5.5%. The 2027–29 factors are custom projections, and the local build expects an untracked ofwat_increases.csv, so I could not reproduce their company-weighted derivation. Please document the inputs, vintage and projection calculation, or bound the effect of using these existing series for the newly uprated columns. I can revisit approval with that provenance and completed current-head CI.

[vahid-ahmadi]

Thanks. Both points are addressed in 7ab0afa, with details under "Provenance" in the description.

1. Population. ons.population for 2022 to 2073 is the population growth row of the OBR's long-term economic determinants for the March 2025 EFO (published 19 June 2025). Each fiscal year is stored at 1 January of the year it starts, and every value matches. It is the OBR forecast to 2029-30 and the ONS 2022-based migration category variant after that. The 2025 gap you found comes from vintage: the per-capita series are March 2026 nominal growth minus this March 2025 population, and the March 2026 tables imply 0.37% for 2025 against 0.72% stored. Re-deriving would raise the per-capita GDP and mixed income factors by about 0.35% by 2025, 0.46% by 2027 and 0.73% by 2030. That is the bound for the seven new per-capita columns. I have not changed it here, because household_weight and every per_capita.* series depend on it and need refreshing together. The YAML now cites the dated release instead of the ONS homepage, and the refresh is Refresh ons.population and re-derive the per-capita series from one vintage #2383.
2. Water bills. 2025-26 is now 27% (Ofwat, 29 January 2026) and 2026-27 is 5.4% (Water UK, 28 January 2026, £639 up £33), with a test that fails on the old 26% and 5.5%. The only runtime user of this series is the water column this PR starts uprating, so the change raises it by 0.7% from 2026 and moves nothing else. I kept the 2027-29 values. They are documented in the YAML and the water bills docs page as a customer-weighted mean of the CCW table, which shows Ofwat's draft determinations, plus a hardcoded CPIH path. The weights were in the untracked CSV, so those three years cannot be reproduced. The rebuild is Water bill projections for 2027-29 cannot be reproduced from the repo #2384.

Following Max's comment, the description now has the Reproducer: and axiom: lines, and .github/check_fix_pr.py passes locally. Guard tests (8), code health and the YAML suite (3,058) pass, and ruff is clean.

@juaristi22 could you re-review?

[vahid-ahmadi]

Merged main into this branch (6c3f5d6). Beyond the conflict, which kept both trading_loss and self_employment_gross_receipts, two things changed that reviewers should know about:

1. The guard caught four inputs main added for UC reg 77. They are now listed under the index each declares: owned_company_capital, owned_company_holding_value and owned_company_trade_assets under per-capita GDP, and owned_company_income_share under per-capita mixed income.
2. Main (Project single-year datasets to 2032, past the OBR horizon #2376) now reads the uprating attribute to grow unlisted columns after 2030. test_unmapped_inputs_preserve_legacy_years_and_grow_after_2030 listed the columns this PR maps as fallbacks, so it now asserts only rail_usage and employee_pension_contributions_reported. The guard docstring no longer calls the attribute dead metadata.

[juaristi22]

Re-reviewed 6c3f5d60e4947512e1bc9a77b2ff40a252dbcd40 after the provenance updates and main merge. The code checks are clear: all 24 changed guard, water-growth and projection tests pass, including the Hypothesis checks. The new company columns use their declared indices, and the post-2030 fallback does not double-uprate them. Captured current-head CI passes.

The dated OBR release attribution is now documented, and I verified Water UK’s 2026–27 increase of 5.4% (£33 to £639). Two numerical source checks remain incomplete in this run:

• The official OBR workbook downloads returned “No Access” HTML or403, so I could not independently check the population row and the newly supplied per-capita vintage bounds.
• Both permitted routes to the cited Ofwat statement were blocked, so I could not newly corroborate the changed 2025–26 value of27%. I have not established that the value is wrong.

The explicitly disclosed 2027–2029 forecast/weights limitation remains tracked by #2384. I am leaving this review PARTIAL with no confirmed code defect. Accessible copies of the exact official workbook/table and Ofwat statement would let me close the remaining source checks without repeating the model tests.