Calibrate SPI income bands to their projected values, and map 2023-24 outturns to 2023

[MaxGhenis]

Data release: hold for the next uk-data batch on Max's go (d833). Merging runs versioning and the Push build, upload and tag.

Problem

The 2025 calibration has targeted the HMRC Survey of Personal Incomes (SPI) income bands at their raw 2023-24 values, with no uprating at all. Two bugs combine.

1. Projected SPI targets never reached calibration.

• targets/build_loss_matrix.py fetches targets with get_all_targets(geographic_level=...), without a year filter.
• targets/registry.py keeps only the first target of each name.
• hmrc_spi.get_targets() emitted each band twice: the ODS outturn (base year only), then the incomes_projection.csv projection (later years) under the same name.
• So the projection was dropped, and _resolve_value(target, 2025) fell back to the nearest earlier year: the outturn, unuprated.

A seeded build of main shows it. The 12,570–15,000 employment-income band is calibrated to £16.90bn, the raw ODS value, while get_all_targets(year=2025) gives the projected £17.53bn. Year-filtered callers and the tests saw the projection; calibration did not. This has been the case since the February 2026 registry refactor (de061fc), which introduced both the unfiltered first-wins fetch and the split targets.

2. The SPI and CGT outturns were mapped one year late.

• PolicyEngine UK's year N is tax year N to N+1. In policyengine-uk 2.122.2, convert_to_fiscal_year_parameters (policyengine_uk/utils/parameters.py) gives every parameter, including the uprating indices behind storage/uprating_factors.csv, its value at 30 April N. So cgt.annual_exempt_amount is £6,000 in 2023 and £3,000 in 2024.
• hmrc_spi.py mapped the 2023-24 SPI to _SPI_YEAR = 2024 ("mapped to calendar 2024"), and hmrc_cgt.py followed with _CGT_BASE_YEAR = 2024. The projection, the local-area income scaling (local_income.get_national_income_projections) and the CGT gains targets therefore carried one year too little growth.
• The rest of the repo already uses the start year: FRS 2024-25 is 2024 (datasets/frs_release.py), the SPI 2022-23 microdata is SPI_FISCAL_YEAR = 2022, OBR FY 2025-26 is 2025, and Restore HMRC salary sacrifice relief calibration targets #533's salary sacrifice tables check that "2024 to 2025" is 2024.
• The end-year mapping first appears in de061fc ("tax year 2022-23, mapped to calendar 2023"). Refresh UK target source vintages #390 and Regenerate SPI projections from 2023-24 base #414 carried it forward without stating a reason, and Refresh HMRC SPI targets to tax year 2023-24 #388 and Regenerate SPI income projection targets from the 2023-24 base #393 give none either. The code before the refactor mapped SPI 2020-21 to fiscal_year=2020.

Fix

• hmrc_spi.get_targets() returns one target per band holding the base year and every projected year (_merge_by_name, which _read_projection_csv already used within the projection).
• _SPI_YEAR = 2023 and _CGT_BASE_YEAR = 2023, with comments that state the convention.
• storage/incomes_projection.csv is regenerated from the same ODS. incomes.csv comes out byte-identical, and the new 2023 rows equal main's 2024 rows.
• The CGT base year is gated at the £6,000 annual exempt amount (AEA) that the 2023-24 outturn was realised under. Projected years keep the £3,000 AEA then in force.

Target changes, as the calibration resolves them (2025)

These come from build_loss_matrix's own path: the unfiltered registry plus _resolve_value. Of the 645 national targets that resolve for 2025, exactly 166 change. All of them are SPI band targets (13 bands × 6 incomes × amount and count) or CGT gains targets.

Targets (summed over the 13 income bands)	Main	This PR	Change
Employment income	£1,084.2bn	£1,176.9bn	+8.55%
Self-employment income	£107.5bn	£117.8bn	+9.64%
State pension	£85.2bn	£90.0bn	+5.62%
Private pension income	£127.8bn	£140.5bn	+9.95%
Property income (after the 1.9x rental scale)	£55.8bn	£59.5bn	+6.63%
Dividend income	£71.8bn	£76.5bn	+6.63%
Every income count			+2.16%
hmrc/capital_gains_total and the 9 CGT band gains	£67.7bn total	£70.3bn total	+3.75%
CGT taxpayer counts	378k	378k	unchanged (held flat by design)

Each SPI change is the variable's uprating index for 2025 divided by that for 2023, which is two years of growth. The CGT change is one year of growth, because CGT targets already listed every year. The constituency and local authority employment and self-employment targets are scaled to the projection's 12,570-and-over row, so they rise by one year of growth (+4.65% and +4.78%).

Calibration fit

CALIBRATION_FIT_PENDING

Invariants

• Calibration sees the projection: for every projected year, band, income type, amount and count (936 cells), the target resolved through build_loss_matrix's path (unfiltered registry plus _resolve_value) equals incomes_projection.csv, with the 1.9x scale on property income. hmrc_spi emits each name once. This is test_calibration_reads_the_projected_spi_targets; without the merge it fails (312 names, 156 unique).
• Year mapping: a tax-year-N/N+1 outturn is PolicyEngine year N. test_spi_year_is_the_start_of_the_configured_tax_year reads the year from the configured ODS name (..._2324.ods). test_base_year_is_the_tax_year_the_outturn_was_realised_in reads the AEA from the real policyengine-uk parameter tree: £6,000 at _CGT_BASE_YEAR, £3,000 a year later.
• Projection identity: each year's projected cell equals the official base cell × index[year] / index[base]. test_projection_is_the_spi_table_uprated_with_the_committed_table is a differential test: it regenerates the projection from incomes.csv and uprating_factors.csv and compares every cell. It is byte-for-byte the same test as in Regenerate the uprating factor table from policyengine-uk's load-time uprating #541, so the two PRs merge cleanly in either order; whichever lands second regenerates incomes_projection.csv.
• With the constants set back to 2024, five tests fail. With the fix, the target test files pass.

Not changed

• ons_demographics also emits three duplicate names. The second ons/uk_population is in thousands. The two registry paths disagree only for 2023, which nothing calibrates to. This is reported to the UK hub separately.
• hmrc_salary_sacrifice.py was handled by Restore HMRC salary sacrifice relief calibration targets #533.

axiom: n/a: data (calibration target years)

🤖 Generated with Claude Code