Repository navigation
Conversation
Every index stopped at 2039 while the growth series run to 2074, so a parameter or input grown by an index froze at its 2039 level. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The indices read each series on 1 January, so a value dated later in a year first reaches them the next year. The pension fixtures relied on every index running to 2039, so their growth series now end there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Aggregate impact on certified data: passBase Checks: 0 broken by this change, 1 failing on both sides, 0 fixed. Moves flagged for review: 0 (thresholds in Notes
Moves flagged for reviewNone. Spending by benefit and service
Caseloads
Standard reform costings (change in gov_balance)
Standard reforms: change in each reform's target
Household net income by decileNo household's net income changes in any year. No change in any year: receipts by tax head; government balance, each component counted once; model totals; household income totals; population; poverty rates; people in poverty; inequality; mean household net income in each side's own deciles. Checks failing on both base and head (1)
Counted-once checkEach of 92 components was raised by £1 per unit in 2026-27, and every head of its totals was read household by household: its own head must rise by exactly £1, no other tax head may move, no other benefit may rise, and the model's own total must move by what its heads moved. A head that breaks a rule is measured again twice: with every other head held at its baseline value, and with every other head held at its value after the first £1 while the component rises again. It fails where either still shows the response. 0 failed; 0 were skipped.
Provenance
Dataset: Runtime differences:
Receipts by tax head: head levels
Spending by benefit and service: head levels
Caseloads: head levels
Government balance, each component counted once: head levels
Model totals: head levels
Household income totals: head levels
Population: head levels
Poverty rates: head levels
People in poverty: head levels
Inequality: head levels
Mean household net income in each side's own deciles: head levels
Standard reform costings (change in gov_balance): head levels
Standard reforms: change in each reform's target: head levels
|
The legacy NI metadata fixture is an exact extract that ended in 2039, so compare it through that year: later records are new, not legacy. Build the nested growth tree without subscripting a call, which the cached-array scan reads as an entity call. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Compare the NI rates in full, since only the thresholds follow indices past the legacy extraction, and check that those thresholds keep rising. Run the synthetic pension checks through 2040, the last year their lagged series reach. State that growth_horizon reads the series before fiscal conversion, pin a dated reform past the data, and let an empty growth node build no indices, as it did before. Name the carer tie allowance as an intended exception to the half-up reference. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A drop right after 2039 would otherwise pass, since the check started in 2040. Start it at the last legacy value so the boundary counts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Hand-off to the UK hub at head GPT-6.1 Sol ( Lint, Changes, Changelog fragments, both documentation builds, the release build, Fix evidence and the four smoke imports pass at this head. Python 1/6 to 6/6, YAML and Aggregate impact hadn't finished at 01:33Z on 31 October. At
This PR leaves d2161 stops any agent merging or enqueueing this PR. No published number moves, so it needs no sign-off from Max. d2222 holds the separate call on long-run paths for the 37 short growth series. Three other pieces of work overlap this PR:
🤖 Generated with Claude Code |
Moves only the policyengine-core entry in uv.lock, from 3.33.12 to 3.34.1, which carries PolicyEngine/policyengine-core#631 (shared parameter metadata in clone, Parameter.interpolated). Core's requirements are the same in both releases, so no other package moves. uv lock --check passes under uv 0.13.6 and 0.12.13. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Three comments and a docs table still said the indices end in 2039. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fixes #2755
What and why
create_economic_assumption_indicesbuilt every cumulative index withrange(start_year + 1, 2040), while the growth series run to 2074. AParametercarries its last value forward rather than raising, so everything grown by an index froze at its 2039 level with no error. That covers the State Pension, the tax thresholds, benefit rates, and the inputs Core grows byvariable.upratingafterPROJECTION_END_YEAR.The indices now end at
growth_horizon(yoy_growth), the last year any growth series covers. They end in 2074, one year pastyoy_growth.yaml, because the lagged series run a year past their source. The indices read each series on 1 January, so a value dated later in a year counts from the year after. Fiscal-year conversion needs no change: af5d07a made it follow each parameter's stated dates.State Pension path and totals, 2035 to 2050
I ran
main(37591f4) and this branch, based on that commit, on the certified dataset,enhanced_frs_2024_25.h5at2526edbb, in fullMicrosimulationruns. The run recorded no SHA for the branch. Weekly rates in pounds; totals in £bn, by model year.gov_tax, maingov_tax, fixgov_spending, maingov_spending, fixAll 12 totals I recorded match in every year from 2035 to 2039, so the change starts in 2040. Tax and spending both rise from 2040 because the tax base and the benefit rates both stop freezing. These rows measure the bug; they don't forecast the public finances.
Build time and CI cost
The longer indices take
mainat 2da212a from 75,993 stored values to 94,445, and everyCountryTaxBenefitSystemandSimulationclones them. The table below comparesmainat 03349c9 with this branch at b5a69da.main03349c9, Core 3.33.12CountryTaxBenefitSystemCountryTaxBenefitSystemin a processSimulationplushousehold_net_incomeThe rounds shared an 18-CPU host with other jobs, at one-minute load averages from 25 to 41, so every time runs above a quiet host's; the columns ran interleaved, so they compare with each other. Under Core 3.33.12, this branch clones the processed tree 18% slower than
main, builds a new system 11% slower and spends 0.18 s more on the first build in a process. Core 3.34.1 takes all three belowmain. The household row varies most between rounds (3,543 to 4,345 ms for this branch under 3.33.12) and doesn't separatemainfrom this branch on this host; a second probe reads 3,159, 3,191 and 2,773 ms for the three columns. Profiles of three household simulations put the system clone at 14.4% of their time onmain, 15.5% on this branch under 3.33.12 and 7.8% under 3.34.1.CI showed the cost before this update. On this branch's runs at 3fbcd89 and 478fc38, the Python shards took 66.3 to 75.5 minutes, and the 75-minute limit cancelled shards 2/6 and 3/6 both times. Three merge-queue runs on
mainafter #2742 merged (4542fff, ada1bba and cc477bd) took 59.6 to 73.5 minutes per shard, somainalone comes within 1.5 minutes of the limit. On both of this branch's runs,Test / YAMLstopped about 7 minutes in, atcharge_parameters.yaml, when its runner received a shutdown signal. Both runs predate #2742, and the three merge-queue runs passed the YAML job in 8.2 to 9.6 minutes. The logs record no memory figures, so I infer memory from the 16.1 GiB local peak onmainthat #2742 reports and this branch's 9% larger retained systems.Two changes outside this PR address those costs, and both have landed:
uv.lockto 3.34.1, so CI tests this branch with it. In the table, it brings this branch's clone and build times belowmain's.This PR keeps the repeated annual records on
main(54,578 of its 75,993 entries repeat the value before them):_fiscal_year_conversion_yearsretains them on purpose, and the API metadata exposes them.What else moves
gov.dft.bus.local_bus_fares_indexparameters usefiscal_year_blend, which reads the January 2040 index.mainhad no January 2040 value, so it understated that one fiscal year (London 257.75 against 259.10). Nothing else changes before 2040, and none of the recorded totals moves.test_live_parameter_metadata_preserves_legacy_scalar_recordspins every rate in full and every threshold through 2039, where the legacy extract ends, and fails if a threshold falls after that.uk-triple-lockpinseconomic_assumption_indices: [None, 2039]and the SHA-256 of these files inmodel_horizon.py, so its next PE-UK bump fails loudly until it re-audits. Its published results end at 2039-40. The API's UK year list readsPROJECTION_END_YEARand the CGT projection, not the indices, so it doesn't widen.uv.lockfrom policyengine-core 3.33.12 to 3.34.1 and moves no other package;pyproject.tomlkeeps>=3.33.0. I compared the two releases on this branch. At f6855cf, every leaf parameter holds the same value at 180 instants from 2015 to 2074, and every node and stored entry holds the same metadata. That holds in a first and a secondCountryTaxBenefitSystemand in a household simulation's system, across 3,142,884 comparisons. At b5a69da, a grid of 24 synthetic households covers earnings, pensions, children, every tenure, all four nations and benefit claims. It gives the samehousehold_net_income,gov_taxandgov_spendingin every year from 2025 to 2050, under current law and under a reform to the personal allowance and the Universal Credit taper, across 3,744 comparisons. Five reforms and thereindex_benefit_capscenario leave the cached and built parameter trees unchanged under both releases. Code here writes metadata only at the top level of a dict or by replacing the dict, never inside a nested value that 3.34.1's clones now share. 3.34.1 also removesReform.from_api,Reform.api_idand thepolicyengine_core.charts.apifunctions, which nothing here calls andtest_no_core_public_api_functions.pykeeps that way, and refuses a non-integer count inbuild_default_simulation, which nothing here calls.Known limitation
Only 10 of the 47 growth series stated in the source tree reach 2073. The other 37 end between 2029 and 2036 and hold their final rate, as they did to 2039 on
main, and the longer horizon compounds that rate for 35 more years. The ones that reach microsim inputs after 2032 includeobr.mortgage_interestat 5.53% andobr.per_capita.mixed_incomeat 4.0%, so an index like mortgage interest reaches 36 times its base by 2074. These series need long-run paths of their own, which this PR leaves out of scope. The README now says so.Invariants
Test plan
policyengine_uk/tests/test_economic_assumption_horizon.pychecks invariants 1 to 4, on the real tree and on 200 Hypothesis-generated growth trees of mixed lengths.test_pension_credit_uprating.pyandtest_pension_age_projection_invariants.pyend in 2039, so their lagged series reach 2040, and the checks run through 2040. Reforms they apply still reach the dependent forecasts.test_parameter_metadata.pyholds no live test.test_dated_reform_routes.pyreached its local time limit with no failure logged, and CI runs it with the six Python shards.policyengine-core testruns the whole YAML tree in one process: 5,409 YAML cases and one Python test pass.make formatchangelog.d/derive-index-horizons.fixed.mdReproducer
Reproducer: policyengine_uk/tests/test_economic_assumption_horizon.py::test_the_state_pension_keeps_rising_after_2039
axiom: n/a: microsim-only
🤖 Generated with Claude Code