Skip to content

Add the UK Budget reform contract runner - #577

Merged
MaxGhenis merged 14 commits into
mainfrom
w1-uk-budget-reform-contract-runner
Oct 12, 2026
Merged

MaxGhenis merged 14 commits into
mainfrom
w1-uk-budget-reform-contract-runner

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #576

Draft: GPT-6.1 Sol requested eight changes in round 1. All are fixed (below), and round 2 is running. Merge #575 first, since the wrapper routes need its interval keys. The interface is stable for W4 (qualification) to import; their gate is #574.

What this adds

policyengine.tax_benefit_models.uk.budget_reform_contract.run_budget_reform_contract(*, routes=None, dataset_path=None, seed=0, max_examples=20, cases=None) -> dict. Contract policyengine-uk/budget-reform-contract, version 1.0.0.

Routes

Route id How a case runs
country.reform_dict policyengine_uk.Simulation(reform=dict) with start.stop keys
country.scenario_parameter_changes Simulation(scenario=Scenario(parameter_changes=dict))
country.reform_from_dict Simulation(reform=Reform.from_dict(dict))
wrapper.household pe.uk.calculate_household(reform=dict): lone date for a change held from a date, start.stop for a bounded one
wrapper.policy Simulation(policy=compile_reform_to_policy(...)) over an in-memory dataset of the case household
api.sim_api unsupported in-process, with a reason. The sim API's own gate imports CASES.

Cases

Each case's expected values come from statute or an annualisation written here, never from model output. Where an expectation needs current law (for example, the allowance a freeze holds back), it reads only current-law parameters.

  • april_rate_change: basic rate 21% from 6 April 2027. FY2027 gets the full rate; the income tax change for £30k is (30,000 − PA) × 1%.
  • finite_rate_change: the same, for 2027-28 only. 2028-29 reverts.
  • extended_freeze: PA and BRL frozen to 2033-34.
    • The aligned NICs thresholds follow: LPL and UPL, and PT and UEL at /52 to the penny.
    • Income tax rises by the basic rate × the allowance held back.
    • FY2034: prior law's projected path (contract 1.0.0). ITA 2007 s57/s21 resumption from the frozen amounts is recorded as pending (cos d1344).
  • april_benefit_uprating: UC single 25+ £500 from April 2027 and £520 from April 2028, giving £6,000 / £6,240 / £6,240.
  • bounded_change_on_uprated_benefit: £500 for 2027-28 only. Later years equal the baseline.
  • mid_year_blended_rate: CGT basic rate 20% from 30 October 2027. FY2027 = (0.18×207 + 0.20×159)/366. For £10,000 of gains within the basic band, capital gains tax rises by (gains − exempt amount) × 2% × 159/366 in 2027 (£60.82) and by the full year's 2% in 2028 (£140).
  • september_cpi_moves_indexed_thresholds: a 5% September 2030 CPI gives PA £13,200 and BRL £39,600 (s57(3), s21(3)). Country routes only.
  • uc_rebalancing_reform_after_data_load: protected LCWRA £500/month gives £6,000.
  • structural_switch: marriage allowance without its income condition gives £1,260 (s55B).
  • baseline_vat_is_current_law (#2188): baseline_vat under reform equals unreformed vat, and VAT rises by £10,000 × 2% / current law's microdata VAT coverage (£526.32), on every route.
  • modifier_then_parameter_changes and parameter_changes_then_modifier: one scenario (country routes) or one Policy sum (wrapper.policy) carries both a modifier that sets the dividend basic rate to 8.75% on the annualised parameters and a basic rate of 21% from 6 April 2027. Income tax changes by (dividends − allowance) × (8.75% − current rate), plus the basic-rate uplift from 2027. On policyengine-uk 2.125.1 the modifier's rate reverted to 10.75%. On wrapper main, a policy carrying both lost its values; Route UK dated reforms through policyengine-uk's statutory pipeline #575 fixes that.
  • unknown_path: every route rejects it.
  • Readings: a bare-year dict key is that year only; Reform.from_dict reads a bare year as onwards; a single-day dict key overrides one day; effective dates given latest first.
  • Isolation. Expected values read a copy of current law's parameters taken before any reform runs. Every route builds its unreformed reference before the reform, and a fresh unreformed run after it must reproduce it (unreformed_after_reform rows). Country routes also check the reformed simulation's own baseline. A run-level module_parameters_unchanged row compares every parameter, 2024–2036, with the copy.

Properties (bounded Hypothesis sampling, no shrinking)

  • annual_values_match_the_oracle: random single changes on rate, uprated, indexed and blended parameters. The annual values match the independent annualisation on every country route.
  • routes_agree_and_baselines_hold: random changes give identical household income tax, capital gains tax, UC standard allowance and UC award on every in-process route, and each route's baseline and later unreformed runs hold.
  • Hypothesis or, without it, a seeded random sampler. Either way a failed check is fail and a crash is error, with the sampled change and income in the counterexample.

Population checks (dataset_path)

These run on a deterministic subsample of 1,000 households:

  • baseline isolation;
  • query order;
  • route agreement;
  • the April change: 2026 is unchanged and total 2027 income tax rises. One person's tax can fall: in Scotland a higher UK basic rate grosses up Gift Aid by more and restores more of a tapered allowance. Executed: £105,000 earnings and £1,000 Gift Aid give −£3.60;
  • finite-change expiry;
  • UC standard allowances +10% and State Pension rates +10%: each award moves by exactly 10%, none falls, and the totals rise.

Report

contract_id, contract_version, prior_law_option, installed versions, dataset (path, sha256), cases (one row per case, route and check, with status in pass / fail / error / unsupported (with reason) / pending (with decision_id)), properties, population, summary and holds (no fail and no error). A run in which nothing passed gets an error row, and unknown or empty case or route selections raise ValueError.

The contract lives here, not in policyengine-uk, so a candidate model release never grades itself (agreed with W4).

Measured runs (this machine under heavy load)

After the review fixes (12 Oct, 71fba3c + #575 + policyengine-uk#2305 at 603fecbb2): the full qualification run, with every route, every case, seed 0, 20 examples and population checks on the certified enhanced_frs_2024_25 (sha256 5ba399ea…). Holds: 682 pass, 0 fail, 0 error, 6 pending (d1344), 79 unsupported with reasons. Both properties passed on 20 examples, and 19 of 19 population checks passed. 24.7 minutes, peak RSS 3.0 GB. 8718f37 then removed a few duplicate VAT isolation rows.

Earlier runs, before the review fixes (11 Oct):

Installed Scope Result Time Peak RSS
policyengine-uk 2.125.1 (pinned), wrapper main all cases, 2 examples 76 pass, 66 fail, 35 error 4.7 min 5.8 GB (before value caching)
policyengine-uk#2305 + #575 + this PR all cases, 5 examples 253 pass, 0 error, 6 pending, 67 unsupported, 1 property fail (an oracle bug, now fixed) 6.8 min 1.8 GB
same properties only, 20 examples both pass 13.6 min 1.8 GB
same population checks on certified enhanced_frs_2024_25 (sha256 5ba399ea…, 1,000 households) 13 pass 0.4 min 2.5 GB

On the pinned model the dated cases fail, as they should; those failures are the defects the companion PRs fix. The 6 pending rows quantify the open d1344 choice: in 2034, prior law's projected path gives PA £13,630 and BRL £41,000; resuming indexation from the frozen amounts (ITA 2007 s57/s21) would give £12,830 and £38,500. A full qualification run (all routes, 20 examples, certified dataset) takes about 20 minutes here. Reports are in ~/reviews/w1-budget-reform-contract-2026-10-11/contract-report-*.json.

Tests

tests/test_uk_budget_reform_contract.py covers:

Since the review, the tests also cover selection validation, fresh-after isolation, the reference copy, both property engines' fail and error rows, the parameter modifier, the modifier cases, and the population checks with and without award recipients.

Locally: 28 passed, 3 skipped on the pinned model. All 31 pass with policyengine-uk#2305 and #575 installed.

🤖 Generated with Claude Code

run_budget_reform_contract() runs dated Budget reforms through the
policyengine-uk routes (reform dict, Scenario parameter_changes,
Reform.from_dict) and the policyengine.py routes (calculate_household,
Simulation(policy=...)). It checks each against expected values derived
independently of the model: statute arithmetic for the indexed thresholds,
and a day-weighted fiscal-year annualisation written here. It also checks
that every reformed simulation's baseline equals an unreformed one, runs
bounded Hypothesis sampling (annual values against the oracle, route
agreement plus baseline isolation), and, given a certified dataset, runs
population checks on a deterministic household subsample. The report is
JSON for release qualification to gate on and receipt.

Contract 1.0.0 expects prior law's projected path to resume after a
bounded change. The statutory-resumption expectations for an extended
freeze are recorded as pending under cos d1344.

Fixes #576.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis and others added 4 commits October 11, 2026 16:04
…pters

Caching values lets each simulation be released after its case (the
previous run peaked at 5.8 GB). The household route's cache key could not
be hashed, the policy route's in-memory dataset lacked a region, and a
pending check whose oracle cannot be computed now stays pending instead of
gating as an error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…al year

The sampled-property oracle built a blended year from effective-date
pieces, so a change that began before 6 April was overridden by prior law
for the whole year. It now takes each day's value directly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis and others added 9 commits October 11, 2026 18:53
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nsion

A reform can reach a parameter without reaching the award, as when an
award is fixed as an input before the reform applies (UC standard allowance
+10% scored £0 across several policyengine-uk releases for that reason).
Two household cases check the award itself on every route: a claimant with
no income is awarded the reformed standard allowance (£500 x 12), and a
full-rate pensioner gets the reformed new State Pension (£280 x 52). Two
population checks raise every UC standard allowance, and the basic and new
State Pension rates, by 10% for 2027-28 and require the allowances and
pensions to scale by exactly 10%, no award to fall and the totals to rise.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The household route summed a variable across every entity that carried it,
so a benefit-unit award also reported under the household counted twice
(universal_credit read £12,000 for a £6,000 award).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ences, modifier cases

Review (GPT-6.1 Sol, review-runner-out.md):
- The population April-rate check tests total 2027 income tax, not each
  person: in Scotland a higher UK basic rate can lower one person's tax
  through Gift Aid and the allowance taper (executed: -£3.60).
- The mid-year CGT case checks capital gains tax (2027: 159/366 of a
  year at 20%), and the VAT case checks the reformed VAT, on every route.
- Expected values read a copy of current law taken before any reform;
  every route builds its unreformed reference before the reform and a
  fresh one after it; a run-level row checks the shared parameters are
  unchanged.
- Unknown or empty case and route selections raise; a run in which
  nothing passed has an error row.
- Both property engines turn any exception into an error row, and every
  counterexample names the sampled change and income.
- The routes-agree property compares capital gains tax and Universal
  Credit as well as income tax.

Also adds W5's shape: one scenario or policy with a parameter-setting
modifier and dated parameter changes, in either order.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…laimant

The responsiveness checks need an award to move; with earners only they
fail, as they should, and a test now says so.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The VAT case reads vat for two checks, which doubled its isolation rows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…er cache

Review round 2 (GPT-6.1 Sol, review-runner-r2-out.md):
- The run-level parameters row passes whenever nothing leaked, so it
  masked the 'nothing passed' guard on any in-process selection; passes
  are now counted before it is added.
- module_parameters_unchanged also compares policyengine-uk's cached
  processed tree (which every new simulation copies), cached statutory
  tree and cached projected timelines, where the installed version has
  them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review round 3 (GPT-6.1 Sol, review-runner-r3-out.md, APPROVE with P3s):
- module_parameters_unchanged compares each parameter's whole timeline
  and metadata, not 1 January samples, so a leak on 6 April or after
  2036, or into uprating metadata, shows.
- The rebuild memo, which the run fills, is documented as unwatched.
- A test covers the projected-timelines branch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis
MaxGhenis marked this pull request as ready for review October 12, 2026 08:50
@MaxGhenis
MaxGhenis merged commit 8e79785 into main Oct 12, 2026
12 checks passed
@MaxGhenis
MaxGhenis deleted the w1-uk-budget-reform-contract-runner branch October 12, 2026 09:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add a UK Budget reform contract runner for release qualification

1 participant