Repository navigation
Conversation
The report measures one model checkout on the pinned certified dataset over 2025-26 to 2031-32: receipts by tax head, spending and caseloads by benefit, poverty by measure and group, inequality, own-decile incomes and the standard reform costings. It checks them against a fiscal partition of the model's totals (policyengine_uk/data/fiscal_partition.yaml): - closure: each total's components are exactly the partition's heads and declared double counts; - accounting identities, household by household; - a counted-once check, which perturbs each component by £1 and requires each tax total to move by exactly £1 per unit, and each spending total by at most £1. compare then sets a base measurement against a head one. It fails on a crash, a different dataset, non-finite figures or a check the base passes and the head breaks. It flags moves above the proposed thresholds, and renders a pull request comment. The partition's tax heads follow the #2159 work's specification, encoded for main before that fix: corporate_sdlt is a declared double count inside expected_sdlt. Spending heads cover gov_spending and household_benefits. Test data: conftest pins enhanced_frs_2024_25 from uk-data 1.58.0 by Hugging Face commit and SHA-256 (tests/certified_dataset.yaml), verifies the bytes before data tests run, and adds a mandatory-data mode (POLICYENGINE_UK_REQUIRE_TEST_DATA). In that mode, data tests fail rather than skip without data, and run only on the pinned dataset. The reform-cost fixtures are rebased onto 1.58.0, keeping each earlier value as a receipt with the model and data that produced it. Refs #2334. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
.github/workflows/aggregate-impact.yaml measures the pull request's base and head on the same runner image. Each side syncs its own uv.lock and runs the head's copy of the tool against its own model, on the pinned dataset whose bytes the tool verifies. The report job then compares the two, writes the job summary and one updated comment, and fails on structural failures. Fork and Dependabot runs, which cannot read the private dataset, record a skip instead. Refs #2334. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sections with no change collapse into one line, an unchanged decile table into one sentence, and the counted-once summary lists only real violations (a tax total moving by other than £1 per unit, or a spending total by more than £1), not the means-test offsets every benefit shows. Refs #2334. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refs #2334. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The aggregate impact report (#2335) adds gov_spending, household_benefits and spending_heads to fiscal_partition.yaml. The tax tests now check the three tax totals only, require that spending heads borrow no tax head id or variable and that no spending head names a tax total, and apply dropped_when to the tax totals alone. Checked against #2335's file with these tax sections merged in: this file's tests and #2335's partition and guard tests pass. The vat head records that 76 households have negative VAT on policyengine-uk-data 1.58.0 (consumption or the full-rate expenditure share is negative in the data; policyengine-uk-data issue 565). Refs #2159 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refs #2334. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lookup - Set POLICYENGINE_UK_REQUIRE_TEST_DATA=1 (W3's conftest switch, #2335) wherever require-data is on: main pushes, merge queue groups and releases. A test that needs the dataset then fails with its own message instead of skipping. Unused until #2335 lands. - A manual dispatch of the main workflow always tests; the merge queue lookup applies only to pushes, and a failed lookup parses to zero. - Document the re-run path's residual and where docs-only pull requests build the documentation; the workflow-shape test covers the release workflow's require-data and the lookup. From the GPT-6.1 Sol delta review of 96e9cec (APPROVE, non-blocking). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The standard reforms move into the tool (standard_reforms.yaml), so a vendored copy carries them: the eight reform-cost fixtures, plus a 10% rise in the Universal Credit standard allowance and in the State Pension. Each names the variable it must move, the direction and a minimum. A reform that moves its target by less, by nothing or the wrong way in any year fails responsive:<id>, so a reform the model silently ignores cannot pass as a small number. A relative reform is built from each side's own baseline parameters: that model year's value times the factor, for every year of the horizon. On uk-data 1.58.0 all ten respond in every year, 2025-26 to 2031-32. The UC standard allowance rise adds £3.0bn to £3.4bn of Universal Credit, and the State Pension rise £12.4bn to £14.4bn. Refs #2334. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The certified dataset is licensed survey microdata, and only aggregates may be published. The workflow uploaded each side's measurement as an artifact, including the household-level arrays the decile comparison uses, and cached the dataset in the Actions cache. Both are removed: - base and head are now measured in one job, so the household detail never travels between jobs; only the report directory (the two result.json files, the comment and the summary) is uploaded; - the dataset is downloaded on each run and never cached; - measure writes household detail under private/, apart from result.json; - check details give counts and weighted totals, not one household's value. test_aggregate_impact_workflow.py fails if the workflow uploads anything but the report, uses the Actions cache, copies anything private into the report, or splits the measurement across jobs. Refs #2334. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The aggregate impact report (#2335) adds gov_spending, household_benefits and spending_heads to fiscal_partition.yaml. The tax tests now check the three tax totals only, require that spending heads borrow no tax head id or variable and that no spending head names a tax total, and apply dropped_when to the tax totals alone. Checked against #2335's file with these tax sections merged in: this file's tests and #2335's partition and guard tests pass. The vat head records that 76 households have negative VAT on policyengine-uk-data 1.58.0 (consumption or the full-rate expenditure share is negative in the data; policyengine-uk-data issue 565). Refs #2159 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Aggregate impact on certified data: passBase Checks: 0 broken by this change, 4 failing on both sides, 0 fixed. Moves flagged for review: 0 (thresholds proposed in Moves flagged for reviewNone. Household net income by decileNo household's net income changes in any year. No change in any year: receipts by tax head; spending by benefit and service; caseloads; government balance, each component counted once; model totals; components the model counts twice; household income totals; population; poverty rates; people in poverty; inequality; mean household net income in each side's own deciles; standard reform costings (change in gov_balance); standard reforms: change in each reform's target. Checks failing on both base and head (4)
Counted-once checkEach of 82 components was raised by £1 per unit in 2026-27: tax totals must rise by exactly £1 per unit and spending totals by at most £1. 2 failed; 0 were skipped (zero for every unit, or not money).
In the model's own totals:
Provenance
Dataset: Receipts by tax head: head levels
Spending by benefit and service: head levels
Caseloads: head levels
Government balance, each component counted once: head levels
Model totals: head levels
Components the model counts twice: head levels
Household income totals: head levels
Population: head levels
Poverty rates: head levels
People in poverty: head levels
Inequality: head levels
Mean household net income in each side's own deciles: head levels
Standard reform costings (change in gov_balance): head levels
Standard reforms: change in each reform's target: head levels
|
Counted-once check. It judged each total's net movement, which a means-test fall, or the eligibility-jump exclusion, could use to hide a double count. It now reads each head's response household by household: - a component's own head must rise by exactly £1; - no other tax head may move; - no other benefit may rise in any household (rises and falls are not netted); - the model's own total must move by what its heads moved, plus one for a declared double count. That last rule catches a total whose formula adds a zero component twice. A head that declares a read may respond, but not one for one. The snapshot of supplied inputs is taken after formula results are discarded, so an earlier result is never frozen as an input. Comparison. - A missing or crashed base now fails. - A check failing in a year where the base passes is a regression, even if it already failed in another year. - A check the base ran that is missing from the head fails (or is noted, when the partition changed), and the head must have run the core checks. - A figure on one side only is shown as added or removed, never as no change. - A non-finite head figure fails directly. - Households must match one to one. Measurement. - Finiteness is checked after the reform costings. - A rebuilt baseline must match in every year. - A skipped check is recorded as skipped. - metrics_version covers the effective spec, horizon and reforms. - Geographic exclusivity guards are checked on the data. Test data. - Data tests are identified by their marker, so the data-free tests under microsimulation/ run without credentials. - The setup hook runs before pytest's skip handling, so a skip marker cannot turn missing data into a skip. - The latest-data smoke test is a named exception to the pin. The partition's notes now describe main before the #2159 fix correctly. Refs #2334. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Qualification compares a candidate's report with the previous pair's. With only a status per check, a check that fails in more years than before looks unchanged; failing_scopes lets it be told apart. Refs #2334. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The tool digest was read from disk when a measurement finished, so editing the tool during a run stamped the result with code that had not produced it. It is now taken when the module loads. Refs #2334. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Two fixes from this PR's W3 report change the lists
Neither PR touches |
Fixes #2334
This is workstream W3 of the Autumn Budget 2026 CI plan. It stays a draft until the independent GPT-6.1 Sol review is back.
The workflow has already run on this PR: see the "Aggregate impact on certified data" comment below. Both sides ran on uk-data 1.58.0 in identical environments (the same environment hash), at about 5 minutes a side. That comment was removed with the data-protection fix below and the next run replaces it. The result was pass, with no moves, as expected for a PR that changes no formula.
What it adds
An aggregate-impact report. The module is
policyengine_uk.utils.aggregate_impact, and the workflow is.github/workflows/aggregate-impact.yaml. On every PR, the workflow measures the base and the head on the same certified dataset, on the same runner image. Each side syncs its ownuv.lockand runs the PR head's copy of the tool. The report covers every year from 2025-26 to 2031-32:Base and head are measured in one job. The dataset is licensed survey microdata, so only aggregates leave that job; a test enforces it (see "Data protection" below).
It posts one comment and updates it on later runs. It fails on:
Moves above the flag thresholds are listed, not failed.
A fiscal partition,
policyengine_uk/data/fiscal_partition.yaml. The tax heads are the specification from the Corporate SDLT is counted twice in household and government tax totals #2159 work (Count corporate SDLT once and let SDLT abolition remove all of it #2235), which was written from the OBR receipts lines and each variable's formula. They are encoded here for main before that fix:corporate_sdltis a declared double count insideexpected_sdlt. The spending heads covergov_spendingandhousehold_benefits.Immutable test data and a mandatory-data mode, in
tests/conftest.py.tests/certified_dataset.yamlpins uk-data 1.58.0'senhanced_frs_2024_25.h5by Hugging Face commit47bbb58d, with sha2565ba399ea…. That is the artifact policyengine.py#555 certified with model 2.125.1.POLICYENGINE_UK_REQUIRE_TEST_DATA=1, every data test fails in setup when the dataset is unavailable, and runs only on the pinned file.Reform fixtures rebased onto 1.58.0.
reforms_config.yamlkeeps every earlier value as a receipt, with the model and data that produced it. It also records current main's figure on 1.40.3, so the data's share of each move is visible. The earlier values were computed on whatever Enhanced FRS was latest until the 1.40.3 pin landed on 2026-04-05.Invariants the report enforces
gov_tax,household_tax,pre_budget_change_household_tax,gov_spendingandhousehold_benefits, the model's own component list is exactly the partition's heads plus its declared double counts, each listed once. Each head's declared entity is its variable's entity.dropped_when(council tax abolition).gov_balanceequalsgov_tax − gov_spending.sign: nonnegativeare never negative.gov_balanceexactly.standard_reforms.yaml). A reform that moves its target by less, by nothing or the wrong way in any year failsresponsive:<id>, so a reform the model silently ignores cannot pass as a small number.Tests
Microsimulation.calculate(...).sum(), comparing person and benefit-unit weights with household-mapped sums.microsimulationmarker):test_aggregate_impact_certified.pyruns 2026-27 with the counted-once check on every tax component and four benefits, and the two relative reforms.Commands run locally:
pytest policyengine_uk/tests/aggregate_impact: 79 passed without credentials (7 data tests skipped); the 7 data tests pass withPOLICYENGINE_UK_REQUIRE_TEST_DATA=1and a token.pytest policyengine_uk/tests -m microsimulationon the 1.58.0 pin: every test outside the rebased fixtures passed before the rebase; the rebased fixtures were then recomputed.The full Python suite was not run locally; CI runs it.
What the report already finds on main
These figures are from certified 1.58.0, 2026-27, main at 6d89338. The first two are filed as #2337 and #2338, and the negative VAT as policyengine-uk-data#565.
gov_tax,household_taxandpre_budget_change_household_taxcountcorporate_sdlttwice: coefficient 2.000, £3.13bn a year (Corporate SDLT is counted twice in household and government tax totals #2159, fixed in Count corporate SDLT once and let SDLT abolition remove all of it #2235).taxable_employment_incomeaddsemployment_benefits, which is SSP plus SMP, so UC deducts their income tax and NI.uc_individual_earned_income_before_mifcounts only employment, miscellaneous and self-employment income as earnings.household_benefitsbut missing fromgov_spending, sogov_balanceleaves out their cost (Scottish Child Payment and Healthy Start are household benefits but missing from government spending #2338).extend_single_year_dataset(end_year=2030).simulations/simulation.py~1121–1215): an input with anupratingindex is uprated from 2030, and other inputs are carried over unchanged. Council tax per household, for example, is flat 2030→2031.microdata_vat_coverage: the value has been 0.383 since 2010 and overshoots OBR VAT receipts on current datasets #1996, Split VAT grossing into household share and survey coverage #2187). This is why the VAT 2pp fixture moves from 21.3 to 34.5.Data protection
An earlier version of this workflow (commits 073e808 to 8d3e482) uploaded each side's measurement as an Actions artifact, including the household-level arrays the decile comparison uses, and cached the dataset in the Actions cache. Those artifacts and the cache were deleted on 11 October, and
fd94415removes both:result.jsonfiles of aggregates, the comment and the summary);private/;test_aggregate_impact_workflow.pyfails if the workflow uploads anything else, uses the cache, or splits the measurement across jobs.Thresholds and methodology
These are proposals pending cos d1344:
report_spec.yaml: max(£100m, 0.1%) for £ totals, max(5,000, 0.5%) for caseloads, 0.1pp for rates and 0.001 for Gini. Flags never fail the job;standard_reforms.yaml, each well below the reform's smallest yearly effect on 1.58.0.No model semantics change in this PR.
Not in this PR
POLICYENGINE_UK_REQUIRE_TEST_DATAin the existing workflows.axiom: n/a: CI, test-data and reporting infrastructure; no policy change
🤖 Generated with Claude Code