Repository navigation
Conversation
Stage 1 of the salary sacrifice imputation ran the QRF for everyone not asked SALSAC and kept positive draws even without pay. The forest learnt the no-pay case from the 30 SALSAC respondents with a job but no recorded pay, three of whom report a sacrifice, so about 1.5k records (about 1.06m weighted, £11.3bn of the £25.8bn total in the 1.56.16 enhanced FRS) had a sacrifice and no pay. From 2029 policyengine-uk returns the excess over the £2,000 cap to employment income, so those people gained pay. - limit_salary_sacrifice_to_pay: every amount, reported or imputed, is between zero and the person's pay. No FRS 2024-25 respondent who reports a sacrifice and has pay reports more than that pay. - The QRF trains only on SALSAC respondents with pay. - Stage 2 moves employee pension to salary sacrifice up to the person's pay; any remainder stays as employee pension. - Tests: Hypothesis properties of the limit and of the whole imputation (stub model), and built-dataset checks that nobody has a sacrifice without pay or above it, and nobody without pay has employment income in 2029. Both built-dataset checks fail on the 1.56.16 file. - hypothesis added to the dev extras. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
policyengine-uk is gaining the same limit (pension_contributions_via_salary_sacrifice_from_pay). Where the installed model has it, the built dataset's salary sacrifice must already equal it. Skipped on a policyengine-uk without the variable. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Stage 2 converts a pension contributor's employee contribution into salary sacrifice but left their pay unchanged. FRS pay is before that deduction, and policyengine-uk treats employment income as pay after any sacrifice, so a converted donor kept the same pay, lost the relief on the contribution and paid more income tax. Removing the phantom no-pay users enlarges stage 2's shortfall: at pre-calibration weights the extra donors (1.32m weighted) moved £2.95bn and paid £0.90bn more income tax for that reason alone. Stage 2 now takes the amount moved out of pay, so taxable pay is unchanged, and moves at most half the pay, so the sacrifice never exceeds the pay that remains. Tests: pay falls by exactly the amount moved, pay plus salary sacrifice and the pension total are unchanged for every donor, nobody else's pay changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review (subfleet 20261002-093645-ukdata-518-ss-review) found the limits used employment_income from a Microsimulation of the 2024 survey-year dataset, which calculates 2025 by default and uprates pay by 5.3% but not salary sacrifice. Sacrifices could therefore reach 105.3% of stored pay, and stage 2 could move up to 52.65% of stored pay and leave less than that. CI on a99a1ef caught it: 4 people in the TESTING build sacrificed more than their pay. - The limits and stage-2 donor pay use the stored employment_income column. QRF predictors still come from the simulation, as in training. - The model cache is now salary_sacrifice_paid_respondents.pkl, so a forest cached from the old training set is not reused. - The no-training-data error names the actual condition; docstrings say the upper bound is an imputation bound, not a legal one, and that only predictions for people with pay are kept. - Wiring tests run at 2024 and 2025; the 2024 cases fail on a99a1ef. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review r2 (subfleet 20261003-090151-ukdata-518-ss-review-r2, APPROVE) nits: the stage 2 comments still said donors keep their full employee pension and become below-cap users, and "keeps taxable pay unchanged" claimed more than the code guarantees (pay plus sacrifice is unchanged). The stored pay array is copied before use. No change in behaviour. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… simulable create_spi labels GORCODE 13 (address abroad), 14 (address unknown) and -1 (composite records) UNKNOWN. policyengine-uk 2.104.5 (PolicyEngine/policyengine-uk#1985) uprates their rent by the UK-wide index, so the dataset now simulates as built. Scottish taxpayer status comes from HMRC's SCOT_TXP flag rather than the region: the two disagree for 1,394 GORCODE-11 records and 606 Scottish taxpayers elsewhere on the 2022-23 tape. load_spi_dataset relabels UNKNOWN as SOUTH_EAST only on releases before 2.104.5, and ensure_spi_dataset rebuilds a cached H5 without the flag. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… uprating storage/uprating_factors.csv was last built on 2026-05-20 from each variable's `uprating` attribute. policyengine-uk 2.93.0 projects a saved dataset with the year-on-year growth parameters in data/uprating_indices.yaml instead, so the build calibrated 2025 values the model does not run on: earnings were uprated 3.7% for 2024-25 against the engine's 5.3%, self-employment income 4.7% against 0.7%, and GDP-indexed incomes and wealth 2.8% against 4.4%. Regenerating with the old generator would not fix it: on 2.93.0 it emits no employment_income row (an `adds` formula with no attribute) and indexes savings interest by household interest income where the engine uses GDP per head. The generator now compounds the growth parameter each variable is listed under in uprating_indices.yaml, keeps the household-weight and road-fuel overrides, and stores six decimal places. The table loses four rows the engine does not uprate at load (benunit_rent, housing_service_charges, pension_contributions_via_salary_sacrifice, water_and_sewerage_charges) and gains bus_fare_spending and private_pension_wealth. incomes_projection.csv is regenerated: the table is its only input that changed (rebuilding it from main's table reproduced main's file exactly). compute_ss_headcount deflated contributions through the salary sacrifice row, which no longer exists; the loss matrix would have skipped all three headcount targets. It now classifies contributions as simulated in the calibration year, the values test_salary_sacrifice_headcount checks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review of #540 round 1 (Sol 6.1, REQUEST CHANGES): - model_simulates_unknown_region now simulates one UNKNOWN household on the imported policyengine-uk instead of reading installed package metadata, which differs from the imported code under make data-local. Only a missing parameter for UNKNOWN reads as "cannot simulate"; any other error raises. - A differential test ties the probe to the release: where the imported model is the installed distribution, the probe agrees with version >= 2.104.5. - The unknown-region test xfails only on the missing private rent index for UNKNOWN; any other error fails it. - The region invariant test checks HMRC's codebook mapping written out in the test, not REGION_MAP itself. - New test: records whose SCOT_TXP agrees with their region keep exactly the income tax of the region-derived rule; the others change. - Income tax tests now cover 2030, the last year policyengine-uk carries dataset inputs to; the create_spi comment records that horizon. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review r1 of #541: policyengine-uk also reassigns student loan plans at load and zeroes the repayments of loans it writes off, which the table's single average-earnings index cannot represent. The generator docstring and the test module now name it beside council tax and rent, and a test projects a kept Plan 2 loan and a written-off Plan 1 loan through both paths: repayments agree for the first, and only the engine zeroes the second. The docstring also states when policyengine-core applies a variable's `uprating` attribute: only to periods with no stored value. A second test fails if the table gains a salary sacrifice row, as policyengine-uk#1863 would add. The headcount targets then move from survey-year to nominal calibration-year contributions, a methodology choice to make at that point. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review r2 of #541: policyengine-uk carries ages unchanged at load, so ages taken from the base year put the engine's university starts at 2013 and 1983 rather than the 2012 and 1982 the comment names. Deriving them from the projected year makes the cohorts exact; 2012 is the first Plan 2 year. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review of #540 round 2 (Sol 6.1, REQUEST CHANGES): - model_simulates_unknown_region no longer matches error text. A failure with region UNKNOWN reads as "cannot simulate" only if the same household labelled SOUTH_EAST simulates, which is exactly when the loader's relabel helps; otherwise the control's error is raised. The household mirrors the SPI-shaped fixture policyengine-uk's own test_rent_uprating.py simulates. (ParameterNotFoundError.name is reset to None by AttributeError.__init__, so the parameter name is not available to match.) - Probe tests cover unrelated parameters ending in .UNKNOWN, unrelated parameters and schema errors (all raised), and a non-parameter failure that relabelling cures (read as "no"). - The release differential test runs only for an unmodified final release installed from a package index and imported from there: it skips pre, dev and local versions, direct-URL or editable installs, and an economic_assumptions.py that does not match the release's RECORD. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…aises Review of #540 round 3 (Sol 6.1, APPROVE WITH CHANGES, three nits): - The probe docstring says what it establishes: that the imported model can build a simulation of an SPI-shaped household in Region.UNKNOWN. - The release differential test describes its checks as they are: a final release not installed from a direct URL, whose imported economic_assumptions.py is the installed file and matches its RECORD. - The shared-failure test gives the UNKNOWN and SOUTH_EAST households distinct errors and requires the control's, with the UNKNOWN error as its context. A mutant that re-raised the UNKNOWN error had survived. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Merge uk-data 1.58.0 (4cbedbe) into uprating-factors-v2. Resolve both uprating CSV conflicts with the PR generator under policyengine-uk 2.122.2 and core 3.32.13, retaining pension_credit_reported_capital. Regenerate the HMRC income projection and retire the obsolete attribute-based three-decimal pension-capital assertion in favour of all-row engine checks. Preserve restored salary-sacrifice relief, the direct 2,000 pound headcount cutoff and d979 guard, and the batch pension-age mask pipeline wiring.
The release updated pyproject.toml but left the editable package version at 1.57.4 in uv.lock, making uv lock --check fail. Update only that metadata entry; dependency versions and hashes, including UK 2.122.2 and core 3.32.13, remain unchanged.
None of these files carries a towncrier type (<name>.<type>.md), so towncrier left each one in changelog.d instead of compiling it, and several releases record "No significant changes." Every fragment's commit is in a release tagged 1.57.3 or earlier, and none of their text is in CHANGELOG.md. Renaming them with a type would list long-shipped changes under the next version, so they are removed; git history keeps their text. They also kept the old changelog check passing for any pull request, because it only counted files in changelog.d, and they stopped bump_version.py from ever reporting that no fragments were present. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The old check passed whenever changelog.d held any file, so it passed for every pull request while the untyped fragments sat on main. The new check (.github/check_changelog_fragment.py) reads the pull request's own diff, the merge commit against its first parent, and: - rejects any fragment the pull request adds or edits unless it is named changelog.d/<name>.<type>.md with a type from pyproject.toml's towncrier config, or if it is empty; - requires at least one such fragment, added by the pull request, whenever it changes a file a release ships. Changes only to docs/, .github/, .claude/, tools/, changelog.d/, root Markdown files or .gitignore need none, so docs and CI pull requests stay out of data releases. Tests cover the rules with Hypothesis properties, differential checks that every accepted name gets the same type from towncrier's own parser and from bump_version.py, the 32 historical untyped names, and end-to-end runs on temporary git repositories. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CONTRIBUTING.md gains a "Changelog fragments" section: the name form, what each type does to the version, which paths need no fragment, and that merging a fragment releases data. This pull request changes the package's tests, so under its own rule it adds a typed fragment. Merging it is therefore a data release. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review of bb41c3c found two names the check accepted that stop versioning, so the release never starts: - 1.fixed.md with 01.fixed.md: towncrier drops a number's leading zeros and stops with "multiple files for 1.fixed". The check now computes the entry towncrier files each fragment under, across every fragment in changelog.d at the pull request's head, and rejects duplicates. - a name such as "².fixed.md": towncrier's isdigit() passes but int() raises. Names are now limited to ASCII letters, digits, ".", "_" and "-", starting with a letter or digit, which also rules out the whitespace towncrier strips ("foo .fixed.md" against "foo.fixed.md"). Also from the review: - git's path output is read as bytes and decoded with os.fsdecode, so a carriage return or a non-UTF-8 byte in a file name no longer changes or breaks the path. A fragment that is not UTF-8 text is rejected. - The exempt paths are described as a release-policy choice: a change under .github/ can still change how the next release is built. Tests add towncrier differentials over arbitrary Unicode names and over collisions, a real "towncrier build --draft" run, and the repo's Hypothesis settings convention (no deadline, too_slow suppressed). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Second review of 8808d47 approved it with two code notes: - A pull request that deletes every file in changelog.d leaves no such directory in the checkout, and the check raised FileNotFoundError instead of passing. It now treats a missing directory as empty. - The property "a badly named fragment is always rejected" could draw the name .gitkeep, which the check deliberately skips, and fail. The test now skips it and pins that case as an explicit example. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s-pay # Conflicts: # pyproject.toml # uv.lock
…ve the 32 untyped fragments Batch member; reviewed head 6af935f. Batch go: Max d1360 (2026-10-12). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…unknown regions simulable Batch member; reviewed head 4fbd8be. Batch go: Max d1360 (2026-10-12). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Batch member; reviewed head 99916cb. Batch go: Max d1360 (2026-10-12). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…'s load-time uprating Batch member; reviewed head 31888a2. Batch go: Max d1360 (2026-10-12). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Author
|
UK hub (2026-10-12 10:40Z): release CI is green at |
33 of 69 tasks
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The second uk-data release batch. Max gave the go on 2026-10-12 (cos d1360, "Release now"). Merging this PR publishes one release through push.yaml.
Members
Each member's reviewed head is merged unchanged, in the order below. No conflicts came up, and this PR adds no release-only commit.
Each member's own description carries its mechanism, invariants, tests and its build's measured impact. Those builds were each measured on their own base.
Checks before merge (runbook)
Release CI green at the head (
gh pr checksexits 0).Combined seeded build. This branch against main, same harness, seed and engine (policyengine-uk 2.122.2, as locked). It measures the four together:
This is where Limit salary sacrifice to the pay it comes out of #518's hand-off item is checked: the saved file's fit to employment-income bands and employee NICs under the locked engine. The build runs through the UK hub's rebuild queue when free disk is above 100 GB.
Merge with a merge commit at the tested head.
After publish: dispatch
policyengine-uk-data-releaseto policyengine.py for qualification (d1362), then certify the pair (d778 pattern).Lock
uv lock --checkpasses. policyengine-uk stays at 2.122.2, the engine every member was built and measured with.Not in this batch
🤖 Generated with Claude Code