Publish only the policyengine-uk release files whose checks passed

[MaxGhenis]

Stacked on #2269 (test sharding) and includes #2303 (stale #1968 xfail), so their commits show here until they merge. Part of workstream W2 of the Autumn Budget CI plan: no policyengine-uk release reaches PyPI unless the exact files published passed the required checks.

The defect

• Publish raced the tests. Publish in versioning.yaml needed only Versioning and checked out moving main. On 11 October, Test for the merge 3fd35c1 started at 11:35:10Z (run 38136097762). Publish checked out the version-bump child 4d80b29 and uploaded 2.126.6 from 11:36:12Z (run 38136097710), while that Test was still running.
• Versioning raced itself. It failed 20 times in two weeks at "Update changelog"; the sampled logs are non-fast-forward races, and there was no concurrency group.
• The tag script was unchecked. It tagged whatever main was and ran git push --tags || true.
• Main runs were cancelled or skipped. A newer commit cancelled the previous main run, so 83001d5 and cfa0c55 never finished a Test. The main push filter left out pyproject.toml, uv.lock and Makefile.

Change

Release identity (versioning.yaml, with prepare_release.py, release_artifacts.py and publish-git-tag.sh):

• Versioning makes the version commit on the latest main, one run at a time, never forcing. It retries when main moves and fails on any other rejected push. It refuses a commit that changes more than the version line, the changelog and the consumed fragments. It counts fragments the way the tip's bump_version.py does, so it works before and after Fail pull requests that add changelog fragments towncrier drops #2324. A main whose tip is already a version commit has nothing to release.
• Release build checks out that exact SHA and runs uv build once, with the build backend pinned (.github/build-constraints.txt). check-dist proves the wheel and sdist hold exactly the policyengine_uk/ files tracked at that SHA, byte for byte, with that version in their metadata. The files' provenance is attested.
• Release checks run on those files:
  • Release lint;
  • Release install: the wheel with the newest dependencies on Python 3.11–3.14, and the sdist;
  • Release test: the full suite on the installed wheel, in the locked environment, with the dataset required. It removes the checkout's package and confirms the import resolves to the wheel.
• Publish runs only if all of them passed, in the pypi environment, one run per version at a time. It re-verifies the hashes and check-dist, and pypi-state refuses if PyPI holds other bytes under the version. It then uploads the same files and waits until PyPI serves exactly them. Next it sends policyengine.py a policyengine-uk-release repository dispatch with the version (W4's certification; that step can't fail the release). Last, it tags that SHA: annotated, with the digests in the message, idempotent, and never moving a tag.
• Recovery: "Re-run failed jobs" re-tests and publishes the same built files. A release-sha dispatch input re-releases an existing version commit after prepare_release.py --verify checks it.
• Pull requests that change this machinery run Build and the release checks as a dry run.

One test suite (test-suite.yaml and the test-environment action), called by PR, merge-queue, main and release runs:

• It runs in the locked environment (uv sync --frozen --extra dev), from the source tree or an installed wheel. uv.lock is refreshed to what CI and production install (policyengine-core 3.33.2, pandas 3.0.6, pytest 9.1.1); the old lock was stale.
• It has four Python shards, and each shard logs its memory every 60 s.
• The checks are Test / YAML, Test / Python K/4 and Test / Result.
• Mandatory data (W3): merge-queue, main and release runs set POLICYENGINE_UK_REQUIRE_TEST_DATA=1. Test / Result fails if the dataset wasn't present. Pull request runs don't require it.

No silent skips:

• pytest_outcome_log.py records each test's outcome under its raw node id, including subtests. It detects XPASS.
• check_test_shards.py fails on any failure, missing or duplicate run, skip, xfail, XPASS or module skip that test_outcome_allowlist.toml doesn't accept for this mode and dataset. There are seven entries:
  • the two strict xfails (Marriage Allowance is booked on the recipient as the transferor's unused allowance, not the fixed s.55B transfer #1947, Benefit cap, targeted childcare and local CTR use the wrong UC earned income figure #1986);
  • one optional-package skip;
  • two wheel-only skips;
  • microsimulation skips, only in runs without the dataset;
  • the DWP rate test's 66 filter skips.
• xfail_strict=true.
• make test-ci reproduces the check locally.

Required checks become requirable:

• pr_code_changes.yaml runs on every PR to main and on merge_group. A Changes job (changed_code.py) decides whether code changed. If none did, the tests skip and Test / Result passes without running, so a required check never hangs on a path filter. Lint runs on every PR, with ruff check (the same steps as Run ruff check in Lint, with F811 and F821 on for the package #2355).
• Release checks are prefixed "Release" so they can't satisfy the PR checks.
• With the repository variable RELEASE_PUSH_AS_APP=true, the org GitHub App pushes the version commit, so a ruleset can let that App bypass required checks.

Main push (code_changes.yaml):

• It also runs on pyproject.toml, uv.lock, Makefile, conftest.py, .gitignore and README.md.
• It has no concurrency group, so every commit on main finishes its run.
• It skips version commits, which the release tests on the wheel.
• It skips a commit that a successful merge-queue run already tested. A manual dispatch always tests.

Invariants

1. Release identity: the files uploaded to PyPI have the sha256 the Build job recorded. Those are the files Release install and Release test installed, and check-dist proved they hold exactly the version commit's tracked package files. That commit is the one tagged. Covered by test_release_artifacts.py: property tests for any single changed byte, plus fault cases.
2. No overwrite: Publish never uploads when PyPI has a file for the version that isn't ours, byte for byte. classify is checked differentially against an independent statement of the rule.
3. History only grows: the version commit's single parent is the tip it was made on. A race re-versions on the new tip; any other push failure fails. Covered by test_prepare_release.py, using real git with a bare remote, with both bump_version.py versions.
4. Every test ran once and passed, or a reviewed entry accepts its outcome. The checker's verdict equals an independent model of the rule on generated runs, and on real pytest runs with and without xdist (test_ci_test_shards.py).
5. Every PR reports Test / Result: the PR workflow has no path filter, and the code patterns equal the main push filter (test_ci_test_shards.py).
6. The dispatch to policyengine.py follows a confirmed upload and can't block the tag. It runs after PyPI serves the files, its steps have continue-on-error and a 2-minute timeout, and the token is scoped to policyengine.py with contents: write (test_ci_test_shards.py).

Verified

• A local uv build of 6d89338 with the pinned backend reproduces PyPI's 2.126.8 wheel and sdist byte for byte. check-dist passes them against that commit, and pypi-state reports complete against live PyPI.
• On CI, the earlier heads built and checked the release files, and the wheel-mode suite ran in four shards. The checker rejected exactly the outcomes it should: main's dataframe round-trip failure (fixed by Export the identifiers for every year an input frame names, so it round-trips #2349), the Class 4 lower profits limit is £11,908 in 2023-24; NICA 2022 s.2 sets £12,570 from 6 April 2023 #1968 XPASS (Assert the 2023-24 Class 4 lower profits limit instead of expecting it to fail #2303) and the DWP test's filter skips, which are now allowlisted.
• 92 tests in test_ci_test_shards.py, test_prepare_release.py and test_release_artifacts.py pass locally. ruff and actionlint are clean, and shellcheck is clean on publish-git-tag.sh.

Reviews

All in ~/reviews/budget-ci-w2/:

• GPT-6.1 Sol: review-2306-release-out.md and review-2306-suite-out.md; delta review-2306-delta-96e9cecae.md; final review-2306-final-0879f6cf6.md (APPROVE).
• Opus 5.5: delta 0879f6c..496bfdb in review-2306-delta2-496bfdbec.md (APPROVE). Its non-blocking suggestions are applied in 6379d45.

Status

• CI at 6379d45.
• Split the CI Test job into shards and cut its runtime #2269 merges first. This PR then shows only its own commits.
• Admin follow-up through the coordinating session (~/reviews/budget-ci-w2/admin-request.md, decision d1368):
  • a ruleset with the App as bypass actor, then RELEASE_PUSH_AS_APP;
  • a merge queue;
  • required checks Test / Result, Lint, Fix evidence and Changelog fragments;
  • limit the pypi environment to main and move PYPI into it, or use trusted publishing.
• The dispatch reaches a workflow only once policyengine.py#574 (W4) merges; until then GitHub accepts it and nothing runs.

axiom: n/a: CI and release infrastructure

🤖 Generated with Claude Code

[MaxGhenis]

Coordination note from #2324 (untyped changelog fragments, #2314).

#2324 deletes the 93 untyped files in changelog.d. It also changes .github/bump_version.py, which now:

• counts only the fragments towncrier consumes;
• warns about untyped files;
• exits 1 when there is no typed fragment;
• requires a [tool.towncrier] table in pyproject.toml.

It does not edit pr_code_changes.yaml or versioning.yaml, and git merge-tree with release-identity is clean.

On a local merge of both branches, six tests in test_prepare_release.py fail:

• the seed pyproject.toml has no [tool.towncrier];
• test_a_versioned_tip_means_no_release expects an untyped-only merge to be released;
• prepare_release.find_fragments counts every file, so an untyped-only tip would call bump_version.py and fail instead of reporting nothing to release.

A patch for this branch is in ~/reviews/uk-changelog-fragment-check-2026-10-11/w2-prepare-release-typed-fragments.patch. With it, all 64 tests in the two files pass on the merge. It:

• counts fragments with the tip's own bump_version.py;
• has the fake towncrier consume only typed fragments;
• adds the towncrier table to the seed pyproject.toml;
• adds test_untyped_files_alone_mean_no_release.

🤖 Generated with Claude Code