Repository navigation
Report Dependabot runs as skipped in Aggregate impact, not as failures - #2665
Conversation
The measure step exited early for Dependabot, which has no data access, before it copied the tool into $RUNNER_TEMP/tool. The compare step then failed with "No module named pe_uk_aggregate_impact", so every Dependabot pull request failed the check. Set the tool up before any early exit, and test the measure and compare steps as a Dependabot run. Fixes #2664 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The review of #2665 found the second half of the invariant untested: any other same-repository run with no token still fails. The new test runs the measure step as a maintainer with no token and checks it fails with the token error and writes no result. It passes on main and with the fix. Towncrier adds the bullet itself, so the fragment no longer starts with one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Aggregate impact on certified data: passBase Checks: 0 broken by this change, 1 failing on both sides, 0 fixed. Moves flagged for review: 0 (thresholds in Notes
Moves flagged for reviewNone. Spending by benefit and service
Caseloads
Standard reform costings (change in gov_balance)
Standard reforms: change in each reform's target
Household net income by decileNo household's net income changes in any year. No change in any year: receipts by tax head; government balance, each component counted once; model totals; household income totals; population; poverty rates; people in poverty; inequality; mean household net income in each side's own deciles. Checks failing on both base and head (1)
Counted-once checkEach of 88 components was raised by £1 per unit in 2026-27, and every head of its totals was read household by household: its own head must rise by exactly £1, no other tax head may move, no other benefit may rise, and the model's own total must move by what its heads moved. A head that breaks a rule is measured again twice: with every other head held at its baseline value, and with every other head held at its value after the first £1 while the component rises again. It fails where either still shows the response. 0 failed; 0 were skipped.
Provenance
Dataset: Receipts by tax head: head levels
Spending by benefit and service: head levels
Caseloads: head levels
Government balance, each component counted once: head levels
Model totals: head levels
Household income totals: head levels
Population: head levels
Poverty rates: head levels
People in poverty: head levels
Inequality: head levels
Mean household net income in each side's own deciles: head levels
Standard reform costings (change in gov_balance): head levels
Standard reforms: change in each reform's target: head levels
|
|
Merge audit (UK hub, Claude Opus 5.5). Landing at head
|
Fixes #2664
Defect
Every Dependabot pull request fails the Aggregate impact check. The workflow means to report such runs as skipped.
Dependabot runs get no secrets, so
HUGGING_FACE_TOKENis empty. "Measure base, then head" in.github/workflows/aggregate-impact.yamlsees that, writes{"status": "skipped"}tomeasured/head/result.jsonand exits 0. But that exit comes before the three lines that copy the tool into$RUNNER_TEMP/tool. "Compare base with head" then runspython -m pe_uk_aggregate_impact comparewithPYTHONPATH="$RUNNER_TEMP/tool", and fails:That gives
status=1, a "FAIL: The comparison did not finish" comment, and a failed job from "Fail on a structural failure". This happened on #2623 (run 40034956076) and #2624. #2540 passed only once a maintainer pushed to its branch (anth-volk's changelog fragment, then a merge ofmain), which made the runs non-Dependabot ones with data access. The regression came in with fd94415, which moved the tool copy out of the compare step.Fix
tool=,mkdir -p,cp -R) to the top of "Measure base, then head", before any early exit. Nothing else changes.compare()returns statusskipped(compare.py, thehead.get("status") == "skipped"branch).cmd_compareexits 1 only forfail.render()writes "Aggregate impact on certified data: skipped".cpnow runs first, but every pull request head carriespolicyengine_uk/utils/aggregate_impact; a dispatch of a head older than the tool fails at the samecpit always did.Invariant: a Dependabot run passes the check and its comment says it was skipped. Any other same-repository run with no token still fails; fork pull requests skip the job through its
if:, as before.Tests
test_a_dependabot_run_is_reported_as_skipped_and_passesruns the workflow's own "Measure base, then head" and "Compare base with head" scripts, as the file's other step tests do (run_step,bash -eo pipefail). The setup is the one a Dependabot run sees:ACTOR=dependabot[bot]and an empty token;.venv/bin/pythonthat runs the test's interpreter;RUNNER_TEMPandGITHUB_STEP_SUMMARYpointed at the test's temporary directory.It checks:
status=0is written toGITHUB_OUTPUT;test_only_dependabot_may_skip_a_run_with_no_data_accesscovers the invariant's other half. It runs the measure step as a maintainer with no token and checks that it fails with the token error and writes no result.Run locally (pytest, one process, existing venv; ruff 0.17.0, the version
uv.lockpins):main's workflow: fails with the same error as CI:No module named pe_uk_aggregate_impact,assert '1' == '0'.ruff format --checkandruff checkon the test file: clean.Impact
CI only. No model, parameter or data change, so no aggregate moves. The
fixedfragment follows #2522, the last fix to this workflow.#2623 and #2624 have had
mainmerged into their branches, which gives them runs with data access; they don't wait on this PR.Reproducer: policyengine_uk/tests/code_health/test_aggregate_impact_workflow.py::test_a_dependabot_run_is_reported_as_skipped_and_passes
axiom: n/a: CI tooling
🤖 Generated with Claude Code