Repository navigation
Give contract assertions unique ids and declare full-run population coverage - #612
Merged
Merged
Conversation
…verage Two additions to the UK Budget reform contract runner (W1's files) that the qualification gate (#574) needs. Neither changes any check's status, expected value or actual value: - Each case row carries assertion_id, its check label plus its index in the case's checks, so two assertions with the same label (one mandatory, one pending) stay distinct in a report. - budget_reform_population.population_check_routes(routes) declares the exact (check, route) rows a full population run emits. tests/test_assertion_ids_change_no_verdict compares every case, route and three fake routes' rows, without assertion_id, with rows recorded from the runner on main before this change (tests/fixtures/uk_contract/ evaluate_case_rows.json, 1,322 rows), so any change to a verdict fails it. A population test checks the declared coverage equals what a run emits. Co-Authored-By: GPT-6.1 Sol via Subfleet <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 14, 2026
Round-1 review of #612: - population_check_routes declares a completed run's coverage; the early or partial error row ("population", "-") is declared separately as POPULATION_ERROR_DIAGNOSTIC, and a report holding it does not hold. Shared constants name every population check, with byte-identical strings, so labels cannot drift from the runner. - Duplicate country routes declare one baseline-isolation row, as the runner emits. - The golden fixture gains 45 rows from five scenarios that reach the missed verdict branches (matching variable comparisons, both sides of a tolerance, a missing value, a matching pending check, an expected rejection). The original 1,322 rows are an unchanged prefix. The fixture was regenerated from main 503d499's runner twice, identically, and the four reported mutants now fail the golden test. Co-Authored-By: GPT-6.1 Sol via Subfleet <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis
added a commit
that referenced
this pull request
Oct 14, 2026
#612's population stub raised universal_credit under the UC allowance reform but not universal_credit_pre_benefit_cap, so on this branch the payment no longer equalled the award less deductions and #614's accounting assertion rightly failed three coverage tests. The stub now raises the award with the payment. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis
marked this pull request as ready for review
October 15, 2026 02:31
Contributor
Author
|
Merged at 0095d7e on the standard gates. These are W1's runner files, changed under the coordinator's ruling; the change only labels and declares.
|
MaxGhenis
added a commit
that referenced
this pull request
Oct 15, 2026
Round-2 review of #614: - Medium: deductions may not reduce the amount payable below a penny an assessment period (SI 2013/380 Sch 6 para 3(1)(a)). The check now requires every payment, in both runs, to be at least the lesser of its award and 12p a year, so deductions that take a whole award fail. - Medium: every value the check reads must be finite in both runs; a NaN deduction, cap reduction or award no longer passes the comparisons. - Low: the award-decline case lowers the award and its payment together, so only the award condition catches it; new tests cover errors that cancel across units, the penny boundary on gains, the 10% scaling tolerance and a payment left at exactly the floor. Removing any one of the four conditions fails a test. - Low: the population stub has no cap reduction or deductions, so its payments equal its awards at every level. - #612 review Low: assertion ids are tested on extraction-error, pending and unsupported rows too. On the full 1.58.1 file with policyengine-uk 2.130.1 the check holds: 0 award falls, 0 accounting breaks, 0 payments below the floor, 0 non-finite values. All 66 runner tests pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #611
These files belong to workstream W1 (the UK Budget reform contract runner:
budget_reform_contract.pyandbudget_reform_population.py). W1's session is idle, so W4 opens this change for W1's runner by the coordinator's ruling. The coordinator's rule for it: it may only label and declare. No check's status, expected value or actual value changes, and no check is added or removed.What changes
assertion_id, its check label plus its index incase.checks. Two assertions with one label (for example a mandatory and a pending one on the same value) stay distinct in a report. The qualification gate (Qualify UK model and data candidates automatically before promotion #574) needs this so a failing mandatory assertion cannot borrow a pending authorization.budget_reform_population.population_check_routes(routes)declares the exact(check_id, route)rows a full population run emits, so the gate can require the complete multiset.Evidence that no verdict changes
test_assertion_ids_change_no_verdictcovers every case inCASES, every route and three fake routes (hashed values, unchanged values, a failing route). It compares the 1,322 report rows, withoutassertion_id, withtests/fixtures/uk_contract/evaluate_case_rows.json.tests/uk_contract_row_scenarios.py. Recording twice gives identical bytes.population_check_routesequals the rowsrun_population_checksemits on a tiny dataset.Invariants
assertion_idare identical to main's for every case, route and fake route.Independent review: GPT-6.1 Sol, round 1 running (
~/reviews/w4-budget-ci-2026-10-11/runner-ids-round1-sol.md).axiom: n/a: test-harness labelling in the contract runner; no policy change
🤖 Generated with Claude Code