Keep a judged case's parse-failure label to cases where a model returned no value (d1514)

[MaxGhenis]

Do not merge until Max rules on decision d1514 (his cos decision queue, deadline 2026-10-20). This PR implements option 1. If he picks option 2, change one line (below). If he picks option 3, close this PR unmerged.

What changes

collect_audit already demotes a row-level parse_contract_failure to llm_error when the model has a prediction (_row_failure_source: "Keep parse-contract labels scoped to genuinely missing predictions"). The judge-reviewed branch never applied that rule to the case note: it published the judge's case_failure_source as it came. A case note could therefore say parse_contract_failure while none of its rows did.

The new _case_failure_source helper (beside _row_failure_source) applies the same scoping to the case note:

• It acts only on a judged case whose class is parse_contract_failure, after the existing budget-exhaustion check. It leaves every other class and the deterministic parse_failure_only branch alone.
• Option 1 (CASE_PARSE_FAILURE_NEEDS_MISSING = "any", the default here): a judged case keeps parse_contract_failure only while at least one wrong model is in missing_models. Otherwise it becomes llm_error.
• Option 2 is a one-line switch: CASE_PARSE_FAILURE_NEEDS_MISSING = "all". A case then keeps the label only when every wrong model is missing, which no judged case is. So every judged case-level parse_contract_failure becomes llm_error, including the mixed one.
• Only the case note changes. A row the verdict leaves unlabelled still falls back to the judge's case class, so no row moves under either option.

Published case notes that change

I rebuilt us_case_notes.csv and us_audit_row_annotations.csv the way the release driver's triage does: collect_audit on the frozen audit tree, then the committed adjudications, then the release's wording amendments.

• On origin/main code, the rebuild is byte-identical to the committed CSVs. That holds for release 20261013 (bc30579, the current annotations/) and for Re-judge five audit verdicts the judge-isolation audit could not clear (release dashboard-data-20261013b) #218's 20261013b (6c44858).
• On this branch, under each option, the row CSV is byte-identical to the committed one. Only case_failure_sources changes in the case notes:

Case note	Committed	Option 1 (this PR)	Option 2	Wrong models (missing)	Rows
scenario_012 child1_head_start_eligible	parse_contract_failure	llm_error	llm_error	claude-haiku-5.5 (none)	llm_error/other
scenario_015 head_chip_eligible	parse_contract_failure	llm_error	llm_error	gemini-3.5-flash-lite (none)	llm_error/other
scenario_044 spouse_medicare_eligible	parse_contract_failure	llm_error	llm_error	gpt-5.4-nano (none)	llm_error/other
scenario_080 head_chip_eligible	parse_contract_failure	parse_contract_failure	llm_error	claude-sonnet-5.5, glm-5.2 (glm-5.2)	claude-sonnet-5.5 llm_error/other; glm-5.2 parse_contract_failure/missing_output

• Option 1 changes three notes, option 2 four, option 3 none. The results are the same on both trees.
• The subtypes stay other and the case annotation text is unchanged.
• In 012, 015 and 044, the model's explanation derives 0 but it submitted 1. None of the four cases is adjudicated.
• The other 312 case-level parse_contract_failure notes are deterministic parse_failure_only cases, and every row in them is parse_contract_failure. Neither option touches them.

Nothing here edits a frozen audit tree or a release artifact. The change reaches publication only when a future release driver runs triage and freezes.

Evidence: ~/reviews/policybench-budget-line-20261013/pr-a-impact/ holds the scripts, the rebuilt case notes per option and their sha256. Background is in REPORT.md ("Separate findings" 1) and sol_review_r3.md (findings 4 and 6).

Invariants

These are stated and tested with Hypothesis in test_judged_case_parse_label_properties. It runs 300 generated cases: one to four wrong models, each with or without a value, a missing model sometimes recording budget_exhausted_at_ceiling, and any case class, row class and subtype the schema allows, with or without per-model entries.

1. The rule (option 1): for a judged case with no wrong model in missing_models, the published case_failure_source is never parse_contract_failure.
2. Option 2: for a judged case with any wrong model holding a value, it never is.
3. Nothing else moves. This is checked differentially against collect_audit with the rule disabled, which is today's output. Under either option:
   • the rows and every other case column are identical;
   • case_failure_source differs only as parse_contract_failure → llm_error, and only where the option's condition fails.

Deliberately not asserted: "the case's class is among its rows' classes". Under option 1 it does not hold, by design. test_case_label_may_match_no_row_while_a_model_is_missing pins the counterexample (sol_review_r3 finding 4) as intended behaviour:

• one parsed wrong answer is normalised to llm_error;
• one missing answer keeps its recorded budget_exhausted_at_ceiling;
• the judge's case-level parse_contract_failure stands.

Mutation check: the property alone catches each of five mutants, and so do the example tests. The mutants:

• the rule not applied;
• the rule also feeding the row fallback;
• the rule demoting another class;
• the rule demoting to the wrong class;
• the two options swapped.

Tests

• tests/test_audit.py:
  • five example tests: the 012/015/044 shape, the 080 shape under both options, the counterexample, the unlabelled-row fallback, and other classes unchanged;
  • the property.
• Locally, pytest passes on test_audit, test_judge_template, test_adjudications, test_annotation_rewrites, and the three freeze and finish_musewonder suites: 1,393 passed, 19 skipped.
• ruff check . and ruff format --check . are clean.

axiom: n/a: benchmark annotation pipeline, no policy rule

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
policybench-site	Ready	Preview	Oct 13, 2026 11:35pm UTC