Re-judge five audit verdicts the judge-isolation audit could not clear (release dashboard-data-20261013b)

[MaxGhenis]

Release dashboard-data-20261013b: re-judges five audit verdicts that a 2026-09-30 audit of the judges' transcripts could not clear, and changes nothing else. It sits on release dashboard-data-20261013 (#217, bc305793). The Gemini 4 Argon release will take this one as its base.

Why

The audit bound each of the 1,342 published verdicts across two audit trees to the transcript that produced it (docs/judge_isolation_2026-09-30/REPORT.md). An independent re-implementation agreed on every row. Five verdicts cannot be presented as independent of the decisions published beside them, and every release since has carried them over unchanged:

Case	Finding
086 federal_refundable_credits	Its 2026-09-05 judge launched a Workflow whose agents searched the repo and read this case's earlier published case note and row annotations.
045 snap	Its 2026-09-24 judge ran in a git checkout. The checkout's git status, injected into the judge's context, named this case's r33 exclusion and fix.
014 federal_refundable_credits, 015 federal_refundable_credits, 015 head_medicaid_eligible	No transcript holds the published verdict's text.

No decision, exclusion or wording amendment rests on any of the five.

What the release does

• Re-judging: each of the five was re-judged by Claude Opus 5.5 on judge template v2 through scripts/run_audit_claude.sh. Each judge ran in an empty directory outside git, with no tools, an allowlisted environment and the pb-judge token login. Every transcript passes the release driver's checks, plus a session-id binding. All five were judged in one round.
• Results: none flags the reference, every case note stays llm_error, every model row keeps its failure source (kimi-k2.6's two rows stay parse_contract_failure, the rest llm_error), and three model rows change subtype. So there is no new reference question to rule on.
• Wording: two independent reviewers (Claude Opus 5.5 and GPT-6.1 Sol) read the 26 new texts against each case's prompt and policyengine-us 2.41.0, and the round-1 review of this PR found three more scenario_015 Medicaid texts and a misstated EITC end point in scenario_014. Fourteen corrections are applied as listed wording amendments (docs/release_20261013b/wording_amendments.json), each checked again against the engine. For scenario_015's Medicaid pathway that check was a calculation of the household on policyengine-us 2.41.0: the head is not SSI-disabled because $3,750 a month of earnings exceeds the $1,690 substantial gainful activity limit, the pathway's income test passes, and the asset test fails ($584,850 against $2,000). The result is the same with the unlisted meets_ssi_disability_criteria input unset, False or True, so the reference does not rest on it. The committed amendments record is release 20261013's with these fourteen appended.
• Payload 0a4b754a4ce3b249e0608efac62e52646e4c6f33c6a28c1a496c4c9fda49b507 (130,982,477 bytes):
  • every section but scenarioPredictions is byte-identical to release 20261013's, including every model's modelStats, the program stats, the heatmap and the exclusions;
  • inside it, 45 annotation fields change, all in the five cases.
  • The annotation CSVs are release 20261013's committed bytes with only the five cases' records replaced: 21 model rows and five case notes.
• Judge tally: Claude Opus 5 goes from 93 to 92, GPT-5.6 Sol from 269 to 266, and Claude Opus 5.5 from 315 to 319. The paper and the benchmark card say so; the paper is re-rendered and re-pinned.
• Seed: docs/release_20261013b/seed_digest.csv records the new judged tree, all 677 cases. The tree is archived at ~/reviews/policybench-judge-isolation/stage-audit-20261013b for the next release.

The driver is scripts/release_20261013b.py (prepare, judge, collect, export, freeze), modelled on scripts/release_20261006.py. Design, gates and commands are in docs/release_20261013b/design.md.

Invariants (enforced by the driver, tested in tests/test_release_20261013b.py)

• Every verdict outside the five keeps its bytes; the seed must be release 20261013's tree, case for case.
• No model, answer, reference, exclusion, adjudication or serving row changes. modelStats, the program stats and the heatmap are byte-identical.
• The payload may differ only at scenarioPredictions/<one of the five>/<model>/{annotation, failureSource, failureSubtype, caseAnnotation, caseFailureSources, caseFailureSubtypes}. Two exports must give the same bytes.
• Annotation records outside the five keep their bytes and order. Within the five, no record is added, dropped or reordered, and only the verdict-decided columns may change. Comparison is as text, so a blank field is blank on both sides.
• Each re-judge's transcript belongs to its sidecar's session and opens with exactly its prompt. Its one accepted structured answer is exactly the verdict, and it shows no tool call, account data, git status or skill.
• A re-judge that flags the reference, or leaves a scored row outside llm_error, parse_contract_failure and budget_exhausted_at_ceiling, stops the release for Max (reference_questions).
• The freeze starts only from release 20261013's working tree (verify_base_working_tree), and after it writes, the pointer is the one it wrote and the manifest moves only in the release's paths.
• The receipt binds every input and every verdict set aside; the freeze refuses anything changed since export.

Tests:

• Hypothesis properties on the payload diff.
• Splice round trips with blank fields, quoted commas, multi-line fields and Unicode line separators.
• Every sidecar and transcript rejection path.
• The retry and stop paths of the judge loop.
• prepare: a faithful stage, and each refusal (seed, audited verdict, base pointer, grounding, decision/exclusion/amendment overlap, v1 header, and a render that changes more than the header).
• collect's reference-question stop for every non-model-error failure source in the schema.
• The freeze's working-tree checks and the receipt's changed-file, outside-path and set-aside branches.
• Checks on the committed frozen state against release 20261013.

Status

• Re-judge, collect, export, freeze, paper render and re-pin (stage results/local/release-20261013b, receipts in ~/reviews/policybench-judge-isolation/stage-receipts).
• Review round 1 (Claude Opus 5.5, REQUEST CHANGES): every finding addressed in 0fcaeb1c and 80c68a85; re-frozen in 6c44858f.
• Targeted tests on this head: tests/test_release_20261013b.py, test_notes.py, test_paper_results.py, test_judge_template.py and test_run_audit_claude.py, 356 passed, 3 skipped (results/local/release-20261013b-pytest-r2.log). CI test and lint green on 6c44858f; the app job waits on the release assets.
• Review round 2 (GPT-6.1 Sol): queued; every Codex and Claude lane is closed until about 01:57Z.
• Publish the GitHub release dashboard-data-20261013b with dashboard-data.json and predictions.csv.gz (needed for the app's CI job, as for Add Claude Haiku 5.5 and move the references to policyengine-us 2.38.6 (release dashboard-data-20261010, 47 models) #208 and Add Meta Muse Wonder and move the references to policyengine-us 2.41.0 (release dashboard-data-20261013, 48 models) #217). This waits on Max's go (d1510).
• CI green and mergeable, merge, mark the release latest, prod check.

This release adds no site note: it changes no model, reference, exclusion or score. The paper and the benchmark card record the re-judge.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
policybench-site	Ready	Preview	Oct 14, 2026 5:54am UTC

[MaxGhenis]

Merge record for release dashboard-data-20261013b.

• Max's ruling, d1510 (2026-10-14): publish once the round-2 GPT-6.1 Sol review agrees and CI is green.
• Round 1: Claude Opus 5.5 requested changes. All of them were addressed in 0fcaeb1c and 80c68a85, and the release was re-frozen in 6c44858f.
• Round 2: GPT-6.1 Sol approved at 6c44858f. Its two low, nonblocking test-fidelity findings are filed as a follow-up.
• Delta 6c44858f..044418fa: test-only. app/tests/notes.test.ts now pins the newest note to release 20261013, because this release adds no note. Every Codex lane was on hold, so Claude Opus 5.5 checked it in-session and agreed. No payload, annotation, pointer or manifest change.
• Release:
  • dashboard-data-20261013b was created with dashboard-data.json (0a4b754a4ce3b249e0608efac62e52646e4c6f33c6a28c1a496c4c9fda49b507, 130,982,477 bytes) and predictions.csv.gz (bb37b988…6746, 43,861,740 bytes, identical to 20261013's).
  • The downloaded sha256 matched the pointer.
  • The Vercel preview had failed before the upload; it was redeployed and passes.
• CI at 044418fa: test, lint, app, paper and Vercel all pass, and the PR is MERGEABLE.
• Local: 356 passed, 3 skipped (release, notes, paper, judge-template and runner tests).
• Reference questions: none. No re-judge flags the reference, and no failure source changes.

Reviews and evidence are in ~/reviews/policybench-judge-isolation/pr-review/.

🤖 Generated with Claude Code