Repository navigation
Conversation
…igest The seed digest pinned each judged case's prompt and verdict, but not the verdict.meta.json sidecar, whose judge_model_requested, judge_runner and judged_at_utc feed the snapshot manifest's judge tally. A kept sidecar rewritten with its verdict_sha256 kept, and its receipt hash updated, passed every gate. - docs/gpt61sol/seed_digest.csv records each of the 674 judged seed cases' sidecar sha256 (meta_sha256), recomputed from the 20260929 audit; SEED_DIGEST_SHA256 pins the new bytes (e80ec95e...) - validate_verdicts, which judge, triage, export and the freeze run, refuses a carried-over verdict whose sidecar is not the seed's - verify_seed refuses a binding without the sidecar hashes; bind-seed re-binds a stage bound before the digest recorded them, and only then - the real stage's stage.json already binds the three-field seed, and all 540 kept sidecars match it, so the stage needs no re-binding Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ns to release 20260929's us_case_reference_explanations.csv could change unnoticed: the receipt did not require it, nothing compared it with release 20260929's, and when the freeze's scratch bundle lacked it, export_full_run's loader quietly read the working directory's committed annotations/<RUN>/ copy instead, so the rebuilt payload still matched. - the freeze's receipt must bind all four annotation files, the case reference explanations included - verify_reference_explanations compares a copy with the file committed at BASE_COMMIT, byte for byte; export checks the staged copy before writing anything, the freeze checks it before any mutation and checks the committed copy after it writes the snapshot - build_payload, which export and the freeze's rebuild share, refuses a bundle that lacks any file export reads (EXPORT_INPUTS), so neither ever falls back to the working directory's annotations Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…against its current verdict verify_restatements checked a re-opened adjudication's judge, dates and judge_previous, but not its top-level judge_reference_suspect and judge_reference_suspect_source, and verify_adjudications_keep_judge_verdicts accepts any raised flag that names a source. So a re-opened entry could say the judge flagged the reference, naming an earlier run, when neither the current verdict nor the 2026-09-22 wave did. - the restate script's reference_flag is now the one definition: the flag is raised when the verdict or the 2026-09-22 wave raised it, and names a source exactly when the wave alone did; the restated top level and the judge_previous item for the replaced verdict both use it - verify_restatements requires it of every re-opened entry, restated or not, against the case's current bound verdict and the wave flags read at BASE_COMMIT; a named source must keep 20260929's wording for the entry or be the script's FLAG_SOURCE_EARLIER_RUN, which a restatement writes when the entry it restates names none (the stage's scenario_022 state income tax entry carries the script's wording in place of 20260929's; the staged record already did before its last restatement) - a Hypothesis differential test restates entries through any sequence of re-judges and checks the gate accepts exactly what the script writes; the stage's 54 restated entries pass (local test) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…fore the freeze The freeze's receipt binds the stage only by a list of hashes stored beside it in release-ready.json, so a stage edited after export, with its receipt re-hashed to match, passed every check that reads the receipt. - export writes docs/gpt61sol/release_receipt.json after release-ready.json: the release tag and the sha256 of the staged payload and of release-ready.json - the freeze reads that file as committed at HEAD (git show HEAD:...), with the working-tree copy required to be HEAD's, straight after the receipt check, and refuses unless it names the tag, the payload and the receipt; so any edit to the stage after export also needs a visible commit - design.md documents the order: export, commit the pin, then freeze - the export test fixture writes its pin to a scratch file, never the repository's Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Export, run again on a copy-on-write clone of the archived GPT-6.1 Sol stage under all round-3 gates, rewrote release-ready.json byte for byte as the archive holds it (sha256 e8e0a51f...) and the payload as published (d1cae745..., the live pointer's). This commits the pin export wrote, so the freeze's check of a committed pin holds for the release as shipped. - tests: the committed pin names the live pointer's tag and payload, in the form export writes (runs anywhere); it names the stage's receipt (local) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
The round-3 review (finding 1) showed the judge tally unit 4 protects was still open by a side route: freeze_snapshot.audit_judge_provenance counts every audit case directory holding a verdict.json, from its sidecar, while validate_verdicts and the receipt cover only the cases cases.jsonl lists as judged. A verdict and bound sidecar added after export to an unlisted or a parse-failure-only case changed the snapshot manifest's tally and passed every gate (reproduced on ceb84ee: the freeze ran past it). - stray_verdicts names any case directory with a verdict.json or verdict.meta.json that is not a judged manifest case; validate_verdicts, which judge, triage, export and the freeze run, refuses one. The real stage has 674 case directories, all judged - the freeze comment and design.md no longer say the receipt pin covers any edit to the stage: it covers files the receipt binds (finding 2); the design row says a pin commit shows that the receipt changed, not what changed in the stage (finding 3) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Author
|
Heads-up: #197 edits #197 corrects one reference explanation, scenario_031
The driver gates (export, freeze rebuild, freeze) check the stage's copy and are unaffected. Only the committed-copy test reads HEAD's working tree. Two ways to keep its intent ("the 20260930 freeze wrote 20260929's explanations") without blocking later corrections:
If #190 merges first, I'll make the change in #197. If #197 merges first, this PR needs it on rebase. Separately, the 20260930b re-judge driver builds from release 20260930's committed annotation bytes. Its freeze refuses a working tree that differs from them, and its gate says the payload differs from the base only in its five cases. Both need to accept #197's ledger once #197 is on main. I've told the session that owns 20260930b. |
Contributor
Author
|
Release |
The second independent review (results/local/adds202609/gpt61sol/verify/ round3_rereview.md, with a shell) reproduced two bypasses and some prose that claimed more than the code does. - stray_verdicts now uses the judge tally's own test, (case / name).is_file() over the case directory's entries: on APFS a Verdict.json passed the glob but was counted by audit_judge_provenance, and a full freeze then published a judge tally of 675 (finding 1). A Hypothesis test checks, on any file system, that every case directory the tally counts is a judged case or one stray_verdicts names - build_payload, which export and the freeze's rebuild share, refuses a bundle holding a file export would read beyond EXPORT_INPUTS: any other file in annotations/ (load_annotations reads every us_*_annotations.csv) and a us/by_model/ folder (preferred to us/predictions.csv). An extra annotations file added before export put an annotation no judge gave on a model's row, and every gate passed (finding 2, pre-existing) - the freeze's post-write checks of the frozen references and explanations move into verify_frozen, which a test now covers (finding 5) - the receipt-pin test no longer flakes on git's background maintenance (finding 3); the flag table covers all 16 combinations - the EXPORT_INPUTS comment, the missing-file and stray refusals, a test name, a docstring and design.md say only what the code does (finding 4) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ates Conflict in tests/test_freeze_gpt61sol.py: keep main's release_blob helpers beside the committed receipt pin's reader. The round-3 tests that read the working tree's references now write release 20260930's committed references (release_references), as main's tests do. Two tests that read files later releases changed now read release 20260930's commit (RELEASE_COMMIT): the committed reference explanations, and the pointer the receipt pin names. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…20260929's The third review (cut off by its provider's filter, its partial output kept at ~/.subfleet/jobs/20261010-180804-pb190-round3-review3/a1/deliverable.md) reproduced that an incumbent's request_completed_at edited in the bundle's predictions passed the freeze: its evidence hash covers only the answer columns, while main's model_response_window reads every row's request_completed_at and analysis.usage_summary_by_model its cost, token, elapsed and timestamp columns, and the frozen predictions.csv.gz publishes them all. - incumbent_row_differences compares every column of every incumbent row, as text, keyed by model, scenario and variable, with release 20260929's predictions read from git at BASE_COMMIT; a column only the bundle has must be empty on incumbent rows. Key equality is compared as values, so a string-dtype frame and an object one agree - export and the freeze run it (verify_incumbent_rows); on the real stage all 26 columns of the 89,280 incumbent rows are the committed text Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…usts it The fourth review (GPT-6.1 Sol, verify/round3_review4.md) reproduced on the real stage that cases.jsonl was trusted: appending GPT-6.1 Sol's correct answer to a parse-failure-only case's wrong_models, and changing a recorded failure source, passed triage, export and the freeze's dry run, and the payload labelled the correct answer parse_contract_failure. collect_audit builds those cases' annotations from the manifest alone, and verify_annotation_amendments rebuilt them from the same file. - manifest_differences calls build_audit_cases, which prepare_audit wrote the manifest with, on the bundle's us/ directory and compares every field of every row but grounding (rendered into the bound prompts and read by nothing published); export and the freeze refuse a difference. On the real stage all 986 rows match - the freeze refuses an extra annotations file or a us/by_model/ folder in the stage's own bundle before anything reads it, bound or not - a predictions column release 20260929 lacks is allowed only as a column of a new model's pinned run file, empty on incumbent rows (review item 2) - the unexpected-file docstring and refusal, design.md and a test docstring say only what the code does (items 3 and 4) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#210's test_the_committed_prompt_records_agree_and_name_a_version read docs/gpt61sol/seed_digest.csv with its old three-column header, which unit 4 extended with each kept case's sidecar sha256 (meta_sha256). The reader now accepts that column beside the three it compares, and still requires one of the two headers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Lands units 4 to 7 of the third review's fix brief for the GPT-6.1 Sol release driver (
scripts/finish_gpt61sol.py,scripts/freeze_gpt61sol.py). Units 1 to 3 shipped with releasedashboard-data-20260930(#187). That release has shipped, so nothing here re-freezes or re-publishes it. These gates harden the driver for the follow-up re-judge release (judge-isolation-20260930b, whose driver imports this one) and any later single-model release.The review's point was that the freeze rebuilt the payload from the staged bundle, but bound the bundle only by a list of hashes stored beside it. Each unit makes the freeze re-derive one more input, or pins it outside the stage.
Units
Each unit is one commit, and its new tests fail on the commit before it.
[review-fix kept-sidecars]d66e6e01docs/gpt61sol/seed_digest.csv) now pins each of the 674 judged seed cases'verdict.meta.jsonsha256.validate_verdicts, which judge, triage, export and the freeze run, refuses a kept sidecar that is not the seed's. Before, a kept sidecar'sjudge_model_requestedcould be rewritten (withverdict_sha256kept and the receipt re-hashed) and the snapshot manifest's judge tally would change.[review-fix reference-explanations]03dbdfacus_case_reference_explanations.csv. Export and the freeze require it to be release 20260929's (git atBASE_COMMIT) byte for byte.build_payload, shared by export and the freeze's rebuild, refuses a bundle that lacks any file export reads (EXPORT_INPUTS). So neither can fall back to the working directory'sannotations/<RUN>/any more.[review-fix reference-flag]ac44dfb3verify_restatementschecks every re-opened entry's top-leveljudge_reference_suspectandjudge_reference_suspect_sourceagainst its current bound verdict and the 2026-09-22 wave (read atBASE_COMMIT). It uses the restate script's ownreference_flag, the one definition: the flag is raised when the verdict or the wave raised it, and names a source exactly when the wave alone did.[review-fix receipt-pin]f6640978release-ready.json, export writesdocs/gpt61sol/release_receipt.json(tag, payload sha256, receipt sha256). The freeze reads it withgit show HEAD:, requires the working-tree copy to match, and refuses unless it names the stage's tag, payload and receipt. A stage edited after export, with its receipt re-hashed, now needs a visible commit.ceb84ee18a3108c6validate_verdictsrefuses averdict.jsonor sidecar in any case directorycases.jsonldoes not list as judged. The snapshot manifest's judge tally counts every case directory holding a verdict, so such a file, added after export, changed the tally past every gate (reproduced onceb84ee1). The pin's documented scope is narrowed to files the receipt binds.6f80237bstray_verdictsuses the tally's ownis_file()test, so a case-variantVerdict.json(counted on APFS) is refused.build_payloadrefuses an extra file inannotations/(export reads everyus_*_annotations.csv) or aus/by_model/folder; this gap predates the PR and was reproduced. The post-freeze checks are now a testedverify_frozen. A flaky test is fixed and the flag table is complete.2ad5d801origin/main(releases 20261006 and 20261010). Tests that read files later releases changed now read release 20260930's commit.abf98242request_completed_at, and the usage summary reads the cost, token and timing columns. Export and the freeze now compare every column of every incumbent row with release 20260929's committed predictions. On the real stage, all 26 columns of all 89,280 rows match.bf827abbcases.jsonlwas trusted. Editing a parse-failure-only case'swrong_modelsand recorded source published GPT-6.1 Sol's correct answer as a parse failure. Export and the freeze now rebuild the manifest withbuild_audit_casesfrom the bound bundle and compare every field butgrounding; all 986 real rows match. The freeze also refuses extra files in the stage's own bundle, and a bundle-only predictions column must come from the new model's run.310a162forigin/main(#210 judge-template versioning, #212, #206). Clean merge.a0ccd0batests/test_judge_template.pyreaddocs/gpt61sol/seed_digest.csvwith its old three-column header, and CI failed on it. The test now reads by column name and accepts unit 4'smeta_sha256column.Invariants these gates enforce, and their tests
SEED_DIGEST_SHA256(e80ec95e…). Tests:test_a_kept_sidecar_must_be_the_seeds,test_bind_seed_*,test_the_freeze_refuses_a_verdict_edited_after_export[edited_kept_sidecar-*].test_the_freeze_refuses_edited_reference_explanations(entry removed, and entry re-hashed),test_the_rebuild_refuses_a_bundle_file_export_reads_and_the_receipt_omits,test_export_refuses_*.test_the_freeze_accepts_exactly_the_reference_flag_restatements_write). Through any sequence of re-judges, the gate accepts what the restate script writes, and refuses the same entry with its flag or source flipped. It also acceptsFLAG_SOURCE_EARLIER_RUNin place of 20260929's own source wording.test_the_freeze_refuses_a_stage_edited_after_its_pin_was_committed, andtest_the_freeze_reads_the_receipt_pin_as_committed_at_head(real git: a pin never committed, then committed, then rewritten and not committed, then committed but naming another payload).docs/gpt61sol/design.md's invariants table has a row or an amendment for each gate.The real stage
The archived stage was not written. The freeze requires a stage inside the checkout, and the 20260930b driver checks the archive's receipt file by file. So the run used an APFS copy-on-write clone of it. Its
stage.jsonalready bound the three-field seed (an earlier partial attempt at unit 4 re-bound it), and all 540 kept sidecars match.--step exportreproducedrelease-ready.jsonbyte for byte, as the archive holds it (e8e0a51f…). It also reproduced the payload as published (d1cae7456cf91ab6fa04644a4d5d570e359522386a7c52f9645d5923bfc11258, the live pointer's). GPT-6.1 Sol: exact 90.595091%, rank 8 of 46, n = 1,928, as released.--dry-run(atceb84ee1,8a3108c6and6f80237b, all before the merge; after it, main's committed references are policyengine-us 2.38.6's, so the reference gate refuses the working tree, as it should):Validated local release inputs: dashboard-data-20260930, 46 models, d1cae745…; references unchanged; 0 adjudications added; 394 wording amendments. The working tree was unchanged.No published number or file changes.
Tests
The suite was run serially at
310a162f:test_finish_gpt61sol.py268,test_freeze_gpt61sol.py224,test_restate_gpt61sol_adjudications.py34,test_run_audit_claude.py98 andtest_judge_provenance.py10 passed, and ruff is clean. The local-only tests ran against the clone.Independent review
The first review (an Opus 5.5 subfleet lane with no shell,
verify/round3_review.md) requested changes. Responses:8a3108c6.FLAG_SOURCE_EARLIER_RUNwhere 20260929's entry had its own wording: kept, because the real stage's scenario_022 state income tax entry carries it. The flag and whether a source is named stay exact.stage.json): corrected.The second review (
verify/round3_rereview.md, with a shell) requested changes, all addressed in6f80237b. A third review, on GPT-6.1 Sol, was refused by the provider's content filter partway through, but its partial output reproduced the incumbent-row gap fixed inabf98242. The fourth review (verify/round3_review4.md, GPT-6.1 Sol with a shell) confirmed the earlier fixes and found the manifest gap, fixed inbf827abb. A fifth review is running. Its first attempt was lost to a lane network fault, so it was dispatched again.Follow-ups (not in this PR)
judge-isolation-20260930b) calls none of these gates yet. It needs to adopt them when it rebases onto main.🤖 Generated with Claude Code