Pin the October 5 audits' leaderboard impact, proposals and answers to the passes' inputs

[MaxGhenis]

Summary

This PR does what #205 does for the reference adversary, for the four other October 5 audits (2026-10-05 SALT withholding, Louisiana, Medicare Part B, payroll). Their scripts/leaderboard_impact.py copied the working tree's run and compared it against the working tree's data.json.gz. Release dashboard-data-20261006 (#202, 9ce4ade8) rewrote that run, so the scripts no longer regenerate their evidence. Three non-engine scripts in the same audits read the run the same way, and this PR pins those too.

No committed evidence, proposal, reference, exclusion or score changes. The diff touches seven scripts, four READMEs and one new test file.

The bug, reproduced

I ran every unmodified script in two scratch clones of main (4db91b5f):

• As main. The run is Stop scoring eight tax outputs and date the response window from the last answer (release dashboard-data-20261006) #202's: payload b1da3eae…, exclusions 92741dfd….
• Pass. Main with the run directory restored from 8b4c0ca1. That commit is Add GPT-6.1 Sol (release dashboard-data-20260930, 46 models) #187, release dashboard-data-20260930, the release each audit's README and policybench/paper_results.py name.

Script	On main	With the run from 8b4c0ca1
2026-10-05/…/leaderboard_impact.py	exit 1: duplicate exclusion for ('scenario_022', federal …). #202 installed all three records.	reproduces all 3 files
2026-10-05-louisiana/…/leaderboard_impact.py	exit 0, rewrites both files: every option scores eight fewer outputs (1,920 instead of 1,928; 1,918 instead of 1,926 for exclude), and the top-five orders and rank changes move	reproduces both files and the log (its stdout)
2026-10-05-medicare-part-b/…/leaderboard_impact.py --weights-check	exit 1: duplicate exclusion for scenario_114 state	reproduces all 3 files
2026-10-05-payroll/…/leaderboard_impact.py --with-salt	exit 1: duplicate exclusion at the salt copy	reproduces all 11 files
2026-10-05/…/propose_exclusions.py	exit 0, rewrites proposed_exclusions.json as "0 proposed, 5 already excluded"	reproduces
2026-10-05-medicare-part-b/…/propose_exclusions.py	exit 1: already-excluded outputs moved	reproduces both files
2026-10-05-payroll/…/model_answers.py	exit 0, rewrites model_answers.csv: scored flips to False on the four outputs' 184 rows, and no other column changes	reproduces both files
2026-10-05-payroll/…/propose_changes.py	reproduces	reproduces (it never reads the run, so it is unchanged here)

The eight records #202 installed are exactly these audits' records: SALT's three, Part B's Virginia record and payroll's four.

The fix

Each script now has one INPUTS table, {repo path: (commit, sha256)}, with an entry for every file it reads from the repository:

• The run. Pinned at 8b4c0ca1: payload, predictions, references, sidecars, exclusions and scenarios. Part B also pins analysis/impact_summary_by_model.csv, and its proposals also pin app/src/modelMeta.ts.
• The audit's own inputs. Pinned at 9ce4ade8, the squash merge that brought these directories in: proposals, sweep CSVs and JSONs, and the classification.

How the scripts read them:

• git_input() / git_bytes() read each file with git show <commit>:<path>, and stop with SystemExit unless the bytes match the pinned sha256. The impact scripts stage every input under <scratch>/pass_inputs/<path> before anything else, and read only that copy. The other three scripts check every pin before they read or write anything.
• --out-dir writes the evidence elsewhere; the default is unchanged. The impact scripts also refuse a --scratch that overlaps the repository, whether inside it, equal to it or containing it, because pass_inputs() and stage() delete and recreate scratch/<name>. The guard, overlaps(), compares filesystem identity (device and inode), so a link or another spelling of case cannot hide an overlap. Pin the reference adversary's leaderboard impact to the pass's inputs #205's guard rejects only the inside case.

#191's SALT records: the payroll script used to read them with an unchecked git show 8af912a0:…. 8af912a0 is #191's head, which exists only on its closed branch, not on main. #202 merged the same bytes (sha256 3c330177…), so the script now reads them from 9ce4ade8. leaderboard_impact.json still records salt_commit: 8af912a0. Part B's committed copy of those records is pinned to the same sha256, and a test checks that #191's head holds those bytes whenever the branch is fetched.

The scoring code (policybench, and scripts/freeze_snapshot.py for Part B) is still the checkout's. Each impact script's published-reproduction check, and the slow test, catch drift there.

READMEs. Each audit gains a "Pinned inputs" section: a table per script with every input, its commit and its full sha256, generated from the scripts' INPUTS. It also says what the unpinned script did on #202's run, and gives the reproduce commands.

Corrections in the touched sentences:

• Step text that listed a leaderboard_impact.log for SALT, Part B and payroll is fixed; none of those logs was committed.
• Part B's "1,928 today" now names the release it scored.

Tests (tests/test_reference_audit_pins.py)

1. Coverage. Each script's pins are exactly the set of files it reads, at the expected commit, and each matches git show. The impact scripts stage exactly those files and remove stale files. The payload is the README's 46 models × 1,928 scored outputs.
2. Differential. Every input shared between scripts has the same pin everywhere. The payload pin equals PAYLOAD_SHA256 in the adversary's definition_conformance.py and publication_sources.py. The SALT records are one sha256 across four pins and equal Audit the state income tax withheld in federal SALT; propose three exclusions #191's head (skipped if 8af912a0 is not fetched).
3. On main. Every pinned commit is an ancestor of HEAD, so no pin depends on a branch that may be deleted.
4. Property (Hypothesis, 80 examples). Flipping, truncating or extending any pinned input of any script is refused, and nothing is returned or written. The unchanged bytes are accepted.
5. Refusal. For every input of every script, a wrong pin stops the script before it scores or writes, and the out dir is never created. Also refused:
   • Stop scoring eight tax outputs and date the response window from the last answer (release dashboard-data-20261006) #202's payload or exclusions;
   • an audit input at 8b4c0ca1, before the audit existed;
   • an unknown commit;
   • a --scratch inside the repository, equal to it, containing it, reached through a link, or spelled in another case (the last runs only on a case-insensitive filesystem).
6. Junk checkout. A git clone --shared --no-checkout in which every input path is junk and every other file is absent, except the script itself (and Part B's scripts/freeze_snapshot.py). The test loads the script from that clone and runs it from inside the clone, with argv[0] naming the clone's copy. So ROOT, HERE and relative paths all point into the junk.
   • Each impact script runs with only analyze (the analyze CLI) stubbed. Every copy starts from the pinned bytes, and each variant changes only the files its records touch (exclusion record, or reference CSV and sidecar). The script writes exactly the committed file names. Every summary field computed from the inputs alone equals the committed one. That includes the always-zero baselines and Part B's legacy impact summaries, whose helpers run for real.
   • The three direct scripts run for real from that clone and reproduce every committed file byte for byte. This needs no mocks and takes a few seconds, so CI runs it.
7. @slow. Each impact script really runs from this checkout, whose run is Stop scoring eight tax outputs and date the response window from the last answer (release dashboard-data-20261006) #202's, and must reproduce every committed file byte for byte; for Louisiana that includes the stdout log. This takes about six minutes in total, so CI deselects it.

Results, all on this branch (outputs saved under ~/reviews/policybench-impact-pins-1005/tests/):

• fast tests for this file: 123 passed at d66e02a2, none skipped;
• full fast suite: 2,132 passed and 8 skipped at 1210c65e. CI runs it again at each head;
• slow tests: 4 passed at d66e02a2, from a checkout whose run Add Claude Haiku 5.5 and move the references to policyengine-us 2.38.6 (release dashboard-data-20261010, 47 models) #208 has rewritten again; they also passed at c86e78c7, 1210c65e and 8e97a320;
• mutation checks. Each mutant puts one read back on the working tree, disables a sha256 check, keeps stale staging, ignores --out-dir, weakens the scratch guard, or moves the pass commit to Stop scoring eight tax outputs and date the response window from the last answer (release dashboard-data-20261006) #202, and must fail a test.
  • mutate.py: 41/41 killed at c86e78c7.
  • At 1210c65e, 51 of 68 mutants ran before I stopped the loop to relieve host disk. 49 were killed. The 2 survivors are equivalent: they drop the up-front pin check in two scripts that read every input before their first write.
  • mutate_review.py: 9/9 targeted mutants for the round-1 and round-2 review findings killed at d66e02a2.

Test runs use a self-cleaning scratch directory rather than tmp_path, which pytest would keep for three runs (about 2 GB).

Invariants

• Pinned inputs. Every byte a script scores, compares or writes from equals the pinned bytes, whatever the working tree holds. Covered by tests 1, 4 and 6.
• Refusal before use. A mismatched input stops the script before any scoring or write. Covered by test 5.
• Reproduction. A regeneration equals the committed evidence byte for byte. Covered by tests 6 and 7.
• Variant isolation. The published copy is byte-identical to the pins, and each variant changes only the files its records touch. Covered by test 6.

Release interplay

No file the Haiku 5.5 release pins is touched. That release has since merged as #208 (dashboard-data-20261010, policyengine-us 2.38.6) and rewrote the run again. This branch is rebased onto it, and the slow regenerations pass from that checkout. #208 only added reference_audit/2026-10-05-louisiana/proposed_exclusions.json in these directories, which this PR leaves alone. Nothing in 2026-10-05-reference-adversary/ changes. #207 moves that audit's pins into scripts/pass_inputs.py; the cross-check test reads either form, so #206 and #207 can merge in either order.

Review

• Round 1 (GPT-6.1 Sol via Subfleet, at 0fbcba80): REQUEST_CHANGES.

  • Should-fix: the scratch guard let through a directory containing the checkout.
  • Should-fix: the junk-checkout tests rebased only ROOT, so a read through HERE escaped.
  • Nit: mocked helpers hid their reads.
  • Nit: a Louisiana count was wrong.

  All four are fixed in 8e97a320.

• Round 2 (at 8e97a320): REQUEST_CHANGES.

  • Should-fix: on a case-insensitive filesystem, a parent spelled in another case passed the lexical guard.
  • Should-fix: the junk-checkout tests still ran from the real checkout, so reads relative to the working directory escaped.

  Both are fixed in d66e02a2.

• Round 3: pending.

Out of scope

• Engine-side scripts in these four audits. That is the sweeps, variants.py, decompose_payroll.py, and Louisiana's check_retirement_exemption.py and verify_convention_module.py. They need the policyengine-us 2.15.17 venv. They read the run, and they build latest_final from 2026-09-28/fixes/, which Stop passing the mortgage interest inputs policyengine-us #9605 deletes #204 changed after these passes. They follow the reference adversary's pass_inputs.py and stage_fixes work (stacked on Pin the reference adversary's leaderboard impact to the pass's inputs #205) and are filed as a follow-up.
• Other audit directories. 2026-10-05-medicaid-031-annotations and the September audits' scripts are filed in the same follow-up.

axiom: n/a: audit tooling; no policy encoding changes.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
policybench-site	Ready	Preview	Oct 10, 2026 12:25pm UTC

[MaxGhenis]

Squash-merged as 2c74e8a2, pinned to head 0bcbba285868132989c9338a6ac43383e2c9e65c:

• Independent review: GPT-6.1 Sol via Subfleet, in four rounds. Round 1 and round 2 returned REQUEST_CHANGES; every finding was fixed. Round 3 approved d66e02a2 with one nit, which 0bcbba28 fixes. Round 4 approves 0bcbba28 with no findings. The verdicts are in ~/reviews/policybench-impact-pins-1005/review*/verdict.md.
• CI: gh pr checks 206 exits 0 on the latest run at this head (run 38051758696: test, lint, app and Vercel). The PR was MERGEABLE and CLEAN, not a draft, with no change requests. Main gained Record two references held against a model consensus, and let the adversary list them (evidence only; no score changes) #213 and Audit the Tax Table against the rate schedule in the federal income tax references (evidence only; no score changes) #214 after that run. Neither touches these scripts or the analyze command, and on a local merge of this head with main cfd851df the 123 fast pin tests pass.
• Tests at this head:
  • 123 fast pin tests pass.
  • The four slow byte-for-byte regenerations pass at d66e02a2. Their scripts are identical at this head.
  • 10 of 10 targeted mutants, one per review finding, are killed.
• No changes to published data: no evidence, proposal, reference, exclusion or score file changes, and nothing Add Claude Haiku 5.5 and move the references to policyengine-us 2.38.6 (release dashboard-data-20261010, 47 models) #208 pins is touched.
• Merge order: Pin the reference adversary's other scripts to the pass's inputs #207 can land before or after this PR. The cross-check test reads either form of the adversary's pins, and passes on a local merge with Pin the reference adversary's other scripts to the pass's inputs #207's head b4f2aac8.