Add Claude Haiku 5.5 and move the references to policyengine-us 2.38.6 (release dashboard-data-20261010, 47 models)

[MaxGhenis]

Release dashboard-data-20261010. Claude Haiku 5.5 joins the board (47 models), Max's rulings of 2026-10-06 apply (d1022, d994), and the references move from policyengine-us 2.15.17 to 2.38.6. That is the first PyPI release with #10027, #10031, #10032 and #10034 (wheel uploaded 2026-10-10 05:39:47Z, version commit e92f4c6b95). Board snapshot: "Snapshot 2026-10-10".

Board

• Claude Haiku 5.5: 79.576%, Fix multi-entity trace truncation in PE derivation narratives #37 of 47 (n = 1,926). Claude Haiku 4.5: 74.840%, Move frontier/provider filters to ScenarioExplorer; show all models on leaderboard #43.
• GPT-6 Sol leads at 95.583 (95.828 in 20261006), then Claude Opus 5.5 at 94.410 (95.052) and GPT-5.6 Sol at 93.490.
• Every one of the 46 earlier models falls, by 0.24 to 1.37 points. The cause is the 14 outputs that 20261006 excluded and this release scores again: half of them are answered within $1 by at most one of the 47 models, and four by none. Had they stayed excluded, every earlier model would have risen, by 0.39 to 0.71 points. Every incumbent's answers and the household weights are 20261006's (asserted in test_release_20261010_note_claims).
• Sensitivity: Claude Haiku 5.5 with tool_choice: auto gets 89.239%, which would rank Fall back to Main when sensitivity slice is empty for the active country #10. The board row forces the answer tool, and a probe with that request returned no thinking block (sensitivity/claude-thinking-2026-08.md).

What moves

Change	Outputs	Evidence
2.38.6 fixes the engine defects behind them; regenerated and scored	18: 14 excluded in 20261006, plus d1022's four engine-defect cells (AZ 018, OH 025, CO 043, NY 082), which stay scored instead of being excluded. 16 land within $1 of their exclusion record's corrected value. Two (005 state, 082 federal) are held to the audit's fix-module values on 2.37.2.	reference_audit/2026-10-09-engine-upgrade/, sidecar regenerated_exclusions
d1022 scope: dependent must file their own return	4 newly excluded (PA 123, MO 093; federal and state income tax)	reference_audit/2026-10-05-reference-adversary/ (#200); ruled records in docs/haiku55/spec.json
d994: Louisiana's 2026 standard deduction, published after the freeze	2 newly excluded (LA 051, 077)	reference_audit/2026-10-05-louisiana/README.md; docs/haiku55/spec.json
2.38.6 counts Indiana county income tax; no prompt names the county	2 newly excluded (IN 015, 067 local income tax)	sidecar new_exclusions
2.38.6 counts Idaho's $10 permanent building fund tax	ID 076 state income tax: 6,818.34 → 6,828.34 (scored)	sidecar changed
Rechecked on 2.38.6; stay excluded	17, including CA 099 state. It computes at its record's corrected value, but its reference rests on a further engine defect: California does not conform to the federal educator-expense deduction (R&TC 17072). The US hub holds a draft fix.	sidecar excluded_outputs_rechecked

• Each model is scored on 1,926 of 1,984 outputs (was 1,920 = 1,920 + 14 − 8). Exclusions go from 64 to 58 in 42 households (42 unlisted input, 14 engine defect, 2 later law). Adjudications go from 77 to 70: 14 decisions on regenerated records are dropped (the Regenerate the SNAP child support reference (r33) now that policyengine-us#9586 has merged: release 20260922c #178 precedent) and 7 are added.
• 23 references change in all: 18 regenerated, 2 newly excluded, Idaho, and two moves within $1. Each changed output has a corrected narrative (reference_audit/2026-10-09-engine-upgrade/hand_corrected_narratives.{json,md}).
• d1029 (whether v2's income tax outputs should cover only the head and spouse's return) is undecided and v2-only. Nothing here depends on it: d1022 excludes the four dependent-return cells.

Driver, gates, invariants

scripts/finish_haiku55.py runs the release: pin-inputs, prepare, install-exclusions, install-references, judge, adjudicate-exclusions, provenance, triage, export. scripts/freeze_haiku55.py freezes it. The references are built by reference_audit/2026-10-09-engine-upgrade/scripts/build_references_upgrade.py, run through sweep_timing.py run. That wrapper checks the installed engine against its wheel RECORD in the computing process, with a fresh bytecode cache. It refuses symlinks, unrecorded importables and editable installs, and it requires fresh outputs. Six pre-T review rounds (GPT-6.1 Sol) covered this machinery.

Invariants (tests):

• Every scored move is explained by an action. An approved output must move, and an excluded output that moves needs a recheck. A new exclusion must move. The records don't depend on the order of the actions. (test_build_references_upgrade.py, Hypothesis.)
• Installing references is idempotent for any build. The ruled records come out in one order whatever the files' order. The excluded-set equality holds for any dropped decisions. (test_finish_haiku55.py, Hypothesis.)
• The frozen manifest changes only where the release may; every other leaf equals 20261006's. Payload differences are empty exactly when the bytes match. (test_freeze_haiku55.py, Hypothesis.)
• A sweep follows its engine's upload and install. verification/sweep_timing.json: 2.38.6 was the newest release when the sweep began (first output 05:45:24Z) and still the newest at 06:53:19Z. (test_sweep_timing.py.)
• Differential: a recomputation of the board from the scorer equals the payload's exact rates to 1e-9 for all 47 models. The note's facts recompute from the frozen snapshot. (tests/test_notes.py.)
• The restored-target sentence counts both kinds of target for any mix of record and fix-module targets (Hypothesis, with the sidecar's own two fix-module targets).

Asset

Uploaded as non-latest before CI: https://github.com/PolicyEngine/policybench/releases/tag/dashboard-data-20261010

• dashboard-data.json: 128,756,688 B, sha256 f834478e65191f8426ef479506f9f6fe32c2e7107f30fdbf4e7ba0e6acba6410. GitHub's asset digest matches, and so does app/src/data.artifact.json.
• predictions.csv.gz: 43,680,233 B, sha256 0183ab75…, byte-identical to the frozen snapshot copy. GitHub's digest matches.

Validation

• Paper render: exit 0 (HTML and PDF). Re-pinned with --rendered-only.
• App: bun test tests 171/171, bun run lint exit 0. uv run ruff check . and ruff format --check . pass.
• Release tests (test_notes, test_paper_results, test_disclosures, test_adjudications): 151 passed, 2 skipped. The full suite runs in this PR's CI; the local run is held because free disk is under the fleet's floor.
• Independent review: Subfleet 20261010-032143-pb-haiku-final-review (GPT-6.1 Sol, hard tier). Its verdict will be posted here.

After merge: gh release edit dashboard-data-20261010 --latest, then a prod check on policybench.org (47 models, Claude Haiku 5.5 at 79.6%, Snapshot 2026-10-10). Max posts the social drafts.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
policybench-site	Ready	Preview	Oct 10, 2026 7:43am UTC

[MaxGhenis]

Final review, round 1 (Subfleet 20261010-032143-pb-haiku-final-review, GPT-6.1 Sol, hard tier, at cb42e48): REQUEST CHANGES, with four findings. All four are answered in 4e10d12.

1. Blocker: test_reference_exclusions and test_snapshot_artifacts still held rehearsal counts. They now pin T's values. This PR's first CI run also failed 16 more tests, all fixed in the same commit:
   • Rehearsal snapshot dates and an other-tag test in test_freeze_haiku55 and test_finish_haiku55.
   • st_birthtime in sweep_timing.py, which Linux runners lack. It now refuses there rather than date the record by a modification time; the tests use a labeled MOCK source only where birth times are missing.
2. The paper said all 18 fixed outputs return to scoring. It now says the 14 that earlier releases excluded return, and the four ruled on 2026-10-06 while still scored stay scored. That sentence comes from a new accessor with a Hypothesis test. The October 6 paragraphs now say "ruled to exclude". The paper is re-rendered and re-pinned.
3. The social drafts no longer say every model's row forces a tool call.
4. The build's final_actions.json (the sidecar's actions_sha256) is now committed byte for byte. A README erratum notes that its free-text note predates CA 099's move to the rechecks, and a test holds its arrays to the sidecar.

The reviewer found these sound:

• all 1,926 scored references against computed.csv;
• every regeneration target, including both fix-module values, recomputed exactly;
• the engine receipts, the upstream-PR ancestry and the publication check;
• the 23 narratives, and the 70 adjudications and 56 verdicts;
• the board, the note's facts, and the card and paper accessors.

Round 2 (20261010-034130-pb-haiku-final-review-r2) reviews the answers at 4e10d12.

[MaxGhenis]

Final review, round 2 (Subfleet 20261010-034130-pb-haiku-final-review-r2, at 4e10d12): APPROVE. All four round-1 findings are resolved. Two non-blocking nits matter only in a future upgrade: the table cell's 0- or 1-ruled wording, and "restored" naming all 18 fixed outputs. Both are queued for a follow-up.

Merged at 4e10d12 (--squash --match-head-commit) once gh pr checks exited 0 and the PR was MERGEABLE/CLEAN.

After the merge:

• gh release edit dashboard-data-20261010 --latest.
• The production deploy of 5a8164a succeeded. policybench.org shows 47 models and "Snapshot 2026-10-10", with Claude Haiku 5.5 at 79.6% (Fix multi-entity trace truncation in PE derivation narratives #37), Claude Haiku 4.5 at 74.8% and GPT-6 Sol at 95.6%. The serving-sensitivity chip shows 89.2% (Fall back to Main when sensitivity slice is empty for the active country #10).
• /notes/2026-10-10-claude-haiku-5-5-joins-the-board is live.
• The served paper matches the merged pins: web 31cd4141373a, PDF 1607c71abaf3.