Add Meta Muse Wonder and move the references to policyengine-us 2.41.0 (release dashboard-data-20261013, 48 models) by MaxGhenis · Pull Request #217 · PolicyEngine/policybench

MaxGhenis · 2026-10-12T22:54:27Z

Adds Meta's Muse Wonder as the board's 48th model and moves the references from policyengine-us 2.38.6 to 2.41.0. This is release dashboard-data-20261013, which follows #208 (Claude Haiku 5.5, dashboard-data-20261010). The card PR #216 merged as fe5fc2e. Design and runbook: docs/musewonder/design.md.

What the release does

Muse Wonder: 90.1% of answers within $1 (household-weighted, no tools), Fall back to Main when sensitivity slice is empty for the active country #10 of 48. On the bounded score, which gives partial credit for near misses, it is Update GitHub Actions for Node 24 runtime #3, behind only GPT-6 Sol and Claude Opus 5.5. It is the board's first Meta model.
- Routing: Meta's API rejects a forced tool call, so the row answers in the JSON contract and reasons at the provider default. It runs through OpenRouter, whose one endpoint for this model is Meta's own.
- Board run: 100/100 households, 1,984/1,984 answers parsed.
- Muse Wonder Full has no API (Muse Code only), so it can't join the board. The release note says so.
Engine move to 2.41.0: the first release carrying every fix that release 20261010 was waiting on.
- Nine excluded outputs return to scoring, each within $1 of its audited corrected value. Twelve rechecked outputs stay excluded.
- No scored reference changes by any amount, and nothing is newly excluded.
- Result: 49 exclusions, so 1,935 scored outputs per model. All nine narratives are hand-corrected against the build's trace.
Publication check: 2.41.2, PyPI's newest at 11:34 UTC, gives 2.41.0's value for all 1,984 outputs (2.41.1 did too at 09:39). Main at 2f3053c845 (the next publish) also matches every output.
Effect on earlier models: every one of them falls 0.18–0.42 points. The cause is the nine restored outputs, which few models get right: half are matched by at most one model, and four by none. Without those nine, each model's rate would equal release 20261010's (tested).
scenario_080 SNAP stays excluded (final review round 1, P1). 2.41.0 fixes its heat-and-eat defect (#10033) and gives the corrected $3,240, but the 2026-09-29 recheck found it indeterminate on the stated facts: the prompt states no 7 U.S.C. 2012(j) route for the disabled head, and not whether the $2,000 energy subsidy is LIHEAP or a State-law payment. The actions recheck it; the drafter and the paper now handle such an output (indeterminate_rechecks, INDETERMINATE).
Paper: has a paragraph and a table row for each October move (PaperResults.upgrade_view). No exclusion rests on a defect that 2.41.0 still has, and the abstract, audit paragraph and limitations now say so instead of printing zero. Optional sentences render inside never-empty accessors, so no literal code shows (review P2). The disability section and the September flag counts are corrected (review P1s).

Invariants (load-bearing logic touched here)

Restored-fix grouping (engine_upgrade_restored_fixes):
- Group counts sum to the number of restored outputs, each counted once.
- Each output's record causes, its fix-module causes and its pull requests appear in one group.
- No group repeats a pull request.
- Groups do not depend on the sidecar's order (outputs join transitively).
- Covered by Hypothesis property tests (one found and fixed a gap: fix modules named for non-rNN causes) and the reviewer's order case. Release 20261010's published phrase is unchanged (recomputed from its commit).
Exclusion partition: present defects + unstated inputs + later law = 48, the whole exclusion record. Unfixed + fixed-kept + further-defect = the engine-defect records.
Counterfactual: the new board with the nine restored outputs unscored equals release 20261010's board for every incumbent, to 1e-9.

Checks

Final review round 1 (GPT-6.1 Sol): REQUEST_CHANGES, five findings, all addressed in 2fad06b.
Release-pinned test files pass at 2fad06b; full Python suite re-running.
App: bun test 173 pass, 0 fail. eslint --max-warnings=0, ruff check and ruff format --check are all clean.
Freeze exit 0 after the final render (paper re-pinned).
Open decisions checked at 10:10Z: d1252, d1263 and d1441 are unruled.
Final review round 2 (GPT-6.1 Sol).
CI green and mergeable (needs the release assets published first, as for Add Claude Haiku 5.5 and move the references to policyengine-us 2.38.6 (release dashboard-data-20261010, 47 models) #208).
Publication check re-run inside the US hub lander hold (15:30–17:00Z requested) if a newer release has published.
Merge, mark the release latest, prod check.

🤖 Generated with Claude Code

Meta released Muse Wonder on 2026-10-12 (TechCrunch, VentureBeat). Its API id on Meta's Model API is muse-wonder-1.0. PolicyBench holds no Meta key, and OpenRouter's only endpoint for meta/muse-wonder is Meta itself (provider "Meta", serving meta/muse-wonder-20261012), so the row runs through OpenRouter as Kimi, GLM, Qwen, MiniMax and Inkling do. Priced at Meta's list rate, $4 / $20 per 1M input/output tokens with $0.20 cached input and no long-context tier (Meta's launch post and the OpenRouter listing, read 2026-10-12). The serving card waits on the onboarding probes (forced tool vs auto reasoning, gauntlet, smoke). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…0's audit seed finish_musewonder.py, freeze_musewonder.py and date_musewonder_judge_verdicts.py start from the Haiku 5.5 release's scripts. The driver now names release 20261010 as its base (5a8164a, 47 models, 58 exclusions, policyengine-us 2.38.6), rules no exclusion, and renders new and re-opened cases on judge template v2. Release 20261010's stage, with the 232 verdicts it judged, went with its worktree on 2026-10-10. scripts/recover_20261010_audit.py rebuilds each verdict and prompt from the judge's Claude Code session transcript, and every one hashes to docs/haiku55/judge_provenance.json. The rebuilt 677-case seed's digest is docs/musewonder/seed_digest.csv. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

vercel · 2026-10-12T22:54:33Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
policybench-site	Ready	Preview	Oct 13, 2026 1:37pm UTC

…'s regeneration target Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…stall reference_audit/2026-10-12-engine-upgrade/ adapts release 20261010's builder, sweep timing, narratives and action drafting to base 5a8164a and policyengine-us 2.38.6. Its evidence on 2.38.6 gives the audited targets of the ten cells the newest engine is expected to fix: Idaho 007/053 (#10023), Wisconsin 042/091 (#10026), SNAP 080 (#10033), EITC 100 and 119 (#10030), and CA 099 state with the new r33 educator-expense addback (#10052). A target may name a fix module this audit adds, at its own path (fix_location), and the freeze's target-pin check accepts such a path. The driver renders install-references' audit per case on the template its sidecar records, so the seed's v1 verdicts survive and re-opened cases take v2. New verdicts must be on v2, and the public provenance record states each verdict's template. The freeze counts exclusions on three earlier engines. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

tests/test_finish_musewonder.py ports the Haiku 5.5 driver's tests: - the ruled-exclusion sections become tests that the release rules nothing, so its record is release 20261010's; - the seed tests check the rebuilt 677-case seed against the two Haiku 5.5 records it derives from, and each rebuilt verdict against its transcript; - the upgrade tests regenerate release 20261010's own engine-defect records; - the date conventions name only a wave this release adds. tests/test_build_references_upgrade_1012.py tests the 2026-10-12 builder on a synthetic base with release 20261010's two earlier upgrades, its real base, and this audit's own fix module (r33). It loads its scripts under their own names, so the 2026-10-09 tests' builder never stands in for this one. Driver: triage decisions are dated by the engine upgrade's revision day, and that wave is named whenever the upgrade decides anything (DecisionWave): with no ruled wave, a flag on a regenerated reference could not be affirmed otherwise. The recovery script records each rebuilt verdict's requested judge model, the one its transcript's assistant turns name, which the restate script reads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…e is tested Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…p check A sidecar keeps every engine_upgrade revision, so the committed snapshot (last revision 2026-10-10, built by the 2026-10-09 builder) and the snapshot this release freezes (last revision 2026-10-12) must both pass read_reference_engine_setup. UPGRADE_BUILDERS lists both notes, oldest first; UPGRADE_BUILDER is the newest. Release 20261010's freeze tests pin their mock builds to their own builder's note. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

tests/test_freeze_musewonder.py is test_freeze_haiku55.py on release 20261010's base: 48 models, 58 exclusions (release 20261010's record, byte for byte) and 1,926 scored outputs. Defects that were defects only against a ruled record (staging release 20261010's exclusion or adjudication record) are replaced by edits that still are; the ruled-entry tests go, since the spec rules none. The new model's fingerprint is derived from the registry as validate_treatment derives it, and the committed input pins are stubbed until the board run is pinned. The installed-exclusions refusal names ruled records only when the spec rules any. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The base, the rebuilt seed, judge template v2's mixed tree, the provider route, the ten engine-defect candidates with their 2.38.6 targets, the audit's own fix module (r33), DecisionWave and the open decisions. The run and the engine build are pending. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The board's first Meta model: its label, order and frontier group, the provider's label and mark (@lobehub/icons Meta.Color, in 5.10.0), and its paper display name. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

MODEL_DISPLAY_NAMES must name exactly the frozen roster, which holds 47 models until this release's freeze adds the 48th; the name goes in with it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…findings Review r1 (GPT-6.1 Sol) found two gates a stage could pass wrongly: - validate_verdicts trusted a new verdict's sidecar for its template version. A v1 prompt with bound hashes and a sidecar claiming v2 passed, and so did export and the freeze, which call it. Every judged case's prompt header must now be the version its sidecar records (policybench.audit.template_version_problems), and a new verdict must record v2 through recorded_template_version. - A fix module loads its siblings from its own directory, but the builder and the driver resolved each sibling's name by fix_location, which prefers the 2026-09-22 audit. A local module loading an audited name would run the copy beside it while the pin named the audited bytes. A sibling that resolves to another folder than its module's is now refused. r33 loads no sibling, so its evidence is unaffected. The action drafter called scenario_005 and scenario_120 federal income tax "not fixed": it compared them with their records' 1.755.4-era alternative values. Release 20261010's recheck found both on the corrected value its 2.38.6 evidence gave, and kept them excluded for a second, unlisted-input reason. 2.41.0 has not moved them, so the draft now keeps that finding. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ords return to scoring 2.41.0 was PyPI's newest release when the sweep began (wheel uploaded 01:44:55Z, installed 01:49:51Z, first output 01:51:12Z; verification/sweep_timing.json), and it is the first release carrying all seven fixes release 20261010 waited on (#10023, #10026, #10030, #10032, #10033, #10034, #10052). Its 1,984 computed outputs are identical to a first pass on its release commit de3815f4cc. The reviewed actions (final_actions.json, the build's actions_sha256) regenerate ten excluded outputs onto their audited targets and recheck eleven that move and stay excluded. No scored reference changes, by any amount, and nothing is newly excluded: 48 exclusion records, 1,936 scored outputs per model. The narrative writer's drafts misstated the change behind six of the ten (four said 401(k) deferrals count as EITC earned income, the reverse of #10030), so every narrative is hand-corrected against the build's trace (hand_corrected_narratives.md gives each basis). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>