Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
8a81d23
Add Meta Muse Wonder's roster entry, price and release date (card WIP)
MaxGhenis Oct 12, 2026
a32d172
WIP: start the Muse Wonder release driver and rebuild release 2026101…
MaxGhenis Oct 12, 2026
23e1dd7
Add the CA educator-expense addback fix module (r33) for scenario_099…
MaxGhenis Oct 12, 2026
ac05eef
WIP: second-upgrade builder, evidence on 2.38.6 and mixed-template in…
MaxGhenis Oct 12, 2026
0271aca
WIP: port the driver and builder tests to release 20261010's base
MaxGhenis Oct 13, 2026
0fb944a
Test the 2026-10-12 audit's sweep timing script, as the 2026-10-09 on…
MaxGhenis Oct 13, 2026
8a7086d
Accept both later builders' notes in the freeze's engine-upgrade setu…
MaxGhenis Oct 13, 2026
cd71b64
Port the freeze tests to a release that rules nothing
MaxGhenis Oct 13, 2026
e569ffb
WIP: the Muse Wonder release's design note
MaxGhenis Oct 13, 2026
85a3593
Add Meta as a provider and name Muse Wonder in the app and the paper
MaxGhenis Oct 13, 2026
3030df9
Leave Muse Wonder's paper display name for the freeze
MaxGhenis Oct 13, 2026
82556d9
Address the infrastructure review's two P1s; keep 20261010's recheck …
MaxGhenis Oct 13, 2026
f9e141b
Build the references on policyengine-us 2.41.0: ten engine-defect rec…
MaxGhenis Oct 13, 2026
569437d
Record the 2.41.0 build in the design note
MaxGhenis Oct 13, 2026
192835d
Date-qualify the SNAP thresholds and show the narratives' roundings (…
MaxGhenis Oct 13, 2026
d087c7e
Add Meta Muse Wonder's model card: the JSON contract, since Meta refu…
MaxGhenis Oct 13, 2026
e6f69d4
Merge the Muse Wonder card into the release branch
MaxGhenis Oct 13, 2026
384321e
Pin Muse Wonder's board run; match the id OpenRouter reports
MaxGhenis Oct 13, 2026
812d538
Format the Muse Wonder card
MaxGhenis Oct 13, 2026
2f91627
Merge the card's formatting fix
MaxGhenis Oct 13, 2026
c198c8b
Lint the release scripts; point the live-stage tests at the real stage
MaxGhenis Oct 13, 2026
925e5fc
Describe why the card needs no reasoning-suppression sensitivity run …
MaxGhenis Oct 13, 2026
4b0d924
Merge the card's wording fix
MaxGhenis Oct 13, 2026
8902156
Accept judge verdicts from Subfleet lane claude-10's login
MaxGhenis Oct 13, 2026
0f680e2
Merge main (the squashed card, #216) into the release branch
MaxGhenis Oct 13, 2026
3ef4590
Record the provenance of the 151 judge verdicts this release re-opened
MaxGhenis Oct 13, 2026
39808af
Freeze release dashboard-data-20261013: 48 models, references on poli…
MaxGhenis Oct 13, 2026
9171817
Write the Muse Wonder release: note, paper, card and sensitivity note…
MaxGhenis Oct 13, 2026
2fad06b
Keep scenario_080's SNAP excluded and fix the paper's review findings…
MaxGhenis Oct 13, 2026
d25b8fc
Apply the final review's round-2 wording fixes
MaxGhenis Oct 13, 2026
0c5236a
Re-run CI and the Vercel preview now that release dashboard-data-2026…
MaxGhenis Oct 13, 2026
88bf8cb
Pin the "all five" wording and re-wrap paper.md (delta review nits)
MaxGhenis Oct 13, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Binary file modified app/public/paper/policybench.pdf
Binary file not shown.
Binary file modified app/public/paper/web/figures/positive_zero_scatter.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
957 changes: 492 additions & 465 deletions app/public/paper/web/index.html

Large diffs are not rendered by default.

2 changes: 2 additions & 0 deletions app/src/components/ProviderMark.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ import {
Anthropic,
DeepSeek,
Google,
Meta,
Minimax,
Moonshot,
OpenAI,
Expand Down Expand Up @@ -78,6 +79,7 @@ export default function ProviderMark({
{provider === "anthropic" && <Anthropic size={size} color="currentColor" aria-hidden="true" />}
{provider === "deepseek" && <DeepSeek.Color size={size} aria-hidden="true" />}
{provider === "google" && <Google.Color size={size} aria-hidden="true" />}
{provider === "meta" && <Meta.Color size={size} aria-hidden="true" />}
{provider === "minimax" && <Minimax.Color size={size} aria-hidden="true" />}
{provider === "moonshot" && <Moonshot size={size} color="currentColor" aria-hidden="true" />}
{provider === "openai" && <OpenAI size={size} color="currentColor" aria-hidden="true" />}
Expand Down
8 changes: 4 additions & 4 deletions app/src/data.artifact.json
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
{
"asset": "dashboard-data.json",
"bytes": 128756688,
"bytes": 130988798,
"repo": "PolicyEngine/policybench",
"sha256": "f834478e65191f8426ef479506f9f6fe32c2e7107f30fdbf4e7ba0e6acba6410",
"tag": "dashboard-data-20261010",
"url": "https://github.com/PolicyEngine/policybench/releases/download/dashboard-data-20261010/dashboard-data.json",
"sha256": "aef5dc73a76433c97994430f8eb6dfad6bdc6518b18ee3ea8f79cee704757477",
"tag": "dashboard-data-20261013",
"url": "https://github.com/PolicyEngine/policybench/releases/download/dashboard-data-20261013/dashboard-data.json",
"version": 1
}
4 changes: 2 additions & 2 deletions app/src/data.versions.json
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,10 @@
"artifact": {
"pointer": "live"
},
"description": "Scored reference outputs from policyengine-us 2.38.6 (the 58 excluded outputs keep the values they were decided on: 38 from policyengine-us 1.755.4, 18 from 2.15.17, 2 from 2.38.6, and the 17 that move on 2.38.6 were re-reviewed and stay excluded); same household facts and requested outputs for every model; answer transport and request shape per model as recorded in the serving-configuration table - 47 models",
"description": "Scored reference outputs from policyengine-us 2.41.0 (the 49 excluded outputs keep the values they were decided on: 29 from policyengine-us 1.755.4, 18 from 2.15.17, 2 from 2.38.6, and the 12 that move on 2.41.0 were re-reviewed and stay excluded); same household facts and requested outputs for every model; answer transport and request shape per model as recorded in the serving-configuration table - 48 models",
"id": "1.1",
"label": "1.1",
"snapshotLabel": "Snapshot 2026-10-10"
"snapshotLabel": "Snapshot 2026-10-13"
},
{
"artifact": {
Expand Down
4 changes: 2 additions & 2 deletions app/src/lib/referenceEngine.ts
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,8 @@ import type { ReferenceExclusion } from "../types";
* the frozen sidecar.
*/
export const ENGINE_UPGRADE_RECHECK = {
engineVersion: "2.38.6",
rechecked: 17,
engineVersion: "2.41.0",
rechecked: 12,
};

function compareVersions(a: string, b: string): number {
Expand Down
14 changes: 8 additions & 6 deletions app/src/lib/servingSensitivity.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,9 @@
* reasoning in both runs, so its comparison is one of transport. Claude Opus
* 5.5 and Claude Sonnet 5.5 reject forced calls the same way and have no auto
* re-run. Claude Haiku 5.5 accepts forced calls, so its board row is a
* forced-tool row like Claude Opus 5's. Scores are
* forced-tool row like Claude Opus 5's. Meta's Muse Wonder, not a Claude row,
* rejects forced calls too and reasons on JSON, so it has no re-run either.
* Scores are
* the pinned three-decimal measurements from sensitivity/data/*.json; round
* only for display. Each entry's "would rank" is derived from the live board
* rows at render time, never typed by hand.
Expand Down Expand Up @@ -36,30 +38,30 @@ const FORCED_TOOL =

export const SERVING_SENSITIVITY: Record<string, ServingSensitivity> = {
"claude-fable-5": {
autoExact: 91.746,
autoExact: 91.451,
boardTreatment: `${FORCED_TOOL}, one output per request`,
autoTreatment:
"re-run with tool_choice: auto and the whole household in one request",
thinkingSuppressedOnBoard: true,
noteHref: SENSITIVITY_DOC_HREF,
},
"claude-opus-5": {
autoExact: 90.456,
autoExact: 90.109,
boardTreatment: FORCED_TOOL,
autoTreatment: "re-run with tool_choice: auto",
thinkingSuppressedOnBoard: true,
noteHref: SENSITIVITY_DOC_HREF,
},
"claude-sonnet-5": {
autoExact: 84.953,
autoExact: 84.574,
boardTreatment: `${FORCED_TOOL}, one output per request`,
autoTreatment:
"re-run with tool_choice: auto and the whole household in one request",
thinkingSuppressedOnBoard: true,
noteHref: SENSITIVITY_DOC_HREF,
},
"claude-fable-5.1": {
autoExact: 92.044,
autoExact: 91.748,
boardTreatment:
"rejects forced tool calls, so its row answers as a JSON object and reasons at the provider default",
autoTreatment:
Expand All @@ -68,7 +70,7 @@ export const SERVING_SENSITIVITY: Record<string, ServingSensitivity> = {
noteHref: "/notes/2026-09-01-claude-fable-5-1-added",
},
"claude-haiku-5.5": {
autoExact: 89.239,
autoExact: 88.82,
boardTreatment: FORCED_TOOL,
autoTreatment: "re-run with tool_choice: auto",
thinkingSuppressedOnBoard: true,
Expand Down
61 changes: 59 additions & 2 deletions app/src/model-serving-config.json
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
},
"evidence_summary": {
"registry": 29,
"run_state": 18
"run_state": 19
},
"models": {
"claude-fable-5": {
Expand Down Expand Up @@ -1493,6 +1493,63 @@
"shared_completion_budget_tokens": 16384,
"tool_choice": "forced"
},
"muse-wonder": {
"answer_contract": "json",
"evidence": {
"compared_columns": [
"scenario_id",
"variable",
"prediction",
"explanation",
"raw_response",
"provider_resolved_model",
"prompt_tokens",
"completion_tokens",
"error"
],
"fields": [
"answer_contract",
"chunk_size",
"completion_budget_ceiling",
"fingerprint_version",
"initial_completion_budget_tokens",
"max_repair_rounds",
"model_id",
"prompt_contract_version",
"request_timeout_seconds",
"thinking",
"tool_choice_mode"
],
"frozen_rows_sha256": "ed4cbd03e82d4874de4243dc454922f4851ff13b53d59213cc34e9ad0870485a",
"kind": "run_state",
"rows_match": true,
"run": "musewonder-runs/musewonder",
"source_predictions_sha256": "2e0eddd68fdd39201f564ed07d485329e13a80450881a3bad63d24f726380d6a",
"source_rows_sha256": "ed4cbd03e82d4874de4243dc454922f4851ff13b53d59213cc34e9ad0870485a",
"treatment_fingerprint": {
"answer_contract": "json",
"chunk_size": null,
"completion_budget_ceiling": 128000,
"fingerprint_version": 3,
"initial_completion_budget_tokens": 16384,
"max_repair_rounds": 2,
"model_id": "openrouter/meta/muse-wonder",
"prompt_contract_version": "2026-08-09-v2-scoring-contract",
"request_timeout_seconds": 300,
"thinking": {
"mode": "provider_default"
},
"tool_choice_mode": null
}
},
"provider_id": "openrouter/meta/muse-wonder",
"reasoning_setup": "provider default; 16,384-token shared budget",
"registry_derived": [],
"request_shape": "whole scenario",
"request_timeout_seconds": 300,
"shared_completion_budget_tokens": 16384,
"tool_choice": null
},
"ox-alpha": {
"answer_contract": "tool",
"evidence": {
Expand Down Expand Up @@ -1585,7 +1642,7 @@
"tool_choice": null
}
},
"registry_commit": "0562d77a9d263c14bb9c9fa93ba3c765fcff95a9",
"registry_commit": "3ef45909db7367abdffd08379babde5bcf5c549e",
"sources": [
"policybench.config.MODELS",
"policybench.model_cards",
Expand Down
6 changes: 6 additions & 0 deletions app/src/modelMeta.ts
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ export const MODEL_ORDER = [
"inkling",
"qwen3.8-max",
"qwen-3.7-max",
"muse-wonder",
] as const;

export const MODEL_LABELS: Record<string, string> = {
Expand Down Expand Up @@ -103,13 +104,15 @@ export const MODEL_LABELS: Record<string, string> = {
"qwen-3.7-max": "Qwen 3.7 Max",
"qwen3.8-max": "Qwen 3.8 Max",
inkling: "Inkling",
"muse-wonder": "Muse Wonder",
};

export type ProviderKey =
| "alibaba"
| "anthropic"
| "deepseek"
| "google"
| "meta"
| "minimax"
| "moonshot"
| "openai"
Expand Down Expand Up @@ -177,6 +180,7 @@ export const PROVIDER_LABELS: Record<ProviderKey, string> = {
anthropic: "Anthropic",
deepseek: "DeepSeek",
google: "Google",
meta: "Meta",
minimax: "MiniMax",
moonshot: "Moonshot AI",
openai: "OpenAI",
Expand All @@ -196,6 +200,7 @@ export function getProviderForModel(model: string): ProviderKey | null {
if (model.startsWith("grok-")) return "xai";
if (model.startsWith("kimi-")) return "moonshot";
if (model.startsWith("minimax-")) return "minimax";
if (model.startsWith("muse-")) return "meta";
if (model.startsWith("qwen")) return "alibaba";
if (model.startsWith("inkling")) return "thinkingmachines";
return null;
Expand Down Expand Up @@ -227,6 +232,7 @@ const FRONTIER_MODEL_GROUPS = [
["minimax-m3"],
["inkling"],
["qwen3.8-max", "qwen-3.7-max"],
["muse-wonder"],
] as const;

export const FRONTIER_MODELS: readonly string[] = FRONTIER_MODEL_GROUPS.map(
Expand Down
62 changes: 62 additions & 0 deletions app/src/notes/2026-10-13-muse-wonder-joins-the-board.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
{
"slug": "2026-10-13-muse-wonder-joins-the-board",
"date": "2026-10-13",
"title": "Meta's Muse Wonder joins the board as the references move to policyengine-us 2.41.0",
"paragraphs": [
"Meta's Muse Wonder joined the board on {releaseDate}, the day after Meta released it, bringing the board to {nModels} models. It is the first Meta model on the board. It gets {museExact}% of answers within $1, weighted by household impact, #{museRank} of {nModels}. On the bounded score, which gives partial credit for amounts close to the reference, it ranks #{museBoundedRank}, behind only GPT-6 Sol and Claude Opus 5.5. GPT-6 Sol still leads on answers within $1 at {solExact}%, ahead of Claude Opus 5.5 ({opusExact}%) and GPT-5.6 Sol ({sol56Exact}%).",
"Meta released Muse Wonder Full the same day, but only to Muse Code subscribers, with no API. PolicyBench runs every model through its provider's API under one protocol, so Muse Wonder Full is not on the board. PolicyBench reaches Muse Wonder through OpenRouter, whose one endpoint for it is Meta's own API. That API accepts only an automatic tool choice and rejects a forced tool call, so the row answers as a JSON object, as Claude Opus 5.5's and Claude Sonnet 5.5's do, and reasons at Meta's default setting.",
"The release also moves PolicyBench's references from policyengine-us {previousEngine} to {engineVersion}, the first release with every fix the previous release was waiting on. Of the outputs the previous release excluded, {restored:words} return to scoring at the new version's values, each within $1 of the corrected value PolicyBench's audits computed for it. Between them they take the upstream fixes for {restoredFixes}. The new version also fixes the defect behind a SNAP output, the heat-and-eat utility allowance, but that output stays excluded because the stated facts do not decide it. No reference PolicyBench already scored changes, and nothing is newly excluded. Every model is now scored on {scoredOutputs} of its {totalOutputs} requested outputs; PolicyBench excludes the other {excluded}.",
"Against the previous release, the exact rate of every one of the {incumbents} earlier models falls, by {fallMin} to {fallMax} points. The cause is the {restored:words} restored outputs, which few models get right: half of them are answered within $1 by at most {restoredHalfAtMost:words} of the {nModels} models, and {restoredUnmatched:words} by none. Muse Wonder answers {museRestored:words} of them within $1. Without them, every earlier model's rate would be the previous release's."
],
"facts": {
"releaseDate": "2026-10-13",
"nModels": 48,
"museExact": 90.1,
"museRank": 10,
"museBoundedRank": 3,
"solExact": 95.2,
"opusExact": 94.1,
"sol56Exact": 93.1,
"previousEngine": "2.38.6",
"engineVersion": "2.41.0",
"restored": 9,
"restoredFixes": "Idaho's subtraction for health insurance premiums (policyengine-us #10023; two outputs), Wisconsin's retirement income exclusion before refundable credits and Wisconsin's capital gain subtraction for distributions (policyengine-us #10026; two outputs), the IRA deduction's active-participant phase-out, California's itemized deduction conformity and California's educator expense addback (policyengine-us #10032, #10034, #10052) and elective deferrals in the earned income behind the EITC and the refundable child tax credit (policyengine-us #10030; four outputs)",
"scoredOutputs": 1935,
"totalOutputs": 1984,
"excluded": 49,
"incumbents": 47,
"fallMin": 0.18,
"fallMax": 0.42,
"restoredHalfAtMost": 1,
"restoredUnmatched": 4,
"museRestored": 1
},
"data": [
{
"label": "Dashboard data release",
"href": "https://github.com/PolicyEngine/policybench/releases/tag/dashboard-data-20261013"
},
{
"label": "Engine upgrade record",
"href": "https://github.com/PolicyEngine/policybench/blob/main/reference_audit/2026-10-12-engine-upgrade"
},
{
"label": "Reference sidecar with every change",
"href": "https://github.com/PolicyEngine/policybench/blob/main/paper/snapshot/20260501/runs/us_full_run_20260612_policyengine_4_16_1_populace/reference_outputs.csv.meta.json"
},
{
"label": "Exclusion record",
"href": "https://github.com/PolicyEngine/policybench/blob/main/paper/snapshot/20260501/runs/us_full_run_20260612_policyengine_4_16_1_populace/reference_exclusions.json"
},
{
"label": "Muse Wonder model page",
"href": "/model/muse-wonder"
},
{
"label": "Previous release note (October 10)",
"href": "/notes/2026-10-10-claude-haiku-5-5-joins-the-board"
}
],
"boardSnapshot": "2026-10-13",
"release": "dashboard-data-20261013"
}
2 changes: 2 additions & 0 deletions app/src/notes/index.ts
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import museWonderJoinsTheBoard from "./2026-10-13-muse-wonder-joins-the-board.json";
import claudeHaiku55JoinsTheBoard from "./2026-10-10-claude-haiku-5-5-joins-the-board.json";
import claudeFable51Added from "./2026-09-01-claude-fable-5-1-added.json";
import claudeSonnet55DebutsFifth from "./2026-09-29-claude-sonnet-5-5-debuts-fifth.json";
Expand Down Expand Up @@ -30,6 +31,7 @@ export type PolicyBenchNote = {
};

export const notes = [
museWonderJoinsTheBoard,
claudeHaiku55JoinsTheBoard,
stopsScoringEightTaxOutputs,
fiveSnapHouseholdsBbce,
Expand Down
8 changes: 4 additions & 4 deletions app/src/paperSnapshot.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"pdfVersion": "1607c71abaf3",
"responseWindow": "June 12 and October 8, 2026",
"snapshotDate": "2026-10-10",
"webVersion": "31cd4141373a"
"pdfVersion": "88e3c81e3e13",
"responseWindow": "June 12 and October 13, 2026",
"snapshotDate": "2026-10-13",
"webVersion": "af35d8e23eaf"
}
18 changes: 9 additions & 9 deletions app/tests/auditUniverse.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -18,17 +18,17 @@ describe("audit universe", () => {
expect(AUDIT_SELECTION_RULE).toBe(
"rows whose legacy threshold score is below 1",
);
// Release 20261010: the counts tests/test_disclosures.py recomputes
// from the frozen annotations and payload (47 models; the ten outputs
// ruled on 2026-10-06 and the two Indiana county outputs leave scoring,
// the outputs the engine upgrade regenerates return to it).
// Release 20261013: the counts tests/test_disclosures.py recomputes
// from the frozen annotations and payload (48 models; Muse Wonder's rows
// join, and the nine outputs the 2.41.0 upgrade regenerates return to
// scoring).
expect(summarizeAuditUniverse(us)).toEqual({
annotatedRowCount: 8_001,
legacyThresholdRowCount: 8_001,
exactMissCount: 7_997,
annotatedExactMissCount: 7_997,
annotatedRowCount: 8_503,
legacyThresholdRowCount: 8_503,
exactMissCount: 8_499,
annotatedExactMissCount: 8_499,
annotatedExactHitCount: 4,
unannotatedBelowFullBoundedScoreCount: 2_138,
unannotatedBelowFullBoundedScoreCount: 2_218,
});
});
});
5 changes: 3 additions & 2 deletions app/tests/expandPage.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -71,18 +71,19 @@ test("expand page derives its exact-score headline from the live summary", async
).text();

expect(leader.model).toBe("gpt-6-sol");
expect(leader.exact).toBeCloseTo(95.583, 3);
expect(leader.exact).toBeCloseTo(95.225, 3);
expect(pageSource).toContain("const leader = headlineExactLeader(dashboard);");
expect(pageSource).toContain(
"const leaderLabel = MODEL_LABELS[leader.model] ?? leader.model;",
);
expect(pageSource).toContain("The best model, {leaderLabel}, computes");
expect(pageSource).toContain("{leader.exact.toFixed(1)}% of");
expect(pageSource).not.toContain("95.2%");
expect(pageSource).not.toContain("95.6%");
expect(pageSource).not.toContain("96.5%");
expect(pageSource).not.toContain("95.0%");
expect(expectedCopy).toBe(
"GPT-6 Sol computes 95.6% of requested outputs exactly",
"GPT-6 Sol computes 95.2% of requested outputs exactly",
);
});

Expand Down
4 changes: 2 additions & 2 deletions app/tests/heatmapMetric.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -39,8 +39,8 @@ describe("program heatmap metric", () => {
expect(Math.round(heatmapValue(federal, "exact"))).toBe(80);
expect(Math.round(heatmapValue(federal, "score"))).toBe(96);
const stateTax = fable51("state_income_tax_before_refundable_credits");
expect(Math.round(heatmapValue(stateTax, "exact"))).toBe(74);
expect(Math.round(heatmapValue(stateTax, "score"))).toBe(94);
expect(Math.round(heatmapValue(stateTax, "exact"))).toBe(69);
expect(Math.round(heatmapValue(stateTax, "score"))).toBe(92);
});

test("the rendered heatmap shows exact-match cells and says so", () => {
Expand Down
Loading
Loading