Repository navigation
Add Gemini 4 Argon and Muse Spark 1.3 (release, 50 models) [WIP] - #221
Merged
Merged
Conversation
Binds each of the 1,342 verdicts in release 20260929's audit tree and the previous release's tree to the transcript that produced it, and lists every tool call. One published judge used tools (086 federal refundable credits), one saw its case's exclusion history through git status (045 snap), and three Codex verdicts have no transcript that holds their text. An independent re-implementation agrees on every row. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Re-judges exactly the five carried-over verdicts the judge-isolation audit found, with the isolated runner, and refuses any other change: carried-over verdicts, the adjudication record, references, predictions and serving evidence stay release 20260930's, every modelStats entry stays byte-identical, and the payload may differ only in the five cases' annotation fields. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rejected re-judges - triage splices the five cases' records into release 20260930's committed CSV bytes instead of regenerating the files, so wording amendments and blank fields keep their bytes; records compare as text, field by field, and only the columns a verdict decides may change - export and the freeze re-check every annotation file against the base and against annotation-changes.json - the freeze checks the working tree is release 20260930's before writing, restores the wording-amendment record the snapshot freezer drops, and allows manifest changes only in this release's scope - each re-judge's transcript must belong to its session, open with its prompt, end with its verdict and come from Opus 5.5; the sidecar must name Opus 5.5 as requested and record a judging time - a re-judge that fails is set aside and judged again, up to three rounds; everything set aside is pinned - prepare validates the stage path and refuses base-stage audit files the receipt does not bind Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… accepts Google opened Gemini 4 Argon to the public on 2026-10-13 (the Gemini API changelog lists gemini-4-argon as generally available that day; the 2026-09-30 launch was limited to Fairwind Program cyber partners). It runs on the first-party Gemini API as gemini/gemini-4-argon, like the other Gemini rows. Price: $2 / $10 per 1M input/output tokens and $0.10 cached input, the rates Google's pricing page lists through 2026-11-30 (they double on 2026-12-01; page last updated and read 2026-10-13). Card: the Gemini API accepts a forced function call on this model, and the gauntlet, run with no card, passed on the tool contract (3/3 in 5s with 442 completion tokens, 16/16 whole-scenario in 24s with 2,619), so the row takes the board's forced answer tool, as Gemini 3.6, 3.7 and 3.8 Flash do. Reasoning: the harness sends no thinkingConfig, and the model decides call by call whether to think. On the board's own request for 13 frozen households (29 calls a shape) it returned reasoning tokens on 20 forced calls, 25 with tool_choice auto and 22 on the JSON contract; all 87 parsed every output. Forced against auto is within sampling error (Fisher exact p = 0.21), so the 100-household runs decide whether a tool_choice auto sensitivity run is published (the rule is fixed in the onboarding record). A two-scenario smoke on the run path completed 2/2 with no missing prediction or explanation for $0.05. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One script, modelled on release_20261006.py: prepare, judge, collect, export and freeze. The bundle comes from git at the base commit and the audit tree from release 20261013's archived seed, checked against the committed seed digest and judge provenance. The five cases render on judge template v2 through prepare_audit; the transcript checks are finish_musewonder's, plus a session-id binding. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 100-household runs decide the question the card left open. The forced-tool board run returned reasoning tokens on 80 households and a tool_choice auto run on 82 (Fisher exact p = 0.86; 70 reasoned in both, 10 only when forced, 12 only under auto, 8 in neither). Under the rule written before the runs, the row gets no tool_choice auto sensitivity run; the auto run stays in the onboarding record as evidence. The notes also record the board run: 100/100 households in one request each, 1,984/1,984 answers parsed, no budget escalation, $2.77. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…efuses a forced tool
Max ruled on 2026-10-13 (d1441) that Muse Spark 1.3 joins the board in the
Gemini 4 Argon release. Meta released it on 2026-09-02 ("Introducing Muse
Spark 1.3": available that day in Muse Code and the Meta Model API). It runs
through OpenRouter, whose one endpoint for meta/muse-spark-1.3 is Meta's own
(meta/muse-spark-1.3-20260902), as Muse Wonder does.
Price: $1.25 / $4.25 per 1M input/output tokens and $0.15 cached input,
Meta's standard-mode rate at every prompt length (Meta's rate card and the
OpenRouter endpoint listing, read 2026-10-13). The contributor-mode listing
is a different endpoint and is not the board's.
Card: Meta's API accepts tool_choice "auto" only (a named-function or
"required" choice returned 400 on each of three tries), so the row runs the
JSON contract whole-scenario and reasons at the provider default (4,311 to
8,174 reasoning tokens on a one-question probe with no tools, 4,844 to
6,022 with tool_choice auto). With no forced tool there is no
reasoning-suppression sensitivity run.
Gauntlet: with no card the 3-variable JSON probe stopped at the 384-token
limit; with the reasoning budget, 3/3 and 16/16 parsed (8,163 and 4,258
completion tokens, 106s and 58s), about $0.020 per scenario. A
four-household smoke on the run path, with the board's two largest
households, parsed 96/96 in one request each for $0.11 (Meta's reported
cost equals the reconstruction from list prices).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…park 1.3 Work in progress; not ready for review. - scripts/finish_argon.py, freeze_argon.py and date_argon_judge_verdicts.py, adapted from the Muse Wonder release's. The release installs no engine upgrade (policyengine-us 2.41.3 gives 2.41.0's value for all 1,984 outputs; verification/publication_check_2_41_3.csv), so the driver has no install-references step and refuses a stage that records an installed build. - Base pins point at release 20261013 for development, marked PENDING(#218): the release's base is dashboard-data-20261013b once PR #218 merges. - tests/test_finish_argon.py passes except the two tests that wait on the committed input pins. tests/test_freeze_argon.py still stages one model and fails; converting it to two additions is next. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…est returns (card review P2, P3) Google's Gemini 4 Argon guide: "One non-streaming request returns up to 32,768 tokens" (read 2026-10-13); the Models API's outputTokenLimit agrees. The model page's 65,536 is the streaming and SDK figure, reached by continuing across requests, and litellm sends one non-streaming REST request. So provider_max_completion_tokens is 32,768, not 65,536. The output-limit probe's 200s at up to 131,072 showed only that the parameter is accepted. The first board run recorded the 65,536 ceiling in its treatment fingerprint, which the freeze checks against the registry, so the board row is a second run on this card. The first run never escalated (largest household 6,374 completion tokens), and it stays in the record, with the tool_choice auto run, as the reasoning comparison's evidence. The reasoning sentence now says what the comparison shows: a forced tool does not switch reasoning off, and no statistically significant reduction in how often the model reasons was detected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Payload c06177e65913fa7886c239f776483982a140cc126e18e734b443dc890b74ea32 (130,982,273 bytes). Every payload section but scenarioPredictions is release 20261013's; 45 annotation fields change in five cases (21 model annotations, five case notes, three llm_error subtypes), with eleven listed wording amendments applied. The judge tally moves by four: Claude Opus 5 93 to 92, GPT-5.6 Sol 269 to 266, Claude Opus 5.5 315 to 319. The paper is re-rendered for those counts and re-pinned. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The second board run, on the 32,768-token ceiling (e4b4d05): all 100 households in one request each, 1,984/1,984 answers parsed, no budget escalation, $2.79, reasoning tokens on 85 households. None of its responses replays the first run's (no shared provider response id). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…on (card review P1, P3) P1: the card and commit 8c400c9 called the smoke's scenarios 099 and 119 (32 and 28 outputs) the board's two largest households. They are not: scenario_008 has 56 outputs, three households have 38 and two have 34. The card now gives the smoke's sizes (16, 20, 28 and 32 outputs) and rests the timeout and starting budget on the board run, which answered every household in one request, scenario_008 included: 1,984/1,984 parsed, no budget escalation, at most 96s and 9,871 completion tokens per household. The squash message will not repeat 8c400c9's claim. The card also says that the board run's four smoke households replay the smoke's responses from the harness cache (identical requests), as the two smoke households of the Muse Wonder, Haiku 5.5 and Opus 5.5 rows do. P3: the notes now say "Meta rejects forced tool choices", the wording the disclosure test classifies as a provider rejection. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ease-gemini-4-argon
…dd the app labels - docs/argon/input_pins.json: the sha256 of each finished run's predictions.csv and run_state.json. Gemini 4 Argon's is the second board run, on the 32,768-token ceiling (card e4b4d05); Muse Spark 1.3's is its only board run (resumed at 95/100 after an app restart, same treatment). - tests/test_freeze_argon.py: the synthetic stage stages both additions' runs, fingerprints, board rows and serving evidence; counts follow release 20261013b (48 incumbents, 49 exclusions, 1,935 scored, 50 models). - app/src/modelMeta.ts: labels, order and provider for both; Gemini 4 Argon becomes Google's frontier pick (Gemini 3.1 Pro Preview stays the fallback); Muse Wonder stays Meta's. - policybench/paper_results.py: display names for both. Still WIP: base pins wait on PR #218; staging, judging, export and freeze not started. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…release-gemini-4-argon
…ovisionally The base is release dashboard-data-20261013b (PR #218: release 20261013 with five audit verdicts re-judged; its references, exclusions, adjudications, predictions and serving config are 20261013's byte for byte). This merges #218's frozen head (6a334d0) and pins it: - BASE_TAG dashboard-data-20261013b; BASE_SHA256 c06177e6…, the payload at that head; BASE_COMMIT 6a334d0, to become #218's squash merge. - The seed digest is docs/release_20261013b/seed_digest.csv, copied (0f599257…): release 20261013's 677 judged cases with the five re-judged written over them. The stage prepared on these pins: 482 seed prompts unchanged, 195 changed and 1 new, so 196 cases go to the Opus 5.5 judge. None of 20261013b's five re-judged cases is among them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… the 2.41.3 check - docs/benchmark_card.md: Muse Spark 1.3 joins the JSON rows whose provider rejects a forced tool call; the publication check is 2.41.3, PyPI's newest at 17:22 UTC on 2026-10-13, equal to 2.41.0 on all 1,984 outputs. - reference_audit/2026-10-12-engine-upgrade/README.md: the same check, for the release that keeps these references. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ule convention Max ruled both on 2026-10-13, with no score change: - d1263: federal income tax references keep the section 1 rate schedule. The benchmark card's reference section and the paper's reference- credibility section now say so: the references apply the schedule to the cent at every income, not the Tax Table the Form 1040 instructions send filers under $100,000 to, so a Tax Table answer can miss the $1 tolerance (reference_audit/2026-10-10-tax-table). The v2 prompt line is #165. - d1252: VA scenario_039 federal and OH scenario_025 state income tax stay scored. The standing held-reference records now carry the ruling (their archived 2026-10-10 copies still say "not ruled", as written); a test checks the standing list. The label fixes go in the v2 prompt (#165). The spec, design note and driver comment record both rulings. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…view P3) The export docstring and scope-check comment describe two additions on a 50-model board with the base's references and exclusions unchanged; the seed re-renders from release 20261013b's snapshot; the freeze tests' counts are 20261013b's (61 decisions, 49 exclusions); the synthetic v1 seed in the driver tests is labelled as a mock. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
All 196 new verdicts ran on the pb-judge setup-token login (two at a time, then one at a time), on judge template v2 at effort xhigh. Calls whose transcript failed the runner's checks (network retry events) or whose verdict missed a model were judged again. The two Subfleet lanes earlier releases borrowed stay in the group table but were not used. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Triage on the release stage stopped on two cases the new models re-opened: - scenario_009 federal income tax: the verdict labels eight models' rows reference_later_law, a developer-only class, for applying the pre-TCJA rules. Decided llm_error with no reference verdict: P.L. 119-21 predates the 2026-07-03 reference freeze, as the 2026-09-29 wave decided scenario_007. - scenario_064 Wisconsin income tax: the verdict flags the reference on a $500 capital loss limit. Affirmed: Wis. Stat. 71.05(10)(c)2. allows $3,000 for taxable years beginning after 2022 (2025 Schedule WD line 28), which policyengine-us applies. The driver read triage decisions only with an engine upgrade installed. It now reads them from the spec without one, dated by triage_decided_on, and names the wave in the record's date conventions. A decision affirms a reference exactly when the verdict flags it, otherwise needs a non-final class left to resolve, and never excludes. The freeze holds the record to these entries; the dating script adds the wave to the evidence uncommitted and stays idempotent. Also commits the judge provenance record (196 verdicts) and points the stage tests at stage2. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Freeze the stage: Gemini 4 Argon 88.7% within $1 (#12), Muse Spark 1.3 87.2% (#18), on release 20261013b's references and exclusions (49 excluded, 1,935 scored per model). Every earlier model's exact rate and bounded score are release 20261013's. - Date the verdicts: docs/argon/judge_verdicts_20261014.json; the wave evidence gains the uncommitted 2026-10-14 triage wave. - Release note 2026-10-14-gemini-4-argon-joins-the-board, with tests that recompute its facts and check its claims; the Muse Wonder note's tests now read release 20261013 from its commit (bc30579). - Adjudication tests: 63 decisions, the triage wave, and a section that holds this release's record to release 20261013b's plus restatements. - Sensitivity rescored on the 50-model board (Sonnet 5 would rank #23). - Benchmark card: 2026-10-14 snapshot; disclose harness cache replay. - Paper re-rendered; app audit-universe and notes tests updated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PyPI's newest policyengine-us at 04:03 UTC on 2026-10-14 was 2.42.4 (2.42.0 to 2.42.3 never reached PyPI). A sweep of every output on it gives 2.41.0's value for all 1,984 (computed.csv byte-identical), so the references stand. Record the check (verification/publication_check_2_42_4.csv, sweep_timing.json), and name it in the README, benchmark card, design note, spec, driver docstring and release note; the note test reads whichever check sweep_timing.json records. Paper re-rendered. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…unts The manifest's judge_flagged_by_reference_verdict counted only the current verdict's flag. This release's re-judge of scenario_005's federal income tax (an engine-defect exclusion) no longer flags it, though the verdict it replaced did (kept under judge_previous), so the paper would have said three of the five engine-defect exclusions had never been flagged. judge_ever_flagged counts a flag on the current verdict, any judge_previous item, or an earlier run a judge_reference_suspect_source names; the paper again says 24 flagged references, 3 confirmed defects, 2 never flagged. Release 20261013b's record gives the same tally either way. Property tests cover the helper; a differential test checks the manifest block against the record. Pin the counts a 50-model board moves, each traced to the two new rows on unchanged references: audit universe 8,765 / 8,761 / 2,329 (Argon 118 and Spark 144 misses), 1,915 rows on excluded outputs, 1,540 prompt-ambiguity rows, 96,750 scored answers, judge tally 92 / 322 / 264 of 678, serving evidence for 21 rows, recorded costs $463.481 of $527.560. The benchmark card and paper guide carry the same counts and 63 adjudications. Re-froze and re-rendered the paper. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ency example (review r1) - P1: verify_adjudication_changes, which both the driver's triage step and the freeze call, now refuses any new decision that is neither a ruled nor a triage decision from the spec, so a hand-written decision can no longer bypass the triage gates (an affirmation of an unflagged reference, say). Tests cover the freeze (flagged and unflagged) and the driver; a new exclusion is refused at the same gate. - P1: the benchmark card's cache-replay paragraph measured households by their slowest row. A row records its call's latency divided among its outputs, so a household's latency is the sum; Claude Haiku 5.5's scenario_001 (0.1175 s) is dropped and every example now sums to under 0.05 s. - P3: the publication check refuses a moved scored output; the design note and driver comment say so. - P3: the paper's judge chronology lists the October 14 additions. Re-rendered and re-froze to pin the render. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # annotations/us_full_run_20260612_policyengine_4_16_1_populace/us_audit_row_annotations.csv # annotations/us_full_run_20260612_policyengine_4_16_1_populace/us_case_notes.csv # app/public/paper/policybench.pdf # app/public/paper/web/index.html # app/src/data.artifact.json # app/src/paperSnapshot.json # app/tests/notes.test.ts # docs/benchmark_card.md # paper/index.qmd # paper/snapshot/20260501/manifest.json # paper/snapshot/20260501/runs/us_full_run_20260612_policyengine_4_16_1_populace/data.json.gz # tests/test_notes.py # tests/test_paper_results.py
Release 20261013b merged as 2cb77a1 (PR #218, 2026-10-14 06:10 UTC). Its tree is #218's frozen head 6c44858 merged with the earlier main plus one test-only commit (app/tests/notes.test.ts); the annotations, snapshot, release-20261013b docs and reference audit are byte-identical to 6c44858's, and the payload stays 0a4b754a... (130,982,477 bytes). So the stage, prepared on 6c44858, keeps every input the driver reads from the base. BASE_COMMIT and the test pins name 2cb77a1; the verdict evidence names it; the freeze re-ran at the new pin and wrote the same files. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…view r2) - P3: tests/test_disclosures.py now recomputes each household latency the benchmark card names (the sum of its rows) from the frozen predictions: Muse Spark 1.3's four replayed smoke households at 0.03 s or less, the other examples under 0.05 s, and round 1's wrong example (Claude Haiku 5.5 scenario_001) at 0.05 s or more. - P3: the freeze's record gate can no longer report extra decisions (the driver refuses any the spec does not list); its docstring says so, the dry-run line names the triage decisions and no other, and the freeze refuses a nonzero count as a backstop. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis
marked this pull request as ready for review
October 14, 2026 08:44
Contributor
Author
|
Merge record: squash-merged as bf845cb at the reviewed head fc2de55, whose tree the squash keeps.
🤖 Generated with Claude Code |
This was referenced Oct 14, 2026
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft: frozen and pushed at c479e59, re-pinned to #218's squash merge (2cb77a1). Review round 1 (GPT-6.1 Sol) asked for changes, all fixed in eb8af75; round 2 is queued. Not published.
The data release
dashboard-data-20261014adds Google's Gemini 4 Argon and Meta's Muse Spark 1.3 (Max, 2026-10-13, d1441), taking the board to 50 models. It builds ondashboard-data-20261013b(#218, merged as 2cb77a1), which is release 20261013 (#217, Muse Wonder) with five audit verdicts re-judged. The card PRs are #219 (Gemini 4 Argon) and #220 (Muse Spark 1.3), both merged.Results (frozen)
Every one of the 48 earlier models keeps release 20261013's exact rate and bounded score, because the references and exclusions are unchanged. Models are scored on 1,935 of 1,984 outputs; 49 are excluded.
What the release does
docs/argon/input_pins.json.reference_audit/2026-10-12-engine-upgrade/verification/publication_check_2_42_4.csv). There is no engine upgrade.docs/argon/judge_provenance.json). 51 committed decisions are restated with their new verdicts (docs/argon/judge_verdicts_20261014.json).reference_later_law(pre-TCJA rules) are decided asllm_error. P.L. 119-21 predates the 2026-07-03 freeze, as decided for scenario_007 on 2026-09-29.triage_decided_on. A decision affirms only a flagged reference, resolves only a non-final class, and never excludes. The freeze holds the record to exactly these entries.judge_flagged_by_reference_verdictnow counts a flag from any recorded verdict (judge_ever_flagged). scenario_005's engine-defect re-judge dropped the flag its previous verdict raised, and counting only current flags would have made the paper say three engine-defect exclusions had never been flagged; it says two, as before. Release 20261013b's tally is the same either way.2026-10-14-gemini-4-argon-joins-the-board(facts recomputed intests/test_notes.py). Sensitivity is rescored on the 50-model board. The benchmark card discloses cache replay. The paper is re-rendered.Invariants (tested)
test_release_20261014_note_claims).Remaining
BASE_COMMITre-pinned, verdict evidence regenerated, freeze re-run at the new pin (same files).🤖 Generated with Claude Code