Skip to content

Add an internal tool-use pilot for Gemini 4 Argon (not a board condition) - #229

Merged
MaxGhenis merged 6 commits into
mainfrom
argon-tool-pilot
Oct 14, 2026
Merged

Add an internal tool-use pilot for Gemini 4 Argon (not a board condition)#229
MaxGhenis merged 6 commits into
mainfrom
argon-tool-pilot

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Oct 14, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

An internal pilot Max asked for on 2026-10-14. It is not a board condition and nothing from it is published. Its results stay internal; this PR carries only the code.

What it does

scripts/tool_pilot_argon.py compares two arms on the board's 100 US households:

  1. No tools. Argon's frozen board predictions from release dashboard-data-20261014 (Add Gemini 4 Argon and Muse Spark 1.3 (release, 50 models) [WIP] #221), read from the snapshot and never re-run.
  2. PolicyEngine tool. The model sees the board prompt plus a short tool notice, the board's submit_outputs answer tool, and a calculate_household(situation, variables, period) function.
    • The function runs the reference engine on JSON the model writes. The reference engine is policyengine-us 2.41.0 plus reference_audit/2026-09-28/fixes/latest_final.py, the convention reform every 2.41.0 reference is computed with.
    • The model gets up to 8 tool turns, at most 4 calculations a turn, and a 60 s timeout per calculation. After that the answer tool is forced, as on the board.
    • The tool only parses JSON. It checks every name against the engine's variable registry and runs in a separate worker process. No model-written code is executed.

The board's single no_tools condition is untouched. The script writes only under --out-dir, labels its rows gemini-4-argon+pe-tool, and never touches the run store, snapshot, cache or dashboard.

The subcommands are oracle, run, score and diagnose.

Invariants (tested)

  • Oracle. The reference situations reproduce all 1,935 scored references through the tool's validator, execute path and output mapping. The plain 2.41.0 engine reproduces 1,913; the 22 that differ are SNAP and state-tax conventions.
  • Differential.
    • Rescoring arm 1 through analyze_no_tools gives the published weighted_exact for Argon exactly.
    • The household score table averages to it.
  • Validator (property-based). Any input gets exactly one of a request or an error. An accepted request names only registry variables, each on its own entity.
  • Diagnosis. Every arm-2 miss lands in exactly one category.
  • Cost. A household's cost is the sum of its turns' costs.
  • Bootstrap. Each interval contains its point estimate, and identical arms give a zero interval.
  • No selection on outcome.
    • Scoring counts every household, and an unparsed output is a miss.
    • A repair never overwrites a valid first answer (the board's _merge_repair_cells).
    • A resume reruns only infrastructure failures.

There are 69 tests. 68 run without network or API access, and one slow test builds the real engine.

🤖 Generated with Claude Code

@MaxGhenis @claude
Add an internal tool-use pilot for Gemini 4 Argon (not a board condit…
854eb4e 
…ion)

scripts/tool_pilot_argon.py gives Argon a calculate_household function
that runs the reference engine (policyengine-us 2.41.0 with the
reference builder's latest_final convention reform) on a situation the
model writes as JSON, and compares it with Argon's frozen no-tool board
row on the same 100 households. It writes only under --out-dir and
labels its rows gemini-4-argon+pe-tool; the board's single no_tools
condition is unchanged.

Subcommands: oracle (reference situations through the tool engine),
run, score (the board's analyze_no_tools on the scored outputs) and
diagnose (one category per arm-2 miss).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vercel

vercel Bot commented Oct 14, 2026 •
edited
Loading

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
policybench-site Ready Ready Preview Oct 14, 2026 7:17pm UTC

Request Review

@MaxGhenis @claude
Attribute each tool-arm miss to its inputs and bootstrap the househol…
2f8023b 
…d difference

diagnose now applies each differing input as the reference sets it and
recomputes the output, so a mapping miss names the input that causes it;
it reads payroll_tax from employee_payroll_tax too (spm_unit_payroll_tax
sums it over the SPM unit's tax units). score adds household weighted
exact scores, checked against analyze_no_tools before use, a paired
household bootstrap, and hit/miss transitions by program.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis @claude
Summarize tool use and compare each household's main situation with t…
435f581 
…he reference

score adds a tool-use summary (first-call success, validation problems
by kind, the names the model guessed wrong). diagnose writes
situation_fidelity.csv from each household's main situation, the
successful call that computed the most of its outputs, since a model
often ends with a narrow probe on a skeleton situation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis @claude
Merge repairs as the board does and score every household
000d905 
Fixes from review of #229:
- A repair now fills only cells still missing, with the board's
  _merge_repair_cells; a valid first value always stands.
- score counts every household by default (--complete-only to opt
  in), and a resume reruns only infrastructure failures, keeping each
  replaced attempt's cost.
- The validator's property test draws structured calls that reach
  acceptance (about 1 in 10), and checks it accepts at least 30 of 600.

Also: every pinned fix module is verified before the reform loads;
deeply nested JSON is refused, not raised; a failed household writes a
harness_error record instead of ending the run; the pool restarts or
reports lost workers; diagnose pins the engine; truncation retries are
counted apart from model turns. New tests cover the merge, truncation
retry, same-turn submit, turn limit, the litellm thought-signature
round trip, pool timeout and crash restart, scoring without selection,
and resume. Rescoring the pilot's run reproduces every reported number.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis @claude
Stop on a lost tool pool, keep partial spend, and fuzz the validator'…
dc38fd6 
…s groups

Fixes from round 2 of the #229 review:
- A replacement worker that dies before ready is killed and counted as
  lost (any failure, not only RuntimeError), and a pool with no worker
  returns pool_unavailable, which ends the household as harness_error,
  so an infrastructure failure is never scored as the model's answer.
  The run writes the pool's counters to run-summary-*.json.
- The structured property test draws each group's unit over a unique,
  non-empty subset of the drawn people, weighted to the group's own
  inputs: of 1,000 calls 273 are accepted and 23 carry a group input
  (floors 50 and 10). It found the validator accepted an input with an
  empty period dict; that is now refused (no call in the pilot's run
  sent one).

Also: an exception in the turn loop keeps the turns and spend so far; an
explanation-only repair names the kept value; score warns when
households still need a rerun; the nested-JSON test reaches the
RecursionError path. Rescoring the pilot's run reproduces score.json.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis @claude
Address the round-3 review nits on #229
bed93f4 
- Kept values in a repair prompt are plain decimals (plain_number),
  which the board's value = X trailer pattern reads back; :g gave
  12345.7 for 12345.67 and 1.23457e+06 for 1234567.
- A household stops at the first pool_unavailable result, and no new
  household starts once every worker is lost.
- The run summary lists each rerunnable household's error.
- The validator fuzz draws typed group inputs and some invalid
  memberships (a ghost, a person twice or in two units). Over four
  seeds it accepts 322-346 of 1,000 calls, 58-71 with units and 14-38
  with group inputs, and rejects 510-557 on membership; the floors
  (100, 20, 5) sit under the worst seed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis
MaxGhenis marked this pull request as ready for review October 14, 2026 19:34
@MaxGhenis

MaxGhenis commented Oct 14, 2026

Copy link
Copy Markdown
Contributor Author

Merging at bed93f4.

  • CI: gh pr checks exits 0 on this head (lint, test, app, Vercel).
  • Mergeability: MERGEABLE and CLEAN, with no change-requesting review.
  • Review: three code-review rounds, independent Opus 5.5 because the Codex lanes were closed. Round 1 asked for changes on Sol's queue and was served by Opus. Round 2 asked for changes. Round 3 approved with nits. A confirmation pass approved bed93f4, with every nit fixed.
  • Tests: 69 (68 fast, 1 slow), all passing locally.
  • Scope: code only. The pilot's results stay internal, and the board's no_tools condition is unchanged.

@MaxGhenis
MaxGhenis merged commit 48f406a into main Oct 14, 2026
5 checks passed
@MaxGhenis
MaxGhenis deleted the argon-tool-pilot branch October 14, 2026 19:35
@MaxGhenis MaxGhenis mentioned this pull request Oct 15, 2026
Merged

This branch was successfully deployed

1 active deployment
Preview — bed93f45 Deployed Oct 14, 2026 by vercel[bot]
View deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@MaxGhenis