Run the internal tool pilot on GPT-6 Sol and Claude Opus 5.5 (not a board condition)

[MaxGhenis]

WIP. This extends the internal tool-use pilot from #229 (scripts/tool_pilot_argon.py) to two more board models. It is not a board condition: nothing changes in the run store, the snapshot, the exports or the dashboard, and the board's single no_tools condition is untouched. Results stay internal and are not posted here.

What changes

run --model <board model> sends each model the way the board does:

• GPT-6 Sol (Responses API, forced-tool contract): the conversation is chained by previous_response_id, and every function call gets a function_call_output. Tools use the Responses API's flat shape, and the forced answer is {"type": "function", "name": "submit_outputs"}.
• Claude Opus 5.5 (JSON contract; its API returns a 400 on a forced tool, per its model card): calculate_household is offered under tool_choice: auto, the answer is the board's JSON reply, and the cap is a user message. Calls after the cap get a not_run result. Each Anthropic request marks its last message as a prompt-cache breakpoint; the stored conversation stays unmarked.
• Gemini 4 Argon keeps its Add an internal tool-use pilot for Gemini 4 Argon (not a board condition) #229 behavior, except for one change: a calculation that arrives in a forced-answer turn is refused instead of run. The property test below found that gap. It never happened in the Argon run (3 forced or repair turns, none carrying a calculation), so Argon's records and scores are unchanged.
• Labels (<model>+pe-tool), the frozen arm-1 rows and score follow the model the records name. Records written before this change carry no model name and are read as Argon's. run refuses a directory that holds another model's records.

diagnose settles unread outputs with the board's own output:

• The gap. A model can read a state's own variable (ca_income_tax_before_refundable_credits) rather than the board's (state_income_tax_before_refundable_credits), so classify_miss files the miss as wrong_variable_or_period.
• The evidence. resolve_unread_output takes the holding calls: successful calls with a result within $1 of the answer, on the entities of the output's engine variables. It computes the board's output on each of their situations and on the main call's.
• Reclassifying. Only if every one equals the answer does the miss become wrong_input_mapping, attributed on the holding call that computed the most outputs.
• Standing. Otherwise the verdict stands, with the main call's situation as evidence and every holding call's value listed; it is marked ambiguous when the values split.
• Never reclassified: binary outputs, and misses in a household whose calls all computed no outputs (so no situation is the main one). A standing verdict says why it stands.

Attribution:

• Necessary inputs. attribute_miss names them by leave-one-out when only the full set of differences fixes the output.
• Mapping kind. Each mapping miss is labelled entered_differently when the model set a cause, or put its amount under a reviewed counterpart in REENTRY_PAIRS; otherwise left_out. Other equal amounts are listed as entered_as_candidates.

misses.csv gains those columns.

Invariants (tested)

For every scripted conversation on all three paths (Hypothesis, 300 derandomized examples):

• Every function call of a response that reaches the conversation gets exactly one result in the next request, before any user text.
• Each Responses request chains to the response before it.
• A reply cut off below the budget ceiling is resent unchanged at the ceiling.
• At most cap turns run calculations, with at most 4 each, and at most MAX_TOTAL_TURNS turns reach the conversation. Requests equal turns plus truncation retries; the extra request for a retried turn is intended.
• The shared spend counter equals the sum of the household's turn costs.
• A JSON-contract request offers only calculate_household, under auto, and a complete household has every output.

Four mutants of the harness each fail that test on the matching assertion:

• a retry below the ceiling;
• user text before function results;
• no result past the per-turn limit;
• broken chaining.

There are also example tests:

• Each path: prompt equality with the board, chaining, cap forcing, truncation retry at the ceiling, JSON repair and nudge, cost at each model's list price with cached input, and the cache breakpoints.
• Diagnosis: settled verdicts (equal, different, and the main-call fallback) and leave-one-out on fakes.

pytest tests/test_tool_pilot_argon.py: 98 tests (97 fast, 1 slow that builds the engine).

Reviews

• r1 (GPT-6.1 Sol, a56b4bc): AGREE. Its coverage notes on the property test (ordering, truncation, text beside calls) are addressed in 7e6c549.
• r2 (e9f9efc): REQUEST_CHANGES. A coincidental probe on any entity could flip a verdict, and one test fabricated a diff. Fixed in fae34f5.
• r3 (fae34f5): REQUEST_CHANGES.
  • A lone same-entity probe could outvote the main situation for a summed answer.
  • Disagreement reported the probe's situation.
  • A numeric coincidence could earn a re-entry label.
  • Fixed in 297b173. A mutant without the main-call check fails the new test.
• r4 (297b173, Claude Opus 5.5 lane while the Codex lanes were busy): AGREE, with two minor notes. A tie at zero outputs could make a probe the main call, and a standing verdict didn't always say why. Both fixed in d4a8852.
• r5 (d4a8852, GPT-6.1 Sol): AGREE, with one nit: the no-holder note could appear on binary outputs. Fixed in 29cff9f.
• r6 (29cff9f, GPT-6.1 Sol): AGREE, no findings.

Plan

1. A 5-household smoke on each model, with a budget cap. Done.
2. The full 100 households for each. Done.
3. Score and diagnose. Done, internal.
4. Merge on green CI at 29cff9f.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
policybench-site	Ready	Preview	Oct 15, 2026 8:31pm UTC