Repository navigation
combined-v9: the published loop is agent-box's pi - #6
Merged
Merged
Conversation
The instrument is v8's: prompt, questions, conditions, tools, research floor, coverage rule, turn limit and debrief are unchanged. The calls go through agent-box's pi loop (redlines/pi_loop.py) instead of redlines.llm's litellm loop, as FRI's ForecastBench runs do. The full panel's same-day shadow (2026-10-14) sat inside litellm's own run-to-run spread for every model: mean |d| 0.019-0.025 against 0.022-0.041. - The condition set names its loop (`"harness": "pi"`, set_harness). `--harness` defaults to it. Another loop runs only as an --experiment, tagged <protocol>+<harness> (row_protocol). The guard runs before the ECI preflight. - cron: the shadow is generic (REDLINES_SHADOW_HARNESS); litellm is now the check. - compare_harness: --pi becomes --other, and the loops are named from the rows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
elsehow
added a commit
that referenced
this pull request
Oct 19, 2026
…ts (#6) The published set now declares its instrument and its variants (data/combined_conditions.json "instrument": "minimal", "variants": five). run_unified asks one call per model per variant -- 4 models x 5 = 20 calls a run -- with no new cron flags; rows carry `variant`, and the existing pooling (redlines.runlog.pool) makes each model's published number the mean of its five, the ensemble median the median of those means. The minimal instrument is the joint prompt with the guidance cut: no system prompt (one space; pi substitutes its coding-assistant prompt for an empty one), no research paragraph, no rationale length, no debrief, mechanical tool descriptions, no research floor or coverage rule. The questions, LEAP and capability conditions, ECI step and delivery mechanics are v9's byte for byte. A variant's instruction is a plain paragraph before the delivery paragraph. The turn limit is a 100-turn runaway guard that also keeps agent-box's submit nudge on (Opus 5.5's thinking-only reply cut at the 128k output limit ended a call, 2026-10-19). Also: - pi runs from an empty neutral directory: pi writes its cwd into the system prompt, and from the checkout that named the project to the model. - A call with no forecasts after every attempt fails instead of writing null rows that were logged as done. - validate_launch_run checks one call per model per variant, each against its own prompt; no research floor or debrief check under the minimal instrument. - Timeline: one light line per variant (panel median of that prompt's calls), hover for the prompt; on variant dates the re-run band is dropped, since the draws are prompts, not re-asks. The FAQ shows the baseline's prompt. - cron_run.sh: --workers 10 for the 20 calls. - COMBINED_PROTOCOL v10, its lineage, and the pins in the tests and paper. Box: airo-dev/jason (claude) Co-authored-by: Nick Merrill <nick@forecastingresearch.org> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 84aaf24)
elsehow
added a commit
that referenced
this pull request
Oct 19, 2026
Linux hands one argument at most 128 KiB (MAX_ARG_STRLEN), and AIRO's joint prompt passed that on 2026-10-19 when every incident rung became a deaths question and a damages question. pi reads piped stdin as its first prompt, verbatim but for surrounding whitespace, so harness.run now writes the prompt to pi's stdin and pi_command carries no positional message. Every caller -- AIRO's instrument, the ForecastBench driver -- is uncapped by it. harness.PROMPT_ON_STDIN lets a caller check before sending a prompt that would not fit an argument. Tests: the fake pi records what arrived on stdin; the real pi (0.99.2) receives a prompt past the limit and the scripted model sees it, verbatim but for pi's trim, as the first user message. Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is the cut-over after the #5 shadow. From Fri 10-16, the published run goes through agent-box's pi loop under a new tag,
unified-joint-combined-v9. The instrument is v8's unchanged; only the loop moves.Evidence. The full panel through pi on 2026-10-14 (
results/experiments/pi-shadow-2026-10-14T2144Zon the box), compared with that day's litellm run:All 3,444 cells were answered for every model. Opus's debrief ended INCOMPLETE because a provider refusal ended pi's run; agent-box#4 fixes that.
Changes
"harness": "pi").--harnessdefaults to the set's loop. Another loop runs only as an--experiment, with rows tagged<protocol>+<harness>, so it never pools with the series.REDLINES_SHADOW_HARNESS; litellm is now the check.compare_harness --pibecomes--other, and the loops are named from the rows.Deploy (box)
git pullin~/Projects/airo.~/.config/redlines/env, replaceREDLINES_PI_SHADOW=1withREDLINES_SHADOW_HARNESS=litellm.FORECASTER_PIand node onPATHare already there.Tests: 338 pass.
test_the_tabs_take_the_newest_instrumentfails on main as well.🤖 Generated with Claude Code