Repository navigation
--harness pi: the instrument through FRI's agent-box, as a shadow of the cron - #5
Merged
Merged
Conversation
…the cron redlines/pi_loop.py keeps redlines.llm.call_tools's contract with pi as the loop: same prompt, tools and schemas, research floor, coverage rule, turn limit, debrief and validator (agent-box's loop options), same cleaning and rows. code/run_unified.py --harness pi swaps it in, only as a --joint --experiment; its rows carry protocol "<protocol>+pi". The debrief state gains a `validator` entry so a loop in another process runs the same check. code/compare_harness.py compares a shadow run with the cron's, cell by cell, beside the previous cron run as litellm's own spread. code/cron_run.sh runs the shadow and the comparison after publication when the env file sets REDLINES_PI_SHADOW=1; it can only notify. The `pi` extra installs agent-box. code/pi_harness/run_pi.py (the side-by-side trial script, never run at scale) is gone: --harness pi is the same comparison with the instrument's own pipeline. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
elsehow
added a commit
that referenced
this pull request
Oct 19, 2026
…ut never pooled (#7) * The sixth call: the baseline prompt with AI forecasts banned, drawn but never pooled combined-v10 asks the minimal instrument under five prompts; each model's published number is the mean of its five. The set now also declares one UNPOOLED variant, "ai-banned" (data/combined_conditions.json `unpooled_variants`, from code/make_combined_conditions.py): the `none` prompt with one added sentence -- forecasts by AI systems carry no information beyond their inputs; do not consult them -- and agent-box's URL guard (FORECASTER_BLOCK_URL_PATTERNS) against AIRO's own pages for that call alone. It is the `both` arm of the anchor-guard experiment of 2026-10-16 (#5), asked every run with no new cron flag: 4 x 6 = 24 calls. - The guard is per call. run_one_joint hands it to the loop as `env`, and redlines/pi_loop.py passes it to agent-box's harness.run(env=...) beside PI_CODING_AGENT_DIR; never os.environ, which the ten workers share. A guard in the process environment is refused outside a --joint --experiment, as #5 had it, and a bad pattern fails before any paid call. - redlines/tools.py mirrors agent-box's url-guard (blocked_patterns, is_blocked; web_search drops matching results and the answer built from them, read_page refuses); tests/test_tools_parity.py now covers the two guard functions, tests/test_tools_guard.py ports agent-box's mocked tests. - The sixth's rows carry `variant`, `url_block_patterns` and `pooled: false`; the five's rows and prompts are unchanged, so the protocol stays v10. redlines.runlog.pooled_rows drops the stamped rows from load_runlog() and load_conditional(), so each model's point, the CI, the ensemble median, Graph 1, the data bank and Policy-levers stay the mean of the five. The download keeps everything. - Timeline: a sixth, dashed line "AI forecasts banned"; hover shows the sentence, the guard and that it is not in the mean; the "N prompts per model" count is of the pooled ones. - A failed sixth call does not fail the run (main_joint counts it apart), and validate_launch_run expects 24 calls with the sixth optional: a missing or short one is a `variant_warnings` entry, a missing pooled call still an error. The sixth's stamps are checked; the five's absence of them too. - tests/test_sixth_call.py: a synthetic 24-call v10 run through a fake loop passes the validator; the views built over its rows show six lines and a five-prompt mean. Production needs agent-box with its url-guard branch merged: an older agent-box ignores the variable and the sixth call would search unguarded while its rows say otherwise. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Box: airo-dev/airo-ban (claude) * validate_launch_run: a defective sixth call is a warning, never a block on the five The reviewer of 1f9d0c6 showed a sixth call with no rationale, or whose ECI elicit never came back ordered (run_one_joint writes rows after its retries whenever the grid is non-empty), still failed the day: `rationale_present` and `elicited_metadata` ran as errors on the unpooled call. Now every check on what a CALL returned -- its grid, its numbers, its rationale, its elicit -- goes through check_call: a variant warning on a `pooled: false` call, an error on a pooled one. What a row must say about its run (protocol, prompt, windows, dates, the unpooled stamp) stays an error on every row. tests/test_sixth_call.py pins the four cases on the sixth (warning, rc 0) and on a pooled call (error, rc 1). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Box: airo-dev/airo-ban (claude) --------- Co-authored-by: Nick Merrill <nick@forecastingresearch.org> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> (cherry picked from commit 2f7ed12)
elsehow
added a commit
that referenced
this pull request
Oct 19, 2026
URL guard: FORECASTER_BLOCK_URL_PATTERNS, off unless set
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
AIRO's published runs go through its own litellm loop (
redlines.llm.call_tools); FRI's ForecastBench runs go through pi, via agent-box. Same panel today (Opus 5.5, GPT-6 Astra, GPT-6.1 Sol, Sonnet 5.5), two loops. This adds the pi loop as a shadow: same instrument, same day, never the series. A switch comes only after the shadow says the loops agree.What changes
redlines/pi_loop.py:call_tools's contract with pi as the loop. Prompt, tools and schemas, the research floor (10 returned calls), the coverage rule, the 40-turn limit, the debrief and its validator, cleaning and rows are the instrument's own; agent-box's loop options (Loop options for AIRO's instrument: floor, coverage, turn limit, debrief agent-box#3) enforce the rules inside pi.code/run_unified.py --harness pi: swaps the loop in. Only as--joint --experiment; rows carry protocol<protocol>+pi. The debrief state gains avalidatorentry so the out-of-process loop runs the same check. The litellm path and its rows are unchanged.code/compare_harness.py: shadow vs cron, cell by cell, beside the previous cron run as litellm's own spread.code/cron_run.sh: after publication, if the env file setsREDLINES_PI_SHADOW=1, runs the shadow and the comparison; it can only notify.pyproject.toml: apiextra (agent-box).code/pi_harness/run_pi.py, the never-run trial script, is gone.Smoke (box, 2026-10-14 19:56 UTC, GPT-6.1 Sol + Sonnet 5.5, $6.91): 574/574 rows each, first attempt; coverage met; debriefs complete; reasoning xhigh / max. Against that day's 12:00 cron run:
One draw per loop, so this only says nothing alarming yet.
Known differences (pi's, not the instrument's): pi cannot force
tool_choice, so every model gets the ask-in-words last turn (call_tools forces it for OpenAI models); the per-turn output cap is pi's for the model; the debrief's text-only fallback is not ported.To turn the shadow on (after a cron run, not before one): pull on the box;
pip install "fri-forecaster @ git+https://github.com/forecastingresearch/agent-box"into the cron's Python; addREDLINES_PI_SHADOW=1,FORECASTER_PI=…/pi-coding-agent/dist/bundle/cli.jsand aPATHwith node to~/.config/redlines/env. About $25–35 per shadow run.Tests:
tests/test_pi_loop.py(fake agent-box; the validator through agent-box when installed). Suite: 337 passed;test_conditional.py::TestGroupedSetOnTheTabs::test_the_tabs_take_the_newest_instrumentfails onupstream/maintoo.🤖 Generated with Claude Code