Repository navigation
The sixth call: the baseline prompt with AI forecasts banned, drawn but never pooled - #10
Merged
Merged
Conversation
…ut never pooled (#7) * The sixth call: the baseline prompt with AI forecasts banned, drawn but never pooled combined-v10 asks the minimal instrument under five prompts; each model's published number is the mean of its five. The set now also declares one UNPOOLED variant, "ai-banned" (data/combined_conditions.json `unpooled_variants`, from code/make_combined_conditions.py): the `none` prompt with one added sentence -- forecasts by AI systems carry no information beyond their inputs; do not consult them -- and agent-box's URL guard (FORECASTER_BLOCK_URL_PATTERNS) against AIRO's own pages for that call alone. It is the `both` arm of the anchor-guard experiment of 2026-10-16 (#5), asked every run with no new cron flag: 4 x 6 = 24 calls. - The guard is per call. run_one_joint hands it to the loop as `env`, and redlines/pi_loop.py passes it to agent-box's harness.run(env=...) beside PI_CODING_AGENT_DIR; never os.environ, which the ten workers share. A guard in the process environment is refused outside a --joint --experiment, as #5 had it, and a bad pattern fails before any paid call. - redlines/tools.py mirrors agent-box's url-guard (blocked_patterns, is_blocked; web_search drops matching results and the answer built from them, read_page refuses); tests/test_tools_parity.py now covers the two guard functions, tests/test_tools_guard.py ports agent-box's mocked tests. - The sixth's rows carry `variant`, `url_block_patterns` and `pooled: false`; the five's rows and prompts are unchanged, so the protocol stays v10. redlines.runlog.pooled_rows drops the stamped rows from load_runlog() and load_conditional(), so each model's point, the CI, the ensemble median, Graph 1, the data bank and Policy-levers stay the mean of the five. The download keeps everything. - Timeline: a sixth, dashed line "AI forecasts banned"; hover shows the sentence, the guard and that it is not in the mean; the "N prompts per model" count is of the pooled ones. - A failed sixth call does not fail the run (main_joint counts it apart), and validate_launch_run expects 24 calls with the sixth optional: a missing or short one is a `variant_warnings` entry, a missing pooled call still an error. The sixth's stamps are checked; the five's absence of them too. - tests/test_sixth_call.py: a synthetic 24-call v10 run through a fake loop passes the validator; the views built over its rows show six lines and a five-prompt mean. Production needs agent-box with its url-guard branch merged: an older agent-box ignores the variable and the sixth call would search unguarded while its rows say otherwise. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Box: airo-dev/airo-ban (claude) * validate_launch_run: a defective sixth call is a warning, never a block on the five The reviewer of 1f9d0c6 showed a sixth call with no rationale, or whose ECI elicit never came back ordered (run_one_joint writes rows after its retries whenever the grid is non-empty), still failed the day: `rationale_present` and `elicited_metadata` ran as errors on the unpooled call. Now every check on what a CALL returned -- its grid, its numbers, its rationale, its elicit -- goes through check_call: a variant warning on a `pooled: false` call, an error on a pooled one. What a row must say about its run (protocol, prompt, windows, dates, the unpooled stamp) stays an error on every row. tests/test_sixth_call.py pins the four cases on the sixth (warning, rc 0) and on a pooled call (error, rc 1). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Box: airo-dev/airo-ban (claude) --------- Co-authored-by: Nick Merrill <nick@forecastingresearch.org> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> (cherry picked from commit 2f7ed12)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
combined-v10 asks the minimal instrument under five prompts; each model's published number is the mean of its five. This adds a sixth call per model, asked every run, drawn on the Timeline and never in the published number: the
noneprompt with the AI-forecast ban sentence, and agent-box's URL guard against AIRO's own pages for that call alone (thebotharm of the anchor-guard experiment of 2026-10-16).unpooled_variantsindata/combined_conditions.json, frommake_combined_conditions.py); the cron command is unchanged, 4 × 6 = 24 calls; the protocol stays v10 and the five's rows and prompts are byte-identical.run_one_jointhands it to the loop asenv,pi_looppasses it toharness.run(env=...), never into the process environment the ten workers share.redlines/tools.pymirrors agent-box's guard byte for byte.variant,url_block_patternsandpooled: false;runlog.pooled_rowskeeps them out of every pooled view. The Timeline draws a dashed "AI forecasts banned" line.Reviewed twice and merged on the development fork as elsehow#7 (366 passed; a synthetic 24-call run passes the validator). This is that squash, cherry-picked onto main.
Deploy (box): needs agent-box with the URL guard (forecastingresearch/agent-box#5) reinstalled in the cron venv, then
git pull --ff-onlyin~/Projects/airo. An older agent-box ignores the guard and the sixth call would search unguarded while its rows say otherwise.🤖 Generated with Claude Code