Repository navigation
Move default, Auto and test-agent model calls onto the Anthropic BYOK key, with a no-fallback guard - #220
Merged
Conversation
…opic BYOK key - Auto Fast and Auto Pro route only to Claude (Haiku 5.5, Sonnet 5.5) and skip the Jev routing call - agentic e2e actor and judge move to anthropic/claude-haiku-5.5 pinned to the Anthropic provider - live verification and the live gate default to Claude Haiku and refuse non-Anthropic models - the live proxy pins rewritten Claude calls to Anthropic with allow_fallbacks false - new guard test fails on a non-Anthropic default, Auto candidate, test model, allow_fallbacks true, or a non-Claude models list
ℹ️ Recent review info
📝 Walkthrough
Note 🎁 Summarized by CodeRabbit Free
Comment |
The OpenRouter provider copies a tool message's cache breakpoint onto every tool result, so an actor turn with parallel tool calls sent Anthropic 5-6 breakpoints and failed. Move it onto the last result.
A Haiku judge failed 23 of 180 checks on a fake-mode run; Sonnet passed 21 of those 23 with the same Haiku actor. Both stay pinned to Anthropic (BYOK) with fallbacks off.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Moves DebateLab's own model spend onto the Anthropic key (BYOK in OpenRouter, covered by Adam's Claude plan) and adds a guard so non-Anthropic models and provider fallbacks can't creep back in. Do not merge or deploy without Adam's OK.
What #185 already did (confirmed on main)
DEFAULT_MODEL=anthropic/claude-haiku-5.5(Basic),DEFAULT_PRO_MODEL=anthropic/claude-sonnet-5.5(Pro),BACKGROUND_MODEL(research sub-steps) = Haiku 5.5.openRouterProviderOptions()sends{ only: ["anthropic"], allow_fallbacks: false, require_parameters: true }for everyanthropic/model, in bothaiandai-stream. No call site sends an OpenRoutermodelsfallback array.anthropic/claude-haiku-5.5is the newest Haiku onGET /api/v1/models(Anthropic is one of its providers; tools, tool_choice, structured outputs and image input are supported there).So the app's default traffic was already on BYOK. The non-Anthropic spend came from the test harness and from Auto routing. This PR fixes those.
Call sites
DEFAULT_MODEL, ai + ai-stream)DEFAULT_PRO_MODEL)BACKGROUND_MODEL)auto-model-router.ts)e2e.config.ts)inclusionai/ling-3.0-flash-vl(Novita/DeepInfra, with fallback)anthropic/claude-haiku-5.5,only: ["anthropic"],allow_fallbacks: falsee2e.config.ts)meta/muse-spark-1.3-contributoranthropic/claude-haiku-5.5, pinned the same wayverify.sh up --live,VERIFY_LIVE_MODEL)meta/muse-spark-1.3-contributoranthropic/claude-haiku-5.5; non-anthropic/refuseddb-gate.sh,live-e2e.ymlviaLIVE_MODEL)meta/muse-spark-1.3-contributoranthropic/claude-haiku-5.5; onlyanthropic/*oras-requestedfake-openrouter.mjs)only/allow_fallbackswhen rewriting the modelmodelslist, refuses a non-ClaudeLIVE_MODELat startupscripts/prompt-examples.tsdefaultopenai/gpt-6-lunaanthropic/claude-haiku-5.5callJev,typesafe/jev-1.13)model-matrix.yml,LIVE_MODEL=as-requested)Kept on Sonnet or Opus
Anti-fallback guarantees
anthropic/request carriesprovider: { only: ["anthropic"], allow_fallbacks: false }, both from the app and from the test agent and live proxy. Nomodelsfallback arrays anywhere.anthropic/models. With one model per pool the Jev router isn't called.e2e.config.ts,verify.sh,db-gate.shand the live proxy fail loudly if a non-anthropic/model is configured (env var or repo variable), instead of quietly billing credits. The only exception isLIVE_MODEL=as-requested(model matrix).src/test/ai/anthropic-only-guard.test.ts(runs innpm test) fails if: a default/background model isn't Claude; Claude isn't pinned toonly: ["anthropic"]+allow_fallbacks: false; an Auto candidate isn'tanthropic/; the e2e actor/judge default or pin changes; the verify/gate/proxy defaults or refusals change; or any non-test source undersupabase/functions,scripts,agentic,e2e, the verify skill,src/libore2e.config.tshasallow_fallbacks: trueor amodels: [...]list with a non-Claude entry. I mutation-checked it: adding GPT-6 Luna to Auto Fast and anallow_fallbacks: true+ mixedmodelslist both failed it.auto-model-router.test.tsrewritten: the Auto pools are Claude-only and skip Jev; the Jev routing logic is still covered throughrouteAmongCandidateswith a two-Claude test pool.Verify (VPS, fresh clone of main at 3aa123a + this commit)
npm run verify: pass. docs:check, lint, typecheck, vitest 592/592 (82 files), collaboration 30/30, Deno edge 457/457, build, service-worker test, Playwright smoke 47/47.verify.sh upin fake mode, thene2e run agentic/ai-tools.e2e.ts:8 :47 --target desktop): 2/2 pass, judgeopenrouter/anthropic/claude-haiku-5.5, 2 model calls, 6.3k tokens, reported $0.0000 (BYOK). Bothagent.actsteps replayed from cache, so the actor wasn't exercised live.e2e/live, the full agentic suite, and the model matrix (they need Actions or a full live run).Config / secrets
aiandai-streamfunctions are next deployed. Note: this PR doesn't deploy anything.E2E_AGENT_MODEL,E2E_JUDGE_MODELandLIVE_MODELare unset (checked), so the new defaults apply. If someone sets one to a non-Claude model, the run now fails instead of spending credits.allow_fallbacks: falsestops other providers, but only that setting stops OpenRouter from serving Anthropic on its own credits if the BYOK key errors. The app already logsclaude_not_byokwhen that happens.labkey needs no change.Risks
agent.assert/waitFor/extract. Haiku passed the two journeys above, but the full suite (about 250 judgments) hasn't run on it. Expect a few judgments to flake until they're tuned. The actor (only used on cache misses and in explorations) is untested live on Haiku.recipes/fixtures.tsrequires OpenRouter's replymodelto start withLIVE_MODEL. If OpenRouter answers Haiku 5.5 with a dated id, live recipes would flag it. Not testable without a live run.provider_unavailablehandling) instead of failing over. That now also applies to the e2e judge and live verification.Expected effect
Based on the last-30-day OpenRouter numbers (~$39 total): the e2e and live-verify traffic, including Muse Spark 1.3 Contributor (~4K requests, $2.04), the Ling actor, and the Auto-routed GPT/Muse calls, moves onto the Anthropic BYOK key. Opus 5.5 ($15.90) and Sonnet 5 ($11.10) were already BYOK. Non-Anthropic OpenRouter spend from DebateLab's own defaults, Auto routing and test tooling should go to about $0.
What stays on OpenRouter credits:
Overlap with in-flight work
package.json, which this PR doesn't touch.Follow-up for Adam (not in this PR)
Should Basic users still be offered GPT-6 Luna and Muse Spark 1.3, and Pro users GPT/Gemini/DeepSeek/GLM, in the model picker? Those are the only way app traffic can still bill OpenRouter credits. Removing them is a UI/product change.
Summary by CodeRabbit
New Features
Improvements