Skip to content

Commit 073dd08

Browse files
authored
Move default, Auto and test-agent model calls onto the Anthropic BYOK key, with a no-fallback guard (#220)
* Route default, Auto and test-agent model calls to Claude on the Anthropic BYOK key - Auto Fast and Auto Pro route only to Claude (Haiku 5.5, Sonnet 5.5) and skip the Jev routing call - agentic e2e actor and judge move to anthropic/claude-haiku-5.5 pinned to the Anthropic provider - live verification and the live gate default to Claude Haiku and refuse non-Anthropic models - the live proxy pins rewritten Claude calls to Anthropic with allow_fallbacks false - new guard test fails on a non-Anthropic default, Auto candidate, test model, allow_fallbacks true, or a non-Claude models list * Keep the testing agent within Anthropic's four cache breakpoints The OpenRouter provider copies a tool message's cache breakpoint onto every tool result, so an actor turn with parallel tool calls sent Anthropic 5-6 breakpoints and failed. Move it onto the last result. * Judge agentic tests with Claude Sonnet; keep Haiku as the actor A Haiku judge failed 23 of 180 checks on a fake-mode run; Sonnet passed 21 of those 23 with the same Haiku actor. Both stay pinned to Anthropic (BYOK) with fallbacks off.
1 parent 79cb1f8 commit 073dd08

17 files changed

Lines changed: 333 additions & 153 deletions

File tree

‎.claude/skills/verify-debatelab/SKILL.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,14 @@
11
---
22
name: verify-debatelab
3-
description: Launch and drive the real DebateLab web app (React client + local Supabase Auth/Postgres/Edge Functions, with either a fake model provider or real OpenRouter pinned to the cheap meta/muse-spark-1.3-contributor model plus real Exa research) to prove a user-facing feature works, with screenshots, ARIA snapshots, provider calls, and database rows as evidence. Use when asked to verify, demo, reproduce, or screenshot DebateLab behavior in a browser, or before calling a DebateLab UI/Edge Function change done.
3+
description: Launch and drive the real DebateLab web app (React client + local Supabase Auth/Postgres/Edge Functions, with either a fake model provider or real OpenRouter pinned to Claude Haiku 5.5 on the Anthropic BYOK key plus real Exa research) to prove a user-facing feature works, with screenshots, ARIA snapshots, provider calls, and database rows as evidence. Use when asked to verify, demo, reproduce, or screenshot DebateLab behavior in a browser, or before calling a DebateLab UI/Edge Function change done.
44
---
55

66
# Verify DebateLab
77

88
DebateLab is a web app. The surface a user touches is the browser client at `/login` and `/app/*`, backed by Supabase Auth, Postgres/RLS, and Deno Edge Functions. This skill runs all of that locally and drives it with Playwright. It has two modes:
99

1010
- **Fake (default, free).** `e2e/full-stack/fake-openrouter.mjs` answers every model call with canned text. Proves wiring, rows, and UI, not model behavior.
11-
- **Live (`up --live`, costs money).** The same container becomes a recording proxy to real OpenRouter. It rewrites every chat call to `meta/muse-spark-1.3-contributor` (override with `VERIFY_LIVE_MODEL`), records OpenRouter's reply model as proof, and stops forwarding after `--max-calls` (default 400). Edge Functions also get a real Exa key, so web research and verified citations are real.
11+
- **Live (`up --live`, costs money).** The same container becomes a recording proxy to real OpenRouter. It rewrites every chat call to `anthropic/claude-haiku-5.5` pinned to the Anthropic provider with fallbacks off, so it bills the Anthropic BYOK key and not OpenRouter credits (override with `VERIFY_LIVE_MODEL`, which must be an `anthropic/` model), records OpenRouter's reply model as proof, and stops forwarding after `--max-calls` (default 400). Edge Functions also get a real Exa key, so web research and verified citations are real.
1212

1313
Everything goes through one helper, run from the repository root (`git rev-parse --show-toplevel`):
1414

@@ -42,7 +42,7 @@ STRIPE_TEST_SECRET_KEY=sk_test_... # optional; live keys are refused
4242
STRIPE_TEST_PRICE_ID=price_... # required with the Stripe key
4343
```
4444

45-
It prints nothing about the values. Ask Adam before the first live run: it spends money and sends prompts to a contributor model that may use them for training, so never put real user data in a live run. Costs: Muse Spark 1.3 contributor is $0.10 per million prompt tokens and $0.20 per million completion tokens, so a full `drive all` is cents of model spend. Exa bills per search and is usually the larger part. Jev decisions go to `/api/alpha/decisions` with Jev's own model and are not pinned. `doctor` shows `live calls used N/max`; once the cap is hit the proxy answers 429 and tests fail rather than spend more.
45+
It prints nothing about the values. Ask Adam before the first live run: it spends money (Exa, Jev, and the Anthropic key), so never put real user data in a live run. Model calls bill the Anthropic BYOK key; Exa bills per search. Jev decisions go to `/api/alpha/decisions` with Jev's own model and are not pinned. `doctor` shows `live calls used N/max`; once the cap is hit the proxy answers 429 and tests fail rather than spend more.
4646

4747
## Launch
4848

@@ -123,7 +123,7 @@ $H e2e cache ls # recorded agent flo
123123

124124
- `e2e` runs `doctor` first and refuses a broken stack, like `drive`. It turns e2e telemetry off and prints `EVIDENCE <dir>`, which holds `summary.md`, `report.json`, and per-attempt screenshots, traces, and failure videos.
125125
- The suite is tool journeys (tag `journey`) and a page sweep (tag `pages`) of every student and public page in English, Spanish and Hungarian (not `/app/admin` or `/share/:code`). Every test fails on an uncaught error, `console.error` (React warnings included), a Supabase 5xx, horizontal overflow, or a request to a hosted project: `agentic/support/capture-errors.js` records them in the page. A failure that names `errors the page logged` is a real app error to fix, not test noise.
126-
- The testing agent has its own OpenRouter models: `inclusionai/ling-3.0-flash-vl` acts (test steps on a cache miss, and explorations) and `meta/muse-spark-1.3-contributor` judges. `E2E_AGENT_MODEL` and `E2E_JUDGE_MODEL` override them; the model is not part of the replay key. The key comes from `OPENROUTER_API_KEY`, `LIVE_OPENROUTER_API_KEY`, or the live keys file, and is needed in fake mode too. A recorded `agent.act` replays with no model call; every judgment and every exploration step is a call. Without a key, agent tests skip. An OpenRouter error such as `[Meta] The requested model was not found` is infrastructure: rerun with `--last-failed`. The Muse Spark judge cannot act.
126+
- The testing agent has its own OpenRouter models: `anthropic/claude-haiku-5.5` acts (test steps on a cache miss, and explorations) and `anthropic/claude-sonnet-5.5` judges, both pinned to the Anthropic provider (BYOK) with fallbacks off. Haiku is not reliable enough to judge screens: it reads content below the fold as cut off. `E2E_AGENT_MODEL` and `E2E_JUDGE_MODEL` override them but must be `anthropic/` models; the model is not part of the replay key. The key comes from `OPENROUTER_API_KEY`, `LIVE_OPENROUTER_API_KEY`, or the live keys file, and is needed in fake mode too. A recorded `agent.act` replays with no model call; every judgment and every exploration step is a call. Without a key, agent tests skip. An OpenRouter error naming Anthropic (an outage, or the key out of credit) is infrastructure: rerun with `--last-failed`.
127127
- `--session student` starts an exploration signed in as a fresh demo user, minted by `agentic/sessions.e2e.ts`. `--agent` picks a persona: `first-timer`, `coach`, `edge-input`, or `hungarian`. `--target mobile` explores the phone-sized viewport.
128128
- Explorations report `issue` (functional) and `warning` (cosmetic) findings, and exit 1 when there is an issue. Treat a finding as a lead: the stack serves Vite dev, so a route that is still compiling can look dead, and fake mode answers every motion with free-transport text. Prove a finding with a repro test tagged `repro` before you report it as a bug.
129129
- Several explorations can run at once (a bug bash), because each gets its own evidence directory. They share this one stack, so give each a different area.

‎.claude/skills/verify-debatelab/bin/verify.sh‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -40,8 +40,11 @@ STATE="$STATE_DIR/stack.env"
4040
USERS="$STATE_DIR/users.tsv"
4141
ARTIFACTS="$VERIFY_ROOT/artifacts" # evidence; never removed by this script
4242
PROJECT_ID="$(basename "$REPO")" # supabase/config.toml has no project_id
43-
# Adam's standing choice for live verification: the cheap contributor tier.
44-
LIVE_MODEL="${VERIFY_LIVE_MODEL:-meta/muse-spark-1.3-contributor}"
43+
# Live verification runs on Claude Haiku through DebateLab's Anthropic key
44+
# (BYOK in OpenRouter). Other families bill OpenRouter credits, so they are
45+
# refused; the proxy pins the rewritten call to the Anthropic provider.
46+
LIVE_MODEL="${VERIFY_LIVE_MODEL:-anthropic/claude-haiku-5.5}"
47+
case "$LIVE_MODEL" in anthropic/*) ;; *) printf 'FAIL VERIFY_LIVE_MODEL must be an anthropic/ model (Anthropic BYOK), not %s\n' "$LIVE_MODEL" >&2; exit 1 ;; esac
4548
LIVE_KEYS_FILE="${VERIFY_LIVE_KEYS_FILE:-$HOME/.config/debatelab/live-verify.env}"
4649

4750
say() { printf '%s\n' "$*"; }
@@ -299,7 +302,7 @@ cmd_e2e() {
299302
[ -n "$key" ] && export OPENROUTER_API_KEY="$key"
300303
fi
301304
if [ -n "${OPENROUTER_API_KEY:-}" ]; then
302-
say "info testing agent: OpenRouter actor ${E2E_AGENT_MODEL:-inclusionai/ling-3.0-flash-vl}, judge ${E2E_JUDGE_MODEL:-meta/muse-spark-1.3-contributor} (replayed steps make no model call)"
305+
say "info testing agent: OpenRouter actor ${E2E_AGENT_MODEL:-anthropic/claude-haiku-5.5}, judge ${E2E_JUDGE_MODEL:-anthropic/claude-sonnet-5.5}, Anthropic BYOK only (replayed steps make no model call)"
303306
else
304307
say "info no OpenRouter key: tests with agent steps will skip"
305308
fi

‎.claude/skills/verify-debatelab/features/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ This directory is the maintained source for verifying DebateLab's user-facing be
88
- The browser client is the `up` instance at `http://127.0.0.1:18081`. Never use the `debatelab` container on `:8080` or a plain `npm run dev`, because both reach production.
99
- Each scripted test signs in with its own fresh demo-code user (a Pro account with a daily limit of 999) and deletes it afterwards. The fixture marks the onboarding tour as seen, and the user starts with no Current Debate.
1010
- Most tools work inside a Current Debate (see [current-debate](./current-debate.md)). Start one with `startDebate(page, motion)` first, or answer the motion dialog that opens after submit with `chooseForMotion(page, "create" | "once")`. The Library's `Add Source` stays disabled until a debate exists.
11-
- In fake mode, model answers come from the fake provider and contain the marker `FAKE-PROVIDER`. In live mode (`up --live`), every chat call goes to real OpenRouter pinned to `meta/muse-spark-1.3-contributor`, research uses real Exa, and answers are real model text.
11+
- In fake mode, model answers come from the fake provider and contain the marker `FAKE-PROVIDER`. In live mode (`up --live`), every chat call goes to real OpenRouter pinned to `anthropic/claude-haiku-5.5` on the Anthropic BYOK key, research uses real Exa, and answers are real model text.
1212
- The feature files quote fake-mode text such as `FAKE-PROVIDER case thesis`. Recipes never assert it: they read the `ai` response body with `aiResponse()` and check that its fields are on screen, so the same recipe passes in both modes.
1313

1414
## Driving conventions

‎.claude/skills/verify-debatelab/features/model-defaults.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,5 +20,5 @@ Every AI tool has a model selector. Basic accounts can pick Claude Haiku 5.5, GP
2020

2121
## Gotchas
2222

23-
- Fake mode records the requested model but answers with fake text, and live mode rewrites every chat call to the Muse Spark contributor pin. Neither proves that OpenRouter accepts the Claude 5.5 IDs; that needs a direct OpenRouter call.
23+
- Fake mode records the requested model but answers with fake text, and live mode rewrites every chat call to the Claude Haiku 5.5 pin. Neither proves that OpenRouter accepts the Sonnet and Opus 5.5 IDs; that needs a direct OpenRouter call.
2424
- Existing saved choices of a model still in the catalog are kept. Only retired IDs and unsaved settings move to the new defaults. Saved Opus 5.5 settings from the old Pro default are moved to Sonnet 5.5 once, by the `20261022120000_opus_default_to_sonnet.sql` data migration, not by the client.

‎.superset/db-gate.sh‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -30,9 +30,11 @@ FAKE_PROVIDER=debatelab-fake-openrouter
3030
# the fake provider. It needs LIVE_OPENROUTER_API_KEY and LIVE_EXA_API_KEY.
3131
# LIVE_MODEL=as-requested runs e2e/model-matrix instead: one case per catalog model.
3232
LIVE_PROVIDERS="${LIVE_PROVIDERS:-0}"
33-
# Muse Spark 1.3 Contributor is the cheapest model that supports every request
34-
# parameter the app sends. It is a test-only model: it is not in the app catalog.
35-
LIVE_MODEL="${LIVE_MODEL:-meta/muse-spark-1.3-contributor}" # empty also means default
33+
# Claude Haiku runs on DebateLab's Anthropic key (BYOK in OpenRouter), so the
34+
# live gate spends no OpenRouter credits. Other families would, so they are
35+
# refused; as-requested (the model matrix) is the one deliberate exception.
36+
LIVE_MODEL="${LIVE_MODEL:-anthropic/claude-haiku-5.5}" # empty also means default
37+
case "$LIVE_MODEL" in anthropic/*|as-requested) ;; *) echo "LIVE_MODEL must be an anthropic/ model (Anthropic BYOK) or as-requested, not $LIVE_MODEL" >&2; exit 1 ;; esac
3638
if [ "$LIVE_PROVIDERS" = 1 ] && { [ -z "${LIVE_OPENROUTER_API_KEY:-}" ] || [ -z "${LIVE_EXA_API_KEY:-}" ]; }; then
3739
echo "LIVE_PROVIDERS=1 requires LIVE_OPENROUTER_API_KEY and LIVE_EXA_API_KEY" >&2
3840
exit 1

‎AGENTS.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -96,14 +96,14 @@ To prove that a user-facing change works, drive the real app with the `verify-de
9696
```bash
9797
H=.claude/skills/verify-debatelab/bin/verify.sh
9898
$H up # fake model provider, free
99-
$H up --live # real AI: OpenRouter pinned to meta/muse-spark-1.3-contributor, real Exa
99+
$H up --live # real AI: OpenRouter pinned to anthropic/claude-haiku-5.5 (Anthropic BYOK), real Exa
100100
$H doctor # proves the stack is local, ours, and (live) pinned
101101
$H drive all # every recipe; or recipes/<name>, e2e/<path>; add --project desktop-chromium
102102
$H e2e run # agentic tests in agentic/ (TesterArmy e2e); `e2e explore "<goal>" --session student --agent <persona>` hunts for bugs
103103
$H down # always, including after failures
104104
```
105105

106-
- Use `--live` when the change touches prompts, model output handling, web research, or citations. The fake mode cannot prove those. URL import fetches pages directly and works the same in both modes. Every chat call is rewritten to the Muse Spark contributor model, so a full run costs cents plus Exa searches, and the proxy stops after 400 upstream calls (`--max-calls N`). Only use made-up debate content: the contributor model may keep prompts.
106+
- Use `--live` when the change touches prompts, model output handling, web research, or citations. The fake mode cannot prove those. URL import fetches pages directly and works the same in both modes. Every chat call is rewritten to Claude Haiku 5.5 pinned to the Anthropic provider, so model calls bill the Anthropic BYOK key, not OpenRouter credits; Exa searches still cost, and the proxy stops after 400 upstream calls (`--max-calls N`). Only use made-up debate content. Never point a default, an Auto pool, or the testing agent at a non-`anthropic/` model or enable `allow_fallbacks`: `src/test/ai/anthropic-only-guard.test.ts` fails on it.
107107
- Live keys come from `~/.config/debatelab/live-verify.env` (mode 600, `LIVE_OPENROUTER_API_KEY` and `LIVE_EXA_API_KEY`). On Adam's VPS it holds copies of the Hermes keys in `~/.hermes/.env`. Never print them.
108108
- The stack holds the same lock as `./.superset/db-gate.sh`, so only one worktree can run it at a time.
109109
- Most tools work inside a Current Debate. Recipes start one with `startDebate` or answer the motion dialog with `chooseForMotion`; see the skill's Drive section.

‎agentic/support/anthropic-cache.ts‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
// The testing agent marks Anthropic prompt-cache breakpoints on the system
2+
// prompt and on the newest message. @openrouter/ai-sdk-provider sends a tool
3+
// message as one OpenRouter message per tool result and copies the message's
4+
// breakpoint onto each, so an actor turn with parallel tool calls asks
5+
// Anthropic for more than its limit of four breakpoints and the call fails
6+
// ("A maximum of 4 blocks with cache_control may be provided"). Moving the
7+
// breakpoint onto the last tool result keeps one per message, which caches
8+
// the same prefix.
9+
10+
type Options = Record<string, Record<string, unknown> | undefined>;
11+
type Part = { providerOptions?: Options } & Record<string, unknown>;
12+
type Message = { role: string; content: unknown; providerOptions?: Options };
13+
type CallOptions = { prompt: Message[] } & Record<string, unknown>;
14+
15+
export function oneBreakpointPerToolMessage(message: Message): Message {
16+
if (message.role !== "tool" || !Array.isArray(message.content) || message.content.length < 2) return message;
17+
const anthropic = message.providerOptions?.anthropic;
18+
const breakpoint = anthropic?.cacheControl ?? anthropic?.cache_control;
19+
if (!anthropic || breakpoint === undefined) return message;
20+
const { cacheControl: _camel, cache_control: _snake, ...rest } = anthropic;
21+
const parts = message.content as Part[];
22+
const last = parts.length - 1;
23+
return {
24+
...message,
25+
providerOptions: { ...message.providerOptions, anthropic: rest },
26+
content: parts.map((part, index) =>
27+
index === last
28+
? { ...part, providerOptions: { ...part.providerOptions, anthropic: { ...part.providerOptions?.anthropic, cacheControl: breakpoint } } }
29+
: part,
30+
),
31+
};
32+
}
33+
34+
/** The same model, with every call's prompt passed through oneBreakpointPerToolMessage. */
35+
export function withAnthropicCacheLimit<M extends object>(model: M): M {
36+
const fix = (options: CallOptions): CallOptions => ({ ...options, prompt: options.prompt.map(oneBreakpointPerToolMessage) });
37+
return new Proxy(model, {
38+
get(target, property) {
39+
const value: unknown = Reflect.get(target, property);
40+
if ((property === "doGenerate" || property === "doStream") && typeof value === "function") {
41+
return (options: CallOptions) => value.call(target, fix(options));
42+
}
43+
return typeof value === "function" ? value.bind(target) : value;
44+
},
45+
});
46+
}

0 commit comments

Comments
 (0)