Skip to content

Commit eacfb43

Browse files
Programcodeclaude
andcommitted
test(agentic): act with Ling 3.0 Flash VL instead of GPT-6 Luna
Ling costs $0.021/$0.062 per million tokens on OpenRouter, about a fifth of Luna. With the replay cache off it passed 23 of 24 actor-heavy tests on both viewports for $0.024; the one failure, a dialog left open, passed twice on rerun. Gemma 3 12B, cheaper still, failed 19 of 24. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
1 parent 4025827 commit eacfb43

5 files changed

Lines changed: 8 additions & 7 deletions

File tree

‎.claude/skills/verify-debatelab/SKILL.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -119,7 +119,7 @@ $H e2e cache ls # recorded agent flo
119119

120120
- `e2e` runs `doctor` first and refuses a broken stack, like `drive`. It turns e2e telemetry off and prints `EVIDENCE <dir>`, which holds `summary.md`, `report.json`, and per-attempt screenshots, traces, and failure videos.
121121
- The suite is tool journeys (tag `journey`) and a page sweep (tag `pages`) of every student and public page in English, Spanish and Hungarian (not `/app/admin` or `/share/:code`). Every test fails on an uncaught error, `console.error` (React warnings included), a Supabase 5xx, horizontal overflow, or a request to a hosted project: `agentic/support/capture-errors.js` records them in the page. A failure that names `errors the page logged` is a real app error to fix, not test noise.
122-
- The testing agent has its own OpenRouter models: `openai/gpt-6-luna` acts (test steps on a cache miss, and explorations) and `meta/muse-spark-1.3-contributor` judges. `E2E_AGENT_MODEL` and `E2E_JUDGE_MODEL` override them; the model is not part of the replay key. The key comes from `OPENROUTER_API_KEY`, `LIVE_OPENROUTER_API_KEY`, or the live keys file, and is needed in fake mode too. A recorded `agent.act` replays with no model call; every judgment and every exploration step is a call. Without a key, agent tests skip. An OpenRouter error such as `[Meta] The requested model was not found` is infrastructure: rerun with `--last-failed`. The Muse Spark judge cannot act.
122+
- The testing agent has its own OpenRouter models: `inclusionai/ling-3.0-flash-vl` acts (test steps on a cache miss, and explorations) and `meta/muse-spark-1.3-contributor` judges. `E2E_AGENT_MODEL` and `E2E_JUDGE_MODEL` override them; the model is not part of the replay key. The key comes from `OPENROUTER_API_KEY`, `LIVE_OPENROUTER_API_KEY`, or the live keys file, and is needed in fake mode too. A recorded `agent.act` replays with no model call; every judgment and every exploration step is a call. Without a key, agent tests skip. An OpenRouter error such as `[Meta] The requested model was not found` is infrastructure: rerun with `--last-failed`. The Muse Spark judge cannot act.
123123
- `--session student` starts an exploration signed in as a fresh demo user, minted by `agentic/sessions.e2e.ts`. `--agent` picks a persona: `first-timer`, `coach`, `edge-input`, or `hungarian`. `--target mobile` explores the phone-sized viewport.
124124
- Explorations report `issue` (functional) and `warning` (cosmetic) findings, and exit 1 when there is an issue. Treat a finding as a lead: the stack serves Vite dev, so a route that is still compiling can look dead, and fake mode answers every motion with free-transport text. Prove a finding with a repro test tagged `repro` before you report it as a bug.
125125
- Several explorations can run at once (a bug bash), because each gets its own evidence directory. They share this one stack, so give each a different area.

‎.claude/skills/verify-debatelab/bin/verify.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -295,7 +295,7 @@ cmd_e2e() {
295295
[ -n "$key" ] && export OPENROUTER_API_KEY="$key"
296296
fi
297297
if [ -n "${OPENROUTER_API_KEY:-}" ]; then
298-
say "info testing agent: OpenRouter actor ${E2E_AGENT_MODEL:-openai/gpt-6-luna}, judge ${E2E_JUDGE_MODEL:-meta/muse-spark-1.3-contributor} (replayed steps make no model call)"
298+
say "info testing agent: OpenRouter actor ${E2E_AGENT_MODEL:-inclusionai/ling-3.0-flash-vl}, judge ${E2E_JUDGE_MODEL:-meta/muse-spark-1.3-contributor} (replayed steps make no model call)"
299299
else
300300
say "info no OpenRouter key: tests with agent steps will skip"
301301
fi

‎AGENTS.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -98,7 +98,7 @@ $H down # always, including after failures
9898
- The stack holds the same lock as `./.superset/db-gate.sh`, so only one worktree can run it at a time.
9999
- Most tools work inside a Current Debate. Recipes start one with `startDebate` or answer the motion dialog with `chooseForMotion`; see the skill's Drive section.
100100
- Neither mode proves Stripe Checkout, real Stripe cancellation, ElevenLabs voice audio, Google sign-in or export, staging, or production.
101-
- Agentic tests (`agentic/*.e2e.ts`) are an AI agent driving goals and judging screens, with exact assertions on the result. They cover a journey per tool and every student and public page in English, Spanish and Hungarian, and fail on page errors, `console.error`, and Supabase 5xx answers. They catch what selectors cannot: stray untranslated text, clipped layouts, off-topic model output in live mode. They also run explorations that hunt for bugs as a persona. The agent calls its own OpenRouter models (GPT-6 Luna acts, the Muse Spark contributor tier judges), even on a fake stack. A flow change that breaks an agentic test updates it in the same PR; commit new or re-recorded `agentic/replay/` entries with it. See [docs/testing.md](docs/testing.md#agentic-browser-tests).
101+
- Agentic tests (`agentic/*.e2e.ts`) are an AI agent driving goals and judging screens, with exact assertions on the result. They cover a journey per tool and every student and public page in English, Spanish and Hungarian, and fail on page errors, `console.error`, and Supabase 5xx answers. They catch what selectors cannot: stray untranslated text, clipped layouts, off-topic model output in live mode. They also run explorations that hunt for bugs as a persona. The agent calls its own OpenRouter models (Ling 3.0 Flash VL acts, the Muse Spark contributor tier judges), even on a fake stack. A flow change that breaks an agentic test updates it in the same PR; commit new or re-recorded `agentic/replay/` entries with it. See [docs/testing.md](docs/testing.md#agentic-browser-tests).
102102

103103
### Keep the skill working
104104

‎docs/testing.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -261,7 +261,7 @@ How it runs:
261261
- Always through `verify.sh e2e <run|explore|list|cache>`. The wrapper runs `doctor`, loads the stack environment, turns e2e telemetry off, and takes the testing agent's key from `OPENROUTER_API_KEY`, `LIVE_OPENROUTER_API_KEY`, or `~/.config/debatelab/live-verify.env`. It writes the report (`summary.md`, `report.json`), screenshots, traces, and failure videos to `.verify/artifacts/<run>-e2e-<command>-<pid>/`. The config refuses a non-loopback URL and never starts a dev server, because a plain `npm run dev` reaches production.
262262
- Each signed-in test signs in a fresh demo-code user through the Login page and deletes it afterwards. Public-page tests use the `visitorTest` fixture with the same checks.
263263
- There are two targets, `desktop` (1280 by 800) and `mobile` (393 by 851). e2e does not emulate touch. The app lays out by viewport width and never reads the user agent, so the viewport is what matters; the Playwright Pixel 5 project still owns touch behavior.
264-
- The testing agent has two models on OpenRouter, separate from the app's provider mode, and a fake-mode stack still needs both. The actor (`agent.act` and explorations) is `openai/gpt-6-luna`. Recorded flows replay without it, so it runs only on a cache miss and in explorations, at a fraction of a cent per step. OpenRouter withdrew the free Qwen tier the suite used before, which now fails with `This model is unavailable for free`. The judge (`agent.assert`, `waitFor`, `extract`) is `meta/muse-spark-1.3-contributor` ($0.10 and $0.20 per million input and output tokens), the same tier as the app's live pin. It answers every judgment on every run. A full run of both viewports makes about 250 judgments and costs about ten cents. It takes about 45 minutes on one machine with three workers, which is why CI splits it into four jobs. The judge cannot act, because Meta's endpoint accepts only `tool_choice: "auto"`. `E2E_AGENT_MODEL` and `E2E_JUDGE_MODEL` override the two. The judge's tier may keep prompts, so use made-up debate content only. A transient OpenRouter error such as `[Meta] The requested model was not found` is infrastructure: rerun with `--last-failed`. The agent's context explains that fake AI answers are canned free-transport text.
264+
- The testing agent has two models on OpenRouter, separate from the app's provider mode, and a fake-mode stack still needs both. The actor (`agent.act` and explorations) is `inclusionai/ling-3.0-flash-vl` ($0.021 and $0.062 per million input and output tokens on Novita, with DeepInfra as fallback). Recorded flows replay without it, so it runs only on a cache miss and in explorations. A small model is enough to click through these flows: with the cache off it passed the actor-heavy specs for about a fifth of GPT-6 Luna's price, while Gemma 3 12B, cheaper still, failed most of them. OpenRouter withdrew the free Qwen tier the suite used before, which now fails with `This model is unavailable for free`. The judge (`agent.assert`, `waitFor`, `extract`) is `meta/muse-spark-1.3-contributor` ($0.10 and $0.20 per million input and output tokens), the same tier as the app's live pin. It answers every judgment on every run. A full run of both viewports makes about 250 judgments and costs about ten cents. It takes about 45 minutes on one machine with three workers, which is why CI splits it into four jobs. The judge cannot act, because Meta's endpoint accepts only `tool_choice: "auto"`. `E2E_AGENT_MODEL` and `E2E_JUDGE_MODEL` override the two. The judge's tier may keep prompts, so use made-up debate content only. A transient OpenRouter error such as `[Meta] The requested model was not found` is infrastructure: rerun with `--last-failed`. The agent's context explains that fake AI answers are canned free-transport text.
265265
- Replay. When a later check passes, an `agent.act` flow is recorded in [`agentic/replay/`](../agentic/replay/). Later runs replay it with no model call and verify that its effect happened, until the app changes and the agent takes over and records again. The model is not part of the cache key, so a flow recorded with one actor replays under another. A step that ends on a passing toast or other transient state can hand off to the agent on replay (`end-mismatch` in the report); that costs a model call, not a failure. Commit new and changed entries with the test that made them. Local runs read and write the cache; CI only reads it. Changing an instruction, its params, the test or target name, the agent's `context`, or e2e's minor version re-records.
266266
- Without a model key or the stack environment, agent tests skip, so a bare `npx e2e run` proves nothing. `failOnSkippedFailure` fails a run in which a test skipped after a failure.
267267

‎e2e.config.ts‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -18,13 +18,14 @@ const live = process.env.VERIFY_MODE === "live";
1818
// The testing agent's models, separate from the app's own. Both need tool
1919
// calling and image input. The actor drives agent.act and explorations; its
2020
// recorded flows replay with no model call, so it runs only on a cache miss
21-
// and in explorations, for a fraction of a cent per step. GPT-6 Luna is fast
22-
// and reliable at it (OpenRouter withdrew the free Qwen tier used before).
21+
// and in explorations. Ling 3.0 Flash VL is about a fifth of GPT-6 Luna's
22+
// price and drives these flows fine; Gemma 3 12B, cheaper still, could not.
23+
// OpenRouter withdrew the free Qwen tier used before.
2324
// The judge answers every agent.assert, waitFor and extract on every run, so it
2425
// must be cheap and reliable: the Muse Spark contributor tier, which like the
2526
// app's live pin may keep prompts (made-up content only).
2627
// `||`, not `??`: an unset Actions variable arrives as an empty string.
27-
const model = openrouter(process.env.E2E_AGENT_MODEL || "openai/gpt-6-luna");
28+
const model = openrouter(process.env.E2E_AGENT_MODEL || "inclusionai/ling-3.0-flash-vl");
2829
const judge = openrouter(process.env.E2E_JUDGE_MODEL || "meta/muse-spark-1.3-contributor");
2930

3031
// `context` reaches every model call, judges included: what the app calls things.

0 commit comments

Comments
 (0)