You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 353f85f
Browse filesBrowse the repository at this point in the historyBrowse files
Copy file name to clipboardExpand all lines: .claude/skills/verify-debatelab/SKILL.md
+15-11Lines changed: 15 additions & 11 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,14 +1,14 @@
1
1
---
2
2
name: verify-debatelab
3
-
description: Launch and drive the real DebateLab web app (React client + local Supabase Auth/Postgres/Edge Functions, with either a fake model provider or real OpenRouter pinned to the cheap meta/muse-spark-1.3-contributor model plus real Exa research) to prove a user-facing feature works, with screenshots, ARIA snapshots, provider calls, and database rows as evidence. Use when asked to verify, demo, reproduce, or screenshot DebateLab behavior in a browser, or before calling a DebateLab UI/Edge Function change done.
3
+
description: Launch and drive the real DebateOwl web app (React client + local Supabase Auth/Postgres/Edge Functions, with either a fake model provider or real OpenRouter pinned to Claude Haiku 5.5 on the Anthropic BYOK key plus real Exa research) to prove a user-facing feature works, with screenshots, ARIA snapshots, provider calls, and database rows as evidence. Use when asked to verify, demo, reproduce, or screenshot DebateOwl behavior in a browser, or before calling a DebateOwl UI/Edge Function change done.
4
4
---
5
5
6
-
# Verify DebateLab
6
+
# Verify DebateOwl
7
7
8
-
DebateLab is a web app. The surface a user touches is the browser client at `/login` and `/app/*`, backed by Supabase Auth, Postgres/RLS, and Deno Edge Functions. This skill runs all of that locally and drives it with Playwright. It has two modes:
8
+
DebateOwl is a web app. The surface a user touches is the browser client at `/login` and `/app/*`, backed by Supabase Auth, Postgres/RLS, and Deno Edge Functions. This skill runs all of that locally and drives it with Playwright. It has two modes:
9
9
10
10
-**Fake (default, free).**`e2e/full-stack/fake-openrouter.mjs` answers every model call with canned text. Proves wiring, rows, and UI, not model behavior.
11
-
-**Live (`up --live`, costs money).** The same container becomes a recording proxy to real OpenRouter. It rewrites every chat call to `meta/muse-spark-1.3-contributor`(override with `VERIFY_LIVE_MODEL`), records OpenRouter's reply model as proof, and stops forwarding after `--max-calls` (default 400). Edge Functions also get a real Exa key, so web research and verified citations are real.
11
+
-**Live (`up --live`, costs money).** The same container becomes a recording proxy to real OpenRouter. It rewrites every chat call to `anthropic/claude-haiku-5.5` pinned to the Anthropic provider with fallbacks off, so it bills the Anthropic BYOK key and not OpenRouter credits (override with `VERIFY_LIVE_MODEL`, which must be an `anthropic/` model), records OpenRouter's reply model as proof, and stops forwarding after `--max-calls` (default 400). Edge Functions also get a real Exa key, so web research and verified citations are real.
12
12
13
13
Everything goes through one helper, run from the repository root (`git rev-parse --show-toplevel`):
14
14
@@ -42,7 +42,7 @@ STRIPE_TEST_SECRET_KEY=sk_test_... # optional; live keys are refused
42
42
STRIPE_TEST_PRICE_ID=price_... # required with the Stripe key
43
43
```
44
44
45
-
It prints nothing about the values. Ask Adam before the first live run: it spends money and sends prompts to a contributor model that may use them for training, so never put real user data in a live run. Costs: Muse Spark 1.3 contributor is $0.10 per million prompt tokens and $0.20 per million completion tokens, so a full `drive all` is cents of model spend. Exa bills per search and is usually the larger part. Jev decisions go to `/api/alpha/decisions` with Jev's own model and are not pinned. `doctor` shows `live calls used N/max`; once the cap is hit the proxy answers 429 and tests fail rather than spend more.
45
+
It prints nothing about the values. Ask Adam before the first live run: it spends money (Exa, Jev, and the Anthropic key), so never put real user data in a live run. Model calls bill the Anthropic BYOK key; Exa bills per search. Jev decisions go to `/api/alpha/decisions` with Jev's own model and are not pinned. `doctor` shows `live calls used N/max`; once the cap is hit the proxy answers 429 and tests fail rather than spend more.
46
46
47
47
## Launch
48
48
@@ -81,13 +81,15 @@ It also prints the commit the stack started at and whether there are uncommitted
81
81
- The Current Debate helpers: `startDebate(page, motion)` starts one from the header, `chooseForMotion(page, "create" | "once")` answers the motion dialog, and `debateControl(page)` is the visible header control on either viewport.
82
82
- The voice helpers: `stubSpeech(page)` installs a scripted microphone, recogniser and browser voice; `say(page, words)` speaks; `spokenByBrowser(page)` reads back what the browser voice said.
83
83
-`admin`: a service-role client for reading rows as a second view.
84
-
-`fake` / `providerCall(feature)`: what the Edge Function actually sent upstream. Use `fake.fail(feature, status)` to force a provider error.
85
-
-`aiResponse(page, feature, operation?)`: the parsed body of the next `ai` Edge Function response. Assert that its fields reach the screen, never on model wording, so a recipe passes in both modes. `LIVE` and `AI_TIMEOUT` (30 s fake, 240 s live) let a recipe branch where a check only makes sense live, and a fake-mode branch adds an `unverified` annotation instead of passing silently.
84
+
-`fake` / `providerCall(feature)`: what the Edge Function actually sent upstream. Use `fake.fail(feature, status)` to force a provider error and `fake.truncate(feature)` (fake mode only) to cut the next reply off at its length cap. Each call records the `maxTokens` it asked for.
85
+
-`aiResponse(page, feature, operation?)`: the parsed body of the next `ai` Edge Function response. Pages ask for live generation, so most `ai` responses are event streams; `aiResponse` and `readAIResponse(response)` read the final `result` event (or a plain JSON body) for you, so never call `response.json()` on an `ai` response. Assert that its fields reach the screen, never on model wording, so a recipe passes in both modes. `LIVE` and `AI_TIMEOUT` (30 s fake, 240 s live) let a recipe branch where a check only makes sense live, and a fake-mode branch adds an `unverified` annotation instead of passing silently.
86
+
-`followSystemTheme(page)`: new accounts start on the light theme. A recipe that compares light and dark calls this first; it picks System in Settings, so `page.emulateMedia({ colorScheme })` then drives the theme.
87
+
-`expectStreamed([...])`: every provider call for those features asked the model to stream, which proves the page showed the answer live.
86
88
-`expectProviderCalls([...])`: what reached the provider. Each test writes `provider-calls.json`, and in live mode the fixture fails the test if any chat call was answered by a model other than the pin.
87
89
-`evidence`: `step(line)`, `aria(name, locator?)`, `shot(name)` and `json(name, value)`.
88
90
- Automatic failure on any uncaught page error, horizontal overflow, or remote Supabase request.
89
91
90
-
Every result, source and practice round belongs to a Current Debate or is explicitly one-off. A tool given a motion when no debate exists opens `Start a debate with this motion?`; a motion that differs from the Current Debate's opens `This is a different motion`. Both open after the submit click and before the AI request, so create the `aiResponse()` promise, click, answer with `chooseForMotion`, then await. Or call `startDebate` first, after which tools prefill its motion and open no dialog. The Library's `Add Source` is disabled until a debate exists. See `features/current-debate.md`.
92
+
Every result, source and practice round belongs to a Current Debate or is explicitly one-off. A tool given a motion when no debate exists opens `Start a debate with this motion?`; a motion that differs from the Current Debate's opens `This is a different motion`. Both open after the submit click and before the AI request, so create the `aiResponse()` promise, click, answer with `chooseForMotion`, then await. Or call `startDebate` first, after which tools prefill its motion and open no dialog. The Library's `Add source` is disabled until a debate exists. See `features/current-debate.md`.
91
93
92
94
```bash
93
95
$H drive all # every recipe, both viewports
@@ -104,9 +106,9 @@ Fake mode also runs `e2e/full-stack/`. Live mode runs the recipes, `e2e/live/`,
104
106
105
107
To prove a feature without a recipe, copy `recipes/case-generation.spec.ts` to `recipes/<feature>.spec.ts` and use the selectors from the matching file in `features/`. Recipes are maintained files. Keep the ones worth keeping.
106
108
107
-
**Interactive.** Run `$H user` to mint a code. It prints `DEMO_CODE=...`, and `down` deletes that user. Then point a browser tool (Playwright MCP `browser_navigate` etc.) at `http://127.0.0.1:18081/login`, choose the `Demo code` tab, fill the `Demo code` textbox, and click `Sign in with demo code`. Save your own screenshots into `.verify/artifacts/<run-id>/`. A new account shows the onboarding tour on `/app`. Its first step offers `Start debate`; to skip the tour, click `Skip Tour` (`[data-test-id="button-skip"]`). Then start a debate from the header (`Start debate`, test id `current-debate-desktop` or `current-debate-mobile`) or answer the motion dialog the first tool opens: `Start debate` files the work in a new debate, `Not now, use it once` keeps it one-off. Interactive runs have no remote-Supabase guard. Doctor's local-mode check is what protects you.
109
+
**Interactive.** Run `$H user` to mint a code. It prints `DEMO_CODE=...`, and `down` deletes that user. Then point a browser tool (Playwright MCP `browser_navigate` etc.) at `http://127.0.0.1:18081/login`, choose the `Demo code` tab, fill the `Demo code` textbox, and click `Sign in with demo code`. Save your own screenshots into `.verify/artifacts/<run-id>/`. The one-time DebateOwl welcome never opens on this stack (`VITE_LOCAL_PREVIEW`) or under Playwright (`VITE_E2E`); open `/app?intro=show` to see it, and close it with `Got it` or Esc. Closing it continues straight into the tour's Start debate step when that account has not finished the tour. A new account shows the onboarding tour on `/app`. Its first step offers `Start debate`; to skip the tour, click `Skip tour` (`[data-test-id="button-skip"]`). Then start a debate from the header (`Start debate`, test id `current-debate-desktop` or `current-debate-mobile`) or answer the motion dialog the first tool opens: `Start debate` files the work in a new debate, `Not now, use it once` keeps it one-off. Interactive runs have no remote-Supabase guard. Doctor's local-mode check is what protects you.
108
110
109
-
Selectors: use accessible names and roles as the feature files list them (`getByRole("button", { name: "Generate Debate Case" })`). The exceptions use `data-testid`: Crossfire's controls, the Current Debate control, strip and dashboard, and the voice controls (`voice-mode`, `voice-mic`). Use `exact: true` for short names such as `Start` or `Save`, because the header's `Start debate` also matches `Start`. Never use coordinates.
111
+
Selectors: use accessible names and roles as the feature files list them (`getByRole("button", { name: "Generate debate case" })`). The exceptions use `data-testid`: Crossfire's controls, the Current Debate control, strip and dashboard, and the voice controls (`voice-mode`, `voice-mic`). Use `exact: true` for short names such as `Start` or `Save`, because the header's `Start debate` also matches `Start`. Never use coordinates.
110
112
111
113
## Agentic tests and exploration
112
114
@@ -122,7 +124,7 @@ $H e2e cache ls # recorded agent flo
122
124
123
125
-`e2e` runs `doctor` first and refuses a broken stack, like `drive`. It turns e2e telemetry off and prints `EVIDENCE <dir>`, which holds `summary.md`, `report.json`, and per-attempt screenshots, traces, and failure videos.
124
126
- The suite is tool journeys (tag `journey`) and a page sweep (tag `pages`) of every student and public page in English, Spanish and Hungarian (not `/app/admin` or `/share/:code`). Every test fails on an uncaught error, `console.error` (React warnings included), a Supabase 5xx, horizontal overflow, or a request to a hosted project: `agentic/support/capture-errors.js` records them in the page. A failure that names `errors the page logged` is a real app error to fix, not test noise.
125
-
- The testing agent has its own OpenRouter models: `inclusionai/ling-3.0-flash-vl` acts (test steps on a cache miss, and explorations) and `meta/muse-spark-1.3-contributor` judges. `E2E_AGENT_MODEL` and `E2E_JUDGE_MODEL` override them; the model is not part of the replay key. The key comes from `OPENROUTER_API_KEY`, `LIVE_OPENROUTER_API_KEY`, or the live keys file, and is needed in fake mode too. A recorded `agent.act` replays with no model call; every judgment and every exploration step is a call. Without a key, agent tests skip. An OpenRouter error such as `[Meta] The requested model was not found`is infrastructure: rerun with `--last-failed`. The Muse Spark judge cannot act.
127
+
- The testing agent has its own OpenRouter models: `anthropic/claude-haiku-5.5` acts (test steps on a cache miss, and explorations) and `anthropic/claude-sonnet-5.5` judges, both pinned to the Anthropic provider (BYOK) with fallbacks off. Haiku is not reliable enough to judge screens: it reads content below the fold as cut off. `E2E_AGENT_MODEL` and `E2E_JUDGE_MODEL` override them but must be `anthropic/` models; the model is not part of the replay key. The key comes from `OPENROUTER_API_KEY`, `LIVE_OPENROUTER_API_KEY`, or the live keys file, and is needed in fake mode too. A recorded `agent.act` replays with no model call; every judgment and every exploration step is a call. Without a key, agent tests skip. An OpenRouter error naming Anthropic (an outage, or the key out of credit) is infrastructure: rerun with `--last-failed`.
126
128
-`--session student` starts an exploration signed in as a fresh demo user, minted by `agentic/sessions.e2e.ts`. `--agent` picks a persona: `first-timer`, `coach`, `edge-input`, or `hungarian`. `--target mobile` explores the phone-sized viewport.
127
129
- Explorations report `issue` (functional) and `warning` (cosmetic) findings, and exit 1 when there is an issue. Treat a finding as a lead: the stack serves Vite dev, so a route that is still compiling can look dead, and fake mode answers every motion with free-transport text. Prove a finding with a repro test tagged `repro` before you report it as a bug.
128
130
- Several explorations can run at once (a bug bash), because each gets its own evidence directory. They share this one stack, so give each a different area.
@@ -172,6 +174,8 @@ It never touches `.verify/artifacts/` or `.verify/logs/`, and it is safe to run
172
174
## Gotchas
173
175
174
176
- Live runs are slow: sparring and crossfire take several minutes each because the model must reason. Use `-g` to run one test while iterating.
177
+
- Tailwind v4 is configured in `src/index.css` (`@theme`, `@custom-variant`, `@utility`); there is no `tailwind.config.ts`. Edits to `src/` and `src/index.css` apply on reload.
178
+
- The fake provider streams a structured answer in 24-character chunks 10 ms apart when the request asks to stream, so a Generate draft is on screen for about a second in fake mode; `fake.streamDelay(ms)` slows it for one test (the next reset restores it). Short replies (a Crossfire turn) finish in well under that, so prove those with `expectStreamed` rather than by catching the draft.
175
179
- The fake provider rejects prompts it doesn't recognize. If a prompt change makes a feature fail with a provider error, update `e2e/full-stack/fake-openrouter.mjs`. Don't mock around it.
176
180
- Once, the fake-provider container was SIGKILLed about 20 seconds after `up`, and the cause was never found. Doctor catches this, and `drive` refuses to run until you `down && up`.
177
181
- Demo-code users are Pro with a daily limit of 999. To test quota or tier paths, change `user_settings` through `admin` inside the recipe, as `e2e/full-stack/case-generation.spec.ts` does.
case"$LIVE_MODEL"in anthropic/*) ;; *) printf'FAIL VERIFY_LIVE_MODEL must be an anthropic/ model (Anthropic BYOK), not %s\n'"$LIVE_MODEL">&2;exit 1 ;; esac
say "info testing agent: OpenRouter actor ${E2E_AGENT_MODEL:-inclusionai/ling-3.0-flash-vl}, judge ${E2E_JUDGE_MODEL:-meta/muse-spark-1.3-contributor} (replayed steps make no model call)"
305
+
say "info testing agent: OpenRouter actor ${E2E_AGENT_MODEL:-anthropic/claude-haiku-5.5}, judge ${E2E_JUDGE_MODEL:-anthropic/claude-sonnet-5.5}, Anthropic BYOK only (replayed steps make no model call)"
301
306
else
302
307
say "info no OpenRouter key: tests with agent steps will skip"
0 commit comments