diff --git a/.claude/skills/verify-debatelab/SKILL.md b/.claude/skills/verify-debatelab/SKILL.md new file mode 100644 index 000000000..89dc753fe --- /dev/null +++ b/.claude/skills/verify-debatelab/SKILL.md @@ -0,0 +1,159 @@ +--- +name: verify-debatelab +description: Launch and drive the real DebateLab web app (React client + local Supabase Auth/Postgres/Edge Functions, with either a fake model provider or real OpenRouter pinned to the cheap meta/muse-spark-1.3-contributor model plus real Exa research) to prove a user-facing feature works, with screenshots, ARIA snapshots, provider calls, and database rows as evidence. Use when asked to verify, demo, reproduce, or screenshot DebateLab behavior in a browser, or before calling a DebateLab UI/Edge Function change done. +--- + +# Verify DebateLab + +DebateLab is a web app. The surface a user touches is the browser client at `/login` and `/app/*`, backed by Supabase Auth, Postgres/RLS, and Deno Edge Functions. This skill runs all of that locally and drives it with Playwright. It has two modes: + +- **Fake (default, free).** `e2e/full-stack/fake-openrouter.mjs` answers every model call with canned text. Proves wiring, rows, and UI, not model behavior. +- **Live (`up --live`, costs money).** The same container becomes a recording proxy to real OpenRouter. It rewrites every chat call to `meta/muse-spark-1.3-contributor` (override with `VERIFY_LIVE_MODEL`), records OpenRouter's reply model as proof, and stops forwarding after `--max-calls` (default 400). Edge Functions also get a real Exa key, so web research and verified citations are real. + +Everything goes through one helper, run from the repository root (`git rev-parse --show-toplevel`): + +```bash +H=.claude/skills/verify-debatelab/bin/verify.sh +$H up # fake mode; add --collab to also start the Workspaces collaboration service +$H up --live [--max-calls N] # real OpenRouter (pinned) + real Exa +$H doctor # read-only: is this instance ours, local, and healthy? +$H drive |e2e/> [--project desktop-chromium|mobile-chromium] [-g ] +$H user [label] # mint a demo-code login for interactive driving +$H down # stop everything `up` started; evidence stays +``` + +## Safety rules, read first + +- **Plain `npm run dev` talks to the PRODUCTION Supabase project** (`src/lib/supabase/env.ts` falls back to production). Never sign in or submit forms on a dev server you started yourself. Drive only the `up` instance. It sets `VITE_LOCAL_PREVIEW=true`, and `doctor` proves the client resolved to `http://127.0.0.1:54321`. +- The Docker container `debatelab` on `127.0.0.1:8080` is Adam's deployed app. Never drive it, restart it, or point tests at it. +- The local Supabase stack uses fixed ports (54321+) and shared container names, so only one worktree can run it at a time. `up` takes `~/.superset/debatelab-db.lock` (the lock `.superset/db-gate.sh` uses) and refuses to start if the lock is held, if any `supabase_*` container is running, or if port 18081 or 18090 is busy. Don't work around a refusal. Another worktree owns the stack. +- Recipes abort any request to `*.supabase.co` and fail the test if one is attempted. +- Never print values from `.verify/state/` (`drive.env`, `function.env`). They are local test keys, but treat them as secrets. + +## Live mode keys and cost + +`up --live` reads keys from the environment or from `~/.config/debatelab/live-verify.env` (override with `VERIFY_LIVE_KEYS_FILE`; keep it `chmod 600`): + +```bash +LIVE_OPENROUTER_API_KEY=... # required +LIVE_EXA_API_KEY=... # required +STRIPE_TEST_SECRET_KEY=sk_test_... # optional; live keys are refused +STRIPE_TEST_PRICE_ID=price_... # required with the Stripe key +``` + +It prints nothing about the values. Ask Adam before the first live run: it spends money and sends prompts to a contributor model that may use them for training, so never put real user data in a live run. Costs: Muse Spark 1.3 contributor is $0.10 per million prompt tokens and $0.20 per million completion tokens, so a full `drive all` is cents of model spend. Exa bills per search and is usually the larger part. Jev decisions go to `/api/alpha/decisions` with Jev's own model and are not pinned. `doctor` shows `live calls used N/max`; once the cap is hit the proxy answers 429 and tests fail rather than spend more. + +## Launch + +`$H up` takes 1–3 minutes on a warm Docker cache. It runs these steps in order: + +1. `supabase start`, which replays `supabase/migrations/`. +2. Writes the function env from `.superset/workspace-test.env`. +3. Starts the fake provider container `debatelab-fake-openrouter` (`127.0.0.1:18090`) on the stack network. +4. `supabase functions serve --no-verify-jwt`. +5. Starts Vite on `http://127.0.0.1:18081` in local-preview mode. + +It is ready when it prints `READY http://127.0.0.1:18081`. Port 18081 is fixed: the Edge Function CORS list (`supabase/functions/_shared/cors.ts`) allows it for the local full-stack gate, and it stays clear of the dev server (8080) and the credential-free browser run (18080), which the list also allows. + +Vite serves the working tree live, so uncommitted edits to `src/` apply on reload. Changes to `supabase/functions/` are picked up by `functions serve`. A new migration needs `down` then `up`. + +If `up` fails, it tears itself down and prints the log directory (`.verify/logs/<run-id>/`: `supabase.log`, `functions.log`, `vite.log`, `collaboration.log`). + +## Doctor + +Run `$H doctor` first whenever anything looks off. It exits 0 only when all of these hold: + +- The stack lock is held by this run's holder process. +- Port 18081 is served by this run's Vite process group. +- Vite's served `src/lib/supabase/env.ts` has `VITE_LOCAL_PREVIEW: "true"` and the local API URL. +- The Edge Functions process is alive, and `demo-code-login` answers CORS for the app origin. +- The fake provider answers. +- The Supabase containers for this worktree's project id are running. + +It also prints the commit the stack started at and whether there are uncommitted changes. `drive` runs doctor itself and refuses a failing stack. Recovery for any failed check is `$H down && $H up`. + +## Drive + +**Scripted (preferred for proof).** Recipes are Playwright specs in `.claude/skills/verify-debatelab/recipes/`. They import from `./fixtures`, which re-exports `e2e/full-stack/stack.ts` and `e2e/helpers/voice.ts` and adds the following: + +- A fresh demo-code user per test, signed in through the real Login page and `demo-code-login`, and deleted afterwards. Use the `signedIn` fixture for the page and `user` for `user.email` and `await user.userId()`. The user has seen the onboarding tour and has no Current Debate. +- The Current Debate helpers: `startDebate(page, motion)` starts one from the header, `chooseForMotion(page, "create" | "once")` answers the motion dialog, and `debateControl(page)` is the visible header control on either viewport. +- The voice helpers: `stubSpeech(page)` installs a scripted microphone, recogniser and browser voice; `say(page, words)` speaks; `spokenByBrowser(page)` reads back what the browser voice said. +- `admin`: a service-role client for reading rows as a second view. +- `fake` / `providerCall(feature)`: what the Edge Function actually sent upstream. Use `fake.fail(feature, status)` to force a provider error. +- `aiResponse(page, feature, operation?)`: the parsed body of the next `ai` Edge Function response. Assert that its fields reach the screen, never on model wording, so a recipe passes in both modes. `LIVE` and `AI_TIMEOUT` (30 s fake, 240 s live) let a recipe branch where a check only makes sense live, and a fake-mode branch adds an `unverified` annotation instead of passing silently. +- `expectProviderCalls([...])`: what reached the provider. Each test writes `provider-calls.json`, and in live mode the fixture fails the test if any chat call was answered by a model other than the pin. +- `evidence`: `step(line)`, `aria(name, locator?)`, `shot(name)` and `json(name, value)`. +- Automatic failure on any uncaught page error, horizontal overflow, or remote Supabase request. + +Every result, source and practice round belongs to a Current Debate or is explicitly one-off. A tool given a motion when no debate exists opens `Start a debate with this motion?`; a motion that differs from the Current Debate's opens `This is a different motion`. Both open after the submit click and before the AI request, so create the `aiResponse()` promise, click, answer with `chooseForMotion`, then await. Or call `startDebate` first, after which tools prefill its motion and open no dialog. The Library's `Add Source` is disabled until a debate exists. See `features/current-debate.md`. + +```bash +$H drive all # every recipe, both viewports +$H drive recipes/ai-tools --project desktop-chromium # one file, one viewport +$H drive e2e/full-stack/billing # a maintained full-stack spec +$H drive e2e/workspace-collaboration # starts its own collaboration process; do not use up --collab +$H drive e2e/live/ --project desktop-chromium # live mode only +``` + +Fake mode also runs `e2e/full-stack/`. Live mode runs the recipes, `e2e/live/`, and only the provider-free `e2e/full-stack/` tests (the config's `grepInvert` skips the ones asserting fake text). + +To prove a feature without a recipe, copy `recipes/case-generation.spec.ts` to `recipes/<feature>.spec.ts` and use the selectors from the matching file in `features/`. Recipes are maintained files. Keep the ones worth keeping. + +**Interactive.** Run `$H user` to mint a code. It prints `DEMO_CODE=...`, and `down` deletes that user. Then point a browser tool (Playwright MCP `browser_navigate` etc.) at `http://127.0.0.1:18081/login`, choose the `Demo code` tab, fill the `Demo code` textbox, and click `Sign in with demo code`. Save your own screenshots into `.verify/artifacts/<run-id>/`. A new account shows the onboarding tour on `/app`. Its first step offers `Start debate`; to skip the tour, click `Skip Tour` (`[data-test-id="button-skip"]`). Then start a debate from the header (`Start debate`, test id `current-debate-desktop` or `current-debate-mobile`) or answer the motion dialog the first tool opens: `Start debate` files the work in a new debate, `Not now, use it once` keeps it one-off. Interactive runs have no remote-Supabase guard. Doctor's local-mode check is what protects you. + +Selectors: use accessible names and roles as the feature files list them (`getByRole("button", { name: "Generate Debate Case" })`). The exceptions use `data-testid`: Crossfire's controls, the Current Debate control, strip and dashboard, and the voice controls (`voice-mode`, `voice-mic`). Use `exact: true` for short names such as `Start` or `Save`, because the header's `Start debate` also matches `Start`. Never use coordinates. + +## Evidence + +Each drive writes to `.verify/artifacts/<UTC-timestamp>-<recipe>/` and prints `EVIDENCE <dir>`: + +- `<project>/<test-slug>/`: `run.json` (commit, project, URL), `steps.log` (user actions in order), `*.aria.yml`, `*.png` and `*.json` (provider calls, DB rows). +- `playwright/`: the full trace (`npx playwright show-trace .../trace.zip`) and the final screenshot. +- `report.json`: the pass/fail result. + +A proof must meet these standards: + +- Drive the real user path: the Login page, visible buttons and the real Edge Functions. Never use store setters or direct function calls to produce the visible result. +- Capture the action and the resulting state. Record `step()` for each user action, and take an ARIA snapshot or screenshot before and after. +- Verify side effects as well as the screen: the `providerCall(...)` payload, and the rows in `ai_requests`, `cases`, `sources`, `sparring_sessions` and so on, read through `admin`. +- Mocks are allowed only at the model-provider boundary, which is the fake provider. Everything else is real. + +What each mode proves: + +- Fake mode does not prove model output, web research and citations (no Exa key, so Factoids drops every unverified source), or real Jev judgement. The affected tests add an `unverified` annotation. URL import (`import-source`) fetches the page directly in both modes, so it needs outbound internet but no key. +- Live mode proves those with one cheap model. It does not prove how the production models (per-feature routing in `supabase/functions/_shared/`) behave, and model text varies run to run. +- Neither mode proves Stripe Checkout or the billing portal UI, real Stripe cancellation on account deletion (the fake container answers the customer search), ElevenLabs voice audio (no key, so the browser voice reads replies), Google sign-in, Google Docs export, or anything about staging, production, or the deployed host. + +For those, see `docs/testing.md` (the live-provider gate and the production canary). + +## Cleanup + +`$H down` does the following: + +1. Deletes users minted by `user`. +2. Kills only the process groups it recorded (functions, Vite, collaboration). +3. Removes `debatelab-fake-openrouter`. +4. Runs `supabase stop --no-backup` (the local DB is wiped). +5. Kills the lock holder and releases the lock if it is still ours. +6. Deletes `.verify/state/`. + +It never touches `.verify/artifacts/` or `.verify/logs/`, and it is safe to run twice. Run it after every session, including failed ones. Never kill processes by name, and never run `supabase stop` for a stack you didn't start. After `down`, `docker ps` should show no `supabase_*` and no `debatelab-fake-openrouter` containers, and `~/.superset/debatelab-db.lock` should be gone. + +`.verify/` is gitignored. Delete old artifact directories by hand when they are no longer needed. + +## Feature map + +`features/README.md` indexes the maintained per-feature recipes, with entry points, selectors and end states. A proof that drives one entry point is incomplete when the map lists others. + +## Gotchas + +- Live runs are slow: sparring and crossfire take several minutes each because the model must reason. Use `-g` to run one test while iterating. +- The fake provider rejects prompts it doesn't recognize. If a prompt change makes a feature fail with a provider error, update `e2e/full-stack/fake-openrouter.mjs`. Don't mock around it. +- Once, the fake-provider container was SIGKILLed about 20 seconds after `up`, and the cause was never found. Doctor catches this, and `drive` refuses to run until you `down && up`. +- Demo-code users are Pro with a daily limit of 999. To test quota or tier paths, change `user_settings` through `admin` inside the recipe, as `e2e/full-stack/case-generation.spec.ts` does. +- A `waitForResponse` timeout right after a submit click usually means a motion dialog is open. See the Current Debate note under Drive. +- The client files a result in its debate after it renders. A `page.goto` straight after the `ai` response can drop it; wait for the result on screen (and its row, if the next step reloads) first. +- In a voice round, wait for the opponent's reply to finish (`spokenByBrowser`) before clicking `voice-mic` again; speaking over it abandons the voice request. `recipes/voice-rounds.spec.ts` shows the wait. +- Account deletion is irreversible. `recipes/account-data.spec.ts` deletes only its own `@fullstack.test` demo user and checks that before clicking. +- `.superset/db-gate.sh` and this skill share the lock. While your stack is up, the gate in any worktree waits (for up to 600 s) and then fails. Don't leave the stack up. diff --git a/.claude/skills/verify-debatelab/bin/demo-user.mjs b/.claude/skills/verify-debatelab/bin/demo-user.mjs new file mode 100644 index 000000000..7fe8ece17 --- /dev/null +++ b/.claude/skills/verify-debatelab/bin/demo-user.mjs @@ -0,0 +1,44 @@ +// Mints or deletes disposable demo-code users on the LOCAL verification stack. +// Mirrors createDemoUser/deleteDemoUser in e2e/full-stack/stack.ts. +// node demo-user.mjs create <label> <users.tsv> +// node demo-user.mjs delete-all <users.tsv> +import { createClient } from "@supabase/supabase-js"; +import { createHash, randomUUID } from "node:crypto"; +import { appendFile, readFile } from "node:fs/promises"; + +const url = process.env.LOCAL_SUPABASE_URL; +const serviceKey = process.env.LOCAL_SUPABASE_SERVICE_ROLE_KEY; +const pepper = process.env.DEMO_CODE_PEPPER; +if (!url || !serviceKey || !pepper) throw new Error("Local stack environment is incomplete; run verify.sh up"); +if (!/^http:\/\/(127\.0\.0\.1|localhost):/.test(url)) throw new Error("Refusing to mint users outside the local stack"); +const admin = createClient(url, serviceKey, { auth: { persistSession: false } }); + +const [command, ...args] = process.argv.slice(2); + +if (command === "create") { + const [label = "verify", usersFile] = args; + const alphabet = "ABCDEFGHJKLMNPQRSTUVWXYZ23456789"; + const part = () => Array.from({ length: 4 }, () => alphabet[Math.floor(Math.random() * alphabet.length)]).join(""); + const code = `DEMO-${part()}-${part()}`; + const email = `${label}-${randomUUID()}@fullstack.test`; + const codeHash = createHash("sha256").update(`${pepper}:${code}`).digest("hex"); + const { data, error } = await admin.from("demo_access_codes") + .insert({ code_hash: codeHash, label, email, max_uses: 10 }) + .select("id") + .single(); + if (error) throw error; + await appendFile(usersFile, `${data.id}\t${email}\n`); + console.log(`DEMO_CODE=${code}\nEMAIL=${email}\nSign in at http://127.0.0.1:18081/login -> "Demo code" tab.`); +} else if (command === "delete-all") { + const lines = (await readFile(args[0], "utf8")).split("\n").filter(Boolean); + const { data } = await admin.auth.admin.listUsers({ perPage: 1000 }); + for (const line of lines) { + const [codeId, email] = line.split("\t"); + const id = data?.users.find((user) => user.email === email)?.id; + if (id) await admin.auth.admin.deleteUser(id); + await admin.from("demo_access_codes").delete().eq("id", codeId); + console.log(`deleted ${email}`); + } +} else { + throw new Error(`Unknown command ${command}`); +} diff --git a/.claude/skills/verify-debatelab/bin/verify.sh b/.claude/skills/verify-debatelab/bin/verify.sh new file mode 100755 index 000000000..27fd8cb9b --- /dev/null +++ b/.claude/skills/verify-debatelab/bin/verify.sh @@ -0,0 +1,310 @@ +#!/usr/bin/env bash +# Verification harness for DebateLab: a long-lived copy of .superset/db-gate.sh +# without the test runs. It starts local Supabase, every Edge Function, the fake +# OpenRouter provider, and a Vite client pinned to the local stack, then holds +# the shared stack lock until `down`. +# +# verify.sh up [--live [--max-calls N]] [--collab] +# start everything; prints READY <url>. --live sends +# every model call to real OpenRouter pinned to +# $LIVE_MODEL, and research to real Exa. +# verify.sh doctor read-only health and isolation check +# verify.sh user [label] mint a disposable local demo-code user +# verify.sh drive <filter> [playwright args...] +# run recipes (`all`, `recipes/<name>`) or a maintained +# suite (`e2e/live/`, `e2e/workspace-collaboration`) +# and keep the evidence +# verify.sh down stop what `up` started; evidence is kept +set -euo pipefail + +SKILL_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +REPO="$(git -C "$SKILL_DIR" rev-parse --show-toplevel)" +cd "$REPO" +source "$REPO/.superset/lib/lock.sh" + +DB_LOCK="$HOME/.superset/debatelab-db.lock" +FAKE_PROVIDER=debatelab-fake-openrouter +APP_PORT=18081 # the only local client origin the Edge Function CORS list and invite fixture allow +FAKE_PORT=18090 +COLLAB_PORT=12345 +APP_ORIGIN="http://127.0.0.1:$APP_PORT" +VERIFY_ROOT="$REPO/.verify" +STATE_DIR="$VERIFY_ROOT/state" # secrets and pids; removed by down +STATE="$STATE_DIR/stack.env" +USERS="$STATE_DIR/users.tsv" +ARTIFACTS="$VERIFY_ROOT/artifacts" # evidence; never removed by this script +PROJECT_ID="$(basename "$REPO")" # supabase/config.toml has no project_id +# Adam's standing choice for live verification: the cheap contributor tier. +LIVE_MODEL="${VERIFY_LIVE_MODEL:-meta/muse-spark-1.3-contributor}" +LIVE_KEYS_FILE="${VERIFY_LIVE_KEYS_FILE:-$HOME/.config/debatelab/live-verify.env}" + +say() { printf '%s\n' "$*"; } +fail() { printf 'FAIL %s\n' "$*" >&2; exit 1; } +record() { printf '%s=%q\n' "$1" "$2" >>"$STATE"; } +load_state() { [ -f "$STATE" ] || fail "no running verification stack (state file missing); run: verify.sh up"; source "$STATE"; } +alive() { [ -n "${1:-}" ] && kill -0 "$1" 2>/dev/null; } +port_busy() { ss -ltnH "sport = :$1" 2>/dev/null | grep -q .; } +wait_url() { + local url="$1" tries="${2:-300}" + shift; [ "$#" -gt 0 ] && shift + for _ in $(seq 1 "$tries"); do + curl --fail --silent --output /dev/null "$@" "$url" && return 0 + sleep 0.2 + done + curl --fail --silent --output /dev/null "$@" "$url" +} + +cmd_up() { + local collab=false live=false max_calls=400 + while [ "$#" -gt 0 ]; do + case "$1" in + --collab) collab=true ;; + --live) live=true ;; + --max-calls) max_calls="${2:?--max-calls needs a number}"; shift ;; + *) fail "unknown up option $1" ;; + esac + shift + done + [ ! -f "$STATE" ] || fail "a verification stack is already recorded in $STATE; run doctor, or down first" + + # Live keys come from the environment or a 0600 key file. They are read, never printed. + local or_key="" exa_key="" stripe_key="" stripe_price="" + if [ "$live" = true ]; then + or_key="${LIVE_OPENROUTER_API_KEY:-}"; exa_key="${LIVE_EXA_API_KEY:-}" + stripe_key="${STRIPE_TEST_SECRET_KEY:-}"; stripe_price="${STRIPE_TEST_PRICE_ID:-}" + if [ -f "$LIVE_KEYS_FILE" ]; then + [ "$(stat -c %a "$LIVE_KEYS_FILE")" = 600 ] || fail "$LIVE_KEYS_FILE must be mode 600" + key_from_file() { sed -n "s/^$1=//p" "$LIVE_KEYS_FILE" | tail -1; } + [ -n "$or_key" ] || or_key="$(key_from_file LIVE_OPENROUTER_API_KEY)" + [ -n "$exa_key" ] || exa_key="$(key_from_file LIVE_EXA_API_KEY)" + [ -n "$stripe_key" ] || stripe_key="$(key_from_file STRIPE_TEST_SECRET_KEY)" + [ -n "$stripe_price" ] || stripe_price="$(key_from_file STRIPE_TEST_PRICE_ID)" + fi + [ -n "$or_key" ] && [ -n "$exa_key" ] || fail "--live needs LIVE_OPENROUTER_API_KEY and LIVE_EXA_API_KEY (environment or $LIVE_KEYS_FILE)" + if [ -n "$stripe_key" ]; then + case "$stripe_key" in sk_test_*|rk_test_*) ;; *) fail "refusing a Stripe key that is not test mode";; esac + [ -n "$stripe_price" ] || fail "STRIPE_TEST_SECRET_KEY needs STRIPE_TEST_PRICE_ID" + fi + fi + + # The stack uses the CLI's default ports and shared container names, so only + # one worktree may run it. Refuse instead of waiting or double-driving. + lock_acquire "$DB_LOCK" 0 || fail "the local Supabase stack lock is held by pid $(cat "$DB_LOCK/pid" 2>/dev/null || echo '?'); another worktree owns it" + if docker ps --format '{{.Names}}' | grep -q '^supabase_'; then + lock_release "$DB_LOCK"; fail "Supabase containers are already running (not started by this run)" + fi + for port in 54321 "$APP_PORT" "$FAKE_PORT"; do + if port_busy "$port"; then lock_release "$DB_LOCK"; fail "port $port is already in use"; fi + done + + mkdir -p "$STATE_DIR" "$ARTIFACTS" + chmod 700 "$STATE_DIR" + local run_id; run_id="$(date -u +%Y%m%dT%H%M%SZ)" + local log_dir="$VERIFY_ROOT/logs/$run_id" + mkdir -p "$log_dir" + : >"$STATE"; : >"$USERS" + record PROJECT_ID "$PROJECT_ID" + record LOG_DIR "$log_dir" + record HEAD "$(git rev-parse HEAD)" + record MODE "$([ "$live" = true ] && echo live || echo fake)" + record LIVE_MODEL "$([ "$live" = true ] && echo "$LIVE_MODEL")" + record STRIPE_MODE "$([ -n "$stripe_key" ] && echo test || echo placeholder)" + + # lock_acquire wrote this shell's pid, which dies when `up` returns and would + # let another worktree reclaim the lock. Hand ownership to a holder we kill in down. + setsid sleep infinity >/dev/null 2>&1 & + local holder=$! + echo "$holder" >"$DB_LOCK/pid" + record HOLDER_PID "$holder" + + trap 'say "up failed; tearing down (logs: $log_dir)"; cmd_down || true' ERR + + say "Starting local Supabase (logs: $log_dir)..." + record STACK_STARTED true + supabase start >>"$log_dir/supabase.log" 2>&1 + + local status_json; status_json="$(supabase status -o json 2>/dev/null)" + sv() { node -e 'const v=JSON.parse(process.argv[1])[process.argv[2]]; if (!v) process.exit(1); process.stdout.write(String(v))' "$status_json" "$1"; } + local api_url anon_key service_key db_url mailer_url mailer_host + api_url="$(sv API_URL)"; anon_key="$(sv ANON_KEY)"; service_key="$(sv SERVICE_ROLE_KEY)" + db_url="$(sv DB_URL)"; mailer_url="$(sv MAILPIT_URL)" + mailer_host="$(docker ps --filter "name=supabase_inbucket_$PROJECT_ID" --format '{{.Names}}')" + [ -n "$mailer_host" ] || { say "Local Mailpit container is unavailable"; false; } + + LOCAL_SERVICE_ROLE_KEY="$service_key" LOCAL_MAILER_HOST="$mailer_host" \ + node scripts/workspace-test-env.mjs .superset/workspace-test.env "$STATE_DIR/function.env" + if [ "$live" = true ]; then + # printf is a builtin, so keys never reach a process list or log. + grep -vE '^(OPENROUTER_API_KEY|EXA_API_KEY)=' "$STATE_DIR/function.env" >"$STATE_DIR/function.env.next" + printf 'OPENROUTER_API_KEY=%s\nEXA_API_KEY=%s\n' "$or_key" "$exa_key" >>"$STATE_DIR/function.env.next" + if [ -n "$stripe_key" ]; then + grep -vE '^(STRIPE_SECRET_KEY|STRIPE_PRICE_ID)=' "$STATE_DIR/function.env.next" >"$STATE_DIR/function.env.tmp" + printf 'STRIPE_SECRET_KEY=%s\nSTRIPE_PRICE_ID=%s\n' "$stripe_key" "$stripe_price" >>"$STATE_DIR/function.env.tmp" + mv -- "$STATE_DIR/function.env.tmp" "$STATE_DIR/function.env.next" + fi + mv -- "$STATE_DIR/function.env.next" "$STATE_DIR/function.env" + chmod 600 "$STATE_DIR/function.env" + fi + # Values the recipes and `user` need. Local, test-only, but never echo them. + ( + umask 077 + { + printf 'LOCAL_SUPABASE_URL=%s\n' "$api_url" + printf 'LOCAL_SUPABASE_ANON_KEY=%s\n' "$anon_key" + printf 'LOCAL_SUPABASE_SERVICE_ROLE_KEY=%s\n' "$service_key" + printf 'FAKE_OPENROUTER_URL=%s\n' "http://127.0.0.1:$FAKE_PORT" + printf 'WORKSPACE_TEST_MAILER_URL=%s\n' "$mailer_url" + printf 'DEMO_CODE_PEPPER=%s\n' "$(sed -n 's/^DEMO_CODE_PEPPER=//p' .superset/workspace-test.env)" + printf 'PLAYWRIGHT_BASE_URL=%s\n' "$APP_ORIGIN" + printf 'VERIFY_MODE=%s\n' "$([ "$live" = true ] && echo live || echo fake)" + printf 'LIVE_MODEL=%s\n' "$([ "$live" = true ] && echo "$LIVE_MODEL")" + printf 'VERIFY_STRIPE=%s\n' "$([ -n "$stripe_key" ] && echo test || echo placeholder)" + [ -z "$stripe_key" ] || printf 'STRIPE_TEST_SECRET_KEY=%s\n' "$stripe_key" + # e2e/workspace-collaboration.spec.ts starts its own collaboration process with these. + printf 'COLLABORATION_DATABASE_URL=%s\n' "$db_url" + printf 'COLLABORATION_PORT=%s\n' "$COLLAB_PORT" + printf 'COLLABORATION_ALLOWED_ORIGINS=%s\n' "$APP_ORIGIN" + printf 'SUPABASE_URL=%s\n' "$api_url" + printf 'SUPABASE_ANON_KEY=%s\n' "$anon_key" + } >"$STATE_DIR/drive.env" + ) + record API_URL "$api_url" + + # Edge Functions reach the fake provider by container name on the stack network. + docker rm -f "$FAKE_PROVIDER" >/dev/null 2>&1 || true + local provider_mode=() + [ "$live" = true ] && provider_mode=(-e UPSTREAM=https://openrouter.ai/api -e "LIVE_MODEL=$LIVE_MODEL" -e "MAX_UPSTREAM_CALLS=$max_calls") + docker run -d --rm --name "$FAKE_PROVIDER" --network "supabase_network_$PROJECT_ID" \ + "${provider_mode[@]}" -p "127.0.0.1:$FAKE_PORT:8787" -v "$REPO/e2e/full-stack/fake-openrouter.mjs:/fake.mjs:ro" \ + node:22-alpine node /fake.mjs >/dev/null + record FAKE_STARTED true + + setsid supabase functions serve --no-verify-jwt --env-file "$STATE_DIR/function.env" \ + >"$log_dir/functions.log" 2>&1 & + record FUNCTIONS_PID "$!" + + VITE_LOCAL_PREVIEW=true \ + VITE_LOCAL_SUPABASE_URL="$api_url" \ + VITE_LOCAL_SUPABASE_ANON_KEY="$anon_key" \ + VITE_COLLABORATION_URL="ws://127.0.0.1:$COLLAB_PORT" \ + setsid npm run dev -- --host 127.0.0.1 --port "$APP_PORT" --strictPort >"$log_dir/vite.log" 2>&1 & + record VITE_PID "$!" + + if [ "$collab" = true ]; then + COLLABORATION_DATABASE_URL="$db_url" \ + COLLABORATION_PORT="$COLLAB_PORT" \ + COLLABORATION_ALLOWED_ORIGINS="$APP_ORIGIN" \ + SUPABASE_URL="$api_url" \ + SUPABASE_ANON_KEY="$anon_key" \ + setsid npm run start:collaboration >"$log_dir/collaboration.log" 2>&1 & + record COLLAB_PID "$!" + fi + + wait_url "$mailer_url/api/v1/messages" + wait_url "$APP_ORIGIN" + wait_url "http://127.0.0.1:$FAKE_PORT/__requests" + for fn in workspace-invites ai ai-stream demo-code-login admin announcements; do + wait_url "$api_url/functions/v1/$fn" 300 --request OPTIONS --header "Origin: $APP_ORIGIN" + done + if [ "$collab" = true ]; then + for _ in $(seq 1 150); do grep -q "Collaboration service listening" "$log_dir/collaboration.log" && break; sleep 0.2; done + grep -q "Collaboration service listening" "$log_dir/collaboration.log" + fi + trap - ERR + say "READY $APP_ORIGIN mode=$([ "$live" = true ] && echo "live model=$LIVE_MODEL max-calls=$max_calls" || echo fake) (run: $0 doctor)" +} + +cmd_doctor() { + load_state + local bad=0 + check() { if eval "$2" >/dev/null 2>&1; then say "ok $1"; else say "FAIL $1"; bad=1; fi; } + check "stack lock held by our holder pid $HOLDER_PID" "[ \"\$(cat '$DB_LOCK/pid')\" = '$HOLDER_PID' ] && kill -0 $HOLDER_PID" + check "Vite process group $VITE_PID alive" "kill -0 $VITE_PID" + local listener_pid + listener_pid="$(ss -ltnpH "sport = :$APP_PORT" 2>/dev/null | grep -o 'pid=[0-9]*' | head -1 | cut -d= -f2)" + check "port $APP_PORT is served by our Vite (pgid $VITE_PID)" "[ -n '$listener_pid' ] && [ \"\$(ps -o pgid= -p '$listener_pid' | tr -d ' ')\" = '$VITE_PID' ]" + # The browser client must resolve the local stack, never production or staging. + local env_module; env_module="$(curl -s "$APP_ORIGIN/src/lib/supabase/env.ts" || true)" + check "client is in local-preview mode" "grep -q '\"VITE_LOCAL_PREVIEW\": \"true\"' <<<\"\$env_module\"" + check "client points at $API_URL" "grep -q '\"VITE_LOCAL_SUPABASE_URL\": \"$API_URL\"' <<<\"\$env_module\"" + check "Edge Functions process group $FUNCTIONS_PID alive" "kill -0 $FUNCTIONS_PID" + check "demo-code-login answers CORS for $APP_ORIGIN" "curl --fail -s -o /dev/null -X OPTIONS -H 'Origin: $APP_ORIGIN' '$API_URL/functions/v1/demo-code-login'" + check "fake OpenRouter answers on $FAKE_PORT" "curl --fail -s -o /dev/null http://127.0.0.1:$FAKE_PORT/__requests" + check "Supabase containers for project $PROJECT_ID running" "docker ps --format '{{.Names}}' | grep -q '^supabase_db_$PROJECT_ID\$'" + local stats; stats="$(curl -s "http://127.0.0.1:$FAKE_PORT/__stats" || true)" + if [ "$MODE" = live ]; then + local proxy_env; proxy_env="$(docker inspect -f '{{range .Config.Env}}{{println .}}{{end}}' "$FAKE_PROVIDER" 2>/dev/null || true)" + check "provider proxy forwards to real OpenRouter" "grep -qx 'UPSTREAM=https://openrouter.ai/api' <<<\"\$proxy_env\"" + check "every chat call is pinned to $LIVE_MODEL" "grep -qx 'LIVE_MODEL=$LIVE_MODEL' <<<\"\$proxy_env\"" + check "Edge Functions hold a real OpenRouter key (not the placeholder)" "! grep -qxF \"\$(grep '^OPENROUTER_API_KEY=' .superset/workspace-test.env)\" '$STATE_DIR/function.env'" + check "Edge Functions hold an Exa key" "grep -q '^EXA_API_KEY=.' '$STATE_DIR/function.env'" + say "info live calls used: $(node -e 'const s=JSON.parse(process.argv[1]); process.stdout.write(`${s.upstreamCalls}/${s.maxUpstreamCalls}`)' "$stats" 2>/dev/null || echo '?')" + else + check "provider is the deterministic fake (no network, no spend)" "grep -q '\"mode\":\"fake\"' <<<\"\$stats\"" + fi + say "info stripe: $STRIPE_MODE" + if [ -n "${COLLAB_PID:-}" ]; then check "collaboration service $COLLAB_PID alive" "kill -0 $COLLAB_PID"; fi + say "info started at commit $HEAD; worktree now at $(git rev-parse HEAD)$(git diff --quiet || echo ' (+ uncommitted changes; Vite serves them live)')" + say "info logs in $LOG_DIR" + return "$bad" +} + +cmd_user() { + load_state + set -a; source "$STATE_DIR/drive.env"; set +a + node "$SKILL_DIR/bin/demo-user.mjs" create "${1:-verify}" "$USERS" +} + +cmd_drive() { + load_state + local recipe="${1:?usage: verify.sh drive <all|recipes/<name>|e2e/<path>> [playwright args]}"; shift + cmd_doctor >/dev/null || { cmd_doctor || true; fail "doctor failed; not driving a broken stack (fix it, or down + up)"; } + # Filters are path substrings: `recipes/ai-tools`, `e2e/live/`, `all` (every recipe). + local filter="${recipe%.spec.ts}" + [ "$recipe" = all ] && filter="verify-debatelab/recipes/" + local name; name="$(basename "$filter")" + local run_id; run_id="$(date -u +%Y%m%dT%H%M%SZ)-$name" + set -a; source "$STATE_DIR/drive.env"; set +a + export VERIFY_ARTIFACT_DIR="$ARTIFACTS/$run_id" VERIFY_HEAD="$(git rev-parse HEAD)" + mkdir -p "$VERIFY_ARTIFACT_DIR" + local status=0 + npx playwright test --config "$SKILL_DIR/playwright.verify.config.ts" "$filter" "$@" || status=$? + say "EVIDENCE $VERIFY_ARTIFACT_DIR (exit $status)" + return "$status" +} + +cmd_down() { + if [ ! -f "$STATE" ]; then say "nothing recorded in $STATE; nothing to stop"; return 0; fi + source "$STATE" + if [ -s "$USERS" ] && [ -f "$STATE_DIR/drive.env" ]; then + ( set -a; source "$STATE_DIR/drive.env"; set +a; node "$SKILL_DIR/bin/demo-user.mjs" delete-all "$USERS" ) || say "warn: could not delete minted demo users (stack is about to be wiped anyway)" + fi + local pid + for pid in "${COLLAB_PID:-}" "${VITE_PID:-}" "${FUNCTIONS_PID:-}"; do + [ -n "$pid" ] && kill -- "-$pid" 2>/dev/null || true + done + for pid in "${COLLAB_PID:-}" "${VITE_PID:-}" "${FUNCTIONS_PID:-}"; do + [ -n "$pid" ] || continue + for _ in $(seq 1 50); do alive "$pid" || break; sleep 0.1; done + alive "$pid" && kill -9 -- "-$pid" 2>/dev/null || true + done + [ "${FAKE_STARTED:-}" = true ] && docker rm -f "$FAKE_PROVIDER" >/dev/null 2>&1 || true + if [ "${STACK_STARTED:-}" = true ]; then + supabase stop --no-backup >>"${LOG_DIR:-/dev/null}/supabase.log" 2>&1 || say "warn: supabase stop failed; see ${LOG_DIR:-}/supabase.log" + fi + if [ -n "${HOLDER_PID:-}" ]; then + if [ "$(cat "$DB_LOCK/pid" 2>/dev/null)" = "$HOLDER_PID" ]; then lock_release "$DB_LOCK"; fi + kill "$HOLDER_PID" 2>/dev/null || true + fi + rm -rf -- "$STATE_DIR" + say "DOWN (evidence kept in $ARTIFACTS, logs in ${LOG_DIR:-$VERIFY_ROOT/logs})" +} + +case "${1:-}" in + up) shift; cmd_up "$@" ;; + doctor) cmd_doctor ;; + user) shift; cmd_user "$@" ;; + drive) shift; cmd_drive "$@" ;; + down) cmd_down ;; + *) sed -n '2,/^set -euo/p' "$0" | sed '$d'; exit 2 ;; +esac diff --git a/.claude/skills/verify-debatelab/features/README.md b/.claude/skills/verify-debatelab/features/README.md new file mode 100644 index 000000000..71491a9e5 --- /dev/null +++ b/.claude/skills/verify-debatelab/features/README.md @@ -0,0 +1,54 @@ +# DebateLab verification map + +This directory is the maintained source for verifying DebateLab's user-facing behavior. Read this index before driving the app, then use the matching feature file as the recipe. The skill's `SKILL.md` covers launch, doctor, evidence, and cleanup. + +## Baseline preconditions + +- `bin/verify.sh up` has printed `READY http://127.0.0.1:18081`, and `bin/verify.sh doctor` exits 0. +- The browser client is the `up` instance at `http://127.0.0.1:18081`. Never use the `debatelab` container on `:8080` or a plain `npm run dev`, because both reach production. +- Each scripted test signs in with its own fresh demo-code user (a Pro account with a daily limit of 999) and deletes it afterwards. The fixture marks the onboarding tour as seen, and the user starts with no Current Debate. +- Most tools work inside a Current Debate (see [current-debate](./current-debate.md)). Start one with `startDebate(page, motion)` first, or answer the motion dialog that opens after submit with `chooseForMotion(page, "create" | "once")`. The Library's `Add Source` stays disabled until a debate exists. +- In fake mode, model answers come from the fake provider and contain the marker `FAKE-PROVIDER`. In live mode (`up --live`), every chat call goes to real OpenRouter pinned to `meta/muse-spark-1.3-contributor`, research uses real Exa, and answers are real model text. +- The feature files quote fake-mode text such as `FAKE-PROVIDER case thesis`. Recipes never assert it: they read the `ai` response body with `aiResponse()` and check that its fields are on screen, so the same recipe passes in both modes. + +## Driving conventions + +- Scripted runs go in `recipes/<feature>.spec.ts`, import from `./fixtures`, and run with `bin/verify.sh drive recipes/<feature>`. +- Interactive runs use `bin/verify.sh user` to get a code, then sign in on `/login`. +- Prefer roles and accessible names. The exceptions use `data-testid`: Crossfire's controls, the header's Current Debate control (`debateControl(page)`), the Current Debate strip and dashboard, and the voice controls (`voice-mode`, `voice-mic`). +- Wait for `aiResponse()` or a named element, never a fixed sleep. Use `AI_TIMEOUT` (30 s fake, 240 s live) for anything behind a model call. +- Run both `desktop-chromium` and `mobile-chromium` when the change touches layout. + +## Proof and skip reporting + +- UI proof is an ARIA snapshot plus a screenshot before and after the action, with `steps.log` naming each action. +- Side-effect proof is `expectProviderCalls([...])` (what the Edge Function sent upstream) plus the rows read back through `admin`. Every test writes `provider-calls.json`; in live mode the fixture fails the test if any chat call was answered by a model other than the pin. +- Record the feature ID and entry point with every artifact. +- Tests that need live mode record an `unverified` annotation in fake mode instead of passing silently. Report those as unverified. +- Never verifiable here: Stripe Checkout and the billing portal, Google sign-in and Google Docs export, staging, production, and the deployed host. Jev decisions are real only in live mode and use Jev's own model, not the pin. +- Do not report a skipped entry point as verified through a different path. + +## Feature entry contract + +Each feature file starts with an H1 title and one paragraph describing the user-visible behavior. It then uses exactly four H2 sections, in this order: + +- `Sub-features` +- `How to get to it (user POV)` +- `Driving it with verify.sh + Playwright` +- `Gotchas` + +## Features + +- [Demo-code sign-in](./auth-demo-login.md): sign-in, reload, rejected code, sign-out (`e2e/full-stack/auth.spec.ts`), and email one-time-code sign-in through Mailpit (`recipes/auth.spec.ts`). +- [Case generation](./case-generation.md): generate, web research, save and reopen from History on a fresh device, PDF and DOCX export, provider failure and retry, quota block (`recipes/case-generation.spec.ts`). +- [Library sources](./library-sources.md): add, pause, delete, grounding, and URL import (`recipes/library.spec.ts`). +- [AI tools](./ai-tools.md): Attack, Brainstorm, Factoids, and Ask streamed, cited, and with web search (`recipes/ai-tools.spec.ts`). +- [Current Debate](./current-debate.md): start, the first-debate offer and the different-motion choice, sources and results scoped to a debate, switching, and the first onboarding step (`recipes/current-debate.spec.ts`); debate context review and pins (`e2e/full-stack/debate-context.spec.ts`). +- [Practice](./practice.md): a full Lincoln-Douglas sparring round with verdict, and Crossfire in both directions with debrief, each filed in its debate (`recipes/practice.spec.ts`). +- [Voice rounds](./voice-rounds.md): spoken Sparring and Crossfire on the browser voice, with no ElevenLabs key (`recipes/voice-rounds.spec.ts`). +- [Account data](./account-data.md): Download my data and Delete account, each proven against a bystander (`recipes/account-data.spec.ts`); Delete All My Data scoped against a bystander (`e2e/full-stack/account-data.spec.ts`). +- Account: output language reaching the model from a new device, flowchart node delete, save and load, and the speech timer (`recipes/account.spec.ts`). +- Billing: expired account blocked, promo code restores access (`recipes/billing.spec.ts`); signed, forged, and replayed Stripe webhooks (`e2e/full-stack/billing.spec.ts`). +- Admin announcements and the non-admin block (`e2e/full-stack/admin.spec.ts`), public share links (`e2e/full-stack/public.spec.ts`), and Workspaces with live collaboration and invites (`e2e/workspace-collaboration.spec.ts`). + +Not yet mapped: theme and UI language, the public landing and SEO pages, and share-link creation (the app has no client caller for the `share` function). diff --git a/.claude/skills/verify-debatelab/features/account-data.md b/.claude/skills/verify-debatelab/features/account-data.md new file mode 100644 index 000000000..c58a3e1ce --- /dev/null +++ b/.claude/skills/verify-debatelab/features/account-data.md @@ -0,0 +1,30 @@ +# Account data: export and deletion + +From Settings, a signed-in user can download everything saved to the account as one JSON file, or delete the account permanently. Both go through the `account` Edge Function, which acts only on the caller's own session and ignores any user id in the request. Deletion first checks Stripe for the user's customers and cancels any subscription, then removes every row that references the user. `Delete All My Data` is the older, narrower erase: it keeps the account and removes its saved work. + +## Sub-features + +- `acct-export`: `Download my data` saves `debatelab-data-YYYY-MM-DD.json` with `format: "debatelab-account-export"`, the caller's `account` (`id`, `email`), and `data` (debates, sources, and saved results). Another account's rows never appear, even when the request names that account's id. +- `acct-delete`: `Delete account` opens `Delete your account?`. Its confirm button stays disabled until the field labelled `Type your email to confirm` holds the account email. After deletion, the toast reads `Account deleted` and the app returns to `/`. The auth user and every row mentioning its id, in the `public` and `private` schemas, are gone. Nobody else's rows change, and the old session gets `401`. +- `acct-erase`: `Delete All My Data` with the same email confirmation shows `Data Deleted` and empties the account's sources and debates. A bystander's rows stay. This is covered by `e2e/full-stack/account-data.spec.ts`, not by a recipe. + +## How to get to it (user POV) + +- Settings: the gear link named `Settings` in the header, or `/app/settings`. `Download my data` is under `Your data` in the Account card. `Delete All My Data` and `Delete account` are in the Danger Zone card. + +## Driving it with verify.sh + Playwright + +Preconditions: + +- Doctor passes. The recipe `recipes/account-data.spec.ts` covers export and deletion: `bin/verify.sh drive recipes/account-data`. The maintained specs are `e2e/full-stack/account-deletion.spec.ts` and `account-data.spec.ts`. +- Each test creates a bystander with `createDemoUser(...)`, signs it in through its own browser context, and gives it a debate and a source. `deleteDemoUser(other)` runs in `finally`. + +- **Export.** Start a debate and add a source (see [current-debate](./current-debate.md)). On `/app/settings`, wait for `page.waitForEvent("download")` while clicking `Download my data`. End state: the filename matches `^debatelab-data-\d{4}-\d{2}-\d{2}\.json$`. The JSON's `account.id` is the user's, `data.debates` and `data.sources` hold only this account's rows, and the bystander's marker is absent. Posting `{action: "export", userId: <bystander>}` with the user's token still returns the caller. +- **Delete.** First, show the guards hold. With no token, `delete` and `export` return `401`. With the user's token, `{action: "delete", confirmation: <bystander email>, userId: <bystander id>}` returns `400` and both users still exist. Then, on `/app/settings`, click `Delete account`. In the dialog, check the dialog's `Delete account` button is disabled, fill `Type your email to confirm` with `user.email`, and click it. End state: `Account deleted` shows and the URL is `/`. `admin.auth.admin.getUserById(userId)` returns no user, and a scan of every `public` and `private` table (through `COLLABORATION_DATABASE_URL`) finds no row mentioning the id. `fake.requests()` has exactly one `stripe-customer-search` for `metadata['user_id']:'<id>'`, the bystander's source survives, and the old token gets `401` from `export`. + +## Gotchas + +- Delete only the fixture's own demo user, whose email ends in `@fullstack.test`. The recipe asserts that before it clicks. Never point it at an account you did not create in the test. +- The fixture's teardown deletes the user again. That is a no-op after a successful deletion and still removes the demo code row. +- Locally, Stripe is the fake provider container (`STRIPE_API_URL` in `.superset/workspace-test.env`), and test accounts have no customers. Real Stripe cancellation is not provable here. +- The direct function calls take the session token from the `debate-auth-*` localStorage key. Keep that token out of evidence files. diff --git a/.claude/skills/verify-debatelab/features/ai-tools.md b/.claude/skills/verify-debatelab/features/ai-tools.md new file mode 100644 index 000000000..7620bed5a --- /dev/null +++ b/.claude/skills/verify-debatelab/features/ai-tools.md @@ -0,0 +1,35 @@ +# AI tools + +The single-shot coaching tools: Attack finds weaknesses in a pasted argument, Brainstorm returns ranked angles for a motion, Factoids researches a statement, and Ask is a streamed coaching chat with an optional citation mode. + +## Sub-features + +- `attack-run` analyses a pasted argument and returns weaknesses. Provider feature: `attack`. +- `brainstorm-run` returns ranked ideas for a motion. Provider feature: `brainstorm`. +- `factoids-run` finishes research and shows search suggestions. It drops unverified sources because there is no web key locally. Provider feature: `factoids`. +- `ask-stream` streams a coaching answer through `ai-stream`. Provider feature: `ask-stream`, recorded as an `ask` usage row. +- `ask-cited` uses the structured endpoint when citation mode is on. Provider feature: `ask`. + +## How to get to it (user POV) + +- Header links: `Attack` and `Ask`. `More` menu (desktop) or `Open menu` (mobile): `Brainstorm` and `Factoids`. +- Routes: `/app/attack`, `/app/brainstorm`, `/app/factoids`, `/app/ask`. + +## Driving it with verify.sh + Playwright + +Preconditions: + +- Doctor passes. The user is signed in. + +- **Attack.** On `/app/attack`, fill the field labelled `Your Argument *` and click `Find Weaknesses`. The text `FAKE-PROVIDER weakness` appears. `providerCall("attack").user` contains the argument. +- **Brainstorm.** On `/app/brainstorm`, fill the field labelled `Motion / Topic *` and click `Start Brainstorming`. With no Current Debate, the dialog `Start a debate with this motion?` opens first: `chooseForMotion(page, "create")` (or `"once"`). The text `FAKE-PROVIDER idea 1` appears. `providerCall("brainstorm").user` contains `Motion: <motion>`, and the `brainstorms` row is filed in the new debate. With a Current Debate, the field is prefilled and no dialog opens unless the motion differs. +- **Factoids.** On `/app/factoids`, fill the field labelled `Statement *` and click `Find Factoids`. The text `FAKE-PROVIDER search suggestion` appears. +- **Ask, streamed.** On `/app/ask`, fill the placeholder `Ask your debate coach...` and click the button named `Send message`. The text `FAKE-PROVIDER streamed answer: ...` appears. `fake.requests()` features contain `ask-stream` and not `ask`. `ai_requests` gets `{feature: "ask", status: "success"}`. +- **Ask, cited.** Click `#citation-mode` first, then send. The text `FAKE-PROVIDER answer` appears, and `providerCall("ask")` fired. + +## Gotchas + +- Ask silently falls back to the non-streaming path if streaming fails, so a visible answer alone does not prove streaming. Check the provider feature list. +- Factoids evidence locally shows only that the pipeline completes without inventing citations. It does not show real sources. +- Attack, Factoids, and Ask take no motion, so they open no motion dialog. Brainstorm does (see [current-debate](./current-debate.md)). +- `ask-web` (`#web-search-mode`) proves web sources only in live mode; fake mode records an `unverified` annotation. diff --git a/.claude/skills/verify-debatelab/features/auth-demo-login.md b/.claude/skills/verify-debatelab/features/auth-demo-login.md new file mode 100644 index 000000000..e0d727f56 --- /dev/null +++ b/.claude/skills/verify-debatelab/features/auth-demo-login.md @@ -0,0 +1,34 @@ +# Demo-code sign-in + +A visitor signs in with a demo access code on the Login page. The `demo-code-login` Edge Function creates or reuses a Pro account, the session survives a reload, and signing out sends protected routes back to `/login`. + +## Sub-features + +- `auth-demo-signin` signs in with a valid code and lands on `/app`. +- `auth-persist` keeps the session after a reload, and Settings shows the account email. +- `auth-reject` shows the alert `Demo code not found.` for an unknown code and stays on `/login`. +- `auth-signout` signs out from Settings, after which `/app/generate` redirects to `/login`. +- `auth-refresh` refreshes an expired access token before an AI request. + +## How to get to it (user POV) + +- Open `/login`, or any `/app/*` route while signed out (it redirects). +- In Settings (`/app/settings`, or the gear link named `Settings` in the header), choose `Sign out`. + +## Driving it with verify.sh + Playwright + +Preconditions: + +- Doctor passes. For interactive runs, `bin/verify.sh user` has printed a `DEMO_CODE`. Scripted runs get one from the `user` fixture. + +- **Sign in.** Go to `/login`, click the tab named `Demo code`, and fill the textbox named `Demo code`. Then click the button named `Sign in with demo code`. The URL becomes `/app`. This is the `signIn()` helper in `e2e/full-stack/stack.ts`. +- **Account row.** Read `user_settings` through `admin` for `await user.userId()`. It shows `subscription_tier: "pro"` and `daily_limit_override: 999`. +- **Persist.** Reload the page. The URL stays `/app`. On `/app/settings`, the page shows `user.email`. +- **Reject.** On a signed-out page, submit `DEMO-ZZZZ-ZZZZ`. The element with role `alert` has the text `Demo code not found.`, and the URL still matches `/login`. +- **Sign out.** On `/app/settings`, click the button named `Sign out`, then go to `/app/generate`. The URL matches `/login`. + +## Gotchas + +- A first sign-in on a new account starts the onboarding tour on `/app`. Its first step (`Welcome to DebateLab!`, `Start your first debate here.`) offers `Start debate`. The fixtures pre-seed `debate-trainer-onboarding` to skip it. Interactive runs click `Skip Tour` (`[data-test-id="button-skip"]`). The tour itself is covered in [current-debate](./current-debate.md). +- Sessions are stored under the `debate-auth-local` localStorage key in local-preview mode. Don't reuse stub-session helpers (`e2e/helpers/auth.ts`). They fake auth and prove rendering only. +- Google sign-in is not wired to the local stack. Only demo codes work here. diff --git a/.claude/skills/verify-debatelab/features/case-generation.md b/.claude/skills/verify-debatelab/features/case-generation.md new file mode 100644 index 000000000..124a9740d --- /dev/null +++ b/.claude/skills/verify-debatelab/features/case-generation.md @@ -0,0 +1,41 @@ +# Case generation + +A signed-in user enters a motion on Generate and receives a structured debate case: thesis, framework, contentions, the predicted opponent case, and counters. The case can be saved to the account and reopened from History, or exported as PDF or DOCX. + +## Sub-features + +- `gen-create` generates a case from the Current Debate's motion. The `ai` Edge Function calls the provider and records `ai_requests` with `feature: generate, status: success`. The case is filed in the debate automatically (`cases.debate_id`, `side: pro`); there is no Save button in a debate. +- `gen-grounded` includes active Library sources in the system prompt (see [library-sources](./library-sources.md)). +- `gen-error` shows `Generation Failed` on a provider error, and the button becomes enabled again for a retry. +- `gen-quota` shows the dialog `Daily Access Paused` when the account has no quota, without calling the provider. +- `gen-save` saves a one-off case (the motion dialog answered with `Not now, use it once`) under a title into `cases` with `debate_id: null`. It appears in History under `One-off and unsorted work`, including from a fresh browser context. +- `gen-export` downloads a `.pdf` (starting with `%PDF`) and a `.docx` (starting with `PK`). + +## How to get to it (user POV) + +- The header link `Generate`, or the Home card with `data-tour="mode-generate"`, or `/app/generate`. +- History: `More` → `History` on desktop, `Open menu` → `History` on mobile, or `/app/history`. + +## Driving it with verify.sh + Playwright + +Preconditions: + +- Doctor passes. The recipe `recipes/case-generation.spec.ts` covers every sub-feature below: `bin/verify.sh drive recipes/case-generation`. + +- **Start the debate.** `await startDebate(page, motion)`. On `/app/generate`, the textbox named `Motion / Topic *` already holds the motion. +- **Generate.** Click the button named `Generate Debate Case`. The text `FAKE-PROVIDER case thesis` and `FAKE-PROVIDER contention 3` appear, along with a toast `Generate complete`. Without a Current Debate, fill the motion yourself and answer the dialog `Start a debate with this motion?` with `chooseForMotion`. +- **Provider side effect.** `providerCall("generate")`'s `user` contains `Motion: <motion>`. `ai_requests` for the user contains `{feature: "generate", status: "success"}`. +- **Filed in the debate.** `cases` for the user is one row whose `debates(motion)` is the motion. On a fresh browser context, `/app/history` shows it as a heading in the `Cases` tab panel, and `Load` reopens it. +- **Save (one-off).** Generate with no debate and choose `chooseForMotion(page, "once")`. Click the button named `Save` (use `exact`), fill the dialog field labelled `Case title`, and click the dialog's `Save` button. `cases` for the user is `[{title, debate_id: null}]`. On a fresh browser context, `/app/history` shows `One-off and unsorted work` with a list item holding the title. +- **Error.** Call `await fake.fail("generate", 500)`, start the debate, and generate. The text `Generation Failed` (use `exact`) appears. Clicking again then succeeds. +- **Quota.** Update `user_settings` to set `daily_limit_override: 0` through `admin`, start the debate, then generate. The dialog shows `Daily Access Paused`. `fake.requests()` has no `generate` entry. +- **Export.** Wait for `page.waitForEvent("download")` while clicking the button named `PDF` or `DOCX` (use `exact`). Check the suggested filename's extension and the file's magic bytes. + +## Gotchas + +- The Generate page shows the heading `Ready to Generate` before the first run. Wait for it after navigating, because lazy route chunks load slowly on a cold Vite. +- `Save` appears both on the result toolbar and in the dialog. Scope the second click to `getByRole("dialog")`. The toolbar `Save` exists only for one-off results. +- A case filed in a debate is titled with the motion, which the header control also shows (truncated and hidden). Look for it inside the History panel, not with a page-wide `getByText`. +- The client files a result after it renders. Wait for the result on screen (and the row, if the next step reloads) before a `page.goto`, or the reload can drop it. +- The model label in the result header (for example `anthropic/claude-sonnet-5`) is the model the function requested. The answer itself came from the fake provider. +- `Google Docs` export needs Google OAuth and cannot be proven locally. diff --git a/.claude/skills/verify-debatelab/features/current-debate.md b/.claude/skills/verify-debatelab/features/current-debate.md new file mode 100644 index 000000000..77128c8eb --- /dev/null +++ b/.claude/skills/verify-debatelab/features/current-debate.md @@ -0,0 +1,44 @@ +# Current Debate + +Each student works in one Current Debate at a time: a motion with its own Library sources and saved results. The header control starts or switches the debate, tools prefill its motion and use only its material, and a tool given a different motion asks first whether to start a debate for it or use it once. Nothing crosses debates. `src/lib/stores/debate-store.ts` owns this state, and the `debates` and `current_debates` tables hold it. + +## Sub-features + +- `debate-start` starts a debate from the header. The control's `aria-label` becomes `Current debate: <motion>`, and `debates` and `current_debates` get the row. +- `debate-offer` opens the dialog `Start a debate with this motion?` when a tool runs with a motion and no Current Debate exists. `Start debate` creates the debate and the run lands in it. `Not now, use it once` runs without one. +- `debate-once` opens the dialog `This is a different motion` when a Current Debate exists and the tool has another motion. `Use once without saving to a debate` runs with no debate material and saves nothing to a debate. `Create a new debate for this motion` switches to a new one. +- `debate-scope` keeps sources and results in their debate. The Library, the dashboard at `/app/debate`, History, and every AI request's system prompt see only the Current Debate's material. +- `debate-switch` switches back to an earlier debate through `Switch debate`, and the Library and History follow. +- `debate-tour` has a brand-new account start its first debate from the first onboarding step (`Start your first debate here.`). + +## How to get to it (user POV) + +- Header: the button with test id `current-debate-desktop` (desktop) or `current-debate-mobile` (mobile, on the second header row). It reads `Start debate` until a debate exists. After that it opens a dialog with `Start new debate` and `Switch debate`. +- Home (`/app`): the card with test id `home-current-debate`, which says `No current debate` for a new account. +- Every tool page shows the strip with test id `current-debate-strip` (`Working in: <motion>`), with a `Use once` button for one-off mode. +- The debate dashboard is `/app/debate`. +- A new account's onboarding tour starts on `/app` with a `Start debate` button in its first step. + +## Driving it with verify.sh + Playwright + +Preconditions: + +- Doctor passes. The recipe `recipes/current-debate.spec.ts` covers every sub-feature: `bin/verify.sh drive recipes/current-debate`. The maintained specs are `e2e/full-stack/current-debate.spec.ts`, `first-run.spec.ts`, and `debate-context.spec.ts`. +- Helpers, re-exported from `recipes/fixtures.ts`: `startDebate(page, motion)`, `chooseForMotion(page, "create" | "once")`, and `debateControl(page)` (the visible header control on either viewport). + +- **Start.** `await startDebate(page, motion)`. It clicks the header control, fills the dialog field labelled `Motion`, and clicks the dialog's `Start debate`. End state: `debateControl(page)` has `aria-label` `Current debate: <motion>`, and `current_debates.debate_id` for the user is that `debates` row. +- **Scope.** Add a source on `/app/library` with a unique marker (see [library-sources](./library-sources.md)); the test id `library-debate` names the motion. Run Brainstorm or Generate: the motion field already holds the debate's motion and no dialog opens. End state: the newest `providerCalls()` entry for the feature has this debate's marker in `system` and not another debate's. `sources.debate_id`, `cases.debate_id`, and `brainstorms.debate_id` point at the debate they were made in. +- **Offer.** With no debate, fill a tool's motion and submit. The dialog `Start a debate with this motion?` contains the motion. Click its `Start debate`. End state: the result shows, the strip reads `Working in: <motion>`, and `debates` holds exactly that motion. +- **Use once.** With a debate, submit a different motion. The dialog `This is a different motion` opens. `chooseForMotion(page, "once")`. End state: the provider call's `system` has no debate marker, and the header still names the old debate. +- **Dashboard.** `/app/debate`: `current-debate-motion` has the motion, `debate-side-pro` reads `In favor`, `debate-side-con` reads `Against`, and `debate-sources` and `debate-recent` list only this debate's work. +- **Switch.** Click `debateControl(page)`, then the dialog's `Switch debate`, then `Switch to <motion>`. End state: the Library shows only that debate's sources. In History, the `Brainstorms` and `Cases` tabs show only its results. +- **Tour.** Seed `debate-trainer-onboarding` with `hasCompletedOnboarding: false` once per session (see the recipe), sign in, and click the tooltip's `Start debate`. Fill `Motion`, click `Start debate`, click `[data-test-id="button-primary"]` twice, then `Get Started!`. End state: one `debates` row is current, and Generate opens prefilled with no dialog. + +## Gotchas + +- The motion dialogs open after the tool's submit button and before its AI request. Create the `aiResponse()` promise before the click, answer the dialog, then await it. Otherwise the wait times out with the dialog still open. +- `chooseForMotion` scopes itself to the two motion dialogs by name, because the next step may open its own dialog, such as the paywall `Free Trial Ended`. +- The Library's `Add Source` button is disabled until a Current Debate exists. Start one first. +- `getByRole("button", { name: "Start" })` also matches the header's `Start debate`. Use `exact: true` for other Start buttons, such as the speech timer's. +- In live mode, research can save web sources to the Library. Check the rows a recipe added by title rather than counting all of them. +- Debate context review (the `debate-context-control` line and its `Review` sheet with pins) is covered by `e2e/full-stack/debate-context.spec.ts`, not by a recipe. diff --git a/.claude/skills/verify-debatelab/features/library-sources.md b/.claude/skills/verify-debatelab/features/library-sources.md new file mode 100644 index 000000000..6630ba93c --- /dev/null +++ b/.claude/skills/verify-debatelab/features/library-sources.md @@ -0,0 +1,36 @@ +# Library sources + +Users save reference texts to their Library. The Library belongs to the Current Debate: each source has a `debate_id`, and the page shows only the Current Debate's sources (test id `library-debate` names the motion). Active sources ground AI requests: the Edge Function loads them server-side and puts them in the system prompt. Paused sources are excluded. Deletions persist across reloads. + +## Sub-features + +- `lib-add` adds a source with a title and content. It appears with its toggle on, and the banner `Grounding Active` shows. +- `lib-toggle` pauses a source through its switch, which keeps it out of prompts. +- `lib-ground` includes only active sources' text in the `system` of later AI calls (Generate, Brainstorm, and others). +- `lib-delete` deletes a source. It stays gone after a reload and in the `sources` table. +- `lib-import` imports a web page by URL through the `import-source` Edge Function, which fetches the page itself (no Exa), so it works in both modes when the VPS has outbound internet. +- `lib-erase` has Settings → `Delete All My Data` remove this account's sources and nobody else's. + +## How to get to it (user POV) + +- Desktop: `More` → `Library`. Mobile: `Open menu` → `Library`. Or go to `/app/library`. +- Settings (`/app/settings`) → `Delete All My Data`, for the erase path. + +## Driving it with verify.sh + Playwright + +Preconditions: + +- Doctor passes. The user is signed in (`signedIn` fixture). A Current Debate exists: `await startDebate(page, motion)`. Without one, `Add Source` stays disabled. + +- **Add.** Click the button named `Add Source`. Fill the textbox named `Source title...`, fill the textbox named `Paste source content...` with a unique marker, and click `Save to Library`. The switch named `Toggle <title>` is checked. `sources` has a row with `is_active: true` and the debate's `debate_id`. +- **Toggle.** Click the switch named `Toggle <title>`. It becomes not checked. +- **Ground.** Go to `/app/brainstorm`; the motion is prefilled from the debate. Run it (see [ai-tools](./ai-tools.md)). `providerCall("brainstorm").system` contains the active marker and does not contain the paused marker. +- **Delete.** Click the button named `Delete <title>` (use `exact`). The switch count drops to 0, and stays 0 after `page.reload()`. `sources` no longer has the title. +- **Import.** In the add panel, fill `Import from URL` and click `Import URL`. `Source Ready` appears, then `Save imported source` stores a row whose `origin` is the URL. +- **Erase.** On Settings, click `Delete All My Data`. In the dialog, fill `Type your email to confirm` with `user.email` and click the dialog's `Delete All My Data`. The text `Data Deleted` appears, and `sources` for the user is `[]`. + +## Gotchas + +- Large pastes, up to about 250 KB, are expected to work because sources load server-side. A 413 error means that regressed. +- Another debate's sources are hidden, not deleted. To prove scoping across debates, see [current-debate](./current-debate.md). +- When proving `lib-erase`, create a second user with `createDemoUser` and show its rows survive. Otherwise "erased everything" is not distinguishable from "erased the right things". diff --git a/.claude/skills/verify-debatelab/features/practice.md b/.claude/skills/verify-debatelab/features/practice.md new file mode 100644 index 000000000..3e93dabea --- /dev/null +++ b/.claude/skills/verify-debatelab/features/practice.md @@ -0,0 +1,36 @@ +# Practice + +Live practice modes. Sparring runs a multi-speech round against an AI opponent and ends with a judged verdict saved to the account. Crossfire runs a question-and-answer exchange with Jev answer signals and ends with a debrief. Both are filed in the Current Debate (see [current-debate](./current-debate.md)), and both can run as spoken rounds (see [voice-rounds](./voice-rounds.md)). + +## Sub-features + +- `spar-round` starts a round with a chosen format, submits the user's speeches, and receives opponent speeches. +- `spar-verdict` judges the finished round. A `sparring_sessions` row reaches `status: complete`, with `side: pro` and the debate for its motion. +- `cross-turn` answers a user question with an AI response and attaches a Jev `Direct answer` signal. +- `cross-debrief` ends the session and shows the debrief. A `crossfires` row is filed in the debate for its resolution. +- `cross-ai-questions` reverses the roles: the AI asks (`mode-ai-questions-user`), the user answers, and the AI follows up. + +## How to get to it (user POV) + +- Header links `Sparring` and `Crossfire`, or `/app/sparring` and `/app/crossfire`. +- Generate result → `Open in Crossfire`, which seeds Crossfire from a generated case. + +## Driving it with verify.sh + Playwright + +Preconditions: + +- Doctor passes. The user is signed in. The recipe is `recipes/practice.spec.ts`. + +- **Start a sparring round.** On `/app/sparring`, fill the field labelled `Motion`. Open the combobox named `Round format` and choose the option matching `Lincoln-Douglas · 5 speeches`. Click `Start round`. A new account has no Current Debate, so `Start a debate with this motion?` opens: `chooseForMotion(page, "create")`. +- **Speeches.** For each user turn, fill the textbox named `Draft for <turn>` (`Affirmative constructive`, `First affirmative rebuttal`, `Second affirmative rebuttal`) and click `Submit speech`. After each of the first two, the last `FAKE-PROVIDER opponent speech` is visible. +- **Verdict.** The heading `Here is what happened in the room.` and the text `FAKE-PROVIDER verdict: the user had the edge.` appear. `providerCall("sparring").user` contains `SPARRING ACTION:`. `sparring_sessions` statuses for the user equal `["complete"]`, and `select("side,debates(motion)")` gives `{side: "pro", debates: {motion}}`. +- **Start Crossfire.** On `/app/crossfire`, fill test id `crossfire-resolution` and `crossfire-argument`. Click `mode-question-ai` (or `mode-ai-questions-user`), then `start-crossfire`, then `chooseForMotion(page, "create")`. +- **Crossfire turn.** Fill `crossfire-input` and click `submit-turn`. The text `FAKE-PROVIDER crossfire answer` and `Direct answer` appear. `providerCall("crossfire")` and `providerCall("jev")` both fired. +- **Debrief.** Click `end-crossfire`. Test id `crossfire-debrief` is visible. `crossfires` for the user equals `[{title: <resolution>, debates: {motion: <resolution>}}]`. + +## Gotchas + +- Opponent speeches carry a server-signed integrity token. Scoring fails if the client drops it, so a verdict proves the round-trip. +- Jev is faked with a fixed confident answer. `Direct answer` proves the wiring, not the quality of the judgement. +- Sparring AI calls happen once per opponent turn. Keep the per-step wait near 30 s rather than one long wait for the verdict. +- In `mode-ai-questions-user` the opening question is requested as soon as the round starts, after the motion dialog. Create the `aiResponse(page, "crossfire", "next-turn")` promise before clicking `start-crossfire`. diff --git a/.claude/skills/verify-debatelab/features/voice-rounds.md b/.claude/skills/verify-debatelab/features/voice-rounds.md new file mode 100644 index 000000000..71de520fd --- /dev/null +++ b/.claude/skills/verify-debatelab/features/voice-rounds.md @@ -0,0 +1,34 @@ +# Voice rounds + +Sparring and Crossfire can run as spoken rounds. The student speaks through the browser's speech recognition, and the words fill the draft box. The opponent's reply is read aloud. With an ElevenLabs key, the `voice` Edge Function produces that audio from server-signed segments. Without one, which is how the local stack runs, the function answers `503 voice_unavailable` once and the browser's own voice reads the replies. The judge receives measured delivery facts (words per minute, fillers, pauses) and is told it did not hear the round. + +## Sub-features + +- `voice-spar` runs a spoken Lincoln-Douglas round. The opponent is told the round is spoken (`SPOKEN ROUND:` in its system prompt). Each reply comes back as clean text plus signed voice segments, which may keep allowed audio tags such as `[sighs]` and drop invented ones. The browser voice reads the clean text, and no tag reaches the page or the saved round. The scorecard shows `Spoken delivery, measured` (test id `spoken-delivery`), and the saved user turns carry `delivery` measurements. +- `voice-cross` runs a spoken Crossfire exchange. The question arrives by microphone, the request carries `SPOKEN EXCHANGE:`, and the clean answer is read aloud. The Jev signal still attaches, and the debrief shows `spoken-delivery`. +- `voice-unavailable`: when the browser has no speech recognition or the microphone is blocked, the switch is replaced by the test id `voice-unavailable`. This is covered by `src/test/pages/SparringVoice.test.tsx` and `CrossfireVoice.test.tsx`, not by a recipe. + +## How to get to it (user POV) + +- `/app/sparring` or `/app/crossfire`: the switch `Voice round` (test id `voice-mode`) on the setup form. The first time it is switched on, the notice `Before you turn on voice` explains where audio and text go. Click its `Turn on voice`. +- During the round, the microphone button (test id `voice-mic`) starts listening, and the page shows `Listening`. + +## Driving it with verify.sh + Playwright + +Preconditions: + +- Doctor passes. The recipe `recipes/voice-rounds.spec.ts` covers both rounds: `bin/verify.sh drive recipes/voice-rounds`. The maintained spec is `e2e/full-stack/voice.spec.ts`. +- Browser automation has no microphone or speech service. `stubSpeech(page)` installs a scripted recogniser, browser voice, and microphone, and `recipes/fixtures.ts` re-exports it from `e2e/helpers/voice.ts`. Call it on the signed-in page before navigating to the tool, because init scripts apply from the next load. `say(page, words)` delivers a final recognition result. `spokenByBrowser(page)` returns everything the browser voice was asked to read. + +- **Set up.** Fill the round as in [practice](./practice.md), click `voice-mode`, then `Turn on voice`, then `Start round` or `start-crossfire`, then `chooseForMotion(page, "create")`. +- **Speak a turn.** Click `voice-mic`, wait for `Listening`, `say(page, words)`. The draft (`Draft for <turn>`, or `crossfire-input`) has the words. Click `Submit speech` or `submit-turn`. +- **Let the opponent finish.** After each opponent reply, wait until the `voice` function has answered (once, `503 voice_unavailable`) and `spokenByBrowser(page)` equals the replies so far. Only then click `voice-mic` again. +- **End state, Sparring.** The verdict heading `Here is what happened in the room.` shows. `spoken-delivery` contains `Nobody listened for tone, confidence or volume`. After `Show transcript`, the page has no `[sighs]` or `[dramatic gasp]`. `sparring_sessions.turns` holds the clean replies, and the first user turn's `delivery` is `{wordCount: 10, fillerCount: 1}` for the scripted words. The `score` call's `system` contains `You did not hear this round`, and its `user` contains `"wordsPerMinute"`. `ai_requests` has no `voice_%` rows. +- **End state, Crossfire.** The answer is on the page with no `[scoffs]`, `spokenByBrowser` equals it, and the `crossfire` call's `user` contains `SPOKEN EXCHANGE:`. After `end-crossfire`, `crossfire-debrief` and `spoken-delivery` are visible. + +## Gotchas + +- Turning the microphone on while the opponent is still speaking stops its voice and abandons the pending `voice` request. A test that does not wait sees zero voice answers and a short `spokenByBrowser`. +- The fake opponent always adds `[sighs]` (allowed) and `[dramatic gasp]` (invented) in a spoken round, and Crossfire adds `[scoffs]`. Only fake mode asserts the exact tagged segment text. Live replies may carry no tag at all. +- Real ElevenLabs audio, its usage rows (`voice_*` in `ai_requests`), and its daily allowance are not provable locally. Neither mode sets an ElevenLabs key. +- The scripted recogniser returns exactly the words given, so `delivery` counts are deterministic. Real recognition can miss words, which the scorecard says. diff --git a/.claude/skills/verify-debatelab/playwright.verify.config.ts b/.claude/skills/verify-debatelab/playwright.verify.config.ts new file mode 100644 index 000000000..7d2fd3ea6 --- /dev/null +++ b/.claude/skills/verify-debatelab/playwright.verify.config.ts @@ -0,0 +1,47 @@ +import { defineConfig, devices } from "@playwright/test"; +import { join } from "node:path"; +import { fileURLToPath } from "node:url"; + +// Driven by `bin/verify.sh drive`, which starts nothing: the stack from +// `verify.sh up` must already be running. Playwright empties outputDir at the +// start of every run, so each drive gets its own directory under .verify/artifacts. +const artifactDir = process.env.VERIFY_ARTIFACT_DIR; +if (!artifactDir) throw new Error("Run recipes through `verify.sh drive`, not npx playwright directly"); +const live = process.env.VERIFY_MODE === "live"; + +export default defineConfig({ + // The repository root, so the maintained e2e suites can run against this stack too. + testDir: fileURLToPath(new URL("../../..", import.meta.url)), + // Recipes run in both modes. e2e/full-stack asserts fake-provider text, so + // live mode keeps only its provider-free files; e2e/live asserts live + // structure (live mode only); the workspace journey starts its own + // collaboration process and runs in either mode. + testMatch: live + ? /(\.claude\/skills\/verify-debatelab\/recipes\/.+|e2e\/live\/.+|e2e\/full-stack\/(admin|public|auth|billing|account-data)|e2e\/workspace-collaboration)\.spec\.ts$/ + : /(\.claude\/skills\/verify-debatelab\/recipes\/.+|e2e\/full-stack\/.+|e2e\/workspace-collaboration)\.spec\.ts$/, + // In live mode the full-stack tests that assert fake-provider text are + // skipped; the recipes cover the same journeys with structural assertions. + grepInvert: live + ? /a promo code redeemed in Settings restores AI access|an expired access token refreshes before an AI request|only active Library sources ground AI requests|the output language setting follows the account/ + : undefined, + outputDir: join(artifactDir, "playwright"), + fullyParallel: false, + workers: 1, + retries: 0, + // Live model calls with mandatory reasoning take far longer than the fake. + timeout: live ? 600_000 : 120_000, + reporter: [["list"], ["json", { outputFile: join(artifactDir, "report.json") }]], + use: { + baseURL: "http://127.0.0.1:18081", + locale: "en-US", + actionTimeout: 15_000, + // Proof runs keep everything, not only failures. + trace: "on", + screenshot: "on", + }, + expect: { timeout: 15_000 }, + projects: [ + { name: "desktop-chromium", use: { ...devices["Desktop Chrome"], viewport: { width: 1280, height: 800 } } }, + { name: "mobile-chromium", use: { ...devices["Pixel 5"] } }, + ], +}); diff --git a/.claude/skills/verify-debatelab/recipes/account-data.spec.ts b/.claude/skills/verify-debatelab/recipes/account-data.spec.ts new file mode 100644 index 000000000..48680fecc --- /dev/null +++ b/.claude/skills/verify-debatelab/recipes/account-data.spec.ts @@ -0,0 +1,170 @@ +import { readFile } from "node:fs/promises"; +import pg from "pg"; +import type { Page } from "@playwright/test"; +import { admin, createDemoUser, deleteDemoUser, expect, fake, signIn, startDebate, test } from "./fixtures"; + +// Feature map: features/account-data.md. Mirrors e2e/full-stack/account-deletion.spec.ts. +// Each test seeds a bystander account and proves the action touched only the +// recipe's own demo user. Delete All My Data is e2e/full-stack/account-data.spec.ts. + +const accountUrl = `${process.env.LOCAL_SUPABASE_URL}/functions/v1/account`; + +async function addSource(page: Page, title: string, content: string) { + await page.goto("/app/library"); + await page.getByRole("button", { name: "Add Source" }).click(); + await page.getByRole("textbox", { name: "Source title..." }).fill(title); + await page.getByRole("textbox", { name: "Paste source content..." }).fill(content); + await page.getByRole("button", { name: "Save to Library" }).click(); + await expect(page.getByRole("switch", { name: `Toggle ${title}` })).toBeChecked(); +} + +/** The signed-in browser session's access token, for probing the function directly. */ +async function accessToken(page: Page) { + return page.evaluate(() => { + const key = Object.keys(localStorage).find((candidate) => candidate.startsWith("debate-auth-")); + return key ? (JSON.parse(localStorage.getItem(key)!) as { access_token: string }).access_token : ""; + }); +} + +function callAccount(token: string | null, body: unknown) { + return fetch(accountUrl, { + method: "POST", + headers: { + "content-type": "application/json", + apikey: process.env.SUPABASE_ANON_KEY ?? "", + ...(token ? { authorization: `Bearer ${token}` } : {}), + }, + body: JSON.stringify(body), + }); +} + +/** Every public and private table holding a row whose text contains the id. */ +async function tablesMentioning(id: string) { + const client = new pg.Client({ connectionString: process.env.COLLABORATION_DATABASE_URL }); + await client.connect(); + try { + const { rows: tables } = await client.query<{ name: string }>( + "SELECT format('%I.%I', schemaname, tablename) AS name FROM pg_tables WHERE schemaname IN ('public', 'private')", + ); + const found: string[] = []; + for (const { name } of tables) { + const { rows } = await client.query(`SELECT 1 FROM ${name} AS t WHERE t::text LIKE $1 LIMIT 1`, [`%${id}%`]); + if (rows.length > 0) found.push(name); + } + return found; + } finally { + await client.end(); + } +} + +/** A second account, signed in through its own browser context, with a debate and a source of its own. */ +async function seedBystander(page: Page, other: Awaited<ReturnType<typeof createDemoUser>>) { + const context = await page.context().browser()!.newContext(); + try { + const bystander = await context.newPage(); + await bystander.addInitScript(() => localStorage.setItem("debate-trainer-onboarding", JSON.stringify({ state: { hasCompletedOnboarding: true, isTourRunning: false }, version: 0 }))); + await signIn(bystander, other); + await startDebate(bystander, "Bystander motion"); + await addSource(bystander, "Bystander study", "BYSTANDER-PRIVATE evidence"); + } finally { + await context.close(); + } +} + +test("acct-export: Download my data saves only this account's data as JSON", async ({ signedIn: page, user, evidence }) => { + const other = await createDemoUser("export-bystander"); + try { + await seedBystander(page, other); + await startDebate(page, "Exported motion"); + await addSource(page, "Exported study", "EXPORTED-PRIVATE evidence"); + await evidence.step("Seeded a bystander account; started 'Exported motion' and added 'Exported study'"); + + await page.goto("/app/settings"); + await evidence.aria("01-settings"); + const download = page.waitForEvent("download"); + await page.getByRole("button", { name: "Download my data" }).click(); + const file = await download; + expect(file.suggestedFilename()).toMatch(/^debatelab-data-\d{4}-\d{2}-\d{2}\.json$/); + const path = `${evidence.dir}/export.json`; + await file.saveAs(path); + const text = await readFile(path, "utf8"); + const exported = JSON.parse(text); + await evidence.step(`Clicked Download my data: ${file.suggestedFilename()} (${text.length} bytes, saved as export.json)`); + + expect(exported.format).toBe("debatelab-account-export"); + expect(exported.account).toMatchObject({ id: await user.userId(), email: user.email }); + expect(exported.data.debates.map((row: { motion: string }) => row.motion)).toEqual(["Exported motion"]); + expect(exported.data.sources.map((row: { title: string }) => row.title)).toEqual(["Exported study"]); + expect(text).not.toContain("BYSTANDER-PRIVATE"); + + // Naming another account in the request still returns only the caller's data. + const response = await callAccount(await accessToken(page), { action: "export", userId: await other.userId() }); + expect(response.status).toBe(200); + const direct = await response.json(); + expect(direct.account.id).toBe(await user.userId()); + expect(JSON.stringify(direct)).not.toContain("BYSTANDER-PRIVATE"); + await evidence.json("02-export-summary", { + file: file.suggestedFilename(), + format: exported.format, + debates: exported.data.debates.length, + sources: exported.data.sources.length, + bystanderLeaked: false, + forgedUserIdRequestReturnedCaller: true, + }); + } finally { + await deleteDemoUser(other); + } +}); + +test("acct-delete: Delete account removes every row that references this user and nobody else's", async ({ signedIn: page, user, evidence }) => { + // Deletion is irreversible: only ever delete the fixture's own throwaway demo user. + expect(user.email, "the account to delete is this recipe's demo user").toMatch(/^[a-z-]+-[0-9a-f-]{36}@fullstack\.test$/); + const other = await createDemoUser("delete-bystander"); + try { + await seedBystander(page, other); + await startDebate(page, "Deleted motion"); + await addSource(page, "Deleted study", "Removed with the account."); + const userId = await user.userId(); + const otherId = await other.userId(); + const before = await tablesMentioning(userId); + expect(before).not.toEqual([]); + await evidence.json("01-tables-before", before); + + // Without a session, or with another account's confirmation, nothing is deleted. + expect((await callAccount(null, { action: "delete", confirmation: user.email })).status).toBe(401); + expect((await callAccount(null, { action: "export" })).status).toBe(401); + const token = await accessToken(page); + const forged = await callAccount(token, { action: "delete", confirmation: other.email, userId: otherId }); + expect(forged.status).toBe(400); + expect((await admin.auth.admin.getUserById(userId)).data.user?.id).toBe(userId); + expect((await admin.auth.admin.getUserById(otherId)).data.user?.id).toBe(otherId); + + await page.goto("/app/settings"); + await page.getByRole("button", { name: "Delete account" }).click(); + const dialog = page.getByRole("dialog"); + await expect(dialog.getByRole("button", { name: "Delete account" })).toBeDisabled(); + await dialog.getByLabel("Type your email to confirm").fill(user.email); + await evidence.shot("02-confirm-dialog"); + await dialog.getByRole("button", { name: "Delete account" }).click(); + await evidence.step("Settings → Delete account, typed the account email, confirmed"); + await expect(page.getByText("Account deleted").first()).toBeVisible({ timeout: 20_000 }); + await expect(page).toHaveURL(/\/$/); + await evidence.shot("03-account-deleted"); + + expect((await admin.auth.admin.getUserById(userId)).data.user).toBeNull(); + expect(await tablesMentioning(userId)).toEqual([]); + // Billing was checked in Stripe (the fake Stripe locally), not only in the database, before anything was deleted. + const searches = (await fake.requests()).filter((request) => request.feature === "stripe-customer-search"); + expect(searches.map((request) => request.user)).toEqual([`metadata['user_id']:'${userId}'`]); + + expect((await admin.auth.admin.getUserById(otherId)).data.user?.id).toBe(otherId); + const { data: theirs } = await admin.from("sources").select("title").eq("user_id", otherId); + expect(theirs?.map((row) => row.title)).toEqual(["Bystander study"]); + await evidence.json("04-after-delete", { userGone: true, tablesMentioningUser: [], bystanderSources: theirs }); + + // The deleted session no longer works. + expect((await callAccount(token, { action: "export" })).status).toBe(401); + } finally { + await deleteDemoUser(other); + } +}); diff --git a/.claude/skills/verify-debatelab/recipes/account.spec.ts b/.claude/skills/verify-debatelab/recipes/account.spec.ts new file mode 100644 index 000000000..f5a588dc2 --- /dev/null +++ b/.claude/skills/verify-debatelab/recipes/account.spec.ts @@ -0,0 +1,83 @@ +import { admin, AI_TIMEOUT, aiResponse, expect, expectProviderCalls, signIn, test } from "./fixtures"; + +// Feature map: the Account entry in features/README.md. Export and account +// deletion are recipes/account-data.spec.ts (features/account-data.md). Delete +// All My Data and the plain flowchart save are covered by +// e2e/full-stack/account-data.spec.ts, which drive also runs. + +test("acct-language: the output language follows the account to a new device and reaches the model", async ({ signedIn: page, user, browser, evidence }) => { + await page.goto("/app/settings"); + await page.locator("div.space-y-2", { has: page.getByText("AI Output Language", { exact: true }) }).getByRole("combobox").click(); + await page.getByRole("option", { name: "Magyar" }).click(); + await expect.poll(async () => { + const { data } = await admin.from("user_settings").select("output_language").eq("user_id", await user.userId()).single(); + return data?.output_language; + }).toBe("hu"); + await evidence.shot("01-settings-hungarian"); + + const context = await browser.newContext(); + try { + const other = await context.newPage(); + await other.addInitScript(() => localStorage.setItem("debate-trainer-onboarding", JSON.stringify({ state: { hasCompletedOnboarding: true, isTourRunning: false }, version: 0 }))); + await signIn(other, user); + await other.goto("/app/attack"); + await other.getByLabel("Your Argument *").fill("Free transit cuts congestion because riders leave their cars at home."); + const response = aiResponse(other, "attack"); + await other.getByRole("button", { name: "Find Weaknesses" }).click(); + const body = await response; + await evidence.json("02-attack-response", body); + const call = (await expectProviderCalls(["attack"])).find((item) => item.feature === "attack")!; + expect(call.user).toContain("Language: Hungarian (hu)"); + await evidence.shot("02-attack-in-hungarian", other); + } finally { + await context.close(); + } +}); + +test("flow-edit: a flowchart node is deleted, the chart is saved, and it loads back", async ({ signedIn: page, user, evidence }) => { + const title = `Verify flow ${Date.now()}`; + await page.goto("/app/flowchart"); + await page.getByText("Argument 1", { exact: true }).first().click(); + await page.getByRole("button", { name: "Delete" }).last().click(); + const confirm = page.getByRole("alertdialog"); + await expect(confirm.getByText("Delete node?")).toBeVisible(); + await confirm.getByRole("button", { name: "Delete node" }).click(); + await expect(page.getByText("Node deleted").first()).toBeVisible(); + await expect(page.getByText("Argument 1", { exact: true })).toHaveCount(0); + await evidence.shot("01-node-deleted"); + + await page.getByRole("button", { name: "Save", exact: true }).click(); + const dialog = page.getByRole("dialog"); + await dialog.getByRole("textbox").fill(title); + await dialog.getByRole("button", { name: "Save Flowchart" }).click(); + await expect(page.getByText("Flowchart saved successfully").first()).toBeVisible(); + const { data } = await admin.from("flowcharts").select("title,nodes").eq("user_id", await user.userId()); + expect(data?.map((row) => row.title)).toEqual([title]); + expect(JSON.stringify(data?.[0].nodes)).not.toContain("Argument 1"); + await evidence.json("02-flowchart-row", data); + + await page.reload(); + await page.getByRole("button", { name: "Load" }).click(); + await page.getByText(title).first().click(); + await expect(page.getByText("Argument 1", { exact: true })).toHaveCount(0); + await evidence.shot("03-loaded"); +}); + +test("timer-run: the speech timer starts, pauses, and resets", async ({ signedIn: page, evidence }) => { + await page.goto("/app/timer"); + const display = page.getByRole("timer", { name: "Timer display" }); + const initial = await display.textContent(); + // Exact: the header's "Start debate" button also matches a substring "Start". + await page.getByRole("button", { name: "Start", exact: true }).click(); + await expect(page.getByText("SPEAKING").first()).toBeVisible(); + await expect.poll(() => display.textContent(), { timeout: 5_000 }).not.toBe(initial); + await page.getByRole("button", { name: "Pause" }).click(); + await expect(page.getByText("PAUSED").first()).toBeVisible(); + const paused = await display.textContent(); + await page.waitForTimeout(1_500); + expect(await display.textContent()).toBe(paused); + await evidence.shot("01-paused"); + await page.keyboard.press("r"); + await expect(display).toHaveText(initial!); + await evidence.shot("02-reset"); +}); diff --git a/.claude/skills/verify-debatelab/recipes/ai-tools.spec.ts b/.claude/skills/verify-debatelab/recipes/ai-tools.spec.ts new file mode 100644 index 000000000..1fb85bf3a --- /dev/null +++ b/.claude/skills/verify-debatelab/recipes/ai-tools.spec.ts @@ -0,0 +1,124 @@ +import { admin, AI_TIMEOUT, aiResponse, chooseForMotion, expect, expectProviderCalls, LIVE, providerCalls, test } from "./fixtures"; + +// Feature map: features/ai-tools.md. + +test("attack-run: Attack analyses a pasted argument", async ({ signedIn: page, evidence }) => { + const argument = "Free transit cuts congestion because riders leave their cars at home, so cities should abolish fares."; + await page.goto("/app/attack"); + await page.getByLabel("Your Argument *").fill(argument); + await evidence.step("Attack: pasted an argument and clicked Find Weaknesses"); + const response = aiResponse(page, "attack"); + await page.getByRole("button", { name: "Find Weaknesses" }).click(); + const body = await response; + await evidence.json("01-attack-response", body); + const weaknesses = body.output.weaknesses as Array<{ description: string }>; + expect(weaknesses.length).toBeGreaterThan(0); + await expect(page.getByText(weaknesses[0].description).first()).toBeVisible(); + await evidence.shot("02-attack-result"); + const call = (await expectProviderCalls(["attack"])).find((item) => item.feature === "attack")!; + expect(call.user).toContain("riders leave their cars at home"); +}); + +test("brainstorm-run: Brainstorm returns ranked angles", async ({ signedIn: page, user, evidence }) => { + const motion = "Universal basic income should replace targeted welfare"; + await page.goto("/app/brainstorm"); + await page.getByLabel("Motion / Topic *").fill(motion); + const response = aiResponse(page, "brainstorm"); + await page.getByRole("button", { name: "Start Brainstorming" }).click(); + // A new account has no Current Debate, so Brainstorm first offers one for this motion. + await chooseForMotion(page, "create"); + await evidence.step(`Brainstorm: "${motion}", accepted the offer to start a debate for it`); + const body = await response; + await evidence.json("01-brainstorm-response", body); + const topFive = body.output.topFive as Array<{ idea: { title: string } }>; + expect(topFive.length).toBeGreaterThan(0); + await expect(page.getByText(topFive[0].idea.title).first()).toBeVisible(); + await evidence.shot("02-brainstorm-result"); + const call = (await expectProviderCalls(["brainstorm"])).find((item) => item.feature === "brainstorm")!; + expect(call.user).toContain(`Motion: ${motion}`); + // The result is saved in the debate the offer created. + await expect.poll(async () => { + const { data } = await admin.from("brainstorms").select("debates(motion)").eq("user_id", await user.userId()); + return data; + }).toEqual([{ debates: { motion } }]); +}); + +test("factoids-run: Factoids researches a statement and keeps only verified citations", async ({ signedIn: page, evidence }) => { + await page.goto("/app/factoids"); + await page.getByLabel("Statement *").fill("Fare-free public transport increases ridership"); + const response = aiResponse(page, "factoids"); + await page.getByRole("button", { name: "Find Factoids" }).click(); + const body = await response; + await evidence.json("01-factoids-response", body); + const factoids = body.output.factoids as Array<{ shortClaim: string; citation: { url?: string; verified: string } }>; + if (LIVE) { + expect(factoids.length, "verified factoids").toBeGreaterThan(0); + for (const factoid of factoids) { + expect(factoid.citation.verified).toBe("verified"); + expect(factoid.citation.url).toMatch(/^https?:\/\//); + } + await expect(page.getByText(factoids[0].shortClaim).first()).toBeVisible(); + } else { + // No Exa key in fake mode: every unverified factoid must be dropped, not invented. + expect(factoids).toEqual([]); + test.info().annotations.push({ type: "unverified", description: "verified citations need --live (Exa)" }); + } + await evidence.shot("02-factoids-result"); + await expectProviderCalls(["factoids"]); +}); + +test("ask-stream: Ask streams a coaching answer and records usage", async ({ signedIn: page, user, evidence }) => { + await page.goto("/app/ask"); + await page.getByPlaceholder("Ask your debate coach...").fill("In two sentences, how should I weigh access against cost?"); + const stream = page.waitForResponse((response) => response.url().endsWith("/functions/v1/ai-stream"), { timeout: AI_TIMEOUT }); + await page.getByRole("button", { name: "Send message" }).click(); + const response = await stream; + expect(response.status()).toBe(200); + const events = await response.text(); + // A 200 stream can still end early; only the final event proves a complete answer. + expect(events).toContain("event: final"); + expect(events).not.toContain("event: error"); + await evidence.json("01-stream-events", events.split("\n\n").slice(-6)); + await expect(page.getByText("Response was interrupted")).toHaveCount(0); + await evidence.shot("02-streamed-answer"); + // The client silently falls back to the structured endpoint if streaming fails. + const features = (await providerCalls()).map((call) => call.feature); + expect(features).toContain("ask-stream"); + expect(features).not.toContain("ask"); + await expect.poll(async () => { + const { data } = await admin.from("ai_requests").select("feature,status").eq("user_id", await user.userId()); + return data; + }, { timeout: AI_TIMEOUT }).toContainEqual(expect.objectContaining({ feature: "ask", status: "success" })); +}); + +test("ask-cited: Ask in citation mode uses the structured endpoint", async ({ signedIn: page, evidence }) => { + await page.goto("/app/ask"); + await page.locator("#citation-mode").click(); + await page.getByPlaceholder("Ask your debate coach...").fill("Which evidence supports fare-free transit?"); + const response = aiResponse(page, "ask"); + await page.getByRole("button", { name: "Send message" }).click(); + const body = await response; + await evidence.json("01-ask-response", body); + expect(typeof body.output.response).toBe("string"); + await evidence.shot("02-cited-answer"); + await expectProviderCalls(["ask"]); +}); + +test("ask-web: Ask with web search shows the sources it used", async ({ signedIn: page, evidence }) => { + await page.goto("/app/ask"); + await page.locator("#web-search-mode").click(); + await page.getByPlaceholder("Ask your debate coach...").fill("What did recent studies find about fare-free public transport ridership?"); + const response = aiResponse(page, "ask"); + await page.getByRole("button", { name: "Send message" }).click(); + const body = await response; + await evidence.json("01-ask-web-response", body); + await evidence.shot("02-ask-web"); + if (!LIVE) { + test.info().annotations.push({ type: "unverified", description: "web search needs --live (Exa)" }); + return; + } + const searchInfo = body.output.searchInfo as { webSearchUsed: boolean; sources: Array<{ url: string }> }; + expect(searchInfo.webSearchUsed).toBe(true); + expect(searchInfo.sources.length).toBeGreaterThan(0); + await expect(page.getByText("Web research used:")).toBeVisible(); +}); diff --git a/.claude/skills/verify-debatelab/recipes/auth.spec.ts b/.claude/skills/verify-debatelab/recipes/auth.spec.ts new file mode 100644 index 000000000..ac1dc992e --- /dev/null +++ b/.claude/skills/verify-debatelab/recipes/auth.spec.ts @@ -0,0 +1,39 @@ +import { expect, test } from "./fixtures"; + +// Feature map: features/auth.md. Demo-code sign-in, reload, rejection, and +// sign-out are covered by e2e/full-stack/auth.spec.ts, which drive also runs. + +const mailer = process.env.WORKSPACE_TEST_MAILER_URL; + +test("auth-email-otp: an emailed one-time code signs an existing account in", async ({ page, user, request, evidence }) => { + test.skip(!mailer, "WORKSPACE_TEST_MAILER_URL is missing; run through verify.sh drive"); + const sentAfter = Date.now(); + await page.goto("/login"); + await page.getByRole("tab", { name: "Email" }).click(); + await page.getByLabel("Email address").fill(user.email); + await page.getByRole("button", { name: "Send code" }).click(); + await expect(page.getByText("Code sent").first()).toBeVisible(); + await evidence.shot("01-code-sent"); + + // Read the real message from the local Mailpit inbox. + let otp = ""; + await expect.poll(async () => { + const payload = await (await request.get(`${mailer}/api/v1/messages`)).json() as { messages?: Array<Record<string, unknown>> }; + for (const message of payload.messages ?? []) { + if (!JSON.stringify(message).includes(user.email)) continue; + if (Date.parse(String(message.Created ?? 0)) < sentAfter - 5_000) continue; + const detail = await (await request.get(`${mailer}/api/v1/message/${message.ID}`)).text(); + otp = detail.match(/\b(\d{6})\b/)?.[1] ?? ""; + if (otp) break; + } + return otp; + }, { timeout: 30_000 }).toMatch(/^\d{6}$/); + await evidence.step("Read the one-time code from Mailpit"); + + await page.getByRole("textbox", { name: "Verification code" }).fill(otp); + await page.getByRole("button", { name: "Verify code" }).click(); + await expect(page).toHaveURL(/\/app/); + await page.reload(); + await expect(page).toHaveURL(/\/app/); + await evidence.shot("02-signed-in-by-email"); +}); diff --git a/.claude/skills/verify-debatelab/recipes/billing.spec.ts b/.claude/skills/verify-debatelab/recipes/billing.spec.ts new file mode 100644 index 000000000..64661910b --- /dev/null +++ b/.claude/skills/verify-debatelab/recipes/billing.spec.ts @@ -0,0 +1,47 @@ +import { randomUUID } from "node:crypto"; +import { admin, AI_TIMEOUT, aiResponse, chooseForMotion, expect, providerCalls, test } from "./fixtures"; + +// Feature map: the Billing entry in features/README.md. Signed, forged, and replayed Stripe +// webhooks are covered by e2e/full-stack/billing.spec.ts, which drive also runs. + +test("bill-promo: an expired account is blocked, redeems a promo code, and gets AI access back", async ({ signedIn: page, user, evidence }) => { + const userId = await user.userId(); + await admin.from("user_settings") + .update({ subscription_tier: "expired", trial_ends_at: null, grace_period_ends_at: null, daily_limit_override: null }) + .eq("user_id", userId); + const code = `VERIFY${randomUUID().slice(0, 8).toUpperCase()}`; + const { data: promo, error } = await admin.from("promo_codes") + .insert({ code, type: "free_month", config: { days: 30 }, max_redemptions: 1 }).select("id").single(); + if (error) throw error; + const brainstorm = async () => { + await page.goto("/app/brainstorm"); + await page.getByLabel("Motion / Topic *").fill("Public transport should be free"); + await page.getByRole("button", { name: "Start Brainstorming" }).click(); + // No Current Debate, so the motion check asks first; the paywall (or the run) follows it. + await chooseForMotion(page, "once"); + }; + try { + await brainstorm(); + await expect(page.getByRole("dialog").getByText("Free Trial Ended")).toBeVisible({ timeout: AI_TIMEOUT }); + await evidence.shot("01-blocked"); + expect((await providerCalls()).filter((call) => call.feature === "brainstorm")).toEqual([]); + + await page.goto("/app/settings"); + await page.getByRole("button", { name: "Upgrade to Pro" }).click(); + await page.getByRole("button", { name: "Have a promo code?" }).click(); + await page.getByPlaceholder("Enter promo code").fill(code); + await page.getByRole("button", { name: "Apply", exact: true }).click(); + await expect(page.getByText("Promo code applied!").first()).toBeVisible(); + await evidence.shot("02-promo-applied"); + const { data } = await admin.from("user_settings").select("subscription_tier").eq("user_id", userId).single(); + expect(data?.subscription_tier).toBe("pro"); + + const response = aiResponse(page, "brainstorm"); + await brainstorm(); + const body = await response; + expect((body.output.topFive as unknown[]).length).toBeGreaterThan(0); + await evidence.shot("03-access-restored"); + } finally { + await admin.from("promo_codes").delete().eq("id", promo.id); + } +}); diff --git a/.claude/skills/verify-debatelab/recipes/case-generation.spec.ts b/.claude/skills/verify-debatelab/recipes/case-generation.spec.ts new file mode 100644 index 000000000..5b0df3ad9 --- /dev/null +++ b/.claude/skills/verify-debatelab/recipes/case-generation.spec.ts @@ -0,0 +1,177 @@ +import { readFile } from "node:fs/promises"; +import { admin, AI_TIMEOUT, aiResponse, chooseForMotion, type Evidence, expect, expectProviderCalls, fake, LIVE, signIn, startDebate, test } from "./fixtures"; + +// Feature map: features/case-generation.md. +const motion = "This House would make public transport free at the point of use"; + +type CaseOutput = { caseStructure: { thesis: string }; contentions: Array<{ claim: string }> }; + +/** + * Starts a Current Debate for the motion from the header, as a student does + * first. Generate then prefills that motion and runs with no motion check. + */ +async function openGenerate(page: import("@playwright/test").Page) { + await startDebate(page, motion); + await page.goto("/app/generate"); + await expect(page.getByRole("heading", { name: "Ready to Generate" })).toBeVisible(); + await expect(page.getByRole("textbox", { name: "Motion / Topic *" })).toHaveValue(motion); +} + +async function generate(page: import("@playwright/test").Page) { + await openGenerate(page); + const response = aiResponse(page, "generate"); + await page.getByRole("button", { name: "Generate Debate Case" }).click(); + const body = await response; + const output = body.output as unknown as CaseOutput; + // Standard length asks for exactly three contentions. + expect(output.contentions).toHaveLength(3); + await expect(page.getByText(output.caseStructure.thesis).first()).toBeVisible(); + return { body, output }; +} + +/** Signs in on a fresh browser context, which has no local copy, so History must hydrate from the database. */ +async function reopenFromHistory(browser: import("@playwright/test").Browser, user: Parameters<typeof signIn>[1], title: string, thesis: string, evidence: Evidence, prefix: string, scope: "debate" | "one-off") { + const context = await browser.newContext(); + try { + const other = await context.newPage(); + await other.addInitScript(() => localStorage.setItem("debate-trainer-onboarding", JSON.stringify({ state: { hasCompletedOnboarding: true, isTourRunning: false }, version: 0 }))); + await signIn(other, user); + await other.goto("/app/history"); + // History shows the Current Debate's work in tabs, or, with no debate, a + // list of one-off work. Scope to it: an auto-filed case is titled with the + // motion, which the header also shows. + let entry; + if (scope === "debate") { + entry = other.getByRole("tabpanel", { name: "Cases" }); + await expect(entry.getByRole("heading", { name: title })).toBeVisible(); + } else { + await expect(other.getByRole("combobox", { name: "Show work from" })).toContainText("One-off and unsorted work"); + entry = other.getByRole("main").getByRole("listitem").filter({ hasText: title }); + await expect(entry).toBeVisible(); + } + await evidence.shot(`${prefix}-history-fresh-device`, other); + await entry.getByRole("button", { name: "Load" }).first().click(); + await expect(other.getByText(thesis).first()).toBeVisible(); + await evidence.shot(`${prefix}-reopened-from-history`, other); + await evidence.step(`Reopened "${title}" from History on a fresh device`); + } finally { + await context.close(); + } +} + +// Inside a Current Debate, Generate files the case in the debate by itself and +// shows no Save button; manual Save with a title is the one-off path (gen-save). +test("gen-create: a case generated in the Current Debate is filed there and reopens from History on a fresh device", async ({ signedIn: page, user, evidence, browser }) => { + await evidence.step(`Header → Start debate, then Generate: motion "${motion}"`); + const { body, output } = await generate(page); + await evidence.json("01-generate-response", body); + await evidence.shot("01-generated"); + await evidence.aria("01-generated"); + for (const contention of output.contentions) await expect(page.getByText(contention.claim).first()).toBeVisible(); + await expect(page.getByRole("button", { name: "Save", exact: true })).toHaveCount(0); + + const call = (await expectProviderCalls(["generate"])).find((item) => item.feature === "generate")!; + expect(call.user).toContain(`Motion: ${motion}`); + const userId = await user.userId(); + const { data: usage } = await admin.from("ai_requests").select("feature,status,model").eq("user_id", userId); + expect(usage).toContainEqual(expect.objectContaining({ feature: "generate", status: "success" })); + await evidence.json("02-ai-requests", usage); + + // The case was recorded in the debate started for its motion, without a Save click. + let title = ""; + await expect.poll(async () => { + const { data } = await admin.from("cases").select("title,side,debates(motion)").eq("user_id", userId); + title = (data?.[0]?.title as string | undefined) ?? ""; + return data?.map((row) => ({ side: row.side, debates: row.debates })); + }).toEqual([{ side: "pro", debates: { motion } }]); + expect(title.length).toBeGreaterThan(0); + await evidence.json("03-case-row", { title, motion }); + + await reopenFromHistory(browser, user, title, output.caseStructure.thesis, evidence, "04", "debate"); +}); + +test("gen-save: a one-off case is saved under a title and reopens from History on a fresh device", async ({ signedIn: page, user, evidence, browser }) => { + await page.goto("/app/generate"); + await expect(page.getByRole("heading", { name: "Ready to Generate" })).toBeVisible(); + await page.getByRole("textbox", { name: "Motion / Topic *" }).fill(motion); + const response = aiResponse(page, "generate"); + await page.getByRole("button", { name: "Generate Debate Case" }).click(); + // No Current Debate: decline the offer, so the case belongs to no debate until saved. + await chooseForMotion(page, "once"); + await evidence.step(`Generate: motion "${motion}", chose Not now, use it once`); + const output = (await response).output as unknown as CaseOutput; + await expect(page.getByText(output.caseStructure.thesis).first()).toBeVisible(); + await evidence.shot("01-generated-one-off"); + + const title = `Verify case ${Date.now()}`; + await page.getByRole("button", { name: "Save", exact: true }).click(); + await page.getByRole("dialog").getByLabel("Case title").fill(title); + await page.getByRole("dialog").getByRole("button", { name: "Save", exact: true }).click(); + await evidence.step(`Saved the case as "${title}"`); + const userId = await user.userId(); + await expect.poll(async () => { + const { data } = await admin.from("cases").select("title,debate_id").eq("user_id", userId); + return data; + }).toEqual([{ title, debate_id: null }]); + const { data: debates } = await admin.from("debates").select("id").eq("user_id", userId); + expect(debates).toEqual([]); + + await reopenFromHistory(browser, user, title, output.caseStructure.thesis, evidence, "02", "one-off"); +}); + +test("gen-research: generation researches the web and cites what it found", async ({ signedIn: page, evidence }) => { + const { body } = await generate(page); + await evidence.json("01-generate-response", body); + const sources = (body.discoveredSources ?? []) as Array<{ url: string }>; + await evidence.json("02-discovered-sources", sources); + if (!LIVE) { + // The fake stack has no Exa key, so research finds nothing. Record it; do not claim it. + test.info().annotations.push({ type: "unverified", description: "web research needs --live (Exa)" }); + return; + } + expect(sources.length, "web sources found by research").toBeGreaterThan(0); + expect(sources.every((source) => /^https?:\/\//.test(source.url))).toBe(true); + const panel = page.getByText("Researched sources").locator(".."); + await expect(panel).toBeVisible(); + await expect(panel.getByRole("link").first()).toHaveAttribute("href", /^https?:\/\//); + await evidence.shot("03-researched-sources"); +}); + +test("gen-export: a generated case exports to PDF and DOCX", async ({ signedIn: page, evidence }) => { + await generate(page); + for (const [button, extension, magic] of [["PDF", ".pdf", "%PDF"], ["DOCX", ".docx", "PK"]] as const) { + const download = page.waitForEvent("download"); + await page.getByRole("button", { name: button, exact: true }).click(); + const file = await download; + expect(file.suggestedFilename()).toMatch(new RegExp(`\\${extension}$`)); + const path = `${evidence.dir}/export${extension}`; + await file.saveAs(path); + const bytes = await readFile(path); + expect(bytes.subarray(0, magic.length).toString("latin1"), `${button} file signature`).toBe(magic); + await evidence.step(`Exported ${button}: ${file.suggestedFilename()} (${bytes.length} bytes, saved as export${extension})`); + } +}); + +test("gen-error: a provider failure shows an error and the retry succeeds", async ({ signedIn: page, evidence }) => { + await fake.fail("generate", 500); + await openGenerate(page); + await page.getByRole("button", { name: "Generate Debate Case" }).click(); + await expect(page.getByText("Generation Failed", { exact: true })).toBeVisible({ timeout: AI_TIMEOUT }); + await evidence.shot("01-failed"); + await expect(page.getByRole("button", { name: "Generate Debate Case" })).toBeEnabled(); + + const response = aiResponse(page, "generate"); + await page.getByRole("button", { name: "Generate Debate Case" }).click(); + const body = await response; + await expect(page.getByText((body.output as unknown as CaseOutput).caseStructure.thesis).first()).toBeVisible(); + await evidence.shot("02-retried"); +}); + +test("gen-quota: an account without quota is told why and the provider is never called", async ({ signedIn: page, user, evidence }) => { + await admin.from("user_settings").update({ daily_limit_override: 0 }).eq("user_id", await user.userId()); + await openGenerate(page); + await page.getByRole("button", { name: "Generate Debate Case" }).click(); + await expect(page.getByRole("dialog").getByText("Daily Access Paused")).toBeVisible({ timeout: AI_TIMEOUT }); + await evidence.shot("01-quota-dialog"); + expect((await fake.requests()).filter((request) => request.feature === "generate")).toEqual([]); +}); diff --git a/.claude/skills/verify-debatelab/recipes/current-debate.spec.ts b/.claude/skills/verify-debatelab/recipes/current-debate.spec.ts new file mode 100644 index 000000000..a7918eca8 --- /dev/null +++ b/.claude/skills/verify-debatelab/recipes/current-debate.spec.ts @@ -0,0 +1,240 @@ +import type { Page } from "@playwright/test"; +import { admin, AI_TIMEOUT, aiResponse, chooseForMotion, debateControl, expect, LIVE, type ProviderCall, providerCalls, signIn, startDebate, test } from "./fixtures"; + +// Feature map: features/current-debate.md. Mirrors e2e/full-stack/current-debate.spec.ts +// and e2e/full-stack/first-run.spec.ts with structural assertions, so it +// passes in both modes. Markers in source text prove which sources reached +// the prompt; model wording is never asserted. + +const HOMEWORK = "This House would ban homework"; +const ZOOS = "This House would abolish zoos"; +// Several AI round trips per test; live reasoning models are slow. +const LIVE_AWARE_TIMEOUT = LIVE ? 900_000 : 150_000; + +async function addSource(page: Page, title: string, content: string) { + await page.goto("/app/library"); + await page.getByRole("button", { name: "Add Source" }).click(); + await page.getByRole("textbox", { name: "Source title..." }).fill(title); + await page.getByRole("textbox", { name: "Paste source content..." }).fill(content); + await page.getByRole("button", { name: "Save to Library" }).click(); + await expect(page.getByRole("switch", { name: `Toggle ${title}` })).toBeChecked(); +} + +async function switchDebate(page: Page, motion: string) { + await debateControl(page).click(); + await page.getByRole("dialog").getByRole("button", { name: "Switch debate" }).click(); + await page.getByRole("dialog").getByRole("button", { name: `Switch to ${motion}` }).click(); + await expect(page.getByRole("dialog")).toBeHidden(); + await expect(debateControl(page)).toHaveAttribute("aria-label", `Current debate: ${motion}`); +} + +/** The newest provider call for a feature, once more than `after` of them exist. */ +async function latestCall(feature: string, after: number) { + let call: ProviderCall | undefined; + await expect.poll(async () => { + const calls = (await providerCalls()).filter((item) => item.feature === feature); + call = calls.length > after ? calls.at(-1) : undefined; + return Boolean(call); + }, { timeout: AI_TIMEOUT }).toBe(true); + return call!; +} + +async function callCount(feature: string) { + return (await providerCalls()).filter((item) => item.feature === feature).length; +} + +test("debate-start, debate-scope, debate-once, debate-switch: two debates keep their motions, sources, and results apart", async ({ signedIn: page, user, evidence }) => { + test.setTimeout(LIVE_AWARE_TIMEOUT); + const userId = await user.userId(); + + // First debate, started from the header. Its source grounds a Brainstorm that prefills the motion. + await expect(page.getByTestId("home-current-debate")).toContainText("No current debate"); + await startDebate(page, HOMEWORK); + await evidence.step(`Header → Start debate: "${HOMEWORK}"`); + await evidence.aria("01-header-after-start", debateControl(page)); + await addSource(page, "Homework study", "HOMEWORK-SOURCE pupils sleep less when homework runs late."); + + await page.goto("/app/brainstorm"); + await expect(page.getByTestId("current-debate-strip")).toContainText(`Working in: ${HOMEWORK}`); + await expect(page.getByLabel("Motion / Topic *")).toHaveValue(HOMEWORK); + let before = await callCount("brainstorm"); + const homeworkRun = aiResponse(page, "brainstorm"); + await page.getByRole("button", { name: "Start Brainstorming" }).click(); + const ideas = (await homeworkRun).output.topFive as Array<{ idea: { title: string } }>; + await expect(page.getByText(ideas[0].idea.title).first()).toBeVisible(); + await expect(page.getByRole("dialog")).toHaveCount(0); + // The client files the result after it renders; a full reload any sooner loses it. + await expect.poll(async () => (await admin.from("brainstorms").select("id").eq("user_id", userId)).data?.length).toBe(1); + const homeworkCall = await latestCall("brainstorm", before); + expect(homeworkCall.user).toContain(`Motion: ${HOMEWORK}`); + expect(homeworkCall.system).toContain("HOMEWORK-SOURCE"); + await evidence.shot("02-brainstorm-in-homework-debate"); + + // Second debate: a fresh Library and a case grounded only in its own source. + await startDebate(page, ZOOS); + await evidence.step(`Header → Start new debate: "${ZOOS}"`); + await page.goto("/app/library"); + await expect(page.getByTestId("library-debate")).toContainText(ZOOS); + await expect(page.getByRole("switch", { name: "Toggle Homework study" })).toHaveCount(0); + await addSource(page, "Zoo study", "ZOO-SOURCE conservation breeding saved several species."); + await evidence.shot("03-library-in-zoo-debate"); + + await page.goto("/app/generate"); + await expect(page.getByRole("textbox", { name: "Motion / Topic *" })).toHaveValue(ZOOS); + before = await callCount("generate"); + const zooRun = aiResponse(page, "generate"); + await page.getByRole("button", { name: "Generate Debate Case" }).click(); + const zooCase = (await zooRun).output as { caseStructure: { thesis: string } }; + await expect(page.getByText(zooCase.caseStructure.thesis).first()).toBeVisible(); + await expect.poll(async () => (await admin.from("cases").select("id").eq("user_id", userId)).data?.length).toBe(1); + const zooCall = await latestCall("generate", before); + expect(zooCall.user).toContain(`Motion: ${ZOOS}`); + expect(zooCall.system).toContain("ZOO-SOURCE"); + expect(zooCall.system).not.toContain("HOMEWORK-SOURCE"); + + // A different motion asks first; "use once" keeps it out of every debate. + await page.goto("/app/brainstorm"); + await page.getByLabel("Motion / Topic *").fill("This House would lower the voting age"); + before = await callCount("brainstorm"); + const oneOffRun = aiResponse(page, "brainstorm"); + await page.getByRole("button", { name: "Start Brainstorming" }).click(); + await expect(page.getByRole("dialog")).toContainText("This is a different motion"); + await evidence.shot("04-different-motion-choice"); + await evidence.aria("04-different-motion-choice", page.getByRole("dialog")); + await chooseForMotion(page, "once"); + await evidence.step("Different motion on Brainstorm → Use once without saving to a debate"); + const oneOffIdeas = (await oneOffRun).output.topFive as Array<{ idea: { title: string } }>; + await expect(page.getByText(oneOffIdeas[0].idea.title).first()).toBeVisible(); + const oneOffCall = await latestCall("brainstorm", before); + expect(oneOffCall.system).not.toContain("ZOO-SOURCE"); + expect(oneOffCall.system).not.toContain("HOMEWORK-SOURCE"); + await expect(debateControl(page)).toHaveAttribute("aria-label", `Current debate: ${ZOOS}`); + + // The dashboard shows only this debate's work, with both sides as equals. + await page.goto("/app/debate"); + await expect(page.getByTestId("current-debate-motion")).toHaveText(ZOOS); + await expect(page.getByTestId("debate-side-pro")).toContainText("In favor"); + await expect(page.getByTestId("debate-side-con")).toContainText("Against"); + await expect(page.getByTestId("debate-sources")).toContainText("Zoo study"); + await expect(page.getByTestId("debate-sources")).not.toContainText("Homework study"); + await expect(page.getByTestId("debate-recent")).toContainText(ZOOS); + await evidence.shot("05-zoo-dashboard"); + + // Switching back switches the Library and History with it. + await switchDebate(page, HOMEWORK); + await evidence.step(`Header → Switch debate → Switch to "${HOMEWORK}"`); + await page.goto("/app/library"); + await expect(page.getByRole("switch", { name: "Toggle Homework study" })).toBeVisible(); + await expect(page.getByRole("switch", { name: "Toggle Zoo study" })).toHaveCount(0); + await page.goto("/app/history"); + await page.getByRole("tab", { name: /Brainstorms/ }).click(); + await expect(page.getByRole("tabpanel").getByText(HOMEWORK)).toBeVisible(); + await page.getByRole("tab", { name: /Cases/ }).click(); + await expect(page.getByRole("tabpanel").getByText("No saved cases yet")).toBeVisible(); + await evidence.shot("06-history-back-in-homework"); + + // The database agrees: every row sits in exactly the debate it came from. + // Live research may save web sources too, so only the two added here are checked. + const { data: debates } = await admin.from("debates").select("id,motion").eq("user_id", userId); + expect(debates?.map((debate) => debate.motion).sort()).toEqual([ZOOS, HOMEWORK].sort()); + const idFor = (motion: string) => debates!.find((debate) => debate.motion === motion)!.id; + const { data: sources } = await admin.from("sources").select("title,debate_id").eq("user_id", userId).in("title", ["Homework study", "Zoo study"]).order("title"); + expect(sources).toEqual([ + { title: "Homework study", debate_id: idFor(HOMEWORK) }, + { title: "Zoo study", debate_id: idFor(ZOOS) }, + ]); + const { data: cases } = await admin.from("cases").select("debate_id,side").eq("user_id", userId); + expect(cases).toEqual([{ debate_id: idFor(ZOOS), side: "pro" }]); + const { data: brainstorms } = await admin.from("brainstorms").select("debate_id").eq("user_id", userId); + expect(brainstorms).toEqual([{ debate_id: idFor(HOMEWORK) }]); + const { data: current } = await admin.from("current_debates").select("debate_id").eq("user_id", userId).single(); + expect(current?.debate_id).toBe(idFor(HOMEWORK)); + await evidence.json("07-rows", { debates, sources, cases, brainstorms, current }); +}); + +test("debate-offer: with no Current Debate, a tool offers to start one for the motion it was given", async ({ signedIn: page, user, evidence }) => { + const motion = "This House would abolish zoos"; + await page.goto("/app/generate"); + await page.getByRole("textbox", { name: "Motion / Topic *" }).fill(motion); + const run = aiResponse(page, "generate"); + await page.getByRole("button", { name: "Generate Debate Case" }).click(); + + const offer = page.getByRole("dialog", { name: "Start a debate with this motion?" }); + await expect(offer).toContainText(motion); + await evidence.shot("01-offer"); + await evidence.aria("01-offer", offer); + await offer.getByRole("button", { name: "Start debate" }).click(); + await expect(offer).toBeHidden(); + await evidence.step("Generate with no Current Debate → Start a debate with this motion? → Start debate"); + + const body = await run; + const thesis = (body.output as { caseStructure: { thesis: string } }).caseStructure.thesis; + await expect(page.getByText(thesis).first()).toBeVisible(); + await expect(page.getByTestId("current-debate-strip")).toContainText(`Working in: ${motion}`); + await expect(debateControl(page)).toHaveAttribute("aria-label", `Current debate: ${motion}`); + await evidence.shot("02-generated-in-new-debate"); + + const userId = await user.userId(); + const { data: debates } = await admin.from("debates").select("id,motion").eq("user_id", userId); + expect(debates?.map((debate) => debate.motion)).toEqual([motion]); + const { data: current } = await admin.from("current_debates").select("debate_id").eq("user_id", userId).single(); + expect(current?.debate_id).toBe(debates![0].id); + await expect.poll(async () => { + const { data } = await admin.from("cases").select("debate_id").eq("user_id", userId); + return data?.map((row) => row.debate_id); + }).toEqual([debates![0].id]); +}); + +test("debate-tour: a brand-new account starts its first debate from the onboarding tour", async ({ page, user, evidence }) => { + test.setTimeout(LIVE_AWARE_TIMEOUT); + // The shared fixture marks the tour as seen. A new student has not seen it, so + // override that once per browser session, as e2e/full-stack/first-run.spec.ts does. + await page.addInitScript(() => { + if (sessionStorage.getItem("first-run-seeded")) return; + sessionStorage.setItem("first-run-seeded", "1"); + localStorage.setItem("debate-trainer-onboarding", JSON.stringify({ state: { hasCompletedOnboarding: false, isTourRunning: false }, version: 0 })); + }); + await signIn(page, user); + + const tooltip = page.locator(".react-joyride__tooltip"); + const next = page.locator('[data-test-id="button-primary"]'); + await expect(tooltip).toContainText("Welcome to DebateLab!", { timeout: 15_000 }); + await expect(tooltip).toContainText("Start your first debate here."); + await evidence.shot("01-tour-first-step"); + + const motion = "This House would ban homework"; + await tooltip.getByRole("button", { name: "Start debate" }).click(); + const dialog = page.getByRole("dialog"); + await dialog.getByLabel("Motion").fill(motion); + await dialog.getByRole("button", { name: "Start debate" }).click(); + await expect(dialog).toBeHidden(); + await evidence.step(`Tour step 1 → Start debate: "${motion}"`); + + await expect(tooltip).toContainText("Library"); + await next.click(); + await expect(tooltip).toContainText("Generate"); + await next.click(); + await page.getByRole("button", { name: "Get Started!" }).click(); + await expect(tooltip).toHaveCount(0); + await evidence.step("Tour: Library → Generate → Get Started!"); + + const userId = await user.userId(); + const { data: debates } = await admin.from("debates").select("id,motion").eq("user_id", userId); + expect(debates?.map((debate) => debate.motion)).toEqual([motion]); + const { data: current } = await admin.from("current_debates").select("debate_id").eq("user_id", userId).single(); + expect(current?.debate_id).toBe(debates![0].id); + + // The first tool already works inside that debate, with no motion check. + await page.locator('main [data-tour="mode-generate"]').click(); + await expect(page.getByTestId("current-debate-strip")).toContainText(`Working in: ${motion}`); + await expect(page.getByRole("textbox", { name: "Motion / Topic *" })).toHaveValue(motion); + const run = aiResponse(page, "generate"); + await page.getByRole("button", { name: "Generate Debate Case" }).click(); + await run; + await expect(page.getByRole("dialog")).toHaveCount(0); + await evidence.shot("02-first-case-in-tour-debate"); + await expect.poll(async () => { + const { data } = await admin.from("cases").select("debate_id").eq("user_id", userId); + return data?.map((row) => row.debate_id); + }).toEqual([debates![0].id]); +}); diff --git a/.claude/skills/verify-debatelab/recipes/fixtures.ts b/.claude/skills/verify-debatelab/recipes/fixtures.ts new file mode 100644 index 000000000..3a87ea4e8 --- /dev/null +++ b/.claude/skills/verify-debatelab/recipes/fixtures.ts @@ -0,0 +1,139 @@ +import { mkdir, appendFile, writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import type { Locator, Page } from "@playwright/test"; +import { expect, fake, stackReady, test as stackTest } from "../../../../e2e/full-stack/stack"; + +// Recipes reuse the full-stack gate's fixtures: a fresh demo-code user per test +// (deleted afterwards), `signedIn`, the provider proxy, and the service-role +// `admin` client for reading rows. This file adds a remote-project guard, an +// evidence writer, and mode-aware provider checks. Recipes must pass in both +// modes, so assert structure (response shape, rows, visible sections), never +// model wording. +export * from "../../../../e2e/full-stack/stack"; +// Scripted microphone, recogniser, and browser voice for spoken rounds. +export { say, spokenByBrowser, stubSpeech } from "../../../../e2e/helpers/voice"; + +if (!stackReady) throw new Error("Stack environment missing; run recipes through `verify.sh drive`"); + +export const LIVE = process.env.VERIFY_MODE === "live"; +export const LIVE_MODEL = process.env.LIVE_MODEL ?? ""; +/** Per-step wait for one AI round trip. Live reasoning models are slow. */ +export const AI_TIMEOUT = LIVE ? 240_000 : 30_000; + +export type ProviderCall = { + feature: string | null; + model?: string; + requestedModel?: string; + upstreamModel?: string; + upstreamProvider?: string; + status?: number; + ms?: number; + stream?: boolean; + system?: string; + user?: string; +}; + +export async function providerCalls(): Promise<ProviderCall[]> { + return (await fake.requests()) as ProviderCall[]; +} + +/** Resolves with the parsed body of the next `ai` Edge Function response for a feature. */ +export function aiResponse(page: Page, feature: string, operation?: string) { + return page.waitForResponse((response) => { + if (!response.url().endsWith("/functions/v1/ai") || response.request().method() !== "POST") return false; + try { + const body = response.request().postDataJSON() as { feature?: string; input?: { operation?: string } }; + return body.feature === feature && (!operation || body.input?.operation === operation); + } catch { + return false; + } + }, { timeout: AI_TIMEOUT }).then(async (response) => { + expect(response.status(), `ai ${feature} status`).toBe(200); + return response.json() as Promise<{ output: Record<string, unknown>; discoveredSources?: unknown[]; [key: string]: unknown }>; + }); +} + +/** The provider was called for each feature, and every call succeeded. */ +export async function expectProviderCalls(features: string[]) { + const calls = await providerCalls(); + for (const feature of features) { + expect(calls.map((call) => call.feature), `provider calls include ${feature}`).toContain(feature); + } + return calls; +} + +export type Evidence = { + dir: string; + /** Full-page screenshot, `<name>.png`. */ + shot(name: string, target?: Page): Promise<string>; + /** ARIA snapshot of a region (default: whole page), `<name>.aria.yml`. */ + aria(name: string, locator?: Locator): Promise<string>; + /** Any JSON value, such as responses or database rows, `<name>.json`. */ + json(name: string, value: unknown): Promise<string>; + /** One line in `steps.log` describing the user action just taken. */ + step(line: string): Promise<void>; +}; + +export const test = stackTest.extend<{ evidence: Evidence }>({ + // Overriding `page` also covers `signedIn`, which navigates during setup. + page: async ({ page }, provide) => { + const remote: string[] = []; + await page.context().route(/^https:\/\/[^/]+\.supabase\.co\//, (route) => { + remote.push(route.request().url()); + return route.abort(); + }); + await provide(page); + expect(remote, "requests to a remote Supabase project").toEqual([]); + }, + evidence: async ({ page }, provide, testInfo) => { + const slug = testInfo.title.toLowerCase().replace(/[^a-z0-9]+/g, "-").replace(/^-|-$/g, "").slice(0, 80); + const dir = join(process.env.VERIFY_ARTIFACT_DIR!, testInfo.project.name, slug); + await mkdir(dir, { recursive: true }); + const file = (name: string) => join(dir, name); + await writeFile(file("run.json"), JSON.stringify({ + test: testInfo.title, + file: testInfo.file, + project: testInfo.project.name, + commit: process.env.VERIFY_HEAD, + mode: LIVE ? "live" : "fake", + liveModel: LIVE ? LIVE_MODEL : null, + baseURL: testInfo.project.use.baseURL, + startedAt: new Date().toISOString(), + }, null, 2)); + await provide({ + dir, + async shot(name, target = page) { + const path = file(`${name}.png`); + await target.screenshot({ path, fullPage: true }); + return path; + }, + async aria(name, locator = page.locator("body")) { + const path = file(`${name}.aria.yml`); + await writeFile(path, `# ${page.url()}\n${await locator.ariaSnapshot()}\n`); + return path; + }, + async json(name, value) { + const path = file(`${name}.json`); + await writeFile(path, JSON.stringify(value, null, 2)); + return path; + }, + async step(line) { + await appendFile(file("steps.log"), `${new Date().toISOString()} ${line}\n`); + }, + }); + + // Every test records what reached the provider. In live mode this is the + // proof of spend and pinning: OpenRouter's own reply names the model. + const calls = await providerCalls(); + await writeFile(file("provider-calls.json"), JSON.stringify(calls.map(({ system, user, ...call }) => ({ + ...call, + systemChars: system?.length, + userHead: user?.slice(0, 300), + })), null, 2)); + if (LIVE) { + const answered = calls.filter((call) => call.feature !== "jev" && call.status === 200); + const offPin = answered.filter((call) => call.model !== LIVE_MODEL || !call.upstreamModel?.startsWith(LIVE_MODEL)); + expect(offPin.map((call) => `${call.feature}: sent ${call.model}, answered by ${call.upstreamModel}`), "live chat calls answered by the pinned model").toEqual([]); + } + }, +}); diff --git a/.claude/skills/verify-debatelab/recipes/library.spec.ts b/.claude/skills/verify-debatelab/recipes/library.spec.ts new file mode 100644 index 000000000..879b2e713 --- /dev/null +++ b/.claude/skills/verify-debatelab/recipes/library.spec.ts @@ -0,0 +1,70 @@ +import type { Page } from "@playwright/test"; +import { admin, AI_TIMEOUT, aiResponse, expect, expectProviderCalls, startDebate, test } from "./fixtures"; + +// Feature map: features/library-sources.md. The Library belongs to the Current +// Debate: `Add Source` stays disabled until one exists, so each test starts one. +const motion = "Public transport should be free"; + +async function addSource(page: Page, title: string, content: string) { + await page.getByRole("button", { name: "Add Source" }).click(); + await page.getByRole("textbox", { name: "Source title..." }).fill(title); + await page.getByRole("textbox", { name: "Paste source content..." }).fill(content); + await page.getByRole("button", { name: "Save to Library" }).click(); + await expect(page.getByRole("switch", { name: `Toggle ${title}` })).toBeChecked(); +} + +test("lib-add, lib-toggle, lib-ground, lib-delete: only active sources ground AI requests, and deletions persist", async ({ signedIn: page, user, evidence }) => { + await startDebate(page, motion); + await evidence.step(`Started the debate "${motion}" from the header`); + await page.goto("/app/library"); + await expect(page.getByTestId("library-debate")).toContainText(motion); + await addSource(page, "Active study", "ACTIVE-SOURCE-MARKER ridership rose after fares were removed in Tallinn."); + await addSource(page, "Paused study", "PAUSED-SOURCE-MARKER congestion did not change."); + await page.getByRole("switch", { name: "Toggle Paused study" }).click(); + await expect(page.getByRole("switch", { name: "Toggle Paused study" })).not.toBeChecked(); + await evidence.step("Added two sources and paused one"); + await evidence.shot("01-library"); + + // Brainstorm prefills the Current Debate's motion, so no motion check opens. + await page.goto("/app/brainstorm"); + await expect(page.getByLabel("Motion / Topic *")).toHaveValue(motion); + const response = aiResponse(page, "brainstorm"); + await page.getByRole("button", { name: "Start Brainstorming" }).click(); + await response; + const call = (await expectProviderCalls(["brainstorm"])).find((item) => item.feature === "brainstorm")!; + expect(call.system).toContain("ACTIVE-SOURCE-MARKER"); + expect(call.system).not.toContain("PAUSED-SOURCE-MARKER"); + await evidence.json("02-grounding", { activeInPrompt: true, pausedInPrompt: false }); + + await page.goto("/app/library"); + await page.getByRole("button", { name: "Delete Paused study", exact: true }).click(); + await expect(page.getByRole("switch", { name: "Toggle Paused study" })).toHaveCount(0); + await page.reload(); + await expect(page.getByRole("switch", { name: "Toggle Active study" })).toBeVisible(); + await expect(page.getByRole("switch", { name: "Toggle Paused study" })).toHaveCount(0); + // Live research can save the web sources it found, so check only the two added here. + const userId = await user.userId(); + const { data: debate } = await admin.from("debates").select("id").eq("user_id", userId).eq("motion", motion).single(); + const { data } = await admin.from("sources").select("title,is_active,debate_id").eq("user_id", userId).in("title", ["Active study", "Paused study"]); + expect(data).toEqual([{ title: "Active study", is_active: true, debate_id: debate?.id }]); + await evidence.json("03-sources-rows", data); + await evidence.shot("03-after-delete"); +}); + +test("lib-import: a web page imports into the Library", async ({ signedIn: page, user, evidence }) => { + const url = "https://en.wikipedia.org/wiki/Free_public_transport"; + await startDebate(page, motion); + await page.goto("/app/library"); + await page.getByRole("button", { name: "Add Source" }).click(); + await page.getByRole("textbox", { name: "Import from URL" }).fill(url); + await page.getByRole("button", { name: "Import URL" }).click(); + await expect(page.getByText("Source Ready", { exact: true })).toBeVisible({ timeout: AI_TIMEOUT }); + await evidence.shot("01-import-ready"); + await page.getByRole("button", { name: "Save imported source" }).click(); + await expect.poll(async () => { + const { data } = await admin.from("sources").select("origin,content").eq("user_id", await user.userId()); + return data?.map((row) => ({ origin: row.origin, long: row.content.length > 2_000 })); + }).toEqual([{ origin: url, long: true }]); + const { data } = await admin.from("sources").select("title,origin").eq("user_id", await user.userId()); + await evidence.json("02-imported-row", data); +}); diff --git a/.claude/skills/verify-debatelab/recipes/practice.spec.ts b/.claude/skills/verify-debatelab/recipes/practice.spec.ts new file mode 100644 index 000000000..ded9dda58 --- /dev/null +++ b/.claude/skills/verify-debatelab/recipes/practice.spec.ts @@ -0,0 +1,107 @@ +import { admin, AI_TIMEOUT, aiResponse, chooseForMotion, expect, expectProviderCalls, test } from "./fixtures"; + +// Feature map: features/practice.md. Voice rounds are recipes/voice-rounds.spec.ts. +// A new account has no Current Debate, so each round first accepts the offer +// to start one for its motion; the finished round is filed in that debate. +const sparMotion = "This House would make public transport free at the point of use."; +const resolution = "Resolved: cities should ban private cars downtown."; + +test("spar-round and spar-verdict: a Lincoln-Douglas round is argued, judged, and saved", async ({ signedIn: page, user, evidence }) => { + await page.goto("/app/sparring"); + await page.getByLabel("Motion").fill(sparMotion); + await page.getByRole("combobox", { name: "Round format" }).click(); + await page.getByRole("option", { name: /Lincoln-Douglas · 5 speeches/ }).click(); + await page.getByRole("button", { name: "Start round" }).click(); + await chooseForMotion(page, "create"); + await evidence.step("Sparring: started a Lincoln-Douglas round and a debate for its motion"); + + const speeches = [ + ["Affirmative constructive", "Universal access removes the direct price barrier to participation in work, school, and civic life."], + ["First affirmative rebuttal", "Targeted discounts leave eligibility gaps and add administrative friction that keeps poor riders out."], + ["Second affirmative rebuttal", "Weigh reliability: the universal policy covers every marginal trip, while targeting misses many of them."], + ] as const; + for (const [turn, content] of speeches) { + const draft = page.getByRole("textbox", { name: `Draft for ${turn}` }); + // The opponent speaks between the user's turns; the next draft box appears after it. + await expect(draft).toBeVisible({ timeout: AI_TIMEOUT }); + await draft.fill(content); + await page.getByRole("button", { name: "Submit speech" }).click(); + await evidence.step(`Submitted ${turn}`); + } + await evidence.shot("01-round-in-progress"); + + await expect(page.getByRole("heading", { name: "Here is what happened in the room." })).toBeVisible({ timeout: AI_TIMEOUT * 2 }); + await evidence.shot("02-verdict"); + await evidence.aria("02-verdict"); + await expect.poll(async () => { + const { data } = await admin.from("sparring_sessions").select("status").eq("user_id", await user.userId()); + return data?.map((row) => row.status); + }).toEqual(["complete"]); + const { data: session } = await admin.from("sparring_sessions").select("status,side,final_evaluation,debates(motion)").eq("user_id", await user.userId()).single(); + await evidence.json("03-sparring-session", session); + // The judged round is filed in the debate created for its motion, with the user's side. + expect(session).toMatchObject({ side: "pro", debates: { motion: sparMotion } }); + const calls = await expectProviderCalls(["sparring"]); + expect(calls.some((call) => call.feature === "sparring" && /SPARRING ACTION: score/m.test(call.user ?? ""))).toBe(true); +}); + +async function startCrossfire(page: import("@playwright/test").Page, mode: "mode-question-ai" | "mode-ai-questions-user") { + await page.goto("/app/crossfire"); + await page.getByTestId("crossfire-resolution").fill(resolution); + await page.getByTestId("crossfire-argument").fill("Private cars create congestion and unsafe streets."); + await page.getByTestId(mode).click(); + await page.getByTestId("start-crossfire").click(); + await chooseForMotion(page, "create"); +} + +/** The finished exchange is recorded in the debate created for its resolution. */ +async function expectCrossfireSaved(userId: string) { + await expect.poll(async () => { + const { data } = await admin.from("crossfires").select("title,debates(motion)").eq("user_id", userId); + return data; + }).toEqual([{ title: resolution, debates: { motion: resolution } }]); +} + +test("cross-turn and cross-debrief: questioning the AI gets an answer, a Jev signal, and a debrief", async ({ signedIn: page, user, evidence }) => { + await startCrossfire(page, "mode-question-ai"); + await page.getByTestId("crossfire-input").fill("What evidence links fewer private cars to safer streets?"); + const turn = aiResponse(page, "crossfire"); + await page.getByTestId("submit-turn").click(); + const body = await turn; + await evidence.json("01-crossfire-turn", body); + const text = (body.output as { text?: string }).text; + expect(text?.length).toBeGreaterThan(0); + await expect(page.getByText(text!.slice(0, 40)).first()).toBeVisible(); + await evidence.shot("02-answer"); + await expectProviderCalls(["crossfire", "jev"]); + + await page.getByTestId("end-crossfire").click(); + await expect(page.getByTestId("crossfire-debrief")).toBeVisible({ timeout: AI_TIMEOUT }); + await evidence.shot("03-debrief"); + await evidence.aria("03-debrief", page.getByTestId("crossfire-debrief")); + await expectCrossfireSaved(await user.userId()); +}); + +test("cross-ai-questions: the AI questions the user, answers are judged, and the debrief completes", async ({ signedIn: page, user, evidence }) => { + const opening = aiResponse(page, "crossfire", "next-turn"); + await startCrossfire(page, "mode-ai-questions-user"); + const question = (await opening).output as { text: string }; + await expect(page.getByText(question.text.slice(0, 40)).first()).toBeVisible(); + await evidence.json("01-ai-question", question); + await evidence.shot("01-ai-question"); + + const answer = "Downtown crash data fell after car-free zones were introduced in several cities."; + await page.getByTestId("crossfire-input").fill(answer); + const next = aiResponse(page, "crossfire", "next-turn"); + await page.getByTestId("submit-turn").click(); + const followUp = (await next).output as { text: string }; + await expect(page.getByText(followUp.text.slice(0, 40)).first()).toBeVisible(); + await evidence.step(`Answered: "${answer}"; the AI asked a follow-up`); + + const debrief = aiResponse(page, "crossfire", "debrief"); + await page.getByTestId("end-crossfire").click(); + await evidence.json("02-debrief", (await debrief).output); + await expect(page.getByTestId("crossfire-debrief")).toBeVisible({ timeout: AI_TIMEOUT }); + await evidence.shot("02-debrief"); + await expectCrossfireSaved(await user.userId()); +}); diff --git a/.claude/skills/verify-debatelab/recipes/voice-rounds.spec.ts b/.claude/skills/verify-debatelab/recipes/voice-rounds.spec.ts new file mode 100644 index 000000000..3f78b3bef --- /dev/null +++ b/.claude/skills/verify-debatelab/recipes/voice-rounds.spec.ts @@ -0,0 +1,169 @@ +import { admin, AI_TIMEOUT, aiResponse, chooseForMotion, expect, expectProviderCalls, LIVE, providerCalls, say, spokenByBrowser, stubSpeech, test } from "./fixtures"; + +// Feature map: features/voice-rounds.md. Mirrors e2e/full-stack/voice.spec.ts +// with structural assertions, so it passes in both modes. The local stack has +// no ElevenLabs key: the voice function answers 503 once and the browser voice +// reads every reply. The recogniser and the browser voice are scripted +// (e2e/helpers/voice.ts); everything behind the browser is real. + +type VoiceSegment = { text: string; token: string }; +type AiBody = { output?: { speech?: { content?: string }; text?: string }; voice?: { segments: VoiceSegment[] } }; + +const squash = (text: string) => text.replace(/\s+/g, " ").trim(); + +test("voice-spar: a spoken sparring round runs on the browser voice, and no audio tag is shown or saved", async ({ signedIn: page, user, evidence }) => { + // Init scripts apply from the next load, which is the goto below. + await stubSpeech(page); + + const voiceAnswers: Array<{ status: number; code?: string }> = []; + page.on("response", async (response) => { + if (response.request().method() !== "POST" || !new URL(response.url()).pathname.endsWith("/functions/v1/voice")) return; + const body = await response.json().catch(() => ({})); + voiceAnswers.push({ status: response.status(), code: body.code }); + }); + + const motion = "This House would make public transport free at the point of use."; + await page.goto("/app/sparring"); + await page.getByLabel("Motion").fill(motion); + await page.getByRole("combobox", { name: "Round format" }).click(); + await page.getByRole("option", { name: /Lincoln-Douglas · 5 speeches/ }).click(); + await page.getByTestId("voice-mode").click(); + await page.getByRole("button", { name: "Turn on voice" }).click(); + await page.getByRole("button", { name: "Start round" }).click(); + await chooseForMotion(page, "create"); + await evidence.step("Sparring: voice mode on, started a Lincoln-Douglas round and a debate for its motion"); + + const speeches = [ + ["Affirmative constructive", "Um, universal access removes the direct price barrier to participation."], + ["First affirmative rebuttal", "Targeted discounts preserve eligibility gaps and administrative friction."], + ["Second affirmative rebuttal", "The universal policy covers every marginal trip, so it is more reliable."], + ] as const; + const replies: string[] = []; + const segments: VoiceSegment[] = []; + for (const [turn, words] of speeches) { + const draft = page.getByRole("textbox", { name: `Draft for ${turn}` }); + await expect(draft).toBeVisible({ timeout: AI_TIMEOUT }); + await page.getByTestId("voice-mic").click(); + await expect(page.getByText("Listening")).toBeVisible(); + await say(page, words); + await expect(draft).toHaveValue(words); + const last = turn === "Second affirmative rebuttal"; + const opponent = last ? null : aiResponse(page, "sparring"); + await page.getByRole("button", { name: "Submit speech" }).click(); + await evidence.step(`Spoke and submitted ${turn}: "${words}"`); + if (!opponent) continue; + const body = (await opponent) as AiBody; + const reply = body.output?.speech?.content ?? ""; + expect(reply.length, "opponent speech text").toBeGreaterThan(0); + replies.push(reply); + segments.push(...(body.voice?.segments ?? [])); + await expect(page.getByText(reply.slice(0, 40)).last()).toBeVisible(); + // Let the opponent finish. Turning the microphone on any sooner speaks over + // the opponent, which stops its voice and abandons the voice request. + await expect.poll(() => voiceAnswers.length, { timeout: AI_TIMEOUT }).toBe(1); + await expect.poll(async () => squash(await spokenByBrowser(page)), { timeout: AI_TIMEOUT }).toBe(squash(replies.join(" "))); + await evidence.step(`The browser voice finished reading opponent reply ${replies.length}`); + } + await evidence.shot("01-round-spoken"); + + await expect(page.getByRole("heading", { name: "Here is what happened in the room." })).toBeVisible({ timeout: AI_TIMEOUT * 2 }); + await evidence.shot("02-verdict"); + + // The server signed segments for the voice service; the page and the browser voice got clean text. + expect(segments.length, "signed voice segments").toBeGreaterThan(0); + for (const segment of segments) expect(segment.token).toMatch(/^\d+\.[a-f0-9]{64}$/); + await evidence.json("03-voice-segments", segments); + if (!LIVE) { + // The fake opponent uses one allowed tag and one invented tag; only the allowed one is signed. + expect(segments[0].text).toBe(`[sighs] ${replies[0]}`); + } + // No key: the voice function said so once, and the browser voice read every reply. + expect(voiceAnswers).toEqual([{ status: 503, code: "voice_unavailable" }]); + expect(await spokenByBrowser(page)).not.toContain("["); + + // The opponent was told the round is spoken; the judge got measured facts and the rule against claiming to have heard it. + const calls = await expectProviderCalls(["sparring"]); + expect(calls.some((call) => call.feature === "sparring" && call.system?.includes("SPOKEN ROUND:"))).toBe(true); + const judge = calls.find((call) => call.feature === "sparring" && /SPARRING ACTION: score/m.test(call.user ?? "")); + expect(judge?.system).toContain("You did not hear this round"); + expect(judge?.user).toContain('"wordsPerMinute"'); + + // Scorecard: measured delivery facts, and no audio tag anywhere on the page. + await expect(page.getByTestId("spoken-delivery")).toContainText("Nobody listened for tone, confidence or volume"); + await page.getByRole("button", { name: "Show transcript" }).click(); + await expect(page.locator("body")).not.toContainText("[sighs]"); + await expect(page.locator("body")).not.toContainText("[dramatic gasp]"); + await evidence.aria("04-scorecard", page.getByTestId("spoken-delivery")); + + // The saved round holds clean opponent text and the user's measurements, in the debate for its motion. + const userId = await user.userId(); + await expect.poll(async () => { + const { data } = await admin.from("sparring_sessions").select("status").eq("user_id", userId); + return data?.map((row) => row.status); + }).toEqual(["complete"]); + const { data: saved } = await admin.from("sparring_sessions").select("turns,debates(motion)").eq("user_id", userId).single(); + await evidence.json("05-sparring-session", saved); + const turns = saved!.turns as Array<{ speaker: string; content: string; delivery?: { wordCount: number; fillerCount: number } }>; + expect(turns.filter((turn) => turn.speaker === "ai").map((turn) => turn.content)).toEqual(replies); + expect(turns[0].delivery).toMatchObject({ wordCount: 10, fillerCount: 1 }); + expect(turns[1].delivery).toBeUndefined(); + expect(saved!.debates).toEqual({ motion }); + + // No provider voice was used, so no voice usage was billed. + const { data: usage } = await admin.from("ai_requests").select("feature").eq("user_id", userId).like("feature", "voice_%"); + expect(usage).toEqual([]); +}); + +test("voice-cross: a spoken Crossfire question gets a clean, signed reply read by the browser voice", async ({ signedIn: page, user, evidence }) => { + await stubSpeech(page); + + const resolution = "Resolved: cities should ban private cars downtown."; + await page.goto("/app/crossfire"); + await page.getByTestId("crossfire-resolution").fill(resolution); + await page.getByTestId("crossfire-argument").fill("Private cars create congestion and unsafe streets."); + await page.getByTestId("mode-question-ai").click(); + await page.getByTestId("voice-mode").click(); + await page.getByRole("button", { name: "Turn on voice" }).click(); + await page.getByTestId("start-crossfire").click(); + await chooseForMotion(page, "create"); + + const question = "What evidence supports the safety mechanism?"; + await page.getByTestId("voice-mic").click(); + await expect(page.getByText("Listening")).toBeVisible(); + await say(page, question); + await expect(page.getByTestId("crossfire-input")).toHaveValue(question); + const turn = aiResponse(page, "crossfire"); + await page.getByTestId("submit-turn").click(); + await evidence.step(`Crossfire: voice on, asked "${question}" through the microphone`); + + const body = (await turn) as AiBody; + await evidence.json("01-crossfire-reply", body); + const answer = body.output?.text ?? ""; + expect(answer.length, "crossfire answer text").toBeGreaterThan(0); + expect(answer).not.toContain("["); + await expect(page.getByText(answer.slice(0, 40)).first()).toBeVisible(); + const segments = body.voice?.segments ?? []; + expect(segments.length, "signed voice segments").toBeGreaterThan(0); + for (const segment of segments) expect(segment.token).toMatch(/^\d+\.[a-f0-9]{64}$/); + if (!LIVE) expect(segments[0].text).toBe(`[scoffs] ${answer}`); + await expect.poll(async () => squash(await spokenByBrowser(page)), { timeout: AI_TIMEOUT }).toBe(squash(answer)); + await expect(page.locator("body")).not.toContainText("[scoffs]"); + await evidence.shot("02-spoken-answer"); + + const call = (await providerCalls()).find((item) => item.feature === "crossfire"); + expect(call?.user).toContain("SPOKEN EXCHANGE:"); + expect(call?.user).toContain(question); + // The Jev signal still attaches: the signed reply is the clean one. + await expectProviderCalls(["crossfire", "jev"]); + if (!LIVE) await expect(page.getByText("Direct answer")).toBeVisible(); + + await page.getByTestId("end-crossfire").click(); + await expect(page.getByTestId("crossfire-debrief")).toBeVisible({ timeout: AI_TIMEOUT }); + await expect(page.getByTestId("spoken-delivery")).toBeVisible(); + await evidence.shot("03-debrief"); + await evidence.aria("03-debrief", page.getByTestId("crossfire-debrief")); + await expect.poll(async () => { + const { data } = await admin.from("crossfires").select("title,debates(motion)").eq("user_id", await user.userId()); + return data; + }).toEqual([{ title: resolution, debates: { motion: resolution } }]); +}); diff --git a/.gitignore b/.gitignore index f770fde4c..038649d67 100644 --- a/.gitignore +++ b/.gitignore @@ -31,6 +31,9 @@ coverage/ .vite/ .playwright-cli/ +# Verification skill state, logs, and evidence +.verify/ + # Playwright artifacts test-results/ playwright-report/ @@ -52,8 +55,9 @@ deno.lock # Personal Superset workspace overrides .superset/config.local.json -# Local agent/task files -.claude/ +# Local agent/task files (project skills under .claude/skills are tracked) +.claude/* +!.claude/skills/ .codex/ tasks.txt humanizer.md @@ -67,6 +71,7 @@ humanizer.md !**/README.md !docs/**/*.md !plans/**/*.md +!.claude/skills/**/*.md .vercel .env* .worktrees/ diff --git a/AGENTS.md b/AGENTS.md index f01992aea..871b4195a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -75,6 +75,36 @@ npx playwright install --with-deps webkit CI=1 PORT=18080 PLAYWRIGHT_BASE_URL=http://127.0.0.1:18080 npm run test:e2e:webkit ``` +## Verifying features in the real app + +To prove that a user-facing change works, drive the real app with the `verify-debatelab` skill ([SKILL.md](.claude/skills/verify-debatelab/SKILL.md)). It runs the React client, local Supabase Auth, Postgres, and Edge Functions, and signs in real demo users. Each run keeps screenshots, ARIA snapshots, provider calls, and database rows in `.verify/artifacts/`. + +```bash +H=.claude/skills/verify-debatelab/bin/verify.sh +$H up # fake model provider, free +$H up --live # real AI: OpenRouter pinned to meta/muse-spark-1.3-contributor, real Exa +$H doctor # proves the stack is local, ours, and (live) pinned +$H drive all # every recipe; or recipes/<name>, e2e/<path>; add --project desktop-chromium +$H down # always, including after failures +``` + +- Use `--live` when the change touches prompts, model output handling, web research, or citations. The fake mode cannot prove those. URL import fetches pages directly and works the same in both modes. Every chat call is rewritten to the Muse Spark contributor model, so a full run costs cents plus Exa searches, and the proxy stops after 400 upstream calls (`--max-calls N`). Only use made-up debate content: the contributor model may keep prompts. +- Live keys come from `~/.config/debatelab/live-verify.env` (mode 600, `LIVE_OPENROUTER_API_KEY` and `LIVE_EXA_API_KEY`). On Adam's VPS it holds copies of the Hermes keys in `~/.hermes/.env`. Never print them. +- The stack holds the same lock as `./.superset/db-gate.sh`, so only one worktree can run it at a time. +- Most tools work inside a Current Debate. Recipes start one with `startDebate` or answer the motion dialog with `chooseForMotion`; see the skill's Drive section. +- Neither mode proves Stripe Checkout, real Stripe cancellation, ElevenLabs voice audio, Google sign-in or export, staging, or production. + +### Keep the skill working + +The skill is maintained code. A change that breaks it or makes it stale must update it in the same PR. That includes a new dialog in a flow, renamed labels or test IDs, moved routes, new stack services, and a new or removed user-facing feature. Specifically: + +- Fix the affected recipes in `.claude/skills/verify-debatelab/recipes/`, and add a recipe for a new feature. +- Update the matching file in `features/` and its index in `features/README.md`. +- Update `SKILL.md` wherever its commands, selectors, or limits no longer match the app. +- Run `$H drive recipes/<name>` for every recipe you touched, on both `desktop-chromium` and `mobile-chromium`, and report the result in the PR. A recipe that no longer passes is a regression, the same as a failing test. + +Never weaken a recipe's assertions to make it pass. + ## Hard boundaries - Start bug fixes with a failing test or repeatable browser reproduction when a suitable test layer exists. diff --git a/docs/testing.md b/docs/testing.md index c7a5b5211..7f4b0afb6 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -196,6 +196,10 @@ Tests assert the rendered result, what the Edge Function sent upstream (read bac These journeys do not prove live provider behavior, web research (the local stack has no search key), real Jev judgement (the fake answers every Jev question with a fixed confident choice), Stripe Checkout, the billing portal, and subscription cancellation during account deletion (they call the Stripe API; the local key is a placeholder, and the Deno tests cover the cancellation logic), the `checkout.session.completed` and `invoice.payment_succeeded` webhooks (they fetch the subscription from Stripe), or Google export. +### Interactive verification stack + +To check one feature by hand, or keep evidence for a change, use the same stack as a long-lived instance through [`.claude/skills/verify-debatelab/`](../.claude/skills/verify-debatelab/SKILL.md). `bin/verify.sh up` starts it, and `doctor` proves the client resolved to the local project. `drive <all|recipes/<name>|e2e/<path>>` runs Playwright recipes or maintained specs and keeps screenshots, ARIA snapshots, provider calls and database rows in `.verify/artifacts/`. `down` stops everything. It holds the same `~/.superset/debatelab-db.lock` as the gate, so the gate waits while it is up. By default it uses the fake provider and proves the same things as the full-stack journeys. `up --live` turns the fake provider into a capped recording proxy to real OpenRouter that pins every chat call to `meta/muse-spark-1.3-contributor`, and gives Edge Functions a real Exa key; it reads keys from `~/.config/debatelab/live-verify.env`, costs money, and needs Adam's approval. Neither mode proves Stripe Checkout, Google export, staging, or production. + ### Live provider journeys ```bash diff --git a/e2e/full-stack/fake-openrouter.mjs b/e2e/full-stack/fake-openrouter.mjs index 1583e2611..9ff7c65bb 100644 --- a/e2e/full-stack/fake-openrouter.mjs +++ b/e2e/full-stack/fake-openrouter.mjs @@ -223,6 +223,13 @@ async function proxy(req, res, path) { : { feature: "jev", model: body.model }; delete record.output; requests.push(record); + // Forced failures (fake.fail) apply in live mode too, before any spend. + const failure = chat && failures.find((item) => item.count > 0 && (!item.feature || item.feature === record.feature)); + if (failure) { + failure.count -= 1; + record.status = failure.status; + return json(res, failure.status, { error: { message: `Forced ${failure.status} for ${record.feature}` } }); + } if (++upstreamCalls > maxUpstreamCalls) { record.status = 429; return json(res, 429, { error: { message: `Live gate stopped after ${maxUpstreamCalls} provider calls` } }); @@ -236,8 +243,16 @@ async function proxy(req, res, path) { }); record.status = response.status; res.writeHead(response.status, { "content-type": response.headers.get("content-type") ?? "application/json" }); - for await (const chunk of response.body ?? []) res.write(chunk); + // Keep the head of the reply: OpenRouter's own `model` and `provider` + // fields prove which model answered, independent of the rewrite above. + let head = ""; + for await (const chunk of response.body ?? []) { + if (head.length < 8192) head += Buffer.from(chunk).toString("utf8"); + res.write(chunk); + } res.end(); + record.upstreamModel = /"model"\s*:\s*"([^"]+)"/.exec(head)?.[1]; + record.upstreamProvider = /"provider"\s*:\s*"([^"]+)"/.exec(head)?.[1]; } catch (error) { record.status = 502; const message = error instanceof Error ? error.message : "upstream connection failed"; @@ -252,6 +267,9 @@ async function proxy(req, res, path) { createServer(async (req, res) => { const url = new URL(req.url ?? "/", "http://fake"); if (url.pathname === "/__requests" && req.method === "GET") return json(res, 200, requests); + if (url.pathname === "/__stats" && req.method === "GET") { + return json(res, 200, { mode: upstream ? "live" : "fake", liveModel: liveModel ?? null, upstreamCalls, maxUpstreamCalls }); + } if (url.pathname === "/__reset" && req.method === "POST") { requests.length = 0; failures = []; diff --git a/scripts/check-docs.mjs b/scripts/check-docs.mjs index 5ac6d78c9..3c6379aa3 100644 --- a/scripts/check-docs.mjs +++ b/scripts/check-docs.mjs @@ -11,6 +11,7 @@ const EXCLUDED_PREFIXES = [ "coverage/", "test-results/", "playwright-report/", + ".verify/", "ios/App/CapApp-SPM/", "plans/", ];