Skip to content

Commit 7df7aa5

Browse files
Programcodeclaude
andauthored
test(e2e): full-stack feature journeys, plus the bugs they found (#67)
* fix(settings): persist preferences to the account again Since the 2026-08-19 hardening revoked INSERT on user_settings, the client's upsert was always denied, so language, output language, theme, model choices, and the custom system prompt never left the browser. The hardening grant also omitted output_language. Update the provisioned row instead and grant the missing column. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(mobile): stop toasts, Library, and Flowchart widening the page Sonner's mobile toaster overflowed by its left offset, the Library grid let a source row's min-content width stretch the card, and the Flowchart toolbar did not wrap. Each widened the phone layout viewport and pushed controls off-screen. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * test(e2e): drive every feature through the real local stack Add full-stack browser journeys that sign in through the real demo-code flow and exercise Auth, RLS, quota, every Edge Function, streaming, and persistence against local Supabase. Only OpenRouter is replaced, by a strict fake container reached through the new OPENROUTER_API_URL override. The case-generation test fails against the pre-#66 client, which silently dropped oversized requests. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * test(e2e): fail the CI gate instead of skipping full-stack specs Also warm the admin and announcements functions before the browser run. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * ci(db-gate): retry supabase start after registry rate limits Both database-tests attempts on this PR failed while pulling Supabase images from ghcr.io with toomanyrequests, before any test ran. Retry the start up to three times and stop a partially started stack between attempts. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
1 parent 653aeac commit 7df7aa5

29 files changed

Lines changed: 868 additions & 46 deletions

‎.superset/db-gate.sh‎

Lines changed: 28 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
#!/usr/bin/env bash
2-
# Runs the Docker-backed Supabase, database, collaboration, and real-Auth browser gate
2+
# Runs the Docker-backed Supabase, database, collaboration, and full-stack browser gate
33
# with exclusive access across
44
# Superset workspaces of this repo. `supabase/config.toml` sets no
55
# project_id or custom port block, so `supabase db start` always binds the
@@ -13,6 +13,7 @@ DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
1313
source "$DIR/lib/lock.sh"
1414

1515
DB_LOCK="$HOME/.superset/debatelab-db.lock"
16+
FAKE_PROVIDER=debatelab-fake-openrouter
1617
stack_started=false
1718
runtime_dir=""
1819
owned_pids=()
@@ -32,6 +33,7 @@ cleanup() {
3233
trap - EXIT
3334

3435
stop_owned_processes
36+
docker rm -f "$FAKE_PROVIDER" >/dev/null 2>&1 || true
3537

3638
if [ -n "$runtime_dir" ] && [ -f "$runtime_dir/function.env" ]; then
3739
rm -- "$runtime_dir/function.env"
@@ -59,8 +61,15 @@ if [ -n "${WORKSPACE_GATE_ARTIFACT_ROOT:-}" ]; then
5961
else
6062
runtime_dir="$(mktemp -d)"
6163
fi
62-
supabase start >"$runtime_dir/supabase.log" 2>&1
64+
# Shared CI runners intermittently hit registry rate limits while pulling images.
6365
stack_started=true
66+
for attempt in 1 2 3; do
67+
if supabase start >>"$runtime_dir/supabase.log" 2>&1; then break; fi
68+
if [ "$attempt" -eq 3 ]; then exit 1; fi
69+
echo "supabase start failed (attempt $attempt); retrying after $((attempt * 30))s"
70+
supabase stop --no-backup >/dev/null 2>&1 || true
71+
sleep $((attempt * 30))
72+
done
6473
npm run test:db
6574

6675
status_json="$(supabase status -o json 2>/dev/null)"
@@ -82,7 +91,13 @@ LOCAL_SERVICE_ROLE_KEY="$service_role_key" \
8291
LOCAL_MAILER_HOST="$mailer_host" \
8392
node scripts/workspace-test-env.mjs .superset/workspace-test.env "$runtime_dir/function.env"
8493

85-
setsid supabase functions serve workspace-invites --no-verify-jwt \
94+
# Edge Functions reach the fake provider by container name on the stack network.
95+
docker rm -f "$FAKE_PROVIDER" >/dev/null 2>&1 || true
96+
docker run -d --rm --name "$FAKE_PROVIDER" --network "supabase_network_$project_id" \
97+
-p 127.0.0.1:18090:8787 -v "$(pwd)/e2e/full-stack/fake-openrouter.mjs:/fake.mjs:ro" \
98+
node:22-alpine node /fake.mjs >/dev/null
99+
100+
setsid supabase functions serve --no-verify-jwt \
86101
--env-file "$runtime_dir/function.env" >"$runtime_dir/functions.log" 2>&1 &
87102
owned_pids+=("$!")
88103

@@ -94,19 +109,21 @@ VITE_COLLABORATION_URL=ws://127.0.0.1:12345 \
94109
owned_pids+=("$!")
95110

96111
mailer_url="$(status_value MAILPIT_URL)"
97-
for url in "$mailer_url/api/v1/messages" "$app_origin"; do
112+
for url in "$mailer_url/api/v1/messages" "$app_origin" "http://127.0.0.1:18090/__requests"; do
98113
for _ in $(seq 1 200); do
99114
if curl --fail --silent --output /dev/null "$url"; then break; fi
100115
sleep 0.1
101116
done
102117
curl --fail --silent --output /dev/null "$url"
103118
done
104-
function_url="$api_url/functions/v1/workspace-invites"
105-
for _ in $(seq 1 200); do
106-
if curl --fail --silent --output /dev/null --request OPTIONS --header "Origin: $app_origin" "$function_url"; then break; fi
107-
sleep 0.1
119+
for function_name in workspace-invites ai ai-stream demo-code-login admin announcements; do
120+
function_url="$api_url/functions/v1/$function_name"
121+
for _ in $(seq 1 200); do
122+
if curl --fail --silent --output /dev/null --request OPTIONS --header "Origin: $app_origin" "$function_url"; then break; fi
123+
sleep 0.1
124+
done
125+
curl --fail --silent --output /dev/null --request OPTIONS --header "Origin: $app_origin" "$function_url"
108126
done
109-
curl --fail --silent --output /dev/null --request OPTIONS --header "Origin: $app_origin" "$function_url"
110127

111128
LOCAL_SUPABASE_URL="$api_url" \
112129
LOCAL_SUPABASE_ANON_KEY="$anon_key" \
@@ -117,5 +134,7 @@ COLLABORATION_PORT=12345 \
117134
COLLABORATION_ALLOWED_ORIGINS="$app_origin" \
118135
SUPABASE_URL="$api_url" \
119136
SUPABASE_ANON_KEY="$anon_key" \
137+
FAKE_OPENROUTER_URL=http://127.0.0.1:18090 \
138+
DEMO_CODE_PEPPER="$(sed -n 's/^DEMO_CODE_PEPPER=//p' .superset/workspace-test.env)" \
120139
PLAYWRIGHT_BASE_URL="$app_origin" \
121140
npx playwright test --config=playwright.workspace.config.ts

‎.superset/workspace-test.env‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,11 @@
1-
# Test-only values for the locked local workspace browser gate.
1+
# Test-only values for the locked local full-stack browser gate.
22
WORKSPACE_INVITE_SECRET=local-workspace-invite-secret-for-browser-gate
33
WORKSPACE_INVITE_MAILER_URL=http://workspace-mailer.invalid:8025/api/v1/send
44
WORKSPACE_INVITE_MAILER_TOKEN=local-workspace-mailer
55
WORKSPACE_INVITE_MAILER_FORMAT=mailpit
66
WORKSPACE_INVITE_BASE_URL=http://127.0.0.1:18081
77
WORKSPACE_INVITE_ALLOWED_ORIGINS=http://127.0.0.1:18081
8+
OPENROUTER_API_KEY=local-fake-openrouter-key
9+
OPENROUTER_API_URL=http://debatelab-fake-openrouter:8787/api
10+
DEMO_CODE_PEPPER=local-demo-code-pepper-for-browser-gate
11+
SPARRING_INTEGRITY_SECRET=local-sparring-integrity-secret-for-browser-gate

‎docs/testing.md‎

Lines changed: 18 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -145,7 +145,7 @@ In Superset, run:
145145
./.superset/db-gate.sh
146146
```
147147

148-
The wrapper acquires `~/.superset/debatelab-db.lock`, starts full local Supabase, runs `npm run test:db`, then holds the same lock while a controlled Mailpit relay, collaboration process, and Vite client run the real-Auth desktop/mobile workspace journey. Pull requests run this same gate in the `database-tests` job and retain its logs and browser traces on failure. It stops every owned process and the stack on exit, including after failure. Outside Superset, run only the database portion with:
148+
The wrapper acquires `~/.superset/debatelab-db.lock`, starts full local Supabase, runs `npm run test:db`, then holds the same lock while a controlled Mailpit relay, collaboration process, fake model provider, and Vite client run the real-Auth desktop/mobile workspace and full-stack feature journeys. Pull requests run this same gate in the `database-tests` job and retain its logs and browser traces on failure. It stops every owned process and the stack on exit, including after failure. Outside Superset, run only the database portion with:
149149

150150
```bash
151151
supabase db start
@@ -155,6 +155,23 @@ supabase stop --no-backup
155155

156156
The stack replays migrations and runs pgTAP tests. The browser portion creates isolated local users and deletes them, captures mail only in local Mailpit, restarts the collaboration process, and proves recovery and revocation. `CREATE INDEX CONCURRENTLY` requires the stated CLI floor so it runs outside the migration batch.
157157

158+
### Full-stack feature journeys
159+
160+
The same gate serves every Edge Function and runs `e2e/full-stack/` on desktop and mobile Chromium. Only the paid model provider is replaced: [`e2e/full-stack/fake-openrouter.mjs`](../e2e/full-stack/fake-openrouter.mjs) runs as a `node:22-alpine` container on the Supabase Docker network, and the functions reach it through `OPENROUTER_API_URL`. Real code handles everything else: Auth, demo-code sign-in, RLS, quota, source loading, prompt building, output validation, streaming, and persistence.
161+
162+
Each test signs in through the Login page with a fresh demo code and deletes that user afterwards. The specs cover:
163+
164+
- sign-in, reload, sign-out, a rejected code, and an expired token that refreshes before an AI request;
165+
- case generation grounded in a 250 KB Library source, provider failure with retry, a zero quota, and a saved case that a fresh browser loads from History;
166+
- Attack, Brainstorm, Factoids, streamed Ask (it fails if the client falls back to the non-streaming path), and Ask with citations;
167+
- a judged Lincoln-Douglas sparring round and a Crossfire debrief;
168+
- Library activation and deletion, output-language sync across devices, and flowchart saves;
169+
- admin announcements seen by another user, denied admin access, and a public share link.
170+
171+
Tests assert the rendered result, what the Edge Function sent upstream (read back from the fake provider), and the resulting database rows. They also fail on any uncaught page error or horizontal page overflow. The fake provider rejects prompts it does not recognize, so a prompt change must update it.
172+
173+
These journeys do not prove live provider behavior, web research (the local stack has no search key), Jev decisions (the fake returns 503 and callers use their fallbacks), Stripe, or Google export.
174+
158175
The local stack begins at port `54321` and is not parallel-safe across worktrees. The task that starts it must stop it before finishing or abandoning the run. Use `supabase stop --no-backup` unless another active task explicitly owns the same stack.
159176

160177
Never substitute a staging or production project for this gate.
Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
import { admin, expect, MARK, providerCall, signIn, stackReady, test } from "./stack";
2+
import type { Page } from "@playwright/test";
3+
4+
test.skip(!stackReady, "Local full stack is required (run ./.superset/db-gate.sh)");
5+
6+
async function addSource(page: Page, title: string, content: string) {
7+
await page.getByRole("button", { name: "Add Source" }).click();
8+
await page.getByRole("textbox", { name: "Source title..." }).fill(title);
9+
await page.getByRole("textbox", { name: "Paste source content..." }).fill(content);
10+
await page.getByRole("button", { name: "Save to Library" }).click();
11+
await expect(page.getByRole("switch", { name: `Toggle ${title}` })).toBeChecked();
12+
}
13+
14+
test("only active Library sources ground AI requests, and deletions persist", async ({ signedIn: page, user }) => {
15+
await page.goto("/app/library");
16+
await addSource(page, "Active study", "ACTIVE-SOURCE-MARKER ridership rose after fares were removed.");
17+
await addSource(page, "Paused study", "PAUSED-SOURCE-MARKER congestion did not change.");
18+
await page.getByRole("switch", { name: "Toggle Paused study" }).click();
19+
await expect(page.getByRole("switch", { name: "Toggle Paused study" })).not.toBeChecked();
20+
21+
await page.goto("/app/brainstorm");
22+
await page.getByLabel("Motion / Topic *").fill("Public transport should be free");
23+
await page.getByRole("button", { name: "Start Brainstorming" }).click();
24+
await expect(page.getByText(`${MARK} idea 1`)).toBeVisible({ timeout: 30_000 });
25+
const call = await providerCall("brainstorm");
26+
expect(call.system).toContain("ACTIVE-SOURCE-MARKER");
27+
expect(call.system).not.toContain("PAUSED-SOURCE-MARKER");
28+
29+
await page.goto("/app/library");
30+
await page.getByRole("button", { name: "Delete Paused study", exact: true }).click();
31+
await expect(page.getByRole("switch", { name: "Toggle Paused study" })).toHaveCount(0);
32+
await page.reload();
33+
await expect(page.getByRole("switch", { name: "Toggle Active study" })).toBeVisible();
34+
await expect(page.getByRole("switch", { name: "Toggle Paused study" })).toHaveCount(0);
35+
36+
const { data } = await admin.from("sources").select("title").eq("user_id", await user.userId());
37+
expect(data?.map((row) => row.title)).toEqual(["Active study"]);
38+
});
39+
40+
test("the output language setting follows the account and reaches the model", async ({ signedIn: page, user, browser }) => {
41+
await page.goto("/app/settings");
42+
await page.locator("div.space-y-2", { has: page.getByText("AI Output Language", { exact: true }) }).getByRole("combobox").click();
43+
await page.getByRole("option", { name: "Magyar" }).click();
44+
await expect.poll(async () => {
45+
const { data } = await admin.from("user_settings").select("output_language").eq("user_id", await user.userId()).single();
46+
return data?.output_language;
47+
}).toBe("hu");
48+
49+
// A new device starts with English defaults and must hydrate from the account.
50+
const context = await browser.newContext();
51+
try {
52+
const other = await context.newPage();
53+
await other.addInitScript(() => localStorage.setItem("debate-trainer-onboarding", JSON.stringify({ state: { hasCompletedOnboarding: true, isTourRunning: false }, version: 0 })));
54+
await signIn(other, user);
55+
await other.goto("/app/attack");
56+
await other.getByLabel("Your Argument *").fill("Free transit cuts congestion.");
57+
await other.getByRole("button", { name: "Find Weaknesses" }).click();
58+
await expect(other.getByText(`${MARK} weakness`)).toBeVisible({ timeout: 30_000 });
59+
expect((await providerCall("attack")).user).toContain("Language: Hungarian (hu)");
60+
} finally {
61+
await context.close();
62+
}
63+
});
64+
65+
test("a saved flowchart is stored on the account", async ({ signedIn: page, user }) => {
66+
const title = `Flow ${Date.now()}`;
67+
await page.goto("/app/flowchart");
68+
await page.getByRole("button", { name: "Save", exact: true }).click();
69+
const dialog = page.getByRole("dialog");
70+
await dialog.getByRole("textbox").fill(title);
71+
await dialog.getByRole("button", { name: "Save Flowchart" }).click();
72+
await expect(page.getByText("Flowchart saved successfully")).toBeVisible();
73+
74+
const { data } = await admin.from("flowcharts").select("title").eq("user_id", await user.userId());
75+
expect(data?.map((row) => row.title)).toEqual([title]);
76+
});

‎e2e/full-stack/admin.spec.ts‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
import { admin, createDemoUser, deleteDemoUser, expect, signIn, stackReady, test } from "./stack";
2+
3+
test.skip(!stackReady, "Local full stack is required (run ./.superset/db-gate.sh)");
4+
5+
test("an admin publishes an announcement that another user sees", async ({ signedIn: page, user, browser }) => {
6+
const title = `Maintenance ${Date.now()}`;
7+
const reader = await createDemoUser("reader");
8+
const readerContext = await browser.newContext();
9+
try {
10+
// Admin is granted out of band, never through demo sign-in.
11+
await admin.from("user_roles").update({ role: "admin" }).eq("user_id", await user.userId());
12+
await page.goto("/app/admin");
13+
await expect(page.getByRole("heading", { name: "Admin Dashboard" })).toBeVisible();
14+
await page.getByRole("tab", { name: "Announcements" }).click();
15+
await page.getByLabel("Title").fill(title);
16+
await page.getByLabel("Message").fill("DebateLab will be unavailable for ten minutes.");
17+
await page.getByRole("button", { name: "Create announcement" }).click();
18+
await expect(page.getByRole("heading", { name: title })).toBeVisible();
19+
20+
const readerPage = await readerContext.newPage();
21+
await readerPage.addInitScript(() => localStorage.setItem("debate-trainer-onboarding", JSON.stringify({ state: { hasCompletedOnboarding: true, isTourRunning: false }, version: 0 })));
22+
await signIn(readerPage, reader);
23+
await expect(readerPage.getByRole("heading", { name: title })).toBeVisible();
24+
await readerPage.getByRole("button", { name: `Dismiss announcement: ${title}` }).click();
25+
await readerPage.reload();
26+
await expect(readerPage.getByRole("main")).toBeVisible();
27+
await expect(readerPage.getByRole("heading", { name: title })).toHaveCount(0);
28+
} finally {
29+
await readerContext.close();
30+
await admin.from("admin_announcements").delete().eq("title", title);
31+
await deleteDemoUser(reader);
32+
}
33+
});
34+
35+
test("a regular user cannot open the admin dashboard", async ({ signedIn: page }) => {
36+
await page.goto("/app/admin");
37+
await expect(page.getByText("Access Denied")).toBeVisible();
38+
await expect(page.getByRole("heading", { name: "Admin Dashboard" })).toHaveCount(0);
39+
});

‎e2e/full-stack/ai-tools.spec.ts‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
import { admin, expect, MARK, fake, providerCall, stackReady, test } from "./stack";
2+
3+
test.skip(!stackReady, "Local full stack is required (run ./.superset/db-gate.sh)");
4+
5+
test("Attack analyses a pasted case", async ({ signedIn: page }) => {
6+
await page.goto("/app/attack");
7+
await page.getByLabel("Your Argument *").fill("Free transit cuts congestion because riders leave their cars at home.");
8+
await page.getByRole("button", { name: "Find Weaknesses" }).click();
9+
10+
await expect(page.getByText(`${MARK} weakness`)).toBeVisible({ timeout: 30_000 });
11+
expect((await providerCall("attack")).user).toContain("riders leave their cars at home");
12+
});
13+
14+
test("Brainstorm returns ranked angles", async ({ signedIn: page }) => {
15+
await page.goto("/app/brainstorm");
16+
await page.getByLabel("Motion / Topic *").fill("Universal basic income should replace targeted welfare");
17+
await page.getByRole("button", { name: "Start Brainstorming" }).click();
18+
19+
await expect(page.getByText(`${MARK} idea 1`)).toBeVisible({ timeout: 30_000 });
20+
expect((await providerCall("brainstorm")).user).toContain("Motion: Universal basic income should replace targeted welfare");
21+
});
22+
23+
test("Factoids completes research without verified web sources", async ({ signedIn: page }) => {
24+
await page.goto("/app/factoids");
25+
await page.getByLabel("Statement *").fill("Fare-free transit raises ridership");
26+
await page.getByRole("button", { name: "Find Factoids" }).click();
27+
28+
// The local stack has no search key, so the server must drop every unverified
29+
// factoid and still finish instead of inventing citations.
30+
await expect(page.getByText(`${MARK} search suggestion`)).toBeVisible({ timeout: 30_000 });
31+
expect((await providerCall("factoids")).user).toContain('"Fare-free transit raises ridership"');
32+
});
33+
34+
test("Ask streams a coaching answer and records usage", async ({ signedIn: page, user }) => {
35+
await page.goto("/app/ask");
36+
await page.getByPlaceholder("Ask your debate coach...").fill("How do I weigh access against cost?");
37+
await page.getByRole("button", { name: "Send message" }).click();
38+
39+
await expect(page.getByText(`${MARK} streamed answer: weigh access against cost.`)).toBeVisible({ timeout: 30_000 });
40+
// The client silently falls back to the non-streaming path when streaming
41+
// fails, so prove the answer really came through ai-stream.
42+
const features = (await fake.requests()).map((request) => request.feature);
43+
expect(features).toContain("ask-stream");
44+
expect(features).not.toContain("ask");
45+
46+
const userId = await user.userId();
47+
await expect.poll(async () => {
48+
const { data } = await admin.from("ai_requests").select("feature,status").eq("user_id", userId);
49+
return data;
50+
}).toContainEqual(expect.objectContaining({ feature: "ask", status: "success" }));
51+
});
52+
53+
test("Ask with citations uses the structured endpoint", async ({ signedIn: page }) => {
54+
await page.goto("/app/ask");
55+
await page.locator("#citation-mode").click();
56+
await page.getByPlaceholder("Ask your debate coach...").fill("Which evidence supports fare-free transit?");
57+
await page.getByRole("button", { name: "Send message" }).click();
58+
59+
await expect(page.getByText(`${MARK} answer`)).toBeVisible({ timeout: 30_000 });
60+
expect((await providerCall("ask")).user).toContain("Which evidence supports fare-free transit?");
61+
});

0 commit comments

Comments
 (0)