Skip to content

Commit 74d11eb

Browse files
Programcodeclaude
Programcode
and
claude
authored
test(e2e): live provider journeys, production canary, and required checks (#68)
* test(e2e): add live provider journeys and a production canary Live mode for the locked gate turns the fake provider into a recording proxy that pins every chat call to one cheap model (Muse Spark 1.3 Contributor by default) and caps provider calls, while Jev, Exa research, and URL import run for real. The specs assert response shape, rendered sources, database rows, and provider health rather than model wording. The production canary checks the live site, every deployed function's CORS, the ai auth guard, and one streamed answer as a dedicated capped account. Database and live jobs now pull Supabase images from ECR, because ghcr.io rate-limits shared runners. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * ci(db-gate): show why supabase start failed before retrying Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * test(e2e): require complete live streams and live Jev signals A 200 Ask stream can still end without its final event, and the Crossfire test passed even when no Jev signal decision ran. Assert both, and record the accepted risk that pull-request runs can read the live provider keys. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * test(canary): check the method and headers a browser preflight needs Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * docs(agents): scope the production canary credential exception Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(crossfire): stop sending client-only turn signals to the server Once a Jev signal attached to an answer, every later Crossfire request resent the turn with a `signal` key, and the ai function rejected it as "transcript[1] is invalid". The live suite caught it. The fake provider now answers Jev so the credential-free gate exercises the same path. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * test(full-stack): cover billing, promo codes, data erasure, and exports - A promo code redeemed in Settings restores AI access for an expired account. - Signed Stripe webhooks grant and revoke Pro; forged and replayed ones change nothing. - Delete All My Data erases the account's rows and leaves another account's intact. - A generated case downloads as a real PDF and DOCX file. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * test(live): generate a case with every catalog model The pinned live suite cannot see a production model that times out or returns invalid JSON. LIVE_MODEL=as-requested makes the proxy forward the model the Edge Function chose and runs e2e/model-matrix, which picks each catalog model in the Generate page and prints its provider latency. The workflow runs daily and on pull requests that touch AI code. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * test(canary): generate a production case and keep AI checks within the account cap Main can deploy more than the canary account's 10 AI requests a day, so deployment runs skip the AI checks; daily and manual runs make them. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * ci(model-matrix): run weekly; one run costs about $1 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * test(canary): check the signed-in app on every production deployment Deployment runs skip the AI requests to stay under the canary account's cap, so they now sign in, open Generate, and require Settings to show the account's Pro plan. That read goes through row-level security, so it fails on a broken session, and it spends no AI request. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(ai): give provider calls the time research leaves, not a fixed 60 s Claude Opus 5 needs about 68 s to write a case, so every Opus case generation hit the 60 s provider timeout and returned 500. Each call now shares one 140 s request budget, which stays under Supabase's 150 s response limit. Research, the main call, and its JSON repair retry all use what is left. The model matrix reproduces the failure and proves the fix. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(ai): keep a finished case when the source-usage check runs late The post-generation Jev source-usage check shared the request budget, so a model that finished near the deadline could lose its completed case to a 500. The check now follows only the client's signal and its own 8 s Jev timeout, and the provider budget drops to 130 s to leave room for it under Supabase's 150 s limit. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * test(ai): cover request budget and balanced routing * test(canary): require complete streamed answers * docs(testing): keep canary identity private * docs(testing): describe the shared AI request budget * test(live): use balanced multi-provider routing --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
1 parent 7df7aa5 commit 74d11eb

28 files changed

Lines changed: 1029 additions & 36 deletions

‎.github/workflows/deploy.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -93,6 +93,8 @@ jobs:
9393

9494
- name: Database and real workspace journey
9595
env:
96+
# ghcr.io rate-limits shared runners; the same images are on ECR.
97+
SUPABASE_INTERNAL_IMAGE_REGISTRY: public.ecr.aws
9698
WORKSPACE_GATE_ARTIFACT_ROOT: ${{ runner.temp }}/workspace-gate
9799
run: ./.superset/db-gate.sh
98100

‎.github/workflows/live-e2e.yml‎

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
name: Live provider E2E
2+
3+
# Real OpenRouter, Jev, and Exa behind the local stack. Runs on pull requests,
4+
# nightly to catch provider drift without code changes, and on demand.
5+
on:
6+
pull_request:
7+
schedule:
8+
- cron: "17 3 * * *"
9+
workflow_dispatch:
10+
11+
permissions:
12+
contents: read
13+
14+
concurrency:
15+
group: live-e2e-${{ github.ref }}
16+
cancel-in-progress: true
17+
18+
jobs:
19+
live-e2e:
20+
runs-on: ubuntu-latest
21+
timeout-minutes: 45
22+
steps:
23+
- name: Checkout code
24+
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
25+
with:
26+
persist-credentials: false
27+
28+
- name: Set up Node
29+
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
30+
with:
31+
node-version: 22
32+
33+
- name: Set up Supabase CLI
34+
uses: supabase/setup-cli@46f7f98c7f948ad727d22c1e67fab04c223a0520 # v3.0.0
35+
with:
36+
version: 2.116.0
37+
38+
- name: Install dependencies
39+
run: npm ci
40+
41+
- name: Install Chromium
42+
run: npx playwright install --with-deps chromium
43+
44+
# db-gate.sh fails before starting anything when either key is missing.
45+
# Accepted risk (2026-09-24): pull-request code runs with these keys, so
46+
# anyone who can push a branch here could read them. Keep them low-limit.
47+
- name: Live provider journeys
48+
env:
49+
LIVE_PROVIDERS: 1
50+
# Repository variable override; empty falls back to db-gate.sh's default.
51+
LIVE_MODEL: ${{ vars.LIVE_MODEL }}
52+
LIVE_OPENROUTER_API_KEY: ${{ secrets.LIVE_OPENROUTER_API_KEY }}
53+
LIVE_EXA_API_KEY: ${{ secrets.LIVE_EXA_API_KEY }}
54+
SUPABASE_INTERNAL_IMAGE_REGISTRY: public.ecr.aws
55+
WORKSPACE_GATE_ARTIFACT_ROOT: ${{ runner.temp }}/workspace-gate
56+
run: ./.superset/db-gate.sh
57+
58+
- name: Upload live failure artifacts
59+
if: failure()
60+
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
61+
with:
62+
name: live-e2e-failure-${{ github.run_id }}
63+
path: |
64+
${{ runner.temp }}/workspace-gate/
65+
test-results/
66+
playwright-report/
67+
if-no-files-found: ignore
68+
retention-days: 7

‎.github/workflows/model-matrix.yml‎

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
name: Model matrix
2+
3+
# Generates one case with every model in the catalog, unpinned, so a model that
4+
# times out or returns invalid JSON fails here before users hit it. One run
5+
# costs about $1, so it runs weekly and when model routing, prompts, or the ai
6+
# function change.
7+
on:
8+
pull_request:
9+
paths:
10+
- supabase/functions/_shared/**
11+
- supabase/functions/ai/**
12+
- src/lib/ai/**
13+
- e2e/model-matrix/**
14+
- .github/workflows/model-matrix.yml
15+
schedule:
16+
- cron: "29 4 * * 1"
17+
workflow_dispatch:
18+
19+
permissions:
20+
contents: read
21+
22+
concurrency:
23+
group: model-matrix-${{ github.ref }}
24+
cancel-in-progress: true
25+
26+
jobs:
27+
model-matrix:
28+
runs-on: ubuntu-latest
29+
timeout-minutes: 45
30+
steps:
31+
- name: Checkout code
32+
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
33+
with:
34+
persist-credentials: false
35+
36+
- name: Set up Node
37+
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
38+
with:
39+
node-version: 22
40+
41+
- name: Set up Supabase CLI
42+
uses: supabase/setup-cli@46f7f98c7f948ad727d22c1e67fab04c223a0520 # v3.0.0
43+
with:
44+
version: 2.116.0
45+
46+
- name: Install dependencies
47+
run: npm ci
48+
49+
- name: Install Chromium
50+
run: npx playwright install --with-deps chromium
51+
52+
# db-gate.sh fails before starting anything when either key is missing.
53+
# Accepted risk (2026-09-24): pull-request code runs with these keys, so
54+
# anyone who can push a branch here could read them. Keep them low-limit.
55+
- name: Case generation with every catalog model
56+
env:
57+
LIVE_PROVIDERS: 1
58+
LIVE_MODEL: as-requested
59+
LIVE_OPENROUTER_API_KEY: ${{ secrets.LIVE_OPENROUTER_API_KEY }}
60+
LIVE_EXA_API_KEY: ${{ secrets.LIVE_EXA_API_KEY }}
61+
SUPABASE_INTERNAL_IMAGE_REGISTRY: public.ecr.aws
62+
WORKSPACE_GATE_ARTIFACT_ROOT: ${{ runner.temp }}/workspace-gate
63+
run: ./.superset/db-gate.sh
64+
65+
- name: Upload live failure artifacts
66+
if: failure()
67+
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
68+
with:
69+
name: model-matrix-failure-${{ github.run_id }}
70+
path: |
71+
${{ runner.temp }}/workspace-gate/
72+
test-results/
73+
playwright-report/
74+
if-no-files-found: ignore
75+
retention-days: 7

‎.github/workflows/prod-canary.yml‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
name: Production canary
2+
3+
# Checks the live site after every Vercel production deployment and daily.
4+
# Read-only on deployments. The daily and manual runs also make two real AI
5+
# requests (a streamed Ask and a case) as the dedicated canary account.
6+
on:
7+
deployment_status:
8+
schedule:
9+
- cron: "41 6 * * *"
10+
workflow_dispatch:
11+
12+
permissions:
13+
contents: read
14+
15+
concurrency:
16+
group: production-canary
17+
cancel-in-progress: false
18+
19+
jobs:
20+
canary:
21+
if: >-
22+
github.event_name != 'deployment_status' ||
23+
(github.event.deployment_status.state == 'success' && github.event.deployment.environment == 'Production')
24+
runs-on: ubuntu-latest
25+
timeout-minutes: 15
26+
steps:
27+
- name: Checkout code
28+
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
29+
with:
30+
persist-credentials: false
31+
32+
- name: Set up Node
33+
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
34+
with:
35+
node-version: 22
36+
37+
- name: Install dependencies
38+
run: npm ci
39+
40+
- name: Install Chromium
41+
run: npx playwright install --with-deps chromium
42+
43+
- name: Production canary
44+
env:
45+
PROD_SUPABASE_ANON_KEY: ${{ vars.PROD_SUPABASE_ANON_KEY }}
46+
PROD_CANARY_EMAIL: ${{ secrets.PROD_CANARY_EMAIL }}
47+
PROD_CANARY_PASSWORD: ${{ secrets.PROD_CANARY_PASSWORD }}
48+
CANARY_AI_REQUESTS: ${{ github.event_name != 'deployment_status' }}
49+
run: npx playwright test --config=playwright.production.config.ts

‎.superset/db-gate.sh‎

Lines changed: 32 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,17 @@ source "$DIR/lib/lock.sh"
1414

1515
DB_LOCK="$HOME/.superset/debatelab-db.lock"
1616
FAKE_PROVIDER=debatelab-fake-openrouter
17+
# LIVE_PROVIDERS=1 runs e2e/live against real OpenRouter, Jev, and Exa instead of
18+
# the fake provider. It needs LIVE_OPENROUTER_API_KEY and LIVE_EXA_API_KEY.
19+
# LIVE_MODEL=as-requested runs e2e/model-matrix instead: one case per catalog model.
20+
LIVE_PROVIDERS="${LIVE_PROVIDERS:-0}"
21+
# GLM Flash is inexpensive and has broad endpoint coverage, so OpenRouter's
22+
# default balanced routing can fail over between providers during this gate.
23+
LIVE_MODEL="${LIVE_MODEL:-z-ai/glm-5.3-flash}" # empty also means default
24+
if [ "$LIVE_PROVIDERS" = 1 ] && { [ -z "${LIVE_OPENROUTER_API_KEY:-}" ] || [ -z "${LIVE_EXA_API_KEY:-}" ]; }; then
25+
echo "LIVE_PROVIDERS=1 requires LIVE_OPENROUTER_API_KEY and LIVE_EXA_API_KEY" >&2
26+
exit 1
27+
fi
1728
stack_started=false
1829
runtime_dir=""
1930
owned_pids=()
@@ -67,10 +78,11 @@ for attempt in 1 2 3; do
6778
if supabase start >>"$runtime_dir/supabase.log" 2>&1; then break; fi
6879
if [ "$attempt" -eq 3 ]; then exit 1; fi
6980
echo "supabase start failed (attempt $attempt); retrying after $((attempt * 30))s"
81+
tail -n 3 "$runtime_dir/supabase.log"
7082
supabase stop --no-backup >/dev/null 2>&1 || true
7183
sleep $((attempt * 30))
7284
done
73-
npm run test:db
85+
if [ "$LIVE_PROVIDERS" != 1 ]; then npm run test:db; fi
7486

7587
status_json="$(supabase status -o json 2>/dev/null)"
7688
status_value() {
@@ -90,11 +102,20 @@ fi
90102
LOCAL_SERVICE_ROLE_KEY="$service_role_key" \
91103
LOCAL_MAILER_HOST="$mailer_host" \
92104
node scripts/workspace-test-env.mjs .superset/workspace-test.env "$runtime_dir/function.env"
105+
provider_mode=()
106+
if [ "$LIVE_PROVIDERS" = 1 ]; then
107+
# printf is a builtin, so the keys never appear in a process list or log.
108+
grep -v '^OPENROUTER_API_KEY=' "$runtime_dir/function.env" >"$runtime_dir/function.env.next"
109+
printf 'OPENROUTER_API_KEY=%s\nEXA_API_KEY=%s\n' "$LIVE_OPENROUTER_API_KEY" "$LIVE_EXA_API_KEY" >>"$runtime_dir/function.env.next"
110+
mv -- "$runtime_dir/function.env.next" "$runtime_dir/function.env"
111+
chmod 600 "$runtime_dir/function.env"
112+
provider_mode=(-e UPSTREAM=https://openrouter.ai/api -e "LIVE_MODEL=$LIVE_MODEL")
113+
fi
93114

94115
# Edge Functions reach the fake provider by container name on the stack network.
95116
docker rm -f "$FAKE_PROVIDER" >/dev/null 2>&1 || true
96117
docker run -d --rm --name "$FAKE_PROVIDER" --network "supabase_network_$project_id" \
97-
-p 127.0.0.1:18090:8787 -v "$(pwd)/e2e/full-stack/fake-openrouter.mjs:/fake.mjs:ro" \
118+
"${provider_mode[@]}" -p 127.0.0.1:18090:8787 -v "$(pwd)/e2e/full-stack/fake-openrouter.mjs:/fake.mjs:ro" \
98119
node:22-alpine node /fake.mjs >/dev/null
99120

100121
setsid supabase functions serve --no-verify-jwt \
@@ -125,6 +146,13 @@ for function_name in workspace-invites ai ai-stream demo-code-login admin announ
125146
curl --fail --silent --output /dev/null --request OPTIONS --header "Origin: $app_origin" "$function_url"
126147
done
127148

149+
if [ "$LIVE_PROVIDERS" = 1 ] && [ "$LIVE_MODEL" = as-requested ]; then
150+
test_selection=(e2e/model-matrix --project=desktop-chromium)
151+
elif [ "$LIVE_PROVIDERS" = 1 ]; then
152+
test_selection=(e2e/live --project=desktop-chromium)
153+
else
154+
test_selection=(e2e/workspace-collaboration.spec.ts e2e/full-stack)
155+
fi
128156
LOCAL_SUPABASE_URL="$api_url" \
129157
LOCAL_SUPABASE_ANON_KEY="$anon_key" \
130158
LOCAL_SUPABASE_SERVICE_ROLE_KEY="$service_role_key" \
@@ -135,6 +163,7 @@ COLLABORATION_ALLOWED_ORIGINS="$app_origin" \
135163
SUPABASE_URL="$api_url" \
136164
SUPABASE_ANON_KEY="$anon_key" \
137165
FAKE_OPENROUTER_URL=http://127.0.0.1:18090 \
166+
LIVE_MODEL="$LIVE_MODEL" \
138167
DEMO_CODE_PEPPER="$(sed -n 's/^DEMO_CODE_PEPPER=//p' .superset/workspace-test.env)" \
139168
PLAYWRIGHT_BASE_URL="$app_origin" \
140-
npx playwright test --config=playwright.workspace.config.ts
169+
npx playwright test --config=playwright.workspace.config.ts "${test_selection[@]}"

‎.superset/workspace-test.env‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,3 +9,5 @@ OPENROUTER_API_KEY=local-fake-openrouter-key
99
OPENROUTER_API_URL=http://debatelab-fake-openrouter:8787/api
1010
DEMO_CODE_PEPPER=local-demo-code-pepper-for-browser-gate
1111
SPARRING_INTEGRITY_SECRET=local-sparring-integrity-secret-for-browser-gate
12+
STRIPE_SECRET_KEY=sk_test_local-full-stack-unused
13+
STRIPE_WEBHOOK_SECRET=whsec_local-full-stack-test

‎AGENTS.md‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -71,7 +71,11 @@ CI=1 PORT=18080 PLAYWRIGHT_BASE_URL=http://127.0.0.1:18080 npm run test:e2e:webk
7171

7272
- Start bug fixes with a failing test or repeatable browser reproduction when a suitable test layer exists.
7373
- Stub sessions prove rendering only. Do not call them authenticated backend proof.
74-
- Never put staging or production credentials in jobs, logs, traces, screenshots, commits, comments, or docs.
74+
- Never put staging or production credentials in jobs, logs, traces, screenshots, commits, comments, or docs. Two scoped exceptions reach jobs only through GitHub Actions secrets:
75+
- the protected staging values;
76+
- the dedicated production canary account (`PROD_CANARY_EMAIL` and `PROD_CANARY_PASSWORD`, a regular user capped at 10 AI requests a day), which Adam approved on 2026-09-24.
77+
78+
Neither exception may appear in logs, traces, or screenshots. No other production credential may enter a job.
7579
- Never substitute the production project when local or staging access is unavailable.
7680
- Production deployments, data changes, migrations, Edge Function deployments, and secret changes require explicit authorization.
7781
- A local pass proves only the local commit. Check the current-head GitHub run before calling CI green.

‎docs/operations.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,8 @@ Pull requests also start isolated local Supabase for pgTAP and run desktop plus
2525

2626
Non-PR runs upload `dist/` briefly and may build a `linux/amd64` and `linux/arm64` Docker image when registry credentials are present. A successful quality run does not by itself prove that an image was pushed.
2727

28+
[`.github/workflows/live-e2e.yml`](../.github/workflows/live-e2e.yml) runs the local stack against real providers on pull requests and nightly. [`.github/workflows/model-matrix.yml`](../.github/workflows/model-matrix.yml) generates a case with every catalog model weekly and when AI code changes. [`.github/workflows/prod-canary.yml`](../.github/workflows/prod-canary.yml) checks production after each Vercel production deployment and daily. See [testing.md](testing.md) for all three. Both Supabase-starting jobs pull images from `public.ecr.aws`, because ghcr.io rate-limits shared runners.
29+
2830
[`.github/workflows/staging-e2e.yml`](../.github/workflows/staging-e2e.yml) owns protected staging. It accepts eligible successful main-branch quality runs or a manual dispatch from `main`. It runs only when `STAGING_E2E_ENABLED=true` and all protected values exist.
2931

3032
## Production image
@@ -111,6 +113,8 @@ Secret names may be documented; values may not. Server-side categories include:
111113

112114
Never place server secrets in `VITE_*`, screenshots, traces, logs, comments, commits, or documentation.
113115

116+
GitHub Actions secrets: `LIVE_OPENROUTER_API_KEY` and `LIVE_EXA_API_KEY` for live provider tests, and `PROD_CANARY_EMAIL` and `PROD_CANARY_PASSWORD` for the production canary account. The public anon key is the `PROD_SUPABASE_ANON_KEY` repository variable. To rotate the canary password, set it with the Supabase Auth admin API and pipe it into `gh secret set` over stdin.
117+
114118
## Rollback
115119

116120
Choose rollback before changing production:

0 commit comments

Comments
 (0)