test(e2e): live provider journeys, production canary, and required checks by Programcode · Pull Request #68 · Programcode/DebateOwl

Programcode · 2026-09-24T06:01:13Z

Why

The full-stack gate from #67 fakes the model provider, so it can't catch real-provider breakage: a model that stops returning valid JSON or runs past a timeout, a research API change, or a production deploy with a missing function or a dead key. Some flows had no end-to-end test at all: billing, promo codes, data erasure, and exports.

What this adds

Live provider journeys (e2e/live/, live-e2e.yml, run on PRs, nightly at 03:17 UTC, and on demand; a required check)

LIVE_PROVIDERS=1 ./.superset/db-gate.sh runs the local stack against real OpenRouter, Jev, and Exa. The fake provider becomes a recording proxy that pins chat calls to LIVE_MODEL (default meta/muse-spark-1.3-contributor) and refuses calls past 200 per run.
Coverage: web-researched case generation, Factoids with verified citations, Ask with web search, URL import, streamed Ask, Attack, Brainstorm, a judged sparring round, and Crossfire with live Jev signals.
Tests assert response shape, rendered sources, database rows, and that every provider call returned 200 on the pinned model. They never assert on model wording.

Model matrix (e2e/model-matrix/, model-matrix.yml, run weekly, on demand, and on PRs that touch AI code)

LIVE_MODEL=as-requested forwards the model the Edge Function chose, and the spec generates one researched case with every model in MODEL_CATALOG, selected through the Generate page. The run log prints each model's provider latency.
One run costs about $1.

New full-stack journeys (credential-free gate, desktop and mobile)

A promo code redeemed in Settings restores AI access for an expired account.
Signed Stripe webhooks grant and revoke Pro; forged and replayed ones change nothing.
Delete All My Data erases the account's rows and leaves another account's intact.
A generated case downloads as a real PDF and DOCX.
Crossfire now runs through a Jev signal. The fake provider answers Jev instead of returning 503.

Production canary (e2e/production/, prod-canary.yml)

On every Vercel production deploy, daily, and on demand: the site loads, app routes are guarded, every browser-facing Edge Function passes the browser's preflight, and ai returns 401 to unauthenticated calls.
The daily and manual runs also make two real requests as the dedicated canary account: a streamed GPT-5.6 Luna Ask and a generated case. Deployment runs skip these because main can deploy more often than the account's cap of 10 requests a day. Instead, every run signs in as the canary and requires Settings to show its Pro plan, which spends no AI request.

Crossfire bug fix (src/lib/ai/client.ts)

Once a Jev signal attached to an answer, every later Crossfire request resent the turn with a signal key, and the ai function rejected it as transcript[1] is invalid. The client now strips it. A unit test and the full-stack spec reproduce it. This is frontend-only, so Vercel ships it.

Branch protection on main

quality, database-tests, browser-tests, and live-e2e are required.
Admins are included, force pushes and deletions are blocked, and a PR is required (0 approvals).

Evidence

Full credential-free gate (pgTAP, then 52 browser journeys on desktop and mobile): all pgTAP tests passed and 51 journeys passed. The skip is the existing mobile skip in workspace-collaboration.spec.ts.
live-e2e: passed in CI on 59487a7 and 949e918.
Model matrix before and after fix(ai): let slow models use the time research leaves #69: see below.
Unit tests: 1034 passed.

What the live checks found

Crossfire broke after the first Jev signal. Fixed here.
Claude Opus 5 can't generate a case in production. It needs about 65 to 68 s after research, and the ai function aborted provider calls at 60 s. Claude Sonnet 5 (the Pro default) took 55.9 s and DeepSeek V4 Pro 75.3 s in one run, so both fail intermittently too. fix(ai): let slow models use the time research leaves #69, stacked on this PR, gives every request one 140 s budget, under Supabase's 150 s response limit. With it, the matrix passes 9/10 models, including Opus 5 and DeepSeek V4 Pro.
DeepSeek V4 Flash is unreliable for case generation. Across three runs today it timed out, passed at 53 s, and returned empty content after 8 s. Not changed here. Keeping or removing it is a product decision.
workspace-invites isn't deployed to production. This is intentional while Workspaces are hidden. The canary excludes it.

Until #69 merges, this PR's model-matrix check fails on Claude Opus 5. That's the bug it exists to catch. It isn't a required check.

Limits

The live suite proves the pipeline with the pinned test model. The model matrix covers production models, but only case generation.
The canary's generate check can't run before merge. It uses the selectors the matrix proved against the same UI.
Not covered: Stripe Checkout and the billing portal (they call the Stripe API), the checkout-completed and payment-succeeded webhooks (they fetch from Stripe), and Google Docs export.
Accepted risk: PR runs of live-e2e and model-matrix expose the provider keys to branch code. Use low-limit keys.

🤖 Generated with Claude Code

Summary by CodeRabbit

Bug Fixes
- Crossfire requests no longer send client-only annotations with conversation transcripts.
- AI requests have a longer time budget for research and other processing, reducing premature timeouts on longer tasks.
Quality & Reliability
- Expanded automated checks cover live AI features, web research, model-generated cases, production access, billing, account data deletion, and document exports.
- Added scheduled and deployment-triggered checks to help catch issues in production and across supported AI models.
AI
- Updated the default model used for live-provider checks.

Final integration

Includes stacked PR fix(ai): let slow models use the time research leaves #69's shared 130-second request budget, which lets slower catalog models use the time left after research.
Keeps OpenRouter on its documented balanced provider routing by omitting provider.sort and provider.order; require_parameters: true still excludes endpoints that cannot honor the request parameters.
Adds deterministic deadline and caller-cancellation tests plus non-streaming and streaming request-body assertions for the routing contract.

Confidence Score: 5/5

No outstanding review finding blocks merging; the documented canary and key-exposure tradeoffs remain.

Summary

The PR adds live-provider journeys, a model matrix, full-stack tests, and a production canary. Deployment runs check the canary account's sign-in but leave live AI requests to daily and manual runs.

_{Reviews (10) · Last reviewed commit: "test(live): use balanced multi-provider ..."}

Live mode for the locked gate turns the fake provider into a recording proxy that pins every chat call to one cheap model (Muse Spark 1.3 Contributor by default) and caps provider calls, while Jev, Exa research, and URL import run for real. The specs assert response shape, rendered sources, database rows, and provider health rather than model wording. The production canary checks the live site, every deployed function's CORS, the ai auth guard, and one streamed answer as a dedicated capped account. Database and live jobs now pull Supabase images from ECR, because ghcr.io rate-limits shared runners. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

vercel · 2026-09-24T06:01:15Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
debate-navigator	Ready	Preview	Sep 24, 2026 5:25pm UTC

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

coderabbitai · 2026-09-24T06:01:24Z

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

@coderabbitai resume to resume automatic reviews.
@coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

▶️ Resume reviews
🔍 Trigger review

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Free

Run ID: 00949e04-5f9a-46a0-9169-e56cfb9f2c35

📥 Commits

Reviewing files that changed from the base of the PR and between 6455bba and da2782f.

📒 Files selected for processing (3)

.superset/db-gate.sh
docs/testing.md
e2e/full-stack/fake-openrouter.mjs

🚧 Files skipped from review as they are similar to previous changes (1)

docs/testing.md

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.

📝 Walkthrough

Walkthrough

The pull request adds live-provider and model-matrix test paths, production canary checks, and full-stack journey coverage. It changes provider proxy behavior, Crossfire transcript requests, and AI request deadlines. New GitHub Actions workflows run the live-provider, model-matrix, and production canary suites.

Changes

Live-provider journeys

Layer / File(s)	Summary
Configure live-provider execution `.superset/db-gate.sh`, `e2e/full-stack/fake-openrouter.mjs`, `.superset/workspace-test.env`	The database gate sets the default live model. The proxy forwards chat and Jev requests within a call limit and records request results. In fake mode, Jev decision requests return generated answers. The workspace test environment adds Stripe test values.
Exercise live model and research features `e2e/live/`, `e2e/model-matrix/`, `playwright.config.ts`, `playwright.workspace.config.ts`	Live suites check model interactions, research results, citations, provider calls, and URL imports. The model-matrix suite tests case generation for each catalog model. Playwright configurations select or exclude the model-matrix specs.
Schedule and document live journeys `.github/workflows/live-e2e.yml`, `.github/workflows/model-matrix.yml`, `.github/workflows/deploy.yml`, `docs/operations.md`, `docs/testing.md`	Workflows run live journeys on configured triggers and upload artifacts after failures. The database journey uses `public.ecr.aws`. Documentation describes credentials, schedules, coverage, and CI requirements.

Production canary

Layer / File(s)	Summary
Define production canary checks `playwright.production.config.ts`, `e2e/production/canary.spec.ts`	The production suite checks generation access, Pro status, streamed answers, and generated contentions. AI-request tests can be disabled for deployment-triggered runs.
Run production canary `.github/workflows/prod-canary.yml`, `AGENTS.md`, `docs/testing.md`, `docs/operations.md`	The workflow runs after successful production deployments, daily, or on demand. The credential rule describes permitted canary credentials and capture restrictions. Documentation describes the workflow, credentials, and checks.

Crossfire transcript and Jev behavior

Layer / File(s)	Summary
Send transcripts without client-only signals `src/lib/ai/client.ts`, `src/test/ai/crossfire-client.test.ts`, `e2e/full-stack/practice.spec.ts`	Crossfire and signal-generation requests omit turn-level `signal` fields from transcripts. Tests check both request paths and the Jev provider call.

Full-stack journey coverage

Layer / File(s)	Summary
Validate account, billing, and export journeys `e2e/full-stack/account-data.spec.ts`, `e2e/full-stack/billing.spec.ts`, `e2e/full-stack/case-generation.spec.ts`, `docs/testing.md`	Tests check account-data deletion isolation, promo-code access, Stripe webhook signature and replay handling, and PDF and DOCX downloads. Documentation lists these journeys and clarifies local coverage limits.

AI request budget

Layer / File(s)	Summary
Apply a shared request deadline `supabase/functions/ai/index.ts`, `supabase/functions/ai/handler-research.test.ts`, `supabase/functions/ai-stream/index.ts`, `supabase/functions/ai-stream/handler_test.ts`	AI requests use a shared 130-second deadline signal for provider and research operations. Tests check abort reasons and OpenRouter provider preferences without `sort` or `order`.

Estimated code review effort: 4 (Complex) | ~60 minutes

Note

🎁 Summarized by CodeRabbit Free

Your organization is on the Free plan. CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please upgrade your subscription to CodeRabbit Essentials by visiting https://app.coderabbit.ai/settings/billing.

_{Comment @coderabbitai help to get the list of available commands.}

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Programcode · 2026-09-24T06:20:04Z

@coderabbitai review

coderabbitai · 2026-09-24T06:20:10Z

⚠️ Action not completed

Review rate limited.

A 200 Ask stream can still end without its final event, and the Crossfire test passed even when no Jev signal decision ran. Assert both, and record the accepted risk that pull-request runs can read the live provider keys. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Programcode · 2026-09-24T06:40:04Z

@coderabbitai review

coderabbitai · 2026-09-24T06:40:10Z

⚠️ Action not completed

Review rate limited.

Programcode · 2026-09-24T07:00:05Z

@coderabbitai review

coderabbitai · 2026-09-24T07:00:12Z

⚠️ Action not completed

Review rate limited.

Programcode · 2026-09-24T07:40:05Z

@coderabbitai review

coderabbitai · 2026-09-24T07:40:11Z

✅ Action performed

Review finished.

Once a Jev signal attached to an answer, every later Crossfire request resent the turn with a `signal` key, and the ai function rejected it as "transcript[1] is invalid". The live suite caught it. The fake provider now answers Jev so the credential-free gate exercises the same path. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

- A promo code redeemed in Settings restores AI access for an expired account. - Signed Stripe webhooks grant and revoke Pro; forged and replayed ones change nothing. - Delete All My Data erases the account's rows and leaves another account's intact. - A generated case downloads as a real PDF and DOCX file. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

The pinned live suite cannot see a production model that times out or returns invalid JSON. LIVE_MODEL=as-requested makes the proxy forward the model the Edge Function chose and runs e2e/model-matrix, which picks each catalog model in the Generate page and prints its provider latency. The workflow runs daily and on pull requests that touch AI code. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…e account cap Main can deploy more than the canary account's 10 AI requests a day, so deployment runs skip the AI checks; daily and manual runs make them. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Programcode · 2026-09-24T12:20:06Z

@coderabbitai review

coderabbitai · 2026-09-24T12:20:12Z

⚠️ Action not completed

Review rate limited.

Deployment runs skip the AI requests to stay under the canary account's cap, so they now sign in, open Generate, and require Settings to show the account's Pro plan. That read goes through row-level security, so it fails on a broken session, and it spends no AI request. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Claude Opus 5 needs about 68 s to write a case, so every Opus case generation hit the 60 s provider timeout and returned 500. Each call now shares one 140 s request budget, which stays under Supabase's 150 s response limit. Research, the main call, and its JSON repair retry all use what is left. The model matrix reproduces the failure and proves the fix. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

The post-generation Jev source-usage check shared the request budget, so a model that finished near the deadline could lose its completed case to a 500. The check now follows only the client's signal and its own 8 s Jev timeout, and the provider budget drops to 130 s to leave room for it under Supabase's 150 s limit. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Programcode · 2026-09-24T12:40:06Z

@coderabbitai review

coderabbitai · 2026-09-24T12:40:11Z

✅ Action performed

Review finished.

Programcode · 2026-09-24T15:20:05Z

@coderabbitai review

coderabbitai · 2026-09-24T15:20:11Z

⚠️ Action not completed

Review rate limited.

Programcode · 2026-09-24T15:40:06Z

@coderabbitai review

coderabbitai · 2026-09-24T15:40:12Z

⚠️ Action not completed

Review rate limited.

Programcode · 2026-09-24T16:00:05Z

@coderabbitai review

coderabbitai · 2026-09-24T16:00:11Z

⚠️ Action not completed

Review rate limited.

Programcode · 2026-09-24T16:40:05Z

@coderabbitai review

coderabbitai · 2026-09-24T16:40:11Z

✅ Action performed

Review finished.

Programcode · 2026-09-24T17:40:05Z

@coderabbitai review

coderabbitai · 2026-09-24T17:40:11Z

✅ Action performed

Review finished.

Copilot AI lite review requested due to automatic review settings September 24, 2026 06:01

Copilot AI reviewed Sep 24, 2026

vercel Bot deployed to Preview September 24, 2026 06:01

View deployment

ci(db-gate): show why supabase start failed before retrying

1279d7c

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

vercel Bot deployed to Preview September 24, 2026 06:12

View deployment

greptile-apps Bot reviewed Sep 24, 2026

View reviewed changes

Comment thread .github/workflows/live-e2e.yml Resolved

Comment thread e2e/live/models.spec.ts Resolved

Comment thread e2e/live/models.spec.ts Resolved

Comment thread e2e/production/canary.spec.ts Resolved

This comment has been minimized.

Sign in to view

Programcode and others added 2 commits September 24, 2026 06:33

test(canary): check the method and headers a browser preflight needs

7f3661d

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

vercel Bot deployed to Preview September 24, 2026 06:34

View deployment

greptile-apps Bot reviewed Sep 24, 2026

View reviewed changes

Comment thread .github/workflows/prod-canary.yml Resolved

docs(agents): scope the production canary credential exception

2f83384

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

vercel Bot deployed to Preview September 24, 2026 06:40

View deployment

vercel Bot deployed to Preview September 24, 2026 11:24

View deployment

Programcode and others added 3 commits September 24, 2026 11:48

vercel Bot deployed to Preview September 24, 2026 11:57

View deployment

Programcode and others added 3 commits September 24, 2026 12:20

vercel Bot deployed to Preview September 24, 2026 12:21

View deployment

test(ai): cover request budget and balanced routing

f0868bb

vercel Bot deployed to Preview September 24, 2026 15:13

View deployment

test(canary): require complete streamed answers

d955a58

vercel Bot deployed to Preview September 24, 2026 15:20

View deployment

greptile-apps Bot reviewed Sep 24, 2026

View reviewed changes

Comment thread docs/testing.md Resolved

docs(testing): keep canary identity private

26cc3f7

vercel Bot deployed to Preview September 24, 2026 15:41

View deployment

docs(testing): describe the shared AI request budget

6455bba

vercel Bot deployed to Preview September 24, 2026 16:22

View deployment

test(live): use balanced multi-provider routing

da2782f

vercel Bot deployed to Preview September 24, 2026 17:25

View deployment

Programcode merged commit 74d11eb into main Sep 24, 2026
7 of 11 checks passed

Conversation

Programcode commented Sep 24, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

What this adds

Evidence

What the live checks found

Limits

Summary by CodeRabbit

Final integration

Confidence Score: 5/5

Summary

Uh oh!

vercel Bot commented Sep 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot commented Sep 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reviews paused

Walkthrough

Changes

Uh oh!

Programcode commented Sep 24, 2026

Uh oh!

coderabbitai Bot commented Sep 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment has been minimized.

Uh oh!

Programcode commented Sep 24, 2026

Uh oh!

coderabbitai Bot commented Sep 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Programcode commented Sep 24, 2026

Uh oh!

coderabbitai Bot commented Sep 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Programcode commented Sep 24, 2026

Uh oh!

coderabbitai Bot commented Sep 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Programcode commented Sep 24, 2026

Uh oh!

coderabbitai Bot commented Sep 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Programcode commented Sep 24, 2026

Uh oh!

coderabbitai Bot commented Sep 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Programcode commented Sep 24, 2026

Uh oh!

coderabbitai Bot commented Sep 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Programcode commented Sep 24, 2026

Uh oh!

coderabbitai Bot commented Sep 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Programcode commented Sep 24, 2026

Uh oh!

coderabbitai Bot commented Sep 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Programcode commented Sep 24, 2026

Uh oh!

coderabbitai Bot commented Sep 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Programcode commented Sep 24, 2026 •

edited by coderabbitai Bot

Loading

vercel Bot commented Sep 24, 2026 •

edited

Loading

coderabbitai Bot commented Sep 24, 2026 •

edited

Loading

coderabbitai Bot commented Sep 24, 2026 •

edited

Loading

coderabbitai Bot commented Sep 24, 2026 •

edited

Loading

coderabbitai Bot commented Sep 24, 2026 •

edited

Loading

coderabbitai Bot commented Sep 24, 2026 •

edited

Loading

coderabbitai Bot commented Sep 24, 2026 •

edited

Loading

coderabbitai Bot commented Sep 24, 2026 •

edited

Loading

coderabbitai Bot commented Sep 24, 2026 •

edited

Loading

coderabbitai Bot commented Sep 24, 2026 •

edited

Loading

coderabbitai Bot commented Sep 24, 2026 •

edited

Loading

coderabbitai Bot commented Sep 24, 2026 •

edited

Loading

coderabbitai Bot commented Sep 24, 2026 •

edited

Loading