Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .superset/db-gate.sh
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,9 @@ FAKE_PROVIDER=debatelab-fake-openrouter
# the fake provider. It needs LIVE_OPENROUTER_API_KEY and LIVE_EXA_API_KEY.
# LIVE_MODEL=as-requested runs e2e/model-matrix instead: one case per catalog model.
LIVE_PROVIDERS="${LIVE_PROVIDERS:-0}"
# GLM Flash is inexpensive and has broad endpoint coverage, so OpenRouter's
# default balanced routing can fail over between providers during this gate.
LIVE_MODEL="${LIVE_MODEL:-z-ai/glm-5.3-flash}" # empty also means default
# Muse Spark 1.3 Contributor is the cheapest model that supports every request
# parameter the app sends. It is a test-only model: it is not in the app catalog.
LIVE_MODEL="${LIVE_MODEL:-meta/muse-spark-1.3-contributor}" # empty also means default
if [ "$LIVE_PROVIDERS" = 1 ] && { [ -z "${LIVE_OPENROUTER_API_KEY:-}" ] || [ -z "${LIVE_EXA_API_KEY:-}" ]; }; then
echo "LIVE_PROVIDERS=1 requires LIVE_OPENROUTER_API_KEY and LIVE_EXA_API_KEY" >&2
exit 1
Expand Down
2 changes: 1 addition & 1 deletion docs/testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -193,7 +193,7 @@ These journeys do not prove live provider behavior, web research (the local stac
LIVE_PROVIDERS=1 LIVE_OPENROUTER_API_KEY=... LIVE_EXA_API_KEY=... ./.superset/db-gate.sh
```

Live mode uses the same locked stack but skips pgTAP and the fake-provider specs. It runs `e2e/live/` on desktop Chromium against real OpenRouter, Jev, and Exa. Imported web pages are fetched live too. The fake provider becomes a recording proxy. It forwards every chat call to OpenRouter with the model pinned to `LIVE_MODEL` (default `z-ai/glm-5.3-flash`; CI reads the `LIVE_MODEL` repository variable), passes Jev decisions through unchanged, and refuses calls past `MAX_UPSTREAM_CALLS` (200). The default is inexpensive and broadly hosted, so requests can use OpenRouter's balanced provider routing instead of depending on one endpoint.
Live mode uses the same locked stack but skips pgTAP and the fake-provider specs. It runs `e2e/live/` on desktop Chromium against real OpenRouter, Jev, and Exa. Imported web pages are fetched live too. The fake provider becomes a recording proxy. It forwards every chat call to OpenRouter with the model pinned to `LIVE_MODEL` (default `meta/muse-spark-1.3-contributor`; CI reads the `LIVE_MODEL` repository variable, which is unset), passes Jev decisions through unchanged, and refuses calls past `MAX_UPSTREAM_CALLS` (200). The default is the cheapest model that supports every request parameter the app sends, including `response_format` and reasoning effort. It is test-only and not in the app catalog, so users cannot pick it. Meta is its only provider, so an outage there fails the gate instead of failing over.

The specs cover web-researched case generation, verified Factoids citations, Ask with web search, URL import, streamed Ask, Attack, Brainstorm, a judged sparring round, and Crossfire with live Jev signals. Outputs vary, so tests assert structure rather than text:

Expand Down
5 changes: 4 additions & 1 deletion e2e/live/models.spec.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { admin, expect, fake, stackReady, test } from "../full-stack/stack";
import { admin, chooseForMotion, expect, fake, stackReady, test } from "../full-stack/stack";
import { aiResponse, expectProviderCallsHealthy, LIVE_TIMEOUT } from "./live";

test.skip(!stackReady, "Live gate is required (LIVE_PROVIDERS=1 ./.superset/db-gate.sh)");
Expand Down Expand Up @@ -33,6 +33,7 @@ test("Attack and Brainstorm return valid structured analyses", async ({ signedIn
await page.getByLabel("Motion / Topic *").fill("Universal basic income should replace targeted welfare");
const brainstorm = aiResponse(page, "brainstorm");
await page.getByRole("button", { name: "Start Brainstorming" }).click();
await chooseForMotion(page, "once");
expect(((await brainstorm).output.topFive as unknown[]).length).toBeGreaterThan(0);
await expectProviderCallsHealthy(["attack", "brainstorm"]);
});
Expand All @@ -43,6 +44,7 @@ test("a live sparring round is judged and saved", async ({ signedIn: page, user
await page.getByRole("combobox", { name: "Round format" }).click();
await page.getByRole("option", { name: /Lincoln-Douglas · 5 speeches/ }).click();
await page.getByRole("button", { name: "Start round" }).click();
await chooseForMotion(page, "create");

const speeches = [
["Affirmative constructive", "Universal access removes the direct price barrier to participation in work, school, and civic life."],
Expand Down Expand Up @@ -70,6 +72,7 @@ test("Crossfire answers live and produces a debrief", async ({ signedIn: page })
await page.getByTestId("crossfire-argument").fill("Private cars create congestion and unsafe streets.");
await page.getByTestId("mode-question-ai").click();
await page.getByTestId("start-crossfire").click();
await chooseForMotion(page, "create");

await page.getByTestId("crossfire-input").fill("What evidence links fewer private cars to safer streets?");
const turn = aiResponse(page, "crossfire");
Expand Down
5 changes: 4 additions & 1 deletion e2e/live/research.spec.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { admin, expect, stackReady, test } from "../full-stack/stack";
import { admin, chooseForMotion, expect, stackReady, startDebate, test } from "../full-stack/stack";
import { aiResponse, expectProviderCallsHealthy, LIVE_TIMEOUT } from "./live";

test.skip(!stackReady, "Live gate is required (LIVE_PROVIDERS=1 ./.superset/db-gate.sh)");
Expand All @@ -11,6 +11,7 @@ test("case generation researches the web and cites what it found", async ({ sign
await page.getByRole("textbox", { name: "Motion / Topic *" }).fill("This House would make public transport free at the point of use");
const response = aiResponse(page, "generate");
await page.getByRole("button", { name: "Generate Debate Case" }).click();
await chooseForMotion(page, "once");
const body = await response;

// Standard length asks for exactly three contentions.
Expand Down Expand Up @@ -63,6 +64,8 @@ test("Ask with web search shows the sources it used", async ({ signedIn: page })

test("a web page imports into the Library", async ({ signedIn: page, user }) => {
const url = "https://en.wikipedia.org/wiki/Free_public_transport";
// Sources belong to a debate, so the Library needs a Current Debate first.
await startDebate(page, "This House would make public transport free");
await page.goto("/app/library");
await page.getByRole("button", { name: "Add Source" }).click();
await page.getByRole("textbox", { name: "Import from URL" }).fill(url);
Expand Down
Loading