Skip to content

Commit da2782f

Browse files
committed
test(live): use balanced multi-provider routing
1 parent 6455bba commit da2782f

3 files changed

Lines changed: 23 additions & 12 deletions

File tree

‎.superset/db-gate.sh‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,9 @@ FAKE_PROVIDER=debatelab-fake-openrouter
1818
# the fake provider. It needs LIVE_OPENROUTER_API_KEY and LIVE_EXA_API_KEY.
1919
# LIVE_MODEL=as-requested runs e2e/model-matrix instead: one case per catalog model.
2020
LIVE_PROVIDERS="${LIVE_PROVIDERS:-0}"
21-
LIVE_MODEL="${LIVE_MODEL:-meta/muse-spark-1.3-contributor}" # empty also means default
21+
# GLM Flash is inexpensive and has broad endpoint coverage, so OpenRouter's
22+
# default balanced routing can fail over between providers during this gate.
23+
LIVE_MODEL="${LIVE_MODEL:-z-ai/glm-5.3-flash}" # empty also means default
2224
if [ "$LIVE_PROVIDERS" = 1 ] && { [ -z "${LIVE_OPENROUTER_API_KEY:-}" ] || [ -z "${LIVE_EXA_API_KEY:-}" ]; }; then
2325
echo "LIVE_PROVIDERS=1 requires LIVE_OPENROUTER_API_KEY and LIVE_EXA_API_KEY" >&2
2426
exit 1

‎docs/testing.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -179,7 +179,7 @@ These journeys do not prove live provider behavior, web research (the local stac
179179
LIVE_PROVIDERS=1 LIVE_OPENROUTER_API_KEY=... LIVE_EXA_API_KEY=... ./.superset/db-gate.sh
180180
```
181181

182-
Live mode uses the same locked stack but skips pgTAP and the fake-provider specs. It runs `e2e/live/` on desktop Chromium against real OpenRouter, Jev, and Exa. Imported web pages are fetched live too. The fake provider becomes a recording proxy. It forwards every chat call to OpenRouter with the model pinned to `LIVE_MODEL` (default `meta/muse-spark-1.3-contributor`; CI reads the `LIVE_MODEL` repository variable), passes Jev decisions through unchanged, and refuses calls past `MAX_UPSTREAM_CALLS` (200). Contributor models only serve accounts that allow paid model training in OpenRouter privacy settings.
182+
Live mode uses the same locked stack but skips pgTAP and the fake-provider specs. It runs `e2e/live/` on desktop Chromium against real OpenRouter, Jev, and Exa. Imported web pages are fetched live too. The fake provider becomes a recording proxy. It forwards every chat call to OpenRouter with the model pinned to `LIVE_MODEL` (default `z-ai/glm-5.3-flash`; CI reads the `LIVE_MODEL` repository variable), passes Jev decisions through unchanged, and refuses calls past `MAX_UPSTREAM_CALLS` (200). The default is inexpensive and broadly hosted, so requests can use OpenRouter's balanced provider routing instead of depending on one endpoint.
183183

184184
The specs cover web-researched case generation, verified Factoids citations, Ask with web search, URL import, streamed Ask, Attack, Brainstorm, a judged sparring round, and Crossfire with live Jev signals. Outputs vary, so tests assert structure rather than text:
185185

‎e2e/full-stack/fake-openrouter.mjs‎

Lines changed: 19 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -223,16 +223,25 @@ async function proxy(req, res, path) {
223223
return json(res, 429, { error: { message: `Live gate stopped after ${maxUpstreamCalls} provider calls` } });
224224
}
225225
const started = Date.now();
226-
const response = await fetch(`${upstream}${path.replace(/^\/api/, "")}`, {
227-
method: "POST",
228-
headers: { authorization: req.headers.authorization ?? "", "content-type": "application/json" },
229-
body: JSON.stringify(chat ? { ...body, model: chatModel(body.model) } : body),
230-
});
231-
record.status = response.status;
232-
res.writeHead(response.status, { "content-type": response.headers.get("content-type") ?? "application/json" });
233-
for await (const chunk of response.body ?? []) res.write(chunk);
234-
res.end();
235-
record.ms = Date.now() - started;
226+
try {
227+
const response = await fetch(`${upstream}${path.replace(/^\/api/, "")}`, {
228+
method: "POST",
229+
headers: { authorization: req.headers.authorization ?? "", "content-type": "application/json" },
230+
body: JSON.stringify(chat ? { ...body, model: chatModel(body.model) } : body),
231+
});
232+
record.status = response.status;
233+
res.writeHead(response.status, { "content-type": response.headers.get("content-type") ?? "application/json" });
234+
for await (const chunk of response.body ?? []) res.write(chunk);
235+
res.end();
236+
} catch (error) {
237+
record.status = 502;
238+
const message = error instanceof Error ? error.message : "upstream connection failed";
239+
console.error(`live provider request failed: ${message}`);
240+
if (!res.headersSent) json(res, 502, { error: { message: "Live provider request failed" } });
241+
else res.destroy();
242+
} finally {
243+
record.ms = Date.now() - started;
244+
}
236245
}
237246

238247
createServer(async (req, res) => {

0 commit comments

Comments
 (0)