Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion e2e/staging/auth.setup.ts
Original file line number Diff line number Diff line change
Expand Up @@ -118,7 +118,6 @@ setup('authenticate with the protected staging demo code', async ({ page }) => {
auth: { persistSession: false, autoRefreshToken: false, detectSessionInUrl: false },
});
const { data, error } = await supabase.auth.verifyOtp({
email: handoff.email,
token_hash: handoff.tokenHash,
type: 'magiclink',
});
Expand Down
4 changes: 2 additions & 2 deletions e2e/staging/authed-flows.spec.ts
Original file line number Diff line number Diff line change
Expand Up @@ -58,15 +58,15 @@ test('real staging session persists, writes and removes a source, and completes
const saveResponseButton = page.getByRole('button', { name: 'Save message' }).last();
await expect(saveResponseButton).toBeAttached({ timeout: 90_000 });
const assistantBubble = saveResponseButton.locator('..');
await expect(assistantBubble.locator(':scope > .text-sm')).toHaveText(/\S/, { timeout: 90_000 });
await expect(assistantBubble.locator(':scope > div.text-sm')).toHaveText(/\S/, { timeout: 90_000 });
} finally {
if (sourceCreated) {
await page.goto('/app/library');
await expect(page.getByText(sourceTitle, { exact: true })).toBeVisible({ timeout: 15_000 });
const sourceDelete = page.waitForResponse((response) =>
response.request().method() === 'DELETE' && response.url().includes('/rest/v1/sources?')
);
await page.getByRole('button', { name: `Delete ${sourceTitle}` }).click();
await page.getByRole('button', { name: `Delete ${sourceTitle}`, exact: true }).click();
expect((await sourceDelete).ok()).toBe(true);
await page.evaluate(() => localStorage.removeItem('debate-trainer-storage'));
await page.reload();
Expand Down
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
"preview": "vite preview",
"test": "vitest run",
"test:watch": "vitest",
"test:edge": "deno test --allow-read --allow-env=DEMO_CODE_PEPPER,EXA_API_KEY,SUPABASE_URL,SUPABASE_SERVICE_ROLE_KEY,SPARRING_INTEGRITY_SECRET,OPENAI_API_KEY --allow-net=127.0.0.1 supabase/functions/",
"test:edge": "deno test --allow-read --allow-env=DEMO_CODE_PEPPER,EXA_API_KEY,SUPABASE_URL,SUPABASE_SERVICE_ROLE_KEY,SPARRING_INTEGRITY_SECRET,OPENAI_API_KEY,GOOGLE_API_KEY,ANTHROPIC_API_KEY --allow-net=127.0.0.1 supabase/functions/",
"test:db": "supabase test db",
"test:e2e": "playwright test --project=desktop-chromium --project=mobile-chromium",
"test:e2e:webkit": "playwright test --project=webkit",
Expand Down
112 changes: 112 additions & 0 deletions plans/prompt-eval-sheet.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,112 @@
# Prompt evaluation v2

Evaluation started 2026-09-07 UTC, resumed 2026-09-09 UTC. This is subjective agent assessment of synthetic model responses, not an automated quality gate or a human panel review.

## Reproduction

Before revision: `235bb49`. After revision: the final shared prompt version committed with this v2 sheet, identified by the system-prompt hashes below. Use the same request configuration for both revisions. Only the system prompt changes.

- Endpoint: `https://openrouter.ai/api/v1/chat/completions`.
- Model: `openai/gpt-4o-mini`, temperature `0`, seed `42`, `max_tokens: 1600`.
- No tools, research, local sources, custom instructions, admin instructions, history, or response-format override. Provider routing is automatic; a seed is not a determinism guarantee.
- General Ask, citations/search/pros-cons/argument-map all disabled.
- Structured system: `buildSystemPrompt({ feature: "ask", dateContext: "Current date: 2026-09-07." })`.
- Structured user: `buildAskBasePrompt` with the synthetic message and all toggles plus `researchAvailable` false.
- Before Markdown system: the five lines in baseline `ai-stream/index.ts`'s `buildMessages`, with `getAskModeGuidance("general")`.
- After Markdown system: `buildStreamingAskSystemPrompt(getAskModeGuidance("general"))`.
- Markdown user: the synthetic message directly. The model was sampled with the streaming system prompt through a complete-response API call. SSE transport is tested separately by the automated streaming suite.
- Validate unmodified structured output with `JSON.parse` and `AskOutputSchema.safeParse`. Do not repair omitted keys before recording validity.

Each of the three motions is crossed with all three complexity instructions, in both Markdown and JSON, before and after. That is 36 responses in the final comparison. An earlier 18-response after batch preceded the review fixes, for 54 completed calls overall; the after observations below use only the final batch. Message construction is exactly:

```text
Motion: {motion}
{complexity instruction}
Respond entirely in {language label}. Do not invent facts, statistics, or citations. No research is attached. Keep the answer under 220 words.
```

Motions and output language:

1. `phones`, English: `Schools should ban smartphones during lessons.`
2. `transit`, Spanish: `El transporte público urbano debería ser gratuito.`
3. `ai`, Hungarian: `Az iskoláknak engedélyezniük kell a generatív mesterséges intelligencia használatát a házi feladatokban.`

Complexity instructions:

- Beginner: `Explain one argument and one counterargument for a beginner in plain language.`
- Intermediate: `Compare two competing mechanisms and explain which warrant a judge should test.`
- Advanced: `Analyze burden of proof, a strong countermodel, and impact weighing under uncertainty.`


System-prompt SHA-256 hashes used in this evaluation:

- before json: `aaa82b8400622789bd0d2ee05eeb69b24c47c9e30faadcadbbc281538a268b49`
- before stream: `dca86dd2269438966851e44c7f6c20a7b8007be69a7168c8dfc6b5b620037542`
- after json: `3ef82d881b727f3f1d894dd182efdea4d38e02414ecc7f4d2cbe8c94c100b325`
- after stream: `07f5c00df06c542c29729918374fde57c518a2006d9170d2a0f280d99a3fd3d4`

## Assessment rules

Quality and control obedience use 1 to 5: 1 is unusable, 3 is useful with material weaknesses, 5 is precise and complete. Evidence honesty distinguishes invented sources from unsupported generalizations. Language consistency means all prose fields stay in the requested language; schema keys remain English. Hallucination risk is a subjective low/medium/high judgment, not a factual audit. JSON syntax and schema validity are separate observations; Markdown has no JSON requirement.

## Before observations

All 18 responses completed without truncation. All nine JSON outputs parsed, but only eight passed the actual Ask schema. The advanced phones response omitted both `followUpQuestions` and `uncertainties`. All 18 used the requested output language. No response invented a named citation, URL, study, or statistic. Several supplied unsupported causal generalizations.

| Run | Quality | Controls | Evidence honesty | Language | Risk | JSON syntax/schema | Observation |
| --- | --- | --- | --- | --- | --- | --- | --- |
| phones beginner Markdown | 3 | 4 | Generalizations | English | Medium | N/A | Clear distraction/tool clash; assumes improved learning. |
| phones beginner JSON | 4 | 4 | Generalizations | English | Medium | Pass/pass | Useful counterproposal; uncertain educational context surfaced. |
| phones intermediate Markdown | 3 | 4 | Generalizations | English | Medium | N/A | Names mechanisms but gives a weak reason to prioritize distraction. |
| phones intermediate JSON | 3 | 4 | Overstatement | English | Medium | Pass/pass | Claims a ban eliminates distractions entirely. |
| phones advanced Markdown | 3 | 4 | Generalizations | English | Medium | N/A | Countermodel is clear; impact comparison stays generic. |
| phones advanced JSON | 3 | 2 | Generalizations | English | Medium | Pass/fail | Required follow-up and uncertainty arrays missing. |
| transit beginner Markdown | 4 | 4 | Conditional claims | Spanish | Low | N/A | Explains access, funding and crowding without invented data. |
| transit beginner JSON | 3 | 4 | Speculation | Spanish | Medium | Pass/pass | Adds an unsupported free-service quality-perception claim. |
| transit intermediate Markdown | 2 | 3 | Generalizations | Spanish | Medium | N/A | Taxes and subsidies are not clearly competing funding mechanisms. |
| transit intermediate JSON | 2 | 3 | Generalizations | Spanish | Medium | Pass/pass | Same mechanism overlap; acknowledges missing local data. |
| transit advanced Markdown | 4 | 4 | Conditional claims | Spanish | Low | N/A | Targeted fares countermodel and service-quality tradeoff. |
| transit advanced JSON | 4 | 4 | Conditional claims | Spanish | Low | Pass/pass | Financing uncertainty is explicit; useful countermodel. |
| ai beginner Markdown | 3 | 4 | Generalizations | Hungarian | Medium | N/A | Accessible help/dependency clash with a generic closing. |
| ai beginner JSON | 4 | 4 | Conditional claims | Hungarian | Low | Pass/pass | Explains learning support versus independent practice. |
| ai intermediate Markdown | 3 | 3 | Generalizations | Hungarian | Medium | N/A | Compares permission/restriction more than causal mechanisms. |
| ai intermediate JSON | 3 | 4 | Generalizations | Hungarian | Medium | Pass/pass | Judge test is concrete; conflates generated work and plagiarism. |
| ai advanced Markdown | 2 | 3 | Speculation | Hungarian | Medium | N/A | Introduces mental health without developing the causal link. |
| ai advanced JSON | 3 | 4 | Generalizations | Hungarian | Medium | Pass/pass | Countermodel and uncertainty exist, but weighing is generic. |

## After observations

All 18 final after responses completed without truncation. All nine structured responses parsed and passed `AskOutputSchema`; all nine Markdown responses stayed freeform. All responses kept the requested language and none invented named citations, URLs, studies, or statistics.

The schema improvement is observed in this sample, not proof of a reliable rate increase. Coaching quality did not improve consistently. All after JSON responses used empty uncertainty arrays; only the beginner phones response supplied follow-up questions. Four responses included disabled optional analysis fields, including a populated argument map in the beginner phones response. Existing server post-processing removes these fields. No schema or control enforcement was weakened.

| Run | Quality | Controls | Evidence honesty | Language | Risk | JSON syntax/schema | Observation |
| --- | --- | --- | --- | --- | --- | --- | --- |
| phones beginner Markdown | 3 | 4 | Generalizations | English | Medium | N/A | Clear clash; unsupported learning-outcome assumptions remain. |
| phones beginner JSON | 3 | 2 | Generalizations | English | Medium | Pass/pass | Emits disabled argument map and pros/cons; server removes them. |
| phones intermediate Markdown | 3 | 4 | Generalizations | English | Medium | N/A | Gives a judge test, but prioritization remains weak. |
| phones intermediate JSON | 3 | 4 | Overstatement | English | Medium | Pass/pass | Again claims a ban eliminates distractions entirely. |
| phones advanced Markdown | 3 | 4 | Generalizations | English | Medium | N/A | Clear regulated-use model; impact comparison stays broad. |
| phones advanced JSON | 3 | 3 | Generalizations | English | Medium | Pass/pass | Required arrays retained; emits disabled empty analysis fields. |
| transit beginner Markdown | 4 | 4 | Conditional claims | Spanish | Low | N/A | Useful access versus financing comparison. |
| transit beginner JSON | 4 | 4 | Conditional claims | Spanish | Low | Pass/pass | Explains mobility benefits and budget displacement. |
| transit intermediate Markdown | 2 | 3 | Generalizations | Spanish | Medium | N/A | Taxes and subsidies remain overlapping mechanisms. |
| transit intermediate JSON | 3 | 4 | Speculation | Spanish | Medium | Pass/pass | Distinct route-optimization mechanism; savings are unproven. |
| transit advanced Markdown | 4 | 4 | Conditional claims | Spanish | Low | N/A | Targeted fares countermodel and funding uncertainty. |
| transit advanced JSON | 3 | 3 | Conditional claims | Spanish | Low | Pass/pass | Funding objection substitutes for a policy countermodel; disabled fields emitted. |
| ai beginner Markdown | 3 | 4 | Generalizations | Hungarian | Medium | N/A | Understandable, but translates motion awkwardly as physical movement. |
| ai beginner JSON | 4 | 4 | Conditional claims | Hungarian | Low | Pass/pass | Clear learning support versus independence clash. |
| ai intermediate Markdown | 3 | 3 | Generalizations | Hungarian | Medium | N/A | Policy labels still substitute for developed mechanisms. |
| ai intermediate JSON | 3 | 4 | Generalizations | Hungarian | Medium | Pass/pass | Concrete judge tests; conflates generated work and plagiarism. |
| ai advanced Markdown | 3 | 3 | Speculation | Hungarian | Medium | N/A | Discusses motivation risk without a developed causal link. |
| ai advanced JSON | 2 | 3 | Generalizations | Hungarian | Medium | Pass/pass | Awkward burden-of-proof terminology; disabled empty fields emitted. |

### Assessment

The narrow contract change is supported: streaming remains Markdown, language stays consistent, and required JSON fields are present in all nine after samples. This baseline does not support a claim of better coaching overall. Retain schema validation, control post-processing and evidence warnings; do not begin provider-specific tuning from these results alone.

All 54 completed calls reported provider `OpenAI`, model `openai/gpt-4o-mini`. The exact prompts, synthetic inputs, request hashes and per-response assessments make the comparison repeatable, although model/backend revisions can change later outputs.

## Limits

These calls exercise prompt behavior through OpenRouter, not the app's native OpenAI provider connection or authenticated Supabase handler. Nine motion/complexity pairs do not establish provider-wide reliability. Language was requested explicitly, so these samples do not prove automatic language inference. No tuning for a particular provider is justified by this small sample alone. Generate’s `needsSource` field is a model-authored warning, not a verified-source badge. Available sources do not establish support for every contention; this work preserves model warnings and forces warnings when no eligible evidence exists, without claiming semantic verification. One transport disconnect in the intermediate batch was retried with the identical request. Raw responses and credentials are not committed. Synthetic response artifacts and request hashes are retained privately under `/tmp/debatelab-prompt-eval-20260907` on the execution host.
8 changes: 5 additions & 3 deletions plans/system-prompt-improvement.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# System prompt improvement plan

**Status:** Remaining work as of 2026-09-01 on `main` (`79b65ca`). Re-verified: PRs #16-#28 did not touch the prompt builders.
**Status:** Implementation and manual baseline completed on the isolated `t3code/system-prompt-improvements` branch, based on `235bb49`. Credential-free and protected staging verification passed. Release verification and rollout are tracked in PR #42. See [prompt-eval-sheet.md](prompt-eval-sheet.md) for the dated 36-response comparison and its limits.

## Preconditions

Expand Down Expand Up @@ -59,7 +59,7 @@ Run focused tests after each fixture group:

```bash
npx vitest run supabase/functions/_shared/prompts.test.ts supabase/functions/ai/prompt-builder.test.ts supabase/functions/ai/control-matrix.test.ts src/test/supabase/prompt-builder.test.ts
npx vitest run src/test/ai/streaming.test.ts src/test/ai/streaming-integration.test.ts
npx vitest run supabase/functions/ai-stream/prompt.test.ts src/test/ai/streaming.test.ts src/test/ai/streaming-integration.test.ts
```

Then run the repository gates:
Expand All @@ -70,10 +70,12 @@ npm run typecheck
npm test -- --coverage
npm run build
deno check supabase/functions/_shared/prompts.ts supabase/functions/ai/index.ts supabase/functions/ai-stream/index.ts
deno test --allow-read --allow-env=EXA_API_KEY --allow-net=127.0.0.1 supabase/functions/
npm run test:edge
CI=1 PORT=18082 PLAYWRIGHT_BASE_URL=http://127.0.0.1:18082 npm run test:e2e -- e2e/authed-pages.spec.ts e2e/sparring.spec.ts e2e/crossfire.spec.ts
```

The maintained `test:edge` script includes the environment permissions required by handler tests. The earlier `--allow-env=EXA_API_KEY` command fails when current tests set their synthetic Supabase and provider environment. Streaming handler tests run under native Deno.

Confirm port `18082` is free before the browser run; the explicit `PORT` and `PLAYWRIGHT_BASE_URL` prevent reuse of the dev server's default `8080`. Use mocked provider responses and authenticated fixtures. Acceptance requires every visible control to have a tested prompt or post-processing effect, all structured outputs to pass their schema gates, streaming Ask to remain Markdown-only, no citation rule to leak into a no-research path, and the manual sheet to be dated and reproducible.

## Rollout and authorization boundary
Expand Down
2 changes: 1 addition & 1 deletion src/lib/ai/schemas/factoids.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ export const FactoidsInputSchema = z.object({
region: z.string().optional(),
sourceType: z.enum(['academic', 'government', 'news', 'any']),
timeframe: z.string().optional(),
sourceCount: z.coerce.number().min(3).max(15).default(5),
sourceCount: z.coerce.number().int().min(3).max(15).default(5),
});

export type FactoidsInput = z.infer<typeof FactoidsInputSchema>;
Expand Down
4 changes: 4 additions & 0 deletions src/test/schemas.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -442,6 +442,10 @@ describe('Factoids Schema', () => {
expect(FactoidsInputSchema.safeParse({ ...validInput, sourceType: 'blog' }).success).toBe(false);
});

it('rejects fractional source counts', () => {
expect(FactoidsInputSchema.safeParse({ ...validInput, sourceCount: 3.5 }).success).toBe(false);
});

it('accepts all valid sourceTypes', () => {
expect(FactoidsInputSchema.safeParse({ ...validInput, sourceType: 'academic' }).success).toBe(true);
expect(FactoidsInputSchema.safeParse({ ...validInput, sourceType: 'government' }).success).toBe(true);
Expand Down
Loading
Loading