From 36aba00207b838968859de8fe7dafbb4c6ff1921 Mon Sep 17 00:00:00 2001 From: Adam Date: Fri, 2 Oct 2026 20:49:24 +0000 Subject: [PATCH 1/2] fix(e2e): start a Current Debate in the staging spec Since #72, Library disables Add Source until a debate is current. The staging spec never started one, so every run since Actions came back on 2026-10-01 waited out its 180-second timeout on a disabled button. The spec now starts a uniquely named debate from Library, sweeps debates an interrupted run left behind, and deletes its debate with the test user's own session after deleting the source through the UI. Actions time out after 30 seconds so cleanup still runs, and a failure logs the step name and URL path only. Docs: record the real cause in operations.md and describe the new suite steps, port and failure output in testing.md. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/operations.md | 6 +- docs/testing.md | 12 +- e2e/staging/authed-flows.spec.ts | 214 +++++++++++++++++++++++-------- 3 files changed, 173 insertions(+), 59 deletions(-) diff --git a/docs/operations.md b/docs/operations.md index 9203a6cfa..740eb7952 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -63,7 +63,9 @@ The staging browser suite disables trace, screenshot, and video capture. It crea Do not enable the staging repository variable before every prerequisite is present. A skipped staging job is not a pass. -`STAGING_E2E_ENABLED` has been `true` since 2026-09-18. The job runs after every successful `Quality` run from a push to `main`. Quality runs from pull requests also trigger it, and those show as skipped by design. Check the latest real result with `gh run list --workflow staging-e2e.yml` and `npm run ci:failures -- `, not this page. The run uploads no artifact, so the job log is the only evidence. +`STAGING_E2E_ENABLED` has been `true` since 2026-09-18. The job runs after every successful `Quality` run from a push to `main`. Quality runs from pull requests also trigger it, and those show as skipped by design. Check the latest real result with `gh run list --workflow staging-e2e.yml` and `npm run ci:failures -- `, not this page. The run uploads no artifact, so the job log is the only evidence. On failure the log names the failed step and the URL path, for example `Staging step "add a source to the debate" failed at path /app/library`. + +No staging run passed after 2026-09-23. Until 2026-10-01 the job did not run, because the `Quality` run it waits for never started. Once Actions came back, it failed on every `main` push through 2026-10-02. The cause was the spec, not staging: since the Current Debate change (#72), Library disables **Add Source** until a debate is current, and the spec never started one. The spec now starts a uniquely named debate and deletes it afterwards. The suite drives the real UI, so a change to a flow it covers can break it with no backend drift; run it locally before blaming staging. ## Staging sync @@ -195,7 +197,7 @@ Do not expose protected data in artifacts. Record the commit, environment, opera ## Incident clues - Every Actions job fails in 2 to 5 seconds with no logs: Actions minutes or billing, not the code. Verify locally. -- Green local E2E plus staging failure: schema, functions, provider access, demo code, or CORS may differ. +- Green local E2E plus staging failure: schema, functions, provider access, demo code, or CORS may differ. Run the staging suite locally first; if it fails there too on a UI step, the spec is stale, not staging. - New frontend errors against Supabase right after a merge: the matching migration or function may not be applied to production yet. Run `npm run backend:status` from the linked checkout. - Function returns unauthorized after deploy: check handler auth and environment propagation before changing gateway settings. - Browser works on one route only: check the `vercel.json` SPA rewrite and cache headers. diff --git a/docs/testing.md b/docs/testing.md index 0c446771b..1f678ddc9 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -293,11 +293,15 @@ The suite: - verifies that the anon key belongs to the supplied staging URL; - authenticates through `demo-code-login`; - proves session reload; -- writes and reads one uniquely named source; -- makes one bounded AI request; -- deletes only the source it created. +- removes debates an interrupted earlier run left behind (only motions starting `E2E staging debate `); +- starts one uniquely named debate from Library, because sources belong to a Current Debate; +- writes one uniquely named source, then reads it back after clearing local state; +- makes one bounded AI request in Ask; +- deletes the source through the UI and the debate through the REST API with the test user's own session (the app cannot delete the Current Debate), which also removes the chat Ask saved to it. -Temporary auth state lives under `test-results/.auth/`. The staging projects disable traces, screenshots, and video because browser captures can retain authenticated data. +Use port `18080`: the staging Edge Functions answer browser requests only from allowlisted origins (see the [staging contract](operations.md#staging-contract)). The CI demo code exists only in the GitHub `staging` environment, so a local run needs its own bounded staging code. + +Temporary auth state lives under `test-results/.auth/`. The staging projects disable traces, screenshots, and video because browser captures can retain authenticated data. Actions time out after 30 seconds, well inside the 180-second test timeout, so cleanup still runs after a failure. Each failure logs the step name and the URL path, never the query or hash. Playwright still writes an ARIA snapshot to `test-results/**/error-context.md` on failure; the workflow does not upload it, and it can contain staging account text. The GitHub staging workflow runs only after an eligible main-branch quality result or a main-branch manual dispatch, and only when `STAGING_E2E_ENABLED=true`. The protected environment must already contain current secrets, functions, schema, provider access, a dedicated test code, and a main-only deployment rule. diff --git a/e2e/staging/authed-flows.spec.ts b/e2e/staging/authed-flows.spec.ts index 897ba58ea..f9c1bf825 100644 --- a/e2e/staging/authed-flows.spec.ts +++ b/e2e/staging/authed-flows.spec.ts @@ -1,12 +1,73 @@ -import { expect, test } from '@playwright/test'; +import { createClient } from '@supabase/supabase-js'; +import { expect, test, type Page } from '@playwright/test'; test.describe.configure({ mode: 'serial' }); +// A failed action throws well inside the test timeout, so the cleanup in `finally` still runs. +test.use({ actionTimeout: 30_000, navigationTimeout: 30_000 }); -test('real staging session persists, writes and removes a source, and completes one AI request', async ({ page }) => { +const MOTION_PREFIX = 'E2E staging debate '; + +/** + * Runs one named phase as a Playwright step. On failure it logs the phase and + * the URL path only: the query and hash can carry auth handoff tokens, and the + * staging projects keep no traces, screenshots or video. + */ +async function step(page: Page, name: string, body: () => Promise): Promise { + return test.step(name, async () => { + try { + return await body(); + } catch (error) { + const path = page.url().startsWith('http') ? new URL(page.url()).pathname : page.url(); + console.error(`Staging step "${name}" failed at path ${path}`); + test.info().annotations.push({ type: 'failed-step', description: `${name} at ${path}` }); + throw error; + } + }); +} + +/** A REST client acting as the signed-in staging test user, for cleanup only. */ +async function userClient(page: Page) { + const session = await page.evaluate(() => localStorage.getItem('debate-auth-staging')); + const accessToken = session ? (JSON.parse(session) as { access_token?: string }).access_token : undefined; + expect(accessToken, 'staging session for cleanup').toBeTruthy(); + return createClient(process.env.STAGING_SUPABASE_URL!, process.env.STAGING_SUPABASE_ANON_KEY!, { + auth: { persistSession: false, autoRefreshToken: false, detectSessionInUrl: false }, + global: { headers: { Authorization: `Bearer ${accessToken}` } }, + }); +} + +/** + * The debate the spec creates is the Current Debate, and the app only lets a + * student delete a debate that is not current, so cleanup uses the same + * signed-in session through the REST API and deletes that one row by id. + * Its sources and saved chat go with it (ON DELETE CASCADE). + */ +async function deleteOwnDebate(page: Page, debateId: string) { + const { data, error } = await (await userClient(page)).from('debates').delete().eq('id', debateId).select('id'); + expect(error, error?.message).toBeNull(); + expect(data).toHaveLength(1); +} + +/** + * A run killed before its cleanup (a cancelled job, a crashed runner) leaves + * its debate behind as the account's Current Debate. Remove only debates this + * suite names, so every run starts from the same state. + */ +async function deleteLeftoverDebates(page: Page) { + const { data, error } = await (await userClient(page)) + .from('debates').delete().like('motion', `${MOTION_PREFIX}%`).select('id'); + expect(error, error?.message).toBeNull(); + if (data?.length) console.log(`Removed ${data.length} debate(s) left by an earlier staging run`); +} + +test('real staging session persists, starts a debate, writes and removes a source, and completes one AI request', async ({ page }) => { test.setTimeout(180_000); const consoleErrors: string[] = []; - const sourceTitle = `E2E staging source ${Date.now()}-${Math.random().toString(36).slice(2, 8)}`; + const runId = `${Date.now()}-${Math.random().toString(36).slice(2, 8)}`; + const motion = `${MOTION_PREFIX}${runId}`; + const sourceTitle = `E2E staging source ${runId}`; const sourceContent = 'A dedicated Playwright staging record used to verify authenticated persistence.'; + let debateId: string | null = null; let sourceCreated = false; page.on('pageerror', (error) => consoleErrors.push(`pageerror: ${error.message}`)); @@ -15,62 +76,109 @@ test('real staging session persists, writes and removes a source, and completes }); try { - await page.goto('/app/settings'); - const testEmail = await page.evaluate(() => localStorage.getItem('debate-staging-e2e-email')); - expect(testEmail).toBeTruthy(); - await expect(page.getByText(testEmail!, { exact: true })).toBeVisible(); - - await page.reload(); - await expect(page).toHaveURL(/\/app\/settings$/); - await expect(page.getByText(testEmail!, { exact: true })).toBeVisible(); + await step(page, 'session persists across a reload', async () => { + await page.goto('/app/settings'); + const testEmail = await page.evaluate(() => localStorage.getItem('debate-staging-e2e-email')); + expect(testEmail).toBeTruthy(); + await expect(page.getByText(testEmail!, { exact: true })).toBeVisible(); - await page.goto('/app/library'); - await page.getByRole('button', { name: 'Add Source' }).click(); - await page.getByPlaceholder('Source title...').fill(sourceTitle); - await page.getByPlaceholder('Paste source content...').fill(sourceContent); - const sourceInsert = page.waitForResponse((response) => - response.request().method() === 'POST' && response.url().includes('/rest/v1/sources') - ); - await page.getByRole('button', { name: 'Save to Library' }).click(); - expect((await sourceInsert).ok()).toBe(true); - sourceCreated = true; - await expect(page.getByText(sourceTitle, { exact: true })).toBeVisible(); - - await page.evaluate(() => localStorage.removeItem('debate-trainer-storage')); - const sourceHydration = page.waitForResponse((response) => - response.request().method() === 'GET' && response.url().includes('/rest/v1/sources?') - ); - await page.reload(); - expect((await sourceHydration).ok()).toBe(true); - await expect(page.getByText(sourceTitle, { exact: true })).toBeVisible({ timeout: 15_000 }); - await page.getByText(sourceTitle, { exact: true }).click(); - await expect(page.locator('#source-detail-content')).toHaveValue(sourceContent); - await page.getByRole('button', { name: 'Close' }).click(); + await page.reload(); + await expect(page).toHaveURL(/\/app\/settings$/); + await expect(page.getByText(testEmail!, { exact: true })).toBeVisible(); + }); - await page.goto('/app/ask'); - await page.getByLabel('Pros/Cons').click(); - await page.getByPlaceholder('Ask your debate coach...').fill('Reply with one short sentence confirming this staging request.'); - const aiResponse = page.waitForResponse((response) => - response.request().method() === 'POST' && response.url().includes('/functions/v1/ai') - ); - await page.getByRole('button', { name: 'Send message' }).click(); - expect((await aiResponse).ok()).toBe(true); - const saveResponseButton = page.getByRole('button', { name: 'Save message' }).last(); - await expect(saveResponseButton).toBeAttached({ timeout: 90_000 }); - const assistantBubble = saveResponseButton.locator('..'); - await expect(assistantBubble.locator(':scope > div.text-sm')).toHaveText(/\S/, { timeout: 90_000 }); - } finally { - if (sourceCreated) { + await step(page, 'start a debate', async () => { await page.goto('/app/library'); - await expect(page.getByText(sourceTitle, { exact: true })).toBeVisible({ timeout: 15_000 }); - const sourceDelete = page.waitForResponse((response) => - response.request().method() === 'DELETE' && response.url().includes('/rest/v1/sources?') + await deleteLeftoverDebates(page); + await page.reload(); + // Sources belong to a debate, so Library asks for one before it allows adding a source. + await expect(page.getByTestId('library-debate')).toContainText('Select or create a debate', { timeout: 15_000 }); + await expect(page.getByRole('button', { name: 'Add Source' })).toBeDisabled(); + await page.getByTestId('library-debate').getByRole('button', { name: 'Start debate' }).click(); + const dialog = page.getByRole('dialog', { name: 'Start new debate' }); + await dialog.getByLabel('Motion').fill(motion); + const debateInsert = page.waitForResponse((response) => + response.request().method() === 'POST' && response.url().includes('/rest/v1/debates') + ); + await dialog.getByRole('button', { name: 'Start debate' }).click(); + const insertResponse = await debateInsert; + expect(insertResponse.ok()).toBe(true); + const created = await insertResponse.json() as { id?: string; motion?: string }; + expect(created.motion).toBe(motion); + debateId = created.id ?? null; + expect(debateId).toBeTruthy(); + await expect(dialog).toBeHidden(); + await expect(page.getByTestId('library-debate')).toContainText(motion); + await expect(page.getByTestId('current-debate-desktop')).toHaveAttribute('aria-label', `Current debate: ${motion}`); + }); + + await step(page, 'add a source to the debate', async () => { + await page.getByRole('button', { name: 'Add Source' }).click(); + await page.getByPlaceholder('Source title...').fill(sourceTitle); + await page.getByPlaceholder('Paste source content...').fill(sourceContent); + const sourceInsert = page.waitForResponse((response) => + response.request().method() === 'POST' && response.url().includes('/rest/v1/sources') ); - await page.getByRole('button', { name: `Delete ${sourceTitle}`, exact: true }).click(); - expect((await sourceDelete).ok()).toBe(true); + await page.getByRole('button', { name: 'Save to Library' }).click(); + expect((await sourceInsert).ok()).toBe(true); + sourceCreated = true; + await expect(page.getByText(sourceTitle, { exact: true })).toBeVisible(); + }); + + await step(page, 'source and debate hydrate from staging after local state is cleared', async () => { await page.evaluate(() => localStorage.removeItem('debate-trainer-storage')); + const sourceHydration = page.waitForResponse((response) => + response.request().method() === 'GET' && response.url().includes('/rest/v1/sources?') + ); await page.reload(); - await expect(page.getByText(sourceTitle, { exact: true })).toHaveCount(0, { timeout: 15_000 }); + expect((await sourceHydration).ok()).toBe(true); + await expect(page.getByTestId('library-debate')).toContainText(motion, { timeout: 15_000 }); + await expect(page.getByText(sourceTitle, { exact: true })).toBeVisible({ timeout: 15_000 }); + await page.getByText(sourceTitle, { exact: true }).click(); + await expect(page.locator('#source-detail-content')).toHaveValue(sourceContent); + await page.getByRole('button', { name: 'Close' }).click(); + }); + + await step(page, 'complete one AI request in Ask', async () => { + await page.goto('/app/ask'); + await page.getByLabel('Pros/Cons').click(); + await page.getByPlaceholder('Ask your debate coach...').fill('Reply with one short sentence confirming this staging request.'); + const aiResponse = page.waitForResponse((response) => + response.request().method() === 'POST' && response.url().includes('/functions/v1/ai'), + { timeout: 90_000 }); + await page.getByRole('button', { name: 'Send message' }).click(); + expect((await aiResponse).ok()).toBe(true); + const saveResponseButton = page.getByRole('button', { name: 'Save message' }).last(); + await expect(saveResponseButton).toBeAttached({ timeout: 90_000 }); + const assistantBubble = saveResponseButton.locator('..'); + await expect(assistantBubble.locator(':scope > div.text-sm')).toHaveText(/\S/, { timeout: 90_000 }); + }); + } finally { + try { + if (sourceCreated) { + await step(page, 'remove the source', async () => { + await page.goto('/app/library'); + await expect(page.getByText(sourceTitle, { exact: true })).toBeVisible({ timeout: 15_000 }); + const sourceDelete = page.waitForResponse((response) => + response.request().method() === 'DELETE' && response.url().includes('/rest/v1/sources?') + ); + await page.getByRole('button', { name: `Delete ${sourceTitle}`, exact: true }).click(); + expect((await sourceDelete).ok()).toBe(true); + await page.evaluate(() => localStorage.removeItem('debate-trainer-storage')); + const sourceHydration = page.waitForResponse((response) => + response.request().method() === 'GET' && response.url().includes('/rest/v1/sources?') + ); + await page.reload(); + expect((await sourceHydration).ok()).toBe(true); + await expect(page.getByTestId('library-debate')).toContainText(motion, { timeout: 15_000 }); + await expect(page.getByText(sourceTitle, { exact: true })).toHaveCount(0, { timeout: 15_000 }); + }); + } + } finally { + if (debateId) { + const id = debateId; + await step(page, 'delete the debate', () => deleteOwnDebate(page, id)); + } } } From 16b2f79adcd960d222679e38cfe1d40c6129fd8b Mon Sep 17 00:00:00 2001 From: Adam Date: Sat, 3 Oct 2026 05:48:51 +0000 Subject: [PATCH 2/2] fix(e2e): prove debate hydration from staging, not the local cache The Current Debate store persists under its own key, so clearing only the session store let the restored motion come from localStorage. Clear both stores, wait for the sources, debates and current_debates reads, and check the current-debate indicator too. Co-Authored-By: Claude Opus 5.5 (1M context) --- e2e/staging/authed-flows.spec.ts | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/e2e/staging/authed-flows.spec.ts b/e2e/staging/authed-flows.spec.ts index f9c1bf825..918515267 100644 --- a/e2e/staging/authed-flows.spec.ts +++ b/e2e/staging/authed-flows.spec.ts @@ -126,13 +126,20 @@ test('real staging session persists, starts a debate, writes and removes a sourc }); await step(page, 'source and debate hydrate from staging after local state is cleared', async () => { - await page.evaluate(() => localStorage.removeItem('debate-trainer-storage')); - const sourceHydration = page.waitForResponse((response) => - response.request().method() === 'GET' && response.url().includes('/rest/v1/sources?') + // Sources and debates persist under separate keys; clear both so every value below comes from staging. + await page.evaluate(() => { + localStorage.removeItem('debate-trainer-storage'); + localStorage.removeItem('debatelab-current-debate'); + }); + const hydrationRead = (table: string) => page.waitForResponse((response) => + response.request().method() === 'GET' && response.url().includes(`/rest/v1/${table}?`) ); + const tables = ['sources', 'debates', 'current_debates']; + const reads = Promise.all(tables.map(hydrationRead)); await page.reload(); - expect((await sourceHydration).ok()).toBe(true); + (await reads).forEach((read, index) => expect(read.ok(), `${tables[index]} read`).toBe(true)); await expect(page.getByTestId('library-debate')).toContainText(motion, { timeout: 15_000 }); + await expect(page.getByTestId('current-debate-desktop')).toHaveAttribute('aria-label', `Current debate: ${motion}`); await expect(page.getByText(sourceTitle, { exact: true })).toBeVisible({ timeout: 15_000 }); await page.getByText(sourceTitle, { exact: true }).click(); await expect(page.locator('#source-detail-content')).toHaveValue(sourceContent);