diff --git a/.claude/skills/e2e/SKILL.md b/.claude/skills/e2e/SKILL.md new file mode 100644 index 000000000..d280768f1 --- /dev/null +++ b/.claude/skills/e2e/SKILL.md @@ -0,0 +1,172 @@ +--- +name: e2e +description: Agentic end-to-end tests with e2e, the e2e runner. Covers scaffolding e2e.config.ts, picking the Playwright browser engine or the agent-device mobile engine, starting the app under test from the config, driving flows with agent.act, judging with agent.assert, agent.waitFor, and agent.extract, pinning values with screen, app, browser, and expect, shaping the agent (context, system prompt, tools, personas), the replay cache, the e2e CLI, reading .e2e/report.json, and bug bashes (parallel explore runs proven with repro tests). Use when a project depends on e2e, when asked for end-to-end, browser, mobile, or agentic UI tests, to bug bash or hunt for bugs, or when an e2e run fails. +--- + +> **DebateLab rules override this skill.** Vendored from `e2e@0.15.2` +> (`node_modules/e2e/skills/e2e`, Apache-2.0, TesterArmy). Refresh it with +> `rm -rf .claude/skills/e2e && cp -r node_modules/e2e/skills/e2e .claude/skills/e2e` +> after an upgrade, then re-add this block. +> +> - Tests live in `agentic/*.e2e.ts` and import `test`, `expect`, and the +> `student` fixture from `agentic/support/fixtures.ts`. The config is +> `e2e.config.ts`. +> - Run everything through the verification stack, never bare `npx e2e`: +> `.claude/skills/verify-debatelab/bin/verify.sh up`, then +> `verify.sh e2e run|explore|list ...`, then `verify.sh down`. The wrapper +> checks the stack, loads its environment and the OpenRouter key, turns +> telemetry off, and writes evidence to `.verify/artifacts/`. +> - Never add `app.command` or point a target at a plain `npm run dev`: that +> client talks to the production Supabase project. +> - Personas for explore and bug bashes are the named agents in +> `e2e.config.ts`. Sessions: `student` (from `agentic/sessions.e2e.ts`). +> - The `verify-debatelab` skill covers the stack, its fake and live modes, +> and what each proves. Read its "Agentic tests and exploration" section. + +# e2e: agentic end-to-end tests in TypeScript + +e2e runs UI tests with agent goals and exact assertions. `agent.act` drives +one goal; `agent.assert`, `agent.waitFor`, and `agent.extract` judge the +screen. `screen`, `app`, `browser`, and `expect` make exact interactions and +checks. The replay cache reruns verified actions and checks their recorded end +state without a model call; agent judgments still run live. UI targets use +`@e2e-dev/web` for browsers or `@e2e-dev/mobile` for iOS simulators and +Android emulators. A test that takes only `app` can check an API with `fetch` +and `expect` (topic `writing-tests`). Model sign-in commands are in +[setup](references/setup.md#subscriptions-and-api-keys). + +```ts +// e2e.config.ts +import type { E2EConfig } from 'e2e'; +import { web } from '@e2e-dev/web'; +import { gateway } from 'ai'; + +export default { + targets: [ + { + engine: web(), + app: { + url: 'http://127.0.0.1:3000', + command: { executable: 'pnpm', args: ['dev'], log: '.e2e/logs/app.log' }, + }, + }, + ], + // The model behind every agent.* step: an AI SDK instance; gateway() from 'ai' reads AI_GATEWAY_API_KEY or a Vercel OIDC token. + agents: { + default: { + model: gateway('openai/gpt-6-luna-fast'), + system: 'You are a thorough QA agent. Verify every outcome on screen.', + }, + }, +} satisfies E2EConfig; +``` + +```ts +// tests/billing.e2e.ts +import { test } from '@e2e-dev/web'; +import { expect } from 'e2e'; + +test('a member upgrades to Pro', async ({ app, agent, screen, browser }) => { + await app.open('/settings/billing'); + await agent.act('upgrade the workspace to the Pro plan'); + await expect(screen.getByRole('status')).toContainText('Pro'); + await expect(browser).toHaveURL('/settings/billing'); +}); +``` + +## Topics + +Read the topic for the job before writing code. The files sit next to this +one; the installed CLI prints the same text with `npx e2e guide ` +(`e2e guide` alone prints this page). For anything the topics do not cover, +the full documentation ships in the `docs/` directory of the installed `e2e` +package (`node_modules/e2e/docs` in a single-package project); a link such as +`/reference/cli` is `docs/reference/cli.mdx`. + +| Topic | File | Read it when | +| --- | --- | --- | +| `setup` | [references/setup.md](references/setup.md) | Adding e2e to a project, writing `e2e.config.ts`, starting the app from the config, mobile targets | +| `writing-tests` | [references/writing-tests.md](references/writing-tests.md) | Writing or fixing tests: fixtures, locators, actions, matchers, sign-in sessions, the `browser` fixture | +| `agent` | [references/agent.md](references/agent.md) | Adding `agent.*` steps, picking a model, cost and budgets, the replay cache | +| `running` | [references/running.md](references/running.md) | CLI flags, reporters, `.e2e/report.json`, exit codes, CI | +| `explore` | [references/explore.md](references/explore.md) | Exploring an app toward a goal without a test file: `e2e explore`, its budgets, verdict, and `run.explore` | +| `debugging` | [references/debugging.md](references/debugging.md) | A run failed: error codes and their fixes, `--headed`, `--debug`, `--ai-trace` | +| `mcp` | [references/mcp.md](references/mcp.md) | Driving the live app from a coding agent over MCP: `e2e mcp`, its tools, and the explore-then-write loop | +| `bug-bash` | [references/bug-bash.md](references/bug-bash.md) | Asked to bug bash, QA, or hunt for bugs across an app or a branch: parallel `e2e explore` charters, merging findings, proving each with a repro test | + +## Workflow + +1. Look at what exists: `e2e.config.ts` or `e2e.config.mts`, the `tests` glob + (default `tests/**/*.e2e.ts`), `e2e` in `package.json`. Nothing there: + follow `setup`. +2. Learn the screens before writing a test: routes, labels, roles, button + text. Semantic locators need the accessible names the app renders, so read + the components, open the page with `--headed`, or drive the live app over + the registered `e2e mcp` server (topic `mcp`): `open_session`, `observe`, + and `locate` show exact names and check a locator before you write it. +3. Write `tests/.e2e.ts`. Drive the flow with `agent.act`, one goal + per call, and pin each outcome right after with `expect` or `agent.assert`. + Exact values go through `screen`: a sign-in form in a setup test, a field + that must receive one specific string, a count that must be one number. +4. Run one file: `npx e2e run tests/.e2e.ts`. Agent steps need a + model in the config and that provider's authentication (a saved + subscription login, an API key); a local endpoint may need none. Tests + without agent steps need no model. +5. Read the failure: the reporter prints the error code, message, and a code + frame; `.e2e/report.json` has every step and artifact path. Fix the + locator, the expectation, or the app. Never add a sleep. + +## Rules + +- Run the CLI as `npx e2e ...` (or `pnpm exec e2e ...`). +- The config is `export default { ... } satisfies E2EConfig` with + `import type { E2EConfig } from 'e2e'`. `targets` is required; a UI target + names an engine and declares the app beside it: `{ engine: web(), app: { url, command } }`. + A tools-only target can omit the engine and set `platform`. +- Import `test`, `describe`, the hooks, `expect`, `credentials`, and + `secrets` from `e2e`. A test that uses the `browser` fixture imports `test`, + `describe`, and the hooks from `@e2e-dev/web`: the same runtime functions, + typed with `browser`. +- Config and tests are ES modules whatever `package.json` sets as `type`. +- Locators resolve when used. Actions wait for readiness and `expect` retries + assertions. Reads such as `textContent()` fail at once on zero matches and + `count()` answers from the current screen; nothing waits for a value to + change, so use a matcher when a value has to settle. +- A locator that matches two nodes fails with `LOCATOR_AMBIGUOUS`; narrow it + (topic `writing-tests`). +- Secrets never appear in test code. Declare accounts under `credentials` and + every other sensitive value under `secrets` in the config; resolve with + `credentials.user(name).password` or `secrets.get(name)` (separate + namespaces: `secrets.get` never returns a password), and hand the opaque + `Secret` only to `fill()` or `agent.act` params. +- Agent instructions: one goal per `act`, the wording on screen, real values + in params. Judge meaning, not phrasing: `toContain('Pro')`, not an exact + sentence a model produced. +- Check each agent goal's outcome. A passing `act` with a recorded check can + be cached and replayed without model calls (topic `agent`). +- Shape the agent for this app: `context` for vocabulary the screens use, + `system` for how it works, tools for a test API, named personas under + `agents`. When a step fails, tighten the goal first, then the context, then + the agent. +- `.e2e/` is output (`report.json`, `artifacts/`, `cache/`, `logs/`; the + config's `output` moves the report and artifacts, never `cache/` or the + app's log). Read it, never edit it. + +## Feedback + +When e2e itself gets in your way, tell the e2e team: a command or API that +broke (`bug`), docs or this skill that misled you (`docs`), or a capability +you needed and did not find (`feature`). Send it once per problem, after you +worked around it or gave up, never for failures of the app under test. + +```bash +npx e2e feedback --type bug -m "" \ + --task "" --expected "<...>" --actual "" \ + --approach "" --command "" --agent "" +``` + +Describe e2e's behavior only: never paste app content, page text, test files, +URLs of private apps, or credentials. Secret-named environment variables and +common token shapes are redacted, but do not rely on it. `--dry-run` prints +what would be sent. Tell the user you sent it and give them the reference id +it prints. diff --git a/.claude/skills/e2e/references/agent.md b/.claude/skills/e2e/references/agent.md new file mode 100644 index 000000000..3d190d459 --- /dev/null +++ b/.claude/skills/e2e/references/agent.md @@ -0,0 +1,310 @@ +# Agent steps + +`agent` is a fixture like `screen` and the main way a test drives the app. +Each call is one bounded invocation: fresh redacted observation, deadline, +model-call budget, no shared transcript. No agent step, no model calls. + +## Configure a model + +Put an AI SDK model under `agents.default`. Vercel AI Gateway reads +`AI_GATEWAY_API_KEY` or, without it, a Vercel OIDC token: + +```ts +import type { E2EConfig } from 'e2e'; +import { web } from '@e2e-dev/web'; +import { gateway } from 'ai'; + +export default { + targets: [{ engine: web(), app: { url: 'http://127.0.0.1:3000' } }], + agents: { default: { model: gateway('openai/gpt-6-luna-fast') } }, +} satisfies E2EConfig; +``` + +Subscription logins and local models: +[setup](setup.md#subscriptions-and-api-keys). Keep `ai@^7` installed with +any provider. + +- Pass a model instance, not a string (`INVALID_CONFIG`). +- An agents entry is one plain object of `model`, `judge`, `system`, + `context`, `tools`, `maxSteps`, `maxModelCalls`, `judgmentTimeout`, + `maxObservationBytes`, `maxInputTokens`, `providerOptions`, or `executor` + for a custom brain; none inherits `model` or `context` from `default`. +- `model` drives `agent.act`. Judgments use `judge` when set, else `model`. +- A missing model for the built-in agent raises one run-level + `MODEL_UNAVAILABLE` when the first test acquires `agent`, exit 2; auth + failures surface on the first model call as `MODEL_PROVIDER_FAILED`. +- `context` is what the app calls things, sent to every model call, judges + included; `system` is how the acting agent works, read only by the act + loop. Both live on the agents entry; `agentContext` on a test or group + adds more. + +### Choose an agent + +Tests use `agents.default` unless selected otherwise: + +- `e2e run --agent buyer` picks another configured agent. +- `{ agent: 'buyer' }` on a test or group pins that agent; a list such as + `{ agent: ['buyer', 'admin'] }` runs each test once per agent. +- `--agent buyer,admin` runs unpinned tests for both, narrows a pinned list + to matching names, never replacing a pin it does not name. +- `{ agent: 'name' }` on an `agent.*` call overrides the test's choice. + +For signed-in personas, pair an agent with a `session` in a describe block +repeated per persona; the cache records per agent step, so a specialised +agent replays too. + +## act: one goal + +```ts +import { credentials } from 'e2e'; + +await agent.act('add a todo named "Buy milk" and mark it done'); +await agent.act('invite {email} as an editor', { params: { email: 'ada@example.test' } }); + +const member = credentials.user('member'); +await agent.act('sign in with the given credentials', { + params: { + username: member.username, + password: member.password, // a Secret: the model sees its name, the runner fills the field + }, +}); +``` + +`act(instruction, options?)` runs a multi-action flow to a verdict. Passed +resolves with `summary`, `modelCalls`, `actions`, and `cache` (the replay +cache's part). Failed or blocked throws an `AgentError` whose `code` says +why: `ACTION_FAILED` (product failure); `STEP_BUDGET_EXHAUSTED`, +`STEP_TIMEOUT`, `CONTEXT_OVERFLOW` (out of room); +`AUTH_CREDENTIAL_UNAVAILABLE`, `AUTH_CREDENTIAL_INVALID`, +`SECRET_UNAVAILABLE`, `ENVIRONMENT_UNAVAILABLE`, `SEED_DATA_MISSING`, +`TEST_SETUP_FAILED`, `AUTOMATION_UNSUPPORTED`, `POLICY_DENIED` (blocked +from outside); +`MODEL_OUTPUT_INVALID` (unusable answer). Full list: topic debugging. + +Options: + +- `params`: values the instruction names; the runner fills a `Secret`, + and a run-unique `unique(\`E2E ${Date.now()}\`)` keeps the cache working + across runs. +- `timeout`: the config `timeout`. +- `maxSteps` (default 25 actions), `maxModelCalls` (default 25): may only + lower the agent's limits, else `INVALID_ARGUMENT`. +- No `schema` (use `extract({ schema })`) and no `vision`: the model decides + when it needs pixels. + +The tools, one per engine action. Name the target as the screen names it +(files in `params`); the agent picks the verb: + +- `observe`: re-read the screen after waiting on work in progress. +- `tap`: button, link, menu item, tab, checkbox, row, field. +- `double_tap`/`long_press`/`right_click`: second-click item, long-press + menu, context menu only. +- `hover`: menus, flyouts, tooltips that open on the pointer. +- `type`: one input; on browser/device no target means focus; `replace` clears. +- `press`: one key to a node or, on browser/device, to focus; `times` up to 20. +- `select`: one option by visible label. +- `check`: set a checkbox, switch, or radio to a state, not flip it. +- `scroll`: viewport or one scrollable node, a screen or a few. +- `scroll_to`: a listed node into view, or by `text` page a list to a row. +- `drag`: one node onto another. +- `upload`: project-root files to a file input; outside it or hidden (`.env`) + is `POLICY_DENIED`. +- `navigate`: a URL or app-relative path. +- `back`: browser history or in-app back. +- `type_secret`: a declared secret by name; plaintext never reaches the model. + Offered only when the step declares secrets and the engine can fill them. +- `screenshot`: attach viewport pixels; every later result then carries one. +- `tap_at`/`hover_at`/`press_at`/`select_at`/`type_at`: a screenshot point. +- `dismiss_keyboard` (device): hide the on-screen keyboard. + +The focused field's selected text is listed as `selection="..."` (never for +a secret), so a repeated `Shift+ArrowLeft` selects one word with visible +feedback. Point tools serve a canvas shape, map pin, image region, or system +sheet control, hit-testing the tree first so a listed control underneath is +acted on by id; `type_at` on nothing listed taps, then types. A screen with +nothing to tap by id opens with a screenshot attached. A device has no +`right_click`, `select`, `upload`, or `scroll_to` by node id. Pixels are +masked and withheld after a secret fill (topic writing-tests): act on pixels +before signing in, or in a test of its own. A result says when an action +closed an on-screen keyboard; on a touch screen that tap was often spent +closing it, so act again. + +## assert, waitFor, extract: one question + +```ts +import { z } from 'zod'; + +await agent.assert('the dashboard shows a trial badge'); // one look, one judgment + +await agent.waitFor('the export finished and a download link appeared', { // polls + interval: 500, + timeout: 120_000, +}); + +const data = await agent.extract('every todo title and how many remain', { // structured output + schema: z.object({ titles: z.array(z.string()), remaining: z.number().int() }), +}); +expect(data.titles).toContain('Buy milk'); +``` + +- `assert` does not poll. False is `ASSERTION_FAILED` with the model's + explanation and a screenshot in the report; too little on screen to + decide is `ASSERTION_INCONCLUSIVE`, also a failure, so reach the right + screen first and ask about what is visible. Judged from the tree alone, + it adds `pass vision: true when the answer is in pixels`, as does + `waitFor`'s timeout after an inconclusive round. +- Judgments see the assertion and the current screen only, never prior + steps or the act loop's summaries; malformed output gets one repair + round, then `MODEL_OUTPUT_INVALID`. +- `waitFor` observes every `interval` (default 3 s), judges only when the + screen changed, and is `STEP_TIMEOUT` after `timeout` (default 30 s). +- `extract` takes any Standard Schema validator (zod works); the model sees + the schema's shape, never its value rules (`min`, `max`, lengths, + patterns), which check what it read. Data the screen does not show is + `ASSERTION_INCONCLUSIVE` naming what was missing, never `""` or `0`; to + accept absence, ask for it (`'the phone, or null when none is shown'` + with `.nullable()`). + +`vision` on a judgment picks the evidence: `false` (default) the tree; +`true` the tree plus a masked screenshot; `'only'` the screenshot alone, for +what the screen presents (an overlay, a broken layout, a chart) where the +tree would answer first. `'only'` never falls back to the tree: unprovably +masked pixels fail with `POLICY_DENIED`. + +## Write instructions the model can execute + +- One goal per `act`; goal order is the test's, the path the model's. +- Use the words on screen: `'open the Billing tab'`, not `'upgrade'`. +- Values go in `params`, never expanded: `act('rename to {name}', { params })`. +- Do not describe mechanics the runner handles: waiting, scrolling, retries. +- Pin every `act` outcome right after it; that check lets the cache record: + +```ts +await agent.act('create a workspace named "Atlas" on the Pro plan'); +await expect(screen.getByRole('status')).toHaveText('Created "Atlas" on the Pro plan'); +``` + +The model gets the instruction verbatim plus the params as a separate +block; the check also makes the test model-portable. Off-screen state (a +database row) can land after `act` returns; `expect.poll` the read instead +of sleeping. + +## What the model sees + +A redacted snapshot of the screen (roles, names, text, states), prior-step +summaries, and your context; never raw HTML, cookies, headers, environment +values, or a `Secret`'s value; password fields masked. The first +screen of a step arrives whole; later action results report what changed, +keyed by node ids stable while an element exists, or the whole screen when +most changed. Pixels arrive through `vision` on a judgment or the act loop's +`screenshot` and point tools, masked and withheld after a secret fill. When +the browser engine's tree capture times out, the model gets a screenshot and +a warning, a judgment needs `vision: true`, and no control may be inferred +absent nor old node ids reused. Nothing the model returns runs as code or +selectors: the runner validates and authorizes every tool call first. + +## Budgets and cost + +| Call | Model calls | Default timeout | +| --- | ---: | --- | +| `act` | up to `agents..maxModelCalls` (25) | the config `timeout`, 120 s | +| `assert` | 2 | 30 s | +| `extract` | 2 | 30 s | +| `waitFor` | up to `agents..maxModelCalls` (25) | 30 s | + +- Slow model calls: raise the step or test `timeout` for `act`, the agent's + `judgmentTimeout` for judgments, `actionTimeout` for slow UI. + `STEP_TIMEOUT` and `STEP_BUDGET_EXHAUSTED` fail the test; smaller goals + help. +- `--debug` prints phase timings and a per-step table (duration, model + calls, tokens, cache share, cost) to stderr and saves step transcripts as + artifacts. + +## The replay cache + +A passing `agent.act` saves its actions once a later check verifies the +outcome; the next run replays them without model calls, and the live agent +continues from the current screen when the app or final state no longer +matches. Misses and hand-offs use the model; `agent.assert`, +`agent.waitFor`, and `agent.extract` are never cached. + +- On by default (`read-write`), `read-only` in CI, off via `cache: 'off'` + or `--no-cache`. Entries live in `.e2e/cache/`; deleting the directory + only slows the next run. +- An entry is written only after a later verification passes (a locator or + engine `expect` matcher, `locator.waitFor`, `browser.waitForURL`, + `agent.assert`, `agent.waitFor`), so an unchecked `act` never replays; a + plain-value `expect`, `expect.poll`, `agent.extract`, another `act`, or + the attempt passing confirms nothing. +- A replay needs the app on the recorded path (unless the recording opens + with a navigation), re-finds each control by role, name, test id, + placeholder, and input purpose, and passes alone only when the recorded + end path and the controls seen during the step are back; otherwise the + agent takes over mid-step. `step.cache.reason` says why: `no-entry`, + `wrong-context`, `target-not-found`, `target-ambiguous`, `end-mismatch`, + and so on. +- A step recording no actions creates no entry; one whose `unique()` value + equals, is spelled inside, or is the encoded form of another param's value + is not recorded either (`step.cache.notRecorded`: `param-collision`). +- `e2e init` gitignores `.e2e/cache/`; remove that line to commit entries + and share replays with CI and teammates (CI stays `read-only` unless + `cache: 'read-write'` is set). +- A failing run evicts the entries it implicates; `--no-cache` rules the + cache out of a failure. A cancelled run and a failure where no model + answered (`MODEL_PROVIDER_FAILED`, `MODEL_UNAVAILABLE`) implicate nothing + and evict nothing. +- With committed recordings, `--strict-cache` in CI fails a recording that + no longer replays with `REPLAY_STALE` instead of quietly spending model + calls every run; re-record locally and commit. Unrecorded steps still run + live. + +## Inspect what the model did + +```bash +npx e2e run tests/checkout.e2e.ts --debug # step table, transcripts as artifacts +npx e2e run tests/checkout.e2e.ts --ai-trace # writes /ai-trace.json (.e2e/ai-trace.json by default) +npx unbox-ai runs .e2e/ai-trace.json # one line per agent step +npx unbox-ai summary .e2e/ai-trace.json --run 0 # turns, tokens, tool calls of one step +``` + +Never read the trace directly: megabytes of resent context, images replaced +by byte counts. `--no-cache` traces the whole flow, since replayed steps +make no model calls. + +## Make the agent yours + +1. **The goal.** A failed step usually named something the screen does not; + reword it with on-screen labels, and `--debug` shows what the model saw + and tried. +2. **`context`.** Vocabulary every step needs (plan names, what a + "workspace" is, which tab holds billing), once on `agents..context` + or per test via `agentContext`. +3. **`system` on the agent.** How carefully it verifies, what it never + does, how it treats a modal; a UX reviewer, a cautious QA persona, and a + fast smoke agent are three `system` prompts on one model. +4. **Tools.** A test API the agent may call mid-flow (seed a cart, mint a + coupon) via `tools`; see below. +5. **The model and its options.** `providerOptions` for reasoning effort, + or another model for one persona. `--agent ` runs the suite as any + configured agent, comparing candidates on the same tests; every result + records which agent ran it. + +## Beyond the built-in agent + +- `tools: { seedCart }` on an agents entry adds AI SDK tools wrapped with + `defineTool(tool({ ... }), { mutates: true })` from `e2e/agent` for a + test API a flow calls mid-step. +- `createToolLoopExecutor` keeps the loop and replaces prompt and tool + vocabulary. +- Any `StepExecutor` (`{ name, version?, cache?, runStep(ctx) }`) goes under + `executor`; the runner still owns observations, actions, budgets, and the + report, and `system` or `tools` beside `executor` is `INVALID_CONFIG`. + +Reference: https://e2e.tester.army/docs/agents + +## In CI + +Run the suite, agent steps included, on every pull request, passing the key +the config's model reads (`env: { AI_GATEWAY_API_KEY }` for `gateway()`) +from secrets. Sharing `.e2e/cache/` lets CI replay verified action steps +without model calls; CI defaults (`read-only` cache, retries): topic running. diff --git a/.claude/skills/e2e/references/bug-bash.md b/.claude/skills/e2e/references/bug-bash.md new file mode 100644 index 000000000..f774c4839 --- /dev/null +++ b/.claude/skills/e2e/references/bug-bash.md @@ -0,0 +1,236 @@ +# Running a bug bash + +A bug bash is many `e2e explore` runs at once, one charter each, then a +verification pass that turns every claimed bug into a repro test that fails +for the reason reported. Plan the charters, fan them out, merge the findings, +prove each bug. Report confirmed bugs only, each with its failing test. + +## 1. Prepare + +- A config with a target and an agent that holds a model (topic `setup`), + and authentication for its provider. +- The app must serve several explorers at once: start it once and set + `reuseExisting: true` on the target's `app.command`, or declare the URL with + port `0` so each run starts its own app on a free port. `reuseExisting` is + ignored when `CI` is set, as in many agent sandboxes: use port `0` or + unset `CI`. A fixed port that is not reused fails every run after the first. +- When the target's `app.command` brings up a stack of its own (a database + it starts and removes on exit), the first explorer to finish removes the + database under the rest. Start the stack once yourself with the config's + command and environment, and explore with a bug-bash config that declares + no command (below). +- Explore a production build when the project has one. A dev server that + compiles a route on first visit reads to the explorer as a dead link. +- Seed one disposable workspace or user per charter with the project's own + fixtures or seed scripts and declare each as a credential (`bb-`). + Explorers that share an account report each other's edits as bugs. +- Start signed-in charters from a saved session: `e2e explore --session + ` runs the setup test that saves it and explores signed in (topic + `explore`). A setup that types a password withholds screenshots from then + on; sign in by cookie or API call to keep them (topic `writing-tests`). + Each charter with its own account needs its own setup and session. +- Put what the local app cannot do in the agent's `context`: integrations + without keys, what is seed data, what must never be clicked (paid runs, + real accounts). Add the explorer's own blind spots from step 5's artifact + bucket (the config below carries the sentence); without them those + families dominate the findings. +- A bash against a deployed site others use is read-only: no signups, + sign-ins, or submissions, nothing injection-shaped in URLs, no request + loops. A WAF block is the firewall working, not a finding, and it can + follow the runner's IP into every later charter. +- Each exploration step needs 40 or more `maxSteps` and `maxModelCalls` on + the agent; a config tuned for test steps (`maxSteps: 15`) starves it. That + is per step, separate from `--max-steps`, which counts a charter's steps. +- On a mobile target, declare one target per device and give each explorer + and verifier its own (topic `setup`). +- Skip `--headed`. On an engine that records, `--video` gives each run a + replay for any confirmed bug. + +A bug-bash config, left untracked, spreads the project's config and +overrides what a bug bash needs. Typing the import as `E2EConfig` keeps +`shared.tests` and `shared.credentials` compiling when the base declares +neither: + +```ts +// e2e.bugbash.config.ts +import type { E2EConfig } from 'e2e'; +import { web } from '@e2e-dev/web'; +import { gateway } from 'ai'; +import base from './e2e.config.ts'; + +const shared: E2EConfig = base; + +// What the local app cannot do, plus the explorer's blind spots (step 5's artifact bucket). +const context = + 'Sign in with the credential the goal names. The local app sends no email and has no AI key. Never start a paid run or connect an integration. ' + + 'Not bugs: a link that opens a new tab leaves this one unchanged; accessible text splits around inline links, so judge copy by the rendered screen when a screenshot is available and never report split text alone as broken copy; an infinite-scroll "Loading more" sentinel loads when scrolled into view; images lazy-load, so scroll and wait before calling one blank.'; +const persona = { model: gateway('openai/gpt-6-luna-fast'), maxSteps: 40, maxModelCalls: 40, context }; + +export default { + ...shared, + // The project's tests, so a repro can use its setup tests' sessions, plus the repro tests from step 6. + tests: [shared.tests ?? 'tests/**/*.e2e.ts', 'tests/bugbash/**/*.e2e.ts'].flat(), + // The app already runs: no command. + targets: [{ name: 'web', engine: web(), app: { url: 'http://127.0.0.1:3000' } }], + retries: 0, + reporters: ['list'], + credentials: { + ...shared.credentials, + // The seed script's password, from the environment rather than the file. + 'bb-cart': { username: 'bb-cart@example.test', password: process.env.BUGBASH_PASSWORD ?? '' }, + 'bb-account': { username: 'bb-account@example.test', password: process.env.BUGBASH_PASSWORD ?? '' }, + }, + // The postures from step 2 as personas: same model and budgets, a different stance; each charter picks one with --agent (step 3). + agents: { + default: persona, + skeptic: { ...persona, system: 'Distrust every number, date, count, and claim on screen; cross-check each against every other place it appears.' }, + fuzzer: { ...persona, system: "At every input, run the goal's input matrix before anything else, judging each entry before the next. Never take the happy path." }, + }, +} satisfies E2EConfig; +``` + +Set `BUGBASH_PASSWORD` to 6 or more characters first, or the config fails to +load. Pass `--config e2e.bugbash.config.ts` to every command below, and +`open_session {config: "e2e.bugbash.config.ts"}` over MCP. + +## 2. Plan charters + +A charter is one `e2e explore` goal: one area, one posture, one sentence, +naming the start route and, when needed, the credential (`Sign in as +credential bb-cart. Starting at /cart, ...`). Read the routes, navigation, +and forms first; for a branch, `git diff --stat` against the base. + +| Posture | Charter shape | +| --- | --- | +| First-time user | `Starting at /signup, sign up and complete onboarding like a first-time user; report anything confusing, broken, or inconsistent` | +| Numbers and copy | `Starting at /cart, change quantities and apply a coupon; check every price, total, and label against the rest of the page` | +| Edge input | `Starting at /settings/profile, submit each field empty, too long, with unicode and with leading spaces; report validation that is missing or wrong` | +| State | `Starting at /projects, create, rename, and delete a project, reloading and going back after each; report state that is lost or stale` | +| Error paths | `Starting at /login, try a wrong password, an unknown account, and a locked account; report errors that are missing, misleading, or leak detail` | + +Aim for five to ten charters, each with its own slug. Overlap is fine; +duplicates merge in step 4. Give each posture its persona (config above), +picked per charter in step 3: a generic agent walks past a stat that +contradicts the same stat on another page, the skeptic catches it. An +edge-input charter names its exact matrix (empty, a 300-character string, +unicode, leading spaces, literal special characters) or it spends the whole +time budget before judging a single result. + +## 3. Fan out + +One `e2e explore` per charter with its own output directory: +`--output .e2e/bugbash/` writes `report.json` and `artifacts/` there, +and `--reporter list,markdown` adds `summary.md`. `e2e init` gitignores +specific `.e2e/` paths, not `.e2e/bugbash/`: add it to `.gitignore` or +delete it when done, and delete it before a new bug bash. `charters.txt` and +the logs under it are yours; the rest is the runner's output, read only. + +Run them as background shell jobs, four at a time, not as subagents. One +line per charter, `slug|target|agent|charter`, the agent naming its persona, +then (`xargs -0 -P` is a GNU and BSD extension, present on macOS and Linux): + +```bash +mkdir -p .e2e/bugbash +cat > .e2e/bugbash/charters.txt <<'CHARTERS' +cart|web|skeptic|Starting at /cart, change quantities and apply a coupon; check every price, total, and label against the rest of the page +account|web|fuzzer|Starting at /settings/profile, submit each field empty, a 300-character value, unicode, and leading spaces; report validation that is missing or wrong +CHARTERS +while IFS='|' read -r slug target agent charter; do + [ -n "$slug" ] && printf '%s\0%s\0%s\0%s\0' "$slug" "$target" "$agent" "$charter" +done < .e2e/bugbash/charters.txt | xargs -0 -n 4 -P 4 sh -c \ + 'npx e2e explore "$4" --config e2e.bugbash.config.ts --target "$2" --agent "$3" --output ".e2e/bugbash/$1" --max-steps 6 --video --reporter list,markdown < /dev/null > ".e2e/bugbash/$1.log" 2>&1' _ +``` + +The log's summary prints `AI` (cost) and `Duration`. Exit code `1` means +issues were reported: read the log either way. Exit codes `2` and `3` are +setup and environment problems; fix them and rerun that charter alone. + +## 4. Merge + +Read each log's `Findings` section (topic `explore`); `summary.md` holds the +same, and `report.json` has the record under `run.explore`. The video is in +the attempt's `video/` directory under `.e2e/bugbash//artifacts/`. + +Merge findings that describe one defect: same path, same broken behavior. +Keep the clearest reproduction and every charter that hit it. Keep warnings +in a separate list unless the user asked for polish. + +A charter that filled a password, or whose session's setup did, prints no +`evidence` line after that point. Its video is not masked: check it for +secrets before sharing (topic `writing-tests`). + +## 5. Triage + +Sort every finding before writing any test. Read the source to sort them. + +| Bucket | Sign | Outcome | +| --- | --- | --- | +| Explorer artifact | A "dead" `target="_blank"` link whose destination opens when clicked with popup capture or navigated to directly, a broken sentence the screenshot renders whole, a "Loading more" sentinel nothing scrolled to, a lazy-loading image or embed | Rejected with the check that settled it; settle this bucket first. A new-tab link whose destination never opens stays a candidate | +| Environment | Fails on a key, a service, or a limit only the local stack lacks (an email provider, an AI key, a billing plan) | Rejected, naming the variable or service; note separately when the app handles the failure badly in a way production users would see, such as showing the raw error | +| Design | The code, its tests, or its copy say the behavior is intended | Rejected, citing where | +| Fixture | The seed data lacks a field real records always have | Rejected, naming the field | +| Candidate | None of the above | Verify it (step 6) | + +## 6. Verify + +A finding is a model's claim; prove each candidate before reporting it. When +your client can start subagents (Claude Code's Agent or Task tool, for one), +start one per area with its three to five candidates as the log printed +them and these steps, up to four at a time, the `e2e mcp` server's default +session limit. Each reports back, per finding: confirmed or rejected, the +root cause as `file:line` when it may read the source, the repro test path, +and the failure it saw. Without subagents, verify one area after another. + +1. Read `actual` against the screenshot, or the video when there is none. + A finding the evidence contradicts is rejected here. Settle the artifact + bucket first: a "dead" link's destination must actually open (a valid + href with a prevented default is still dead), a copy claim must show in + the rendered screenshot. +2. Write a repro test that follows the reproduction and asserts the + expected behavior, so it fails today and passes once the bug is fixed. + Put it under `bugbash/` inside the directory the config's `tests` glob + covers (`tests/bugbash/.e2e.ts` for the default): a file the glob + does not match is never selected, whatever path you pass to `e2e run`. + Build on the project's fixtures (a `test.extend` fixture, a setup test's + session). Prefer `screen` actions and `expect` with exact values; + `agent.act` for a step that varies, `agent.assert` for an outcome only + judgment can check (topic `writing-tests`). Tag it `{ tags: ['bugbash'] }`. +3. Get exact locators from the live app: with the `e2e mcp` server + registered, `open_session` with the bug-bash config, pass its session id + to every call, walk the reproduction, and `locate` each locator before + writing it (topic `mcp`). Close your session when done. +4. Run the file alone: `npx e2e run tests/bugbash/.e2e.ts`. Confirmed + only when it fails with `ASSERTION_FAILED` on the assertion that encodes + the bug. Any other failure (`LOCATOR_NOT_FOUND`, a timeout, a setup + error) means the test is wrong: fix it and rerun. A passing test means + the bug did not reproduce: reject the finding, say so, and move the test + out of `tests/bugbash/` (to `.e2e/bugbash/extra-tests/`, or offer it as + a regression test). Only failing repro tests stay there. + +## 7. Report + +Lead with the confirmed bugs, most severe first. For each: title, path, one +line of expected against actual, root cause when known, steps, screenshot +and video paths, repro test, and the charters that found it. Then the +environment candidates that are production risks, marked unverified. Then +the rejected findings grouped by reason (environment, design, fixture, did +not reproduce), and the warnings. End with the charters run, their cost, and +the areas no charter reached. + +The repro tests fail until the bugs are fixed: where the project's `tests` +glob covers `tests/bugbash/`, its gating run leaves them out +(`npx e2e run --exclude-tag bugbash`) or they stay uncommitted. Offer to fix +each bug: the repro test turning green is the proof, and it stays as the +regression test with its `bugbash` tag removed. + +## Rules + +- Never report an unverified finding as a bug. "The explorer reported" is + not "confirmed". +- One charter, one area. A charter that spans the whole app ends at its + step budget having skimmed everything. +- Seeded data and test accounts the app ships for development are not + bugs; say so in the agent's `context`. +- Leave the project's files as you found them: the bug-bash config, seed + script, and repro tests stay untracked until the user asks otherwise, and + stop the stack you started when the user is done. diff --git a/.claude/skills/e2e/references/debugging.md b/.claude/skills/e2e/references/debugging.md new file mode 100644 index 000000000..abb9176b3 --- /dev/null +++ b/.claude/skills/e2e/references/debugging.md @@ -0,0 +1,111 @@ +# Debugging a failing run + +Paths below sit under the configured `output` directory; `.e2e` is the +default. + +## Read the failure + +1. Run with `--reporter list,markdown`: `list` ends with a `Failed Tests` + section, the markdown reporter's `Failures` line names `.e2e/failures/`. +2. Open the failed test's page under `.e2e/failures/` and grep `Expected:` + / `Observed:` (an `expect`), `Asked for:` and `Waited:` (a locator), + `Look at:` (the line it unwound through), whether every attempt + failed alike (a bug, not a flake), the steps, the last model turns of a + failed agent step, and the accessibility tree at failure, one node per + line. Fix from what was there. +3. `.e2e/report.json` backs the pages: + +```bash +jq '.run | {status, exitCode, errors}' .e2e/report.json +jq '.run.results[] | select(.selected and .status != "passed") | {titlePath, file, status}' .e2e/report.json +jq '.run.results[] | select(.selected and .status != "passed") | .attempts[-1] + | {status, error, failure, steps: [.steps[] | select(.status != "passed") | {api, label, source, status, error}], artifacts}' .e2e/report.json +``` + + `error.details` holds the facts, `error.source` the line, `failure` the + `url`, `screen` and `screenshot` artifact ids, and `candidates`; a + failed agent step has `turns`; `selected` drops filtered-out tests + (recorded as `skipped`). +4. Artifacts, under `.e2e/artifacts/`: `failure/screen.txt` + and the engine's screenshot per failed attempt; a Playwright trace per + traced attempt (`npx playwright show-trace `); downloads; with + `--video` the recording (`video/video.webm` in a local browser, each + later page `video/video-part.webm` with its own `startedAt`; + `video/video.mp4` on a device; a provider's file or link); with + `--debug` every agent step's transcript. + +## Error codes + +| Code | Usual cause | Fix | +| --- | --- | --- | +| `CONFIG_LOAD_FAILED` | The config throws while loading (a refused engine option is `INVALID_CONFIG` instead) or imports a missing package, subpath, or removed export such as `defineConfig` | Install the dependency, or fix the import or line quoted | +| `INVALID_CONFIG`, `INVALID_GLOB` | Unknown or foreign key (`app`, `webServer`, `use`, `projects`, `baseURL`), `json` with `list`; a `tests` glob with braces, classes, an absolute path, or no wildcard | The message names the key or glob to write; the app is declared in the target's `app` | +| `CONFIG_NOT_FOUND`, `CONFIG_AMBIGUOUS` | Wrong `--config` path; both `.ts` and `.mts` present | Fix the path; keep one | +| `NO_TESTS` | The glob, a positional, or a filter matched nothing; the message names each empty positional and each undeclared `--tag` with the nearest declared one | Check the config `tests`, the `.e2e.ts` suffix, the tag names | +| `NO_LAST_RUN` | `--last-failed` found no `.e2e/report.json` | Run once without it | +| `COLLECTION_ERROR` | `async` describe body, `test.setup` inside `describe`, an option forbidden in a serial group, registration outside collection | Restructure per `writing-tests` | +| `HOOK_FAILED` | `beforeAll` or `afterAll` threw; its scope's tests skip | Fix the hook; the report carries its error | +| `UNSUPPORTED_ARTIFACT` | A test's or target's `trace` or `video` on an engine that cannot record | Drop it there, or set it at the config root or CLI (such targets skip with a notice) | +| `BROWSER_INSTALL_FAILED`, `LAUNCH_TIMEOUT` | Browser download failed; engine init or attempt start exceeded `launchTimeout` | Run the quoted `npx playwright install ` (`--with-deps` on bare Linux); raise the root `launchTimeout` (60 s default) | +| `APP_UNREACHABLE` | `app.command` never answered `readyUrl` within `startupTimeout`; on a device, a message naming the iOS automation runner: the runner failed, not the app | Read the quoted log lines; check the port, `app.url`, `app.command.env`, `app.command.startupTimeout`. Runner: rerun, else `npx agent-device daemon stop` and reboot the simulator | +| `APP_ALREADY_RUNNING` | Something already serves `url` when `command` should start | Stop it, or `reuseExisting: true` locally | +| `APP_URL_REQUIRED`, `APP_NOT_OPEN` | A navigation on a target without `app.url`; a `screen` call before `app.open()` | Add `app.url` to the target; open the app first | +| `INVALID_ARGUMENT`, `INVALID_LOCATOR` | A step argument failed validation; a locator got a bad option or filter key | Fix the call the code frame names | +| `LOCATOR_NOT_FOUND` | Wrong role or name, inexact text, element off screen or in an iframe, page not open | Read the markup for the accessible name; `exact: false` or a RegExp; `browser.frameLocator` for iframes; `app.open()` first; `--headed` | +| `LOCATOR_AMBIGUOUS` | Two matches (hidden duplicate, repeated label) | `{ name }`, a container scope, `filter`, `first()`, or `{ visible: true }` | +| `ASSERTION_FAILED` | Wrong expectation, or the state settles later than 5 s; for `agent.assert`, a false judgment (explained in the report) | Compare with the report's actual text or the screenshot; `{ timeout }` on the matcher; rewrite the question | +| `ASSERTION_INCONCLUSIVE` | `agent.assert` or `agent.extract` asked about what the screen does not show (another page, still loading, only in pixels); a failure, never a pass | `app.open()` or `agent.waitFor` the right screen first; ask about what is shown; `vision: true` when the answer is in pixels (the message says so) | +| `MODEL_OUTPUT_INVALID` | An `extract` or `assert` answer failed the schema after one repair round | Simplify the schema or question; pick a stronger model | +| `ACTION_FAILED` | Element not actionable (covered, disabled, detached), or an operation timed out | `expect` the condition first; close overlays; check `actionTimeout` | +| `TEST_TIMEOUT` | The attempt exceeded `timeout` (120 s) | Split the test, or raise `timeout` | +| `STEP_NOT_AWAITED` | The body returned while a step still ran: a call without `await` | `await` the call the code frame names (every `app`, `agent`, `screen`, `expect` call) | +| `MODEL_UNAVAILABLE` | `agents..model` holds no AI SDK instance: reported once under `run.errors` at the first `agent` fixture; the run stops. A model-less custom executor fails only `waitFor` and `extract` this way | Construct one, e.g. `gateway('openai/gpt-6-luna-fast')` from `ai`, and export its key (`AI_GATEWAY_API_KEY`) | +| `MODEL_PROVIDER_FAILED` | Network, 5xx, rate limit, no credits, or no response in 120 s, after the transport retries | Check the credential (the key; for keyless `gateway()` the Vercel CLI login and `.vercel/project.json`) and quota; retry (exit 3) | +| `REPLAY_STALE` | `--strict-cache` or `cache.strict`, and a committed recording no longer replays; `step.cache.reason` says why | Rerun read-write with the knob the message names off; commit the changed entry under the directory it names (a custom `cache.store` is written by that run) | +| `AUTOMATION_UNSUPPORTED` | The step needs an interaction the engine's toolset lacks (drag on a device, say); blocked, exit 1 | Do that step with `screen` actions, or run the test on a target that supports it | +| `STEP_TIMEOUT`, `STEP_BUDGET_EXHAUSTED` | Goal too big or ambiguous, or a slow provider | Split the goal, use on-screen wording, add vocabulary via `agents..context` or `agentContext`; raise `maxSteps` or `maxModelCalls` (budget), the config `timeout` (`act`), `judgmentTimeout` (judgments); `--debug` | +| `CONTEXT_OVERFLOW` | Screen plus step history did not fit the model's context window (`act` after one shrink-and-retry; a judgment on the first overflow) | Lower `agents..maxObservationBytes`, split the step, or pick a larger-window model | +| `POLICY_DENIED` | A `file:`, `data:`, or `javascript:` URL; a password `Secret` into a non-password sink; reading a secure field, `toHaveValue`, `toHaveText`, `toContainText`, `toHaveAttribute` included; `app.screenshot()` after a secret fill | http(s) only; passwords into password inputs only; assert the outcome, not the value; screenshot before filling secrets | +| `UNSUPPORTED_CAPABILITY` | A fixture the engine lacks (`browser` on a device), `schema` or `vision` on `act`, an action the surface lacks | Declare `requires: ['browser']`; drop the option or action | +| `SESSION_UNAVAILABLE`, `SESSION_CONTRACT` | `session: 'x'` with no setup saving `x`; a setup that skipped a declared name. `SESSION_MISMATCH`, `SESSION_EXPIRED`, `SESSION_INVALID`: the stored session is another run's or app's, expired, or corrupt | Add or fix the `test.setup`; rerun it | +| `ONLY_IN_CI` | `test.only` reached CI | Remove it | +| `AUTH_CREDENTIAL_UNAVAILABLE`, `AUTH_CREDENTIAL_INVALID` | `credentials.user('x')` for an undeclared name; the app rejected a configured credential | Add it to `config.credentials`; fix the stored value | +| `SECRET_UNAVAILABLE` | `secrets.get('x')` for an undeclared name | Add it to `config.secrets`; `E2E_SECRET_X` only overrides a declared one | + +## Tools + +| Do | When | +| --- | --- | +| `--headed` | Watch the failing step | +| `--workers 1 --retries 0` | Take parallelism and retries out of the picture | +| `--no-cache` | Rule out a stale `agent.act` replay | +| `--debug` | Each agent step's duration, model calls, cost, transcript | +| `--ai-trace`, then `npx unbox-ai runs .e2e/ai-trace.json` | What the model saw and called | +| `--video`, `--video=retain-on-failure` | Watch the failed attempt; `step.startedAt` minus the segment's `startedAt` is the step's offset into it | +| `command.log: '.e2e/logs/app.log'` | The app's output when it never gets ready or errors mid-test | +| `await app.screenshot('before-submit')` | Evidence before a secret is filled; later calls fail with `POLICY_DENIED` | +| `CI=1 npx e2e run` | Reproduce CI-only behaviour (defaults: topic running) | + +## Flaky tests + +- A read (`textContent()`, `count()`) caught a value mid-update: use a + matcher. +- Shared data: unique names per run, `afterEach` cleanup, or a `serial` + group. +- App not ready: assert on the element you are about to use, not the + previous page. +- An agent judgment asserts exact phrasing: judge the fact, add a + deterministic `expect` beside it. +- Load timing: `retries` masks it; `--workers 1 --headed` shows it. +- Measure a fix: `npx e2e run --repeat-each 10 --retries 0` + (`--no-cache` for agent steps). `Repeats` reads `0 of 1 test passed all + 10 runs` over `3/10 passed · repeat 0 ASSERTION_FAILED · ...` (0-based) + before, `1 of 1 test passed all 10 runs` after. + +## Is it the app? + +A deterministic step failing every run at the same place with the same code +is a product bug or a changed screen, not a flake: reproduce once with +`--headed`, then fix the app or update locator and expectation together. A +blocked agent step (`AUTH_CREDENTIAL_UNAVAILABLE`, `ENVIRONMENT_UNAVAILABLE`, +`SEED_DATA_MISSING`) exits 2 or 3 on purpose: fix the environment. diff --git a/.claude/skills/e2e/references/explore.md b/.claude/skills/e2e/references/explore.md new file mode 100644 index 000000000..b3b709c5b --- /dev/null +++ b/.claude/skills/e2e/references/explore.md @@ -0,0 +1,128 @@ +# Exploring without a test + +`e2e explore` runs the agent against the app with a goal instead of a test +file: see what it can do with an app before tests exist, hunt for regressions +on a branch, or find what is worth turning into a test. It needs a config with +a target and an agent that holds a model, nothing else. + +```bash +npx e2e explore # goal: "Explore the app and find bugs" +npx e2e explore 'Explore checkout like a first-time buyer and report anything off' +npx e2e explore --target web --max-steps 4 --headed +npx e2e explore 'Hunt for broken forms' --video +npx e2e explore --session admin 'Explore the admin settings' +``` + +## What a run does + +1. Opens the app on the target's URL. +2. Plans one step: a structured model call reads the goal, the steps and + findings so far, and the current screen, and answers with a title and a + concrete charter for one flow, or decides the goal is covered. +3. Runs the charter as an `agent.act()` step with the project's tools plus + `report_finding`. The agent reports each defect the moment it has evidence: + title, `issue` or `warning`, severity 1 to 5, expected, actual, reproduction + steps. The runner adds the path and a redacted screenshot. +4. Repeats until the planner finishes, the step limit, the clock, or three + failed or blocked steps in a row that reported nothing; then asks for a + closing assessment. + +A failed step does not end the run: it is recorded, and only reported issues +fail the product verdict. A run whose charters were all blocked stays blocked. +A step that hits its action or time budget ended at its limit and counts as +neither. Configured `credentials` reach the explorer as step secrets: the +planner knows the account names and usernames, and the agent fills passwords +with `type_secret` by name. That tool is offered only when the engine can fill +secrets; otherwise the agent skips signing in and says so in the step summary. + +## Start signed in + +`--session ` starts from a session a setup test saves +(`test.setup('...', { sessions: ['admin'] }, ...)` and `session.save('admin')`, +topic `writing-tests`). The run collects the config's test files, runs exactly +the setup that declares the session, then restores it into the exploration +and, on a target with a URL, opens the app, as a test with +`{ session: 'admin' }` does. No other test runs. The setup runs as for +`e2e run`, with the configured agents, cache, and retries; only the +exploration runs as the explorer. The planner and the agent are told they +start signed in, so no charter is spent signing in again. A name no setup +declares fails before any app process starts with `COLLECTION_ERROR`, naming +the declared sessions. + +A setup that filled a secret taints the restored session, so every finding +goes without a screenshot; one that signed in without a fill (a `browser.setCookies` +session cookie, say) keeps them where the engine captures pixels, and only +configured secrets are redacted (topic `writing-tests`). + +## Flags + +| Flag | Default | Effect | +| --- | --- | --- | +| `[goal]` | `Explore the app and find bugs` | One quoted sentence: the area and the posture. | +| `--target ` | first configured target | The one target to explore. | +| `--agent ` | `default` | Build the explorer from another configured agent (`agents.`). | +| `--session ` | none | Run the setup that saves this session, then explore with it restored. | +| `--max-steps ` | 8 (1 to 12) | Exploration steps at most. | +| `--timeout ` | 600000 (180000 to 900000) | Wall clock; the last minute is for the assessment. | +| `--headed`, `--reporter`, `--output`, `--debug`, `--ai-trace`, `--trace [mode]`, `--video [mode]` | as `run` | Same meaning as for `e2e run`. One attempt, so a retry mode (`on-first-retry`, `on-all-retries`, the CI trace default) records nothing (`CI=1 e2e explore` needs `--trace on`; the run's notice says so); put the goal before a bare `--trace` or `--video`. | + +Per-step action and model-call budgets default to 40 each; +`agents..maxSteps` and `agents..maxModelCalls` in the config +override them. The replay cache is off and retries are zero for the +exploration. + +## Reading the result + +Exit code `0`: steps ran, no `issue` was reported, and not every charter was +blocked; warnings are allowed. Exit code `1`: at least one `issue`, or no step +ran and nothing was found. If every charter was blocked and no `issue` was +reported, the run is `blocked` even with warnings, and the exit code is the +first blocker's own, as for `run`: 1 for an automation limit, 2 for a missing +model, credentials, seed data, or test setup, 3 for an unavailable +environment. When a charter passes, fails, or exhausts its budget, reported +issues decide the verdict. Other errors use `2` and `3` as for `run`. + +The terminal shows each step by its title with its duration, actions, and +findings, and each finding the moment it is reported as +`⚑ high issue Title (/path)`. At the end a `Findings` section lists every +finding, issues first and the most severe first, each with where it was seen, +its screenshot path, expected against actual, and the steps that reach it; +then the `Assessment` and the `run` summary with `Findings` and `Steps` in +place of `Test Files` and `Tests`. Severity words: critical 5, high 4, +medium 3, low 2, trivial 1. `.e2e/report.json` has the record under +`run.explore`: + +```json +{ + "goal": "...", + "budgets": { "maxSteps": 8, "timeoutMs": 600000 }, + "ended": "finished | step-limit | time | stuck | aborted", + "summary": "the closing assessment", + "steps": [{ "index": 1, "title": "...", "instruction": "...", "status": "passed | failed | blocked | exhausted", "summary": "...", "errorCode": "...", "startedAt": "...", "durationMs": 0 }], + "findings": [{ "id": "", "index": 0, "step": 1, "kind": "issue", "severity": 4, "title": "...", "expected": "...", "actual": "...", "reproduction": ["..."], "path": "/cart", "observationRevision": "...", "artifactId": ":artifact:2", "reportedAt": "..." }] +} +``` + +A step carries `errorCode` only when it did not pass; a finding carries +`step`, `path`, `observationRevision`, and `artifactId` only when known. +`artifactId` names the evidence screenshot among the attempt's `artifacts` in +the result whose `file` is `explore`, where its path, size, and digest are. +With `--session`, `run.results` also holds the setup's result and the +project's other tests as skipped (`filtered`). The attempt directory is +`.e2e/artifacts//explore--//attempt-0/`: the slug +is the goal's first words in lowercase ASCII, capped, and the digest keeps +distinct goals apart while the same goal always maps to the same directory. +For example +`.e2e/artifacts/web/explore-check-the-cart-totals-1a2b3c4d5e6f7a8b/default/attempt-0/finding-1.png`. + +Turn a finding into a test: its `reproduction` steps are the `agent.act()` +instructions or `screen.*` actions, and `expected` is the assertion. + +## When it does not fit + +The agents entry's `tools` and `system` carry over to the explorer; a +hand-rolled `StepExecutor` under `executor` is replaced by the built-in agent +for the run, with a notice on stderr, and needs a `model` on the entry (or the +executor's) or explore fails before it starts. Findings are the model's claims +plus evidence, not verified reproductions: read `actual` against the +screenshot before filing a bug. diff --git a/.claude/skills/e2e/references/mcp.md b/.claude/skills/e2e/references/mcp.md new file mode 100644 index 000000000..4ce194a05 --- /dev/null +++ b/.claude/skills/e2e/references/mcp.md @@ -0,0 +1,114 @@ +# Driving the app over MCP + +`e2e mcp` serves a project's live app to a coding agent over MCP (stdio). The +coding agent drives the app the way the testing agent does: look at a screen +before writing a test, check a locator before committing to it. Running tests +and reading a failed run stay on the CLI (topics `running` and `debugging`). + +## Setup + +The server ships with `e2e`. `e2e init` offers to register it; by hand: + +```bash +claude mcp add e2e -- npx e2e mcp # Claude Code +``` + +Or declare it in the client's project config (`.mcp.json` for Claude Code, +`.cursor/mcp.json` for Cursor, `.vscode/mcp.json` for VS Code): + +```json +{ "mcpServers": { "e2e": { "command": "npx", "args": ["e2e", "mcp"] } } } +``` + +Flags: `--config ` names the default config file, `--target ` +fixes the target every session opens on, `--headless` hides the browser or +simulator (sessions are headed by default outside CI), `--max-sessions ` +sets how many sessions may be open at once (default 4, 1 through 16). + +## Tools + +Four tools; everything a session can do is a catalog behind `call`. + +| Tool | Does | +| --- | --- | +| `open_session` | Loads the config (`config` names another file; default the nearest `e2e.config.ts`), starts the declared app command if any, boots the engine, opens the app URL, and returns the session id, the catalog, and the first observation. `target` is required when the config declares several. Each session has its own browser or device. | +| `tools` | The catalog: one line per tool with its argument names (`?` marks optional), the first sentence of its description, and `[read-only]` where it changes nothing. `tools {tool}` shows the full description and the JSON Schema of its arguments. | +| `call` | Runs one catalog tool: `call {tool: "tap", args: {target: "n42"}}`. Arguments are checked against the tool's schema first; a wrong one fails with `INVALID_ARGUMENT` naming the field. | +| `close_session` | Saves a recording still running, ends the attempt, disposes the engine, stops the app processes the session started once no other session uses them. | + +The catalog, per session: + +| Catalog tool | Does | +| --- | --- | +| `observe` | A fresh observation: one node per line as `#id role "name" ...`, plus the current path. On the web engine a link's `href` is its path on the app URL's origin, else origin and path (`mailto:` and `tel:` keep their scheme, `blob:` its inner origin and path); `?…` and `#…` mark a dropped query or fragment, `data:…` and `javascript:…` an inline payload, a trailing `…` a target cut at 256 characters. | +| `tap`, `double_tap`, `long_press`, `right_click`, `hover`, `type`, `press`, `select`, `check`, `scroll`, `scroll_to`, `drag`, `upload`, `navigate`, `back`, `dismiss_keyboard` | The grammar verbs, as the testing agent gets them: `check` takes `checked`, `drag` a `to` node, `upload` project-relative `files`; `dismiss_keyboard` comes with device engines. Each reports what changed on screen; `observe` shows the whole screen. A verb the engine cannot honor is not listed and fails with `UNSUPPORTED_CAPABILITY`. | +| `type_secret` | Fills a configured secret by name: a credential's password into a password field, a `secrets` entry into any editable input; the plaintext never reaches the agent. Listed when the config declares `credentials` or `secrets` and the engine can fill secrets. | +| `locate` | Tries a semantic locator with exactly one of `role`, `text`, `label`, `placeholder`, `testId` (`name` only with `role`; `exact` optional; anything else is `INVALID_ARGUMENT`) and returns how many nodes match, which, and the `screen.*` call to write. | +| `screenshot` | The masked pixels as an image. | +| `tap_at` | Taps a point (`x`, `y` in the latest screenshot's pixels): a listed control under it by id, otherwise the bare point, which an engine without a bare-point tap cannot do. Listed when the engine declares `tap` or a bare-point tap. | +| `hover_at` | Hovers a point the same way: a listed control under it by id, otherwise the bare point. Listed when the engine declares `hover` or a bare-point hover. | +| `type_at` | Types `value` into the field at a point: a listed input under it is filled by id; with a keyboard, anything else is tapped to focus it and typed into, at the caret unless `replace` is set. Without a keyboard a point on nothing listed fails. Listed when the engine declares `type` or a keyboard. | +| `press_at` | Sends one `key` (`Enter`, `Escape`, `Tab`) to the control at a point: a listed control gets it by id; with a keyboard, anything else is tapped to focus it and the key goes through the keyboard. Listed when the engine declares `press` or a keyboard. | +| `select_at` | Picks the option whose visible label is `value` in the select-like control at a point; the point must land on a listed select. Listed when the engine declares `select`. | +| `start_recording` | Starts a video of the app (`name` optional, for the file name). Listed when the engine records video. | +| `stop_recording` | Stops it and returns the absolute path of each video file, under `/videos//` (`.e2e` by default), or the URL of a provider's own recording. | +| Project tools | Every `defineTool` in the agent's `tools` that applies to the target's platform, under its own name; an engine pack such as `mobileTools` adds `open_app`, `swipe`, `alert`. | + +Once a secret is filled in the session, `screenshot` and the point tools +(`tap_at`, `hover_at`, `type_at`, `press_at`, `select_at`) answer with a +`PIXEL_TAINTED` line for the rest of it; act on listed nodes by id (topic +`writing-tests`). Before the session's first `screenshot`, a point tool only +says to take one. + +Resources: `e2e://guide` and `e2e://guide/` hold this skill. + +## Workflow + +1. `open_session`, then `call {tool: "observe"}` and act until the screen you + want to test is in front of you. Node ids are valid only for the newest + observation; an action reports what changed, so observe again before + using new ids. +2. `call {tool: "locate", args: {...}}` for each locator you intend to write. + One match: use the printed `screen.getByRole(...)` call. Zero or several: + adjust before writing the test; the same failure would hit the test as + `LOCATOR_NOT_FOUND` or `LOCATOR_AMBIGUOUS`. +3. Write `tests/.e2e.ts` (topic `writing-tests`). Deterministic steps + where you saw exact names; `agent.act` where the flow varies. +4. Run it from the shell: `npx e2e run tests/.e2e.ts`, read the + failure (topic `debugging`), fix, repeat. +5. `close_session` when you are done; an idle session closes on its own after + 30 minutes and never outlives 4 hours. When the client exits, every session + closes and the app commands stop. To look at another project or config, + `open_session {config: "path/to/e2e.config.ts"}`; no restart needed. + +## Rules + +- Record a demo or a bug for a pull request with `start_recording` once the + screen is set up, and `stop_recording` when the part worth watching is + over; `close_session` saves one still running. Videos are not masked: + keep secrets off screen while recording. +- A failed action is an error result that leads with the tool, its target, + and the code (`tap #n9 failed: LOCATOR_NOT_FOUND: ...`) and still shows + the screen it re-observed: re-aim from that screen. +- Nothing a session does is recorded as a test or into the replay cache. A + session is for looking and trying; the test is what you write afterwards. +- A run from the shell and a live session share the app only if the target's + app `command` sets `reuseExisting` (ignored in CI); otherwise close the + session before running. +- Parallel agents (subagents) share one server: each opens its own session + and passes its session id to every `tools`, `call`, and `close_session`. + A call may leave `session` out only while one session is open. Sessions + open at once share one config. Sessions on the same app command share its + process, which stops when the last of them closes. On mobile, two sessions + on one simulator fight over it: one target per device (topic `setup`), each + session on its own target. +- `TARGET_REQUIRED`: pass `target` to `open_session` or start with `--target`. + `UNKNOWN_TARGET`: no target of that name; the message lists the declared + ones. `NO_SESSION`: call `open_session` first, or the session named has + ended (the message says why). `SESSION_REQUIRED`: several sessions are open; + pass `session`. `SESSION_OPEN`: every session slot is taken (close one, or + raise `--max-sessions`). `CONFIG_IN_USE`: open sessions use another config; + open on theirs, or close them first. `ENGINE_IN_USE`: the target's engine + comes from a package every session shares; create it in the config or a + file it imports by path. `UNKNOWN_TOOL`: the name is not in this session's + catalog; the message lists what is. diff --git a/.claude/skills/e2e/references/running.md b/.claude/skills/e2e/references/running.md new file mode 100644 index 000000000..e7c788d2b --- /dev/null +++ b/.claude/skills/e2e/references/running.md @@ -0,0 +1,209 @@ +# Running tests + +## Commands + +```bash +npx e2e run [files...] [options] # run tests +npx e2e explore [goal] [options] # explore toward a goal without a test file (topic explore) +npx e2e list [files...] [options] # print what run would select +npx e2e init [directory] [--yes] # scaffold a project, refresh the agent skill +npx e2e guide [topic] # print this skill; topics: setup, writing-tests, agent, + # running, explore, debugging, mcp, bug-bash +npx e2e cache ls|clear|stats # inspect or empty the replay cache +npx e2e login|logout|models [provider] # e2e/oauth subscription logins: openai, + # github-copilot, spacexai +npx e2e mcp [--target ] # MCP server for a coding agent (topic mcp) +npx e2e feedback -m [opts] # report a problem with e2e itself +npx e2e telemetry [disable|enable] # anonymous usage telemetry: status or switch +``` + +`run` flags: + +| Flag | Effect | +| --- | --- | +| `[files...]` | Files, directories, quoted globs relative to the project root, or a bare name (`signup`, `signup.e2e.ts`, `agent/signup.e2e.ts` all select `tests/agent/signup.e2e.ts`); `file:line` is the test whose `test(` opens on that line. They narrow the config `tests` glob, never bypass it. | +| `--config ` | Config file; default `e2e.config.ts` or `.mts`, found upward. | +| `--target ` | Target names, comma-separated or repeated; only these start their app commands. Unknown names fail before startup. | +| `--tag ` | Any of the tags, comma-separated or repeated; all of them with `--tag-mode all`. An empty `--target`, `--tag`, or `--agent` value is a usage error, exit 2. | +| `--exclude-tag ` | Drop tests carrying any of these tags, however selected. | +| `--grep `, `--grep-invert ` | Keep, or drop, tests whose title (describe titles and test title joined by spaces, `checkout pays`; not file or tags) matches a regular expression. Bare pattern or `'/pattern/i'`; repeat for alternatives. | +| `--last-failed` | The tests the previous run (`/report.json`) did not pass, plus every test in a failed `beforeAll` or `afterAll` scope. No report is `NO_LAST_RUN`, exit 2. | +| `--shard ` | One contiguous slice of the selected tests (`--shard 2/3`), cut after every other filter; serial groups stay whole, each shard brings its own setup tests. | +| `--headed` | Visible browser or simulator when the engine supports it. | +| `--agent ` | Run unpinned tests as these `agents.` entries (default `agents.default`), comma-separated or repeated; several names run each such test once per agent. | +| `--workers `, `--retries ` | Override the resolved values; retries 0-10, workers 1-1024. | +| `--max-failures ` | Stop after n failures: the rest skip (cause `failure-limit`), running tests end `interrupted`; exit 1. | +| `--repeat-each ` | Run every selected test n times, each run its own result (`repeat` 0 through n-1); add `--no-cache` or later runs replay the first's recording. The `Repeats` summary row names each unstable test's failed runs. | +| `--reporter ` | `list`, `json`, `junit`, `markdown`, comma-separated; `json` cannot combine with `list`. | +| `--output ` | Results directory, over the config's `output` (default `.e2e`). | +| `--no-cache` | Replay cache off for this run. | +| `--strict-cache` | Fail a step whose committed recording no longer replays (`REPLAY_STALE`, exit 2) instead of handing it to the agent. | +| `--pass-with-no-tests` | Exit 0, not `NO_TESTS`, when nothing matches. | +| `--debug` | Phase timings and an agent step table on stderr; transcripts as artifacts. | +| `--ai-trace` | Every model call, to `/ai-trace.json`. | +| `--trace [mode]`, `--video [mode]` | Which attempts record a trace, or a video (WebM on browsers, MP4 on devices), over the config and every target: bare is `on`; `--trace off` skips the cost; `retain-on-failure` (video) keeps only failed attempts; `on-first-retry` records first retries, `on-all-retries` every retry. A test's own `trace` or `video` still wins; targets whose engine cannot record are skipped with a notice. Both are greedy: write `--video=` or put test files first. The failure recap names the video. | + +```bash +npx e2e run tests/signup.e2e.ts +npx e2e run signup.e2e.ts:12 # by bare name, the test declared at line 12 +npx e2e run tests/agent --tag smoke --exclude-tag slow --grep checkout +npx e2e run --last-failed # the loop after a red run +npx e2e run --shard 2/3 # one CI job of three +npx e2e run 'tests/**/*.smoke.e2e.ts' --target chromium --workers 1 --retries 0 +CI=1 npx e2e run # the CI defaults, locally +``` + +`list` takes the same files and selection flags (`--config`, `--target`, +`--tag`, `--tag-mode`, `--exclude-tag`, `--grep`, `--grep-invert`, +`--last-failed`, `--shard`, `--pass-with-no-tests`), prints one line per +test-target pair, `file › title [target] #tag`, skipped pairs ending in +` (skipped: )`, and starts no app, engine, or worker. +`--reporter json` prints `{ "pairs": [...] }`. + +```bash +npx e2e list tests/signup.e2e.ts --tag smoke --reporter json +``` + +With a `package.json` script `"test:e2e": "e2e run"`, pnpm forwards `--` +literally: `pnpm test:e2e -- --headed` reaches e2e as `run -- --headed` and +exits 2. Write `pnpm test:e2e --headed` or `pnpm exec e2e run --headed`. + +## The replay cache + +Entries live under `.e2e/cache/`, one file per key, named by key digest. +`cache` commands read the same config as `run` (`--config`, `cache.dir`); +with a custom `cache.store` they refuse (exit 2), inspect that store with +its own tools. + +| Command | Prints | +| --- | --- | +| `e2e cache ls` | One row per entry: test, target, instruction digest, age, action count. | +| `e2e cache stats` | Directory, entry count, total size. | +| `e2e cache clear` | Deletes the entries and the directory; files the runner never wrote stay. | + +## Output + +`` (`.e2e` by default) holds `report.json`, `junit.xml`, +`summary.md`, `failures/`, `ai-trace.json`, `sessions/`, and `artifacts/` +(screenshots, Playwright traces, videos, `--debug` transcripts, downloads). +`artifacts/` is cleared once a run's tests start; a run stopping before +leaves the last run's files. The report records every artifact path, a +hosted service's video by URL. + +- `list` (default): setup steps, one line per file and target, a `Failed + Tests` section (error, code, failing line, code frame), then the + summary rows `Test Files`, `Tests`, `AI`, `Cache` (when the replay cache + ran), `Repeats` (with `--repeat-each`), `Errors`, `Start at`, `Duration`, + `Report`, `AI trace` (with `--ai-trace`). Past a minute `Duration` + repeats as minutes and seconds, setup time split out as `startup` in the + same parenthetical (`682.97s (11m 23s, startup 43.00s)`). +- `report.json`, written whatever the reporters, holds `run.status`, + `run.exitCode`, `run.errors[]` (run-level, such as `APP_UNREACHABLE`), + and `run.results[]`, one per test and target: `titlePath`, `file`, + `source`, `tags` (`[]` when none), `agent`, `repeat` (0 unless + `--repeat-each`), `selected`, `status`, `attempts[]` of `steps[]`, + `artifacts[]`, `error`. +- `junit`: `junit.xml` for CI summaries; `--reporter list,junit` keeps the + terminal output. +- `markdown` (`--reporter list,markdown`): `summary.md` plus one page per + failed or flaky test under `failures/`. The summary holds counts and + spend, a block per failed test (error, facts, failing step, whether every + attempt failed alike, last model turns, screen location and closest + nodes, the line to look at, evidence paths), the flaky tests folded + alike, and every test as one folded table, a row per file (or an + exploration's findings and assessment); a failure page adds every step, + every kept turn, and the screen at failure inline. Read the page first; + paste the summary into a pull request or handoff. +- `json`: the report on stdout. +- Custom reporters get step progress with `identity` (`attemptId`, + `attemptIndex`, `stepId`, `stepIndex`: report IDs, zero-based indexes; + retries change the attempt, serial members share the group's attempt with + distinct step IDs) and an `end` phase carrying the redacted error and the + `blocked` and `cancelled` statuses. Older streams may lack `identity`. +- `github()` from `@e2e-dev/github`: on GitHub Actions, one pull request + comment per run (edited on rerun) plus the job summary; needs + `pull-requests: write` and `GITHUB_TOKEN` (or `GH_TOKEN`) in the step's + env. + +## Exit codes + +| Code | Meaning | +| ---: | --- | +| 0 | Every selected test passed, was flaky, or was skipped | +| 1 | A test or setup test failed or timed out | +| 2 | CLI, config, collection, credential, model-config, or policy error | +| 3 | Engine, app process, model provider, artifact, or cleanup failure | +| 4 | Internal runner error | +| 130 | Interrupted by an external signal | + +The highest code present wins (`130 > 4 > 3 > 2 > 1 > 0`). 130 needs an +external signal: a `--max-failures` stop or a run-level error that +interrupted workers keeps the failures' or the error's code. Exit 2 is +deterministic, never retry it; only exit 3 is worth a job-level retry. + +The first SIGINT or SIGTERM interrupts and writes the report if any test +had started; the second forces engine teardown; the third kills the app +process groups and exits 130 at once. + +## Continuous integration + +CI mode is on when `CI` is set (not `0` or `false`): `retries` 1, +`workers` 1, `trace` `on-first-retry`, `test.only` rejected with +`ONLY_IN_CI`, the replay cache `read-only` unless the config sets a mode explicitly +(`cache: 'read-write'` or `cache.mode`), `reuseExisting` ignored. + +```yaml +# .github/workflows/e2e.yml +# Omits the github() reporter: it needs pull-requests: write and +# GITHUB_TOKEN in the run step's env. +name: e2e +on: + pull_request: + push: + branches: [main] +permissions: + contents: read +jobs: + e2e: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + - uses: pnpm/action-setup@9fd676a19091d4595eefd76e4bd31c97133911f1 # v4.2.0 + - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 + with: + node-version: 26 # any Node >= 22.12 + cache: pnpm + - run: pnpm install --frozen-lockfile + - run: npx playwright install chromium --with-deps + - run: npx e2e run --reporter list,junit + env: + AI_GATEWAY_API_KEY: ${{ secrets.AI_GATEWAY_API_KEY }} + E2E_USER_ADMIN_USERNAME: ${{ secrets.E2E_USER_ADMIN_USERNAME }} + E2E_USER_ADMIN_PASSWORD: ${{ secrets.E2E_USER_ADMIN_PASSWORD }} + - if: ${{ !cancelled() }} + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 + with: + name: e2e-report + path: | + .e2e/report.json + .e2e/junit.xml + if-no-files-found: warn + - if: ${{ !cancelled() }} + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 + with: + name: e2e-artifacts + path: .e2e/artifacts + if-no-files-found: warn + retention-days: 7 +``` + +- Install browsers as their own step so the download stays out of the + launch timeout. +- Upload artifacts unless cancelled, so a test that failed then passed on + retry keeps its evidence. +- Start the app through the target's `app.command`; the runner tears it down + on every exit path. +- Agent steps run in the same job: pass the key the config's model reads + (`AI_GATEWAY_API_KEY` for `gateway()` from `ai`) as a secret in the run + step's `env`, and commit `.e2e/cache/` so recorded steps replay with no + model call. diff --git a/.claude/skills/e2e/references/setup.md b/.claude/skills/e2e/references/setup.md new file mode 100644 index 000000000..7c42c1e01 --- /dev/null +++ b/.claude/skills/e2e/references/setup.md @@ -0,0 +1,329 @@ +# Setting up e2e + +## Requirements + +- Node.js 22.12 or newer. +- ES modules: `.ts` config, tests, helpers, and workspace packages exporting + `.ts` source load as ESM regardless of the nearest `package.json` `type` + (CommonJS packages need no change); never `require` or `module.exports`. +- Browser tests: `@e2e-dev/web` plus `playwright` (`>=1.63.0 <2`), a peer the + engine does not install: an existing Playwright keeps its version and + browser cache, one out of range fails install as an unmet peer (npm's + `ERESOLVE`): upgrade `playwright` within the range. Missing + browsers download on first boot; in CI run `npx playwright install chromium + --with-deps`. Mobile tests: `@e2e-dev/mobile`, pinning `agent-device` + exactly; the pin moves with each engine release. + +## Scaffold + +Fresh project: + +```bash +npx e2e init # npm +pnpm dlx e2e init # pnpm +``` + +With `e2e` installed, run the installed version: `npx e2e init` or `pnpm exec +e2e init`; `npx e2e init my-app` scaffolds into a new directory. + +The wizard picks an engine and a model provider (None for tests without AI) +and offers to install this skill, register the MCP server for your coding +agent, and install dependencies. `--yes` picks Playwright and Vercel AI +Gateway, installs the skill in `.agents/skills/` (`.claude/skills/e2e` +symlinks to it), registers MCP in `.mcp.json` and `.cursor/mcp.json`, skips +installing dependencies. + +Init adds dependencies and a `test:e2e` script to `package.json`, writes +`e2e.config.ts` and `tests/example.e2e.ts`, updates `.gitignore`, leaves +existing configs and tests alone. Re-run after upgrading to refresh skill and +MCP entries. + +Without the wizard (`ai`, Vercel AI SDK v7, only for `agent.*` steps): + +```bash +npm install --save-dev e2e @e2e-dev/web playwright ai@^7 +``` + +## Subscriptions and API keys + +`e2e init` writes model config and dependencies for a subscription, an API +key, or a local endpoint. Authenticate: + +| Choice | Setup | +| --- | --- | +| ChatGPT Plus or Pro | `npx e2e login openai` | +| GitHub Copilot | `npx e2e login github-copilot` (GitHub CLI signed in, or your own `--client-id`) | +| SuperGrok or X Premium+ | `npx e2e login spacexai` | +| Vercel AI Gateway | Set `AI_GATEWAY_API_KEY`, or sign in to the Vercel CLI and `npx vercel link`; without the key `gateway()` uses a Vercel OIDC token | +| OpenRouter | Set `OPENROUTER_API_KEY` | +| Local or self-hosted endpoint | Set the endpoint URL and a model it serves, plus a key if required | + +Switching an existing config to ChatGPT: install `ai` and `@ai-sdk/openai`, +set `model: chatgpt('gpt-6-luna')` from `e2e/oauth/chatgpt`, run `npx e2e +login openai`. `npx e2e models` lists the ids each login serves. Use API keys +in CI. + +## The config + +`e2e.config.ts` sits at the project root and default-exports an object literal +ending in `satisfies E2EConfig`; unknown keys are `INVALID_CONFIG` at load. + +```ts +import type { E2EConfig } from 'e2e'; +import { web } from '@e2e-dev/web'; +import { gateway } from 'ai'; + +export default { + tests: 'tests/**/*.e2e.ts', + targets: [ + { + engine: web(), + app: { + url: 'http://127.0.0.1:3000', + command: { executable: 'pnpm', args: ['dev'], log: '.e2e/logs/app.log' }, + }, + }, + ], + // Model behind every agent.* step. + agents: { + default: { + model: gateway('openai/gpt-6-luna-fast'), + system: 'You are a thorough QA agent. Verify every outcome on screen.', + }, + }, + credentials: { + admin: { username: 'admin@example.test', password: process.env.ADMIN_PASSWORD ?? '' }, + }, +} satisfies E2EConfig; +``` + +A string `password` is checked at load (6+ code points): set `ADMIN_PASSWORD` +or `E2E_USER_ADMIN_PASSWORD` first, or defer to fill time with +`() => process.env.ADMIN_PASSWORD ?? ''`. + +| Key | Default | Notes | +| --- | --- | --- | +| `targets` | required | Non-empty; `--target` takes `name`. UI targets set `engine` and `app` (`platform` and `name` default to the engine's platform); tools-only targets may omit `engine` and must set `platform`. | +| `tests` | `'tests/**/*.e2e.ts'` | Globs relative to the project root (optional `./`): `*`, `?`, whole `**` segments, leading `!` excludes (`'!tests/wip/**'`; only exclusions is `INVALID_CONFIG`). Braces, character classes, extglobs, `..`, absolute paths, and a wildcard-free directory entry (`'!tests/wip'`) are `INVALID_GLOB`. | +| `timeout` | `120000` | Per test attempt, ms; also the default `agent.act` deadline. | +| `launchTimeout` | `60000` | Engine init and attempt start, ms. | +| `actionTimeout` | `30000` | Each locator action and engine operation, agent-step observations included. | +| `assertionTimeout` | `5000` | `expect` polling window. | +| `cleanupTimeout` | `30000` | Each `afterEach` hook, fixture teardown, engine cleanup, ms. | +| `retries` | `0`, `1` in CI | 0 to 10. | +| `workers` | half the cores, `1` in CI | Parallel test files, capped by the `workers` the engine declares per target. | +| `reporters` | `['list']` | `list`, `json`, `junit`, `markdown`, or `{ name, onEvent?, onRunFinished? }` objects; `json` excludes `list`, `--reporter` keeps the objects. | +| `cache` | `'read-write'` | `'read-write'`, `'read-only'`, `'off'`, or `{ mode, store, dir, strict }`; CI demotes only a defaulted mode to `'read-only'`. `strict` (`--strict-cache`) fails a step whose recording no longer replays (`REPLAY_STALE`) instead of handing it to the agent. | +| `agents` | `{ default: built-in }` | Tests run with `default`, `e2e run --agent ` picks another, entries never inherit from `default`. Options: topic `agent`. | +| `credentials` | `{}` | Named `{ username, password }`; `password` is a 6+ code point string or a function returning it. | +| `secrets` | `{}` | Named values the model never sees (API keys, tokens), same value rule. A separate namespace: a credential's password is `credentials.user(name).password` (named `.password`), never `secrets.get()`, so a name may be both. | +| `output` | `'.e2e'` | Results directory; `--output ` for one run. Inside the project root, not the root, not a tests glob's directory, never the cache dir (`cache.dir` stays `.e2e/cache`). | +| `artifacts` | none | `{ store }`: artifacts go to the host `ArtifactStore` (`{ put(artifact), putLink?(link) }`); `putLink` gets provider-hosted video links (never a passed `retain-on-failure` attempt's). Failure screenshots are always captured when the engine can. | +| `trace` | `'on'`, `'on-first-retry'` in CI | Attempts that record a Playwright trace: `'off'`, `'on'`, `'retain-on-failure'`, `'on-first-retry'`, `'on-all-retries'`; precedence and capability rule as `video`. | +| `video` | `'off'` | Same modes; `'retain-on-failure'` records all, keeps those that did not pass. Precedence: the test's `video`, `--video [mode]`, the target's (`{ engine, video }`), the config's. | +| `projectId` | the package name | Report and cache identity. | + +- `tests` discovery enters only directories a glob can match; symlinks are + not followed. +- `trace` and `video`: a config or flag mode skips targets whose engine + cannot record (one notice), a target or test mode requires it + (`UNSUPPORTED_ARTIFACT`); a retry mode with `retries: 0` prints a notice; + neither invalidates the replay cache. + +## The app under test + +The target declares the app; the engine only drives it. `web({ url })`, +`mobile({ app })`, and the other old app options are unknown keys. The +target's `app`: + +| Key | Meaning | +| --- | --- | +| `url` | Base URL for `app.open()` and relative navigation; required on a `web()` target, not supported on a mobile target yet. Missing scheme: `https://`, `http://` for loopback; port `0` on `127.0.0.1` or `[::1]` takes a free port. | +| `bundleId` | Device targets: the bundle id, package name, or display name (`Settings`) `app.open()` launches. | +| `appPath` | Device targets: the `.app` or `.apk` under test. A device target needs `bundleId` or `appPath`. | +| `launchArguments`, `permissions` | Device targets: arguments and permission states (`grant`, `deny`, `reset`) every fresh launch gets. | +| `command` | The process serving `url`: `{ executable, args, cwd, env, startupTimeout, shutdownTimeout, log, reuseExisting }`; `{port}` in `args` and `env` expands to `url`'s port. Targets declaring the same command share one process. | +| `readyUrl` | Readiness probe when it differs from `url`; `{port}` expands too. | +| `environment` | `'test'`, `'staging'`, `'production'`; inferred from the host, labels the report and cache key. | +| `identity` | Stable identity for cache and session keys when the origin changes per deploy (preview URLs). Defaults to the URL's origin and path, else `bundleId`, else `appPath`. | + +A Next.js 16 dev server blocks cross-origin requests to its dev resources, so +a target that opens `127.0.0.1` or `[::1]` while `next dev` identifies as +`localhost` never hydrates. Add the host to `allowedDevOrigins` in +`next.config.ts` (`'127.0.0.1'`, or `'[::1]'` for the bracket form). A +`localhost` target needs no entry, and a production `next start` target is +unaffected. + +There is no `services` key in this version: it is an unknown key wherever it +appears. Start dependency processes before the run, or have `app.command` +start a script that brings them up and serves the app. + +`web()` options: + +| Option | Meaning | +| --- | --- | +| `browser` | `'chromium'` (default), `'firefox'`, `'webkit'`, or a `BrowserProvider` leasing hosted browsers over CDP (`kernel()` from `@e2e-dev/kernel`, or your own), which implies chromium and excludes `connect`. Scope `'worker'` (default): one browser per worker slot from `prepare` to `finish`; `'attempt'`: one per attempt, with `reconnectEndpoint`'s limits. | +| `viewport` | `{ width, height }`, default 1280x720; `null` follows the browser window (hosted live view, headed run). On a headed hosted browser (Kernel) use `null` and size the service's screen; a fixed size gives a smaller, unmaximized window. | +| `connect` | `{ cdpEndpoint }` attaches to a remote Chromium over CDP; both it and `reconnectEndpoint` are resolvers `(signal) => url`, not strings. With `reconnectEndpoint` it rides one persistent default context and reconnects only to the original browser and page. | +| `headers` | Sent to the app's site only (Vercel's `x-vercel-protection-bypass`, ngrok's `ngrok-skip-browser-warning`), `agent.act` included; disables the browser HTTP cache and service workers. | +| `basicAuth` | `{ username, password }` for a `401` challenge; `password` may be `secrets.get('name')`, resolved per attempt and redacted like any secret, the base64 `Authorization` credential too. | +| `userAgent` | The `User-Agent` every attempt sends and `navigator.userAgent` reports. | +| `testIdAttribute` | What `getByTestId` reads; default `data-testid`. | +| `screencast` | `{ size?, quality? }` for the engine's own video: frame size (default the viewport's), JPEG quality 0 to 100. | + +- `basicAuth` via `secrets.get()` masks text, not screenshots or model pixels + showing the password. An undeclared name is `INVALID_CONFIG` at load, as is + the handle (a reference, not the value) in `app.command.env`, a template + literal, or `context`; read those from `process.env`. +- `reconnectEndpoint` or an attempt-scoped provider rides one persistent + context without `headers`, `basicAuth`, `userAgent`, `app.clearState()`, or + session state. Recovery never repeats a dispatched operation; exhausting + the budget is `OPERATION_TIMEOUT`. Without `reconnectEndpoint` a dropped + connection is reacquired at the next attempt. + +Two browsers, one app declaration: + +```ts +const app = { url: 'http://127.0.0.1:3000' }; +export default { + targets: [ + { name: 'chromium', engine: web(), app }, + { name: 'mobile-webkit', engine: web({ browser: 'webkit', viewport: { width: 390, height: 844 } }), app }, + ], +} satisfies E2EConfig; +``` + +### Let the runner start the app + +Prefer `app.command` to a hand-started dev server: self-contained locally +and in CI. + +```ts +engine: web(), +app: { + url: 'http://127.0.0.1:3000', + command: { + executable: 'pnpm', + args: ['dev'], + env: { PORT: '3000', DATABASE_URL: process.env.DATABASE_URL ?? '' }, + startupTimeout: 120_000, + log: '.e2e/logs/app.log', + }, +}, +``` + +- The runner spawns `command`, polls `readyUrl` (default `url`) for a 200 to + 499 status within `startupTimeout` (default 60 s), and stops it when the run + ends, fails, or is interrupted (`shutdownTimeout`, default 10 s). Never + ready is `APP_UNREACHABLE`; `.e2e/report.json` is still written. +- The child gets only `PATH`, `HOME`, the temp-directory variables, + `SystemRoot` and `COMSPEC` on Windows, and `command.env`; pass the rest + through `env`. Model keys and `E2E_USER_*` values are never inherited. +- Output is discarded unless `log` names a file under an ignored directory + such as `.e2e/logs/`; without it a server dying on boot is invisible. +- A `url` answering before the spawn is `APP_ALREADY_RUNNING`; + `command.reuseExisting: true` attaches to a running dev server (CI ignores + it). +- `executable` resolves on `PATH`, never through a shell; name the server + itself, not a wrapper script. +- `url: 'http://127.0.0.1:0'` (or `[::1]:0`, never `localhost:0`) takes a + free port; the command receives it as + `{port}` in `args` or `env` (`args: ['dev', '--port', '{port}']`), which + also expands in `readyUrl`. Tests read the URL from `app.baseUrl`; the + cache identity keeps `:0`. A port-0 `url` with no `command`, and + `reuseExisting` beside one, are `INVALID_CONFIG`. + A port grabbed between allocation and spawn fails the start with + `APP_UNREACHABLE`; rerun. + +For an app started elsewhere, point `app.url` at it, literally or via +`process.env.APP_URL ?? 'http://localhost:3000'`; the runner reads no +`APP_URL` and loads no `.env`, so put `process.loadEnvFile('.env')` atop +`e2e.config.ts` (workers re-import it). + +## Environment variables the runner reads + +| Variable | Effect | +| --- | --- | +| `AI_GATEWAY_API_KEY`, `OPENROUTER_API_KEY`, `OPENAI_API_KEY`, ... | Read by provider packages, not the runner. | +| `E2E_USER__USERNAME`, `E2E_USER__PASSWORD` | Override `credentials.`; `` is the name uppercased, other characters `_`. | +| `E2E_SECRET_` | Overrides `secrets.`, same rule. | +| `CI` | CI defaults; list in topic `running`. | +| `E2E_TELEMETRY_DISABLED`, `DO_NOT_TRACK` | Disable anonymous telemetry, as does `e2e telemetry disable`; `E2E_TELEMETRY_DEBUG=1` prints events instead of sending. | + +## Mobile targets + +`@e2e-dev/mobile` drives iOS simulators and Android emulators through +[agent-device](https://github.com/callstack/agent-device); needs Xcode with a +simulator runtime or the Android SDK with an emulator; run +`npx agent-device doctor` once. + +```ts +import type { E2EConfig } from 'e2e'; +import { mobile } from '@e2e-dev/mobile'; +import { mobileTools } from '@e2e-dev/mobile/tools'; +import { gateway } from 'ai'; + +const iphone = mobile({ platform: 'ios' }); + +export default { + targets: [{ engine: iphone, app: { bundleId: 'com.example.app' } }], + workers: 1, + agents: { + default: { + model: gateway('openai/gpt-6-luna-fast'), + tools: mobileTools(iphone), + }, + }, +} satisfies E2EConfig; +``` + +- `app.bundleId`: bundle id, package name, or display name `app.open()` + launches fresh (an attempt launches nothing itself; without it, the + installed build). `app.appPath`: the `.app` or `.apk` under test; the + engine installs nothing, so a fixture every test takes calls + `device.installApp()` once per device (no path installs `app.appPath`). + `app.launchArguments` and `app.permissions` ride every fresh launch: + arguments reach the app process (iOS) or `am start` (Android), permissions + are set first. +- One worker per device. No `device`: every booted simulator or emulator of + the platform is the pool, up to `workers` (none booted: agent-device boots + one); one `device`: one worker whatever `workers` says; a list (`device: + ['iPhone 17', 'iPhone 17 Pro']`): an explicit pool. Devices boot in + `prepare`, before the run's clock. Two sessions on one device fight over + it: for parallel MCP sessions or bug-bash explorers, declare one target per + device, each naming its `device`. +- `device` can be a `DeviceProvider` leasing hosted devices, one per worker + slot: `easSimulators({ buildId })` from `@e2e-dev/eas` takes the project + from the app config's `extra.eas.projectId` (`projectId` overrides), reads + `EXPO_TOKEN`, else the `eas login` session, needs no Xcode or Android SDK, and with `buildId` EAS installs + the app (omit `app.appPath`). A run must fit one session: `maxDurationMinutes`, + absent, is the account's cap (40 on a standard plan). `videoTouches: false` + on the engine for video there. +- Only a control that appeared or moved with the previous action waits out + `transition` (default 500 ms); agent actions settle `settle` ms (default + 150) before the next observation, `settle: false` skips it. +- `screen`, `expect`, `app`, `agent` work unchanged; `test` from + `@e2e-dev/mobile` types the `device` fixture (`installApp`, `openLink`, + `setPermission`, `setNetwork`, `setAppearance`, `clearKeychain`, `fold` + for an iPhone Duo's hinge, `locator`, more). Portable suites declare + `requires: ['device']`. +- No `state` capability: `test.setup` and `session` are unavailable; sign in + per test with `screen` actions or `agent.act`, both fill a `Secret` (topic + `writing-tests`, Sign-in sessions). +- A deterministic check naming a platform label runs there only: + `test('...', { platforms: ['ios'] }, ...)`. +- `selectOption`, `setInputFiles`, `scrollIntoView`, `secondaryTap`, and + `modifiers` on `tap` or `doubleTap` are `UNSUPPORTED_CAPABILITY` on a + device. +- React Native on iOS: checkbox and radio role and state come off the + accessibility value (`getByRole`, `check()`, `toBeChecked` work); a tab is + `other` with `selected`, query by test id or label; a plain `View` is a + leaf, scope to the `ScrollView` or give children test ids. + +## Done when + +- `npx e2e run tests/example.e2e.ts` passes against the app. +- `package.json` has a script such as `"test:e2e": "e2e run"`. +- `.gitignore` lists the `.e2e/` outputs (init adds them); drop the + `.e2e/cache/` line to commit replays. +- CI runs the whole suite, agent steps included, on PRs; see `running`. diff --git a/.claude/skills/e2e/references/writing-tests.md b/.claude/skills/e2e/references/writing-tests.md new file mode 100644 index 000000000..f7601c2e0 --- /dev/null +++ b/.claude/skills/e2e/references/writing-tests.md @@ -0,0 +1,389 @@ +# Writing tests + +## A complete file + +```ts +// tests/todos.e2e.ts +import { beforeEach, describe, test } from '@e2e-dev/web'; +import { expect } from 'e2e'; + +describe('todos', { tags: ['todos'] }, () => { + beforeEach(async ({ app }) => { + await app.open('/todos'); + }); + + test('adds and completes a todo', async ({ agent, screen, browser }) => { + await agent.act('add a todo named {title}', { params: { title: 'Write the release notes' } }); + await expect(screen.getByRole('listitem')).toHaveCount(1); + await expect(screen.getByRole('status', 'Remaining')).toHaveText('1 remaining'); + + await agent.act('mark the todo as done'); + await expect(screen.getByRole('status', 'Remaining')).toHaveText('0 remaining'); + await expect(browser).toHaveURL('/todos'); + }); + + test('ignores an empty submission', async ({ screen }) => { + // An exact interaction: the empty submit is the point of the test. + await screen.getByRole('button', 'Add').tap(); + await expect(screen.getByRole('listitem')).toHaveCount(0); + }); +}); +``` + +The agent does the flow; `expect` pins what must be true after each goal, +and that check lets the replay cache rerun the step later. `screen` actions +are for exact interactions and values. Files match the config `tests` glob, +default `tests/**/*.e2e.ts`. In a browser every test starts from a fresh +context with no page open, so it calls `app.open()` first; on a device a test +that skips `app.open()` starts where the previous test left the app. + +## Registration + +`test` registers everything, and `describe`, `beforeEach`, `afterEach`, +`beforeAll`, and `afterAll` are also top-level imports of the same functions +(`import { describe, beforeEach } from 'e2e'`; `@e2e-dev/web` and +`@e2e-dev/mobile` export `test`, `describe`, and the hooks typed with their fixture; a +`test.extend()` chain registers hooks that see its fixtures through +`test.beforeEach`). Everything registers at import, so a `describe` body is +synchronous (`async` is a `COLLECTION_ERROR`). + +```ts +test('title', async ({ app, screen }) => {}); +test('title', { tags: ['smoke'], retries: 2, timeout: 60_000 }, async ({ app }) => {}); +describe('group', { tags: ['billing'] }, () => { /* tests and hooks */ }); +describe('checkout flow', { serial: true }, () => { /* ordered, shared app state */ }); +beforeEach(async ({ app }) => {}); // per attempt, with test fixtures +afterEach(async ({ screen }) => {}); // runs after failures too, with its own cleanup budget +beforeAll(async ({ platform }) => {}); // per suite realm, no app fixtures +afterAll(async () => {}); +test.skip('later', async () => {}); +test.only('focus', async () => {}); // local only: CI fails with ONLY_IN_CI +test('conditional', async () => { test.skip(await onlyOneOrg(), 'nothing to switch to'); }); // throws: the body stops here, reported skipped; call it before the first step +test('later', async () => { test.skip('waiting on the API'); }); // bare skip from the body +test.setup('sign in', { sessions: ['admin'] }, async ({ app, screen, session }) => {}); // see Sign-in sessions +const wsTest = test.extend<{ ws: Ws }>({ ws: async ({ browser }, use) => { await use(await seed()); await drop(); } }); +wsTest('uses the workspace', async ({ ws }) => {}); // code after use() is teardown, runs after failures too +``` + +A setup test cannot skip from its body (`INVALID_ARGUMENT`). + +| Option | Default | Notes | +| --- | --- | --- | +| `timeout` | `config.timeout`, 120 s | Covers `beforeEach` and the body. | +| `retries` | `config.retries` | 0 to 10; a serial group's value applies to its members. | +| `tags` | `[]` | Distinct, non-blank, no comma or edge whitespace (`'Login Form'` is fine); union across layers. `--tag smoke` selects, `--tag-mode all` requires every tag. | +| `skip` | unset | `true` or a reason string. | +| `platforms` | unset | Only targets with these platforms, e.g. `['ios']`. | +| `requires` | `[]` | Engine capabilities, e.g. `['browser']`; missing ones skip the test at selection rather than failing it with `UNSUPPORTED_CAPABILITY`. | +| `session` | unset | Restore state saved by a setup test. | +| `agentContext` | unset | Extra context for `agent.*` calls in this test or group. | +| `agent` | the run's agent | A configured name (`agents.`) or a list run once per agent; `--agent` narrows the list, a setup test takes one name. Innermost wins; `agent.act(..., { agent })` names another for one call. | +| `trace`, `video` | the target's | `'off'`, `'on'`, `'retain-on-failure'`, `'on-first-retry'`, `'on-all-retries'`. Innermost wins over `--trace` / `--video`, the target, and the config; recording where the engine cannot is `UNSUPPORTED_ARTIFACT` for the run. | +| `serial` | `false` | Groups only. Members share app state, run in order on one worker, retry as a whole, and take the group's `trace` and `video`. | + +Serial members cannot set `retries`, `trace`, `video`, `session`, +`platforms`, `requires`, `skip`, or `only`, nor can a nested `describe` set +`trace` or `video`; nesting serial groups is a `COLLECTION_ERROR`. + +Hooks nest: outer `beforeEach` first, inner `afterEach` first. `beforeAll` +reruns per retry and per serial group (each a fresh module realm) on the +test timeout; `afterAll` runs on `config.cleanupTimeout`. A failing +`beforeAll` is a `HOOK_FAILED` run error skipping every test in its scope. + +## Fixtures + +Built-in fixtures are lazy; destructure them. Your own `test.extend` +fixtures set up for every test registered through that `test`, destructured +or not, so a state-changing fixture belongs on its own `test`. + +`app` (`App`) and `screen` (`Screen`): always. `agent` (`Agent`): needs a +configured model, else `MODEL_UNAVAILABLE` (topic `agent`). `platform` +(`string`): always, hooks included; `web`, `ios`, `android`, or an engine's +label. `browser` (`Browser`): browser targets, import `test` from `@e2e-dev/web`. +`device` (`Device`): device targets, import `test` from `@e2e-dev/mobile`. +`session` (`SetupSession`): only in `test.setup`. + +### app + +- `baseUrl` (`string | undefined`): the target's app URL with the run's + port; `undefined` when the target declares no `app.url`. +- `open(path?)`: opens the target's `app.url`, a relative path, or an absolute + http(s) URL. On a device it takes no path and relaunches the pinned app. +- `back()`: one history step back. +- `restart()`: recreates the context keeping persisted state (a restored + session included), then reopens the base URL. +- `clearState()`: clears cookies and storage, recreates the context, reopens + the base URL. +- `screenshot(label?)`: saves a redacted screenshot artifact and returns its + path. Denied after a secret fill (see Sign-in sessions). + +`clearState()` is `UNSUPPORTED_CAPABILITY` with `connect.reconnectEndpoint` +or an attempt-scoped browser provider; on a device with no pinned `app`, both +methods are. + +## Locators + +`screen.getBy*` builds a lazy query, resolved only by an action, read, or +assertion. Every query also exists on a locator, scoped to its subtree. + +| Query | Matches | +| --- | --- | +| `getByRole(role, name?, { exact?, checked?, disabled?, selected?, expanded?, pressed?, level?, visible? })` | Semantic role, optionally by accessible name (`getByRole('button', 'Save')`; the object form `{ name }` works too) and state (`level`: heading level 1 to 6). First choice. | +| `getByLabel(text, { exact?, visible? })` | Form controls by label. | +| `getByPlaceholder(text, { exact?, visible? })` | Inputs by placeholder. | +| `getByText(text, { exact?, visible? })` | Visible text. | +| `getByDisplayValue(value, { exact?, visible? })` | Inputs by current value; on the web it cannot scope child queries or be a `filter({ has })` target. | +| `getByTestId(id, { visible? })` | `data-testid` on the web (or `web({ testIdAttribute })`), accessibility identifier or resource id on a device; a string matches the whole id, a RegExp tests it. Last resort. | + +Roles: `button`, `link`, `textbox`, `searchbox`, `combobox`, `listbox`, +`option`, `checkbox`, `radio`, `radiogroup`, `switch`, `slider`, `spinbutton`, +`progressbar`, `meter`, `image`, `heading`, `tab`, `tablist`, `tabpanel`, +`menu`, `menubar`, `menuitem`, `menuitemcheckbox`, `menuitemradio`, `toolbar`, +`tooltip`, `tree`, `treeitem`, `list`, `listitem`, `table`, `grid`, `row`, +`rowgroup`, `rowheader`, `cell`, `gridcell`, `columnheader`, `separator`, +`group`, `article`, `figure`, `form`, `status`, `alert`, `dialog`, +`alertdialog`, `main`, `navigation`, `banner`, `contentinfo`, `complementary`, +`region`. The union is closed (anything else is a type error); `img` aliases +`image`; a role the platform lacks (`tooltip` on a phone) matches nothing. +On the web a `contenteditable` host is a `textbox` for the agent and takes +`fill`; from a test use `getByLabel`, `getByTestId`, or `role="textbox"` on +the host. + +Text matching is exact after whitespace normalization (whole string, +case-sensitive) for a `getByRole` name, `getByLabel`, `getByPlaceholder`, and +`getByText`: `name: 'Save'` misses `Save changes` (unlike Playwright). A +missed query makes a negated assertion pass, so check it positively. +`getByText` and `getByLabel` return the innermost match, so a container +echoing its child does not count twice. `exact: false` is a case-insensitive +substring; a `RegExp` matches as written. + +- Exactly one match per action, read, or assertion. Two fail at once with + `LOCATOR_AMBIGUOUS`; at zero, actions and assertions poll until the + timeout then `LOCATOR_NOT_FOUND`, a read fails at once. Exceptions: + `toHaveCount`, `toBeVisible`, `toBeHidden`, `toBeAttached`, list-form + `toHaveText` and `toContainText`, `isVisible()` (false at zero), + `isHidden()` (true), `count()`, `all()`, `allTextContents()`. +- Narrow with `filter({ hasText })` (a case-insensitive substring, unlike a + query), `filter({ has: locator })`, `first()`, `last()`, `nth(i)`, or + scoping under another locator. Any other `filter` key (`hasNot`, + `hasNotText`) or an empty `filter({})` is `INVALID_LOCATOR`. +- `visible: true` drops nodes the page hides (a closed drawer) before the + exactly-one rule. +- On the web, queries reach open shadow roots and closed roots attached with + `attachShadow`, not declarative closed roots. `browser.locator(css)`, + `frameLocator`, and `filter({ hasText })` stop at a closed root; query the + text inside or filter with `has`. +- `screen.scrollUntilVisible(locator, { direction?, momentum?, timeout? })` + scrolls the viewport (`down`, `slow` by default) until the locator resolves + visibly, else `LOCATOR_NOT_FOUND`; on a locator it scrolls that node. + +### Actions + +Each action resolves one node, waits up to `config.actionTimeout` (30 s, or +`{ timeout }`) for it to be actionable, and does one thing: `tap()` (alias +`click()`), `doubleTap()`, `secondaryTap()` (each takes +`{ modifiers: ['Shift'] }`), `longPress({ duration? })` +(100 to 10000 ms; default 500 on the web, 1000 on a device), +`fill(value | Secret)`, `pressSequentially(text, { delay? })`, `clear()`, +`press(key)`, `check()`, `uncheck()`, +`selectOption(label | { label } | { value } | { index })` (one option; an +array is `INVALID_ARGUMENT`), `focus()`, `hover()`, `setInputFiles(paths)` +(relative to the project root), `dragTo(locator)`, `scrollIntoView()`, +`swipe({ direction, momentum? })`. Web only, `UNSUPPORTED_CAPABILITY` on a +device: `secondaryTap`, `selectOption`, `setInputFiles`, `scrollIntoView`, +and `modifiers`. + +`fill` sets the value with no key events; when the app reacts to keystrokes +(autocomplete, a masked input) use `pressSequentially`, which focuses the +field and types one character per `delay`, plain string only (a `Secret` is +`INVALID_ARGUMENT` and goes through `fill`). + +Coordinates are CSS pixels, for what the tree does not list: `tap({ position: +{ x, y } })` offsets from the node's top-left corner (with `modifiers` it is +`INVALID_ARGUMENT`), `screen.tapAt({ x, y })` taps a viewport point, +`screen.swipe({ from, to })` swipes along a path, +`screen.swipe({ direction, momentum? })` swipes the viewport. Prefer a +locator; a point moves with the layout. + +### Reads + +Reads resolve once, no retry: `textContent()`, `inputValue()`, +`getAttribute(name)`, `isVisible()`, `isHidden()`, `isEnabled()`, +`isDisabled()`, `isChecked()`, `boundingBox()`, `count()`. `all()` gives one +`nth(i)` locator per current match and `allTextContents()` every match's +text, both `[]` at zero. Text is the rendered text, whitespace collapsed: on +the web what `innerText` reads (`text-transform` applies, `display: none` +drops out, `
` is a space); `toHaveText` reads the same. `isChecked()` is +`false`, not an error, on a node with no checked state, so query checkable +controls by role. `waitFor({ state?: 'visible' | 'hidden', timeout? })` waits +within `actionTimeout`, else `LOCATOR_NOT_FOUND`. For a value that has to +settle use `expect`, not a read. Reading a password field's value or +attributes is `POLICY_DENIED`, as is `toHaveAttribute` on one, negated too. + +## expect + +`expect(locator)` polls up to `config.assertionTimeout` (5 s) or +`{ timeout }`; `.not` inverts and passes once the negation has held 1 s +continuously, so it never returns in under a second. `expect(value, +message?)` is synchronous. +`expect.poll(read, { timeout?, interval?, message? })` re-reads until a value +matcher passes (`assertionTimeout` and 100 ms by default, stopping with the +attempt); a throwing read keeps polling, and it is not a report step. +`expect.soft(x)` keeps a failure instead of throwing; the attempt fails +after the body with every soft failure listed. +`expect.any(Class)`, `expect.anything()`, `expect.objectContaining(obj)`, +`expect.arrayContaining(arr)`, `expect.stringContaining(s)`, and +`expect.stringMatching(s | RegExp)` stand in for values inside `toEqual`, +`toMatchObject`, `toContain`, and `toHaveProperty`. + +```ts +await expect(screen.getByRole('dialog')).not.toBeVisible({ timeout: 10_000 }); +await expect(browser).toHaveURL('/dashboard'); // relative to the base URL, or a RegExp +expect(order).toMatchObject({ id: expect.any(Number), lines: [{ sku: 'a' }] }); +expect.soft(await screen.getByTestId('tax').textContent()).toBe('$8.00'); // kept, body runs on +const users = expect(await response.json()).toMatchSchema(z.array(User)); // any Standard Schema; typed output +``` + +`toMatchSchema(schema)` takes a synchronous Standard Schema (Zod, Valibot, +ArkType), fails listing every issue by path, and returns the parsed value +typed; prefer it when the app already has a schema for the response. + +| Locator matchers | Browser matchers | Value matchers | +| --- | --- | --- | +| `toBeVisible`, `toBeHidden`, `toBeAttached`, `toBeEnabled`, `toBeDisabled`, `toBeChecked`, `toBeSelected`, `toBeExpanded`, `toBeFocused`, `toHaveText`, `toContainText`, `toHaveValue`, `toHaveAttribute`, `toHaveCount`, `toHaveAccessibleName` | `toHaveURL`, `toHaveTitle`, `toHaveClass(locator, expected)` | `toBe`, `toEqual`, `toMatchObject`, `toBeTruthy`, `toBeFalsy`, `toBeNull`, `toBeUndefined`, `toBeDefined`, `toHaveLength`, `toHaveProperty`, `toContain`, `toMatch`, `toBeGreaterThan`, `toBeGreaterThanOrEqual`, `toBeLessThan`, `toBeLessThanOrEqual`, `toBeCloseTo`, `toMatchSchema` | + +`toHaveText` compares the whole normalized text, `toContainText` a substring +or RegExp, `toHaveValue` a form control's value as is, whitespace included +(it fails on a node with none); on a password field all three are +`POLICY_DENIED`, never a comparison against `''`. List forms: +`toHaveText(['Alpha', /^Beta/])` needs exactly two matches with those texts +in order; `toContainText(['Alpha', 'Beta'])` needs each entry in a distinct +match, in order, extra matches allowed. `toHaveAttribute(name)` checks +presence, `toHaveAttribute(name, value)` the value; `toBeAttached` waits for +a match, hidden or not; `toHaveClass` compares the whole normalized class +list or tests a RegExp. A failed matcher is `ASSERTION_FAILED`, exit code 1. + +## Sign-in sessions + +Sign in once in a setup test, save the state under a name, and let other +tests declare it. Selecting a dependent test alone still runs its setup. + +```ts +// tests/auth.setup.e2e.ts +import { test } from '@e2e-dev/web'; +import { expect, credentials } from 'e2e'; + +test.setup('authenticate as admin', { sessions: ['admin'] }, async ({ app, screen, session, browser }) => { + const admin = credentials.user('admin'); + await app.open('/login'); + await screen.getByLabel('Email').fill(admin.username); + await screen.getByLabel('Password').fill(admin.password); + await screen.getByRole('button', 'Sign in').tap(); + await expect(browser).toHaveURL('/dashboard'); // prove the sign-in worked before saving + await session.save('admin'); +}); +``` + +```ts +// tests/dashboard.e2e.ts +import { test, expect } from 'e2e'; + +test('the dashboard opens directly', { session: 'admin' }, async ({ app, screen }) => { + await app.open('/dashboard'); + await expect(screen.getByRole('heading', 'Dashboard')).toBeVisible(); +}); +``` + +- Setup tests are top-level; exactly one setup saves a given name and the + body saves every declared name once (`SESSION_CONTRACT` otherwise). Names + match `[A-Za-z0-9_.-]{1,128}`. +- A session holds cookies, local storage, and IndexedDB for one run, + encrypted and deleted at cleanup; server state is not part of it. +- Once a secret is filled, model pixels and assertion screenshots are + withheld for the rest of that session (later serial members included) and + `app.screenshot()` is `POLICY_DENIED`. A restored session keeps its setup's + taint; a setup that signs in without a fill (`browser.setCookies`, say) leaves + screenshots available. +- Credentials live in the config, values in the environment: + +```ts +credentials: { + admin: { username: 'admin@example.test', password: process.env.ADMIN_PASSWORD ?? '' }, +}, +``` + +- A static password or secret needs 6 or more code points, else + `INVALID_CONFIG` at config load, so an unset variable fails every command. + A function is read at fill time and redacted only from that fill on. +- `E2E_USER__USERNAME` and `E2E_USER__PASSWORD` override either + field per run, even over a function; `` is the credential name + uppercased, every character outside `[A-Z0-9]` as `_`. +- `credentials.user('admin').password` is a `Secret` with no plaintext + accessor, named `admin.password` to the agent and in reports; `secrets.get()` + never returns it (separate namespaces). Only `fill()` and `agent.act` params accept it, stringifying it + is `INVALID_CONFIG`, and `credentials.user()` outside a run throws + `AUTH_CREDENTIAL_UNAVAILABLE`. +- Any other sensitive value (an API key) is a `secrets` entry, + `secrets: { 'stripe-key': process.env.STRIPE_KEY ?? '' }`, overridable with + `E2E_SECRET_STRIPE_KEY`; `secrets.get('stripe-key')` is the same kind of + handle, fills any editable input, and is redacted by name everywhere the + runner writes. + +## The browser fixture (browser only) + +Prefer `app` and `screen`; `browser` is for what only a browser has, and a +portable suite declares `requires: ['browser']`. Page methods act on the one +active tab; cookies, routes, and `onDialog` cover the whole browser. A tab +the app opens itself (`target="_blank"`, `window.open`) is not followed: +`browser.goto` its URL instead. + +- `goto(url, { waitUntil?, timeout? })`, `reload({ timeout? })`, + `back({ timeout? })`, `forward({ timeout? })`: navigation on the test + timeout by default; `goto` takes a base-relative path. +- `url()`, `title()`, `waitForURL(url | RegExp, { timeout? })`: reads and a + URL wait; `waitForURL` matches like `toHaveURL`. +- `locator(css)`: raw CSS or XPath; not portable, a last resort. +- `frameLocator(css)`: a `Screen` scoped to one iframe + (`browser.frameLocator('#payment').getByLabel('Card number')`); keeps + `locator(css)` and `frameLocator(css)` for nesting. +- `evaluate(fn | source, arg?)`: runs a function or source string in the + page, JSON in and out, no closures; a throw in the page is + `EVALUATE_FAILED`. +- `route(pattern, handler)`, `unroute(pattern)`: intercept requests; + `route.request` has `url`, `method`, `headers`, `postData`. The handler + calls exactly one of `fulfill({ status?, headers?, json | body })`, + `continue()`, or `abort()`; none or two fails the next step with + `ACTION_FAILED`. +- `waitForResponse(pattern, { timeout? })`: resolves with + `{ url, status, headers, json(), text() }`; `text()` and `json()` reject + with `ACTION_FAILED` when the body could not be read. +- `cookies()`, `setCookies([...])`: a target is an http(s) URL or a domain. +- `setViewport({ width, height })`: resize. +- `onDialog('accept' | 'dismiss' | handler)`: awaited; resolves to an async + unsubscribe. Register it before the tap that opens the dialog. A handler + gets `{ message, accept(text?), dismiss() }`, `accept` taking the prompt + text; no handler, or one that neither accepts nor dismisses, fails the + next step with `INVALID_STATE`. +- `waitForDownload(() => trigger, { timeout? })`: returns + `{ path, suggestedFilename }`. +- `keyboard.press(key)`, `keyboard.type(text)`, `mouse.*`: unfocused input; + prefer `locator.press` and `locator.fill`. + +A `route`, `unroute`, or `waitForResponse` pattern is a glob string or +`RegExp` matched against the full URL (`*` stays within one path segment, +`**` crosses `/`, `?` is one character); a predicate function is +`INVALID_ARGUMENT`. + +## Habits + +- Selectors come from the source (labels, roles, text); add an `aria-label` + or heading where the app has no accessible name rather than fall back to + `browser.locator('.btn-primary')`. +- Test data gets a run-unique name (`Invoice ${Date.now()}`) and `afterEach` + cleanup, so replays and retries never trip over leftovers. +- APIs live in the same suite: `fetch(new URL('/api/users', app.baseUrl))` + plus value matchers, in a `test.extend` fixture that reads `browser.cookies()` + when the API needs the session. +- No sleeps or polling loops; a matcher with a longer `timeout` instead. +- `await` every step call, else `STEP_NOT_AWAITED` at the line of the call. +- Assert the fact a model produced with `toContain`, not its exact sentence. diff --git a/.claude/skills/verify-debatelab/SKILL.md b/.claude/skills/verify-debatelab/SKILL.md index 89dc753fe..e650f142d 100644 --- a/.claude/skills/verify-debatelab/SKILL.md +++ b/.claude/skills/verify-debatelab/SKILL.md @@ -19,6 +19,7 @@ $H up --live [--max-calls N] # real OpenRouter (pinned) + real Exa $H doctor # read-only: is this instance ours, local, and healthy? $H drive |e2e/> [--project desktop-chromium|mobile-chromium] [-g ] $H user [label] # mint a demo-code login for interactive driving +$H e2e <run|explore|list|cache> [args] # agentic tests in agentic/ and explorations (TesterArmy e2e) $H down # stop everything `up` started; evidence stays ``` @@ -104,6 +105,23 @@ To prove a feature without a recipe, copy `recipes/case-generation.spec.ts` to ` Selectors: use accessible names and roles as the feature files list them (`getByRole("button", { name: "Generate Debate Case" })`). The exceptions use `data-testid`: Crossfire's controls, the Current Debate control, strip and dashboard, and the voice controls (`voice-mode`, `voice-mic`). Use `exact: true` for short names such as `Start` or `Save`, because the header's `Start debate` also matches `Start`. Never use coordinates. +## Agentic tests and exploration + +`$H e2e` runs [e2e](https://e2e.tester.army/docs) against this stack: an AI agent drives goals and judges screens, and exact assertions pin the result. The tests are `agentic/*.e2e.ts`, the config is `e2e.config.ts`, and the API is in the `e2e` skill (`.claude/skills/e2e/SKILL.md`). docs/testing.md, under "Agentic browser tests", says what the suite covers and what a pass proves. + +```bash +$H e2e run # the suite, desktop and mobile targets +$H e2e run agentic/hungarian.e2e.ts --target mobile # one file, one viewport +$H e2e explore "Use Brainstorm and Attack on one motion and report anything off" --session student --agent coach +$H e2e cache ls # recorded agent flows in agentic/replay +``` + +- `e2e` runs `doctor` first and refuses a broken stack, like `drive`. It turns e2e telemetry off and prints `EVIDENCE <dir>`, which holds `summary.md`, `report.json`, and per-attempt screenshots, traces, and failure videos. +- The testing agent has its own OpenRouter models: free `qwen/qwen3.8-27b:free` acts (test steps and explorations) and `meta/muse-spark-1.3-contributor` judges, under two cents per suite run. `E2E_AGENT_MODEL` and `E2E_JUDGE_MODEL` override them. The key comes from `OPENROUTER_API_KEY`, `LIVE_OPENROUTER_API_KEY`, or the live keys file, and is needed in fake mode too. A recorded `agent.act` replays with no model call; every judgment and every exploration step is a call. Without a key, agent tests skip. A `temporarily rate-limited upstream` error from the free actor is infrastructure: rerun with `--last-failed`. For a long or important exploration, `E2E_AGENT_MODEL=openai/gpt-6-luna` is faster and more reliable for cents (the Muse Spark judge cannot act). +- `--session student` starts an exploration signed in as a fresh demo user, minted by `agentic/sessions.e2e.ts`. `--agent` picks a persona: `first-timer`, `coach`, `edge-input`, or `hungarian`. `--target mobile` explores the phone-sized viewport. +- Explorations report `issue` (functional) and `warning` (cosmetic) findings, and exit 1 when there is an issue. Treat a finding as a lead: the stack serves Vite dev, so a route that is still compiling can look dead, and fake mode answers every motion with free-transport text. Prove a finding with a repro test tagged `repro` before you report it as a bug. +- Several explorations can run at once (a bug bash), because each gets its own evidence directory. They share this one stack, so give each a different area. + ## Evidence Each drive writes to `.verify/artifacts/<UTC-timestamp>-<recipe>/` and prints `EVIDENCE <dir>`: diff --git a/.claude/skills/verify-debatelab/bin/verify.sh b/.claude/skills/verify-debatelab/bin/verify.sh index 6fdf21f87..5e1dd31ac 100755 --- a/.claude/skills/verify-debatelab/bin/verify.sh +++ b/.claude/skills/verify-debatelab/bin/verify.sh @@ -14,6 +14,10 @@ # run recipes (`all`, `recipes/<name>`) or a maintained # suite (`e2e/live/`, `e2e/workspace-collaboration`) # and keep the evidence +# verify.sh e2e <run|explore|list|cache> [e2e args...] +# run the agentic suite in agentic/ (TesterArmy e2e) +# or an exploration against this stack; the testing +# agent's model is billed to the OpenRouter key # verify.sh down stop what `up` started; evidence is kept set -euo pipefail @@ -273,6 +277,50 @@ cmd_drive() { return "$status" } +cmd_e2e() { + load_state + local sub="${1:-run}"; [ "$#" -eq 0 ] || shift + case "$sub" in run|explore|list|cache) ;; *) fail "usage: verify.sh e2e <run|explore|list|cache> [e2e args]" ;; esac + cmd_doctor >/dev/null || { cmd_doctor || true; fail "doctor failed; not driving a broken stack (fix it, or down + up)"; } + set -a; source "$STATE_DIR/drive.env"; set +a + export E2E_TELEMETRY_DISABLED=1 + # The testing agent's own model key, separate from the app's provider mode: + # a fake-mode stack still needs a real model to act and judge. + if [ -z "${OPENROUTER_API_KEY:-}" ]; then + local key="${LIVE_OPENROUTER_API_KEY:-}" + if [ -z "$key" ] && [ -f "$LIVE_KEYS_FILE" ]; then + [ "$(stat -c %a "$LIVE_KEYS_FILE")" = 600 ] || fail "$LIVE_KEYS_FILE must be mode 600" + key="$(sed -n 's/^LIVE_OPENROUTER_API_KEY=//p' "$LIVE_KEYS_FILE" | tail -1)" + fi + [ -n "$key" ] && export OPENROUTER_API_KEY="$key" + fi + if [ -n "${OPENROUTER_API_KEY:-}" ]; then + say "info testing agent: OpenRouter actor ${E2E_AGENT_MODEL:-qwen/qwen3.8-27b:free}, judge ${E2E_JUDGE_MODEL:-meta/muse-spark-1.3-contributor} (replayed steps make no model call)" + else + say "info no OpenRouter key: tests with agent steps will skip" + fi + case "$sub" in + run|explore) + # The pid keeps parallel explorations (a bug bash) in separate directories. + local run_id; run_id="$(date -u +%Y%m%dT%H%M%SZ)-e2e-$sub-$$" + local out=".verify/artifacts/$run_id" + local status=0 + # --last-failed reads <output>/report.json, so seed the new directory with + # the latest run's report (timestamped names sort oldest first). + if [ "$sub" = run ] && [[ " $* " == *" --last-failed "* ]]; then + local reports=(.verify/artifacts/*-e2e-run*/report.json) + [ -f "${reports[-1]}" ] || fail "--last-failed needs a previous \`verify.sh e2e run\`; none left a report in .verify/artifacts/" + mkdir -p "$out" && cp "${reports[-1]}" "$out/report.json" + say "info retrying the failures recorded in ${reports[-1]%/report.json}" + fi + npx e2e "$sub" --output "$out" "$@" || status=$? + say "EVIDENCE $REPO/$out (exit $status)" + return "$status" + ;; + *) npx e2e "$sub" "$@" ;; + esac +} + cmd_down() { if [ ! -f "$STATE" ]; then say "nothing recorded in $STATE; nothing to stop"; return 0; fi source "$STATE" @@ -305,6 +353,7 @@ case "${1:-}" in doctor) cmd_doctor ;; user) shift; cmd_user "$@" ;; drive) shift; cmd_drive "$@" ;; + e2e) shift; cmd_e2e "$@" ;; down) cmd_down ;; *) sed -n '2,/^set -euo/p' "$0" | sed '$d'; exit 2 ;; esac diff --git a/.github/workflows/agentic-e2e.yml b/.github/workflows/agentic-e2e.yml new file mode 100644 index 000000000..6dec72d43 --- /dev/null +++ b/.github/workflows/agentic-e2e.yml @@ -0,0 +1,125 @@ +name: Agentic E2E + +# The e2e suite in agentic/ against the local stack in fake mode: an AI agent +# drives goal-level flows and judges screens. Recorded flows replay from +# agentic/replay without model calls; judgments always call the model. +# On demand it runs one exploration (a persona looking for bugs) instead. +# Not a required check. See docs/testing.md#agentic-browser-tests. +on: + pull_request: + paths: + - "agentic/**" + - "e2e.config.ts" + - "e2e/full-stack/stack-core.ts" + - ".claude/skills/verify-debatelab/bin/verify.sh" + - ".github/workflows/agentic-e2e.yml" + schedule: + - cron: "47 3 * * *" + workflow_dispatch: + inputs: + explore: + description: "Exploration goal. Empty runs the test suite instead." + required: false + default: "" + persona: + description: "Agent from e2e.config.ts that explores" + type: choice + default: first-timer + options: [default, first-timer, coach, edge-input, hungarian] + target: + description: "Viewport to explore on" + type: choice + default: desktop + options: [desktop, mobile] + +permissions: + contents: read + +concurrency: + group: agentic-e2e-${{ github.ref }} + cancel-in-progress: true + +jobs: + agentic-e2e: + runs-on: ubuntu-latest + timeout-minutes: 45 + steps: + - name: Checkout code + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + + - name: Set up Node + uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 + with: + node-version: 22 + + - name: Set up Supabase CLI + uses: supabase/setup-cli@46f7f98c7f948ad727d22c1e67fab04c223a0520 # v3.0.0 + with: + version: 2.116.0 + + - name: Install dependencies + run: npm ci + + - name: Install Chromium + run: npx playwright install --with-deps chromium + + # Without the key every agent test skips, which would read as a pass. + # Same accepted risk as live-e2e.yml: pull-request code runs with it. + - name: Require the testing agent's model key + env: + LIVE_OPENROUTER_API_KEY: ${{ secrets.LIVE_OPENROUTER_API_KEY }} + run: test -n "$LIVE_OPENROUTER_API_KEY" || { echo "LIVE_OPENROUTER_API_KEY is not set" >&2; exit 1; } + + - name: Start the local stack (fake model provider) + env: + SUPABASE_INTERNAL_IMAGE_REGISTRY: public.ecr.aws + run: .claude/skills/verify-debatelab/bin/verify.sh up + + - name: Agentic suite + if: inputs.explore == '' + env: + LIVE_OPENROUTER_API_KEY: ${{ secrets.LIVE_OPENROUTER_API_KEY }} + E2E_AGENT_MODEL: ${{ vars.E2E_AGENT_MODEL }} + E2E_JUDGE_MODEL: ${{ vars.E2E_JUDGE_MODEL }} + # Bug-bash repro tests fail until their bug is fixed; they run on demand. + run: .claude/skills/verify-debatelab/bin/verify.sh e2e run --exclude-tag repro --reporter list,markdown,junit + + - name: Exploration + if: inputs.explore != '' + env: + LIVE_OPENROUTER_API_KEY: ${{ secrets.LIVE_OPENROUTER_API_KEY }} + E2E_AGENT_MODEL: ${{ vars.E2E_AGENT_MODEL }} + E2E_JUDGE_MODEL: ${{ vars.E2E_JUDGE_MODEL }} + GOAL: ${{ inputs.explore }} + PERSONA: ${{ inputs.persona }} + TARGET: ${{ inputs.target }} + run: >- + .claude/skills/verify-debatelab/bin/verify.sh e2e explore "$GOAL" + --session student --agent "$PERSONA" --target "$TARGET" + --trace=on --video=on --reporter list,markdown + + - name: Publish the report + if: always() + run: | + for summary in .verify/artifacts/*-e2e-*/summary.md; do + [ -f "$summary" ] && cat "$summary" >>"$GITHUB_STEP_SUMMARY" + done + true + + - name: Stop the local stack + if: always() + run: .claude/skills/verify-debatelab/bin/verify.sh down + + # Local test data only: fake model output, throwaway demo users. + - name: Upload evidence + if: always() + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 + with: + name: agentic-e2e-${{ github.run_id }} + path: | + .verify/artifacts/ + .verify/logs/ + if-no-files-found: ignore + retention-days: 7 diff --git a/.gitignore b/.gitignore index de5855aba..5e8a49898 100644 --- a/.gitignore +++ b/.gitignore @@ -75,3 +75,6 @@ humanizer.md .vercel .env* .worktrees/ + +# e2e (agentic tests) output when run outside verify.sh; the replay cache in agentic/replay is committed +.e2e/ diff --git a/AGENTS.md b/AGENTS.md index 37e1c1a71..ce9457aef 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -89,6 +89,7 @@ $H up # fake model provider, free $H up --live # real AI: OpenRouter pinned to meta/muse-spark-1.3-contributor, real Exa $H doctor # proves the stack is local, ours, and (live) pinned $H drive all # every recipe; or recipes/<name>, e2e/<path>; add --project desktop-chromium +$H e2e run # agentic tests in agentic/ (TesterArmy e2e); `e2e explore "<goal>" --session student --agent <persona>` hunts for bugs $H down # always, including after failures ``` @@ -97,6 +98,7 @@ $H down # always, including after failures - The stack holds the same lock as `./.superset/db-gate.sh`, so only one worktree can run it at a time. - Most tools work inside a Current Debate. Recipes start one with `startDebate` or answer the motion dialog with `chooseForMotion`; see the skill's Drive section. - Neither mode proves Stripe Checkout, real Stripe cancellation, ElevenLabs voice audio, Google sign-in or export, staging, or production. +- Agentic tests (`agentic/*.e2e.ts`) are an AI agent driving goals and judging screens, with exact assertions on the result. They catch what selectors cannot: stray untranslated text, clipped layouts, off-topic model output in live mode. They also run explorations that hunt for bugs as a persona. The agent calls its own OpenRouter models (free Qwen acts, the Muse Spark contributor tier judges), even on a fake stack. A flow change that breaks an agentic test updates it in the same PR; commit new or re-recorded `agentic/replay/` entries with it. See [docs/testing.md](docs/testing.md#agentic-browser-tests). ### Keep the skill working diff --git a/agentic/case-generation.e2e.ts b/agentic/case-generation.e2e.ts new file mode 100644 index 000000000..febeb4965 --- /dev/null +++ b/agentic/case-generation.e2e.ts @@ -0,0 +1,26 @@ +import { admin, agentic, AI_TIMEOUT, debateControl, expect, LIVE, test } from "./support/fixtures"; + +const MOTION = "This House would replace school exams with continuous assessment"; + +test("a student starts a debate and generates a case for it", agentic, async ({ app, agent, browser, screen, student }) => { + await app.open("/app"); + await agent.act("Start a new debate on the motion {motion}. Then open Generate and generate a debate case for that motion.", { + params: { motion: MOTION }, + }); + + await expect(debateControl(browser)).toHaveAttribute("aria-label", `Current debate: ${MOTION}`); + await expect(screen.getByRole("button", "Regenerate Arguments")).toBeVisible({ timeout: AI_TIMEOUT }); + + // The client files the case in its debate after it renders. + const userId = await student.userId(); + await expect.poll(async () => { + const { data } = await admin.from("cases").select("debates(motion)").eq("user_id", userId); + return data; + }, { timeout: 15_000 }).toEqual([{ debates: { motion: MOTION } }]); + + await agent.assert("The page shows a generated debate case: a thesis and several contentions, each with a claim and supporting reasoning."); + if (LIVE) { + // Recipes never assert on model wording; a judge can check that the case is on topic. + await agent.assert(`The generated case argues about the motion "${MOTION}", not about an unrelated topic.`); + } +}); diff --git a/agentic/first-debate.e2e.ts b/agentic/first-debate.e2e.ts new file mode 100644 index 000000000..d85f681c0 --- /dev/null +++ b/agentic/first-debate.e2e.ts @@ -0,0 +1,16 @@ +import { agentic, debateControl, debatesOf, expect, firstRunTest as test } from "./support/fixtures"; + +const MOTION = "This House would make debate compulsory in secondary schools"; + +// The onboarding copy and step order change often. The goal stays the same: +// a new student finishes the tour with their first debate started. +test("a new student starts their first debate from the onboarding tour", agentic, async ({ agent, browser, student }) => { + await agent.act("Follow the onboarding tour. At its first step, start a debate on the motion {motion}, then go through the remaining steps until the tour is finished.", { + params: { motion: MOTION }, + }); + + await expect(debateControl(browser)).toHaveAttribute("aria-label", `Current debate: ${MOTION}`); + await expect(browser.locator(".react-joyride__tooltip")).toHaveCount(0); + expect((await debatesOf(student)).map((debate) => debate.motion)).toEqual([MOTION]); + await agent.assert("The onboarding tour is closed and the app shows the new motion as the current debate."); +}); diff --git a/agentic/hungarian.e2e.ts b/agentic/hungarian.e2e.ts new file mode 100644 index 000000000..08fe81e75 --- /dev/null +++ b/agentic/hungarian.e2e.ts @@ -0,0 +1,21 @@ +import { agentic, expect, hungarianTest as test } from "./support/fixtures"; + +// e2e/hungarian-parity.spec.ts checks lang, headings and chosen labels, and by +// design cannot reject stray English. A judge reading each screen can. +const PAGES = ["/app", "/app/generate", "/app/brainstorm", "/app/sparring", "/app/library", "/app/history", "/app/settings"]; + +for (const path of PAGES) { + test(`${path} reads in Hungarian and fits the screen`, agentic, async ({ app, agent, browser }) => { + await app.open(path); + await expect(browser.locator("html")).toHaveAttribute("lang", "hu"); + await expect(browser.locator("main h1, main h2").first()).toBeVisible(); + await agent.assert( + "Every piece of interface text on this screen (navigation, headings, buttons, field labels, placeholders, hints and empty states) is in Hungarian. Product names such as DebateLab, AI model names (Claude Sonnet 5) and plan names (Pro), debate format codes, language names offered as choices (English, Español, Magyar), and user content do not count. An English loanword inflected with a Hungarian ending, such as \"coachingot\", is Hungarian.", + { vision: true }, + ); + await agent.assert( + "No label, heading, button, navigation or message text on this screen is cut off, overlapping other text, or running past the edge of its button or the screen. A placeholder or value that scrolls inside a text field is normal and does not count.", + { vision: "only" }, + ); + }); +} diff --git a/agentic/replay/185f3f3b58f5231404f8aece6743cb319e69c2bd78482e64b82cab4207b3f413.json b/agentic/replay/185f3f3b58f5231404f8aece6743cb319e69c2bd78482e64b82cab4207b3f413.json new file mode 100644 index 000000000..a015a2a00 --- /dev/null +++ b/agentic/replay/185f3f3b58f5231404f8aece6743cb319e69c2bd78482e64b82cab4207b3f413.json @@ -0,0 +1,97 @@ +{ + "schemaVersion": "trace-1", + "createdAt": "2026-10-02T16:03:12.575Z", + "payload": { + "actions": [ + { + "name": "tap", + "summary": "tap button \"Start debate\" (3 of 3)", + "target": { + "role": "button", + "name": "Start debate", + "position": { + "index": 2, + "of": 3 + } + } + }, + { + "name": "type", + "summary": "type \"This House would make debate compulsory…\" into textbox \"Motion\" in \"Start new debate\"", + "target": { + "role": "textbox", + "name": "Motion", + "placeholder": "This House would…", + "within": "Start new debate" + }, + "value": "This House would make debate compulsory in secondary schools" + }, + { + "name": "tap", + "summary": "tap button \"Start debate\" in \"Start new debate\"", + "target": { + "role": "button", + "name": "Start debate", + "within": "Start new debate" + } + }, + { + "name": "tap", + "summary": "tap button \"Next (Step 2 of 4)\"", + "target": { + "role": "button", + "name": "Next (Step 2 of 4)" + } + }, + { + "name": "tap", + "summary": "tap button \"Next (Step 3 of 4)\"", + "target": { + "role": "button", + "name": "Next (Step 3 of 4)" + } + }, + { + "name": "tap", + "summary": "tap button \"Get Started!\"", + "target": { + "role": "button", + "name": "Get Started!" + } + } + ], + "executor": { + "name": "e2e-default-agent", + "version": "2" + }, + "recordedFor": { + "testId": "agentic/first-debate.e2e.ts::a%20new%20student%20starts%20their%20first%20debate%20from%20the%20onboarding%20tour", + "targetId": "desktop", + "instructionDigest": "027693f6c76e21fe239c9a427a38bfdbab2bd0577d771749dc8a73dd02fb46f8" + }, + "summary": "Followed the onboarding tour: at step 1 of 4 started a debate with motion \"This House would make debate compulsory in secondary schools\", then advanced through the Library, Generate steps and the final \"You're Ready!\" step via Get Started!. The tour is finished; home shows the Current Debate with t…", + "startPath": "/app", + "endPath": "/app", + "endAnchors": [ + { + "text": "Current debate" + }, + { + "text": "This House would make debate compulsory in secondary schools" + }, + { + "text": "Last activity less than a minute ago · 0 sources · 0 outputs" + }, + { + "role": "link", + "name": "Open debate" + }, + { + "role": "button", + "name": "Current debate: This House would make debate compulsory in secondary schools", + "testId": "current-debate-desktop" + } + ], + "endWaitMs": 14375 + } +} diff --git a/agentic/replay/b4819cfeff66096f89bd15a116b3d35d58a8b3bbea01d479ba86a4b32575157f.json b/agentic/replay/b4819cfeff66096f89bd15a116b3d35d58a8b3bbea01d479ba86a4b32575157f.json new file mode 100644 index 000000000..07f4dbccf --- /dev/null +++ b/agentic/replay/b4819cfeff66096f89bd15a116b3d35d58a8b3bbea01d479ba86a4b32575157f.json @@ -0,0 +1,99 @@ +{ + "schemaVersion": "trace-1", + "createdAt": "2026-10-02T16:08:26.536Z", + "payload": { + "actions": [ + { + "name": "tap", + "summary": "tap button \"Start debate\" (2 of 2)", + "target": { + "role": "button", + "name": "Start debate", + "position": { + "index": 1, + "of": 2 + } + } + }, + { + "name": "type", + "summary": "type \"This House would replace school exams w…\" into textbox \"Motion\" in \"Start new debate\"", + "target": { + "role": "textbox", + "name": "Motion", + "placeholder": "This House would…", + "within": "Start new debate" + }, + "value": "This House would replace school exams with continuous assessment" + }, + { + "name": "tap", + "summary": "tap button \"Start debate\" in \"Start new debate\"", + "target": { + "role": "button", + "name": "Start debate", + "within": "Start new debate" + } + }, + { + "name": "tap", + "summary": "tap link \"Generate Build a complete debate case.\"", + "target": { + "role": "link", + "name": "Generate Build a complete debate case." + } + }, + { + "name": "tap", + "summary": "tap button \"Generate Debate Case\"", + "target": { + "role": "button", + "name": "Generate Debate Case" + } + } + ], + "executor": { + "name": "e2e-default-agent", + "version": "2" + }, + "recordedFor": { + "testId": "agentic/case-generation.e2e.ts::a%20student%20starts%20a%20debate%20and%20generates%20a%20case%20for%20it", + "targetId": "mobile", + "instructionDigest": "86311d0b0c95064361ba72d803d642b0c0266b78be93afeba829dde446989804" + }, + "summary": "Started a new debate on the motion \"This House would replace school exams with continuous assessment\" (now shown as Current Debate in header and home), opened Generate (working-in strip shows the motion) and generated the case. Result confirmed: \"Generate complete\" with Case Structure, Thesis/3 con…", + "startPath": "/app", + "endPath": "/app/generate", + "endAnchors": [ + { + "text": "Current debate" + }, + { + "text": "This House would replace school exams with continuous assessment" + }, + { + "role": "heading", + "name": "Generate" + }, + { + "text": "Working in:" + }, + { + "text": "In favor" + }, + { + "role": "button", + "name": "Change" + }, + { + "role": "button", + "name": "Use once" + }, + { + "role": "heading", + "name": "Generate Debate Case" + } + ], + "endWaitMs": 16389 + } +} diff --git a/agentic/replay/c194c806fb663c3248e8b760c4857321ddf1b33facc8effb258d09fea6fcf918.json b/agentic/replay/c194c806fb663c3248e8b760c4857321ddf1b33facc8effb258d09fea6fcf918.json new file mode 100644 index 000000000..c50666e29 --- /dev/null +++ b/agentic/replay/c194c806fb663c3248e8b760c4857321ddf1b33facc8effb258d09fea6fcf918.json @@ -0,0 +1,97 @@ +{ + "schemaVersion": "trace-1", + "createdAt": "2026-10-02T16:10:12.968Z", + "payload": { + "actions": [ + { + "name": "tap", + "summary": "tap button \"Start debate\" (3 of 3)", + "target": { + "role": "button", + "name": "Start debate", + "position": { + "index": 2, + "of": 3 + } + } + }, + { + "name": "type", + "summary": "type \"This House would make debate compulsory…\" into textbox \"Motion\" in \"Start new debate\"", + "target": { + "role": "textbox", + "name": "Motion", + "placeholder": "This House would…", + "within": "Start new debate" + }, + "value": "This House would make debate compulsory in secondary schools" + }, + { + "name": "tap", + "summary": "tap button \"Start debate\" in \"Start new debate\"", + "target": { + "role": "button", + "name": "Start debate", + "within": "Start new debate" + } + }, + { + "name": "tap", + "summary": "tap button \"Next (Step 2 of 4)\"", + "target": { + "role": "button", + "name": "Next (Step 2 of 4)" + } + }, + { + "name": "tap", + "summary": "tap button \"Next (Step 3 of 4)\"", + "target": { + "role": "button", + "name": "Next (Step 3 of 4)" + } + }, + { + "name": "tap", + "summary": "tap button \"Get Started!\"", + "target": { + "role": "button", + "name": "Get Started!" + } + } + ], + "executor": { + "name": "e2e-default-agent", + "version": "2" + }, + "recordedFor": { + "testId": "agentic/first-debate.e2e.ts::a%20new%20student%20starts%20their%20first%20debate%20from%20the%20onboarding%20tour", + "targetId": "mobile", + "instructionDigest": "027693f6c76e21fe239c9a427a38bfdbab2bd0577d771749dc8a73dd02fb46f8" + }, + "summary": "Followed the onboarding tour: started the debate on motion \"This House would make debate compulsory in secondary schools\" at step 1 (dialog \"Start new debate\"), then clicked Next through steps 2 (Library), 3 (Generate), and 4 (\"You're Ready!\" → \"Get Started!\"). Tour dialog closed; header now shows …", + "startPath": "/app", + "endPath": "/app", + "endAnchors": [ + { + "text": "Current debate" + }, + { + "text": "This House would make debate compulsory in secondary schools" + }, + { + "text": "Last activity less than a minute ago · 0 sources · 0 outputs" + }, + { + "role": "link", + "name": "Open debate" + }, + { + "role": "button", + "name": "Current debate: This House would make debate compulsory in secondary schools", + "testId": "current-debate-mobile" + } + ], + "endWaitMs": 13620 + } +} diff --git a/agentic/replay/c7adcf13daafc4d7b7fdb639f07aee1d7a7d8028597c925bd41fdeff99dc3848.json b/agentic/replay/c7adcf13daafc4d7b7fdb639f07aee1d7a7d8028597c925bd41fdeff99dc3848.json new file mode 100644 index 000000000..d42c1ab99 --- /dev/null +++ b/agentic/replay/c7adcf13daafc4d7b7fdb639f07aee1d7a7d8028597c925bd41fdeff99dc3848.json @@ -0,0 +1,99 @@ +{ + "schemaVersion": "trace-1", + "createdAt": "2026-10-02T16:08:35.794Z", + "payload": { + "actions": [ + { + "name": "tap", + "summary": "tap button \"Start debate\" (2 of 2)", + "target": { + "role": "button", + "name": "Start debate", + "position": { + "index": 1, + "of": 2 + } + } + }, + { + "name": "type", + "summary": "type \"This House would replace school exams w…\" into textbox \"Motion\" in \"Start new debate\"", + "target": { + "role": "textbox", + "name": "Motion", + "placeholder": "This House would…", + "within": "Start new debate" + }, + "value": "This House would replace school exams with continuous assessment" + }, + { + "name": "tap", + "summary": "tap button \"Start debate\" in \"Start new debate\"", + "target": { + "role": "button", + "name": "Start debate", + "within": "Start new debate" + } + }, + { + "name": "tap", + "summary": "tap link \"Generate Build a complete debate case.\"", + "target": { + "role": "link", + "name": "Generate Build a complete debate case." + } + }, + { + "name": "tap", + "summary": "tap button \"Generate Debate Case\"", + "target": { + "role": "button", + "name": "Generate Debate Case" + } + } + ], + "executor": { + "name": "e2e-default-agent", + "version": "2" + }, + "recordedFor": { + "testId": "agentic/case-generation.e2e.ts::a%20student%20starts%20a%20debate%20and%20generates%20a%20case%20for%20it", + "targetId": "desktop", + "instructionDigest": "86311d0b0c95064361ba72d803d642b0c0266b78be93afeba829dde446989804" + }, + "summary": "Started a new debate with motion \"This House would replace school exams with continuous assessment\" (now shown as Current Debate in the header), then opened Generate and generated a case: thesis, framework, 3 structured arguments, predicted opponent case, and prepared counters are displayed (FAKE-P…", + "startPath": "/app", + "endPath": "/app/generate", + "endAnchors": [ + { + "text": "Current debate" + }, + { + "role": "heading", + "name": "Generate" + }, + { + "text": "Working in:" + }, + { + "text": "This House would replace school exams with continuous assessment" + }, + { + "text": "In favor" + }, + { + "role": "button", + "name": "Change" + }, + { + "role": "button", + "name": "Use once" + }, + { + "role": "heading", + "name": "Generate Debate Case" + } + ], + "endWaitMs": 18577 + } +} diff --git a/agentic/sessions.e2e.ts b/agentic/sessions.e2e.ts new file mode 100644 index 000000000..7bace91b4 --- /dev/null +++ b/agentic/sessions.e2e.ts @@ -0,0 +1,24 @@ +import { test } from "@e2e-dev/web"; +import { expect } from "e2e"; +import { createDemoUser, stackReady } from "../e2e/full-stack/stack-core"; + +// The signed-in start for `verify.sh e2e explore --session student`. Explore +// runs only this setup. The user lives until `verify.sh down` wipes the stack. +test.setup("a student signs in for exploration", { sessions: ["student"] }, async ({ app, screen, browser, session }) => { + if (!stackReady) throw new Error("No verification stack environment; run through verify.sh e2e"); + const user = await createDemoUser("explore"); + await app.open("/login"); + await browser.evaluate(() => { + localStorage.setItem("debate-trainer-onboarding", JSON.stringify({ state: { hasCompletedOnboarding: true, isTourRunning: false }, version: 0 })); + return null; + }); + await browser.reload(); + // A plain fill, not a Secret: the code is a throwaway local one, and a Secret + // fill would withhold the explorer's screenshots for the whole run. + await screen.getByRole("tab", "Demo code").tap(); + await screen.getByRole("textbox", "Demo code").fill(user.code); + await screen.getByRole("button", "Sign in with demo code").tap(); + await browser.waitForURL(/\/app$/, { timeout: 20_000 }); + await expect(browser.locator('[data-testid="home-current-debate"]')).toBeVisible(); + await session.save("student"); +}); diff --git a/agentic/support/fixtures.ts b/agentic/support/fixtures.ts new file mode 100644 index 000000000..18f7a52db --- /dev/null +++ b/agentic/support/fixtures.ts @@ -0,0 +1,110 @@ +import { test as base, type Browser } from "@e2e-dev/web"; +import { expect, type Screen } from "e2e"; +import { admin, createDemoUser, deleteDemoUser, stackReady, type DemoUser } from "../../e2e/full-stack/stack-core"; + +export { admin, fake, MARK } from "../../e2e/full-stack/stack-core"; +export { expect }; + +export const LIVE = process.env.VERIFY_MODE === "live"; +/** Budget for anything behind a model call: fake answers are instant, live ones are not. */ +export const AI_TIMEOUT = LIVE ? 240_000 : 30_000; + +const hasModel = Boolean(process.env.OPENROUTER_API_KEY); + +/** + * Options for a test with agent steps. `agent.assert`, `waitFor` and `extract` + * always call the model, so without a key the test skips at collection rather + * than spending a minute on provider retries. The stack check keeps a plain + * `npx e2e run` from minting users nowhere. + */ +export const agentic = { + skip: !stackReady + ? "no verification stack environment; run through verify.sh e2e" + : !hasModel + ? "OPENROUTER_API_KEY is not set; run through verify.sh e2e" + : false, +} as const; + +const labels = { + en: { tab: "Demo code", field: "Demo code", submit: "Sign in with demo code" }, + hu: { tab: "Demókód", field: "Demókód", submit: "Bejelentkezés demókóddal" }, +} as const; + +type Language = keyof typeof labels; +type StudentOptions = { tour?: boolean; language?: Language }; +export type Student = DemoUser & { codeId: string }; + +/** Persisted client state the app reads at boot. e2e has no init scripts, so the caller reloads. */ +async function seedClientState(browser: Browser, { tour = false, language = "en" }: StudentOptions) { + await browser.evaluate((seed: { tourPending: boolean; language: string }) => { + localStorage.setItem("debate-trainer-onboarding", JSON.stringify({ + state: { hasCompletedOnboarding: !seed.tourPending, isTourRunning: false }, + version: 0, + })); + localStorage.setItem("debate-trainer-settings", JSON.stringify({ + state: { language: seed.language, outputLanguage: seed.language, theme: "system" }, + version: 0, + })); + return null; + }, { tourPending: tour, language }); +} + +/** Signs in through the real Login page and demo-code-login Edge Function. */ +async function signIn(screen: Screen, browser: Browser, code: string, language: Language) { + const copy = labels[language]; + await screen.getByRole("tab", copy.tab).tap(); + await screen.getByRole("textbox", copy.field).fill(code); + await screen.getByRole("button", copy.submit).tap(); + await browser.waitForURL(/\/app$/, { timeout: 20_000 }); +} + +/** + * A `test` whose `student` fixture is a fresh demo-code user (Pro, daily limit + * 999, no Current Debate), signed in and deleted afterwards. Any request to a + * hosted Supabase project is aborted and fails the test, and so does a page + * wider than its viewport. + */ +function studentTest(options: StudentOptions = {}) { + const language = options.language ?? "en"; + return base.extend<{ student: Student }>({ + student: async ({ app, screen, browser }, provide) => { + const remote: string[] = []; + await browser.route(/^https:\/\/[^/]+\.supabase\.co\//, async (route) => { + remote.push(route.request.url); + await route.abort(); + }); + const user = await createDemoUser(`agentic-${language}`); + try { + await app.open("/login"); + await seedClientState(browser, options); + await browser.reload(); + await signIn(screen, browser, user.code, language); + await provide(user); + const overflow = await browser.evaluate(() => document.documentElement.scrollWidth - document.documentElement.clientWidth); + expect(overflow, "horizontal page overflow in px").toBeLessThanOrEqual(1); + expect(remote, "requests to a hosted Supabase project").toEqual([]); + } finally { + await deleteDemoUser(user); + } + }, + }); +} + +/** A signed-in student who has already seen the onboarding tour. */ +export const test = studentTest(); +/** A brand-new account: the onboarding tour starts on /app. */ +export const firstRunTest = studentTest({ tour: true }); +/** A signed-in student using the Hungarian interface. */ +export const hungarianTest = studentTest({ language: "hu" }); + +/** The visible header Current Debate control on either viewport. */ +export function debateControl(browser: Browser) { + return browser.locator('[data-testid="current-debate-desktop"]:visible, [data-testid="current-debate-mobile"]:visible'); +} + +/** The `debates` rows a user owns, read through the service role. */ +export async function debatesOf(student: Student) { + const { data, error } = await admin.from("debates").select("id, motion").eq("user_id", await student.userId()); + if (error) throw error; + return data; +} diff --git a/docs/README.md b/docs/README.md index ecb0de9c4..c0fcad39c 100644 --- a/docs/README.md +++ b/docs/README.md @@ -23,7 +23,7 @@ Start with the smallest document that answers the task instead of loading every ## Test and verify -- [Testing guide](testing.md), current. Open before running coverage, Edge Function, database, browser, WebKit, live-provider, canary, or protected staging gates. +- [Testing guide](testing.md), current. Open before running coverage, Edge Function, database, browser, agentic, WebKit, live-provider, canary, or protected staging gates. - [`package.json`](../package.json) owns npm script definitions. - [`vitest.config.ts`](../vitest.config.ts) and [`playwright.config.ts`](../playwright.config.ts) own test selection and browser behavior. diff --git a/docs/development.md b/docs/development.md index 88a9c81a0..5e47ca50a 100644 --- a/docs/development.md +++ b/docs/development.md @@ -150,6 +150,8 @@ Failure artifacts go to ignored `test-results/` and `playwright-report/`. Protec Read [testing.md](testing.md) before running WebKit or staging. +Agentic tests (`agentic/*.e2e.ts`, the e2e runner) drive the verification stack, never a dev server you started. Run them with `.claude/skills/verify-debatelab/bin/verify.sh e2e run` after `verify.sh up`. See [testing.md](testing.md#agentic-browser-tests). + ## iOS workflow ```bash diff --git a/docs/testing.md b/docs/testing.md index 8a99be1fb..8a0627f8d 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -29,7 +29,7 @@ npm run lint npm run typecheck ``` -Lint checks repository ESLint rules. Type checking uses `tsconfig.app.json` and emits no files. Neither command replaces runtime tests. +Lint checks repository ESLint rules. Type checking uses `tsconfig.app.json` for the client and `tsconfig.agentic.json` for `e2e.config.ts` and `agentic/` (the e2e runner itself does not type-check), and emits no files. Neither command replaces runtime tests. ## Unit and component tests @@ -230,6 +230,41 @@ LIVE_PROVIDERS=1 LIVE_MODEL=as-requested LIVE_OPENROUTER_API_KEY=... LIVE_EXA_AP `.github/workflows/model-matrix.yml` runs this on Mondays at 04:29 UTC, on demand, and on pull requests that change `supabase/functions/_shared/`, `supabase/functions/ai/`, or `src/lib/ai/`. It is not a required check, because it runs only on those paths. One run costs about $1 in model and search usage. +## Agentic browser tests + +```bash +H=.claude/skills/verify-debatelab/bin/verify.sh +$H up +$H e2e run # the suite on both viewports +$H e2e run agentic/case-generation.e2e.ts --target desktop # one file, one viewport +$H e2e explore "Practise a Sparring round and report anything broken" --session student --agent first-timer +$H down +``` + +[`agentic/`](../agentic/) holds tests for [e2e](https://e2e.tester.army/docs), TesterArmy's agentic test runner (`e2e@0.15.2`, configured in [`e2e.config.ts`](../e2e.config.ts)). In each test, an AI agent works toward a goal in plain language (`agent.act`) and judges what is on screen (`agent.assert`). Locators and database reads then pin the result exactly. The suite covers what the Playwright layers cannot: + +- Goal-level journeys that survive copy and layout changes: a new student starts their first debate from the onboarding tour, and a student starts a debate and generates a case. +- Judgments a selector cannot make: every main page reads in Hungarian (`e2e/hungarian-parity.spec.ts` deliberately does not reject stray English), and no text is cut off or overflowing on either viewport. +- In live mode (`up --live`), whether the generated case argues the motion at all. Recipes never assert model wording. +- Exploration. `e2e explore` sends a persona through the app with a goal and reports findings with expected and observed behavior, reproduction steps, and screenshots. The personas are the named agents in `e2e.config.ts`: `first-timer`, `coach`, `edge-input`, and `hungarian`. + +How it runs: + +- Always through `verify.sh e2e <run|explore|list|cache>`. The wrapper runs `doctor`, loads the stack environment, turns e2e telemetry off, and takes the testing agent's key from `OPENROUTER_API_KEY`, `LIVE_OPENROUTER_API_KEY`, or `~/.config/debatelab/live-verify.env`. It writes the report (`summary.md`, `report.json`), screenshots, traces, and failure videos to `.verify/artifacts/<run>-e2e-<command>-<pid>/`. The config refuses a non-loopback URL and never starts a dev server, because a plain `npm run dev` reaches production. +- Each test signs in a fresh demo-code user through the Login page and deletes it afterwards. A request to a hosted Supabase project, or a page wider than its viewport, fails the test. +- There are two targets, `desktop` (1280 by 800) and `mobile` (393 by 851). e2e 0.15 sets only the viewport size: it does not emulate touch or a phone user agent, so the Playwright Pixel 5 project still owns mobile behavior. +- The testing agent has two models on OpenRouter, separate from the app's provider mode, and a fake-mode stack still needs both. The actor (`agent.act` and explorations) is `qwen/qwen3.8-27b:free`. It is free, and recorded flows replay without it, but it is slow and its upstream sometimes answers `temporarily rate-limited upstream`. Treat that as infrastructure and rerun with `--last-failed`. The judge (`agent.assert`, `waitFor`, `extract`) is `meta/muse-spark-1.3-contributor` ($0.10 and $0.20 per million input and output tokens), the same tier as the app's live pin. It answers every judgment on every run. A full suite run costs under two cents, and a 4-step exploration well under one. The judge cannot act, because Meta's endpoint accepts only `tool_choice: "auto"`. `E2E_AGENT_MODEL` and `E2E_JUDGE_MODEL` override the two; `E2E_AGENT_MODEL=openai/gpt-6-luna` gives fast, reliable explorations for cents. Both default models may keep prompts, so use made-up debate content only. The agent's context explains that fake AI answers are canned free-transport text. +- Replay. When a later assertion passes, an `agent.act` flow is recorded in [`agentic/replay/`](../agentic/replay/). Later runs replay it with no model call until the app changes, at which point the agent takes over and records again. Commit new and changed entries with the test that made them. Local runs read and write the cache; CI only reads it. `agent.assert`, `waitFor`, and `extract` always call the model. +- Without a model key or the stack environment, agent tests skip, so a bare `npx e2e run` proves nothing. + +To write a test, import `test`, `expect`, and the fixtures from [`agentic/support/fixtures.ts`](../agentic/support/fixtures.ts) and pass the `agentic` options. Follow every `agent.act` with a locator assertion, because only verified flows are recorded. Prefer `agent.assert` for what a selector cannot express, and an exact `expect` for everything else. The [`e2e` skill](../.claude/skills/e2e/SKILL.md), `npx e2e guide <topic>`, and `node_modules/e2e/docs` cover the API for the installed version. + +A bug bash runs several explorations in parallel, one charter per area and persona, then proves each finding with a repro test. Ask an agent to "bug bash <area>"; the `e2e` skill's `bug-bash` topic has the procedure. A finding counts only when its repro test fails for the reported reason. Tag repro tests `{ tags: ["repro"] }` so the scheduled suite skips them until the fix lands, then remove the tag so they become regression tests. The client runs on a Vite dev server, so a route that is still compiling can look like a dead link to an explorer. Reproduce before you believe it. + +[`.github/workflows/agentic-e2e.yml`](../.github/workflows/agentic-e2e.yml) runs the suite nightly at 03:47 UTC, on pull requests that change it, and on demand. On demand with a goal, it runs one exploration with the chosen persona and viewport instead. It uses the `LIVE_OPENROUTER_API_KEY` secret (the `E2E_AGENT_MODEL` and `E2E_JUDGE_MODEL` repository variables override the models), fails when the key is missing, and writes the report to the job summary. It is not a required check: model judgments can flake, and a red nightly is a finding to triage. + +e2e is pre-1.0 and its minor releases have broken APIs, so it is pinned exactly. Read its changelog before upgrading, and expect the replay cache to re-record after an engine minor bump. + ## Production canary ```bash @@ -321,7 +356,7 @@ Tests are not the proof that a change works; using it is. Two local recipes that [`.github/workflows/staging-e2e.yml`](../.github/workflows/staging-e2e.yml) owns protected staging conditions. -[`.github/workflows/live-e2e.yml`](../.github/workflows/live-e2e.yml) runs the live provider journeys, and [`.github/workflows/prod-canary.yml`](../.github/workflows/prod-canary.yml) runs the production canary. `main` has no branch protection or rulesets (checked 2026-10-01; GitHub Free does not enforce them on private repositories), so no check is required. Path-filtered workflows that skip therefore don't block merges. Read the PR's checks before merging. +[`.github/workflows/agentic-e2e.yml`](../.github/workflows/agentic-e2e.yml) runs the agentic suite and on-demand explorations. [`.github/workflows/live-e2e.yml`](../.github/workflows/live-e2e.yml) runs the live provider journeys, and [`.github/workflows/prod-canary.yml`](../.github/workflows/prod-canary.yml) runs the production canary. `main` has no branch protection or rulesets (checked 2026-10-01; GitHub Free does not enforce them on private repositories), so no check is required. Path-filtered workflows that skip therefore don't block merges. Read the PR's checks before merging. To see why a run failed, use `npm run ci:failures` (current branch's PR), `npm run ci:failures -- <pr>` or `npm run ci:failures -- <run-id>`. It works while the run is still going, as soon as a job fails. It lists failed jobs and steps, prints the Playwright or Vitest summary from each job log, downloads the `*-failure-<run-id>` artifacts into ignored `.ci-failures/`, and prints each failed test's error plus the dialogs and headings from its `error-context.md` page snapshot. A dialog the spec never answered shows up there. Protected staging uploads no artifact, so only its job log is available. @@ -344,3 +379,4 @@ When Actions cannot run at all, the local `npm run verify` and `./.superset/db-g - Staging pass: no authorization to deploy or mutate production. - Live provider pass: no proof for the production models, only for the pinned test model. - Canary pass: production was healthy at that moment; it covers one AI path, not every feature. +- Agentic pass: a model judged the screens it was shown. It is not exhaustive, a judgment can be wrong in either direction, and fake mode proves nothing about model output. An exploration finding is a lead until a repro test fails on it. diff --git a/e2e.config.ts b/e2e.config.ts new file mode 100644 index 000000000..4a5346e34 --- /dev/null +++ b/e2e.config.ts @@ -0,0 +1,95 @@ +import type { E2EConfig } from "e2e"; +import { web } from "@e2e-dev/web"; +import { openrouter } from "@openrouter/ai-sdk-provider"; + +// Agentic browser tests (TesterArmy e2e) against the verify-debatelab stack. +// Run them through `.claude/skills/verify-debatelab/bin/verify.sh e2e`, which +// checks the stack, loads its environment and the OpenRouter key, and keeps +// the evidence. See docs/testing.md#agentic-browser-tests. + +// Only the `verify.sh up` client: a plain dev server talks to production Supabase. +const appUrl = process.env.PLAYWRIGHT_BASE_URL || "http://127.0.0.1:18081"; +if (!/^http:\/\/(127\.0\.0\.1|localhost):\d+\/?$/.test(appUrl)) { + throw new Error(`Agentic tests only drive the local verification stack, not ${appUrl}`); +} + +const live = process.env.VERIFY_MODE === "live"; + +// The testing agent's models, separate from the app's own. Both need tool +// calling and image input. The actor drives agent.act and explorations; its +// recorded flows replay with no model call, so a free, slow model is enough. +// The judge answers every agent.assert, waitFor and extract on every run, so it +// must be reliable: the Muse Spark contributor tier is the cheapest that is, +// and like the app's live pin it may keep prompts (made-up content only). +// `||`, not `??`: an unset Actions variable arrives as an empty string. +const model = openrouter(process.env.E2E_AGENT_MODEL || "qwen/qwen3.8-27b:free"); +const judge = openrouter(process.env.E2E_JUDGE_MODEL || "meta/muse-spark-1.3-contributor"); + +// `context` reaches every model call, judges included: what the app calls things. +const context = [ + "DebateLab is a debate training app for students. A motion is the debate topic, such as \"This House would ban homework\".", + "Each student works in one Current Debate at a time. The header control reads \"Start debate\" until one exists, then names the motion. Tool pages show a strip \"Working in: <motion>\".", + "Tools: Generate (a full debate case: thesis, contentions, predicted opponent case, prepared counters), Attack, Brainstorm, Factoids, Ask, Sparring (a practice round against an AI opponent, judged at the end), Crossfire (cross-examination practice), Flowchart, Timer, Library (sources for the Current Debate), and History (saved results).", + "A tool given a motion with no Current Debate asks \"Start a debate with this motion?\". A motion that differs from the Current Debate asks \"This is a different motion\".", + "The interface language can be English, Spanish or Hungarian. Source titles, citations, user content and model names may stay in their own language. Hungarian abbreviates AI as MI (mesterséges intelligencia), so \"MI-modell\" is Hungarian.", + "Long values in selects, chips and narrow fields are shortened with an ellipsis (…) on purpose.", + "This is a local test stack. Google sign-in, Stripe checkout, ElevenLabs voices and Google Docs export are not configured here, so their failures are expected.", + "The client runs on a Vite dev server, so the first visit to a page can take a few seconds to render.", + live + ? "AI answers come from a real model, so their wording varies between runs." + : "AI answers come from a fake provider: every AI result is canned text containing the marker FAKE-PROVIDER, and it argues about free public transport whatever the motion. That mismatch is expected here and is not a defect.", +].join("\n"); + +// `system` is read by the acting agent only, never by the judges. +const rules = [ + "Use the visible labels on screen. Verify a visible confirmation before completing a step.", + "Never delete the account or all data, sign out, or open billing checkout unless the instruction asks for it.", +]; + +// The judge cannot act: Meta's endpoint accepts only tool_choice "auto", and +// the act loop requires a tool call. Explorations act with the actor too. +const agent = (persona?: string) => ({ + model, + judge, + context, + // Generous for slow providers; e2e's default is 30 s per judgment. + judgmentTimeout: 120_000, + system: [...rules, ...(persona ? [persona] : [])].join("\n"), +}); + +export default { + projectId: "debatelab", + tests: "agentic/**/*.e2e.ts", + targets: [ + { + name: "desktop", + engine: web({ viewport: { width: 1280, height: 800 } }), + app: { url: appUrl, identity: "debatelab-verify-stack", environment: "test" }, + }, + { + // A phone-sized viewport only: e2e 0.15 has no touch or user-agent emulation. + name: "mobile", + engine: web({ viewport: { width: 393, height: 851 } }), + app: { url: appUrl, identity: "debatelab-verify-stack", environment: "test" }, + }, + ], + // AI steps wait on Edge Functions, and a free model can take minutes per judgment. + timeout: 600_000, + // More parallel act steps only trip the free actor's upstream rate limit. + workers: process.env.CI ? 1 : 2, + // The dev server compiles lazy routes on first visit. + assertionTimeout: 15_000, + video: "retain-on-failure", + reporters: ["list", "markdown"], + // Recorded agent.act() flows replay with no model call. Committed so every + // checkout and CI share them; CI reads them and never writes. + cache: { dir: "agentic/replay" }, + agents: { + default: agent(), + // Bug-bash personas for `verify.sh e2e explore --agent <name>`. + "first-timer": agent("You are a 15-year-old who has never used DebateLab and has a debate tournament next week. Go where a newcomer would go, read what the app tells you, and report anything confusing, broken, or misleading."), + coach: agent("You are a debate coach checking DebateLab before recommending it to a team. Check that numbers, labels, counts, dates and copy agree across pages, and that saved work shows up where the app says it will."), + "edge-input": agent("You probe input handling: very long motions, emoji, right-to-left text, leading and trailing spaces, empty fields, double submits, and leaving a page mid-request. Report crashes, lost work, unclear errors, and states you cannot recover from."), + hungarian: agent("You are a Hungarian student. Switch the interface language to Hungarian in Settings first if it is not already. Report untranslated interface text, mixed-language screens, and text that is cut off or overflows."), + }, +} satisfies E2EConfig; diff --git a/e2e/full-stack/stack-core.ts b/e2e/full-stack/stack-core.ts new file mode 100644 index 000000000..4761b4c66 --- /dev/null +++ b/e2e/full-stack/stack-core.ts @@ -0,0 +1,69 @@ +import { createClient } from "@supabase/supabase-js"; +import { createHash, randomUUID } from "node:crypto"; + +// These specs drive the real local stack: Supabase Auth, Postgres/RLS, every +// Edge Function, and a fake OpenRouter (e2e/full-stack/fake-openrouter.mjs). +// Only the paid model provider is replaced. `.superset/db-gate.sh` provides the +// environment below; without it the specs skip. +// +// This module is the Playwright-free half of stack.ts. The agentic suite in +// agentic/ runs under the e2e runner and cannot import @playwright/test. +const supabaseUrl = process.env.LOCAL_SUPABASE_URL; +const serviceKey = process.env.LOCAL_SUPABASE_SERVICE_ROLE_KEY; +const fakeUrl = process.env.FAKE_OPENROUTER_URL; +const demoPepper = process.env.DEMO_CODE_PEPPER; + +export const MARK = "FAKE-PROVIDER"; +export const stackReady = Boolean(supabaseUrl && serviceKey && fakeUrl && demoPepper); +// Only the locked gate loads these specs in CI; a skip there would read as a pass. +if (process.env.CI && !stackReady) throw new Error("Full-stack gate environment is incomplete"); + +export const admin = stackReady + ? createClient(supabaseUrl!, serviceKey!, { auth: { persistSession: false } }) + : (null as never); + +export type FakeRequest = { feature: string | null; model?: string; stream?: boolean; system: string; user: string }; + +export const fake = { + async reset() { + await fetch(`${fakeUrl}/__reset`, { method: "POST" }); + }, + async requests(): Promise<FakeRequest[]> { + return (await fetch(`${fakeUrl}/__requests`)).json(); + }, + async fail(feature: string, status = 500, count = 1) { + await fetch(`${fakeUrl}/__fail`, { method: "POST", body: JSON.stringify({ feature, status, count }) }); + }, +}; + +export type DemoUser = { code: string; email: string; userId: () => Promise<string> }; + +async function userIdFor(email: string) { + // The local project holds only test users, so one page covers it. + const { data, error } = await admin.auth.admin.listUsers({ perPage: 1000 }); + if (error) throw error; + const user = data.users.find((candidate) => candidate.email === email); + if (!user) throw new Error(`No user for ${email}`); + return user.id; +} + +export async function createDemoUser(label: string): Promise<DemoUser & { codeId: string }> { + const alphabet = "ABCDEFGHJKLMNPQRSTUVWXYZ23456789"; + const part = () => Array.from({ length: 4 }, () => alphabet[Math.floor(Math.random() * alphabet.length)]).join(""); + const code = `DEMO-${part()}-${part()}`; + const email = `${label}-${randomUUID()}@fullstack.test`; + const codeHash = createHash("sha256").update(`${demoPepper}:${code}`).digest("hex"); + const { data, error } = await admin.from("demo_access_codes") + .insert({ code_hash: codeHash, label, email, max_uses: 10 }) + .select("id") + .single(); + if (error) throw error; + return { code, email, codeId: data.id, userId: () => userIdFor(email) }; +} + +export async function deleteDemoUser(user: DemoUser & { codeId: string }) { + const { data } = await admin.auth.admin.listUsers({ perPage: 1000 }); + const id = data?.users.find((candidate) => candidate.email === user.email)?.id; + if (id) await admin.auth.admin.deleteUser(id); + await admin.from("demo_access_codes").delete().eq("id", user.codeId); +} diff --git a/e2e/full-stack/stack.ts b/e2e/full-stack/stack.ts index d4937b7d4..041eb8d78 100644 --- a/e2e/full-stack/stack.ts +++ b/e2e/full-stack/stack.ts @@ -1,70 +1,7 @@ -import { createClient } from "@supabase/supabase-js"; import { expect, test as base, type Page } from "@playwright/test"; -import { createHash, randomUUID } from "node:crypto"; +import { createDemoUser, deleteDemoUser, fake, type DemoUser, type FakeRequest } from "./stack-core"; -// These specs drive the real local stack: Supabase Auth, Postgres/RLS, every -// Edge Function, and a fake OpenRouter (e2e/full-stack/fake-openrouter.mjs). -// Only the paid model provider is replaced. `.superset/db-gate.sh` provides the -// environment below; without it the specs skip. -const supabaseUrl = process.env.LOCAL_SUPABASE_URL; -const serviceKey = process.env.LOCAL_SUPABASE_SERVICE_ROLE_KEY; -const fakeUrl = process.env.FAKE_OPENROUTER_URL; -const demoPepper = process.env.DEMO_CODE_PEPPER; - -export const MARK = "FAKE-PROVIDER"; -export const stackReady = Boolean(supabaseUrl && serviceKey && fakeUrl && demoPepper); -// Only the locked gate loads these specs in CI; a skip there would read as a pass. -if (process.env.CI && !stackReady) throw new Error("Full-stack gate environment is incomplete"); - -export const admin = stackReady - ? createClient(supabaseUrl!, serviceKey!, { auth: { persistSession: false } }) - : (null as never); - -export type FakeRequest = { feature: string | null; model?: string; stream?: boolean; system: string; user: string }; - -export const fake = { - async reset() { - await fetch(`${fakeUrl}/__reset`, { method: "POST" }); - }, - async requests(): Promise<FakeRequest[]> { - return (await fetch(`${fakeUrl}/__requests`)).json(); - }, - async fail(feature: string, status = 500, count = 1) { - await fetch(`${fakeUrl}/__fail`, { method: "POST", body: JSON.stringify({ feature, status, count }) }); - }, -}; - -export type DemoUser = { code: string; email: string; userId: () => Promise<string> }; - -async function userIdFor(email: string) { - // The local project holds only test users, so one page covers it. - const { data, error } = await admin.auth.admin.listUsers({ perPage: 1000 }); - if (error) throw error; - const user = data.users.find((candidate) => candidate.email === email); - if (!user) throw new Error(`No user for ${email}`); - return user.id; -} - -export async function createDemoUser(label: string): Promise<DemoUser & { codeId: string }> { - const alphabet = "ABCDEFGHJKLMNPQRSTUVWXYZ23456789"; - const part = () => Array.from({ length: 4 }, () => alphabet[Math.floor(Math.random() * alphabet.length)]).join(""); - const code = `DEMO-${part()}-${part()}`; - const email = `${label}-${randomUUID()}@fullstack.test`; - const codeHash = createHash("sha256").update(`${demoPepper}:${code}`).digest("hex"); - const { data, error } = await admin.from("demo_access_codes") - .insert({ code_hash: codeHash, label, email, max_uses: 10 }) - .select("id") - .single(); - if (error) throw error; - return { code, email, codeId: data.id, userId: () => userIdFor(email) }; -} - -export async function deleteDemoUser(user: DemoUser & { codeId: string }) { - const { data } = await admin.auth.admin.listUsers({ perPage: 1000 }); - const id = data?.users.find((candidate) => candidate.email === user.email)?.id; - if (id) await admin.auth.admin.deleteUser(id); - await admin.from("demo_access_codes").delete().eq("id", user.codeId); -} +export * from "./stack-core"; /** Sign in through the real Login page and demo-code-login Edge Function. */ export async function signIn(page: Page, user: DemoUser) { diff --git a/package-lock.json b/package-lock.json index 11f0edf01..4bcace633 100644 --- a/package-lock.json +++ b/package-lock.json @@ -81,8 +81,10 @@ "zustand": "^5.0.10" }, "devDependencies": { + "@e2e-dev/web": "0.11.1", "@eslint/js": "^9.32.0", - "@playwright/test": "^1.60.0", + "@openrouter/ai-sdk-provider": "^3.1.0", + "@playwright/test": "^1.63.0", "@sentry/vite-plugin": "^5.4.0", "@tailwindcss/typography": "^0.5.16", "@testing-library/jest-dom": "^6.6.0", @@ -93,7 +95,9 @@ "@types/react-dom": "^18.3.7", "@vitejs/plugin-react-swc": "^3.11.0", "@vitest/coverage-v8": "^4.1.8", + "ai": "^7.0.127", "autoprefixer": "^10.4.21", + "e2e": "0.15.2", "eslint": "^9.32.0", "eslint-plugin-react-hooks": "^5.2.0", "eslint-plugin-react-refresh": "^0.4.20", @@ -118,6 +122,57 @@ "dev": true, "license": "MIT" }, + "node_modules/@ai-sdk/gateway": { + "version": "4.0.103", + "resolved": "https://registry.npmjs.org/@ai-sdk/gateway/-/gateway-4.0.103.tgz", + "integrity": "sha512-nnGTUHzPRQ14jhv9CL4OQrzcRBIFTBMIPKGIi9Z9u/4GtwSRxa/nxUX6np2Iwp5phOzO0a0njj/j9Jolf5AEaA==", + "dev": true, + "license": "Apache-2.0", + "dependencies": { + "@ai-sdk/provider": "4.0.21", + "@ai-sdk/provider-utils": "5.0.53", + "@vercel/oidc": "3.2.0" + }, + "engines": { + "node": ">=22" + }, + "peerDependencies": { + "zod": "^3.25.76 || ^4.1.8" + } + }, + "node_modules/@ai-sdk/provider": { + "version": "4.0.21", + "resolved": "https://registry.npmjs.org/@ai-sdk/provider/-/provider-4.0.21.tgz", + "integrity": "sha512-UpbC9C1oht8dhfKPbVXSLRZS3DI8uk8n5v2uMjBPNIYGsC2kL045ywq7D8Hh9KgnH9rP5W/GxQdYtWScRouqlA==", + "dev": true, + "license": "Apache-2.0", + "dependencies": { + "json-schema": "^0.4.0" + }, + "engines": { + "node": ">=22" + } + }, + "node_modules/@ai-sdk/provider-utils": { + "version": "5.0.53", + "resolved": "https://registry.npmjs.org/@ai-sdk/provider-utils/-/provider-utils-5.0.53.tgz", + "integrity": "sha512-VVe6UDd0y0/B4TfauuKKzcppqvhTmSluwwUVT4WG2/8h7gFbyFm7PM4UGM+ZXe3/RI6bv+Vl1cw/7aiBoQqdxg==", + "dev": true, + "license": "Apache-2.0", + "dependencies": { + "@ai-sdk/provider": "4.0.21", + "@standard-schema/spec": "^1.1.0", + "@workflow/serde": "4.1.0", + "eventsource-parser": "^3.0.8", + "undici": "^7.29.0" + }, + "engines": { + "node": ">=22" + }, + "peerDependencies": { + "zod": "^3.25.76 || ^4.1.8" + } + }, "node_modules/@alloc/quick-lru": { "version": "5.3.0", "resolved": "https://registry.npmjs.org/@alloc/quick-lru/-/quick-lru-5.3.0.tgz", @@ -522,6 +577,50 @@ "@capacitor/core": ">=8.0.0" } }, + "node_modules/@clack/core": { + "version": "1.5.1", + "resolved": "https://registry.npmjs.org/@clack/core/-/core-1.5.1.tgz", + "integrity": "sha512-iHTrHA8MtVuLl2TfZySmcKv1qO2PoyC9Z7pfSDozEuV5vtY3/wcOPKJXlqJ5Oq2Cx5DDGQGAMVx6HZfRRoVEbQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "fast-wrap-ansi": "^0.2.0", + "sisteransi": "^1.0.5" + }, + "engines": { + "node": ">= 20.12.0" + } + }, + "node_modules/@clack/prompts": { + "version": "1.8.1", + "resolved": "https://registry.npmjs.org/@clack/prompts/-/prompts-1.8.1.tgz", + "integrity": "sha512-dlT1m5e/0yUL0kRNcQn7yGLVThkgbB0Ga/1AmfDDC/8ik6AIiSf2QLQO2zPYvefsHP0aFgxO93cVLCCfDp7kzQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "@clack/core": "1.5.1", + "fast-string-width": "^3.0.2", + "fast-wrap-ansi": "^0.2.0", + "sisteransi": "^1.0.5" + }, + "engines": { + "node": ">= 20.12.0" + } + }, + "node_modules/@e2e-dev/web": { + "version": "0.11.1", + "resolved": "https://registry.npmjs.org/@e2e-dev/web/-/web-0.11.1.tgz", + "integrity": "sha512-ZmCTlkVei9sLRyObs4jftNewarv1tKYVE18ThOv3+96w3M5PC7vwBjYJxxPti9byzdSNF2MJYlQCXLAnguMIJQ==", + "dev": true, + "license": "Apache-2.0", + "engines": { + "node": ">=22.12.0" + }, + "peerDependencies": { + "e2e": ">=0.15.0 <1", + "playwright": ">=1.63.0 <2" + } + }, "node_modules/@esbuild/aix-ppc64": { "version": "0.28.2", "resolved": "https://registry.npmjs.org/@esbuild/aix-ppc64/-/aix-ppc64-0.28.2.tgz", @@ -1209,6 +1308,19 @@ "yjs": "^13.6.8" } }, + "node_modules/@hono/node-server": { + "version": "2.1.3", + "resolved": "https://registry.npmjs.org/@hono/node-server/-/node-server-2.1.3.tgz", + "integrity": "sha512-TA//nWMqPhbfdfneACk6t5a9eqbS9lABEPyKn0/xZTah3H3U2XaVg85rJFl0/Fyit0I552YDHgXGVSf3GwqbUw==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=20" + }, + "peerDependencies": { + "hono": "^4" + } + }, "node_modules/@hookform/resolvers": { "version": "3.10.0", "resolved": "https://registry.npmjs.org/@hookform/resolvers/-/resolvers-3.10.0.tgz", @@ -1487,6 +1599,71 @@ "integrity": "sha512-QZqem4QuAnAyzfz+Gj5/+SLxqwCAw2qmt7732ZXodr6VDWGeYLG6w1i/vYLa55JQM9wRuBKLmXmiZ2P0LtE5rw==", "license": "MIT" }, + "node_modules/@modelcontextprotocol/sdk": { + "version": "1.30.1", + "resolved": "https://registry.npmjs.org/@modelcontextprotocol/sdk/-/sdk-1.30.1.tgz", + "integrity": "sha512-H2HxLvC3HDNybePJaLdSrU1hhUK5iQw+WvV1b01myFyI7sdVGe1u/IPTE5D9fGCiJDVtgMV/lmFkQXLmQyIFYA==", + "dev": true, + "license": "MIT", + "dependencies": { + "@hono/node-server": "^1.19.9 || ^2.0.5", + "ajv": "^8.17.1", + "ajv-formats": "^3.0.1", + "content-type": "^1.0.5", + "cors": "^2.8.5", + "cross-spawn": "^7.0.5", + "eventsource": "^3.0.2", + "eventsource-parser": "^3.0.0", + "express": "^5.2.1", + "express-rate-limit": "^8.2.1", + "hono": "^4.11.4", + "jose": "^6.1.3", + "json-schema-typed": "^8.0.2", + "pkce-challenge": "^5.0.0", + "raw-body": "^3.0.0", + "zod": "^3.25 || ^4.0", + "zod-to-json-schema": "^3.25.1" + }, + "engines": { + "node": ">=18" + }, + "peerDependencies": { + "@cfworker/json-schema": "^4.1.1", + "zod": "^3.25 || ^4.0" + }, + "peerDependenciesMeta": { + "@cfworker/json-schema": { + "optional": true + }, + "zod": { + "optional": false + } + } + }, + "node_modules/@modelcontextprotocol/sdk/node_modules/ajv": { + "version": "8.20.0", + "resolved": "https://registry.npmjs.org/ajv/-/ajv-8.20.0.tgz", + "integrity": "sha512-Thbli+OlOj+iMPYFBVBfJ3OmCAnaSyNn4M1vz9T6Gka5Jt9ba/HIR56joy65tY6kx/FCF5VXNB819Y7/GUrBGA==", + "dev": true, + "license": "MIT", + "dependencies": { + "fast-deep-equal": "^3.1.3", + "fast-uri": "^3.0.1", + "json-schema-traverse": "^1.0.0", + "require-from-string": "^2.0.2" + }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/epoberezkin" + } + }, + "node_modules/@modelcontextprotocol/sdk/node_modules/json-schema-traverse": { + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/json-schema-traverse/-/json-schema-traverse-1.0.0.tgz", + "integrity": "sha512-NM8/P9n3XjXhIZn1lLhkFaACTOURQXjWhV4BA/RnOv8xvgqtqpAX9IO4mRQxSx1Rlo4tqzeqb0sOlruaOy3dug==", + "dev": true, + "license": "MIT" + }, "node_modules/@napi-rs/canvas": { "version": "1.0.8", "resolved": "https://registry.npmjs.org/@napi-rs/canvas/-/canvas-1.0.8.tgz", @@ -1807,14 +1984,28 @@ "node": ">= 8" } }, + "node_modules/@openrouter/ai-sdk-provider": { + "version": "3.1.0", + "resolved": "https://registry.npmjs.org/@openrouter/ai-sdk-provider/-/ai-sdk-provider-3.1.0.tgz", + "integrity": "sha512-BQy1TA9fKrh47s9ivsO6FY5QimN7GUF0i8Qon3pkxQclEgShso0dClpcWn8OIyPLEZZPFaFKgI8TW4S/O0p0Dw==", + "dev": true, + "license": "Apache-2.0", + "engines": { + "node": ">=22" + }, + "peerDependencies": { + "ai": "^7.0.0", + "zod": "^3.25.76 || ^4.1.8" + } + }, "node_modules/@playwright/test": { - "version": "1.62.1", - "resolved": "https://registry.npmjs.org/@playwright/test/-/test-1.62.1.tgz", - "integrity": "sha512-DTcUc8qii+cpHvtOwggMtBRMjKZHXYWdw8syRYu2vtzuq4Wxphqq4NfCs5Zt44L6mA8rfDfj+PHnxFc/FeK6mQ==", + "version": "1.63.0", + "resolved": "https://registry.npmjs.org/@playwright/test/-/test-1.63.0.tgz", + "integrity": "sha512-oxMK4vllB9RK5NQ2l1pq1IfOf2AvnEuj/vYGDj0H2nMtmtZpKtCwt/l00GEO6xjGfpBNAvjovvYdCm50dRQkpQ==", "dev": true, "license": "Apache-2.0", "dependencies": { - "playwright": "1.62.1" + "playwright": "1.63.0" }, "bin": { "playwright": "cli.js" @@ -5582,6 +5773,16 @@ } } }, + "node_modules/@vercel/oidc": { + "version": "3.2.0", + "resolved": "https://registry.npmjs.org/@vercel/oidc/-/oidc-3.2.0.tgz", + "integrity": "sha512-UycprH3T6n3jH0k44NHMa7pnFHGu/N05MjojYr+Mc6I7obkoLIJujSWwin1pCvdy/eOxrI/l3uDLQsmcrOb4ug==", + "dev": true, + "license": "Apache-2.0", + "engines": { + "node": ">= 20" + } + }, "node_modules/@vercel/speed-insights": { "version": "2.0.0", "resolved": "https://registry.npmjs.org/@vercel/speed-insights/-/speed-insights-2.0.0.tgz", @@ -5778,6 +5979,13 @@ "url": "https://opencollective.com/vitest" } }, + "node_modules/@workflow/serde": { + "version": "4.1.0", + "resolved": "https://registry.npmjs.org/@workflow/serde/-/serde-4.1.0.tgz", + "integrity": "sha512-pav4F2BoirECWR7Nf1TKt+2eETcBj7jj4cBefQ8VXQCA6NPkaKeLfj/zMgi+3zYV5ZIBT4GuUiphsj0/b9hPQQ==", + "dev": true, + "license": "Apache-2.0" + }, "node_modules/@xmldom/xmldom": { "version": "0.8.15", "resolved": "https://registry.npmjs.org/@xmldom/xmldom/-/xmldom-0.8.15.tgz", @@ -5795,6 +6003,47 @@ "dev": true, "license": "BSD-3-Clause" }, + "node_modules/accepts": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/accepts/-/accepts-2.0.0.tgz", + "integrity": "sha512-5cvg6CtKwfgdmVqY1WIiXKc3Q1bkRqGLi+2W/6ao+6Y7gu/RCwRuAhGEzh5B4KlszSuTLgZYuqFqo5bImjNKng==", + "dev": true, + "license": "MIT", + "dependencies": { + "mime-types": "^3.0.0", + "negotiator": "^1.0.0" + }, + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/accepts/node_modules/mime-db": { + "version": "1.54.0", + "resolved": "https://registry.npmjs.org/mime-db/-/mime-db-1.54.0.tgz", + "integrity": "sha512-aU5EJuIN2WDemCcAp2vFBfp/m4EAhWJnUNSSw0ixs7/kXbd6Pg64EmwJkNdFhB8aWt1sH2CTXrLxo/iAGV3oPQ==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/accepts/node_modules/mime-types": { + "version": "3.0.2", + "resolved": "https://registry.npmjs.org/mime-types/-/mime-types-3.0.2.tgz", + "integrity": "sha512-Lbgzdk0h4juoQ9fCKXW4by0UJqj+nOOrI9MJ1sSj4nI8aI2eo1qmvQEie4VD1glsS250n15LsWsYtCugiStS5A==", + "dev": true, + "license": "MIT", + "dependencies": { + "mime-db": "^1.54.0" + }, + "engines": { + "node": ">=18" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, "node_modules/acorn": { "version": "8.18.0", "resolved": "https://registry.npmjs.org/acorn/-/acorn-8.18.0.tgz", @@ -5855,6 +6104,24 @@ "node": ">= 6.0.0" } }, + "node_modules/ai": { + "version": "7.0.127", + "resolved": "https://registry.npmjs.org/ai/-/ai-7.0.127.tgz", + "integrity": "sha512-JNsNPk4ZvRGEJuqdVo2lYFkm7uAuf3p/RwKFWn1Wh/+IK3U1RyFWw2GXoFnhzqNfxEuj7XJB4WpIUhl1sHToWg==", + "dev": true, + "license": "Apache-2.0", + "dependencies": { + "@ai-sdk/gateway": "4.0.103", + "@ai-sdk/provider": "4.0.21", + "@ai-sdk/provider-utils": "5.0.53" + }, + "engines": { + "node": ">=22" + }, + "peerDependencies": { + "zod": "^3.25.76 || ^4.1.8" + } + }, "node_modules/ajv": { "version": "6.15.0", "resolved": "https://registry.npmjs.org/ajv/-/ajv-6.15.0.tgz", @@ -5872,6 +6139,48 @@ "url": "https://github.com/sponsors/epoberezkin" } }, + "node_modules/ajv-formats": { + "version": "3.0.1", + "resolved": "https://registry.npmjs.org/ajv-formats/-/ajv-formats-3.0.1.tgz", + "integrity": "sha512-8iUql50EUR+uUcdRQ3HDqa6EVyo3docL8g5WJ3FNcWmu62IbkGUue/pEyLBW8VGKKucTPgqeks4fIU1DA4yowQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "ajv": "^8.0.0" + }, + "peerDependencies": { + "ajv": "^8.0.0" + }, + "peerDependenciesMeta": { + "ajv": { + "optional": true + } + } + }, + "node_modules/ajv-formats/node_modules/ajv": { + "version": "8.20.0", + "resolved": "https://registry.npmjs.org/ajv/-/ajv-8.20.0.tgz", + "integrity": "sha512-Thbli+OlOj+iMPYFBVBfJ3OmCAnaSyNn4M1vz9T6Gka5Jt9ba/HIR56joy65tY6kx/FCF5VXNB819Y7/GUrBGA==", + "dev": true, + "license": "MIT", + "dependencies": { + "fast-deep-equal": "^3.1.3", + "fast-uri": "^3.0.1", + "json-schema-traverse": "^1.0.0", + "require-from-string": "^2.0.2" + }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/epoberezkin" + } + }, + "node_modules/ajv-formats/node_modules/json-schema-traverse": { + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/json-schema-traverse/-/json-schema-traverse-1.0.0.tgz", + "integrity": "sha512-NM8/P9n3XjXhIZn1lLhkFaACTOURQXjWhV4BA/RnOv8xvgqtqpAX9IO4mRQxSx1Rlo4tqzeqb0sOlruaOy3dug==", + "dev": true, + "license": "MIT" + }, "node_modules/ansi-regex": { "version": "5.0.1", "resolved": "https://registry.npmjs.org/ansi-regex/-/ansi-regex-5.0.1.tgz", @@ -6137,6 +6446,62 @@ "integrity": "sha512-iD3898SR7sWVRHbiQv+sHUtHnMvC1o3nW5rAcqnq3uOn07DSAppZYUkIGslDz6gXC7HfunPe7YVBgoEJASPcHA==", "license": "MIT" }, + "node_modules/body-parser": { + "version": "2.3.0", + "resolved": "https://registry.npmjs.org/body-parser/-/body-parser-2.3.0.tgz", + "integrity": "sha512-2cGmJupaNgg+QUwVLAucDuWuoMZ6EX9iHDRswZ5lsNYEmwPaRknMPCLZz07yTzVq/83p4o/wzbDZbBrTvGGTIw==", + "dev": true, + "license": "MIT", + "dependencies": { + "bytes": "^3.1.2", + "content-type": "^2.0.0", + "debug": "^4.4.3", + "http-errors": "^2.0.1", + "iconv-lite": "^0.7.2", + "on-finished": "^2.4.1", + "qs": "^6.15.2", + "raw-body": "^3.0.2", + "type-is": "^2.1.0" + }, + "engines": { + "node": ">=18" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/body-parser/node_modules/content-type": { + "version": "2.1.0", + "resolved": "https://registry.npmjs.org/content-type/-/content-type-2.1.0.tgz", + "integrity": "sha512-mj7UPXE0jaqaOsukNZRUEfEi2AcL7C/vwmwcHV0O97eO1E1pxBZuyjlZrx5seTaNBg1U6+o35wpa35Qfcc+7ag==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=18" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/body-parser/node_modules/iconv-lite": { + "version": "0.7.3", + "resolved": "https://registry.npmjs.org/iconv-lite/-/iconv-lite-0.7.3.tgz", + "integrity": "sha512-IKXpvIzjnC9XTAUbVBcMfGS0EPaIXtW6v+zr+RRp+hqULEpo0owZax6wyRwPOJbWbzjYspQwusTsfVr0ifh4uQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "safer-buffer": ">= 2.1.2 < 3.0.0" + }, + "engines": { + "node": ">=0.10.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, "node_modules/bplist-creator": { "version": "0.1.0", "resolved": "https://registry.npmjs.org/bplist-creator/-/bplist-creator-0.1.0.tgz", @@ -6224,6 +6589,16 @@ "node": "*" } }, + "node_modules/bytes": { + "version": "3.1.2", + "resolved": "https://registry.npmjs.org/bytes/-/bytes-3.1.2.tgz", + "integrity": "sha512-/Nf7TyzTx6S3yRJObOAV7956r8cr2+Oj8AC5dt8wSP3BQAoeX58NoHyCU8P8zGkNXStjTSi6fzO6F0pBdcYbEg==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">= 0.8" + } + }, "node_modules/call-bind-apply-helpers": { "version": "1.0.2", "resolved": "https://registry.npmjs.org/call-bind-apply-helpers/-/call-bind-apply-helpers-1.0.2.tgz", @@ -6238,6 +6613,23 @@ "node": ">= 0.4" } }, + "node_modules/call-bound": { + "version": "1.0.4", + "resolved": "https://registry.npmjs.org/call-bound/-/call-bound-1.0.4.tgz", + "integrity": "sha512-+ys997U96po4Kx/ABpBCqhA9EuxJaQWDQg7295H4hBphv3IZg0boBKuwYpt4YXp6MZ5AmZQnU/tyMTlRpaSejg==", + "dev": true, + "license": "MIT", + "dependencies": { + "call-bind-apply-helpers": "^1.0.2", + "get-intrinsic": "^1.3.0" + }, + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, "node_modules/callsites": { "version": "3.1.0", "resolved": "https://registry.npmjs.org/callsites/-/callsites-3.1.0.tgz", @@ -6504,6 +6896,30 @@ "dev": true, "license": "MIT" }, + "node_modules/content-disposition": { + "version": "1.1.0", + "resolved": "https://registry.npmjs.org/content-disposition/-/content-disposition-1.1.0.tgz", + "integrity": "sha512-5jRCH9Z/+DRP7rkvY83B+yGIGX96OYdJmzngqnw2SBSxqCFPd0w2km3s5iawpGX8krnwSGmF0FW5Nhr0Hfai3g==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=18" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/content-type": { + "version": "1.0.5", + "resolved": "https://registry.npmjs.org/content-type/-/content-type-1.0.5.tgz", + "integrity": "sha512-nTjqfcBFEipKdXCv4YDQWCfmcLZKm81ldF0pAopTvyrFGVbcR6P/VAAd5G7N+0tTr8QqiU0tFadD6FK4NtJwOA==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">= 0.6" + } + }, "node_modules/convert-source-map": { "version": "2.0.0", "resolved": "https://registry.npmjs.org/convert-source-map/-/convert-source-map-2.0.0.tgz", @@ -6524,6 +6940,16 @@ "url": "https://opencollective.com/express" } }, + "node_modules/cookie-signature": { + "version": "1.2.2", + "resolved": "https://registry.npmjs.org/cookie-signature/-/cookie-signature-1.2.2.tgz", + "integrity": "sha512-D76uU73ulSXrD1UXF4KE2TMxVVwhsnCgfAyTg9k8P6KGZjlXKrOLe4dJQKI3Bxi5wjesZoFXJWElNWBjPZMbhg==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=6.6.0" + } + }, "node_modules/core-js": { "version": "3.50.0", "resolved": "https://registry.npmjs.org/core-js/-/core-js-3.50.0.tgz", @@ -6544,6 +6970,24 @@ "integrity": "sha512-ZQBvi1DcpJ4GDqanjucZ2Hj3wEO5pZDS89BWbkcrvdxksJorwUDDZamX9ldFkp9aw2lmBDLgkObEA4DWNJ9FYQ==", "license": "MIT" }, + "node_modules/cors": { + "version": "2.8.6", + "resolved": "https://registry.npmjs.org/cors/-/cors-2.8.6.tgz", + "integrity": "sha512-tJtZBBHA6vjIAaF6EnIaq6laBBP9aq/Y3ouVJjEfoHbRBcHBAHYcMh/w8LDrk2PvIMMq8gmopa5D4V8RmbrxGw==", + "dev": true, + "license": "MIT", + "dependencies": { + "object-assign": "^4", + "vary": "^1" + }, + "engines": { + "node": ">= 0.10" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, "node_modules/cross-spawn": { "version": "7.0.6", "resolved": "https://registry.npmjs.org/cross-spawn/-/cross-spawn-7.0.6.tgz", @@ -6931,6 +7375,16 @@ "node": ">=0.4.0" } }, + "node_modules/depd": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/depd/-/depd-2.0.0.tgz", + "integrity": "sha512-g7nH6P6dyDioJogAAGprGpCtVImJhpPk/roCzdb3fIh61/s/nPsfR6onyMwkCAR/OlC3yBC0lESvUoQEAssIrw==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">= 0.8" + } + }, "node_modules/dequal": { "version": "2.0.3", "resolved": "https://registry.npmjs.org/dequal/-/dequal-2.0.3.tgz", @@ -7087,6 +7541,155 @@ "node": ">= 0.4" } }, + "node_modules/e2e": { + "version": "0.15.2", + "resolved": "https://registry.npmjs.org/e2e/-/e2e-0.15.2.tgz", + "integrity": "sha512-QBbrovhKlr++kc4J2Pt+2oJ4tEl36MgIDbaZmuMaE+mjTokfIxnSWJHAgFY6ox8rfuOkvCGYXBSFIKXyiqv7Cw==", + "dev": true, + "license": "Apache-2.0", + "dependencies": { + "@ai-sdk/provider": "4.0.18", + "@clack/prompts": "1.8.1", + "@modelcontextprotocol/sdk": "1.30.1", + "@vitest/expect": "5.0.2", + "commander": "15.0.0", + "picocolors": "1.1.1", + "pngjs": "7.0.0", + "tsx": "4.23.14", + "zod": "4.6.1" + }, + "bin": { + "e2e": "dist/cli/bin.js" + }, + "engines": { + "node": ">=22.12.0" + }, + "peerDependencies": { + "@ai-sdk/openai": "^4.0.0", + "@ai-sdk/openai-compatible": "^3.0.0", + "@ai-sdk/xai": "^5.0.0", + "ai": "^7.0.0" + }, + "peerDependenciesMeta": { + "@ai-sdk/openai": { + "optional": true + }, + "@ai-sdk/openai-compatible": { + "optional": true + }, + "@ai-sdk/xai": { + "optional": true + }, + "ai": { + "optional": true + } + } + }, + "node_modules/e2e/node_modules/@ai-sdk/provider": { + "version": "4.0.18", + "resolved": "https://registry.npmjs.org/@ai-sdk/provider/-/provider-4.0.18.tgz", + "integrity": "sha512-+GZJIgz1jk86pwEbb3f1BD2bdoSKyWE4Jg4YUc7NMnMozbWemSKYZcw2F4nMaO5qwsL5A8RmioAKW81YktRpIQ==", + "dev": true, + "license": "Apache-2.0", + "dependencies": { + "json-schema": "^0.4.0" + }, + "engines": { + "node": ">=22" + } + }, + "node_modules/e2e/node_modules/@vitest/expect": { + "version": "5.0.2", + "resolved": "https://registry.npmjs.org/@vitest/expect/-/expect-5.0.2.tgz", + "integrity": "sha512-pd6YDkhOHptwC65NYNpGpNxWlJH9M9Gxwtej73osTddb1NpiE0Jz5CIjTTEGXRgK0YtiSN7qBokmBXZ5uPJL1w==", + "dev": true, + "license": "MIT", + "dependencies": { + "@standard-schema/spec": "^1.1.0", + "@types/chai": "^5.2.2", + "@vitest/spy": "5.0.2", + "@vitest/utils": "5.0.2", + "chai": "^6.2.2", + "tinyrainbow": "^3.1.1" + }, + "funding": { + "url": "https://opencollective.com/vitest" + } + }, + "node_modules/e2e/node_modules/@vitest/pretty-format": { + "version": "5.0.2", + "resolved": "https://registry.npmjs.org/@vitest/pretty-format/-/pretty-format-5.0.2.tgz", + "integrity": "sha512-YHM+mQQ7N1ROHMugJ7P/M+fC/q1G4zs/iAMZZCvhHcY8WkwPQcXBHD+z18oz+808Cfa12K7jOr3XPQMcLIovjw==", + "dev": true, + "license": "MIT", + "dependencies": { + "tinyrainbow": "^3.1.1" + }, + "funding": { + "url": "https://opencollective.com/vitest" + } + }, + "node_modules/e2e/node_modules/@vitest/spy": { + "version": "5.0.2", + "resolved": "https://registry.npmjs.org/@vitest/spy/-/spy-5.0.2.tgz", + "integrity": "sha512-Ijc7T1nT9efNb5LxvjaBrEqw3f/QwUv5EE0nKqZxgqsaV/FxAAZ8baGylA8X/Z2oS4Lp+K74Jr6dTJsDKxJDeg==", + "dev": true, + "license": "MIT", + "funding": { + "url": "https://opencollective.com/vitest" + } + }, + "node_modules/e2e/node_modules/@vitest/utils": { + "version": "5.0.2", + "resolved": "https://registry.npmjs.org/@vitest/utils/-/utils-5.0.2.tgz", + "integrity": "sha512-mgpUtxFhKeOKVaArBVHNJGdYHAeRh135o59l13uVmPUCS1OUtis5pBqXEgE25iqRgFqEBwXvJlkhknRsYnMeqg==", + "dev": true, + "license": "MIT", + "dependencies": { + "@vitest/pretty-format": "5.0.2", + "convert-source-map": "^2.0.0", + "tinyrainbow": "^3.1.1" + }, + "funding": { + "url": "https://opencollective.com/vitest" + } + }, + "node_modules/e2e/node_modules/commander": { + "version": "15.0.0", + "resolved": "https://registry.npmjs.org/commander/-/commander-15.0.0.tgz", + "integrity": "sha512-z67u4ZhzCL/Tydu1lJARtEZYWbWaN7oYLHbsuzocr6y4N6WZAagG3RQ4FW61V1/0+jImpj293XfrcYnd1qxtPg==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=22.12.0" + } + }, + "node_modules/e2e/node_modules/tsx": { + "version": "4.23.14", + "resolved": "https://registry.npmjs.org/tsx/-/tsx-4.23.14.tgz", + "integrity": "sha512-yFwMnbAsUFz/T3kR8P2ENc/DDUaYd9tKg4oVYo0lhkX0LlK4UuqYAO6mpQ2y6lmcyip1uPJ34C2kPACopDwG4w==", + "dev": true, + "license": "MIT", + "dependencies": { + "esbuild": "~0.28.0" + }, + "bin": { + "tsx": "dist/cli.mjs" + }, + "engines": { + "node": ">=18.0.0" + }, + "optionalDependencies": { + "fsevents": "~2.3.3" + } + }, + "node_modules/ee-first": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/ee-first/-/ee-first-1.1.1.tgz", + "integrity": "sha512-WMwm9LhRUo+WUaRN+vRuETqG89IgZphVSNkdFgeb6sS/E4OrDIN7t48CAewSHXc6C8lefD8KKfr5vY61brQlow==", + "dev": true, + "license": "MIT" + }, "node_modules/electron-to-chromium": { "version": "1.5.420", "resolved": "https://registry.npmjs.org/electron-to-chromium/-/electron-to-chromium-1.5.420.tgz", @@ -7112,6 +7715,16 @@ "integrity": "sha512-MSjYzcWNOA0ewAHpz0MxpYFvwg6yjy1NG3xteoqz644VCo/RPgnr1/GGt+ic3iJTzQ8Eu3TdM14SawnVUmGE6A==", "license": "MIT" }, + "node_modules/encodeurl": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/encodeurl/-/encodeurl-2.0.0.tgz", + "integrity": "sha512-Q0n9HRi4m6JuGIV1eFlmvJB7ZEVxu93IrMyiMsGC0lrMJMWzRgx6WGquyfQgZVb31vhGgXnfmPNNXmxnOkRBrg==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">= 0.8" + } + }, "node_modules/entities": { "version": "6.0.1", "resolved": "https://registry.npmjs.org/entities/-/entities-6.0.1.tgz", @@ -7241,6 +7854,13 @@ "node": ">=6" } }, + "node_modules/escape-html": { + "version": "1.0.3", + "resolved": "https://registry.npmjs.org/escape-html/-/escape-html-1.0.3.tgz", + "integrity": "sha512-NiSupZ4OeuGwr68lGIeym/ksIZMJodUGOSCZ/FSnTxcrekbvqrgdUxlJOMpijaKZVjAJrWrGs/6Jy8OMuyj9ow==", + "dev": true, + "license": "MIT" + }, "node_modules/escape-string-regexp": { "version": "4.0.0", "resolved": "https://registry.npmjs.org/escape-string-regexp/-/escape-string-regexp-4.0.0.tgz", @@ -7488,12 +8108,45 @@ "node": ">=0.10.0" } }, + "node_modules/etag": { + "version": "1.8.1", + "resolved": "https://registry.npmjs.org/etag/-/etag-1.8.1.tgz", + "integrity": "sha512-aIL5Fx7mawVa300al2BnEE4iNvo1qETxLrPI/o05L7z6go7fCw1J6EQmbK4FmJ2AS7kgVF/KEZWufBfdClMcPg==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">= 0.6" + } + }, "node_modules/eventemitter3": { "version": "4.0.7", "resolved": "https://registry.npmjs.org/eventemitter3/-/eventemitter3-4.0.7.tgz", "integrity": "sha512-8guHBZCwKnFhYdHr2ysuRWErTwhoN2X8XELRlrRwpmfeY2jjuUN4taQMsULKUVo1K4DvZl+0pgfyoysHxvmvEw==", "license": "MIT" }, + "node_modules/eventsource": { + "version": "3.0.7", + "resolved": "https://registry.npmjs.org/eventsource/-/eventsource-3.0.7.tgz", + "integrity": "sha512-CRT1WTyuQoD771GW56XEZFQ/ZoSfWid1alKGDYMmkt2yl8UXrVR4pspqWNEcqKvVIzg6PAltWjxcSSPrboA4iA==", + "dev": true, + "license": "MIT", + "dependencies": { + "eventsource-parser": "^3.0.1" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/eventsource-parser": { + "version": "3.1.1", + "resolved": "https://registry.npmjs.org/eventsource-parser/-/eventsource-parser-3.1.1.tgz", + "integrity": "sha512-EKN1vKAMcZ8MlYMpaNuxN6R9yakzH6uajHcHVTqWJzvu5pWw9DyhbP35HH8MVBQ+dZjAfDxk+A8NiR9KWaXiyQ==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=18.0.0" + } + }, "node_modules/expect-type": { "version": "1.4.0", "resolved": "https://registry.npmjs.org/expect-type/-/expect-type-1.4.0.tgz", @@ -7504,6 +8157,107 @@ "node": ">=12.0.0" } }, + "node_modules/express": { + "version": "5.2.1", + "resolved": "https://registry.npmjs.org/express/-/express-5.2.1.tgz", + "integrity": "sha512-hIS4idWWai69NezIdRt2xFVofaF4j+6INOpJlVOLDO8zXGpUVEVzIYk12UUi2JzjEzWL3IOAxcTubgz9Po0yXw==", + "dev": true, + "license": "MIT", + "dependencies": { + "accepts": "^2.0.0", + "body-parser": "^2.2.1", + "content-disposition": "^1.0.0", + "content-type": "^1.0.5", + "cookie": "^0.7.1", + "cookie-signature": "^1.2.1", + "debug": "^4.4.0", + "depd": "^2.0.0", + "encodeurl": "^2.0.0", + "escape-html": "^1.0.3", + "etag": "^1.8.1", + "finalhandler": "^2.1.0", + "fresh": "^2.0.0", + "http-errors": "^2.0.0", + "merge-descriptors": "^2.0.0", + "mime-types": "^3.0.0", + "on-finished": "^2.4.1", + "once": "^1.4.0", + "parseurl": "^1.3.3", + "proxy-addr": "^2.0.7", + "qs": "^6.14.0", + "range-parser": "^1.2.1", + "router": "^2.2.0", + "send": "^1.1.0", + "serve-static": "^2.2.0", + "statuses": "^2.0.1", + "type-is": "^2.0.1", + "vary": "^1.1.2" + }, + "engines": { + "node": ">= 18" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/express-rate-limit": { + "version": "8.7.0", + "resolved": "https://registry.npmjs.org/express-rate-limit/-/express-rate-limit-8.7.0.tgz", + "integrity": "sha512-hOwV7WOxXfjRpAM1DSJWZDXx3GhplwD8IfwuwvogD8i1Qnkgosw/H45s4ZnFAUHDAhPjlY9hLBvJhKmGMyY26g==", + "dev": true, + "license": "MIT", + "dependencies": { + "debug": "^4.4.3", + "ip-address": "^10.2.0" + }, + "engines": { + "node": ">= 16" + }, + "funding": { + "url": "https://github.com/sponsors/express-rate-limit" + }, + "peerDependencies": { + "express": ">= 4.11" + } + }, + "node_modules/express/node_modules/cookie": { + "version": "0.7.2", + "resolved": "https://registry.npmjs.org/cookie/-/cookie-0.7.2.tgz", + "integrity": "sha512-yki5XnKuf750l50uGTllt6kKILY4nQ1eNIQatoXEByZ5dWgnKqbnqmTrBE5B4N7lrMJKQ2ytWMiTO2o0v6Ew/w==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/express/node_modules/mime-db": { + "version": "1.54.0", + "resolved": "https://registry.npmjs.org/mime-db/-/mime-db-1.54.0.tgz", + "integrity": "sha512-aU5EJuIN2WDemCcAp2vFBfp/m4EAhWJnUNSSw0ixs7/kXbd6Pg64EmwJkNdFhB8aWt1sH2CTXrLxo/iAGV3oPQ==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/express/node_modules/mime-types": { + "version": "3.0.2", + "resolved": "https://registry.npmjs.org/mime-types/-/mime-types-3.0.2.tgz", + "integrity": "sha512-Lbgzdk0h4juoQ9fCKXW4by0UJqj+nOOrI9MJ1sSj4nI8aI2eo1qmvQEie4VD1glsS250n15LsWsYtCugiStS5A==", + "dev": true, + "license": "MIT", + "dependencies": { + "mime-db": "^1.54.0" + }, + "engines": { + "node": ">=18" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, "node_modules/extend": { "version": "3.0.2", "resolved": "https://registry.npmjs.org/extend/-/extend-3.0.2.tgz", @@ -7579,6 +8333,50 @@ "pako": "^2.1.0" } }, + "node_modules/fast-string-truncated-width": { + "version": "3.0.3", + "resolved": "https://registry.npmjs.org/fast-string-truncated-width/-/fast-string-truncated-width-3.0.3.tgz", + "integrity": "sha512-0jjjIEL6+0jag3l2XWWizO64/aZVtpiGE3t0Zgqxv0DPuxiMjvB3M24fCyhZUO4KomJQPj3LTSUnDP3GpdwC0g==", + "dev": true, + "license": "MIT" + }, + "node_modules/fast-string-width": { + "version": "3.0.2", + "resolved": "https://registry.npmjs.org/fast-string-width/-/fast-string-width-3.0.2.tgz", + "integrity": "sha512-gX8LrtNEI5hq8DVUfRQMbr5lpaS4nMIWV+7XEbXk2b8kiQIizgnlr12B4dA3ZEx3308ze0O4Q1R+cHts8kyUJg==", + "dev": true, + "license": "MIT", + "dependencies": { + "fast-string-truncated-width": "^3.0.2" + } + }, + "node_modules/fast-uri": { + "version": "3.1.8", + "resolved": "https://registry.npmjs.org/fast-uri/-/fast-uri-3.1.8.tgz", + "integrity": "sha512-GZMtZUTNRpOVIECoXwLNZS5xUGE+mVNbTB8h/7Rwh2TFWcBQiPzTgyZi05BF9UMZKkLJv8XBRJTlU7zg8+ZfMg==", + "dev": true, + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/fastify" + }, + { + "type": "opencollective", + "url": "https://opencollective.com/fastify" + } + ], + "license": "BSD-3-Clause" + }, + "node_modules/fast-wrap-ansi": { + "version": "0.2.2", + "resolved": "https://registry.npmjs.org/fast-wrap-ansi/-/fast-wrap-ansi-0.2.2.tgz", + "integrity": "sha512-7F2Fl+TjRSenLqlU3UjSH0iyqopqoZIu7eZVpEirP2g1GtWa2G/ecEmBdgz31+Mxr+ELclgg6sokpSFIQiZ02Q==", + "dev": true, + "license": "MIT", + "dependencies": { + "fast-string-width": "^3.0.2" + } + }, "node_modules/fastq": { "version": "1.20.3", "resolved": "https://registry.npmjs.org/fastq/-/fastq-1.20.3.tgz", @@ -7628,6 +8426,28 @@ "node": ">=8" } }, + "node_modules/finalhandler": { + "version": "2.1.1", + "resolved": "https://registry.npmjs.org/finalhandler/-/finalhandler-2.1.1.tgz", + "integrity": "sha512-S8KoZgRZN+a5rNwqTxlZZePjT/4cnm0ROV70LedRHZ0p8u9fRID0hJUZQpkKLzro8LfmC8sx23bY6tVNxv8pQA==", + "dev": true, + "license": "MIT", + "dependencies": { + "debug": "^4.4.0", + "encodeurl": "^2.0.0", + "escape-html": "^1.0.3", + "on-finished": "^2.4.1", + "parseurl": "^1.3.3", + "statuses": "^2.0.1" + }, + "engines": { + "node": ">= 18.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, "node_modules/find-up": { "version": "5.0.0", "resolved": "https://registry.npmjs.org/find-up/-/find-up-5.0.0.tgz", @@ -7683,6 +8503,16 @@ "node": ">= 6" } }, + "node_modules/forwarded": { + "version": "0.2.0", + "resolved": "https://registry.npmjs.org/forwarded/-/forwarded-0.2.0.tgz", + "integrity": "sha512-buRG0fpBtRHSTCOASe6hD258tEubFoRLb4ZNA6NxMVHNw2gOcwHo9wyablzMzOA5z9xA9L1KNjk/Nt6MT9aYow==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">= 0.6" + } + }, "node_modules/fraction.js": { "version": "5.3.4", "resolved": "https://registry.npmjs.org/fraction.js/-/fraction.js-5.3.4.tgz", @@ -7724,6 +8554,16 @@ } } }, + "node_modules/fresh": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/fresh/-/fresh-2.0.0.tgz", + "integrity": "sha512-Rx/WycZ60HOaqLKAi6cHRKKI7zxWbJ31MhntmtwMoaTeF7XFH9hhBp8vITaMidfljRQ6eYWCKkaTK+ykVJHP2A==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">= 0.8" + } + }, "node_modules/fs-extra": { "version": "11.4.0", "resolved": "https://registry.npmjs.org/fs-extra/-/fs-extra-11.4.0.tgz", @@ -7739,9 +8579,9 @@ } }, "node_modules/fsevents": { - "version": "2.3.2", - "resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.3.2.tgz", - "integrity": "sha512-xiqMQR4xAeHTuB9uWm+fFRcIOgKBMiOBP+eXiyT7jsgVCq1bkVygt00oASowB7EdtpOHaaPgKt812P9ab+DDKA==", + "version": "2.3.3", + "resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.3.3.tgz", + "integrity": "sha512-5xoDfX+fL7faATnagmWPpbFtwh/R77WmMMqqHGS65C3vvB0YHrgF+B1YmZ3441tMj5n63k0212XNoJwzlhffQw==", "hasInstallScript": true, "license": "MIT", "optional": true, @@ -8030,6 +8870,16 @@ "url": "https://opencollective.com/unified" } }, + "node_modules/hono": { + "version": "4.13.12", + "resolved": "https://registry.npmjs.org/hono/-/hono-4.13.12.tgz", + "integrity": "sha512-6E2QDAc9Ick9Sq77ZrGS/dk2WUYni91aufTw6LJKpV7w8kW5/GxVUc650FOADOmlwg3K+f7Pun6XlV+pYmW6gw==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=16.9.0" + } + }, "node_modules/html-encoding-sniffer": { "version": "3.0.0", "resolved": "https://registry.npmjs.org/html-encoding-sniffer/-/html-encoding-sniffer-3.0.0.tgz", @@ -8089,6 +8939,27 @@ "node": ">=8.0.0" } }, + "node_modules/http-errors": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/http-errors/-/http-errors-2.0.1.tgz", + "integrity": "sha512-4FbRdAX+bSdmo4AUFuS0WNiPz8NgFt+r8ThgNWmlrjQjt1Q7ZR9+zTlce2859x4KSXrwIsaeTqDoKQmtP8pLmQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "depd": "~2.0.0", + "inherits": "~2.0.4", + "setprototypeof": "~1.2.0", + "statuses": "~2.0.2", + "toidentifier": "~1.0.1" + }, + "engines": { + "node": ">= 0.8" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, "node_modules/http-proxy-agent": { "version": "5.0.0", "resolved": "https://registry.npmjs.org/http-proxy-agent/-/http-proxy-agent-5.0.0.tgz", @@ -8252,6 +9123,26 @@ "integrity": "sha512-DRebOWuqDvxunfkNJAlc3IzWIPD5xVxwUNbHr7xKB8E6aLJxIPfNX3CoMJghcFjpv6RWQsrcJbghtEwSPoJqMA==", "license": "MIT" }, + "node_modules/ip-address": { + "version": "10.7.3", + "resolved": "https://registry.npmjs.org/ip-address/-/ip-address-10.7.3.tgz", + "integrity": "sha512-A1kdq/tSb5QjvKvAMgIoEvDBIgL7qaqVP/jkvSwYYRZ9iEzvPpopxp2wQfu3SuZRHtpHNxMn8Fs0bS+gf5Xmwg==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">= 12" + } + }, + "node_modules/ipaddr.js": { + "version": "1.9.1", + "resolved": "https://registry.npmjs.org/ipaddr.js/-/ipaddr.js-1.9.1.tgz", + "integrity": "sha512-0KI/607xoxSToH7GjN1FfSbLoU0+btTicjsQSWQlh/hZykN8KpmMf7uYwPW3R+akZ6R/w18ZlXSHBYXiYUPO3g==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">= 0.10" + } + }, "node_modules/is-alphabetical": { "version": "2.0.1", "resolved": "https://registry.npmjs.org/is-alphabetical/-/is-alphabetical-2.0.1.tgz", @@ -8402,6 +9293,13 @@ "dev": true, "license": "MIT" }, + "node_modules/is-promise": { + "version": "4.0.0", + "resolved": "https://registry.npmjs.org/is-promise/-/is-promise-4.0.0.tgz", + "integrity": "sha512-hvpoI6korhJMnej285dSg6nu1+e6uxs7zG3BYAm5byqDsgJNWwxzM6z6iZiAgQR4TJ30JmBTOwqZUw3WlyH3AQ==", + "dev": true, + "license": "MIT" + }, "node_modules/is-wsl": { "version": "2.2.0", "resolved": "https://registry.npmjs.org/is-wsl/-/is-wsl-2.2.0.tgz", @@ -8484,6 +9382,16 @@ "jiti": "bin/jiti.js" } }, + "node_modules/jose": { + "version": "6.2.12", + "resolved": "https://registry.npmjs.org/jose/-/jose-6.2.12.tgz", + "integrity": "sha512-9NiFmJEex0sy2Dk58j2UGBSHgUs2ypF9eZSu4L6vjOX3Dp96Sw1F3uL+H+D1sx02jZZdzUT0HgvCy59CuvXcWw==", + "dev": true, + "license": "MIT", + "funding": { + "url": "https://github.com/sponsors/panva" + } + }, "node_modules/js-tokens": { "version": "4.0.0", "resolved": "https://registry.npmjs.org/js-tokens/-/js-tokens-4.0.0.tgz", @@ -8579,6 +9487,13 @@ "dev": true, "license": "MIT" }, + "node_modules/json-schema": { + "version": "0.4.0", + "resolved": "https://registry.npmjs.org/json-schema/-/json-schema-0.4.0.tgz", + "integrity": "sha512-es94M3nTIfsEPisRafak+HDLfHXnKBhV3vU5eqPcS3flIWqcxJWgXHXiey3YrpaNsanY5ei1VoYEbOzijuq9BA==", + "dev": true, + "license": "(AFL-2.1 OR BSD-3-Clause)" + }, "node_modules/json-schema-traverse": { "version": "0.4.1", "resolved": "https://registry.npmjs.org/json-schema-traverse/-/json-schema-traverse-0.4.1.tgz", @@ -8586,6 +9501,13 @@ "dev": true, "license": "MIT" }, + "node_modules/json-schema-typed": { + "version": "8.0.2", + "resolved": "https://registry.npmjs.org/json-schema-typed/-/json-schema-typed-8.0.2.tgz", + "integrity": "sha512-fQhoXdcvc3V28x7C7BMs4P5+kNlgUURe2jmUT1T//oBRMDrqy1QPelJimwZGo7Hg9VPV3EQV5Bnq4hbFy2vetA==", + "dev": true, + "license": "BSD-2-Clause" + }, "node_modules/json-stable-stringify-without-jsonify": { "version": "1.0.1", "resolved": "https://registry.npmjs.org/json-stable-stringify-without-jsonify/-/json-stable-stringify-without-jsonify-1.0.1.tgz", @@ -9074,6 +9996,33 @@ "url": "https://opencollective.com/unified" } }, + "node_modules/media-typer": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/media-typer/-/media-typer-1.1.1.tgz", + "integrity": "sha512-yz3xRaG20c6/BOzvYoDaGtPmGscs7YivItZEEqe6GbwNfHuxu9YNmvnEkMzKldAGY4/80pRcQRZSEnhquk9XuQ==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">= 0.8" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/merge-descriptors": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/merge-descriptors/-/merge-descriptors-2.0.0.tgz", + "integrity": "sha512-Snk314V5ayFLhp3fkUREub6WtjBfPdCPY1Ln8/8munuLuiYhsABgBVWsozAG+MWMbVEvcdcpbi9R7ww22l9Q3g==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=18" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, "node_modules/merge2": { "version": "1.4.1", "resolved": "https://registry.npmjs.org/merge2/-/merge2-1.4.1.tgz", @@ -9693,6 +10642,37 @@ "dev": true, "license": "MIT" }, + "node_modules/negotiator": { + "version": "1.1.0", + "resolved": "https://registry.npmjs.org/negotiator/-/negotiator-1.1.0.tgz", + "integrity": "sha512-NMPBRMJgiQHjbd8phG3Vebdx4kZ1H121rbl5IkMqeOsahptB9BKo/d7oJ3zTXqTgagn2bWlNSXkh0QUGM31RYg==", + "dev": true, + "license": "MIT", + "dependencies": { + "content-type": "^2.1.0" + }, + "engines": { + "node": ">=18" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/negotiator/node_modules/content-type": { + "version": "2.1.0", + "resolved": "https://registry.npmjs.org/content-type/-/content-type-2.1.0.tgz", + "integrity": "sha512-mj7UPXE0jaqaOsukNZRUEfEi2AcL7C/vwmwcHV0O97eO1E1pxBZuyjlZrx5seTaNBg1U6+o35wpa35Qfcc+7ag==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=18" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, "node_modules/node-fetch": { "version": "2.7.0", "resolved": "https://registry.npmjs.org/node-fetch/-/node-fetch-2.7.0.tgz", @@ -9783,6 +10763,19 @@ "node": ">= 6" } }, + "node_modules/object-inspect": { + "version": "1.13.4", + "resolved": "https://registry.npmjs.org/object-inspect/-/object-inspect-1.13.4.tgz", + "integrity": "sha512-W67iLl4J2EXEGTbfeHCffrjDfitvLANg0UlX3wFUUSTx92KXRFegMHUVgSqE+wvhAbi4WqjGg9czysTV2Epbew==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, "node_modules/obug": { "version": "2.1.4", "resolved": "https://registry.npmjs.org/obug/-/obug-2.1.4.tgz", @@ -9797,6 +10790,29 @@ "node": ">=12.20.0" } }, + "node_modules/on-finished": { + "version": "2.4.1", + "resolved": "https://registry.npmjs.org/on-finished/-/on-finished-2.4.1.tgz", + "integrity": "sha512-oVlzkg3ENAhCk2zdv7IJwd/QUD4z2RxRwpkcGY8psCVcCYZNq4wYnVWALHM+brtuJjePWiYF/ClmuDr8Ch5+kg==", + "dev": true, + "license": "MIT", + "dependencies": { + "ee-first": "1.1.1" + }, + "engines": { + "node": ">= 0.8" + } + }, + "node_modules/once": { + "version": "1.4.0", + "resolved": "https://registry.npmjs.org/once/-/once-1.4.0.tgz", + "integrity": "sha512-lNaJgI+2Q5URQBkccEKHTQOPaXdUxnZZElQTZY0MFUAuaEqe1E+Nyvgdz/aIyNi6Z9MzO5dv1H8n58/GELp3+w==", + "dev": true, + "license": "ISC", + "dependencies": { + "wrappy": "1" + } + }, "node_modules/open": { "version": "8.4.2", "resolved": "https://registry.npmjs.org/open/-/open-8.4.2.tgz", @@ -9949,6 +10965,16 @@ "url": "https://github.com/inikulin/parse5?sponsor=1" } }, + "node_modules/parseurl": { + "version": "1.3.3", + "resolved": "https://registry.npmjs.org/parseurl/-/parseurl-1.3.3.tgz", + "integrity": "sha512-CiyeOxFT/JZyN5m0z9PfXw4SCBJ6Sygz1Dpl0wqjlhDEGGBP1GnsUVEL0p63hoG1fcj3fHynXi9NYO4nWOL+qQ==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">= 0.8" + } + }, "node_modules/path-exists": { "version": "4.0.0", "resolved": "https://registry.npmjs.org/path-exists/-/path-exists-4.0.0.tgz", @@ -9999,6 +11025,17 @@ "url": "https://github.com/sponsors/isaacs" } }, + "node_modules/path-to-regexp": { + "version": "8.4.2", + "resolved": "https://registry.npmjs.org/path-to-regexp/-/path-to-regexp-8.4.2.tgz", + "integrity": "sha512-qRcuIdP69NPm4qbACK+aDogI5CBDMi1jKe0ry5rSQJz8JVLsC7jV8XpiJjGRLLol3N+R5ihGYcrPLTno6pAdBA==", + "dev": true, + "license": "MIT", + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, "node_modules/pathe": { "version": "2.0.3", "resolved": "https://registry.npmjs.org/pathe/-/pathe-2.0.3.tgz", @@ -10147,29 +11184,36 @@ "node": ">= 6" } }, + "node_modules/pkce-challenge": { + "version": "5.0.1", + "resolved": "https://registry.npmjs.org/pkce-challenge/-/pkce-challenge-5.0.1.tgz", + "integrity": "sha512-wQ0b/W4Fr01qtpHlqSqspcj3EhBvimsdh0KlHhH8HRZnMsEa0ea2fTULOXOS9ccQr3om+GcGRk4e+isrZWV8qQ==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=16.20.0" + } + }, "node_modules/playwright": { - "version": "1.62.1", - "resolved": "https://registry.npmjs.org/playwright/-/playwright-1.62.1.tgz", - "integrity": "sha512-0M+L3LAD8/nm554LOla9Ayx0j0tmFZ0FBcoQ7F1VuVHpM/XpiC8RcDzBQB8W5+hA8L22THxELzeF+2WcUzvcLg==", + "version": "1.63.0", + "resolved": "https://registry.npmjs.org/playwright/-/playwright-1.63.0.tgz", + "integrity": "sha512-+7ziBLidS4NaNCdt57SUDT+wYmmd5fmiQejUic/kb+YsYSCPyOOE9sebzMjNmQrsnNpDJqd4WHvV/8lfKfUDUg==", "dev": true, "license": "Apache-2.0", "dependencies": { - "playwright-core": "1.62.1" + "playwright-core": "1.63.0" }, "bin": { "playwright": "cli.js" }, "engines": { "node": ">=20" - }, - "optionalDependencies": { - "fsevents": "2.3.2" } }, "node_modules/playwright-core": { - "version": "1.62.1", - "resolved": "https://registry.npmjs.org/playwright-core/-/playwright-core-1.62.1.tgz", - "integrity": "sha512-wPYSwEBJY9GHraISXqyqtx0na0LpO3XEX7jNDhntbex7tzUS7kLnZsOlFruFJB4Hi/rhDMjXGqHewDZ68nYZVw==", + "version": "1.63.0", + "resolved": "https://registry.npmjs.org/playwright-core/-/playwright-core-1.63.0.tgz", + "integrity": "sha512-rYCsBF/M5HjUch52bbtVONEFjv6Xu8sm8h72dNlR5bzIE1fvC/bxgspzkjSfU+MweEMmPM8KJebG6nnyxo5mCg==", "dev": true, "license": "Apache-2.0", "bin": { @@ -10211,6 +11255,16 @@ "node": ">=8.0" } }, + "node_modules/pngjs": { + "version": "7.0.0", + "resolved": "https://registry.npmjs.org/pngjs/-/pngjs-7.0.0.tgz", + "integrity": "sha512-LKWqWJRhstyYo9pGvgor/ivk2w94eSjE3RGVuzLGlr3NmD8bf7RcYGze1mNdEHRP6TRP6rMuDHk5t44hnTRyow==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=14.19.0" + } + }, "node_modules/popper.js": { "version": "1.16.1", "resolved": "https://registry.npmjs.org/popper.js/-/popper.js-1.16.1.tgz", @@ -10747,6 +11801,24 @@ "prosemirror-transform": "^1.1.0" } }, + "node_modules/proxy-addr": { + "version": "2.0.8", + "resolved": "https://registry.npmjs.org/proxy-addr/-/proxy-addr-2.0.8.tgz", + "integrity": "sha512-5nnx0yGyVUcY6t9RnWcARWtwT9F1D8O9rt08htPvnd49W1IgZtmLkhu9WfMzQj1cFxjHIO6connUNVW5k7AVyQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "forwarded": "0.2.0", + "ipaddr.js": "1.9.1" + }, + "engines": { + "node": ">= 0.10" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, "node_modules/proxy-from-env": { "version": "1.1.0", "resolved": "https://registry.npmjs.org/proxy-from-env/-/proxy-from-env-1.1.0.tgz", @@ -10777,6 +11849,23 @@ "node": ">=6" } }, + "node_modules/qs": { + "version": "6.16.0", + "resolved": "https://registry.npmjs.org/qs/-/qs-6.16.0.tgz", + "integrity": "sha512-h6fhOIaRrID2CbEY2fqs+7t+UXZo+MLAnU5gRIq85uFtdiUPCdsApMlHhXogKVM4HM2DVbIjGNTTYH2OcmP1vA==", + "dev": true, + "license": "BSD-3-Clause", + "dependencies": { + "es-define-property": "^1.0.1", + "side-channel": "^1.1.1" + }, + "engines": { + "node": ">=0.6" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, "node_modules/query-selector-shadow-dom": { "version": "1.0.1", "resolved": "https://registry.npmjs.org/query-selector-shadow-dom/-/query-selector-shadow-dom-1.0.1.tgz", @@ -10810,14 +11899,61 @@ ], "license": "MIT" }, - "node_modules/raf": { - "version": "3.4.1", - "resolved": "https://registry.npmjs.org/raf/-/raf-3.4.1.tgz", - "integrity": "sha512-Sq4CW4QhwOHE8ucn6J34MqtZCeWFP2aQSmrlroYgqAV1PjStIhJXxYuTgUIfkEk7zTLjmIjLmU5q+fbD1NnOJA==", + "node_modules/raf": { + "version": "3.4.1", + "resolved": "https://registry.npmjs.org/raf/-/raf-3.4.1.tgz", + "integrity": "sha512-Sq4CW4QhwOHE8ucn6J34MqtZCeWFP2aQSmrlroYgqAV1PjStIhJXxYuTgUIfkEk7zTLjmIjLmU5q+fbD1NnOJA==", + "license": "MIT", + "optional": true, + "dependencies": { + "performance-now": "^2.1.0" + } + }, + "node_modules/range-parser": { + "version": "1.3.0", + "resolved": "https://registry.npmjs.org/range-parser/-/range-parser-1.3.0.tgz", + "integrity": "sha512-hek2mFQpPuI4E1BBKrSto+BU3e3x4xuarsbiwr3+lf7p44juvFMV0XFWQAP3xUyqXA4RrXLIoaSUGbSt056ZMw==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">= 0.6" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/raw-body": { + "version": "3.0.2", + "resolved": "https://registry.npmjs.org/raw-body/-/raw-body-3.0.2.tgz", + "integrity": "sha512-K5zQjDllxWkf7Z5xJdV0/B0WTNqx6vxG70zJE4N0kBs4LovmEYWJzQGxC9bS9RAKu3bgM40lrd5zoLJ12MQ5BA==", + "dev": true, "license": "MIT", - "optional": true, "dependencies": { - "performance-now": "^2.1.0" + "bytes": "~3.1.2", + "http-errors": "~2.0.1", + "iconv-lite": "~0.7.0", + "unpipe": "~1.0.0" + }, + "engines": { + "node": ">= 0.10" + } + }, + "node_modules/raw-body/node_modules/iconv-lite": { + "version": "0.7.3", + "resolved": "https://registry.npmjs.org/iconv-lite/-/iconv-lite-0.7.3.tgz", + "integrity": "sha512-IKXpvIzjnC9XTAUbVBcMfGS0EPaIXtW6v+zr+RRp+hqULEpo0owZax6wyRwPOJbWbzjYspQwusTsfVr0ifh4uQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "safer-buffer": ">= 2.1.2 < 3.0.0" + }, + "engines": { + "node": ">=0.10.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" } }, "node_modules/react": { @@ -11282,6 +12418,16 @@ "url": "https://opencollective.com/unified" } }, + "node_modules/require-from-string": { + "version": "2.0.2", + "resolved": "https://registry.npmjs.org/require-from-string/-/require-from-string-2.0.2.tgz", + "integrity": "sha512-Xf0nWe6RseziFMu+Ap9biiUbmplq6S9/p+7w7YXP/JBHhrUDDUhwa+vANyubuqfZWTveU//DYVGsDG7RKL/vEw==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=0.10.0" + } + }, "node_modules/requires-port": { "version": "1.0.0", "resolved": "https://registry.npmjs.org/requires-port/-/requires-port-1.0.0.tgz", @@ -11421,6 +12567,23 @@ "integrity": "sha512-UT5EDe2cu2E/6O4igUr5PSFs23nvvukicWHx6GnOPlHAiiYbzNuCRQCuiUdHJQcqKalLKlrYJnjY0ySGsXNQXQ==", "license": "MIT" }, + "node_modules/router": { + "version": "2.2.0", + "resolved": "https://registry.npmjs.org/router/-/router-2.2.0.tgz", + "integrity": "sha512-nLTrUKm2UyiL7rlhapu/Zl45FwNgkZGaCpZbIHajDYgwlJCOzLSk+cIPAnsEqV955GjILJnKbdQC1nVPz+gAYQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "debug": "^4.4.0", + "depd": "^2.0.0", + "is-promise": "^4.0.0", + "parseurl": "^1.3.3", + "path-to-regexp": "^8.0.0" + }, + "engines": { + "node": ">= 18" + } + }, "node_modules/run-parallel": { "version": "1.2.0", "resolved": "https://registry.npmjs.org/run-parallel/-/run-parallel-1.2.0.tgz", @@ -11509,6 +12672,80 @@ "node": ">=10" } }, + "node_modules/send": { + "version": "1.2.1", + "resolved": "https://registry.npmjs.org/send/-/send-1.2.1.tgz", + "integrity": "sha512-1gnZf7DFcoIcajTjTwjwuDjzuz4PPcY2StKPlsGAQ1+YH20IRVrBaXSWmdjowTJ6u8Rc01PoYOGHXfP1mYcZNQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "debug": "^4.4.3", + "encodeurl": "^2.0.0", + "escape-html": "^1.0.3", + "etag": "^1.8.1", + "fresh": "^2.0.0", + "http-errors": "^2.0.1", + "mime-types": "^3.0.2", + "ms": "^2.1.3", + "on-finished": "^2.4.1", + "range-parser": "^1.2.1", + "statuses": "^2.0.2" + }, + "engines": { + "node": ">= 18" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/send/node_modules/mime-db": { + "version": "1.54.0", + "resolved": "https://registry.npmjs.org/mime-db/-/mime-db-1.54.0.tgz", + "integrity": "sha512-aU5EJuIN2WDemCcAp2vFBfp/m4EAhWJnUNSSw0ixs7/kXbd6Pg64EmwJkNdFhB8aWt1sH2CTXrLxo/iAGV3oPQ==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/send/node_modules/mime-types": { + "version": "3.0.2", + "resolved": "https://registry.npmjs.org/mime-types/-/mime-types-3.0.2.tgz", + "integrity": "sha512-Lbgzdk0h4juoQ9fCKXW4by0UJqj+nOOrI9MJ1sSj4nI8aI2eo1qmvQEie4VD1glsS250n15LsWsYtCugiStS5A==", + "dev": true, + "license": "MIT", + "dependencies": { + "mime-db": "^1.54.0" + }, + "engines": { + "node": ">=18" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/serve-static": { + "version": "2.2.1", + "resolved": "https://registry.npmjs.org/serve-static/-/serve-static-2.2.1.tgz", + "integrity": "sha512-xRXBn0pPqQTVQiC8wyQrKs2MOlX24zQ0POGaj0kultvoOCstBQM5yvOhAVSUwOMjQtTvsPWoNCHfPGwaaQJhTw==", + "dev": true, + "license": "MIT", + "dependencies": { + "encodeurl": "^2.0.0", + "escape-html": "^1.0.3", + "parseurl": "^1.3.3", + "send": "^1.2.0" + }, + "engines": { + "node": ">= 18" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, "node_modules/set-cookie-parser": { "version": "2.7.2", "resolved": "https://registry.npmjs.org/set-cookie-parser/-/set-cookie-parser-2.7.2.tgz", @@ -11521,6 +12758,13 @@ "integrity": "sha512-MATJdZp8sLqDl/68LfQmbP8zKPLQNV6BIZoIgrscFDQ+RsvK/BxeDQOgyxKKoh0y/8h3BqVFnCqQ/gd+reiIXA==", "license": "MIT" }, + "node_modules/setprototypeof": { + "version": "1.2.0", + "resolved": "https://registry.npmjs.org/setprototypeof/-/setprototypeof-1.2.0.tgz", + "integrity": "sha512-E5LDX7Wrp85Kil5bhZv46j8jOeboKq5JMmYM3gVGdGH8xFpPWXUMsNrlODCrkoxMEeNi/XZIwuRvY4XNwYMJpw==", + "dev": true, + "license": "ISC" + }, "node_modules/shebang-command": { "version": "2.0.0", "resolved": "https://registry.npmjs.org/shebang-command/-/shebang-command-2.0.0.tgz", @@ -11542,6 +12786,82 @@ "node": ">=8" } }, + "node_modules/side-channel": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/side-channel/-/side-channel-1.1.1.tgz", + "integrity": "sha512-6x6dK6zJdpTzF4sQeNYxwtvBzf6Eg4GtlesS94HOvTudUeyK2WXAaIfmDgsyslYrRBeFIlsi54AYsFGUuhmvrQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "es-errors": "^1.3.0", + "object-inspect": "^1.13.4", + "side-channel-list": "^1.0.1", + "side-channel-map": "^1.0.1", + "side-channel-weakmap": "^1.0.2" + }, + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/side-channel-list": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/side-channel-list/-/side-channel-list-1.0.1.tgz", + "integrity": "sha512-mjn/0bi/oUURjc5Xl7IaWi/OJJJumuoJFQJfDDyO46+hBWsfaVM65TBHq2eoZBhzl9EchxOijpkbRC8SVBQU0w==", + "dev": true, + "license": "MIT", + "dependencies": { + "es-errors": "^1.3.0", + "object-inspect": "^1.13.4" + }, + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/side-channel-map": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/side-channel-map/-/side-channel-map-1.0.1.tgz", + "integrity": "sha512-VCjCNfgMsby3tTdo02nbjtM/ewra6jPHmpThenkTYh8pG9ucZ/1P8So4u4FGBek/BjpOVsDCMoLA/iuBKIFXRA==", + "dev": true, + "license": "MIT", + "dependencies": { + "call-bound": "^1.0.2", + "es-errors": "^1.3.0", + "get-intrinsic": "^1.2.5", + "object-inspect": "^1.13.3" + }, + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/side-channel-weakmap": { + "version": "1.0.2", + "resolved": "https://registry.npmjs.org/side-channel-weakmap/-/side-channel-weakmap-1.0.2.tgz", + "integrity": "sha512-WPS/HvHQTYnHisLo9McqBHOJk2FkHO/tlpvldyrnem4aeQp4hai3gythswg6p01oSoTl58rcpiFAjF2br2Ak2A==", + "dev": true, + "license": "MIT", + "dependencies": { + "call-bound": "^1.0.2", + "es-errors": "^1.3.0", + "get-intrinsic": "^1.2.5", + "object-inspect": "^1.13.3", + "side-channel-map": "^1.0.1" + }, + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, "node_modules/siginfo": { "version": "2.0.0", "resolved": "https://registry.npmjs.org/siginfo/-/siginfo-2.0.0.tgz", @@ -11679,6 +12999,16 @@ "node": ">=0.1.14" } }, + "node_modules/statuses": { + "version": "2.0.2", + "resolved": "https://registry.npmjs.org/statuses/-/statuses-2.0.2.tgz", + "integrity": "sha512-DvEy55V3DB7uknRo+4iOGT5fP1slR8wQohVdknigZPMpMstaKJQWhwiYBACJE3Ul2pTnATihhBYnRhZQHGBiRw==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">= 0.8" + } + }, "node_modules/std-env": { "version": "4.2.0", "resolved": "https://registry.npmjs.org/std-env/-/std-env-4.2.0.tgz", @@ -12090,6 +13420,16 @@ "node": ">=8.0" } }, + "node_modules/toidentifier": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/toidentifier/-/toidentifier-1.0.1.tgz", + "integrity": "sha512-o5sSPKEkg/DIQNmH43V0/uerLrpzVedkUh8tGNvaeXpfpuwjKenlSox/2O/BTlZUtEe+JG7s5YhEz608PlAHRA==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=0.6" + } + }, "node_modules/tough-cookie": { "version": "4.1.4", "resolved": "https://registry.npmjs.org/tough-cookie/-/tough-cookie-4.1.4.tgz", @@ -12637,20 +13977,6 @@ "@esbuild/win32-x64": "0.23.1" } }, - "node_modules/tsx/node_modules/fsevents": { - "version": "2.3.3", - "resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.3.3.tgz", - "integrity": "sha512-5xoDfX+fL7faATnagmWPpbFtwh/R77WmMMqqHGS65C3vvB0YHrgF+B1YmZ3441tMj5n63k0212XNoJwzlhffQw==", - "hasInstallScript": true, - "license": "MIT", - "optional": true, - "os": [ - "darwin" - ], - "engines": { - "node": "^8.16.0 || ^10.6.0 || >=11.0.0" - } - }, "node_modules/type-check": { "version": "0.4.0", "resolved": "https://registry.npmjs.org/type-check/-/type-check-0.4.0.tgz", @@ -12676,6 +14002,66 @@ "url": "https://github.com/sponsors/sindresorhus" } }, + "node_modules/type-is": { + "version": "2.1.0", + "resolved": "https://registry.npmjs.org/type-is/-/type-is-2.1.0.tgz", + "integrity": "sha512-faYHw0anBbc/kWF3zFTEnxSFOAGUX9GFbOBthvDdLsIlEoWOFOtS0zgCiQYwIskL9iGXZL3kAXD8OoZ4GmMATA==", + "dev": true, + "license": "MIT", + "dependencies": { + "content-type": "^2.0.0", + "media-typer": "^1.1.0", + "mime-types": "^3.0.0" + }, + "engines": { + "node": ">= 18" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/type-is/node_modules/content-type": { + "version": "2.1.0", + "resolved": "https://registry.npmjs.org/content-type/-/content-type-2.1.0.tgz", + "integrity": "sha512-mj7UPXE0jaqaOsukNZRUEfEi2AcL7C/vwmwcHV0O97eO1E1pxBZuyjlZrx5seTaNBg1U6+o35wpa35Qfcc+7ag==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=18" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/type-is/node_modules/mime-db": { + "version": "1.54.0", + "resolved": "https://registry.npmjs.org/mime-db/-/mime-db-1.54.0.tgz", + "integrity": "sha512-aU5EJuIN2WDemCcAp2vFBfp/m4EAhWJnUNSSw0ixs7/kXbd6Pg64EmwJkNdFhB8aWt1sH2CTXrLxo/iAGV3oPQ==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/type-is/node_modules/mime-types": { + "version": "3.0.2", + "resolved": "https://registry.npmjs.org/mime-types/-/mime-types-3.0.2.tgz", + "integrity": "sha512-Lbgzdk0h4juoQ9fCKXW4by0UJqj+nOOrI9MJ1sSj4nI8aI2eo1qmvQEie4VD1glsS250n15LsWsYtCugiStS5A==", + "dev": true, + "license": "MIT", + "dependencies": { + "mime-db": "^1.54.0" + }, + "engines": { + "node": ">=18" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, "node_modules/typescript": { "version": "5.9.3", "resolved": "https://registry.npmjs.org/typescript/-/typescript-5.9.3.tgz", @@ -12720,6 +14106,16 @@ "integrity": "sha512-DXtD3ZtEQzc7M8m4cXotyHR+FAS18C64asBYY5vqZexfYryNNnDc02W4hKg3rdQuqOYas1jkseX0+nZXjTXnvQ==", "license": "MIT" }, + "node_modules/undici": { + "version": "7.30.0", + "resolved": "https://registry.npmjs.org/undici/-/undici-7.30.0.tgz", + "integrity": "sha512-dkrQXeHSaoamnItlYbmzG0wFYrM0ZwDxCIg0A7aKjTyyhh9svRzCNFEzV+Vm05/yehjCzjDZ31KXfGEjYSztDQ==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=20.18.1" + } + }, "node_modules/undici-types": { "version": "6.21.0", "resolved": "https://registry.npmjs.org/undici-types/-/undici-types-6.21.0.tgz", @@ -12822,6 +14218,16 @@ "node": ">= 10.0.0" } }, + "node_modules/unpipe": { + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/unpipe/-/unpipe-1.0.0.tgz", + "integrity": "sha512-pjy2bYhSsufwWlKwPc+l3cN7+wuJlK6uz0YdJEOlQDbl6jo/YlPi4mb8agUkVC8BF7V8NuzeyPNqRksA3hztKQ==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">= 0.8" + } + }, "node_modules/untildify": { "version": "4.0.0", "resolved": "https://registry.npmjs.org/untildify/-/untildify-4.0.0.tgz", @@ -12964,6 +14370,16 @@ "uuid": "dist/esm/bin/uuid" } }, + "node_modules/vary": { + "version": "1.1.2", + "resolved": "https://registry.npmjs.org/vary/-/vary-1.1.2.tgz", + "integrity": "sha512-BNGbWLfd0eUPabhkXUVm0j8uuvREyTh5ovRa/dyow/BqAbZJyC+5fU+IzQOzmAKzYqYRAISoRhdQr3eIZ/PXqg==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">= 0.8" + } + }, "node_modules/vfile": { "version": "6.0.3", "resolved": "https://registry.npmjs.org/vfile/-/vfile-6.0.3.tgz", @@ -13107,21 +14523,6 @@ } } }, - "node_modules/vite/node_modules/fsevents": { - "version": "2.3.3", - "resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.3.3.tgz", - "integrity": "sha512-5xoDfX+fL7faATnagmWPpbFtwh/R77WmMMqqHGS65C3vvB0YHrgF+B1YmZ3441tMj5n63k0212XNoJwzlhffQw==", - "dev": true, - "hasInstallScript": true, - "license": "MIT", - "optional": true, - "os": [ - "darwin" - ], - "engines": { - "node": "^8.16.0 || ^10.6.0 || >=11.0.0" - } - }, "node_modules/vite/node_modules/picomatch": { "version": "4.0.7", "resolved": "https://registry.npmjs.org/picomatch/-/picomatch-4.0.7.tgz", @@ -13386,6 +14787,13 @@ "url": "https://github.com/chalk/wrap-ansi?sponsor=1" } }, + "node_modules/wrappy": { + "version": "1.0.2", + "resolved": "https://registry.npmjs.org/wrappy/-/wrappy-1.0.2.tgz", + "integrity": "sha512-l4Sp/DRseor9wL6EvV2+TuQn63dMkPjZ/sp9XkghTEbV9KlPS1xUsZ3u7/IQO4wxtcFB4bgpQPRcR3QCvezPcQ==", + "dev": true, + "license": "ISC" + }, "node_modules/ws": { "version": "8.21.3", "resolved": "https://registry.npmjs.org/ws/-/ws-8.21.3.tgz", @@ -13575,14 +14983,24 @@ } }, "node_modules/zod": { - "version": "4.5.4", - "resolved": "https://registry.npmjs.org/zod/-/zod-4.5.4.tgz", - "integrity": "sha512-sC95tT5iHHH9gtpj6A81kh+NEaRAUFN+qlUPDUbRfOMvNf5QCBqsb3WgvnpVtK5Y+4UfA6KqufotuTvMGiTlsA==", + "version": "4.6.1", + "resolved": "https://registry.npmjs.org/zod/-/zod-4.6.1.tgz", + "integrity": "sha512-341aRWQsve0rvronKNTqZpjmzdbUDlFuzHaI/XLg/Ej82qffDJRRfBTCuv7+9q/rMjB6LSLyEBnW4InJeMtt/Q==", "license": "MIT", "funding": { "url": "https://github.com/sponsors/colinhacks" } }, + "node_modules/zod-to-json-schema": { + "version": "3.25.2", + "resolved": "https://registry.npmjs.org/zod-to-json-schema/-/zod-to-json-schema-3.25.2.tgz", + "integrity": "sha512-O/PgfnpT1xKSDeQYSCfRI5Gy3hPf91mKVDuYLUHZJMiDFptvP41MSnWofm8dnCm0256ZNfZIM7DSzuSMAFnjHA==", + "dev": true, + "license": "ISC", + "peerDependencies": { + "zod": "^3.25.28 || ^4" + } + }, "node_modules/zustand": { "version": "5.0.15", "resolved": "https://registry.npmjs.org/zustand/-/zustand-5.0.15.tgz", diff --git a/package.json b/package.json index 88a6baa46..55004e120 100644 --- a/package.json +++ b/package.json @@ -17,7 +17,7 @@ "ios:build": "vite build --mode capacitor && cap sync ios", "ios:open": "cap open ios", "lint": "eslint .", - "typecheck": "tsc --noEmit -p tsconfig.app.json", + "typecheck": "tsc --noEmit -p tsconfig.app.json && tsc --noEmit -p tsconfig.agentic.json", "preview": "vite preview", "smoke:jev": "node scripts/smoke-jev.mjs", "test": "vitest run", @@ -31,6 +31,7 @@ "test:e2e:webkit": "playwright test --project=webkit", "test:sw": "node scripts/test-service-worker-offline.mjs", "test:e2e:staging": "playwright test --project=staging", + "test:agentic": "bash .claude/skills/verify-debatelab/bin/verify.sh e2e run", "verify": "npm run docs:check && npm run lint && npm run typecheck && npm test && npm run test:collaboration && npm run test:edge && npm run build && npm run test:sw && P=$(bash .superset/e2e-port.sh) && CI=1 PORT=$P PLAYWRIGHT_BASE_URL=http://127.0.0.1:$P npm run test:e2e:smoke", "verify:full": "npm run verify && P=$(bash .superset/e2e-port.sh) && CI=1 PORT=$P PLAYWRIGHT_BASE_URL=http://127.0.0.1:$P npm run test:e2e" }, @@ -108,8 +109,10 @@ "zustand": "^5.0.10" }, "devDependencies": { + "@e2e-dev/web": "0.11.1", "@eslint/js": "^9.32.0", - "@playwright/test": "^1.60.0", + "@openrouter/ai-sdk-provider": "^3.1.0", + "@playwright/test": "^1.63.0", "@sentry/vite-plugin": "^5.4.0", "@tailwindcss/typography": "^0.5.16", "@testing-library/jest-dom": "^6.6.0", @@ -120,7 +123,9 @@ "@types/react-dom": "^18.3.7", "@vitejs/plugin-react-swc": "^3.11.0", "@vitest/coverage-v8": "^4.1.8", + "ai": "^7.0.127", "autoprefixer": "^10.4.21", + "e2e": "0.15.2", "eslint": "^9.32.0", "eslint-plugin-react-hooks": "^5.2.0", "eslint-plugin-react-refresh": "^0.4.20", diff --git a/plans/2026-10-02-missing-translation-keys.md b/plans/2026-10-02-missing-translation-keys.md new file mode 100644 index 000000000..a043e0b95 --- /dev/null +++ b/plans/2026-10-02-missing-translation-keys.md @@ -0,0 +1,35 @@ +# Missing translation keys + +**Status:** Backlog. Found on 2026-10-02 by the agentic Hungarian checks (`agentic/hungarian.e2e.ts`). Recheck the list against current source before starting. + +## Problem + +Components call `t('some.key', 'English fallback')`, but 88 of those literal keys are in no catalog. i18next then renders the inline English fallback, so Spanish and Hungarian users see English in toasts, confirmation dialogs, Library folder actions, Flowchart controls, the Google Drive dialog, and more. `src/test/i18n.test.ts` cannot catch this: it compares key sets between catalogs, and these keys are missing from all three. + +The agentic suite found the first four on every main page (`themeToggle.toggle`, `sparring.setup.speakingOrder`, `modelSelector.label`, `modelSelector.selectFor`), and they are fixed. A scan of `src/` for literal `t()` keys absent from `src/locales/en.json` (plural `_one`/`_other` forms count as present) finds the rest. + +## Plan + +1. Add a unit test next to `src/test/i18n.test.ts` that scans `src/` (excluding `src/test/`) for literal `t('a.b')` keys and fails when one is missing from `en.json`, allowing plural suffixes. Dynamic keys such as ``t(`nav.${x}`)`` are out of its reach. +2. Add every missing key to `en.json` with the current fallback text, then to `es.json` and `hu.json` with real translations. Keep the inline fallbacks or drop them, but do it consistently. +3. Verify with the unit test, `e2e/hungarian-parity.spec.ts`, and `verify.sh e2e run agentic/hungarian.e2e.ts`. For toasts and dialogs the main-page checks never open, also run `verify.sh e2e explore "Use the Library, History and Flowchart and report any English text" --session student --agent hungarian`. + +## Missing keys (88, listed per file) + +- `src/components/OfflineIndicator.tsx`: `offlineIndicator.label` +- `src/components/ThemeToggle.tsx`: `themeToggle.current` +- `src/components/features/ExportToolbar.tsx`: `common.discuss` +- `src/components/features/ask/AskChat.tsx`: `askChat.save` +- `src/components/features/brainstorm/BrainstormResults.tsx`: `brainstormResults.noIdeasDesc`, `brainstormResults.noIdeasTitle` +- `src/components/features/export/GoogleDriveAuthModal.tsx`: `googleDriveAuth.connect`, `googleDriveAuth.description`, `googleDriveAuth.failedDesc`, `googleDriveAuth.failedTitle`, `googleDriveAuth.redirecting`, `googleDriveAuth.scopeNote`, `googleDriveAuth.title` +- `src/components/features/factoids/FactoidsResults.tsx`: `factoidsResults.noVerified`, `factoidsResults.useSuggestion` +- `src/components/features/generate/GenerateResults.tsx`: `generateResults.copiedToClipboard`, `generateResults.copyFailed`, `generateResults.undo` +- `src/pages/AskPage.tsx`: `ask.chatSaved`, `ask.chatSavedDesc`, `ask.retryableFailedDesc`, `ask.saveFailed`, `ask.saveFailedDesc`, `common.retry` +- `src/pages/AttackPage.tsx`: `attack.complete`, `attack.completeDesc` +- `src/pages/BrainstormPage.tsx`: `brainstorm.complete`, `brainstorm.completeDesc` +- `src/pages/FactoidsPage.tsx`: `factoids.complete`, `factoids.completeDesc`, `factoids.saveFailed` +- `src/pages/FlowchartPage.tsx`: `flowchart.autoLayout`, `flowchart.edgeAttack`, `flowchart.edgeRebuttal`, `flowchart.edgeSupport`, `flowchart.exportFailedToast`, `flowchart.selectedNode` +- `src/pages/GeneratePage.tsx`: `ask.saveFailed`, `ask.saveFailedDesc`, `generate.caseTitle`, `generate.complete`, `generate.completeDesc`, `generate.noUndo`, `generate.regenerateUnavailable`, `generate.regenerateUnavailableDesc`, `generate.regenerated`, `generate.regeneratedDesc`, `generate.saveCase`, `generate.undone`, `generate.undoneDesc` +- `src/pages/HistoryPage.tsx`: `common.deleted`, `historyPage.caseDeleted`, `historyPage.deleteCaseDesc`, `historyPage.deleteCaseTitle`, `historyPage.deleteFailed` +- `src/pages/LibraryPage.tsx`: `common.clear`, `common.copied`, `common.copy`, `common.deleted`, `libraryPage.activateFolder`, `libraryPage.clearSuccess`, `libraryPage.copyFailed`, `libraryPage.deactivateFolder`, `libraryPage.deleteFolderDesc`, `libraryPage.deleteFolderTitle`, `libraryPage.deleteSourceDesc`, `libraryPage.deleteSourceTitle`, `libraryPage.folderActions`, `libraryPage.folderActionsDesc`, `libraryPage.folderActivated`, `libraryPage.folderActivatedDesc`, `libraryPage.folderDeactivated`, `libraryPage.folderDeactivatedDesc`, `libraryPage.folderDeleted`, `libraryPage.folderDeletedDesc`, `libraryPage.folderMergeConfirm`, `libraryPage.folderRenamed`, `libraryPage.folderRenamedDesc`, `libraryPage.noMatchingSources`, `libraryPage.noMatchingSourcesDesc`, `libraryPage.renameFolder`, `libraryPage.sourceCopied`, `libraryPage.sourceDeleted`, `libraryPage.sourceUpdated`, `libraryPage.sourceUpdatedDesc`, `libraryPage.typeLabel`, `libraryPage.untitledSource`, `libraryPage.useForGrounding`, `libraryPage.useForGroundingDesc` +- `src/pages/TimerPage.tsx`: `timer.disableSound`, `timer.enableSound` diff --git a/plans/README.md b/plans/README.md index 4c1068380..d8afe2944 100644 --- a/plans/README.md +++ b/plans/README.md @@ -10,6 +10,7 @@ Nothing is in progress. ## Backlog +- **Missing translation keys.** 88 literal `t()` keys are in no catalog, so Spanish and Hungarian users see English in toasts, dialogs and Library actions. Plan: [2026-10-02 missing translation keys](2026-10-02-missing-translation-keys.md). - **Next 15.** Security and data protection, operational safety, code health, and product polish, ordered by risk. Items 3, 5, 6, 8 and 13 have shipped; the plan lists the follow-ups they left. Plan: [2026-10-01 next 15](2026-10-01-next-15.md). Item 15 covers the PR #63 decision. - **Leaner CI and two previews per PR.** Three-minute PR check, a staging Vercel project for per-PR staging previews, and Edge Function CORS for preview hosts. Plan: [2026-10-01 leaner CI and previews](2026-10-01-ci-and-preview-verification.md). diff --git a/src/components/ModelSelector.tsx b/src/components/ModelSelector.tsx index 86423ded4..6e0149c65 100644 --- a/src/components/ModelSelector.tsx +++ b/src/components/ModelSelector.tsx @@ -65,7 +65,7 @@ export function ModelSelector({ feature, className, showLabel = false }: ModelSe } }} > - <SelectTrigger className="h-10 w-full text-sm" aria-label={t('modelSelector.selectFor', 'Select AI model for {{feature}}', { feature })}> + <SelectTrigger className="h-10 w-full text-sm" aria-label={t('modelSelector.selectFor', 'Select AI model for {{feature}}', { feature: t(`nav.${feature}`, feature).toLowerCase() })}> <SelectValue /> </SelectTrigger> <SelectContent> diff --git a/src/locales/en.json b/src/locales/en.json index 2af16c88d..431e9a2ca 100644 --- a/src/locales/en.json +++ b/src/locales/en.json @@ -258,7 +258,9 @@ "reasoning": "Reasoning", "reasoningLow": "Low", "reasoningMedium": "Medium", - "reasoningHigh": "High" + "reasoningHigh": "High", + "label": "AI model", + "selectFor": "Select AI model for {{feature}}" }, "voice": { "permissionDenied": "Microphone access is blocked. Allow microphone access in your browser settings and try again.", @@ -1853,7 +1855,8 @@ "openingRoom": "Opening room...", "startRound": "Start round", "draftNotice": "Your draft is saved in this tab as you type. You can pause without losing your place.", - "offlineCannotStart": "Reconnect to start a round. Your setup is kept while you are offline." + "offlineCannotStart": "Reconnect to start a round. Your setup is kept while you are offline.", + "speakingOrder": "Speaking order" }, "round": { "inProgress": "Round in progress", @@ -2239,5 +2242,8 @@ "lastShortened_other": "{{count}} sources were shortened to fit.", "lastUnavailable_one": "{{count}} item was no longer in this debate.", "lastUnavailable_other": "{{count}} items were no longer in this debate." + }, + "themeToggle": { + "toggle": "Toggle theme" } } diff --git a/src/locales/es.json b/src/locales/es.json index 0e8b12f3b..c5b5b6a25 100644 --- a/src/locales/es.json +++ b/src/locales/es.json @@ -258,7 +258,9 @@ "reasoning": "Razonamiento", "reasoningLow": "Bajo", "reasoningMedium": "Medio", - "reasoningHigh": "Alto" + "reasoningHigh": "Alto", + "label": "Modelo de IA", + "selectFor": "Seleccionar modelo de IA para {{feature}}" }, "voice": { "permissionDenied": "El acceso al micrófono está bloqueado. Permite el acceso al micrófono en el navegador e inténtalo de nuevo.", @@ -1853,7 +1855,8 @@ "openingRoom": "Abriendo la sala...", "startRound": "Iniciar ronda", "draftNotice": "Tu borrador se guarda en esta pestaña mientras escribes. Puedes pausar sin perder tu lugar.", - "offlineCannotStart": "Vuelve a conectarte para iniciar una ronda. La configuración se conserva mientras estás sin conexión." + "offlineCannotStart": "Vuelve a conectarte para iniciar una ronda. La configuración se conserva mientras estás sin conexión.", + "speakingOrder": "Orden de intervención" }, "round": { "inProgress": "Ronda en curso", @@ -2239,5 +2242,8 @@ "lastShortened_other": "{{count}} fuentes se recortaron para que cupieran.", "lastUnavailable_one": "{{count}} elemento ya no estaba en este debate.", "lastUnavailable_other": "{{count}} elementos ya no estaban en este debate." + }, + "themeToggle": { + "toggle": "Cambiar tema" } } diff --git a/src/locales/hu.json b/src/locales/hu.json index 5e599e5df..bc4a1fa0b 100644 --- a/src/locales/hu.json +++ b/src/locales/hu.json @@ -258,7 +258,9 @@ "reasoning": "Érvelés", "reasoningLow": "Alacsony", "reasoningMedium": "Közepes", - "reasoningHigh": "Magas" + "reasoningHigh": "Magas", + "label": "MI-modell", + "selectFor": "MI-modell kiválasztása: {{feature}}" }, "voice": { "permissionDenied": "A mikrofonhozzáférés blokkolva van. Engedélyezd a mikrofont a böngésző beállításaiban, majd próbáld újra.", @@ -1139,7 +1141,7 @@ "sidePro": "Csak mellette", "sideCon": "Csak ellene", "outputFormat": "Kimeneti Formátum", - "outputTop5": "Top 5 Legjobb Szög", + "outputTop5": "Az 5 Legjobb Szög", "outputMindmap": "Gondolattérkép Stílus", "outputBuckets": "Kategorizált Csoportok", "existing": "Ami Már Megvan (Opcionális)", @@ -1155,7 +1157,7 @@ "brainstormResults": { "clarification": "Pontosítás Szükséges", "clarificationDesc": "Válaszolj ezekre a kérdésekre a jobb eredményekért", - "top5": "Top 5 Érvelési Szög", + "top5": "Az 5 Legerősebb Érvelési Szög", "top5Desc": "Legjobb lehetőségek potenciál szerint rangsorolva", "argMap": "Érvtérkép", "argMapDesc": "Ötletek kategória szerint rendezve", @@ -1853,7 +1855,8 @@ "openingRoom": "Szoba megnyitása...", "startRound": "Forduló indítása", "draftNotice": "A piszkozat írás közben ezen a böngészőfülön mentődik. Szüneteltetheted anélkül, hogy elveszítenéd a helyed.", - "offlineCannotStart": "Csatlakozz újra a forduló indításához. A beállításaid offline állapotban is megmaradnak." + "offlineCannotStart": "Csatlakozz újra a forduló indításához. A beállításaid offline állapotban is megmaradnak.", + "speakingOrder": "Felszólalási sorrend" }, "round": { "inProgress": "Forduló folyamatban", @@ -2239,5 +2242,8 @@ "lastShortened_other": "{{count}} forrást rövidítettünk, hogy beférjen.", "lastUnavailable_one": "{{count}} elem már nem volt ebben a vitában.", "lastUnavailable_other": "{{count}} elem már nem volt ebben a vitában." + }, + "themeToggle": { + "toggle": "Téma váltása" } } diff --git a/src/pages/SettingsPage.tsx b/src/pages/SettingsPage.tsx index 148e15e46..849893bd8 100644 --- a/src/pages/SettingsPage.tsx +++ b/src/pages/SettingsPage.tsx @@ -298,12 +298,13 @@ export default function SettingsPage() { </CardDescription> </CardHeader> <CardContent className="space-y-4"> - <div className="flex items-center justify-between"> - <div> + <div className="flex items-center justify-between gap-3"> + {/* A long email wraps instead of pushing Sign out off a phone screen. */} + <div className="min-w-0"> <p className="text-sm text-muted-foreground">{t('settings.signedInAs')}</p> - <p className="font-medium">{user?.email || t('settings.unknownUser')}</p> + <p className="font-medium break-all">{user?.email || t('settings.unknownUser')}</p> </div> - <Button variant="outline" onClick={handleSignOut}> + <Button variant="outline" className="shrink-0" onClick={handleSignOut}> {t('settings.signOut')} </Button> </div> diff --git a/tsconfig.agentic.json b/tsconfig.agentic.json new file mode 100644 index 000000000..5e27f570c --- /dev/null +++ b/tsconfig.agentic.json @@ -0,0 +1,15 @@ +{ + "compilerOptions": { + "target": "ES2022", + "lib": ["ES2023", "DOM"], + "module": "ESNext", + "moduleResolution": "bundler", + "types": ["node"], + "skipLibCheck": true, + "isolatedModules": true, + "moduleDetection": "force", + "noEmit": true, + "strict": true + }, + "include": ["e2e.config.ts", "agentic/**/*.ts"] +}