test(agentic): cover every tool and page, fail on hidden errors, fix what it found

.claude/skills/e2e/SKILL.md

@@ -3,14 +3,17 @@ name: e2e
 description: Agentic end-to-end tests with e2e, the e2e runner. Covers scaffolding e2e.config.ts, picking the Playwright browser engine or the agent-device mobile engine, starting the app under test from the config, driving flows with agent.act, judging with agent.assert, agent.waitFor, and agent.extract, pinning values with screen, app, browser, and expect, shaping the agent (context, system prompt, tools, personas), the replay cache, the e2e CLI, reading .e2e/report.json, and bug bashes (parallel explore runs proven with repro tests). Use when a project depends on e2e, when asked for end-to-end, browser, mobile, or agentic UI tests, to bug bash or hunt for bugs, or when an e2e run fails.
 ---
 
-> **DebateLab rules override this skill.** Vendored from `e2e@0.15.2`
+> **DebateLab rules override this skill.** Vendored from `e2e@0.17.0`
 > (`node_modules/e2e/skills/e2e`, Apache-2.0, TesterArmy). Refresh it with
 > `rm -rf .claude/skills/e2e && cp -r node_modules/e2e/skills/e2e .claude/skills/e2e`
 > after an upgrade, then re-add this block.
 >
 > - Tests live in `agentic/*.e2e.ts` and import `test`, `expect`, and the
 >   `student` fixture from `agentic/support/fixtures.ts`. The config is
->   `e2e.config.ts`.
+>   `e2e.config.ts`. Tag a tool journey `journey`; a new route joins the page
+>   sweep in `agentic/pages.e2e.ts`. The fixtures fail a test on page errors,
+>   `console.error`, Supabase 5xx answers and overflow; fix the app, never the
+>   check.
 > - Run everything through the verification stack, never bare `npx e2e`:
 >   `.claude/skills/verify-debatelab/bin/verify.sh up`, then
 >   `verify.sh e2e run|explore|list ...`, then `verify.sh down`. The wrapper
@@ -30,8 +33,8 @@ one goal; `agent.assert`, `agent.waitFor`, and `agent.extract` judge the
 screen. `screen`, `app`, `browser`, and `expect` make exact interactions and
 checks. The replay cache reruns verified actions and checks their recorded end
 state without a model call; agent judgments still run live. UI targets use
-`@e2e-dev/web` for browsers or `@e2e-dev/mobile` for iOS simulators and
-Android emulators. A test that takes only `app` can check an API with `fetch`
+`@e2e-dev/web` for browsers or `@e2e-dev/mobile` for iOS simulators,
+Android emulators, and connected phones. A test that takes only `app` can check an API with `fetch`
 and `expect` (topic `writing-tests`). Model sign-in commands are in
 [setup](references/setup.md#subscriptions-and-api-keys).
 

.claude/skills/e2e/references/agent.md

@@ -6,8 +6,13 @@ model-call budget, no shared transcript. No agent step, no model calls.
 
 ## Configure a model
 
-Put an AI SDK model under `agents.default`. Vercel AI Gateway reads
-`AI_GATEWAY_API_KEY` or, without it, a Vercel OIDC token:
+Put an AI SDK model under `agents.default`. You can use all AI SDK providers
+that support tool calls and the language model specification v2 or later
+(https://ai-sdk.dev/providers). Examples: `@ai-sdk/openai`,
+`@ai-sdk/anthropic`, `@ai-sdk/google`, `@ai-sdk/amazon-bedrock`,
+`@openrouter/ai-sdk-provider`, `ollama-ai-provider-v2`. For a server with
+`/v1/chat/completions`, use `@ai-sdk/openai-compatible`. Vercel AI Gateway
+reads `AI_GATEWAY_API_KEY` or, without it, a Vercel OIDC token:
 
 ```ts
 import type { E2EConfig } from 'e2e';
@@ -32,7 +37,8 @@ any provider.
 - `model` drives `agent.act`. Judgments use `judge` when set, else `model`.
 - A missing model for the built-in agent raises one run-level
   `MODEL_UNAVAILABLE` when the first test acquires `agent`, exit 2; auth
-  failures surface on the first model call as `MODEL_PROVIDER_FAILED`.
+  failures surface on the first model call as `MODEL_PROVIDER_FAILED`, or as
+  `CONFIG_LOAD_FAILED` for a provider that reads its key when the config loads.
 - `context` is what the app calls things, sent to every model call, judges
   included; `system` is how the acting agent works, read only by the act
   loop. Both live on the agents entry; `agentContext` on a test or group
@@ -50,8 +56,8 @@ Tests use `agents.default` unless selected otherwise:
 - `{ agent: 'name' }` on an `agent.*` call overrides the test's choice.
 
 For signed-in personas, pair an agent with a `session` in a describe block
-repeated per persona; the cache records per agent step, so a specialised
-agent replays too.
+repeated per persona; the cache keys every entry by the agent and its
+context, so each persona records and replays its own steps.
 
 ## act: one goal
 
@@ -236,14 +242,19 @@ matches. Misses and hand-offs use the model; `agent.assert`,
   `agent.assert`, `agent.waitFor`), so an unchecked `act` never replays; a
   plain-value `expect`, `expect.poll`, `agent.extract`, another `act`, or
   the attempt passing confirms nothing.
-- A replay needs the app on the recorded path (unless the recording opens
-  with a navigation), re-finds each control by role, name, test id,
-  placeholder, and input purpose, and passes alone only when the recorded
-  end path and the controls seen during the step are back; otherwise the
-  agent takes over mid-step. `step.cache.reason` says why: `no-entry`,
+- A replay needs the app on the recorded route (origin, path, and query,
+  ids and tokens aside; unless the recording opens with a navigation),
+  re-finds each control by role, name, test id, placeholder, input purpose,
+  and its named row (an unnamed control with twins and no named container
+  hands off), and passes alone only when the recorded end route is back,
+  every control that appeared (with its checked or selected state) is
+  there, every one that went away is gone, at least one of those changed
+  during the replay, and no new alert showed; otherwise the agent takes
+  over mid-step. Entries are keyed per agent and per `agentContext`. `step.cache.reason` says why: `no-entry`,
   `wrong-context`, `target-not-found`, `target-ambiguous`, `end-mismatch`,
   and so on.
-- A step recording no actions creates no entry; one whose `unique()` value
+- A step recording no actions, or changing nothing on screen or in the
+  route, creates no entry; one whose `unique()` value
   equals, is spelled inside, or is the encoded form of another param's value
   is not recorded either (`step.cache.notRecorded`: `param-collision`).
 - `e2e init` gitignores `.e2e/cache/`; remove that line to commit entries

.claude/skills/e2e/references/debugging.md

@@ -28,7 +28,7 @@ jq '.run.results[] | select(.selected and .status != "passed") | .attempts[-1]
    (recorded as `skipped`).
 4. Artifacts, under `.e2e/artifacts/`: `failure/screen.txt`
    and the engine's screenshot per failed attempt; a Playwright trace per
-   traced attempt (`npx playwright show-trace <file>`); downloads; with
+   traced attempt (`npx playwright-core@1.63.0 show-trace <file>`; `pnpm dlx` under pnpm); downloads; with
    `--video` the recording (`video/video.webm` in a local browser, each
    later page `video/video-part<n>.webm` with its own `startedAt`;
    `video/video.mp4` on a device; a provider's file or link); with
@@ -46,7 +46,7 @@ jq '.run.results[] | select(.selected and .status != "passed") | .attempts[-1]
 | `COLLECTION_ERROR` | `async` describe body, `test.setup` inside `describe`, an option forbidden in a serial group, registration outside collection | Restructure per `writing-tests` |
 | `HOOK_FAILED` | `beforeAll` or `afterAll` threw; its scope's tests skip | Fix the hook; the report carries its error |
 | `UNSUPPORTED_ARTIFACT` | A test's or target's `trace` or `video` on an engine that cannot record | Drop it there, or set it at the config root or CLI (such targets skip with a notice) |
-| `BROWSER_INSTALL_FAILED`, `LAUNCH_TIMEOUT` | Browser download failed; engine init or attempt start exceeded `launchTimeout` | Run the quoted `npx playwright install <names>` (`--with-deps` on bare Linux); raise the root `launchTimeout` (60 s default) |
+| `BROWSER_INSTALL_FAILED`, `LAUNCH_TIMEOUT` | Browser download failed; engine init or attempt start exceeded `launchTimeout` | Run the quoted `npx @e2e-dev/web install <names>` or `pnpm exec e2e-web install <names>` (`--with-deps` on bare Linux); raise the root `launchTimeout` (60 s default) |
 | `APP_UNREACHABLE` | `app.command` never answered `readyUrl` within `startupTimeout`; on a device, a message naming the iOS automation runner: the runner failed, not the app | Read the quoted log lines; check the port, `app.url`, `app.command.env`, `app.command.startupTimeout`. Runner: rerun, else `npx agent-device daemon stop` and reboot the simulator |
 | `APP_ALREADY_RUNNING` | Something already serves `url` when `command` should start | Stop it, or `reuseExisting: true` locally |
 | `APP_URL_REQUIRED`, `APP_NOT_OPEN` | A navigation on a target without `app.url`; a `screen` call before `app.open()` | Add `app.url` to the target; open the app first |
@@ -58,14 +58,14 @@ jq '.run.results[] | select(.selected and .status != "passed") | .attempts[-1]
 | `MODEL_OUTPUT_INVALID` | An `extract` or `assert` answer failed the schema after one repair round | Simplify the schema or question; pick a stronger model |
 | `ACTION_FAILED` | Element not actionable (covered, disabled, detached), or an operation timed out | `expect` the condition first; close overlays; check `actionTimeout` |
 | `TEST_TIMEOUT` | The attempt exceeded `timeout` (120 s) | Split the test, or raise `timeout` |
-| `STEP_NOT_AWAITED` | The body returned while a step still ran: a call without `await` | `await` the call the code frame names (every `app`, `agent`, `screen`, `expect` call) |
+| `STEP_NOT_AWAITED` | The body returned while a step or `expect.poll` still ran, or an `afterEach` hook or fixture teardown returned while an `expect.poll` still ran: a call without `await` | `await` the call the code frame names (every `app`, `agent`, `screen`, `expect`, `expect.poll` call) |
 | `MODEL_UNAVAILABLE` | `agents.<name>.model` holds no AI SDK instance: reported once under `run.errors` at the first `agent` fixture; the run stops. A model-less custom executor fails only `waitFor` and `extract` this way | Construct one, e.g. `gateway('openai/gpt-6-luna-fast')` from `ai`, and export its key (`AI_GATEWAY_API_KEY`) |
 | `MODEL_PROVIDER_FAILED` | Network, 5xx, rate limit, no credits, or no response in 120 s, after the transport retries | Check the credential (the key; for keyless `gateway()` the Vercel CLI login and `.vercel/project.json`) and quota; retry (exit 3) |
 | `REPLAY_STALE` | `--strict-cache` or `cache.strict`, and a committed recording no longer replays; `step.cache.reason` says why | Rerun read-write with the knob the message names off; commit the changed entry under the directory it names (a custom `cache.store` is written by that run) |
 | `AUTOMATION_UNSUPPORTED` | The step needs an interaction the engine's toolset lacks (drag on a device, say); blocked, exit 1 | Do that step with `screen` actions, or run the test on a target that supports it |
 | `STEP_TIMEOUT`, `STEP_BUDGET_EXHAUSTED` | Goal too big or ambiguous, or a slow provider | Split the goal, use on-screen wording, add vocabulary via `agents.<name>.context` or `agentContext`; raise `maxSteps` or `maxModelCalls` (budget), the config `timeout` (`act`), `judgmentTimeout` (judgments); `--debug` |
 | `CONTEXT_OVERFLOW` | Screen plus step history did not fit the model's context window (`act` after one shrink-and-retry; a judgment on the first overflow) | Lower `agents.<name>.maxObservationBytes`, split the step, or pick a larger-window model |
-| `POLICY_DENIED` | A `file:`, `data:`, or `javascript:` URL; a password `Secret` into a non-password sink; reading a secure field, `toHaveValue`, `toHaveText`, `toContainText`, `toHaveAttribute` included; `app.screenshot()` after a secret fill | http(s) only; passwords into password inputs only; assert the outcome, not the value; screenshot before filling secrets |
+| `POLICY_DENIED` | A URL whose scheme is not `http:` or `https:` (`file:`, `view-source:`, `data:`); a password `Secret` into a non-password sink; reading a secure field, `toHaveValue`, `toHaveText`, `toContainText`, `toHaveAttribute` included; `app.screenshot()` after a secret fill | http(s) or `about:blank` only; passwords into password inputs only; assert the outcome, not the value; screenshot before filling secrets |
 | `UNSUPPORTED_CAPABILITY` | A fixture the engine lacks (`browser` on a device), `schema` or `vision` on `act`, an action the surface lacks | Declare `requires: ['browser']`; drop the option or action |
 | `SESSION_UNAVAILABLE`, `SESSION_CONTRACT` | `session: 'x'` with no setup saving `x`; a setup that skipped a declared name. `SESSION_MISMATCH`, `SESSION_EXPIRED`, `SESSION_INVALID`: the stored session is another run's or app's, expired, or corrupt | Add or fix the `test.setup`; rerun it |
 | `ONLY_IN_CI` | `test.only` reached CI | Remove it |

.claude/skills/e2e/references/mcp.md

@@ -20,6 +20,9 @@ Or declare it in the client's project config (`.mcp.json` for Claude Code,
 { "mcpServers": { "e2e": { "command": "npx", "args": ["e2e", "mcp"] } } }
 ```
 
+No model is needed: without the optional `ai` package, sessions open and the
+deterministic tools work.
+
 Flags: `--config <path>` names the default config file, `--target <name>`
 fixes the target every session opens on, `--headless` hides the browser or
 simulator (sessions are headed by default outside CI), `--max-sessions <n>`

.claude/skills/e2e/references/running.md

@@ -11,7 +11,7 @@ npx e2e guide [topic]              # print this skill; topics: setup, writing-te
                                    # running, explore, debugging, mcp, bug-bash
 npx e2e cache ls|clear|stats       # inspect or empty the replay cache
 npx e2e login|logout|models [provider]  # e2e/oauth subscription logins: openai,
-                                   # github-copilot, spacexai
+                                   # github-copilot, opencode-console, spacexai
 npx e2e mcp [--target <name>]      # MCP server for a coding agent (topic mcp)
 npx e2e feedback -m <text> [opts]  # report a problem with e2e itself
 npx e2e telemetry [disable|enable] # anonymous usage telemetry: status or switch
@@ -27,7 +27,7 @@ npx e2e telemetry [disable|enable] # anonymous usage telemetry: status or switch
 | `--tag <tags>` | Any of the tags, comma-separated or repeated; all of them with `--tag-mode all`. An empty `--target`, `--tag`, or `--agent` value is a usage error, exit 2. |
 | `--exclude-tag <tags>` | Drop tests carrying any of these tags, however selected. |
 | `--grep <pattern>`, `--grep-invert <pattern>` | Keep, or drop, tests whose title (describe titles and test title joined by spaces, `checkout pays`; not file or tags) matches a regular expression. Bare pattern or `'/pattern/i'`; repeat for alternatives. |
-| `--last-failed` | The tests the previous run (`<output>/report.json`) did not pass, plus every test in a failed `beforeAll` or `afterAll` scope. No report is `NO_LAST_RUN`, exit 2. |
+| `--last-failed` | The tests the previous run (`<output>/report.json`) did not pass or never ran (`--max-failures` included), plus every test in a failed `beforeAll` or `afterAll` scope. A test another filter leaves out stays owed (`run.carried` in the report) until a rerun runs it. No report is `NO_LAST_RUN`, exit 2. |
 | `--shard <index/total>` | One contiguous slice of the selected tests (`--shard 2/3`), cut after every other filter; serial groups stay whole, each shard brings its own setup tests. |
 | `--headed` | Visible browser or simulator when the engine supports it. |
 | `--agent <names>` | Run unpinned tests as these `agents.<name>` entries (default `agents.default`), comma-separated or repeated; several names run each such test once per agent. |
@@ -87,7 +87,8 @@ its own tools.
 `summary.md`, `failures/`, `ai-trace.json`, `sessions/`, and `artifacts/`
 (screenshots, Playwright traces, videos, `--debug` transcripts, downloads).
 `artifacts/` is cleared once a run's tests start; a run stopping before
-leaves the last run's files. The report records every artifact path, a
+leaves the last run's files, and a `--last-failed` rerun keeps the files
+the report it reruns names and writes its own under `artifacts/rerun-<n>/`. The report records every artifact path, a
 hosted service's video by URL.
 
 - `list` (default): setup steps, one line per file and target, a `Failed
@@ -102,11 +103,14 @@ hosted service's video by URL.
   and `run.results[]`, one per test and target: `titlePath`, `file`,
   `source`, `tags` (`[]` when none), `agent`, `repeat` (0 unless
   `--repeat-each`), `selected`, `status`, `attempts[]` of `steps[]`,
-  `artifacts[]`, `error`.
+  `artifacts[]`, `error`. `run.summary` counts the selected results:
+  `passed`, `failed` (timed out included), `interrupted` (stopped by the
+  run: a signal, `--max-failures`, a run-level error; not a failure), `flaky`, and `skipped` add up
+  to `selected`; `discovered - selected` were left out.
 - `junit`: `junit.xml` for CI summaries; `--reporter list,junit` keeps the
   terminal output.
 - `markdown` (`--reporter list,markdown`): `summary.md` plus one page per
-  failed or flaky test under `failures/`. The summary holds counts and
+  failed or flaky test under `failures/` (an interrupted test gets none). The summary holds counts and
   spend, a block per failed test (error, facts, failing step, whether every
   attempt failed alike, last model turns, screen location and closest
   nodes, the line to look at, evidence paths), the flaky tests folded
@@ -142,8 +146,9 @@ interrupted workers keeps the failures' or the error's code. Exit 2 is
 deterministic, never retry it; only exit 3 is worth a job-level retry.
 
 The first SIGINT or SIGTERM interrupts and writes the report if any test
-had started; the second forces engine teardown; the third kills the app
-process groups and exits 130 at once.
+had started; the second forces engine teardown and still writes the
+report, plus `junit.xml` and `summary.md` when those reporters are on; the
+third kills the app process groups and exits 130 at once.
 
 ## Continuous integration
 
@@ -174,7 +179,7 @@ jobs:
           node-version: 26 # any Node >= 22.12
           cache: pnpm
       - run: pnpm install --frozen-lockfile
-      - run: npx playwright install chromium --with-deps
+      - run: pnpm exec e2e-web install chromium --with-deps
       - run: npx e2e run --reporter list,junit
         env:
           AI_GATEWAY_API_KEY: ${{ secrets.AI_GATEWAY_API_KEY }}