Repository navigation
test: agentic browser tests with TesterArmy e2e - #99
Conversation
Adds an e2e suite in agentic/ that runs against the verify-debatelab stack. An AI agent drives goals and judges screens, and exact locator and database assertions pin the result. It covers what the Playwright layers cannot: goal-level onboarding and case-generation journeys, Hungarian copy and clipped-text judgments on every main page, and, in live mode, whether a generated case is on topic. Explorations with named personas hunt for bugs. - verify.sh e2e <run|explore|list|cache> runs doctor, loads the stack env and the OpenRouter key, disables telemetry, and keeps evidence - agentic-e2e.yml workflow: nightly, on suite changes, and on-demand exploration; not a required check - Playwright moves to 1.63 (e2e's web engine peer) so both runners share one browser build - the Playwright-free half of e2e/full-stack/stack.ts moves to stack-core.ts for reuse - vendors the e2e agent skill with DebateLab rules on top Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The first agentic run judged every main page in Hungarian and found English the parity spec cannot reject: the theme toggle's label, the model selector's label and accessible name, and Sparring's "Speaking order". Their keys were in no catalog, so every non-English user saw the inline English fallback. The selector's accessible name now uses the translated tool name instead of the raw feature id. The same run caught mobile Settings scrolling sideways: a long email pushed Sign out 4 px past a 393 px screen in Hungarian. The email now wraps. Also records the first replay entries, makes free Qwen the default testing model with budgets that fit it, and adds a backlog plan for the 88 other literal t() keys missing from every catalog. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Free Qwen stays the actor: its recorded flows replay without a model call. As the judge it failed about a third of judgment-heavy tests on rate limits, timeouts and false positives, so judgments now go to meta/muse-spark-1.3-contributor (E2E_JUDGE_MODEL overrides it). A full suite run costs about half a cent. The judge cannot act, because Meta only accepts tool_choice "auto", so explorations keep the actor. The Hungarian judgment now exempts language names offered as choices and inflected English loanwords, and the context names MI as the Hungarian AI abbreviation and ellipsis truncation as intended. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Translating the tool name capitalized it ("Factoids"), which changed the
English accessible name and broke the authed-pages spec that finds the
selector by "Select AI model for factoids".
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
An unset GitHub Actions variable reaches the job as an empty string, and `??` kept it, so CI would have asked OpenRouter for model "". Also correct the suite cost to the measured figure (under two cents). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Navigate logical layers of code changes, visualize relationships, and explore their blast radius. Note Currently processing new changes in this PR. This may take a few minutes, please wait... ⚙️ Run configuration
⛔ Files ignored due to path filters (1)
📒 Files selected for processing (5)
ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Free Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (38)
Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review. 📝 WalkthroughWalkthroughThe pull request adds agentic E2E configuration, test fixtures and scenarios, verification and CI workflows, and usage guides. It also updates UI translations and settings layout, and adds a plan that inventories missing translation keys. ChangesAgentic E2E Testing
UI Labels and Settings Layout
Translation Backlog Plan
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant Developer
participant VerifyScript
participant LocalStack
participant E2ERunner
participant DebateLabApp
Developer->>VerifyScript: Run e2e run or e2e explore
VerifyScript->>LocalStack: Check stack health and load environment
VerifyScript->>E2ERunner: Start the selected E2E operation
E2ERunner->>DebateLabApp: Run browser actions and assertions
E2ERunner-->>VerifyScript: Return status and write evidence
Security Architecture ReviewSecurity architecture risk: 🔵 Low · up to The new automated tests reuse an existing external-service credential and operate against a local test database. No new production privilege expansion was established, but approval rules, credential limits, and protections for retained test output remain unverified. Retained concerns Security review detailsSecurity Blast Radius
Security Findings and Attack Paths
Trust Boundaries and Controls
Resilience and Maintainability Implications
Hardening Proposals
Note 🎁 Summarized by CodeRabbit FreeYour organization is on the Free plan. CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please upgrade your subscription to CodeRabbit Essentials by visiting https://app.coderabbit.ai/settings/billing. Comment |
# Conflicts: # package.json
e2e reads the previous failures from <output>/report.json, and every wrapped run gets a fresh output directory, so the documented retry exited NO_LAST_RUN. Seed the new directory with the latest run's report. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
@coderabbitai review |
|
|
@coderabbitai review |
|
|
@coderabbitai review |
✅ Action performedReview finished.
|
The Hungarian agentic test flagged "Top 5 Legjobb Szög" on /app/brainstorm. Both Brainstorm labels now read "Az 5 ...", matching the export heading. The judge prompt now names AI model names and plan names (Pro) as exempt product names, since the model picker shows "Claude Sonnet 5" with a Pro badge. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
@coderabbitai review |
|
|
@coderabbitai review |
|
# Conflicts: # docs/testing.md # package.json
|
@coderabbitai review |
|
|
@coderabbitai review |
|
What this adds
An agentic browser suite in
agentic/, built on TesterArmy's e2e runner (e2e@0.15.2) and run against the verify-debatelab stack. In each test an AI agent works toward a goal written in plain language (agent.act) and judges the screen (agent.assert). Exact locators and database reads then check the result.It covers what the Playwright layers cannot:
agentic/replay/and replay with no model call until the UI changes.verify.sh e2e explore "<goal>" --session student --agent <persona>sends a persona (first-timer,coach,edge-inputorhungarian) through the app. It reports findings with reproduction steps, screenshots and a trace.Bugs it found, fixed here
t()keys missing from every catalog, mostly in toasts, dialogs and Library folder actions. That work is a backlog plan, not part of this PR:plans/2026-10-02-missing-translation-keys.md.How it runs
verify.sh e2e <run|explore|list|cache>runsdoctor, loads the stack environment and the OpenRouter key, turns e2e telemetry off, and writessummary.md,report.json, traces and failure videos to.verify/artifacts/. The config refuses any non-loopback URL and never starts a dev server, because a plainnpm run devtalks to production.qwen/qwen3.8-27b:freeacts andmeta/muse-spark-1.3-contributorjudges. A full suite run costs under two cents.E2E_AGENT_MODELandE2E_JUDGE_MODELoverride them..github/workflows/agentic-e2e.ymlruns the suite nightly at 03:47 UTC, on pull requests that change the suite, and on demand. A manual run can instead run one exploration with a chosen persona and viewport. It is not a required check, and it writes the report to the job summary.Supporting changes
@playwright/testmoves from 1.62 to 1.63, the version e2e's web engine requires, so both runners share one Playwright and one browser build.e2e/full-stack/stack.tsmoved tostack-core.ts, andstack.tsre-exports it.tsconfig.agentic.jsonis chained intonpm run typecheck, because the e2e runner does not type-check..claude/skills/e2e/with DebateLab rules on top. Agents must go throughverify.shand must never addapp.command.docs/testing.md, plus updates todocs/development.md, the verify-debatelab skill, AGENTS.md and the plans index.Verification
Final head
6aac107, which is59bc164merged withmain(PRs #100, #101, #107). CI did not run on either head. GitHub stopped every job after 2 seconds with "recent account payments have failed or your spending limit needs to be increased". The local results below are the evidence, and none of them is a CI pass.At
6aac107, run one after another so no two Playwright runs sharedtest-results/:npm run verify:fullpassed. It covers docs, lint, typecheck, 265 unit tests in 24 files (after test: cut tests that can't catch a regression, fix the misbuilt ones #107's test cut), 30 collaboration tests, edge, build, service worker, the desktop@smokeset (29 passed), and the full desktop and mobile Playwright suite (237 passed, 5 skipped)../.superset/db-gate.shpassed with pgTAP 419 of 419 and full-stack Playwright 71 passed, 1 skipped.verify.sh e2e run --last-failedthen selected only that test, and it passed.At
59bc164, before the merge:npm run verifypassed (1228 unit tests and 249 Playwright tests, with 1 flaky inadmin-announcements.spec.ts), the database gate passed, and the agentic suite passed 18 of 18. A Codex review of the whole branch at59bc164(gpt-6.1-sol, high effort) found nothing; the later merge brought in onlymainand two small conflict resolutions. An earlier Codex pass found that--last-failedcould never find the previous report;ee533e6fixes that.The last CI run that executed was on
ee533e6. Quality, browser tests, database tests and live E2E passed. Agentic E2E failed on the Hungarian Brainstorm copy fixed in59bc164. The same run marked/app/settingsflaky because the judge flagged the "Pro" plan badge; the judge prompt now lists model names and plan names as product names.Earlier in the branch: the first
npm run verifycaught a real regression inauthed-pages.spec.ts:58(the translated tool name capitalized the English accessible name), fixed by theModelSelectorlowercase commit. Exploration asfirst-timeron desktop finished 4 of 4 steps with no findings for $0.003.Known limits and decisions for Adam
temporarily rate-limited upstream. The Muse Spark judge cannot act, because Meta's endpoint only acceptstool_choice: "auto". The reliable paid actor isE2E_AGENT_MODEL=openai/gpt-6-luna, at cents per exploration.AuthProviderkeeps the interface language and resets output language to English, and the existing first-run spec checks only the interface language. This PR leaves that alone because it is a product decision.E2E_TELEMETRY_DISABLED=1, but one event (command and versions, no content) went out from a barenpx e2e listduring setup.🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Documentation
Bug Fixes