Skip to content

test(agentic): cover every tool and page, fail on hidden errors, fix what it found - #110

Merged
Programcode merged 14 commits into
mainfrom
feat/agentic-e2e-coverage
Oct 6, 2026
Merged

Programcode merged 14 commits into
mainfrom
feat/agentic-e2e-coverage

Conversation

@Programcode

@Programcode Programcode commented Oct 6, 2026 •

Copy link
Copy Markdown
Owner

What this changes

The agentic suite from #99 was well built but thin, and it had never run in CI. This PR widens it to every tool and to every page a student or visitor reaches. It also makes the suite fail on errors the screen hides, and fixes the app bugs it found on its first runs.

The suite

  • e2e 0.17.0 and @e2e-dev/web 0.12.0 (from 0.15.2 and 0.11.1). This upgrade brings init scripts and failOnSkippedFailure. The replay cache re-recorded, as it must after a minor bump.
  • Hidden-error capture. agentic/support/capture-errors.js runs before the app in every document. It records uncaught errors, rejected promises, console.error (React warnings and the route error boundary included) and 5xx answers from Supabase. Every test fails on them.
  • More layout guards. Tests already failed on a page wider than its viewport. They now also fail on visible text that crosses the screen edge, which an overflow-x: hidden wrapper would otherwise hide. The guard measures each text node and clips it the way the browser does: along its containing-block chain, and by clip and clip-path. Text clipped away entirely (an off-canvas drawer, a panned canvas node, an sr-only label) or inside a sideways scroller does not count. Text cut short inside the screen, such as an ellipsis or a scroll area, is out of its scope on purpose, and the judge checks it. agentic/edge-guard.e2e.ts pins the rules on 17 injected layouts. Each one puts its text across the edge before clipping: removing clipping fails 7 of them, removing the containing-block rule fails the fixed-position case, and the two cases added in bdd73db fail on the guard before it.
  • Page sweep (agentic/pages.e2e.ts): all 14 student pages and 5 public pages, in English, Spanish and Hungarian, on desktop and mobile, for 114 tests. It skips /app/admin (admins only) and /share/:code (no client code creates a share link yet). Selectors check lang, a heading, and that no undefined, NaN or [object Object] leaked into the page. The judge checks for error notices, raw translation keys, untranslated interface text and clipped text. It replaces hungarian.e2e.ts, which covered 7 pages in one language.
  • Tool journeys, one goal per tool, each pinned by a locator and the database rows the app wrote. They cover the onboarding tour, Generate plus reopening the case from History, Attack, Brainstorm, Factoids, Ask, a full Sparring round, Crossfire, Library add and pause, Flowchart, Timer, and the Settings language switch. agentic/forms.e2e.ts covers empty submits on the four AI forms.
  • The actor model. OpenRouter withdrew qwen/qwen3.8-27b:free, so every cache miss and every exploration failed with This model is unavailable for free. The actor is now openai/gpt-6-luna. It only runs on a cache miss or in an exploration.
  • CI (agentic-e2e.yml):
    • It now runs on pull requests that change the app (src/, public/, Edge Functions, migrations, dependencies), not only the suite.
    • It runs four jobs: the page sweep and everything else, on each viewport. The journeys job excludes only pages and repro, so an untagged test still runs.
    • A weekly bug bash (Sundays 05:13 UTC) sends each of the four personas exploring with its own charter. Explorations never fail the workflow.
    • The suite and exploration jobs load the OpenRouter key on pull requests and manual dispatch from any ref, as test: agentic browser tests with TesterArmy e2e #99 already did. That falls under the live-key exception recorded in docs/standards/security.md.
  • Docs: docs/testing.md, the verify-debatelab skill, AGENTS.md, and the vendored e2e skill, refreshed from 0.17.0 with the DebateLab block re-added.

App bugs the suite found, fixed here

  • Empty submits on Generate, Attack, Brainstorm and Factoids showed no message and threw an unhandled ZodError. @hookform/resolvers 3 cannot read zod 4's errors and rethrows them. The fix is an upgrade to 5.4.0, the newest release without the @typeschema peer chain that npm rejects. It is pinned exactly, because a caret range resolves to 5.9.1 whenever the lockfile is regenerated. FactoidsForm now types its input and output separately, because its schema coerces and defaults fields.
  • The mobile landing hero ran past a 393 px screen in Hungarian (416 px wide), cutting "minden" to "mind". The hero grid had no explicit column, so its auto track grew to its widest content. grid-cols-1 caps it.
  • Attack results nested a <div> inside a <p> (p > div > p). React's DOM-nesting warning caught it.
  • The three Settings dropdowns had no accessible name, so the replay cache could not record them. Their labels are now associated. The account recipe and account-data.spec.ts find the output-language dropdown by name, which pins the fix.
  • Flowchart showed English in Spanish and Hungarian: the strength badges, React Flow's zoom, fit-view and lock buttons, and the minimap's name. The canvas now renders the same four buttons with translated names, disabled at the zoom limits like the originals. The Hungarian landing badge "Napi MI prep" now reads "Napi MI-felkészülés".

Not changed

  • An exploration also reported a double motion dialog in Brainstorm. A repro test did not reproduce it, so it is dropped. Two more findings came from fake mode, where the provider ignores the side and format settings.
  • When the agent unchecks a Library source switch, e2e reports the uncheck as failed even though the switch turns off a moment later. That step therefore never records and costs about two actor calls per run. This is an e2e engine issue, not an app bug.
  • OpenRouter's Muse Spark judge answers [Meta] The requested model was not found on roughly one call in a hundred. CI's default retry absorbs it; locally, use --last-failed.

Verification

All of this ran locally. GitHub Actions cannot run on this account until its billing block is cleared, and every Agentic E2E run since #99 stopped after 2 seconds with no steps. None of this is a CI pass.

  • On the final head bdd73db, everything except the page sweep ran on both viewports and passed 179 of 180. That covers the 32 journey and form tests, all 34 guard cases (17 layouts on each viewport), and a guard-only probe of all 114 page, language and viewport combinations. The one failure was the transient provider error, and it passed on --last-failed.
  • The page sweep last ran in full at b45a069: 140 of 146 passed in 49 minutes for $0.11. Of the 6 failures, 2 were the transient provider error, 3 were the judge flagging Spanish loanwords (prompt, tokens, chats, email), and 1 was the hero overflow. After the loanword wording and the hero fix, those pages and Flowchart passed at 5276486 (the 2 provider errors among them passed on --last-failed). Later commits change only the fixtures' edge guard, test files and docs.
  • A CI simulation of the journeys (default actor, read-only cache, CI=1) passed 24 of 24 in 3 minutes for under a cent. 28 flows replayed, and 4 fell back to the actor.
  • The new checks catch real bugs. The forms test failed 8 of 8 before the resolver upgrade and passed 8 of 8 after. The edge guard fails on main's hero and passes with the fix.
  • An exploration on 0.17 with the capture script active finished 4 of 8 steps with 5 findings for $0.02 before a transient provider error stopped it.
  • npm run verify on the merged head 9339f03 passed docs, lint (with knip and the repo rules from Review standards and lint checks from past PR reviews #108), both typechecks, 273 unit tests, 30 collaboration tests, 318 Edge tests, build, service worker, and 29 desktop @smoke Playwright tests. The account recipe passed 5 of 5 on desktop and on mobile Chromium, and e2e/full-stack/account-data.spec.ts passed 4 of 4. At bdd73db, the agentic typecheck, lint and docs:check pass.
  • The database gate (./.superset/db-gate.sh) passed at bdd73db: pgTAP 419 of 419, and full-stack Playwright 71 passed with 1 skipped.
  • In the browser, the Hungarian Flowchart shows translated badges and button names. Zoom-in disables at the limit, and the lock toggles.
  • actionlint is clean on every workflow.
  • Codex review (gpt-6.1-sol, high), six rounds. Round 1 found one low issue (zoom limits), fixed in c61a4c2. Round 2 found two medium issues in the first version of the edge guard (it measured element boxes and ignored clipping), fixed in b1cba08. Round 3 found the guard flagging an ellipsis-truncated motion (my mobile journey run caught it too, fixed in 8e7a499) and two clipping gaps, fixed in 8a5e455. Round 4 found two clipping gaps (fixed-position escape, clip areas) and two guard cases that did not exercise their rule, all fixed in 0199c6e. It also flagged text cut short inside the screen without an ellipsis, which I left out of the guard's scope: scroll areas, carousels and line clamps clip there by design, and the judge checks cut-off text. Round 5 found two low gaps (percentage insets, more containing-block properties) and an overstated "every page" claim, all fixed in bdd73db. Round 6 on bdd73db found no code issues; its one note, stale guard counts in this description, is fixed here.

Not verified: live mode (up --live), the LIVE branches of the journeys, WebKit, and any GitHub Actions run.

🤖 Generated with Claude Code

Programcode and others added 9 commits October 6, 2026 07:51
Upgrade e2e to 0.17.0 and @e2e-dev/web to 0.12.0 for init scripts and
failOnSkippedFailure. An init script records uncaught errors,
console.error and Supabase 5xx answers; every test fails on them.

Add a journey per tool and a sweep of every page in English, Spanish
and Hungarian on both viewports. Run CI per viewport on app changes,
and a weekly bug bash with each persona.

Fix the p > div > p nesting in Attack results that the new check caught.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…tic flows

The agentic suite's page-error check found that empty submits on
Generate, Attack, Brainstorm and Factoids threw an unhandled ZodError
and showed no message: @hookform/resolvers 3 cannot read zod 4 errors.
Upgrade it to 5.4.0, the newest release without the @typeschema peer
chain npm rejects, and type FactoidsForm's input and output apart.
agentic/forms.e2e.ts pins all four.

The replay cache could not record Settings because its three selects
had no accessible name. Associate their labels, and find the output
language select by name in the two specs that used a CSS container.

OpenRouter withdrew the free Qwen actor, so every cache miss and
exploration failed. Default the actor to openai/gpt-6-luna. Split CI
into page-sweep and journey legs per viewport, fix the truncated
workflow, and commit the re-recorded replay cache.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
React Flow 11 gives its zoom, fit-view and lock buttons fixed English
names, which the agentic page sweep flagged in Spanish and Hungarian.
Render the same four as ControlButtons with translated names, and pass
the MiniMap a translated aria-label.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…re fail-safe

Codex review: the translated zoom buttons stayed enabled at the zoom
limits, unlike React Flow's own. The error-capture fetch wrapper now
cannot throw into the app's request.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…st the edge

The page sweep's judge saw the Hungarian landing hero cut off on a
393 px phone: the hero grid had no explicit column, so its auto track
grew to its widest word (416 px) while an overflow-x: hidden wrapper hid
the page-level overflow. grid-cols-1 caps the track. The fixtures now
fail on visible text that crosses the screen edge, which the page-width
check alone cannot see. The sweep's judgment lists the technical
loanwords the Spanish copy uses on purpose.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… edge

Codex review round 2: the edge guard measured element boxes, so nowrap
text cut off inside a box at the edge passed, a wide box holding short
text failed, and text clipped away entirely (a panned canvas node, an
off-canvas drawer) failed. Measure each text node and clip it through
every clipping ancestor; exempt the pannable Flowchart canvas.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings October 6, 2026 10:47
@vercel

vercel Bot commented Oct 6, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
debate-navigator Ready Ready Preview Oct 6, 2026 11:41am UTC
debatelab-staging Ready Ready Preview Oct 6, 2026 11:41am UTC

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@coderabbitai

coderabbitai Bot commented Oct 6, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Repository YAML (base), Organization UI (inherited)
  • Review profile: CHILL
  • Plan: Free
  • Run ID: 9a653d90-9e5d-4709-8068-f64ed094a542
📥 Commits

Reviewing files that changed from the base of the PR and between 9339f03 and bdd73db.

📒 Files selected for processing (8)
  • .claude/skills/verify-debatelab/SKILL.md
  • AGENTS.md
  • agentic/edge-guard.e2e.ts
  • agentic/pages.e2e.ts
  • agentic/replay/20966190b26e96794bfe4506de2677da462ca3057158879501ad8326af309b56.json
  • agentic/replay/23755b48030e2ba88ae7593c0e65506ecd8b0b7d98a4006cfda221eda5edb068.json
  • agentic/support/fixtures.ts
  • docs/testing.md
🚧 Files skipped from review as they are similar to previous changes (3)
  • agentic/pages.e2e.ts
  • AGENTS.md
  • docs/testing.md

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.


📝 Walkthrough

Walkthrough

The pull request expands agentic end-to-end coverage and updates fixtures, replay records, runner configuration, and CI execution. It also adds multilingual page checks, localized Flowchart controls, and several form and page adjustments.

Changes

Agentic E2E coverage and execution

Layer / File(s) Summary
Runner configuration and usage guidance
.claude/skills/e2e/*, .claude/skills/verify-debatelab/*, package.json, e2e.config.ts, docs/testing.md
The E2E runner is updated to 0.17.0. Guidance covers providers, replay behavior, browser APIs, reports, secrets, and connected devices. The default actor model changes to openai/gpt-6-luna.
Shared browser guards and fixtures
agentic/support/*, e2e.config.ts, agentic/edge-guard.e2e.ts
Browser scripts and fixtures record page errors, rejected promises, console errors, matching 5xx responses, overflow, and text past the screen edge. Student and visitor fixtures support language-specific checks.
Journey and multilingual page coverage
agentic/*.e2e.ts, agentic/replay/*, e2e/full-stack/account-data.spec.ts
The tests add AI-tool, form, page-sweep, sparring, Crossfire, workspace, and case-reopening coverage. Replay files record desktop and mobile interactions and expected states. The former Hungarian-only test is removed.
Suite matrix and exploration workflow
.github/workflows/agentic-e2e.yml, AGENTS.md, .claude/skills/verify-debatelab/*, docs/testing.md
The workflow separates desktop and mobile page and journey runs. It adds manual and scheduled exploration legs and uploads evidence by viewport and test part.

Application UI updates

Layer / File(s) Summary
Localized Flowchart controls and labels
src/pages/FlowchartPage.tsx, src/locales/{en,es,hu}.json, plans/2026-10-02-missing-translation-keys.md
The Flowchart page adds translated zoom, fit-view, and canvas-interactivity controls. Strength badges and the minimap label use localized text.
Form behavior and page semantics
src/components/features/factoids/FactoidsForm.tsx, src/components/features/attack/AttackResults.tsx, src/components/landing/Hero.tsx, src/pages/SettingsPage.tsx
The factoid form separates schema input and output types and normalizes the source count. Attack Markdown wrappers use div elements, landing grids specify one column below lg, and settings labels target their select controls.

Estimated code review effort: 4 (Complex) | ~45 minutes

  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Note

🎁 Summarized by CodeRabbit Free

Your organization is on the Free plan. CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please upgrade your subscription to CodeRabbit Essentials by visiting https://app.coderabbit.ai/settings/billing.

Comment @coderabbitai help to get the list of available commands.

@Programcode

Copy link
Copy Markdown
Owner Author

Screenshots from the local verification stack (fake mode, made-up data). Mobile landing in Hungarian at 393 px, before and after the hero grid fix, and the Hungarian Flowchart with translated badges and canvas controls.

Landing, Hungarian, 393 px: before

Landing, Hungarian, 393 px: after

Flowchart, Hungarian: after

Programcode and others added 2 commits October 6, 2026 10:56
The clipped-text guard from b1cba08 flagged the mobile header's
truncated motion: its text box runs past the screen, but its own
ellipsis box cuts it off at 360 px. Flag text only when what shows of
it reaches the screen edge. Nine injected layouts pin the rule; the
guard still fails on main's mobile hero.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@Programcode

Copy link
Copy Markdown
Owner Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Oct 6, 2026 •

Copy link
Copy Markdown
⚠️ Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

…e guard

Codex review round 3: any clip or clip-path exempted text outright, and
a clip on one axis clipped both. Clip each axis only where its overflow
clips, clip to the box for clip and clip-path, and skip only a
zero-area clip (sr-only). agentic/edge-guard.e2e.ts pins the guard on
eleven injected layouts, sized to the viewport so both targets run them.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@Programcode

Copy link
Copy Markdown
Owner Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Oct 6, 2026 •

Copy link
Copy Markdown
⚠️ Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

…areas

Codex review round 4: a fixed or absolute box escapes the overflow of
ancestors outside its containing block, a clip-path inset or a
zero-height clip can hide a whole box, and two guard cases passed even
with clipping removed. Clip only along the containing-block chain,
compute clip and inset areas, and rebuild every case so its text
crosses the screen edge before clipping. Removing clipping fails 7
cases; removing the containing-block rule fails the fixed-escape case.
Text cut short inside the screen stays out of scope: the judge checks it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…page claims

Codex review round 5: inset(10%) took the height for its horizontal
cuts, and backdrop-filter and the translate, rotate and scale
properties also make a containing block for fixed boxes. Two new guard
cases fail on the previous guard. The page sweep covers the student and
public pages, not /app/admin or /share/:code, and the docs now say so.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@Programcode

Copy link
Copy Markdown
Owner Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Oct 6, 2026 •

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@Programcode
Programcode merged commit 4025827 into main Oct 6, 2026
8 of 16 checks passed
@Programcode
Programcode deleted the feat/agentic-e2e-coverage branch October 6, 2026 12:08

This branch was successfully deployed

2 active deployments
Preview – debate-navigator — bdd73dbc Deployed Oct 6, 2026 by vercel[bot]
Preview – debatelab-staging — bdd73dbc Deployed Oct 6, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants