Skip to content

Commit dd31e49

Browse files
committed
chore: add verify script, drop stale bun.lockb, add verification plan
npm run verify chains lint, typecheck, coverage-gated tests, and build as the one command agents run before opening a PR. bun.lockb is stale drift: every pipeline path uses npm and package-lock.json. The plan doc describes the full agent verification loop this change starts.
1 parent c930d2d commit dd31e49

2 files changed

Lines changed: 189 additions & 1 deletion

File tree

‎package.json‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,8 @@
1919
"test:coverage": "vitest run --coverage",
2020
"test:watch": "vitest",
2121
"test:e2e": "playwright test",
22-
"test:e2e:ui": "playwright test --ui"
22+
"test:e2e:ui": "playwright test --ui",
23+
"verify": "npm run lint && npm run typecheck && npm test -- --coverage && npm run build"
2324
},
2425
"dependencies": {
2526
"@capacitor/app": "^8.1.0",

‎plans/agent-verification-loop.md‎

Lines changed: 187 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,187 @@
1+
# Agent Verification Loop
2+
3+
Goal: make DebateLab verifiable by agents end to end, so bugs are caught inside the development loop instead of by users on production.
4+
5+
Basis: Theo's thesis that coding is solved and the verification layer is not. This repo proves the point. Agents here can already write code well. What they cannot do today is prove their work: the e2e suite uses fake JWTs, coverage thresholds never fire, four backend test files never execute, PRs gate on nothing, and there is no way for an agent to hold a real authenticated session. This plan closes those gaps in five phases.
6+
7+
## Current state, verified 2026-08-26
8+
9+
What exists:
10+
11+
- Commands: `npm run lint`, `npm run typecheck`, `npm test`, `npm run test:coverage`, `npm run test:e2e`, `npm run build`.
12+
- CI (`.github/workflows/deploy.yml`): on push to main only. Runs lint, typecheck, vitest, build, then pushes a multi-arch image to Docker Hub (`adamgaborjaniszabo/debate-trainer:latest`). VPS pickup happens off-repo.
13+
- Vitest: ~63 test files, ~630 cases under `src/test`. Coverage thresholds (90/80/89/90) configured over 8 audited modules, enforced only when `--coverage` is passed manually.
14+
- Playwright: two render-smoke specs, Chromium-only, auto-starts `npm run dev` on :8080. Auth is a stubbed localStorage session with a bogus token, good only for render checks.
15+
- Supabase: 12 edge functions, manual CLI deploys, currently linked to prod project `uziwdtkzofptihahfooo`. A staging project `brifeyypjauyusmkptkx` exists and its anon creds ship in `src/lib/supabase/env.ts`, but migrations, functions, secrets, and CORS were never applied or verified. The env switcher (`setActiveEnv`) has zero callers.
16+
17+
What is broken or missing, ranked by risk:
18+
19+
1. `vitest.config.ts:11` includes only `src/**`, so the 28 cases in `supabase/functions/**/*.test.ts` (prompt-builder, source-quality, url-validation, native-cors) never run anywhere.
20+
2. Coverage gates never run in CI; `deploy.yml` invokes bare `npm test`.
21+
3. No `pull_request` trigger, so PRs merge ungated.
22+
4. Edge function handlers (Stripe checkout, portal, webhook, redeem-promo, admin) have zero automated execution. Money paths ride on review alone.
23+
5. No real authenticated path in e2e. `demo-code-login` works server-side but nothing in `src/` calls it, no tooling mints codes, no seed users exist.
24+
6. Staging Supabase is created but almost certainly empty; every function's `ALLOWED_ORIGINS` lacks a staging origin.
25+
7. No WebKit/Firefox projects although `plans/smoke-test-checklist.md` names Safari streaming as the likeliest breakage.
26+
8. TypeScript `strict: false`; typecheck is a weak signal. ESLint has `no-unused-vars` off.
27+
9. Observability dead in the Docker path: Sentry DSN unset at build time and nginx CSP omits the Sentry/PostHog origins that `vercel.json` allows.
28+
10. README drift: wrong repo name, Pi/port-3001 hosting story, obsolete `VITE_SUPABASE_*` guidance. Dual lockfiles: `package-lock.json` canonical, stale `bun.lockb` tracked.
29+
30+
## Target outcome
31+
32+
One command an agent runs on any branch:
33+
34+
```
35+
npm run verify
36+
```
37+
38+
runs lint, typecheck, unit tests with enforced coverage, build, and e2e against a real staging environment with a real session, and produces pass/fail plus failure evidence (trace, screenshot, video). CI enforces the same on every PR. When an agent cannot verify something, the repo tells it why and what to build next, via AGENTS.md.
39+
40+
## Phase 0: Make the checks that exist actually run
41+
42+
Small, high-leverage, no new infrastructure.
43+
44+
### Task 0.1: Include Supabase function tests in vitest
45+
46+
- Modify: `vitest.config.ts:11`
47+
- Change include to `["src/**/*.{test,spec}.{ts,tsx}", "supabase/functions/**/*.{test,spec}.{ts,tsx}"]`.
48+
- Risk: those tests may use Deno-flavored imports. Run `npx vitest run supabase/functions` first. If a test fails on Deno globals, either shim in `src/test/setup.ts` or move the tested pure logic into `_shared/` modules importable by both. Do not weaken assertions to make them pass.
49+
- Verify: `npx vitest run supabase/functions` exits 0 with 28+ passing cases.
50+
51+
### Task 0.2: Enforce coverage in CI
52+
53+
- Modify: `.github/workflows/deploy.yml`
54+
- Change the test step to `npm test -- --coverage`. Thresholds in `vitest.config.ts` then fail the job on regression. Keep the inline TODO comment accurate afterward.
55+
- Verify: push a branch with a deliberate assertion deletion, confirm the workflow goes red, revert.
56+
57+
### Task 0.3: Gate pull requests
58+
59+
- Modify: `.github/workflows/deploy.yml`
60+
- Add `pull_request:` trigger (any branch) running lint, typecheck, tests with coverage, and build. Docker push steps stay behind the existing secrets check so PR runs never publish.
61+
- Verify: open a draft PR with a failing test, confirm required checks fail, then fix.
62+
63+
### Task 0.4: Lockfile hygiene
64+
65+
- Delete `bun.lockb` (stale, unused by CI/Docker/Vercel, all npm). Optionally gitignore it. One-line decision, removes drift bait.
66+
67+
### Task 0.5: Umbrella verify script
68+
69+
- Modify: `package.json` scripts.
70+
- Add `"verify": "npm run lint && npm run typecheck && npm test -- --coverage && npm run build"`. E2E joins in Phase 3 once it needs no human.
71+
72+
## Phase 1: Write down how to verify (AGENTS.md)
73+
74+
Agents need the playbook in the repo. Create `AGENTS.md` at root:
75+
76+
- Canonical commands (the ones above), node 22, npm, ports 8080/54321.
77+
- Environment model: creds resolve in `src/lib/supabase/env.ts` from `localStorage['debate-supabase-env']`, prod vs staging refs, what IS_STAGING changes.
78+
- Verification ladder: static checks, unit, coverage, build, e2e smoke, e2e authed flows (once Phase 3 lands), then manual browser only for visual judgment.
79+
- Rule: if a change cannot be verified because tooling is missing, stop and propose the missing piece rather than merging blind. This encodes Theo's "agent notices it can't test, then builds the missing piece".
80+
- Evidence rule: attach trace/screenshot output or state plainly that verification did not happen. Never claim untested work passed.
81+
- Fix README.md drift in the same commit: repo name, Hetzner + port 8080 + cloudflared story, remove `VITE_SUPABASE_URL/ANON_KEY` guidance, correct workflow filename.
82+
83+
Verify: fresh-agent test, give a subagent only AGENTS.md and ask it to describe how to run and verify the app. It should answer correctly without reading other files.
84+
85+
## Phase 2: Finish staging so agents have a safe sandbox
86+
87+
Staging project `brifeyypjauyusmkptkx` exists. Everything below targets it, never prod.
88+
89+
### Task 2.1: Audit staging state
90+
91+
- `npx supabase link --project-ref brifeyypjauyusmkptkx`, then `npx supabase migration list` and a probe request to one function URL.
92+
- Record what exists. Expect: empty.
93+
94+
### Task 2.2: Apply schema and functions
95+
96+
- `npx supabase db push` (all 19 migrations).
97+
- `npx supabase functions deploy` each of the 12 functions.
98+
- Set non-provider secrets: `SUPABASE_SERVICE_ROLE_KEY` handled by platform; set `DEMO_CODE_PEPPER` to a staging-only random value.
99+
- Provider keys (`OPENAI_API_KEY`, `GOOGLE_API_KEY`, `ANTHROPIC_API_KEY`, `EXA_API_KEY`): Adam supplies values via `supabase secrets set`; never through chat, never in files. Stripe keys deferred until payment flows get e2e coverage.
100+
101+
### Task 2.3: CORS for staging callers
102+
103+
- Each function hardcodes an `ALLOWED_ORIGINS` array including `localhost:8080/5173/54321`. Localhost already covers the agent loop, so no change strictly required for Phase 3. If a hosted staging domain is wanted later (e.g. `debate-staging.pi-stack.org` via cloudflared), add it to every array in the same commit. Adam owns the tunnel route.
104+
105+
### Task 2.4: Deterministic env for tests
106+
107+
- Modify `src/lib/supabase/env.ts`: read `import.meta.env.VITE_FORCE_SUPABASE_ENV` as initial default for the active env (localStorage still wins). Build-time override, no runtime UI needed.
108+
- Verify: `VITE_FORCE_SUPABASE_ENV=staging npm run dev`, check network tab hits `brifeyypjauyusmkptkx.supabase.co`.
109+
110+
### Task 2.5: Seed script
111+
112+
- Create `scripts/seed-staging.ts`: uses service-role key from `STAGING_SERVICE_ROLE_KEY` env var (read at runtime, never committed). Creates one test user, mints a demo access code row with `sha256(pepper:code)` matching the staged `DEMO_CODE_PEPPER`, prints the plaintext code to stdout once.
113+
- Document in AGENTS.md. Rollback: staging is disposable; wipe and re-run.
114+
115+
## Phase 3: Real authenticated e2e, the actual bug catcher
116+
117+
This is the phase that catches Peter's-truncated-text-class bugs: things invisible in code review.
118+
119+
### Task 3.1: Auth setup project
120+
121+
- Create `e2e/auth.setup.ts`: calls `demo-code-login` on staging with the seeded code, completes `verifyOtp({ type: 'magiclink' })` via the staging anon client, saves valid storageState to `test-results/.auth/staging.json`.
122+
- Modify `playwright.config.ts`: add a `setup` project depending on nothing, authed projects depend on it with `use: { storageState }`. Public specs stay storage-free.
123+
- Env needed in CI/local: `STAGING_URL`, seeded demo code. Values come from the seed step; store locally in `.env.e2e` (gitignored).
124+
125+
### Task 3.2: First real flow specs
126+
127+
- Create `e2e/authed-flows.spec.ts` covering, in order of value:
128+
1. Login persists across reload; user menu shows the test identity.
129+
2. Ask: submit a prompt, assert streamed tokens arrive (SSE works, error boundary silent).
130+
3. History: conversation appears after the ask; reload keeps it.
131+
4. Sources: import `https://example.com`, assert it lands in the library.
132+
5. Share: share a session, open the public URL logged-out, assert content renders.
133+
- Cost control: pin these specs to the cheapest provider/model via the staging `AI_DEFAULT_MODEL` secret; three short conversations per run is cents.
134+
135+
### Task 3.3: Browser breadth and failure evidence
136+
137+
- Add `webkit` project to `playwright.config.ts` (Safari parity, the checklist's known risk). Install deps once: `npx playwright install --with-deps webkit`.
138+
- Config: `video: 'on'`, `trace: 'retain-on-failure'`, screenshot on failure.
139+
- CI: upload `playwright-artifacts` on failure; add `test:e2e` to the PR workflow (public smoke at minimum; authed flows require the staging secrets as GitHub secrets).
140+
- Extend `npm run verify` with e2e once green locally.
141+
142+
## Phase 4: Prove it, observe it
143+
144+
- Evidence convention in AGENTS.md: agents attach traces/screenshots via `gh pr comment` uploads or link CI artifacts. Never commit media to the repo.
145+
- Observability: pass `VITE_SENTRY_DSN` as a Docker build arg in `deploy.yml`; align `nginx.conf` CSP `connect-src` with `vercel.json` (add `*.ingest.sentry.io`, `eu.i.posthog.com`). Prod errors become visible to agents fixing incidents instead of invisible to everyone.
146+
- Commit the VPS pickup step: whatever pulls `debate-trainer:latest` on codex-remote (watchtower or systemd timer) gets a compose/unit file in-repo, making the last invisible hop observable and re-runnable. Adam confirms which mechanism runs today before we codify it.
147+
- Optional, decide explicitly: Vercel PR previews (project still linked) would give every PR a live URL humans and agents can poke. Conflicts with Docker-as-canonical unless kept preview-only.
148+
149+
## Phase 5: The autonomous loop
150+
151+
Only after Phases 0-3 are green:
152+
153+
- Nightly full-suite cron on codex-remote (tmux or systemd timer): `npm run verify` + authed e2e against staging, failures reported with artifacts. Fits Adam's watchdog pattern: execute safe diagnosis automatically, alert with evidence.
154+
- Bug intake rule in AGENTS.md: every bug report becomes a failing repro test first, fix second. Mirrors Theo's claim that speced bugs plus minimal verification get solved by frontier agents.
155+
- Later candidates, not committed here: deno check/lint of `supabase/functions` in CI, incremental `strict: true` tsconfig ramp, CodeRabbit re-run against the new baseline.
156+
157+
## Who does what
158+
159+
Agent-executable: all of Phase 0, AGENTS.md, README fixes, seed script, e2e code, config changes, docs.
160+
161+
Adam-only:
162+
163+
- Provide staging provider secret values via `supabase secrets set` (never chat, never files).
164+
- Choose and disclose the VPS image-pickup mechanism.
165+
- Decide: delete `bun.lockb`, enable Vercel previews or not, staging tunnel domain or localhost-only.
166+
- Approve any GitHub settings change making PR checks required/branch protection.
167+
168+
Secret handling rules: service-role and pepper values only via env vars or `supabase secrets set`; `.env*` stays gitignored; no secret ever printed in logs, commits, or PR comments; staging secrets distinct from prod.
169+
170+
## Acceptance test for the whole plan
171+
172+
On a fresh clone, an agent with only AGENTS.md and `.env.e2e`:
173+
174+
1. Runs `npm ci && npm run verify`, all green including coverage gates.
175+
2. Makes a deliberate bug (break the share page render), runs verify, watches e2e fail with a saved trace naming the broken page.
176+
3. Opens a PR; CI independently reproduces the red gate.
177+
4. Reverts, everything green again.
178+
179+
That loop, repeated hundreds of times, is the verification layer. Bugs die in the loop instead of on production.
180+
181+
## Risks and rollback
182+
183+
- Flaky authed e2e erodes trust fast. Mitigation: retries only in CI, tight selectors, staging pinned cheap model, quarantine-and-fix policy for flaky specs within 48h.
184+
- Staging drift (migrations applied out of order, secrets rotated silently). Mitigation: seed script is idempotent; AGENTS.md documents refresh procedure; staging is disposable by design.
185+
- Provider cost runaway from scheduled e2e. Mitigation: daily-limit override caps the demo user; nightly cron is the only scheduled consumer.
186+
- Phase 0 coverage enforcement may immediately redden CI if the 81.56% branches baseline dipped since the TODO was written. Mitigation: run `npm test -- --coverage` locally first; fix gaps before merging the workflow change, not by lowering thresholds.
187+
- Rollback: every phase is additive config plus docs. Reverting the workflow edit restores old CI; deleting the e2e auth setup restores smoke-only; staging can be wiped without touching prod. Nothing here writes to prod except the eventual nginx/Sentry header alignment, which ships through the normal gated pipeline.

0 commit comments

Comments
 (0)