Repository navigation
Cut CI to a two-job gate and weekly E2E and AI evals - #283
Merged
Merged
Conversation
Quality now runs one `check` job (docs, lint, types, guards, Edge Function tests, build) beside the `@smoke` browser job. The unit-test job is gone: 89 component, store and page tests are deleted, and Vitest keeps 12 guard files that read repo config and content (Claude-only routing, Vercel headers, production-target refusals, quota and Stripe ownership, i18n parity, SEO rules). The browser suite, full database gate, agentic suite and AI evals move from daily to weekly on staggered days. The model matrix becomes a job of live-e2e.yml, the bug bash runs monthly, and protected staging runs daily and on demand instead of after every push. Prod canary is unchanged. AGENTS.md and the docs now ask for end-to-end or backend tests, not unit tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Important Review skippedToo many files! This PR contains 116 files, which is 16 over the limit of 100. To get a review, reduce the PR to 100 files or fewer by splitting it into smaller PRs or changing its base branch. Upgrade to a paid plan to raise the limit. This review couldn't start because sufficient usage credits or metered capacity aren't available. Add credits or update usage-based reviews in the billing tab, then retry.
 |
A commit merged just before the daily schedule can reach staging while its Quality push run is still queued. The step now polls for up to 10 minutes instead of failing on the first look. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Owner
Author
|
Codex review r1 (head 8015a0b) had one finding. Medium: on the scheduled path, staging E2E checked Quality's result once and would fail while Quality on The pgTAP failure on 8015a0b was |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cuts GitHub Actions from about 4,400 to about 1,500 billed minutes a month. The pull-request gate drops from five jobs to two and from about 2.5 minutes to 1.6 minutes of wall time (2.2 minutes with a cold cache). CI no longer runs unit tests. The end-to-end suites and the AI evals still run, weekly or on demand, instead of daily.
What changed
Quality (
deploy.yml). The gate is now two jobs:check(pull requests and pushes tomain) runs docs, lint, typecheck, the guards, Edge Function tests and build. The build step catches a broken build before Vercel does.browser-smoke(pull requests only) is unchanged.The
lint,unit,edgeandtypecheck-buildjobs are gone. GitHub rounds each job up to a whole minute, so four short jobs cost four minutes even though the work fits in about two.Unit tests. I deleted 89 Vitest files, plus their setup, helpers and mocks, and uninstalled
jsdom,@testing-library/*and@vitest/coverage-v8. Most were component, page, store and hook tests. The set also included tests you may want to veto one by one:src/test/ai/streaming.test.ts,src/lib/ai/partial-json.test.ts)session-expiry,auth-callback,AuthProvider.behavior) and checkout return (checkout-return)backend-status.test.tsanddb-gate.test.ts(the second landed in Make local Supabase verification parallel-safe with two slots #280 this morning)Every file is recoverable with
git show e237d9f2:<path>. Deleting them beats leaving them unrun: tests nobody runs rot and still mislead agents. The collaboration service keeps its own tests (npm run test:collaboration, 30 tests, 3 s). The service is hidden, so neither CI norverifyruns them. Vitest now runs 12 guard files, listed explicitly invitest.config.ts, innodein about 2 seconds. Each one reads repo config or content and checks a rule no browser run can see:Kept on purpose: Edge Function tests (Deno, about 25 s inside
check). These are unit-level tests too. I kept them because Edge Functions own auth, quota, billing and cost accounting, and no pull-request browser test reaches a real Edge Function. They run insidecheck, so they add no billed job. Say the word and they go.Schedules. Daily suites become weekly, on staggered days so no two start local Supabase the same day. Today's Database run failed with
toomanyrequests: Data limit exceededfrompublic.ecr.aws.live-e2e.yml): Mondays. The model matrix is now a second job in this workflow (model-matrix.ymlis deleted). Dispatch with-f suite=live|model-matrix|both.browser-nightly.ymltobrowser-weekly.yml.main, instead of after every push tomain. About half of each old run's billed time went to polling for the staging sync. Before a backend production deploy, dispatch it (docs/operations.mdnow says so). The VPS sync's hold-off matches on workflow file and job name, so a scheduled run still holds the sync.mainand daily, and the daily run makes two real AI requests, so provider drift still shows within a day.Instructions. AGENTS.md,
docs/testing.md,docs/operations.md(CI section rewritten),docs/development.md,docs/staging.md,docs/standards/security.md, README and three verify-debatelab feature notes now ask for end-to-end or backend tests (Playwright, agentic, Deno, pgTAP). None of them asks for unit tests anymore.npm run verifymirrors the new gate.Before and after
Assumptions: about 150 pull-request runs, 50 pushes to
mainand 15 schema pull requests a month, and 4.33 weeks a month. Durations come from today's runs unless marked est. Billed minutes round each job up to a whole minute.October's real spend was higher. Live, model-matrix and agentic still ran on pull requests then, and the canary ran on every
deployment_status. Those triggers were already gone frommain, so the "before" column usesmainas it stood, not October's history. Model matrix, full gate and bug bash have no successful run on record (Actions billing failed most October runs), so their durations are estimates.Verification
npm run verifyat the final head: all 9 steps passed in 96.5 s (lint 36.6 s, build 35.1 s, smoke 29.7 s, typecheck 25.6 s).node(94 tests, 1.9 s).npm ciran:check128 s,browser-smoke115 s, 2.2 min of wall time, 5 billed minutes. The run on b931cb1 was warm:check91 s,browser-smoke69 s, 1.6 min of wall time, 4 billed minutes. pgTAP runs becausedatabase.ymlchanged. Its first run failed onpublic.ecr.awstoomanyrequests: Data limit exceededwhile pulling the Postgres image, which is infrastructure, not this diff. On b931cb1 it passed, so all 5 checks are green at the final head.Not verified: the new schedules and the
live-e2e.ymldispatch matrix only run onmainor by dispatch, so no run here exercised them. actionlint checked the matrix expression's types. The first scheduled runs are Monday's AI evals and Tuesday's browser suite.Already red before this PR: today's daily production canary failed at
canary.spec.ts:97(streamed Ask), whicht3/fix-canary-ask-model-pickeris fixing.Merge rule
This PR is not low-risk under AGENTS.md: it changes
.github/,package.jsonandpackage-lock.json. It needs Adam's OK to merge.🤖 Generated with Claude Code