ci(repo): publish a weekly test-suite health report by jgoux · Pull Request #6891 · supabase/cli

jgoux · 2026-09-29T14:53:16Z

Problem

Nothing showed how the test suite grows, how slow each part is, or which tests flake. Pruning campaigns and ratchets need that data to pick targets and set budgets.

Change

The unit, integration, CLI e2e and cli-e2e jobs upload their Vitest results caches as artifacts, retained for 14 days. Each is staged per workspace by .github/scripts/stage-vitest-results.sh, so the workspace survives the upload. This is the same pipeline the e2e timings file already uses; the results-cache parser moves to apps/cli/scripts/vitest-results-cache.ts and both consumers share it.
A weekly workflow, test-health-report.yml, also runs on demand. It writes a job summary from those artifacts plus git:
- test lines of code by workspace and tier, compared with a week earlier;
- the fastest-growing and slowest test files;
- flaky tests. That covers files that failed in develop or merge-queue runs, and runs that only passed on a rerun. For reruns it names the job, not the file, because the API only reports first-attempt outcomes per job.
The report is advisory and gates nothing. Its logic lives in .github/scripts/test-health-report/ and is tested by the existing GitHub scripts CI.

Makes suite growth, speed, and flakiness visible from CI data so later pruning routines and ratchets have something to act on. - Uploads Vitest results-cache timings from the unit and integration jobs, and the cli-e2e suite, alongside the existing e2e timings, staged under a per-workspace path so they stay attributable; raises retention to 14 days. - Shares the results-cache parser between the e2e timings refresh script and the new report instead of duplicating it. - Adds a weekly scheduled workflow that builds a job-summary report: test LOC by workspace/tier with week-over-week change, the fastest growing and slowest test files, and flaky-test signals from the week's develop/merge-queue Test runs. Advisory only.

Puts the report where bun test ./.github/scripts and its type-check already run it, and de-duplicates the per-job results-cache staging into one script. - Moves tools/test-health-report.ts and tools/lib/test-health-report.ts into .github/scripts/test-health-report/ as report.ts and logic.ts, with logic.test.ts running under the existing github-scripts-ci.yml. - Replaces the copy-pasted staging loop in the unit, integration, and e2e jobs with .github/scripts/stage-vitest-results.sh. - Adds apps/cli/scripts/vitest-results-cache.ts to github-scripts-ci.yml's trigger paths, since the report imports it. - Trims comments to the repo's one-or-two-line, why-only convention.

github-actions

🤖 AI Review

Both reviews completed. The most serious confirmed issues are that the pinned upload action excludes the hidden staging directory, the report’s download pattern misses three new artifact names, and successful retries can encounter duplicate artifact names once uploads work. Ten distinct findings remain after merging two overlaps. The invalid --days example in one finding was narrowed: it causes an error, while a missing historical commit can produce a misleading zero change.

Findings

Severity	Location	Category	Sources	Claim
🟠 MAJOR	`.github/scripts/test-health-report/report.ts:159`	`artifact-selection`	claude+codex	The download pattern excludes the new unit, integration, and cli-e2e timing artifacts, leaving only e2e shard timings in the report.
🟠 MAJOR	`.github/workflows/test.yml:116`	`artifact-upload`	claude+codex	All four timing uploads exclude their staged files because `.timings-staging` is hidden and `include-hidden-files` is unset.
🟠 MAJOR	`.github/workflows/test.yml:115`	`ci-retries`	codex	Once timing uploads contain files, a retried unit, integration, or cli-e2e job can fail while uploading an artifact whose name was used by its earlier attempt.
🟡 MINOR	`.github/scripts/test-health-report/report.ts:263`	`error-handling`	claude	One failed first-attempt jobs API request aborts the entire weekly report.
🟡 MINOR	`.github/scripts/test-health-report/report.ts:87`	`correctness`	claude	When no commit precedes the requested window, the report silently compares HEAD with itself and shows zero LOC change.
🟡 MINOR	`.github/scripts/test-health-report/report.ts:177`	`pagination`	claude	First-attempt failures beyond the jobs API’s first page would be omitted if the Test workflow grows beyond the default page size.
🟡 MINOR	`.github/scripts/test-health-report/report.ts:163`	`error-handling`	codex	Authentication, API, and network download failures are treated like absent artifacts, allowing a successful report with incomplete timing data.
🟡 MINOR	`.github/scripts/test-health-report/report.ts:232`	`argument-validation`	codex	`--max-runs 0` keeps all runs, and negative values select an unintended subset.
🟡 MINOR	`.github/scripts/test-health-report/report.ts:36`	`repo-convention`	codex	The new effectful TypeScript report script bypasses the repository’s documented Effect runtime convention.
⚪ NIT	`.github/scripts/test-health-report/report.ts:187`	`ci`	claude	`writeStepSummary` replaces the step-summary file instead of appending to it.

Stats

Claude findings: 6 · Codex findings: 6 · Confirmed: 10 · Refuted: 0 · Uncertain: 0

Models: claude-opus-5-5 + gpt-6-sol · Trigger: auto · Workflow run

This review runs once per PR. A maintainer can request another with a /ai-review comment.

github-actions · 2026-09-29T15:02:58Z

+    "--repo",
+    repo,
+    "--pattern",
+    "*-timings-*",


🟠 MAJOR · artifact-selection · source: claude+codex

The download pattern excludes the new unit, integration, and cli-e2e timing artifacts, leaving only e2e shard timings in the report.

Evidence: report.ts:158-159 uses *-timings-*; test.yml:115,155,305 names the excluded artifacts unit-timings, integration-timings, and cli-e2e-timings. Only e2e-timings-${{ matrix.shard }} at test.yml:296 matches.

Suggested fix: Use a pattern that matches all four artifact name forms, or select the names explicitly.

github-actions · 2026-09-29T15:02:58Z

+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: unit-timings
+          path: .timings-staging


🟠 MAJOR · artifact-upload · source: claude+codex

All four timing uploads exclude their staged files because .timings-staging is hidden and include-hidden-files is unset.

Evidence: test.yml:107,147,289 stages into .timings-staging; upload paths at :116,156,297,306 use that directory without include-hidden-files. The pinned upload action defaults this input to false and its bundled glob skips a search item whose basename begins with a dot.

Suggested fix: Use a staging directory without a leading dot, or set include-hidden-files: true on every timing upload.

github-actions · 2026-09-29T15:02:58Z

+            runUrl: workflowRun.html_url,
+            finalAttempt: workflowRun.run_attempt,
+            finalConclusion: workflowRun.conclusion,
+            attempt1Jobs: await attempt1Jobs(repo, workflowRun.id),


🟡 MINOR · error-handling · source: claude

One failed first-attempt jobs API request aborts the entire weekly report.

Evidence: report.ts:263 awaits attempt1Jobs without a local catch; ghApiJson throws on API failure at :50-55, and the only catch is the outer handler at :296-299.

Suggested fix: Handle a failed jobs request per run and add a note identifying the omitted retry data.

github-actions · 2026-09-29T15:02:58Z

+async function resolveWeekAgoSha(headSha: string, days: number): Promise<string> {
+  const result = await runGit(["log", "-1", `--before=${days} days ago`, "--format=%H", headSha]);
+  const sha = result.stdout.trim();
+  return sha === "" ? headSha : sha;


🟡 MINOR · correctness · source: claude

When no commit precedes the requested window, the report silently compares HEAD with itself and shows zero LOC change.

Evidence: report.ts:84-87 falls back to headSha on empty git output; :220-228 compares the two snapshots, and logic.ts:271 renders them as a week-over-week comparison.

Suggested fix: Add a report note when the historical commit is unavailable and validate --days as a positive integer.

github-actions · 2026-09-29T15:02:58Z

+  runId: number,
+): Promise<ReadonlyArray<{ readonly name: string; readonly conclusion: string | null }>> {
+  const data = await ghApiJson<{ jobs: Array<{ name: string; conclusion: string | null }> }>(
+    `repos/${repo}/actions/runs/${runId}/attempts/1/jobs`,


🟡 MINOR · pagination · source: claude

First-attempt failures beyond the jobs API’s first page would be omitted if the Test workflow grows beyond the default page size.

Evidence: report.ts:176-179 requests attempts/1/jobs once without per_page or pagination, whereas :110-126 paginates workflow runs.

Suggested fix: Paginate first-attempt jobs, or request per_page=100 and handle further pages.

github-actions · 2026-09-29T15:02:58Z

+  if (!summaryPath) {
+    return;
+  }
+  await Bun.write(summaryPath, `${markdown}\n`, { createPath: true });


⚪ NIT · ci · source: claude

writeStepSummary replaces the step-summary file instead of appending to it.

Evidence: report.ts:182-188 uses Bun.write for $GITHUB_STEP_SUMMARY; the existing summary writer in .github/scripts/ai-review/resolve.ts:383-392 appends.

Suggested fix: Append the markdown to the summary file.

github-actions · 2026-09-29T15:02:58Z

+        if: always()
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: unit-timings


🟠 MAJOR · ci-retries · source: codex

Once timing uploads contain files, a retried unit, integration, or cli-e2e job can fail while uploading an artifact whose name was used by its earlier attempt.

Evidence: test.yml:111-118,151-158,301-308 uploads fixed names under if: always() without overwrite or an attempt suffix. The pinned upload action declares overwrite: false by default and says an existing name causes failure.

Suggested fix: Give artifacts attempt-specific names and update selection, or overwrite the earlier artifact intentionally.

github-actions · 2026-09-29T15:02:58Z

+  if (result.exitCode !== 0) {
+    if (!/no artifacts? found|no valid artifacts/i.test(result.stderr)) {
+      console.warn(`[test-health-report] gh run download ${runId} failed: ${result.stderr.trim()}`);
+    }
+    return false;


🟡 MINOR · error-handling · source: codex

Authentication, API, and network download failures are treated like absent artifacts, allowing a successful report with incomplete timing data.

Evidence: report.ts:163-167 returns false for every nonzero gh run download exit; :244-255 then continues. The note at :271-276 is added only when all timing entries are absent.

Suggested fix: Distinguish a confirmed no-artifacts result from other failures; fail or add a report note for omitted runs.

github-actions · 2026-09-29T15:02:58Z

+    if (maxRuns !== undefined && runs.length > maxRuns) {
+      notes.push(
+        `sampled the ${maxRuns} most recent of ${runs.length} matching runs from the past ` +
+          `${days} day(s) (--max-runs).`,
+      );
+      runs = runs.slice(-maxRuns);


🟡 MINOR · argument-validation · source: codex

--max-runs 0 keeps all runs, and negative values select an unintended subset.

Evidence: report.ts:212 converts the argument with Number; :232-237 applies runs.slice(-maxRuns) without validation.

Suggested fix: Require --max-runs to be a positive integer before applying the cap.

github-actions · 2026-09-29T15:02:58Z

+async function run(command: readonly string[]): Promise<ProcessResult> {
+  const proc = Bun.spawn([...command], { cwd: repoRoot, stdout: "pipe", stderr: "pipe" });
+  const [exitCode, stdout, stderr] = await Promise.all([
+    proc.exited,
+    new Response(proc.stdout).text(),
+    new Response(proc.stderr).text(),
+  ]);
+  return { exitCode, stdout, stderr };


🟡 MINOR · repo-convention · source: codex

The new effectful TypeScript report script bypasses the repository’s documented Effect runtime convention.

Evidence: report.ts:36-43 defines a Promise-returning subprocess helper and the script uses async functions throughout. trusted/CLAUDE.md:30-35 says to write TypeScript runtime code in Effect and reserves Promise-returning APIs for package exports to non-Effect consumers.

Suggested fix: Represent subprocess execution and report construction as Effect operations and run the program at the script entry point.

jgoux · 2026-09-29T15:06:08Z

Closing: an advisory weekly report gates nothing, so it's unlikely to change behaviour. The work moves to shrink-only ratchets that fail PRs (CLI-2545). The branch stays for its shared results-cache parser.

jgoux added 2 commits September 29, 2026 16:44

jgoux self-assigned this Sep 29, 2026

jgoux requested a review from a team as a code owner September 29, 2026 14:53

github-actions Bot reviewed Sep 29, 2026

View reviewed changes

jgoux closed this Sep 29, 2026

Navigation Menu

Repository navigation

ci(repo): publish a weekly test-suite health report - #6891

ci(repo): publish a weekly test-suite health report#6891
jgoux wants to merge 2 commits into
developfrom
juliengoux/cli-2541-publish-a-weekly-test-suite-health-report-from-ci-data

jgoux commented Sep 29, 2026

Uh oh!

github-actions Bot left a comment

Uh oh!

github-actions Bot Sep 29, 2026

Uh oh!

github-actions Bot Sep 29, 2026

Uh oh!

github-actions Bot Sep 29, 2026

Uh oh!

github-actions Bot Sep 29, 2026

Uh oh!

github-actions Bot Sep 29, 2026

Uh oh!

github-actions Bot Sep 29, 2026

Uh oh!

github-actions Bot Sep 29, 2026

Uh oh!

github-actions Bot Sep 29, 2026

Uh oh!

github-actions Bot Sep 29, 2026

Uh oh!

github-actions Bot Sep 29, 2026

Uh oh!

jgoux commented Sep 29, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

jgoux commented Sep 29, 2026

Problem

Change

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

🤖 AI Review

Findings

Stats

Uh oh!

github-actions Bot Sep 29, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Sep 29, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Sep 29, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Sep 29, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Sep 29, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Sep 29, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Sep 29, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Sep 29, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Sep 29, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Sep 29, 2026

Choose a reason for hiding this comment

Uh oh!

jgoux commented Sep 29, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant