ci(cli): judge block-scoped benchmark timings by per-block spread by Prashansa-K · Pull Request #7433 · supabase/cli

Prashansa-K · 2026-11-02T06:30:29Z

Follow-up to #7412.

Block-scoped workloads started one stack per block of five launches, so a ten-run workload gave each build only two stack sessions. Launches in one block share a stack session, so session-to-session spread could push the branch median outside the base range, and an A/A run (same CLI code as base and branch) was flagged as slower or faster.

Blocks are now two launches, so each build gets five stack sessions at ten runs, and each measured launch records its block index in the report.
With at least three paired blocks, the block is the unit of evidence. The median of the per-block deltas must clear the existing 5 percent and 5 ms floor, a strict majority of blocks must move the same way past that floor, and the median delta must exceed 1.5 times the larger interquartile range of the two builds' block medians. One odd block cannot create or hide a verdict, and mixed-sign blocks read as within noise.
Run-scoped workloads keep the existing rule (the branch median outside the base min-max range) and also need the median delta to exceed 1.5 times the larger interquartile range of the two builds' launches, so a wide spread reads as noise.
estimatedSeconds for block-scoped workloads is rescaled for the extra stack sessions, and tracing-monitoring.md describes the new rule.

False positives may not be entirely eliminated. A follow-up could allow re-running one or more chosen workloads, so a reported regression can be confirmed without re-running the whole benchmark.

🤖 Generated with Claude Code

Block-scoped workloads now run in blocks of two launches, so each build gets five stack sessions at ten runs. Launches record their block index, and the timing verdict needs a strict majority of paired blocks to move the same way past the noise floor before it reads slower or faster. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

github-actions · 2026-11-02T06:32:39Z

CLI benchmark (Lightweight Commands): no changes found

Commit d848714 against develop at a728de0 · 10 runs each · report only, never blocks merging

Command	Workload	Before	After	Change
`supabase --version`	`version`	176 ms	179 ms	no real change
`supabase --help`	`help`	177 ms	174 ms	no real change
`supabase init`	`init`	178 ms	181 ms	no real change
`supabase migration new bench`	`migration-new`	189 ms	190 ms	no real change
`supabase gen types --lang typescript --project-id replayfixtureprojref`	`gen-types`	216 ms	216 ms	no real change
`supabase projects list`	`projects-list`	192 ms	198 ms	no real change
`supabase functions list --project-ref replayfixtureprojref`	`functions-list`	201 ms	205 ms	no real change

API calls: unchanged for all 7 commands.

Binary size: 134.0 MB, unchanged.

Details: timing ranges, API calls per command, slowest phases

Timing ranges

Command	Workload	Before range	After range	Change	Verdict
`supabase --version`	`version`	168-189 ms	166-195 ms	+3 ms (+2%)	within noise
`supabase --help`	`help`	169-185 ms	169-193 ms	-3 ms (-2%)	within noise
`supabase init`	`init`	171-185 ms	171-189 ms	+3 ms (+2%)	within noise
`supabase migration new bench`	`migration-new`	182-211 ms	180-199 ms	+1 ms (+0%)	within noise
`supabase gen types --lang typescript --project-id replayfixtureprojref`	`gen-types`	201-232 ms	206-224 ms	-1 ms (0%)	within noise
`supabase projects list`	`projects-list`	186-202 ms	191-210 ms	+7 ms (+4%)	within noise
`supabase functions list --project-ref replayfixtureprojref`	`functions-list`	191-211 ms	194-210 ms	+4 ms (+2%)	within noise

API calls per command

Command	Workload	Before	After
`supabase --version`	`version`	0	0
`supabase --help`	`help`	0	0
`supabase init`	`init`	0	0
`supabase migration new bench`	`migration-new`	0	0
`supabase gen types --lang typescript --project-id replayfixtureprojref`	`gen-types`	2	2
`supabase projects list`	`projects-list`	1	1
`supabase functions list --project-ref replayfixtureprojref`	`functions-list`	2	2

Slowest phases

No command moved beyond noise, so there are no phase deltas.

Workflow run

Comment /bench to run the lightweight benchmark (this comment) again on the latest commit, or /bench full to also run the integration-heavy benchmark, which posts in its own comment.

Previous runs

4573f67 · 2026-11-02 06:32 UTC · no changes found · run

github-actions

🤖 AI Review

Both independent reviews were available. Confirmed Claude’s test-coverage nit; no runtime defects identified. Corrected the finding’s line numbers to match the checked-out code.

Findings

Severity	Location	Category	Sources	Claim
⚪ NIT	`apps/cli/scripts/bench-report.unit.test.ts:220`	`test-coverage`	claude	The mixed-sign blocks test passes even if the block agreement check is bypassed. It therefore does not verify that disagreement suppresses a verdict the fallback range check would flag.

Stats

Claude findings: 1 · Codex findings: 0 · Confirmed: 1 · Refuted: 0 · Uncertain: 0

Models: claude-opus-5-5 + gpt-6.1-sol · Trigger: auto · Workflow run

This review runs once per PR. A maintainer can request another with a /ai-review comment.

Prashansa-K · 2026-11-02T06:37:55Z

/bench full

github-actions · 2026-11-02T06:38:21Z

CLI benchmark (Integration Heavy Commands): 1 thing to check

Commit d848714 against develop at a728de0 · 5-10 runs each · report only, never blocks merging

Command	Workload	Runtime	Before	After	Change
`supabase start`	`start`	docker	17124 ms	17237 ms	no real change
`supabase stop --no-backup`	`stop`	docker	4818 ms	5146 ms	no real change (+7%, within normal variation)
`supabase db start`	`db-start`	docker	10999 ms	11086 ms	no real change
`supabase status`	`status`	docker	284 ms	269 ms	no real change (-6%, within normal variation)
`supabase db query --local select 1`	`db-query`	docker	493 ms	392 ms	no real change (-20%, within normal variation)
`supabase db reset`	`db-reset`	docker	16656 ms	16587 ms	no real change
`supabase test db`	`test-db`	docker	643 ms	705 ms	no real change (+10%, within normal variation)
`supabase gen types --lang typescript --local`	`gen-types-small`	docker	256 ms	246 ms	no real change
`supabase gen types --lang typescript --local`	`gen-types-medium`	docker	282 ms	264 ms	no real change (-6%, within normal variation)
`supabase gen types --lang typescript --local`	`gen-types-large`	docker	343 ms	343 ms	no real change
`supabase db diff --use-pg-delta`	`db-diff-cold`	docker	10054 ms	9893 ms	no real change
`supabase db diff --use-pg-delta`	`db-diff-warm`	docker	1708 ms	1670 ms	no real change
`supabase db dump --local`	`db-dump`	docker	429 ms	412 ms	no real change
`supabase db lint --local`	`db-lint`	docker	226 ms	258 ms	no real change (+14%, within normal variation)
`supabase migration up --local`	`migration-up`	docker	259 ms	256 ms	no real change
`supabase db push --local`	`db-push-local`	docker	295 ms	310 ms	no real change (+5%, within normal variation)
`supabase migration list --local`	`migration-list`	docker	320 ms	280 ms	no real change (-12%, within normal variation)
`supabase stack start --runtime docker --eager`	`stack-start-docker`	stack, docker	9514 ms	8833 ms	no real change (-7%, within normal variation)
`supabase stack status`	`stack-status-docker`	stack, docker	228 ms	239 ms	no real change (+5%, within normal variation)
`supabase stack stop`	`stack-stop-docker`	stack, docker	7003 ms	6960 ms	no real change
`supabase db query --local select 1`	`stack-db-query-docker`	stack, docker	393 ms	400 ms	no real change
`supabase db reset`	`stack-db-reset-docker`	stack, docker	13505 ms	14002 ms	no real change
`supabase stack start --runtime native --eager`	`stack-start-native`	stack, native	7706 ms	6915 ms	faster by 791 ms (-10%)
`supabase stack status`	`stack-status-native`	stack, native	250 ms	232 ms	no real change (-7%, within normal variation)
`supabase stack stop`	`stack-stop-native`	stack, native	3383 ms	3409 ms	no real change
`supabase db query --local select 1`	`stack-db-query-native`	stack, native	393 ms	372 ms	no real change (-5%, within normal variation)
`supabase db reset`	`stack-db-reset-native`	stack, native	10404 ms	10593 ms	no real change

API calls: none, before or after, for all 27 commands.

Container operations: unchanged for all 17 commands.

Binary size: 134.0 MB, unchanged.

Details: timing ranges, API calls per command, container operations, slowest phases

Timing ranges

Command	Workload	Before range	After range	Change	Verdict
`supabase start`	`start`	16721-18086 ms	16735-18255 ms	+114 ms (+1%)	within noise
`supabase stop --no-backup`	`stop`	4642-5465 ms	4652-5702 ms	+328 ms (+7%)	within noise
`supabase db start`	`db-start`	10546-11503 ms	10693-11411 ms	+88 ms (+1%)	within noise
`supabase status`	`status`	201-400 ms	212-377 ms	-16 ms (-6%)	within noise
`supabase db query --local select 1`	`db-query`	197-660 ms	208-547 ms	-101 ms (-20%)	within noise
`supabase db reset`	`db-reset`	16061-18633 ms	16088-17201 ms	-69 ms (0%)	within noise
`supabase test db`	`test-db`	592-783 ms	651-745 ms	+61 ms (+10%)	within noise
`supabase gen types --lang typescript --local`	`gen-types-small`	239-392 ms	219-281 ms	-9 ms (-4%)	within noise
`supabase gen types --lang typescript --local`	`gen-types-medium`	246-303 ms	242-316 ms	-17 ms (-6%)	within noise
`supabase gen types --lang typescript --local`	`gen-types-large`	332-456 ms	282-458 ms	+0 ms (+0%)	within noise
`supabase db diff --use-pg-delta`	`db-diff-cold`	8694-12234 ms	6072-10676 ms	-162 ms (-2%)	within noise
`supabase db diff --use-pg-delta`	`db-diff-warm`	1522-1760 ms	1599-1691 ms	-38 ms (-2%)	within noise
`supabase db dump --local`	`db-dump`	379-599 ms	340-515 ms	-17 ms (-4%)	within noise
`supabase db lint --local`	`db-lint`	208-276 ms	205-307 ms	+32 ms (+14%)	within noise
`supabase migration up --local`	`migration-up`	255-350 ms	240-324 ms	-3 ms (-1%)	within noise
`supabase db push --local`	`db-push-local`	249-526 ms	275-586 ms	+15 ms (+5%)	within noise
`supabase migration list --local`	`migration-list`	199-578 ms	215-644 ms	-40 ms (-12%)	within noise
`supabase stack start --runtime docker --eager`	`stack-start-docker`	8244-14772 ms	8359-13416 ms	-681 ms (-7%)	within noise
`supabase stack status`	`stack-status-docker`	204-270 ms	209-313 ms	+12 ms (+5%)	within noise
`supabase stack stop`	`stack-stop-docker`	6711-7119 ms	6771-7367 ms	-44 ms (-1%)	within noise
`supabase db query --local select 1`	`stack-db-query-docker`	321-685 ms	285-649 ms	+7 ms (+2%)	within noise
`supabase db reset`	`stack-db-reset-docker`	13341-27975 ms	12812-18992 ms	+497 ms (+4%)	within noise
`supabase stack start --runtime native --eager`	`stack-start-native`	6942-8318 ms	6403-6930 ms	-791 ms (-10%)	faster
`supabase stack status`	`stack-status-native`	221-309 ms	223-280 ms	-18 ms (-7%)	within noise
`supabase stack stop`	`stack-stop-native`	3343-3441 ms	3348-3496 ms	+26 ms (+1%)	within noise
`supabase db query --local select 1`	`stack-db-query-native`	324-542 ms	282-441 ms	-21 ms (-5%)	within noise
`supabase db reset`	`stack-db-reset-native`	9578-10710 ms	9468-10638 ms	+189 ms (+2%)	within noise

API calls per command

Command	Workload	Before	After	Endpoint changes
`supabase start`	`start`	0	0
`supabase stop --no-backup`	`stop`	0	0
`supabase db start`	`db-start`	0	0
`supabase status`	`status`	0	0
`supabase db query --local select 1`	`db-query`	0	0
`supabase db reset`	`db-reset`	0	0
`supabase test db`	`test-db`	0	0
`supabase gen types --lang typescript --local`	`gen-types-small`	0	0
`supabase gen types --lang typescript --local`	`gen-types-medium`	0	0
`supabase gen types --lang typescript --local`	`gen-types-large`	0	0
`supabase db diff --use-pg-delta`	`db-diff-cold`	0	0
`supabase db diff --use-pg-delta`	`db-diff-warm`	0	0
`supabase db dump --local`	`db-dump`	0	0
`supabase db lint --local`	`db-lint`	0	0
`supabase migration up --local`	`migration-up`	0	0
`supabase db push --local`	`db-push-local`	0	0
`supabase migration list --local`	`migration-list`	0	0
`supabase stack start --runtime docker --eager`	`stack-start-docker`	0	0
`supabase stack status`	`stack-status-docker`	0	0
`supabase stack stop`	`stack-stop-docker`	0	0
`supabase db query --local select 1`	`stack-db-query-docker`	0	0
`supabase db reset`	`stack-db-reset-docker`	0	0
`supabase stack start --runtime native --eager`	`stack-start-native`	0	0
`supabase stack status`	`stack-status-native`	0	0
`supabase stack stop`	`stack-stop-native`	0	0
`supabase db query --local select 1`	`stack-db-query-native`	0	0
`supabase db reset`	`stack-db-reset-native`	0	0

Container operations per command

Command	Workload	Runtime	Operation	Before	After
`supabase start`	`start`	docker	`cp`	3	3
`supabase start`	`start`	docker	`create`	12	12
`supabase start`	`start`	docker	`image inspect`	17	17
`supabase start`	`start`	docker	`network create`	1	1
`supabase start`	`start`	docker	`network inspect`	2	2
`supabase start`	`start`	docker	`run`	3	3
`supabase start`	`start`	docker	`start`	12	12
`supabase start`	`start`	docker	`volume create`	3	3
`supabase start`	`start`	docker	`volume inspect`	1	1
`supabase stop --no-backup`	`stop`	docker	`container prune`	1	1
`supabase stop --no-backup`	`stop`	docker	`network prune`	1	1
`supabase stop --no-backup`	`stop`	docker	`ps`	1	1
`supabase stop --no-backup`	`stop`	docker	`stop`	12	12
`supabase stop --no-backup`	`stop`	docker	`volume ls`	1	1
`supabase stop --no-backup`	`stop`	docker	`volume prune`	1	1
`supabase db start`	`db-start`	docker	`cp`	1	1
`supabase db start`	`db-start`	docker	`create`	1	1
`supabase db start`	`db-start`	docker	`image inspect`	4	4
`supabase db start`	`db-start`	docker	`network create`	1	1
`supabase db start`	`db-start`	docker	`network inspect`	1	1
`supabase db start`	`db-start`	docker	`run`	3	3
`supabase db start`	`db-start`	docker	`start`	1	1
`supabase db start`	`db-start`	docker	`volume create`	1	1
`supabase db start`	`db-start`	docker	`volume inspect`	1	1
`supabase status`	`status`	docker	`ps`	1	1
`supabase db reset`	`db-reset`	docker	`container rm`	1	1
`supabase db reset`	`db-reset`	docker	`cp`	1	1
`supabase db reset`	`db-reset`	docker	`create`	1	1
`supabase db reset`	`db-reset`	docker	`exec`	1	1
`supabase db reset`	`db-reset`	docker	`image inspect`	4	4
`supabase db reset`	`db-reset`	docker	`network inspect`	1	1
`supabase db reset`	`db-reset`	docker	`restart`	4	4
`supabase db reset`	`db-reset`	docker	`run`	3	3
`supabase db reset`	`db-reset`	docker	`start`	1	1
`supabase db reset`	`db-reset`	docker	`stop`	4	4
`supabase db reset`	`db-reset`	docker	`volume create`	1	1
`supabase db reset`	`db-reset`	docker	`volume rm`	1	1
`supabase test db`	`test-db`	docker	`image inspect`	1	1
`supabase test db`	`test-db`	docker	`run`	1	1
`supabase gen types --lang typescript --local`	`gen-types-small`	docker	`container inspect`	1	1
`supabase gen types --lang typescript --local`	`gen-types-medium`	docker	`container inspect`	1	1
`supabase gen types --lang typescript --local`	`gen-types-large`	docker	`container inspect`	1	1
`supabase db diff --use-pg-delta`	`db-diff-cold`	docker	`cp`	3	3
`supabase db diff --use-pg-delta`	`db-diff-cold`	docker	`create`	1	1
`supabase db diff --use-pg-delta`	`db-diff-cold`	docker	`image inspect`	4	4
`supabase db diff --use-pg-delta`	`db-diff-cold`	docker	`network inspect`	1	1
`supabase db diff --use-pg-delta`	`db-diff-cold`	docker	`rm`	1	1
`supabase db diff --use-pg-delta`	`db-diff-cold`	docker	`run`	3	3
`supabase db diff --use-pg-delta`	`db-diff-cold`	docker	`start`	2	2
`supabase db diff --use-pg-delta`	`db-diff-cold`	docker	`stop`	1	1
`supabase db diff --use-pg-delta`	`db-diff-warm`	docker	`cp`	2	2
`supabase db diff --use-pg-delta`	`db-diff-warm`	docker	`create`	1	1
`supabase db diff --use-pg-delta`	`db-diff-warm`	docker	`image inspect`	1	1
`supabase db diff --use-pg-delta`	`db-diff-warm`	docker	`network inspect`	1	1
`supabase db diff --use-pg-delta`	`db-diff-warm`	docker	`rm`	1	1
`supabase db diff --use-pg-delta`	`db-diff-warm`	docker	`start`	1	1
`supabase db dump --local`	`db-dump`	docker	`image inspect`	1	1
`supabase db dump --local`	`db-dump`	docker	`run`	1	1

Slowest phases

Command	Workload	Span	Change	Before -> after
`supabase stack start --runtime native --eager`	`stack-start-native`	`experimental.stack.start`	-932 ms	7437 -> 6505 ms
`supabase stack start --runtime native --eager`	`stack-start-native`	`command.stack.start`	-932 ms	7438 -> 6506 ms
`supabase stack start --runtime native --eager`	`stack-start-native`	`cli.run`	-928 ms	7460 -> 6533 ms

Workflow run

Comment /bench full to run this benchmark again on the latest commit, which also refreshes the lightweight comment if it is behind. /bench runs only the lightweight benchmark.

Previous runs

d848714 · 2026-11-02 11:54 UTC · 1 thing to check · run
d848714 · 2026-11-02 07:24 UTC · 2 things to check · run
4573f67 · 2026-11-02 06:55 UTC · 2 things to check · run

…timing verdict Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Prashansa-K · 2026-11-02T07:05:46Z

/bench full

avallete

LGTM, only a few questions.

Also, I wonder if instead of a global rule (2%-5ms) we could instead have this declared as data with later on a way to automatically update it within the workload.json ?

That would allow an "per workload tolerance" rather than a global once. As for instance I expect the variance for a supabase --version to be much smaller than a supabase start.

I'm thinking if we could have hand-written tolerance per workload in workloads.*.json (for example noisePct: 3 for --version, 20 for db start) that might allow to take the variance/side-effect in consideration better and reduce the false negatives ?

Just an idea and could be done in a follow-up PR though.

Block-scoped workloads judge the median of per-block deltas against the block medians' spread, so one odd stack session can neither flag nor hide a change. Run-scoped workloads keep the base range check and also need the change to clear the launches' spread. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Prashansa-K and others added 2 commits November 2, 2026 11:32

ci(cli): rescale block-scoped benchmark estimates for blocks of two

4573f67

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Prashansa-K requested a review from a team as a code owner November 2, 2026 06:30

github-actions Bot reviewed Nov 2, 2026

View reviewed changes

Comment thread apps/cli/scripts/bench-report.unit.test.ts Resolved

ci(cli): require each build's median outside the other's range for a …

d848714

…timing verdict Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

avallete approved these changes Nov 2, 2026

View reviewed changes

Comment thread apps/cli/scripts/bench-report.ts Resolved

Comment thread apps/cli/scripts/bench-report.ts Resolved

Prashansa-K changed the title ~~ci(cli): judge block-scoped benchmark timings by per-block agreement~~ ci(cli): judge block-scoped benchmark timings by per-block spread Nov 2, 2026

Prashansa-K added this pull request to the merge queue Nov 2, 2026

Merged via the queue into develop with commit c3f3ef2 Nov 2, 2026
61 of 62 checks passed

Prashansa-K deleted the ci/cli-bench-block-verdict branch November 2, 2026 15:01

supabase-cli-releaser Bot mentioned this pull request Nov 3, 2026

chore: production deploy #7452

Merged

Navigation Menu

Repository navigation

ci(cli): judge block-scoped benchmark timings by per-block spread - #7433

ci(cli): judge block-scoped benchmark timings by per-block spread#7433
Prashansa-K merged 4 commits into
developfrom
ci/cli-bench-block-verdict

Prashansa-K commented Nov 2, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Nov 2, 2026 •

edited

Loading

Timing ranges

API calls per command

Slowest phases

Uh oh!

github-actions Bot left a comment

Uh oh!

Uh oh!

Prashansa-K commented Nov 2, 2026

Uh oh!

github-actions Bot commented Nov 2, 2026 •

edited

Loading

Timing ranges

API calls per command

Container operations per command

Slowest phases

Uh oh!

Prashansa-K commented Nov 2, 2026

Uh oh!

avallete left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Prashansa-K commented Nov 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Nov 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CLI benchmark (Lightweight Commands): no changes found

Timing ranges

API calls per command

Slowest phases

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

🤖 AI Review

Findings

Stats

Uh oh!

Uh oh!

Prashansa-K commented Nov 2, 2026

Uh oh!

github-actions Bot commented Nov 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CLI benchmark (Integration Heavy Commands): 1 thing to check

Timing ranges

API calls per command

Container operations per command

Slowest phases

Uh oh!

Prashansa-K commented Nov 2, 2026

Uh oh!

avallete left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Prashansa-K commented Nov 2, 2026 •

edited

Loading

github-actions Bot commented Nov 2, 2026 •

edited

Loading

github-actions Bot commented Nov 2, 2026 •

edited

Loading