fix: keep test-runner retries from laundering failures - #21
Merged
Conversation
Shared-process batch failures retry in the same mode; isolated passes cannot stamp the suite green. The overlap lock covers all bun test processes, and retries forward caller flags including --timeout. Co-authored-by: pavelhov <pavelhov@users.noreply.github.com>
--bail followed by a test path was swallowed as a flag value, which would have dropped files from the run and from retries. Co-authored-by: pavelhov <pavelhov@users.noreply.github.com>
pavelhov
marked this pull request as ready for review
August 27, 2026 18:58
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Test-runner correctness only. No packaging, OAuth, or product-code changes.
Findings
P1 — shared-process batch fail then isolate-pass was a false green
scripts/test-parallel.tsretried a failed--no-isolatebatch by splitting it into per-file--isolateruns. Cross-file contamination that only shows up in a shared process could fail the batch, pass in isolation, and stamp CI green.Contract: a shared-process batch failure is still a failure unless those files pass again in the same shared-process mode. Isolated reruns are diagnostic only and cannot mark the suite green. Isolate-lane files still retry isolated. Default workers and batch size are unchanged; this is not a “always isolate / drop coverage / lower workers” workaround.
Fix:
retryQueueForFailures/retryFailuresInSameModeretry each failed item in the mode it failed in (batches stay batches). If a batch is still red after that,diagnosticIsolateRetryItemsruns per-file--isolatefor signal and prints a contamination warning when every file passed isolated.P1 — overlap lock only saw
--isolatefindCompetingTestRunnersusedpgrep -f "bun.*test --isolate". The parallel runner’s--no-isolatebatches and the GUI suite’s--parallelspawn were invisible, so two full suites could overlap.Contract: the lock covers the
bun testperiod, not only--isolate. Wrapper scripts (bun scripts/test*.ts) are not treated as competing runners. GUI now waits on the same lock.Fix: match
bun test/bun.exe test(including--no-isolateand--parallel).gui/scripts/test.tscallswaitForExclusiveRunbefore spawning.P2 — retries dropped caller flags
GUI retries spawned
bun test --isolate <file>and dropped caller argv such as--timeout 1, so a timeout failure could pass on retry. The proxy runner never forwarded extrabun testflags on any spawn.Contract: retries forward the caller’s flags unchanged, including
--timeout.Fix:
partitionBunTestCliArgssplits flags from targets (and does not treat a following test path as a value for boolean flags like--bail). ProxybunTestArgvForWorkItemand GUIretryTestArgssplice those flags into every retry spawn.What was verified in the tree
scripts/test-plan.ts). Unchanged.tests/preload.tsHOME sandbox were not touched.min(4, CPU); default batch size remains 8.How tested
Regression tests that fail on the old behavior:
a shared-process batch failure stays a failure if files only pass isolated— batch retry stays--no-isolate; isolate diagnostics passing does not recover the run. A sibling test allows recovery only when the same batch passes in shared-process mode.overlap lock matches bun test including --no-isolateplus a livebun test --no-isolatechild thatfindCompetingTestRunnersmust see.--timeout/ extra flags.Local checks on this revision (
70b324d):bun test tests/test-runner.test.ts— 14 passcd gui && bun test tests/gui-test-runner.test.ts— 5 passbun run typecheckcd gui && bun run lintcd gui && bun run test— 869 pass, 139 filesbun run privacy:scanbun run test:parallel— exit 0 at default workers (~109s, 606 files)GitHub
ciis the merge gate. Target isdevelopment, notmain.