Skip to content

fix: keep test-runner retries from laundering failures - #21

Merged
cursor[bot] merged 2 commits into
developmentfrom
cursor/test-runner-correctness-f78e
Aug 27, 2026
Merged

fix: keep test-runner retries from laundering failures#21
cursor[bot] merged 2 commits into
developmentfrom
cursor/test-runner-correctness-f78e

Conversation

@pavelhov

@pavelhov pavelhov commented Aug 27, 2026

Copy link
Copy Markdown
Owner

Test-runner correctness only. No packaging, OAuth, or product-code changes.

Findings

P1 — shared-process batch fail then isolate-pass was a false green

scripts/test-parallel.ts retried a failed --no-isolate batch by splitting it into per-file --isolate runs. Cross-file contamination that only shows up in a shared process could fail the batch, pass in isolation, and stamp CI green.

Contract: a shared-process batch failure is still a failure unless those files pass again in the same shared-process mode. Isolated reruns are diagnostic only and cannot mark the suite green. Isolate-lane files still retry isolated. Default workers and batch size are unchanged; this is not a “always isolate / drop coverage / lower workers” workaround.

Fix: retryQueueForFailures / retryFailuresInSameMode retry each failed item in the mode it failed in (batches stay batches). If a batch is still red after that, diagnosticIsolateRetryItems runs per-file --isolate for signal and prints a contamination warning when every file passed isolated.

P1 — overlap lock only saw --isolate

findCompetingTestRunners used pgrep -f "bun.*test --isolate". The parallel runner’s --no-isolate batches and the GUI suite’s --parallel spawn were invisible, so two full suites could overlap.

Contract: the lock covers the bun test period, not only --isolate. Wrapper scripts (bun scripts/test*.ts) are not treated as competing runners. GUI now waits on the same lock.

Fix: match bun test / bun.exe test (including --no-isolate and --parallel). gui/scripts/test.ts calls waitForExclusiveRun before spawning.

P2 — retries dropped caller flags

GUI retries spawned bun test --isolate <file> and dropped caller argv such as --timeout 1, so a timeout failure could pass on retry. The proxy runner never forwarded extra bun test flags on any spawn.

Contract: retries forward the caller’s flags unchanged, including --timeout.

Fix: partitionBunTestCliArgs splits flags from targets (and does not treat a following test path as a value for boolean flags like --bail). Proxy bunTestArgvForWorkItem and GUI retryTestArgs splice those flags into every retry spawn.

What was verified in the tree

  • Planner still batches cheap files and isolates server/spawn/HOME-mutating/e2e files (scripts/test-plan.ts). Unchanged.
  • Linux hermetic/systemd fixtures and tests/preload.ts HOME sandbox were not touched.
  • Default worker cap remains min(4, CPU); default batch size remains 8.
  • Full parallel run: 606 files, 239 isolated, 46 batches of up to 8, 4 workers. No worker-count or coverage change.

How tested

Regression tests that fail on the old behavior:

  1. a shared-process batch failure stays a failure if files only pass isolated — batch retry stays --no-isolate; isolate diagnostics passing does not recover the run. A sibling test allows recovery only when the same batch passes in shared-process mode.
  2. overlap lock matches bun test including --no-isolate plus a live bun test --no-isolate child that findCompetingTestRunners must see.
  3. Proxy and GUI: retry argv keeps --timeout / extra flags.

Local checks on this revision (70b324d):

  • bun test tests/test-runner.test.ts — 14 pass
  • cd gui && bun test tests/gui-test-runner.test.ts — 5 pass
  • bun run typecheck
  • cd gui && bun run lint
  • cd gui && bun run test — 869 pass, 139 files
  • bun run privacy:scan
  • bun run test:parallel — exit 0 at default workers (~109s, 606 files)

GitHub ci is the merge gate. Target is development, not main.

Open in Web Open in Cursor 

cursoragent and others added 2 commits August 27, 2026 18:50
Shared-process batch failures retry in the same mode; isolated passes
cannot stamp the suite green. The overlap lock covers all bun test
processes, and retries forward caller flags including --timeout.

Co-authored-by: pavelhov <pavelhov@users.noreply.github.com>
--bail followed by a test path was swallowed as a flag value, which
would have dropped files from the run and from retries.

Co-authored-by: pavelhov <pavelhov@users.noreply.github.com>
@pavelhov
pavelhov marked this pull request as ready for review August 27, 2026 18:58
@cursor
cursor Bot merged commit 70b324d into development Aug 27, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants