Skip to content

fix(ci): stop kill-grace from bounding Windows helper spawns - #1823

Open
tthayer wants to merge 2 commits into
DeusData:mainfrom
tthayer:ci/windows-helper-spawn-timeout
Open

fix(ci): stop kill-grace from bounding Windows helper spawns#1823
tthayer wants to merge 2 commits into
DeusData:mainfrom
tthayer:ci/windows-helper-spawn-timeout

Conversation

@tthayer

@tthayer tthayer commented Aug 24, 2026

Copy link
Copy Markdown

Split out of #1426 at maintainer request — process-control change, reviewed on its own.

Problem

The Windows shard failed Step 0i (parallel suite scheduler contract) with cleanup failed: hang_after_summary: leader exited leaving live descendants while its sibling shard passed. Nothing was actually leaking a process.

run-test-wave.py passed --kill-grace as the subprocess timeout for two external Windows helpers: taskkill.exe /T /F and the powershell.exe Get-CimInstance descendant probe. Those are different quantities. kill_grace budgets how long a doomed process may take to die; the helpers also pay process spawn plus, for PowerShell, CIM startup. Contract fixtures run with --kill-grace 1, which is not reliably enough to launch either helper on a loaded runner.

The two timeouts compounded: a timed-out taskkill is reported as "could not prove process-tree cleanup", which raises out of the wave loop; the finally-block cleanup re-enters with the leader already dead, falls to the descendant probe, which times out on the same one-second budget and takes its "cannot prove absence -> assume the worst" branch. Phantom descendants, red wave.

Change

  • scripts/run-test-wave.py: helper invocations get their own floor, max(kill_grace, WINDOWS_HELPER_TIMEOUT_SECONDS=30). kill_grace still governs every actual death wait, so nothing fails open — the timeout-race contract still refuses with rc=2 over a genuinely surviving descendant, now because PowerShell answered rather than because it timed out. The production path already passed --kill-grace 15 and is unchanged in behaviour.
  • tests/test_parallel_harness_contract.sh: the flat 8s budget for the scheduler's refusal now reads the floor out of the scheduler on Windows rather than restating it — hard-coding it turns a slow runner into a harness failure the moment the two drift. The refusal path can spend the floor twice (proving descendants, then cleanup re-entry), so it allows both plus interpreter startup. POSIX is unchanged at 8s: the refusal there is signal-driven.

The two halves ship together because the contract imports WINDOWS_HELPER_TIMEOUT_SECONDS from the scheduler.

Validation

tests/test_parallel_harness_contract.sh passes locally on darwin arm64. The Windows behaviour it guards is exercised by the Windows shard in CI.

🤖 Generated with Claude Code

Tony Thayer-Osborne and others added 2 commits August 24, 2026 14:29
The Windows shard failed Step 0i (parallel suite scheduler contract) with
"cleanup failed: hang_after_summary: leader exited leaving live
descendants" while its sibling shard passed the same contract. Nothing in
the wave was actually leaking a process.

run-test-wave.py passed --kill-grace as the subprocess timeout for the two
external Windows helpers: taskkill.exe /T /F and the powershell.exe
Get-CimInstance descendant probe. Those are different quantities.
kill_grace budgets how long a doomed process may take to die; the helpers
also have to pay process spawn plus, for PowerShell, CIM startup. The
contract fixtures run with --kill-grace 1, and one second is not reliably
enough to launch either helper on a loaded runner.

The two timeouts then compounded. A timed-out taskkill is reported as
"could not prove process-tree cleanup", which raises out of the wave loop;
the finally-block cleanup re-enters with the leader already dead, so it
falls to the descendant probe, which times out on the same one-second
budget and takes its "cannot prove absence -> assume the worst" branch.
Phantom descendants, red wave.

Give helper invocations their own floor, max(kill_grace, 30). kill_grace
still governs every actual death wait, so nothing fails open: the
timeout-race contract still refuses with rc=2 over a genuinely surviving
descendant, now because PowerShell answered rather than because it timed
out. The production path already passed --kill-grace 15 and is unchanged
in behaviour.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Tony Thayer-Osborne <tony.thayerosborne@conductorone.com>
The parallel-suite scheduler contract gave the scheduler a flat 8-second
budget to finish refusing. On Windows that refusal costs external helper
spawns (taskkill.exe, and powershell.exe for the descendant probe), which
the scheduler now budgets with its own floor rather than --kill-grace.

Read that floor out of the scheduler instead of restating it: hard-coding
a number here silently turns a slow runner into a harness failure the
moment the two drift apart. The refusal path can spend the floor twice --
once proving descendants, once in the cleanup re-entry -- so allow both
plus interpreter startup. POSIX is unchanged: the refusal is signal-driven
and still lands inside the original 8 seconds.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Tony Thayer-Osborne <tony.thayerosborne@conductorone.com>
@github-actions

Copy link
Copy Markdown

Thanks for opening this — it has been seen, and it is queued.

This note is automated, but it is not a brush-off: it exists so you know where your PR stands instead of having to guess from silence.

Current review status: working through a backlog. 0.9.1-rc.1 is out, so the release freeze that held reviews is over — but it left a large queue of open pull requests behind it, and we are reading through them oldest-first. The background is in discussion #1144.

What that means for this PR, concretely:

  • It will not be closed for inactivity. No stale bot touches pull requests here.
  • It may still sit a while before a human reads it. That is on us, not on you.
  • Older PRs are read first, so a recent one is not being skipped — it is behind a queue.

Things that will genuinely speed it up whenever review does happen:

  • Keep it rebased on main — the tree is moving quickly right now, and a conflicting branch cannot be reviewed as the diff you intended.
  • Get CI green, or say which failures you believe are pre-existing.
  • Keep the change to one claim. Bundled features and refactors get split before they get merged, which costs you a round trip.
  • Every commit needs a sign-off (git commit -s) — CI enforces DCO.

If this fixes a bug, a reproduction we can run is worth more than a description of the symptom.

Thanks for contributing, and sorry in advance for the wait.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant