Repository navigation
src: fix hang when joining workers in process.exit() - #66171
manzoorwanijk wants to merge 2 commits into
Conversation
|
Welcome to Node.js, and thank you for your first contribution! Before review, please take a moment to read:
Please make sure every commit is signed off. For a first pull request, GitHub Actions require collaborator approval and Jenkins CI must be started by a collaborator or triager, so an initial wait is normal. |
|
I have a question: Will this fix be backported to v24 when/if it lands? |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #66171 +/- ##
==========================================
- Coverage 90.74% 90.72% -0.03%
==========================================
Files 798 798
Lines 280083 280115 +32
Branches 53900 53895 -5
==========================================
- Hits 254166 254137 -29
- Misses 16217 16264 +47
- Partials 9700 9714 +14
🚀 New features to boost your workflow:
|
…13 of 30 against 0 of 30 The 28 September list, item 15: find the mechanism, add no timeout. Reproduced, then read off the stuck process: its main thread waits in uv_thread_join under node::DefaultProcessExitHandler, and the one Node thread left sleeps on a condition variable inside V8 — nodejs/node#54918 and #64274, process.exit() joining platform workers while a V8 background job is parked waiting for a GC the exiting thread never runs. The upstream fix (nodejs/node#66171) is unmerged; this machine runs Node 24.12.0. Separated by treatment rather than argued: 30 alternating pairs under the same launch, the proof as shipped hung 13 times after printing its pass, and the same file ending by process.exitCode hung 0 times. The class is 289 process.exit( calls in 147 scripts plus the engine host entries; converting them is its own unit (a script relying on process.exit to cut an open handle would trade this hang for another), recorded with the shape it should take. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
8da904b to
8816d66
Compare
…ent compilers On Node 24 `process.exit` can deadlock joining V8's platform workers (a concurrent Sparkplug or Maglev compile waits for a GC the main thread never runs), so a run that got Ctrl-C or `sandcastle stop` could write its end and then never exit (nodejs/node#66171, still open). `exitOnSignal` no longer calls `process.exit`: it sets the exit code to 128 + n, emits `exit` once so every listener runs, drops its own listener and re-sends the signal with the default action back. The kernel ends the process, and the shell still sees 129, 130 or 143. `bin/sandcastle` passes `--no-maglev --no-concurrent-sparkplug` on both exec lines; a detached run inherits them through `process.execArgv`. Tests: the signals fixture runs with the launcher's flags, asserts death by the signal, the exit listeners once with 128 + n, and no `process.exit` call; the launcher, mod and notify tests follow the new exec line and the signal death. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…63788) On Node 24 and 26, process.exit() joins V8's platform workers without disposing the isolate. A concurrent Maglev or Sparkplug compile job parked waiting for a main-thread GC then never finishes, so CLI commands, the hook relay, or a stopping Gateway can sit at 0% CPU forever after printing their output (nodejs/node#64274; fix pending in nodejs/node#66171). Every OpenClaw executable entrypoint (CLI entry, Gateway/service index, native hook relay, macOS node worker) now turns off Maglev and concurrent Sparkplug at startup when it runs as main, matching Node 22's tiering and the existing Vitest policy. Library imports keep stock V8; explicit --maglev or --concurrent-sparkplug flags still win. Testbox A/B (4 vCPU, Node 24.19, e2e CLI child replay, 8-way): 6/1019 hung with defaults, 0/1019 with the policy; re-enabling only Maglev hung 2/1019 and only concurrent Sparkplug 12/944, so both flags are needed.
|
Can we please expedite this bug fix? |
…enclaw#163788) On Node 24 and 26, process.exit() joins V8's platform workers without disposing the isolate. A concurrent Maglev or Sparkplug compile job parked waiting for a main-thread GC then never finishes, so CLI commands, the hook relay, or a stopping Gateway can sit at 0% CPU forever after printing their output (nodejs/node#64274; fix pending in nodejs/node#66171). Every OpenClaw executable entrypoint (CLI entry, Gateway/service index, native hook relay, macOS node worker) now turns off Maglev and concurrent Sparkplug at startup when it runs as main, matching Node 22's tiering and the existing Vitest policy. Library imports keep stock V8; explicit --maglev or --concurrent-sparkplug flags still win. Testbox A/B (4 vCPU, Node 24.19, e2e CLI child replay, 8-way): 6/1019 hung with defaults, 0/1019 with the policy; re-enabling only Maglev hung 2/1019 and only concurrent Sparkplug 12/944, so both flags are needed.
…nd a time limit Tests that spawned the CLI ran tsx's cli.mjs or .bin/tsx without the launcher's --no-maglev --no-concurrent-sparkplug, so they could hit the Node 24 exit deadlock (nodejs/node#66171), and their spawnSync calls had no timeout, so one stuck exit hung pnpm test, every sandbox gate run and full-check.sh. test/cli-spawn.ts (runKit, runNode, startKit, startNode) reads the flags from bin/sandcastle, runs node with tsx's loader in one process as the launcher does, and kills a child that outlives its limit (60 s, SIGKILL) with the test failing and the command named. Every test that spawned a kit script now uses it, and test/cli-spawn.test.ts refuses a test file that names a tsx entry itself. pnpm test gets --test-timeout=300000, which fails an async hang by name; it cannot interrupt a synchronous spawnSync, which is what the helper's limit is for. Where the ticket left a choice: the helper also covers the Herdr plugin entry (script option) and the non-CLI fixtures that import src/, since the guard test would otherwise need an allowlist for them. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…enclaw#163788) On Node 24 and 26, process.exit() joins V8's platform workers without disposing the isolate. A concurrent Maglev or Sparkplug compile job parked waiting for a main-thread GC then never finishes, so CLI commands, the hook relay, or a stopping Gateway can sit at 0% CPU forever after printing their output (nodejs/node#64274; fix pending in nodejs/node#66171). Every OpenClaw executable entrypoint (CLI entry, Gateway/service index, native hook relay, macOS node worker) now turns off Maglev and concurrent Sparkplug at startup when it runs as main, matching Node 22's tiering and the existing Vitest policy. Library imports keep stock V8; explicit --maglev or --concurrent-sparkplug flags still win. Testbox A/B (4 vCPU, Node 24.19, e2e CLI child replay, 8-way): 6/1019 hung with defaults, 0/1019 with the policy; re-enabling only Maglev hung 2/1019 and only concurrent Sparkplug 12/944, so both flags are needed. (cherry picked from commit 1aa0aeb) Co-authored-by: RomneyDa <6581799+RomneyDa@users.noreply.github.com>
review wanted
|
Thank you |
|
Independent reproduction on macOS (Darwin 25.5), Apple Silicon, Node v26.5.0 (Homebrew):
|
7511be4 to
ed0841e
Compare
aduh95
left a comment
There was a problem hiding this comment.
The added test seems to be flaky on main (i.e. without the fix), maybe it's related to the new V8 version. Here's the result of tools/test.py -t 20 --repeat=99 test/parallel/test-process-exit-background-gc.js on my machine
# Without my suggestion
[02:11|% 100|+ 64|- 35]: Done
# With my suggestion
[03:05|% 100|+ 54|- 45]: Done
This comment was marked as resolved.
This comment was marked as resolved.
6c081ed to
58e671d
Compare
Done. Thank you for the reminder. |
A pack-installs-alone test hung in a pre-merge test run. An npm install the script ran never exited, and the test timed out at 600 s. Run on a loaded host with Node 24.22.0 and npm 11.19.1, the test hung 6 times in 862 runs, each time in the script's npm install after npm had logged "info ok". Each hung npm's main thread was waiting to join one of its own V8 worker threads. That is nodejs/node#64274, where process.exit() joins V8's workers while one is parked waiting for a garbage collection only the main thread runs. npm 11 ends every command with process.exit(). The reproduction in the proposed fix, nodejs/node#66171, hung 11 of 50 times on that host, and 0 of 50 with --single-threaded, which disables V8's background tasks. scripts/single-threaded.mjs says how npm and Node are started: - npmCommand(cwd, path) takes the first executable npm in path's directories, each resolved against cwd, the directory the child runs in, which is where the child reads a relative or empty entry from. Where that npm's real path is a JavaScript file, as an npm installed with Node's is, it runs under this Node with --single-threaded, which NODE_OPTIONS refuses, by the path it was found at. Any other npm, such as a test's shim or a shell script, is started by name as before, and so is npm when PATH is unset. - The script asks once for npm pack, from its own directory, and once for npm install, from the install's. Its loads, the bin's load and a CommonJS bin's --check run under --single-threaded. The test starts the real npm the same way, in its own npm pack and in the shims that hand calls to it, and its stand-in for bun starts Node with the flag. Its own npm pack, which had no limit, now kills a hung npm at two minutes and fails, naming npm pack and its directory, rather than blocking the test's process past the runner's timeout. Tests: - npmCommand's walk: .js, .cjs and .mjs entries; a non-executable npm and a directory named npm passed over; a first npm that isn't JavaScript, a PATH with no npm, and an unset PATH, by name; relative and empty entries read from the child's directory. - npm starts under the flag where PATH's npm is a JavaScript file, and by name otherwise, in the script and in the tests' own npm pack. - npm pack, npm install and the tests' own npm pack each read a relative PATH entry from their own directory. - Through a NODE_OPTIONS preload: both loads, each bin's load, a CommonJS bin's --check and a JavaScript npm start under the flag, and each Node the script starts with its environment is one of them. - The tests' own npm pack fails at its limit, naming the command and its directory, and other failures stay npm's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…the scratch dir (code-yeongyu#10911) npm 11 ends every command with process.exit(). On Node 24 that call joins V8's platform workers while one can be parked waiting for a GC only the main thread runs (nodejs/node#64274, fix pending in nodejs/node#66171), so npm can hang after printing the tarball name: the 25 s npm pack stall on hosted Windows. The packed-sidecar test now runs npm's JavaScript entry as node --single-threaded npm-cli.js pack <pkg>, from the filesystem root: one process instead of cmd.exe -> npm.cmd -> node, so the tree kill always reaches it, and no process keeps the scratch dir as its current directory.
58e671d to
da20842
Compare
On macos-15 a compiled CLI that rejected its input printed the VALIDATION_ERROR and then never exited (tests/boundary/cli/nativeDemangleCli.test.ts timed out after 20 s). A diagnostic run reproduced it in 2 of 700 runs; spindump showed a deadlock inside process.exit(): the main thread was in DisposePlatform, joining V8's platform workers, while those workers (concurrent Sparkplug and Maglev compiles) waited in CollectionBarrier::AwaitCollectionBackground for a garbage collection that only the exiting main thread could run. This is nodejs/node#64274; its fix, nodejs/node#66171, is not released. Incur ends a failed command with process.exit() unless serve() is given an exit option. The dispatcher now passes one that sets process.exitCode, as its other exits already do, so the event loop ends the process and V8 disposes the isolate first. If an error ever leaves work scheduled, an unref'd timer still ends the process after 5 s, as process.exit did; it never delays a clean exit. A new CLI boundary test preloads a probe that records process.exit() calls and checks that a validation error ends without one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DffptKBR2VS1iaZ72o8Hga
) The file parsed all 67 catalogs (about 80 MB of JSON) in the test process, so --test-force-exit called process.exit() with the heap at its limit and Sparkplug/Maglev jobs still running. Those jobs park until the main thread collects garbage while process.exit() joins them, which hung whichever unit shard held this file until the job timed out (nodejs/node#64274, fix proposed in nodejs/node#66171). A short-lived child now parses the catalogs and ends on its own, and the test process receives only the keys it asserts on. All assertions stay. Locally, with the CI flags on Node 24.22.2, about 600 runs hung 4 times before this change and 1260 runs hung 0 times after it. Ref #19203
|
Given the number of projects affected, which are linked to this PR, can we please have another look at this? |
…otnet#136897) ## Problem `JIT/jit64/regress/vsw/539509/test1_539509` intermittently times out on the `R2R-CG2 browser wasm Checked` leg. It prints `Passed.` and then `node` never exits (dotnet#135042, about 25 hits a month). Any runtime test run under `corerun.js` on Node can hit this; this test hits it most often. ## Root cause This is a Node/V8 shutdown deadlock, nodejs/node#64274. The upstream fix, nodejs/node#66171, is not merged yet, so the Helix image's node v24.22.2 is affected. corerun's `_proc_exit` override (dotnet#132078) calls `process.exit()`. That joins V8's platform worker threads without disposing the isolate. If a background compile job is parked in `CollectionBarrier::AwaitCollectionBackground`, waiting for a GC that only the main thread can run, the join never returns. Evidence, using the exact payloads from the failing Helix job (crossgen2 + `corerun.js` + node v24.22.2) on Linux x64: - **Repro:** the test hangs after `Passed.` in about 1 of every 1,000–2,000 runs. - **Hung process:** it has only `MainThread`, one `V8Worker` and `SignalInspector` left, all in `futex_wait_queue` with no CPU time. It doesn't answer `--report-on-signal`, which fits a main thread blocked in a join. - **Stress:** with `--stress-concurrent-allocation` the hang rate rises to 18 of 170 runs. `--single-threaded` (0 of 60) and natural exit both remove it. ## Fix In `libCorerun.js`, on Node: - Keep the hard `process.exit()` only when the exit is requested from inside wasm, i.e. a wasm frame is on the JS stack (for example `Environment.Exit`). That case must not unwind `ExitStatus` through live CLR frames (dotnet#131937). - Otherwise, when main has returned or a timer tick has finished, fall through to Emscripten's own `_proc_exit`. It sets `process.exitCode` and lets Node exit when the event loop drains, which disposes the isolate and releases the parked worker. - Ignore the late `_exit` that `SystemJS_ResolveMainPromise` queues for corerun once the runtime has already exited. Previously it never ran because `process.exit()` ended the process first. Without the guard it would trip Emscripten's `exitRuntime` assertion for exit code 0. ## Validation - **Stress:** test1_539509 with the CI payloads under `--stress-concurrent-allocation` had 0 hangs in 180 runs with the fix vs 3 in 60 with the current host. Without stress, the average run time is unchanged (656 vs 652 ms). - **Exit-mode matrix:** I compared the current and patched hosts with a test app covering return 0, return 100, `Environment.Exit(42)`, `Environment.Exit(0)`, async Main returning 3 and 0, an unhandled exception, and GC with finalizers. Exit codes and output were identical. - **Real Emscripten link:** I linked the changed node block with Emscripten 6.0.2 at `-O0` and `-O2`. Main returning exits naturally (`beforeExit` fires), `exit()` from wasm still hard-exits, and a late `_exit` after runtime exit is silent. - **Not done:** I didn't rebuild the full browser corerun locally. The runs above used the same code patched into the CI-generated `corerun.js`, so this PR's browser legs provide the full build validation. The Windows x86 gcstress timeouts added to dotnet#135042 are a different mechanism (native corerun, no Node), and this PR does not address them. Fixes dotnet#135042 > [!NOTE] > This PR description was generated with GitHub Copilot. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
V8 background jobs, such as concurrent Sparkplug and Maglev compiles, can run into the heap limit. They then post a GC request to the main thread and park in CollectionBarrier::AwaitCollectionBackground() until the main thread collects garbage or the isolate is torn down. process.exit() does neither. DefaultProcessExitHandlerInternal() calls DisposePlatform() while the isolate is still alive and not running JS, and NodePlatform::Shutdown() joins the worker threads. A worker that is parked at that point never wakes up, and the join hangs forever. Pass the exiting isolate down to NodePlatform::Shutdown(). While the workers finish their current tasks, the main thread checks every 10ms whether a foreground task was posted for that isolate. If so, it drops the queued tasks, which would never run on this path, and performs a GC so that the parked workers can finish. Normal exits dispose the isolate before the platform and are not affected. Fixes: nodejs#64274 Signed-off-by: Manzoor Wani <manzoorwani.jk@gmail.com>
The test only hung on an unfixed build about a third of the time, because the window in which the main thread is out of JS depended on how fast the machine runs pbkdf2, and the stress allocation task had to run into the heap limit inside that window. Calibrate the iteration count from a short probe so that the window is about a second everywhere, and use up the old generation budget just before exiting so that the next background allocation fails and parks the task. Run in the test process instead of spawning a child, which also drops the child process overhead. On an unfixed build the test now hangs in 20 of 20 runs, and it passes 30 of 30 with the fix. Signed-off-by: Manzoor Wani <manzoorwani.jk@gmail.com>
da20842 to
d4492d9
Compare
marcastben-hash
left a comment
There was a problem hiding this comment.
I read this at d4492d9 together with the V8 code it interacts with (CollectionBarrier, Heap::CollectAllAvailableGarbage, Heap::PerformRequestedGC). I did not build or run the branch, so everything below comes from reading, and the points are questions for people who know this shutdown path better than I do.
The mechanism looks right to me: workers are stopped and joined by stop_sub_worker_contexts() before this point, so only the main isolate can have a parked background job, and LowMemoryNotification() ends in ResumeThreadsAwaitingCollection() for both kMajor (via each CollectGarbage) and kLastResort, which is what releases AwaitCollectionBackground().
-
The trigger is "any foreground task is queued", not "a GC was requested".
DiscardForegroundTasks()returns true for whatever V8 has posted to the isolate's foreground runner, and those are not onlyBackgroundCollectionInterruptTasks. So anyprocess.exit()where a platform worker needs more than 10 ms to finish its current task while some unrelated foreground task is pending will also take this path. That would be harmless if the reaction were cheap, but see the next point. -
LowMemoryNotification()is the heaviest GC entry point. It callsCollectAllAvailableGarbage(kLowMemoryNotification), which runs at least 2 and up to 7 forced mark-compacts withkReduceMemoryFootprint. For comparison, the task that V8 itself posted would have runPerformRequestedGC(), which is a singleCollectAllGarbage()for akMajorrequest. On a large heap this could makeprocess.exit()noticeably slower in cases that would not have hung. Is there a reason to prefer the fullLowMemoryNotification()over a single major GC here? Any completed major GC clears thekMajorrequest and wakes the waiters. -
This runs a GC after
uv_library_shutdown().BaseObjectweak callbacks callOnGCCollect()synchronously during the collection, so object destructors now run at a point where the libuv threadpool is already shut down andcan_call_into_jsis false. Before this change no GC could happen that late. I do not have a concrete failing case, but it seems worth a maintainer confirming that destructors which close handles or queue threadpool work are fine there. -
As far as I can tell,
CollectAllAvailableGarbage()finishes withCheckHeapLimitReached(), so if the heap really is at its limit,process.exit(0)could end in a fatal OOM instead of exiting with the requested code. That is still better than hanging, but it may deserve a line in the comment.
Disclosure: this review was prepared with AI assistance (Claude).
process.exit()can hang forever when a V8 background job (for example a concurrent Sparkplug or Maglev compile) is parked waiting for the main thread to collect garbage. The exit handler joins the platform workers without disposing the isolate, and only isolate teardown or a main-thread GC wakes such a worker. This makes the main thread perform that GC while it waits for the workers to exit.Fixes #64274
The test uses
--stress-concurrent-allocationplus a threadpool job thatuv_library_shutdown()waits on, which parks a worker inCollectionBarrier::AwaitCollectionBackground()before the join. Without the fix it hangs in 20 of 20 local runs; with it, it passes 30 of 30.This hang stalls our WordPress core builds that end with
process.exit()on Node.js 24 (WordPress/wordpress-develop#13608).