Repository navigation
release blocker: cc --help segfaults on main (NULL deref in run_microtasks) — clean at 6c880be77b, broken at 42d0f45685 and still broken at tip #9305
Description
Activity
Resolved: not a clobber — a
setjmpstack-slot-coloring hazard on the exception-return pathA hardware watchpoint on the slot settles the "callee scribbled" vs "a path stored 0" question from the caveat above. It is the second one, and the two writes are both
run_microtasks's own:Hardware watchpoint -location *(unsigned long*)($rsp+0x38) Old = 5609279237992 New = 140737352661952 at run_microtasks+2618 (0x7ffff7e97bc0, valid) Old = 140737352661952 New = 0 at run_microtasks+... (0xc6fd5d0) then: RELOAD slot=(nil) -> SIGSEGVThe two writers share one stack slot, and one of them is on the longjmp path
The pointer is stored before perry's exception transport arms
setjmp:c6fc5e5: mov %rax,0x38(%rsp) ; <-- the pointer, stored PRE-setjmp ... c6fc72d: call perry_runtime::exception::try_push_with_kind c6fc732: mov %rax,%rdi c6fc735: call setjmp ; <-- SETJMP c6fc73b: test %eax,%eax c6fc742: je c6fc7f1 ; ==0 -> first pass ; fallthrough = NON-ZERO return = arrived here via LONGJMP c6fc748: mov %r14,0x30(%rsp) c6fc74d: mov MICROTASK_PREV_CONTEXTS::SLOT,%r14d ... c6fc808: mov 0x38(%rsp),%rax ; <-- reload, on the longjmp path c6fc814: movzbl -0x37db0(%rax),%eax ; <-- FAULT, %rax == 0The zero comes from a completely unrelated loop that LLVM gave the same slot, on the normal (post-
setjmp) path — it is copying a 0x30-byte record out of an array:c6fd5c6: mov 0x18(%r8,%rcx,1),%rdx c6fd5cb: mov %rdx,0x38(%rsp) ; <-- unrelated temporary, SAME SLOT c6fd5d0: mov 0x20(%r8,%rcx,1),%rdx c6fd5d5: mov %rdx,0x90(%rsp)Traced with breakpoints, the order in a failing run is exactly:
SETJMP call STORE_ZERO rdx=(nil) x5 <- the loop, after setjmp RELOAD slot=(nil) <- the longjmp-return path SIGSEGVWhat this means
This is the textbook
setjmprule: a non-volatilelocal that is written beforesetjmpand read after alongjmpreturn has an indeterminate value. The compiler is entitled to color that stack slot for something else on the path between, and here it did. The crash therefore needs a JSthrowinside a microtask to fire at all, which is why it is invisible to smaller corpora and why--versionis fine.Two consequences for the fix:
- It is not specific to the value that happens to be in
0x38(%rsp)— any live-across-setjmplocal in a function that uses thejs_eh_try_push/setjmptransport has the same exposure. The commits in the bisect window did not create the hazard; they changed inlining/layout enough to make LLVM take the coloring. That means a "fix" that only perturbs this one function will make the symptom move rather than go away. - The durable fix is on perry's side of the transport: values that must survive a
longjmpreturn have to bevolatile(or re-loaded from memory after thesetjmp, or held somewhere the transport itself restores).panic="abort"plus the-C force-unwind-tables=yespairing documented in[profile.release]already makes this transport special; this is the other half of that contract and it is currently unenforced.
Suggested instrument
A regression that throws from inside a microtask and reads a pre-
setjmplocal afterwards, compiled at-O3, would pin the contract. Reproducing it reliably will likely need the same inlining pressure as the bundle, so a plain unit test may not be enough — the bundle itself, gated on--helpbeing byte-identical to node, is the honest gate, and it is currently the one that is red.- It is not specific to the value that happens to be in
Independently confirmed, from a different session, using the preserved known-good/known-bad pair rather than a fresh build.
Same box, same
HOME=/tmp/cc_home, three runs each:binary source result app_A6c880be77b+ #9272's contentrc=0, 9,175 bytes, 3/3 — byte-identical to node cli_2.1.112.js --helpapp_Dpristine 42d0f45685SIGSEGV 3/3, rc=139, 0 bytes (core dumped) So it is deterministic, not a flake, and the good/bad boundary is where the report places it.
Why this is worse than one crashing binary
It takes the campaign's primary correctness gate offline. Every recent runtime performance PR — #9272, #9216, #9177, #9214, #9190 and the stack-map work in #9299/#9300 — gates its numbers on
cc --helpbeing byte-identical to node, on the grounds that a performance number from a binary producing wrong output is not a number. That gate cannot run onmaintoday.The second-order effect is the dangerous one: a segfaulting baseline is indistinguishable from "my change broke it" until someone builds a control pair. Anyone measuring a runtime change against
mainright now will see their own binary crash and reasonably conclude they caused it. That costs an afternoon per person, and it silently discourages exactly the parity check we want people running.The root-cause analysis is the actionable part
The report's diagnosis — a non-
volatilelocal written beforesetjmp(perry'stry_push_with_kindexception transport) and read on thelongjmpreturn path, with LLVM giving the same stack slot0x38(%rsp)to an unrelated array-copy loop on the normal path — is the textbooksetjmpindeterminacy rule, and it has a consequence worth stating loudly:The commits in the bisect window did not create this hazard. They changed inlining enough for LLVM to take the coloring. So a fix that only perturbs this one function will move the symptom rather than remove it, and it will look fixed. The durable fix is at the transport: any local live across
setjmpand read after alongjmpreturn must bevolatileor spilled somewhere the optimiser cannot color, and that property wants a test that fails when it is violated — not a rebuild that happens to reassign the slot.It also explains the shape of the report: it needs a JS
throwinside a microtask to fire, which is why smaller corpora miss it entirely and--versionis clean.The known-good/known-bad pair is preserved at
/root/claude-reg3/{app_A,app_D}on perrymaster with anOWNERfile asking they be kept for this issue.
claude --helpon the natively compiledcli_2.1.112.jsbundle segfaults deterministically on currentmain, with no GC knobs, no environment, and no flags.--versionworks;--helpand-pcrash.This is not #9261 (that needs
PERRY_GC_SCHEDULE_SEED/RATE;--helpwith no knobs is explicitly node-identical there). It is not #8770 (that was non-deterministic agent-loop init).Repro
PERRY_DEBUG_SYMBOLS=1, x86-64 Linux (perrymaster). The compile itself is clean —CC_RC=0, only the usual deferred-API warnings.Good/bad, six binaries, one box, one toolchain
Every arm below is the same host, same nightly, same
cli_2.1.112.js, same compile command. Three consecutive runs each.--help6c880be77b+ #9272's content42d0f45685d20fb4fd25(maintip)So the regression is in
6c880be77b..42d0f45685, excluding #9272 (which arm A already carries), and it is still present onmaintip. The five non-test, non-CI commits in that window are #9282, #9257, #9254-phase-1, #9263, #9274, #9279.Where it crashes
perry_runtime::promise::microtasks::run_microtasks, at the same file offset every run and across ASLR:Fault address
0xfffffffffffc8250is exactly0 - 0x37db0, i.e. a NULL base with a large negative displacement — the shape of a thread-local access whose base pointer is null.btis#0 run_microtasks #1 main.The slot holds a live pointer, then holds NULL — same frame
gdb (ASLR off), breakpoints on the store and the reload of that spill slot:
Same
%rsp, so the same frame and the same slot: it held a valid pointer and reads back as NULL. Between the two there is acall setjmpand an indirectcall *%r12into runtime/generated code.Caveat, stated rather than glossed:
objdumpfinds six writers of0x38(%rsp)in this (heavily LLVM-merged, ~30 KB) function, so "a callee scribbled on the caller's frame" and "a different control-flow path legitimately stored 0 and the reload does not check it" are both consistent with the evidence above. Distinguishing them is the next step, and a watchpoint on the slot should do it in one run.Why this matters beyond the crash
cc --helpis the corpus this repo's whole perf campaign measures against — #9272, #9216, #9177 and every A/B in that family gate on "--helpbyte-identical to node, 9,175 bytes". Onmaintoday that gate cannot run at all. Any perf work that rebases ontomainright now gets a baseline that segfaults, which is indistinguishable from "my change broke it" until you build a control. It also means the bundle is not exercising the microtask pump at all, so nothing downstream of it is covered.