Repository navigation
main health 2026-09-08: what the local CI replay found (7 reds on main, 3 with fix PRs, the obsolete #7254 pin, a Linux hang, a load-sensitive SIGSEGV) #9974
Description
Activity
Item 4 (the obsolete #7254 pin): PR #9975. Verified: under the smoke's pairing the reproducer's stdout is byte-identical to node's and the run is live (12,528 objects copied under the verifier), so the pinned panic was the false positive #4644 fixed. The bisect that named 9f790a5 used the smoke script itself as the predicate (a hand-copied arm-5 recipe is silent on every tree because the script exports PERRY_GC_SCHEDULE_ALLOC_KB=0 and PERRY_NO_AUTO_OPTIMIZE=1 globally). https://claude.ai/code/session_011dhBmdn4vGgNibjo3oZqTo
Eighth known-red-on-main job (Linux, local ladder):
cargo-test-perrynamed jobasync_hooks_constructors_expose_real_prototype_methodsfails on main @ 8b7dc33 with the compiled fixture seeing everyAsyncLocalStorage.prototype.*/AsyncResource.prototype.*method as missing — filed as #9980 with the verbatim output and the CI command. Attributed by replaying the parent of #9958 (whose ladder first ran the job); macOS runners had shown main green on that shard before they stalled, so it may be Linux-only. Added to the LCI-main2 baseline set.Update from the completed #9998 CI replay against current
main(dc39693e6):- Correction to item 6:
test_gap_iterator_prototype_next_patchwas not oracle-only. Node's console formatter crashed while the built-in iterator prototypes were patched, which hid a real Perry product bug. After buffering output until the prototypes are restored, the oracle shows that Perry's eager Array/Map/Set/String materializers bypassed patchednextwhile manual iteration honored it. fix(release): clear publishing and runtime blockers #9998 fixes the eager spread,Array.from, collection-constructor, Promise-combinator, and call/push-spread paths. The 41-line fixture byte-matches Node normally and under forced moving GC; full gap shard 1 passes. - Two new product reds landed with the perf(json): streamline parse and stringify construction #9849 JSON train. Current-main push run 34252807132 and fix(release): clear publishing and runtime blockers #9998's independent run 34256962396 both show them.
test_gap_json_nested_records: exact difference is the inherited root-arrayObject.prototype.toJSONcase. Node returns"global:"; Perry serializes the array contents.array_get_to_jsonchecks only the array's own named-property side table and skips prototype-chainGet. PR perf(json): streamline parse and stringify construction #9849's checked-in benchmark notes acknowledged this mismatch, but the required gap gate correctly rejects it. No complete fix or tracker exists.test_gap_2899_2779_2777_static_helpers: Node and Perry exit 0 and the first line matches, but a later line differs. The current runner logs only the first line; an isolated local recovery attempt rebuilt the runtime and then stalled compiling the fixture, so the exact later divergence/root cause remains unmeasured.
- Infrastructure status: fix(release): clear publishing and runtime blockers #9998's
e2e-scopedfailure is solely the process-global typed-feedback race fixed by test(codegen): isolate the default feedback lane #9999; the stale public benchmark artifact remains the only lint failure. The full GC stress matrix and fan-in pass. [scheduled-gate:gc-native-roots.yml] gc-native-roots is failing on main #9915's native-roots workflow is green again and the bot has closed that tracker.
Release classification: the iterator and JSON findings are product behavior and affect what ships. The typed-feedback race, stale benchmark evidence, artifact-download failures, GC-ratchet receipts/retries, and obsolete pins are gate infrastructure.
- Correction to item 6:
Item 8 (the
lintpublic-baseline step) is still red on main295473cd5, 20 days on, and the cause is now pinned down — it is one benchmark file, not a broken gate.Full script-tier replay on main
295473cd5(perrymaster, Linux x86_64): 103 of 105 gates pass,cargo fmt --all -- --checkclean. The two failures are this one andcargo xwin check, the latter only becausecargo-xwinis not installed on the host.For the public baseline, the gate is right and the artifact is stale:
- The committed artifact is
benchmarks/results/public-node-bun-v1.json,generated_at 2026-09-23T01:34:16Z, recordedcommit b77aba63433b100b9482d47181b5ccc5e724ff81,perry 0.5.1639— 121 commits behind main. - Age is not the problem: 5 days against
MAX_AGE_DAYS = 45. - The
harness_fingerprintmatches exactly (513dba8f…), so nothing inpublic-baseline-config.jsonorhonest_bench/results/expected.jsonmoved. Onlysource_fingerprintdiffers (recorded9c87723d…, current6a0e4479…). - Diffing the 60 tracked input files between
b77aba63andHEADgives exactly two:
Cargo.toml | 137 +++++++++++++++++--------------- benchmarks/suite/13_factorial.ts | 7 +-So the trigger is a real benchmark edit —
benchmarks/suite/13_factorial.ts— and the artifact genuinely needs regenerating with./benchmarks/run_public_baseline.shagainst node and bun on a quiet host. That is a measurement, not a code change, so it wants doing deliberately rather than folded into an unrelated PR; I have not run it, since the only Linux host I have is shared and loaded.Worth noting for whoever picks it up:
Cargo.tomlbeing inSOURCE_PATHSmeans every release train's version bump alone re-reddens this gate even when no benchmark input changed. That is probably why it reads as "fails on every host" indefinitely — after each regeneration it goes red again at the next train. Narrowing the tracked part ofCargo.toml, or excluding it in favour of the workspace members that actually affect benchmark results, would stop the gate crying wolf between genuine input changes like this one.- The committed artifact is
Context: GitHub's hosted runners stalled on 2026-09-07, so the PR gates were replayed locally (perrymaster, Linux x86_64, Ubuntu-like; plus macOS arm64 for some) from
.github/workflowsverbatim, with main 8b7dc33 as the baseline. The replay found the following on main itself. Fix PRs exist for the first three.warningsjob red — 7 dead-code errors in the product build.RUSTFLAGS="-D warnings" cargo check -p perry --binscompilesperry-runtimewithoutregex-engine, sonewborn_parent_needs_barrierand the sixREGEXP_PROTOTYPE_*statics inobject/regex_proto_thunks.rsare dead there. Fix: fix(runtime): gate the seven regex-engine/test-only items so the product-scope -D warnings check compiles again #9970 (cfg-gatedany(test, feature = "regex-engine"); ladder-verified: warnings green, runtime suite 3247/0).cargo-testred — one perry unit test.commands::compile::build_cache::tests::codegen_env_vars_are_build_cache_inputs:PERRY_SEGVIEW/PERRY_SEGVIEW_DIAG(segment-view lowering, perf(intl): take the view mode's canonicality proof and UTF-8 re-validation off the per-grapheme path #9893) are read by codegen but registered neither as build-cache inputs nor as exclusions — a stale cached object can be served across a lowering-knob change. Fix: fix(cache): PERRY_SEGVIEW keys the build cache, PERRY_SEGVIEW_DIAG is a recorded diagnostic exclusion — fixes codegen_env_vars_are_build_cache_inputs on main #9971 (input + reasoned exclusion + a real cache test).scripts/run_lint_gates.shwas green by omission. Its extractor accepted onlypython3 scripts/,./scripts/andcargo fmtcommands, silently dropping the "Public benchmark evidence freshness" step; its compile tier never ran the product-scope-D warningscheck. Fix: tooling(lint): run_lint_gates.sh fails loudly when a lint step yields no command — it silently dropped the public-baseline step #9969 (derives everyrun:step, fails loudly on an empty derivation, compile tier derived from thewarnings/checkjobs).scripts/gc_instrument_smoke.sharm 5b is obsolete by design.git bisect runwith the script as predicate names 9f790a5 (Add a bridge for SolidJS as on-ramp which maps to perry/ui underneath #4644, "allow retained array-growth aliases in copying verification") as the first commit where the reproducer stops panicking, on Linux and macOS alike; arm 5a stays live (3,894 objects copied under the verifier). Add a bridge for SolidJS as on-ramp which maps to perry/ui underneath #4644 reclassifies gc: PERRY_GC_ZEAL + PERRY_GC_VERIFY_EVACUATION reports a stale forwarded pointer in shadow-stack roots (10/10, pre-existing, no CI arm sets the pair) #7254's Ptr growth alias from "stale forwarded pointer" to a retained alias (its changelog: a false abort). The script's own instruction applies: retire the pinned block and run the file under an ordinary arm — with the reproducer's output checked against node under the pairing. PR in preparation. Note for the smoke's readers: the script exportsPERRY_GC_SCHEDULE_ALLOC_KB=0andPERRY_NO_AUTO_OPTIMIZE=1globally; a hand-copied arm-5 recipe without them is silent everywhere.test_gap_webcrypto_async_threadpoolhangs on Linux main. Alone,PERRY_RUN_TIMEOUT=60, Node 26.5.1, fast mode: 3/3 "TIMEOUT (killed after 60s)" at 64.5–64.7 s wall on main's build (and identically on an unrelated PR's build). The gap snapshot records it as passing.test_gap_iterator_prototype_next_patch(node itself exits 1 insidenode:internal/per_context/primordials:449while perry prints the expected output — oracle-side),test_gap_9616_readable_to_web_fs,test_gap_crypto_scrypt_options,test_gap_disposablestack_2875,test_gap_promisify_pipeline_custom_6692— all "pass → parity_fail" against the snapshot; identical sets on every PR head tested, so they are main's.RUST_TEST_THREADS=1 cargo test --lib -p perry-runtimethe binary died 3/3 atasync_hooks::test_support::tests::native_async_resource_accepts_string_and_symbol_expandoswhile a release build ran alongside, 0/18 without that load, never under gdb. Load-sensitive race; a native AsyncResource with string/symbol expandos is a headerless pointer-tagged receiver with side-table expandos.lint's public-baseline step fails on every host: "public artifact benchmark inputs changed; regenerate it with ./benchmarks/run_public_baseline.sh" (the workflow's own comment dates it to 2026-07-29).Local-replay notes:
gc_ratchet.py checkis hard-locked to darwin-arm64 (runmeasureonly on Linux); the gap harnesses refuse an unpinned Node (26.5.1 pinned);cargo test --workspace --libis not CI's shape (it links ext-crate lib tests CI never builds).https://claude.ai/code/session_011dhBmdn4vGgNibjo3oZqTo