ci: perry compile --no-link pays for a full runtime build it discards — 343s nested cargo inside a 300s budget (intermittent compiler-output-regression timeout) #10782

Activity

1. proggeramlug

   changed the title [-]ci: compiler-output-regression has been red for 34h — perry compile stalls on a nested cargo build because the harness never sets PERRY_NO_AUTO_OPTIMIZE[/-] [+]ci: perry compile --no-link pays for a full runtime build it discards — 343s nested cargo inside a 300s budget (intermittent compiler-output-regression timeout)[/+] on Sep 20, 2026
2. proggeramlug commented on Sep 20, 2026

   

   proggeramlug

   on Sep 20, 2026

   ContributorAuthor

   More actions

   Correction to this issue's framing — the title and the "34 hours" claim were wrong, and I wrote them.

   I sampled one failing run (bd24fb32a2), found TimeoutExpired in it, confirmed the job's conclusion was failure across 26 consecutive sweeps, and wrote that up as one cause. That was checking whether the failing set was non-empty instead of whether it was the same set.

   Measured across six recent main sweeps, counting occurrences in each compiler-output-regression job log:

   SHA	TimeoutExpired	unexpected_missed_reason_kinds	raw-field-get	no_dynamic_property_runtime
   dd00a00305	0	7	8	3
   3f5b5b1424	0	7	8	3
   bd24fb32a2	2	6	4	3
   14ca314481	0	7	8	3
   14ca314481	0	7	8	3
   2f9dc8e692	0	7	8	3

   The timeout is intermittent — one run in six, and the run I happened to open. The persistent red is a set of structural verdicts that has nothing to do with auto-optimize, tracked separately as #10784.

   This issue stays open on its own merits, rescoped. The mechanism is real and independently confirmed: run_pipeline.rs:6169 resolves optimized_libs unconditionally, while --no-link's early return is at :6730 — so a --no-link compile performs the entire runtime build and then discards it 560 lines later. PERRY_RUNTIME_DIR does not suppress it. Measured on an M-series mini, the nested build alone takes 343 s, against a 300 s --compile-timeout: it cannot fit the budget it is charged to, on hardware faster than a GitHub runner. That is why it fires sometimes rather than always — it is a cache-state race, not a constant.

   Two further corrections to what I originally wrote here:

   • I said the four failing steps "pass --no-link --print-hir" and that suppressing auto-optimize was therefore unambiguously safe for them. Each of those steps runs both a --no-link probe and a linking compile whose binary is then executed — no step passes --skip-run, and --perf-counters off disables only run_perf_stat, not run_benchmark. The suppression is safe for the probe, not for the step.
   • Suppressing it on the linking compile would be actively wrong. runtime_budgets' allocations_traced / gc_collections_traced / write_barriers_traced are derived (analyzers.py:434-481) solely from the executed binary's PERRY_GC_TRACE stderr, and optimized_libs/freshness.rs:288-293 adds perry-runtime/diagnostics to the rebuild because the compiler sees PERRY_GC_TRACE=1 — a derivation that runs only on the auto-optimize branch. A runtime emitting no trace scores 0 on all three and passes every budget vacuously, since they are maxima. That is CLAUDE.md failure mode useEffect + setState panics with RefCell already borrowed on macOS ARM64 #4.

   So the fix sets PERRY_NO_AUTO_OPTIMIZE=1 at exactly one call site — capture.py:242, the HIR probe — and deliberately not at capture.py:261. repsel_census.py:681 already sets it and has done for a while, with a docstring making the same argument; this was the one site missing from an existing convention. A test asserts the classification per call site and was sabotage-checked: moving the suppression to the linking compile turns it red.

   Known remainder, not closed by the fix: the cold build still happens, just in the linking compile's budget instead of the probe's. Only one perry-auto hash is ever created across all 23 workloads, so CI pays exactly one cold build; the fix changes which step absorbs it, not whether 300 s is enough. The durable fix is a pre-warm step in test.yml with its own explicit budget — asserting target/perry-auto-* exists afterwards, or it is a no-op nobody notices.

   Worth its own issue: --no-link could decline to auto-optimize outright — a two-line guard at run_pipeline.rs:6169 — which fixes the class rather than this instance. The risk is the undefined-symbol stub scan at :6179, which consumes optimized_libs.runtime to seed its defined-symbol set; skipping the rebuild falls back to find_runtime_library and could change the synthesized stubs in an emitted object. repsel_determinism and repsel_knob_isolation hash --no-link objects, so their baselines would move once.

   One incidental worth recording: when the harness times out, Python kills perry but the nested cargo survives as an orphan and keeps building.
3. proggeramlug

   added a commit that references this issue on Sep 20, 2026

   fix(ci): stop the compiler-output HIR probe triggering a nested runti…

   ...

   5548fd8
4. proggeramlug commented on Sep 20, 2026

   

   proggeramlug

   on Sep 20, 2026

   ContributorAuthor

   More actions

   Fixed on main as of v0.5.1616 (8e9f6f09e8), via merge train 237 (#10792).

   compiler-output-regression now passes both suites — the first time since 2026-09-02, eighteen days. It is a main-gate input, so every merge in that window was bypassing it.

   The refreshed gate was proven able to fail, on the assembled tree rather than only per-branch:

   clean      → rc=0, "failed_workloads": []
   sabotaged  → rc=1   (one require_records consumer reverted to js_object_get_field_by_name_f64)
   

   That check took four attempts, and the three that failed are worth recording because each one looked like a pass: BSD sed has no 0,/re/ address so the file was never modified; the next edit landed on a comment rather than the consumer = line; and the third used the wrong entry point, which produced zero bytes of output and exited 0. Asserting that an edit landed is not sufficient — it has to land on the load-bearing line, and the harness has to actually be running.

   Both fixes are in the train body in full, including the reasoning that PERRY_NO_AUTO_OPTIMIZE=1 is set at the HIR probe and deliberately not at the linking compile, since runtime_budgets are maxima derived from the executed binary's PERRY_GC_TRACE output and would pass vacuously against a runtime that emits no trace.

   Two things left open rather than quietly closed:

   The cold build still happens, just in the linking compile's 300 s budget instead of the probe's. Only one perry-auto hash is created across all 23 workloads, so CI pays exactly one cold build — the fix changes which step absorbs it, not whether 300 s is enough. The durable answer is a pre-warm step in test.yml with its own explicit budget, asserting target/perry-auto-* exists afterwards or it is a no-op nobody notices.

   --no-link could decline to auto-optimize outright — a two-line guard at run_pipeline.rs:6169 — which fixes the class rather than this instance. The risk is the undefined-symbol stub scan at :6179, which consumes optimized_libs.runtime to seed its defined-symbol set; skipping the rebuild falls back to find_runtime_library and could change synthesized stubs in an emitted object. repsel_determinism and repsel_knob_isolation hash --no-link objects, so their baselines would move once. Worth its own issue.

   And the structural finding, which this PR deliberately did not attempt: allowed_missed_reason_kinds is a module-wide sweep and main's event loop guarantees a five-remark floor in every program, so for any fixture without un-vectorizable loops of its own the list detects compiler boilerplate rather than the fixture. Excluding main is necessary but not sufficient — the GC loop back-edge poll injects control flow into the fixture's own functions, accounting for 2 of native_owned_typed_views' 3 control_flow remarks. -fsave-optimization-record=yaml gives exact per-function attribution with stable keys and is the right mechanism, ~150–250 lines, but the real cost is re-deriving all 25 workloads' lists. Its own issue, so that diff is reviewable on its own.
5. proggeramlug

   closed this as completedon Sep 20, 2026
6. proggeramlug

   added 2 commits that reference this issue on Sep 20, 2026

   ci: give compiler-output-regression's cold runtime builds their own b…

   ...

   27c373f

   ci: give compiler-output-regression's cold runtime builds their own b…

   ...

   f7afd3b
7. proggeramlug commented on Sep 20, 2026

   

   proggeramlug

   on Sep 20, 2026

   ContributorAuthor

   More actions

   Landed via merge train 243 (#10838) as v0.5.1622 — ba303c1f5f.

   This should make compiler-output-regression green for the first time in weeks. Its structural verdicts were already fixed as of v0.5.1618 (failed_workloads: []); the only thing left was the 2 × TimeoutExpired residual recorded when #10782 was first closed — the cold build moving from the probe's budget into the linking compile's.

   The pre-warm found something my design missed: there are two cold builds, not one. The perry-auto-<hash> directory name excludes the cross-feature set while the build stamp inside it includes it, and loop_bound_semantics is the one gated workload with no *_traced budget — so it wants the other feature set, hits a stamp mismatch, and re-runs cargo. A single-warm step would have left exactly one gate to absorb a cold build, and the red would have looked unchanged.

   Both liveness assertions are in and both are load-bearing: the presence check (a perry-auto-*/ with a build stamp and a non-empty archive) and the verification compile that must land under half the 300 s budget. The second is the one that actually discriminates — presence can pass on a stale directory.

   A gate caught the new step, and that is the gate working. compiler_output_step_liveness.py pins the job's post-build step inventory and refused the addition. Registered in COMPILER_SUBJECTS rather than exempted, since the pre-warm depends on the build and already carries the matching guard; the gate now passes at 10 subjects and its own self-test still passes at 8 cases, which I checked separately because a registry edit that broke the checker would be worse than the drift it hides.

   Worth stating why it has to be listed rather than exempted: the pre-warm carries its own assertions and can fail, so it is a subject. If it were ever silently removed, every gate below would quietly inherit the timeout it exists to prevent — and the failure would look exactly like the red it just fixed.

   On provenance: this work came from an agent that died on an authentication error after committing but before pushing. I recovered the commit, rebased it twice as main moved beneath it, and pushed. Since it never reached lint, I verified the workflow change here instead — test.yml parses at 28 jobs, the pre-warm sits at step 10 with all seven gates at 11–17, and ci_plan.py --self-test passes.

   #10804 rode along: six areas, 197 fixtures weighted to its Ptr<Shape> surface, zero unexplained regressions.