ci: repack eval matrix, cancel superseded runs, centralize nix setup - #49
Open
wyattgill9 wants to merge 1 commit into
Open
wyattgill9 wants to merge 1 commit into
wyattgill9 wants to merge 1 commit into
Conversation
The on-push pipeline took ~1.5h wall despite ~20min of real work. Job timings from run 33041628057 show all linux work finishing 20 minutes in; the remainder was 24 macOS nixpkgs-eval jobs (each ~3.5min of runner setup for ~6min of eval) queuing on the ~5-slot macOS pool, compounded by overlapping runs from rapid pushes starving each other. Repack the 72-job eval matrix into 12 group jobs: each walks 3 of the 12 chunks, both gc-budget lanes, cold+warm each. Coverage is byte-identical — the compiled-chunk cache (~/.cache/fix/chunks) is wiped before each (chunk, lane) pair so cold runs stay genuinely cold. Oracle caches move to per-group v2 keys. Add a concurrency group with cancel-in-progress so a new push releases the superseded run's runners immediately, and split build-darwin out of build so a macOS runner drought can no longer block the linux evals, gc detector, or daemon arms. Fold the nix preamble every job repeated (install, snapshot restore + self-heal, channel pin) into a setup-nix composite action replacing nix-store-cache; jobs are now checkout + one setup step. Snapshot saves stay explicit last steps so a broken job never publishes a poisoned store. Also fix nixpkgs-gc-detector restoring a store key nothing ever saved (it ran cold every run); it now shares the x86_64-linux eval key.
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The on-push pipeline took ~1.5h wall clock for ~20 minutes of real work. Job timings from recent runs show all linux jobs finishing 20 minutes in; the rest was the 24 macOS nixpkgs-eval jobs queuing on the ~5-slot, slow-provisioning macOS runner pool — each job spending ~3.5 minutes on runner setup for ~6 minutes of eval. Rapid pushes made it worse: each superseded run kept holding macOS runners, so overlapping runs starved each other (25–30 minute dead gaps mid-run were common).
Solution
Three structural changes, no coverage change:
Repack the 72-job eval matrix into 12 group jobs. Each group walks 3 of the 12 chunks, running both gc-budget lanes per chunk, cold+warm each. Coverage is byte-identical: the compiled-chunk cache (
~/.cache/fix/chunks) is wiped before each (chunk, lane) pair so cold runs stay genuinely cold. Setup cost amortizes 6x, and darwin evals fit the macOS pool in one wave instead of five. Oracle caches move to per-group v2 keys (one-time re-eval on first run).Cancel superseded runs. A concurrency group per ref with cancel-in-progress, so a new push releases the previous run's runners immediately.
Centralize the nix preamble. Every job repeated the same 5–6 steps (install nix, snapshot restore + self-heal, cold reinstall, channel pin). That moves into a
setup-nixcomposite action replacingnix-store-cache; jobs are now checkout + one setup step. Snapshot saves stay explicit last steps so a broken job never publishes a poisoned store snapshot. The nightly workflow gets the same treatment.Also splits
build-darwinout of the build matrix so a macOS runner drought can't block the linux evals, gc detector, or daemon arms — and fixesnixpkgs-gc-detectorrestoring a store snapshot key that nothing ever saved (it ran on a cold store every run; it now shares the x86_64-linux eval key).Results
Measured on this branch (run 33049343496):
Net −116 lines across the workflow files.