Skip to content

ci: repack eval matrix, cancel superseded runs, centralize nix setup - #49

Open
wyattgill9 wants to merge 1 commit into
psyclyx:mainfrom
wyattgill9:main
Open

wyattgill9 wants to merge 1 commit into
psyclyx:mainfrom
wyattgill9:main

Conversation

@wyattgill9

@wyattgill9 wyattgill9 commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

tl;dr: ci 2x faster, LLM assisted PR

Problem

The on-push pipeline took ~1.5h wall clock for ~20 minutes of real work. Job timings from recent runs show all linux jobs finishing 20 minutes in; the rest was the 24 macOS nixpkgs-eval jobs queuing on the ~5-slot, slow-provisioning macOS runner pool — each job spending ~3.5 minutes on runner setup for ~6 minutes of eval. Rapid pushes made it worse: each superseded run kept holding macOS runners, so overlapping runs starved each other (25–30 minute dead gaps mid-run were common).

Solution

Three structural changes, no coverage change:

Repack the 72-job eval matrix into 12 group jobs. Each group walks 3 of the 12 chunks, running both gc-budget lanes per chunk, cold+warm each. Coverage is byte-identical: the compiled-chunk cache (~/.cache/fix/chunks) is wiped before each (chunk, lane) pair so cold runs stay genuinely cold. Setup cost amortizes 6x, and darwin evals fit the macOS pool in one wave instead of five. Oracle caches move to per-group v2 keys (one-time re-eval on first run).

Cancel superseded runs. A concurrency group per ref with cancel-in-progress, so a new push releases the previous run's runners immediately.

Centralize the nix preamble. Every job repeated the same 5–6 steps (install nix, snapshot restore + self-heal, cold reinstall, channel pin). That moves into a setup-nix composite action replacing nix-store-cache; jobs are now checkout + one setup step. Snapshot saves stay explicit last steps so a broken job never publishes a poisoned store snapshot. The nightly workflow gets the same treatment.

Also splits build-darwin out of the build matrix so a macOS runner drought can't block the linux evals, gc detector, or daemon arms — and fixes nixpkgs-gc-detector restoring a store snapshot key that nothing ever saved (it ran on a cold store every run; it now shares the x86_64-linux eval key).

Results

Measured on this branch (run 33049343496):

before after
wall clock ~77–90 min ~23 min
jobs per push 82 22
macOS jobs per push 26 6

Net −116 lines across the workflow files.

The on-push pipeline took ~1.5h wall despite ~20min of real work. Job
timings from run 33041628057 show all linux work finishing 20 minutes
in; the remainder was 24 macOS nixpkgs-eval jobs (each ~3.5min of
runner setup for ~6min of eval) queuing on the ~5-slot macOS pool,
compounded by overlapping runs from rapid pushes starving each other.

Repack the 72-job eval matrix into 12 group jobs: each walks 3 of the
12 chunks, both gc-budget lanes, cold+warm each. Coverage is
byte-identical — the compiled-chunk cache (~/.cache/fix/chunks) is
wiped before each (chunk, lane) pair so cold runs stay genuinely cold.
Oracle caches move to per-group v2 keys.

Add a concurrency group with cancel-in-progress so a new push releases
the superseded run's runners immediately, and split build-darwin out of
build so a macOS runner drought can no longer block the linux evals,
gc detector, or daemon arms.

Fold the nix preamble every job repeated (install, snapshot restore +
self-heal, channel pin) into a setup-nix composite action replacing
nix-store-cache; jobs are now checkout + one setup step. Snapshot saves
stay explicit last steps so a broken job never publishes a poisoned
store. Also fix nixpkgs-gc-detector restoring a store key nothing ever
saved (it ran cold every run); it now shares the x86_64-linux eval key.
@wyattgill9

Copy link
Copy Markdown
Contributor Author

@psyclyx

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant