ci: stop the build cache from crossing macOS images - #22
Open
drakulavich wants to merge 2 commits into
Open
Conversation
`target/` was cached under `${{ runner.os }}-cargo-…` with a
`${{ runner.os }}-cargo-` prefix fallback. `runner.os` is "macOS" for every
macOS image, so that key is shared by macos-14, macos-15 and whatever comes
next: a job on one image can restore a build tree produced on another.
`.build/` was not cached at all. That is where build.rs puts
libFluidAudioBridge.a, and `target/` holds binaries already linked against
it — so the two could also drift apart on their own, with a linked binary
surviving a library that did not.
Split accordingly:
* registry and git checkouts are downloaded sources, portable, prefix
fallback kept;
* `target/` plus `.build/` are build output, cached together, keyed on the
image and its Xcode (via ImageVersion), and with **no** restore-keys —
a prefix fallback is precisely what makes this cache unsound, since it
hands over the nearest tree rather than a matching one.
Cost of the stricter key is a full rebuild whenever the image updates. That
is under a minute here.
This is a latent fault today — only one job exists and it is macos-15, so
nothing collides yet. It becomes live the moment a second image is added.
I hit it while probing a macos-14 job in a fork: after that job populated
the cache, a macos-15 job restored its tree and the example died with
`Bus error: 10` before printing anything. Clean builds of the same commit
on the same image synthesized fine, and the crash never reproduced without
a restored cache. I could not close the loop on the exact mechanism — by
the time I tested the cache directly, a later run had already refreshed the
key — so treat the crash as the symptom that pointed here rather than as
proof. The key sharing and the missing `.build/` are plainly wrong on
inspection either way.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MKfMMVQbSEgEca4nMYey4D
…ty string
The first version keyed on `${{ env.ImageVersion }}`. That renders empty:
the `env` context sees workflow/job/step `env:` blocks, not the runner's
ambient environment, so the key came out as
build-macos-15-ARM64--9697397a3ff6…
^^ nothing here
Image separation still worked, since the label is a literal, so the fault
this PR is about was fixed — but the "an Xcode update inside the same image
also invalidates" part was doing nothing.
Take the digest from `swift --version` instead, emitted by the step that
already prints it. That is also the more direct signal: what decides
whether a cached `target/` + `.build/` pair is still valid is the toolchain
that produced them, not the image serial that happens to carry it.
Caught by reading the rendered key in the CI log rather than assuming the
expression worked.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MKfMMVQbSEgEca4nMYey4D
Member
|
from ai |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Two things about the cache step are wrong independently of each other.
The key is shared across macOS images.
${{ runner.os }}ismacOSon every macOS runner — 13, 14, 15 alike. Witha job on one image can restore a
target/produced on another. The prefix fallback is what makes this reachable: even when the hash differs,macOS-cargo-matches whatever tree was saved last, from any image..build/was not cached, buttarget/was.build.rsbuilds the Swift package into.build/and links againstlibFluidAudioBridge.athere;target/then holds binaries already linked against it. Caching one without the other lets a linked binary be restored while the library it points at is not — and if cargo considers the build script fresh, it will not regenerate.build/either.Change
Split one cache into two, by what the contents actually are.
Downloaded sources —
~/.cargo/registry,~/.cargo/git. Portable across images and toolchains, a stale hit is harmless, prefix fallback kept.Build output —
target/and.build/, together, since they are only valid as a pair. Keyed on the image label,runner.arch, and a digest ofswift --version, so a toolchain change invalidates even when the image label does not move. Norestore-keys.The toolchain digest is emitted by the step that already prints the Swift version, which is why that step now has an
idand sits ahead of the cache. (First push keyed on${{ env.ImageVersion }}instead; that renders empty — theenvcontext reads workflow/job/stepenv:blocks, not the runner's ambient environment — so it contributed nothing. Visible in that run's log asbuild-macos-15-ARM64--9697…. The toolchain digest is the more direct signal anyway.)That last part is the point rather than an oversight: a prefix fallback is exactly what makes an artifact cache unsound. It hands the job the nearest build tree instead of a matching one, which is the opposite of what you want from build output. Exact key or rebuild.
The cost is a full rebuild whenever the image updates. CI here is well under a minute, so that is cheap next to the alternative.
How I ran into it
This is latent today — there is one job, it is
macos-15, so nothing collides yet. It goes live the moment a second image is added.I was probing a
macos-14job in a fork (context: #21). Once that job had populated the cache, amacos-15job restored its tree and the example died withBus error: 10, printing nothing. Clean builds of the same commit on the same image synthesized fine — nine of them, across two toolchains and both with and without the compute-units branch — and the crash never appeared without a restored cache.I could not close the loop on the exact mechanism: by the time I tested the cache in isolation, a later successful run had already refreshed that key, so the restore no longer reproduced the original condition. So the crash is what pointed me here, not proof of causation, and I would rather say that plainly than overclaim it. The shared key and the missing
.build/are wrong on inspection regardless of what that particular crash turns out to have been.Verification
The check on this PR builds green from a cold cache — both keys miss on the first run, as intended, and the build takes ~2m50s from scratch. The rendered keys are in that run's log, which is how the empty
ImageVersionabove was caught; worth a glance at the current run to confirm the toolchain digest is non-empty.Nothing outside
.github/workflows/ci.ymlis touched, and the job'sname:is unchanged, so any required-check configuration keeps matching.