Skip to content

ci: stop the build cache from crossing macOS images - #22

Open
drakulavich wants to merge 2 commits into
FluidInference:mainfrom
drakulavich:fix/ci-cache-scoping
Open

ci: stop the build cache from crossing macOS images#22
drakulavich wants to merge 2 commits into
FluidInference:mainfrom
drakulavich:fix/ci-cache-scoping

Conversation

@drakulavich

@drakulavich drakulavich commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Problem

Two things about the cache step are wrong independently of each other.

The key is shared across macOS images. ${{ runner.os }} is macOS on every macOS runner — 13, 14, 15 alike. With

key: ${{ runner.os }}-cargo-${{ hashFiles('**/Cargo.lock', 'Package.resolved') }}
restore-keys: |
  ${{ runner.os }}-cargo-

a job on one image can restore a target/ produced on another. The prefix fallback is what makes this reachable: even when the hash differs, macOS-cargo- matches whatever tree was saved last, from any image.

.build/ was not cached, but target/ was. build.rs builds the Swift package into .build/ and links against libFluidAudioBridge.a there; target/ then holds binaries already linked against it. Caching one without the other lets a linked binary be restored while the library it points at is not — and if cargo considers the build script fresh, it will not regenerate .build/ either.

Change

Split one cache into two, by what the contents actually are.

Downloaded sources~/.cargo/registry, ~/.cargo/git. Portable across images and toolchains, a stale hit is harmless, prefix fallback kept.

Build outputtarget/ and .build/, together, since they are only valid as a pair. Keyed on the image label, runner.arch, and a digest of swift --version, so a toolchain change invalidates even when the image label does not move. No restore-keys.

The toolchain digest is emitted by the step that already prints the Swift version, which is why that step now has an id and sits ahead of the cache. (First push keyed on ${{ env.ImageVersion }} instead; that renders empty — the env context reads workflow/job/step env: blocks, not the runner's ambient environment — so it contributed nothing. Visible in that run's log as build-macos-15-ARM64--9697…. The toolchain digest is the more direct signal anyway.)

That last part is the point rather than an oversight: a prefix fallback is exactly what makes an artifact cache unsound. It hands the job the nearest build tree instead of a matching one, which is the opposite of what you want from build output. Exact key or rebuild.

The cost is a full rebuild whenever the image updates. CI here is well under a minute, so that is cheap next to the alternative.

How I ran into it

This is latent today — there is one job, it is macos-15, so nothing collides yet. It goes live the moment a second image is added.

I was probing a macos-14 job in a fork (context: #21). Once that job had populated the cache, a macos-15 job restored its tree and the example died with Bus error: 10, printing nothing. Clean builds of the same commit on the same image synthesized fine — nine of them, across two toolchains and both with and without the compute-units branch — and the crash never appeared without a restored cache.

I could not close the loop on the exact mechanism: by the time I tested the cache in isolation, a later successful run had already refreshed that key, so the restore no longer reproduced the original condition. So the crash is what pointed me here, not proof of causation, and I would rather say that plainly than overclaim it. The shared key and the missing .build/ are wrong on inspection regardless of what that particular crash turns out to have been.

Verification

The check on this PR builds green from a cold cache — both keys miss on the first run, as intended, and the build takes ~2m50s from scratch. The rendered keys are in that run's log, which is how the empty ImageVersion above was caught; worth a glance at the current run to confirm the toolchain digest is non-empty.

Nothing outside .github/workflows/ci.yml is touched, and the job's name: is unchanged, so any required-check configuration keeps matching.

drakulavich and others added 2 commits August 5, 2026 20:37
`target/` was cached under `${{ runner.os }}-cargo-…` with a
`${{ runner.os }}-cargo-` prefix fallback. `runner.os` is "macOS" for every
macOS image, so that key is shared by macos-14, macos-15 and whatever comes
next: a job on one image can restore a build tree produced on another.

`.build/` was not cached at all. That is where build.rs puts
libFluidAudioBridge.a, and `target/` holds binaries already linked against
it — so the two could also drift apart on their own, with a linked binary
surviving a library that did not.

Split accordingly:

  * registry and git checkouts are downloaded sources, portable, prefix
    fallback kept;
  * `target/` plus `.build/` are build output, cached together, keyed on the
    image and its Xcode (via ImageVersion), and with **no** restore-keys —
    a prefix fallback is precisely what makes this cache unsound, since it
    hands over the nearest tree rather than a matching one.

Cost of the stricter key is a full rebuild whenever the image updates. That
is under a minute here.

This is a latent fault today — only one job exists and it is macos-15, so
nothing collides yet. It becomes live the moment a second image is added.
I hit it while probing a macos-14 job in a fork: after that job populated
the cache, a macos-15 job restored its tree and the example died with
`Bus error: 10` before printing anything. Clean builds of the same commit
on the same image synthesized fine, and the crash never reproduced without
a restored cache. I could not close the loop on the exact mechanism — by
the time I tested the cache directly, a later run had already refreshed the
key — so treat the crash as the symptom that pointed here rather than as
proof. The key sharing and the missing `.build/` are plainly wrong on
inspection either way.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MKfMMVQbSEgEca4nMYey4D
…ty string

The first version keyed on `${{ env.ImageVersion }}`. That renders empty:
the `env` context sees workflow/job/step `env:` blocks, not the runner's
ambient environment, so the key came out as

    build-macos-15-ARM64--9697397a3ff6…
                          ^^ nothing here

Image separation still worked, since the label is a literal, so the fault
this PR is about was fixed — but the "an Xcode update inside the same image
also invalidates" part was doing nothing.

Take the digest from `swift --version` instead, emitted by the step that
already prints it. That is also the more direct signal: what decides
whether a cached `target/` + `.build/` pair is still valid is the toolchain
that produced them, not the image serial that happens to carry it.

Caught by reading the rendered key in the CI log rather than assuming the
expression worked.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MKfMMVQbSEgEca4nMYey4D
@Alex-Wengg

Copy link
Copy Markdown
Member

from ai


──────────────────────────────────────────────────────────────────────────────────

• I found two issues; neither invalidates the core macOS-image fix, but I’d address them before
  merging.

  - P2 – Include the Rust compiler in the build-cache key. The workflow installs the moving stable
    toolchain, but the key only covers macOS, architecture, Swift, and lockfiles (workflow L30–64

    (https://github.com/FluidInference/fluidaudio-rs/blob/b1c45cad5f5f2c5879f9a6c53c08b47e0097058e/.github/workflows/ci.yml#L30-L64)).
    When stable advances, the old exact cache remains immutable, so Cargo must repeatedly rebuild
    it. Give the Rust setup step an id and add its existing cachekey output:

    - name: Setup Rust
      id: rust
      uses: dtolnay/rust-toolchain@stable

    key: build-macos-
    15-${{ runner.arch }}-rust${{ steps.rust.outputs.cachekey }}-swift${{ steps.swift.outputs.id }
    }-${{ hashFiles('**/Cargo.lock', 'Package.resolved') }}
    The action documents cachekey specifically for this purpose: dtolnay/rust-toolchain outputs
    (https://github.com/dtolnay/rust-toolchain#outputs).
    (https://github.com/dtolnay/rust-toolchain#outputs).

  - P3 – .build is not where this project’s Swift artifacts are produced. The workflow says
    build.rs writes the static library into .build and that it must be cached alongside target
    (workflow L45–62

    (https://github.com/FluidInference/fluidaudio-rs/blob/b1c45cad5f5f2c5879f9a6c53c08b47e0097058e/.github/workflows/ci.yml#L45-L62)).
    In reality, build.rs passes --build-path "$OUT_DIR/swift-build" (build.rs L9–26

    (https://github.com/FluidInference/fluidaudio-rs/blob/b1c45cad5f5f2c5879f9a6c53c08b47e0097058e/build.rs#L9-L26));
    Cargo’s OUT_DIR is already under target. Thus target contains both the Rust and Swift outputs,
    while root .build is unused. Remove .build and correct the comment/PR body.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants