ci: retry transient buildkit Canceled failures, add memory headroom and failure diagnostics

[gilescope]

Extracted from #442 (the buildkit upgrade) so the CI flake mitigations can land independently of the bump.

Retry wrappers

Non-deterministic earthly Canceled failures — different targets each run, no signal received — get a single retry across 10 workflows (16 loops). remote-cache gets two extra attempts, being the memory-heaviest: it spawns a registry container plus several earthly invocations over ~90s and flaked back-to-back even with the standard 2-attempt wrapper.

Between attempts the buildkitd container, the cache volumes and ~/.earthly/buildkit are all dropped. Dropping the container alone isn't enough — earthly's client-side state still references the dead session and the retry fails immediately with no active sessions.

Genuine failures still fail: attempts are capped at 2–3, and the final attempt's exit code propagates.

Memory headroom

• stage2-setup now adds the same 12G swap the build path already had. Tests on docker runners were failing with earthly Canceled mid-build plus runc file already closed, consistent with silently OOM-killed buildkit-runc children on the 16 GiB runner.
• The swapfile alone nearly fills the runner's ~14G free root disk, so ~25G of unused preinstalled toolchains (dotnet, android, ghc, CodeQL) is reclaimed first — otherwise allocation fails with No space left on device.
• Earthly bootstrap moved to after the GCR mirror config in build-earthly.yml, so the daemon starts with the mirror in place rather than being configured post-hoc. The source-built binary now gets the mirror too; previously it had none and pulled from docker.io directly, exposing it to rate limits.
• Docker Hub login is continue-on-error: it only raises pull rate limits (the GCR mirror serves most pulls), so a transient registry-1.docker.io timeout must not fail the whole job.

Diagnostics

• New failure-diagnostics composite action, wired into reusable-test and reusable-wait-block-target.
• The buildkitd log is dumped inside the retry loop, before the reset destroys it — attempt-1 session-loss failures are undiagnosable otherwise.
• Fixes build-earthly.yml's failure handler, which logged earth-buildkitd — a container name that has never existed. The released binary names its daemon earthly-buildkitd, the source-built one earthly-dev-buildkitd. That step has been silently logging nothing on every failed build; it now logs both.

Tick-Tock is opt-in

Running the earthly-next workflow on every push/PR roughly doubled CI load on the 16-GiB runners and amplified the Canceled/OOM flake rate. It now runs nightly (04:17 UTC) and on manual dispatch, with a concurrency group. The next: job is removed from ci.yml; no other job depended on it.

Deliberately excluded

Stays in #442, each for its own PR:

• the buildkit image bump (buildkitd-v0.8.17-fix.1 → fix.5) and the actions-setup version pin
• the test-group resplit (WAIT blocks, group13/group14)
• the buildkitd-artifact rework (always ship the image as a GHA artifact; point the outer buildkit at the PR's own staging image) — that changes what CI actually tests, so it deserves its own review
• buildkit_max_parallelism 1 — not needed on main

Verification

• yq parses all 34 workflows and both composite actions.
• actionlint clean on all 14 changed files.
• No dangling needs: after the ci.yml job removal; remaining jobs are fast-check, docker, podman.
• Confirmed free of bump content, group-resplit content, and the artifact rework by grep.

One pre-existing lint finding surfaced while widening the actionlint glob and is not addressed here: reusable-secrets-integrations.yml:59 passes EARTHLY_TOKEN to stage2-setup, which declares no such input (GHA silently drops it). Verified pre-existing on main at f1f2d6b. Filed separately — the obvious fix of adding the input would quiet the lint without establishing whether the token was ever needed.

🤖 Generated with Claude Code

https://claude.ai/code/session_01GQK1DssrULoDFoX5JU9njo

[github-actions]

⚠️ Are we earthbuild yet?

Warning: "earthly" occurrences have increased by 77 (1.62%)

📈 Overall Progress

Branch	Total Count
main	4748
This PR	4825
Difference	+77 (1.62%)

---

Keep up the great work migrating from Earthly to Earthbuild! 🚀

💡 Tips for finding more occurrences

Run locally to see detailed breakdown:

./.github/scripts/count-earthly.sh

Note that the goal is not to reach 0.
There is anticipated to be at least some occurences of earthly in the source code due to backwards compatibility with config files and language constructs.

[janishorsts]

I remember the GHA machine has an FS mounted with plenty of space.

Out of curiosity, I pointed Docker to use it #747. No need to deal with the symptom of running out of space on FS.

  === Available disk space before Docker setup ===
  Filesystem      Size  Used Avail Use% Mounted on
  /dev/root       145G   58G   88G  40% /

[janishorsts]

My PR is partially wrong; I relied on AI too much. 😃 However, the large-size mount is still true.

Since the last time I looked at free disk space, the mounts have changed.

Previously, there was an additional mount with plenty of space.

This time, the root mount already contains 80GB of free space.

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       145G   66G   80G  46% /
tmpfs           7.9G   84K  7.9G   1% /dev/shm
tmpfs           3.2G  1.1M  3.2G   1% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
efivarfs        128M   32K  128M   1% /sys/firmware/efi/efivars
/dev/sda16      881M   64M  756M   8% /boot
/dev/sda15      105M  6.2M   99M   6% /boot/efi
tmpfs           1.6G   12K  1.6G   1% /run/user/1001

If there's always a mount with enough space, we could conditionally use it by pointing Docker or Podman to it. No need to free space.