CI build optimizations from profiling: enable mold linker in CI; tame llvm-cov write churn (sccache verified healthy)

[Claudius-Maginificent]

Summary

While profiling the Rust workspace test suite (the test-mac steps in .github/workflows/tests-rs-workspace.yml) for cold-vs-hot timing and IO/CPU/memory, two software-side improvements surfaced (no runner changes). A third candidate — sccache — was investigated and found already healthy in CI (see Verified / no action).

Measured (Linux, 14 cores, 20 GB RAM, mold; exact Mac package set: 22-pkg nextest + 4-pkg shielded cargo test, --all-features)

• Cold suite 450 s (compile ~239 s / 53%, test exec ~211 s / 47%); hot 211 s → warm target/ = 2.13× speedup, all of it from skipped compilation.
• Plain target/ 12.6 GB vs llvm-cov-instrumented 77 GB; disk writes 21 GB vs ~85 GB @ ~335 MB/s (llvm-cov).
• Compile+test working set ~10–14 GB (RAM not the bottleneck).

Recommendations (software/config only)

1. Enable the mold linker in CI Linux builds — .cargo/config.toml, Dockerfile, Linux rust setup

Confirmed against CI: neither the Dockerfile nor the repo .cargo/config.toml set a fast linker for x86_64-unknown-linux-gnu (the repo config only sets an aarch64 cross linker=), so CI Linux/Docker builds link with the default ld. -fuse-ld=mold currently lives only in contributors' personal ~/.cargo/config.toml.

• Proposal: add -C link-arg=-fuse-ld=mold to the repo Linux rustflags + install mold in the Dockerfile builder stage / Linux setup.
• Note: a ci/mold-linker branch already exists — this issue captures the measured justification to land it.
• Benefit: shorter link step on the big crates (drive-abci, dash-sdk).
• Trade-off: mold must be present wherever Linux builds run; not applicable to the macOS runner.

2. Tame the per-PR llvm-cov write churn (keep coverage on PRs) — runner env / build wrapper

Per-PR coverage is intentional and stays. But the instrumented build writes ~85 GB/run (almost all regenerable) on the self-hosted Mac, wearing the SSD. To cut actual writes without dropping coverage:

• Wrap the cargo invocation in eatmydata (no-op fsync/fdatasync) and relax vm.dirty_ratio/vm.dirty_background_ratio so ephemeral artifacts die in page cache before writeback; and/or put target/llvm-cov-target on a dedicated fast scratch volume.
• Benefit: large drop in bytes hitting disk; coverage preserved. Trade-off: durability traded for speed — fine for regenerable CI scratch only.

Verified / no action

• sccache is healthy in CI. Checked the nightly Drive build (run 28136181686, job Build Drive image): Cache hits (Rust) 528, Cache misses 0, Rust hit-rate 100% (C/C++ 100% too). An earlier local repro showed ~0% Rust hits, but that was a local-cache mismatch — not a CI issue. No change needed; the Dockerfile already emits sccache --show-stats in build-images logs for future audits.

Already good — keep

• macOS runner uses clean: false (warm target/) — the single biggest lever (2.13×). ✅
• CARGO_PROFILE_DEV_DEBUG=0, CARGO_INCREMENTAL=0, CARGO_PROFILE_DEV_CODEGEN_UNITS=256. ✅
• Per-PR llvm-cov coverage — intentional (per-PR diff coverage); keep. ✅
• sccache S3 cache — 100% Rust hit-rate on warm builds. ✅

Out of scope

Runner hardware sizing is being evaluated separately; this issue covers only .github/* and build-script/config changes.

---

_{Filed from CI profiling analysis. Co-authored by Claudius the Magnificent AI Agent}