Repository navigation
ci: modernize Rust test caching and split test jobs - #739
Merged
Merged
Conversation
The test/lint/fmt/netlify workflows all cached ~/.cargo plus the entire target/ under a single key derived only from Cargo.lock, with no restore-keys. Any dependency change was a total cache miss (every dep recompiled), and target/ was frozen at first-save and grew unbounded. Switch Rust caching to actions-rust-lang/setup-rust-toolchain (which wraps Swatinem/rust-cache): deps-only caching, smart restore-keys, and a per-purpose shared key. Remove the runner's bundled `stable` toolchain first so the cache key doesn't drift as the image updates (the toolchain is pinned to 1.88.0 via rust-toolchain.toml). Other changes, mirroring the icp-cli CI: - Split the single ubuntu test job into concurrent jobs — `test` (all three OSes), `ref-tests` + SoftHSM (Linux), and `wasm` (Linux) — each with its own cache key. Wall-clock is now the slowest job, not the sum. Because tests run per-crate and ref-tests is filtered out, pocket-ic only compiles in the ref-tests job, keeping the other caches small. - Add CI env tuning: CARGO_INCREMENTAL=0, CARGO_NET_RETRY=10, CARGO_PROFILE_DEV_DEBUG=line-tables-only, and the Windows lld-link linker (windows pinned to 2025 so the LLVM path is guaranteed), plus a Windows Defender exclusion step. - Skip test/lint on docs-only changes via dorny/paths-filter; the required aggregate checks pass via the skip. - Delete the long-dead netlify docs workflow and drop the dead netlify.app "master" doc links from the ic-agent and ic-identity-hsm READMEs (docs.rs is the published-docs convention). Test commands themselves are unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two failures from splitting the single test job: - setup-rust-toolchain defaults RUSTFLAGS to "-D warnings", which the old actions/cache did not. The WASM build then failed on pre-existing dead-code warnings (test helpers unused on the wasm target). The test jobs were never warnings-as-errors (only lint is), so set rustflags: "" on the test, ref-tests, and wasm jobs. - The ic-agent doctests use ref-tests' pocket-ic helpers and need the downloaded assets. The single job used to download them before the test loop; re-add the download step to the test job (non-Windows, as before). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
test_route_with_initially_unhealthy_seeds_becoming_healthy has been flaky since it was added and regularly needs CI reruns. The failure is assert_routed_domains at test_utils.rs:47 reporting a subset of the two expected domains — not a timing issue: both seeds are healthy, but with two equal-score nodes each route() is a uniform 50/50 draw, so route_n_times(6) misses one node with probability 2*(1/2)^6 = 1/32 ≈ 3.1%. Sibling tests were already bumped to route_n_times(30) "for probabilistic routing"; this test (and test_routing_with_topology_and_node_health_updates, which has the same two-node/6-draw pattern) were missed. Bump both to 30, dropping the miss probability to 2*(1/2)^30 ≈ 2e-9. The single-node call sites (n=3/6) cannot undersample and are left as-is. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
lwshang
marked this pull request as ready for review
July 23, 2026 17:58
|
✅ No security or compliance issues detected. Reviewed everything up to abcdbb4. Security Overview
Detected Code Changes
|
adamspofford
approved these changes
Jul 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The
testCI was slow and its cache strategy was not smart.test.yml,lint.yml,fmt.yml, andnetlify.ymlall cached~/.cargoplus the entiretarget/under a single key derived only fromCargo.lock, with norestore-keys— so any dependency change was a total cache miss (every dep recompiled), andtarget/was frozen at first-save and grew unbounded.This applies the caching/CI practices already in use in
icp-cli.Performance (measured on this PR's own runs)
Per-job wall time, cache miss (cold) vs cache hit (warm) — same jobs, same commits, the only difference is whether
rust-cacherestored the dependency build. Cache hits are verified in the logs (e.g. ubuntu:Restored from cache key "v0-rust-Linux-test-…" full match: true).Testsworkflow)Run Tests· ubuntu-latestRun Tests· macos-latestRun Tests· windows-2025ref-tests· ubuntuWASM tests· ubuntuBecause the jobs now run concurrently, the whole workflow's wall-clock is the slowest single job rather than the old sequential sum:
main, one sequential job per OS (Run Tests → ref-tests → WASM → SoftHSM)So a typical warm-cache PR iteration goes from ~15–18m down to ~8m (~2×), and even a cold run beats the old sequential job thanks to the split. (The old naive cache also invalidated on any
Cargo.lockchange — e.g. every release — forcing a full cold rebuild; the new key restores via prefix instead.)Changes
Caching — switch Rust caching to
actions-rust-lang/setup-rust-toolchain(wrapsSwatinem/rust-cache): deps-only caching, smart restore-keys, per-purpose shared keys. Each job first removes the runner's bundledstabletoolchain so the cache key doesn't drift as the runner image updates (the toolchain is pinned to1.88.0viarust-toolchain.toml).Test-job split — the single ubuntu job (which ran
Run Tests→ref-tests→WASM→SoftHSMsequentially) is split into concurrent jobs, each with its own cache key:test(Run Tests loop)<os>-testref-tests+ SoftHSMLinux-reftestswasmLinux-wasmlintLinux-lintWall-clock is now the slowest job instead of the sum. Because tests run per-crate and
ref-testsis filtered out of thetestjob,pocket-iconly compiles in theref-testsjob — keeping the other caches small. Test commands are unchanged.CI env tuning (
test.yml,lint.yml) —CARGO_INCREMENTAL=0,CARGO_NET_RETRY=10,CARGO_PROFILE_DEV_DEBUG=line-tables-only, the Windowslld-linklinker (Windows pinned towindows-2025so the LLVM path is guaranteed), and a Windows Defender exclusion step.Docs-only skip —
dorny/paths-filtergate ontest.ymlandlint.yml; docs-only PRs skip the heavy jobs and the required*:requiredaggregates pass via the skip. (Not added tofmt.yml— the filter runner costs more than the fmt check.)Cleanup — delete the long-dead
netlify.ymlworkflow, and drop the deadagent-rust.netlify.app"master" doc links from theic-agent/ic-identity-hsmREADMEs (docs.rs is the published-docs convention; matchesic-utils).Follow-up fixes (from CI on this PR)
Two regressions the job split surfaced (
ci: fix test-job split regressions):setup-rust-toolchaindefaultsRUSTFLAGS=-D warnings(the oldactions/cachedid not), so the WASM build failed on pre-existing dead-code warnings. Tests were never built warnings-as-errors (only lint is), sorustflags: ""is set on thetest,ref-tests, andwasmjobs.ic-agentdoctests spin uppocket-icvia ref-tests' helpers and need the downloaded assets. The old single job downloaded them before the test loop;Download test assetsis re-added to thetestjob (non-Windows, as before).De-flake a long-standing flaky test (
test: de-flake dynamic-routing seed tests) — surfaced by the Windows run here.test_route_with_initially_unhealthy_seeds_becoming_healthyhas been flaky since it was added and regularly needed reruns. It is not a timing issue: with two equal-score healthy seeds eachroute()is a uniform 50/50 draw, soroute_n_times(6)misses one node with probability2·(1/2)⁶ = 1/32 ≈ 3.1%. Sibling tests were already bumped toroute_n_times(30)"for probabilistic routing"; this test andtest_routing_with_topology_and_node_health_updates(same two-node/6-draw pattern) were missed. Both bumped to30→ miss probability≈ 2e-9. Test-only change (touches only#[cfg(test)]route counts).Validation
actionlint(no expression/needs/syntax errors; remaining shellcheck infos are pre-existing, in verbatim-copied test scripts).test(ubuntu/macos/windows-2025),ref-tests,wasm,lint,fmt, and the*:requiredaggregates all pass.🤖 Generated with Claude Code