Skip to content

ci: modernize Rust test caching and split test jobs - #739

Merged
lwshang merged 3 commits into
mainfrom
ci/modernize-test-cache
Jul 23, 2026
Merged

lwshang merged 3 commits into
mainfrom
ci/modernize-test-cache

Conversation

@lwshang

@lwshang lwshang commented Jul 23, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The test CI was slow and its cache strategy was not smart. test.yml, lint.yml, fmt.yml, and netlify.yml all cached ~/.cargo plus the entire target/ under a single key derived only from Cargo.lock, with no restore-keys — so any dependency change was a total cache miss (every dep recompiled), and target/ was frozen at first-save and grew unbounded.

This applies the caching/CI practices already in use in icp-cli.

Performance (measured on this PR's own runs)

Per-job wall time, cache miss (cold) vs cache hit (warm) — same jobs, same commits, the only difference is whether rust-cache restored the dependency build. Cache hits are verified in the logs (e.g. ubuntu: Restored from cache key "v0-rust-Linux-test-…" full match: true).

Job (new Tests workflow) Cache miss Cache hit
Run Tests · ubuntu-latest 12.6m 7.1m
Run Tests · macos-latest 14.4m 7.3m
Run Tests · windows-2025 10.9m 7.8m
ref-tests · ubuntu 6.7m 4.1m
WASM tests · ubuntu 3.6m 2.7m

Because the jobs now run concurrently, the whole workflow's wall-clock is the slowest single job rather than the old sequential sum:

Scenario Workflow wall-clock
Before — main, one sequential job per OS (Run Tests → ref-tests → WASM → SoftHSM) ~14–18m
After — cold cache ~14m
After — warm cache (hit) ~8m

So a typical warm-cache PR iteration goes from ~15–18m down to ~8m (~2×), and even a cold run beats the old sequential job thanks to the split. (The old naive cache also invalidated on any Cargo.lock change — e.g. every release — forcing a full cold rebuild; the new key restores via prefix instead.)

Changes

Caching — switch Rust caching to actions-rust-lang/setup-rust-toolchain (wraps Swatinem/rust-cache): deps-only caching, smart restore-keys, per-purpose shared keys. Each job first removes the runner's bundled stable toolchain so the cache key doesn't drift as the runner image updates (the toolchain is pinned to 1.88.0 via rust-toolchain.toml).

Test-job split — the single ubuntu job (which ran Run Tests → ref-tests → WASM → SoftHSM sequentially) is split into concurrent jobs, each with its own cache key:

Job OS Cache key
test (Run Tests loop) ubuntu / macos / windows-2025 <os>-test
ref-tests + SoftHSM ubuntu Linux-reftests
wasm ubuntu Linux-wasm
lint ubuntu Linux-lint

Wall-clock is now the slowest job instead of the sum. Because tests run per-crate and ref-tests is filtered out of the test job, pocket-ic only compiles in the ref-tests job — keeping the other caches small. Test commands are unchanged.

CI env tuning (test.yml, lint.yml) — CARGO_INCREMENTAL=0, CARGO_NET_RETRY=10, CARGO_PROFILE_DEV_DEBUG=line-tables-only, the Windows lld-link linker (Windows pinned to windows-2025 so the LLVM path is guaranteed), and a Windows Defender exclusion step.

Docs-only skip — dorny/paths-filter gate on test.yml and lint.yml; docs-only PRs skip the heavy jobs and the required *:required aggregates pass via the skip. (Not added to fmt.yml — the filter runner costs more than the fmt check.)

Cleanup — delete the long-dead netlify.yml workflow, and drop the dead agent-rust.netlify.app "master" doc links from the ic-agent / ic-identity-hsm READMEs (docs.rs is the published-docs convention; matches ic-utils).

Follow-up fixes (from CI on this PR)

Two regressions the job split surfaced (ci: fix test-job split regressions):

  • setup-rust-toolchain defaults RUSTFLAGS=-D warnings (the old actions/cache did not), so the WASM build failed on pre-existing dead-code warnings. Tests were never built warnings-as-errors (only lint is), so rustflags: "" is set on the test, ref-tests, and wasm jobs.
  • The ic-agent doctests spin up pocket-ic via ref-tests' helpers and need the downloaded assets. The old single job downloaded them before the test loop; Download test assets is re-added to the test job (non-Windows, as before).

De-flake a long-standing flaky test (test: de-flake dynamic-routing seed tests) — surfaced by the Windows run here. test_route_with_initially_unhealthy_seeds_becoming_healthy has been flaky since it was added and regularly needed reruns. It is not a timing issue: with two equal-score healthy seeds each route() is a uniform 50/50 draw, so route_n_times(6) misses one node with probability 2·(1/2)⁶ = 1/32 ≈ 3.1%. Sibling tests were already bumped to route_n_times(30) "for probabilistic routing"; this test and test_routing_with_topology_and_node_health_updates (same two-node/6-draw pattern) were missed. Both bumped to 30 → miss probability ≈ 2e-9. Test-only change (touches only #[cfg(test)] route counts).

Validation

  • All workflows parse as YAML and pass actionlint (no expression/needs/syntax errors; remaining shellcheck infos are pre-existing, in verbatim-copied test scripts).
  • Full CI run on this branch is green: test (ubuntu/macos/windows-2025), ref-tests, wasm, lint, fmt, and the *:required aggregates all pass.

🤖 Generated with Claude Code

lwshang and others added 3 commits July 23, 2026 12:49
The test/lint/fmt/netlify workflows all cached ~/.cargo plus the entire
target/ under a single key derived only from Cargo.lock, with no
restore-keys. Any dependency change was a total cache miss (every dep
recompiled), and target/ was frozen at first-save and grew unbounded.

Switch Rust caching to actions-rust-lang/setup-rust-toolchain (which
wraps Swatinem/rust-cache): deps-only caching, smart restore-keys, and a
per-purpose shared key. Remove the runner's bundled `stable` toolchain
first so the cache key doesn't drift as the image updates (the toolchain
is pinned to 1.88.0 via rust-toolchain.toml).

Other changes, mirroring the icp-cli CI:
- Split the single ubuntu test job into concurrent jobs — `test` (all
  three OSes), `ref-tests` + SoftHSM (Linux), and `wasm` (Linux) — each
  with its own cache key. Wall-clock is now the slowest job, not the sum.
  Because tests run per-crate and ref-tests is filtered out, pocket-ic
  only compiles in the ref-tests job, keeping the other caches small.
- Add CI env tuning: CARGO_INCREMENTAL=0, CARGO_NET_RETRY=10,
  CARGO_PROFILE_DEV_DEBUG=line-tables-only, and the Windows lld-link
  linker (windows pinned to 2025 so the LLVM path is guaranteed), plus a
  Windows Defender exclusion step.
- Skip test/lint on docs-only changes via dorny/paths-filter; the
  required aggregate checks pass via the skip.
- Delete the long-dead netlify docs workflow and drop the dead
  netlify.app "master" doc links from the ic-agent and ic-identity-hsm
  READMEs (docs.rs is the published-docs convention).

Test commands themselves are unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two failures from splitting the single test job:

- setup-rust-toolchain defaults RUSTFLAGS to "-D warnings", which the old
  actions/cache did not. The WASM build then failed on pre-existing
  dead-code warnings (test helpers unused on the wasm target). The test
  jobs were never warnings-as-errors (only lint is), so set rustflags: ""
  on the test, ref-tests, and wasm jobs.

- The ic-agent doctests use ref-tests' pocket-ic helpers and need the
  downloaded assets. The single job used to download them before the test
  loop; re-add the download step to the test job (non-Windows, as before).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
test_route_with_initially_unhealthy_seeds_becoming_healthy has been flaky
since it was added and regularly needs CI reruns. The failure is
assert_routed_domains at test_utils.rs:47 reporting a subset of the two
expected domains — not a timing issue: both seeds are healthy, but with
two equal-score nodes each route() is a uniform 50/50 draw, so
route_n_times(6) misses one node with probability 2*(1/2)^6 = 1/32 ≈ 3.1%.

Sibling tests were already bumped to route_n_times(30) "for probabilistic
routing"; this test (and test_routing_with_topology_and_node_health_updates,
which has the same two-node/6-draw pattern) were missed. Bump both to 30,
dropping the miss probability to 2*(1/2)^30 ≈ 2e-9. The single-node call
sites (n=3/6) cannot undersample and are left as-is.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@lwshang
lwshang marked this pull request as ready for review July 23, 2026 17:58
@lwshang
lwshang requested a review from a team as a code owner July 23, 2026 17:58
@zeropath-ai

zeropath-ai Bot commented Jul 23, 2026

Copy link
Copy Markdown

✅ No security or compliance issues detected. Reviewed everything up to abcdbb4.

Security Overview
Detected Code Changes
Change Type Relevant files
Enhancement ► .github/workflows/fmt.yml
    Enable updated checkout and rust toolchain handling
► .github/workflows/lint.yml
    Add CI changeset filtering, remove bundled Rust toolchain, update rust-toolchain setup for lint, adjust aggregate condition
► .github/workflows/test.yml
    Add environment vars for cargo, doc/tests filter, multi-OS test matrix changes, introduce wasm and ref-tests workflows, adjust test aggregation
► ic-agent/README.md
    Remove link formatting for Documentation (master) in README
► ic-agent/src/agent/route_provider/dynamic_routing/dynamic_route_provider.rs
    Increase route_n_times calls in tests to 30 (probabilistic routing)
Bug Fix ► ic-identity-hsm/README.md
    Remove Documentation (master) link line

@lwshang
lwshang merged commit a53a273 into main Jul 23, 2026
27 checks passed
@lwshang
lwshang deleted the ci/modernize-test-cache branch July 23, 2026 18:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants