Skip to content

Benchmarks: a changed benchmark runs only the benchmarks of its modules - #488

Merged
alex merged 2 commits into
mainfrom
claude/optimistic-mayer-jdgs35-bench
Oct 1, 2026
Merged

alex merged 2 commits into
mainfrom
claude/optimistic-mayer-jdgs35-bench

Conversation

@alex

@alex alex commented Oct 1, 2026 •

Copy link
Copy Markdown
Member

ci/bench_arches.py treats every change under bench/ as shared, so it runs every benchmark on every architecture. A full run now takes about 14 minutes on the aarch64 runners and more than the job's 15-minute timeout on the others. Two changes:

  1. A changed benchmark runs only its own modules' benchmarks. Adding an algorithm to an architecture always edits that algorithm's benchmark, because check_arch_gates.py requires its cfg to match the library module's. Such a PR therefore ran the whole suite in all nine configurations, and the jobs were cancelled at the timeout. That is what happened to the AES-CMAC PRs Implement AES-CMAC on AArch64, with an AES-extension variant #469, Implement AES-CMAC on ARMv7 #477 and Implement AES-CMAC on x86 #485, whose only benchmark change is the cfg line.

    Now a changed bench/benches/primitives/<name>.rs narrows the run to the modules its USES lists, on every architecture. Those modules' benchmarks include the changed one. For example, cmac_aes.rs (USES = ["cmac_aes", "aes"]) now runs the CMAC and AES-GCM benchmarks. Everything still runs for:

    • a change to main.rs;
    • a benchmark whose USES can't be read, such as a deleted one;
    • any other file under bench/.

    A narrowed run never benchmarks less than a change's library modules need: the narrowing only replaces "everything" with the changed benchmark's own modules.

  2. The compare job's timeout goes from 15 to 30 minutes. Shared changes (e.g. Cargo.lock, src/lib.rs, the comparison scripts) and runs by hand still benchmark everything. On this PR's own first run, every x86-64, x86 and ARMv7 job was cancelled at 15 minutes, and the aarch64 ones finished in about 14. If maintainers would rather keep runner time down, the alternative is to cut the number of rounds bench_compare.py runs for a full run.

Testing

I ran git diff --name-only … | python3 ci/bench_arches.py on file lists:

  • The diff of Implement AES-CMAC on x86 #485 now gives aes cmac_aes on each architecture, instead of an empty modules (everything).
  • bench/benches/primitives/main.rs and a missing benchmark file still give everything.
  • --all is unchanged (nine entries).

This PR's own Benchmarks run still runs everything, because it changes ci/bench_arches.py and bench.yml, which are shared. The run on the second commit shows whether 30 minutes is enough.

🤖 Generated with Claude Code

https://claude.ai/code/session_01WkLN6tAYk76HACiEWLbAMD

ci/bench_arches.py treated every change under bench/ as shared, running
every benchmark on every architecture. A pull request that only gates an
algorithm's benchmark on another architecture (as adding one does, since
check_arch_gates.py requires it) then ran the whole suite in each of the
nine configurations, past the 15-minute timeout on the slower runners.
A changed bench/benches/primitives/<name>.rs now narrows to the modules
its USES lists, whose benchmarks include it; main.rs, and a benchmark
whose USES cannot be read (e.g. a deleted one), still run everything.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WkLN6tAYk76HACiEWLbAMD
Every benchmark, which a shared change (or a run by hand) still runs,
now takes about 14 minutes on the aarch64 runners and more than the
15-minute timeout on the x86-64, x86 and ARMv7 ones, which were
cancelled on this pull request's own run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WkLN6tAYk76HACiEWLbAMD
@alex
alex added this pull request to the merge queue Oct 1, 2026
Merged via the queue into main with commit abbd54c Oct 1, 2026
40 checks passed
@alex
alex deleted the claude/optimistic-mayer-jdgs35-bench branch October 1, 2026 21:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants