Benchmarks: a changed benchmark runs only the benchmarks of its modules - #488
Merged
Merged
Conversation
ci/bench_arches.py treated every change under bench/ as shared, running every benchmark on every architecture. A pull request that only gates an algorithm's benchmark on another architecture (as adding one does, since check_arch_gates.py requires it) then ran the whole suite in each of the nine configurations, past the 15-minute timeout on the slower runners. A changed bench/benches/primitives/<name>.rs now narrows to the modules its USES lists, whose benchmarks include it; main.rs, and a benchmark whose USES cannot be read (e.g. a deleted one), still run everything. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WkLN6tAYk76HACiEWLbAMD
This was referenced Oct 1, 2026
Every benchmark, which a shared change (or a run by hand) still runs, now takes about 14 minutes on the aarch64 runners and more than the 15-minute timeout on the x86-64, x86 and ARMv7 ones, which were cancelled on this pull request's own run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WkLN6tAYk76HACiEWLbAMD
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ci/bench_arches.pytreats every change underbench/as shared, so it runs every benchmark on every architecture. A full run now takes about 14 minutes on the aarch64 runners and more than the job's 15-minute timeout on the others. Two changes:A changed benchmark runs only its own modules' benchmarks. Adding an algorithm to an architecture always edits that algorithm's benchmark, because
check_arch_gates.pyrequires itscfgto match the library module's. Such a PR therefore ran the whole suite in all nine configurations, and the jobs were cancelled at the timeout. That is what happened to the AES-CMAC PRs Implement AES-CMAC on AArch64, with an AES-extension variant #469, Implement AES-CMAC on ARMv7 #477 and Implement AES-CMAC on x86 #485, whose only benchmark change is thecfgline.Now a changed
bench/benches/primitives/<name>.rsnarrows the run to the modules itsUSESlists, on every architecture. Those modules' benchmarks include the changed one. For example,cmac_aes.rs(USES = ["cmac_aes", "aes"]) now runs the CMAC and AES-GCM benchmarks. Everything still runs for:main.rs;USEScan't be read, such as a deleted one;bench/.A narrowed run never benchmarks less than a change's library modules need: the narrowing only replaces "everything" with the changed benchmark's own modules.
The compare job's timeout goes from 15 to 30 minutes. Shared changes (e.g.
Cargo.lock,src/lib.rs, the comparison scripts) and runs by hand still benchmark everything. On this PR's own first run, every x86-64, x86 and ARMv7 job was cancelled at 15 minutes, and the aarch64 ones finished in about 14. If maintainers would rather keep runner time down, the alternative is to cut the number of roundsbench_compare.pyruns for a full run.Testing
I ran
git diff --name-only … | python3 ci/bench_arches.pyon file lists:aes cmac_aeson each architecture, instead of an emptymodules(everything).bench/benches/primitives/main.rsand a missing benchmark file still give everything.--allis unchanged (nine entries).This PR's own Benchmarks run still runs everything, because it changes
ci/bench_arches.pyandbench.yml, which are shared. The run on the second commit shows whether 30 minutes is enough.🤖 Generated with Claude Code
https://claude.ai/code/session_01WkLN6tAYk76HACiEWLbAMD