Skip to content

ci: hot-path performance gate (instruction counts) and wall-clock A/B - #270

Draft
unbalancedparentheses wants to merge 4 commits into
mainfrom
ci/perf-gate
Draft

unbalancedparentheses wants to merge 4 commits into
mainfrom
ci/perf-gate

Conversation

@unbalancedparentheses

@unbalancedparentheses unbalancedparentheses commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

What and why

The 0.8 release gate found async burst −37% and default request() −24% against main, by hand, on fede. This adds a check that catches that class of regression on every PR touching the hot path.

  • benchmarks/hotpath/driver.rs: one portable driver (#[actor] handlers, request(), send() only, so it builds on any recent revision). hotpath.py copies the head's driver into both trees, so base and head run the same workload.
  • benchmarks/hotpath.py gate (CI): instructions per operation under callgrind, at 2,000 and 6,000 operations (differenced, so process start and setup cancel), median of 3, head vs merge base. Fails if any benchmark grows more than 3% (--threshold, HOTPATH_THRESHOLD in the workflow). Per-benchmark table in the job summary.
  • Gated benchmarks are the deterministic ones: async on a Tokio current-thread runtime (burst, sequential request(), 16-task request(), send from 1 and 2 producers: identical counts run to run) and threads-mode sequential request() (within 0.5% on fede, 0.9% on the runner). Multi-thread runtimes and free-running OS-thread producers vary up to ±20% in instructions from run to run, so threads-mode burst/send and the multi-thread async variants are wall-clock only.
  • benchmarks/hotpath.py abba (manual, pre-release): wall-clock ABBA A/B on fixed cores, median and range; added to the release checklist.
  • .github/workflows/perf-gate.yml: on every PR and merge_group, but it only builds and measures when concurrency/src, rt/src, macros/src or the driver changed; otherwise it passes with a "skipped" summary, so it can be a required check (a paths: filter would leave a required check pending).

How it was verified

Self-test on fede (nice 19).

main vs main (two runs), all PASS: every async benchmark 0.0% (spread ±0), threads_request_seq +0.1% / +0.2%. Same on the GitHub runner (this PR's own perf-gate run): async 0.0%, threads −0.9%.

#240 head 7f716152 vs main 0791ace5: FAIL, exit 1:

benchmark base Ir/op head Ir/op change max spread result
async_ct_burst 3,825 4,467 +16.8% ±0 FAIL
async_ct_request_seq 3,395 4,059 +19.6% ±1 FAIL
async_ct_request_conc16 2,512 3,278 +30.5% ±0 FAIL
async_ct_send_p1 839 1,015 +21.0% ±0 FAIL
async_ct_send_p2 833 1,011 +21.3% ±0 FAIL
threads_request_seq 3,309 2,397 −27.6% ±23 ok

Limit: instruction counts miss contention, wake-ups and cache-line traffic, so the wall-clock A/B stays a release step (docs/TESTING.md, "Performance gate").

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant