bench: one set of drivers, tuned load, and results that name their tree - #188
Merged
Conversation
Three things made a number here impossible to trust, and all three cost a day of archaeology chasing a regression that did not exist. The drivers had drifted apart. run.sh drove HTTP/3 as "-t 4 --connections 64 -m 8 --send-batch 8" and any.sh as "--connections 64 -m 32"; the same server measured 260k under one and 505k under the other, and neither file said they differed. Every driver now lives in lib.sh and both scripts source it. The load shape was fixed at a value that is not the peak. Sweeping it shows Http3/Nghttp3Buffered doing 820k req/s at 16 connections and 497k at 64, with CPU per request nearly doubling across that step - so the default was reporting 497k for a server that does 848k. --tune sweeps the ladder, keeps the winner per sample in bench/tuned.tsv, and later runs reuse it, which makes a measurement both reproducible and near the peak. Each row records the shape it was taken at. Results could not be tied to a tree. One recorded run names commit 96137b9 alongside samples that did not exist for another fifteen hours, because they were uncommitted when it ran. Runs now record dirty, cpu, governor and server_cpus. Two guards for what utilisation cannot see. A run can peg both reactors and still be driver-bound - fewer requests in flight means less batching per wakeup - so after each measurement the load is doubled briefly and the row is flagged if throughput climbs. And the server is pinned to the performance cores, because on a hybrid CPU the scheduler may put a reactor on an efficiency core and the same server then measures a fifth of its throughput. All of them, not a chosen few: narrower pinning measured worse than letting the scheduler pick among P-cores.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Benchmarks here could not be reproduced, and that cost a day chasing an HTTP/3 "regression" that did not exist. Three causes, all fixed.
The drivers had drifted apart
run.shdrove HTTP/3 as-t 4 --connections 64 -m 8 --send-batch 8,any.shas--connections 64 -m 32. The same server measured 260k under one and 505k under the other — both real, neither comparable, and neither file said they differed.Every driver now lives in
bench/lib.sh; both scripts source it. Zero divergent invocations remain.The fixed load shape was not the peak
bash bench/any.sh --tunesweeps connection counts per sample, keeps the winner inbench/tuned.tsv, and later runs reuse it — reproducible and near the server's peak. On this hardware:Per-connection cost falls off a cliff past ~16 connections and the default sat on the wrong side of it: 848,582 req/s tuned versus 497,467 at the default. A plain rerun reproduced 805,569 (−0.6%). Each row now records the shape it was measured at, and rows are only comparable when those match.
Results could not be tied to a tree
bench/results/20260810T001150Z.jsonnamescommit: 96137b9alongside samples that did not exist until ~15 hours later — they were uncommitted when it ran. The commit was stamped correctly; the tree was dirty and nothing said so.Runs now record
dirty,cpu,governorandserver_cpus.Two guards for what utilisation cannot see
Driver-bound detection. A run can peg both reactors and still be limited by the load generator — fewer requests in flight means less batching per wakeup, so the server burns more CPU per request while looking fully busy. The case that started this sat at 97% utilisation and 7.5 µs/req; the same server under a heavier driver did 3.8. After each measurement the load is doubled briefly, and the row is flagged if throughput climbs.
Core placement. On an i9-14900K the same HTTP/3 server measures ~490k req/s on P-cores and 96k on E-cores; left alone the scheduler picks and its choice reads as a regression. The server is pinned to the performance cores — all of them, since narrower is worse:
Denying the scheduler any choice costs more than the placement gains. The driver is not pinned; confining it would only move the bottleneck onto the driver.
Also
bench/README.mddocuments each trap with the numbers that prove it, including thepkill -fself-match that kills the shell doing the cleanup, and the SO_REUSEPORT case where a leaked server is silently blended into the next result.