Skip to content

bench: one set of drivers, tuned load, and results that name their tree - #188

Merged
MDA2AV merged 1 commit into
mainfrom
bench/reproducible
Aug 16, 2026
Merged

bench: one set of drivers, tuned load, and results that name their tree#188
MDA2AV merged 1 commit into
mainfrom
bench/reproducible

Conversation

@MDA2AV

@MDA2AV MDA2AV commented Aug 15, 2026

Copy link
Copy Markdown
Owner

Benchmarks here could not be reproduced, and that cost a day chasing an HTTP/3 "regression" that did not exist. Three causes, all fixed.

The drivers had drifted apart

run.sh drove HTTP/3 as -t 4 --connections 64 -m 8 --send-batch 8, any.sh as --connections 64 -m 32. The same server measured 260k under one and 505k under the other — both real, neither comparable, and neither file said they differed.

Every driver now lives in bench/lib.sh; both scripts source it. Zero divergent invocations remain.

The fixed load shape was not the peak

bash bench/any.sh --tune sweeps connection counts per sample, keeps the winner in bench/tuned.tsv, and later runs reuse it — reproducible and near the server's peak. On this hardware:

conns=4    799,260  2.51us/req      conns=32   794,518  2.52us/req
conns=8    812,137  2.46us/req      conns=64   497,467  4.03us/req  <- old default
conns=16   819,938  2.44us/req  <-  conns=128  456,692  4.38us/req

Per-connection cost falls off a cliff past ~16 connections and the default sat on the wrong side of it: 848,582 req/s tuned versus 497,467 at the default. A plain rerun reproduced 805,569 (−0.6%). Each row now records the shape it was measured at, and rows are only comparable when those match.

Results could not be tied to a tree

bench/results/20260810T001150Z.json names commit: 96137b9 alongside samples that did not exist until ~15 hours later — they were uncommitted when it ran. The commit was stamped correctly; the tree was dirty and nothing said so.

Runs now record dirty, cpu, governor and server_cpus.

Two guards for what utilisation cannot see

Driver-bound detection. A run can peg both reactors and still be limited by the load generator — fewer requests in flight means less batching per wakeup, so the server burns more CPU per request while looking fully busy. The case that started this sat at 97% utilisation and 7.5 µs/req; the same server under a heavier driver did 3.8. After each measurement the load is doubled briefly, and the row is flagged if throughput climbs.

Core placement. On an i9-14900K the same HTTP/3 server measures ~490k req/s on P-cores and 96k on E-cores; left alone the scheduler picks and its choice reads as a regression. The server is pinned to the performance cores — all of them, since narrower is worse:

pinned to two cores    465k        pinned to all P-cores  494k
pinned to the 6GHz pair 482k       unpinned               505k

Denying the scheduler any choice costs more than the placement gains. The driver is not pinned; confining it would only move the bottleneck onto the driver.

Also

bench/README.md documents each trap with the numbers that prove it, including the pkill -f self-match that kills the shell doing the cleanup, and the SO_REUSEPORT case where a leaked server is silently blended into the next result.

Three things made a number here impossible to trust, and all three cost a day of
archaeology chasing a regression that did not exist.

The drivers had drifted apart. run.sh drove HTTP/3 as
"-t 4 --connections 64 -m 8 --send-batch 8" and any.sh as "--connections 64 -m 32";
the same server measured 260k under one and 505k under the other, and neither
file said they differed. Every driver now lives in lib.sh and both scripts source
it.

The load shape was fixed at a value that is not the peak. Sweeping it shows
Http3/Nghttp3Buffered doing 820k req/s at 16 connections and 497k at 64, with
CPU per request nearly doubling across that step - so the default was reporting
497k for a server that does 848k. --tune sweeps the ladder, keeps the winner per
sample in bench/tuned.tsv, and later runs reuse it, which makes a measurement
both reproducible and near the peak. Each row records the shape it was taken at.

Results could not be tied to a tree. One recorded run names commit 96137b9
alongside samples that did not exist for another fifteen hours, because they were
uncommitted when it ran. Runs now record dirty, cpu, governor and server_cpus.

Two guards for what utilisation cannot see. A run can peg both reactors and still
be driver-bound - fewer requests in flight means less batching per wakeup - so
after each measurement the load is doubled briefly and the row is flagged if
throughput climbs. And the server is pinned to the performance cores, because on
a hybrid CPU the scheduler may put a reactor on an efficiency core and the same
server then measures a fifth of its throughput. All of them, not a chosen few:
narrower pinning measured worse than letting the scheduler pick among P-cores.
@MDA2AV
MDA2AV merged commit 07b3708 into main Aug 16, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant