Repository navigation
Bench: time signing operations with Tinybench and cold starts with Hyperfine, not hand-written timers - #57
Merged
Merged
Conversation
…perfine, not hand-written timers
…, and fail loudly on old-format import rows
![bompus-ai-review[bot]](https://avatars.githubusercontent.com/u/253635?s=60&v=4){kind=small}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The release phases timed with hand-written
performance.now()loops; the host's runner choices are Tinybench for JavaScript operations and Hyperfine for whole commands. This moves the phases onto them. It changes the method, so figures from before and after are not directly comparable; the release reruns every arm.bench/timing.mjs(Tinybench 6.2.2, pinned) times a signing call in batches of 1,000, at least 10 batches in about a second after a warm-up of at least 4 batches (Tinybench's defaults of 64 and 16 batches would run slow signers far past their windows).speed.mjsandspeed-more.mjsuse it.import-once.mjsreplacesimport-time.mjs(no in-process timing; exits 1 on a wrong first signature; candidatenoneis the harness without a library), onehyperfinecall per round and runtime over the control and all arms, rows built byhyperfine-rows.mjs. The start-up metric is the arm's command time minus the control's. Part D keeps the 64 MiB swap-drop rejection, andanalyze-memory.mjsfails loudly on import rows from the earlier in-process timer instead of dropping them.memory.mjsloses its hand-written wall timer; CPU time, GC and heap counters stay runtime counters, which neither tool reports.run-admitted.shdefaults to the host lock scheduled jobs honor (~/.cache/setupLinuxHost/post-upgrade-bench.lock) instead of a path nothing else checks. The olderrun-matrix.sh,run-memory.shandrun-more.share marked historical.await-cost.mjsandhash-encode.mjsstill use hand timers; they are outside the release phases.Checked:
npm run check; part D ran end to end throughrun-admitted.sh(admitted, 182 rows). Before/after on the same artifact (two full passes, old then new, and two partial repeats in new then old order, so drift is only partly controlled): 78 speed cells differ by a median of 2.0% (71 within 10%); the cells that moved more than 10% were single-pass noise (a second pass of the old timer agrees with the new runner); cold-start figures land within 1.5 ms. Independent review (GPT-6.1 Sol, xhigh) found three defects (old-format rows dropped silently, part D lost its swap rejection, the batch minimums broke the timing windows); all are fixed in the last commit.