Repository navigation
Measure every comparison figure again with Tinybench and Hyperfine on Node 26.13.0 and Bun 1.4.4 - #58
Conversation
…et a part continue past a unit out of attempts
…out, check the matrix is whole, admit the calibration, add the campaign controller; part B analysis reports rounds, range and settled, and works without part R
…dict in the analyzers
…ze.mjs; empty or short Hyperfine reports and units with no rows are failures; tidy the raw files at the end of a campaign
…shorter windows, one admission window per unit), extend only unsettled comparisons; the stopped first plan's rows kept apart
…ows, one admission window per unit; analyzers use a true median and the A/A gap, reused checks are timed from the window's end, receipt and tidy failures fail, too-few-chunks is not admitted; the freeze ignores analysis files
…d, the host rules require it); two rounds declared as an owner-requested count
… the declared round count
… and the analyzer inputs placed at the fixed paths
…egenerated tables and charts
…es from the new results; analysis test accepts a release without a regression part
![bompus-ai-review[bot]](https://avatars.githubusercontent.com/u/253635?s=60&v=4){kind=small}
…heck for the regression branch, scrub home paths from receipts, fresh-directory reproduction, record the harness limits
…o measured code changed)
There was a problem hiding this comment.
Bompus AI Review (GPT-6.1 Terra, max reasoning): 9 finding(s).
bench/make-assets.mjs:824medium: Use the analyzer's unsettled verdict in speed charts
medium: Use the analyzer's unsettled verdict in speed charts
analyze-more.mjs now records "settled: !unsettled", including when a library has fewer than two rounds, but the chart decides whether to show "no clear change" solely from the rounded multiplier and band.
Failure: If one of a library's two units exhausts its attempts, a remaining 3× sample is marked unsettled by the analyzer but rendered as an unqualified 3× result in a chart claiming two processes per library.
Suggested fix: Render the saved settled verdict and round count; apply the corresponding clear verdict to the S3 speed chart as well.
…mary, scrub cgroup paths, qualify process peaks, fresh-directory reproduction, cohort pointer, baseline-less regression cell, chart verdict from the analyzer
…group paths; qualify process peaks; fresh-directory reproduction; cohort pointer; baseline-less regression cell; chart verdict from the analyzer
|
Re the summary finding on the speed charts: the other-service chart now takes its 'no clear change' mark from the analyzer's settled verdict (fixed in this push). The S3 speed chart is unchanged: every cell of that part has both rounds in this run (n=2 for all 12 arms), so the analyzer's verdict and the chart agree; a missing-unit case would need the same guard, which goes in the later harness change. |
…tial copy when the cleanup grep fails
… contradict (#59) * Docs: correct statements the review of #58 found stale or too strong (measured clients, rounds, multipliers, earlier experiments, exit codes, README endpoints) * Docs: separate the driver's exit 9 from the campaign's status * Docs: per-chart round counts, the ncc legend's twice-band condition, and the scope of the campaign's gap line
Sigflint signs an S3 GET in 0.99 µs on Node 26.13.0, 3.5 times as fast as the next fastest library (
lean-s3, 3.44 µs); on Bun 1.4.4 it takes 0.65 µs, 2.9 times as fast aslean-s3. Across the two runtimes the other libraries take 2.0 to 87 times as long on the requests they sign correctly. The code under test is unchanged since 1.0.2 (src/tree90a032d5, same built-file hashes). The README had saidaws4was the fastest other library; since the four S3 clients joined the comparison,lean-s3is, so the headline claim changes.What this changes:
bench/protocol-release-1.0.4.mdis the plan declared before the run (two rounds, a count chosen to keep it short), with the stopped plans' rows kept apart. The 66-unit first pass admitted every unit; the extension decision (bench/results/release-1.0.4-extension-decisions.json) added rounds 3 to 5 of memory and import for the comparisons two rounds left unsettled. 38 of its 39 units were admitted; round 4 of thes3-lite-clientmemory unit used its four attempts and stays a recorded gap (that arm has four rounds).docs/tables.md, the SVG charts, the memory block ofdocs/benchmarks.md) come frombench/make-assets.mjsandbench/make-memory-assets.mjs; the prose, README and changelog were edited by hand and checked against the saved results. The Deno table is unchanged and now says it used the earlier timer.test/benchmark-analysis.test.jsaccepts a release directory without a regression file;bench/make-memory-assets.mjsnames Node 26.13.0;bench/run-campaign.sh,tidy-results.mjsand the analyzers carry the 1.0.4 plan's fixes (already described in the plan).Checks:
npm run checkpasses (96 tests). The measured units each passed a fresh 10 s admission window before and after, plus the continuous unit observer. Deviation recorded in the plan: the campaign controller retried a refused admission every 15 s instead of the declared 60 s.Not in this PR: the version bump, tag, publish and announcement (held for the owner's go). No local review pass was run on this PR; the private reviewer reviews it.
Update after the first review round: the committed admission receipts have the home directory replaced by ~ and the checkout by ; the memory-table text now qualifies the allocation range and the s3mini note; the protocol records three harness limits found in review (left unfixed so the measured code stays as frozen) and a fresh-directory reproduction. The freeze record was rewritten for the final protocol text; no measured code changed.