Repository navigation
test: qualify benchmark timing and paired comparisons - #67
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Workers could begin before timing started, and soak payload bandwidth depended on optional library counters. Start timers before releasing workers, count accepted and returned payload bytes in the harness, and separate workload duration from the final drain. Add an optional repeated
put_steadyphase with persistent writers, fixed warmup and measured rounds, bounded work chunks, and a drain after each batch; retain the cold-write result.Use per-worker soak observers, support disabled library statistics without losing payload validation, sample macOS RSS through the shared harness, and drop the closed cache before reopening to avoid overlapping cache instances in memory measurements.
Add a serial ABBA/BAAB comparison tool that retains raw logs, configuration, binary hashes, and power samples. Report paired bootstrap intervals and reject inadequate duration, unstable power, excessive block drift, or excessive spread between same-binary repeats. The repeat check fixes a reproduced case where alternating 1000/500 ops/s A/A runs previously passed because pairing canceled all noise. It uses the existing drift threshold and has regression coverage for both A/A and A/B. Integrate the tool tests into
cargo x testand document the workflow.Validation
Synchronized with
mainat3d29b6e, including #66. Validated PR head6acc5fc:cargo x checkcargo x test: 369 Rust tests plus 9 Python comparison-tool testscargo x lint: formatting, code, documentation, packaging, and dependency policy checks pass, including Tokio 1.54.1 advisoriescargo test --workspace --release --all-features: 367 tests passA fresh byte-identical A/A control used disabled library statistics, eight blocks, all 32 samples, and five-second unpaced mixed workloads with unchanged default thresholds. The tool correctly rejected this host's run: effective-hit block drift was 12.99% and within-block spread was 8.16% against the 5% limit; the paired 95% interval was [-3.12%, +1.48%]. Raw logs, hashes, configuration, and power samples were retained. This result does not qualify small performance changes; this PR makes no production-throughput claim.
GitHub CI on the validated head: all checks pass, including Linux/macOS tests with Rust 1.98.0 and stable, Check, AddressSanitizer, ThreadSanitizer, the Required gate, and Semantic PR.