Skip to content

test: qualify benchmark timing and paired comparisons - #67

Merged
leiysky merged 3 commits into
mainfrom
dev/benchmark-noise-control
Oct 27, 2026
Merged

leiysky merged 3 commits into
mainfrom
dev/benchmark-noise-control

Conversation

@leiysky

@leiysky leiysky commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Workers could begin before timing started, and soak payload bandwidth depended on optional library counters. Start timers before releasing workers, count accepted and returned payload bytes in the harness, and separate workload duration from the final drain. Add an optional repeated put_steady phase with persistent writers, fixed warmup and measured rounds, bounded work chunks, and a drain after each batch; retain the cold-write result.

Use per-worker soak observers, support disabled library statistics without losing payload validation, sample macOS RSS through the shared harness, and drop the closed cache before reopening to avoid overlapping cache instances in memory measurements.

Add a serial ABBA/BAAB comparison tool that retains raw logs, configuration, binary hashes, and power samples. Report paired bootstrap intervals and reject inadequate duration, unstable power, excessive block drift, or excessive spread between same-binary repeats. The repeat check fixes a reproduced case where alternating 1000/500 ops/s A/A runs previously passed because pairing canceled all noise. It uses the existing drift threshold and has regression coverage for both A/A and A/B. Integrate the tool tests into cargo x test and document the workflow.

Validation

Synchronized with main at 3d29b6e, including #66. Validated PR head 6acc5fc:

  • cargo x check
  • cargo x test: 369 Rust tests plus 9 Python comparison-tool tests
  • cargo x lint: formatting, code, documentation, packaging, and dependency policy checks pass, including Tokio 1.54.1 advisories
  • cargo test --workspace --release --all-features: 367 tests pass
  • Release benchmark builds and 13 serial smoke scenarios: cold/read/recovery, 16 KiB with 8 writers and partial work chunks, idle workers, statistics on/off, Disabled/Adaptive mixed soak, read-only warm reopening, unpaced mixed load, and rejected unobservable coverage settings
  • Verified exact operation/payload-byte accounting, admission-plus-drain timing, application bytes against enabled library counters, nonzero macOS RSS, and close/drop/open ordering
  • Restoring the original noise gate causes the new alternating-noise regression to fail for both A/A and A/B

A fresh byte-identical A/A control used disabled library statistics, eight blocks, all 32 samples, and five-second unpaced mixed workloads with unchanged default thresholds. The tool correctly rejected this host's run: effective-hit block drift was 12.99% and within-block spread was 8.16% against the 5% limit; the paired 95% interval was [-3.12%, +1.48%]. Raw logs, hashes, configuration, and power samples were retained. This result does not qualify small performance changes; this PR makes no production-throughput claim.

GitHub CI on the validated head: all checks pass, including Linux/macOS tests with Rust 1.98.0 and stable, Check, AddressSanitizer, ThreadSanitizer, the Required gate, and Semantic PR.

@leiysky
leiysky merged commit e9ec2bb into main Oct 27, 2026
8 checks passed
@leiysky
leiysky deleted the dev/benchmark-noise-control branch October 27, 2026 16:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant