workload-replay: Confirm performance regressions with paired reruns - #38712
Open
aljoscha wants to merge 2 commits into
Open
workload-replay: Confirm performance regressions with paired reruns#38712aljoscha wants to merge 2 commits into
aljoscha wants to merge 2 commits into
Conversation
Require the same metric to exceed its existing threshold in two fresh reference/current pairs. Rerun the baseline as well so an unusually fast reference does not poison confirmation. Preserve both comparison tables and plots, and keep query errors fatal without retrying them. Closes: QAR-153
aljoscha
marked this pull request as ready for review
September 8, 2026 12:27
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Workload replay gates performance on a single reference/current pair. Short object-creation timings and variable CPU averages can cross the 20% threshold on unchanged code. A single outlier should trigger confirmation, not fail the workload outright.
Changes
Keep all existing thresholds. When a pair reports performance regressions, run one additional fresh reference/current pair with the same seed and workload settings. Fail only for metrics that exceed their thresholds in both pairs. Rerun the reference too, since an unusually fast baseline can produce a false regression just as an unusually slow current run can.
Passing comparisons and runs without a reference do not incur extra work. Query-error failures and replay exceptions are not retried. A requested reference that cannot be resolved fails explicitly rather than silently bypassing comparison.
Both comparison tables remain in the log and in confirmed failure details. Comparison plots have attempt-specific names. The cost is at most one extra pair per regressing workload, including setup and hydration. This filters one-off noise, but does not establish statistical significance or eliminate correlated measurement noise.
Coverage
Add executor-level tests using real metric comparisons and controlled replay results. They cover transient and persistent regressions, an anomalously fast baseline, mismatched metrics across attempts, query errors in either pair, no-reference execution, unresolved references, cleanup counts, and retained comparison evidence.
The full benchmark remains unverified locally because workload repository cloning stalled. Full repository checks are also blocked by missing tools and a Rust toolchain older than the workspace requires. CI confirmation is still needed.
Closes: QAR-153
Release notes
None.