Repository navigation
ci(codspeed): a merge that cannot change the measurement is not measured - #577
Merged
Merged
Conversation
A release commit is a strict duplicate. release-please writes CHANGELOG.md and .release-please-manifest.json and touches no source -- the git tag is the version -- so the metered macro runner reproduces the previous commit's number at full price. Over the last 40 merges that is 17 of them, 43%, every one a release. The rule is what cannot change what the benchmark executes, which is narrower than "not code" and deliberately excludes entries that look safe: pyproject.toml and uv.lock decide which polars is installed and a floor moving under us is what this series exists to catch, bench/ is the workload, and this file is the instrument. Checked against the last 40 merges: nothing touching src/, bench/, pyproject.toml or uv.lock is skipped. The concurrency note calls a missing number a hole the next pull request falls into, and it is right in general. It does not apply to these paths: the hole sits at a commit whose executable tree is identical to its parent, so the fallback compares against the same code it would have. The cost is that a pull request merged after one is footnoted `unexpected-base`, which reads like a fault -- it misled this session for one round trip on #520 -- so the comment says so where somebody will find it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
FBumann
enabled auto-merge (squash)
August 11, 2026 12:30
Merging this PR will degrade performance by 3.09%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | Memory | test_emit[profiled-s-lpspec-lp] |
22.8 MB | 26.2 MB | -12.9% |
| ⚡ | Memory | test_emit[sector-s-lpspec-highs] |
4.5 MB | 4.1 MB | +7.84% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing ci/codspeed-skips-the-release-commit (81db3f4) with main (cf794c7)
Footnotes
-
38 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
FBumann
added a commit
that referenced
this pull request
Aug 11, 2026
#577 left the ignore list one obvious step away from a silent hole. The step is `**.yaml`, and it would stop measuring `bench/models/*.yaml` -- the benchmarked models themselves -- along with `examples/**`, `tests/golden/model.yaml` and this workflow. Counted: an extension glob over yaml/yml/json silences 48 files that reach a build or define the instrument. An extension cannot distinguish a tool's configuration from a program's input, so the inert config files are named one at a time and the rule is written where the next person to tidy the list will read it. The five paths added are worth almost nothing on their own -- over the last 100 merges they skip one more than #577's list already did. They are here to make the rule concrete rather than to save a run, and the commit says so instead of implying a saving it does not deliver. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Skips the metered macro runner on merges that cannot change what it measures.
Why
A release commit is a strict duplicate.
release-pleasewritesCHANGELOG.mdand.release-please-manifest.jsonand touches no source — the git tag is the version — so CodSpeed spends a metered macro run reproducing the previous commit's number.Measured against the last 40 merges to
main: 17 would be skipped, 43%, and every one of them is a release commit.The rule, and what it deliberately excludes
Only what cannot change what the benchmark executes — narrower than "not code", and it keeps out three things that look obviously safe:
pyproject.tomlanduv.lockdecide which polars gets installed, and a floor moving under us is exactly what this series is forbench/is the workloadcodspeed.ymlis the instrumentThe failure mode to watch is not the release commit; it is a path added to this list later that turns out to reach the build, which then hides a regression in silence. The comment says so at the list.
paths-ignoreskips only when every changed file matches, and merges here are squashes, so a pull request touching docs andsrc/together is still measured.The
docs/**and**.mdentries are principle rather than measurement: no docs-only merge appears in the last 40, so they save nothing yet and are included because they satisfy the same rule.The cost, named
A skipped commit has no number, which the
concurrencynote calls a hole the next pull request falls into. That note is right in general and does not apply here: the hole sits at a commit whose executable tree is identical to its parent, so the fallback compares against the same code it would have.What it does cost is legible — a pull request merged after one is footnoted
unexpected-base. That footnote misled this session for a round trip on #520: it reads as "the baseline is stale, the comparison is unsafe" when the truth was "the baseline is the same code under a different sha". Hence the comment where somebody hitting it will look.🤖 Generated with Claude Code