Skip to content

ci(codspeed): a merge that cannot change the measurement is not measured - #577

Merged
FBumann merged 1 commit into
mainfrom
ci/codspeed-skips-the-release-commit
Aug 11, 2026
Merged

FBumann merged 1 commit into
mainfrom
ci/codspeed-skips-the-release-commit

Conversation

@FBumann

@FBumann FBumann commented Aug 11, 2026

Copy link
Copy Markdown
Collaborator

Skips the metered macro runner on merges that cannot change what it measures.

Why

A release commit is a strict duplicate. release-please writes CHANGELOG.md and .release-please-manifest.json and touches no source — the git tag is the version — so CodSpeed spends a metered macro run reproducing the previous commit's number.

Measured against the last 40 merges to main: 17 would be skipped, 43%, and every one of them is a release commit.

SKIP  cf794c7  chore(main): release 0.0.1-alpha.89 (#572)
SKIP  839a50c  chore(main): release 0.0.1-alpha.88 (#569)
SKIP  15c9776  chore(main): release 0.0.1-alpha.87 (#567)
...
of the last 40 merges: 17 skipped, 23 measured
commits that would be skipped despite touching src/bench/pyproject/uv.lock: NONE

The rule, and what it deliberately excludes

Only what cannot change what the benchmark executes — narrower than "not code", and it keeps out three things that look obviously safe:

  • pyproject.toml and uv.lock decide which polars gets installed, and a floor moving under us is exactly what this series is for
  • bench/ is the workload
  • codspeed.yml is the instrument

The failure mode to watch is not the release commit; it is a path added to this list later that turns out to reach the build, which then hides a regression in silence. The comment says so at the list.

paths-ignore skips only when every changed file matches, and merges here are squashes, so a pull request touching docs and src/ together is still measured.

The docs/** and **.md entries are principle rather than measurement: no docs-only merge appears in the last 40, so they save nothing yet and are included because they satisfy the same rule.

The cost, named

A skipped commit has no number, which the concurrency note calls a hole the next pull request falls into. That note is right in general and does not apply here: the hole sits at a commit whose executable tree is identical to its parent, so the fallback compares against the same code it would have.

What it does cost is legible — a pull request merged after one is footnoted unexpected-base. That footnote misled this session for a round trip on #520: it reads as "the baseline is stale, the comparison is unsafe" when the truth was "the baseline is the same code under a different sha". Hence the comment where somebody hitting it will look.

🤖 Generated with Claude Code

A release commit is a strict duplicate. release-please writes CHANGELOG.md and
.release-please-manifest.json and touches no source -- the git tag is the
version -- so the metered macro runner reproduces the previous commit's number
at full price. Over the last 40 merges that is 17 of them, 43%, every one a
release.

The rule is what cannot change what the benchmark executes, which is narrower
than "not code" and deliberately excludes entries that look safe: pyproject.toml
and uv.lock decide which polars is installed and a floor moving under us is what
this series exists to catch, bench/ is the workload, and this file is the
instrument. Checked against the last 40 merges: nothing touching src/, bench/,
pyproject.toml or uv.lock is skipped.

The concurrency note calls a missing number a hole the next pull request falls
into, and it is right in general. It does not apply to these paths: the hole
sits at a commit whose executable tree is identical to its parent, so the
fallback compares against the same code it would have. The cost is that a pull
request merged after one is footnoted `unexpected-base`, which reads like a
fault -- it misled this session for one round trip on #520 -- so the comment
says so where somebody will find it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@FBumann
FBumann enabled auto-merge (squash) August 11, 2026 12:30
@codspeed

codspeed Bot commented Aug 11, 2026

Copy link
Copy Markdown

Merging this PR will degrade performance by 3.09%

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 1 improved benchmark
❌ 1 regressed benchmark
✅ 12 untouched benchmarks
⏩ 38 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ Memory test_emit[profiled-s-lpspec-lp] 22.8 MB 26.2 MB -12.9%
⚡ Memory test_emit[sector-s-lpspec-highs] 4.5 MB 4.1 MB +7.84%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing ci/codspeed-skips-the-release-commit (81db3f4) with main (cf794c7)

Open in CodSpeed

Footnotes

  1. 38 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@FBumann
FBumann merged commit b71ef49 into main Aug 11, 2026
6 checks passed
FBumann added a commit that referenced this pull request Aug 11, 2026
#577 left the ignore list one obvious step away from a silent hole. The step is
`**.yaml`, and it would stop measuring `bench/models/*.yaml` -- the benchmarked
models themselves -- along with `examples/**`, `tests/golden/model.yaml` and
this workflow. Counted: an extension glob over yaml/yml/json silences 48 files
that reach a build or define the instrument.

An extension cannot distinguish a tool's configuration from a program's input,
so the inert config files are named one at a time and the rule is written where
the next person to tidy the list will read it.

The five paths added are worth almost nothing on their own -- over the last 100
merges they skip one more than #577's list already did. They are here to make
the rule concrete rather than to save a run, and the commit says so instead of
implying a saving it does not deliver.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant