From 81db3f4868558d34a99a4c20a9bb65b23274d03a Mon Sep 17 00:00:00 2001 From: FBumann <117816358+FBumann@users.noreply.github.com> Date: Tue, 11 Aug 2026 14:29:01 +0200 Subject: [PATCH] ci(codspeed): a merge that cannot change the measurement is not measured A release commit is a strict duplicate. release-please writes CHANGELOG.md and .release-please-manifest.json and touches no source -- the git tag is the version -- so the metered macro runner reproduces the previous commit's number at full price. Over the last 40 merges that is 17 of them, 43%, every one a release. The rule is what cannot change what the benchmark executes, which is narrower than "not code" and deliberately excludes entries that look safe: pyproject.toml and uv.lock decide which polars is installed and a floor moving under us is what this series exists to catch, bench/ is the workload, and this file is the instrument. Checked against the last 40 merges: nothing touching src/, bench/, pyproject.toml or uv.lock is skipped. The concurrency note calls a missing number a hole the next pull request falls into, and it is right in general. It does not apply to these paths: the hole sits at a commit whose executable tree is identical to its parent, so the fallback compares against the same code it would have. The cost is that a pull request merged after one is footnoted `unexpected-base`, which reads like a fault -- it misled this session for one round trip on #520 -- so the comment says so where somebody will find it. Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/codspeed.yml | 39 ++++++++++++++++++++++++++++++---- 1 file changed, 35 insertions(+), 4 deletions(-) diff --git a/.github/workflows/codspeed.yml b/.github/workflows/codspeed.yml index 2222f3a47..7f840264f 100644 --- a/.github/workflows/codspeed.yml +++ b/.github/workflows/codspeed.yml @@ -46,10 +46,11 @@ name: CodSpeed # is indistinguishable from that one — so check there first, before reading # anything here as broken. # -# That machine is metered, which is the whole of why this is fenced: `main` -# always, a pull request only when somebody applies `trigger:bench` — the label -# bench.yml is already driven by. Bare metal should not run fork code unasked -# either. +# That machine is metered, which is the whole of why this is fenced: every merge +# to `main` that changes code, a pull request only when somebody applies +# `trigger:bench` — the label bench.yml is already driven by. Bare metal should +# not run fork code unasked either. The one merge that does not qualify is the +# release commit; see `paths-ignore` under `on:`. # # What it buys is the signal the tree has nowhere else. bench.yml gates on # memory, this job's other half measures memory, and the wall-clock ladder is a @@ -89,6 +90,36 @@ name: CodSpeed on: push: branches: [main] + # **Only what cannot change what the benchmark executes.** A merge that + # leaves the built model, the workload and the installed versions identical + # would spend a metered macro run reproducing the previous commit's number. + # The release commit is the standing case — release-please writes CHANGELOG + # and manifest and touches no source, the git tag being the version — and at + # the alpha cadence those are a large share of merges. + # + # That principle is the whole of the list, and it excludes some entries that + # look obviously safe. `pyproject.toml` and `uv.lock` decide which polars is + # installed, and a floor moving under us is precisely what this series is for. + # `bench/` is the workload. This file is the instrument. None of the three + # may be added here however documentation-shaped a particular diff to them is. + # + # `paths-ignore` skips only when *every* changed file matches, and merges here + # are squashes, so a pull request touching docs and `src/` together is still + # measured. The failure mode to watch is the reverse: a path added to this + # list that turns out to reach the build hides a regression in silence. + # + # A skipped commit has no number, which the concurrency note below calls a + # hole the next pull request falls into. It is not one for these paths: the + # hole sits at a commit whose executable tree is identical to its parent, so + # the fallback compares against the same code it would have. The cost is + # cosmetic and worth naming because it reads like a fault — a pull request + # merged after one of these is footnoted `unexpected-base`. + paths-ignore: + - '**.md' + - docs/** + - .release-please-manifest.json + - LICENSE + - .gitignore # required for a report to have somewhere to land. `labeled` is on the list # because the default triple stops at `reopened`, and without it applying # `trigger:bench` to an open pull request would re-trigger nothing — the