Repository navigation
ci(codspeed): config is listed by path, never by extension - #578
Merged
Merged
Conversation
#577 left the ignore list one obvious step away from a silent hole. The step is `**.yaml`, and it would stop measuring `bench/models/*.yaml` -- the benchmarked models themselves -- along with `examples/**`, `tests/golden/model.yaml` and this workflow. Counted: an extension glob over yaml/yml/json silences 48 files that reach a build or define the instrument. An extension cannot distinguish a tool's configuration from a program's input, so the inert config files are named one at a time and the rule is written where the next person to tidy the list will read it. The five paths added are worth almost nothing on their own -- over the last 100 merges they skip one more than #577's list already did. They are here to make the rule concrete rather than to save a run, and the commit says so instead of implying a saving it does not deliver. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Merging this PR will regress 1 benchmark
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | Memory | test_emit[sector-s-lpspec-highs] |
4.3 MB | 4.9 MB | -12.15% |
| ⚡ | Memory | test_emit[transport-s-lpspec-lp] |
20.9 MB | 17.7 MB | +18.16% |
| ⚡ | Memory | test_emit[profiled-s-lpspec-lp] |
24.4 MB | 22.4 MB | +8.72% |
| ⚡ | Memory | test_emit[sector-s-lpspec-lp] |
5.6 MB | 5.2 MB | +7.2% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing ci/config-by-path-not-by-extension (b7e876c) with main (b71ef49)
Footnotes
-
38 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
FBumann
added a commit
that referenced
this pull request
Aug 11, 2026
…race The build is deterministic given a pool size: repeated runs produce identical chunk counts and identical bytes. Peak is not, because peak is max simultaneous liveness and whether two morsels are alive at the same instant is scheduling. Measured on transport/s, eight runs per arm: spread 1.5% at one thread, 3.2% at four, 3.6% at eight. The level moves further than the spread. Per-thread buffers dwarf a small model -- 176 MB, 194 MB, 216 MB at 1, 4 and 8 threads for a model whose frames total 4.27 MB -- and chunk count rises with the pool for the same data (sector/s: 4, 7, 13, 25 chunks at 1, 2, 4, 8, all at 590,000 bytes). memray counts allocations, so an unpinned pool measures the runner as much as the code. That is the s-rung instability seen on #578, a pull request whose entire diff is a YAML comment and which moved four benchmarks across a 30-point range. Only the memory job. The walltime job stays multi-threaded: polars is multi-threaded native code, single-threading it would measure something nobody runs, and its macro runner is dedicated hardware whose pool is already stable. PYTHONHASHSEED is not pinned beside it. Five seeds produced byte-identical chunk counts and sizes, so it would be a knob added on plausibility. This puts a step in the series at this commit; points before and after are not comparable and nothing makes them so. The trade is deliberate -- this job exists to make a series readable, and bench.yml with docs/benchmarks.md own the honest multi-threaded peak measured as RSS. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
FBumann
added a commit
that referenced
this pull request
Aug 11, 2026
…race (#579) The build is deterministic given a pool size: repeated runs produce identical chunk counts and identical bytes. Peak is not, because peak is max simultaneous liveness and whether two morsels are alive at the same instant is scheduling. Measured on transport/s, eight runs per arm: spread 1.5% at one thread, 3.2% at four, 3.6% at eight. The level moves further than the spread. Per-thread buffers dwarf a small model -- 176 MB, 194 MB, 216 MB at 1, 4 and 8 threads for a model whose frames total 4.27 MB -- and chunk count rises with the pool for the same data (sector/s: 4, 7, 13, 25 chunks at 1, 2, 4, 8, all at 590,000 bytes). memray counts allocations, so an unpinned pool measures the runner as much as the code. That is the s-rung instability seen on #578, a pull request whose entire diff is a YAML comment and which moved four benchmarks across a 30-point range. Only the memory job. The walltime job stays multi-threaded: polars is multi-threaded native code, single-threading it would measure something nobody runs, and its macro runner is dedicated hardware whose pool is already stable. PYTHONHASHSEED is not pinned beside it. Five seeds produced byte-identical chunk counts and sizes, so it would be a knob added on plausibility. This puts a step in the series at this commit; points before and after are not comparable and nothing makes them so. The trade is deliberate -- this job exists to make a series readable, and bench.yml with docs/benchmarks.md own the honest multi-threaded peak measured as RSS. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #577. The five added paths are nearly worthless; the comment is the point. Stating that up front so the diff is not read as a saving it does not deliver.
The hole this closes
#577's list ends with
LICENSEand.gitignore, and the obvious next tidy-up is**.yaml— it looks exactly as inert as**.md. It is not. Counted against the tree:bench/models/*.yamlare the benchmarked models. A glob that reads as housekeeping would stop measuring a change todispatch.yaml, and stop measuring changes to the instrument that measures it.examples/**andtests/golden/model.yamlare models too.**.mdis safe for the opposite reason: no markdown in this tree reaches a build. The only files it matches undersrc/,bench/,examples/andtests/are five READMEs.So the rule that survives is an extension cannot distinguish a tool's configuration from a program's input, and inert config gets named one line at a time.
What the paths buy
One.
mkdocs.yml,.readthedocs.yaml,.pre-commit-config.yaml,.github/dependabot.ymland.release-please-config.jsonare added to make the rule concrete — a named example of what qualifies — not to save runs. #577 already took the 43%.Verified
The guard that matters, replayed over every tracked file under
src/,bench/,examples/,tests/, pluspyproject.toml,uv.lock,codspeed.ymlandbench.yml— 178 files: nothing that reaches a build or defines the instrument is ignored, the five READMEs aside.🤖 Generated with Claude Code