[FBumann]

profile_phases (#497) answers where the time goes. This answers did my change do anything, or is the machine noisy — and it was meant to be part of #497, but landed on that branch after it had already merged, so it needs a PR of its own.

Why

Every wrong perf number produced today came from reading two minima and calling the difference a result:

• a "16% sort improvement" that was a volume at 98% full
• two arms that were the same code — git stash push -- <path> stashes nothing when the change is already committed
• a real 24% win declared unresolvable and reverted
• another understated by half

What it does

uv run python -m bench.ab record main    --case profiled --size l
...switch the code...
uv run python -m bench.ab record aligned --case profiled --size l --append
uv run python -m bench.ab compare main aligned

• A verdict, not a ratio. Bootstraps a confidence interval on the difference of the minima — resampling, because the statistic is a minimum and the distribution has a long right tail — and prints NOT RESOLVABLE when it straddles zero.
• Interleaving. --append lets arms alternate, so both meet the same drift. Block ordering is the layout most vulnerable to a machine that changes, and the one you get by default.
• A fingerprint of what ran. Each arm hashes src/; compare refuses two arms that hash alike.
• Preconditions. Free disk and idle CPU, before the first round rather than after the last.
• --phase build|emit, because writing 1 GB measures the volume, not the writer.

It earned itself three times

#511 was hand-measured at −11%; interleaved with an interval it is −19.7% (95%: −20.2 .. −17.5), and the scatter I had reverted as unmeasurable came back as −24.2%.

#517: a code comment claiming a 10% penalty turned out to be +2.6%, 95%: −10.1% .. +8.9% — not resolvable.

And it caught a comparison of nothing: after #492 merged, "main's lp_file" and "the branch's" were the same bytes, and the two arms had already produced 927 ms against 1074 ms — a 16% apparent difference from identical code, the exact magnitude of everything being chased.

Samples land in bench/results/ab/ and are gitignored: they describe one laptop for one afternoon.

824 passed · ruff clean.

[codspeed]

Merging this PR will degrade performance by 5.58%

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 1 improved benchmark
❌ 4 regressed benchmarks
✅ 9 untouched benchmarks
⏩ 24 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	Memory	test_emit[sector-s-lpspec-highs]	6.5 MB	7.3 MB	-11.05%
❌	Memory	test_emit[sector-s-lpspec-lp]	6.7 MB	7.4 MB	-9.7%
❌	Memory	test_emit[profiled-s-lpspec-highs]	32.4 MB	35.1 MB	-7.62%
❌	Memory	test_emit[dispatch-s-lpspec-lp]	36.3 MB	38.6 MB	-5.91%
⚡	Memory	test_emit[profiled-s-lpspec-lp]	34.7 MB	32.3 MB	+7.51%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing bench/ab-tool (60f8087) with main (5ca09f0)²}

Footnotes

1. 24 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

2. No successful run was found on main (e3eff99) during the generation of this report, so 5ca09f0 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[FBumann]

Not an issue for now