fix(bench): compare like with like, and add a gurobi arm by FBumann · Pull Request #438 · fluxopt/specsolve

FBumann · 2026-07-31T18:14:46Z

Asked for a gurobi arm in bench/, since nothing would have caught the
regression #434 fixed — the sink is correctness-tested to death and had no
performance gate at all. Adding it turned up a comparability bug in the arm
that already shipped.

The arm

uv run python -m bench.run --cases dispatch --sinks gurobi

	`lp`	`highs`	`gurobi`
`lpspec`	`ex.write(...)`	`build_highs(...)`	`build_gurobi(...)`
`linopy`	`to_file(io_api='lp-polars')`	`to_highspy(set_names=False)`	`to_gurobipy(set_names=False)`

Same discipline as the highs arm: build_gurobi never calls optimize(),
to_gurobipy never returns a solved model, so neither number is really a
number about Gurobi's search. Opt-in via --sinks gurobi because it needs the
[gurobi] extra, where the other two need nothing a contributor does not
already have.

The size-limited wheel licence is not an obstacle, which I had assumed it
was: it caps optimize(), not model building, and this harness never
optimizes. The full ladder is reachable with the pip licence.

The bug it surfaced

linopy names every variable and constraint by default. Neither of our solver
sinks names anything. So to_highspy() — the call the published highs
comparison uses — was being timed doing work our arm does not do.

It is not a rounding error:

seam	`set_names=True` (default)	`set_names=False`
`to_highspy`	0.11s	0.02s
`to_gurobipy`	0.34s	0.22s

(200k variables, 200k constraints.) Naming is 82% of linopy's HiGHS
hand-off and 35% of its Gurobi one. Both seams now pass
set_names=False, so the two arms end holding the same artifact — which
bench/README.md already claimed was "the only reason the two are comparable".

What this costs us

The correction runs against lpspec, which is the direction an honest
harness should err. On dispatch s the two arms now emit in the same time
(0.10s vs 0.10s on gurobi, 0.01s vs 0.01s on highs), where the old call showed
linopy slower. Published emit figures for the highs sink will narrow when
the ladder is next regenerated.

The claim that survives is the one the harness exists for: build — 0.01s
against 0.17s on that rung. That was never the phase the naming touched.

docs/benchmarks.md is unchanged here: regenerating it needs a machine at rest
and is a separate, deliberate act.

Not included

A bench/regressions/ case for the gurobi hand-off — that suite needs its own
group and memray, and the ladder is the thing that was asked for. Worth a
follow-up if you want the gate automated rather than run.

Follows #418 and #434.

🤖 Generated with Claude Code

coderabbitai · 2026-07-31T18:14:56Z

Warning

Review limit reached

@FBumann, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 38 seconds

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 91a518db-ed22-4153-b5d0-dcede9b84f27

📥 Commits

Reviewing files that changed from the base of the PR and between 782f76b and fdc6e6c.

📒 Files selected for processing (4)

bench/README.md
bench/_run_case.py
bench/report.py
bench/run.py

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

Adds `--sinks gurobi`: lpspec through `build_gurobi`, linopy through `to_gurobipy`, neither starting the search. Opt-in, because it needs the [gurobi] extra where the other two need nothing a contributor lacks. The size-limited wheel licence is no obstacle — it caps `optimize()`, which this harness never calls. Adding it surfaced a comparability bug in the arm that already shipped. **linopy names every variable and constraint by default and neither of our solver sinks names anything**, so `to_highspy()` was being timed doing work our arm does not do — and it is not a rounding error: naming is 82% of linopy's HiGHS hand-off (0.11s against 0.02s at 200k variables) and 35% of its Gurobi one. Both seams now pass `set_names=False`, so the two arms end holding the same artifact. That correction runs against us: published `emit` numbers for the highs sink will narrow when the ladder is next regenerated, and on `dispatch s` the two arms now emit in the same time. The claim that survives is the one this harness exists for — `build`, where lpspec is 0.01s against 0.17s. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

#438 fixed a comparability bug this branch's numbers were taken across: linopy's `to_highspy()` was being timed *with* variable naming, which neither of our solver sinks does — 82% of its HiGHS hand-off. Every `highs` ratio here was therefore measured against a linopy doing work we do not do, and flatters us. #439 moved the engine again on top of that. So the tables and all three results files go back to main's, and the whole measurement is re-run from a clean tree at a commit this branch can name.

…ug made Every number on the page now comes from one run at `98f382d` on a clean tree, and the page says so — the fingerprint hash is what makes a published number traceable to the code that produced it, and the previous one recorded `782f76b-dirty`, which identifies nothing. **The `highs` comparison was not like for like, and it flattered us.** #438 found that linopy's `to_highspy()` was being timed *with* variable naming, which neither of our solver sinks does — 82% of its hand-off, and memory as well as time. Corrected: highs sink, l rung was now dispatch peak 0.87x 1.46x fleet peak 0.76x 1.10x profiled peak 0.82x 1.27x transport peak 0.75x 1.10x So the summary changes from "ahead on both, on all six" to **ahead on wall on five of six, and behind on peak on four of six through `highs`**. HiGHS's own dense model dominates both arms there and our COO triples are what is left. The `lp` sink barely moved, because that comparison was already like for like, and it is where the representation actually pays: ahead on peak on four of six. `profiled` is the case in the ladder we lose, on both sinks (1.27x, 1.07x). Said plainly rather than left to the tables. The scaling section's reversal stands and is re-measured: peak 0.68-0.78x from 10k to 120M variables, against the 1.07x this page used to conclude the advantage "runs out" from. That section now also says it is the LP sink, and therefore not evidence for the `highs` peak row.

…ug made (#441) * docs(bench): re-measure the published tables, and reverse a conclusion The committed results were taken at `ef8558c` — version `0.0.0a27`, before the package was even renamed. Eight perf changes have landed since (#408, #412, #413, #414, #415, #434, #421, #433), so every table on the page understated the engine, and one paragraph drew a conclusion the numbers no longer support. Re-run in one sitting on an idle machine: the ladder, the density sweep and the scaling run, so all three files describe the same tree. Parity gate passes on all six cases (`fleet` to 4.6e-16) before anything is timed. **What moved**, against the previously published numbers, `l` rung, our arm: wall -10% to -39% on nine of twelve rungs, peak -2% to -21% on nine. **What got worse, and is not buried.** `dispatch` build is 10-12% slower — that is `maintain_order='left'` from #433, which buys -21% peak on the same rung and is the trade that PR argued for. `fleet/highs` peak is up 8.2%, outside its 1.0% floor, and I have no explanation for it. **A conclusion is reversed.** `## Absurd sizes` read 1.07x peak at 120M variables and concluded that our memory headroom "is a small-and-sparse-model property, not a scaling one". Peak is now 0.73-0.83x across four orders of magnitude. Two things moved it and only one is us: #433 took ~20% off this case, and the earlier reading came from a run *that section itself flagged as noisy* — it recorded linopy at 38.5 s and 106.2 s for the same rung. The old number was a conclusion drawn across the noise floor. The section now says so rather than quietly presenting a better one. **`profiled` is no longer the case in the ladder we lose** (1.09x -> 0.74x through the highs sink), and `fleet/lp` has crossed on wall (1.03x -> 0.90x). Both paragraphs are rewritten to say what changed rather than to delete the sentences that admitted the losses. Two numbers still go against us and are still named: `profiled` 1.31x on wall through the LP sink, `fleet` 1.47x on peak. Not touched: `## Sink capabilities` still says Gurobi's row wants a spike before it is relied on. #418 shipped a real Gurobi sink since, so that caveat is probably obsolete — but exercising it is not this change, and asserting capabilities I have not run is how that table would come to lie. * docs(bench): re-measure at 98f382d, and correct a claim the harness bug made Every number on the page now comes from one run at `98f382d` on a clean tree, and the page says so — the fingerprint hash is what makes a published number traceable to the code that produced it, and the previous one recorded `782f76b-dirty`, which identifies nothing. **The `highs` comparison was not like for like, and it flattered us.** #438 found that linopy's `to_highspy()` was being timed *with* variable naming, which neither of our solver sinks does — 82% of its hand-off, and memory as well as time. Corrected: highs sink, l rung was now dispatch peak 0.87x 1.46x fleet peak 0.76x 1.10x profiled peak 0.82x 1.27x transport peak 0.75x 1.10x So the summary changes from "ahead on both, on all six" to **ahead on wall on five of six, and behind on peak on four of six through `highs`**. HiGHS's own dense model dominates both arms there and our COO triples are what is left. The `lp` sink barely moved, because that comparison was already like for like, and it is where the representation actually pays: ahead on peak on four of six. `profiled` is the case in the ladder we lose, on both sinks (1.27x, 1.07x). Said plainly rather than left to the tables. The scaling section's reversal stands and is re-measured: peak 0.68-0.78x from 10k to 120M variables, against the 1.07x this page used to conclude the advantage "runs out" from. That section now also says it is the LP sink, and therefore not evidence for the `highs` peak row.

FBumann added perf Wall time, memory, or I/O cost — not a wrong answer area:solver Solver sink, options, statuses, duals, value read-back labels Jul 31, 2026

FBumann force-pushed the bench/gurobi-arm branch from c9f48bc to d03f7ba Compare July 31, 2026 18:16

FBumann force-pushed the bench/gurobi-arm branch from d03f7ba to fdc6e6c Compare July 31, 2026 18:16

FBumann changed the title ~~bench: a gurobi arm, and the naming that was flattering us~~ fix(bench): compare like with like, and add a gurobi arm Jul 31, 2026

FBumann merged commit ea15cb4 into main Jul 31, 2026
4 of 5 checks passed

fluxopt-release Bot mentioned this pull request Jul 31, 2026

chore(main): release 0.0.1-alpha.56 #440

Merged

FBumann mentioned this pull request Jul 31, 2026

docs(bench): re-measure at 98f382d, and correct a claim the harness bug made #441

Merged

FBumann mentioned this pull request Jul 31, 2026

docs(bench): plot the results, and fold the tables under them #451

Merged

Navigation Menu

Repository navigation

fix(bench): compare like with like, and add a gurobi arm - #438

fix(bench): compare like with like, and add a gurobi arm#438
FBumann merged 1 commit into
mainfrom
bench/gurobi-arm

FBumann commented Jul 31, 2026

Uh oh!

coderabbitai Bot commented Jul 31, 2026 •

edited

Loading

Review limit reached

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

FBumann commented Jul 31, 2026

The arm

The bug it surfaced

What this costs us

Not included

Uh oh!

coderabbitai Bot commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review limit reached

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

coderabbitai Bot commented Jul 31, 2026 •

edited

Loading