Skip to content

ci(codspeed): modernize CodSpeed integration to v4 with OIDC and simulation mode - #26

Closed
codspeed[bot] wants to merge 2 commits into
masterfrom
codspeed/wizard-1780659936896
Closed

codspeed[bot] wants to merge 2 commits into
masterfrom
codspeed/wizard-1780659936896

Conversation

@codspeed

@codspeed codspeed Bot commented Jun 5, 2026

Copy link
Copy Markdown

Summary

This PR finalizes the CodSpeed continuous performance testing setup for linopy by bringing the existing workflows up to current CodSpeed standards. The repository already ships a mature pytest-based benchmark suite under benchmarks/ (730 benchmarks across build, matrix export, LP write, netCDF I/O, and solver handoff phases) with pytest-codspeed pinned in the [benchmarks] extra. The changes here focus on the CI integration itself.

Changes

  • Upgraded the CodSpeed action from v3 to v4 (latest) in both codspeed.yml and codspeed-macro.yml.
  • Switched to OIDC authentication: added permissions: id-token: write and removed the manual CODSPEED_TOKEN secret reference. OIDC relies on the CodSpeed GitHub App installation rather than a manually managed secret.
  • Set mode: simulation explicitly on the instrumentation workflow (CPU instruction-count measurement), the recommended mode for Python.
  • Removed continue-on-error so the workflow status now meaningfully reflects benchmark execution.
  • Added the CodSpeed badge to the README.

Workflows

  • codspeed.yml — instrumentation (simulation) benchmarks on every PR to master and on master pushes. Runs on ubuntu-latest, consumes no metered macro-runner minutes.
  • codspeed-macro.yml — walltime benchmarks on CodSpeed's codspeed-macro bare-metal runners, master-only plus manual dispatch (macro-runner minutes are metered). Available because the repository lives under a GitHub organization.

Validation

The benchmark suite was run locally under CodSpeed simulation. Benchmarks collect cleanly (730 collected) and a sample benchmark uploaded successfully, confirming the harness, simulation mode, and result upload to fluxopt/linopy all work end-to-end.

Next steps

  • Ensure the CodSpeed GitHub App is installed on the organization so OIDC authentication succeeds.
  • After merge, the first run on master establishes the performance baseline; subsequent PRs are compared against it automatically.

@codspeed

codspeed Bot commented Jun 5, 2026 •

Copy link
Copy Markdown
Author

Merging this PR will create unknown performance changes

🆕 174 new benchmarks
⏩ 174 skipped benchmarks1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
🆕 Simulation test_build[basic-n=10] N/A 193.6 ms N/A
🆕 Simulation test_build[cumsum-severity=0] N/A 81.8 ms N/A
🆕 Simulation test_build[cumsum-severity=25] N/A 87.2 ms N/A
🆕 Simulation test_build[cumsum-severity=50] N/A 108.4 ms N/A
🆕 Simulation test_build[expression_arithmetic-n=10] N/A 211.4 ms N/A
🆕 Simulation test_build[knapsack-n=100] N/A 50 ms N/A
🆕 Simulation test_build[kvl_cycles-severity=0] N/A 302.7 ms N/A
🆕 Simulation test_build[kvl_cycles-severity=25] N/A 301.9 ms N/A
🆕 Simulation test_build[kvl_cycles-severity=50] N/A 301.2 ms N/A
🆕 Simulation test_build[masked-n=10] N/A 107.5 ms N/A
🆕 Simulation test_build[merge_balance-severity=0] N/A 879.9 ms N/A
🆕 Simulation test_build[merge_balance-severity=25] N/A 912.9 ms N/A
🆕 Simulation test_build[merge_balance-severity=50] N/A 925.5 ms N/A
🆕 Simulation test_build[milp-n=10] N/A 156.4 ms N/A
🆕 Simulation test_build[nodal_balance-severity=0] N/A 81.3 ms N/A
🆕 Simulation test_build[nodal_balance-severity=25] N/A 87.3 ms N/A
🆕 Simulation test_build[nodal_balance-severity=50] N/A 94.7 ms N/A
🆕 Simulation test_build[piecewise-n=10] N/A 376.4 ms N/A
🆕 Simulation test_build[qp-n=10] N/A 106.7 ms N/A
🆕 Simulation test_build[rolling-severity=0] N/A 84 ms N/A
... ... ... ... ... ...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.


Comparing codspeed/wizard-1780659936896 (60f2938) with master (ec473c8)

Open in CodSpeed

Footnotes

  1. 174 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@FBumann FBumann closed this Jun 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant