Repository navigation

bench: internal performance benchmark suite + CodSpeed CI - #781

Merged

FBumann merged 127 commits into

PyPSA:masterfrom

fluxopt:bench/strip-to-codspeed

Jun 23, 2026

Merged

bench: internal performance benchmark suite + CodSpeed CI#781
FBumann merged 127 commits into
PyPSA:masterfrom
fluxopt:bench/strip-to-codspeed

Commits on May 27, 2026
- docs: update benchmark readme
  FBumann
  committed
  8c908af
Commits on May 28, 2026
- benchmarks: reusable model registry, new model types, new phases, CI smoke
  
  FBumann
  and
  claude
  committed
  413f1c6
- benchmarks: add --long flag, gate super-long sizes by default
  
  FBumann
  and
  claude
  committed
  a6cc83b
- benchmarks: make --quick truly quick (35s → 18s)
  
  FBumann
  and
  claude
  committed
  300abb5
- benchmarks: add registry-usage notebook + execute in CI
  
  FBumann
  and
  claude
  committed
  c725c68
- benchmarks: switch walkthrough to .ipynb, add reprs to ModelSpec
  
  FBumann
  and
  claude
  committed
  99483f8
- benchmarks: typer-based CLI as the single entry point
  
  FBumann
  and
  claude
  committed
  751aa78
- benchmarks: pin typer==0.26.2, use ctx.args for pytest pass-through
  
  FBumann
  and
  claude
  committed
  8b124e2
- benchmarks: pin test infra + add transitive lockfile
  
  FBumann
  and
  claude
  committed
  86fd036
- benchmarks: add ``sweep`` subcommand for cross-version perf runs
  
  FBumann
  and
  claude
  committed
  9be18e1
- benchmarks: collapse README to a pointer, kill duplication
  
  FBumann
  and
  claude
  committed
  51f418d
- benchmarks: make pypsa optional, expand notebook into proper guide
  
  FBumann
  and
  claude
  committed
  c0f3fee
- benchmarks: sweep gains --phase / --model / --filter + pytest pass-through
  
  FBumann
  and
  claude
  committed
  0522a75
- benchmarks: add ``compare`` subcommand wrapping pytest-benchmark compare
  
  FBumann
  and
  claude
  committed
  7bb464e
- benchmarks: compare lists snapshots as relative paths (easier to copy-paste)
  
  FBumann
  and
  claude
  committed
  83bdeda
- benchmarks: tighter defaults for ``compare`` (median/iqr, sorted by name)
  
  FBumann
  and
  claude
  committed
  8e378b5
- benchmarks: compare gains ``--group-by=fullname`` default + ctx.args arg split
  
  FBumann
  and
  claude
  committed
  f67721b
- benchmarks: revert compare to manual arg-split + acknowledge typer wart
  
  FBumann
  and
  claude
  committed
  3ac333b
- benchmarks: add ``plot`` subcommand (compare / sweep / scaling views)
  
  FBumann
  and
  claude
  committed
  919e061
- benchmarks: move plotting to benchmarks/plotting.py + text_auto + hover absolutes
  
  FBumann
  and
  claude
  committed
  c921b78
- benchmarks: switch primary metric to ``min``, allow ``--metric`` override
  
  FBumann
  and
  claude
  committed
  4c6f328
- benchmarks: plot compare sorts/bars by absolute time delta by default
  
  FBumann
  and
  claude
  committed
  d703cb1
- benchmarks: add ``scatter`` plot view for two-snapshot exploration
  
  FBumann
  and
  claude
  committed
  69693c0
- benchmarks: scatter view handles N snapshots via plotly animation_frame
  
  FBumann
  and
  claude
  committed
  2f08aa6
- benchmarks: scatter — include baseline as frame 0, clip colour to p95 |Δ|
  
  FBumann
  and
  claude
  committed
  321d2d9
- benchmarks: scatter — center y-axis symmetrically around 1.0
  
  FBumann
  and
  claude
  committed
  a0d4b7a
- benchmarks: address review — row height, scaling-from-params, mismatch warn
  
  FBumann
  and
  claude
  committed
  45700e7
- benchmarks: plot returns Figure, default output → .benchmarks/plots/<view>.html
  
  FBumann
  and
  claude
  committed
  ad7aa53
- benchmarks: plot renderers return (Figure, n_tests) — drop trace introspection
  
  FBumann
  and
  claude
  committed
  7c7bab2
- benchmarks: notebook plot demo uses the CLI + tqdm progress
  
  FBumann
  and
  claude
  committed
  6a8a16d
- benchmarks: notebook plot demo accepts the full CLI command string
  
  FBumann
  and
  claude
  committed
  09dad9d
- benchmarks: memory tracks all phases via memray.Tracker; README accuracy fixes
  
  FBumann
  and
  claude
  committed
  2ece2c1
- benchmarks: plot subcommand auto-detects memory snapshots alongside timing
  
  FBumann
  and
  claude
  committed
  ea4bc76
- benchmarks: compare view drops unchanged tests (esp. memory)
  
  FBumann
  and
  claude
  committed
  cccd476
- benchmarks: fix compare y-axis collision; revert unchanged-row filter
  
  FBumann
  and
  claude
  committed
  d34824a
- benchmarks: compare view renders value text outside bars
  
  FBumann
  and
  claude
  committed
  d88f235
- benchmarks: compare bars keep alphabetical test-id order
  
  FBumann
  and
  claude
  committed
  abb3f14
- benchmarks: plot gains ``--facets {phase,model}`` for compare + scatter
  
  FBumann
  and
  claude
  committed
  914efbf
- benchmarks: faceted compare/scatter share one x + y axis label
  
  FBumann
  and
  claude
  committed
  eb687f1
- benchmarks: notebook showcases ``--facets phase`` after compare/scatter
  
  FBumann
  and
  claude
  committed
  5a08e79
- benchmarks: faceted compare — per-facet rows, shared y-tick labels per row
  
  FBumann
  and
  claude
  committed
  e24451a
- benchmarks: scatter as default compare view + expose load_long_df
  
  FBumann
  and
  claude
  committed
  f4917dd
- benchmarks: memory sweep + --rounds/--repeats overrides + centralized pins
  
  FBumann
  and
  claude
  committed
  2993b95
- benchmarks: CodSpeed CI + Dependabot perf attribution loop
  
  FBumann
  and
  claude
  committed
  ac1df53
- benchmarks: drop lockfile, relocate walkthrough, Jupytext --build flow
  
  FBumann
  and
  claude
  committed
  0e6ec41
- benchmarks: add CLI walkthrough as Jupytext MyST notebook
  
  FBumann
  and
  claude
  committed
  3981cad
- benchmarks: bump pinned jupytext to 1.19.3 (matches installed)
  
  FBumann
  and
  claude
  committed
  59eadb3
- benchmarks: sweep --smoke for cross-version sanity checks
  
  FBumann
  and
  claude
  committed
  cbf517a
- benchmarks: small cleanups (dead __iter__, naming, stale comments)
  
  FBumann
  and
  claude
  committed
  4ba6fb4
- benchmarks: delete unused SOLVER_BUILD phase + collapse models re-exports
  
  FBumann
  and
  claude
  committed
  d86b111
- benchmarks: share phase verbs via benchmarks/phases.py + guard the seam
  
  FBumann
  and
  claude
  committed
  b153239
- benchmarks: extract _provision_venvs helper to dedupe sweep plumbing
  
  FBumann
  and
  claude
  committed
  754e0ec
Commits on May 29, 2026
- benchmarks: bump pinned numpy 1.26.4 → 2.4.6
  
  FBumann
  and
  claude
  committed
  7d3e474
- benchmarks: relax numpy pin to <2.0 for wider sweep coverage
  
  FBumann
  and
  claude
  committed
  2621a7b
- benchmarks: pin numpy back to ==1.26.4 (last 1.x)
  
  FBumann
  and
  claude
  committed
  2656178
- benchmarks: fix sweep silently measuring dev linopy + getattr SOLVER_HANDOFFS
  
  FBumann
  and
  claude
  committed
  e7f9c5b
- benchmarks: pin xarray to 2025.1.2 to extend sweep coverage to 0.4.4
  
  FBumann
  and
  claude
  committed
  b35fafe
- benchmarks: shim write_lp for linopy <0.4.1, extending sweep floor to 0.2.0
  
  FBumann
  and
  claude
  committed
  11f56d2
- benchmarks: add --as-of <DATE> for cross-time-reproducible sweeps
  
  FBumann
  and
  claude
  committed
  3091c64
- benchmarks: harden the sweep isolation seam (preflight + no bytecode)
  
  FBumann
  and
  claude
  committed
  e74ae1e
- benchmarks: copy harness into sweep venvs instead of symlinking
  
  FBumann
  and
  claude
  committed
  55612f5
- benchmarks: add ad-hoc `bench` helper for arbitrary callables
  
  FBumann
  and
  claude
  committed
  c031153
- benchmarks: make the suite mypy-clean
  
  FBumann
  and
  claude
  committed
  3df647c
- benchmarks: extract snapshot.py + calibrate bench.time
  
  FBumann
  and
  claude
  committed
  c5f23ec
- benchmarks: split sweep orchestration out of cli.py
  
  FBumann
  and
  claude
  committed
  2839145
- benchmarks: drop the "Other CLI surfaces" table from the walkthrough
  
  FBumann
  and
  claude
  committed
  4502fed
- benchmarks: show load_long_df from-file diff in the walkthrough
  
  FBumann
  and
  claude
  committed
  99f4f56
- benchmarks: label sweep snapshots by ref/sha for git/file specs
  
  FBumann
  and
  claude
  committed
  927750f
Commits on Jun 4, 2026
- feat(benchmarks): add user-pattern specs swept over a severity axis
  
  FBumann
  and
  claude
  committed
  ee8d89a
- feat(benchmarks): end-to-end pipeline memory + CLI/docs for patterns
  
  FBumann
  and
  claude
  committed
  cfcd4b2
- refactor(benchmarks): one spec selector (--filter) + spec_param_id helper
  
  FBumann
  and
  claude
  committed
  9063ec8
- fix(benchmarks): make the suite mypy-clean + repair the plot dependency guard
  
  FBumann
  and
  claude
  committed
  8a4a8a6
- feat(benchmarks): add merge_balance and flow_sum patterns
  
  FBumann
  and
  claude
  committed
  abe6329
- feat(benchmarks): add storage model + rolling pattern (intertemporal coupling)
  
  FBumann
  and
  claude
  committed
  f2e63a2
Commits on Jun 5, 2026
- feat(benchmarks): add cumsum pattern
  
  FBumann
  and
  claude
  committed
  816a5ce
- ci(mypy): type-check benchmarks/ (anchor the legacy benchmark/ exclude)
  
  FBumann
  and
  claude
  committed
  ac1fda4
- benchmarks: harden pypsa example fetch + CodSpeed continue-on-error
  
  FBumann
  and
  claude
  committed
  7f2585b
- Merge branch 'master' into benchmark-suite-charter
  FBumann
  authored
  827a947
- Merge remote-tracking branch 'origin/benchmark-suite-charter' into feat/benchmark-patterns
  FBumann
  committed
  919e766
- refactor(benchmarks): split cli.py into a typer sub-app package
  FBumann
  committed
  be65b12
- test(benchmarks): group harness unit tests under _tests/
  
  FBumann
  and
  claude
  committed
  f07fa3c
- docs(benchmarks): trim verbose inline comments in CI/config
  
  FBumann
  and
  claude
  committed
  14b6445
- feat(benchmarks): drop flow_sum pattern (irreducible, not a sparsity dial)
  
  FBumann
  and
  claude
  committed
  e9934f0
- perf(benchmarks): size pattern dims so peak-RSS cliffs clear the noise floor
  
  FBumann
  and
  claude
  committed
  1db88ec
- fix: read tuple coords entries as xarray's (dim_name, values)
  
  FBumann
  and
  claude
  committed
  8daaa2d
- test: parameterize and reorganize alignment coords tests
  
  FBumann
  and
  claude
  committed
  049a940
- fix: correct coords-entry TypeError to not list tuple as a bare sequence
  
  FBumann
  and
  claude
  committed
  e08b44f
- fix(benchmarks): clearer scaling-plot axis labels via a per-axis table
  
  FBumann
  and
  claude
  committed
  03beba4
- refactor(benchmarks): rename the spec-name field model -> spec
  
  FBumann
  and
  claude
  committed
  942dec8
- [pre-commit.ci] auto fixes from pre-commit.com hooks
  pre-commit-ci[bot]
  committed
  ac3a6d1
- ci: split CodSpeed into its own master-only workflow
  
  FBumann
  and
  claude
  committed
  fb4a8bd
- merge: benchmark suite onto master (sparsity/memory baseline)
  FBumann
  committed
  1ddba73
- ci(codspeed): instrumentation on PRs-to-master, add walltime macro job
  
  FBumann
  and
  claude
  committed
  f235cc4
- merge: tuple coords-entry regression fix (#766)
  FBumann
  committed
  d0ca7d3
- fix(benchmarks): walkthrough groups by spec, not the renamed-away model
  
  FBumann
  and
  claude
  committed
  0ef08c0
- ci(codspeed): authenticate via OIDC instead of a token secret
  
  FBumann
  and
  claude
  committed
  dc8d404
- ci(codspeed): set mode=simulation (required input in action v4)
  
  FBumann
  and
  claude
  committed
  ce449ce
- ci(codspeed): add memory instrument job (heap allocations)
  
  FBumann
  and
  claude
  committed
  ec473c8
- perf(benchmarks): scope CodSpeed runs, drop netcdf disk I/O
  
  FBumann
  and
  claude
  committed
  755085a
- perf(benchmarks): thin default severity sweep 5->3 (0,50,100)
  
  FBumann
  and
  claude
  committed
  ab61e27
- refactor(benchmarks): rename phases to to_/from_ scheme; drop matrices from CodSpeed
  
  FBumann
  and
  claude
  committed
  1e2f49d
- perf(benchmarks): per-instrument CodSpeed subsets via --codspeed-set
  
  FBumann
  and
  claude
  committed
  0478494
- perf(benchmarks): simplify --codspeed-set to full|simulation
  
  FBumann
  and
  claude
  committed
  d0d0c52
- perf(benchmarks): scope walkthrough run/memory cells to one model
  
  FBumann
  and
  claude
  committed
  8593377
- ci(codspeed): combine simulation+memory upload; per-PR memory; label-gated full run
  
  FBumann
  and
  claude
  committed
  e2276dc
- refactor(benchmarks): lean up — trim docstrings/comments, dedupe plotting (#28)
  
  FBumann
  and
  claude
  authored
  f37c8be
- ci(codspeed): drop simulation (cachegrind); memory is the always-on baseline (#29)
  
  FBumann
  and
  claude
  authored
  520bb43
Commits on Jun 6, 2026
- feat(benchmarks): log₂ sweep colouring + --clip colour clamp (#30)
  
  FBumann
  and
  claude
  authored
  3fdb9b4
- Benchmark selection rework + time/memory CLI unification (#31)
  
  FBumann
  and
  claude
  authored
  0dc9c0a
- fix(benchmarks): sweep colourbar keeps several round fold labels at small --clip (#32)
  
  FBumann
  and
  claude
  authored
  4db3c76
- refactor(benchmarks): explicit QUICK_SIZES/LONG_SIZES per spec (drop thresholds) (#33)
  
  FBumann
  and
  claude
  authored
  6d47dfa
Commits on Jun 7, 2026
- Merge branch 'PyPSA:master' into master
  FBumann
  authored
  a64c0f2
- pre-commit
  FBumann
  committed
  85c103c
Commits on Jun 8, 2026
- refactor(benchmarks): decouple the CI baseline from the local analytics layer
  
  FBumann
  and
  claude
  committed
  72b3f90
- refactor(benchmarks): one measurements() contract for timing + memory
  
  FBumann
  and
  claude
  committed
  d2a5d54
Commits on Jun 13, 2026
- bench: strip local-analysis layer — CodSpeed-only suite
  
  FBumann
  and
  claude
  committed
  bff1fb8
- bench: trim framework docstrings + drop dead Jupyter repr
  
  FBumann
  and
  claude
  committed
  3f6242e
- bench: collapse to one size set per spec + drop dead spec metadata
  
  FBumann
  and
  claude
  committed
  d7084d0
- bench: drop dead PhaseCase.value + fix stale references
  
  FBumann
  and
  claude
  committed
  ed0044a
- bench: merge ModelSpec + PatternSpec into one BenchSpec
  
  FBumann
  and
  claude
  committed
  61c4e64
- bench: delete the case layer — inline setup into the pytest drivers
  
  FBumann
  and
  claude
  committed
  be3f4a1
- bench: emit spec/phase/axis dims into extra_info for pytest-benchmem plots
  
  FBumann
  and
  claude
  committed
  bb8d4be
Commits on Jun 14, 2026
- bench: modernize the benchmark deps + run on Python 3.13
  
  FBumann
  and
  claude
  committed
  4cd85d8
- ci(codspeed): merge the memory + walltime workflows into one
  
  FBumann
  and
  claude
  committed
  4cf8522
Commits on Jun 15, 2026
- Merge branch 'master' into bench/strip-to-codspeed
  FBumann
  authored
  7a570df
Commits on Jun 22, 2026
- bench: move phase drivers into benchmarks/drivers/
  FabianHofmann
  committed
  839472f
Commits on Jun 23, 2026
- Merge branch 'master' into bench/strip-to-codspeed
  FabianHofmann
  authored
  074e4b1

Commits

Commits on May 27, 2026

Commits on May 28, 2026

Commits on May 29, 2026

Commits on Jun 4, 2026

Commits on Jun 5, 2026

Commits on Jun 6, 2026

Commits on Jun 7, 2026

Commits on Jun 8, 2026

Commits on Jun 13, 2026

Commits on Jun 14, 2026

Commits on Jun 15, 2026

Commits on Jun 22, 2026

Commits on Jun 23, 2026