Skip to content

Run a single IDAKLU solver outside any OpenMP parallel region - #5837

Open
aabills wants to merge 3 commits into
mainfrom
fix/idaklu-single-solver-openmp
Open

aabills wants to merge 3 commits into
mainfrom
fix/idaklu-single-solver-openmp

Conversation

@aabills

@aabills aabills commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

Description

IDAKLUSolverGroup::solve runs its solvers under #pragma omp parallel num_threads(team_size). That's also true when there is only one solver, the usual case, so a single solve ran inside a one-thread parallel region. OpenMP code that the model calls during the solve, such as a CasADi external function, therefore ran nested at omp_get_level() == 1 and took the nested-level ICVs instead of the top-level ones.

The most visible effect shows up with a per-level OMP_NUM_THREADS list. Under OMP_NUM_THREADS=8,1, an OpenMP region in a CasADi external called from the solve got 1 thread instead of 8. The 8,1 setting asks for a team of 8 at the top level, and that's the level the external is logically at.

Fix. The team body becomes a lambda, run_thread(thread). When team_size == 1 it's called directly on the calling thread, with no parallel region. Otherwise it runs under the same #pragma omp parallel num_threads(team_size) as before. The multi-solver path is unchanged.

Reproducer

A CasADi external, built with -fopenmp against the same libomp as pybammsolvers, is used both as the model's RHS and as an output variable. It reports what OpenMP looks like from inside the solve:

int probe(const double** arg, double** res, casadi_int* iw, double* w, int mem) {
  int team = 0;
  #pragma omp parallel
  {
    #pragma omp single
    team = omp_get_num_threads();
  }
  res[0][0] = omp_get_level();
  res[0][1] = omp_in_parallel();
  res[0][2] = team;
  return 0;
}

The probe is passed to create_casadi_solver_group as an output variable, with num_solvers=1 and num_threads=1. Results on macOS arm64 with Homebrew libomp:

Environment Build omp_get_level() in solve Team size of a region in solve
default (14 cores) main 1 14
default (14 cores) this PR 0 14
OMP_NUM_THREADS=8,1 main 1 1
OMP_NUM_THREADS=8,1 this PR 0 8
OMP_NUM_THREADS=8,2 main 1 2
OMP_NUM_THREADS=8,2 this PR 0 8

With default settings the per-region overhead was the same either way: about 23 µs for a small parallel for at 8 threads, both inside and outside a solve. This PR is therefore about OpenMP semantics, not speed. A nested setting meant for code that runs inside the external's regions no longer gets applied to the external itself.

Type of change

Bug fix. There's an entry under ## Bug fixes in CHANGELOG.md.

Tests

No new test yet. The reproducer needs an OpenMP-compiled shared library loaded as a CasADi external, and the test suite has no existing way to build one. Pointers to an acceptable way to cover this in CI would be welcome.

The existing test_parallel_solver_group_uses_multiple_solvers covers the multi-solver path, which still opens the parallel region.

Important checks:

  • No style issues (pre-commit hooks pass on the commits)
  • All tests pass. Run locally against a wheel built from this branch: packages/pybamm/tests/unit/test_solvers and packages/pybamm/tests/integration/test_solvers (467 passed, 95 skipped) and packages/pybammsolvers/tests (64 passed). The full nox -s tests hasn't been run.
  • The documentation builds: nox -s doctests not run (no documentation changes)
  • Code is commented for hard-to-understand areas
  • Tests added that prove the fix is effective (see Tests above)

🤖 Generated with Claude Code

aabills and others added 3 commits October 2, 2026 14:36
With one solver the solve still ran inside a one-thread parallel region, so
OpenMP code the model calls during the solve (a CasADi external function, for
example) ran nested at level 1. It then took the nested-level ICVs, so under
OMP_NUM_THREADS=8,1 its regions got one thread instead of eight. A single
solver now runs directly on the calling thread; several solvers keep the
self-scheduled team.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov

codecov Bot commented Oct 2, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.09%. Comparing base (445ba33) to head (57793e5).

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #5837   +/-   ##
=======================================
  Coverage   98.09%   98.09%           
=======================================
  Files         346      346           
  Lines       35354    35354           
=======================================
  Hits        34680    34680           
  Misses        674      674           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@aabills
aabills marked this pull request as ready for review October 2, 2026 22:50
@aabills
aabills requested a review from a team as a code owner October 2, 2026 22:50
@github-actions

github-actions Bot commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

🐰 Bencher Report

ProjectPyBaMM
Branchfix/idaklu-single-solver-openmp
Testbedbare-metal

⚠️ WARNING: Truncated view!

The full continuous benchmarking report exceeds the maximum length allowed on this platform.

🐰 View full continuous benchmarking report in Bencher

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant