Skip to content

Seed scib's kBET eigensolver in the solver-equivalence test - #337

Open
timtreis wants to merge 1 commit into
mainfrom
fix/flaky-solver-test
Open

timtreis wants to merge 1 commit into
mainfrom
fix/flaky-solver-test

Conversation

@timtreis

@timtreis timtreis commented Nov 15, 2026 •

Copy link
Copy Markdown
Member

test_evaluate_integration_baseline_solver_auto_matches_arpack flaked because of kBET, not the PCA solver. scib's kbet_per_label calls scipy.sparse.linalg.eigsh with an unseeded random start vector. On some runners, 3 of 60 unseeded calls on the same kNN graph gave 0.991667 where the rest gave 0.941667. Start-vector seeds 4, 11 and 19 always give 0.991667.

The test now patches eigsh with a fixed v0, so both runs use the same start vector. The tolerance stays at 1e-4. The two solvers' baselines agree to a subspace angle below 1e-5.

Verified on CI with a temporary diagnostic, since removed: on all three hatch envs, a fixed start vector gave the same kBET in 60 of 60 calls, and the patched test failed 0 of 8 times. tests/test_metrics.py, ruff, ruff format and mypy pass locally.

@read-the-docs-community

read-the-docs-community Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

Documentation build overview

📚 mantispy | 🛠️ Build #35898918 | 📁 Comparing 0bc66c3 against latest (d19f209)

  🔍 Preview build  

1 file changed
± api/generated/mantispy.settings.html

The flake was kBET, which moved between 0.941667 and 0.991667 while every other score, and the auto and arpack PCA baselines (subspace angle under 1e-5), agreed. kBET never sees the baseline: scib diffuses each label's graph through scipy's eigsh with an unseeded random start vector, and on some CI runners about one start in twenty lands on the other kBET value. The test now gives eigsh a fixed start vector, so it checks the solver and nothing else.
@timtreis
timtreis force-pushed the fix/flaky-solver-test branch from df898bf to 0bc66c3 Compare November 15, 2026 00:48
@timtreis timtreis changed the title Make the solver test independent of scib's kBET randomness Seed scib's kBET eigensolver in the solver-equivalence test Nov 15, 2026
@timtreis
timtreis marked this pull request as ready for review November 15, 2026 00:48
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79.60%. Comparing base (d19f209) to head (0bc66c3).

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #337   +/-   ##
=======================================
  Coverage   79.60%   79.60%           
=======================================
  Files          97       97           
  Lines       12354    12354           
=======================================
  Hits         9835     9835           
  Misses       2519     2519           
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants