[WIP] Fix rng by garrettwrong · Pull Request #1411 · ComputationalCryoEM/ASPIRE-Python

garrettwrong · 2026-09-21T15:17:00Z

Resolves #1406 and #1405.

Still WIP. Stashing here for potential discussion during weekly dev meeting.

Still need to replace more instances of randi randn and random, but the main uses in the code have been converted at 87b4

Refactors Simulation to consolidate the randomized components. Legacy MATLAB behaviors isolated to LegacySimulation which should repro old work/sequences. Simulation should now use the modern numpy rng implementation and be randomly seeded by default.

garrettwrong · 2026-09-22T14:56:15Z

At fd8f704 this branch has migrated all the code not tied to hardcoded references towards modern Numpy RNG practices. With the exception noted below, legacy references/implementations have been isolated. They are currently still able to reproduce. Code has been refactored to allow easily dropping them entirely in the future if the related tests are replaced. Legacy random utils have been deprecated into our third level of imports, ie they are no longer exposed as aspire.utils. I'll plan to update the docstrings more after this patch is otherwise ready to go.

One problem area that remains is NoiseAdder. I found that a collection of tests require the exact sequence of generated noise to reproduce hard coded references. This is similar to what I found for generate_random_rotations, Simulation, and LegacyVolume, except that in those cases it was more straightforward to replicate and seperate the behavior for the legacy situations. I'm not sure exactly what to do with the NoiseAdder quite yet. Need to think on the options a little.

The other area remaining is the collection of commonlines tests. Most of the tests were not robust and some were not actually functional. The tolerances appear to be derived from whatever results come out with the prior seed/rng. While I can adjust the test tolerances, its probably better that Josh does this since most of these are his recent work. Some of them (the commonlines indices one for example) date back to Junchoa and should probably replaced with a legitimate test.

The following are the remaining test failures in the regular suite:

FAILED tests/test_orient_sdp.py::test_gram_matrix[resolution=33-offsets=0-dtype=<class 'numpy.float32'>]
FAILED tests/test_orient_lud.py::test_estimate_rotations[resolution=32-offsets=0-dtype=<class 'numpy.float32'>-alpha=0.6666666666666666-adp_proj=True]
FAILED tests/test_orient_lud.py::test_estimate_rotations[resolution=32-offsets=0-dtype=<class 'numpy.float32'>-alpha=0.6666666666666666-adp_proj=False]
FAILED tests/test_orient_d2.py::test_estimate_rotations[n images=10-resolution=48-dtype=<class 'numpy.float32'>-offsets=0]
FAILED tests/test_orient_lud.py::test_estimate_rotations[resolution=33-offsets=0-dtype=<class 'numpy.float32'>-alpha=0.6666666666666666-adp_proj=True]
FAILED tests/test_orient_lud.py::test_estimate_rotations[resolution=33-offsets=0-dtype=<class 'numpy.float32'>-alpha=0.6666666666666666-adp_proj=False]
FAILED tests/test_orient_d2.py::test_estimate_rotations[n images=10-resolution=49-dtype=<class 'numpy.float32'>-offsets=0]
FAILED tests/test_rotation.py::test_common_lines[dtype=<class 'numpy.float64'>]
FAILED tests/test_orient_sdp.py::test_construct_S[resolution=32-offsets=0-dtype=<class 'numpy.float32'>]
FAILED tests/test_orient_sdp.py::test_construct_S[resolution=33-offsets=0-dtype=<class 'numpy.float32'>]
FAILED tests/test_orient_sdp.py::test_gram_matrix[resolution=32-offsets=0-dtype=<class 'numpy.float32'>]
FAILED tests/test_rotation.py::test_common_lines[dtype=<class 'numpy.float32'>]
FAILED tests/test_orient_irls.py::test_estimate_rotations[resolution=32-offsets=0-dtype=<class 'numpy.float32'>-alpha=0.6666666666666666-adp_proj=True]
FAILED tests/test_orient_irls.py::test_estimate_rotations[resolution=32-offsets=0-dtype=<class 'numpy.float32'>-alpha=0.6666666666666666-adp_proj=False]
FAILED tests/test_orient_symmetric.py::test_relative_viewing_directions[24-45-4-float64]
FAILED tests/test_orient_irls.py::test_estimate_rotations[resolution=32-offsets=None-dtype=<class 'numpy.float32'>-alpha=0.6666666666666666-adp_proj=True]
FAILED tests/test_orient_irls.py::test_estimate_rotations[resolution=32-offsets=None-dtype=<class 'numpy.float32'>-alpha=0.6666666666666666-adp_proj=False]

These are the failures in the expensive suite:

FAILED tests/test_orient_sdp.py::test_gram_matrix[resolution=32-offsets=0-dtype=<class 'numpy.float64'>]                           
FAILED tests/test_orient_sdp.py::test_gram_matrix[resolution=33-offsets=0-dtype=<class 'numpy.float64'>]                           
FAILED tests/test_orient_lud.py::test_estimate_rotations[resolution=33-offsets=None-dtype=<class 'numpy.float64'>-alpha=None-adp_pr
oj=True]                                                                                                                           
FAILED tests/test_orient_irls.py::test_estimate_rotations[resolution=32-offsets=None-dtype=<class 'numpy.float64'>-alpha=0.66666666
66666666-adp_proj=True]                                                                                                            
FAILED tests/test_orient_lud.py::test_estimate_rotations[resolution=33-offsets=None-dtype=<class 'numpy.float64'>-alpha=None-adp_pr
oj=False]                                                                                                                          
FAILED tests/test_orient_irls.py::test_estimate_rotations[resolution=32-offsets=None-dtype=<class 'numpy.float64'>-alpha=0.66666666
66666666-adp_proj=False]                                                                                                           
FAILED tests/test_orient_d2.py::test_estimate_rotations[n images=10-resolution=48-dtype=<class 'numpy.float64'>-offsets=None]      
FAILED tests/test_orient_symmetric.py::test_relative_viewing_directions[24-44-4-float32]                                           
FAILED tests/test_orient_d2.py::test_scl_scores[n images=10-resolution=48-dtype=<class 'numpy.float64'>-offsets=None]              
FAILED tests/test_orient_d2.py::test_estimate_rotations[n images=10-resolution=48-dtype=<class 'numpy.float32'>-offsets=None]      
FAILED tests/test_orient_symmetric.py::test_relative_viewing_directions[24-44-4-float64]                                           FAILED tests/test_orient_lud.py::test_estimate_rotations[resolution=33-offsets=0-dtype=<class 'numpy.float64'>-alpha=0.666666666666
6666-adp_proj=True]                                                                                                                FAILED tests/test_orient_symmetric.py::test_relative_viewing_directions[24-45-4-float32]                                           
FAILED tests/test_orient_irls.py::test_estimate_rotations[resolution=33-offsets=0-dtype=<class 'numpy.float32'>-alpha=0.66666666666
66666-adp_proj=True]                                                                                                               FAILED tests/test_orient_lud.py::test_estimate_rotations[resolution=33-offsets=0-dtype=<class 'numpy.float64'>-alpha=0.666666666666
6666-adp_proj=False]                                                                                                               
FAILED tests/test_orient_d2.py::test_estimate_rotations[n images=10-resolution=48-dtype=<class 'numpy.float64'>-offsets=0]
FAILED tests/test_orient_irls.py::test_estimate_rotations[resolution=33-offsets=0-dtype=<class 'numpy.float32'>-alpha=0.6666666666666666-adp_proj=False]  
FAILED tests/test_orient_lud.py::test_estimate_rotations[resolution=32-offsets=None-dtype=<class 'numpy.float32'>-alpha=None-adp_proj=True]                
FAILED tests/test_orient_lud.py::test_estimate_rotations[resolution=32-offsets=None-dtype=<class 'numpy.float32'>-alpha=None-adp_pr
oj=False]                                         
FAILED tests/test_orient_d2.py::test_estimate_rotations[n images=10-resolution=49-dtype=<class 'numpy.float64'>-offsets=None]
FAILED tests/test_orient_lud.py::test_estimate_rotations[resolution=32-offsets=None-dtype=<class 'numpy.float64'>-alpha=None-adp_pr
oj=True]
FAILED tests/test_orient_irls.py::test_estimate_rotations[resolution=33-offsets=0-dtype=<class 'numpy.float64'>-alpha=0.66666666666
66666-adp_proj=True]
FAILED tests/test_orient_irls.py::test_estimate_rotations[resolution=32-offsets=0-dtype=<class 'numpy.float64'>-alpha=0.66666666666
66666-adp_proj=True]
FAILED tests/test_orient_lud.py::test_estimate_rotations[resolution=32-offsets=None-dtype=<class 'numpy.float64'>-alpha=None-adp_pr
oj=False]
FAILED tests/test_orient_d2.py::test_estimate_rotations[n images=10-resolution=49-dtype=<class 'numpy.float64'>-offsets=0]
FAILED tests/test_orient_irls.py::test_estimate_rotations[resolution=33-offsets=0-dtype=<class 'numpy.float64'>-alpha=0.66666666666
66666-adp_proj=False]
FAILED tests/test_orient_irls.py::test_estimate_rotations[resolution=32-offsets=0-dtype=<class 'numpy.float64'>-alpha=0.66666666666
66666-adp_proj=False]
FAILED tests/test_orient_lud.py::test_estimate_rotations[resolution=32-offsets=0-dtype=<class 'numpy.float64'>-alpha=0.666666666666
6666-adp_proj=True]
FAILED tests/test_orient_lud.py::test_estimate_rotations[resolution=32-offsets=0-dtype=<class 'numpy.float64'>-alpha=0.666666666666
6666-adp_proj=False]
FAILED tests/test_orient_irls.py::test_estimate_rotations[resolution=33-offsets=None-dtype=<class 'numpy.float32'>-alpha=0.66666666
66666666-adp_proj=True]
FAILED tests/test_orient_d2.py::test_estimate_rotations[n images=10-resolution=49-dtype=<class 'numpy.float32'>-offsets=None]
FAILED tests/test_orient_irls.py::test_estimate_rotations[resolution=33-offsets=None-dtype=<class 'numpy.float32'>-alpha=0.66666666
66666666-adp_proj=False]
FAILED tests/test_orient_sdp.py::test_construct_S[resolution=32-offsets=0-dtype=<class 'numpy.float64'>]
FAILED tests/test_orient_sdp.py::test_construct_S[resolution=33-offsets=0-dtype=<class 'numpy.float64'>]
FAILED tests/test_orient_irls.py::test_estimate_rotations[resolution=33-offsets=None-dtype=<class 'numpy.float64'>-alpha=0.66666666
66666666-adp_proj=True]
FAILED tests/test_orient_irls.py::test_estimate_rotations[resolution=33-offsets=None-dtype=<class 'numpy.float64'>-alpha=0.66666666
66666666-adp_proj=False]
FAILED tests/test_synthetic_volume.py::test_volume_symmetry[ISymmetricVolume, res=70-float32]
FAILED tests/test_synthetic_volume.py::test_volume_symmetry[ISymmetricVolume, res=71-float32]
FAILED tests/test_synthetic_volume.py::test_volume_symmetry[ISymmetricVolume, res=70-float64]
FAILED tests/test_synthetic_volume.py::test_volume_symmetry[ISymmetricVolume, res=71-float64]
FAILED tests/test_orient_symmetric.py::test_estimate_rotations[24-45-6-float64]
FAILED tests/test_orient_sync_voting.py::test_estimate_shifts_with_est_rots[resolution=41-offsets=None-dtype=<class 'numpy.float64'
>]
FAILED tests/test_orient_sync_voting.py::test_estimate_shifts_with_est_rots[resolution=41-offsets=None-dtype=<class 'numpy.float32'
>]
FAILED tests/test_orient_symmetric.py::test_estimate_rotations[24-44-8-float32]
FAILED tests/test_orient_symmetric.py::test_estimate_rotations[24-45-9-float64]

@j-c-c please make a PR (into this fix_rng branch) that corrects these tests (either by updating tolerances or addressing code problems) .

While that is happening I will setup to run the experimental data reconstructions to see how they are performing with this implementation.

j-c-c · 2026-09-22T15:33:13Z

@j-c-c please make a PR (into this fix_rng branch) that corrects these tests (either by updating tolerances or addressing code problems) .

While that is happening I will setup to run the experimental data reconstructions to see how they are performing with this implementation.

@garrettwrong Working on it now.

codecov · 2026-09-25T18:46:52Z

Codecov Report

❌ Patch coverage is 95.33679% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.11%. Comparing base (e4a05cc) to head (cec9d3a).

Files with missing lines	Patch %	Lines
src/aspire/utils/random.py	61.90%	8 Missing ⚠️
src/aspire/source/simulation.py	98.24%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #1411      +/-   ##
===========================================
- Coverage    90.65%   90.11%   -0.55%     
===========================================
  Files          135      135              
  Lines        14932    15001      +69     
===========================================
- Hits         13537    13518      -19     
- Misses        1395     1483      +88

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

garrettwrong · 2026-10-01T17:36:21Z

Just kicking off the review suite, not ready for review yet.

[skip ci]

…of file rng

garrettwrong · 2026-10-12T19:00:28Z

Looks like most of the remaining failures are in tests/test_orient_sync_voting.py::test_estimate_rotations_fuzzy_mask where it often comes just slightly outside 10% for the test now that it is randomly seeded.

garrettwrong · 2026-10-23T12:25:13Z

Manually re-running the ampere jobs to see if there is any flaky tests that pop up before those machines go down for maintenance.

garrettwrong · 2026-11-02T16:01:33Z

@j-c-c , what's the ETA looking like on this review?

j-c-c · 2026-11-02T16:20:48Z

@j-c-c , what's the ETA looking like on this review?

@garrettwrong still working through it. Should have it sometime after lunch.

garrettwrong self-assigned this Sep 21, 2026

garrettwrong added cleanup support User Support labels Sep 21, 2026

garrettwrong force-pushed the fix_rng branch from 47354bb to fd8f704 Compare September 22, 2026 14:30

j-c-c reviewed Oct 1, 2026

View reviewed changes

Comment thread tests/test_orient_sync_voting.py Resolved

garrettwrong force-pushed the fix_rng branch from 31ef042 to 555bc27 Compare October 1, 2026 15:34

garrettwrong marked this pull request as ready for review October 1, 2026 17:36

garrettwrong requested a review from janden as a code owner October 1, 2026 17:36

garrettwrong changed the title ~~Fix rng~~ [WIP] Fix rng Oct 1, 2026

garrettwrong force-pushed the fix_rng branch from d9e54b4 to 867949e Compare October 1, 2026 20:00

j-c-c mentioned this pull request Oct 2, 2026

Improvetest_orient_sync_voting #1340

Open

j-c-c reviewed Oct 2, 2026

View reviewed changes

Comment thread tests/test_orient_symmetric.py Resolved

j-c-c reviewed Oct 6, 2026

View reviewed changes

Comment thread tests/test_orient_symmetric.py Resolved

j-c-c reviewed Oct 6, 2026

View reviewed changes

Comment thread tests/test_orient_symmetric.py Resolved

garrettwrong added 12 commits October 8, 2026 08:50

stash1 all tests still passing

bdf183f

[skip ci]

remedy bot_align

db5f5a2

remedy cov3d tutorial

a86cacd

remediate test files

69fd80e

cleanup

3f56385

remedy micrograph sim

bd2b88e

remedy cl cn

7dc2627

minor tol adjustment

d051b82

minor tol adjustment preproc

09f6f1e

progress

26ba5fd

Better handle and log seed init

7832b85

pin denoiser utest seed similar to old sim

0b5e88e

garrettwrong added 9 commits October 8, 2026 08:50

migrate src code from seed to rng

cc16936

migrate test code from seed to rng

06db7f3

migrate gallery code from seed to rng

df90f8e

fix copy paste error per jcc

c708851

update minimal scipy version in conda envs and pyproj

64a2842

change tolerance per discussion with jcc

364dad6

remove semi-fixed seeds from test_orient_symmetric, replace with top …

18cfb9f

…of file rng

update min_dist_cls for C2 case per jcc

c283830

more orient sym test patches per JCC

de73229

garrettwrong force-pushed the fix_rng branch from b888c82 to de73229 Compare October 8, 2026 12:51

garrettwrong added 2 commits October 8, 2026 09:04

patch jcc's patch

702e1e6

split off the legacy NoiseAdder

8cc01c2

garrettwrong force-pushed the fix_rng branch from 9b84213 to 8cc01c2 Compare October 9, 2026 14:01

garrettwrong added 3 commits October 9, 2026 10:25

minor tweak denoiser test tol (by 1e-7)

1f0a6a7

remove seed refereces in gallery

9bd5cd4

bump white noise estimate from 1 to 2% for random seeded case

00d7e46

j-c-c reviewed Oct 13, 2026

View reviewed changes

Comment thread tests/test_orient_sync_voting.py Resolved

use emdb volume instead of builtins for fuzzy mask unit test

a5d056a

garrettwrong force-pushed the fix_rng branch from 8233812 to a5d056a Compare October 13, 2026 14:26

loosen frc/fsc tols

16a092a

garrettwrong added 2 commits October 23, 2026 09:42

add docstring for legacy white noise adder

06b2491

try to stablize the preproc tests with a seed

cec9d3a

garrettwrong force-pushed the fix_rng branch from afffc6d to cec9d3a Compare October 23, 2026 16:35

garrettwrong requested a review from j-c-c October 23, 2026 17:18

garrettwrong mentioned this pull request Oct 26, 2026

Remove sph_harm workaround for older (<1.15) scipy #1418

Draft

j-c-c added a commit that referenced this pull request Oct 27, 2026

Flake8 fix also contained in PR #1411

63996ad

Navigation Menu

Repository navigation

[WIP] Fix rng - #1411

[WIP] Fix rng#1411
garrettwrong wants to merge 59 commits into
developfrom
fix_rng

garrettwrong commented Sep 21, 2026

Uh oh!

garrettwrong commented Sep 22, 2026

Uh oh!

j-c-c commented Sep 22, 2026

Uh oh!

codecov Bot commented Sep 25, 2026 •

edited

Loading

Uh oh!

Uh oh!

garrettwrong commented Oct 1, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

garrettwrong commented Oct 12, 2026

Uh oh!

Uh oh!

garrettwrong commented Oct 23, 2026

Uh oh!

garrettwrong commented Nov 2, 2026

Uh oh!

j-c-c commented Nov 2, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

garrettwrong commented Sep 21, 2026

Uh oh!

garrettwrong commented Sep 22, 2026

Uh oh!

j-c-c commented Sep 22, 2026

Uh oh!

codecov Bot commented Sep 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

garrettwrong commented Oct 1, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

garrettwrong commented Oct 12, 2026

Uh oh!

Uh oh!

garrettwrong commented Oct 23, 2026

Uh oh!

garrettwrong commented Nov 2, 2026

Uh oh!

j-c-c commented Nov 2, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codecov Bot commented Sep 25, 2026 •

edited

Loading