Docs: Sphinx → properdocs migration + descending-drive Poisson fix (+ Optuna parallelization)#19
Merged
Conversation
Design proposal for a backend-swappable, vectorized MyoGen API: a shared flat SimState buffer stepped in lockstep by a backend-agnostic driver, with two backend seams (neural dynamics NEURON->jaxley; muscle/force analytic->FEM) and a closed-loop stage graph for afferent feedback. Buffer-first data model with neo/NWB as opt-in adapters. Maps the full current public surface onto the new layers and lists net-new work and open questions for team review.
Bite-sized TDD plan for the backend-agnostic simulation kernel (myogen/kernel): SimState shared buffer, Stage/Backend protocols, the Simulation lockstep driver with one-tick closed-loop feedback, and the buffer-first SimResult with lazy neo/NWB adapters. Seven tasks, fully CI-testable without NEURON/jaxley/FEM. NEURONBackend, real stages, jaxley and FEM are scoped as follow-on plans.
SimResult.from_state() extracts spike times in seconds, copies force and surface EMG arrays, infers grid_shape from EMG buffer dimensions, and handles missing optional buffers gracefully. Simulation.run() now returns SimResult instead of bare SimState; all kernel tests updated to read buffers through sim.state.
…rocesses Extract simulation logic and constants into _oscillating_dc_helpers.py, add _optimize_dc_worker.py subprocess worker, and rewrite the launch in 02_optimize_oscillating_dc.py to fan out N workers that all write to one shared SQLite Optuna study. Avoids NEURON h singleton / macOS spawn constraints; gallery-safe via conf.py ignore_pattern for the _ files.
Example 14 (calibrate_noise_from_real) imports h5py directly to load real recordings, but it was only available transitively. Declaring it in the nwb extra makes 'uv sync --extra nwb' sufficient to run the NWB/HDF5 examples (11, 13, 14).
Many MyoGen examples use multiprocessing/joblib (which breaks under the gallery's exec context) or form slow inter-dependent chains, so executing the gallery in CI is unreliable. Render examples as source + prose with full-code download links instead; opt into execution with MKDOCS_GALLERY_PLOT=true.
Add MuscleGeometry/JointGeometry/JointBiomechanics to the simulator API, the four missing type aliases, and a RANDOM_GENERATOR/SEED note; restore the How-to-cite, Contributing, and License sections on the home page and the MUAP simulate/access step in the quick-start (per migration-coverage audit).
Capture one representative figure per example (run standalone, where they work) and wire them as mkdocs-gallery thumbnails via per-example mkdocs_gallery_thumbnail_path directives + a logo default_thumb_file fallback. Gives the source-only gallery real pictures on the overview grid without executing examples in the docs build (which spawn-bombs on the multiprocessing examples).
- CI: execute the mkdocs-gallery examples (MKDOCS_GALLERY_PLOT=true) in
the docs build so figures render inline in the deployed site, matching
the previous Sphinx docs. Runs on ubuntu-latest, where NEURON + joblib
forking is stable (local macOS builds stay source-only to avoid a
fork-related segfault in the joblib examples).
- Home: constrain the logo with an inline style so Material's
`.md-typeset img { height:auto }` can no longer override the height
attribute and stretch it to full column width.
- Correct the conductivity parameter names in the Muscle docstring to
match the constructor: muscle_conductivity_{radial,longitudinal}__S_per_m
(were __S_m, so griffe reported them as absent from the signature).
- Move the "Results are stored in ..." prose in generate_muscle_fiber_centers
and assign_mfs2mns out of the Parameters section into Notes, so griffe no
longer misreads it as a phantom "Results" parameter.
The docs now build with zero griffe warnings.
Every gallery example that calls plt.show() (24 of them) emitted a "FigureCanvasAgg is non-interactive, and thus cannot be shown" UserWarning into its rendered Out block, dragging the absolute source path along with it. mkdocs-gallery captures figures via plt.get_fignums() independently of show(), so show() serves no purpose during a headless build. No-op it (as sphinx-gallery does) and pin the Agg backend.
plt.cm.get_cmap is deprecated in recent matplotlib; the warning leaked into the rendered Out block of examples 03 and 04. plt.get_cmap returns an identical colormap, so the figures are unchanged.
mkdocs-gallery scrapes stderr, so the libraries' and examples' live tqdm progress bars were captured verbatim into every example's rendered Out block (e.g. "Running simulation: 99%|...| 14881/15000 ms"), along with a "TqdmWarning: clamping frac to range [0,1]". tqdm reads TQDM_DISABLE only at import, too early to set from here, so force-disable every tqdm instance created during the build.
neuron.run(tstop) is deprecated (emits a DeprecationWarning that leaked into
the docs). Its body is simply `tstop = X; while (t < tstop) { fadvance() }`
— it does NOT re-initialise. The call sites already run h.finitialize()
with a hand-set initial voltage, so the documented replacement
`h.stdinit()` would wrongly re-initialise and change results; we replicate
neuron.run()'s exact behaviour instead. Also drops the now-unused
`import neuron` at both sites.
mean_firing_rate(st.time_slice(st.min(), st.max())) divides by the spike
span (last - first spike), which is zero for a train with a single spike:
divide-by-zero -> inf/nan that then poisoned the downstream np.mean/np.std
("invalid value in subtract"), all leaking into the rendered docs (and making
the printed motor-neuron rate inf/nan). Require >=2 spikes, since a rate over
the spike span is only defined then. Affects examples 03 and finetune/04;
example 02 was already guarded this way.
# Conflicts: # docs/source/conf.py # docs/source/examples.rst
RaulSimpetru
force-pushed
the
feat/puffer-api
branch
from
July 12, 2026 16:39
c0c064f to
be6907f
Compare
…ples The MKDOCS_GALLERY_PLOT=true CI docs build segfaulted (exit 139) while executing the gallery. mkdocs-gallery runs every example sequentially in one long-lived process; once NEURON has initialised native runtime state that the reset_neuron() hook cannot unwind, MyoGen spawning joblib `loky` worker processes (parallel muscle-fibre / MUAP generation in SurfaceEMG, muscle assignment, intramuscular EMG, and example 09's explicit n_jobs=2) crashes the process. Reproduced on Linux CI and macOS. Force every joblib.Parallel to n_jobs=1 for the build: in-process, no worker boundary (and, as a bonus, the tqdm-disable patch stays effective since no fresh worker re-imports it). Also wires the examples/04_clinical category (SCI iEMG + PIC spasticity, merged from main) into the properdocs gallery: add it to gallery_conf.py examples/gallery dirs, ignore the _pic_protocols helper, add a Clinical nav entry, and convert its README.rst to README.md.
RaulSimpetru
force-pushed
the
feat/puffer-api
branch
from
July 12, 2026 16:49
be6907f to
7884ad2
Compare
…teness Executing the whole example gallery in one process segfaults from NEURON native-state accumulation (mkdocs-gallery has no per-example process isolation). scripts/build_docs.py wraps `properdocs build`: because mkdocs-gallery md5-caches each executed example, retrying resumes past the crash in a fresh process with less accumulated state — the same mechanism the old Sphinx CI leaned on with `make html || make html`. The wrapper aborts if a retry makes no progress (no infinite loop) and validates that every executable example has its rendered page and figures before the site can deploy (a NEURON SIGSEGV lands before the md5 stamp is written, so crashing examples re-run rather than caching incomplete). docs.yml now runs the wrapper.
Executing the NEURON example gallery is slow and needs retries to survive state-accumulation segfaults (the heavy watanabe/clinical tail clears only a couple of examples per fresh process). It only affects the *deployed* site, so build source-only on pull requests (fast review builds) and execute the gallery only on push-to-main and manual workflow_dispatch. Raise the retry cap to 12 for the executed deploy build.
_pic_protocols.py called DescendingDrive__Pool(poisson_batch_size=5) — the pre-Poisson-fix API this branch removed — which broke the clinical example and test_pic_spasticity (merged from main) with "unexpected keyword argument 'poisson_batch_size'". poisson_batch_size=5 was Gamma(5,5) regular firing (CV=1/sqrt(5)); the equivalent new API is process_type="gamma", shape=5.
…only The retry wrapper does get the full gallery to execute on Linux, but the paper (03_papers/watanabe) and clinical (04_clinical) examples render broken: they use __file__ (undefined under mkdocs-gallery's exec()) to locate a results dir and load each other's .pkl outputs. Restrict gallery execution to the self-contained 01_basic + 02_finetune (filename_pattern) so those get inline plots reliably, and leave watanabe/clinical source-only. Also harden scripts/build_docs.py: parse failed example names and drop their stamps so a broken example re-runs (a deterministic failure then surfaces as "no progress" instead of a silently cached broken page), and scope the completeness target to the executed subdirs.
6 tasks
The migration removed docs/source/ and switched to MkDocs directory URLs, but README.md still pointed at the old Sphinx paths (review findings 4, 5): - logo src docs/source/_static/ -> docs/images/ (was a 404 on GitHub/PyPI) - Installation #installation -> getting-started/#install (install moved there) - Examples examples.html -> auto_examples/01_basic/ - User Guide neo_blocks_guide.html -> neo-blocks/ (#how-to-cite is left as-is: index.md still has a "How to cite" section.)
The workflow-level `concurrency: group: pages` serialized every Docs run, so PR "Docs" checks queued behind a long main-branch executed build (up to MAX_ATTEMPTS retries). Move the concurrency block to the deploy job, where the actual Pages-deploy race is.
Address Optuna-example review findings (1, 3, 7): - Workers now pass sampler=TPESampler(seed=...) to load_study(). Optuna doesn't persist samplers to storage, so each worker was silently building a fresh UNSEEDED sampler -> non-reproducible dc_offset suggestions despite the study's seed=42. (Full bit-reproducibility across parallel workers is still not possible; this removes the unseeded-sampler drift.) - Drive "noise" was np.clip(N(0,1), 0, inf) -> mean ~0.4 Hz, a systematic DC bias on every trial's drive. Make it zero-mean and clip the total instead. - Lower the default worker count 8 -> min(4, N_TRIALS) (each worker builds a memory-heavy 800-MN+400-DD network; 8 could OOM a laptop/CI; env override raises it), and delete any stale study at start so reruns don't accumulate trials into best_trial.
…x review) Two issues the reviewer and I missed, caught by Codex adversarial verification: - The subprocess parallelization dropped the wall-clock guard: TIMEOUT_SECONDS was defined/exported but unused, so a stalled worker would run/wait forever. Restore it by passing timeout=TIMEOUT_SECONDS to each worker's study.optimize (workers run concurrently, bounding the whole optimization to ~TIMEOUT_SECONDS). - The drive-noise unbias fix was half-done: 01_compute_baseline_force.py still used one-sided clip(N(0,1),0,inf) (~+0.4 Hz), so the reference force was computed with biased noise while trials used the corrected zero-mean noise. Make the baseline match (zero-mean, clip the total). - Minor: validate_cache() now checks the rendered .md is non-empty (matching its own docstring), not merely present.
…w finding 2)
The objective normalizes each simulation to its own peak (%MVC methodology, per
Watanabe) and matches the normalized steady-state force *profile* against the
identically-normalized baseline — not absolute force in Newtons. The old
docstring/comments ("match reference force", "scale to Newtons") read as
absolute-force matching, which the reviewer and Codex both flagged as
misleading. Clarify the intent in scripts 01 and 02; no logic change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The pufferlib-inspired API-redesign proposal and its kernel core have been moved to a separate PR — they are superseded by a new functional/differentiable API design, and keeping them here mixed unfinished design work with shippable fixes. This PR is now scoped to three verified, independently-reviewable workstreams.
1. Docs: Sphinx → properdocs (MkDocs)
Migrates the documentation to properdocs (mkdocs-material + mkdocstrings), matching MyoGestic / Virtual-Hand-Interface:
properdocs.yml(material theme, mkdocstrings API, section-index), flat markdown underdocs/(home, getting-started, neo-blocks), the mkdocstrings API pages, and a mkdocs-gallery example gallery. The Sphinx config (conf.py, rst pages,make/Makefile) is removed;.github/workflows/docs.ymlbuilds + deploys to GitHub Pages.Example gallery with inline plots. Executing the NEURON gallery in mkdocs-gallery's single, long-lived process segfaults as NEURON's native runtime state accumulates (it has no per-example process isolation), so:
docs/gallery_conf.pyforcesjoblib.Parallel→n_jobs=1(spawninglokyworker processes after NEURON init is one crash source), no-opsplt.show(removes the "FigureCanvasAgg non-interactive" warning that leaked into every Out block; figures are still captured viaplt.get_fignums()), and disables tqdm progress bars (they were being scraped into the rendered output).scripts/build_docs.pywraps the build with bounded retries — mkdocs-gallery md5-caches each executed example, so a retry resumes past the crash in a fresh process — plus a no-progress guard (no infinite loop) and completeness validation (a failed example re-runs; a deterministically-failing one aborts loudly instead of deploying a broken page).01_basic+02_finetunegalleries execute (inline plots); the paper (03_papers/watanabe) and clinical (04_clinical) galleries render source-only because they rely on__file__(undefined underexec()) and load each other's.pkloutputs. Execution runs only on themain-branch deploy (+ manualworkflow_dispatch); pull requests build source-only so review stays fast (~10s).Docstring/gallery hygiene: leftover Sphinx rST (
:func:/:class:/.. note::) swept from public docstrings; docstring↔signature mismatches fixed (AffIb__Pool,DescendingDrive__Pool.timestep__ms, a staleMusclecross-ref, andmuscle.py's conductivity + phantomResultsparams → zero griffe warnings); deprecatedRANDOM_GENERATOR/SEEDlabelled; the noise/binning utilities +DescendingDrive_Gamma__Pooldocumented; deprecatedplt.cm.get_cmap→plt.get_cmapandneuron.run()→an explicitfadvanceloop; and a single-spike divide-by-zero guard in the firing-rate examples.2. Fix: the descending-drive "Poisson" process was actually a Gamma process
DescendingDrive__Pool(process_type="poisson")withpoisson_batch_size/N > 1did not emit a Poisson process. The inter-spike threshold was drawn as the average of NExp(1)variates (-(1/N)·log(∏U)), i.e. a Gamma(N, N) renewal process (ISI CV =1/√N), progressively more regular as N grows. OnlyN = 1was ever correct, and the knob silently duplicated the existingDD_Gammapath.Exp(1)threshold (-log1p(-U), which also removes a latentlog(0)that could permanently silence a unit) — a discrete-time Poisson process (CV = 1, exact in thedt → 0limit).poisson_batch_sizeis removed; regular/Gamma firing goes throughprocess_type="gamma", shape=N.poisson_batch_size— really the Gamma shape of theAffIa/AffII/AffIbgenerators, not a Poisson batch size — is renamed toshape, and the docstrings that called it a "maximum firing rate" are corrected.process_type="gamma", shape=Nwhere regular firing was deliberately used (behaviour-preserving).tests/test_poisson_process.pyproves — through both the publicDescendingDrive__PoolAPI and the raw generator — that the Poisson path fits a Poisson process (KS vs Exponential, χ² goodness-of-fit on spike counts with the full upper tail, Fano factor, serial independence) and that the Gamma path is a genuine Gamma of the requested shape (KS fit, shape recovery, wrong-shape rejection). The same battery, run on the Gamma process, is the discriminating control.3. Optuna example parallelization (~5×)
examples/03_papers/watanabe/02now fans its 25 independent Optuna trials across worker processes sharing one SQLite study: ~2000 s → 139 s, equivalent optimum. Verified faithful (byte-identical sim/objective, deterministic per seed, independently audited).h5pyadded to thenwbextra so the NWB/HDF5 examples are fully declarable.Notes for reviewers
docs.ymldeploys via GitHub Actions, so repo Settings → Pages → Source must be set to "GitHub Actions" (the old workflow pushed to agh-pagesbranch).