Skip to content

Ingest: progress bars and parallel per-epoch work (mirrors NDI-matlab#1079) - #345

Merged
stevevanhooser merged 3 commits into
mainfrom
claude/ingest-progress-parallel
Oct 31, 2026
Merged

stevevanhooser merged 3 commits into
mainfrom
claude/ingest-progress-parallel

Conversation

@stevevanhooser

Copy link
Copy Markdown
Contributor

Mirrors VH-Lab/NDI-matlab#1079 (merged).

What

  • Progress bars: ndi_session.ingest / ndi_daq_system.ingest report progress through ndi_gui_component_ProgressBarWindow, with one session bar and one bar per DAQ system.
    • On by default; ShowProgress=False turns it off.
    • ProgressBar= / ProgressBarTag= reuse an existing window.
  • Parallel per-epoch work: the per-epoch reader and metadata-reader work moves to the new ndi.daq.fun.ingestEpochDocuments. It runs on spawned worker processes when the Parallel.* preferences allow.
    • Database writes stay serial and in epoch order.
    • Readers are sent to workers with _session cleared; a reader that can't be pickled falls back to serial with a warning.
  • Supporting ports:
    • ProgressBarWindow gets MATLAB's silent (headless) mode (Silent, isHeadless, silentModeDefault, resolveSilentMode) and per-bar Timeout.
    • New Parallel.OpenPoolIfNone / UseDefaultNumberOfWorkers / NumberOfWorkers preferences and ndi.fun.parallelWorkers. Python has no persistent pool, so the default (OpenPoolIfNone=false) is serial.
  • Bridge entries are updated for ndi.daq.system, ndi.session, ndi.preferences, ndi.fun.parallelWorkers and ProgressBarWindow, and added for ndi.daq.fun.ingestEpochDocuments.
  • Adds tests/test_ingest_progress_parallel.py (20 tests).

Notes


Generated by Claude Code

…#1079)

ndi_session.ingest / ndi_daq_system.ingest report progress through
ndi_gui_component_ProgressBarWindow (one session bar, one bar per DAQ
system; ShowProgress/ProgressBar/ProgressBarTag), and the per-epoch
reader/metadata-reader work moves to ndi.daq.fun.ingestEpochDocuments,
which runs it on spawned worker processes when the Parallel.* preferences
allow. Database writes stay serial and in epoch order. Readers are sent
to workers with _session cleared; an unpicklable reader falls back to
serial with a warning.

Supporting ports:
- ProgressBarWindow: MATLAB's silent (headless) mode (Silent,
  isHeadless, silentModeDefault, resolveSilentMode) and per-bar Timeout.
- Parallel.OpenPoolIfNone / UseDefaultNumberOfWorkers / NumberOfWorkers
  preferences and ndi.fun.parallelWorkers. Python has no persistent pool,
  so the default (OpenPoolIfNone=false) is serial.

Bridge entries updated for ndi.daq.system, ndi.session, ndi.preferences,
ndi.fun.parallelWorkers (matlab_only -> ported) and ProgressBarWindow, and
added for ndi.daq.fun.ingestEpochDocuments; Silent leaves the property
allowlist. Adds tests/test_ingest_progress_parallel.py.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011v7R5rnKYwncJkrNZRXhtw

Copy link
Copy Markdown
Contributor Author

MATLAB/Python bridge completeness fails, but not because of this PR.

It reports four NDI-matlab files with no bridge entry:

  • +ndi/+gui/+nav/updateDialog.m
  • +ndi/+update/packageStatus.m
  • +ndi/+update/private/findCalculators.m
  • +ndi/+update/private/pullRepo.m

These were added on NDI-matlab main with the update-dialog work (e.g. 749b1a266). They aren't touched here, and main here has been red since #344. Every daq, session, preferences and ProgressBarWindow entry this PR changes passes.

No fix exists yet. The fix is bridge entries for those four files in src/ndi/gui and src/ndi/update (ported, or not_yet_ported with a decision log). That belongs in its own PR rather than this one. Re-running won't help: the result is deterministic.


Generated by Claude Code

Resolves the CHANGELOG conflict with #347 by keeping both entries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011v7R5rnKYwncJkrNZRXhtw

Copy link
Copy Markdown
Contributor Author

I merged main into this branch (633b17f). The only conflict was in CHANGELOG.md, where #347 and this PR each added an entry; both are kept.

MATLAB/Python bridge completeness will still be red, now for a different reason. #347 fixed the update-dialog entries. What remains is hash drift on two entries:

  • ingestEpochDocuments
  • mfdaq_reader

NDI-matlab #1090 has since changed both MATLAB files. The port of #1090 is in #346, which is stacked on this PR and moves both entries to NDI-matlab main.

Locally, against NDI-matlab main, everything else passes: 6502 passed, 352 skipped. The checks go green once #346 merges after this one.


Generated by Claude Code

@stevevanhooser
stevevanhooser merged commit 0898f8a into main Oct 31, 2026
14 of 15 checks passed
@stevevanhooser
stevevanhooser deleted the claude/ingest-progress-parallel branch October 31, 2026 18:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants