Repository navigation
Ingest: progress bars and parallel per-epoch work (mirrors NDI-matlab#1079) - #345
Conversation
…#1079) ndi_session.ingest / ndi_daq_system.ingest report progress through ndi_gui_component_ProgressBarWindow (one session bar, one bar per DAQ system; ShowProgress/ProgressBar/ProgressBarTag), and the per-epoch reader/metadata-reader work moves to ndi.daq.fun.ingestEpochDocuments, which runs it on spawned worker processes when the Parallel.* preferences allow. Database writes stay serial and in epoch order. Readers are sent to workers with _session cleared; an unpicklable reader falls back to serial with a warning. Supporting ports: - ProgressBarWindow: MATLAB's silent (headless) mode (Silent, isHeadless, silentModeDefault, resolveSilentMode) and per-bar Timeout. - Parallel.OpenPoolIfNone / UseDefaultNumberOfWorkers / NumberOfWorkers preferences and ndi.fun.parallelWorkers. Python has no persistent pool, so the default (OpenPoolIfNone=false) is serial. Bridge entries updated for ndi.daq.system, ndi.session, ndi.preferences, ndi.fun.parallelWorkers (matlab_only -> ported) and ProgressBarWindow, and added for ndi.daq.fun.ingestEpochDocuments; Silent leaves the property allowlist. Adds tests/test_ingest_progress_parallel.py. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011v7R5rnKYwncJkrNZRXhtw
…nly) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011v7R5rnKYwncJkrNZRXhtw
|
It reports four NDI-matlab files with no bridge entry:
These were added on NDI-matlab No fix exists yet. The fix is bridge entries for those four files in Generated by Claude Code |
Resolves the CHANGELOG conflict with #347 by keeping both entries. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011v7R5rnKYwncJkrNZRXhtw
|
I merged MATLAB/Python bridge completeness will still be red, now for a different reason. #347 fixed the update-dialog entries. What remains is hash drift on two entries:
NDI-matlab #1090 has since changed both MATLAB files. The port of #1090 is in #346, which is stacked on this PR and moves both entries to NDI-matlab main. Locally, against NDI-matlab main, everything else passes: 6502 passed, 352 skipped. The checks go green once #346 merges after this one. Generated by Claude Code |
Mirrors VH-Lab/NDI-matlab#1079 (merged).
What
ndi_session.ingest/ndi_daq_system.ingestreport progress throughndi_gui_component_ProgressBarWindow, with one session bar and one bar per DAQ system.ShowProgress=Falseturns it off.ProgressBar=/ProgressBarTag=reuse an existing window.ndi.daq.fun.ingestEpochDocuments. It runs on spawned worker processes when theParallel.*preferences allow._sessioncleared; a reader that can't be pickled falls back to serial with a warning.ProgressBarWindowgets MATLAB's silent (headless) mode (Silent,isHeadless,silentModeDefault,resolveSilentMode) and per-barTimeout.Parallel.OpenPoolIfNone/UseDefaultNumberOfWorkers/NumberOfWorkerspreferences andndi.fun.parallelWorkers. Python has no persistent pool, so the default (OpenPoolIfNone=false) is serial.ndi.daq.system,ndi.session,ndi.preferences,ndi.fun.parallelWorkersandProgressBarWindow, and added forndi.daq.fun.ingestEpochDocuments.tests/test_ingest_progress_parallel.py(20 tests).Notes
mainhas been red since Navigator: DAQ system browser, cloud dataset id, Assign to workspace; faster daqsystem_load #344 on bridge drift from NDI-matlab main (+ndi/+update,+ndi/+gui/+nav,+ndi/+setup,+ndi/+fun/+epoch). That drift is unrelated to this change and will show here too.Generated by Claude Code