[stevevanhooser]

Ports VH-Lab/NDI-matlab#1089, #1090 and #1091, all of which are merged. CI checks against NDI-matlab main and installs ndi-compress from NDI-compress-python main, which now has Waltham-Data-Science/NDI-compress-python#4.

Why

Python ingestion made each epoch's documents without any data. After a Python ingest, reading an ingested epoch failed with channel_list.bin not found. NDI-matlab leads, so this ports three things from it: its writer, its segment-level parallel scheduler, and its raw-vs-ingested comparison tool.

What

• ndi_daq_reader_mfdaq ingestion (NDI-matlab 6ff0264e5, 569fdab67):
  • writes the channel list (channel_list.bin, with each channel's sample rate, offset and scale);
  • writes the data in compressed segments: 1e6 samples for analog/time, 1e7 for digital;
  • stores analog data in the reader's underlying type, and stores event/marker/text channels too;
  • is split, as in MATLAB, into ingest_epochfiles_plan, ingest_segment and the static ingest_attach_segments. ingest_epochfiles(..., ProgressFcn=) runs the three in sequence.
• ingestEpochDocuments (b192ae605, 569fdab67):
  • On workers, every epoch is planned in the calling process. Each segment, and each epoch's metadata readers, is one task for a spawn ProcessPoolExecutor, so every worker stays busy even with one long epoch.
  • Segments are attached in plan order, so the documents match a serial ingest.
  • ProgressFcn's numDone is fractional, counted in the calling process from the plan weights.
• ndi.test.ingest.compare / mfdaq_compare (#1091) compare a raw session with an ingested copy of it:
  • checks every epoch, the channel list, every sample of every continuous type (with RelativeTolerance and TimeTolerance), and every event;
  • returns a report with one entry per check and never stops on a mismatch.
• Intan reader:
  • underlying_datatype gives uint16, as in MATLAB;
  • aux inputs are read at their own rate;
  • digital channels are requested as NDR numbers them (before, digital reads always failed);
  • reads past the end of the file are no longer padded.
• Metadata readers store their parameters as data.bin, and readmetadata_ingested reads them back.
• mfdaq_epoch_channel.writeToFile writes MATLAB's saveStructArray format.

Differences from MATLAB (in the bridge decision logs)

• Digital-output segments are written as do_group*. MATLAB writes di_group*, which collides with digital inputs of the same group; this looks like a MATLAB bug.
• Float underlying data are stored as floats. MATLAB's datatypestring rejects 'float64'.
• The time channel computes its own sample range.
• In the compare tool, a channel-list field is compared only when the raw reader set it. Python's ChannelInfo always has every field, whereas MATLAB compares only the fields both lists share.

Bridge

• mfdaq_reader is synced to NDI-matlab 5c5784c6a, with entries for the three new methods.
• ingestEpochDocuments is synced to 569fdab67.
• These were the last drifted entries, so with this PR main passes the bridge checks again. The fun/epoch and setup notes are Update button: package table with tick boxes; bridge catch-up with NDI-matlab main #347's, kept in the merge.

Testing

• tests/test_ingest_data_writer.py (10 tests) ingests the example Intan session and reads every channel type back, comparing against the raw reads. This includes epochs split across several segments and windows that cross segment boundaries.
• tests/test_ingest_compare.py (8 tests):
  • mirrors NDI-matlab's ingestCompareTest;
  • a corrupted sample must be reported;
  • a two-worker parallel ingest with many segments must compare equal;
  • mirrors MATLAB's plan/segment/attach equivalence test.
• Full suite locally after merging main, against NDI-matlab main: 6525 passed, 352 skipped, 0 failed. CI on 21716f8 is green, bridge completeness included.

🤖 Generated with Claude Code

https://claude.ai/code/session_011v7R5rnKYwncJkrNZRXhtw

[stevevanhooser]

MATLAB/Python bridge completeness fails here on the same four files as #345: +ndi/+gui/+nav/updateDialog.m and three under +ndi/+update (packageStatus.m, private/findCalculators.m, private/pullRepo.m). This PR doesn't touch them; see the comment on #345. Every entry this PR adds or changes passes, including against NDI-matlab's paired claude/ingest-within-epoch-progress branch. No fix exists yet, and it belongs in its own small bridge PR.

---

Generated by Claude Code

[stevevanhooser]

test (3.11), test (3.12) and MATLAB ↔ Python symmetry tests fail with one error: compress_ephys() got an unexpected keyword argument 'output_size_bits'.

• On 30435c0, each test job has 18 failures: the ingest, ingest-compare and session-ingest tests. The symmetry job has 3, the Intan, Intan-NDR and Axon-NDR ingestion artifacts.
• CI installs ndi-compress from NDI-compress-python main (see pyproject.toml). The format options this PR needs exist only in compress_ephys: pass the data format (unsigned, 8/32/64-bit, float) NDI-compress-python#4, which is not merged yet.
• the function ndi.cloud.downloadDataset is not present #4 now has binaries for all three platforms, built by its workflow, and is green.
• Locally, with the function ndi.cloud.downloadDataset is not present #4 installed, all these tests pass. The full suite there: 6487 passed.

Once #4 merges, re-running CI here should leave only the bridge-completeness failures, which #347 fixes.

---

Generated by Claude Code

[stevevanhooser]

MATLAB/Python bridge completeness is red on 30435c0 for reasons outside this PR. The checks for every entry this PR touches (daq, daq/fun, fun, setup) pass.

The two failures are entries missing for MATLAB's update dialog (VH-Lab/NDI-matlab 19b8ab573 and 749b1a266):

• +ndi/+gui/+nav/updateDialog.m
• +ndi/+update/packageStatus.m, private/findCalculators.m and private/pullRepo.m

main fails the same way against NDI-matlab main. The fix is #347: it ports the update dialog itself, adds these entries, and brings the +ndi/+update and ndiPane hashes up to 749b1a266. I haven't copied that port into this ingestion PR, because it's a separate feature. This check goes green once #347 merges and this branch takes in its base. I haven't re-run the job, since a re-run would fail the same way.

---

Generated by Claude Code