[stevevanhooser]

Summary

Two ports from NDI-matlab, plus a fix for the lightsheet test fixture that main's nightly cloud run has been failing on.

1. Upload documents in bulk and wait for them to be ingested (VH-Lab/NDI-matlab#1047)

uploadDocumentCollection's bridge entry said "Exact match", but the port only had MATLAB's serial path: every document went up as its own addDocument request, whatever NDI_CLOUD_UPLOAD_NO_ZIP said. That is one request per document, which for a 100,000-document dataset takes hours.

• Bulk upload by default. Each chunk (Cloud.Upload.Max_Document_Batch_Count) is one JSON array, zipped by the new zip_documents_for_upload and PUT to getBulkUploadURL.
  • The archive matches MATLAB's: a single .json holding the documents' JSON array, encoded with did's json_encode_nan.
  • NDI_CLOUD_UPLOAD_NO_ZIP=true keeps the serial path, as in MATLAB.
  • upload_type now reports 'batch' or 'serial' truthfully.
• New ndi.cloud.api.documents.waitForDocumentCount. A bulk upload returns once the zip is stored, before the server has ingested it, and has no job id to wait on. A listing made straight afterwards can therefore miss documents; that's how Cloud CI 1 on NDI-matlab main failed.
  • It polls countDocuments with backoff until the count reaches a target or the timeout elapses. A failed poll is retried.
  • It returns state, count, expectedCount and elapsed.
• The batch path waits for ingestion. It reads the count before uploading, then waits for it to grow by the number of documents in the stored batches.
  • wait_for_ingestion / ingestion_timeout (300 s) mirror MATLAB's options.
  • A timeout logs a warning and leaves status alone. The result is in report["ingestion"].
• Bridge.
  • waitForDocumentCount added.
  • uploadDocumentCollection synced to 004585c.
  • zip_documents_for_upload is now ported. It had been recorded as covered by zipForUpload, which zips files, not documents.
• CHANGELOG entry under Unreleased → Added.

2. ctest.reportSummary says how far out of tolerance each field was (VH-Lab/NDI-matlab#1046)

• Each out-of-tolerance field now gets a line with:
  • its actual value,
  • its expected value,
  • their absolute difference (when both are numeric and the same shape),
  • the comparison's comment.
• New static valueString renders values the way MATLAB's mat2str(v,6) does. Text is quoted, anything else is shown as its size and type, and the result is cut at 200 characters.
• compare_mock_docs now also carries DocComparison's value1/value2/detail under MATLAB's names (actualValue/expectedValue/comment), which reportSummary reads.
• Bridge: ctest synced to 0374dd2, and valueString added. This clears the test_matlab_bridge_hashes drift failure that was already on main.

3. Point the lightsheet pyramid tests at the rebuilt fixture

• test_the_coarsest_level_computes_and_is_not_all_zero has failed in the User 1 / prod Cloud API cell since Oct 20, on main and on this PR.
• The coarsest pyramid level of fixture 6accf3a0044c77d93a6ebdb7 read entirely as fill_value. That fits its chunk files having been removed from the cloud while its documents stayed.
• The fixture was rebuilt today with ndi.test.cloud.lightsheet_blob_cloud_roundtrip as test user 1 on prod, as 6adb9d501dabe363f1268c3f. tests/fixtures/lightsheet_cloud_fixture.json now points at it.

Tests

• New tests/test_cloud_upload_bulk_documents.py (20 tests) covers:
  • the zip format;
  • the batch path: chunking, a partial failure, cleanup, the wait's target, a timeout, no starting count, and the wait turned off;
  • the serial switch;
  • waitForDocumentCount's poll logic, ported from WaitForDocumentCountMockTest.m.
• Sync tests that model the transport by patching addDocument (26 tests) run on the serial path through a new serial_document_upload fixture. Their assertions are unchanged.
• TestReportSummarySaysHowFarOut (8 tests) ports TestCtestReportSummary.m, plus a check that compare_mock_docs' report carries the values.

Validation

• black --check src/ tests/ and ruff check src/ tests/: clean.
• pytest tests/ locally (with a full-history NDI-matlab checkout at 004585c): 6130 passed, 0 failed.
• CI on 912ff61: all 19 checks pass, including all four live Cloud API cells (users 1 and 2, prod and dev) and the bridge, symmetry and Windows jobs.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KChjwyZT9314dmEuS5ovUP

[stevevanhooser]

Cloud API (User 1, prod) fails on one test, tests/test_pyramid_loader_integration.py::TestLoaderReadsRealBytes::test_the_coarsest_level_computes_and_is_not_all_zero ("coarsest-level channel-0 slice is entirely fill_value"). The job's other 803 tests pass, including the upload tests.

This failure isn't caused by this PR:

• The test downloads the permanent lightsheet fixture and reads its pyramid. It doesn't touch uploadDocumentCollection, waitForDocumentCount or ctest.
• The same assertion fails in main's nightly Test Cloud API run, on the same User 1 / prod cell. It failed last night (run 39894603671), and that workflow has been red on main since Oct 20.

Update: the cause was the fixture itself. The coarsest level of 6accf3a0044c77d93a6ebdb7 read entirely as fill_value, which fits its chunk files having been removed from the cloud while the documents stayed. The fixture was rebuilt today with ndi.test.cloud.lightsheet_blob_cloud_roundtrip as test user 1 on prod, as 6adb9d501dabe363f1268c3f. Commit 912ff61 points tests/fixtures/lightsheet_cloud_fixture.json at it, and that also fixes main's nightly once this PR merges.

---

Generated by Claude Code