Bulk document upload with ingestion wait (NDI-matlab#1047); ctest failure values (NDI-matlab#1046)

CHANGELOG.md

@@ -8,6 +8,20 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 
 ### Added
 
+- **Bulk document upload, and waiting for it to be ingested** (mirroring
+  VH-Lab/NDI-matlab#1047). `ndi.cloud.upload.uploadDocumentCollection` now
+  uploads documents in bulk, as NDI-matlab does: each chunk
+  (`Cloud.Upload.Max_Document_Batch_Count`) is one zipped JSON array
+  (new `zip_documents_for_upload`) PUT to `getBulkUploadURL`, instead of one
+  `addDocument` request per document, which for a dataset of 100,000
+  documents took hours. `NDI_CLOUD_UPLOAD_NO_ZIP=true` keeps the
+  one-at-a-time path. A bulk upload returns before the server has ingested
+  the zip and has no job to wait on, so afterwards it waits, with the new
+  `ndi.cloud.api.documents.waitForDocumentCount`, for the dataset to list the
+  uploaded documents (`wait_for_ingestion`, `ingestion_timeout`, default
+  300 s; a timeout is a warning, and the result is in `report["ingestion"]`).
+  `upload_type` now says `'batch'` or `'serial'` truthfully.
+
 - **Checking for, and installing, a newer NDI-python** (mirroring
   VH-Lab/NDI-matlab#1038). New `ndi.update`: `check()` compares this copy with
   the latest commit on main -- a git clone through `git fetch` (plus the

src/ndi/cloud/api/documents.py

@@ -10,10 +10,12 @@
 
 from __future__ import annotations
 
+import math
 import re
+import time
 from typing import Annotated, Any
 
-from pydantic import SkipValidation, validate_call
+from pydantic import Field, SkipValidation, validate_call
 
 from ..client import APIResponse, CloudClient, _auto_client
 from ._validators import VALIDATE_CONFIG, CloudId, FilePath, PageNumber, PageSize, Scope
@@ -182,6 +184,77 @@ def countDocuments(dataset_id: CloudId, *, client: _Client = None) -> int:
     return ds.get("documentCount", 0)
 
 
+@_auto_client
+@validate_call(config=VALIDATE_CONFIG)
+def waitForDocumentCount(
+    dataset_id: CloudId,
+    expected_count: Annotated[int, Field(ge=0)],
+    *,
+    timeout: float = 300.0,
+    initial_interval: float = 1.0,
+    max_interval: float = 10.0,
+    backoff_factor: float = 2.0,
+    client: _Client = None,
+) -> dict[str, Any]:
+    """Wait until a dataset lists at least *expected_count* documents.
+
+    Repeatedly calls :func:`countDocuments` at exponentially growing
+    intervals until the count is at least *expected_count* or the overall
+    timeout elapses. A failed poll is retried, not treated as the end of
+    the wait.
+
+    Use it after a bulk document upload: the upload returns once the zip is
+    stored, before the server has ingested the documents, so a listing made
+    straight away can miss some of them. Bulk document uploads have no job
+    id, so :func:`ndi.cloud.api.files.waitForAllBulkUploads` (which waits on
+    bulk FILE uploads) does not cover them.
+
+    Args:
+        dataset_id: Cloud dataset ID.
+        expected_count: The number of documents the dataset should list.
+        timeout: Overall deadline in seconds. Default 300.
+        initial_interval: First sleep between polls (s). Default 1.
+        max_interval: Cap on the per-poll sleep (s). Default 10.
+        backoff_factor: Multiplier applied after each poll. Default 2.
+
+    Returns:
+        Dict with ``state`` (``'complete'`` or ``'timeout'``), ``count``
+        (the last count read; ``nan`` if no poll succeeded),
+        ``expectedCount`` and ``elapsed`` (seconds). ``state == 'complete'``
+        is the success verdict.
+
+    MATLAB equivalent: +cloud/+api/+documents/waitForDocumentCount.m
+    """
+    start = time.monotonic()
+    interval = initial_interval
+    answer: dict[str, Any] = {
+        "state": "unknown",
+        "count": math.nan,
+        "expectedCount": expected_count,
+        "elapsed": 0.0,
+    }
+    while True:
+        try:
+            count = countDocuments(dataset_id, client=client)
+            if isinstance(count, (int, float)) and not isinstance(count, bool):
+                answer["count"] = count
+                if count >= expected_count:
+                    answer["state"] = "complete"
+                    answer["elapsed"] = time.monotonic() - start
+                    return answer
+        except Exception:
+            # A failed poll is transient: keep the last good count and try
+            # again rather than giving up on a gateway blip.
+            pass
+        elapsed = time.monotonic() - start
+        if elapsed >= timeout:
+            answer["state"] = "timeout"
+            answer["elapsed"] = elapsed
+            return answer
+        time.sleep(min(interval, max_interval, timeout - elapsed))
+        interval = min(interval * backoff_factor, max_interval)
+
+
 @_auto_client
 @validate_call(config=VALIDATE_CONFIG)
 def bulkFetch(

src/ndi/cloud/api/ndi_matlab_python_bridge.yaml

@@ -554,6 +554,49 @@ functions:
     decision_log: >
       Exact match. MATLAB also has documentCount.m which is a duplicate/alias.
 
+  - name: waitForDocumentCount
+    matlab_path: "+ndi/+cloud/+api/+documents/waitForDocumentCount.m"
+    matlab_last_sync_hash: "004585c"
+    python_path: "ndi/cloud/api/documents.py"
+    input_arguments:
+      - name: dataset_id
+        type_matlab: "string"
+        type_python: "CloudId"
+      - name: expected_count
+        type_matlab: "double"
+        type_python: "int"
+      - name: timeout
+        type_matlab: "double"
+        type_python: "float"
+        default: "300.0"
+      - name: initial_interval
+        type_matlab: "double"
+        type_python: "float"
+        default: "1.0"
+      - name: max_interval
+        type_matlab: "double"
+        type_python: "float"
+        default: "10.0"
+      - name: backoff_factor
+        type_matlab: "double"
+        type_python: "float"
+        default: "2.0"
+      - name: client
+        type_python: "_Client"
+        default: "None"
+    output_arguments:
+      - name: result
+        type_python: "dict[str, Any]"
+    decision_log: >
+      Ported from VH-Lab/NDI-matlab#1047 (matlab 004585c). Polls
+      countDocuments at exponentially growing intervals until the count is at
+      least expected_count or the timeout elapses; a failed poll is retried,
+      not fatal. Returns MATLAB's ANSWER struct (state 'complete' | 'timeout',
+      count -- nan if no poll succeeded -- expectedCount, elapsed); MATLAB's
+      leading b flag is state == 'complete', the same convention
+      waitForFileTierJob follows. WaitForDocumentCountMockTest ports over as
+      TestWaitForDocumentCount in tests/test_cloud_upload_bulk_documents.py.
+
   - name: bulkUpload
     matlab_path: "N/A"
     python_path: "ndi/cloud/api/documents.py"

src/ndi/cloud/ndi_matlab_python_bridge.yaml

@@ -884,7 +884,7 @@ functions:
   # --- upload.py functions ---
   - name: uploadDocumentCollection
     matlab_path: "+ndi/+cloud/+upload/uploadDocumentCollection.m"
-    matlab_last_sync_hash: "7180cc8"
+    matlab_last_sync_hash: "004585c"
     python_path: "ndi/cloud/upload.py"
     input_arguments:
       - name: dataset_id
@@ -900,6 +900,14 @@ functions:
       - name: max_chunk
         type_python: "int | None"
         default: "None"
+      - name: wait_for_ingestion
+        type_matlab: "logical"
+        type_python: "bool"
+        default: "True"
+      - name: ingestion_timeout
+        type_matlab: "double"
+        type_python: "float"
+        default: "300.0"
       - name: client
         type_python: "CloudClient | None"
         default: "None"
@@ -908,6 +916,24 @@ functions:
         type_python: "dict[str, Any]"
     decision_log: >
       Exact match.
+      2026-10-23: synchronized with NDI-matlab 004585c (VH-Lab/NDI-matlab#1047).
+      Until now this entry said "Exact match" but the port only had MATLAB's
+      SERIAL path: every document went up as its own addDocument request,
+      whatever NDI_CLOUD_UPLOAD_NO_ZIP said -- one request per document, hours
+      for a 100,000-document dataset. It now has MATLAB's default BATCH path:
+      each chunk is one JSON array zipped by zip_documents_for_upload and PUT
+      to getBulkUploadURL; NDI_CLOUD_UPLOAD_NO_ZIP=true keeps the serial path,
+      as in MATLAB. #1047's ingestion wait comes with it: a bulk upload
+      returns once the zip is stored, not ingested, and has no job id, so the
+      batch path reads countDocuments before uploading and afterwards calls
+      waitForDocumentCount for that count plus the documents in the stored
+      batches (wait_for_ingestion / ingestion_timeout mirror MATLAB's
+      waitForIngestion / ingestionTimeout). A timeout logs a warning and
+      leaves status alone; the result is in report["ingestion"].
+      upload_type now reports 'batch' or 'serial' truthfully. Tests that model
+      the transport by patching addDocument run on the serial path through the
+      serial_document_upload fixture; the batch path is tested in
+      tests/test_cloud_upload_bulk_documents.py.
       2026-10-17: NDI-matlab 7180cc8 (VH-Lab/NDI-matlab#1033) makes maxDocumentChunk default to the preference
       Cloud.Upload.Max_Document_Batch_Count (was Inf). max_chunk=None now
       reads the same preference; 0 or negative still means all at once.
@@ -1877,10 +1903,34 @@ porting_deferred:
   - name: zip_documents_for_upload
     matlab_path: "+ndi/+cloud/+upload/+internal/zip_documents_for_upload.m"
     matlab_last_sync_hash: "c6aef552"
-    status: ported_differently
+    python_path: "ndi/cloud/upload.py"
+    input_arguments:
+      - name: documents
+        type_matlab: "cell"
+        type_python: "list[dict[str, Any]]"
+      - name: dataset_id
+        type_matlab: "char"
+        type_python: "str"
+      - name: target_folder
+        type_matlab: "string"
+        type_python: "str | Path | None"
+        default: "None"
+    output_arguments:
+      - name: zip_path
+        type_python: "Path"
+      - name: id_manifest
+        type_python: "list[str]"
     decision_log: >
-      MATLAB internal helper. Python uses zipForUpload which handles
-      this internally with zipfile module.
+      2026-10-23: ported (was ported_differently). The old note said
+      zipForUpload covered this; it does not -- zipForUpload zips binary FILES
+      for getFileCollectionUploadURL, and nothing zipped DOCUMENTS, which is
+      why uploadDocumentCollection had no bulk path. Same archive as MATLAB:
+      one .json file holding the documents as a JSON array, encoded with
+      did.datastructures.json_encode_nan (MATLAB's jsonencodenan), the zip
+      named <dataset_id>.<random>.zip. Lives in ndi.cloud.upload rather than
+      an internal subpackage, next to its one caller. Python's own zip entry
+      name is random hex rather than ndi.file.temp_name's; the server reads
+      the archive's single .json, whatever it is called.
   # =======================================================================
   # +ndi/+cloud/+ui - MATLAB's cloud dialogs.
   #