Lightsheet and gene-pyramid viewers: one dask layer per level; store fetched tiles without scanning DID's cache index

CHANGELOG.md

@@ -118,6 +118,27 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 
 ### Fixed
 
+- **Lightsheet viewer: slow tiles from the local cache.** Each pyramid level was
+  built as one dask graph layer per chunk (`from_delayed` + `da.block`), and
+  dask walks every layer each time napari computes a slice, so the cost grew
+  with the size of the level rather than the view: 16 cached chunks took ~1.4 s
+  to show on a 16k-chunk level and ~26 s on a 131k-chunk one, against ~15 ms to
+  read the files. This is the "Loading view 50/112" with nothing in flight.
+  A level is now a single `map_blocks` layer: the same view takes ~20 ms at
+  any level size, and building the level at open time is ~20x faster.
+  The viewer also now prints one `[lightsheet] tiles:` line per burst of reads
+  showing where tile time goes (path lookup, queue wait, disk read, decode,
+  and the burst's wall-clock span); `NDI_LIGHTSHEET_TIMING=0` turns it off.
+  The gene-expression pyramid viewer built its levels the same way and gets
+  the same fix.
+  Downloaded tiles also took about a minute each to be stored in DID's file
+  cache: with dozens of download threads finishing together, each scanned the
+  cache's index row by row and waited on its lock a second at a time. The
+  viewer now checks the disk instead of the index and lets one thread at a
+  time add to the cache; the debug fetch line reports the add as
+  `cache add Xs`. The underlying cache fixes are VH-Lab/DID-python#89 and
+  VH-Lab/DID-matlab#221.
+
 - **Saving preferences no longer drops NDI-matlab's own entries.**
   `~/.ndi/NDI_Preferences.json` is shared with NDI-matlab, and each side
   rewrote it with only the preferences it registers, so saving from Python

src/ndi/cloud/api/files.py

@@ -260,8 +260,11 @@ def _download_session():
     connection across sibling ``getFile`` calls. Every tile of a
     pyramid goes to the same S3 host, so a shared Session amortises
     the TLS handshake (~100-500ms) across N chunks. Thread-safe for
-    GET; the default pool holds 10 connections which is comfortable
-    for the lightsheet fetcher's 8 workers.
+    GET. The pool holds ``_DOWNLOAD_POOL_SIZE`` connections per host:
+    the lightsheet fetcher runs up to ``ndi.pyramid.multiscale.MAX_WORKERS``
+    (64) downloads at once, and with requests' default of 10 every
+    download past the tenth opened a connection of its own and threw it
+    away afterwards ("Connection pool is full, discarding connection").
 
     A previous version opened a fresh connection per call
     (``requests.get(url, ...)``) which added handshake latency to
@@ -271,11 +274,23 @@ def _download_session():
 
     session = getattr(_download_session, "_session", None)
     if session is None:
+        from requests.adapters import HTTPAdapter
+
         session = requests.Session()
+        adapter = HTTPAdapter(
+            pool_connections=_DOWNLOAD_POOL_SIZE, pool_maxsize=_DOWNLOAD_POOL_SIZE
+        )
+        session.mount("https://", adapter)
+        session.mount("http://", adapter)
         _download_session._session = session
     return session
 
 
+#: Connections kept per host by the download session; matches the most
+#: downloads ndi.pyramid.multiscale.MAX_WORKERS runs at once.
+_DOWNLOAD_POOL_SIZE = 64
+
+
 def getFile(
     url: NonEmptyStr,
     target_path: str | Path,

src/ndi/gui/app/genepyramid/multiscale.py

@@ -275,7 +275,9 @@ def levelArrays(session, pyr_doc, gene_rows=None, density: bool = True) -> list[
     Returns:
         A list of dask arrays, finest first.
     """
-    dask, da = _require_dask()
+    _dask, da = _require_dask()
+    from dask.base import tokenize
+
     levels, _frame = levelTable(session, pyr_doc)
     fetcher = _TileFetcher(session)
 
@@ -303,8 +305,7 @@ def levelArrays(session, pyr_doc, gene_rows=None, density: bool = True) -> list[
         # The block being lazy never helped, because the cost was already
         # paid by the time dask chose its blocks.
         def block(
-            r,
-            c,
+            block_info=None,
             _fetch=fetcher,
             _doc=tile_doc,
             _cols=cols,
@@ -315,6 +316,7 @@ def block(
             _name=tileFileName,
             _lv=lv_props,
         ):
+            r, c = block_info[None]["chunk-location"]
             name = _name(_lv, r * _cols + c)
             if name not in _stored:
                 return np.zeros((_th, _tw), np.float32)  # missing tile costs nothing
@@ -330,21 +332,32 @@ def block(
                 tile = readTileFile(_fetch.tilePath(_doc, name))
             return renderTile(tile, gene_rows, _th, _tw, binSize=_scale)
 
-        grid = [
-            [
-                da.from_delayed(dask.delayed(block)(r, c), shape=(th, tw), dtype=np.float32)
-                for c in range(cols)
-            ]
-            for r in range(rows)
-        ]
+        # ONE graph layer per level, one block per tile. Building the level
+        # as one from_delayed array per tile joined by da.block gave dask one
+        # layer per tile, and dask walks every layer each time napari
+        # computes a slice -- so drawing a view cost time in proportion to
+        # the whole level, not to the tiles on screen (seconds on a level of
+        # tens of thousands of tiles). The name is explicit: tokenizing the
+        # closure would hash the fetcher, and id(fetcher) keeps two viewers
+        # of the same pyramid from sharing keys.
+        token = tokenize(
+            getattr(tile_doc, "id", str(tile_doc)), b, gene_rows, density, th, tw, id(fetcher)
+        )
+        level = da.map_blocks(
+            block,
+            chunks=((th,) * rows, (tw,) * cols),
+            dtype=np.float32,
+            meta=np.empty((0, 0), np.float32),
+            name=f"ndi-genepyramid-level-{token}",
+        )
         # Crop to the level's TRUE size. The assembled grid is
         # rows*th by cols*tw, which overshoots whenever the level does not
         # divide evenly by the grid. Leaving the padding on makes a level
         # cover more ground than the extent it claims, so the levels of one
         # pyramid disagree about their own field of view and a viewer
         # registering them by shape ratio misaligns them -- worse at coarse
         # levels, where the rounding is proportionally larger.
-        out.append(da.block(grid)[:lh, :lw])
+        out.append(level[:lh, :lw])
 
     return out