diff --git a/CITATION.cff b/CITATION.cff new file mode 100644 index 0000000..24f7b69 --- /dev/null +++ b/CITATION.cff @@ -0,0 +1,64 @@ +cff-version: 1.2.0 +message: If you use this software, please cite the JOSS paper below. +title: 'PixelMap: A Browser-Based Tool for Wiring-Aware, Anatomy- and Activity-Guided Design of Neuropixels + Channelmaps' +type: software +authors: +- given-names: Maxime + family-names: Beau + orcid: https://orcid.org/0000-0002-8907-6612 +- given-names: Julie M. J. + family-names: Fabre + orcid: https://orcid.org/0000-0003-0550-0410 +- given-names: Christian + family-names: Tabedzki + orcid: https://orcid.org/0000-0001-8409-6094 +- given-names: Jorge + family-names: Yanar + orcid: https://orcid.org/0000-0003-1416-3567 +- given-names: Carlos D. + family-names: Brody + orcid: https://orcid.org/0000-0002-4201-561X +version: 1.4.0 +date-released: '2026-09-19' +doi: 10.5281/zenodo.22850328 +identifiers: +- type: doi + value: 10.5281/zenodo.22850327 + description: Zenodo concept DOI (all versions) +- type: doi + value: 10.5281/zenodo.22850328 + description: Zenodo archive of the v1.4.0 JOSS release +- type: doi + value: 10.21105/joss.11008 + description: JOSS paper +repository-code: https://github.com/m-beau/pixelmap +url: https://pixelmap.pni.princeton.edu +license: GPL-3.0-only +preferred-citation: + type: article + title: 'PixelMap: A Browser-Based Tool for Wiring-Aware, Anatomy- and Activity-Guided Design of + Neuropixels Channelmaps' + authors: + - given-names: Maxime + family-names: Beau + orcid: https://orcid.org/0000-0002-8907-6612 + - given-names: Julie M. J. + family-names: Fabre + orcid: https://orcid.org/0000-0003-0550-0410 + - given-names: Christian + family-names: Tabedzki + orcid: https://orcid.org/0000-0001-8409-6094 + - given-names: Jorge + family-names: Yanar + orcid: https://orcid.org/0000-0003-1416-3567 + - given-names: Carlos D. + family-names: Brody + orcid: https://orcid.org/0000-0002-4201-561X + journal: Journal of Open Source Software + publisher: + name: Open Journals + year: 2026 + start: 11008 + doi: 10.21105/joss.11008 + url: https://doi.org/10.21105/joss.11008 diff --git a/docker-compose.yml b/docker-compose.yml index 755f733..83f11d7 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -18,6 +18,11 @@ services: - ANALYTICS_COOKIE_MAX_AGE_DAYS=365 - ANALYTICS_COOKIE_SAMESITE=Lax - ANALYTICS_COOKIE_SECURE=false + # Log a "[watchdog]" WARNING (with the blocked stack) when the event + # loop is stuck this long, and once when RSS crosses this many MB. See + # pixelmap/gui/watchdog.py. Set PIXELMAP_STALL_WARN_S=0 to disable. + - PIXELMAP_STALL_WARN_S=5 + - PIXELMAP_RSS_WARN_MB=3000 volumes: - ./output:/app/output - brainglobe_cache:/root/.brainglobe diff --git a/paper/paper.md b/paper/paper.md index 6a67072..13c8af1 100644 --- a/paper/paper.md +++ b/paper/paper.md @@ -29,9 +29,9 @@ authors: orcid: 0000-0002-4201-561X affiliation: "1, 2" affiliations: - - name: Princeton Neuroscience Institute, Princeton University, USA + - name: Princeton Neuroscience Institute, Princeton University, United States of America index: 1 - - name: Howard Hughes Medical Institute, USA + - name: Howard Hughes Medical Institute, United States of America index: 2 date: 29 May 2026 bibliography: paper.bib @@ -52,7 +52,7 @@ Beyond satisfying hardware constraints, channelmaps must be tailored to the expe 1. **Being available on any machine installation-free** at [https://pixelmap.pni.princeton.edu](https://pixelmap.pni.princeton.edu). 2. **Visualising wiring constraints in real time**: When users select electrodes (red), the interface immediately shows which become unavailable (black) due to shared lines or ADCs, preventing invalid configurations. 3. **Supporting arbitrary electrode geometries** through 1) common preset geometries, 2) entering electrode ranges as text for reproducibility, 3) directly clicking or dragging on the probe visualisation, or 4) loading pre-existing `.imro` files. These four selection methods are compatible with one another and meant to be combined. -4. **Guiding channelmap design with anatomy and activity overlays**: Users can overlay anatomical region boundaries from any brain atlas available through BrainGlobe [@claudi2020] to plan recordings around specific brain regions of interest, and **simultaneously** a SpikeGLX activity survey heatmap to identify the most active channels and confirm anatomical predictions after probe implantation. +4. **Guiding channelmap design with anatomy and activity overlays**: Users can overlay anatomical region boundaries from any brain atlas available through BrainGlobe [@claudi2020] to plan recordings around specific brain regions of interest. They can also **simultaneously** overlay a SpikeGLX activity survey heatmap to identify the most active channels and confirm anatomical predictions after probe implantation. | Probe Version | Physical Electrodes | Readout Channels | |---------------|---------------------|------------------| @@ -94,6 +94,28 @@ PixelMap's architecture is designed to easily support future hardware and atlase **Contributor extensibility.** The codebase is written as comprehensively as possible with contributions in mind. New selection presets can be added to `backend.py`, new overlay types can follow the pattern established by the activity-survey and anatomy overlays, and the GUI layer can be extended or replaced without modifying the core constraint logic. This modular design, together with comprehensive tests and a [contributor guide](https://pixelmap-neuropixels.readthedocs.io/en/latest/development.html), is intended to make contribution straightforward. These design choices have already enabled two new contributors to add significant features in 2026, warranting authorship on this paper: the anatomical overlay (J.M.J.F.) and the activity survey overlay (J.Y.). +```{=latex} +\makeatletter +% JOSS's template forces every figure to [H] (float package), which wraps the +% figure in a single unbreakable box. Figure 1's caption is long, so that box +% does not fit in the space left on the page and LaTeX pushes the whole figure +% to the next page, leaving a large gap. Contribute the figure to the main +% vertical list instead, and typeset the caption as an ordinary paragraph, so +% the caption can break across pages. PDF only: the markdown is untouched, so +% the JATS/HTML outputs are unaffected. +\renewenvironment{figure}[1][]{% + \par\addvspace{\intextsep}% + \def\@captype{figure}% +}{% + \par\addvspace{\intextsep}% +} +\long\def\@makecaption#1#2{% + \vskip\abovecaptionskip\nobreak + \begingroup\captionfont\noindent{\captionlabelfont #1:} #2\par\endgroup + \vskip\belowcaptionskip} +\makeatother +``` + ![PixelMap's browser-based graphical user interface, shown here for a four-shank Neuropixels 2.0 probe.\ **Left:** *Probe and recording metadata* panel (these metadata are also written into `.imro` files), followed by three of the four methods of electrode selection: *Preset Selection* (preset geometries), *Textual Selection* (manual input of electrode ranges), and *Selection from IMRO file* (pre-loading an existing `.imro` file). These three methods can be mixed with each other and with the five interactive click-and-drag box tools, whose toolbar sits along the right edge of the central panel. Controls for the activity survey and anatomical overlays are further down this panel, below the cropped view.\ **Centre:** main panel featuring the probe's physical layout with one or four shanks that exhibit the 960 (Neuropixels 1.0) or 1,280 (Neuropixels 2.0, Quad Base) physical electrodes per shank to be selected from. Electrodes available for selection are light grey, selected electrodes turn red, and electrodes that become unavailable due to hardware wiring constraints turn black. In this example, 384 electrodes have been selected (matching the maximum simultaneous recording capacity of this probe), with a pattern distributed across multiple banks and shanks, illustrating that PixelMap allows selection of arbitrary channelmap geometries. The optional anatomical overlay, which guides channelmap design, is rendered beside the shanks as coloured depth bands with region labels.\ @@ -111,7 +133,7 @@ PixelMap addresses a practical bottleneck in Neuropixels experimental workflows. PixelMap is released under an open-source licence (GPLv3), provides comprehensive documentation including a [contributor guide](https://pixelmap-neuropixels.readthedocs.io/en/latest/development.html), and is accessible via web application, Python package, Docker container, or programmatic API. It builds on the authors' track record using Neuropixels probes [@steinmetz2021; @bondy2024; @beau2025; @fabre2026basal] and developing Neuropixels software [@beau2021]. -Adoption is evidenced by deployment on PNI's public server, 59 GitHub stars, and web traffic of roughly 400 unique visitors between March and May 2026 (~45/week; see the [analytics](https://github.com/m-beau/pixelmap/tree/main/analytics) folder of the repository). The package has been under active development for over ten months, with external contributors implementing new features (anatomical and activity overlays). +Adoption is evidenced by deployment on Princeton Neuroscience Institute's (PNI) public server, 59 GitHub stars, and web traffic of roughly 400 unique visitors between March and May 2026 (~45/week; see the [analytics](https://github.com/m-beau/pixelmap/tree/main/analytics) folder of the repository). The package has been under active development for over ten months, with external contributors implementing new features (anatomical and activity overlays). # AI Usage Disclosure diff --git a/paper/paper.pdf b/paper/paper.pdf index 658d3c4..1c6be07 100644 Binary files a/paper/paper.pdf and b/paper/paper.pdf differ diff --git a/pixelmap/__init__.py b/pixelmap/__init__.py index 330d4c3..42aaeb3 100644 --- a/pixelmap/__init__.py +++ b/pixelmap/__init__.py @@ -7,6 +7,6 @@ from .backend import * -__version__ = "1.3.1" +__version__ = "1.4.0" __author__ = "Maxime Beau" __email__ = "maxime@princeton.edu" diff --git a/pixelmap/anatomy/atlas.py b/pixelmap/anatomy/atlas.py index 7fc1090..92f6154 100644 --- a/pixelmap/anatomy/atlas.py +++ b/pixelmap/anatomy/atlas.py @@ -26,6 +26,19 @@ ~9 MB of compressed chunks, against ~1.3 GB of tiffs under v2. That is what makes baking them into the Docker image cheap and reliable. +Only one annotation volume is ever resident +------------------------------------------- +The decompressed annotation is by far the largest thing PixelMap holds: 308 MB +for allen_mouse_25um, 1.07 GB for whs_sd_rat_39um, 4.8 GB for allen_mouse_10um +(mean across the registry: ~700 MB). Caching more than one of those is what +took the server to 3 GB RSS with a single user simply trying atlases. + +So exactly one atlas's annotation is kept in RAM at a time, and switching +atlases *releases the old one before loading the new* — see +:func:`_make_resident`. The atlas *objects* are a separate, cheap cache +(:func:`get_atlas`, ~5 MB each, metadata only): keeping several of those costs +almost nothing and is what lets metadata queries stay lock-free. + Why the network never runs inline --------------------------------- The deployed app is a single-process Panel/Bokeh server: a blocking call on @@ -39,20 +52,28 @@ from __future__ import annotations +import ctypes +import ctypes.util import functools +import gc import threading from dataclasses import dataclass from pathlib import Path import numpy as np from brainglobe_atlasapi import BrainGlobeAtlas -from brainglobe_atlasapi.descriptors import V3_ANNOTATION_NAME, V3_ATLAS_ROOTDIR +from brainglobe_atlasapi.descriptors import ( + V3_ANNOTATION_NAME, + V3_ATLAS_ROOTDIR, + remote_url_s3, +) from brainglobe_atlasapi.list_atlases import ( get_all_atlases_lastversions, get_downloaded_atlases, ) from pixelmap.anatomy._registry_snapshot import REGISTRY_SNAPSHOT +from pixelmap.anatomy.volume import CanonicalVolume _DEFAULT_ATLAS = "allen_mouse_25um" @@ -67,37 +88,232 @@ class RegionInfo: rgb: tuple[int, int, int] # 0-255 color as defined by the atlas -@functools.lru_cache(maxsize=4) +@functools.lru_cache(maxsize=8) def get_atlas(name: str = _DEFAULT_ATLAS): - """Return a cached :class:`BrainGlobeAtlas` instance. + """Return a cached, **metadata-only** :class:`BrainGlobeAtlas` instance. We delegate the download/caching to brainglobe — its on-disk cache is shared across processes. ``check_latest=False`` skips the remote version check so the app doesn't hang when the atlas host is unreachable. - Construction fetches only the manifest and metadata (a few hundred KB); - the annotation array is pulled from S3 later, when something first reads - ``atlas.annotation``. Either step is free once the data is local, but - neither is free on a cold cache — see :func:`ensure_downloaded` for - getting that cost off the event loop. - - The cache is bounded so a server that sees many atlases doesn't hold every - annotation volume it has ever decoded. Nothing else may keep a strong - reference to an atlas object, or that bound stops meaning anything (see - :func:`_region_info_from_id`). + Construction fetches only the manifest and metadata (a few hundred KB, ~5 MB + resident, ~65 ms); the annotation array is pulled from S3 later, when + something first reads ``atlas.annotation``. **Nothing may read that + attribute except** :func:`_make_resident`, which is what keeps the + one-annotation-at-a-time invariant true — every other function here is a + metadata query and must stay that way. + + Because the objects this returns are small, the bound is generous and this + function takes no lock: metadata queries (:func:`is_downloaded`, + :func:`volume_center_um`, :func:`origin_corner`, region lookups) stay cheap + and never block, even while another thread is mid-download. The annotation + is bounded separately, and far more tightly, by :func:`_make_resident`. """ return BrainGlobeAtlas(name, check_latest=False) +#: Serialises atlas *switches*. Held across a download, so a second session +#: asking for a different atlas waits rather than putting two annotation +#: volumes in RAM at once — the whole point of this module. Deliberately not +#: taken by :func:`get_atlas`, so metadata queries never block on a download. +_resident_lock = threading.RLock() + +#: The single atlas whose annotation is currently open, held by a *strong* +#: reference so it survives eviction from :func:`get_atlas`'s cache and stays +#: releasable. ``None`` when nothing is open. +_resident = None +_resident_name: str | None = None +#: Its :class:`~pixelmap.anatomy.volume.CanonicalVolume`. Normally a handle on +#: the zarr store plus a few hundred KB of cached projections; only the +#: fallback path (test doubles, unresolvable layouts) makes it a full array. +_resident_volume: CanonicalVolume | None = None + + +def _malloc_trim() -> None: + """Hand glibc's freed heap back to the OS. No-op where unavailable. + + Freeing the array is not enough: the annotation is decoded from thousands + of small OME-Zarr chunks, and those land on the heap, where glibc keeps + them after ``free()``. Measured on one 1.07 GB atlas: dropping every + reference returned the array itself but left ~585 MB of high-water behind, + and that is what made RSS ratchet up across atlas switches instead of + returning to baseline. ``MALLOC_ARENA_MAX=2`` (see the Dockerfile) bounds + how many arenas can hoard it; this returns what they are already holding. + + Absent on macOS/musl, where there is no ``malloc_trim`` — the caller has + still dropped its references either way. + """ + trim = _malloc_trim_fn() + if trim is None: + return + try: + trim(0) + except Exception: # noqa: BLE001 - best-effort reclaim, never fatal + pass + + +@functools.lru_cache(maxsize=1) +def _malloc_trim_fn(): + """Resolve glibc's ``malloc_trim`` once, or ``None`` where there isn't one. + + ``libc.so.6`` by name first: it is the glibc soname on every platform that + has this function, including the Ubuntu base image, and it avoids + ``find_library``, which shells out to ``ldconfig``/``gcc`` and returns + ``None`` in slim containers that have neither. Resolved once per process + because that fallback is far too expensive to run on every atlas switch. + """ + candidates = ("libc.so.6", ctypes.util.find_library("c")) + for name in candidates: + if not name: + continue + try: + return ctypes.CDLL(name).malloc_trim + except (OSError, AttributeError): + continue + return None + + +def _release_resident() -> None: + """Drop the resident annotation volume and return its pages to the OS. + + Order matters. ``canonical_annotation`` memoises a *view* onto the array, + which keeps the whole buffer alive, so its memo has to go first or clearing + ``_annotation`` frees nothing. Callers hold :data:`_resident_lock`. + """ + global _resident, _resident_name, _resident_volume + + if _resident is None: + return + canonical_annotation.cache_clear() + _resident_volume = None + # brainglobe has no public "unload" hook; ``_annotation`` is the attribute + # its ``annotation`` property memoises into, and clearing it makes the next + # read re-fetch from the (already local) on-disk cache in ~130 ms. Guarded + # because test doubles expose ``annotation`` as a plain attribute instead. + try: + _resident._annotation = None + except AttributeError: + pass + _resident = None + _resident_name = None + gc.collect() + _malloc_trim() + + +def _make_resident(name: str): + """Return ``name``'s atlas with its annotation loaded, and no other's. + + The single place that opens annotation data, so the "one volume at a time" + invariant holds by construction rather than by everyone remembering it. + Switching releases the previous atlas *before* opening the new one, which + keeps the peak at whichever of the two is larger instead of their sum. + + Normally "opening" is just a zarr handle — reads happen per query, through + :class:`~pixelmap.anatomy.volume.CanonicalVolume`. The eager fallback in + :func:`_open_volume` is the exception, and it is the only case where this + costs the whole array. + """ + global _resident, _resident_name, _resident_volume + + with _resident_lock: + if _resident_volume is not None and _resident_name == name: + return _resident_volume + _release_resident() + atlas = get_atlas(name) + volume = _open_volume(atlas) # downloads the chunks if they aren't local + _resident, _resident_name, _resident_volume = atlas, name, volume + return volume + + +def _open_volume(atlas) -> CanonicalVolume: + """Open ``atlas``'s annotation for lazy reads, fetching the data if needed. + + Prefers the OME-Zarr store on disk, so reads decompress only the chunks they + touch. Falls back to brainglobe's ``atlas.annotation`` — which materialises + the whole array — when the store cannot be resolved: test doubles, and any + future layout this does not understand. The fallback is correct, just + expensive, so it must never raise on the happy path. + """ + axes = anatomical_axes(atlas) + resolution = (axes["AP"].res_um, axes["DV"].res_um, axes["ML"].res_um) + try: + store = _open_annotation_store(atlas) + except Exception: # noqa: BLE001 - any failure falls back to the eager read + store = atlas.annotation + return CanonicalVolume(store, axes, resolution) + + +def _annotation_level_dir(atlas) -> Path: + """Directory of the OME-Zarr pyramid level matching the atlas's resolution. + + brainglobe picks the level at construction (``_annotation_pyramid_level``); + we only have to turn that index into its on-disk name, which the multiscale + metadata beside the levels spells out. + """ + import json + + location = atlas.metadata["annotation_set"]["location"][1:] + root = Path(atlas.root_dir) / location / V3_ANNOTATION_NAME + meta = json.loads((root / "zarr.json").read_text()) + attrs = meta.get("attributes", meta) + ome = attrs.get("ome", attrs) + multiscales = ome["multiscales"] + datasets = (multiscales[0] if isinstance(multiscales, list) else multiscales)["datasets"] + return root / datasets[atlas._annotation_pyramid_level]["path"] + + +def _open_annotation_store(atlas): + """A ``zarr.Array`` over the atlas's annotation, downloading it if absent. + + Mirrors what ``brainglobe_atlasapi.core.Atlas.annotation`` does to fetch a + level, minus the ``.compute()`` that turns it into an in-memory array — the + whole point being to leave the data on disk. Coupled to brainglobe's v3 + layout, as :func:`_annotation_is_cached` already is. + """ + import zarr + + level_dir = _annotation_level_dir(atlas) + if not (level_dir / "c").exists(): + location = atlas.metadata["annotation_set"]["location"][1:] + remote = remote_url_s3.format( + f"{location}/{V3_ANNOTATION_NAME}/{level_dir.name}/" + ) + atlas.fs.get(remote, str(level_dir), recursive=True) + return zarr.open_array(str(level_dir), mode="r") + + +def reclaim_free_memory() -> None: + """Return already-freed heap to the OS, without dropping the open atlas. + + Reading a slice allocates and frees a burst of chunk buffers, and the C + allocator keeps that space for reuse rather than unmapping it. Nothing is + leaking in Python — object counts are flat across hundreds of renders — but + RSS is what a container limit measures, so the space has to be handed back + explicitly. Cheap (a few ms) and safe to call after any read-heavy + operation; a no-op where the platform offers no way to do it (macOS). + """ + _malloc_trim() + + +def release_atlas_memory() -> None: + """Drop the resident annotation volume, freeing its RAM. + + For callers that know no atlas is needed for a while. The next lookup + reloads from the local on-disk cache. + """ + with _resident_lock: + _release_resident() + + def ensure_downloaded(name: str = _DEFAULT_ATLAS) -> None: """Materialise ``name``'s annotation, downloading it if necessary. The one call in this module that can block for seconds on a cold cache, in a single place so callers can push it onto a worker thread — which the GUI does, because on the server the calling thread also serves every other - session. A no-op once the atlas is local. + session. A no-op once the atlas is local *and* already resident. """ - _ = get_atlas(name).annotation + _make_resident(name) @functools.lru_cache(maxsize=1) @@ -366,7 +582,9 @@ def derive_origin_from_ac(name: str) -> tuple[float, float, float] | None: if not ac_ids: return None ann, res = canonical_annotation(name) # (AP, DV, ML) - ap, dv, ml = np.where(np.isin(ann, ac_ids)) + # Slab-wise: ``np.isin`` over a whole volume would allocate a bool array the + # size of the voxel count on top of the volume itself. + ap, dv, ml = ann.find_label_voxels(ac_ids) if ap.size == 0: return None midline = float(ml.mean()) # AC ~symmetric → centroid ML = midline @@ -398,8 +616,9 @@ def _atlas_shape(atlas) -> tuple[int, ...]: brainglobe publishes ``shape`` from the atlas manifest, which is the cheap way to ask: reading ``annotation.shape`` instead would pull the entire - array from S3. Test doubles only carry ``annotation``, so fall back to - that. + array from S3 — outside :func:`_make_resident`, and so outside the + one-volume-at-a-time bound. Every real v3 atlas carries ``shape``; only + test doubles reach the fallback. """ shape = getattr(atlas, "shape", None) if shape is not None: @@ -434,7 +653,7 @@ def anatomical_axes(atlas) -> dict[str, _AnatAxis]: return axes -@functools.lru_cache(maxsize=4) +@functools.lru_cache(maxsize=1) def canonical_annotation(atlas_name: str): """Return ``(annotation, resolution)`` reoriented to canonical ``(AP, DV, ML)``. @@ -444,18 +663,19 @@ def canonical_annotation(atlas_name: str): assumes this layout, so funnelling every atlas through here is what lets non-Allen orientations work. For an already-``asr`` atlas this is a no-op (identity transpose, no flips), so Allen behavior is unchanged. + + The first element is a :class:`~pixelmap.anatomy.volume.CanonicalVolume`, + not an ``ndarray``: it indexes like one for the ints and step-1 slices this + package uses, but each read pulls only the chunks it touches instead of + materialising hundreds of MB. Use ``.gather()`` for scattered points and + ``.projections()`` for a whole-brain silhouette — both are far cheaper than + the equivalent numpy expression, and neither needs the full array. + + Memoised at ``maxsize=1`` to match the one-atlas-at-a-time bound; + :func:`_release_resident` clears it, which is what actually enforces that. """ - atlas = get_atlas(atlas_name) - axes = anatomical_axes(atlas) - order = (axes["AP"].array_axis, axes["DV"].array_axis, axes["ML"].array_axis) - arr = np.transpose(atlas.annotation, order) - flip_axes = tuple(i for i, kind in enumerate(("AP", "DV", "ML")) if axes[kind].flip) - if flip_axes: - arr = np.flip(arr, axis=flip_axes) - res = np.array( - [axes["AP"].res_um, axes["DV"].res_um, axes["ML"].res_um], dtype=float - ) - return arr, res + volume = _make_resident(atlas_name) + return volume, np.array(volume.resolution, dtype=float) def volume_center_um(name: str = _DEFAULT_ATLAS) -> tuple[float, float, float]: @@ -502,17 +722,26 @@ def lookup_regions( ml_idx = np.round(coords[:, 1] / voxel_size[2]).astype(int) shape = annotation.shape + inside = ( + (ap_idx >= 0) & (ap_idx < shape[0]) + & (dv_idx >= 0) & (dv_idx < shape[1]) + & (ml_idx >= 0) & (ml_idx < shape[2]) + ) + + # One read of the trajectory's bounding box, rather than a chunk + # decompression per voxel — see :meth:`CanonicalVolume.gather`. + ids = np.zeros(len(coords), dtype=np.int64) + if inside.any(): + ids[inside] = annotation.gather( + ap_idx[inside], dv_idx[inside], ml_idx[inside] + ) results: list[RegionInfo | None] = [] - for ap, dv, ml in zip(ap_idx, dv_idx, ml_idx): - if not (0 <= ap < shape[0] and 0 <= dv < shape[1] and 0 <= ml < shape[2]): - results.append(None) - continue - region_id = int(annotation[ap, dv, ml]) - if region_id == 0: # outside-brain or undefined + for is_inside, region_id in zip(inside, ids): + if not is_inside or region_id == 0: # outside the volume, or undefined results.append(None) continue - results.append(_region_info_from_id(atlas_name, region_id)) + results.append(_region_info_from_id(atlas_name, int(region_id))) return results diff --git a/pixelmap/anatomy/regions.py b/pixelmap/anatomy/regions.py index d7f33ff..90f7ccd 100644 --- a/pixelmap/anatomy/regions.py +++ b/pixelmap/anatomy/regions.py @@ -96,9 +96,15 @@ def tip_depth_below_surface_um( & (ml_idx >= 0) & (ml_idx < n_ml) ) inside = np.zeros(distances.shape, dtype=bool) - inside[in_volume] = ( - annotation[ap_idx[in_volume], dv_idx[in_volume], ml_idx[in_volume]] != 0 - ) + if in_volume.any(): + # One bounding-box read for the whole search line: the samples are + # collinear, so the box is thin even when the line is 30 mm long. + inside[in_volume] = ( + annotation.gather( + ap_idx[in_volume], dv_idx[in_volume], ml_idx[in_volume] + ) + != 0 + ) if not inside[0]: return None # tip is not in the brain — "depth below surface" is meaningless diff --git a/pixelmap/anatomy/schematic.py b/pixelmap/anatomy/schematic.py index 141d915..b5c0f4c 100644 --- a/pixelmap/anatomy/schematic.py +++ b/pixelmap/anatomy/schematic.py @@ -14,8 +14,6 @@ from __future__ import annotations -import functools - import numpy as np from matplotlib.figure import Figure from matplotlib.lines import Line2D @@ -31,12 +29,19 @@ _OUTLINE_RGBA = (0.2, 0.2, 0.25, 0.8) -@functools.lru_cache(maxsize=4) def _atlas_data(atlas_name: str): - """Cached ``(atlas, annotation, resolution)`` in canonical ``(AP, DV, ML)``. + """``(atlas, annotation, resolution)`` in canonical ``(AP, DV, ML)``. Going through :func:`~pixelmap.anatomy.atlas.canonical_annotation` means the slicing/projection below works for any atlas orientation, not just Allen's. + + Deliberately **not** memoised. It used to be an ``lru_cache(maxsize=4)``, + which made it a third independent cache holding atlas objects and their + annotation volumes — keyed and evicted independently of the two in + :mod:`pixelmap.anatomy.atlas`, so between them they could pin a dozen + volumes (~700 MB each) while each looked individually well-bounded. That + was the server's monotonic RSS climb. Both calls below are already cached + upstream, so there is nothing to memoise here anyway. """ atlas = get_atlas(atlas_name) annotation, resolution = canonical_annotation(atlas_name) @@ -75,13 +80,20 @@ def _outline_rgba(label_img: np.ndarray) -> np.ndarray: return rgba -def _draw_view(ax, label_img, x_um, y_um, atlas, fallback_mask, title, warp=None): +def _draw_view(ax, label_img, x_um, y_um, atlas, silhouette, title, warp=None): """Draw one slice (colored regions + outlines), or a silhouette fallback. - ``label_img`` and ``fallback_mask`` are shaped ``(len(y_um), len(x_um))``; - y is DV in both views. The y-axis is oriented dorsal-up. ``warp``, if given, - is ``(scale_x, scale_y, rotate_deg, pivot_x, pivot_y)`` applied to the *brain - image only* (not the probe) so the atlas visibly squashes/tilts about bregma. + ``label_img`` and the mask ``silhouette`` returns are shaped + ``(len(y_um), len(x_um))``; y is DV in both views. The y-axis is oriented + dorsal-up. ``warp``, if given, is ``(scale_x, scale_y, rotate_deg, pivot_x, + pivot_y)`` applied to the *brain image only* (not the probe) so the atlas + visibly squashes/tilts about bregma. + + ``silhouette`` is a *callable*, not a mask: building it is the one operation + here that has to read every voxel, and it is only needed when the tip falls + outside the volume and there is no slice to draw. Passing the array instead + meant paying for it on every render, including the overwhelmingly common + case where it goes unused. """ # extent=(left, right, bottom, top); top = y_um[0] (DV 0) puts dorsal up. extent = (float(x_um[0]), float(x_um[-1]), float(y_um[-1]), float(y_um[0])) @@ -95,8 +107,9 @@ def _draw_view(ax, label_img, x_um, y_um, atlas, fallback_mask, title, warp=None ax.set_ylim(extent[2], extent[3]) else: # Tip is outside the volume here — show the whole-brain silhouette. - artists.append(ax.contourf(x_um, y_um, fallback_mask, levels=[0.5, 1.5], colors=[_FILL])) - artists.append(ax.contour(x_um, y_um, fallback_mask, levels=[0.5], colors=[_EDGE], linewidths=0.6)) + mask = silhouette() + artists.append(ax.contourf(x_um, y_um, mask, levels=[0.5, 1.5], colors=[_FILL])) + artists.append(ax.contour(x_um, y_um, mask, levels=[0.5], colors=[_EDGE], linewidths=0.6)) ax.invert_yaxis() if warp is not None: @@ -123,6 +136,7 @@ def render_locator( ml_squish: float = 1.0, dv_squish: float = 1.0, tilt_deg: float = 0.0, + fig: Figure | None = None, ) -> Figure: """Render the locator figure for the current insertion pose. @@ -136,6 +150,13 @@ def render_locator( shape with the probe straight. Returns a bare matplotlib :class:`~matplotlib.figure.Figure` for a Panel ``Matplotlib`` pane (no pyplot global state). + + Pass ``fig`` to draw into an existing figure instead of building one. The + overlay re-renders on every pose edit, and constructing a figure per render + — with its Agg canvas and renderer — was measured at 3.1 MB of unreturned + RSS per call inside the container, more than everything the atlas itself + costs. Reusing one figure per session removes that; the caller owns it and + must not keep references to artists across renders, since this clears it. """ atlas, ann, res = _atlas_data(atlas_name) n_ap, n_dv, n_ml = ann.shape @@ -158,11 +179,13 @@ def render_locator( ap_idx = int(np.clip(round(tip_ap / res[0]), 0, n_ap - 1)) dv_idx = int(np.clip(round(tip_dv / res[1]), 0, n_dv - 1)) ml_idx = int(np.clip(round(tip_ml / res[2]), 0, n_ml - 1)) - inside = ann > 0 # 2×2 grid (4th cell empty) so the figure stays narrow enough to fit the # side panel without overflowing the page; a 1×3 row would be too wide. - fig = Figure(figsize=(3.7, 3.3), dpi=110) + if fig is None: + fig = Figure(figsize=(3.7, 3.3), dpi=110) + else: + fig.clear() # reuse the canvas and renderer; drop the previous render fig.subplots_adjust(left=0.02, right=0.98, top=0.92, bottom=0.02, wspace=0.08, hspace=0.20) ax_sag = fig.add_subplot(2, 2, 1) @@ -181,14 +204,17 @@ def render_locator( # Labels report the coordinate along each panel's visible horizontal axis: # AP on the sagittal view, ML on the coronal view. + # ``projections()`` returns the silhouette reduced over ML, AP and DV + # respectively, and caches, so at most one whole-volume scan happens — and + # only if some view actually has no slice to draw. _draw_view(ax_sag, ann[:, :, ml_idx].T, ap_um, dv_um, atlas, - fallback_mask=inside.any(axis=2).T, + silhouette=lambda: ann.projections()[0].T, title=f"Sagittal · AP {ap_um[ap_idx]:.0f} µm", warp=warp_sag) _draw_view(ax_cor, ann[ap_idx, :, :], ml_um, dv_um, atlas, - fallback_mask=inside.any(axis=0), + silhouette=lambda: ann.projections()[1], title=f"Coronal · ML {ml_um[ml_idx]:.0f} µm", warp=warp_cor) _draw_view(ax_hor, ann[:, dv_idx, :], ml_um, ap_um, atlas, - fallback_mask=inside.any(axis=1), + silhouette=lambda: ann.projections()[2], title=f"Horizontal · DV {dv_um[dv_idx]:.0f} µm", warp=warp_hor) # Probe (red) + bregma estimate (black) per view: (axis, x-col, y-col) diff --git a/pixelmap/anatomy/volume.py b/pixelmap/anatomy/volume.py new file mode 100644 index 0000000..41eae4c --- /dev/null +++ b/pixelmap/anatomy/volume.py @@ -0,0 +1,211 @@ +"""Lazy, canonically-oriented access to a brainglobe annotation volume. + +PixelMap never needs a whole annotation. It reads a few hundred voxels along a +probe, three 2D slices for the locator, and — only when the tip falls outside +the brain — a whole-brain silhouette. brainglobe's ``atlas.annotation`` +nevertheless materialises the entire array, which is the single largest cost in +the app: 308 MB for allen_mouse_25um, 692 MB for danionella_cerebrum_mixed_2.5um, +4.8 GB for allen_mouse_10um. + +Worse, materialising costs far more than the array itself. The data is stored +as chunked OME-Zarr, and assembling it decompresses every chunk: measured 3.4x +the array size through brainglobe's dask path, 2.4x through a plain zarr read. +danionella's chunks are 322**3 — 134 MB decompressed apiece, sixteen of them for +a 692 MB array — so simply opening it peaked at 2.3 GB. + +:class:`CanonicalVolume` reads from the zarr store instead, so only the chunks +covering the requested region are ever decompressed. It also hides the +orientation remap: every atlas is presented as canonical ``(AP, DV, ML)`` +(brainglobe ``"asr"``) regardless of its native axis order, which is what the +rest of :mod:`pixelmap.anatomy` assumes. + +The backing store is anything supporting numpy basic indexing and ``.shape`` — +a ``zarr.Array`` in production, a plain ``ndarray`` for test doubles and for +atlases whose on-disk layout cannot be resolved. +""" + +from __future__ import annotations + +import numpy as np + +#: Target bytes per slab for the whole-volume scans (silhouette, label search). +#: These are the only operations that must touch every voxel, and this is what +#: keeps them O(slab) in RAM instead of O(volume). +_SLAB_BYTES = 64 * 1024 * 1024 + + +class CanonicalVolume: + """An annotation volume indexed as canonical ``(AP, DV, ML)``, read lazily. + + Supports ``vol[i, j, k]`` with ints and step-1 slices, which is everything + PixelMap asks for. Each read is translated into the store's native axis + order, fetched, then transposed and flipped back into canonical order — so + only the requested region is ever decompressed and held. + """ + + def __init__(self, store, axes, resolution): + self._store = store + self._axes = axes # {"AP"|"DV"|"ML": _AnatAxis} + self._order = ("AP", "DV", "ML") + self.resolution = tuple(float(r) for r in resolution) + self.shape = tuple(int(axes[k].n) for k in self._order) + self.dtype = np.dtype(getattr(store, "dtype", np.int32)) + self._projections = None + + # -- indexing ---------------------------------------------------------- + + def __getitem__(self, key) -> np.ndarray: + if not isinstance(key, tuple): + key = (key,) + if len(key) != 3: + raise IndexError( + f"CanonicalVolume needs a 3-axis index (AP, DV, ML); got {len(key)}" + ) + + native: list = [slice(None)] * 3 + kept: list[tuple[str, bool]] = [] # canonical axes that survive, + flip + for kind, item in zip(self._order, key): + axis = self._axes[kind] + if isinstance(item, (int, np.integer)): + native[axis.array_axis] = _native_index(int(item), axis.n, axis.flip) + elif isinstance(item, slice): + native[axis.array_axis] = _native_slice(item, axis.n, axis.flip) + kept.append((kind, axis.flip)) + else: + raise TypeError( + "CanonicalVolume supports ints and step-1 slices only; " + f"got {type(item).__name__} on the {kind} axis" + ) + + out = np.asarray(self._store[tuple(native)]) + if not kept: + return out + + # The read came back in *native* axis order; put the surviving axes + # back into canonical (AP, DV, ML) order, then undo any flipped reads. + by_native = sorted(kept, key=lambda kf: self._axes[kf[0]].array_axis) + out = np.transpose(out, [by_native.index(kf) for kf in kept]) + flipped = tuple(i for i, (_, flip) in enumerate(kept) if flip) + return np.flip(out, axis=flipped) if flipped else out + + def __array__(self, dtype=None, copy=None): + """Materialise the whole volume — the thing this class exists to avoid. + + Present so ``np.asarray(vol)`` and ``np.testing.assert_array_equal`` + work on small test volumes. Never call it on a real atlas. + """ + out = self[:, :, :] + return out.astype(dtype) if dtype is not None else out + + def gather(self, ap, dv, ml) -> np.ndarray: + """Values at N canonical voxel indices, via a single bounding-box read. + + Reading the points one at a time would be far worse than reading the + whole volume: every scalar access decompresses the entire chunk holding + it, so 380 lookups cost 289 MB and 475 ms against 340 MB and 78 ms for + the full array. One read of their bounding box costs 15 MB and 5 ms — + a probe trajectory is a thin box, which is the case that matters. + + Indices must already be inside the volume. + """ + ap = np.asarray(ap, dtype=int) + dv = np.asarray(dv, dtype=int) + ml = np.asarray(ml, dtype=int) + if ap.size == 0: + return np.zeros(0, dtype=self.dtype) + + lo = (int(ap.min()), int(dv.min()), int(ml.min())) + hi = (int(ap.max()) + 1, int(dv.max()) + 1, int(ml.max()) + 1) + box = self[lo[0]:hi[0], lo[1]:hi[1], lo[2]:hi[2]] + return box[ap - lo[0], dv - lo[1], ml - lo[2]] + + # -- whole-volume scans, done a slab at a time -------------------------- + + def _slabs(self): + """Yield ``(start, stop, block)`` over canonical AP, bounded in RAM. + + Slabs are aligned to the store's native chunking along whichever axis + AP maps to, so a chunk is decompressed once rather than once per slab. + """ + n_ap = self.shape[0] + plane = int(np.prod(self.shape[1:])) * self.dtype.itemsize + thickness = max(1, _SLAB_BYTES // max(plane, 1)) + + chunks = getattr(self._store, "chunks", None) + if chunks: + native_chunk = int(chunks[self._axes["AP"].array_axis]) + if native_chunk > 0: + thickness = max(native_chunk, thickness - thickness % native_chunk) + + for start in range(0, n_ap, thickness): + stop = min(start + thickness, n_ap) + yield start, stop, self[start:stop, :, :] + + def projections(self) -> tuple[np.ndarray, np.ndarray, np.ndarray]: + """Whole-brain silhouette, as the three orthogonal max-projections. + + Returns ``(over_ML, over_AP, over_DV)`` with shapes ``(AP, DV)``, + ``(DV, ML)`` and ``(AP, ML)`` — the masks the locator falls back to when + the tip lies outside the volume. Computed slab-wise, so peak RAM is one + slab rather than the volume plus a full-size bool array, and cached + because the result is a few hundred KB and the scan is the expensive + part. + """ + if self._projections is not None: + return self._projections + + n_ap, n_dv, n_ml = self.shape + over_ml = np.zeros((n_ap, n_dv), dtype=bool) + over_ap = np.zeros((n_dv, n_ml), dtype=bool) + over_dv = np.zeros((n_ap, n_ml), dtype=bool) + for start, stop, block in self._slabs(): + inside = block > 0 + over_ml[start:stop] = inside.any(axis=2) + over_ap |= inside.any(axis=0) + over_dv[start:stop] = inside.any(axis=1) + + self._projections = (over_ml, over_ap, over_dv) + return self._projections + + def find_label_voxels(self, ids) -> tuple[np.ndarray, np.ndarray, np.ndarray]: + """Canonical ``(AP, DV, ML)`` indices of every voxel labelled in ``ids``. + + Slab-wise for the same reason as :meth:`projections`: ``np.isin`` over a + whole volume allocates a bool array the size of the voxel count on top + of the volume itself — 1.2 GB for a 10 µm atlas. + """ + ids = np.asarray(list(ids)) + ap_out: list = [] + dv_out: list = [] + ml_out: list = [] + for start, _stop, block in self._slabs(): + ap, dv, ml = np.where(np.isin(block, ids)) + if ap.size: + ap_out.append(ap + start) + dv_out.append(dv) + ml_out.append(ml) + if not ap_out: + empty = np.zeros(0, dtype=int) + return empty, empty.copy(), empty.copy() + return ( + np.concatenate(ap_out), + np.concatenate(dv_out), + np.concatenate(ml_out), + ) + + +def _native_index(i: int, n: int, flip: bool) -> int: + """Canonical position ``i`` as an index into a possibly-reversed native axis.""" + return n - 1 - i if flip else i + + +def _native_slice(item: slice, n: int, flip: bool) -> slice: + """Canonical slice as a *forward* native slice (the caller re-flips the read). + + Zarr has no negative-step indexing, so a flipped axis is read forward over + the mirrored range and the resulting block reversed afterwards. + """ + start, stop, step = item.indices(n) + if step != 1: + raise TypeError("CanonicalVolume supports step-1 slices only") + return slice(n - stop, n - start) if flip else slice(start, stop) diff --git a/pixelmap/gui/app_util.py b/pixelmap/gui/app_util.py index ac5a315..e996fdd 100644 --- a/pixelmap/gui/app_util.py +++ b/pixelmap/gui/app_util.py @@ -8,6 +8,7 @@ import panel as pn import psutil +from pixelmap.gui import watchdog from pixelmap.gui.gui import create_app pn.extension(notifications=True) @@ -85,5 +86,8 @@ def main(show=True, local=True): verbose=True, ) else: + # Once per process (app.py re-runs per session); we're on the server's + # event loop here, which is the loop the stall watchdog must watch. + watchdog.install() _log_session_start() create_app().servable(title="Neuropixels Channelmap Generator") diff --git a/pixelmap/gui/gui.py b/pixelmap/gui/gui.py index b6b3d9d..164effb 100644 --- a/pixelmap/gui/gui.py +++ b/pixelmap/gui/gui.py @@ -11,6 +11,7 @@ import json import logging import re +import threading from dataclasses import dataclass, field from io import BytesIO, StringIO from pathlib import Path @@ -773,11 +774,15 @@ def on_electrode_selection(self, _, old, new): self.update_electrode_counter() self.refresh_undo_buttons() - # Clear the selection to allow for new interactions + # Clear the selection to allow for new interactions. + # + # No gc.collect() here: a full collection walks every live object in + # the process, and on the production server (GBs of RSS, hundreds of + # thousands of Bokeh models across sessions) that froze the single + # event loop for 20-90 s per click -- failing healthchecks and getting + # users' websockets dropped by the proxy. self.electrode_source.selected.indices = [] - gc.collect() - def get_zigzag_subset(self): N_per_shank = PROBE_FEATURES[self.probe_subtype]["n_sites_per_shank"] @@ -1373,6 +1378,9 @@ def compute_anatomy_overlay(self): tilt_deg = float(self.tilt_input.value) try: + # Reuse this session's figure: building one per render cost 3.1 MB + # of RSS per overlay compute, and the overlay recomputes on every + # pose edit. _fig = render_locator( atlas_name, tip_atlas=tip, @@ -1386,12 +1394,18 @@ def compute_anatomy_overlay(self): ml_squish=ml_squish, dv_squish=dv_squish, tilt_deg=tilt_deg, + fig=self._locator_fig, ) + self._locator_fig = _fig self.anatomy_locator.object = self._fig_to_locator_html(_fig) - plt.close(_fig) except Exception as exc: # a failed locator shouldn't sink the overlay print(f"Locator render failed: {exc}") + # Hand back the chunk buffers this render churned through. They are + # already freed; this is what unmaps them, so repeated overlay computes + # don't ratchet RSS upward for the life of the process. + anatomy_atlas.reclaim_free_memory() + # Overlay now exists → later input edits live-update it. self._anatomy_overlay_active = True self.anatomy_locator_section.visible = True @@ -1414,6 +1428,25 @@ def clear_anatomy_overlay(self): self.anatomy_locator.object = "" self.anatomy_locator_section.visible = False self._update_tip_depth_readout() # back to a blank readout + if self._locator_fig is not None: + plt.close(self._locator_fig) + self._locator_fig = None + + # Hand back the atlas's RAM: nothing on screen needs it any more, and an + # annotation volume is 308 MB for allen_mouse_25um and up to 4.8 GB for + # a 10 µm atlas. Recomputing reloads it from the local on-disk cache in + # ~130 ms, which is a fair price for not sitting on gigabytes. + # + # Off-thread on purpose: the release takes the atlas lock, which a + # concurrent download can hold for seconds, and this method runs on the + # Bokeh event loop that serves every other session. Fire-and-forget — + # if it loses a race with someone else loading an atlas, that atlas is + # simply the resident one and the memory is reclaimed on the next switch. + threading.Thread( + target=anatomy_atlas.release_atlas_memory, + name="pixelmap-atlas-release", + daemon=True, + ).start() def _fig_to_locator_html(self, fig) -> str: """Convert a matplotlib figure to an HTML img with click-to-lightbox.""" @@ -2661,6 +2694,9 @@ def create_widgets(self): self.clear_anatomy_button.on_click(lambda event: self.clear_anatomy_overlay()) self.anatomy_legend = pn.pane.HTML(self._empty_legend_html(), sizing_mode="stretch_width") self.anatomy_locator = pn.pane.HTML("", sizing_mode="stretch_width") + # Reused across renders (see compute_anatomy_overlay); dropped when the + # overlay is cleared so an idle session doesn't sit on a canvas. + self._locator_fig = None # Hidden until the first overlay is computed; cleared by "Clear overlay". self.anatomy_locator_section = pn.Column( self.anatomy_locator, diff --git a/pixelmap/gui/watchdog.py b/pixelmap/gui/watchdog.py new file mode 100644 index 0000000..616ba62 --- /dev/null +++ b/pixelmap/gui/watchdog.py @@ -0,0 +1,199 @@ +"""Early warnings for the deployed server: event-loop stalls and high RSS. + +The production container used to die with no output at all -- a SIGKILL from +the OOM killer or an orchestrator acting on a failed healthcheck leaves no +Python traceback. The only clue was that the periodic ``Memory usage`` log +sometimes fired 20-90 s late, i.e. the single Bokeh event loop had been +blocked that long. + +This module turns those conditions into loud, greppable ``[watchdog]`` WARNING +lines while the app is still alive, so the Slack log forwarder flags them: + +* :class:`StallWatchdog` -- the event loop beats a heartbeat every second; a + separate thread notices when it stops and logs the *blocked thread's stack*, + which names the handler responsible. Logs again with the duration once the + loop recovers. +* :class:`MemoryWatch` -- warns once when RSS crosses a soft limit, re-arming + only after it drops back below, so RSS hovering at the limit doesn't spam. + +Neither ever raises into the app or kills anything: they only log. + +Configuration (environment, read by :func:`install`): + +``PIXELMAP_STALL_WARN_S`` seconds of blocking before warning (default 5; + ``0`` disables the watchdog entirely) +``PIXELMAP_RSS_WARN_MB`` RSS soft limit in MB (default unset = no memory + warnings) +""" + +from __future__ import annotations + +import logging +import os +import sys +import threading +import time +import traceback +from collections.abc import Callable + +log = logging.getLogger("pixelmap.watchdog") + +#: The process-wide watchdog, so :func:`install` is idempotent -- ``app.py`` is +#: re-executed for every Bokeh session, but this module is imported once. +_installed: _Watchdog | None = None +_install_lock = threading.Lock() + + +def _env_float(name: str, default: float) -> float: + raw = os.environ.get(name, "").strip() + if not raw: + return default + try: + return float(raw) + except ValueError: + log.warning("[watchdog] ignoring %s=%r: not a number, using %s", name, raw, default) + return default + + +def _rss_mb() -> float: + import psutil + + return psutil.Process().memory_info().rss / 1024 / 1024 + + +class StallWatchdog: + """Detect a blocked event loop and report what it is stuck on. + + :meth:`beat` runs *on* the loop; :meth:`check` runs from any other thread. + """ + + def __init__(self, threshold_s: float = 5.0, clock: Callable[[], float] = time.monotonic): + if threshold_s <= 0: + raise ValueError(f"threshold_s must be positive, got {threshold_s}") + self.threshold_s = threshold_s + self._clock = clock + self._last_beat = clock() + self._loop_thread_id: int | None = None + self._reported = False + + def beat(self) -> None: + now = self._clock() + if self._reported: + log.warning( + "[watchdog] event loop recovered after being blocked %.1f s", + now - self._last_beat, + ) + self._reported = False + self._last_beat = now + self._loop_thread_id = threading.get_ident() + + def check(self) -> bool: + """Log a warning if the loop is newly stalled. Returns whether it did.""" + lag = self._clock() - self._last_beat + if lag < self.threshold_s or self._reported: + return False + self._reported = True + log.warning( + "[watchdog] event loop blocked for %.1f s (threshold %.1f s); " + "all sessions and the healthcheck are frozen. Blocked thread:\n%s", + lag, + self.threshold_s, + self._loop_stack(), + ) + return True + + def _loop_stack(self) -> str: + frame = sys._current_frames().get(self._loop_thread_id) if self._loop_thread_id else None + if frame is None: + return " (stack unavailable: loop thread unknown)" + return "".join(traceback.format_stack(frame)) + + +class MemoryWatch: + """Warn once when RSS reaches ``limit_mb``; re-arm below ``rearm_fraction``.""" + + def __init__(self, limit_mb: float | None, rearm_fraction: float = 0.9): + if not 0 < rearm_fraction <= 1: + raise ValueError(f"rearm_fraction must be in (0, 1], got {rearm_fraction}") + self.limit_mb = limit_mb if limit_mb and limit_mb > 0 else None + self.rearm_fraction = rearm_fraction + self._armed = True + + def check(self, rss_mb: float) -> bool: + """Log a warning if RSS newly crossed the limit. Returns whether it did.""" + if self.limit_mb is None: + return False + if rss_mb < self.limit_mb * self.rearm_fraction: + self._armed = True + return False + if rss_mb < self.limit_mb or not self._armed: + return False + self._armed = False + log.warning( + "[watchdog] RSS %.0f MB reached the %.0f MB warning threshold; " + "the container risks an OOM kill", + rss_mb, + self.limit_mb, + ) + return True + + +class _Watchdog: + """Wires both checks to the running loop and a daemon polling thread.""" + + def __init__(self, stall: StallWatchdog, memory: MemoryWatch, poll_s: float = 1.0): + from tornado.ioloop import PeriodicCallback + + self.stall = stall + self.memory = memory + self._stop = threading.Event() + self._poll_s = poll_s + # Created on the loop thread, so it schedules onto the server's loop. + self._beat = PeriodicCallback(stall.beat, poll_s * 1000) + self._beat.start() + stall.beat() + self._thread = threading.Thread(target=self._run, name="pixelmap-watchdog", daemon=True) + self._thread.start() + + def _run(self) -> None: + while not self._stop.wait(self._poll_s): + try: + self.stall.check() + self.memory.check(_rss_mb()) + except Exception: # a broken watchdog must never take the app down + log.exception("[watchdog] check failed") + + def stop(self) -> None: + self._stop.set() + self._beat.stop() + self._thread.join(timeout=5) + + +def install( + stall_threshold_s: float | None = None, + rss_warn_mb: float | None = None, +) -> _Watchdog | None: + """Start the watchdogs once per process. Call from the event-loop thread. + + Arguments default to the environment (see module docstring). Returns the + running watchdog, or ``None`` when disabled by a non-positive threshold. + """ + global _installed + + if stall_threshold_s is None: + stall_threshold_s = _env_float("PIXELMAP_STALL_WARN_S", 5.0) + if rss_warn_mb is None: + rss_warn_mb = _env_float("PIXELMAP_RSS_WARN_MB", 0.0) + + with _install_lock: + if _installed is not None: + return _installed + if stall_threshold_s <= 0: + return None + _installed = _Watchdog(StallWatchdog(stall_threshold_s), MemoryWatch(rss_warn_mb)) + log.info( + "[watchdog] started: stall warning at %.1f s, RSS warning at %s", + stall_threshold_s, + f"{rss_warn_mb:.0f} MB" if rss_warn_mb and rss_warn_mb > 0 else "off", + ) + return _installed diff --git a/pyproject.toml b/pyproject.toml index cf9ce85..0def759 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -72,6 +72,10 @@ docs = [ "sphinx-design", ] +[tool.pyright] +venvPath = "." +venv = ".venv" + [dependency-groups] dev = [ "furo>=2025.12.19", diff --git a/scripts/prefetch_atlases.py b/scripts/prefetch_atlases.py index 86924b9..b951334 100755 --- a/scripts/prefetch_atlases.py +++ b/scripts/prefetch_atlases.py @@ -21,13 +21,20 @@ Set ``PREFETCH_ALLOW_PARTIAL=1`` to downgrade the failure to a warning. That is for local experimentation only — never for a release build. -Why reading ``.annotation`` matters ------------------------------------ +Why this goes through PixelMap's own loader +------------------------------------------- ``BrainGlobeAtlas(...)`` only fetches the manifest and metadata — a few -hundred KB — and pulls the annotation array from S3 lazily, the first time -something reads ``atlas.annotation``. Constructing the atlas would therefore -"succeed" while leaving the expensive part for the first user to hit at -runtime, which is exactly what this script exists to prevent. +hundred KB — and pulls the annotation from S3 lazily. Constructing the atlas +would therefore "succeed" while leaving the expensive part for the first user +to hit at runtime, which is exactly what this script exists to prevent. + +So it calls ``pixelmap.anatomy.atlas.ensure_downloaded`` and then asserts +``is_downloaded``: the same two functions the GUI uses to decide whether +opening an atlas is free. Going through the app's loader rather than +brainglobe's means "the build warmed the cache" and "the app considers the +atlas local" cannot drift apart — and it puts the chunks on disk without +materialising the array, which brainglobe's ``.annotation`` would (3.4x the +array size in peak RAM, for data we only want cached). """ import os @@ -61,13 +68,19 @@ def warm_registry() -> bool: def fetch(atlas_name: str) -> bool: """Download one atlas, retrying transient failures. True if it landed.""" - from brainglobe_atlasapi import BrainGlobeAtlas + from pixelmap.anatomy.atlas import ( + ensure_downloaded, + is_downloaded, + release_atlas_memory, + ) for attempt, delay in enumerate((*RETRY_DELAYS, None), start=1): try: - atlas = BrainGlobeAtlas(atlas_name, check_latest=False) - # Force the annotation onto disk (see the module docstring). - _ = atlas.annotation + ensure_downloaded(atlas_name) + # Ask the same question the GUI asks before calling an atlas free. + if not is_downloaded(atlas_name): + raise RuntimeError("annotation chunks are still missing") + release_atlas_memory() print(f"[prefetch] {atlas_name}: ok", flush=True) return True except Exception as exc: # noqa: BLE001 - retried, then reported below diff --git a/tests/README.md b/tests/README.md index fcf40fd..91bfaa0 100644 --- a/tests/README.md +++ b/tests/README.md @@ -182,7 +182,7 @@ Tests run automatically via GitHub Actions on: - Every pull request to `main` - Manual workflow dispatch -The workflow tests across multiple Python versions (3.10, 3.11, 3.12) to ensure compatibility. +The workflow tests across multiple Python versions (3.11, 3.12) to ensure compatibility. ## Adding New Tests diff --git a/tests/test_anatomy_atlas.py b/tests/test_anatomy_atlas.py index 504763d..05bd59a 100644 --- a/tests/test_anatomy_atlas.py +++ b/tests/test_anatomy_atlas.py @@ -43,11 +43,13 @@ def _must_not_be_called(*args, **kwargs): @pytest.fixture(autouse=True) def _reset_atlas_cache(): """Make sure no real atlas leaks across tests via the lru_cache.""" + atlas_module.release_atlas_memory() atlas_module.get_atlas.cache_clear() atlas_module.canonical_annotation.cache_clear() atlas_module.derive_origin_from_ac.cache_clear() atlas_module._region_info_from_id.cache_clear() yield + atlas_module.release_atlas_memory() atlas_module.get_atlas.cache_clear() atlas_module.canonical_annotation.cache_clear() atlas_module.derive_origin_from_ac.cache_clear() @@ -89,6 +91,7 @@ def __init__(self, name, **kwargs): self.annotation = np.zeros((4, 4, 4), dtype=np.int32) monkeypatch.setattr(atlas_module, "BrainGlobeAtlas", Zeros) + atlas_module.release_atlas_memory() atlas_module.get_atlas.cache_clear() atlas_module.canonical_annotation.cache_clear() @@ -149,6 +152,7 @@ def test_canonical_annotation_reorients_to_asr(self, monkeypatch): # Native "sla" volume is indexed (DV, ML, AP); canonical must be (AP, DV, ML). native = np.arange(2 * 3 * 4, dtype=np.int32).reshape(2, 3, 4) monkeypatch.setattr(atlas_module, "BrainGlobeAtlas", _atlas_cls("sla", native)) + atlas_module.release_atlas_memory() atlas_module.get_atlas.cache_clear() atlas_module.canonical_annotation.cache_clear() arr, _ = atlas_module.canonical_annotation("x") @@ -163,6 +167,7 @@ def test_derive_origin_from_ac_finds_midline_crossing(self, monkeypatch): structs = {5: {"id": 5, "acronym": "ac", "name": "anterior commissure", "rgb_triplet": [1, 2, 3]}} monkeypatch.setattr(atlas_module, "BrainGlobeAtlas", _atlas_cls("asr", ann, structs)) + atlas_module.release_atlas_memory() atlas_module.get_atlas.cache_clear() atlas_module.canonical_annotation.cache_clear() atlas_module.derive_origin_from_ac.cache_clear() @@ -175,6 +180,7 @@ def test_derive_origin_from_ac_none_without_ac(self, monkeypatch): structs = {9: {"id": 9, "acronym": "x", "name": "some nucleus", "rgb_triplet": [0, 0, 0]}} monkeypatch.setattr(atlas_module, "BrainGlobeAtlas", _atlas_cls("asr", ann, structs)) + atlas_module.release_atlas_memory() atlas_module.get_atlas.cache_clear() atlas_module.canonical_annotation.cache_clear() atlas_module.derive_origin_from_ac.cache_clear() @@ -187,6 +193,7 @@ def test_lookup_resolves_through_ap_flip(self, monkeypatch): native[0, :, :] = 2 # native index 0 = posterior native[1:, :, :] = 1 monkeypatch.setattr(atlas_module, "BrainGlobeAtlas", _atlas_cls("psr", native)) + atlas_module.release_atlas_memory() atlas_module.get_atlas.cache_clear() atlas_module.canonical_annotation.cache_clear() anterior = atlas_module.lookup_regions("x", np.array([[0.0, 0.0, 0.0]])) @@ -371,12 +378,26 @@ def test_unreadable_cache_falls_back_instead_of_raising(self, monkeypatch, tmp_p class TestEnsureDownloaded: - def test_materialises_the_annotation(self, monkeypatch): + """``ensure_downloaded`` must actually fetch the data, not just construct. + + Constructing a ``BrainGlobeAtlas`` pulls only the manifest and metadata — + a few hundred KB — and leaves the annotation on S3. An ``ensure_downloaded`` + that stopped there would shift the real download onto the first user to open + the anatomy panel, on the Bokeh event loop, which is what it exists to + prevent. (It no longer *materialises* the array: with the lazy volume it + only guarantees the data is on disk. For a double with no zarr store to + resolve, that falls back to reading ``annotation``.) + """ + + def test_fetches_the_annotation_data(self, monkeypatch): reads = [] class _A: def __init__(self, name, **_kwargs): self.name = name + self.orientation = "asr" + self.resolution = (25.0, 25.0, 25.0) + self.shape = (2, 2, 2) @property def annotation(self): @@ -387,6 +408,28 @@ def annotation(self): atlas_module.ensure_downloaded("some_atlas") assert reads == ["some_atlas"] + def test_constructing_alone_is_not_enough(self, monkeypatch): + """Guards the failure that shipped three cold images: a build that + 'succeeded' because the atlas object existed, while the data never + left S3.""" + reads = [] + + class _A: + def __init__(self, name, **_kwargs): + self.name = name + self.orientation = "asr" + self.resolution = (25.0, 25.0, 25.0) + self.shape = (2, 2, 2) + + @property + def annotation(self): + reads.append(self.name) + return np.zeros((2, 2, 2), np.int32) + + monkeypatch.setattr(atlas_module, "BrainGlobeAtlas", _A) + atlas_module.get_atlas("some_atlas") + assert reads == [], "constructing an atlas must not fetch the annotation" + class TestAtlasesAreReleased: """Atlas objects must not outlive ``get_atlas``'s cache. @@ -454,3 +497,122 @@ def test_rotating_past_the_bound_does_not_accumulate(self, monkeypatch): # Rebuilding on eviction is expected; *retaining* the rebuilds is not. assert len(built) > bound, "expected evictions to force rebuilds" assert self._live(built) <= bound + + + +class TestOnlyOneAnnotationIsResident: + """At most one annotation volume may be in RAM at any moment. + + Regression test for the production leak that took RSS from 86 MB to 3 GB + while a single user tried out atlases. Three independent ``lru_cache``es + (``get_atlas``, ``canonical_annotation`` and ``schematic._atlas_data``, + ``maxsize=4`` apiece) each pinned atlases and their annotation volumes, and + because they were keyed and evicted independently they could hold a dozen + between them — ~700 MB each on average across the registry, 5.2 GB at the + top end. Each cache looked individually well-bounded; the total was not. + + Liveness is asserted on the annotation *arrays*, via weakrefs, not on the + atlas objects that own them: ``canonical_annotation`` memoises a transposed + *view*, which keeps the whole buffer alive on its own, so counting atlas + objects would miss precisely the leak that matters. + """ + + @staticmethod + def _atlas_cls(alive: list, peak: list[int]): + """A fake that weak-references every annotation array it materialises.""" + + class _LoadTrackingAtlas: + def __init__(self, name: str, **_kwargs): + self.name = name + self.orientation = "asr" + self.resolution = (25.0, 25.0, 25.0) + self.shape = (4, 4, 4) + self.structures = { + 1: {"acronym": "R", "name": "Region", "rgb_triplet": [1, 2, 3]} + } + self._annotation = None + + @property + def annotation(self): + if self._annotation is None: + ann = np.ones((4, 4, 4), dtype=np.int32) + alive.append(weakref.ref(ann)) + self._annotation = ann + gc.collect() + peak[0] = max(peak[0], sum(r() is not None for r in alive)) + return self._annotation + + return _LoadTrackingAtlas + + @staticmethod + def _live(alive: list) -> int: + gc.collect() + return sum(ref() is not None for ref in alive) + + def test_switching_atlases_never_holds_two_volumes(self, monkeypatch): + alive: list = [] + peak = [0] + monkeypatch.setattr( + atlas_module, "BrainGlobeAtlas", self._atlas_cls(alive, peak) + ) + coords = np.array([[0.0, 0.0, 0.0]]) + + # Many more distinct atlases than any of the old caches could hold. + for i in range(24): + name = f"atlas_{i}" + atlas_module.ensure_downloaded(name) + atlas_module.lookup_regions(name, coords) + atlas_module.canonical_annotation(name) + live = self._live(alive) + assert live == 1, ( + f"{live} annotation volumes in RAM after {name}; 1 is allowed" + ) + + assert peak[0] == 1, ( + f"peak of {peak[0]} concurrently-live annotations: the previous " + "atlas must be released *before* the next is read, or a switch " + "transiently costs the sum of the two" + ) + + def test_revisiting_the_resident_atlas_does_not_reload(self, monkeypatch): + alive: list = [] + peak = [0] + monkeypatch.setattr( + atlas_module, "BrainGlobeAtlas", self._atlas_cls(alive, peak) + ) + coords = np.array([[0.0, 0.0, 0.0]]) + + atlas_module.ensure_downloaded("a") + for _ in range(5): + atlas_module.ensure_downloaded("a") + atlas_module.lookup_regions("a", coords) + assert len(alive) == 1, "resident atlas was needlessly reloaded" + assert self._live(alive) == 1 + + def test_release_atlas_memory_frees_everything(self, monkeypatch): + alive: list = [] + monkeypatch.setattr( + atlas_module, "BrainGlobeAtlas", self._atlas_cls(alive, [0]) + ) + atlas_module.ensure_downloaded("a") + atlas_module.canonical_annotation("a") + assert self._live(alive) == 1 + atlas_module.release_atlas_memory() + assert self._live(alive) == 0, "release_atlas_memory left a volume in RAM" + + def test_metadata_queries_do_not_load_annotations(self, monkeypatch): + """The cheap path must stay cheap — and must not evict the resident.""" + alive: list = [] + monkeypatch.setattr( + atlas_module, "BrainGlobeAtlas", self._atlas_cls(alive, [0]) + ) + + atlas_module.ensure_downloaded("resident") + assert len(alive) == 1 + + for i in range(12): # more than get_atlas's own bound + atlas_module.volume_center_um(f"other_{i}") + atlas_module.orientation_code(f"other_{i}") + + assert len(alive) == 1, "a metadata query materialised an annotation" + assert self._live(alive) == 1, "a metadata query evicted the resident atlas" diff --git a/tests/test_anatomy_schematic.py b/tests/test_anatomy_schematic.py index a9c23e0..c73ce03 100644 --- a/tests/test_anatomy_schematic.py +++ b/tests/test_anatomy_schematic.py @@ -6,6 +6,9 @@ from __future__ import annotations +import gc +import weakref + import numpy as np import pytest @@ -33,13 +36,15 @@ def __init__(self, name: str, **_kwargs): @pytest.fixture(autouse=True) def _reset_caches(): + # ``schematic._atlas_data`` is deliberately uncached now (it used to pin a + # fourth set of annotation volumes), so there is nothing to clear for it. + atlas_module.release_atlas_memory() atlas_module.get_atlas.cache_clear() atlas_module.canonical_annotation.cache_clear() - schematic_module._atlas_data.cache_clear() yield + atlas_module.release_atlas_memory() atlas_module.get_atlas.cache_clear() atlas_module.canonical_annotation.cache_clear() - schematic_module._atlas_data.cache_clear() @pytest.fixture @@ -106,3 +111,161 @@ def test_falls_back_to_silhouette_when_tip_outside_volume(fake_brainglobe): y_range=(0.0, 80.0), ) assert len(fig.axes) == 4 # 3 slices + legend; renders without crashing + + + +class _LoadTrackingAtlas: + """Fake atlas whose annotation arrays are individually weak-referenced. + + Liveness is tracked on the *array*, not on the atlas object: a cache that + stores the array directly keeps it alive even after the owning atlas has + dropped its own reference, and that is exactly the failure mode being + guarded against. + """ + + alive: list = [] # weakrefs to every annotation array ever materialised + + def __init__(self, name: str, **_kwargs): + self.name = name + self.orientation = "asr" + self.resolution = (25.0, 25.0, 25.0) + self.shape = (8, 6, 10) + self.structures = { + 1: {"acronym": "L", "name": "Left", "rgb_triplet": [200, 0, 0]}, + 2: {"acronym": "R", "name": "Right", "rgb_triplet": [0, 0, 200]}, + } + self._annotation = None + + @property + def annotation(self): + if self._annotation is None: + ann = np.zeros((8, 6, 10), dtype=np.int32) + ann[1:7, 1:5, 1:5] = 1 + ann[1:7, 1:5, 5:9] = 2 + type(self).alive.append(weakref.ref(ann)) + self._annotation = ann + return self._annotation + + @classmethod + def live_annotations(cls) -> int: + gc.collect() + return sum(ref() is not None for ref in cls.alive) + + +def test_rendering_many_atlases_keeps_one_annotation_resident(monkeypatch): + """The locator must not build an atlas cache of its own. + + ``_atlas_data`` used to be an ``lru_cache(maxsize=4)`` holding + ``(atlas, annotation, resolution)``. Keyed and evicted independently of the + caches in :mod:`pixelmap.anatomy.atlas`, it pinned up to four more + annotation volumes — hundreds of MB apiece — which is part of why the + server's RSS climbed monotonically as a user browsed atlases. Note it held + the *array*, so releasing the atlas object would not have freed it. + """ + monkeypatch.setattr(atlas_module, "BrainGlobeAtlas", _LoadTrackingAtlas) + monkeypatch.setattr(_LoadTrackingAtlas, "alive", []) + + for i in range(12): # three times the old cache bound + render_locator( + f"atlas_{i}", + tip_atlas=(100.0, 100.0, 75.0), + pitch_deg=0.0, + yaw_deg=0.0, + shank_orientation_deg=0.0, + shank_positions={0: 0.0}, + y_range=(0.0, 80.0), + ) + live = _LoadTrackingAtlas.live_annotations() + assert live == 1, ( + f"{live} annotation volumes still in RAM after rendering atlas_{i}; " + "exactly 1 is allowed" + ) + + +class TestSilhouetteIsOnlyBuiltWhenNeeded: + """The whole-brain silhouette is the one thing here that reads every voxel. + + It is the fallback for a tip outside the volume, so on the normal path — + tip in the brain, three real slices to draw — it must never be built. It + used to be computed unconditionally (``inside = ann > 0``), which cost a + full-size bool array on every single render: 1.2 GB for a 10 µm atlas. + """ + + @staticmethod + def _count_projections(monkeypatch) -> list[int]: + from pixelmap.anatomy.volume import CanonicalVolume + + calls = [0] + original = CanonicalVolume.projections + + def counted(self): + calls[0] += 1 + return original(self) + + monkeypatch.setattr(CanonicalVolume, "projections", counted) + return calls + + def test_tip_inside_the_volume_never_builds_it(self, fake_brainglobe, monkeypatch): + calls = self._count_projections(monkeypatch) + _render() + assert calls[0] == 0, "silhouette was built even though every view had a slice" + + def test_tip_outside_the_volume_falls_back_to_it(self, fake_brainglobe, monkeypatch): + calls = self._count_projections(monkeypatch) + # Far outside the 8x6x10-voxel fake brain. + render_locator( + "fake", + tip_atlas=(50_000.0, 50_000.0, 50_000.0), + pitch_deg=0.0, + yaw_deg=0.0, + shank_orientation_deg=0.0, + shank_positions={0: 0.0}, + y_range=(0.0, 80.0), + ) + assert calls[0] > 0, "no slice to draw, but the silhouette was never built" + + +class TestFigureReuse: + """``render_locator`` must be able to draw into a caller-owned figure. + + Building a ``Figure`` per render — with its Agg canvas and renderer — cost + 3.1 MB of unreturned RSS per call in the container, more than the atlas read + it wraps, and the overlay re-renders on every pose edit. + """ + + def test_passing_a_figure_draws_into_it(self, fake_brainglobe): + first = _render() + second = render_locator( + "fake", + tip_atlas=(100.0, 100.0, 75.0), + pitch_deg=0.0, + yaw_deg=0.0, + shank_orientation_deg=0.0, + shank_positions={0: 0.0, 1: 50.0}, + y_range=(0.0, 80.0), + fig=first, + ) + assert second is first, "render_locator built a new figure instead of reusing" + + def test_reuse_does_not_accumulate_axes(self, fake_brainglobe): + """Without a clear, every render would stack another set of axes on.""" + fig = _render() + n_axes = len(fig.axes) + for _ in range(5): + fig = render_locator( + "fake", + tip_atlas=(100.0, 100.0, 75.0), + pitch_deg=0.0, + yaw_deg=0.0, + shank_orientation_deg=0.0, + shank_positions={0: 0.0, 1: 50.0}, + y_range=(0.0, 80.0), + fig=fig, + ) + assert len(fig.axes) == n_axes, ( + f"{len(fig.axes)} axes after 6 renders, expected {n_axes} — the " + "figure is not being cleared between renders" + ) + + def test_omitting_the_figure_still_builds_a_fresh_one(self, fake_brainglobe): + assert _render() is not _render() diff --git a/tests/test_anatomy_surface_depth.py b/tests/test_anatomy_surface_depth.py index 7011393..fe5ce8d 100644 --- a/tests/test_anatomy_surface_depth.py +++ b/tests/test_anatomy_surface_depth.py @@ -35,10 +35,12 @@ def __init__(self, name, **_kwargs): @pytest.fixture(autouse=True) def _reset_cache(): + atlas_module.release_atlas_memory() atlas_module.get_atlas.cache_clear() atlas_module.canonical_annotation.cache_clear() atlas_module._region_info_from_id.cache_clear() yield + atlas_module.release_atlas_memory() atlas_module.get_atlas.cache_clear() atlas_module.canonical_annotation.cache_clear() atlas_module._region_info_from_id.cache_clear() diff --git a/tests/test_anatomy_visualization.py b/tests/test_anatomy_visualization.py index 4fc66b4..c40b0e2 100644 --- a/tests/test_anatomy_visualization.py +++ b/tests/test_anatomy_visualization.py @@ -29,10 +29,12 @@ def __init__(self, name: str, **_kwargs): @pytest.fixture(autouse=True) def _reset_cache(): + atlas_module.release_atlas_memory() atlas_module.get_atlas.cache_clear() atlas_module.canonical_annotation.cache_clear() atlas_module._region_info_from_id.cache_clear() yield + atlas_module.release_atlas_memory() atlas_module.get_atlas.cache_clear() atlas_module.canonical_annotation.cache_clear() atlas_module._region_info_from_id.cache_clear() diff --git a/tests/test_anatomy_volume.py b/tests/test_anatomy_volume.py new file mode 100644 index 0000000..01d5d10 --- /dev/null +++ b/tests/test_anatomy_volume.py @@ -0,0 +1,210 @@ +"""Tests for the lazy, canonically-oriented annotation reader. + +Two things have to hold, and they pull in opposite directions: + +* **Correctness** — a ``CanonicalVolume`` must be indistinguishable from the + eagerly reoriented array it replaces, for every atlas orientation. The + reference here is literally the expression the old ``canonical_annotation`` + used, so any divergence in the transpose/flip logic shows up immediately. +* **Laziness** — it must not read more of the store than the caller asked for. + Correctness alone would be satisfied by materialising everything, which is + precisely the behaviour being removed, so the reads are spied on. +""" + +from __future__ import annotations + +import itertools + +import numpy as np +import pytest + +from pixelmap.anatomy.atlas import anatomical_axes +from pixelmap.anatomy.volume import CanonicalVolume + +# Every orientation brainglobe can express: one letter per anatomical axis, +# in any axis order — 3! orderings x 2^3 directions. +ALL_ORIENTATIONS = [ + "".join(p) + for axes in itertools.permutations("ASL") + for p in itertools.product(*[{"A": "ap", "S": "si", "L": "lr"}[c] for c in axes]) +] + + +class _FakeAtlas: + def __init__(self, orientation, arr): + self.orientation = orientation + self.annotation = arr + self.shape = arr.shape + self.resolution = (25.0, 25.0, 25.0) + + +class _SpyStore: + """Wraps an ndarray and records every index tuple asked of it.""" + + def __init__(self, arr): + self._arr = arr + self.shape = arr.shape + self.dtype = arr.dtype + self.chunks = (2, 2, 2) + self.reads: list = [] + + def __getitem__(self, key): + self.reads.append(key) + return self._arr[key] + + def read_volume(self) -> int: + """Total voxels pulled out of the store across all reads.""" + total = 0 + for key in self.reads: + shape = self._arr[key].shape + total += int(np.prod(shape)) if shape else 1 + return total + + +def _build(orientation, arr): + """A CanonicalVolume plus the eagerly-reoriented reference array.""" + atlas = _FakeAtlas(orientation, arr) + axes = anatomical_axes(atlas) + vol = CanonicalVolume(arr, axes, (25.0, 25.0, 25.0)) + + order = (axes["AP"].array_axis, axes["DV"].array_axis, axes["ML"].array_axis) + ref = np.transpose(arr, order) + flips = tuple(i for i, k in enumerate(("AP", "DV", "ML")) if axes[k].flip) + if flips: + ref = np.flip(ref, axis=flips) + return vol, ref + + +@pytest.mark.parametrize("orientation", ALL_ORIENTATIONS) +class TestMatchesTheEagerReorientation: + """The lazy reader must agree with transpose+flip on the full array.""" + + ARR = np.arange(4 * 5 * 6, dtype=np.int32).reshape(4, 5, 6) + + def test_shape_and_full_materialisation(self, orientation): + vol, ref = _build(orientation, self.ARR) + assert vol.shape == ref.shape + np.testing.assert_array_equal(np.asarray(vol), ref) + + def test_every_orthogonal_slice(self, orientation): + vol, ref = _build(orientation, self.ARR) + n_ap, n_dv, n_ml = ref.shape + for i in range(n_ap): + np.testing.assert_array_equal(vol[i, :, :], ref[i, :, :]) + for j in range(n_dv): + np.testing.assert_array_equal(vol[:, j, :], ref[:, j, :]) + for k in range(n_ml): + np.testing.assert_array_equal(vol[:, :, k], ref[:, :, k]) + + def test_sub_boxes_and_scalars(self, orientation): + vol, ref = _build(orientation, self.ARR) + rng = np.random.default_rng(0) + n_ap, n_dv, n_ml = ref.shape + for _ in range(15): + a0, a1 = sorted(rng.integers(0, n_ap + 1, 2)) + d0, d1 = sorted(rng.integers(0, n_dv + 1, 2)) + m0, m1 = sorted(rng.integers(0, n_ml + 1, 2)) + np.testing.assert_array_equal( + vol[a0:a1, d0:d1, m0:m1], ref[a0:a1, d0:d1, m0:m1] + ) + for a, d, m in itertools.product(range(n_ap), range(n_dv), range(n_ml)): + assert vol[a, d, m] == ref[a, d, m] + + def test_gather_matches_fancy_indexing(self, orientation): + vol, ref = _build(orientation, self.ARR) + rng = np.random.default_rng(1) + ap = rng.integers(0, ref.shape[0], 50) + dv = rng.integers(0, ref.shape[1], 50) + ml = rng.integers(0, ref.shape[2], 50) + np.testing.assert_array_equal(vol.gather(ap, dv, ml), ref[ap, dv, ml]) + + def test_projections_match_the_eager_silhouette(self, orientation): + arr = self.ARR.copy() + arr[arr % 3 == 0] = 0 # carve holes so the projections aren't all-True + vol, ref = _build(orientation, arr) + inside = ref > 0 + over_ml, over_ap, over_dv = vol.projections() + np.testing.assert_array_equal(over_ml, inside.any(axis=2)) + np.testing.assert_array_equal(over_ap, inside.any(axis=0)) + np.testing.assert_array_equal(over_dv, inside.any(axis=1)) + + def test_find_label_voxels_matches_isin(self, orientation): + vol, ref = _build(orientation, self.ARR) + ids = [int(v) for v in np.unique(ref)[3:7]] + got = vol.find_label_voxels(ids) + expected = np.where(np.isin(ref, ids)) + for g, e in zip(got, expected): + np.testing.assert_array_equal(np.sort(g), np.sort(e)) + + +class TestReadsAreBounded: + """The whole point: never pull more out of the store than was asked for.""" + + ARR = np.arange(8 * 8 * 8, dtype=np.int32).reshape(8, 8, 8) % 5 + + def test_a_2d_slice_does_not_read_the_volume(self): + atlas = _FakeAtlas("asr", self.ARR) + spy = _SpyStore(self.ARR) + vol = CanonicalVolume(spy, anatomical_axes(atlas), (25.0, 25.0, 25.0)) + + vol[4, :, :] + + assert spy.read_volume() == 8 * 8, ( + f"read {spy.read_volume()} voxels for a 64-voxel slice — the " + "whole volume is being materialised" + ) + + def test_gather_reads_only_the_bounding_box(self): + atlas = _FakeAtlas("asr", self.ARR) + spy = _SpyStore(self.ARR) + vol = CanonicalVolume(spy, anatomical_axes(atlas), (25.0, 25.0, 25.0)) + + ap = np.array([1, 2, 1]) + dv = np.array([0, 1, 1]) + ml = np.array([5, 6, 5]) + out = vol.gather(ap, dv, ml) + + np.testing.assert_array_equal(out, self.ARR[ap, dv, ml]) + assert len(spy.reads) == 1, "gather must issue exactly one read" + assert spy.read_volume() == 2 * 2 * 2, ( + f"read {spy.read_volume()} voxels for a 2x2x2 bounding box" + ) + + def test_whole_volume_scans_are_slabbed_not_materialised(self, monkeypatch): + import pixelmap.anatomy.volume as volume_module + + # One slab per 2 AP planes, so a full scan needs several reads. + monkeypatch.setattr(volume_module, "_SLAB_BYTES", 2 * 8 * 8 * 4) + atlas = _FakeAtlas("asr", self.ARR) + spy = _SpyStore(self.ARR) + vol = CanonicalVolume(spy, anatomical_axes(atlas), (25.0, 25.0, 25.0)) + + vol.projections() + + assert len(spy.reads) > 1, "a full scan must be slabbed, not one big read" + biggest = max(int(np.prod(self.ARR[k].shape)) for k in spy.reads) + assert biggest < self.ARR.size, ( + "a single slab pulled the whole volume — peak RAM is unbounded" + ) + + +class TestRejectsUnsupportedIndexing: + """Fail loudly rather than silently returning the wrong voxels.""" + + ARR = np.zeros((4, 4, 4), dtype=np.int32) + + def _vol(self): + atlas = _FakeAtlas("asr", self.ARR) + return CanonicalVolume(self.ARR, anatomical_axes(atlas), (25.0, 25.0, 25.0)) + + def test_wrong_number_of_axes(self): + with pytest.raises(IndexError): + self._vol()[1, 2] + + def test_strided_slice(self): + with pytest.raises(TypeError): + self._vol()[::2, :, :] + + def test_fancy_indexing_points_at_gather(self): + with pytest.raises(TypeError): + self._vol()[np.array([0, 1]), :, :] diff --git a/tests/test_gui_overlays.py b/tests/test_gui_overlays.py index 1324ec7..eae78dc 100644 --- a/tests/test_gui_overlays.py +++ b/tests/test_gui_overlays.py @@ -14,6 +14,7 @@ import asyncio import inspect import threading +import time import numpy as np import pandas as pd @@ -85,6 +86,7 @@ def __init__(self, name, **_kwargs): 2: {"acronym": "SUP", "name": "Superficial region", "rgb_triplet": [200, 100, 50]}, } + atlas_module.release_atlas_memory() atlas_module.get_atlas.cache_clear() atlas_module.canonical_annotation.cache_clear() atlas_module._region_info_from_id.cache_clear() @@ -92,6 +94,7 @@ def __init__(self, name, **_kwargs): # Pretend it's cached so the GUI treats it as a local atlas throughout. monkeypatch.setattr(atlas_module, "is_downloaded", lambda name=FAKE_ATLAS: True) yield ann + atlas_module.release_atlas_memory() atlas_module.get_atlas.cache_clear() atlas_module.canonical_annotation.cache_clear() atlas_module._region_info_from_id.cache_clear() @@ -696,3 +699,67 @@ def test_no_unlocked_async_gui_callbacks(): f"async GUI callbacks missing @pn.io.with_lock: {unlocked}. Panel " "schedules these nolock, so any Bokeh model write inside them raises." ) + + +class TestClearOverlayReleasesAtlasMemory: + """"Clear overlay" must hand the annotation volume back, not just blank the plot. + + Blanking the Bokeh sources left the atlas resident — 308 MB for + allen_mouse_25um, up to 4.8 GB for a 10 µm atlas — so a user who cleared the + overlay saw RSS stay exactly where it was. + """ + + @staticmethod + def _wait_for_release(timeout: float = 5.0) -> bool: + """The release runs on a daemon thread; give it a moment to land.""" + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + if atlas_module._resident is None: + return True + time.sleep(0.01) + return atlas_module._resident is None + + def test_clearing_the_overlay_releases_the_atlas(self, posed_gui): + posed_gui.compute_anatomy_overlay() + assert atlas_module._resident is not None, "overlay should have loaded an atlas" + + posed_gui.clear_anatomy_overlay() + + assert self._wait_for_release(), "clear overlay left the atlas resident in RAM" + + def test_recomputing_after_a_clear_still_works(self, posed_gui): + """Releasing must be transparent — the next compute reloads from disk.""" + posed_gui.compute_anatomy_overlay() + posed_gui.clear_anatomy_overlay() + assert self._wait_for_release() + + posed_gui.compute_anatomy_overlay() + assert atlas_module._resident is not None + assert len(posed_gui.region_band_source.data["acronym"]) > 0, ( + "overlay did not come back after a release" + ) + + +class TestLocatorFigureIsReusedAcrossRenders: + """The GUI must keep one locator figure per session, not one per render.""" + + def test_recomputing_reuses_the_same_figure(self, posed_gui): + posed_gui.compute_anatomy_overlay() + first = posed_gui._locator_fig + assert first is not None, "no locator figure was retained" + + for _ in range(4): + posed_gui.compute_anatomy_overlay() + + assert posed_gui._locator_fig is first, ( + "a new figure was built per overlay compute — 3.1 MB of RSS each, " + "on a path that re-runs for every pose edit" + ) + + def test_clearing_the_overlay_drops_the_figure(self, posed_gui): + posed_gui.compute_anatomy_overlay() + assert posed_gui._locator_fig is not None + posed_gui.clear_anatomy_overlay() + assert posed_gui._locator_fig is None, ( + "an idle session is still holding a matplotlib canvas" + ) diff --git a/tests/test_selection_responsiveness.py b/tests/test_selection_responsiveness.py new file mode 100644 index 0000000..46ce316 --- /dev/null +++ b/tests/test_selection_responsiveness.py @@ -0,0 +1,79 @@ +"""Regression tests: an electrode click must not stall the server. + +Production logs showed the single-threaded Bokeh event loop freezing for +20-90 s after nearly every ``Selection changed: [] -> [N]``. The handler ran a +full ``gc.collect()`` on every tap, which walks every live object in a process +holding several GB and hundreds of thousands of Bokeh models. While frozen, +the healthcheck failed and the reverse proxy dropped users' websockets. +""" + +from __future__ import annotations + +import pytest + +from pixelmap.gui import gui as gui_module +from pixelmap.gui.gui import ChannelmapGUI +from pixelmap.types import Electrode + + +@pytest.fixture(scope="module") +def gui(): + """One GUI instance for the module — construction loads all probe data.""" + return ChannelmapGUI() + + +@pytest.fixture(autouse=True) +def clean_selection(gui): + gui.electrodes.clear_selection() + gui.select_mode = "select" + yield + gui.electrodes.clear_selection() + + +@pytest.fixture +def gc_calls(monkeypatch): + """Count full collections triggered from the GUI module.""" + calls = [] + monkeypatch.setattr(gui_module.gc, "collect", lambda *a, **k: calls.append(a) or 0) + return calls + + +def _electrode_at(gui, idx): + data = gui.electrode_source.data + return Electrode(data["shank_id"][idx], data["electrode_id"][idx]) + + +def test_single_tap_does_not_force_gc(gui, gc_calls): + gui.on_electrode_selection("indices", [], [0]) + assert gc_calls == [] + + +def test_box_select_does_not_force_gc(gui, gc_calls): + gui.on_electrode_selection("indices", [], [0, 1, 2, 3]) + assert gc_calls == [] + + +def test_many_taps_do_not_force_gc(gui, gc_calls): + for _ in range(20): + gui.on_electrode_selection("indices", [], [0]) + assert gc_calls == [] + + +def test_single_tap_still_toggles(gui): + target = _electrode_at(gui, 0) + gui.on_electrode_selection("indices", [], [0]) + assert target in gui.electrodes.selected + gui.on_electrode_selection("indices", [], [0]) + assert target not in gui.electrodes.selected + + +def test_tap_clears_bokeh_selection(gui): + gui.on_electrode_selection("indices", [], [0]) + assert gui.electrode_source.selected.indices == [] + + +@pytest.mark.parametrize("empty", [[], None]) +def test_empty_selection_is_noop(gui, empty): + """The handler's own reset (``indices = []``) re-fires it with an empty list.""" + gui.on_electrode_selection("indices", [0], empty) + assert gui.electrodes.selected == set() diff --git a/tests/test_watchdog.py b/tests/test_watchdog.py new file mode 100644 index 0000000..7212f82 --- /dev/null +++ b/tests/test_watchdog.py @@ -0,0 +1,224 @@ +"""Tests for the event-loop stall and memory watchdogs. + +They exist so production problems show up as a loud log line with a stack +trace (which the Slack log forwarder picks up) long before the container gets +OOM-killed or restarted for failing its healthcheck. +""" + +from __future__ import annotations + +import asyncio +import logging +import threading + +import pytest + +from pixelmap.gui import watchdog +from pixelmap.gui.watchdog import MemoryWatch, StallWatchdog + + +class FakeClock: + def __init__(self, t: float = 1000.0): + self.t = t + + def __call__(self) -> float: + return self.t + + +@pytest.fixture +def clock(): + return FakeClock() + + +@pytest.fixture +def caplog_warn(caplog): + caplog.set_level(logging.INFO, logger="pixelmap.watchdog") + return caplog + + +def _warnings(caplog): + return [r for r in caplog.records if r.levelno == logging.WARNING] + + +# -------------------------------------------------------------------------- +# StallWatchdog +# -------------------------------------------------------------------------- + +@pytest.mark.parametrize("bad", [0, -1, -0.5]) +def test_stall_threshold_must_be_positive(bad): + with pytest.raises(ValueError): + StallWatchdog(threshold_s=bad) + + +def test_no_report_below_threshold(clock, caplog_warn): + wd = StallWatchdog(threshold_s=5, clock=clock) + wd.beat() + clock.t += 4.99 + assert wd.check() is False + assert _warnings(caplog_warn) == [] + + +def test_report_at_exact_threshold(clock, caplog_warn): + wd = StallWatchdog(threshold_s=5, clock=clock) + wd.beat() + clock.t += 5 + assert wd.check() is True + + +def test_stall_reported_once_per_stall(clock, caplog_warn): + wd = StallWatchdog(threshold_s=5, clock=clock) + wd.beat() + clock.t += 6 + assert wd.check() is True + clock.t += 30 + assert wd.check() is False + assert len(_warnings(caplog_warn)) == 1 + assert "[watchdog]" in _warnings(caplog_warn)[0].getMessage() + + +def test_recovery_logs_duration_and_rearms(clock, caplog_warn): + wd = StallWatchdog(threshold_s=5, clock=clock) + wd.beat() + clock.t += 42 + wd.check() + wd.beat() # loop came back + recovered = [r for r in caplog_warn.records if "recovered" in r.getMessage()] + assert len(recovered) == 1 + assert "42.0" in recovered[0].getMessage() + + # A second, separate stall is reported again. + clock.t += 10 + assert wd.check() is True + assert len([r for r in _warnings(caplog_warn) if "blocked for" in r.getMessage()]) == 2 + + +def test_beat_without_stall_logs_nothing(clock, caplog_warn): + wd = StallWatchdog(threshold_s=5, clock=clock) + for _ in range(3): + wd.beat() + clock.t += 1 + assert caplog_warn.records == [] + + +def test_check_before_first_beat_does_not_raise(clock, caplog_warn): + """Before the loop ever beats there is no thread to dump.""" + wd = StallWatchdog(threshold_s=5, clock=clock) + clock.t += 10 + assert wd.check() is True + assert "unavailable" in _warnings(caplog_warn)[0].getMessage() + + +def test_report_includes_blocked_threads_stack(clock, caplog_warn): + """The whole point: the warning says *what* is blocking the loop.""" + wd = StallWatchdog(threshold_s=5, clock=clock) + beaten = threading.Event() + release = threading.Event() + + def _blocking_handler(): + wd.beat() + beaten.set() + release.wait(timeout=10) + + t = threading.Thread(target=_blocking_handler, daemon=True) + t.start() + try: + assert beaten.wait(timeout=5) + clock.t += 10 + assert wd.check() is True + finally: + release.set() + t.join(timeout=5) + assert "_blocking_handler" in _warnings(caplog_warn)[0].getMessage() + + +# -------------------------------------------------------------------------- +# MemoryWatch +# -------------------------------------------------------------------------- + +@pytest.mark.parametrize("limit", [None, 0, -100]) +def test_memory_watch_disabled(limit, caplog_warn): + mw = MemoryWatch(limit_mb=limit) + assert mw.check(10_000_000) is False + assert caplog_warn.records == [] + + +def test_memory_below_limit_is_quiet(caplog_warn): + mw = MemoryWatch(limit_mb=1000) + assert mw.check(999.9) is False + assert caplog_warn.records == [] + + +def test_memory_at_limit_warns(caplog_warn): + mw = MemoryWatch(limit_mb=1000) + assert mw.check(1000) is True + assert "[watchdog]" in _warnings(caplog_warn)[0].getMessage() + + +def test_memory_warns_once_until_rearmed(caplog_warn): + mw = MemoryWatch(limit_mb=1000, rearm_fraction=0.9) + assert mw.check(1500) is True + assert mw.check(1600) is False + # Dipping below the limit but above the re-arm point doesn't re-arm -- + # otherwise RSS hovering around the limit would spam the channel. + assert mw.check(950) is False + assert mw.check(1100) is False + # Dropping below 90% re-arms. + assert mw.check(899) is False + assert mw.check(1200) is True + assert len(_warnings(caplog_warn)) == 2 + + +@pytest.mark.parametrize("bad", [0, -0.1, 1.5]) +def test_memory_rearm_fraction_validated(bad): + with pytest.raises(ValueError): + MemoryWatch(limit_mb=1000, rearm_fraction=bad) + + +# -------------------------------------------------------------------------- +# Environment config +# -------------------------------------------------------------------------- + +@pytest.mark.parametrize( + "raw, expected", + [(None, 7.0), ("", 7.0), ("12", 12.0), ("2.5", 2.5), ("0", 0.0), ("nope", 7.0)], +) +def test_env_float(monkeypatch, raw, expected): + if raw is None: + monkeypatch.delenv("PIXELMAP_TEST_KNOB", raising=False) + else: + monkeypatch.setenv("PIXELMAP_TEST_KNOB", raw) + assert watchdog._env_float("PIXELMAP_TEST_KNOB", 7.0) == expected + + +# -------------------------------------------------------------------------- +# install() +# -------------------------------------------------------------------------- + +@pytest.fixture +def fresh_install(monkeypatch): + monkeypatch.setattr(watchdog, "_installed", None) + yield + if watchdog._installed is not None: + watchdog._installed.stop() + + +def test_install_is_idempotent(fresh_install): + """app.py is re-executed per session; the watchdog must start once per process.""" + + async def run(): + first = watchdog.install(stall_threshold_s=5, rss_warn_mb=None) + second = watchdog.install(stall_threshold_s=5, rss_warn_mb=None) + return first, second + + first, second = asyncio.run(run()) + assert first is not None + assert first is second + assert sum(t.name == "pixelmap-watchdog" for t in threading.enumerate()) == 1 + + +@pytest.mark.parametrize("threshold", [0, -1]) +def test_install_disabled_by_nonpositive_threshold(fresh_install, threshold): + async def run(): + return watchdog.install(stall_threshold_s=threshold, rss_warn_mb=None) + + assert asyncio.run(run()) is None