Skip to content

fix: one byte-bounded owner for atlas volumes, trim heap on eviction - #2

Closed
tabedzki wants to merge 1 commit into
mainfrom
fix/atlas-volume-cache
Closed

tabedzki wants to merge 1 commit into
mainfrom
fix/atlas-volume-cache

Conversation

@tabedzki

@tabedzki tabedzki commented Sep 18, 2026 •

Copy link
Copy Markdown
Owner

🤖 Human guided, AI assisted PR (using this skill). AI text below. 🤖

Problem. Decoded atlas volumes were held by three independent lru_caches plus brainglobe's own _annotation. Evicting one freed nothing (the others hold views of the same array), the next load decoded a duplicate (778 → 1,088 MB), maxsize=4 counted entries not bytes, and freed arrays stayed in glibc's heap (1,188 MB RSS after clearing everything; 211 MB after malloc_trim).

Change. One byte-bounded _VolumeCache owns all volumes (PIXELMAP_ATLAS_CACHE_MB, default ½ the cgroup limit). get_atlas keeps metadata only. malloc_trim after every load/eviction. Concurrent loads of one atlas share a decode.

before after
allen_mouse_25um resident 778 MB 520 MB
evict + reload 1,088 MB 520 MB
clear 1,188 MB 214 MB

Decode peak unchanged; see #3. _memory.py is identical to #4's. #3 touches the same lines; second to land needs a small rebase (combined tree: 243 tests pass).

Assisted-by: Claude-Code:Opus-5

@tabedzki

tabedzki commented Sep 18, 2026 •

Copy link
Copy Markdown
Owner Author

🤖 Human guided, AI assisted comment (using this skill). AI text below. 🤖

Findings

RSS trace loading allen_mouse_25um (308 MB array):

annotation loaded                        778 MB
get_atlas.cache_clear()                  778 MB   canonical view still pins it (arr.base is ann)
ensure_downloaded again                 1088 MB   second copy
rat loaded, all caches cleared          1188 MB   glibc keeps freed pages
malloc_trim(0)                           211 MB

So: eviction freed nothing, reload duplicated, and "freed" never left RSS. lru_cache can't fix this — it bounds entries not bytes, two caches can't share a lifetime, and eviction has no trim hook. A third owner, schematic._atlas_data, pinned both the atlas and the array too.

After this PR the same trace reads 520 / 520 / 214 MB.

Assisted-by: Claude-Code:Opus-5

@tabedzki
tabedzki force-pushed the fix/atlas-volume-cache branch from 5bfcf76 to 7dc418f Compare September 18, 2026 15:47
Loading atlases on the hosted server leaked memory in three ways, all
rooted in caching decoded annotation volumes with entry-counted lru_caches:

* `get_atlas`, `canonical_annotation` and `schematic._atlas_data` were
  three independent LRUs (maxsize=4 each) that all held the same arrays,
  and brainglobe caches the array on the atlas object too. They evicted on
  separate schedules: once `get_atlas` dropped an atlas, the transposed
  view in `canonical_annotation` still pinned its base, and the next
  `ensure_downloaded` re-decoded a second copy (measured 778 -> 1088 MB
  for allen_mouse_25um).
* `maxsize=4` bounds a count, not bytes: four 38 MB atlases or four 4.8 GB
  ones. It never related to the container's memory limit.
* Dropping the last reference didn't lower RSS. glibc keeps freed pages in
  its arenas, so after clearing every cache the process still sat at
  1,188 MB until malloc_trim (then 211 MB). The cgroup limit counts RSS,
  so evictions were invisible to the OOM killer.

Now a single `_VolumeCache` owns every decoded volume, keyed by atlas
name and bounded in bytes (PIXELMAP_ATLAS_CACHE_MB, defaulting to half
the cgroup limit). `get_atlas` holds metadata-only objects: the volume is
detached from the brainglobe instance on load. Every load and eviction
ends with gc + malloc_trim, so RSS follows the cache. Concurrent requests
for the same atlas wait for one decode instead of running two.

Measured on real atlases: allen_mouse_25um settles at 520 MB instead of
778; evict-and-reload stays at 520 instead of 1,088; clearing returns to
214 MB instead of 1,188. The ~2x transient peak while decoding is
unchanged and addressed separately.

Assisted-by: Claude-Code:Opus-5
@tabedzki
tabedzki force-pushed the fix/atlas-volume-cache branch from 7dc418f to 0652556 Compare September 18, 2026 15:49
@tabedzki tabedzki closed this Sep 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant