Repository navigation
Conversation
This was referenced Sep 18, 2026
Owner
Author
|
🤖 Human guided, AI assisted comment (using this skill). AI text below. 🤖 FindingsRSS trace loading So: eviction freed nothing, reload duplicated, and "freed" never left RSS. After this PR the same trace reads 520 / 520 / 214 MB. Assisted-by: Claude-Code:Opus-5 |
tabedzki
force-pushed
the
fix/atlas-volume-cache
branch
from
September 18, 2026 15:47
5bfcf76 to
7dc418f
Compare
Loading atlases on the hosted server leaked memory in three ways, all rooted in caching decoded annotation volumes with entry-counted lru_caches: * `get_atlas`, `canonical_annotation` and `schematic._atlas_data` were three independent LRUs (maxsize=4 each) that all held the same arrays, and brainglobe caches the array on the atlas object too. They evicted on separate schedules: once `get_atlas` dropped an atlas, the transposed view in `canonical_annotation` still pinned its base, and the next `ensure_downloaded` re-decoded a second copy (measured 778 -> 1088 MB for allen_mouse_25um). * `maxsize=4` bounds a count, not bytes: four 38 MB atlases or four 4.8 GB ones. It never related to the container's memory limit. * Dropping the last reference didn't lower RSS. glibc keeps freed pages in its arenas, so after clearing every cache the process still sat at 1,188 MB until malloc_trim (then 211 MB). The cgroup limit counts RSS, so evictions were invisible to the OOM killer. Now a single `_VolumeCache` owns every decoded volume, keyed by atlas name and bounded in bytes (PIXELMAP_ATLAS_CACHE_MB, defaulting to half the cgroup limit). `get_atlas` holds metadata-only objects: the volume is detached from the brainglobe instance on load. Every load and eviction ends with gc + malloc_trim, so RSS follows the cache. Concurrent requests for the same atlas wait for one decode instead of running two. Measured on real atlases: allen_mouse_25um settles at 520 MB instead of 778; evict-and-reload stays at 520 instead of 1,088; clearing returns to 214 MB instead of 1,188. The ~2x transient peak while decoding is unchanged and addressed separately. Assisted-by: Claude-Code:Opus-5
tabedzki
force-pushed
the
fix/atlas-volume-cache
branch
from
September 18, 2026 15:49
7dc418f to
0652556
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 Human guided, AI assisted PR (using this skill). AI text below. 🤖
Problem. Decoded atlas volumes were held by three independent
lru_caches plus brainglobe's own_annotation. Evicting one freed nothing (the others hold views of the same array), the next load decoded a duplicate (778 → 1,088 MB),maxsize=4counted entries not bytes, and freed arrays stayed in glibc's heap (1,188 MB RSS after clearing everything; 211 MB aftermalloc_trim).Change. One byte-bounded
_VolumeCacheowns all volumes (PIXELMAP_ATLAS_CACHE_MB, default ½ the cgroup limit).get_atlaskeeps metadata only.malloc_trimafter every load/eviction. Concurrent loads of one atlas share a decode.allen_mouse_25umresidentDecode peak unchanged; see #3.
_memory.pyis identical to #4's. #3 touches the same lines; second to land needs a small rebase (combined tree: 243 tests pass).Assisted-by: Claude-Code:Opus-5