feat(server): add --idle-unload-ms and pre-load memory guard - #306
Conversation
The 5-minute idle unload used to live in an external Python monitor inside the audio-server wrapper, which polled the server log mtime and called /v1/tasks/unload_all_models. Move it into audiocpp_server itself: - new ServerConfig field idle_unload_ms (default 0 = disabled), parsed from server.json and overridable via --idle-unload-ms - a background thread unloads every resident non-busy model once the server has been idle that long without a model load/run; the next request reloads lazily This drops the log-mtime heuristic (which required --log trace spam) and removes the need for the external Python wrapper.
Before every lazy model load, estimate the model's resident footprint (weights plus runtime overhead) and compare against free host memory and, for GPU backends, the backend device's free memory. Refuse the load with HTTP 503 insufficient_memory when estimate + configured headroom does not fit, instead of exhausting the machine (the previous failure mode was kIOGPUCommandBufferCallbackErrorOutOfMemory after models accumulated on a 16GB Mac). - add ServerConfig.min_free_memory_mb (default 512 MiB headroom), parsed from server.json and overridable via --min-free-memory-mb - add a macOS implementation of available_host_memory_bytes() using Mach VM stats (free + inactive + purgeable pages); Linux/Windows were already covered - add InsufficientMemoryError, mapped to 503 insufficient_memory
Independent review found no criticals; fix the actionable findings: - --min-free-memory-mb help text now matches the actual 512 MiB default - estimate_model_memory_bytes() sums directory-style model trees (with depth/file limits) instead of counting only regular files, so directory models are no longer estimated as 0 - the model load path is serialized even when max_loaded_models is 0, so concurrent lazy loads cannot both pass the memory pre-check - expose engine::core::ensure_backends_loaded() and call it before the GPU memory query so the very first load actually runs the device check - the idle-unload thread now wakes on shutdown in <=250ms slices instead of waiting out a full poll interval - document idle_unload_ms / min_free_memory_mb in app/server/README.md and example.json - add server_config_test coverage for the new fields (defaults, overrides, negative rejection)
Run audiocpp_server build + server_config_test on ubuntu/windows/macos to prove the idle-unload and pre-load memory-check changes compile and behave on all three desktop platforms. GPU backends are off here (the changes query memory through the backend-agnostic ggml_backend_dev_memory; existing workflows already cover CUDA/Vulkan/Metal builds).
The Linux build step used nproc and its condition also matched macOS, so macOS ran both build steps and the nproc one stalled. Scope the nproc step to Linux and let macOS use sysctl -n hw.logicalcpu.
|
Hi! This is my first contribution to this repo, so the workflow runs are waiting for maintainer approval (marked For reference, the same matrix already passed on my fork (ubuntu-24.04 / windows-2022 / macos-14, |
|
@gqf2008 Thanks! I'm glad the Metal backend is starting to get more attention. Will review the PR this morning. |
|
@gqf2008 Some issues during testing:
It can also mask the real loader error. I validated this with a multi-GGUF Chatterbox directory:
So the direct selected file can load, but the directory path is rejected earlier by the new memory estimator.
Same two concurrent /v1/models/load requests: Chatterbox Q8 and F16. Main: PR: |
Address 0xShug0's review on 0xShug0#306: 1. Make the memory guard opt-in: min_free_memory_mb now defaults to 0 and 0 disables ensure_model_fits_memory entirely, so existing configs that never opted in see no load-behavior change. 2. Stop over-summing directory models and stop masking the real loader error. estimate_model_memory_bytes now estimates only what the loader will read: a single file, the one GGUF a directory selects (find_directory_gguf), or a full safetensors/HF tree. A directory with several GGUFs and no model.gguf is ambiguous, so the guard estimates nothing and the loader's own "contains N GGUF files" error surfaces instead of a misleading 503. 3. Resolve relative session-option paths against the model directory when model.path is a directory (parent_path only when it is a file). 4. Measure idle time from request completion, not start: run_model and run_streaming_model_impl now stamp last_activity_ms_ on completion, so a long inference is not unloaded the moment it returns. 5. Only serialize lazy loads through model_load_mutex_ when a guard needs it (max_loaded_models > 0 or min_free_memory_mb > 0); with both off, unrelated first-load requests keep their original concurrency. Verified on macOS (Apple M4): audiocpp_server + server_config_test build and pass; startup smoke shows a single-GGUF dir estimates just the selected file (10.83 GiB -> 503 when guard on), and an ambiguous multi-GGUF dir surfaces the real "contains 2 GGUF files" loader error with the guard both on and off.
Brings in c79e588 (tag-driven release pipeline, CI/docs only) and re-triggers upstream CI; the prior Nix (vulkan) failure was a flaky ggml-vulkan shader-gen link error (same tree builds green on vulkan on the fork).
|
@gqf2008 Thanks a lot! I’m going to merge the PR. There is still one minor issue, but it’s not a blocker. A follow-up PR would be appreciated. For issue (2) My test: With With |
|
@0xShug0 Thanks for the thorough testing — all five points are addressed in 1. Guard is now opt-in. 2. Directory estimation no longer over-sums or masks the loader error. The estimator now measures only what the loader will read: a single file; the one GGUF a directory selects via 3. Relative option paths now resolve against 4. Idle clock measures from completion. 5. Lazy loads serialize only when a guard needs it. CI is green across the board (Linux cpu/vulkan, Nix cpu/vulkan, Windows, macOS, and the memory-guard build+test matrix). The earlier |
|
Thanks for merging! The minor follow-up on issue (2) is addressed in #308: an ambiguous multi-GGUF directory now reports an indeterminate footprint, so the guard skips it and the loader's own |
Merge of PR #24 (conflict only in .gitignore: kept fork rules plus upstream's audio-vtest-*). Brings AudioSR (speech-to-speech enhancement, existing task kind), ControlFoley, FireRed (TTS3/Audio + reusable speech modules: s3_tokenizer, HiFi-GAN), MiDashengLM-Gen decode lifecycle fixes, server --idle-unload-ms + pre-load memory guard (0xShug0#306), tag-driven prebuilt release CI (0xShug0#286), and Arena comparison WebUI. Shared-module changes verified additive (Conv3d new; qwen_causal_decode sliding_window defaults to legacy path); upstream did not touch BackendWeightStore, so fork's kMetadataPoolBudget over-commit fix stays intact. fork_regression 9/9 green, loader catalog in sync. Generated-by: zcode
…ate (#308) * fix(server): skip memory guard when the model footprint is indeterminate Follow-up to #306 per maintainer review: a model directory holding several GGUFs and no model.gguf is ambiguous, and the loader rejects it with its own "contains N GGUF files" error. The estimator already contributes no weights for such a directory, but ensure_model_fits_memory still compared the remaining fixed floor plus the configured headroom against free memory, so a large headroom (e.g. min_free_memory_mb=1000000) still answered 503 and masked the real loader error. estimate_model_memory_bytes now returns nullopt for that ambiguous case, and ensure_model_fits_memory skips the guard entirely when the footprint is indeterminate: the load can never allocate anyway, so the loader's error surfaces no matter how large the headroom is. Determinate footprints (single file, selected GGUF, safetensors/HF tree) still guard as before. Verified on macOS: ambiguous 2-GGUF dir with min_free_memory_mb=1000000 now fails with the loader's "contains 2 GGUF files" error (was 503); the same headroom on a single-GGUF dir still 503s; guard-off behavior unchanged; server_config_test passes. * fix(server): harden model memory estimation and add regression tests Code-review follow-up to the ambiguous-directory skip: - extract estimate_model_memory_bytes into app/server/model_memory.[h|cpp] so the estimator is unit-testable instead of a private ServerState member - list the directory once, mirroring the loader's selection: model.gguf wins, the sole *.gguf is used alone, and several GGUFs without model.gguf stay ambiguous - ignore files whose size cannot be read instead of folding the file_size failure value into the sum - log when the guard skips an indeterminate model instead of failing silently: family-specific layouts (e.g. minimax_music3) may load such a directory successfully, so a skipped guard is worth surfacing - fix the --min-free-memory-mb help text and README to describe the opt-in default and the skip behavior - add estimator tests to server_config_test: single file, sole GGUF, model.gguf disambiguation, ambiguous directory, checkpoint tree, and relative aux resolution * fix(test): avoid Windows-reserved directory name in estimator tests "aux" is a reserved DOS device name, so creating .../aux under the temp root throws on Windows and fails server_config_test there. Rename the test directory to "sidecar".
2 participants
Motivation
On a 16GB Apple Silicon machine, a long-running batch transcription loaded model after model with no eviction or idle unload, exhausting unified memory and surfacing as
kIOGPUCommandBufferCallbackErrorOutOfMemory+ HTTP 500s. This PR givesaudiocpp_servertwo server-side memory controls that complement the existingmax_loaded_models(#298).Changes
--idle-unload-ms <ms>(idle_unload_msin server.json, default0= disabled): a background thread unloads every resident non-busy model once the server has gone that long without a model load/run. The next request reloads lazily. Replaces the previous external Python monitor (which polled the log mtime and required--logtrace spam); no wrapper process needed.--min-free-memory-mb <mb>(min_free_memory_mbin server.json, default512,0disables the extra headroom): before every lazy load, estimate the model footprint (weights, including session auxiliary files and directory trees, × 1.5 + 128 MiB floor) and refuse with HTTP 503insufficient_memorywhen estimate + headroom does not fit the free host memory (WindowsGlobalMemoryStatusEx/ LinuxMemAvailable/ new macOS Mach VM stats) or the backend device memory (ggml_backend_dev_memory, CUDA/HIP/Vulkan/Metal; CPU skips the device check).max_loaded_models = 0, so concurrent lazy loads cannot both pass the pre-check.engine::core::available_host_memory_bytes().Verification
server_config_testpassed; smoke tests for 503 refusal on low host/GPU memory, normal load, and idle auto-unload (15 s timeout).audiocpp_serverbuild +server_config_testpassed on ubuntu-24.04 / windows-2022 / macos-14.Notes
--idle-unload-msand--min-free-memory-mbboth default to safe values (0 / 512) so existing deployments see no behavior change unless configured.trim_backend_poolson the unload path as a follow-up.