feat(server): add --idle-unload-ms and pre-load memory guard

.github/workflows/server-memory-guard.yml

@@ -0,0 +1,80 @@
+name: server-memory-guard cross-platform
+
+# Verifies the server memory-management changes (--idle-unload-ms and
+# --min-free-memory-mb) compile and the config unit tests pass on all three
+# desktop platforms. GPU backends are disabled here: the changes themselves are
+# backend-agnostic (they query memory through ggml_backend_dev_memory), and the
+# existing linux/mac/windows workflows already cover CUDA/Vulkan/Metal builds.
+
+on:
+  workflow_dispatch:
+  pull_request:
+    paths:
+      - "app/server/**"
+      - "src/framework/core/host_memory.cpp"
+      - "include/engine/framework/core/backend.h"
+      - "src/framework/core/backend.cpp"
+      - "tests/unittests/test_server_config.cpp"
+      - ".github/workflows/server-memory-guard.yml"
+
+jobs:
+  build-test:
+    name: ${{ matrix.os }} build+test
+    runs-on: ${{ matrix.os }}
+    timeout-minutes: 45
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-24.04, windows-2022, macos-14]
+
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Configure (Linux/macOS)
+        if: runner.os != 'Windows'
+        run: |
+          cmake -S . -B build \
+            -DCMAKE_BUILD_TYPE=Release \
+            -DENGINE_BUILD_TESTS=ON \
+            -DENGINE_ENABLE_OPENMP=OFF \
+            -DENGINE_ENABLE_CUDA=OFF \
+            -DENGINE_ENABLE_VULKAN=OFF \
+            -DENGINE_ENABLE_METAL=OFF
+
+      - name: Configure (Windows)
+        if: runner.os == 'Windows'
+        shell: pwsh
+        run: |
+          cmake -S . -B build `
+            -DCMAKE_BUILD_TYPE=Release `
+            -DENGINE_BUILD_TESTS=ON `
+            -DENGINE_ENABLE_OPENMP=OFF `
+            -DENGINE_ENABLE_CUDA=OFF `
+            -DENGINE_ENABLE_VULKAN=OFF `
+            -DENGINE_ENABLE_METAL=OFF
+
+      - name: Build (Linux)
+        if: runner.os == 'Linux'
+        run: |
+          cmake --build build --target audiocpp_server server_config_test --parallel "$(nproc)"
+
+      - name: Build (macOS)
+        if: runner.os == 'macOS'
+        run: |
+          cmake --build build --target audiocpp_server server_config_test --parallel "$(sysctl -n hw.logicalcpu)"
+
+      - name: Build (Windows)
+        if: runner.os == 'Windows'
+        shell: pwsh
+        run: |
+          cmake --build build --config Release --target audiocpp_server server_config_test --parallel
+
+      - name: Run server_config_test (Linux/macOS)
+        if: runner.os != 'Windows'
+        run: ./build/bin/server_config_test
+
+      - name: Run server_config_test (Windows)
+        if: runner.os == 'Windows'
+        shell: pwsh
+        run: .\build\bin\Release\server_config_test.exe

app/server/README.md

@@ -97,6 +97,10 @@ Set top-level `"lazy_load": true` to register all configured model ids at startu
 
 Set top-level `"max_loaded_models"` to bound how many models are resident in memory at once. When a request needs a model that is not loaded and the limit is already reached, the server first unloads the least recently used idle model (freeing VRAM on GPU backends) and reloads it on its own next request. `1` enforces a single loaded model at a time, which is the practical choice when each model alone nearly fills the device. Higher values keep that many most recently used models warm. The default `0` disables the limit. A model that is mid-inference is never unloaded; if the limit is reached and every loaded model is busy, the request fails with `503` so the client can retry. With more non-lazy models configured than the limit allows, startup loads the first `max_loaded_models` of them and defers the rest to their first request. The equivalent command-line option is `--max-loaded-models <n>`.
 
+Set top-level `"idle_unload_ms"` to have the server unload every resident model after it has gone that long without any model load/run. The next request reloads lazily; a model mid-inference is never unloaded. This complements `max_loaded_models`: that bounds peak residency, this frees memory during quiet periods. Defaults to `0` (disabled). The `--idle-unload-ms <ms>` command-line flag overrides the config value.
+
+Set top-level `"min_free_memory_mb"` to refuse a model load when the host or the GPU backend does not have that much free memory left after the estimated footprint of the new model. The estimate covers only what the loader will actually read: a single-file model's weights, the one GGUF a model directory selects (`model.gguf` or the sole `*.gguf`), or a full safetensors/HF checkpoint tree, plus any session auxiliary files. A directory holding several GGUFs with no `model.gguf` is ambiguous, so the guard makes no estimate there and the loader's own error surfaces instead. The estimate is scaled by a runtime overhead factor plus a fixed floor. When the check fails, the request returns HTTP 503 with `insufficient_memory`; the client may retry later. Defaults to `0`, which disables the guard entirely so existing deployments are unaffected; set a positive value to opt in. The `--min-free-memory-mb <mb>` command-line flag overrides the config value.
+
 Set per-model `"default_request_options"` to apply request-option defaults to every request for that model. Values supplied by the actual request body override these defaults.
 
 Set top-level `"max_request_body_bytes"` to bound the largest HTTP request body buffered in host RAM before routing. This protects endpoints that accept JSON or audio uploads from unbounded `Content-Length` claims. The default is `2147483648` bytes (2 GiB). Raise or lower it to match the largest upload your deployment intends to accept. Values above `2^53 - 1` are rejected because this config parser stores JSON numbers as doubles.

app/server/busy_guard.h

@@ -18,6 +18,14 @@ class ServerBusyError : public std::runtime_error {
     using std::runtime_error::runtime_error;
 };
 
+// The host/GPU does not currently have enough free memory to load a model.
+// Mapped to HTTP 503: a transient server condition the caller may retry after
+// other models have been evicted or the system has freed memory.
+class InsufficientMemoryError : public std::runtime_error {
+public:
+    explicit InsufficientMemoryError(const std::string & message) : std::runtime_error(message) {}
+};
+
 inline std::int64_t steady_now_ms() {
     return std::chrono::duration_cast<std::chrono::milliseconds>(
                std::chrono::steady_clock::now().time_since_epoch())

app/server/config.cpp

@@ -235,6 +235,8 @@ ServerConfig load_server_config(const std::filesystem::path & path) {
     }
     config.busy_timeout_ms = engine::io::json::optional_i32(root, "busy_timeout_ms", config.busy_timeout_ms);
     config.max_loaded_models = engine::io::json::optional_i32(root, "max_loaded_models", config.max_loaded_models);
+    config.idle_unload_ms = engine::io::json::optional_i32(root, "idle_unload_ms", config.idle_unload_ms);
+    config.min_free_memory_mb = engine::io::json::optional_i32(root, "min_free_memory_mb", config.min_free_memory_mb);
     if (const auto * value = root.find("live_ingest")) {
         config.live_ingest = parse_live_ingest_limits(*value, config.live_ingest, "server live_ingest");
     }
@@ -256,6 +258,12 @@ ServerConfig load_server_config(const std::filesystem::path & path) {
     if (config.max_loaded_models < 0) {
         throw std::runtime_error("server max_loaded_models must be >= 0 (0 disables the limit)");
     }
+    if (config.idle_unload_ms < 0) {
+        throw std::runtime_error("server idle_unload_ms must be >= 0 (0 disables idle unload)");
+    }
+    if (config.min_free_memory_mb < 0) {
+        throw std::runtime_error("server min_free_memory_mb must be >= 0 (0 disables the memory guard)");
+    }
     if (config.threads <= 0) {
         throw std::runtime_error("server threads must be positive");
     }

app/server/config.h

@@ -90,6 +90,18 @@ struct ServerConfig {
     // limit and keeps the original behavior: once loaded, a model stays in memory
     // until it is unloaded explicitly or the server exits.
     int max_loaded_models = 0;
+    // Unload every resident model once the server has been idle this long without
+    // any model load/run (steady-clock ms). 0 disables idle unload. Complements
+    // max_loaded_models: that bounds peak residency, this frees memory during
+    // quiet periods. The next request reloads lazily.
+    int idle_unload_ms = 0;
+    // Minimum free memory (host and GPU, each) the server must retain after
+    // loading a model, in MiB. Before every lazy load the server estimates the
+    // model's resident footprint from its weights and refuses to load when
+    // estimate + this headroom would not fit. 0 disables the memory guard
+    // entirely (the default), so existing deployments see no behavior change
+    // unless they opt in.
+    int min_free_memory_mb = 0;
     // Fleet-wide bounds for incrementally delivered request bodies. The defaults are
     // in LiveIngestLimits; a model entry may override any subset of them.
     LiveIngestLimits live_ingest;

app/server/example.json

@@ -5,6 +5,9 @@
   "device": 0,
   "threads": 1,
   "lazy_load": true,
+  "max_loaded_models": 0,
+  "idle_unload_ms": 0,
+  "min_free_memory_mb": 0,
   "models": [
     {
       "id": "pocket-tts",

app/server/main.cpp

@@ -62,7 +62,7 @@ void print_help() {
     std::cout
         << "audiocpp_server [--config <server.json>] [--ui] [--host <ip>] [--port <port>] [--backend <backend>]\n"
         << "                [--device <id>] [--list-devices] [--threads <n>] [--busy-timeout-ms <ms>]\n"
-        << "                [--max-loaded-models <n>]\n"
+        << "                [--max-loaded-models <n>] [--idle-unload-ms <ms>] [--min-free-memory-mb <mb>]\n"
         << "                [--model-spec-override <json-or-directory>] [--voice-dir <directory>]\n"
         << "                [--log] [--log-file <path>]\n"
         << "                [--cors-origins <origins>]\n"
@@ -76,7 +76,11 @@ void print_help() {
         << "                                   busy this long; default 300000, 0 disables\n"
         << "  --max-loaded-models <n>          keep at most n models resident in memory, unloading\n"
         << "                                   the least recently used idle model first; 1 enforces\n"
-        << "                                   a single loaded model, default 0 (no limit)\n"
+        << "                                   a single loaded model, default 0 (no limit)\n"        << "  --idle-unload-ms <ms>            unload all resident models after this many ms without\n"
+        << "                                   any model load/run; default 0 (disabled), next request\n"
+        << "                                   reloads lazily\n"        << "  --min-free-memory-mb <mb>        refuse a model load unless host and GPU each keep at\n"
+        << "                                   least this many MiB free after the load; default 512,\n"
+        << "                                   0 disables the extra headroom\n"
         << "  --voice-dir <directory>          override the shared reference voice library directory\n"
         << "  --cors-origins \"*\"              experimental; disabled by default. Allows browser\n"
         << "                                   requests from any origin for trusted local demos only\n"
@@ -188,6 +192,12 @@ int main(int argc, char ** argv) {
         if (const auto max_loaded_models = arg_value(argc, argv, "--max-loaded-models")) {
             config.max_loaded_models = std::stoi(*max_loaded_models);
         }
+        if (const auto idle_unload_ms = arg_value(argc, argv, "--idle-unload-ms")) {
+            config.idle_unload_ms = std::stoi(*idle_unload_ms);
+        }
+        if (const auto min_free_memory_mb = arg_value(argc, argv, "--min-free-memory-mb")) {
+            config.min_free_memory_mb = std::stoi(*min_free_memory_mb);
+        }
         if (const auto model_spec = arg_value(argc, argv, "--model-spec-override")) {
             config.model_spec_override = std::filesystem::path(*model_spec);
         }
@@ -206,6 +216,12 @@ int main(int argc, char ** argv) {
         if (config.max_loaded_models < 0) {
             throw std::runtime_error("--max-loaded-models must be >= 0 (0 disables the limit)");
         }
+        if (config.idle_unload_ms < 0) {
+            throw std::runtime_error("--idle-unload-ms must be >= 0 (0 disables idle unload)");
+        }
+        if (config.min_free_memory_mb < 0) {
+            throw std::runtime_error("--min-free-memory-mb must be >= 0 (0 disables the memory guard)");
+        }
 
         const auto ui_resource_anchor = executable_directory(argc > 0 ? argv[0] : nullptr);
         minitts::server::ServerState state(