Family of Models: Sprint 5 (V1) + Sprint 6 (V1.5 receptionist) implementation

Nexus-LLM-Runtime-4090/README.md

@@ -0,0 +1,68 @@
+# Nexus-LLM-Runtime-4090
+
+Inference runtime for the 4090 host. As of Sprint 5, the runtime direction is
+**llama.cpp (`llama-server`)**, not vLLM/TensorRT-LLM — see
+`docs/architecture_v2_family_of_models.md` Section 6. `compose.yaml` (the
+trtllm setup) stays in this directory as historical reference until Drew
+retires it; do not use it for new deployments.
+
+## Two ways to run the runtime
+
+**A. Model manager spawns llama-server natively (V1 default).**
+`nodes/model_manager_4090` (`manager.py`) owns weights tiering, the
+llama-server process lifecycle, and presence reporting to the hub. It calls
+`llama-server` as a plain subprocess with the flags below — no Docker
+involved. This is the default because the manager needs direct control over
+staging weights onto the hot tier (`ensure_hot()`) before each spawn, and
+needs to poll `/health` and manage SIGTERM/SIGKILL directly. Use this path
+whenever the model manager service is running.
+
+**B. `compose.llamacpp.yaml` (standalone / manual bring-up).**
+Use this compose file when you need to run llama-server by hand — the model
+manager isn't running, you're debugging a GGUF outside the manager's
+lifecycle, or you want a quick manual smoke test. It runs the official CUDA
+server image (`ghcr.io/ggml-org/llama.cpp:server-cuda`), mounts the hot tier
+at `/models`, and passes the same flags the manager would pass for family
+member #1 ("vera"). It does **not** do tiering or presence reporting — those
+only happen when the manager is in the loop (path A).
+
+## Model manager environment variables
+
+The manager reads settings via `MODELMGR_`-prefixed env vars
+(`nodes/model_manager_4090/config.py`):
+
+| Variable | Purpose |
+|---|---|
+| `MODELMGR_HOT_DIR` | Hot tier path (2 TB Gen4 NVMe) — active/loadable weights. |
+| `MODELMGR_WARM_DIR` | Warm tier path (1 TB Gen2 NVMe) — occasional members. |
+| `MODELMGR_COLD_DIR` | Cold tier path (6 TB HDD/NAS) — archive. |
+| `MODELMGR_HUB_URL` | Hub base URL the manager reports presence to. |
+| `MODELMGR_HUB_TOKEN` | Bearer token for the hub's presence endpoint. |
+| `MODELMGR_LLAMA_SERVER_BIN` | Path/name of the `llama-server` binary the manager spawns. |
+
+## Weights filename convention
+
+`weights_filename()` in `nodes/model_manager_4090/manager.py` builds the
+on-disk filename as `<source repo tail>.<quant>.<format>`, where the source
+tail is the last path segment of `model.source` from `family/registry.yaml`.
+
+For family member #1 ("vera": `hf:Qwen/Qwen3-30B-A3B-Instruct-2507`, quant
+`Q4_K_M`, format `gguf`), the exact expected filename is:
+
+```
+Qwen3-30B-A3B-Instruct-2507.Q4_K_M.gguf
+```
+
+Download tooling, the hot/warm/cold tier directories, and both runtime paths
+above (A and B) must all agree on this name.
+
+## Why tiering exists: don't mmap off the cold tier
+
+`llama.cpp` does not tier weights for us — `mmap` will happily page a GGUF
+straight off the HDD/cold tier, and it works, but page-in latency makes
+inference miserable. That's the exact failure mode
+`nodes/model_manager_4090`'s tiering (`ensure_hot()`) exists to avoid: it
+stages weights onto the hot NVMe tier *before* starting llama-server, so the
+model only ever loads/mmaps from fast storage. If you bring the runtime up
+by hand (path B), make sure the file under `D:/family_weights/hot` is
+actually there and not a broken symlink back to cold storage.

Nexus-LLM-Runtime-4090/compose.llamacpp.yaml

@@ -0,0 +1,40 @@
+services:
+  llamacpp:
+    container_name: llamacpp-vera
+    image: ghcr.io/ggml-org/llama.cpp:server-cuda
+
+    # --- GPU ---
+    runtime: nvidia
+
+    # --- Networking ---
+    ports:
+      - "8000:8000"
+
+    # --- Weights mount (hot tier only — see README on why cold/mmap is avoided) ---
+    volumes:
+      - "D:/family_weights/hot:/models"
+
+    # --- Environment ---
+    environment:
+      NVIDIA_VISIBLE_DEVICES: all
+      NVIDIA_DRIVER_CAPABILITIES: compute,utility
+
+    # --- llama-server flags (mirrors what nodes/model_manager_4090/manager.py
+    #     builds for family/registry.yaml member #1, "vera": Qwen3-30B-A3B
+    #     GGUF Q4_K_M, context_length 32768, offload_policy vram_then_ram) ---
+    command:
+      - "--model"
+      - "/models/Qwen3-30B-A3B-Instruct-2507.Q4_K_M.gguf"
+      - "--host"
+      - "0.0.0.0"
+      - "--port"
+      - "8000"
+      - "--ctx-size"
+      - "32768"
+      - "--n-gpu-layers"
+      - "999"
+      - "--no-mmap"
+
+    # --- Keep container alive for interactive or server use ---
+    tty: true
+    stdin_open: true

clients/README.md

@@ -11,8 +11,10 @@ talking to the same public endpoint on the 4070 brainstem.
 
 Both are deliberately a **separate artifact** from the brainstem service.
 They never import or edit brainstem code. They speak only its public HTTP
-contract: `POST /generate`. That keeps the client free to evolve, ship,
-and break without touching the running fabric.
+contract: the legacy `POST /generate`, and, as of Sprint 5, the family hub
+member API (`GET /family`, `POST /members/{id}/chat`,
+`GET /members/{id}/inbox/{msg_id}`). That keeps the client free to evolve,
+ship, and break without touching the running fabric.
 
 ## The contract these clients speak
 
@@ -28,6 +30,30 @@ and returns:
 { "text": "...", "model": "...", "finish_reason": "stop", "usage": { ... }, "source": "cortex_4090" }
 ```
 
+### Sprint 5: the family hub member API
+
+The brainstem's Sprint 5 family hub adds a member-aware surface alongside
+`/generate`, which keeps working unchanged as the hub's default member.
+
+- `GET /family` (anonymous) - the household roster: `{members: [{id,
+  display_name, presence, queue_depth, model: {source, quant,
+  context_length}}]}`.
+- `POST /members/{id}/chat` (Bearer auth, **no** `X-Session-Id` - member
+  sessions are hub-minted per person+member on the server) - body
+  `{prompt, system?, max_tokens?, temperature?}`. Three possible
+  responses:
+  - `200` - answered live: `{member_id, display_name, text, model,
+    finish_reason, usage, session_id, turn_idx, memory_written}`.
+  - `202` - the member is asleep or busy: `{queued, msg_id, member_id,
+    presence, status_url}`. The message waits in their inbox.
+  - `503 member_loading` - the member is waking up: structured body with
+    `retry_after_seconds` and a `Retry-After` header, same shape as the
+    older `cortex_unavailable`/`cortex_timeout` 503s (which can still
+    pass through this endpoint too).
+- `GET /members/{id}/inbox/{msg_id}` (Bearer auth, sender only) - status
+  of a queued message: `{msg_id, member_id, status: "queued"|"answered",
+  queued_at, result}`. `result.text` carries the reply once answered.
+
 On every request both clients also send:
 
 - `X-Session-Id` - a session id the client generates once and persists.
@@ -56,6 +82,7 @@ On every request both clients also send:
     "tailscale": "http://<REDACTED_TAILSCALE_IP>:5001"
   },
   "default_target": "tailscale",
+  "default_member": "vera",
   "auth_token": "",
   "generation": { "max_tokens": 512, "temperature": 0.7 }
 }
@@ -64,7 +91,9 @@ On every request both clients also send:
 Two named targets, same brainstem, different paths to it. The LAN address
 works on the home network. The Tailscale address works from anywhere on
 the tailnet, on or off the home network, which is why it is the default.
-Either client can also be pointed at an explicit `--url`.
+Either client can also be pointed at an explicit `--url`. `default_member`
+(Sprint 5) is which family member the CLI's `--member` flag talks to when
+you don't name one explicitly.
 
 ## CLI client
 
@@ -87,6 +116,38 @@ the same conversation thread across runs. In the REPL, `/new` starts a
 fresh session, `/session` shows the current one, `/target` shows the
 brainstem url, `/exit` quits. `python nexus_cli.py --help` lists every flag.
 
+### Talking to a family member (Sprint 5)
+
+```
+python nexus_cli.py --family                       # list the roster: id, presence, queue depth, model
+python nexus_cli.py --member vera --prompt "hi"     # one-shot chat with member "vera"
+python nexus_cli.py --member --prompt "hi"          # same, using config's default_member
+python nexus_cli.py --member vera                   # REPL in member mode
+python nexus_cli.py --member vera --check-inbox <msg_id>  # resume a queued reply later
+```
+
+`--member` (with or without an id) switches the chat round trip from the
+legacy `/generate` to `POST /members/{id}/chat`. Three things can happen:
+
+- **Answered live (200)**: printed exactly like a `/generate` reply, with
+  the member's name in the footer.
+- **Queued (202)** - the member is asleep or busy: the CLI prints the
+  `msg_id` and polls the returned `status_url` every `--poll-interval`
+  seconds (default 5s) until it is answered or `--max-wait` (default
+  300s) elapses. If it gives up, it prints the exact command to resume
+  polling later with `--check-inbox`; the message is not lost, it is
+  still sitting in the member's inbox.
+- **Waking (503 `member_loading`)**: the CLI retries up to 3 times,
+  honoring the server's `Retry-After` each time (capped at 60s), before
+  giving up with a clear error. The older `cortex_unavailable`/
+  `cortex_timeout` 503s still get the original Sprint 3c one-retry
+  treatment on this path.
+
+In the REPL, `/family` lists the roster, `/member <id>` switches to
+chatting with that member, and `/legacy` switches back to `/generate`.
+Member chat does not send `X-Session-Id`: the hub mints and persists a
+session per (person, member) itself.
+
 ## Web client
 
 The web client is `web/index.html`, a single self-contained file. The
@@ -97,9 +158,11 @@ this client needs. The browser blocks it at the preflight.
 The fix that does **not** require touching the brainstem is to serve the
 page and the API from the same origin. `web/serve.py` does exactly that:
 it serves `index.html` and reverse-proxies a small allowlist of brainstem
-endpoints (`/generate`, `/fabric/status`, `/cortex/health`, `/health`) to
-the configured 4070 address. The browser only ever talks to `serve.py`,
-same origin, no CORS, brainstem untouched.
+endpoints (`/generate`, `/fabric/status`, `/cortex/health`, `/health`,
+and, as of Sprint 5, `/family`, `/members/{id}/chat`,
+`/members/{id}/inbox/{msg_id}`) to the configured 4070 address. The
+browser only ever talks to `serve.py`, same origin, no CORS, brainstem
+untouched.
 
 ```
 cd clients/web
@@ -123,6 +186,33 @@ an optional auth token, generation parameters, and the session id with a
 "new session" button. The session id is persisted in `localStorage`, so a
 phone keeps its conversation thread across reloads.
 
+### Talking to a family member (Sprint 5)
+
+The people icon in the header opens the family sheet, populated from
+`GET /family`: each member's presence (green dot = awake, amber = busy or
+waking, grey = asleep) and queue depth, plus a "Legacy /generate" row to
+go back to the original endpoint. Tapping a member routes chat through
+`POST /members/{id}/chat` instead; the header subtitle shows who you're
+currently talking to. The chosen member is persisted in `localStorage`,
+same as the session id, so a phone remembers it across reloads.
+
+- **Answered live (200)**: rendered exactly like a `/generate` reply,
+  with the member's name in the footer.
+- **Queued (202)** - the member is asleep or busy: the page shows
+  "*&lt;name&gt; is asleep -- message queued (msg_id). Waiting...*" and
+  polls the inbox status URL at the interval set in Settings ("Queue
+  poll interval", default 5s) until answered, or until "Queue max wait"
+  (default 300s) elapses. If it gives up, a "Check again" button appears
+  so you can resume polling without retyping the message.
+- **Waking (503 `member_loading`)**: shown as a countdown ("*retrying in
+  Ns... (attempt a/3)*") honoring the server's `Retry-After`, up to 3
+  attempts, before surfacing a clear error. The older
+  `cortex_unavailable`/`cortex_timeout` 503s still get the original
+  Sprint 3c one-retry treatment on this path.
+
+Member chat deliberately does not send `X-Session-Id`: the hub mints and
+persists a session per (person, member) itself.
+
 ### Future: serving the page from the brainstem directly
 
 The proxy exists because the brainstem has no CORS and the client may not