diff --git a/docs/ops/OPS-PATTERNS.md b/docs/ops/OPS-PATTERNS.md new file mode 100644 index 0000000..82d5e96 --- /dev/null +++ b/docs/ops/OPS-PATTERNS.md @@ -0,0 +1,95 @@ +# Operational patterns — what is normal, what is not + +Consolidated from the 2026-10-07 host stall and the 2026-10-08 unit clean-up. The matching snapshot is +[`SYSTEM-BASELINE-2026-10-08.md`](SYSTEM-BASELINE-2026-10-08.md); regenerate with +`just ops-baseline` (read-only, no sudo). + +**Status of the numbers.** "Observed" values were measured on this host on 2026-10-08. "Proposed" thresholds are +judgement calls that have not been tuned against a longer history; treat them as starting points, not facts. + +## 1. Reading the snapshot + +| Signal | Observed 2026-10-08 | Proposed "investigate" line | Why | +|---|---|---|---| +| Memory PSI `some avg10` | 0.00 | > 5 for a minute | memory stalls precede the hard stalls seen on 2026-10-07 | +| IO PSI `some avg10` | **77–88 (abnormal)** | > 20 sustained | root is a spinning disk (~100 random IOPS); see §3 | +| CPU PSI `some avg10` | ~1 | > 25 | 4 CPUs; cargo builds are the usual cause | +| `MemAvailable` | 13 GB of 32 GB | < 3 GB | `scripts/guard-run.sh` trips on this | +| Swap used | 6.3 GB, all on the NVMe file | any use of the HDD `/swap.img` | HDD swap should be overflow only | +| tmpfs `/tmp` | **12 of 16 GB** | > 2 GB | `/tmp` is RAM; see §3 | +| `NRestarts >= 5` | 2 historical (pi-agent 11, qwen36 6) | any *currently* restarting unit | see §2 | +| GPU memory | 20.7 of 24.6 GB (Qwen3.8 + mmproj) | n/a | the GPU is owned by `heretic-neo-coder`; do not start the qwen36 unit | + +Normal listeners on loopback: 8000 (socat → 8002), 8002 (Qwen3.8), 8010 (Kroki compat), 8790 (SysML MCP). +On all interfaces: 8787 (kr0ki), 8789 (agent), 22, 111, 631. The k0s control plane (6443, 2380, 10250) and +NATS (4222) and Redis/valkey (6379) are also up. + +## 2. Failure patterns seen, and the fix for each + +**P1 — The restart counter is not the health signal.** `b00t-historian` showed 2,323 restarts and +`container-spire-agent` showed a restart storm in the journal, yet both had been stable for 18 hours. Check +`systemctl --user status` ("Active … since") and `journalctl --since -1h` first. `systemctl --user reset-failed ` +clears the counter once you have confirmed the unit is healthy. + +**P2 — A missing `ExecStart` binary loops forever.** `buildkit.service` pointed at `~/nerdctl/extras/rootless/…`, +which does not exist, exited `203/EXEC` and restarted every 2 s: 36,000 restarts. systemd's default start-rate limit +did not stop it because `RestartSec=2` spaces the starts just outside the window. For any `Restart=always` unit use +`RestartSec=5` or more, plus `StartLimitIntervalSec=300` and `StartLimitBurst=5`. +*Fixed:* the unit now runs `/usr/local/bin/containerd-rootless-setuptool.sh` (backup kept beside the scratchpad). + +**P3 — A unit that needs a service the host does not wire up.** `b00t-hive-serena` runs +`kubectl apply -f …/serena.yaml`, but `~/.kube/config` has no current context. A k0s controller *is* running on this +host (`k0scontroller.service`), so the cluster exists; it is the kubeconfig that is missing. The unit retried every +30 s, 2,646 times. *Disabled for now* (`systemctl --user disable --now`). To bring it back, create an admin +kubeconfig for k0s and give the unit its own `KUBECONFIG` instead of editing your main config (commands in the PR +description). + +**P4 — Expired credentials in a pre-start step.** `b00t-historian` failed `ExecStartPre=sync-nats-secrets.sh` with +`AADSTS700024: Client assertion is not within its valid time range` until the assertion was refreshed. If it +loops again, read the first error line in the journal, not the restart count. + +**P5 — `Conflicts=` is applied even when the unit's own condition fails.** Test-starting the Qwen3.6 unit stopped +Qwen3.8 mid-load. Never `start` a conflicting unit "to see what happens". The boot-order cause was fixed at its +source, `~/.b00t/_b00t_/opencode.agent.toml`. + +**P6 — `/tmp` is a 16 GB tmpfs, so scratch space is RAM.** 12 GB of it was a cargo `target/` directory from an +earlier Claude session. That shows up as `Shmem` (9.4 GB) in `/proc/meminfo`, shrinks the page cache, and is +swapped out when memory is tight. Put build trees under `~/.cache` or `/c0de`, never the scratchpad. Anything in +`/tmp` is also lost on reboot. The earlier scratchpad held four git *worktrees* (`b00t-pr`, `infra-pr`, +`infra-spire`, `infra-vultr`); their commits and branches live in `~/.b00t` and `~/promptexecution/infrastructure`, so +only uncommitted edits (none were present) would be lost, not the commits. Check `git status` and the `.git` file +of anything in a scratchpad before assuming either way. + +**P7 — Spinning-disk root, so random reads dominate IO pressure.** With swap now on NVMe, the remaining IO PSI is +file-backed page faults against `/` (observed: `dm-0` 100 % util at ~190 random 4 KB reads/s, NVMe idle). Heavy +readers were the Kubernetes API server and its datastore, plus long-lived CLI processes. Keeping hot data +(`/var/lib/k0s`, container storage, build caches) on `/c0de` is the structural fix; it has not been done. + +**P8 — The kernel log needs the `adm` group.** `/var/log/kern.log*` is `syslog:adm 0640` and `dmesg` is restricted, so +an agent running as the user cannot check for OOM kills. Either add the user to `adm` or use +`sudo grep -iE "out of memory|oom-kill|hung task|blocked for more|NVRM: Xid" /var/log/kern.log.1 /var/log/kern.log` +(use `NVRM: Xid`, not bare `Xid`: the bare form also matches the r8169 NIC line "XID 480" and overlayfs layer names). + +## 3. Kernel-log findings (read 2026-10-08, `kern.log.1` + `kern.log`) + +| When (UTC) | Event | Reading | +|---|---|---| +| 2026-09-29 15:42–15:46 | `containerd` tasks blocked 122–245 s, then "future hung task reports are suppressed" | disk stall; the kernel stopped reporting further ones | +| 2026-10-02 14:37–14:43 | **global OOM**: kernel killed k0s pods first (envoy-gateway, coredns, metrics-server, mlflow, kube-rbac-proxy, local-path-prov, envoy), then **`llama-server` (8.3 GB resident) in the Qwen container** | the only real out-of-memory event in the log; Kubernetes pods carry `oom_score_adj` ~1000 so they die before the model | +| 2026-10-05 09:11 and 14:28 | `containerd-shim` blocked >122 s | same disk-stall signature as 29 Sep | +| 2026-10-07 | **no OOM-kill, no hung-task, no GPU error** | see below | +| any | GPU `NVRM: Xid` | **none**; the two "XID" hits were a NIC and an overlayfs layer name | + +**What this means for the 7 Oct stall.** The kernel logged nothing on 7 Oct. That fits a stall in which the machine stopped +responding without the kernel noticing an out-of-memory condition (heavy swap and IO wait), or one that ended in a hard reset +that gave the kernel no chance to write. It does *not* confirm either; it only rules out an OOM kill and a reported hung task as +the visible cause. The pattern is recurring, though: three separate disk-stall or OOM episodes in nine days, all involving +containerd/k0s on the spinning root disk, which supports moving `/var/lib/k0s` and container storage to `/c0de` (P7). + +## 4. Standing rules + +1. Before changing a unit, read its journal and confirm the failure is the unit's own (P1–P4). +2. Never run a `Conflicts=` unit to test it (P5). +3. Build trees and clones live on disk, not `/tmp` (P6). +4. Wrap anything memory-hungry in `scripts/guard-run.sh` (capped scope + PSI watchdog). +5. Take a baseline before and after a change: `just ops-baseline > …`. diff --git a/docs/ops/SYSTEM-BASELINE-2026-10-08.md b/docs/ops/SYSTEM-BASELINE-2026-10-08.md new file mode 100644 index 0000000..7ac381b --- /dev/null +++ b/docs/ops/SYSTEM-BASELINE-2026-10-08.md @@ -0,0 +1,173 @@ +# System baseline — sm3llsl1k3s0ld3r + +Captured 2026-10-08T09:38:22Z UTC · kernel 7.0.0-38-generic · 4 CPUs + +## Memory and swap + +``` + total used free shared buff/cache available +Mem: 32038 10373 2180 373 20635 21664 +Swap: 36863 3789 33074 + +NAME TYPE SIZE USED PRIO +/swap.img file 4G 0B -1 +/c0de/swap/swapfile file 32G 3.7G 100 + +Cached: 12816616 kB +SwapCached: 84672 kB +Dirty: 1332 kB +Mapped: 1462276 kB +Shmem: 382064 kB +vm.swappiness=60 vm.page-cluster=0 +``` + +## Pressure (PSI) and load + +``` +cpu: some avg10=1.64 avg60=1.38 avg300=0.87 total=2049483252 +memory: some avg10=0.00 avg60=0.00 avg300=0.00 total=246242973 +io: some avg10=2.02 avg60=5.42 avg300=1.99 total=7832778547 +loadavg: 0.68 0.52 0.75 2/1178 188267 +``` + +## tmpfs (RAM-backed; counts against memory) + +``` +Filesystem Size Used Avail Use% Mounted on +tmpfs 3.2G 3.3M 3.2G 1% /run +tmpfs 16G 168K 16G 1% /dev/shm +tmpfs 16G 39M 16G 1% /tmp +tmpfs 3.2G 416K 3.2G 1% /run/user/1000 +``` + +## Disks + +``` +Filesystem Size Used Avail Use% Mounted on +/dev/mapper/ubuntu--vg-ubuntu--lv 7.2T 4.6T 2.4T 67% / +/dev/nvme0n1p5 468G 277G 168G 63% /c0de +``` + +## GPU + +``` +NVIDIA GeForce RTX 3090, 20678 MiB, 24576 MiB, 0 %, 35 +``` + +## Listening sockets + +``` +:179 +:34973 +:50051 +:53 +0.0.0.0:111 +0.0.0.0:22 +0.0.0.0:631 +0.0.0.0:8787 +0.0.0.0:8789 +*:10249 +*:10250 +*:10256 +[::]:111 +127.0.0.1:10248 +127.0.0.1:10257 +127.0.0.1:10259 +127.0.0.1:20241 +127.0.0.1:3000 +127.0.0.1:33767 +127.0.0.1:39043 +127.0.0.1:50051 +127.0.0.1:6011 +127.0.0.1:6012 +127.0.0.1:6013 +127.0.0.1:8000 +127.0.0.1:8790 +127.0.0.53%lo:53 +127.0.0.54:53 +[::1]:6011 +[::1]:6012 +[::1]:6013 +*:20244 +[::]:22 +*:2380 +*:4222 +[::]:631 +*:6379 +*:6443 +*:8002 +*:8010 +*:8080 +*:8222 +[fd7a:115c:a1e0::3c01:cedc]:55066 +``` + +## systemd --user: running services + +``` +b00t-historian.service +b00t-hive-hive-watchdog.service +b00t-hive-inference-heretic-neo-coder.service +b00t-hive-opencode-agent.service +buildkit.service +container-spire-agent.service +containerd.service +d35d3961e3bd9927ff06fe22628efeb4126e4f849894e83fa21aa7de6a5c1949-3f2513007f457aad.service +dbus.service +hive-b00t-relay.service +kr0ki-agent.service +kr0ki-kroki.service +kr0ki-llm-socat.service +kr0ki-server.service +kr0ki-sysml-mcp.service +valkey.service +xdg-document-portal.service +xdg-permission-store.service +``` + +## systemd --user: problem units (failed / restarting) + +``` +--- NRestarts >= 5 (running or not): +b00t-hive-inference-qwen36-35b-a3b-llamacpp.service NRestarts=6 +b00t-hive-pi-agent.service NRestarts=11 +``` + +## systemd (system): failed + +``` +logrotate.service +``` + +## Containers + +``` +kr0ki-sysml-mcp Up 25 hours localhost/kr0ki-sysml-mcp:dev +kr0ki-dev-kroki Up 25 hours localhost/kr0ki-kroki-compat:dev +valkey Up 22 hours (healthy) docker.io/valkey/valkey:8-alpine +spire-agent Up 20 hours ghcr.io/spiffe/spire-agent:1.15.3 +b00t-heretic Up 13 hours ghcr.io/ggml-org/llama.cpp:server-cuda +``` + +## Kernel log: OOM kills, hung tasks, GPU Xid + +UNREADABLE: /var/log/kern.log is syslog:adm 0640 and this user is not in adm. + +Run: `sudo grep -iE "out of memory|oom-kill|hung task|blocked for more|NVRM: Xid" /var/log/kern.log.1 /var/log/kern.log` +Or: `sudo usermod -aG adm "$USER"` and log in again so this section can run unattended. + +## Largest resident processes + +``` + RSS COMMAND +5238164 llama-server +490640 kube-apiserver +377228 claude +276680 opencode +236844 systemd-journal +220836 codex +161636 java +150736 mlflow +``` + +_Thresholds for reading this are in docs/ops/OPS-PATTERNS.md._ diff --git a/justfile b/justfile index 30cadd0..990c969 100644 --- a/justfile +++ b/justfile @@ -568,6 +568,10 @@ assurance-gaps: delegate +prompt: scripts/delegate-qwen38.sh {{prompt}} +# Read-only host snapshot (memory, PSI, swap, tmpfs, units, containers) as markdown. See docs/ops/OPS-PATTERNS.md. +ops-baseline: + scripts/system-baseline.sh + # Revision-bound compiler evidence and compressed derived artifacts. docgen-image revision="HEAD": bash scripts/docgen.sh image "{{revision}}" diff --git a/scripts/system-baseline.sh b/scripts/system-baseline.sh new file mode 100755 index 0000000..be1741b --- /dev/null +++ b/scripts/system-baseline.sh @@ -0,0 +1,77 @@ +#!/usr/bin/env bash +# system-baseline.sh — read-only "what is normal on this host" snapshot, as markdown on stdout. +# +# No sudo, no writes, no network calls beyond loopback. Safe to run any time (it does not touch +# services). Redirect it to keep a dated snapshot: +# scripts/system-baseline.sh > docs/ops/SYSTEM-BASELINE-$(date -u +%F).md +# +# Reads: /proc (meminfo, pressure, loadavg), systemd --user, podman, ss, df, nvidia-smi (optional). +# Each section degrades to "n/a" when a tool is missing. Environment values are never printed. +set -uo pipefail + +have() { command -v "$1" >/dev/null 2>&1; } +section() { printf '\n## %s\n\n' "$1"; } +fence() { printf '```\n'; cat; printf '```\n'; } + +printf '# System baseline — %s\n\n' "$(hostname)" +printf 'Captured %s UTC · kernel %s · %s CPUs\n' "$(date -u +%FT%TZ)" "$(uname -r)" "$(nproc)" + +section "Memory and swap" +{ free -m; echo; swapon --show 2>/dev/null || echo "swapon: n/a"; echo; + grep -E '^(Shmem|Cached|Dirty|Mapped|SwapCached):' /proc/meminfo; + echo "vm.swappiness=$(cat /proc/sys/vm/swappiness) vm.page-cluster=$(cat /proc/sys/vm/page-cluster)"; } | fence + +section "Pressure (PSI) and load" +{ for r in cpu memory io; do echo "$r: $(head -1 /proc/pressure/$r 2>/dev/null || echo n/a)"; done; + echo "loadavg: $(cat /proc/loadavg)"; } | fence + +section "tmpfs (RAM-backed; counts against memory)" +df -h -t tmpfs 2>/dev/null | awk 'NR==1 || $3!="0"' | fence + +section "Disks" +df -h / /c0de 2>/dev/null | fence + +section "GPU" +if have nvidia-smi; then + nvidia-smi --query-gpu=name,memory.used,memory.total,utilization.gpu,temperature.gpu --format=csv,noheader | fence +else echo "n/a"; fi + +section "Listening sockets" +# Specific host addresses are masked: tracked files must not carry machine addresses (AGENTS.md). +if have ss; then + ss -ltnH | awk '{print $4}' | sed -E 's/^(127\.|\[::1?\]|0\.0\.0\.0|\*|\[::\]|127\.0\.0\.5)/\1/; t; s/^[0-9.]+:/:/' | sort -u | fence +else echo "n/a"; fi + +section "systemd --user: running services" +systemctl --user list-units --type=service --state=running --no-legend --plain 2>/dev/null | awk '{print $1}' | fence + +section "systemd --user: problem units (failed / restarting)" +{ systemctl --user list-units --state=failed,activating,auto-restart --no-legend --plain 2>/dev/null | awk '{print $1, $3, $4}'; + echo "--- NRestarts >= 5 (running or not):"; + for u in $(systemctl --user list-units --type=service --all --no-legend --plain 2>/dev/null | awk '{print $1}'); do + n=$(systemctl --user show "$u" -p NRestarts --value 2>/dev/null); [ "${n:-0}" -ge 5 ] 2>/dev/null && echo "$u NRestarts=$n" + done; } | fence + +section "systemd (system): failed" +systemctl list-units --state=failed --no-legend --plain 2>/dev/null | awk '{print $1}' | fence + +section "Containers" +if have podman; then podman ps --format '{{.Names}}\t{{.Status}}\t{{.Image}}' | cut -c1-140 | fence; else echo "n/a"; fi + +section "Kernel log: OOM kills, hung tasks, GPU Xid" +# Needs the adm group (or root). Without it this section says so rather than reporting a false "clean". +pat='out of memory|oom-kill|hung task|blocked for more|NVRM: Xid' +if [ -r /var/log/kern.log ]; then + hits=$(grep -ihE "$pat" /var/log/kern.log /var/log/kern.log.1 2>/dev/null | tail -20) + if [ -n "$hits" ]; then printf '%s\n' "$hits" | cut -c1-200 | fence; else echo "none in kern.log / kern.log.1"; fi +else + echo "UNREADABLE: /var/log/kern.log is syslog:adm 0640 and this user is not in adm." + echo + echo 'Run: `sudo grep -iE "'"$pat"'" /var/log/kern.log.1 /var/log/kern.log`' + echo 'Or: `sudo usermod -aG adm "$USER"` and log in again so this section can run unattended.' +fi + +section "Largest resident processes" +ps -eo rss,comm --sort=-rss | head -9 | fence + +printf '\n_Thresholds for reading this are in docs/ops/OPS-PATTERNS.md._\n'