From bf372e934f825b9c13e8c82e7d03cd5867a75c3d Mon Sep 17 00:00:00 2001 From: Brian H Date: Thu, 8 Oct 2026 07:27:37 +0000 Subject: [PATCH 1/3] ops: read-only system baseline script, snapshot and operational patterns scripts/system-baseline.sh (just ops-baseline) captures memory/swap, PSI, tmpfs, disks, GPU, listeners, running and failing user units and containers as markdown. docs/ops/OPS-PATTERNS.md consolidates the 2026-10-07 stall and 2026-10-08 unit clean-up into eight failure patterns, with observed values and proposed (untuned) thresholds kept separate. The first snapshot is committed beside it. Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_01GgXQRGsprSZc2oXGNj9n8X --- docs/ops/OPS-PATTERNS.md | 78 ++++++++++++ docs/ops/SYSTEM-BASELINE-2026-10-08.md | 166 +++++++++++++++++++++++++ justfile | 4 + scripts/system-baseline.sh | 64 ++++++++++ 4 files changed, 312 insertions(+) create mode 100644 docs/ops/OPS-PATTERNS.md create mode 100644 docs/ops/SYSTEM-BASELINE-2026-10-08.md create mode 100755 scripts/system-baseline.sh diff --git a/docs/ops/OPS-PATTERNS.md b/docs/ops/OPS-PATTERNS.md new file mode 100644 index 0000000..a423375 --- /dev/null +++ b/docs/ops/OPS-PATTERNS.md @@ -0,0 +1,78 @@ +# Operational patterns — what is normal, what is not + +Consolidated from the 2026-10-07 host stall and the 2026-10-08 unit clean-up. The matching snapshot is +[`SYSTEM-BASELINE-2026-10-08.md`](SYSTEM-BASELINE-2026-10-08.md); regenerate with +`just ops-baseline` (read-only, no sudo). + +**Status of the numbers.** "Observed" values were measured on this host on 2026-10-08. "Proposed" thresholds are +judgement calls that have not been tuned against a longer history; treat them as starting points, not facts. + +## 1. Reading the snapshot + +| Signal | Observed 2026-10-08 | Proposed "investigate" line | Why | +|---|---|---|---| +| Memory PSI `some avg10` | 0.00 | > 5 for a minute | memory stalls precede the hard stalls seen on 2026-10-07 | +| IO PSI `some avg10` | **77–88 (abnormal)** | > 20 sustained | root is a spinning disk (~100 random IOPS); see §3 | +| CPU PSI `some avg10` | ~1 | > 25 | 4 CPUs; cargo builds are the usual cause | +| `MemAvailable` | 13 GB of 32 GB | < 3 GB | `scripts/guard-run.sh` trips on this | +| Swap used | 6.3 GB, all on the NVMe file | any use of the HDD `/swap.img` | HDD swap should be overflow only | +| tmpfs `/tmp` | **12 of 16 GB** | > 2 GB | `/tmp` is RAM; see §3 | +| `NRestarts >= 5` | 2 historical (pi-agent 11, qwen36 6) | any *currently* restarting unit | see §2 | +| GPU memory | 20.7 of 24.6 GB (Qwen3.8 + mmproj) | n/a | the GPU is owned by `heretic-neo-coder`; do not start the qwen36 unit | + +Normal listeners on loopback: 8000 (socat → 8002), 8002 (Qwen3.8), 8010 (Kroki compat), 8790 (SysML MCP). +On all interfaces: 8787 (kr0ki), 8789 (agent), 22, 111, 631. The k0s control plane (6443, 2380, 10250) and +NATS (4222) and Redis/valkey (6379) are also up. + +## 2. Failure patterns seen, and the fix for each + +**P1 — The restart counter is not the health signal.** `b00t-historian` showed 2,323 restarts and +`container-spire-agent` showed a restart storm in the journal, yet both had been stable for 18 hours. Check +`systemctl --user status` ("Active … since") and `journalctl --since -1h` first. `systemctl --user reset-failed ` +clears the counter once you have confirmed the unit is healthy. + +**P2 — A missing `ExecStart` binary loops forever.** `buildkit.service` pointed at `~/nerdctl/extras/rootless/…`, +which does not exist, exited `203/EXEC` and restarted every 2 s: 36,000 restarts. systemd's default start-rate limit +did not stop it because `RestartSec=2` spaces the starts just outside the window. For any `Restart=always` unit use +`RestartSec=5` or more, plus `StartLimitIntervalSec=300` and `StartLimitBurst=5`. +*Fixed:* the unit now runs `/usr/local/bin/containerd-rootless-setuptool.sh` (backup kept beside the scratchpad). + +**P3 — A unit that needs a service the host does not wire up.** `b00t-hive-serena` runs +`kubectl apply -f …/serena.yaml`, but `~/.kube/config` has no current context. A k0s controller *is* running on this +host (`k0scontroller.service`), so the cluster exists; it is the kubeconfig that is missing. The unit retried every +30 s, 2,646 times. *Disabled for now* (`systemctl --user disable --now`). To bring it back, create an admin +kubeconfig for k0s and give the unit its own `KUBECONFIG` instead of editing your main config (commands in the PR +description). + +**P4 — Expired credentials in a pre-start step.** `b00t-historian` failed `ExecStartPre=sync-nats-secrets.sh` with +`AADSTS700024: Client assertion is not within its valid time range` until the assertion was refreshed. If it +loops again, read the first error line in the journal, not the restart count. + +**P5 — `Conflicts=` is applied even when the unit's own condition fails.** Test-starting the Qwen3.6 unit stopped +Qwen3.8 mid-load. Never `start` a conflicting unit "to see what happens". The boot-order cause was fixed at its +source, `~/.b00t/_b00t_/opencode.agent.toml`. + +**P6 — `/tmp` is a 16 GB tmpfs, so scratch space is RAM.** 12 GB of it was a cargo `target/` directory from an +earlier Claude session. That shows up as `Shmem` (9.4 GB) in `/proc/meminfo`, shrinks the page cache, and is +swapped out when memory is tight. Put build trees under `~/.cache` or `/c0de`, never the scratchpad. Anything in +`/tmp` is also lost on reboot. The earlier scratchpad held four git *worktrees* (`b00t-pr`, `infra-pr`, +`infra-spire`, `infra-vultr`); their commits and branches live in `~/.b00t` and `~/promptexecution/infrastructure`, so +only uncommitted edits (none were present) would be lost, not the commits. Check `git status` and the `.git` file +of anything in a scratchpad before assuming either way. + +**P7 — Spinning-disk root, so random reads dominate IO pressure.** With swap now on NVMe, the remaining IO PSI is +file-backed page faults against `/` (observed: `dm-0` 100 % util at ~190 random 4 KB reads/s, NVMe idle). Heavy +readers were the Kubernetes API server and its datastore, plus long-lived CLI processes. Keeping hot data +(`/var/lib/k0s`, container storage, build caches) on `/c0de` is the structural fix; it has not been done. + +**P8 — The kernel log needs the `adm` group.** `/var/log/kern.log*` is `syslog:adm 0640` and `dmesg` is restricted, so +an agent running as the user cannot check for OOM kills. Either add the user to `adm` or use +`sudo grep -iE "out of memory|oom-kill|hung task|blocked for more|Xid" /var/log/kern.log /var/log/kern.log.1`. + +## 3. Standing rules + +1. Before changing a unit, read its journal and confirm the failure is the unit's own (P1–P4). +2. Never run a `Conflicts=` unit to test it (P5). +3. Build trees and clones live on disk, not `/tmp` (P6). +4. Wrap anything memory-hungry in `scripts/guard-run.sh` (capped scope + PSI watchdog). +5. Take a baseline before and after a change: `just ops-baseline > …`. diff --git a/docs/ops/SYSTEM-BASELINE-2026-10-08.md b/docs/ops/SYSTEM-BASELINE-2026-10-08.md new file mode 100644 index 0000000..d06ddbb --- /dev/null +++ b/docs/ops/SYSTEM-BASELINE-2026-10-08.md @@ -0,0 +1,166 @@ +# System baseline — sm3llsl1k3s0ld3r + +Captured 2026-10-08T07:27:27Z UTC · kernel 7.0.0-38-generic · 4 CPUs + +## Memory and swap + +``` + total used free shared buff/cache available +Mem: 32038 18863 540 9043 22456 13174 +Swap: 36863 6555 30308 + +NAME TYPE SIZE USED PRIO +/swap.img file 4G 0B -1 +/c0de/swap/swapfile file 32G 6.4G 100 + +Cached: 21339972 kB +SwapCached: 116024 kB +Dirty: 6220 kB +Mapped: 1489716 kB +Shmem: 9260160 kB +vm.swappiness=60 vm.page-cluster=0 +``` + +## Pressure (PSI) and load + +``` +cpu: some avg10=1.25 avg60=1.09 avg300=1.03 total=1952094421 +memory: some avg10=0.00 avg60=0.00 avg300=0.06 total=245595628 +io: some avg10=83.67 avg60=79.85 avg300=68.97 total=3805792283 +loadavg: 5.71 4.69 2.77 1/1195 3999723 +``` + +## tmpfs (RAM-backed; counts against memory) + +``` +Filesystem Size Used Avail Use% Mounted on +tmpfs 3.2G 3.3M 3.2G 1% /run +tmpfs 16G 168K 16G 1% /dev/shm +tmpfs 16G 12G 4.5G 72% /tmp +tmpfs 3.2G 416K 3.2G 1% /run/user/1000 +``` + +## Disks + +``` +Filesystem Size Used Avail Use% Mounted on +/dev/mapper/ubuntu--vg-ubuntu--lv 7.2T 4.6T 2.4T 67% / +/dev/nvme0n1p5 468G 277G 168G 63% /c0de +``` + +## GPU + +``` +NVIDIA GeForce RTX 3090, 20678 MiB, 24576 MiB, 0 %, 37 +``` + +## Listening sockets + +``` +:179 +:34973 +:50051 +:53 +0.0.0.0:111 +0.0.0.0:22 +0.0.0.0:631 +0.0.0.0:8787 +0.0.0.0:8789 +*:10249 +*:10250 +*:10256 +[::]:111 +127.0.0.1:10248 +127.0.0.1:10257 +127.0.0.1:10259 +127.0.0.1:20241 +127.0.0.1:3000 +127.0.0.1:33767 +127.0.0.1:39043 +127.0.0.1:50051 +127.0.0.1:6011 +127.0.0.1:6012 +127.0.0.1:6013 +127.0.0.1:8000 +127.0.0.1:8790 +127.0.0.53%lo:53 +127.0.0.54:53 +[::1]:6011 +[::1]:6012 +[::1]:6013 +*:20244 +[::]:22 +*:2380 +*:4222 +[::]:631 +*:6379 +*:6443 +*:8002 +*:8010 +*:8080 +*:8222 +[fd7a:115c:a1e0::3c01:cedc]:55066 +``` + +## systemd --user: running services + +``` +b00t-historian.service +b00t-hive-hive-watchdog.service +b00t-hive-inference-heretic-neo-coder.service +b00t-hive-opencode-agent.service +buildkit.service +container-spire-agent.service +containerd.service +d35d3961e3bd9927ff06fe22628efeb4126e4f849894e83fa21aa7de6a5c1949-3f2513007f457aad.service +dbus.service +hive-b00t-relay.service +kr0ki-agent.service +kr0ki-kroki.service +kr0ki-llm-socat.service +kr0ki-server.service +kr0ki-sysml-mcp.service +valkey.service +xdg-document-portal.service +xdg-permission-store.service +``` + +## systemd --user: problem units (failed / restarting) + +``` +--- NRestarts >= 5 (running or not): +b00t-hive-inference-qwen36-35b-a3b-llamacpp.service NRestarts=6 +b00t-hive-pi-agent.service NRestarts=11 +``` + +## systemd (system): failed + +``` +logrotate.service +``` + +## Containers + +``` +kr0ki-sysml-mcp Up 22 hours localhost/kr0ki-sysml-mcp:dev +kr0ki-dev-kroki Up 22 hours localhost/kr0ki-kroki-compat:dev +valkey Up 20 hours (healthy) docker.io/valkey/valkey:8-alpine +spire-agent Up 18 hours ghcr.io/spiffe/spire-agent:1.15.3 +b00t-heretic Up 11 hours ghcr.io/ggml-org/llama.cpp:server-cuda +``` + +## Largest resident processes + +``` + RSS COMMAND +5263696 llama-server +514684 kube-apiserver +423272 claude +230124 opencode +227224 systemd-journal +218088 codex +161636 java +150736 mlflow +``` + +_Thresholds for reading this are in docs/ops/OPS-PATTERNS.md._ diff --git a/justfile b/justfile index 30cadd0..990c969 100644 --- a/justfile +++ b/justfile @@ -568,6 +568,10 @@ assurance-gaps: delegate +prompt: scripts/delegate-qwen38.sh {{prompt}} +# Read-only host snapshot (memory, PSI, swap, tmpfs, units, containers) as markdown. See docs/ops/OPS-PATTERNS.md. +ops-baseline: + scripts/system-baseline.sh + # Revision-bound compiler evidence and compressed derived artifacts. docgen-image revision="HEAD": bash scripts/docgen.sh image "{{revision}}" diff --git a/scripts/system-baseline.sh b/scripts/system-baseline.sh new file mode 100755 index 0000000..780426f --- /dev/null +++ b/scripts/system-baseline.sh @@ -0,0 +1,64 @@ +#!/usr/bin/env bash +# system-baseline.sh — read-only "what is normal on this host" snapshot, as markdown on stdout. +# +# No sudo, no writes, no network calls beyond loopback. Safe to run any time (it does not touch +# services). Redirect it to keep a dated snapshot: +# scripts/system-baseline.sh > docs/ops/SYSTEM-BASELINE-$(date -u +%F).md +# +# Reads: /proc (meminfo, pressure, loadavg), systemd --user, podman, ss, df, nvidia-smi (optional). +# Each section degrades to "n/a" when a tool is missing. Environment values are never printed. +set -uo pipefail + +have() { command -v "$1" >/dev/null 2>&1; } +section() { printf '\n## %s\n\n' "$1"; } +fence() { printf '```\n'; cat; printf '```\n'; } + +printf '# System baseline — %s\n\n' "$(hostname)" +printf 'Captured %s UTC · kernel %s · %s CPUs\n' "$(date -u +%FT%TZ)" "$(uname -r)" "$(nproc)" + +section "Memory and swap" +{ free -m; echo; swapon --show 2>/dev/null || echo "swapon: n/a"; echo; + grep -E '^(Shmem|Cached|Dirty|Mapped|SwapCached):' /proc/meminfo; + echo "vm.swappiness=$(cat /proc/sys/vm/swappiness) vm.page-cluster=$(cat /proc/sys/vm/page-cluster)"; } | fence + +section "Pressure (PSI) and load" +{ for r in cpu memory io; do echo "$r: $(head -1 /proc/pressure/$r 2>/dev/null || echo n/a)"; done; + echo "loadavg: $(cat /proc/loadavg)"; } | fence + +section "tmpfs (RAM-backed; counts against memory)" +df -h -t tmpfs 2>/dev/null | awk 'NR==1 || $3!="0"' | fence + +section "Disks" +df -h / /c0de 2>/dev/null | fence + +section "GPU" +if have nvidia-smi; then + nvidia-smi --query-gpu=name,memory.used,memory.total,utilization.gpu,temperature.gpu --format=csv,noheader | fence +else echo "n/a"; fi + +section "Listening sockets" +# Specific host addresses are masked: tracked files must not carry machine addresses (AGENTS.md). +if have ss; then + ss -ltnH | awk '{print $4}' | sed -E 's/^(127\.|\[::1?\]|0\.0\.0\.0|\*|\[::\]|127\.0\.0\.5)/\1/; t; s/^[0-9.]+:/:/' | sort -u | fence +else echo "n/a"; fi + +section "systemd --user: running services" +systemctl --user list-units --type=service --state=running --no-legend --plain 2>/dev/null | awk '{print $1}' | fence + +section "systemd --user: problem units (failed / restarting)" +{ systemctl --user list-units --state=failed,activating,auto-restart --no-legend --plain 2>/dev/null | awk '{print $1, $3, $4}'; + echo "--- NRestarts >= 5 (running or not):"; + for u in $(systemctl --user list-units --type=service --all --no-legend --plain 2>/dev/null | awk '{print $1}'); do + n=$(systemctl --user show "$u" -p NRestarts --value 2>/dev/null); [ "${n:-0}" -ge 5 ] 2>/dev/null && echo "$u NRestarts=$n" + done; } | fence + +section "systemd (system): failed" +systemctl list-units --state=failed --no-legend --plain 2>/dev/null | awk '{print $1}' | fence + +section "Containers" +if have podman; then podman ps --format '{{.Names}}\t{{.Status}}\t{{.Image}}' | cut -c1-140 | fence; else echo "n/a"; fi + +section "Largest resident processes" +ps -eo rss,comm --sort=-rss | head -9 | fence + +printf '\n_Thresholds for reading this are in docs/ops/OPS-PATTERNS.md._\n' From 7757b0cf9894aba7ff6bd5bca875e9d8e8c69447 Mon Sep 17 00:00:00 2001 From: Brian H Date: Thu, 8 Oct 2026 08:59:41 +0000 Subject: [PATCH 2/3] ops: baseline reports the kernel-log check (OOM/hung task/Xid) or why it cannot run Without adm membership the section states it is unreadable and gives the exact command, instead of silently reporting a clean log. Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_01GgXQRGsprSZc2oXGNj9n8X --- docs/ops/SYSTEM-BASELINE-2026-10-08.md | 62 ++++++++++++++------------ scripts/system-baseline.sh | 13 ++++++ 2 files changed, 47 insertions(+), 28 deletions(-) diff --git a/docs/ops/SYSTEM-BASELINE-2026-10-08.md b/docs/ops/SYSTEM-BASELINE-2026-10-08.md index d06ddbb..cb641fc 100644 --- a/docs/ops/SYSTEM-BASELINE-2026-10-08.md +++ b/docs/ops/SYSTEM-BASELINE-2026-10-08.md @@ -1,33 +1,33 @@ # System baseline — sm3llsl1k3s0ld3r -Captured 2026-10-08T07:27:27Z UTC · kernel 7.0.0-38-generic · 4 CPUs +Captured 2026-10-08T08:59:36Z UTC · kernel 7.0.0-38-generic · 4 CPUs ## Memory and swap ``` total used free shared buff/cache available -Mem: 32038 18863 540 9043 22456 13174 -Swap: 36863 6555 30308 +Mem: 32038 10160 2668 373 20361 21877 +Swap: 36863 3858 33005 NAME TYPE SIZE USED PRIO /swap.img file 4G 0B -1 -/c0de/swap/swapfile file 32G 6.4G 100 +/c0de/swap/swapfile file 32G 3.8G 100 -Cached: 21339972 kB -SwapCached: 116024 kB -Dirty: 6220 kB -Mapped: 1489716 kB -Shmem: 9260160 kB +Cached: 12547860 kB +SwapCached: 122100 kB +Dirty: 1716 kB +Mapped: 1482656 kB +Shmem: 382056 kB vm.swappiness=60 vm.page-cluster=0 ``` ## Pressure (PSI) and load ``` -cpu: some avg10=1.25 avg60=1.09 avg300=1.03 total=1952094421 -memory: some avg10=0.00 avg60=0.00 avg300=0.06 total=245595628 -io: some avg10=83.67 avg60=79.85 avg300=68.97 total=3805792283 -loadavg: 5.71 4.69 2.77 1/1195 3999723 +cpu: some avg10=0.80 avg60=0.98 avg300=1.00 total=2021747944 +memory: some avg10=0.00 avg60=0.00 avg300=0.00 total=246204516 +io: some avg10=10.80 avg60=57.07 avg300=74.69 total=7819758245 +loadavg: 3.52 4.21 4.12 1/1173 75054 ``` ## tmpfs (RAM-backed; counts against memory) @@ -36,7 +36,7 @@ loadavg: 5.71 4.69 2.77 1/1195 3999723 Filesystem Size Used Avail Use% Mounted on tmpfs 3.2G 3.3M 3.2G 1% /run tmpfs 16G 168K 16G 1% /dev/shm -tmpfs 16G 12G 4.5G 72% /tmp +tmpfs 16G 39M 16G 1% /tmp tmpfs 3.2G 416K 3.2G 1% /run/user/1000 ``` @@ -51,7 +51,7 @@ Filesystem Size Used Avail Use% Mounted on ## GPU ``` -NVIDIA GeForce RTX 3090, 20678 MiB, 24576 MiB, 0 %, 37 +NVIDIA GeForce RTX 3090, 20678 MiB, 24576 MiB, 0 %, 36 ``` ## Listening sockets @@ -112,7 +112,6 @@ b00t-hive-opencode-agent.service buildkit.service container-spire-agent.service containerd.service -d35d3961e3bd9927ff06fe22628efeb4126e4f849894e83fa21aa7de6a5c1949-3f2513007f457aad.service dbus.service hive-b00t-relay.service kr0ki-agent.service @@ -142,25 +141,32 @@ logrotate.service ## Containers ``` -kr0ki-sysml-mcp Up 22 hours localhost/kr0ki-sysml-mcp:dev -kr0ki-dev-kroki Up 22 hours localhost/kr0ki-kroki-compat:dev -valkey Up 20 hours (healthy) docker.io/valkey/valkey:8-alpine -spire-agent Up 18 hours ghcr.io/spiffe/spire-agent:1.15.3 -b00t-heretic Up 11 hours ghcr.io/ggml-org/llama.cpp:server-cuda +kr0ki-sysml-mcp Up 24 hours localhost/kr0ki-sysml-mcp:dev +kr0ki-dev-kroki Up 24 hours localhost/kr0ki-kroki-compat:dev +valkey Up 21 hours (healthy) docker.io/valkey/valkey:8-alpine +spire-agent Up 20 hours ghcr.io/spiffe/spire-agent:1.15.3 +b00t-heretic Up 12 hours ghcr.io/ggml-org/llama.cpp:server-cuda ``` +## Kernel log: OOM kills, hung tasks, GPU Xid + +UNREADABLE: /var/log/kern.log is syslog:adm 0640 and this user is not in adm. + +Run: `sudo grep -iE "out of memory|oom-kill|hung task|blocked for more|Xid" /var/log/kern.log.1 /var/log/kern.log` +Or: `sudo usermod -aG adm "$USER"` and log in again so this section can run unattended. + ## Largest resident processes ``` RSS COMMAND -5263696 llama-server -514684 kube-apiserver -423272 claude -230124 opencode -227224 systemd-journal -218088 codex +5238164 llama-server +490388 kube-apiserver +393200 claude +260212 opencode +234952 systemd-journal +218292 codex +171984 codebase-memory 161636 java -150736 mlflow ``` _Thresholds for reading this are in docs/ops/OPS-PATTERNS.md._ diff --git a/scripts/system-baseline.sh b/scripts/system-baseline.sh index 780426f..57786b5 100755 --- a/scripts/system-baseline.sh +++ b/scripts/system-baseline.sh @@ -58,6 +58,19 @@ systemctl list-units --state=failed --no-legend --plain 2>/dev/null | awk '{prin section "Containers" if have podman; then podman ps --format '{{.Names}}\t{{.Status}}\t{{.Image}}' | cut -c1-140 | fence; else echo "n/a"; fi +section "Kernel log: OOM kills, hung tasks, GPU Xid" +# Needs the adm group (or root). Without it this section says so rather than reporting a false "clean". +pat='out of memory|oom-kill|hung task|blocked for more|Xid' +if [ -r /var/log/kern.log ]; then + hits=$(grep -ihE "$pat" /var/log/kern.log /var/log/kern.log.1 2>/dev/null | tail -20) + if [ -n "$hits" ]; then printf '%s\n' "$hits" | cut -c1-200 | fence; else echo "none in kern.log / kern.log.1"; fi +else + echo "UNREADABLE: /var/log/kern.log is syslog:adm 0640 and this user is not in adm." + echo + echo 'Run: `sudo grep -iE "'"$pat"'" /var/log/kern.log.1 /var/log/kern.log`' + echo 'Or: `sudo usermod -aG adm "$USER"` and log in again so this section can run unattended.' +fi + section "Largest resident processes" ps -eo rss,comm --sort=-rss | head -9 | fence From c98d370674f50ce15e05bab587e4168570882234 Mon Sep 17 00:00:00 2001 From: Brian H Date: Thu, 8 Oct 2026 09:38:23 +0000 Subject: [PATCH 3/3] ops: record kernel-log findings; fix Xid pattern false positives kern.log shows one real global OOM (2026-10-02, k0s pods then llama-server), containerd hung-task episodes on 09-29 and 10-05, nothing on 10-07, no GPU Xid. Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_01GgXQRGsprSZc2oXGNj9n8X --- docs/ops/OPS-PATTERNS.md | 23 ++++++++++-- docs/ops/SYSTEM-BASELINE-2026-10-08.md | 51 +++++++++++++------------- scripts/system-baseline.sh | 2 +- 3 files changed, 47 insertions(+), 29 deletions(-) diff --git a/docs/ops/OPS-PATTERNS.md b/docs/ops/OPS-PATTERNS.md index a423375..82d5e96 100644 --- a/docs/ops/OPS-PATTERNS.md +++ b/docs/ops/OPS-PATTERNS.md @@ -67,9 +67,26 @@ readers were the Kubernetes API server and its datastore, plus long-lived CLI pr **P8 — The kernel log needs the `adm` group.** `/var/log/kern.log*` is `syslog:adm 0640` and `dmesg` is restricted, so an agent running as the user cannot check for OOM kills. Either add the user to `adm` or use -`sudo grep -iE "out of memory|oom-kill|hung task|blocked for more|Xid" /var/log/kern.log /var/log/kern.log.1`. - -## 3. Standing rules +`sudo grep -iE "out of memory|oom-kill|hung task|blocked for more|NVRM: Xid" /var/log/kern.log.1 /var/log/kern.log` +(use `NVRM: Xid`, not bare `Xid`: the bare form also matches the r8169 NIC line "XID 480" and overlayfs layer names). + +## 3. Kernel-log findings (read 2026-10-08, `kern.log.1` + `kern.log`) + +| When (UTC) | Event | Reading | +|---|---|---| +| 2026-09-29 15:42–15:46 | `containerd` tasks blocked 122–245 s, then "future hung task reports are suppressed" | disk stall; the kernel stopped reporting further ones | +| 2026-10-02 14:37–14:43 | **global OOM**: kernel killed k0s pods first (envoy-gateway, coredns, metrics-server, mlflow, kube-rbac-proxy, local-path-prov, envoy), then **`llama-server` (8.3 GB resident) in the Qwen container** | the only real out-of-memory event in the log; Kubernetes pods carry `oom_score_adj` ~1000 so they die before the model | +| 2026-10-05 09:11 and 14:28 | `containerd-shim` blocked >122 s | same disk-stall signature as 29 Sep | +| 2026-10-07 | **no OOM-kill, no hung-task, no GPU error** | see below | +| any | GPU `NVRM: Xid` | **none**; the two "XID" hits were a NIC and an overlayfs layer name | + +**What this means for the 7 Oct stall.** The kernel logged nothing on 7 Oct. That fits a stall in which the machine stopped +responding without the kernel noticing an out-of-memory condition (heavy swap and IO wait), or one that ended in a hard reset +that gave the kernel no chance to write. It does *not* confirm either; it only rules out an OOM kill and a reported hung task as +the visible cause. The pattern is recurring, though: three separate disk-stall or OOM episodes in nine days, all involving +containerd/k0s on the spinning root disk, which supports moving `/var/lib/k0s` and container storage to `/c0de` (P7). + +## 4. Standing rules 1. Before changing a unit, read its journal and confirm the failure is the unit's own (P1–P4). 2. Never run a `Conflicts=` unit to test it (P5). diff --git a/docs/ops/SYSTEM-BASELINE-2026-10-08.md b/docs/ops/SYSTEM-BASELINE-2026-10-08.md index cb641fc..7ac381b 100644 --- a/docs/ops/SYSTEM-BASELINE-2026-10-08.md +++ b/docs/ops/SYSTEM-BASELINE-2026-10-08.md @@ -1,33 +1,33 @@ # System baseline — sm3llsl1k3s0ld3r -Captured 2026-10-08T08:59:36Z UTC · kernel 7.0.0-38-generic · 4 CPUs +Captured 2026-10-08T09:38:22Z UTC · kernel 7.0.0-38-generic · 4 CPUs ## Memory and swap ``` total used free shared buff/cache available -Mem: 32038 10160 2668 373 20361 21877 -Swap: 36863 3858 33005 +Mem: 32038 10373 2180 373 20635 21664 +Swap: 36863 3789 33074 NAME TYPE SIZE USED PRIO /swap.img file 4G 0B -1 -/c0de/swap/swapfile file 32G 3.8G 100 +/c0de/swap/swapfile file 32G 3.7G 100 -Cached: 12547860 kB -SwapCached: 122100 kB -Dirty: 1716 kB -Mapped: 1482656 kB -Shmem: 382056 kB +Cached: 12816616 kB +SwapCached: 84672 kB +Dirty: 1332 kB +Mapped: 1462276 kB +Shmem: 382064 kB vm.swappiness=60 vm.page-cluster=0 ``` ## Pressure (PSI) and load ``` -cpu: some avg10=0.80 avg60=0.98 avg300=1.00 total=2021747944 -memory: some avg10=0.00 avg60=0.00 avg300=0.00 total=246204516 -io: some avg10=10.80 avg60=57.07 avg300=74.69 total=7819758245 -loadavg: 3.52 4.21 4.12 1/1173 75054 +cpu: some avg10=1.64 avg60=1.38 avg300=0.87 total=2049483252 +memory: some avg10=0.00 avg60=0.00 avg300=0.00 total=246242973 +io: some avg10=2.02 avg60=5.42 avg300=1.99 total=7832778547 +loadavg: 0.68 0.52 0.75 2/1178 188267 ``` ## tmpfs (RAM-backed; counts against memory) @@ -51,7 +51,7 @@ Filesystem Size Used Avail Use% Mounted on ## GPU ``` -NVIDIA GeForce RTX 3090, 20678 MiB, 24576 MiB, 0 %, 36 +NVIDIA GeForce RTX 3090, 20678 MiB, 24576 MiB, 0 %, 35 ``` ## Listening sockets @@ -112,6 +112,7 @@ b00t-hive-opencode-agent.service buildkit.service container-spire-agent.service containerd.service +d35d3961e3bd9927ff06fe22628efeb4126e4f849894e83fa21aa7de6a5c1949-3f2513007f457aad.service dbus.service hive-b00t-relay.service kr0ki-agent.service @@ -141,18 +142,18 @@ logrotate.service ## Containers ``` -kr0ki-sysml-mcp Up 24 hours localhost/kr0ki-sysml-mcp:dev -kr0ki-dev-kroki Up 24 hours localhost/kr0ki-kroki-compat:dev -valkey Up 21 hours (healthy) docker.io/valkey/valkey:8-alpine +kr0ki-sysml-mcp Up 25 hours localhost/kr0ki-sysml-mcp:dev +kr0ki-dev-kroki Up 25 hours localhost/kr0ki-kroki-compat:dev +valkey Up 22 hours (healthy) docker.io/valkey/valkey:8-alpine spire-agent Up 20 hours ghcr.io/spiffe/spire-agent:1.15.3 -b00t-heretic Up 12 hours ghcr.io/ggml-org/llama.cpp:server-cuda +b00t-heretic Up 13 hours ghcr.io/ggml-org/llama.cpp:server-cuda ``` ## Kernel log: OOM kills, hung tasks, GPU Xid UNREADABLE: /var/log/kern.log is syslog:adm 0640 and this user is not in adm. -Run: `sudo grep -iE "out of memory|oom-kill|hung task|blocked for more|Xid" /var/log/kern.log.1 /var/log/kern.log` +Run: `sudo grep -iE "out of memory|oom-kill|hung task|blocked for more|NVRM: Xid" /var/log/kern.log.1 /var/log/kern.log` Or: `sudo usermod -aG adm "$USER"` and log in again so this section can run unattended. ## Largest resident processes @@ -160,13 +161,13 @@ Or: `sudo usermod -aG adm "$USER"` and log in again so this section can run unat ``` RSS COMMAND 5238164 llama-server -490388 kube-apiserver -393200 claude -260212 opencode -234952 systemd-journal -218292 codex -171984 codebase-memory +490640 kube-apiserver +377228 claude +276680 opencode +236844 systemd-journal +220836 codex 161636 java +150736 mlflow ``` _Thresholds for reading this are in docs/ops/OPS-PATTERNS.md._ diff --git a/scripts/system-baseline.sh b/scripts/system-baseline.sh index 57786b5..be1741b 100755 --- a/scripts/system-baseline.sh +++ b/scripts/system-baseline.sh @@ -60,7 +60,7 @@ if have podman; then podman ps --format '{{.Names}}\t{{.Status}}\t{{.Image}}' | section "Kernel log: OOM kills, hung tasks, GPU Xid" # Needs the adm group (or root). Without it this section says so rather than reporting a false "clean". -pat='out of memory|oom-kill|hung task|blocked for more|Xid' +pat='out of memory|oom-kill|hung task|blocked for more|NVRM: Xid' if [ -r /var/log/kern.log ]; then hits=$(grep -ihE "$pat" /var/log/kern.log /var/log/kern.log.1 2>/dev/null | tail -20) if [ -n "$hits" ]; then printf '%s\n' "$hits" | cut -c1-200 | fence; else echo "none in kern.log / kern.log.1"; fi