Skip to content

ops: system baseline script + operational patterns (2026-10-08) #102

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
elasticdotventures merged 3 commits into from
Oct 15, 2026
Merged

ops: system baseline script + operational patterns (2026-10-08) #102

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
bf372e9
ops: read-only system baseline script, snapshot and operational patterns
elasticdotventures Oct 8, 2026
7757b0c
ops: baseline reports the kernel-log check (OOM/hung task/Xid) or why…
elasticdotventures Oct 8, 2026
c98d370
ops: record kernel-log findings; fix Xid pattern false positives
elasticdotventures Oct 8, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
95 changes: 95 additions & 0 deletions docs/ops/OPS-PATTERNS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
# Operational patterns — what is normal, what is not

Consolidated from the 2026-10-07 host stall and the 2026-10-08 unit clean-up. The matching snapshot is
[`SYSTEM-BASELINE-2026-10-08.md`](SYSTEM-BASELINE-2026-10-08.md); regenerate with
`just ops-baseline` (read-only, no sudo).

**Status of the numbers.** "Observed" values were measured on this host on 2026-10-08. "Proposed" thresholds are
judgement calls that have not been tuned against a longer history; treat them as starting points, not facts.

## 1. Reading the snapshot

| Signal | Observed 2026-10-08 | Proposed "investigate" line | Why |
|---|---|---|---|
| Memory PSI `some avg10` | 0.00 | > 5 for a minute | memory stalls precede the hard stalls seen on 2026-10-07 |
| IO PSI `some avg10` | **77–88 (abnormal)** | > 20 sustained | root is a spinning disk (~100 random IOPS); see §3 |
| CPU PSI `some avg10` | ~1 | > 25 | 4 CPUs; cargo builds are the usual cause |
| `MemAvailable` | 13 GB of 32 GB | < 3 GB | `scripts/guard-run.sh` trips on this |
| Swap used | 6.3 GB, all on the NVMe file | any use of the HDD `/swap.img` | HDD swap should be overflow only |
| tmpfs `/tmp` | **12 of 16 GB** | > 2 GB | `/tmp` is RAM; see §3 |
| `NRestarts >= 5` | 2 historical (pi-agent 11, qwen36 6) | any *currently* restarting unit | see §2 |
| GPU memory | 20.7 of 24.6 GB (Qwen3.8 + mmproj) | n/a | the GPU is owned by `heretic-neo-coder`; do not start the qwen36 unit |

Normal listeners on loopback: 8000 (socat → 8002), 8002 (Qwen3.8), 8010 (Kroki compat), 8790 (SysML MCP).
On all interfaces: 8787 (kr0ki), 8789 (agent), 22, 111, 631. The k0s control plane (6443, 2380, 10250) and
NATS (4222) and Redis/valkey (6379) are also up.

## 2. Failure patterns seen, and the fix for each

**P1 — The restart counter is not the health signal.** `b00t-historian` showed 2,323 restarts and
`container-spire-agent` showed a restart storm in the journal, yet both had been stable for 18 hours. Check
`systemctl --user status` ("Active … since") and `journalctl --since -1h` first. `systemctl --user reset-failed <unit>`
clears the counter once you have confirmed the unit is healthy.

**P2 — A missing `ExecStart` binary loops forever.** `buildkit.service` pointed at `~/nerdctl/extras/rootless/…`,
which does not exist, exited `203/EXEC` and restarted every 2 s: 36,000 restarts. systemd's default start-rate limit
did not stop it because `RestartSec=2` spaces the starts just outside the window. For any `Restart=always` unit use
`RestartSec=5` or more, plus `StartLimitIntervalSec=300` and `StartLimitBurst=5`.
*Fixed:* the unit now runs `/usr/local/bin/containerd-rootless-setuptool.sh` (backup kept beside the scratchpad).

**P3 — A unit that needs a service the host does not wire up.** `b00t-hive-serena` runs
`kubectl apply -f …/serena.yaml`, but `~/.kube/config` has no current context. A k0s controller *is* running on this
host (`k0scontroller.service`), so the cluster exists; it is the kubeconfig that is missing. The unit retried every
30 s, 2,646 times. *Disabled for now* (`systemctl --user disable --now`). To bring it back, create an admin
kubeconfig for k0s and give the unit its own `KUBECONFIG` instead of editing your main config (commands in the PR
description).

**P4 — Expired credentials in a pre-start step.** `b00t-historian` failed `ExecStartPre=sync-nats-secrets.sh` with
`AADSTS700024: Client assertion is not within its valid time range` until the assertion was refreshed. If it
loops again, read the first error line in the journal, not the restart count.

**P5 — `Conflicts=` is applied even when the unit's own condition fails.** Test-starting the Qwen3.6 unit stopped
Qwen3.8 mid-load. Never `start` a conflicting unit "to see what happens". The boot-order cause was fixed at its
source, `~/.b00t/_b00t_/opencode.agent.toml`.

**P6 — `/tmp` is a 16 GB tmpfs, so scratch space is RAM.** 12 GB of it was a cargo `target/` directory from an
earlier Claude session. That shows up as `Shmem` (9.4 GB) in `/proc/meminfo`, shrinks the page cache, and is
swapped out when memory is tight. Put build trees under `~/.cache` or `/c0de`, never the scratchpad. Anything in
`/tmp` is also lost on reboot. The earlier scratchpad held four git *worktrees* (`b00t-pr`, `infra-pr`,
`infra-spire`, `infra-vultr`); their commits and branches live in `~/.b00t` and `~/promptexecution/infrastructure`, so
only uncommitted edits (none were present) would be lost, not the commits. Check `git status` and the `.git` file
of anything in a scratchpad before assuming either way.

**P7 — Spinning-disk root, so random reads dominate IO pressure.** With swap now on NVMe, the remaining IO PSI is
file-backed page faults against `/` (observed: `dm-0` 100 % util at ~190 random 4 KB reads/s, NVMe idle). Heavy
readers were the Kubernetes API server and its datastore, plus long-lived CLI processes. Keeping hot data
(`/var/lib/k0s`, container storage, build caches) on `/c0de` is the structural fix; it has not been done.

**P8 — The kernel log needs the `adm` group.** `/var/log/kern.log*` is `syslog:adm 0640` and `dmesg` is restricted, so
an agent running as the user cannot check for OOM kills. Either add the user to `adm` or use
`sudo grep -iE "out of memory|oom-kill|hung task|blocked for more|NVRM: Xid" /var/log/kern.log.1 /var/log/kern.log`
(use `NVRM: Xid`, not bare `Xid`: the bare form also matches the r8169 NIC line "XID 480" and overlayfs layer names).

## 3. Kernel-log findings (read 2026-10-08, `kern.log.1` + `kern.log`)

| When (UTC) | Event | Reading |
|---|---|---|
| 2026-09-29 15:42–15:46 | `containerd` tasks blocked 122–245 s, then "future hung task reports are suppressed" | disk stall; the kernel stopped reporting further ones |
| 2026-10-02 14:37–14:43 | **global OOM**: kernel killed k0s pods first (envoy-gateway, coredns, metrics-server, mlflow, kube-rbac-proxy, local-path-prov, envoy), then **`llama-server` (8.3 GB resident) in the Qwen container** | the only real out-of-memory event in the log; Kubernetes pods carry `oom_score_adj` ~1000 so they die before the model |
| 2026-10-05 09:11 and 14:28 | `containerd-shim` blocked >122 s | same disk-stall signature as 29 Sep |
| 2026-10-07 | **no OOM-kill, no hung-task, no GPU error** | see below |
| any | GPU `NVRM: Xid` | **none**; the two "XID" hits were a NIC and an overlayfs layer name |

**What this means for the 7 Oct stall.** The kernel logged nothing on 7 Oct. That fits a stall in which the machine stopped
responding without the kernel noticing an out-of-memory condition (heavy swap and IO wait), or one that ended in a hard reset
that gave the kernel no chance to write. It does *not* confirm either; it only rules out an OOM kill and a reported hung task as
the visible cause. The pattern is recurring, though: three separate disk-stall or OOM episodes in nine days, all involving
containerd/k0s on the spinning root disk, which supports moving `/var/lib/k0s` and container storage to `/c0de` (P7).

## 4. Standing rules

1. Before changing a unit, read its journal and confirm the failure is the unit's own (P1–P4).
2. Never run a `Conflicts=` unit to test it (P5).
3. Build trees and clones live on disk, not `/tmp` (P6).
4. Wrap anything memory-hungry in `scripts/guard-run.sh` (capped scope + PSI watchdog).
5. Take a baseline before and after a change: `just ops-baseline > …`.
173 changes: 173 additions & 0 deletions docs/ops/SYSTEM-BASELINE-2026-10-08.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,173 @@
# System baseline — sm3llsl1k3s0ld3r

Captured 2026-10-08T09:38:22Z UTC · kernel 7.0.0-38-generic · 4 CPUs

## Memory and swap

```
total used free shared buff/cache available
Mem: 32038 10373 2180 373 20635 21664
Swap: 36863 3789 33074

NAME TYPE SIZE USED PRIO
/swap.img file 4G 0B -1
/c0de/swap/swapfile file 32G 3.7G 100

Cached: 12816616 kB
SwapCached: 84672 kB
Dirty: 1332 kB
Mapped: 1462276 kB
Shmem: 382064 kB
vm.swappiness=60 vm.page-cluster=0
```

## Pressure (PSI) and load

```
cpu: some avg10=1.64 avg60=1.38 avg300=0.87 total=2049483252
memory: some avg10=0.00 avg60=0.00 avg300=0.00 total=246242973
io: some avg10=2.02 avg60=5.42 avg300=1.99 total=7832778547
loadavg: 0.68 0.52 0.75 2/1178 188267
```

## tmpfs (RAM-backed; counts against memory)

```
Filesystem Size Used Avail Use% Mounted on
tmpfs 3.2G 3.3M 3.2G 1% /run
tmpfs 16G 168K 16G 1% /dev/shm
tmpfs 16G 39M 16G 1% /tmp
tmpfs 3.2G 416K 3.2G 1% /run/user/1000
```

## Disks

```
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv 7.2T 4.6T 2.4T 67% /
/dev/nvme0n1p5 468G 277G 168G 63% /c0de
```

## GPU

```
NVIDIA GeForce RTX 3090, 20678 MiB, 24576 MiB, 0 %, 35
```

## Listening sockets

```
<host-ip>:179
<host-ip>:34973
<host-ip>:50051
<host-ip>:53
0.0.0.0:111
0.0.0.0:22
0.0.0.0:631
0.0.0.0:8787
0.0.0.0:8789
*:10249
*:10250
*:10256
[::]:111
127.0.0.1:10248
127.0.0.1:10257
127.0.0.1:10259
127.0.0.1:20241
127.0.0.1:3000
127.0.0.1:33767
127.0.0.1:39043
127.0.0.1:50051
127.0.0.1:6011
127.0.0.1:6012
127.0.0.1:6013
127.0.0.1:8000
127.0.0.1:8790
127.0.0.53%lo:53
127.0.0.54:53
[::1]:6011
[::1]:6012
[::1]:6013
*:20244
[::]:22
*:2380
*:4222
[::]:631
*:6379
*:6443
*:8002
*:8010
*:8080
*:8222
[fd7a:115c:a1e0::3c01:cedc]:55066
```

## systemd --user: running services

```
b00t-historian.service
b00t-hive-hive-watchdog.service
b00t-hive-inference-heretic-neo-coder.service
b00t-hive-opencode-agent.service
buildkit.service
container-spire-agent.service
containerd.service
d35d3961e3bd9927ff06fe22628efeb4126e4f849894e83fa21aa7de6a5c1949-3f2513007f457aad.service
dbus.service
hive-b00t-relay.service
kr0ki-agent.service
kr0ki-kroki.service
kr0ki-llm-socat.service
kr0ki-server.service
kr0ki-sysml-mcp.service
valkey.service
xdg-document-portal.service
xdg-permission-store.service
```

## systemd --user: problem units (failed / restarting)

```
--- NRestarts >= 5 (running or not):
b00t-hive-inference-qwen36-35b-a3b-llamacpp.service NRestarts=6
b00t-hive-pi-agent.service NRestarts=11
```

## systemd (system): failed

```
logrotate.service
```

## Containers

```
kr0ki-sysml-mcp Up 25 hours localhost/kr0ki-sysml-mcp:dev
kr0ki-dev-kroki Up 25 hours localhost/kr0ki-kroki-compat:dev
valkey Up 22 hours (healthy) docker.io/valkey/valkey:8-alpine
spire-agent Up 20 hours ghcr.io/spiffe/spire-agent:1.15.3
b00t-heretic Up 13 hours ghcr.io/ggml-org/llama.cpp:server-cuda
```

## Kernel log: OOM kills, hung tasks, GPU Xid

UNREADABLE: /var/log/kern.log is syslog:adm 0640 and this user is not in adm.

Run: `sudo grep -iE "out of memory|oom-kill|hung task|blocked for more|NVRM: Xid" /var/log/kern.log.1 /var/log/kern.log`
Or: `sudo usermod -aG adm "$USER"` and log in again so this section can run unattended.

## Largest resident processes

```
RSS COMMAND
5238164 llama-server
490640 kube-apiserver
377228 claude
276680 opencode
236844 systemd-journal
220836 codex
161636 java
150736 mlflow
```

_Thresholds for reading this are in docs/ops/OPS-PATTERNS.md._
4 changes: 4 additions & 0 deletions justfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -568,6 +568,10 @@ assurance-gaps:
delegate +prompt:
scripts/delegate-qwen38.sh {{prompt}}

# Read-only host snapshot (memory, PSI, swap, tmpfs, units, containers) as markdown. See docs/ops/OPS-PATTERNS.md.
ops-baseline:
scripts/system-baseline.sh

# Revision-bound compiler evidence and compressed derived artifacts.
docgen-image revision="HEAD":
bash scripts/docgen.sh image "{{revision}}"
Expand Down
77 changes: 77 additions & 0 deletions scripts/system-baseline.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
#!/usr/bin/env bash
# system-baseline.sh — read-only "what is normal on this host" snapshot, as markdown on stdout.
#
# No sudo, no writes, no network calls beyond loopback. Safe to run any time (it does not touch
# services). Redirect it to keep a dated snapshot:
# scripts/system-baseline.sh > docs/ops/SYSTEM-BASELINE-$(date -u +%F).md
#
# Reads: /proc (meminfo, pressure, loadavg), systemd --user, podman, ss, df, nvidia-smi (optional).
# Each section degrades to "n/a" when a tool is missing. Environment values are never printed.
set -uo pipefail

have() { command -v "$1" >/dev/null 2>&1; }
section() { printf '\n## %s\n\n' "$1"; }
fence() { printf '```\n'; cat; printf '```\n'; }

printf '# System baseline — %s\n\n' "$(hostname)"
printf 'Captured %s UTC · kernel %s · %s CPUs\n' "$(date -u +%FT%TZ)" "$(uname -r)" "$(nproc)"

section "Memory and swap"
{ free -m; echo; swapon --show 2>/dev/null || echo "swapon: n/a"; echo;
grep -E '^(Shmem|Cached|Dirty|Mapped|SwapCached):' /proc/meminfo;
echo "vm.swappiness=$(cat /proc/sys/vm/swappiness) vm.page-cluster=$(cat /proc/sys/vm/page-cluster)"; } | fence

section "Pressure (PSI) and load"
{ for r in cpu memory io; do echo "$r: $(head -1 /proc/pressure/$r 2>/dev/null || echo n/a)"; done;
echo "loadavg: $(cat /proc/loadavg)"; } | fence

section "tmpfs (RAM-backed; counts against memory)"
df -h -t tmpfs 2>/dev/null | awk 'NR==1 || $3!="0"' | fence

section "Disks"
df -h / /c0de 2>/dev/null | fence

section "GPU"
if have nvidia-smi; then
nvidia-smi --query-gpu=name,memory.used,memory.total,utilization.gpu,temperature.gpu --format=csv,noheader | fence
else echo "n/a"; fi

section "Listening sockets"
# Specific host addresses are masked: tracked files must not carry machine addresses (AGENTS.md).
if have ss; then
ss -ltnH | awk '{print $4}' | sed -E 's/^(127\.|\[::1?\]|0\.0\.0\.0|\*|\[::\]|127\.0\.0\.5)/\1/; t; s/^[0-9.]+:/<host-ip>:/' | sort -u | fence
else echo "n/a"; fi

section "systemd --user: running services"
systemctl --user list-units --type=service --state=running --no-legend --plain 2>/dev/null | awk '{print $1}' | fence

section "systemd --user: problem units (failed / restarting)"
{ systemctl --user list-units --state=failed,activating,auto-restart --no-legend --plain 2>/dev/null | awk '{print $1, $3, $4}';
echo "--- NRestarts >= 5 (running or not):";
for u in $(systemctl --user list-units --type=service --all --no-legend --plain 2>/dev/null | awk '{print $1}'); do
n=$(systemctl --user show "$u" -p NRestarts --value 2>/dev/null); [ "${n:-0}" -ge 5 ] 2>/dev/null && echo "$u NRestarts=$n"
done; } | fence

section "systemd (system): failed"
systemctl list-units --state=failed --no-legend --plain 2>/dev/null | awk '{print $1}' | fence

section "Containers"
if have podman; then podman ps --format '{{.Names}}\t{{.Status}}\t{{.Image}}' | cut -c1-140 | fence; else echo "n/a"; fi

section "Kernel log: OOM kills, hung tasks, GPU Xid"
# Needs the adm group (or root). Without it this section says so rather than reporting a false "clean".
pat='out of memory|oom-kill|hung task|blocked for more|NVRM: Xid'
if [ -r /var/log/kern.log ]; then
hits=$(grep -ihE "$pat" /var/log/kern.log /var/log/kern.log.1 2>/dev/null | tail -20)
if [ -n "$hits" ]; then printf '%s\n' "$hits" | cut -c1-200 | fence; else echo "none in kern.log / kern.log.1"; fi
else
echo "UNREADABLE: /var/log/kern.log is syslog:adm 0640 and this user is not in adm."
echo
echo 'Run: `sudo grep -iE "'"$pat"'" /var/log/kern.log.1 /var/log/kern.log`'
echo 'Or: `sudo usermod -aG adm "$USER"` and log in again so this section can run unattended.'
fi

section "Largest resident processes"
ps -eo rss,comm --sort=-rss | head -9 | fence

printf '\n_Thresholds for reading this are in docs/ops/OPS-PATTERNS.md._\n'
Loading