lxc-oomd: node-resident PSI-based OOM responder with freeze + webhook hook

[runleveldev]

Part of #431 (memory-exhaustion thrash). Big feature.

Problem

A single 4GB LXC in sustained memory thrash drove node load from ~8 to ~54, degrading Ceph and every co-tenant container. The kernel OOM killer never fired (3,889 limit hits, 1.8B refaults, 0 oom_kills). Manual diagnosis + rescue took ~an hour; a manual cgroup.freeze + limit raise resolved it in seconds (load 54→16 instantly, fully reversible, zero data loss). We need this automated.

None of the effective levers (PSI, cgroup.freeze, memory.reclaim, memory.high, cgroup OOM group) are reachable from the Proxmox REST API or from today's single per-cluster agent LXC. lxc-oomd must be node-resident (installed on each Proxmox host) — this is a new deployment target for the project and the main cost of this work.

What makes this different from existing tools

systemd-oomd, earlyoom, and Facebook oomd only kill. lxc-oomd's differentiator is freeze-not-just-kill: an instant, reversible, zero-data-loss circuit breaker validated in #431. Lead with that.

Deliverable

A single-file daemon (Python 3 stdlib only) + systemd unit, packaged as .deb/.rpm (fpm/nfpm, matching the repo's existing .fpm packaging in agent/ and manager-control-program/).

Detection

• Poll /sys/fs/cgroup/lxc/*/memory.pressure + memory.stat every ~5s.
• Trigger on full avg10 > T sustained for N seconds plus rising workingset_refault rate (the definitive thrash-vs-merely-full signature). PSI + refault based, not OOM-count based (see swap-policy issue Decide swap policy: does swap affect our memory-pressure metrics? #433).

Escalation ladder (each level emits an event to the hook)

• L0 DETECT → L1 CONTAIN (cgroup.freeze + notify) → L2 TRIAGE (snapshot top-RSS procs, memory.stat, memory.events, PSI) → L3 RESCUE (temporary runtime memory.max bump → thaw → SIGTERM→SIGKILL largest offender) → L4 VERIFY (PSI<5 → restore limit; else reboot CT) → L5 CIRCUIT BREAKER (>N events/hour → stop + quarantine, require human re-enable).

Hook script

• Packaged-but-overridable hook that POSTs the event payload to the notification webhook (Notification queue + inbound webhook in create-a-container #434).
• Payload must include the CT owner resolved from the CT config tags: field, so notifications route to a person rather than being anonymous.

Safety / config

• Ship dry_run = true / notify-only defaults (safe to fleet-install before enabling actions).
• Never act on privileged/system CTs; per-CT opt-out (e.g. memwatch:ignore tag); per-CT threshold overrides.
• Config file for thresholds, ladder toggles, circuit-breaker limits, and notify_cmd.

Acceptance criteria

• Synthetic hog (stress-ng --vm) detected < 60s; node load impact contained < 90s.
• Freeze stops thrash and is reversible with no data loss.
• Owner receives a notification (via Notification queue + inbound webhook in create-a-container #434) with an evidence bundle.
• 4th event/hour stops + quarantines the CT.
• No action ever taken on tagged/system CTs; dry-run produces alerts only.

Rollout

Phase 1 dry-run on one node (pve1) for a week → tune thresholds → Phase 2 freeze+alert only → Phase 3 full ladder fleet-wide.

Dependencies

• Notification queue + inbound webhook in create-a-container #434 notification queue/webhook (hook target).
• Decide swap policy: does swap affect our memory-pressure metrics? #433 swap policy (confirms PSI+refault triggering, not OOM counts).