Memory Exhausting in a container causes performance issue for other containers on the same host

[horner]

> User: I am interested in why high memory in a single container with only 4gb or ram would cause this load spike in proxmox

GitHub Copilot: Short answer: because Linux "load" isn't CPU usage — it counts both runnable tasks and tasks stuck in uninterruptible sleep (D-state), and a container thrashing against its memory cgroup limit generates lots of both.

What happens when an LXC container hits its 4GB cap:

1. Cgroup memory pressure → direct reclaim. LXC memory limits are enforced by the host kernel via cgroups. When the container approaches 4GB, every allocation inside it triggers direct reclaim — the allocating process itself runs kernel reclaim code on the host CPU. That's real host CPU/kernel work, not contained "inside" the CT.

2. Swap thrashing / page cache refaults. The kernel starts swapping the container's pages out and dropping its page cache. Processes then immediately fault those pages back in. Each fault means synchronous disk I/O, and the process waits in D-state. Load average counts every one of those blocked tasks, so a container with 20 thrashing threads can add ~20 to the node's load while CPU utilization looks modest. (This matches your graph: tall load spikes in a plateau/spike pattern typical of thrash cycles, not sustained CPU work.)

3. Kernel threads run at host level. kswapd, writeback threads, and (if you use ZFS) ARC reclaim/arc_prune all do the shoveling on behalf of the container — that work shows up as node load, not container CPU.

4. OOM-kill cycles. If the workload keeps growing, the cgroup OOM killer kills it, the service restarts, re-allocates, and the cycle repeats — producing repeated spikes like the separated humps in your chart.

How to confirm on the node:

# Per-cgroup memory pressure (PSI) — the smoking gun
cat /sys/fs/cgroup/lxc/<CTID>/memory.pressure
cat /proc/pressure/memory /proc/pressure/io

# Swap-in/out activity during a spike
vmstat 1        # watch si/so columns

# D-state processes during a spike
ps -eo state,pid,comm | awk '$1=="D"'

# OOM events for that container
journalctl -k | grep -i oom

Fixes: raise the CT's memory limit, reduce or disable the CT's swap (swap: 0 in the CT config stops thrash-via-swap and fails fast to OOM instead), or fix the app's memory footprint. If the node uses ZFS, also check that ARC isn't competing for the same RAM (arc_summary, cap via zfs_arc_max).

If you share the Proxmox browser tab or give me SSH access to the node from this terminal, I can help dig into the actual PSI/OOM evidence.

> User: lets get into opensource-phxdc-pve1 and look at what is going on.

GitHub Copilot: Ran terminal command: ssh -o ConnectTimeout=10 -o BatchMode=yes root@opensource-phxdc-pve1 "hostname && uptime && free -h"

Ran terminal command: ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new root@opensource-phxdc-pve1 "hostname && uptime && free -h"

The SSH session is waiting for the root password. Please type it directly into the terminal (I can't handle secrets) — then I'll continue investigating.

> User: I just sudo su -

GitHub Copilot: Run in Terminal

> User: now we can jump betwen hosts

GitHub Copilot: Ran terminal command: whoami && ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new root@opensource-phxdc-pve1 "hostname && uptime && free -h && echo --- && cat /proc/pressure/memory"

Ran terminal command: ps -eo state,pid,ppid,wchan:30,comm | awk '$1=="D"' | head -40; echo ===; ps -eo state | grep -c D

Ran terminal command: for d in /sys/fs/cgroup/lxc/*/; do ctid=$(basename $d); p=$(grep '^full' $d/memory.pressure 2>/dev/null | awk '{print $2}' | cut -d= -f2); [ -n "$p" ] && echo "$ctid full_avg10=$p"; done | sort -t= -k2 -rn | head; echo ===; top -bn1 -o %CPU | head -25

Found it: container 392 is at 83% memory-stall (full_avg10=83.63) — it's the one thrashing. Let's see what's inside it.

Ran terminal command: pct config 392 | head -20; echo ===; cat /sys/fs/cgroup/lxc/392/memory.current /sys/fs/cgroup/lxc/392/memory.max /sys/fs/cgroup/lxc/392/memory.swap.current 2>/dev/null; echo ===; grep -E 'oom|refault' /sys/fs/cgroup/lxc/392/memory.events /sys/fs/cgroup/lxc/392/memory.stat 2>/dev/null | grep -Ev ' 0$' | head

Ran terminal command: export PATH=$PATH:/usr/sbin; pct config 392; echo ===; grep -cE '^oom' /sys/fs/cgroup/lxc/392/memory.events; cat /sys/fs/cgroup/lxc/392/memory.events

Ran terminal command: cat /etc/pve/lxc/392.conf 2>&1 | head; echo ===; systemctl is-active pve-cluster; echo ===; for p in $(cat /sys/fs/cgroup/lxc/392/ns/cgroup.procs 2>/dev/null || cat /sys/fs/cgroup/lxc/392/cgroup.procs); do echo "$p $(cat /proc/$p/comm 2>/dev/null) rss=$(awk '/VmRSS/{print $2,$3}' /proc/$p/status 2>/dev/null)"; done | sort -k3 -rn -t= | head -20

Ran terminal command: pvecm status 2>&1 | tail -15; echo ===; ps --ppid $(head -1 /proc/$(pgrep -f "lxc-start.392" | head -1)/task//children 2>/dev/null) 2>/dev/null; find /sys/fs/cgroup/lxc/392 -name cgroup.procs -exec cat {} ; 2>/dev/null | sort -u | wc -l

Ran terminal command: sudo -n true 2>&1 && echo PASSWORDLESS_OK || echo NEEDS_PASSWORD

Checked terminal output

Ran terminal command: whoami; hostname; sudo -n whoami 2>&1

Passwordless sudo works. Continuing:

Ran terminal command: sudo cat /etc/pve/lxc/392.conf; echo ===; sudo grep -E '^(anon|file|slab_reclaimable|kernel|inactive_file|active_file) ' /sys/fs/cgroup/lxc/392/memory.stat

Ran terminal command: sudo find /sys/fs/cgroup/lxc/392 -name cgroup.procs -exec cat {} ; 2>/dev/null | sort -un | while read p; do rss=$(awk '/VmRSS/{print $2}' /proc/$p/status 2>/dev/null); [ -n "$rss" ] && echo "$rss $p $(tr '\0' ' ' < /proc/$p/cmdline 2>/dev/null | cut -c1-100)"; done | sort -rn | head -15

Culprit identified. CT 392 is a code-server dev workspace and it's simply oversubscribed:

RSS	Process
1.78 GB	code-server VS Code extension host
678 MB	second code-server bootstrap process
418 MB	vite --port 3000 dev server (support-crm)
357 MB	kerebron-server (node)
187 MB	code-server entry
93 + 74 MB	two more duplicate vite instances

That's ~3.6 GB of unswappable anon memory in a 4 GB container. The kernel can't reclaim any of it (swap: 0), so it strangles the remaining ~80 MB of file cache — every process constantly refaults its own code pages from disk (1.8B refaults, cgroup hit memory.max 3,889 times). Those stalls are what's pumping the node load to ~53.

Recommendations, in order of impact:

1. Kill the duplicate vite dev servers — pids 1527703/1527685 and 1378423/1378404 look like abandoned npm run dev sessions alongside the active one (1353407).
2. Bump the CT to 6–8 GB — code-server + extension host + a Vite dev server realistically needs ~6 GB. It's an interactive dev box on a host with 430 GB available; 4 GB is just too tight.
3. The 1.78 GB extension host suggests a heavy extension (Copilot/language servers) or a leak — a code-server "Reload Window" would reset it.

Also flagging: that container config embeds LDAP bind and Wazuh registration passwords in plaintext lxc.environment.runtime lines (readable by anyone with node shell access, and now in this terminal's scrollback). Worth rotating/moving to a secrets mechanism.

Want me to kill the stale vite processes and/or raise the memory limit (pct set 392 -memory 8192)?

> User: not yet. I am glad we have a root cause. I want to discuss ideas for detection and mitigation for future events. When a memory exhaustion occurs what should we do on the host computer to resuce the situation and prevent other containers from paying a price for the thrashing?

GitHub Copilot: Good question — this is a classic "cgroups contain memory, but not the side effects of memory pressure" problem. Here's how I'd think about it in three layers:

Why other containers pay a price

The thrash in CT 392 escapes containment via three channels:

• Disk I/O: 1.8B refaults means the CT hammers Ceph/pool0 with reads — shared with everyone.
• Kernel CPU time: direct reclaim runs in-kernel on the host's cores.
• Load-driven scheduling noise and pressure on shared daemons (Ceph OSDs on the same node).

So mitigation = detect early, and convert "slow thrash" into either "reclaim gently" or "fail fast (OOM kill)".

1. Detection (know within seconds, not when the graph spikes)

PSI is the right signal — it's exactly what caught this today:

• Alert on per-cgroup full avg10 > 10–20 sustained for 30–60s: /sys/fs/cgroup/lxc/<CTID>/memory.pressure
• Alert on host-level memory full avg10 > ~5 (host of this size should be near zero).
• Watch memory.events max counter deltas — CT 392 hit its limit 3,889 times silently, no OOM ever fired.

Practical options:

• Prometheus: node_exporter with --collector.pressure + cadvisor-style cgroup metrics, or a tiny textfile-collector script that dumps per-CT PSI. Alertmanager rule on full avg10.
• You already run Wazuh — a custom command check that scans /sys/fs/cgroup/lxc/*/memory.pressure each minute is a 10-line script.
• Key insight: workingset_refault rate + PSI full is the definitive "thrashing, not just full" signature. memory.current ≈ memory.max alone is normal and not alertable.

2. Automatic mitigation (make it fail fast instead of thrash)

The core problem: with swap: 0 and mostly-anon memory, the kernel can't reclaim, but it also never OOMs because it can technically keep evicting the tiny file cache. That's the thrash trap. Fixes:

1. memory.oom.group=1 on the CT cgroup — when OOM finally fires, kill the whole cgroup atomically instead of picking off one process and re-thrashing.

2. Set memory.high below memory.max (e.g. high=3.5G, max=4G). This throttles + reclaims the offender early and gradually, before it reaches the death-spiral zone. Proxmox doesn't expose this, but a hookscript or systemd drop-in can set it at CT start:

   lxc.hook.start-host: /usr/local/sbin/set-memory-high.sh
   

3. Run a userspace PSI-driven killer — this is the industry answer (Facebook's oomd, systemd's systemd-oomd, or earlyoom). Configure it per-cgroup: "if lxc/392 has memory full avg10 > 40 for 30s, kill the cgroup." It acts in seconds, whereas the kernel OOM killer can stall for many minutes while thrash destroys the node. On PVE, systemd-oomd with ManagedOOMMemoryPressure=kill on the LXC slice is the least-custom path.

4. Cap the collateral I/O: set an io.max / blkio limit on CT cgroups (or at least the known dev-box class) so a refault storm can't saturate the Ceph pool. This directly protects the other containers.

3. Manual rescue playbook (when it's happening now)

In order of preference — least to most destructive:

# 1. Confirm the offender (10 seconds)
grep -H . /sys/fs/cgroup/lxc/*/memory.pressure | grep full | sort -t= -k2 -rn | head

# 2. Freeze it — instantly stops the thrash, buys time to think, fully reversible
echo 1 > /sys/fs/cgroup/lxc/392/cgroup.freeze
# ... investigate, then: echo 0 > .../cgroup.freeze

# 3. Give it headroom live (no restart needed) — often the real fix
pct set 392 -memory 8192   # applies to running CT

# 4. Force targeted reclaim (kernel ≥5.19) instead of waiting for kswapd
echo 512M > /sys/fs/cgroup/lxc/392/memory.reclaim

# 5. Kill just the hog inside the CT (from host)
pct exec 392 -- kill -9 <pid>

# 6. Last resort
pct stop 392

cgroup.freeze is the underused gem — it's the SIGSTOP of cgroups. The instant you freeze the CT, all its reclaim/refault activity ceases, node load drains, other containers recover, and you've lost nothing: you can inspect its processes while frozen and thaw after raising the limit.

Suggested minimal package for your fleet

1. Wazuh/Prometheus alert on per-CT PSI full avg10 > 20 for 60s.
2. Hookscript on dev-class CTs: set memory.high = 90% of max and memory.oom.group=1.
3. A rescue-ct helper script on each node implementing steps 1–4 above.
4. Policy: interactive code-server CTs get ≥6 GB.

[horner]

I ran "echo 1 > /sys/fs/cgroup/lxc/392/cgroup.freeze"

[horner]

The impact is dramatic and immediate:

Metric	Before freeze	Now
Load (1-min)	~54	16.6 and falling (5-min 39, 15-min 47 — classic decay curve)
CT 392 memory PSI full avg10	83%	0.00
Host memory PSI full avg10	~20%	0.00
Host I/O PSI full avg10	—	0.22 (refault storm on Ceph stopped)

The avg60/avg300 tails still show the history (CT 392 was at 58% stall over the last 5 minutes — it had actually gotten worse since we first looked), but the avg10 windows going to zero proves the thrash stopped the instant the cgroup froze.

A few things worth noting while it's frozen:

• The 1-min load will settle to the node's true baseline in another minute or two — that number tells you what the Ceph OSDs etc. normally contribute (~5-10 by the look of it).
• The container's memory is still allocated (4 GB of anon pages held), it's just not executing. Anyone using that code-server right now sees it hang — so don't leave it frozen too long if mbachelder is working in it.
• This validates the rescue playbook: freeze is a zero-data-loss circuit breaker.

Suggested next step while frozen: raise the limit, then thaw —

pct set 392 -memory 8192
echo 0 > /sys/fs/cgroup/lxc/392/cgroup.freeze

Then we can watch PSI to confirm it stays at zero with the headroom. Want me to do that?

[horner]

Design critiques

1. "Kill the biggest process" is often wrong for graceful recovery. In CT 392 the biggest process was the code-server extension host — killing it loses the user's session state anyway. Killing inside first is still right, but treat it as targeted (biggest descendant cgroup or process group, like oomd does), and expect that SIGTERM may not work — a thrashing process can take minutes to respond to SIGTERM because it's stalled in reclaim. Freeze first, then signal — a frozen process can't run its signal handler either, so the sequence matters (thaw after delivering, or bump memory before signaling so the handler can actually run).

2. Don't permanently raise memory automatically. Auto-raising rewards runaway workloads and silently eats node capacity ("limit creep"). Instead: grant a temporary headroom bump (runtime memory.max only, not pct set) so the graceful shutdown can execute, then restore the configured limit on restart. Permanent raises stay a human decision.

3. Restarting the whole container may be overkill if the graceful kill worked — PSI going back to ~0 means recovered; leave it running. Restart only if the CT is still unhealthy after the kill. Fewer disruptions, same protection.

4. You already have an owner-identity mechanism: the CT config tags: mbachelder. Use that for notification routing.

Refined escalation ladder

Level 0  DETECT     per-CT PSI full avg10 > T1 (e.g. 40) sustained 30s
                    + workingset_refault rate rising  → confirmed thrash
Level 1  CONTAIN    cgroup.freeze the CT (instant node relief) + alert owner/dashboard
Level 2  TRIAGE     snapshot evidence: top-RSS procs, memory.stat, memory.events, PSI
                    → attach to notification/ticket automatically
Level 3  RESCUE     temp bump memory.max +50% (runtime only) → thaw
                    → SIGTERM the largest offender process (grace 30s) → SIGKILL
Level 4  VERIFY     wait 60s; PSI back < 5? → restore original memory.max, done
                    still thrashing? → pct reboot the CT
Level 5  CIRCUIT    ≥N events (e.g. 3-4) in 60 min → pct stop, mark CT
         BREAKER    "quarantined: memory", require human to re-enable

Every level emits an event → owner notification + dashboard.

Ticket-ready plan

Title: Automated per-container memory-thrash detection & remediation for PVE nodes

Problem statement: A single 4 GB LXC (CT 392) in sustained memory thrash drove node load from ~8 to ~54 on pve4/pve1, degrading Ceph and all co-tenant containers. Kernel OOM never fired (3,889 limit hits, 1.8B refaults, 0 oom_kills). Manual diagnosis/rescue took ~an hour; freeze + limit raise resolved it in seconds. We need this automated.

Deliverables:

1. Detector daemon (ct-memwatch, systemd service on each PVE node)
   • Poll /sys/fs/cgroup/lxc/*/memory.pressure + memory.stat every 5s
   • Trigger: full avg10 > 40 for 30s (thresholds configurable per-CT via a config tag or file)
   • State machine implementing the ladder above; per-CT event history in local state file
2. Remediation actions (as above: freeze → snapshot → temp-bump → graceful kill → verify → restart → circuit-break)
   • All actions logged with structured JSON to journal + a per-event evidence bundle
3. Notification integration
   • Resolve owner from CT tags: → email/Slack/whatever your standard channel is
   • PVE dashboard visibility: write a CT description/tag (memwatch:quarantined) and/or push to your monitoring (Wazuh custom event so it lands in existing alerting)
4. Circuit breaker: >3 remediation events per CT per hour → pct stop, tag quarantined, notify owner + ops; document manual re-enable procedure
5. Config & safety
   • Per-CT opt-out / threshold overrides (some CTs legitimately run hot)
   • Never act on CTs tagged memwatch:ignore; never act on privileged/system CTs
   • Dry-run mode for initial fleet rollout (detect + notify only, no actions)
6. Docs/runbook: manual rescue playbook (freeze/bump/thaw/reclaim commands) for cases the daemon punts to humans

Rollout: Phase 1 dry-run on one node (pve1) for a week → tune thresholds → Phase 2 enable Level 1–2 (freeze+alert only) → Phase 3 full ladder fleet-wide.

Acceptance criteria:

• Synthetic memory-hog CT (e.g. stress-ng --vm) is detected < 60s, node load impact contained < 90s
• Owner receives notification with evidence bundle
• 4th event in an hour stops the CT and quarantines it
• No action ever taken on tagged/system CTs; dry-run produces alerts only

Open questions for the ticket:

• Notification channel of record (email from tags? Slack? your provisioning system — since these CTs look auto-provisioned from Docker images, does that system have an owner DB/API?)
• Should Level 3 kill inside the CT at all, or is freeze→notify→wait-for-human enough for interactive dev boxes? (Batch workloads: auto-kill fine; dev boxes: maybe freeze-only during business hours)
• Where does "manager dashboard" live — PVE UI notes/tags, or an external dashboard you already run?

Want me to also draft a prototype of the detector loop (bash or Python) to attach to the ticket as a starting point?

[horner]

Fastest path: a single-file Python daemon (stdlib only) + systemd unit, packaged with nfpm — one YAML spec produces both .deb and .rpm, no debhelper/rpmbuild ceremony.

Why this stack

• Python3 stdlib only: everything the daemon does is reading cgroup files, writing cgroup.freeze, exec'ing pct, and journald logging — zero pip dependencies means zero Python packaging pain. Runtime dep is just python3 (present on every PVE node and any EL/Fedora box).
• nfpm (nfpm.goreleaser.com): a single static Go binary that turns a directory of files + a YAML manifest into .deb, .rpm (also .apk) with systemd scriptlets. Vastly simpler than maintaining debian/ + a .spec file in parallel.
• CI in GitHub Actions publishes both artifacts per tag; later you can graduate to a real apt/dnf repo with aptly/createrepo_c or just point hosts at GitHub releases.

Repo layout

ct-memwatch/
├── ct-memwatch.py          # the daemon (~200-300 lines)
├── ct-memwatch.service     # systemd unit
├── ct-memwatch.conf        # /etc/ct-memwatch.conf defaults (thresholds, dry-run=true)
├── nfpm.yaml               # package manifest (deb+rpm)
├── Makefile                # make deb / make rpm / make lint
└── .github/workflows/release.yml

The three key files

nfpm.yaml — the whole packaging story:

name: ct-memwatch
arch: all
platform: linux
version: ${VERSION}
maintainer: MIE Ops <ops@mieweb.com>
description: PSI-based per-container memory-thrash watchdog (freeze, notify, circuit-break)
license: MIT
depends: [python3]
contents:
  - src: ct-memwatch.py
    dst: /usr/sbin/ct-memwatch
    file_info: { mode: 0755 }
  - src: ct-memwatch.service
    dst: /usr/lib/systemd/system/ct-memwatch.service
  - src: ct-memwatch.conf
    dst: /etc/ct-memwatch.conf
    type: config|noreplace     # survives upgrades, both deb & rpm
scripts:
  postinstall: packaging/postinstall.sh   # systemctl daemon-reload; enable if configured
  preremove: packaging/preremove.sh       # stop + disable

ct-memwatch.service:

[Unit]
Description=Per-container memory pressure watchdog
After=multi-user.target

[Service]
ExecStart=/usr/sbin/ct-memwatch --config /etc/ct-memwatch.conf
Restart=on-failure
# Hardening — it only needs cgroupfs write + exec pct + journal
ProtectHome=yes
ProtectSystem=strict
ReadWritePaths=/sys/fs/cgroup /run
NoNewPrivileges=yes

[Install]
WantedBy=multi-user.target

Makefile:

VERSION ?= $(shell git describe --tags --always)
build:
	VERSION=$(VERSION) nfpm pkg --packager deb --target dist/
	VERSION=$(VERSION) nfpm pkg --packager rpm --target dist/

That's genuinely it — make build on any box with the nfpm binary emits dist/ct-memwatch_x.y.z_all.deb and .rpm. Install with apt install ./dist/*.deb or dnf install ./dist/*.rpm.

Design notes for the daemon itself

• Ship with dry_run = true and action = notify as defaults — safe to fleet-install immediately, flip per-node when confident.
• Config: thresholds (psi_full_avg10=40, sustain_secs=30), action ladder toggles, circuit-breaker (max_events=4/hour), notify hook = arbitrary command (notify_cmd = /usr/local/bin/notify-owner %CTID %ACTION) so Wazuh/Slack/email routing stays out of the package.
• Cgroup path is configurable (/sys/fs/cgroup/lxc/* on PVE; machine.slice/* pattern would cover libvirt/incus on the dnf side).
• State in /run/ct-memwatch/ (event history for the circuit breaker), tmpfiles.d entry included.

Distribution later (when you outgrow "download the .deb")

• apt repo: aptly or a plain deb [trusted=yes] https://pkgs.mieweb.org/apt ... pool built in CI; nfpm output drops straight in.
• dnf repo: createrepo_c on a static webroot + a .repo file.
• Both can be a single GitHub Pages/S3 bucket job in the same release workflow.

Want me to scaffold the repo now — daemon skeleton with the detect→freeze→notify→circuit-breaker state machine, unit, config, nfpm.yaml, and CI workflow — so you can dry-run it on pve1 today?

[runleveldev]

Roadmap: task breakdown into deliverable sub-issues

Following the discussion above, the work is broken into concrete sub-issues, organized by how achievable each is against what exists in the codebase today.

Key architectural constraint driving this split: the manager reaches Proxmox purely over the REST API, and the site agent runs as a single per-cluster LXC — nothing in the stack has host-level access to /sys/fs/cgroup/lxc/* on the nodes. So the effective levers (PSI reads, cgroup.freeze, memory.high, memory.reclaim, io.max) are all below the API line and require a new node-resident deployment target. Only API-settable CT config (memory, swap, tags) is a true easy win.

Easy wins (Proxmox REST API / app config only)

• Raise default container maxmem to 8GB #432 — Raise default container maxmem to 8GB
• Decide swap policy: does swap affect our memory-pressure metrics? #433 — Decide swap policy: does swap affect our memory-pressure metrics? (spoiler: only matters if detection is OOM-count-based; PSI+refault detection is swap-agnostic)

Prerequisites (enable the big feature)

• Notification queue + inbound webhook in create-a-container #434 — Notification queue + inbound webhook in create-a-container (the one piece fully buildable today with zero node access; hook target for lxc-oomd)
• Add friendly 504 error page for paused/OOMing (frozen) containers #435 — Friendly 504 error page for paused/OOMing (frozen) containers (frozen CT → nginx proxy_read_timeout → 504, not the existing 502)

Big feature

• lxc-oomd: node-resident PSI-based OOM responder with freeze + webhook hook #436 — lxc-oomd: node-resident PSI-based OOM responder with freeze-not-just-kill + packaged/overridable webhook hook

Dependency ordering

#434 (notification queue) → provides the hook target for #436 (lxc-oomd).
#433 (swap policy) → confirms lxc-oomd triggers on PSI+refault, not OOM counts.
#432 and #435 are independent and can land anytime.