Resource Accounting, Health Detection & Owner Attribution

[horner]

Draft plan for a ticket. Parent context: #431 (memory-exhaustion thrash), #436 (lxc-oomd).

Problem

We have no idea who is using what.

The cluster runs 296 containers across 132 owners on 4 Proxmox nodes, and there is currently
no mechanism to answer any of these questions:

• Which user is consuming the most CPU / RAM / disk / I/O?
• Which containers are allocated resources they never use?
• Is the cluster over-committed, and by how much?
• When a container degrades its neighbours, who do we contact?

The incident in #431 made the cost concrete: a single 4 GB container degraded all four nodes,
and identifying the owner was a manual forensic exercise. On 2026-08-02 an identical incident
recurred (CT 284, owner zbarrell) and again took manual investigation to isolate.

Evidence from the 2026-08-02 incident

Recorded here because it defines the detection thresholds and validates the design.

Single container, ozwell-studio-92e3d441 (CT 284, 4 GB cap) on pve2:

Signal	Value
Memory	3.97 GB / 4.00 GB (100%)
anon / file split	3896 MB anon, 60 MB file cache
memory.swap.max	0 (swap disabled)
CPU split	99.3% system / 0.7% user
PSI memory some avg10	99.00
memory.high breaches	321,586,574
Direct reclaim scan	841 B pages scanned, 40.8 B stolen — 4.85% efficiency
workingset_refault_file	30.6 billion
pgmajfault	99.4 million
Block reads	662 MiB/s
oom_kill	0 — kernel OOM never fired

Mechanism: anon memory filled the cap and, with swap disabled, was 100% unreclaimable. The only
reclaimable memory left was file-backed page cache — the executables and shared libraries of the
running processes. The kernel evicted code that was immediately needed again, faulted it back
from Ceph, and repeated. The container was re-reading its own program text continuously.

Effect of stopping the single container:

Metric	Before	After (T+3min)	Change
Ceph client reads	559 MiB/s	210 KiB/s	−99.96%
pve2 load1	56.00	13.04	−77%
pve2 CPU PSI	21.04	0.12	−99%
pve2 IO PSI	16.85	0.25	−99%
pve3 load1	25.02	22.99	−8%
pve4 load1	19.68	16.65	−15%
pve1 load1	6.86	5.80	−15%

One container generated 99.96% of all Ceph read traffic cluster-wide. This is why the problem
is not containable per-node: cgroups cap memory, but not the I/O consequences of memory pressure.

Current state of the codebase

Area	State
Ownership (DB)	Container.username → User.uid
Ownership (Proxmox)	tags: field, 100% coverage verified (72/72 on pve2), synced at routers/api/v1/containers.js:739
Proxmox client	create-a-container/utils/proxmox-api.js — 28 methods, API-token auth, has clusterResources()
Background jobs	create-a-container/job-runner.js — custom DB polling loop, 2 s interval, transactional claim
Quota model	ResourceRequest tracks requested resources only. No enforcement, no actual-usage comparison
Usage tracking	None — no Prometheus, RRD, metrics, or billing code anywhere
Notifications	Password-reset email only (utils/email.js). #434 not started
Agent	Per-site LXC, 30 s systemd check-in. No Proxmox API access, no node access

Key finding: most of this needs no node-resident code

#436 states that none of the useful levers are reachable from the REST API. That is only half
true, and it changes the delivery order.

Verified against the live cluster:

• /cluster/resources?type=vm returns, in one call, for every container:
  tags (= owner), cpu, maxcpu, mem, maxmem, disk, maxdisk, diskread, diskwrite,
  netin, netout, uptime, status, node, vmid.
• /nodes/{node}/lxc/{vmid}/rrddata already exports PSI:
  pressurememoryfull, pressurememorysome, pressurecpufull, pressurecpusome,
  pressureiofull, pressureiosome — at 60 s resolution, 60 points per hour.

Capability	REST API	Node-resident required
Per-CT PSI detection	yes (60 s lag)	only for sub-60 s reaction
Accounting: cpu / mem / disk / bytes / net	yes	—
Owner attribution	yes (tags)	—
stop / shutdown / reboot	yes	—
set memory, cores, swap	yes (updateLxcConfig)	—
IOPS (rios/wios)	no	yes
cgroup.freeze	no — verified: LXC status endpoints are only current, migrate, reboot, shutdown, start, stop	yes
memory.high, memory.reclaim, io.max	no	yes

Consequence: the accounting layer and detection levels L0–L2 ship with zero node-resident
deployment. lxc-oomd is still required, but only for IOPS collection, freeze, and the cgroup
tunables. It moves off the critical path.

Proof of concept: live report, API-only

Generated from one /cluster/resources call:

OWNER              CTs RUN CPU_USED CPU_ALLOC  MEM_USED  MEM_ALLOC     DISK    IO_TOT
horner              23  22     4.28        92     10.4G       126G     144G      865G
pdharamkar          17  17     1.43        68      7.3G        96G      98G      679G
rgingras            14  13     3.31        72      8.4G        76G      73G      585G
rgara               12  12     2.08        48      8.3G        48G      47G      491G
cmyers               8   8     0.03        36      7.5G        46G      27G      380G
sbarber              3   3     0.75        12      2.1G        16G      11G     1690G
TOTAL (132 owners) 296 290    52.45      1218    171.3G      1335G    1314G

Physical capacity: 288 cores, ~1.9 TiB RAM across 4 nodes.

Resource	Allocated	Used	Over-subscription
CPU	1218 cores	52.45 (18.2%)	4.23x
RAM	1335 GiB	171.3 GiB (9.0%)	0.70x

Immediately actionable from this table alone:

• cmyers holds 36 allocated cores at 0.03 used — pure idle reclamation.
• sbarber shows 1690 G of I/O from 3 containers, an order of magnitude above peers per container.
• CPU is 4.23x over-committed but only 18.2% used — the allocation model is disconnected from reality.

Warning for #432 (raise default maxmem to 8 GB)

RAM is currently 1335 GiB allocated of ~1900 GiB — not over-subscribed. Moving 296 containers
to an 8 GB default raises allocation to roughly 2.4 TiB → 1.25x over-subscribed.

Since a memory-thrash incident is what started this whole thread, raising the default before
detection exists inverts the risk. Recommend landing accounting + detection before #432.

Swap policy (#433) — answer

swap: 0 is the direct cause of the thrash-instead-of-OOM behaviour. With no swap, anon memory
is 100% unreclaimable, so the kernel can only evict executable pages.

283 of 291 capped containers cluster-wide are configured swap: 0. Every one has the same
failure mode latent. Setting ~1 GB swap converts a cliff into a slope and is API-settable.

Note this is the opposite of the advice in the original #431 discussion, which suggested
swap: 0 to "fail fast to OOM instead". The measured evidence contradicts it: with swap: 0
the kernel did not OOM — it thrashed indefinitely with oom_kill = 0 across 321 M limit
breaches.

IOPS vs bytes

io.stat per container exposes rios / wios against the RBD device (major 251), attributing
Ceph IOPS directly to a container. Not available via the API.

Fleet baseline measured with CT 284 stopped:

Metric	Value
Containers doing I/O	151
Total IOPS	853
Total throughput	5.7 MiB/s
Fleet average I/O size	6.8 KB

Small random I/O — the profile Ceph handles worst. At replication factor 3 the real OSD load is
roughly 2,500 IOPS, not 853.

The two metrics correlate in steady state (top-20 overlap: 17/20) but diverge exactly at the
failure modes, and they catch different incidents:

Failure mode	Signature	Caught by
Refault / bandwidth storm (CT 284)	662 MiB/s at ~1.3k op/s = 476 KB/op (readahead)	bytes
Small-random-I/O storm	high op count, trivial bandwidth	IOPS

Today's incident was a bandwidth problem — IOPS alone would have missed it. The inverse case is
equally capable of degrading co-tenants and is invisible to bytes. Collect both; io.stat emits
all four counters in a single read.

Design

Schema

ContainerUsageSample          -- raw, 5-min interval, 30-day retention
  containerId, username, siteId, nodeId, sampledAt, status
  cpuUsed, cpuAlloc
  memUsed, memAlloc
  diskUsed, diskAlloc
  diskReadBytes, diskWriteBytes
  diskReadOps, diskWriteOps        -- tier 3 only (node-resident)
  netInBytes, netOutBytes
  psiMemSome, psiMemFull, psiCpuSome, psiCpuFull, psiIoSome, psiIoFull

ContainerUsageDaily           -- LEAF LEDGER: one row per container per day.
  username, containerId, day      -- source of truth, retained indefinitely
  cpuCoreHours, memGbHours, diskGbDay
  ioReadGb, ioWriteGb, ioOpsTotal, netGb
  psiMemFullMinutes            -- health/abuse signal, not just consumption
  sampleCount

cpuCoreHours and memGbHours are the billing-grade primitives: integrable, comparable across
users, and directly chargeable if showback is ever wanted. Charge I/O on ioOpsTotal rather than
bytes — it tracks the cost Ceph actually bears.

Org hierarchy (new — nothing equivalent exists today):

OrgUnit                       -- cost-centre tree; stable across reorgs
  id, name, parentId, costCentreCode, active

User (extend existing model)
  managerUid                   -- self-referential FK -> User.uid; NULL = root
  orgUnitId                    -- optional FK -> OrgUnit

OrgUsageDaily                 -- DERIVED rollup; safe to drop and recompute
  scopeType                    -- 'user' | 'orgunit'
  scopeId, day, depth
  directCpuCoreHours,  directMemGbHours,  directIoOpsTotal,  ...
  subtreeCpuCoreHours, subtreeMemGbHours, subtreeIoOpsTotal, ...
  memberCount, containerCount

Direct vs subtree must be stored separately. A leader needs to see "my own containers" and
"my organisation" as distinct numbers, and keeping them in separate columns is what prevents
double counting. ContainerUsageDaily is the only ledger; OrgUsageDaily is derived and can be
rebuilt from it at any time.

Collector

New create-a-container/usage-collector.js, reusing the job-runner.js polling pattern and the
existing ProxmoxApi class.

Two-tier polling matters: /cluster/resources is 1 call, but rrddata is 1 call per
container (296 per cycle). Bulk scan first, drill into outliers only.

// every 5 min — one bulk call, cheap
const rows = await pve.clusterResources('vm');
// attribute owner via tags, cross-check against Container.username
// then, only for containers whose derived rates look suspicious:
//   pve.rrdData(node, vmid, 'hour')  → PSI series

Collection tiers

Tier	Source	Access	Metrics
1	/cluster/resources	API	owner, cpu, mem, disk, bytes, net
2	rrddata	API	PSI (60 s)
3	io.stat	node	IOPS, per-device

Tiers 1–2 ship immediately. Tier 3 rides along with lxc-oomd (#436).

Attribution

Use Proxmox tags as primary — it lives on the container itself, survives DB loss, and has 100%
coverage today. Cross-check against Container.username. Divergence between the two is itself
an alertable drift condition.

Org hierarchy & rollup

No org hierarchy exists anywhere in the system today. Verified:

• User has uidNumber, uid, gidNumber, cn, sn, givenName, mail, oidcSubject,
  oidcIssuer — but no manager, department, or title.
• Group is flat (gidNumber, cn, isAdmin) with no parent, and only two groups exist:
  sysadmins and ldapusers (migrations/20260616000000-seed-groups.js). They are authorisation
  roles, not org structure — they must not be overloaded for accounting.
• Authentik is the identity source and LDAP (ldaps://ldap1.cluster.mieweb.org:6636,
  dc=cluster,dc=mieweb,dc=org) is an outpost of it. Authentik supports arbitrary
  user.attributes, so a manager attribute can be populated there and flow through
  LDAP → manager DB without schema surgery on the directory.

So hierarchy is net-new work, and it is the single largest unknown in this plan.

Two edges, both supported, used for different purposes:

Edge	Source	Answers
User.managerUid	Authentik user.attributes.manager → LDAP → sync	"who is accountable for this person"
User.orgUnitId	manually curated, or HR feed	"which cost centre does this land in"

Start with managerUid only — it directly satisfies "leaders accountable for their staff" and
needs no new curation process. OrgUnit is deferred until someone actually asks for cost centres.

Rollup is a recursive walk up the manager tree, materialised nightly into OrgUsageDaily:

WITH RECURSIVE reports AS (
  SELECT uid, uid AS root, 0 AS depth FROM "Users"
  UNION ALL
  SELECT u.uid, r.root, r.depth + 1
  FROM "Users" u JOIN reports r ON u."managerUid" = r.uid
  WHERE r.depth < 10                    -- hard depth cap: cycle protection
)
SELECT root, SUM(d."cpuCoreHours") ...
FROM reports r JOIN "ContainerUsageDaily" d ON d.username = r.uid
GROUP BY root;

Edge cases that must be handled explicitly, because all of them exist in real directories:

• Cycles (A manages B manages A) — depth cap plus a cycle detector that logs and breaks.
• Orphans — users whose managerUid points at a departed user. Reparent to a synthetic
  unassigned root rather than silently dropping their usage from every total.
• Departed owners — containers outlive people. Usage must still roll up; attribute to the
  last known manager and flag for reclamation.
• Self-management — roots have managerUid = NULL, not self-referencing.
• Reorgs — OrgUsageDaily is snapshot-per-day, so historical rollups reflect the hierarchy
  as it was. Do not retroactively rewrite history when someone changes manager.

Collection: build vs OpenTelemetry

Worth considering, and my recommendation is not to adopt OTel as the system of record for
accounting — but to stay compatible with it.

The two workloads have genuinely different requirements:

	Accounting	Health / observability
Needs	exact, durable, auditable, per-entity totals	timely, approximate, high-resolution
Tolerates loss	no	yes
Retention	years	days–weeks
Query shape	recursive tree rollups, joins to users	time-series, rate/percentile
Natural store	relational (Postgres)	TSDB

Against OTel for the accounting path:

• OTel pipelines are lossy by design. Collectors drop under backpressure and downsample on
  retention. That is correct for monitoring and disqualifying for anything chargeback-grade.
• It is a large scope add. There is no collector, no TSDB, and no Grafana in this stack today.
  Adopting OTel means standing up and operating an entire observability platform as a prerequisite
  to shipping a usage report.
• Hierarchy rollups are awkward in PromQL. Recursive manager-tree aggregation is a natural SQL
  recursive CTE and an unnatural fit for a TSDB without pre-baked recording rules per level.
• The data volume does not justify it: ~2.5 M sample rows per 30 days is unremarkable for Postgres,
  which is already deployed and backed up.

For OTel:

• The health/PSI path is exactly what OTel is good at, and lxc-oomd: node-resident PSI-based OOM responder with freeze + webhook hook #436's daemon will produce
  high-frequency metrics that nobody wants in a relational table.
• It buys vendor-neutral export if Grafana/Prometheus arrives later.

Decision: SQL ledger for accounting, OTel-compatible export for health. Concretely:

1. Persist accounting to Postgres via Sequelize (the ledger is authoritative).
2. Name metrics using OTel semantic conventions from day one
   (container.cpu.time, container.memory.usage, container.disk.io), and carry owner/manager
   as resource attributes.
3. Ship an optional OTLP exporter behind a config flag as a later milestone, so metrics can
   feed a collector if one is ever stood up — without the ledger depending on it.

This keeps the door open at near-zero cost and avoids making a usage report block on an
observability platform.

Owner reporting

Individual

1. GET /api/v1/users/:username/usage — self-service, reusing the existing owner+collaborator
   visibility pattern (routers/api/v1/containers.js:230). Users see their own; admins see all.
2. Weekly digest email via existing nodemailer. The strongest lever is idle reclamation,
   not billing — cmyers at 0.03 of 36 cores is a polite-email problem, not a technical one.
3. Incident notification — the payload already knows the owner, satisfying lxc-oomd: node-resident PSI-based OOM responder with freeze + webhook hook #436's requirement
   that notifications route to a person. Hook target is Notification queue + inbound webhook in create-a-container #434.

Hierarchical

4. GET /api/v1/org/:uid/usage?depth=n — returns direct and subtree totals, plus a ranked
   breakdown of direct reports. Authorisation rule: a user may read any scope at or below
   themselves in the manager tree; sysadmins may read any scope. This reuses the existing
   visibility-clause pattern rather than inventing a second authorisation model.
5. Leader digest — monthly, to anyone with at least one direct report:
   team total, ranked list of reports, biggest movers month-over-month, and an idle-resource
   list (allocated-vs-used) for reclamation. The idle list is the part that produces action.
6. Fleet report for sysadmins — the per-owner table already prototyped, plus
   over-subscription ratios and attribution-drift warnings.

Reports must show allocated alongside used, never used alone. The single most actionable
number in the prototype was cmyers holding 36 cores at 0.03 used — invisible if you only
report consumption.

Detection thresholds

Three independent triggers, not one:

Trigger	Condition	Catches
Memory thrash	psiMemFull > 40 sustained 30 s and rising workingset_refault	CT 284 / CT 392 class
Bandwidth hog	diskReadBytes rate > 200 MiB/s sustained	CT 284 (would have fired)
IOPS hog	ioOps rate > 2000/s sustained	the case bytes misses

memory.current ≈ memory.max alone is not alertable — 6 containers sit at 91–99% of cap with
psiMemFull = 0.00 and are perfectly healthy. PSI plus refault rate is the discriminator.

Interventions

Ladder per #431/#436, with three amendments from the measured evidence:

1. Freeze before notify, not after. pct exec on CT 284 hung — the container was too starved
   to respond. A thrashing process cannot run a signal handler. Freeze is the only reliable first
   action, and it is node-only.
2. io.max throttling is the most valuable intervention and a stronger justification for
   lxc-oomd than freeze. Unlike freeze or kill it is graceful — it slows the offender rather
   than stopping it, protecting co-tenants without an outage. No API equivalent exists.
3. Never auto-raise memory permanently. Temporary runtime memory.max bump for graceful
   shutdown only; permanent raises stay a human decision to avoid limit creep.

Milestones

Split deliberately so that nothing in A1–A5 is blocked by another ticket. The accounting and
reporting core ships standalone; only the IOPS tier and live intervention wait on #436.

Accounting-specific (no external dependency)

ID	Milestone	Delivers	Node access
A1	Collection foundation	ContainerUsageSample schema + migration; usage-collector.js on the job-runner.js pattern; tier-1 /cluster/resources poll at 5 min; owner attribution via tags with Container.username cross-check	no
A2	PSI collection	Tier-2 rrddata poll for outliers only; PSI columns populated; two-tier throttling so we do not issue 296 calls/cycle	no
A3	Rollup & retention	Nightly job building ContainerUsageDaily; 30-day purge of raw samples; cpuCoreHours / memGbHours primitives; backfill from existing RRD where available	no
A4	Org hierarchy	User.managerUid + Authentik attribute → LDAP → sync; recursive-CTE rollup into OrgUsageDaily; cycle/orphan/departed-owner handling	no
A5	Reporting API + digests	/users/:uid/usage, /org/:uid/usage; fleet report; weekly owner digest and monthly leader digest via existing nodemailer; idle-resource report	no

Dependent on other tickets

ID	Milestone	Depends on	Why blocked
B1	Detection → notification wiring	#434	Needs the notification queue/webhook as a hook target. Detection logic itself lands in A2
B2	Swap policy rollout	#433	Decision ticket. Evidence in this doc argues for swap: 1024 fleet-wide; API-settable, no code
B3	8 GB default	#432, and should follow A1+A5	Pushes RAM allocation to ~1.25x over-subscribed; needs accounting in place to see the effect
B4	IOPS collection (tier 3)	#436	rios/wios exist only in host-side io.stat. No API equivalent
B5	Live intervention	#436	cgroup.freeze, io.max, memory.high are all node-only. Verified: LXC has no suspend endpoint
B6	Optional OTLP export	none, but sequenced last	Only useful once a collector/TSDB exists. Deliberately deferred

Dependency graph

graph LR
  A1[A1 Collection] --> A2[A2 PSI]
  A1 --> A3[A3 Rollup]
  A3 --> A4[A4 Hierarchy]
  A4 --> A5[A5 Reporting]
  A3 --> A5
  A2 --> B1[B1 Detection wiring]
  T434[#434 notify queue] --> B1
  T433[#433 swap policy] --> B2[B2 Swap rollout]
  A5 --> B3[B3 8GB default]
  T432[#432 8GB] --> B3
  T436[#436 lxc-oomd] --> B4[B4 IOPS tier]
  T436 --> B5[B5 Intervention]
  A3 --> B4
  A5 --> B6[B6 OTLP export]

Loading

Critical path to something useful: A1 → A3 → A5. That produces the per-owner report and
digests with no node access and no dependency on any other ticket.

Acceptance criteria

A1–A3 (collection & rollup)

• Per-owner usage report queryable via API and reproducible from /cluster/resources.
• Daily rollups produce cpuCoreHours / memGbHours per user, retained indefinitely.
• Raw samples purge at 30 days; rollups survive.
• Attribution drift (Proxmox tag vs Container.username) is detected and reported.
• Collector issues ≤ 2 API calls per cycle in the steady state (no per-container fan-out unless
  an outlier is flagged).

A4 (hierarchy)

• Subtree totals for any manager equal the sum of leaf ContainerUsageDaily rows beneath them —
  verified by a reconciliation test asserting no double counting.
• A cycle in the manager tree is detected, logged, and does not hang or inflate the rollup.
• A container whose owner has departed still appears in exactly one subtree total.
• Changing a user's manager does not retroactively alter historical rollups.

A5 (reporting)

• A leader can retrieve their team's usage; a non-leader cannot read a peer's or a superior's scope.
• Reports show allocated alongside used for every resource.
• Idle-resource report identifies containers by allocated-vs-used ratio.

B1/B4/B5 (dependent)

• A synthetic memory hog is detected within 60 s and attributed to the correct owner.
• Owner receives a notification containing an evidence bundle.

Open questions

Hierarchy

• Where does manager data come from — is it already in an HR system that can feed Authentik, or
  does it need manual curation for all 132 owners? We should be discussing this with Andy and making sure we populate this automatically from BambooHR
• Is a single manager tree sufficient, or are cost centres (OrgUnit) needed for cross-functional
  teams where the person's manager is not who owns the budget? We could do a hash tag like approach.
• Should a leader see per-container detail for their reports, or only aggregates? (Privacy vs
  usefulness — container hostnames can be revealing.) Leaders should see it all.
• What happens to usage attribution when someone leaves and their containers are reassigned? They should be shut down or reassigned.

Accounting

• Retention: is 30 days of 5-minute samples enough? (~296 CTs × 288 samples/day × 30 = ~2.5 M rows.). Maybe 90 days, but keep daily logs for years.
• Is showback/chargeback ever intended, or is this purely visibility + idle reclamation?
• Should quota enforcement be in scope, or is reporting sufficient for now?
  ResourceRequest already models approved quotas but nothing compares them to actual usage.
• Should digests be opt-in, or default-on for all 132 owners and their leaders?

Collection

• Confirm the OTel decision: SQL ledger now, optional OTLP export later — or is there an existing
  observability platform commitment that would change this?
• Does io.max throttling need per-owner or per-team policy, or a single fleet default?

[horner]

Field evidence: a waste class the current detection triggers would miss

While investigating high load on pve1–pve4 today (the CT 284 incident documented above), I
found a second, larger problem that none of the three proposed detection triggers would fire
on. It is worth folding into this plan because it changes both the baseline numbers and the
detection design.

Finding

mcp-proxy.service is in a permanent crash loop inside 87 containers cluster-wide.

ImportError: cannot import name 'request_ctx' from 'mcp.server.lowlevel.server'
systemd[1]: mcp-proxy.service: Main process exited, code=exited, status=1/FAILURE
systemd[1]: mcp-proxy.service: Scheduled restart job, restart counter is at 33055
systemd[1]: session-c33061.scope: Consumed 1.887s CPU time, 48M memory peak

Root cause is an unpinned dependency:

Container	mcp	mcp-proxy	Result
CT 314 (pve4)	2.0.0	0.12.0	ImportError, NRestarts=33072
CT 177 (pve3)	2.0.0	0.12.0	ImportError, NRestarts=29054
CT 163 (pve3)	1.28.1	0.12.0	Healthy, NRestarts=0

mcp 2.0.0 is a breaking major release that moved request_ctx out of
mcp.server.lowlevel.server. Containers built before that release got 1.28.1 and work;
everything built after got 2.0.0 and crash-loops forever. The unit has Restart=always /
RestartUSec=5s with no start limit, so it retries indefinitely.

Cost

Each failed start burns 1.887 s of CPU (Python interpreter boot + import chain, dying at the
import) every 5 s:

1.887s / 5s = 0.377 cores, permanently, per container

Measured per-container cgroup CPU confirms it — affected containers sit at a uniform 0.32–0.35
core floor.

Node	CTs	Container CPU	Crash-looping	Wasted
pve1	78	5.6 cores	6	1.5
pve2	71	9.2 cores	12	3.2
pve3	71	19.3 cores	41	14.3
pve4	70	15.0 cores	28	9.9
Total	290	49.1 cores	87	28.9

28.9 of 49.1 cores — 59% of all container CPU in the cluster — is this one bug.
At ~33,000 restarts × 5 s it has been running for roughly 1.9 days.

It also fully explains the residual load ranking after CT 284 was stopped: pve3 (41 affected) and
pve4 (28) were the two elevated nodes, pve1 (6) the quietest. That correlation is the entire
remaining signal.

Why this matters for the plan

1. It invalidates the CPU baseline in the proof-of-concept table.

The plan reports CPU used 52.45 of 1218 allocated (18.2%) and concludes the allocation model is
disconnected from reality. That is still true, but ~29 of those 52 cores are not workload at all —
they are a crash loop. Real utilisation is closer to 20 cores, ~1.7%, and the
over-subscription argument is correspondingly stronger, not weaker.

2. None of the three detection triggers catch it.

Trigger	Fires?	Why not
Memory thrash (psiMemFull > 40)	no	PSI memory is 0.00 in these containers
Bandwidth hog (>200 MiB/s)	no	negligible I/O
IOPS hog (>2000/s)	no	negligible I/O

There is no PSI signature, no memory pressure, no I/O. Every host-level metric looks healthy —
pve4 showed 79.6% idle, PSI cpu some avg10 = 0.18, procs_blocked averaging 0.2. The waste is
only visible in aggregate: 0.377 cores per container is a rounding error individually, and 29
cores collectively.

3. It was invisible to per-process sampling.

Worth recording as a collector implementation note. Sampling utime+stime from /proc/<pid>/stat
attributes 0.005 cores to CT 314 while its cgroup reports 0.249. The CPU belongs to processes
that die between samples; it only appears in the parent's cutime/cstime (reaped children).

Per-container cpu.stat (cgroup) is the only correct source for tier-1 CPU accounting.
Any collector built on per-process sampling will silently under-report crash loops and other
short-lived-process workloads by orders of magnitude. /cluster/resources cpu field is derived
from cgroup accounting and is safe.

Proposed additions

A fourth detection trigger — process churn / crash loop:

Trigger	Condition	Catches
Crash loop	NRestarts delta > 10/hour for any unit, or container fork rate > 5/s sustained	this class

Both are cheap. Fork rate is derivable host-side from /proc/<pid>/stat on the container init, and
NRestarts is one systemctl show per container — better suited to the tier-2 "drill into
outliers only" pattern than a full fleet sweep.

A useful cheap heuristic that needs no new collection at all: a container whose CPU is nearly all
attributable to PID 1's reaped children is, by definition, doing nothing useful. That ratio is
computable from data tier 1 already gathers.

For the digests (A5): this is exactly the "idle reclamation" lever the plan identifies as the
strongest one. A container burning 0.377 cores on a service that has failed 33,000 consecutive
times should appear in the weekly owner digest as broken, not as usage. Suggest the idle
report distinguish three states rather than two:

• allocated and unused → reclaim
• allocated and used → fine
• allocated and burning CPU on a failing unit → fix or reclaim ← new

Immediate remediation (outside this ticket)

/opt/mcp-proxy/bin/pip install 'mcp==1.28.1'
systemctl restart mcp-proxy

The durable fix is pinning mcp<2.0.0 in the ozwell-studio image build (not in this repo —
mcp-proxy does not appear anywhere in mieweb/opensource-server), plus adding a start limit to
the unit so a service that fails 33,000 times eventually stops trying:

StartLimitIntervalSec=300
StartLimitBurst=5

That alone would have capped the damage at 5 attempts instead of 29 cores for two days.

---

_{Measurements: per-container cpu.stat deltas over 10–15 s windows on all four nodes,
2026-08-02. One earlier sampling run was discarded — the instrumentation itself forked ~1500
processes and reported lxc = 29.67 cores against a measured total busy of 16.80, which is
impossible. Figures above are from cgroup-file-only sampling, cross-checked against
procs_running, /proc/stat idle deltas, and slice totals (14.25 lxc + 2.08 system.slice ≈ 16.80
busy).}