diff --git a/content/en/docs/next/operations/configuration/system-memory-limits.md b/content/en/docs/next/operations/configuration/system-memory-limits.md index 9a554153..c5b448b4 100644 --- a/content/en/docs/next/operations/configuration/system-memory-limits.md +++ b/content/en/docs/next/operations/configuration/system-memory-limits.md @@ -21,6 +21,8 @@ memory_max.hasValue() ? 0.0 : double(memory_current.orValue(0u)) ``` +Both default blocks on this page, this one and the trigger [below](#when-the-trigger-itself-is-the-problem), are the shipped defaults at that tag: `DefaultOOMCgroupRankingExpression` and `DefaultOOMTriggerExpression` in `pkg/machinery/constants/constants.go`. Cross-check them against the constant at the tag rather than against a rendered configuration reference. The rendered `OOMConfig` reference does build its expressions from those same constants, through the `examples:` annotation on the document type, but what it renders is one example document rather than a table of defaults: it re-serialises the expressions with different parenthesisation, and it sets `sampleInterval: 100ms` against a `DefaultOOMSampleInterval` of `500ms`, with nothing marking which of its fields are shipped defaults and which are illustration. + Any cgroup scoring zero is dropped from the candidate set entirely. A pod whose containers all carry a memory limit has `memory.max` set on its pod cgroup, scores `0.0`, and can never be selected. A pod without one stays a candidate no matter how little memory it is using. Three consequences follow, and all three are easy to get wrong: @@ -35,7 +37,9 @@ Victim selection is also decoupled from the trigger. The cgroup that caused the **A default LimitRange in eligible system namespaces.** The `cozystack-operator` maintains a `LimitRange` named `cozystack-system-defaults` in each namespace it reconciles whose name does not begin with `tenant-`, defaulting container memory for anything that declares none. It covers containers whose upstream chart exposes no `resources` knob. Namespaces with a foreign memory policy or an unresolved safety hold are excluded. -**Explicit requests and limits on node DaemonSets.** Charts additionally set real values on the DaemonSets that run on every node — the cilium agent and its init containers, metallb speaker, the frr-k8s controller and its sidecars, the linstor satellite along with plunger and drbd-logger, virt-handler, fluent-bit, node-exporter, multus and its init container, velero node-agent, and all three kubevirt-csi-node containers. Requests there are fitted to observed usage, which is a real signal for the scheduler; the LimitRange cannot supply that, because it applies one number to every container in the namespace and so has to keep its default request deliberately tiny. kube-ovn is absent from that list because its vendored chart already sets both. +**Explicit requests and limits on node DaemonSets.** Charts additionally set real values on the DaemonSets that run on every node — the cilium agent and its init containers, metallb speaker, the frr-k8s controller and its sidecars, the linstor satellite along with plunger and drbd-logger, virt-handler, fluent-bit, node-exporter, multus and its init container, velero node-agent, and all three kubevirt-csi-node containers. Requests there are fitted to observed usage, which is a real signal for the scheduler; the LimitRange cannot supply that, because it applies one number to every container in the namespace and so has to keep its default request deliberately tiny. kube-ovn is absent from that list because its vendored chart already sets both on its own containers; its init containers are a separate gap, covered [below](#what-this-does-not-cover). + +**One JVM carries a request and no chart-level limit on purpose.** `linstor-controller` is a Java service, and on a JVM a `memory.max` is not only a ceiling on process RSS: it is the figure container-aware heap sizing reads, so a chart-level ceiling shrinks the heap rather than merely capping the process. It therefore declares requests only in `packages/system/linstor/values.yaml`, pinned with `notExists` on the limit in `packages/system/linstor/tests/cluster_test.yaml`, so re-adding a guessed ceiling fails the suite rather than passing quietly. Be precise about what that buys it, because the namespace LimitRange still defaults a limit wherever it is in place: the pod then does get `memory.max` and does leave the victim set, at the namespace-wide default rather than at a figure fitted to the workload. Replacing that with a fitted ceiling needs a measured peak plus an explicit heap share for the JVM rather than a guessed value, which is why the chart states none today. The operator's LimitRange stops at the `tenant-` prefix, so tenant namespaces never receive it. A tenant namespace gets a default of its own only from the tenant chart's `tenant-range-limits`, which is rendered only when `resourceQuotas` is configured on the tenant and is empty by default. A tenant workload left without a memory limit therefore stays an eviction candidate, which is the upstream design working as intended and is what restores the ordering described above. @@ -50,18 +54,28 @@ Two installer values, both empty by default so the operator's own defaults apply The limit is a ceiling, not a reservation, and the default is deliberately far above any real working set. The handler discards a cgroup for carrying a `memory.max` at all, whatever its value, so what the default buys is immunity rather than a fitted ceiling. That is also why it is set so high: the scan described below sees *declared requests*, never a container's actual resident set, so a ceiling low enough to bind is a ceiling that can convert a rare pressure-driven kill into a deterministic one for any component whose real usage nobody measured. Raising it is close to free; lowering it is where the risk lives, and it is worth doing per component in that component's own chart before doing it fleet-wide here. -**The limit must stay above the largest memory request in any system namespace.** A defaulted limit below a container's own request is rejected at admission, and the pod simply will not start. List the requests before lowering it — the operator marks exactly the namespaces it treats as system with `cozystack.io/system=true`, the same condition under which it creates the LimitRange: +**The limit must stay above the largest memory request declared by a system-namespace container that carries no memory limit of its own.** A container whose own request is above the defaulted limit is rejected at admission, and the pod simply will not start. `LimitRanger` is an in-tree admission plugin and runs well ahead of every mutating webhook in the API server's plugin order, so a request a webhook raises later is checked against a container that already carries the defaulted limit. VPA keeps the ratio between request and limit under `controlledValues: RequestsAndLimits`, which is the upstream default and which no VPA in this repository overrides, so it rescales that limit along with the request. A VPA set to `RequestsOnly`, or any other webhook that raises a request and leaves the limit alone, would leave the defaulted limit behind and get the pod rejected, and nothing can list that in advance; the `RequestsOnly` case is [listed below](#what-this-does-not-cover) as outside the scan. What can be listed in advance is a request declared on the workload itself, so list those before lowering the limit, workload templates as well as live pods, because a template is the only place a workload with no pods yet appears while a live pod is the only place a container still running from before the LimitRange was applied appears. The operator marks exactly the namespaces it treats as system with `cozystack.io/system=true`, the same condition under which it creates the LimitRange: ```bash -for ns in $(kubectl get ns -l cozystack.io/system=true -o jsonpath='{.items[*].metadata.name}'); do - kubectl get pods -n "$ns" -o json | jq -r --arg ns "$ns" ' +( + set -eu -o pipefail + namespaces=$(kubectl get namespaces -l cozystack.io/system=true -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}') + [ -n "$namespaces" ] || { echo "no namespace carries cozystack.io/system=true" >&2; exit 1; } + printf '%s\n' "$namespaces" | while read -r ns; do + kubectl get pods,deployments,daemonsets,statefulsets,jobs,cronjobs -n "$ns" -o json + done | jq -r ' .items[] - | [.spec.containers[], (.spec.initContainers // [])[]][] - | select(.resources.requests.memory != null) - | "\($ns)\t\(.name)\t\(.resources.requests.memory)"' -done | sort -u + | .metadata.namespace as $ns + | "\(.kind)/\(.metadata.name)" as $owner + | (.spec.template.spec // .spec.jobTemplate.spec.template.spec // .spec) as $pod + | (($pod.containers // []) + ($pod.initContainers // []))[] + | select(.resources.requests.memory != null and .resources.limits.memory == null) + | "\($ns)\t\($owner)\t\(.name)\t\(.resources.requests.memory)"' | sort -u +) ``` +Each line names the namespace, the object the container came from, the container and its request. A `Deployment/...` line is a workload whose pods need not exist yet; a `Pod/...` line is a container running right now with no limit of its own, which in a namespace that already has the LimitRange means a pod admitted before it was applied. Containers already declaring a memory limit are left out, because a LimitRange default never reaches them, and a pod whose template lives in a custom resource shows up only once the workload it generates does. Quantities print as declared, so compare the largest by hand rather than trusting the sort. The subshell is what makes an empty result trustworthy: an unreadable cluster or a missing label exits non-zero instead of printing nothing and reading as a clean bill of health. + **Keep the request small.** The operator always pairs the default limit with a default request, because a `LimitRange` that sets `default` without `defaultRequest` makes each container's request equal its limit and reserves the whole ceiling at schedule time. Note that leaving `systemNamespaceMemoryRequest` empty does not produce that state: an empty installer value omits the flag, which selects the operator's own `32Mi`. The knob is there to be raised or lowered, and clearing it is not a way to switch the request off. The operator also refuses to start when the request exceeds the limit. A `LimitRange` whose `defaultRequest` is above its `default` is rejected by the API server, which would wedge namespace reconciliation for every system package, so the check happens at startup rather than at apply time. @@ -116,13 +130,23 @@ Confirm a pod cgroup actually carries `memory.max` — this is the property that ```bash POD_UID=$(kubectl get pod -n -o jsonpath='{.metadata.uid}') NODE=$(kubectl get pod -n -o jsonpath='{.spec.nodeName}') -NODE_IP=$(kubectl get node "$NODE" -o jsonpath='{.status.addresses[?(@.type=="InternalIP")].address}') -talosctl -n "$NODE_IP" read "/sys/fs/cgroup/kubepods/burstable/pod$POD_UID/memory.max" +NODE_IP=$([ -z "$NODE" ] || kubectl get node "$NODE" -o jsonpath='{.status.addresses[?(@.type=="InternalIP")].address}') +( + set -eu + [ -n "$POD_UID" ] && [ -n "$NODE_IP" ] || { echo "could not resolve the pod UID or the node address" >&2; exit 1; } + case $(kubectl get pod -n -o jsonpath='{.status.qosClass}') in + Guaranteed) QOS_DIR= ;; + Burstable) QOS_DIR=burstable/ ;; + BestEffort) QOS_DIR=besteffort/ ;; + *) echo "could not read the pod's QoS class" >&2; exit 1 ;; + esac + talosctl -n "$NODE_IP" read "/sys/fs/cgroup/kubepods/${QOS_DIR}pod$POD_UID/memory.max" +) ``` -A byte count rather than the literal `max` means the pod is out of the victim set. Note that `talosctl -n` takes a machine address rather than a Kubernetes node name, which is why the address is looked up instead of being reused from `nodeName`. +A byte count rather than the literal `max` means the pod is out of the victim set. Note that `talosctl -n` takes a machine address rather than a Kubernetes node name, which is why the address is looked up instead of being reused from `nodeName`. The three lookups sit outside the subshell so that `NODE` and `NODE_IP` are still set for the two commands further down, while `set -eu` and the `exit` stay inside it and so cannot take an interactive shell down with them. They are guarded explicitly rather than left to `set -e`, because `kubectl get -o jsonpath` exits zero and prints nothing when the field it asks for is absent, which is the ordinary state of a pod that has not been scheduled yet. -Mind the QoS segment of that path, and do not extrapolate it. BestEffort and Burstable pods live under `kubepods/besteffort/` and `kubepods/burstable/`, but there is no `guaranteed` directory: kubelet creates QoS-level cgroups for those two classes only, so a Guaranteed pod sits directly at `kubepods/pod`. To avoid guessing, list the pod directories two levels down instead: +The `case` is there because the QoS segment of that path is not uniform. BestEffort and Burstable pods live under `kubepods/besteffort/` and `kubepods/burstable/`, but there is no `guaranteed` directory: kubelet creates QoS-level cgroups for those two classes only, so a Guaranteed pod sits directly at `kubepods/pod`. Reading `.status.qosClass` picks the right one of the three rather than assuming Burstable, and a class that cannot be read stops the command instead of sending it to a path that does not exist. To see the layout for yourself, list the pod directories two levels down: ```bash talosctl -n "$NODE_IP" ls -d 2 -t d /sys/fs/cgroup/kubepods @@ -154,7 +178,7 @@ Both halves of that signature — `OOM controller triggered`, then `no eligible **Future Pods outside the scanned templates.** A custom controller can change its desired request without producing a Deployment, StatefulSet, DaemonSet, CronJob or Job. If its first replacement Pod is rejected at admission, there is no new Pod for this scanner to inspect. An ownerless ReplicaSet at zero replicas has the same gap. A `RequestsOnly` VPA can likewise introduce an unseen request after defaulting. The hold prevents automatic lowering and automatic recovery after an observed blocker, but does not validate arbitrary future custom-resource changes. Configure explicit resources for those workloads and inspect their admission errors. -**A handful of vendored containers with no upstream `resources` knob** — the four frr-k8s `cp-*` init containers, the kube-ovn `hostpath-init` and `install-cni` init containers, and `cozy-proxy`, whose chart exposes no resources value at all. These are covered by the namespace LimitRange and nothing else. That is enough for OOM immunity, since the defaulted limit is what puts `memory.max` on the cgroup, but their request is then the generic default rather than a figure fitted to measured usage, so the scheduler gets a weaker signal for them than for the DaemonSets above. Lifting that needs changes upstream. +**A handful of vendored containers with no upstream `resources` knob** — the four frr-k8s `cp-*` init containers, `cozy-proxy`, whose chart exposes no resources value at all, and two kube-ovn init containers. These are covered by the namespace LimitRange and nothing else. Mind the reach of the kube-ovn pair, because it is not confined to the node DaemonSets: `install-cni` runs only on the `kube-ovn-cni` DaemonSet, but in the default render `hostpath-init` runs on six workloads that are as much control-plane as node, three DaemonSets (`ovs-ovn`, `kube-ovn-pinger`, `kube-ovn-cni`) and three Deployments (`ovn-central`, `kube-ovn-controller`, `kube-ovn-monitor`), with a seventh, `ovn-ic-controller`, when `func.ENABLE_IC` is set. That is enough for OOM immunity, since the defaulted limit is what puts `memory.max` on the cgroup, but their request is then the generic default rather than a figure fitted to measured usage, so the scheduler gets a weaker signal for them than for the DaemonSets above. Lifting that needs changes upstream. **Optional components** `hami` and `kilo` carry no chart-level values, on the grounds that inventing numbers for components with no usage measurements behind them is worse than the blanket default. diff --git a/content/en/docs/v1.7/operations/configuration/system-memory-limits.md b/content/en/docs/v1.7/operations/configuration/system-memory-limits.md index 9a554153..c5b448b4 100644 --- a/content/en/docs/v1.7/operations/configuration/system-memory-limits.md +++ b/content/en/docs/v1.7/operations/configuration/system-memory-limits.md @@ -21,6 +21,8 @@ memory_max.hasValue() ? 0.0 : double(memory_current.orValue(0u)) ``` +Both default blocks on this page, this one and the trigger [below](#when-the-trigger-itself-is-the-problem), are the shipped defaults at that tag: `DefaultOOMCgroupRankingExpression` and `DefaultOOMTriggerExpression` in `pkg/machinery/constants/constants.go`. Cross-check them against the constant at the tag rather than against a rendered configuration reference. The rendered `OOMConfig` reference does build its expressions from those same constants, through the `examples:` annotation on the document type, but what it renders is one example document rather than a table of defaults: it re-serialises the expressions with different parenthesisation, and it sets `sampleInterval: 100ms` against a `DefaultOOMSampleInterval` of `500ms`, with nothing marking which of its fields are shipped defaults and which are illustration. + Any cgroup scoring zero is dropped from the candidate set entirely. A pod whose containers all carry a memory limit has `memory.max` set on its pod cgroup, scores `0.0`, and can never be selected. A pod without one stays a candidate no matter how little memory it is using. Three consequences follow, and all three are easy to get wrong: @@ -35,7 +37,9 @@ Victim selection is also decoupled from the trigger. The cgroup that caused the **A default LimitRange in eligible system namespaces.** The `cozystack-operator` maintains a `LimitRange` named `cozystack-system-defaults` in each namespace it reconciles whose name does not begin with `tenant-`, defaulting container memory for anything that declares none. It covers containers whose upstream chart exposes no `resources` knob. Namespaces with a foreign memory policy or an unresolved safety hold are excluded. -**Explicit requests and limits on node DaemonSets.** Charts additionally set real values on the DaemonSets that run on every node — the cilium agent and its init containers, metallb speaker, the frr-k8s controller and its sidecars, the linstor satellite along with plunger and drbd-logger, virt-handler, fluent-bit, node-exporter, multus and its init container, velero node-agent, and all three kubevirt-csi-node containers. Requests there are fitted to observed usage, which is a real signal for the scheduler; the LimitRange cannot supply that, because it applies one number to every container in the namespace and so has to keep its default request deliberately tiny. kube-ovn is absent from that list because its vendored chart already sets both. +**Explicit requests and limits on node DaemonSets.** Charts additionally set real values on the DaemonSets that run on every node — the cilium agent and its init containers, metallb speaker, the frr-k8s controller and its sidecars, the linstor satellite along with plunger and drbd-logger, virt-handler, fluent-bit, node-exporter, multus and its init container, velero node-agent, and all three kubevirt-csi-node containers. Requests there are fitted to observed usage, which is a real signal for the scheduler; the LimitRange cannot supply that, because it applies one number to every container in the namespace and so has to keep its default request deliberately tiny. kube-ovn is absent from that list because its vendored chart already sets both on its own containers; its init containers are a separate gap, covered [below](#what-this-does-not-cover). + +**One JVM carries a request and no chart-level limit on purpose.** `linstor-controller` is a Java service, and on a JVM a `memory.max` is not only a ceiling on process RSS: it is the figure container-aware heap sizing reads, so a chart-level ceiling shrinks the heap rather than merely capping the process. It therefore declares requests only in `packages/system/linstor/values.yaml`, pinned with `notExists` on the limit in `packages/system/linstor/tests/cluster_test.yaml`, so re-adding a guessed ceiling fails the suite rather than passing quietly. Be precise about what that buys it, because the namespace LimitRange still defaults a limit wherever it is in place: the pod then does get `memory.max` and does leave the victim set, at the namespace-wide default rather than at a figure fitted to the workload. Replacing that with a fitted ceiling needs a measured peak plus an explicit heap share for the JVM rather than a guessed value, which is why the chart states none today. The operator's LimitRange stops at the `tenant-` prefix, so tenant namespaces never receive it. A tenant namespace gets a default of its own only from the tenant chart's `tenant-range-limits`, which is rendered only when `resourceQuotas` is configured on the tenant and is empty by default. A tenant workload left without a memory limit therefore stays an eviction candidate, which is the upstream design working as intended and is what restores the ordering described above. @@ -50,18 +54,28 @@ Two installer values, both empty by default so the operator's own defaults apply The limit is a ceiling, not a reservation, and the default is deliberately far above any real working set. The handler discards a cgroup for carrying a `memory.max` at all, whatever its value, so what the default buys is immunity rather than a fitted ceiling. That is also why it is set so high: the scan described below sees *declared requests*, never a container's actual resident set, so a ceiling low enough to bind is a ceiling that can convert a rare pressure-driven kill into a deterministic one for any component whose real usage nobody measured. Raising it is close to free; lowering it is where the risk lives, and it is worth doing per component in that component's own chart before doing it fleet-wide here. -**The limit must stay above the largest memory request in any system namespace.** A defaulted limit below a container's own request is rejected at admission, and the pod simply will not start. List the requests before lowering it — the operator marks exactly the namespaces it treats as system with `cozystack.io/system=true`, the same condition under which it creates the LimitRange: +**The limit must stay above the largest memory request declared by a system-namespace container that carries no memory limit of its own.** A container whose own request is above the defaulted limit is rejected at admission, and the pod simply will not start. `LimitRanger` is an in-tree admission plugin and runs well ahead of every mutating webhook in the API server's plugin order, so a request a webhook raises later is checked against a container that already carries the defaulted limit. VPA keeps the ratio between request and limit under `controlledValues: RequestsAndLimits`, which is the upstream default and which no VPA in this repository overrides, so it rescales that limit along with the request. A VPA set to `RequestsOnly`, or any other webhook that raises a request and leaves the limit alone, would leave the defaulted limit behind and get the pod rejected, and nothing can list that in advance; the `RequestsOnly` case is [listed below](#what-this-does-not-cover) as outside the scan. What can be listed in advance is a request declared on the workload itself, so list those before lowering the limit, workload templates as well as live pods, because a template is the only place a workload with no pods yet appears while a live pod is the only place a container still running from before the LimitRange was applied appears. The operator marks exactly the namespaces it treats as system with `cozystack.io/system=true`, the same condition under which it creates the LimitRange: ```bash -for ns in $(kubectl get ns -l cozystack.io/system=true -o jsonpath='{.items[*].metadata.name}'); do - kubectl get pods -n "$ns" -o json | jq -r --arg ns "$ns" ' +( + set -eu -o pipefail + namespaces=$(kubectl get namespaces -l cozystack.io/system=true -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}') + [ -n "$namespaces" ] || { echo "no namespace carries cozystack.io/system=true" >&2; exit 1; } + printf '%s\n' "$namespaces" | while read -r ns; do + kubectl get pods,deployments,daemonsets,statefulsets,jobs,cronjobs -n "$ns" -o json + done | jq -r ' .items[] - | [.spec.containers[], (.spec.initContainers // [])[]][] - | select(.resources.requests.memory != null) - | "\($ns)\t\(.name)\t\(.resources.requests.memory)"' -done | sort -u + | .metadata.namespace as $ns + | "\(.kind)/\(.metadata.name)" as $owner + | (.spec.template.spec // .spec.jobTemplate.spec.template.spec // .spec) as $pod + | (($pod.containers // []) + ($pod.initContainers // []))[] + | select(.resources.requests.memory != null and .resources.limits.memory == null) + | "\($ns)\t\($owner)\t\(.name)\t\(.resources.requests.memory)"' | sort -u +) ``` +Each line names the namespace, the object the container came from, the container and its request. A `Deployment/...` line is a workload whose pods need not exist yet; a `Pod/...` line is a container running right now with no limit of its own, which in a namespace that already has the LimitRange means a pod admitted before it was applied. Containers already declaring a memory limit are left out, because a LimitRange default never reaches them, and a pod whose template lives in a custom resource shows up only once the workload it generates does. Quantities print as declared, so compare the largest by hand rather than trusting the sort. The subshell is what makes an empty result trustworthy: an unreadable cluster or a missing label exits non-zero instead of printing nothing and reading as a clean bill of health. + **Keep the request small.** The operator always pairs the default limit with a default request, because a `LimitRange` that sets `default` without `defaultRequest` makes each container's request equal its limit and reserves the whole ceiling at schedule time. Note that leaving `systemNamespaceMemoryRequest` empty does not produce that state: an empty installer value omits the flag, which selects the operator's own `32Mi`. The knob is there to be raised or lowered, and clearing it is not a way to switch the request off. The operator also refuses to start when the request exceeds the limit. A `LimitRange` whose `defaultRequest` is above its `default` is rejected by the API server, which would wedge namespace reconciliation for every system package, so the check happens at startup rather than at apply time. @@ -116,13 +130,23 @@ Confirm a pod cgroup actually carries `memory.max` — this is the property that ```bash POD_UID=$(kubectl get pod -n -o jsonpath='{.metadata.uid}') NODE=$(kubectl get pod -n -o jsonpath='{.spec.nodeName}') -NODE_IP=$(kubectl get node "$NODE" -o jsonpath='{.status.addresses[?(@.type=="InternalIP")].address}') -talosctl -n "$NODE_IP" read "/sys/fs/cgroup/kubepods/burstable/pod$POD_UID/memory.max" +NODE_IP=$([ -z "$NODE" ] || kubectl get node "$NODE" -o jsonpath='{.status.addresses[?(@.type=="InternalIP")].address}') +( + set -eu + [ -n "$POD_UID" ] && [ -n "$NODE_IP" ] || { echo "could not resolve the pod UID or the node address" >&2; exit 1; } + case $(kubectl get pod -n -o jsonpath='{.status.qosClass}') in + Guaranteed) QOS_DIR= ;; + Burstable) QOS_DIR=burstable/ ;; + BestEffort) QOS_DIR=besteffort/ ;; + *) echo "could not read the pod's QoS class" >&2; exit 1 ;; + esac + talosctl -n "$NODE_IP" read "/sys/fs/cgroup/kubepods/${QOS_DIR}pod$POD_UID/memory.max" +) ``` -A byte count rather than the literal `max` means the pod is out of the victim set. Note that `talosctl -n` takes a machine address rather than a Kubernetes node name, which is why the address is looked up instead of being reused from `nodeName`. +A byte count rather than the literal `max` means the pod is out of the victim set. Note that `talosctl -n` takes a machine address rather than a Kubernetes node name, which is why the address is looked up instead of being reused from `nodeName`. The three lookups sit outside the subshell so that `NODE` and `NODE_IP` are still set for the two commands further down, while `set -eu` and the `exit` stay inside it and so cannot take an interactive shell down with them. They are guarded explicitly rather than left to `set -e`, because `kubectl get -o jsonpath` exits zero and prints nothing when the field it asks for is absent, which is the ordinary state of a pod that has not been scheduled yet. -Mind the QoS segment of that path, and do not extrapolate it. BestEffort and Burstable pods live under `kubepods/besteffort/` and `kubepods/burstable/`, but there is no `guaranteed` directory: kubelet creates QoS-level cgroups for those two classes only, so a Guaranteed pod sits directly at `kubepods/pod`. To avoid guessing, list the pod directories two levels down instead: +The `case` is there because the QoS segment of that path is not uniform. BestEffort and Burstable pods live under `kubepods/besteffort/` and `kubepods/burstable/`, but there is no `guaranteed` directory: kubelet creates QoS-level cgroups for those two classes only, so a Guaranteed pod sits directly at `kubepods/pod`. Reading `.status.qosClass` picks the right one of the three rather than assuming Burstable, and a class that cannot be read stops the command instead of sending it to a path that does not exist. To see the layout for yourself, list the pod directories two levels down: ```bash talosctl -n "$NODE_IP" ls -d 2 -t d /sys/fs/cgroup/kubepods @@ -154,7 +178,7 @@ Both halves of that signature — `OOM controller triggered`, then `no eligible **Future Pods outside the scanned templates.** A custom controller can change its desired request without producing a Deployment, StatefulSet, DaemonSet, CronJob or Job. If its first replacement Pod is rejected at admission, there is no new Pod for this scanner to inspect. An ownerless ReplicaSet at zero replicas has the same gap. A `RequestsOnly` VPA can likewise introduce an unseen request after defaulting. The hold prevents automatic lowering and automatic recovery after an observed blocker, but does not validate arbitrary future custom-resource changes. Configure explicit resources for those workloads and inspect their admission errors. -**A handful of vendored containers with no upstream `resources` knob** — the four frr-k8s `cp-*` init containers, the kube-ovn `hostpath-init` and `install-cni` init containers, and `cozy-proxy`, whose chart exposes no resources value at all. These are covered by the namespace LimitRange and nothing else. That is enough for OOM immunity, since the defaulted limit is what puts `memory.max` on the cgroup, but their request is then the generic default rather than a figure fitted to measured usage, so the scheduler gets a weaker signal for them than for the DaemonSets above. Lifting that needs changes upstream. +**A handful of vendored containers with no upstream `resources` knob** — the four frr-k8s `cp-*` init containers, `cozy-proxy`, whose chart exposes no resources value at all, and two kube-ovn init containers. These are covered by the namespace LimitRange and nothing else. Mind the reach of the kube-ovn pair, because it is not confined to the node DaemonSets: `install-cni` runs only on the `kube-ovn-cni` DaemonSet, but in the default render `hostpath-init` runs on six workloads that are as much control-plane as node, three DaemonSets (`ovs-ovn`, `kube-ovn-pinger`, `kube-ovn-cni`) and three Deployments (`ovn-central`, `kube-ovn-controller`, `kube-ovn-monitor`), with a seventh, `ovn-ic-controller`, when `func.ENABLE_IC` is set. That is enough for OOM immunity, since the defaulted limit is what puts `memory.max` on the cgroup, but their request is then the generic default rather than a figure fitted to measured usage, so the scheduler gets a weaker signal for them than for the DaemonSets above. Lifting that needs changes upstream. **Optional components** `hami` and `kilo` carry no chart-level values, on the grounds that inventing numbers for components with no usage measurements behind them is worse than the blanket default.