Volume-bound pods can be stranded Pending because stateless pods fill the baseline workers #4031

🤖 Generated by the Agentic Engineer

Evidence

On 2026-09-21 the merge-group deploy for #4029 restarted the Coroot ClickHouse shard replicas. One replica came back fine; the other has stayed Pending since ~21:28Z with Insufficient memory, and the deploy's own rollout gate timed out after 15 minutes, which evicted #4029 from the merge queue.

Read-only checks against prod (aggregate figures only; detail is in private operator notes):

• The replica can only run on a baseline worker: its data is an hcloud volume, and require-baseline-workers-for-hcloud-volumes pins such pods there. Its sibling replica occupies the third baseline worker (pod anti-affinity).
• The baseline workers are at 87–94 % of memory requested. The replica's VPA-sized request (~1.3 GiB, matching live usage of ~1.1 GiB, so it is not over-sized) does not fit on either candidate.
• The autoscaler cannot help: autoscaled nodes are not baseline workers, so it reports didn't match Pod's node affinity and never scales up.
• Meanwhile, stateless pods with no volumes (for example Velero at ~1.2 GiB and the Kubescape operator at ~0.9 GiB) are sitting on those same baseline workers, because prefer-baseline-nodes deliberately steers every pod there for FinOps. The one autoscaled node is at ~23 % memory requested.
• Preemption does not rescue it: the replica runs at priority 0, the same as those stateless pods.

Who is affected and how

Coroot's ClickHouse shard is running on one replica instead of two, so storage redundancy is lost until capacity frees up. More broadly, any restart of a volume-bound pod — a node drain, the pending Talos upgrade (#3713), an eviction — can strand it the same way, because stateless pods fill the only nodes it is allowed to use. Flux reports Ready throughout, so nothing alerts on it.

Expected behaviour

A pod that can only run on a baseline worker always gets a place there. Stateless pods may use baseline capacity while it is spare, but they give it up (and move to autoscaled nodes, scaling up if needed) when a volume-bound pod needs it.

Candidate fixes (to decide in the implementing PR)

1. Priority for volume-bound pods. A dedicated PriorityClass above 0 for pods carrying hcloud volumes, so the scheduler preempts stateless pods, which then reschedule onto autoscaled nodes. The Coroot operator does not expose priorityClassName for ClickHouse, so this needs an admission mutation; whether a webhook-set priorityClassName is honoured after the built-in Priority plugin has already resolved .spec.priority must be verified first.
2. Keep large stateless pods off baseline workers. Narrower, but it trades against the FinOps intent of prefer-baseline-nodes and does not protect against the next stateless workload that grows.

Option 1 fixes the class; option 2 only moves the current pods.

Acceptance criteria

• A volume-bound pod restarted while baseline workers are full is scheduled (by preempting stateless pods) within a bounded time, shown on a fixture or live observation.
• Stateless pods still prefer baseline workers when there is room (the FinOps behaviour is unchanged).
• fix(security): activate the Coroot baseline context with an admission write per template #4029's merge-group deploy passes its Coroot rollout gate.

Rough size: M.