[Aleksei Sviridkin (lexfrei)]

This adds a design proposal for restarting the workloads of a dead node without manual action. Today a VM on a node that lost power often does not start elsewhere: its virt-launcher pod stays Terminating, and KubeVirt does not fail the VMI while that pod still looks alive. The Piraeus HA Controller may release a VM whose disks are all on replicated, but that is not verified end to end on Cozystack.

Releasing a pod is easy. Proving that the old copy is gone is the hard part, because a partitioned node looks the same as a dead one. The proposal adds an optional node-remediation package on the Medik8s operators. NodeHealthCheck finds a node that stays unready, Fence Agents Remediation powers it off through its BMC, and only after the BMC confirms the off does the out-of-service taint release the node's pods and volumes. Without a BMC that answers, the package releases nothing and the documented manual procedure applies.

I'd like review mostly on section 4, which signal counts as proof that a node is dead, and on the control-plane rule in section 2.

It is proposed together with the GPUDevice (gpudevice, #99) and GPU virtual machines (vgpu-vms, #100) proposals and lands with them or after them.

Before review

• If this revises a merged design proposal: it adds a decision record under that proposal's decisions/ directory, or says below why none is needed.

This is a new proposal and revises none.

DCO

• Commits are signed off (git commit --signoff).

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

• Configuration used: defaults
• Review profile: CHILL
• Plan: Advanced
• Run ID: 99cb2f85-97bd-4aeb-8647-815c54f5d4f8

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[Andrei Kvapil (kvaps)]

LGTM as of dc644d4, the direction is very right and we need this. Long ago I did the same thing in kube-fencing, and I didn't know someone carried this approach on. Medik8s is a much better base than writing our own.

Good that piraeus-ha-controller is covered too, it already fences on the DRBD storage level, so the two work side by side.

I checked FAR v0.8.0 against the source. It never deletes the Node object, unlike the node lifecycle controller of a CCM: with OutOfServiceTaint it only adds the taint, and the default ResourceDeletion force-deletes the node's pods. NHC deletes only its remediation CRs. FAR's ClusterRole still has delete on nodes that no code uses, so I'd drop it from the vendored manifests. The other claims hold as well: only reboot and off are accepted, the taint comes only after FenceAgentActionSucceeded, and retrycount: 0 records success without running the agent.

One thing I'd like before the package ships, it can be done in the implementation: the node to BMC mapping should be checked, not only documented. With a wrong address FAR powers off another, healthy machine, reports success, and taints the node we think is dead, so if that one is only partitioned its VMs run twice. FAR's template check runs status, which proves the BMC answers, not which machine is behind it. Comparing the system UUID from Redfish with status.nodeInfo.systemUUID would close it.

nits:

• I'd remediate only Ready=Unknown by default, Ready=False comes from a kubelet that is still alive.
• Please drop --ssl-insecure: "" from the example, people will copy it.

[myasnikovdaniil]

Hi! We removed Status and Date from proposal headers in #111. Could you remove these fields when you next update your proposal? Decision records keep their metadata.

Sorry for the extra work, I wanted to mention this now so it doesn't come up late in review.