Migrate the production storage-placement mutation ClusterPolicies to MutatingPolicy #4998

🤖 Generated by the Agentic Engineer (interactive Claude Code session)

Evidence and problem

Kyverno 1.20 is planned to remove ClusterPolicy (#4984). These 12 mutation policies in k8s/providers/hetzner/infrastructure/cluster-policies/ place storage and databases on the right production workers and keep Longhorn's node components drain-safe:

add-existing-csi-sidecar-baseline-context, add-existing-longhorn-csi-plugin-baseline-context, add-existing-longhorn-engine-image-baseline-context, drain-autoscale-node-storage, label-baseline-workers, label-existing-baseline-workers, prefer-baseline-nodes, prioritise-database-pods, prioritise-hcloud-volume-pods, remove-existing-longhorn-registrar-prestop, remove-longhorn-registrar-prestop, restrict-storage-to-baseline-workers.

Several of them mutate existing resources (targets or mutateExistingOnPolicyUpdate) and act on Nodes and on Longhorn's own components, and nine have no Kyverno CLI test. A mistake here moves database or volume Pods onto the wrong workers, which is the failure class #4980 is about.

What has to be decided before this is Ready

For each existing-resource mutation, whether MutatingPolicy at Kyverno 1.19.1 can express it with the same trigger (on policy update, on change of the target, or both), and how the handover avoids a moment where neither or both policies rewrite a Node or DaemonSet. Record the answer per policy here; split any policy that needs a staged handover into its own child.

How every migration batch works

• Test first. Of the 49 ClusterPolicy manifests, 28 are named in no tests/**/kyverno-test.yaml (counted 2026-10-17 by policy name). Before rewriting a policy, add a Kyverno CLI test that pins its current behaviour — at least one resource it must act on and one it must leave alone — and see it pass against the ClusterPolicy. The same test, pointed at the new kind, must pass unchanged in its expected results.
• Keep the action. Each rewritten policy keeps its failure action as recorded in tests/policy-failure-actions.json (Enforce stays Enforce, Audit stays Audit), its match scope and its exclusions.
• No gap in enforcement. Add the new policy and delete the ClusterPolicy in the same PR, so Flux applies the replacement before it prunes the old one. Kinds whose two versions could both act on the same request (mutation or generation) say how double application is avoided.
• Confirm the capability first. For each rule, confirm the replacement kind can express it at Kyverno 1.19.1 (the running version). A rule it cannot express is recorded on Migrate Kyverno ClusterPolicy and Policy before Kyverno 1.20 removes them #4984 and blocks only that policy, not the batch.
• Validate with kyverno test ./tests and scripts/validate-kyverno-fixture-evaluation.sh (CI's ⚖️ Validate cluster policies step), the layer builds in platform/AGENTS.md, and after deploy a read-only check that each new policy reports ready and the old one is gone.

Acceptance criteria

• The capability and handover decision above is recorded per policy.
• Every policy has a CLI test that passed against the ClusterPolicy before the rewrite.
• Each is a MutatingPolicy with the same scope and resulting objects, and the old ClusterPolicy is gone after deploy.
• After deploy, database and volume Pods keep their placement and priority, read from the live cluster, and no Longhorn component restarts because of the change.

Priority High, Effort High, Service class Risk reduction. Highest-risk batch of the migration because it touches production placement; kept in Backlog until the decision above is recorded. Part of #4984.