Skip to content

Migrate the workload, Flux and provider validation ClusterPolicies to ValidatingPolicy #4996

Closed
12 of 12 issues completed

Description

@devantler

🤖 Generated by the Agentic Engineer (interactive Claude Code session)

Evidence and problem

Kyverno 1.20 is planned to remove ClusterPolicy (#4984). These 10 validation policies cover workload hygiene, Flux and the production provider; each must exist as a ValidatingPolicy (policies.kyverno.io) before 1.20 is installed:

  • k8s/bases/infrastructure/cluster-policies/best-practices/: disallow-latest-tag, validate-host-restrictions, validate-pdb-drain-safe, validate-pod-security, validate-replica-ceiling, validate-replica-floor
  • k8s/bases/infrastructure/cluster-policies/flux/: enforce-flux-best-practices
  • k8s/providers/hetzner/infrastructure/cluster-policies/: require-baseline-workers-for-hcloud-volumes, restrict-arc-openbao-certificate, restrict-backstage-retirement

Four have no Kyverno CLI test today: disallow-latest-tag, validate-pdb-drain-safe, validate-replica-ceiling and restrict-backstage-retirement.

How every migration batch works

  • Test first. Of the 49 ClusterPolicy manifests, 28 are named in no tests/**/kyverno-test.yaml (counted 2026-10-17 by policy name). Before rewriting a policy, add a Kyverno CLI test that pins its current behaviour — at least one resource it must act on and one it must leave alone — and see it pass against the ClusterPolicy. The same test, pointed at the new kind, must pass unchanged in its expected results.
  • Keep the action. Each rewritten policy keeps its failure action as recorded in tests/policy-failure-actions.json (Enforce stays Enforce, Audit stays Audit), its match scope and its exclusions.
  • No gap in enforcement. Add the new policy and delete the ClusterPolicy in the same PR, so Flux applies the replacement before it prunes the old one. Kinds whose two versions could both act on the same request (mutation or generation) say how double application is avoided.
  • Confirm the capability first. For each rule, confirm the replacement kind can express it at Kyverno 1.19.1 (the running version). A rule it cannot express is recorded on Migrate Kyverno ClusterPolicy and Policy before Kyverno 1.20 removes them #4984 and blocks only that policy, not the batch.
  • Validate with kyverno test ./tests and scripts/validate-kyverno-fixture-evaluation.sh (CI's ⚖️ Validate cluster policies step), the layer builds in platform/AGENTS.md, and after deploy a read-only check that each new policy reports ready and the old one is gone.

Acceptance criteria

  • Every policy listed above has a CLI test that passed against the ClusterPolicy before the rewrite.
  • Each is a ValidatingPolicy with the same failure action, scope and exclusions, and the old ClusterPolicy is deleted in the same PR.
  • kyverno test ./tests and the fixture-evaluation check pass in CI.
  • After deploy, each new policy reports ready and no ClusterPolicy of these names remains.

Priority High, Effort High, Service class Risk reduction. These rules keep workloads drain-safe, pod security enforced and production storage on the right workers. Effort includes adding the missing tests, the CEL rewrite, review and one production deploy. No prerequisite; #4994 removes the time pressure but is not required first. Part of #4984.

Activity

  1. added theissue type on Oct 17, 2026
  2. moved this to 🫴 Ready in 🌊 Project Boardon Oct 17, 2026
  3. updated Effort High and Priority High on Oct 17, 2026
  4. moved this from 🫴 Ready to 🏃🏻‍♂️ In Progress in 🌊 Project Boardon Oct 18, 2026
  5. devantler commented on Oct 19, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    Test-first checkpoint, 19 October: #5080 merged at exact head 14a4e38 through merge commit d4f3e11. Its protected merge-group deployment and queue-membership check succeeded, and the separated platform health readback is OK.

    Every policy this issue lists now has a Kyverno CLI test. For restrict-backstage-retirement, two of its five rules are pinned: preserve-retirement-journal and fence-native-targets. preserve-retained-identities, fence-retiring-dependencies and fence-chart-consumers still need allowed and denied cases before that policy is rewritten; those fixtures need writer identities and owner graphs.

    When writing those fixtures, keep in mind that kyverno test merges a per-resource request.oldObject into the resource. A key only the new object has is therefore invisible: change a value, never add a key.

  6. devantler commented on Oct 19, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    Test-first checkpoint, 19 October (later): #5081 pins the preserve-retained-identities rule of restrict-backstage-retirement and is in the merge queue at 3a4363330bf69a78d4d4b13013931de7c32d32f6. It covers the general case (an untouched retained object is allowed; a label change, a data change, or any update under an unbound journal is denied; an unretained object is out of scope) and the Helm history writer exception (only the recorded Helm controller may move the retained latest release forward to uninstalled).

    Writer identities are not a blocker for fixtures: a per-resource request.userInfo with username and uid in values.yaml drives the writer variables, and a wrong uid alone flips the result.

    Still unpinned before the rewrite: the CNPG certificate, CNPG role-label and CA-bundle writer exceptions of preserve-retained-identities, and the fence-retiring-dependencies and fence-chart-consumers rules.

  7. added 4 commits that reference this issue on Oct 19, 2026
  8. 26 remaining items

  9. added a commit that references this issue on Oct 20, 2026
  10. devantler commented on Oct 20, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    Host-restriction pre-deployment comparison, 20 October UTC: independently evaluated #5141 at 401f4e284d18eef102e09c11ebd74344333861c2 with Kyverno 1.19.1.

    • Both the predecessor and replacement allow all 324 Pods in one minimized read-only production snapshot, with zero evaluation errors. Complete UID, resource-identity and expected-rule joins passed. The predecessor produces 232 rule results across 112 in-scope Pods and excludes the other 212; the replacement produces one combined pass per Pod, including its exemptions. This establishes equal authorization decisions for this snapshot, rather than identical report output. A deliberately missing result is rejected by the completeness check.
    • All 14 reviewed controls pass the replacement's fixture suite. Independent application of both policies to those controls gives the same seven allowed and seven denied resources, including infrastructure exclusions, namespace lookalikes, host access, the user-namespace opt-out and the database-label exception.
    • At the live readback, the predecessor ClusterPolicy remains present and the replacement ValidatingPolicy is absent. This is prospective comparison evidence, not deployment acceptance. No production resource was changed.

    The remaining check is to bind the protected merge and successful deployment, then confirm the rendered replacement is ready with the intended Deny/background behavior and the old kind is gone. #4996 stays open with its original acceptance criteria unchanged. This checkpoint does not take over #5141 or claim its implementation or merge.

  11. added a commit that references this issue on Oct 20, 2026
  12. devantler commented on Oct 20, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    Host-restriction deployment acceptance, 20 October UTC: the pending deployment check from the previous checkpoint now passes.

    • feat(kyverno): migrate validate-host-restrictions to ValidatingPolicy #5141 merged as 4c7507df70b4d22d7d79e03e5a466c5bc3a3fdfb. The native merge-group production deployment job executed successfully; the complete 23-job roster and required aggregate were read back.
    • An independent read-only production check confirms that validate-host-restrictions is a ready ValidatingPolicy, its entire specification exactly equals the policy rendered from the merged production infrastructure layer, and it retains Deny, background evaluation and disabled controller autogen. Both reporting-permission and webhook conditions succeed. A separate successful read confirms that the old ClusterPolicy is absent.
    • Platform main subsequently advanced to 53f60460cf9bcb3583f49a5f0240cb541544038d; that change does not touch this policy or its rendering inputs.

    This completes the deployment portion of the host-restriction checkpoint, alongside the earlier comparison and #5140's delivery receipt. The remaining migrations keep #4996 open with its original acceptance criteria. No production resource was changed by this verification.

  13. devantler commented on Oct 20, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    Progress and a scoping note, 20 October UTC.

    • require-baseline-workers-for-hcloud-volumes is now tracked by Migrate require-baseline-workers-for-hcloud-volumes to ValidatingPolicy #5150, with draft feat(kyverno): migrate require-baseline-workers-for-hcloud-volumes to ValidatingPolicy #5151 in review. It is the first policy here that looks up other objects at admission. Kyverno 1.19.1 handles that: the new kind can list the namespace claims, and the CLI test supplies them from a Context file. The audit companion is now its own Audit ValidatingPolicy.
    • The last unclaimed policy, restrict-backstage-retirement, still has no CLI test. Since Complete Backstage retirement by clearing the terminal journal #5042 closed on 18 October, the portal retirement it fenced is complete. Its five Enforce rules split into two groups:
      • Three guard the retirement itself: native targets, retiring dependencies and chart consumers.
      • Two still protect retained data: retained identities and the journal namespace.
    • Before anyone writes a test for it, decide the scope for that slice. The options are to retire the three retirement rules and migrate only the two data-protection rules, or to migrate all five. Migrating rules that can no longer fire would add test and review cost and protect nothing.
  14. devantler commented on Oct 21, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    Scope decision for restrict-backstage-retirement, 21 October UTC: remove it, do not migrate it. This is now tracked by #5159, which blocks this issue natively.

    The other nine policies are unchanged: #5129 (Flux), #5144 (pod security) and #5151 (hcloud volumes) are in the merge queue.

  15. devantler commented on Oct 21, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    Post-deploy readback, 21 October ~02:15Z (read-only, production): #5129 merged 01:42Z and #5151 merged 02:11Z.

    Policy ValidatingPolicy Old ClusterPolicy
    enforce-flux-best-practices Deny, ready gone
    require-baseline-workers-for-hcloud-volumes Deny, ready (plus report-hcloud-claim-lookup-failure, Audit, ready) gone
    disallow-latest-tag, validate-pdb-drain-safe, validate-replica-ceiling, validate-replica-floor Audit, ready gone
    validate-host-restrictions, restrict-arc-openbao-certificate Deny, ready gone
    validate-pod-security — still a ClusterPolicy (behaviour now pinned by #5144, merged 01:12Z)
    restrict-backstage-retirement — still a ClusterPolicy (removal in #5159 / draft #5161)

    8 of 10 are done. What remains is the validate-pod-security rewrite against the #5144 matrix, and the backstage removal.

  16. devantler commented on Oct 21, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    All twelve child issues are closed, and a read-only production check at about 04:59Z confirms the last acceptance criterion: each of the nine migrated policies (disallow-latest-tag, validate-host-restrictions, validate-pdb-drain-safe, validate-pod-security, validate-replica-ceiling, validate-replica-floor, enforce-flux-best-practices, require-baseline-workers-for-hcloud-volumes, restrict-arc-openbao-certificate) exists as a ValidatingPolicy reporting ready, and none of the ten names remains as a cluster policy. The tenth, restrict-backstage-retirement, was retired rather than migrated (#5159 / #5161). The same read listed 36 other cluster policies, so the absence is a real answer. Closing as complete.

  17. moved this from 🏃🏻‍♂️ In Progress to ✅ Done in 🌊 Project Boardon Oct 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Fields

    Priority

    High

    Effort

    High

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions