Skip to content

EAI-7854 Skip the keycloak CNPG PDB for a single instance - #798

Open
pre wants to merge 2 commits into
mainfrom
EAI-7854-keycloak-cnpg-enablepdb
Open

EAI-7854 Skip the keycloak CNPG PDB for a single instance#798
pre wants to merge 2 commits into
mainfrom
EAI-7854-keycloak-cnpg-enablepdb

Conversation

@pre

@pre pre commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

A CloudNativePG Cluster with instances: 1 gets a <name>-primary PodDisruptionBudget at minAvailable: 1, which permits zero disruptions for as long as the cluster stays single-instance.

That PDB protects nothing. There is no second replica to keep serving, so the only thing it can do is refuse. What it refuses is kubectl drain on whichever node the pod happens to sit on, which stalls node maintenance until an operator disables it by hand on the live cluster, out of band from Git.

This came out of a maintenance window on app-install-test, where keycloak/keycloak-cnpg-1 was one of two drain blockers on the node being rebooted. The other was Longhorn's per-node instance-manager PDB, which turned out to be held open by the database pod: Longhorn refuses to release it while volumes are still attached to the node. Clearing the CNPG PDB let the drain evict the database pod, the volumes detached, and Longhorn released its own PDB on the next reconcile.

Change

enablePDB is now rendered on the Cluster, following instances unless set:

cnpg.instances cnpg.enablePDB rendered
1 unset false
2+ unset true
any true true
any false false

The default is derived rather than hardcoded, so a cluster scaled to two instances gets its PDB back without a second values change, and the field stays overridable in both directions.

Applied to both charts that render a keycloak-cnpg Cluster:

  • sources/keycloak-old is the one wired up via root/values.yaml (path: keycloak-old). It already had a cnpg values block, so this extends it.
  • sources/keycloak-config renders the same resource with a fully static manifest and had no values block beyond domain. The cnpg block is introduced there, keeping the two charts identical in behaviour so whichever one is wired up next does not reintroduce the blocker.

enablePDB is CNPG's own recommendation over the alternative. Patching nodeMaintenanceWindow.inProgress instead draws a warning from the operator:

Warning: Consider using `.spec.enablePDB` instead of the node maintenance window feature

It is also the only one of the two that survives ArgoCD. nodeMaintenanceWindow is templated by these charts, so ArgoCD owns it and selfHeal reverts a manual patch within the reconcile interval.

Verification

helm template across the matrix above, run separately for each chart, all five combinations rendering as expected with enablePDB typed !!bool and not a string:

default(instances=1)       enablePDB=false instances=1
explicit-true              enablePDB=true  instances=1
explicit-false             enablePDB=false instances=1
instances=2                enablePDB=true  instances=2
instances=2,pdb=false      enablePDB=false instances=2

For keycloak-old, cnpg.enabled=false still renders no Cluster at all, and helm lint passes.

keycloak-config does not render or lint cleanly on main either: templates/keycloak-realm-templates-cm.yaml fails to parse at line 2123, unchanged by this PR and untouched by it. The matrix above was produced with that one file moved aside; the rest of the chart, including the Cluster, renders correctly.

The rendered manifest was validated against the live CNPG v1.27.0 CRD on the app-install-test cluster with kubectl apply --dry-run=server, which reported configured and converges to the same state as the hand patch currently applied there.

Note on the sibling chart

sources/eai-infra/airm-cnpg is also single-instance with no enablePDB, so it carries the same latent drain blocker. aiwb-cnpg defaults to instances: 3 and is unaffected. Left alone here to keep this PR to the keycloak charts, but worth a follow-up.

A CloudNativePG Cluster with instances: 1 gets a <name>-primary
PodDisruptionBudget at minAvailable: 1, which permits zero disruptions
for as long as the cluster is single-instance. That PDB protects
nothing, since there is no second replica to keep serving, and it
blocks kubectl drain on whichever node the pod sits on. Node
maintenance then stalls until an operator disables it by hand on the
live cluster, out of band from Git.

enablePDB now defaults to false while instances is 1 and true from two
instances up, and an explicit cnpg.enablePDB wins either way.
@pre
pre requested a review from a team as a code owner August 5, 2026 10:41
keycloak-config renders the same keycloak-cnpg Cluster as keycloak-old,
so it carries the same drain blocker. A single-instance PDB permits zero
disruptions, which protects nothing and only stalls `kubectl drain` on
whichever node the primary sits on.

Introduce the cnpg values block this chart lacked and derive enablePDB
from instances, with an explicit value winning. Matches the pattern
already applied to keycloak-old in this branch.
@silokimmo

silokimmo commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

We could have affinity rules like following:

  affinity:
    enablePodAntiAffinity: true
    topologyKey: {{ .Values.cnpg.affinity.topologyKey }}
    podAntiAffinityType: {{ .Values.cnpg.affinity.podAntiAffinityType }}

  With defaults:
  cnpg:
    affinity:
      topologyKey: kubernetes.io/hostname    # spread across nodes, not zones
      podAntiAffinityType: preferred         # don't block scheduling on single-node clusters

@pre

pre commented Aug 5, 2026

Copy link
Copy Markdown
Contributor Author

We could have affinity rules like following:

yeah, although it should be a separate pull request.

@silokimmo silokimmo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - let's improve also affinity rules

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants