Repository navigation
Application pods depending on listener get stuck in pending when GKE nodes restart #342
Description
Affected Stackable version
25.7
Current and expected behavior
When Google Kubernetes Engine restarts nodes, the Superset pod gets stuck in a pending state. It refuses to start due an issue with being unable to scale-up. Presumably this would extend to other pods from other operators.
I traced this back to the listener PersistentVolumeClaim. It seems to be bound to the node(s) that no longer exists in the GKE cluster due to that node being restarted/removed/whatever. Deleting the PVC causes the listener operator to re-sync, and then everything is good again. But it requires manual intervention, which is not a good situation.
So basically whenever GKE swaps out nodes due to upgrades or other infrastructure reasons, the listener and everything connected to it get stuck.
Possible solution
Update operator code to delete PVC when nodes list changes.
Additional context
No response
Environment
v1.33.5-gke.1080000
Would you like to work on fixing this bug?
no
Activity
lfrancke commentedon Oct 22, 2025
I think this is relevant: stackabletech/issues#770
In short: We're on it, but I'll let @sbernauer comment better.
jeffsweep commentedon Oct 22, 2025
Sounds good. In the meantime, I am thinking of adding a monitoring pod/job/something that deletes the PVC when nodes change. Does that make sense as a stopgap solution?
sbernauer commentedon Oct 22, 2025
With SDP 25.11 our defaults will change, so users should not run into this using the defaults.
Do you need to use a NodePort or are you willing to use a LoadBalancer instead? The pinning is only done for NodePorts.
In that case you can define this ListenerClass and use that instead:
listener-operator/deploy/helm/listener-operator/templates/listener-classes.yaml
Lines 18 to 23 in f7b4bf8
As an alternative you can switch to the nightly listener-operator 0.0.0-dev with the feature we just merged this morning: #340. With this version you can configure if pinning should happen or not
sbernauer commentedon Nov 3, 2025
Hi @jeffsweep this should be addressed with the upcoming SDP 25.11 release. In case you still run into problems please feel free to re-open!