Repository navigation
fix(proxy): promote shutdown-manager to native sidecar to close preStop drain race on k8s 1.33+ #9484
Description
On Kubernetes 1.33+, envoy and shutdown-manager are ordinary sibling containers with no termination ordering guarantee. When kubelet sends SIGTERM during a pod rollout, both containers receive it simultaneously.
Envoy's preStop hook (httpGet :19002/shutdown/ready) is designed to block until shutdown-manager confirms that in-flight connections have drained. But because shutdown-manager can exit before the hook completes its polling loop, the hook fails immediately with FailedPreStopHook rather than waiting out the drain window. Kubelet then kills envoy with requests still in flight, producing client-visible 5xx errors on every proxy rollout.
Confirmed on: Envoy AI Gateway v1.7.0, Kubernetes 1.33
Reproduce
- Deploy a proxy with active traffic
- Trigger a proxy pod rollout (update
EnvoyProxyor any setting causing pod recreation) - Observe
FailedPreStopHookevents followed by connection-refused on port 19002:
Warning FailedPreStopHook pod/envoy-<hash> preStop hook for container "envoy" failed
Surfaced during a routine gateway rollout — a cluster of 5xx responses traceable to FailedPreStopHook events immediately followed by refused connections to the shutdown-manager port.
Expected behaviour
shutdown-manager should outlive envoy so the preStop drain handshake always completes before kubelet kills the proxy process.
Proposed fix
On Kubernetes 1.33+, promote shutdown-manager from spec.containers to spec.initContainers with restartPolicy: Always. This makes it a native sidecar, which kubelet guarantees will keep running until all non-sidecar containers (envoy) have fully exited, closing the race entirely.
The change belongs in the proxy pod-spec builder (roughly internal/infrastructure/kubernetes/proxy/resource.go) behind a version gate:
if k8sVersion >= 1.33 {
shutdownMgr.RestartPolicy = ptr.To(corev1.ContainerRestartPolicyAlways)
spec.InitContainers = append(spec.InitContainers, shutdownMgr)
// remove from spec.Containers
} else {
spec.Containers = append(spec.Containers, shutdownMgr)
}Why not make the preStop hook retry-tolerant?
If shutdown-manager has already exited, drain has not completed and Envoy should not be killed. Masking a missing shutdown-manager in the hook would paper over the race rather than fix it. The correct invariant is that shutdown-manager must be alive for the entire duration of Envoy's preStop window.
Notes
- This requires Kubernetes 1.33+ and should be a no-op on older clusters
- It would be useful to export
shutdownManagerContainerName(andenvoyContainerName) as package-level constants — downstream consumers currently have to hardcode"shutdown-manager"as a bare string with no compile-time contract
Activity
github-actions commentedon Aug 12, 2026
This issue has been automatically marked as stale because it has not had activity in the last 30 days.
ChrisJBurns commentedon Aug 12, 2026
not stale
github-actions commentedon Sep 12, 2026
This issue has been automatically marked as stale because it has not had activity in the last 30 days.
cilindrox commentedon Oct 19, 2026
Getting hit by this too - willing to open a PR but I'd like to hear from the maintainers first
zirain commentedon Oct 20, 2026
If there's no way to detect the kubernetes version in controller, we should fallback.
Ryan02I5 commentedon Oct 30, 2026
Another data point: this also reproduces with plain Envoy Gateway (not only Envoy AI Gateway) on Kubernetes 1.35.
Description:
During a proxy rollout, kubelet emitted
FailedPreStopHookfor theenvoycontainer about 1s before the old pod was gone, and clients saw errors while the replacement started. This matches the race described above:shutdown-managerexited before theenvoypreStophttpGet :19002/shutdown/readyfinished, so the drain handshake did not complete.Repro steps:
strategy: Recreate, hostPort 80/443 on one node), defaultshutdown-managersidecar.kubectl -n envoy-gateway-system rollout restart deployment/<envoy-proxy-deployment>Environment:
v1.9.1distroless-v1.39.1v1.35.9+k3s1(K3s, arm64)Logs (Kubernetes events, UTC, from a single rollout):
External probing once per second saw 3 failed requests (07:50:17–07:50:19) before traffic recovered. With a single replica some gap is expected, but the failed preStop means the drain was cut short rather than completed.
On @zirain's question about version detection: on this cluster the server reports
v1.35.9+k3s1through the standard discovery/versionendpoint, so a semver check against1.33should work even for distributions with a+k3sbuild suffix. Happy to test a build or a PR here.mvanhorn commentedon Oct 31, 2026
shutdown-manager was rendered as a regular container, so kubelet could terminate it concurrently with Envoy. The preStop hook depends on shutdown-manager remaining available while connections drain.
Kubernetes 1.33+ API server versions now cause shutdown-manager to be rendered as an init container with restartPolicy Always, making it a native sidecar that outlives Envoy. Older, unavailable, or invalid versions retain the existing layout.
#10469