Skip to content

feat(drain): add drainStrategy to EnvoyProxy shutdown config - #10295

Open
wbpcode wants to merge 8 commits into
envoyproxy:mainfrom
wbpcode:drain-strategy
Open

wbpcode wants to merge 8 commits into
envoyproxy:mainfrom
wbpcode:drain-strategy

Conversation

@wbpcode

@wbpcode wbpcode commented Oct 16, 2026

Copy link
Copy Markdown
Member

What this PR does / why we need it:

Envoy Gateway always starts Envoy with --drain-strategy immediate, and the flag cannot be overridden through extraArgs because Envoy rejects a duplicated flag.

This adds an optional drainStrategy field (Gradual or Immediate) to EnvoyProxy.spec.shutdown, which maps to Envoy's --drain-strategy. When unset, it defaults to Immediate, so existing behavior is unchanged. No +kubebuilder:default is set, so the default is not persisted into stored objects.

Which issue(s) this PR fixes:

Fixes #


PR Checklist

  • Authorship & ownership: Coding agents / AI assistants are welcome, but I have reviewed every change, understand how and why it works, can explain and maintain it, and take full responsibility for this PR. I have not submitted generated output I do not understand.
  • DCO: All commits are signed off (git commit -s). See DCO: Sign your work.
  • API agreed first: If this PR contains API changes (changes under /api), the API was discussed and agreed before the implementation. The API change can be in a separate PR, or in the same PR, but the API must be agreed before implementation. N/A if this PR does not contain API changes.
  • Required checks pass: make generate gen-check, make lint, and the unit-test/coverage build pass. (Flaky e2e failures are not considered breakages, but gen-check, lint, and coverage MUST pass.)
  • Tests added/updated: New/changed code is covered by appropriate tests. N/A if this PR does not contain code changes.
  • Docs: User-facing changes update the docs, either in this PR or a follow-up PR. N/A if this PR does not contain user-facing changes.
  • Release notes: For any non-trivial change, added a release-note fragment under release-notes/current/<section>/<pr-number>-<slug>.md (see release-notes/current/README.md for sections and naming). N/A if this PR does not contain non-trivial changes.
  • Generated files committed: Ran make gen-check and committed the result if API/helm charts/modules changed.
  • Scope & compatibility: The PR is reasonably scoped (no unrelated changes) and preserves backward compatibility, or any breaking change is called out above and documented in release-notes/current/breaking_changes/.
  • Codex review: Requested a Codex review and addressed all of its comments.
  • Copilot review: Requested a Copilot review and addressed all of its comments.

@wbpcode
wbpcode requested a review from a team as a code owner October 16, 2026 03:22
@netlify

netlify Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

✅ Deploy Preview for cerulean-figolla-1f9435 ready!

Name Link
🔨 Latest commit b3b26bc
🔍 Latest deploy log https://app.netlify.com/projects/cerulean-figolla-1f9435/deploys/6ae531350a9585f5a08b2b8f
😎 Deploy Preview https://deploy-preview-10295--cerulean-figolla-1f9435.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

Signed-off-by: wbpcode <wbphub@gmail.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-31T00:11:37.014702Z b3b26bc New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@zirain zirain added this to the v1.10.0-rc.1 Release milestone Oct 16, 2026
@codecov

codecov Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 87.50000% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.21%. Comparing base (79c6478) to head (b3b26bc).

Files with missing lines Patch % Lines
internal/cmd/envoy/shutdown_manager.go 0.00% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #10295      +/-   ##
==========================================
+ Coverage   82.19%   82.21%   +0.01%     
==========================================
  Files         277      277              
  Lines       44014    44027      +13     
==========================================
+ Hits        36178    36196      +18     
+ Misses       7835     7830       -5     
  Partials        1        1              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d5ec7a23f3

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment on lines +31 to +32
// Defaults to immediate when unset.
return "immediate"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Reject invalid strategies outside CRD admission

When this setting is supplied through the file-loaded EnvoyGateway.envoyProxy defaults rather than an EnvoyProxy CRD, the kubebuilder enum marker is never enforced and validateEnvoyProxySpec does not validate Shutdown; an unsupported value such as Graduel therefore reaches this catch-all and is emitted as --drain-strategy immediate. That configuration passes startup validation but silently defeats the requested gradual drain, so validate the enum in the non-CRD path or return an error here rather than treating every non-Gradual value as the unset default.

Useful? React with 👍 / 👎.

@arkodg

arkodg commented Oct 18, 2026

Copy link
Copy Markdown
Contributor

thanks for adding this !

  • do we need to set drain_type: MODIFY_ONLY, and have the shutdown manager explicitly call /drain_listeners?graceful&skip_exit when strategy is set to Gradual ?

@wbpcode

wbpcode commented Oct 19, 2026

Copy link
Copy Markdown
Member Author
  • do we need to set drain_type: MODIFY_ONLY, and have the shutdown manager explicitly call /drain_listeners?graceful&skip_exit when strategy is set to Gradual ?

I agree to set the drain_type: MODIFY_ONLY, so the healcheck/fail won't affect existing traffic. And agree to use the graceful when /drain_listeners.

But all these may needn't to be part of this PR because they could be relatively independent changes?

@zirain
zirain requested a review from arkodg October 26, 2026 02:01
@arkodg

arkodg commented Oct 26, 2026

Copy link
Copy Markdown
Contributor
  • do we need to set drain_type: MODIFY_ONLY, and have the shutdown manager explicitly call /drain_listeners?graceful&skip_exit when strategy is set to Gradual ?

I agree to set the drain_type: MODIFY_ONLY, so the healcheck/fail won't affect existing traffic. And agree to use the graceful when /drain_listeners.

But all these may needn't to be part of this PR because they could be relatively independent changes?

without the changes in the shutdown code logic, will TCP listeners gracefully drain or immediately with drainStrategy: Gradual in the API field

@zirain zirain added the area/api API-related issues label Oct 27, 2026
@wbpcode

wbpcode commented Oct 27, 2026 •

Copy link
Copy Markdown
Member Author

without the changes in the shutdown code logic, will TCP listeners gracefully drain or immediately with drainStrategy: Gradual in the API field

gracefully, drainStrategy: Gradual will be respected. The graceful in the drain_listeners only decide whether to accept new connections when draining existing connections. That's why say they actually could be independent changes.

I can create related PRs also anyway. :)

@wbpcode

wbpcode commented Oct 28, 2026 •

Copy link
Copy Markdown
Member Author

without the changes in the shutdown code logic, will TCP listeners gracefully drain or immediately with drainStrategy: Gradual in the API field

gracefully, drainStrategy: Gradual will be respected. The graceful in the drain_listeners only decide whether to accept new connections when draining existing connections. That's why say they actually could be independent changes.

I can create related PRs also anyway. :)

I see why. The EG use the healthcheck/fail to shutdown. I initially thought the shutdown manager only use the /drain_listeners.

Then, yeah, if the healthcheck/fail is part of our shutdown manager, we need to change the drain_type to MODIFY_ONLY and always call the /drain_listeners?graceful&skip_exit.
Then at the immediate strategy, nothing will be changed because unconditional /drain_listeners will notify all connections to drain now, and the immediate will be applied. Same effect with previous healthcheck/fail + drain_type: MODIFY_ONLY.
And at the gradual strategy, the graceful draining will be applied and won't be affected by the healthcheck/fail.

Good catch. Thanks.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4a1ec08c2a

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread internal/xds/translator/listener.go Outdated
MaxConnectionsToAcceptPerSocketEvent: maxAcceptPerSocketEvent,
// Drain on listener modification/removal and on the drain sequence started by the shutdown
// manager, but not on /healthcheck/fail, which would bypass the configured drain strategy.
DrainType: listenerv3.Listener_MODIFY_ONLY,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Keep old shutdown sidecars able to drain

During a Kubernetes rolling upgrade from a previous Envoy Gateway release (or with a pinned older shutdownManager.image), the old pods still run the prior Shutdown path, which at the default zero healthCheckFailureDelay only posts /healthcheck/fail. As soon as the new control plane sends this MODIFY_ONLY listener to those pods, that endpoint explicitly no longer starts their listener drain, so long-lived connections can remain open until the old manager hits drainTimeout and the rollout kills them; preserve the old manager's drain behavior during version skew or gate this listener mode until every sidecar is upgraded.

Useful? React with 👍 / 👎.

Signed-off-by: wbpcode <wbphub@gmail.com>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 313b85ea71

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread internal/xds/translator/listener.go Outdated
MaxConnectionsToAcceptPerSocketEvent: maxAcceptPerSocketEvent,
// Drain on listener modification/removal and on the drain sequence started by the shutdown
// manager, but not on /healthcheck/fail, which would bypass the configured drain strategy.
DrainType: listenerv3.Listener_MODIFY_ONLY,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Avoid draining every listener during control-plane upgrades

When existing proxies receive the first LDS snapshot from the upgraded control plane, unconditionally serializing MODIFY_ONLY changes every application listener even if drainStrategy is unset. As the adjacent comment notes, Envoy drains listeners on modification/removal, and those existing pods were launched with --drain-strategy immediate, so this update starts draining connections across the whole fleet at once rather than through the Deployment's rolling termination. Gate this xDS change on an opted-in strategy/version or stage the migration so default upgrades do not trigger a fleet-wide connection drain.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah. Control plane upgrading will result in a draining of listener because we changed the MODIFY_ONLY changes.

🤔

@wbpcode wbpcode Oct 29, 2026 •

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now, EG will only change the drain_type when the drain-strategy is set to gradual.

@zhaohuabing zhaohuabing removed this from the v1.10.0-rc.1 Release milestone Oct 29, 2026
Signed-off-by: wbpcode <wbphub@gmail.com>

@arkodg arkodg left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM thanks

@wbpcode

wbpcode commented Oct 31, 2026

Copy link
Copy Markdown
Member Author

/retest

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/api API-related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants