Skip to content

fix(translator): drop backendRefs without ready endpoints when other backends are available - #10180

Open
kkk777-7 wants to merge 9 commits into
envoyproxy:mainfrom
kkk777-7:skip-no-endpoints-backendrefs
Open

kkk777-7 wants to merge 9 commits into
envoyproxy:mainfrom
kkk777-7:skip-no-endpoints-backendrefs

Conversation

@kkk777-7

@kkk777-7 kkk777-7 commented Oct 4, 2026 •

Copy link
Copy Markdown
Member

What this PR does / why we need it:

When an HTTPRoute/GRPCRoute rule has multiple backendRefs and some of them have no ready endpoints, their share of traffic is currently sent to invalid-backend-cluster and returns 503, even though healthy backends exist in the same rule.

This PR adds the DropBackendRefsWithoutEndpoints runtime flag (disabled by default). When enabled, backendRefs without ready endpoints are dropped from the rule's destination when at least one other backendRef is available, so the remaining backends receive all the traffic per their relative weights.

This is opt-in because redistributing the traffic may send more traffic than intended to the remaining backends (e.g. a canary when the stable backend temporarily has no ready endpoints). We can consider enabling it by default once the behavior is stable.

  • The BackendsAvailable=False (EndpointsNotFound) condition is still reported.
  • The 503 direct response is kept when all backendRefs in the rule have no ready endpoints.
Rule backendRefs Flag disabled (default) Flag enabled
All have no endpoints gatewayapi: 503 direct response Unchanged
Valid + no endpoints xds: no-endpoints share routed to invalid-backend-cluster (503), rule split into per-backend weighted clusters No-endpoints backendRefs dropped; 100% to valid backends, rule stays a single cluster
Valid + invalid + no endpoints xds: invalid + no-endpoints share routed to invalid-backend-cluster (503) Unchanged
Valid + invalid xds: invalid share routed to invalid-backend-cluster (500) Unchanged

Additionally, this PR fixes the following regardless of the flag:

  • GRPCRoute: mark backendRefs that failed to be processed as invalid.
  • GRPCRoute: return 503 (mapped to gRPC UNAVAILABLE) instead of 500 for invalid GRPCBackendRefs in weighted clusters, as required by the Gateway API spec.

Which issue(s) this PR fixes:

Fixes #8031


PR Checklist

  • Authorship & ownership: Coding agents / AI assistants are welcome, but I have reviewed every change, understand how and why it works, can explain and maintain it, and take full responsibility for this PR. I have not submitted generated output I do not understand.
  • DCO: All commits are signed off (git commit -s). See DCO: Sign your work.
  • API agreed first: If this PR contains API changes (changes under /api), the API was discussed and agreed before the implementation. The API change can be in a separate PR, or in the same PR, but the API must be agreed before implementation. N/A if this PR does not contain API changes.
  • Required checks pass: make generate gen-check, make lint, and the unit-test/coverage build pass. (Flaky e2e failures are not considered breakages, but gen-check, lint, and coverage MUST pass.)
  • Tests added/updated: New/changed code is covered by appropriate tests. N/A if this PR does not contain code changes.
  • Docs: User-facing changes update the docs, either in this PR or a follow-up PR. N/A if this PR does not contain user-facing changes.
  • Release notes: For any non-trivial change, added a release-note fragment under release-notes/current/<section>/<pr-number>-<slug>.md (see release-notes/current/README.md for sections and naming). N/A if this PR does not contain non-trivial changes.
  • Generated files committed: Ran make gen-check and committed the result if API/helm charts/modules changed.
  • Scope & compatibility: The PR is reasonably scoped (no unrelated changes) and preserves backward compatibility, or any breaking change is called out above and documented in release-notes/current/breaking_changes/.
  • Codex review: Requested a Codex review and addressed all of its comments.
  • Copilot review: Requested a Copilot review and addressed all of its comments.

…backends are available

Signed-off-by: kkk777-7 <kota.kimura0725@gmail.com>
@netlify

netlify Bot commented Oct 4, 2026 •

Copy link
Copy Markdown

✅ Deploy Preview for cerulean-figolla-1f9435 ready!

Name Link
🔨 Latest commit c4f081b
🔍 Latest deploy log https://app.netlify.com/projects/cerulean-figolla-1f9435/deploys/6ae432c56e4997bbece34f3d
😎 Deploy Preview https://deploy-preview-10180--cerulean-figolla-1f9435.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

Signed-off-by: kkk777-7 <kota.kimura0725@gmail.com>
@kkk777-7 kkk777-7 added this to the v1.10.0-rc.1 Release milestone Oct 4, 2026
@codecov

codecov Bot commented Oct 4, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.21%. Comparing base (b02cef4) to head (c4f081b).

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #10180      +/-   ##
==========================================
+ Coverage   82.20%   82.21%   +0.01%     
==========================================
  Files         277      277              
  Lines       44014    44026      +12     
==========================================
+ Hits        36181    36197      +16     
+ Misses       7832     7828       -4     
  Partials        1        1              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@kkk777-7

kkk777-7 commented Oct 4, 2026

Copy link
Copy Markdown
Member Author

@codex

@kkk777-7 kkk777-7 mentioned this pull request Oct 4, 2026
9 of 11 tasks
@arkodg

arkodg commented Oct 5, 2026

Copy link
Copy Markdown
Contributor

@codex review

1 similar comment
@zirain

zirain commented Oct 7, 2026

Copy link
Copy Markdown
Member

@codex review

@kkk777-7

kkk777-7 commented Oct 9, 2026

Copy link
Copy Markdown
Member Author

@codex

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-30T06:07:54.152522Z c4f081b New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: df32bd78fd

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread internal/gatewayapi/route.go Resolved
Comment thread internal/gatewayapi/route.go Resolved
Signed-off-by: kkk777-7 <kota.kimura0725@gmail.com>
@kkk777-7

Copy link
Copy Markdown
Member Author

@codex

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: af32c80dba

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread internal/gatewayapi/route.go Resolved
@kkk777-7

Copy link
Copy Markdown
Member Author

@codex

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: af32c80dba

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread internal/gatewayapi/route.go Resolved
@kkk777-7

Copy link
Copy Markdown
Member Author

@codex

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Breezy!

Reviewed commit: af32c80dba

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@arkodg

arkodg commented Oct 11, 2026

Copy link
Copy Markdown
Contributor

hey @envoyproxy/gateway-maintainers @envoyproxy/gateway-reviewers should this be made opt in, else we might end up sending too much traffic to the canary which may not be the intention

@kkk777-7

Copy link
Copy Markdown
Member Author

hey @envoyproxy/gateway-maintainers @envoyproxy/gateway-reviewers should this be made opt in, else we might end up sending too much traffic to the canary which may not be the intention

thanks, sounds reasonable to me, I'll update.

… a runtime flag

Signed-off-by: kkk777-7 <kota.kimura0725@gmail.com>
@kkk777-7

Copy link
Copy Markdown
Member Author

@codex

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d61d02f0f8

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread internal/gatewayapi/route.go Resolved
@zirain

zirain commented Oct 22, 2026

Copy link
Copy Markdown
Member

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Bravo.

Reviewed commit: d61d02f0f8

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

zirain
zirain previously approved these changes Oct 22, 2026
@kkk777-7

Copy link
Copy Markdown
Member Author

/retest

@github-actions github-actions Bot added the area/api API-related issues label Oct 30, 2026
@kkk777-7

Copy link
Copy Markdown
Member Author

/retest

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/api API-related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Rebalancing traffic from failed backend to healthy one in httproute

3 participants