Repository navigation
test(e2e): apply gRPC health check routes once the backend serves - #10405
Merged
Merged
Conversation
✅ Deploy Preview for cerulean-figolla-1f9435 ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #10405 +/- ##
==========================================
- Coverage 82.20% 82.18% -0.02%
==========================================
Files 277 277
Lines 44014 44014
==========================================
- Hits 36181 36173 -8
- Misses 7832 7840 +8
Partials 1 1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
9 of 11 tasks
nobbs
force-pushed
the
test/grpc-health-check-flake
branch
from
October 27, 2026 07:18
2903c77 to
599a952
Compare
zirain
approved these changes
Oct 28, 2026
nobbs
force-pushed
the
test/grpc-health-check-flake
branch
2 times, most recently
from
October 29, 2026 07:45
0e5e4e6 to
7fd319e
Compare
BackendHealthCheckActiveGRPC applied grpc-health-backend together with its routes and BackendTrafficPolicies. EG publishes the endpoint of a pod that is not Ready yet as draining, so Envoy received the health checked clusters, and ran each first health check, while the gRPC server was still starting. That check failed with a network error, and Envoy checks a cluster that has seen no traffic again only after no_traffic_interval (60s), past the 30s the test waits for a success. Give the backend a gRPC readiness probe, wait for it, and only then apply the policies and routes, policies first: a route processed before its policy yields a cluster without the health check, and the update that adds it waits for EDS until initial_fetch_timeout (15s). Signed-off-by: Alexej Disterhoft <alexej@disterhoft.de>
nobbs
force-pushed
the
test/grpc-health-check-flake
branch
from
October 29, 2026 13:26
7fd319e to
ff9c694
Compare
kkk777-7
approved these changes
Oct 30, 2026
Member
|
LGTM, thanks! |
Member
|
/retest |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
TestE2E/BackendHealthCheckActiveGRPC/explicit_hostname_is_used_as_authority, added in #10093, is flaky. It failed six times since 19 Oct: four times on #10093, once on its merge commit to main, and once on #10402. Each time, the health check success counter of the first cluster the test checks stayed at 0 for the whole 30s wait.The test applied grpc-health-backend together with its routes and BackendTrafficPolicies:
This PR applies the health checked resources only once the backend serves:
The BTPs come before the routes because a route processed before its policy yields a cluster without the health check. In my reproduction, the update that then added the health check waited for EDS until
initial_fetch_timeout(15s), half the test's budget. Waiting first and applying afterwards followsoidc.go, which applies its SecurityPolicy once Keycloak is up.Evidence, from a kind cluster on main (4e77313) that applies the manifests in a loop and reads Envoy's admin API every 0.1s. An iteration fails when the explicit-hostname cluster has no health check success within 30s of the backend turning Ready. A 10m CPU limit on the backend slows its startup to make the race likely:
network_failure, and itsattemptstays at 1.interval: 5s.Not covered locally: the gRPC requests through the Gateway, since the macOS host can't reach the Gateway's address. CI runs them.
Merge danger: a two-way door that changes only the test. The blast radius is this one e2e test.
Two observations outside this test: EG doesn't set
no_traffic_interval, so a cluster without traffic recovers from a failed first check only after 60s, whatever interval the BTP sets. And adding a health check to an existing cluster took 15s to apply in my reproduction, waiting for EDS.Which issue(s) this PR fixes:
No issue filed. The flake came in with #10093.
PR Checklist
git commit -s). See DCO: Sign your work./api), the API was discussed and agreed before the implementation. The API change can be in a separate PR, or in the same PR, but the API must be agreed before implementation. N/A if this PR does not contain API changes. (N/A: no API changes.)make generate gen-check,make lint, and the unit-test/coverage build pass. (Flaky e2e failures are not considered breakages, butgen-check,lint, and coverage MUST pass.) (make lintpasses locally. The PR changes only e2e test files, sogen-checkand coverage are left to CI.)release-notes/current/<section>/<pr-number>-<slug>.md(seerelease-notes/current/README.mdfor sections and naming). N/A if this PR does not contain non-trivial changes. (N/A: test-only.)make gen-checkand committed the result if API/helm charts/modules changed. (N/A: nothing generated changed.)release-notes/current/breaking_changes/.