Repository navigation
Conversation
Health checking is only covered in passing today, inside the failover task and the health check logging task. Add a task that shows how to configure active health checks with BackendTrafficPolicy and check the result through the Envoy admin interface, how panic mode keeps a backend serving when its endpoints fail, the TCP and gRPC checkers, and passive health checking with outlier detection. Fixes envoyproxy#2418 Signed-off-by: Lakhan Yelgate <39281685+lyelgate@users.noreply.github.com>
An unexpected HTTP status marks an endpoint unhealthy right away; only timeouts, connection failures and retriableStatuses count toward unhealthyThreshold. The passive example needs alwaysEjectOneEndpoint to eject the quickstart backend's single endpoint, combined checks are still subject to panic mode, and readers without a LoadBalancer need GATEWAY_HOST pointed at the port-forward. Signed-off-by: Lakhan Yelgate <39281685+lyelgate@users.noreply.github.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3b2dd8fba4
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| An ejected endpoint is reported with the `failed_outlier_check` health flag. It stays ejected for `baseEjectionTime` | ||
| multiplied by the number of times it has been ejected, so an endpoint that keeps failing is ejected for longer. See [PassiveHealthCheck][] for every field and its default. |
There was a problem hiding this comment.
Because the merge patch above retains the successful active GET / check, Envoy's default successful_active_health_check_uneject_host=true clears a passive outlier ejection on the next successful probe. On the quickstart backend that probe runs every 3 seconds, so an endpoint ejected after five 5xx responses does not stay ejected for the stated 30 seconds (let alone the multiplied duration); remove the active check for this demonstration or document that active recovery can terminate the ejection early.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Right, the active check from earlier in the task stays in the policy, so a successful probe returns the endpoint to service before baseEjectionTime. The page now says the ejection can end early for that reason, in cb56044.
Envoy returns an ejected endpoint to service as soon as it passes an active health check, and the passive example keeps the active check from earlier in the task, so the ejection can end before baseEjectionTime. Signed-off-by: Lakhan Yelgate <39281685+lyelgate@users.noreply.github.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cb56044bab
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| return too many errors. This is Envoy's [Outlier Detection][]. | ||
|
|
||
| Health checking is configured in the `healthCheck` field of a [BackendTrafficPolicy][], which can target a [Gateway][], | ||
| an [HTTPRoute][], a [GRPCRoute][] or the other xRoute kinds. Each backendRef of a route has its own set of endpoints, |
There was a problem hiding this comment.
For a BackendTrafficPolicy targeting a UDPRoute, “the other xRoute kinds” promises functionality that the translator deliberately drops: the UDP branches in internal/gatewayapi/backendtrafficpolicy.go:1777-1794 and :2150-2165 copy only load-balancer/DNS settings, and the comment at :2340-2344 states that a health check never reaches a udp_proxy cluster. Such a policy can therefore be accepted while none of its endpoints are checked, so narrow this claim to route kinds whose IR carries HealthCheck or explicitly call out UDP as unsupported.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Right, ir.UDPRoute only carries load balancer and DNS settings, so a health check never reaches a UDP backend. The page now lists the supported route kinds (HTTPRoute, GRPCRoute, TCPRoute, TLSRoute) and says UDPRoute backends aren't health-checked, in 970e56b.
A UDPRoute backend's cluster is built from ir.UDPRoute, which only carries load balancer and DNS settings, so a health check set by a BackendTrafficPolicy never reaches it. List the route kinds that do support health checks instead of "the other xRoute kinds". Signed-off-by: Lakhan Yelgate <39281685+lyelgate@users.noreply.github.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 970e56ba7a
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| Health checking is configured in the `healthCheck` field of a [BackendTrafficPolicy][], which can target a [Gateway][], | ||
| an [HTTPRoute][], a [GRPCRoute][], a TCPRoute or a TLSRoute. Health checks are not applied to the backends of a |
There was a problem hiding this comment.
With a ListenerSet or MergeBackends deployment, this sentence incorrectly makes the target list look exhaustive: BackendTrafficPolicySpec also permits ListenerSet, Service, ServiceImport, and Backend targets (api/v1alpha1/backendtrafficpolicy_types.go:42-48), and the backend-target translation carries HealthCheck into non-UDP merged clusters (internal/gatewayapi/backendtrafficpolicy.go:641-652,2346-2374). Include those scopes with the MergeBackends caveat so users are not directed unnecessarily to per-route policies.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Right, the policy can also target a ListenerSet and, with MergeBackends, a Service, ServiceImport or Backend. The intro no longer lists the targets exhaustively; it says health checks apply to the backends of the resources the policy targets, with the UDPRoute exception, in 9d20329.
| ## Passive Health Checking | ||
|
|
||
| Passive health checking needs no probes: Envoy counts the errors returned by each endpoint while serving real requests. | ||
| Apply a policy that ejects an endpoint for 30 seconds after 5 consecutive `5xx` responses, checking every 3 seconds. |
There was a problem hiding this comment.
With consecutive5XxErrors configured as here, Envoy evaluates the counter inline on each upstream response and can eject the host immediately on the fifth 5xx; interval controls periodic outlier-analysis sweeps rather than batching this detector. Describing the policy as “checking every 3 seconds” therefore gives the wrong timing to users trying to observe the walkthrough; state the immediate trigger and explain the interval separately.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Agreed, the consecutive 5xx detector ejects as soon as the threshold is reached. The text now says so and describes interval as how often the periodic outlier analysis runs, which also returns endpoints to service, in 9d20329.
| An ejected endpoint is reported with the `failed_outlier_check` health flag. It stays ejected for `baseEjectionTime` | ||
| multiplied by the number of times it has been ejected, so an endpoint that keeps failing is ejected for longer. Because |
There was a problem hiding this comment.
After repeated ejections, Envoy uses min(base_ejection_time * ejection count, max_ejection_time), and the unset max_ejection_time defaults to 300 seconds. For this 30-second policy the duration stops growing after 300 seconds rather than always remaining the stated product, so qualify the multiplication rule with the cap to avoid misleading long-running tests.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Good point. Envoy Gateway doesn't expose max_ejection_time, so Envoy's 300 second default always applies. The page now says the ejection time grows only up to that maximum, in 9d20329.
✅ Deploy Preview for cerulean-figolla-1f9435 ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 46e06242ca
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| ## Active Health Checking | ||
|
|
||
| Apply a `BackendTrafficPolicy` that probes every endpoint of the `backend` HTTPRoute with an HTTP `GET /` request every | ||
| 3 seconds. An endpoint is marked unhealthy after 3 failed probes in a row, and healthy again after 1 successful probe. |
There was a problem hiding this comment.
For HTTP checks, api/v1alpha1/healthcheck_types.go:125-128 specifies that unhealthyThreshold is honored for statuses configured in retriableStatuses; an unexpected non-retriable status makes the host unhealthy immediately. The later patch expects 204 while this backend returns 200, and no retriable status is configured, so the endpoint transitions after the first probe rather than after three. Make 200 retriable for the simulation or describe the immediate transition.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Right, thanks: without retriableStatuses an unexpected status marks the endpoint unhealthy on the first probe. Fixed the intro to say the threshold applies to timeouts, connection failures and retriable statuses, and the walkthrough now says the next probe marks it unhealthy right away, in 3b2dd8f.
| always in panic mode while that endpoint is unhealthy. | ||
|
|
||
| ```shell | ||
| curl --verbose --header "Host: www.example.com" http://$GATEWAY_HOST/ |
There was a problem hiding this comment.
When users follow this page's "Without LoadBalancer Support" prerequisite, site/content/en/latest/boilerplates/testing.md:23-46 starts a proxy on localhost:8888 but never defines GATEWAY_HOST. This command therefore expands to http:///, so the panic-mode request and the subsequent 503 comparison cannot be exercised on kind or other clusters without an external load balancer. Provide the matching localhost request path or initialize the variable in that prerequisite branch.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Good point. Added a line before the request telling readers who followed the "Without LoadBalancer Support" prerequisite to export GATEWAY_HOST=localhost:8888, in 3b2dd8f.
|
|
||
| ```shell | ||
| kubectl patch backendtrafficpolicy/health-check --type merge \ | ||
| -p '{"spec":{"healthCheck":{"passive":{"consecutive5XxErrors":5,"interval":"3s","baseEjectionTime":"30s","maxEjectionPercent":50}}}}' |
There was a problem hiding this comment.
The task establishes above that the quickstart backend has one endpoint, so ejecting it would mean ejecting 100% of the cluster. With maxEjectionPercent set to 50 and alwaysEjectOneEndpoint omitted (its default is false), this policy cannot eject that endpoint and therefore cannot exhibit the claimed 30-second ejection or failed_outlier_check state after five 5xx responses. Use a 100% cap or enable alwaysEjectOneEndpoint for this single-endpoint example.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Correct, 50% of a single endpoint rounds down to zero. The example now sets alwaysEjectOneEndpoint: true and explains why, in 3b2dd8f. I checked it with egctl x translate: the cluster gets alwaysEjectOneHost: true.
| An ejected endpoint is reported with the `failed_outlier_check` health flag. It stays ejected for `baseEjectionTime` | ||
| multiplied by the number of times it has been ejected, so an endpoint that keeps failing is ejected for longer. See [PassiveHealthCheck][] for every field and its default. | ||
|
|
||
| Active and passive health checking can be combined in the same policy: an endpoint is only used when it passes both. |
There was a problem hiding this comment.
This flow restores the default panicThreshold before adding passive checking, so with the one-endpoint backend any failed check drops health below 50% and activates panic mode. As the page itself explains above, Envoy then ignores health status and can route to that endpoint, making the categorical claim that an endpoint is only used when it passes both checks false in this example. State the panic-mode exception or keep panic mode disabled when promising this behavior.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Agreed. It now says an endpoint is only considered healthy when it passes both, and that panic mode still applies when fewer than panicThreshold percent of endpoints are healthy, in 3b2dd8f.
BackendTrafficPolicy also targets ListenerSets and, with MergeBackends, Services, ServiceImports and Backends, so describe the targets without an exhaustive list. A consecutive 5xx ejection happens as soon as the threshold is reached, while interval sets how often the periodic outlier analysis runs. The ejection time grows with each ejection only up to Envoy's default max_ejection_time of 300 seconds. Signed-off-by: Lakhan Yelgate <39281685+lyelgate@users.noreply.github.com>
What this PR does / why we need it:
There's no task page for health checking today.
healthCheckonly appears in passing in the Failover task and the Health Check Logging task, so users have to work out the API from the reference.This PR adds
tasks/traffic/health-check.md, following the structure of the other BackendTrafficPolicy tasks (circuit breakers, retry), on top of the quickstart:backendroute, and how to see each endpoint's health flags through the Envoy admin/clustersendpoint (the same port-forward the backend utilization task uses).expectedStatuses: [204], which the echo backend never returns, to show/failed_active_hc. Then it explains the panic threshold, which surprises a lot of people: with the default 50%, a backend whose only endpoint is unhealthy keeps receiving traffic. SettingpanicThreshold: 0makes the same request fail with503 no healthy upstream.send/receive, gRPC withservice, and HTTP withexpectedStatusesandoverrides.port.consecutive5XxErrors,interval,baseEjectionTimeandmaxEjectionPercent, and how the ejection time grows.It links to the
ActiveHealthCheckandPassiveHealthCheckAPI reference for the full field list, and to the existing health check logging task.Testing:
expectedStatusesandpanicThresholdpatches, TCP, gRPC, HTTP withoverrides.port, and passive) was run throughegctl x translateagainst the quickstart resources. Each one isAccepted, and the generatedhttproute/default/backend/rule/0cluster has the expectedhealthChecks(httpHealthCheckwith expected statuses[200,201)or[204,205),tcpHealthCheck,grpcHealthCheck),commonLbConfig.healthyPanicThresholdof 0 forpanicThreshold: 0, andoutlierDetectionfor the passive example.../../../api/...convention of the other task pages. The API anchors#activehealthcheckand#passivehealthcheckexist inextension_types.md.codespellis clean, and the enabled markdownlint rules (MD001, MD009, MD037, MD038) were checked by hand.Which issue(s) this PR fixes:
Fixes #2418
PR Checklist
git commit -s). See DCO: Sign your work./api.make generate gen-check,make lint, and the unit-test/coverage build pass. (Flaky e2e failures are not considered breakages, butgen-check,lint, and coverage MUST pass.) N/A: docs only.release-notes/current/breaking_changes/.