From 46e06242ca256db4091713ee44306819484ed3fe Mon Sep 17 00:00:00 2001 From: Lakhan Yelgate <39281685+lyelgate@users.noreply.github.com> Date: Sun, 1 Nov 2026 14:19:20 +0530 Subject: [PATCH 1/5] docs: add a task for active and passive health checking Health checking is only covered in passing today, inside the failover task and the health check logging task. Add a task that shows how to configure active health checks with BackendTrafficPolicy and check the result through the Envoy admin interface, how panic mode keeps a backend serving when its endpoints fail, the TCP and gRPC checkers, and passive health checking with outlier detection. Fixes #2418 Signed-off-by: Lakhan Yelgate <39281685+lyelgate@users.noreply.github.com> --- .../en/latest/tasks/traffic/health-check.md | 224 ++++++++++++++++++ 1 file changed, 224 insertions(+) create mode 100644 site/content/en/latest/tasks/traffic/health-check.md diff --git a/site/content/en/latest/tasks/traffic/health-check.md b/site/content/en/latest/tasks/traffic/health-check.md new file mode 100644 index 0000000000..71ec16cebb --- /dev/null +++ b/site/content/en/latest/tasks/traffic/health-check.md @@ -0,0 +1,224 @@ +--- +title: "Health Checking" +--- + +Envoy Gateway can check the health of the endpoints behind a backend, so that Envoy only sends requests to endpoints +that are able to serve them. Two kinds of health checking are supported, and they can be used together: + +- **Active health checking**: Envoy periodically sends a probe (HTTP, TCP or gRPC) to every endpoint and marks an + endpoint unhealthy once enough consecutive probes fail. See [Envoy Active Health Checking][]. +- **Passive health checking**: Envoy watches the responses of real requests and temporarily ejects endpoints that + return too many errors. This is Envoy's [Outlier Detection][]. + +Health checking is configured in the `healthCheck` field of a [BackendTrafficPolicy][], which can target a [Gateway][], +an [HTTPRoute][], a [GRPCRoute][] or the other xRoute kinds. Each backendRef of a route has its own set of endpoints, +and each endpoint is checked independently. + +## Prerequisites + +{{< boilerplate prerequisites >}} + +## Active Health Checking + +Apply a `BackendTrafficPolicy` that probes every endpoint of the `backend` HTTPRoute with an HTTP `GET /` request every +3 seconds. An endpoint is marked unhealthy after 3 failed probes in a row, and healthy again after 1 successful probe. + +{{< tabpane text=true >}} +{{% tab header="Apply from stdin" %}} + +```shell +cat <}} + +Check that the policy was accepted: + +```shell +kubectl get backendtrafficpolicy/health-check -o yaml +``` + +The health of each endpoint is visible through the Envoy admin interface: + +```shell +ENVOY_POD=$(kubectl get pods -n envoy-gateway-system -l gateway.envoyproxy.io/owning-gateway-name=eg -o jsonpath='{.items[0].metadata.name}') +kubectl -n envoy-gateway-system port-forward pod/${ENVOY_POD} 19000:19000 & +curl -s localhost:19000/clusters | grep "httproute/default/backend" | grep health_flags +``` + +The quickstart backend answers `GET /` with a `200`, so the endpoint is reported as healthy: + +```console +httproute/default/backend/rule/0::10.244.0.11:3000::health_flags::healthy +``` + +### Simulate a failing endpoint + +By default, only a `200` response counts as healthy. To see what happens when an endpoint fails its health check, +expect a `204` instead, which the quickstart backend never returns: + +```shell +kubectl patch backendtrafficpolicy/health-check --type merge \ + -p '{"spec":{"healthCheck":{"active":{"http":{"expectedStatuses":[204]}}}}}' +``` + +After three probes, the endpoint is marked as failing the active health check: + +```console +httproute/default/backend/rule/0::10.244.0.11:3000::health_flags::/failed_active_hc +``` + +Requests to the backend still succeed, because of Envoy's [panic threshold][Panic Threshold]: when fewer than 50% of a +backend's endpoints are healthy, Envoy ignores health status and balances across all of its endpoints, so that a +failing health check can't take down a whole backend at once. The quickstart backend has a single endpoint, so it is +always in panic mode while that endpoint is unhealthy. + +```shell +curl --verbose --header "Host: www.example.com" http://$GATEWAY_HOST/ +``` + +To stop sending requests to unhealthy endpoints even when all of them are unhealthy, set `panicThreshold` to `0`: + +```shell +kubectl patch backendtrafficpolicy/health-check --type merge \ + -p '{"spec":{"healthCheck":{"panicThreshold":0}}}' +``` + +The same request now fails with `503 Service Unavailable` and the body `no healthy upstream`. + +Restore the expected statuses and the default panic threshold before continuing: + +```shell +kubectl patch backendtrafficpolicy/health-check --type json \ + -p '[{"op":"remove","path":"/spec/healthCheck/active/http/expectedStatuses"},{"op":"remove","path":"/spec/healthCheck/panicThreshold"}]' +``` + +### Other active health checkers + +The `type` field selects the health checker, and the matching field configures it. These snippets replace the +`healthCheck` block of the policy above. + +A TCP health checker that only checks that a connection can be opened, or, with `send` and `receive`, that the endpoint +answers a payload with the expected bytes: + +```yaml + healthCheck: + active: + type: TCP + tcp: + send: + type: Text + text: ping + receive: + type: Text + text: pong +``` + +A gRPC health checker that uses the [gRPC health checking protocol][], optionally for a specific service: + +```yaml + healthCheck: + active: + type: GRPC + grpc: + service: helloworld.Greeter +``` + +The HTTP health checker can also send a different `method`, `hostname` or `requestBody`, accept a list of +`expectedStatuses`, and match the response body with `expectedResponse`. When the health check is served on a different +port than the application, set `overrides.port`: + +```yaml + healthCheck: + active: + type: HTTP + http: + path: /healthz + expectedStatuses: + - 200 + - 204 + overrides: + port: 9090 +``` + +See [ActiveHealthCheck][] for every field and its default. To log the result of each probe, see +[Health Check Logging](../../observability/proxy-health-check-log). + +## Passive Health Checking + +Passive health checking needs no probes: Envoy counts the errors returned by each endpoint while serving real requests. +Apply a policy that ejects an endpoint for 30 seconds after 5 consecutive `5xx` responses, checking every 3 seconds, and +never ejects more than half of the endpoints: + +```shell +kubectl patch backendtrafficpolicy/health-check --type merge \ + -p '{"spec":{"healthCheck":{"passive":{"consecutive5XxErrors":5,"interval":"3s","baseEjectionTime":"30s","maxEjectionPercent":50}}}}' +``` + +An ejected endpoint is reported with the `failed_outlier_check` health flag. It stays ejected for `baseEjectionTime` +multiplied by the number of times it has been ejected, so an endpoint that keeps failing is ejected for longer. See [PassiveHealthCheck][] for every field and its default. + +Active and passive health checking can be combined in the same policy: an endpoint is only used when it passes both. + +## Clean-Up + +```shell +kubectl delete backendtrafficpolicy/health-check +``` + +[Envoy Active Health Checking]: https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/upstream/health_checking +[Outlier Detection]: https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/upstream/outlier +[Panic Threshold]: https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/upstream/load_balancing/panic_threshold +[gRPC health checking protocol]: https://github.com/grpc/grpc/blob/master/doc/health-checking.md +[BackendTrafficPolicy]: ../../../api/extension_types#backendtrafficpolicy +[ActiveHealthCheck]: ../../../api/extension_types#activehealthcheck +[PassiveHealthCheck]: ../../../api/extension_types#passivehealthcheck +[Gateway]: https://gateway-api.sigs.k8s.io/reference/api-types/gateway/ +[HTTPRoute]: https://gateway-api.sigs.k8s.io/reference/api-types/httproute/ +[GRPCRoute]: https://gateway-api.sigs.k8s.io/reference/api-types/grpcroute/ From 3b2dd8fba450d7572f924c1a497fcf073f3b0de7 Mon Sep 17 00:00:00 2001 From: Lakhan Yelgate <39281685+lyelgate@users.noreply.github.com> Date: Sun, 1 Nov 2026 14:41:47 +0530 Subject: [PATCH 2/5] docs: correct health check thresholds, ejection and panic mode notes An unexpected HTTP status marks an endpoint unhealthy right away; only timeouts, connection failures and retriableStatuses count toward unhealthyThreshold. The passive example needs alwaysEjectOneEndpoint to eject the quickstart backend's single endpoint, combined checks are still subject to panic mode, and readers without a LoadBalancer need GATEWAY_HOST pointed at the port-forward. Signed-off-by: Lakhan Yelgate <39281685+lyelgate@users.noreply.github.com> --- .../en/latest/tasks/traffic/health-check.md | 20 +++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-) diff --git a/site/content/en/latest/tasks/traffic/health-check.md b/site/content/en/latest/tasks/traffic/health-check.md index 71ec16cebb..72ddbeb428 100644 --- a/site/content/en/latest/tasks/traffic/health-check.md +++ b/site/content/en/latest/tasks/traffic/health-check.md @@ -21,7 +21,9 @@ and each endpoint is checked independently. ## Active Health Checking Apply a `BackendTrafficPolicy` that probes every endpoint of the `backend` HTTPRoute with an HTTP `GET /` request every -3 seconds. An endpoint is marked unhealthy after 3 failed probes in a row, and healthy again after 1 successful probe. +3 seconds. An endpoint is marked unhealthy after 3 probes in a row time out or fail to connect, and healthy again after +1 successful probe. A response with an unexpected status marks the endpoint unhealthy right away, unless that status is +listed in `retriableStatuses`, in which case it counts toward `unhealthyThreshold` like a timeout. {{< tabpane text=true >}} {{% tab header="Apply from stdin" %}} @@ -108,7 +110,8 @@ kubectl patch backendtrafficpolicy/health-check --type merge \ -p '{"spec":{"healthCheck":{"active":{"http":{"expectedStatuses":[204]}}}}}' ``` -After three probes, the endpoint is marked as failing the active health check: +The next probe gets a `200`, which is neither expected nor retriable, so the endpoint is marked as failing the active +health check right away: ```console httproute/default/backend/rule/0::10.244.0.11:3000::health_flags::/failed_active_hc @@ -119,6 +122,8 @@ backend's endpoints are healthy, Envoy ignores health status and balances across failing health check can't take down a whole backend at once. The quickstart backend has a single endpoint, so it is always in panic mode while that endpoint is unhealthy. +If you followed the "Without LoadBalancer Support" prerequisite, run `export GATEWAY_HOST=localhost:8888` first. + ```shell curl --verbose --header "Host: www.example.com" http://$GATEWAY_HOST/ ``` @@ -193,18 +198,21 @@ See [ActiveHealthCheck][] for every field and its default. To log the result of ## Passive Health Checking Passive health checking needs no probes: Envoy counts the errors returned by each endpoint while serving real requests. -Apply a policy that ejects an endpoint for 30 seconds after 5 consecutive `5xx` responses, checking every 3 seconds, and -never ejects more than half of the endpoints: +Apply a policy that ejects an endpoint for 30 seconds after 5 consecutive `5xx` responses, checking every 3 seconds. +It never ejects more than half of the endpoints, but `alwaysEjectOneEndpoint` still allows one endpoint to be ejected +when that limit rounds down to zero, as it does for the single endpoint of the quickstart backend: ```shell kubectl patch backendtrafficpolicy/health-check --type merge \ - -p '{"spec":{"healthCheck":{"passive":{"consecutive5XxErrors":5,"interval":"3s","baseEjectionTime":"30s","maxEjectionPercent":50}}}}' + -p '{"spec":{"healthCheck":{"passive":{"consecutive5XxErrors":5,"interval":"3s","baseEjectionTime":"30s","maxEjectionPercent":50,"alwaysEjectOneEndpoint":true}}}}' ``` An ejected endpoint is reported with the `failed_outlier_check` health flag. It stays ejected for `baseEjectionTime` multiplied by the number of times it has been ejected, so an endpoint that keeps failing is ejected for longer. See [PassiveHealthCheck][] for every field and its default. -Active and passive health checking can be combined in the same policy: an endpoint is only used when it passes both. +Active and passive health checking can be combined in the same policy: an endpoint is only considered healthy when it +passes both. Panic mode still applies, so when fewer than `panicThreshold` percent of the endpoints are healthy, Envoy +sends requests to all of them regardless. ## Clean-Up From cb56044bab440cdb2eb11118622cb0c41b59bdac Mon Sep 17 00:00:00 2001 From: Lakhan Yelgate <39281685+lyelgate@users.noreply.github.com> Date: Sun, 1 Nov 2026 14:52:11 +0530 Subject: [PATCH 3/5] docs: note that active health checks can end a passive ejection early Envoy returns an ejected endpoint to service as soon as it passes an active health check, and the passive example keeps the active check from earlier in the task, so the ejection can end before baseEjectionTime. Signed-off-by: Lakhan Yelgate <39281685+lyelgate@users.noreply.github.com> --- site/content/en/latest/tasks/traffic/health-check.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/site/content/en/latest/tasks/traffic/health-check.md b/site/content/en/latest/tasks/traffic/health-check.md index 72ddbeb428..419d9852e6 100644 --- a/site/content/en/latest/tasks/traffic/health-check.md +++ b/site/content/en/latest/tasks/traffic/health-check.md @@ -208,7 +208,9 @@ kubectl patch backendtrafficpolicy/health-check --type merge \ ``` An ejected endpoint is reported with the `failed_outlier_check` health flag. It stays ejected for `baseEjectionTime` -multiplied by the number of times it has been ejected, so an endpoint that keeps failing is ejected for longer. See [PassiveHealthCheck][] for every field and its default. +multiplied by the number of times it has been ejected, so an endpoint that keeps failing is ejected for longer. Because +this policy also has an active health check, an ejection can end early: Envoy returns an ejected endpoint to service as +soon as it passes an active health check. See [PassiveHealthCheck][] for every field and its default. Active and passive health checking can be combined in the same policy: an endpoint is only considered healthy when it passes both. Panic mode still applies, so when fewer than `panicThreshold` percent of the endpoints are healthy, Envoy From 970e56ba7a66f71fe3a18748e1b89d6816dd6f67 Mon Sep 17 00:00:00 2001 From: Lakhan Yelgate <39281685+lyelgate@users.noreply.github.com> Date: Sun, 1 Nov 2026 15:02:04 +0530 Subject: [PATCH 4/5] docs: exclude UDPRoute from health-check targets A UDPRoute backend's cluster is built from ir.UDPRoute, which only carries load balancer and DNS settings, so a health check set by a BackendTrafficPolicy never reaches it. List the route kinds that do support health checks instead of "the other xRoute kinds". Signed-off-by: Lakhan Yelgate <39281685+lyelgate@users.noreply.github.com> --- site/content/en/latest/tasks/traffic/health-check.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/site/content/en/latest/tasks/traffic/health-check.md b/site/content/en/latest/tasks/traffic/health-check.md index 419d9852e6..19938015fb 100644 --- a/site/content/en/latest/tasks/traffic/health-check.md +++ b/site/content/en/latest/tasks/traffic/health-check.md @@ -11,8 +11,8 @@ that are able to serve them. Two kinds of health checking are supported, and the return too many errors. This is Envoy's [Outlier Detection][]. Health checking is configured in the `healthCheck` field of a [BackendTrafficPolicy][], which can target a [Gateway][], -an [HTTPRoute][], a [GRPCRoute][] or the other xRoute kinds. Each backendRef of a route has its own set of endpoints, -and each endpoint is checked independently. +an [HTTPRoute][], a [GRPCRoute][], a TCPRoute or a TLSRoute. Health checks are not applied to the backends of a +UDPRoute. Each backendRef of a route has its own set of endpoints, and each endpoint is checked independently. ## Prerequisites From 9d2032949cbb3c74f0b02d141ee736eb8ef76d67 Mon Sep 17 00:00:00 2001 From: Lakhan Yelgate <39281685+lyelgate@users.noreply.github.com> Date: Sun, 1 Nov 2026 23:02:15 +0530 Subject: [PATCH 5/5] docs: clarify health check targets, outlier ejection timing and its cap BackendTrafficPolicy also targets ListenerSets and, with MergeBackends, Services, ServiceImports and Backends, so describe the targets without an exhaustive list. A consecutive 5xx ejection happens as soon as the threshold is reached, while interval sets how often the periodic outlier analysis runs. The ejection time grows with each ejection only up to Envoy's default max_ejection_time of 300 seconds. Signed-off-by: Lakhan Yelgate <39281685+lyelgate@users.noreply.github.com> --- .../en/latest/tasks/traffic/health-check.md | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/site/content/en/latest/tasks/traffic/health-check.md b/site/content/en/latest/tasks/traffic/health-check.md index 19938015fb..7b215cc658 100644 --- a/site/content/en/latest/tasks/traffic/health-check.md +++ b/site/content/en/latest/tasks/traffic/health-check.md @@ -10,9 +10,10 @@ that are able to serve them. Two kinds of health checking are supported, and the - **Passive health checking**: Envoy watches the responses of real requests and temporarily ejects endpoints that return too many errors. This is Envoy's [Outlier Detection][]. -Health checking is configured in the `healthCheck` field of a [BackendTrafficPolicy][], which can target a [Gateway][], -an [HTTPRoute][], a [GRPCRoute][], a TCPRoute or a TLSRoute. Health checks are not applied to the backends of a -UDPRoute. Each backendRef of a route has its own set of endpoints, and each endpoint is checked independently. +Health checking is configured in the `healthCheck` field of a [BackendTrafficPolicy][], and applies to the backends of +the resources the policy targets, such as a [Gateway][], an [HTTPRoute][] or a [GRPCRoute][]. Health checks are not +applied to the backends of a UDPRoute. Each backendRef of a route has its own set of endpoints, and each endpoint is +checked independently. ## Prerequisites @@ -198,8 +199,9 @@ See [ActiveHealthCheck][] for every field and its default. To log the result of ## Passive Health Checking Passive health checking needs no probes: Envoy counts the errors returned by each endpoint while serving real requests. -Apply a policy that ejects an endpoint for 30 seconds after 5 consecutive `5xx` responses, checking every 3 seconds. -It never ejects more than half of the endpoints, but `alwaysEjectOneEndpoint` still allows one endpoint to be ejected +Apply a policy that ejects an endpoint for 30 seconds as soon as it returns 5 consecutive `5xx` responses. `interval` +sets how often Envoy runs its periodic outlier analysis, which also returns endpoints to service once their ejection +time has passed. It never ejects more than half of the endpoints, but `alwaysEjectOneEndpoint` still allows one endpoint to be ejected when that limit rounds down to zero, as it does for the single endpoint of the quickstart backend: ```shell @@ -208,7 +210,8 @@ kubectl patch backendtrafficpolicy/health-check --type merge \ ``` An ejected endpoint is reported with the `failed_outlier_check` health flag. It stays ejected for `baseEjectionTime` -multiplied by the number of times it has been ejected, so an endpoint that keeps failing is ejected for longer. Because +multiplied by the number of times it has been ejected, up to Envoy's default maximum of 300 seconds, so an endpoint +that keeps failing is ejected for longer. Because this policy also has an active health check, an ejection can end early: Envoy returns an ejected endpoint to service as soon as it passes an active health check. See [PassiveHealthCheck][] for every field and its default.