Repository navigation
docs: add a task for active and passive health checking #10474
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
lyelgate
wants to merge
5
commits into
envoyproxy:main
Choose a base branch
from
lyelgate:docs/health-checking
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
46e0624
docs: add a task for active and passive health checking
lyelgate 3b2dd8f
docs: correct health check thresholds, ejection and panic mode notes
lyelgate cb56044
docs: note that active health checks can end a passive ejection early
lyelgate 970e56b
docs: exclude UDPRoute from health-check targets
lyelgate 9d20329
docs: clarify health check targets, outlier ejection timing and its cap
lyelgate File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,237 @@ | ||
| --- | ||
| title: "Health Checking" | ||
| --- | ||
|
|
||
| Envoy Gateway can check the health of the endpoints behind a backend, so that Envoy only sends requests to endpoints | ||
| that are able to serve them. Two kinds of health checking are supported, and they can be used together: | ||
|
|
||
| - **Active health checking**: Envoy periodically sends a probe (HTTP, TCP or gRPC) to every endpoint and marks an | ||
| endpoint unhealthy once enough consecutive probes fail. See [Envoy Active Health Checking][]. | ||
| - **Passive health checking**: Envoy watches the responses of real requests and temporarily ejects endpoints that | ||
| return too many errors. This is Envoy's [Outlier Detection][]. | ||
|
|
||
| Health checking is configured in the `healthCheck` field of a [BackendTrafficPolicy][], and applies to the backends of | ||
| the resources the policy targets, such as a [Gateway][], an [HTTPRoute][] or a [GRPCRoute][]. Health checks are not | ||
| applied to the backends of a UDPRoute. Each backendRef of a route has its own set of endpoints, and each endpoint is | ||
| checked independently. | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| {{< boilerplate prerequisites >}} | ||
|
|
||
| ## Active Health Checking | ||
|
|
||
| Apply a `BackendTrafficPolicy` that probes every endpoint of the `backend` HTTPRoute with an HTTP `GET /` request every | ||
| 3 seconds. An endpoint is marked unhealthy after 3 probes in a row time out or fail to connect, and healthy again after | ||
| 1 successful probe. A response with an unexpected status marks the endpoint unhealthy right away, unless that status is | ||
| listed in `retriableStatuses`, in which case it counts toward `unhealthyThreshold` like a timeout. | ||
|
|
||
| {{< tabpane text=true >}} | ||
| {{% tab header="Apply from stdin" %}} | ||
|
|
||
| ```shell | ||
| cat <<EOF | kubectl apply -f - | ||
| apiVersion: gateway.envoyproxy.io/v1alpha1 | ||
| kind: BackendTrafficPolicy | ||
| metadata: | ||
| name: health-check | ||
| spec: | ||
| targetRefs: | ||
| - group: gateway.networking.k8s.io | ||
| kind: HTTPRoute | ||
| name: backend | ||
| healthCheck: | ||
| active: | ||
| type: HTTP | ||
| timeout: 1s | ||
| interval: 3s | ||
| unhealthyThreshold: 3 | ||
| healthyThreshold: 1 | ||
| http: | ||
| path: / | ||
| EOF | ||
| ``` | ||
|
|
||
| {{% /tab %}} | ||
| {{% tab header="Apply from file" %}} | ||
| Save and apply the following resource to your cluster: | ||
|
|
||
| ```yaml | ||
| --- | ||
| apiVersion: gateway.envoyproxy.io/v1alpha1 | ||
| kind: BackendTrafficPolicy | ||
| metadata: | ||
| name: health-check | ||
| spec: | ||
| targetRefs: | ||
| - group: gateway.networking.k8s.io | ||
| kind: HTTPRoute | ||
| name: backend | ||
| healthCheck: | ||
| active: | ||
| type: HTTP | ||
| timeout: 1s | ||
| interval: 3s | ||
| unhealthyThreshold: 3 | ||
| healthyThreshold: 1 | ||
| http: | ||
| path: / | ||
| ``` | ||
|
|
||
| {{% /tab %}} | ||
| {{< /tabpane >}} | ||
|
|
||
| Check that the policy was accepted: | ||
|
|
||
| ```shell | ||
| kubectl get backendtrafficpolicy/health-check -o yaml | ||
| ``` | ||
|
|
||
| The health of each endpoint is visible through the Envoy admin interface: | ||
|
|
||
| ```shell | ||
| ENVOY_POD=$(kubectl get pods -n envoy-gateway-system -l gateway.envoyproxy.io/owning-gateway-name=eg -o jsonpath='{.items[0].metadata.name}') | ||
| kubectl -n envoy-gateway-system port-forward pod/${ENVOY_POD} 19000:19000 & | ||
| curl -s localhost:19000/clusters | grep "httproute/default/backend" | grep health_flags | ||
| ``` | ||
|
|
||
| The quickstart backend answers `GET /` with a `200`, so the endpoint is reported as healthy: | ||
|
|
||
| ```console | ||
| httproute/default/backend/rule/0::10.244.0.11:3000::health_flags::healthy | ||
| ``` | ||
|
|
||
| ### Simulate a failing endpoint | ||
|
|
||
| By default, only a `200` response counts as healthy. To see what happens when an endpoint fails its health check, | ||
| expect a `204` instead, which the quickstart backend never returns: | ||
|
|
||
| ```shell | ||
| kubectl patch backendtrafficpolicy/health-check --type merge \ | ||
| -p '{"spec":{"healthCheck":{"active":{"http":{"expectedStatuses":[204]}}}}}' | ||
| ``` | ||
|
|
||
| The next probe gets a `200`, which is neither expected nor retriable, so the endpoint is marked as failing the active | ||
| health check right away: | ||
|
|
||
| ```console | ||
| httproute/default/backend/rule/0::10.244.0.11:3000::health_flags::/failed_active_hc | ||
| ``` | ||
|
|
||
| Requests to the backend still succeed, because of Envoy's [panic threshold][Panic Threshold]: when fewer than 50% of a | ||
| backend's endpoints are healthy, Envoy ignores health status and balances across all of its endpoints, so that a | ||
| failing health check can't take down a whole backend at once. The quickstart backend has a single endpoint, so it is | ||
| always in panic mode while that endpoint is unhealthy. | ||
|
|
||
| If you followed the "Without LoadBalancer Support" prerequisite, run `export GATEWAY_HOST=localhost:8888` first. | ||
|
|
||
| ```shell | ||
| curl --verbose --header "Host: www.example.com" http://$GATEWAY_HOST/ | ||
| ``` | ||
|
|
||
| To stop sending requests to unhealthy endpoints even when all of them are unhealthy, set `panicThreshold` to `0`: | ||
|
|
||
| ```shell | ||
| kubectl patch backendtrafficpolicy/health-check --type merge \ | ||
| -p '{"spec":{"healthCheck":{"panicThreshold":0}}}' | ||
| ``` | ||
|
|
||
| The same request now fails with `503 Service Unavailable` and the body `no healthy upstream`. | ||
|
|
||
| Restore the expected statuses and the default panic threshold before continuing: | ||
|
|
||
| ```shell | ||
| kubectl patch backendtrafficpolicy/health-check --type json \ | ||
| -p '[{"op":"remove","path":"/spec/healthCheck/active/http/expectedStatuses"},{"op":"remove","path":"/spec/healthCheck/panicThreshold"}]' | ||
| ``` | ||
|
|
||
| ### Other active health checkers | ||
|
|
||
| The `type` field selects the health checker, and the matching field configures it. These snippets replace the | ||
| `healthCheck` block of the policy above. | ||
|
|
||
| A TCP health checker that only checks that a connection can be opened, or, with `send` and `receive`, that the endpoint | ||
| answers a payload with the expected bytes: | ||
|
|
||
| ```yaml | ||
| healthCheck: | ||
| active: | ||
| type: TCP | ||
| tcp: | ||
| send: | ||
| type: Text | ||
| text: ping | ||
| receive: | ||
| type: Text | ||
| text: pong | ||
| ``` | ||
|
|
||
| A gRPC health checker that uses the [gRPC health checking protocol][], optionally for a specific service: | ||
|
|
||
| ```yaml | ||
| healthCheck: | ||
| active: | ||
| type: GRPC | ||
| grpc: | ||
| service: helloworld.Greeter | ||
| ``` | ||
|
|
||
| The HTTP health checker can also send a different `method`, `hostname` or `requestBody`, accept a list of | ||
| `expectedStatuses`, and match the response body with `expectedResponse`. When the health check is served on a different | ||
| port than the application, set `overrides.port`: | ||
|
|
||
| ```yaml | ||
| healthCheck: | ||
| active: | ||
| type: HTTP | ||
| http: | ||
| path: /healthz | ||
| expectedStatuses: | ||
| - 200 | ||
| - 204 | ||
| overrides: | ||
| port: 9090 | ||
| ``` | ||
|
|
||
| See [ActiveHealthCheck][] for every field and its default. To log the result of each probe, see | ||
| [Health Check Logging](../../observability/proxy-health-check-log). | ||
|
|
||
| ## Passive Health Checking | ||
|
|
||
| Passive health checking needs no probes: Envoy counts the errors returned by each endpoint while serving real requests. | ||
| Apply a policy that ejects an endpoint for 30 seconds as soon as it returns 5 consecutive `5xx` responses. `interval` | ||
| sets how often Envoy runs its periodic outlier analysis, which also returns endpoints to service once their ejection | ||
| time has passed. It never ejects more than half of the endpoints, but `alwaysEjectOneEndpoint` still allows one endpoint to be ejected | ||
| when that limit rounds down to zero, as it does for the single endpoint of the quickstart backend: | ||
|
|
||
| ```shell | ||
| kubectl patch backendtrafficpolicy/health-check --type merge \ | ||
| -p '{"spec":{"healthCheck":{"passive":{"consecutive5XxErrors":5,"interval":"3s","baseEjectionTime":"30s","maxEjectionPercent":50,"alwaysEjectOneEndpoint":true}}}}' | ||
| ``` | ||
|
|
||
| An ejected endpoint is reported with the `failed_outlier_check` health flag. It stays ejected for `baseEjectionTime` | ||
| multiplied by the number of times it has been ejected, up to Envoy's default maximum of 300 seconds, so an endpoint | ||
| that keeps failing is ejected for longer. Because | ||
| this policy also has an active health check, an ejection can end early: Envoy returns an ejected endpoint to service as | ||
| soon as it passes an active health check. See [PassiveHealthCheck][] for every field and its default. | ||
|
|
||
| Active and passive health checking can be combined in the same policy: an endpoint is only considered healthy when it | ||
| passes both. Panic mode still applies, so when fewer than `panicThreshold` percent of the endpoints are healthy, Envoy | ||
| sends requests to all of them regardless. | ||
|
|
||
| ## Clean-Up | ||
|
|
||
| ```shell | ||
| kubectl delete backendtrafficpolicy/health-check | ||
| ``` | ||
|
|
||
| [Envoy Active Health Checking]: https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/upstream/health_checking | ||
| [Outlier Detection]: https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/upstream/outlier | ||
| [Panic Threshold]: https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/upstream/load_balancing/panic_threshold | ||
| [gRPC health checking protocol]: https://github.com/grpc/grpc/blob/master/doc/health-checking.md | ||
| [BackendTrafficPolicy]: ../../../api/extension_types#backendtrafficpolicy | ||
| [ActiveHealthCheck]: ../../../api/extension_types#activehealthcheck | ||
| [PassiveHealthCheck]: ../../../api/extension_types#passivehealthcheck | ||
| [Gateway]: https://gateway-api.sigs.k8s.io/reference/api-types/gateway/ | ||
| [HTTPRoute]: https://gateway-api.sigs.k8s.io/reference/api-types/httproute/ | ||
| [GRPCRoute]: https://gateway-api.sigs.k8s.io/reference/api-types/grpcroute/ | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When users follow this page's "Without LoadBalancer Support" prerequisite,
site/content/en/latest/boilerplates/testing.md:23-46starts a proxy onlocalhost:8888but never definesGATEWAY_HOST. This command therefore expands tohttp:///, so the panic-mode request and the subsequent 503 comparison cannot be exercised on kind or other clusters without an external load balancer. Provide the matching localhost request path or initialize the variable in that prerequisite branch.Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good point. Added a line before the request telling readers who followed the "Without LoadBalancer Support" prerequisite to
export GATEWAY_HOST=localhost:8888, in 3b2dd8f.