feat(management-proxy): expose replicas, panic threshold and admin bind address - #2717
Open
philxiao wants to merge 1 commit into
Open
feat(management-proxy): expose replicas, panic threshold and admin bind address#2717philxiao wants to merge 1 commit into
philxiao wants to merge 1 commit into
Conversation
…nd address Adds managementProxy.replicas (was hardcoded to 2), managementProxy.healthyPanicThreshold and managementProxy.adminBindAddress. healthy_panic_threshold was never emitted, so Envoy's 50% default applied: once fewer than half the upstreams pass their health check, Envoy stops honouring health checks and load balances across all hosts, including ones it knows are down. Where only some management IPs are reachable from the pod network this turns a partial outage into a near-total one; setting it to 0 keeps traffic on the reachable subset. adminBindAddress allows restricting Envoy's unauthenticated admin API to loopback, which matters with hostNetwork. The probes switch from HTTPGet /ready on 9901 to a TCP check on the proxy listener when it is not 0.0.0.0, since kubelet probes the pod IP and cannot reach loopback. Also pass --use-dynamic-base-id when hostNetwork is enabled. Envoy keys its shared memory and abstract domain sockets off --base-id, defaulting to 0, and the abstract namespace is per netns, so it collides with any other Envoy in the host netns and crash-loops with "unable to bind domain socket with base_id=0, id=0, errno=98". Defaults preserve current behaviour. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
graphite-app
Bot
requested review from
assafgi,
kristina-solovyova and
tigrawap
August 4, 2026 22:54
Graphite Automations"Add anton/matt/sergey/kristina as reviwers on operator PRs" took an action on this PR • (08/04/26)3 reviewers were added to this PR based on Anton Bykov's automation. |
There was a problem hiding this comment.
Pull request overview
This PR makes several Envoy management-proxy behaviors configurable via the Helm chart (and operator env/config), aiming to preserve existing defaults while enabling safer operation under partial upstream reachability and hostNetwork.
Changes:
- Add configurable
managementProxy.replicas,managementProxy.healthyPanicThreshold, andmanagementProxy.adminBindAddresssurfaced through chart → env vars → operator config. - Emit
healthy_panic_thresholdand configurable admin bind address into the generated Envoy config; adjust probes and Envoy command line (dynamic base-id underhostNetwork). - Update the management-proxy Deployment to use the new settings (replicas, command, probe handler, hostNetwork).
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| internal/controllers/wekacluster/funcs_management_proxy.go | Generate Envoy config with new LB/admin settings; update Deployment wiring (replicas/command/probes/hostNetwork). |
| internal/config/env.go | Add operator config fields and env parsing for the new management-proxy settings. |
| charts/weka-operator/values.yaml | Expose new managementProxy values and document their intent/defaults. |
| charts/weka-operator/templates/manager.yaml | Plumb new chart values into MANAGEMENT_PROXY_* env vars for the operator Deployment. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+601
to
+605
| Config.ManagementProxyReplicas = getIntEnvOrDefault("MANAGEMENT_PROXY_REPLICAS", 2) | ||
| // Envoy's own default is 50. Set to 0 to disable panic mode, so the proxy | ||
| // only ever routes to upstreams that pass their health check. | ||
| Config.ManagementProxyHealthyPanicThreshold = getIntEnvOrDefault("MANAGEMENT_PROXY_HEALTHY_PANIC_THRESHOLD", 50) | ||
| Config.ManagementProxyAdminBindAddress = env.GetString("MANAGEMENT_PROXY_ADMIN_BIND_ADDRESS", "0.0.0.0") |
Comment on lines
+457
to
+459
| # Defaults for every cluster this operator manages. Each of these can be | ||
| # overridden per cluster via WekaCluster.spec.managementProxy. | ||
| # |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Makes three management-proxy settings configurable via the chart. Defaults preserve current behaviour.
managementProxy.replicas22managementProxy.healthyPanicThreshold50managementProxy.adminBindAddress0.0.0.00.0.0.0Why
healthyPanicThresholdmattershealthy_panic_thresholdwas never emitted into the Envoy config, so Envoy's 50% default applied. Once fewer than half the upstreams fail their health check, Envoy enters panic mode and load balances across all hosts — including ones it knows are down.Where only some management IPs are reachable from the pod network, this turns a partial outage into a near-total one: most requests get routed into connect timeouts even though a healthy subset exists. Setting the threshold to
0keeps traffic on the reachable subset and degrades gracefully.We hit this on a cluster where a subset of the management IPs was unreachable from the pod network — the proxy was healthy, the reachable backends were healthy, and the API was still effectively down.
adminBindAddressEnvoy's admin API is unauthenticated and includes
/quitquitquitand/drain_listeners. WithhostNetwork: truethat is exposed on the node's network, so this allows restricting it to loopback.When the admin address is not
0.0.0.0, the liveness/readiness probes switch fromHTTPGet /readyon 9901 to a TCP check on the proxy listener — kubelet probes the pod IP and cannot reach loopback, so the HTTP probe would fail by construction.--use-dynamic-base-idunder hostNetworkEnvoy keys its shared-memory region and its
@envoy_domain_socket_*abstract sockets off--base-id, which defaults to 0. The abstract socket namespace is per network namespace, so underhostNetworkthe proxy collides with any other Envoy in the host netns — notablycilium-envoy, which runs hostNetwork on every node in a Cilium cluster using L7 features. It's a hard startup error and a crash loop:--use-dynamic-base-idmakes Envoy pick a free base id. Applied only whenhostNetworkis enabled, since in a pod netns base id 0 is unambiguous and changing it would be a needless behaviour change.Notes
healthyPanicThresholdwith akindIs "invalid"check rather thandefault, becausedefaultswallows an explicit0— which is the value most worth setting.MANAGEMENT_PROXY_*env is read per operator instance, so these are effectively per-Kubernetes-cluster; they are shared between multiple WekaClusters managed by the same operator. Happy to move any of them ontoWekaCluster.spec.managementProxyfor per-cluster override if you'd prefer that shape.go build ./internal/...andgo veton the changed packages are clean.🤖 Generated with Claude Code