[do not merge] hack: check the atenet gateways on every IP family - #1047
Draft
Yuan Gao (ygao-g) wants to merge 5 commits into
Draft
[do not merge] hack: check the atenet gateways on every IP family#1047Yuan Gao (ygao-g) wants to merge 5 commits into
Yuan Gao (ygao-g) wants to merge 5 commits into
Conversation
The HTTP and HTTPS ingress listeners bound 0.0.0.0 only, so on a dual-stack cluster Envoy answered on the router Service's IPv4 ClusterIP and on nothing at all for IPv6. Each primary socket now carries an additional "::" address on the same port. Ipv4Compat stays false on it: setting it would clear IPV6_V6ONLY and collide with the primary already bound to that port, and Envoy rejects the whole listener when an additional address fails to bind, taking down all ingress rather than the IPv6 half. IPv4-only clusters are unaffected -- the primary is untouched, and a host without IPv6 simply has no second socket to bind. First of three commits binding atenet's gateways dual-stack.
The Envoy admin socket bound 0.0.0.0, and the atenet-router Service carried no ipFamilyPolicy -- which the API server defaults to SingleStack, one IPv4 ClusterIP and nothing else. Between them the router had no IPv6 address to answer on. The socket now binds "::" with ipv4_compat, one socket for both families, and the Service asks for PreferDualStack. ipv4_compat is load-bearing rather than incidental: dataplane.go health-checks the admin listener over http://127.0.0.1:9901/ready, so a bare "::" would report the dataplane component of /statusz unhealthy. Prefer, not Require, keeps the Service valid on a single-stack cluster, where it is a no-op; spec.ipFamilies is left alone because the primary family is immutable and the API server appends the secondary itself.
The gateway's admin and :443 sockets bound 0.0.0.0, so on an IPv6-primary cluster the kubelet probed the pod on its only address and atenet-egress crashlooped -- Envoy started fine and logged "admin address: 0.0.0.0:15000" -- while an actor's CONNECT had no v6 path in. Both sockets now bind "::" with ipv4_compat, and the Service asks for PreferDualStack so a dual-stack cluster hands out an IPv6 ClusterIP to reach them on. ipv4_compat matters on the admin socket in particular: the ext-proc sidecar's drainer dials 127.0.0.1:15000, and envoydrain.go reads a refusal there as "Envoy already exited" and skips the drain silently.
This was referenced Aug 18, 2026
Yuan Gao (ygao-g)
added
kind/bug
Yuan Gao (ygao-g)
changed the title
Yuan Gao (ygao-g)
force-pushed
the
atenet-dualstack-verify
branch
from
August 19, 2026 00:22
4bad809 to
cf41bc7
Compare
On a fresh IP_FAMILY=ipv6 cluster nothing resolves from inside a pod and no actor boots: CoreDNS inherits the node's IPv4 resolver, which a v6-only pod cannot reach. Two Corefile clauses fix it -- a hosts entry mapping kind-registry to its IPv6 address, and a forward to an IPv6 upstream, overridable with IPV6_DNS_UPSTREAM. IPv4 and dual-stack clusters are unchanged, and atenet-egress still crashloops on v6-only for an unrelated Envoy bind bug. Asking once was not enough to prove that: about half of fresh clusters do not answer the first query, and a pod that goes unanswered stays unanswered, so the check re-asks with a new pod and prints what the pod saw when it gives up. It lives in hack/verify-ipv6-dns.sh rather than inline, because the hosts entry records an address the registry can move off and there was no way to re-check a cluster without rebuilding it.
Whether the router and egress gateways really serve on every family the cluster has was only ever checked by hand, and the checks are easy to get wrong -- Envoy's /listeners reports resolved addresses and never emits ipv4_compat, so an assertion written against it passes on a manifest that lost the flag. This script reads the families off the Service and reports every check it makes, so one run is valid on ipv4, dual, and ipv6. Worth running on all three, because they fail differently. A v4-wildcard bind is a silent data-path gap on dual-stack, where the Service has an IPv6 ClusterIP nothing listens on, and a crashloop on IPv6-only, where the kubelet probes the pod on its only address.
Yuan Gao (ygao-g)
force-pushed
the
atenet-dualstack-verify
branch
from
August 19, 2026 17:24
cf41bc7 to
84fc597
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #246.
hack/check-atenet-ip-families.shchecks that both atenet gateways bind and serve on every IP family the cluster has, so the dual-stack work stops being verified by hand. It reads the families off the Service and reports every check rather than stopping at the first failure, so one run is valid on ipv4, dual or ipv6 — worth running on each, because they fail differently. A v4-wildcard bind is a silent data-path gap on dual-stack, where the Service has an IPv6 ClusterIP nothing listens on, and a crashloop on IPv6-only, where the kubelet probes the pod on its only address.Dependencies
Only the last commit is new; the branch carries two open PRs so the script has something to run against. This branch is for reference only and is not intended to merge — #911 and #958 are the PRs to review.
IP_FAMILYinhack/create-kind-cluster.shis already on main (#877).Testing
IP_FAMILY=ipv6 hack/create-kind-cluster.sh, thenhack/install-ate-kind.sh --deploy-ate-system, on a single-stack IPv6 kind cluster. The install completed — includingatenet-egress, which times out on main. All 14 checks pass:atenet-routerandatenet-egressboth 2/2 Running with 0 restarts.admin address: [::]:9901and[::]:15000in the Envoy logs, and/readyreturns 200 on both127.0.0.1and::1for each — the loopback that the router's health check and the egress drainer dial as IPv4 literals.ipv4_compatcounted from/config_dump. Not/listeners, which reports resolved bound addresses and never emits the flag, so a check written against it would pass on a manifest that had lost it.ingress_http_listenerandingress_https_listenerkeep their0.0.0.0primary and gain a::additional address.Negative control: reverting only the two bind lines to
0.0.0.0flipped exactly the six egress checks to FAIL, left every router check passing, and reproducedStartup probe failed: ... connect: connection refused. Restoring them turned it green again, so the assertions are not vacuous.The e2e suites do run on IPv6-only once #958 is in —
TestActorDirectAccesspasses, including through the router's ingress.TestActorEgressstill fails, but inside the actor's netns (dial udp [fd00:10:96::a]:53: connect: network is unreachable), which is #979's scope, not this one's.The script itself has so far only been run on IPv6-only; the ipv4 and dual-stack paths are unexercised from this branch.
🤖 Generated with Claude Code