atenet: bind the Envoy sockets and Services dual-stack - #911
atenet: bind the Envoy sockets and Services dual-stack#911Yuan Gao (ygao-g) wants to merge 4 commits into
Conversation
|
Bowei Du (@bowei) mind taking a look? |
5308c09 to
3162d63
Compare
| Address: &corev3.Address_SocketAddress{ | ||
| SocketAddress: &corev3.SocketAddress{ | ||
| Address: "::", | ||
| Ipv4Compat: false, |
There was a problem hiding this comment.
Do we want to sync up the configuration in the YAML with the configuration here?
There was a problem hiding this comment.
Actually, the comment here might have confused me. This is talking about two different sockets? If it is, can you remove the comment above.
There was a problem hiding this comment.
Yes, two different sockets. This is the router's ingress data-plane listener; the ipv4_compat: true you saw in the YAML is the Envoy admin socket on 9901. I've cut the comment down to the one line explaining why Ipv4Compat is false here — the cross-reference to the admin socket was the confusing part and it's gone.
On the original question: the admin sockets can't take this shape anyway, since bootstrap.v3.Admin has a single address field and no additional_addresses — one :: socket with ipv4_compat is the only way to serve both families there. The egress :443 listener is a real Listener, so that one could match the ingress shape if you'd rather have a single idiom across listeners. Happy to do it; it's a few lines plus the test that asserts the current shape.
5257d6b to
580afef
Compare
18169a0 to
378c535
Compare
The HTTP and HTTPS ingress listeners bound 0.0.0.0 only, so on a dual-stack cluster Envoy answered on the router Service's IPv4 ClusterIP and on nothing at all for IPv6. Each primary socket now carries an additional "::" address on the same port. Ipv4Compat stays false on the additional address: clearing IPV6_V6ONLY would collide with the primary already bound to that port. Leaving the primary alone is what keeps an IPv4-only cluster unchanged -- with the caveat that a node lacking AF_INET6 entirely could not bind "::" and the listener would not come up. First of three commits binding atenet's gateways dual-stack.
The Envoy admin socket bound 0.0.0.0, and the atenet-router Service carried no ipFamilyPolicy -- which the API server defaults to SingleStack, one IPv4 ClusterIP and nothing else. Between them the router had no IPv6 address to answer on. The socket now binds "::" with ipv4_compat, one socket for both families, and the Service asks for PreferDualStack. bootstrap.v3.Admin takes a single address and has no additional_addresses, so the ingress listeners' shape is not available here; ipv4_compat is what makes the one socket serve both families. It is load-bearing: dataplane.go health-checks the admin listener over http://127.0.0.1:9901/ready, so a bare "::" would report the dataplane component of /statusz unhealthy. Prefer, not Require, keeps the Service valid on a single-stack cluster; spec.ipFamilies is left alone because the primary family is immutable and the API server appends the secondary itself.
The gateway's admin and :443 sockets bound 0.0.0.0, so on an IPv6-primary cluster the kubelet probed the pod on its only address and atenet-egress crashlooped -- Envoy started fine and logged "admin address: 0.0.0.0:15000" -- while an actor's CONNECT had no v6 path in. Both sockets now bind "::" with ipv4_compat, and the Service asks for PreferDualStack so a dual-stack cluster hands out an IPv6 ClusterIP to reach them on. One socket here rather than the ingress listeners' pair: IPv4 peers then arrive as ::ffff: addresses, and nothing on this path reads the peer -- actor identity comes from the client certificate and the access log records the cert SAN. ipv4_compat also has to stay on the admin socket, because the ext-proc sidecar's drainer dials 127.0.0.1:15000 and envoydrain.go reads a refusal there as "Envoy already exited", skipping the drain silently. Last of three.
Both gateway admin sockets bind "::" with ipv4_compat, and the flag is what keeps their in-pod callers working: dataplane.go health-checks the router's over IPv4 loopback, and envoydrain.go dials the egress one the same way and reads a refusal as "Envoy already exited", skipping the drain without reporting an error. No Go test, golden file, or verify script read either manifest, so dropping the flag would have failed silently. make verify now rejects an admin socket that binds "::" without it.
378c535 to
5c35bdc
Compare
Fixes #910
Part of #246
Every Envoy socket in atenet binds the IPv4 wildcard, and neither gateway's Service asks for a second IP family. On a dual-stack cluster the router answers on its IPv4 ClusterIP and on nothing at all for IPv6. On an IPv6-primary cluster it is worse than a dark data path: the kubelet probes the pod on its only address, so
atenet-egresscrashloops while Envoy itself starts fine.Both gateways now bind
::as well, and both Services ask forPreferDualStack. Where the config takes more than one address — the two ingress Listeners — the0.0.0.0primary is left untouched and::is added alongside it. Where it takes only one — the two Envoy admin sockets, whose bootstrap field is singular, and the egress:443Listener — the socket binds::withipv4_compat. That flag is load-bearing, not decoration: the router's dataplane health check dials127.0.0.1:9901and the egress drainer dials127.0.0.1:15000, and a refused drain is reported as a completed one. A newhack/verifycheck keeps it that way, since nothing else in the tree reads these manifests.This makes both gateways accept IPv6; it does not make them reach IPv6 destinations. The egress
dns_lookup_family, the DNS AAAA path, and atunnel's original-destination lookup stay IPv4 and are tracked in #246 and #686.Testing
CI has no IP-family matrix, so the green e2e run here is the IPv4 regression check and nothing more. The other two families were verified by hand on kind, using the
IP_FAMILYsupport from #877.go test ./cmd/atenet/internal/router/...— asserts the0.0.0.0primary, exactly one::additional address,ipv4_compatfalse, and the port, on both ingress listeners.[::]:443socket. Reverting only the two bind lines turns those assertions red, so they are not passing vacuously.ipv4_compatwas read from/config_dump—/listenersreports resolved bound addresses and never emits the flag.atenet-egressgoes from1/2 CrashLoopBackOff(64 restarts, startup probe refused) to2/2 Running, Envoy logsadmin address: [::]:15000, and both127.0.0.1:15000/readyand[::1]:15000/readyreturn 200.TestActorDirectAccesspasses, including through the router's ingress.TestActorEgressstill fails, but inside the actor's netns, which is ateomnet: enable IPv6 forwarding in worker pod netns #979's scope.A node with no
AF_INET6at all could not bind::and the listener would not come up; every node image we target has it, and the IPv4 e2e above is the check.🤖 Generated with Claude Code