Skip to content

atenet: bind the Envoy sockets and Services dual-stack - #911

Open
Yuan Gao (ygao-g) wants to merge 4 commits into
agent-substrate:mainfrom
ygao-g:atenet-envoy-dualstack
Open

atenet: bind the Envoy sockets and Services dual-stack#911
Yuan Gao (ygao-g) wants to merge 4 commits into
agent-substrate:mainfrom
ygao-g:atenet-envoy-dualstack

Conversation

@ygao-g

@ygao-g Yuan Gao (ygao-g) commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator

Fixes #910
Part of #246

It's a good idea to open an issue first for discussion.

  • Tests pass
  • Appropriate changes to documentation are included in the PR (none needed — no flags, APIs, or user-facing behaviour change)

Every Envoy socket in atenet binds the IPv4 wildcard, and neither gateway's Service asks for a second IP family. On a dual-stack cluster the router answers on its IPv4 ClusterIP and on nothing at all for IPv6. On an IPv6-primary cluster it is worse than a dark data path: the kubelet probes the pod on its only address, so atenet-egress crashloops while Envoy itself starts fine.

Both gateways now bind :: as well, and both Services ask for PreferDualStack. Where the config takes more than one address — the two ingress Listeners — the 0.0.0.0 primary is left untouched and :: is added alongside it. Where it takes only one — the two Envoy admin sockets, whose bootstrap field is singular, and the egress :443 Listener — the socket binds :: with ipv4_compat. That flag is load-bearing, not decoration: the router's dataplane health check dials 127.0.0.1:9901 and the egress drainer dials 127.0.0.1:15000, and a refused drain is reported as a completed one. A new hack/verify check keeps it that way, since nothing else in the tree reads these manifests.

This makes both gateways accept IPv6; it does not make them reach IPv6 destinations. The egress dns_lookup_family, the DNS AAAA path, and atunnel's original-destination lookup stay IPv4 and are tracked in #246 and #686.

Testing

CI has no IP-family matrix, so the green e2e run here is the IPv4 regression check and nothing more. The other two families were verified by hand on kind, using the IP_FAMILY support from #877.

  • go test ./cmd/atenet/internal/router/... — asserts the 0.0.0.0 primary, exactly one :: additional address, ipv4_compat false, and the port, on both ingress listeners.
  • Dual-stack. Both Services come up with two ClusterIPs and both gateway pods run with 0 restarts. The networking suite passes with zero skips: an actor is reached over each of the router's ClusterIPs, and an IPv4 actor transits the egress [::]:443 socket. Reverting only the two bind lines turns those assertions red, so they are not passing vacuously. ipv4_compat was read from /config_dump/listeners reports resolved bound addresses and never emits the flag.
  • IPv6-only, with the kind CoreDNS fix from hack: fix DNS on IPv6-only kind clusters #958 underneath. atenet-egress goes from 1/2 CrashLoopBackOff (64 restarts, startup probe refused) to 2/2 Running, Envoy logs admin address: [::]:15000, and both 127.0.0.1:15000/ready and [::1]:15000/ready return 200. TestActorDirectAccess passes, including through the router's ingress. TestActorEgress still fails, but inside the actor's netns, which is ateomnet: enable IPv6 forwarding in worker pod netns #979's scope.

A node with no AF_INET6 at all could not bind :: and the listener would not come up; every node image we target has it, and the IPv4 e2e above is the check.

🤖 Generated with Claude Code

@ygao-g

Copy link
Copy Markdown
Collaborator Author

Bowei Du (@bowei) mind taking a look?

Address: &corev3.Address_SocketAddress{
SocketAddress: &corev3.SocketAddress{
Address: "::",
Ipv4Compat: false,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want to sync up the configuration in the YAML with the configuration here?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, the comment here might have confused me. This is talking about two different sockets? If it is, can you remove the comment above.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, two different sockets. This is the router's ingress data-plane listener; the ipv4_compat: true you saw in the YAML is the Envoy admin socket on 9901. I've cut the comment down to the one line explaining why Ipv4Compat is false here — the cross-reference to the admin socket was the confusing part and it's gone.

On the original question: the admin sockets can't take this shape anyway, since bootstrap.v3.Admin has a single address field and no additional_addresses — one :: socket with ipv4_compat is the only way to serve both families there. The egress :443 listener is a real Listener, so that one could match the ingress shape if you'd rather have a single idiom across listeners. Happy to do it; it's a few lines plus the test that asserts the current shape.

@ygao-g
Yuan Gao (ygao-g) force-pushed the atenet-envoy-dualstack branch 2 times, most recently from 5257d6b to 580afef Compare August 18, 2026 14:59
@ygao-g Yuan Gao (ygao-g) changed the title atenet: bind the Envoy listeners, admin sockets, and Services dual-stack atenet: bind the Envoy sockets and Services dual-stack Aug 18, 2026
@ygao-g Yuan Gao (ygao-g) added kind/bug Something isn't working / bugfixes area/network labels Aug 18, 2026
@ygao-g
Yuan Gao (ygao-g) force-pushed the atenet-envoy-dualstack branch 2 times, most recently from 18169a0 to 378c535 Compare August 19, 2026 03:07
The HTTP and HTTPS ingress listeners bound 0.0.0.0 only, so on a
dual-stack cluster Envoy answered on the router Service's IPv4
ClusterIP and on nothing at all for IPv6. Each primary socket now
carries an additional "::" address on the same port.

Ipv4Compat stays false on the additional address: clearing IPV6_V6ONLY
would collide with the primary already bound to that port. Leaving the
primary alone is what keeps an IPv4-only cluster unchanged -- with the
caveat that a node lacking AF_INET6 entirely could not bind "::" and
the listener would not come up. First of three commits binding atenet's
gateways dual-stack.
The Envoy admin socket bound 0.0.0.0, and the atenet-router Service
carried no ipFamilyPolicy -- which the API server defaults to
SingleStack, one IPv4 ClusterIP and nothing else. Between them the
router had no IPv6 address to answer on. The socket now binds "::" with
ipv4_compat, one socket for both families, and the Service asks for
PreferDualStack.

bootstrap.v3.Admin takes a single address and has no
additional_addresses, so the ingress listeners' shape is not available
here; ipv4_compat is what makes the one socket serve both families.
It is load-bearing: dataplane.go health-checks the admin listener over
http://127.0.0.1:9901/ready, so a bare "::" would report the dataplane
component of /statusz unhealthy. Prefer, not Require, keeps the Service
valid on a single-stack cluster; spec.ipFamilies is left alone because
the primary family is immutable and the API server appends the
secondary itself.
The gateway's admin and :443 sockets bound 0.0.0.0, so on an
IPv6-primary cluster the kubelet probed the pod on its only address
and atenet-egress crashlooped -- Envoy started fine and logged "admin
address: 0.0.0.0:15000" -- while an actor's CONNECT had no v6 path in.
Both sockets now bind "::" with ipv4_compat, and the Service asks for
PreferDualStack so a dual-stack cluster hands out an IPv6 ClusterIP to
reach them on.

One socket here rather than the ingress listeners' pair: IPv4 peers
then arrive as ::ffff: addresses, and nothing on this path reads the
peer -- actor identity comes from the client certificate and the access
log records the cert SAN. ipv4_compat also has to stay on the admin
socket, because the ext-proc sidecar's drainer dials 127.0.0.1:15000
and envoydrain.go reads a refusal there as "Envoy already exited",
skipping the drain silently. Last of three.
Both gateway admin sockets bind "::" with ipv4_compat, and the flag is
what keeps their in-pod callers working: dataplane.go health-checks the
router's over IPv4 loopback, and envoydrain.go dials the egress one the
same way and reads a refusal as "Envoy already exited", skipping the
drain without reporting an error. No Go test, golden file, or verify
script read either manifest, so dropping the flag would have failed
silently.

make verify now rejects an admin socket that binds "::" without it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/network kind/bug Something isn't working / bugfixes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

atenet Envoy sockets and Services lack IPv6 support (hardcoded to 0.0.0.0)

2 participants