network: publish guest listeners to the host's private networks by ppenna · Pull Request #495 · microsoft/nvx

Pedro Henrique Penna (ppenna) · 2026-10-11T20:06:25Z

Fixes #275 with nanvix/openvmm#133: with --network-ingress allow, clients on the host's private networks reach every TCP and UDP port on which the guest listens, without a port list. Internet clients never reach the guest, the host itself is admitted only with explicit --host-loopback allow, and replies to admitted clients pass even when egress is denied. The guest announces its listeners to OpenVMM, run and sandbox pass the option through, aci_edge_sandboxes applies network.ingress.default: allow, and a new test-microvm scenario exercises each policy.

Why

The portable profile rejected --network-ingress allow before launch, because its user-mode NAT had no way to reach a guest listener: every inbound connection needs a host socket, and Consomme bound only the ports that explicit --host-loopback-forward options named. MXC's network.ingress.default: allow means LAN or private-network inbound access to the ports where the workload listens, with no port list, no WAN inbound, host loopback as a separate setting, and responses allowed under an egress deny.

Commits

Commit	OpenVMM pin	Change
`280b62cdc` openvmm: publish guest listeners to the host's private networks	`fc2de0506`	Private-network ingress of nanvix/openvmm#133; the run guide's new section, the usage reference, and the cold-boot, machine ABI, current limits, and status designs; `directional-network-policy` no longer expects ingress `allow` to be rejected
`5307643cd` guest: announce the guest's listeners for private-network ingress		`nvx-ingress`, a static musl helper that the common init starts when the command line carries `virtnet_ingress=allow`; the Alpine, Ubuntu, and Azure Linux builds; its test mode and `scripts/test_guest_ingress.py`, which validate-nvx runs
`1a8e54c10` aci_edge_sandboxes: apply private-network ingress in OpenVMM		The OpenVMM backend reports `ingress_allow` and passes the ingress default; unit, fake OpenVMM, and end-to-end tests; the README
`10971062a` test-microvm: verify private-network ingress		The `private-network-ingress` scenario in the default suite, its unit tests, and the validation design

Each commit passes nvx.py verify, ruff, pyright for Linux and Windows, and the NVX unit suites on its own.

Behavior

The guest's nvx-ingress reads the kernel's socket tables every 100 ms and announces TCP listeners and unconnected UDP sockets that persist for a second (each UDP socket, identified by its inode, stays announced while it exists, even once a server connects it to a client, and a port announces the families of its announced sockets), except those bound to loopback or IPv6 link-local addresses; a TCP listener on :: counts for IPv4 unless the kernel's socket diagnostics report it IPv6-only, and a UDP one unless net.ipv6.bindv6only is set. It sends the list in a UDP datagram to port 1021 of the IPv4 gateway, from a source port below 1024 so that only guest root can publish, on every change and at least every 2 seconds. OpenVMM consumes the announcement before the egress policy and, for each announced port and family, listens on that port of every host address.

Policy	Client on the host's private network	Internet client	The host itself	Guest-initiated traffic
`--network-ingress deny` (default)	Nothing is published	Nothing is published	Nothing is published	Per egress policy
`--network-ingress allow`	Reaches every TCP and UDP listener; replies pass under any egress policy	TCP reset, UDP dropped	TCP reset, UDP dropped	Per egress policy
`--network-ingress allow --host-loopback allow`	Same	TCP reset, UDP dropped	Reaches every listener, over loopback or the host's own addresses	Per egress policy

"Private network" means an address that is not globally reachable: the IPv4 private-use, shared (100.64.0.0/10), link-local, benchmarking, and documentation ranges, and the IPv6 unique local, site-local, documentation, and benchmarking ranges. Clients inside the guest's own networks, such as 192.0.2.0/24 and fd00::c000:200/120 for --net 192.0.2.2/24, are refused, because the guest would take them for its gateway or its neighbors, and so are IPv6 link-local clients, whose replies would leave from the guest's link-local address.
The host is a peer at loopback, at the address that it connected to, or at one of the host's own unicast addresses, listed with getifaddrs or GetUnicastIpAddressTable. If the host's addresses cannot be listed, a private-network peer is admitted only when the host would be, and a globally reachable one never.
Replies stay narrow: the egress policy admits a TCP segment other than an initial SYN, or a UDP datagram, that its rules deny only when Consomme carries its flow, a connection in its TCP table or the association of a published UDP port with a peer that recently sent to it, and it admits the ARP and Neighbor Solicitations that answering needs under egress deny. A UDP reply leaves through the publishing host socket, so the client sees it come from the port it reached.
--host-loopback allow without forwards is now accepted when ingress is allow. Snapshot capture and restore reject ingress allow, because its listeners are live host attachments. Ingress allow records the ingress action in canonical policy encoding version 5; the digests of existing policies do not change.
aci_edge_sandboxes applies network.ingress.default: allow with host loopback still denied; hostLoopback: allow remains rejected.
Limits: at most 64 announced ports; publication follows the guest within about 100 ms for TCP and a second for UDP; a host port that another process holds, or on Linux a privileged one that OpenVMM may not bind, is not published, and OpenVMM retries it with the next announcement; clients on other hosts also need the host firewall to admit OpenVMM, such as an inbound rule for openvmm.exe on Windows.

This also covers part of #276: with ingress allow, --host-loopback allow lets the host reach every guest listener without a port list. Without ingress allow, generic bidirectional host loopback remains unsupported.

Review

A code review of the OpenVMM change before opening this PR found five issues, all fixed in fc2de0506 with tests: with the host admitted, a failure to list the host's addresses would have admitted every peer, including internet ones; a UDP reply to a client whose address equals the guest's gateway could be reinterpreted as gateway DNS or host-loopback traffic, which is why clients inside the guest's networks are refused and replies to a published port's peers now leave through its listener before gateway handling; IPv6 link-local clients could not be answered under an active policy and lost their interface scope, so they are refused and UDP peers keep their complete host-side address; and an explicit forward to an announced guest port blocked its publication, so publications now have their own listener key. Validation also found that Python's socket module has no SIO_UDP_CONNRESET, which the scenario no longer uses.
Copilot's reviews, all addressed and folded into their commits: the aci_edge_sandboxes end-to-end host probe now sends a marker and half-closes, so a connection that wrongly reached the workload would show in its output; the agent backend's network paragraph, which still requires ingress deny, now says that it differs from the OpenVMM backend; the announcer reads each TCP listener's IPV6_V6ONLY from the kernel's socket diagnostics instead of assuming net.ipv6.bindv6only, which UDP sockets still follow because the guest kernel builds no UDP socket diagnostics; UDP sockets settle and stay announced per socket, by inode, with their own families, so a client that reuses a port cannot inherit its publication and separate IPv4 and IPv6-only servers on one port each keep their family once connected; the scenario's guest script waits, bounded, for the servers to record what they received; and the announcer's state machine now has automated tests through a test mode. A comment wording fix is folded too.

Validation

OpenVMM, on fc2de0506: cargo nextest for net_backend_resources, consomme, net_consomme, net_tap, openvmm_defs, openvmm_entry, and virtio_net: 502 passed, 1 skipped, on Windows with Rust 1.95.0, and 567 passed, 1 skipped, on Linux on prometheus30. clippy --all-targets on Windows and Linux and rustdoc with -D warnings are clean for every modified package, and xtask fmt passes.
NVX, on 10971062a: nvx.py verify; the validate-nvx unit suites (979 tests on Windows, where the 8 Linux-only test_guest_ingress.py tests skip; those 8 pass on Linux) and the adversarial suite; ruff; pyright for Linux and Windows; shellcheck and shfmt with the pinned images for the init and the rendered scenario script; and the announcer compiled with the guest build's musl-gcc -Werror flags and with gcc -Wpedantic -Wconversion -Werror. Mutating the announcer's settle time, its IPv6-only flag, or its socket identity check fails the new tests. For aci_edge_sandboxes: fmt --all, clippy with -D warnings for both feature sets, the tests with all features and with the defaults, the artifact bundling check, rustdoc with -D warnings, and the Rust 1.89 MSRV check. Each commit also passes nvx.py verify, ruff, pyright, and the unit suites on its own.
The announcer against the real kernel, in a Linux network namespace with a local stand-in for the gateway: an IPv6-only TCP listener on :: is announced for IPv6 alone, dual-stack and default ones for both families, and a port that both kinds share for both until the dual-stack socket closes; a settled UDP server is announced, a socket that replaces it is withdrawn and announced again only after it settles (1.17 s), a server that connects its settled socket keeps the port, a connected client that reuses the port is never announced, and separate IPv4 and IPv6-only servers on one port each keep their family once connected.
Bare metal, with OpenVMM built from fc2de0506 for each platform and the Alpine guest built from 10971062a: private-network-ingress, host-loopback-policy, directional-network-policy, virtio-net, and network-snapshot passed on prometheus32 (KVM), prometheus30 (MSHV), and prometheus28 (WHP), and test-aci-edge-sandboxes, including the new private_network_ingress_publishes_workload_listeners, passed 16 of 16 on each. Earlier in the review, with the same OpenVMM and the guest before the last announcer changes, the full default test-microvm suite (40 scenario runs) passed on KVM and MSHV, and ipv6-egress-policy, l3-l4-egress-policy, and endpoint-policy-snapshot passed on all three hosts.
Cross-host, on the lab's benchmarking network (198.18.0.0/15), with the final guest: a guest on prometheus32 (KVM) with --net 192.0.2.2/24 --network-egress deny --network-ingress allow served TCP and UDP on port 52245. The host held the port for both protocols within two seconds of the servers listening; the host's own connections to 198.19.2.32:52245 and 127.0.0.1:52245 were reset and its datagrams unanswered; a client on prometheus30 (198.19.2.30) received the TCP server's line and sent its own, and got the UDP reply from 198.19.2.32:52245; and the guest received both requests. The reverse direction was not run: prometheus30 and prometheus28 have host firewalls that drop unsolicited inbound traffic, which this change leaves to the operator.

Dependencies

nanvix/openvmm#133, open, based on that repository's main (2130f24b5), which dev pins. Merge it by fast-forwarding main so that fc2de0506 stays on main.

Copilot

🟡 Changes recommended

The host-denial E2E test can miss leaked connections, and the README contains contradictory ingress guidance.

3 open findings

What changed in this PR

Adds private-network ingress by publishing guest TCP/UDP listeners through OpenVMM while preserving host-loopback and egress policies.

Changes:

Pins OpenVMM’s ingress implementation and adds the guest listener announcer.
Enables ingress policy translation for ACI edge sandboxes.
Adds policy scenarios, tests, and documentation.

File	Description
`openvmm`	Pins the OpenVMM listener-publication implementation.
`guest/common/nvx-ingress.c`	Announces eligible guest listeners.
`guest/common/init`	Starts the announcer for ingress-allow boots.
`docker/Dockerfile`	Builds the static announcer helper.
`scripts/nvx_tools/build_constants.py`	Adds the helper to guest builds.
`scripts/nvx_tools/microvm_tests.py`	Adds the ingress policy scenario.
`scripts/nvx_tools/microvm_test_scripts/private-network-ingress.sh.in`	Implements guest-side scenario checks.
`scripts/test_microvm_tests.py`	Tests scenario orchestration and scripts.
`scripts/test_nvx_tools.py`	Verifies helper build configuration.
`aci_edge_sandboxes/src/openvmm/mod.rs`	Advertises ingress-allow capability.
`aci_edge_sandboxes/src/openvmm/network.rs`	Passes ingress policy to OpenVMM.
`aci_edge_sandboxes/tests/openvmm_fake.rs`	Tests generated launch arguments.
`aci_edge_sandboxes/tests/openvmm_e2e.rs`	Exercises workload listener publication.
`aci_edge_sandboxes/README.md`	Documents sandbox ingress behavior.
`doc/usage.md`	Updates CLI option references.
`doc/run.md`	Documents private-network ingress operation.
`doc/design/validation.md`	Records validation coverage.
`doc/design/machine-and-device-abi.md`	Defines the announcement ABI.
`doc/design/integrated-implementation-status.md`	Updates implementation status.
`doc/design/current-limits.md`	Revises networking limitations.
`doc/design/cold-boot.md`	Documents the new kernel token.

🧠 Review effort: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Copilot

🟡 Changes recommended

Per-socket IPv6 behavior is classified incorrectly, and the new traffic scenario contains a synchronization race.

2 open findings

3 resolved since last review

🧠 Review effort: Balanced

The portable profile rejected --network-ingress allow before launch, because its user-mode NAT had no way to reach a guest listener: every inbound connection needs a host socket, and Consomme bound only the ports that explicit --host-loopback-forward options named. MXC's network.ingress.default: allow means that clients on the host's private networks reach every port on which the workload listens, without a port list, while the internet never reaches it and host loopback stays a separate decision (#275). fc2de0506 (nanvix/openvmm#133) implements that contract. With ingress allow, OpenVMM appends the host-owned virtnet_ingress=allow token, which callers may not supply, and the guest is to announce the TCP and UDP ports on which it listens in UDP datagrams to port 1021 of the IPv4 gateway, from a privileged source port so that only guest root can publish. For each announced port and family, OpenVMM listens on that port of every host address and relays only peers whose addresses are not globally reachable, lie outside the guest's own networks, and are not IPv6 link-local; it resets the TCP connections and drops the datagrams of other peers, so internet clients never reach the guest. The host itself, at loopback or one of its own addresses, is admitted only with explicit --host-loopback allow, which ingress allow now accepts without forwards. Replies to admitted peers pass whatever the egress policy, which admits a TCP segment other than an initial SYN, or a UDP datagram, only when the endpoint carries its flow. Ingress allow records the ingress action in canonical policy encoding version 5, leaving the digests of existing policies unchanged, and snapshot capture and restore reject it, because its listeners are live host attachments. The run guide gains a private-network ingress section, the usage reference describes --network-ingress and --host-loopback for run and sandbox, the cold-boot and machine ABI designs document the token, the announcement, and the policy, and the current limits and the implementation status describe what ingress admits and refuses. The directional-network-policy scenario no longer expects OpenVMM to reject ingress allow, and the validation design no longer lists that check. The guest does not announce its listeners until the next commit adds its announcer, so ingress allow publishes nothing yet. nvx.py verify passes with this pin. Part of #275. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

🟡 Changes recommended

UDP state is keyed only by port, allowing replacement or unrelated sockets to bypass settling or keep stale publications.

2 open findings

2 resolved since last review

🧠 Review effort: Balanced

Copilot

🟡 Changes recommended

UDP identity merging can withdraw a still-active address family, and the PR validation metadata does not match the current head.

2 open findings

1 resolved since last review

Key UDP publication state by socket identity, not port

🧠 Review effort: Balanced

Copilot

🔵 Needs a closer look

The security-sensitive networking change depends on an open OpenVMM PR, has in-progress exact-head CI, and lacks automated helper-state coverage.

1 open finding

Add automated coverage for listener state machine edge cases · New

2 resolved since last review

🧠 Review effort: Balanced

The pinned OpenVMM publishes the guest's listeners to the host's private networks when the kernel command line carries virtnet_ingress=allow, but only the listeners that the guest announces to its gateway (#275). nvx-ingress, a static musl helper beside nvx-time and nvx-exit, reads the kernel's TCP and UDP socket tables every 100 milliseconds. A TCP socket in the LISTEN state is a listener, and so is an unconnected UDP socket that persists for a second, which skips short-lived client sockets, such as those of DNS queries. Each UDP socket settles on its own, identified by its inode in the socket tables, and stays announced while it exists, even once a server connects it to its client as busybox nc -u -l does; another socket that reuses a port, such as a client's, settles anew, and a port announces the families of its announced sockets. Sockets bound to loopback addresses, or to IPv6 link-local ones, are unreachable through the gateway. A TCP listener on :: also takes IPv4 connections unless it is IPv6-only, which the helper reads from the kernel's socket diagnostics (INET_DIAG_SKV6ONLY) on every scan; a UDP socket on ::, which the guest kernel's diagnostics do not cover, follows net.ipv6.bindv6only. It sends the canonical announcement to UDP port 1021 of the IPv4 gateway from the highest free port from 1023 down to 512, which only root can bind, whenever the set changes and at least every 2 seconds, and marks the listeners beyond the 64 that an announcement carries as omitted. The common init starts it in the background, as root and with standard input from /dev/null, after configuring the network when the token is present, and reports a network error when the helper or the IPv4 gateway is missing. The Alpine and Ubuntu builds compile it with the other musl helpers, and the Azure Linux Dockerfile compiles it in the same loop, which a unit test now keeps in step with the musl helper list. Its test subcommand runs numbered snapshots of the socket tables, net.ipv6.bindv6only, and a socket diagnostics dump through the same scans, and prints each announcement. scripts/test_guest_ingress.py, which validate-nvx now runs, compiles the helper on Linux and checks the announcement encoding against a golden vector, which listeners are reachable through the gateway, the IPv6-only classification and its fallback, malformed diagnostics, UDP settling, socket identity, and per-socket families, and the truncation to 64 listeners. The diagnostics query times out after a scan interval, so a dump that never ends cannot stall the scans. Part of #275. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The OpenVMM backend rejected network.ingress.default: allow with policy_validation before provisioning, because OpenVMM could not publish a workload's listeners (#275). The pinned OpenVMM and guest publish them, so the backend now reports the ingress_allow capability and passes the ingress default to OpenVMM as --network-ingress beside the egress posture, with host loopback still denied. A policy that denies egress without allow rules still attaches the network device when it allows ingress. hostLoopback: allow stays unsupported, so the fake OpenVMM test that covered the ingress rejection now checks that rejection instead, and the launch-arguments test checks the ingress arguments under denied egress. A unit test covers the OpenVMM options for ingress allow under egress deny. The new end-to-end test private_network_ingress_publishes_workload_listeners runs nc -l in a sandbox that denies egress and allows ingress, waits until the host holds the workload's port, requires the host's own connection to be refused before it reaches the workload, waits until the closed listener's port is withdrawn, and checks that ingress deny publishes nothing. The README's policy honor matrix and network rules, and the backend's documentation, describe what ingress admits and refuses. Part of #275. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The new private-network-ingress scenario, part of the default suite, boots the guest three times with busybox nc TCP and UDP servers on one port and checks from the host what each policy lets through (#275): - With egress denied and ingress and host loopback allowed, the port is published for TCP and UDP on every host IPv4 address. The host exchanges a line with each server over 127.0.0.1, gets the UDP reply from the port that it reached, and sees the TCP port withdrawn once its server exits, while the guest still cannot connect to a host listener of its own. - With host loopback denied, the host's connections from loopback and from its own addresses are reset before they reach the guest, and its datagrams go unanswered. - With ingress denied, nothing is published, and the host's connection fails. The scenario then requires OpenVMM to reject a snapshot destination with ingress allowed before boot. The guest script checks that the virtnet_ingress token and the announcer match the policy before it starts its servers, and that each server received exactly what the policy admits. A single host has no private-network peer other than itself, so the pinned OpenVMM's unit tests classify the peers of other hosts. Unit tests run the guest script under a POSIX shell with stubbed servers and socket tables for each policy and failure, check the policies and the snapshot rejection of the scenario, and check the host's publication probe. The validation design describes the coverage. Fixes #275. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

🔵 Needs a closer look

The security-sensitive cross-platform networking change depends on a large, currently open and blocked OpenVMM pull request requiring final human review.

0 open findings

1 resolved since last review

Add automated coverage for listener state machine edge cases

🧠 Review effort: Balanced

Copilot AI balanced review requested due to automatic review settings October 11, 2026 20:06

Pedro Henrique Penna (ppenna) mentioned this pull request Oct 11, 2026

microVM: publish guest listeners to the host's private networks nanvix/openvmm#133

Open

Copilot started reviewing on behalf of Pedro Henrique Penna (ppenna) October 11, 2026 20:06 View session