TCP connections to published ports get reset on networks whose runtime state is stale (ECONNRESET after first data exchange)

[itxtoledo]

Published-port forwarding breaks for containers attached to networks with stale runtime state: TCP connects and the listener accepts, but multi-message connections are reset after the first data exchange (Connection reset by peer, errno 54). Single round-trips sometimes pass; anything with more than one request/response cycle fails. Containers on freshly created networks are unaffected.

Impact

Any protocol that keeps a connection alive and exchanges several messages hangs forever (e.g. TigerBeetle's VSR client: getConnectionStatus() never returns — reproduced with timeouts > 60s). Plain HTTP/1.1 with keep-alive or any request/reply loop breaks.

Reproduction

Echo server (python) mounted into a container:

# /tmp/echo-server.py
import socket, threading
s = socket.socket(); s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind(("0.0.0.0", 9000)); s.listen(16)
def handle(c):
    try:
        while True:
            data = c.recv(65536)
            if not data: break
            c.sendall(data)
    except Exception: pass
    finally: c.close()
while True:
    conn, _ = s.accept()
    threading.Thread(target=handle, args=(conn,), daemon=True).start()

1. Network created days earlier (e.g. a compose project network) — BROKEN:

   mocker network inspect hotgate-panel-default -> Created: 2026-08-15 (4 days old)

   mocker run -d --name echo-broken --network hotgate-panel-default -p 43104:9000 -v /tmp/echo-server.py:/app/echo-server.py:ro python:3.13-alpine python /app/echo-server.py

   Client: 100 sequential round-trips over one connection -> round 0 -> [Errno 54] Connection reset by peer.

2. Freshly created network — WORKS:

   mocker network create fresh-net

   mocker run -d --name echo-fresh --network fresh-net -p 43102:9000 -v /tmp/echo-server.py:/app/echo-server.py:ro python:3.13-alpine python /app/echo-server.py

   Client: 100/100 round-trips OK.

Client used: plain Python socket — connect, sendall(68 bytes), loop recv until the payload length, repeat 100x.

Key observations

• Network membership is the trigger, not how the container was created: same result via mocker run --network X and mocker compose up for service X.
• Internal container-to-container traffic on the broken network works: 50/50 round-trips OK between two containers on the same network via their container IPs — only the host -> published-port path breaks.
• Fresh networks are always fine (compose-created or mocker network create).
• After mocker compose down removed a project's network, the default network's published ports also started resetting (previously working containers broke), suggesting the runtime's host-side forwarder state degrades over time / with network churn, and only fresh networks get a working forwarder.
• Restarting the affected container does NOT fix it; recreating/replacing the network does (containers moved to a fresh network work immediately).

Environment

• mocker 0.9.3 (us/tap/mocker, brew)
• macOS 26+ / Apple Silicon, Apple Containerization runtime
• TigerBeetle 0.17.6 in a container reproduces the same hang (VSR client never completes)

Workaround

Use a freshly created network for containers that need published-port TCP: mocker network create <name> (or let compose create a new project network) and attach with --network <name>.

💘 Generated with Crush

[us]

Thanks for the incredibly detailed report and the echo-server matrix — this made it easy to pinpoint.

mocker delegates all network and published-port plumbing to the Apple Containerization runtime: -p is passed straight through to container run, which wires host forwarding via vmnet, and mocker keeps no per-network forwarder state of its own. The behaviour you describe (fresh networks fine, old networks resetting after the first exchange, container restart not helping but network recreation fixing it) matches a known upstream problem where the runtime's vmnet network reference goes stale after system state changes and is never re-created: apple/container#1321 (see also apple/container#919 and apple/container#794).

So the fix has to land upstream. Until then your workaround is the right one: recreate the network (mocker network create <name>) and attach containers to it. We'll track the upstream issues and pick up the fix as soon as a patched runtime ships.