sam-node: tools/call succeeds on the backend but consumer sees EOF (producer->consumer relay drops the response)

[fer-marino]

Environment

• SAM v0.1.0-alpha.9 (Windows x86_64 binaries)
• Windows 11, local sam-one standalone mesh (control plane + router, sqlite backend)
• Two sam-node instances joined via bootstrap token, --allow-loopback
• Producer node's sam-node.yaml registers a type: mcp service via command (stdio subprocess)

What's broken

A tools/call (and tools/list) issued from a consumer node via call_remote_tool / find_remote_tools against a service on a different node consistently fails with:

failed after 3 retries: failed to call tool mcp://<service>/<tool>: calling "tools/call": EOF

This reproduces with two independent clients (the bundled mcp-client.exe, and Python's langchain-mcp-adapters talking to the consumer node's /mcp endpoint) - not a client bug.

Root-caused to the producer→consumer response relay, not the backend

Instrumented the command-spawned subprocess (a stdio<->HTTP relay to a warm backend) to log every raw line in/out with timestamps. Evidence:

1. The subprocess produces the correct tools/call response, twice (once per retry), matched to the request:

   IN  {"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"...","arguments":{...}}}
   OUT ok {"jsonrpc":"2.0","id":3,"result":{"content":[{"text":"...correct payload..."}],"isError":false,...}}
   

2. The producer's own log shows a clean write-back to the consumer at the exact moment of that success, no error:

   2026-09-09T09:17:09.770+0200  Stream Accounting  {"target": "mcp://<service>", "bytes_read": 1218, "bytes_written": 834}
   

3. The consumer logs EOF 1ms later, for the same call:

   2026-09-09T09:17:09.771+0200  [MCP] Tool call failed, retrying in 2s: ... calling "tools/call": EOF
   

Repeats identically on the retry 2s later. The producer believes it wrote a complete response; the consumer sees EOF at essentially the same instant. The backend, the subprocess relay, and the client are demonstrated correct by direct log evidence - the gap is specifically in the producer→consumer P2P response relay (or the consumer's read of it).

Positive control (rules out "your backend is unusual")

Tried the same pattern fronting @modelcontextprotocol/server-everything (the reference server sam-node skill show itself references, mcp://everything/get-sum) instead of our own backend:

• Direct command: [npx, -y, "@modelcontextprotocol/server-everything"] (no relay): failed registration outright - npx takes ~4s to start even warm, exceeding the backend health-check deadline. (Separately worth noting: this deadline appears to have no configurable flag, and is tight enough that even SAM's own reference example can't reliably meet it on Windows.)
• Via the same relay pattern, fronting a warm server-everything instance: different failure signature during health-check - unmarshaling jsonrpc message: EOF on the first attempt, then HTTP Error 400: Bad Request on retries.

Two different backends, two different but both-broken symptoms around retries/session handling for command-spawned services.

Ask

Would appreciate a pointer to whether this is a known issue (didn't find an obvious match in the open issues) and, if useful, I'm happy to share the full wire-level debug log and minimal repro scripts (a ~50-line stdlib-only stdio<->HTTP relay + sam-node.yaml).

Tested on Windows only - haven't yet confirmed whether this reproduces on Linux.

[fer-marino]

Root cause

Found it, with live debug-level evidence, not just correlation. Reproduced against a real local mesh (control plane + two nodes + a real HTTP MCP backend) with --log-level debug. The decisive log sequence from the provider node, in order:

DEBUG mcp_service.go:215 [MCPService] stvv-compliance-docs: backend read error: unmarshaling jsonrpc message: EOF
INFO  middleware.go:98    Stream Accounting {"bytes_read": 1218, "bytes_written": 834}
DEBUG mcp_service.go:190  [MCPService] stvv-compliance-docs: client read error: stream reset (local): code: 0x0: transport error: stream 4 canceled by local with error code 0

HandleStreamPassThrough (internal/node/mcp_service.go) is a dumb bidirectional pipe between the client-facing P2P stream and the backend connection, using a single-receive errc channel: the first error from either direction tears the whole thing down, via the function's top-level defer s.Close().

The backend here is a one-shot HTTP-style MCP server: it answers exactly one request, then its connection is done. So after successfully writing the 834-byte response to the client (bytes_written: 834 above), the backend-read goroutine loops back, calls Read again, and gets a completely normal EOF - not a fault. That EOF is treated as fatal, triggers <-errc, and the deferred s.Close() fires right behind the write that had just reported success.

network.Stream.Close's own doc is explicit about why this is unsafe:

Close may be asynchronous and does not guarantee receipt of the data... If acknowledgment is required, the caller must call CloseWrite, then wait on the stream for a response (or an EOF), then call Close().

On the QUIC transport specifically, Close() also cancels the read side over the wire - CancelRead sends a STOP_SENDING frame to the peer (p2p/transport/quic/stream.go in go-libp2p: Close() = CancelRead(0) + graceful Stream.Close()). A plain TCP+yamux stream's CloseRead, by contrast, is documented as local-only ("Remote is not notified"). That's consistent with what's reported here: the producer's write reports success, but the response never fully reaches the consumer, which sees EOF in its place.

Fix

Opened #379: the backend leg ending now only half-closes our write side to the client (s.CloseWrite(), not a full Close()) and stops the backend->client relay; it no longer touches the read side or frees the stream. The client leg - the client itself finishing its read and hanging up, or a genuine transport error - is what triggers the final close, bounded by a 5s drain timeout so a client that never hangs up can't leak the stream.

One caveat I want to be upfront about: I added a regression test that drives 50 real CallTool round trips over a real local QUIC connection between two nodes, specifically because a TCP+yamux stream or a mocknet-backed one structurally can't exercise this (see PR description for why). It passes reliably with the fix, but run against the pre-fix code it did not reproduce the race on loopback across 1500+ iterations - loopback has essentially no latency/jitter, which likely closes the window that a real multi-host deployment (or a relayed connection, like the one in this report) would have. So the test proves the new code is correct and follows network.Stream's documented safe-shutdown sequence, but it isn't proof it reproduces this exact wire-level loss. If anyone who hit this can verify against the original repro conditions, that'd be the strongest confirmation.