Skip to content

fix(api): keep relaying the request body after the sandbox answers - #5

Merged
matej21 merged 1 commit into
mainfrom
fix/proxy-full-duplex
Oct 24, 2026
Merged

matej21 merged 1 commit into
mainfrom
fix/proxy-full-duplex

Conversation

@matej21

@matej21 matej21 commented Oct 23, 2026

Copy link
Copy Markdown
Member

What

The sandbox proxy now calls http.NewResponseController(w).EnableFullDuplex() before ReverseProxy.ServeHTTP.

Why

Clients saw envd streams (process.Process/Start) drop mid-response from time to time. On a loaded host it happened twice within a few tens of ms. The client saw socket connection was closed unexpectedly, and edvabe logged this at the same moment:

httputil: ReverseProxy read error during body copy: read tcp <edvabe>-><sandbox>:49983: use of closed network connection

This is golang/go#40747:

  1. ReverseProxy hands the inbound request body to the transport as the outbound body. The transport reads it twice: once for the Content-Length bytes, then once more expecting io.EOF.
  2. envd has the whole message once the Content-Length bytes arrive, so it can answer between those two reads. That needs the host to be busy enough to deschedule the transport's write goroutine.
  3. When the proxied response starts, Go's HTTP/1 server drains and closes the inbound request body, unless the handler opted into full duplex.
  4. The transport's EOF read then fails with http: invalid Read on closed Body. It closes the upstream connection.
  5. The proxy's body copy fails with use of closed network connection. ReverseProxy panics with ErrAbortHandler, and the server drops the client connection mid-stream.

With full duplex the server leaves the body alone, and the transport finishes the request write normally. HTTP/2 requests are always full duplex. On those, EnableFullDuplex returns ErrNotSupported, which is why the error is ignored.

The same class of failure needs only a backend that starts its response before it has read the whole body, so any proxied user-port service could hit it too.

Verification

  • New test TestProxyStreamsResponseWhileRequestBodyInFlight: the backend answers before reading the body, and the client sends the body 10 ms late, 20 attempts.
    • Without the fix it fails on attempt 0 with response = "start\n", unexpected EOF and the log line above.
    • With the fix: go test ./internal/api/ -run TestProxyStreamsResponseWhileRequestBodyInFlight -count=50 passes (1,000 requests).
  • go test ./... and go vet ./... pass. golangci-lint is not installed here, so it was not run.
  • Against real envd in a local sandbox, each case below is 3 runs × 100 process.Process/Start streams from a Bun client. The Go toolchain is patched only for the repro, with a 20 ms sleep before the transport's EOF read (net/http/transfer.go, via go build -overlay). That widens the race window so it shows up every time:
    • v0.4.0 code: 49/100 streams dropped per run, with exactly one use of closed network connection log line per drop.
    • This branch: 0/300 dropped, no log lines.
  • Unpatched toolchain, a user-port backend in a sandbox that answers before reading the body, and a client that sends the body 300 ms late, 40 attempts each:
    • ghcr.io/contember/edvabe:0.4.0: 34/40 truncated.
    • This branch: 0/40 truncated.
  • No regressions on this branch with the unpatched toolchain:
    • 200 concurrent exec streams.
    • 3 Bun worker threads × 90 streams.
    • 20 rounds of 3 MiB multipart upload, download, stat, move and remove.

Not verified: an unmodified-toolchain repro of the envd-specific timing. It depends on host scheduling, and it reproduces only on busy CI runners.

🤖 Generated with Claude Code

Go's HTTP/1 server drains and closes the inbound request body when the
response starts, unless the handler enables full duplex. ReverseProxy
hands that same body to the transport as the outbound request body, and
the transport reads it twice: the Content-Length bytes, then once more
expecting io.EOF. When the backend answers in between (envd's Connect
process.Process/Start does when the host is loaded), the server closes
the body first, the transport's request write fails, it closes the
upstream connection, and ReverseProxy aborts the half-sent response:

  httputil: ReverseProxy read error during body copy: read tcp ...:
  use of closed network connection

The client sees the stream cut off mid-response. See golang/go#40747.

Enable full duplex on every proxied request so the body stays readable
for the transport after the response begins.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@matej21
matej21 marked this pull request as ready for review October 24, 2026 10:10
@matej21
matej21 merged commit a07806c into main Oct 24, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant