Repository navigation
fix(js-sdk): send envd file uploads over HTTP/1.1, retire poisoned HTTP/2 dispatchers - #2086
Conversation
…TP/2 dispatchers Node's HTTP/2 client charges each queued request body in full against the session's 10 MB memory budget and resets whichever stream's response headers arrive while it is exhausted (NGHTTP2_ENHANCE_YOUR_CALM). Since undici 8 every envd REST request of a process multiplexes over one session, so concurrent or large files.write() uploads reset each other and the reads next to them (SDK-425). files.write() now POSTs /files through the HTTP/1.1 envd fetcher — one request per pooled connection, no shared budget, as undici 7 effectively gave 2.27.1 — while everything else stays on HTTP/2. A dispatcher whose session Node reports as out of memory is retired and replaced; the failed request is not replayed since envd may already have applied it. Co-Authored-By: mish@e2b.dev <mish@e2b.dev>
|
I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".
|
🦋 Changeset detectedLatest commit: 8256418 The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Package ArtifactsBuilt from 7869b88. Download artifacts from this workflow run. JS SDK ( npm install ./e2b-dockerfile-utils-0.1.2-devin-1793727850-envd-uploads-http1.0.tgz ./e2b-2.62.2-devin-1793727850-envd-uploads-http1.0.tgzCLI ( npm install ./e2b-cli-2.23.1-devin-1793727850-envd-uploads-http1.0.tgzCode Interpreter JS SDK ( npm install ./e2b-code-interpreter-2.9.2-devin-1793727850-envd-uploads-http1.0.tgzDesktop JS SDK ( npm install ./e2b-desktop-2.5.1-devin-1793727850-envd-uploads-http1.0.tgzPython SDK ( pip install ./e2b_dockerfile_utils-0.1.1+devin.1793727850.envd.uploads.http1-py3-none-any.whl ./e2b-2.62.0+devin.1793727850.envd.uploads.http1-py3-none-any.whlCode Interpreter Python SDK ( pip install ./e2b_code_interpreter-2.11.7+devin.1793727850.envd.uploads.http1-py3-none-any.whlDesktop Python SDK ( pip install ./e2b_desktop-2.7.2+devin.1793727850.envd.uploads.http1-py3-none-any.whl |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
🔍 Devin Review: 1 flag
Not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)
Uploads shared the 10-connection envd REST pool, so more than ten concurrent files.write() calls in a process queued in undici while their request timeout kept running. They now get a pool of their own (E2B_ENVD_UPLOAD_CONNECTIONS, default 50). The ConnectionOpts.httpVersion docs state the upload exception. Co-Authored-By: mish@e2b.dev <mish@e2b.dev>
Summary
Fixes SDK-425: concurrent
sandbox.files.write()calls fail withTypeError: fetch failed→NGHTTP2_ENHANCE_YOUR_CALMsincee2b≥ 2.35 (undici 8). Simpler alternative to #2083 (byte budget + one-off sessions), same measurements.Cause. The RST is sent by Node's own http2 client, not by E2B infra. Node queues each buffered request body in full against the session's 10 MB
maxSessionMemory; if a response HEADERS frame arrives on any stream while that budget is exhausted, Node resets that stream withENHANCE_YOUR_CALM. undici 7 kept one request per session (so our 10-connection pool = 10 budgets); undici 8 multiplexes, so every envd REST request in the process shares one session and one budget — parallel or large uploads reset each other and the reads next to them. Streams destroyed before their body is flushed (timeouts, aborts, killed sandboxes) additionally leak their queued bytes on Node ≤ 24 (fixed onmainby nodejs/node#67113), so a session degrades until every small write on it fails — the customer's "climbing since Monday".Fix (JS SDK only — Python uses pyqwest/hyper, not Node http2):
FilesystemPOSTs/files(octet-stream, streamed, gzip and multipart uploads) throughcreateEnvdUploadFetch— an HTTP/1.1 dispatcher pool of its own (E2B_ENVD_UPLOAD_CONNECTIONS, default 50, likeE2B_ENVD_RPC_CONNECTIONS); everything else (read,makeDir, RPC, health…) stays on HTTP/2, andhttpVersion/E2B_HTTP_VERSIONkeep governing those. TheConnectionOpts.httpVersionJSDoc documents the exception.buildDispatchedFetch(src/undici.ts): a request that fails withERR_HTTP2_STREAM_ERROR+NGHTTP2_ENHANCE_YOUR_CALM/NGHTTP2_INTERNAL_ERROR(isPoisonedHTTP2SessionError) retires that dispatcher and builds a fresh one for subsequent requests. The failed request is not replayed (envd may already have applied the write). Only the dispatcher that served the failing request can replace the current one. Safety net for whatever else might fill a session.No API change.
Measured (built package, real sandboxes, Node 24.22; ALPN confirmed: writes
http/1.1, readsh2):files.read()maxSessionMemory=1, aborted big writes)Upstream: backport request on nodejs/node#67113 (nodejs/node#67113 (comment)) and undici issue nodejs/undici#6149.
Link to Devin session: https://app.devin.ai/sessions/98f09d080013403b954d3f7b07c51b3b
Open in Devin Desktop: https://app.devin.ai/desktop/session/98f09d080013403b954d3f7b07c51b3b?variant=devin
Requested by: @mishushakov