Repository navigation
fix(python-sdk): download files over HTTP/1.1 to avoid h2 too_many_data_frames - #2039
devin-ai-integration[bot] wants to merge 4 commits into
Conversation
…ta_frames The sandbox proxy pads file responses with an empty non-final DATA frame every ~32 KiB. h2 (bundled by pyqwest) counts those over a connection's lifetime and closes it with ENHANCE_YOUR_CALM after 100, failing reads of a few MB every time and smaller ones once a pooled connection accumulates the count. Route files.read over the HTTP/1.1 envd transport. Fixes #2027 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".
|
🦋 Changeset detectedLatest commit: 978aead The changes in this PR will be included in the next version bump. This PR includes changesets to release 2 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Package ArtifactsBuilt from b05fcaa. Download artifacts from this workflow run. JS SDK ( npm install ./e2b-dockerfile-utils-0.1.2-devin-1793031570-h2-empty-data-frames.0.tgz ./e2b-2.61.1-devin-1793031570-h2-empty-data-frames.0.tgzCLI ( npm install ./e2b-cli-2.22.4-devin-1793031570-h2-empty-data-frames.0.tgzCode Interpreter JS SDK ( npm install ./e2b-code-interpreter-2.9.1-devin-1793031570-h2-empty-data-frames.0.tgzDesktop JS SDK ( npm install ./e2b-desktop-2.5.1-devin-1793031570-h2-empty-data-frames.0.tgzPython SDK ( pip install ./e2b_dockerfile_utils-0.1.1+devin.1793031570.h2.empty.data.frames-py3-none-any.whl ./e2b-2.61.0+devin.1793031570.h2.empty.data.frames-py3-none-any.whlCode Interpreter Python SDK ( pip install ./e2b_code_interpreter-2.11.5+devin.1793031570.h2.empty.data.frames-py3-none-any.whlDesktop Python SDK ( pip install ./e2b_desktop-2.7.2+devin.1793031570.h2.empty.data.frames-py3-none-any.whl |
There was a problem hiding this comment.
Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.
Besides the inline finding, I checked the sync streaming download path, which the diff hunks don't show. It always goes through _envd_api_streaming, which is now pinned to HTTP/1.1, so sync and async reads both download over HTTP/1.1 in every format.
Extended reasoning...
The sync Filesystem.read stream branch builds and sends its request on self._envd_api_streaming, and this PR now creates that client with http_version=ENVD_API_FILE_DOWNLOAD_HTTP_VERSION. That makes it match the async version, where both stream branches end up on a client pinned to HTTP/1.1.
| # pyqwest) closes a connection with ENHANCE_YOUR_CALM ("too_many_data_frames") | ||
| # after 100 such frames over its lifetime — failing the download and every | ||
| # other stream multiplexed on that connection. | ||
| ENVD_API_FILE_DOWNLOAD_HTTP_VERSION: HttpVersion = "1.1" |
There was a problem hiding this comment.
🟡 (optional) High-concurrency processes now stall or time out on files.read past 200 concurrent downloads per edge address. Before, HTTP/2 multiplexing allowed about 20,000. The pinned HTTP/1.1 pools keep max_connections_per_address=max_connections (default 200). That cap was sized for 100 streams per connection, and every sandbox resolves to the shared envd edge. Fix: size the HTTP/1.1 download pools separately from the HTTP/2 cap (their own larger limit or env var). Update the README section that says files traffic shares the HTTP/2 pool. Non-stream reads then fail after REQUEST_TIMEOUT 60s, and streamed reads wait with no default bound.
Why this was flagged
An eval-style process runs sandbox.files.read concurrently across many sandboxes. Before this change each read was one HTTP/2 stream. pyqwest packs up to 100 streams per connection, capped at 200 connections per resolved address (e2b/api/init.py:98), so about 20,000 concurrent reads were possible. Now ENVD_API_FILE_DOWNLOAD_HTTP_VERSION = "1.1" (e2b/envd/api.py:31) routes them to pools built with HTTPVersion.HTTP1 and the same max_connections_per_address=max_connections (client_async/init.py:131, client_sync/init.py:136). Each download holds a whole connection, and all sandboxes share the edge address. So the 201st concurrent non-stream read waits for a connection and fails once the 60s REQUEST_TIMEOUT whole-request deadline passes. A streamed read sends stream_timeout=None unless request_timeout is set (sandbox_async/filesystem/filesystem.py:243), so it waits until another caller closes its reader. README.md:71 still says files traffic shares the HTTP/2 pool.
Verification: envd/api.py:31 sets ENVD_API_FILE_DOWNLOAD_HTTP_VERSION = "1.1". That builds a separate SyncHTTPTransport with max_connections_per_address=max_connections (client_sync __init__.py:136) and http_version=HTTPVersion.HTTP1 (line 141). max_connections defaults to 200 (e2b/api/__init__.py:98). HTTP/1.1 carries one request per connection.
…indows Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Summary
Fixes #2027. Python
files.readof files over ~500 KB failed at random withhttpx.ReadError ... too_many_data_frames, and files of 4.5 MB or more failed on every read.Root cause. The sandbox proxy splits envd
/filesresponses into DATA frames with an empty, non-final DATA frame after every ~32 KiB (16384, 16384, 0, 16384, 16384, 0, …). The h2 version thatpyqwest0.12.0 bundles (anuraaga/h2 fork @286de9d, which has upstream hyperium/h2 #935/#945) keeps a count of empty DATA frames for the whole lifetime of a connection, limited toMAX_RECV_EMPTY_DATA_FRAMES = 100. Going past it sends a GOAWAY withENHANCE_YOUR_CALM/too_many_data_frames. The count never goes back down, so:The small-frame byte budget is not the cause: it resolves to 2.5 MB here and was never touched. I confirmed this by building pyqwest against an instrumented h2. Every failure was
record len=0 … empty=101 err=true. hyperium/h2master(v0.4.20+) still has the same lifetime counter, and hyper doesn't let you configure it, so the SDK can't raise the limit.Fix.
Filesystem.read(sync and async, all formats) now downloads over HTTP/1.1 using the existing HTTP/1.1-pinned transport pool, whateverhttp_versionis set to:get_envd_apigets anhttp_versionpassthrough toget_transport. Uploads, RPC (commands output had no empty frames in testing) and health checks stay on HTTP/2. The trade-off: concurrent downloads now use one connection each instead of being multiplexed. The JS SDK is not affected (undici/nghttp2): 0 of 15 reads failed at 2, 4.5 and 8 MB.Usage doesn't change:
await sandbox.files.read(path, "bytes")now works for large files.Verification
tests/test_envd_file_download_empty_data_frames.py: an HTTP/2 server sends 150 empty DATA frames, the way the proxy does, and also answers HTTP/1.1. Onmainthehttp_version="2"cases fail withtoo_many_data_frames; with the fix all of them pass (bytes, text, stream, and async stream withstream_idle_timeout).files/test_read.pytests (sync and async).pnpm run format,lintandtypecheckare clean.Follow-ups outside this repo: it would help if the sandbox proxy stopped sending empty non-final DATA frames, and/or if pyqwest/h2 made the empty-frame limit per stream or configurable. Either one would let downloads go back to HTTP/2.
Link to Devin session: https://app.devin.ai/sessions/550e475556994f51b1ba7970bc4590f5
Open in Devin Desktop: https://app.devin.ai/desktop/session/550e475556994f51b1ba7970bc4590f5?variant=devin