Skip to content

fix(python-sdk): download files over HTTP/1.1 to avoid h2 too_many_data_frames - #2039

Open
devin-ai-integration[bot] wants to merge 4 commits into
mainfrom
devin/1793031570-h2-empty-data-frames
Open

devin-ai-integration[bot] wants to merge 4 commits into
mainfrom
devin/1793031570-h2-empty-data-frames

Conversation

@devin-ai-integration

@devin-ai-integration devin-ai-integration Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Fixes #2027. Python files.read of files over ~500 KB failed at random with httpx.ReadError ... too_many_data_frames, and files of 4.5 MB or more failed on every read.

Root cause. The sandbox proxy splits envd /files responses into DATA frames with an empty, non-final DATA frame after every ~32 KiB (16384, 16384, 0, 16384, 16384, 0, …). The h2 version that pyqwest 0.12.0 bundles (anuraaga/h2 fork @ 286de9d, which has upstream hyperium/h2 #935/#945) keeps a count of empty DATA frames for the whole lifetime of a connection, limited to MAX_RECV_EMPTY_DATA_FRAMES = 100. Going past it sends a GOAWAY with ENHANCE_YOUR_CALM / too_many_data_frames. The count never goes back down, so:

  • a read of about 3.2 MB or more fails every time,
  • smaller reads fail once the pooled connection has collected 100 empty frames, which is why it looked random. The GOAWAY also kills every other stream multiplexed on that shared connection.

The small-frame byte budget is not the cause: it resolves to 2.5 MB here and was never touched. I confirmed this by building pyqwest against an instrumented h2. Every failure was record len=0 … empty=101 err=true. hyperium/h2 master (v0.4.20+) still has the same lifetime counter, and hyper doesn't let you configure it, so the SDK can't raise the limit.

Fix. Filesystem.read (sync and async, all formats) now downloads over HTTP/1.1 using the existing HTTP/1.1-pinned transport pool, whatever http_version is set to:

ENVD_API_FILE_DOWNLOAD_HTTP_VERSION: HttpVersion = "1.1"   # e2b/envd/api.py

self._envd_api_download  = get_envd_api(cfg, url, http_version=ENVD_API_FILE_DOWNLOAD_HTTP_VERSION)
self._envd_api_streaming = get_envd_api(cfg, url, for_streaming=True, http_version=ENVD_API_FILE_DOWNLOAD_HTTP_VERSION)

get_envd_api gets an http_version passthrough to get_transport. Uploads, RPC (commands output had no empty frames in testing) and health checks stay on HTTP/2. The trade-off: concurrent downloads now use one connection each instead of being multiplexed. The JS SDK is not affected (undici/nghttp2): 0 of 15 reads failed at 2, 4.5 and 8 MB.

Usage doesn't change: await sandbox.files.read(path, "bytes") now works for large files.

Verification

  • Live reproduction from the issue, 10 reads per size. Before: 0.5 MB 1/10, 1 MB 2/10, 2 MB 4/10, 4.5 MB 10/10 failed. After: 0/10 failed at 0.25, 0.5, 1, 2, 4.5 and 8 MB.
  • New offline test tests/test_envd_file_download_empty_data_frames.py: an HTTP/2 server sends 150 empty DATA frames, the way the proxy does, and also answers HTTP/1.1. On main the http_version="2" cases fail with too_many_data_frames; with the fix all of them pass (bytes, text, stream, and async stream with stream_idle_timeout).
  • Offline Python unit tests pass, as do the live files/test_read.py tests (sync and async). pnpm run format, lint and typecheck are clean.

Follow-ups outside this repo: it would help if the sandbox proxy stopped sending empty non-final DATA frames, and/or if pyqwest/h2 made the empty-frame limit per stream or configurable. Either one would let downloads go back to HTTP/2.

Link to Devin session: https://app.devin.ai/sessions/550e475556994f51b1ba7970bc4590f5
Open in Devin Desktop: https://app.devin.ai/desktop/session/550e475556994f51b1ba7970bc4590f5?variant=devin


Devin Review

…ta_frames

The sandbox proxy pads file responses with an empty non-final DATA frame
every ~32 KiB. h2 (bundled by pyqwest) counts those over a connection's
lifetime and closes it with ENHANCE_YOUR_CALM after 100, failing reads of
a few MB every time and smaller ones once a pooled connection accumulates
the count. Route files.read over the HTTP/1.1 envd transport.

Fixes #2027

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

@changeset-bot

changeset-bot Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 978aead

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 2 packages
Name Type
@e2b/python-sdk Patch
@e2b/code-interpreter-python Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-26T16:25:47.989129Z ae4ed1a PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

devin-ai-integration[bot]

This comment was marked as resolved.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Devin Review

@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Package Artifacts

Built from b05fcaa. Download artifacts from this workflow run.

JS SDK (e2b@2.61.1-devin-1793031570-h2-empty-data-frames.0, with @e2b/dockerfile-utils@0.1.2-devin-1793031570-h2-empty-data-frames.0):

npm install ./e2b-dockerfile-utils-0.1.2-devin-1793031570-h2-empty-data-frames.0.tgz ./e2b-2.61.1-devin-1793031570-h2-empty-data-frames.0.tgz

CLI (@e2b/cli@2.22.4-devin-1793031570-h2-empty-data-frames.0):

npm install ./e2b-cli-2.22.4-devin-1793031570-h2-empty-data-frames.0.tgz

Code Interpreter JS SDK (@e2b/code-interpreter@2.9.1-devin-1793031570-h2-empty-data-frames.0):

npm install ./e2b-code-interpreter-2.9.1-devin-1793031570-h2-empty-data-frames.0.tgz

Desktop JS SDK (@e2b/desktop@2.5.1-devin-1793031570-h2-empty-data-frames.0):

npm install ./e2b-desktop-2.5.1-devin-1793031570-h2-empty-data-frames.0.tgz

Python SDK (e2b==2.61.0+devin.1793031570.h2.empty.data.frames, with e2b-dockerfile-utils==0.1.1+devin.1793031570.h2.empty.data.frames):

pip install ./e2b_dockerfile_utils-0.1.1+devin.1793031570.h2.empty.data.frames-py3-none-any.whl ./e2b-2.61.0+devin.1793031570.h2.empty.data.frames-py3-none-any.whl

Code Interpreter Python SDK (e2b-code-interpreter==2.11.5+devin.1793031570.h2.empty.data.frames):

pip install ./e2b_code_interpreter-2.11.5+devin.1793031570.h2.empty.data.frames-py3-none-any.whl

Desktop Python SDK (e2b-desktop==2.7.2+devin.1793031570.h2.empty.data.frames):

pip install ./e2b_desktop-2.7.2+devin.1793031570.h2.empty.data.frames-py3-none-any.whl

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.

Besides the inline finding, I checked the sync streaming download path, which the diff hunks don't show. It always goes through _envd_api_streaming, which is now pinned to HTTP/1.1, so sync and async reads both download over HTTP/1.1 in every format.

Extended reasoning...

The sync Filesystem.read stream branch builds and sends its request on self._envd_api_streaming, and this PR now creates that client with http_version=ENVD_API_FILE_DOWNLOAD_HTTP_VERSION. That makes it match the async version, where both stream branches end up on a client pinned to HTTP/1.1.

# pyqwest) closes a connection with ENHANCE_YOUR_CALM ("too_many_data_frames")
# after 100 such frames over its lifetime — failing the download and every
# other stream multiplexed on that connection.
ENVD_API_FILE_DOWNLOAD_HTTP_VERSION: HttpVersion = "1.1"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 (optional) High-concurrency processes now stall or time out on files.read past 200 concurrent downloads per edge address. Before, HTTP/2 multiplexing allowed about 20,000. The pinned HTTP/1.1 pools keep max_connections_per_address=max_connections (default 200). That cap was sized for 100 streams per connection, and every sandbox resolves to the shared envd edge. Fix: size the HTTP/1.1 download pools separately from the HTTP/2 cap (their own larger limit or env var). Update the README section that says files traffic shares the HTTP/2 pool. Non-stream reads then fail after REQUEST_TIMEOUT 60s, and streamed reads wait with no default bound.

Why this was flagged

An eval-style process runs sandbox.files.read concurrently across many sandboxes. Before this change each read was one HTTP/2 stream. pyqwest packs up to 100 streams per connection, capped at 200 connections per resolved address (e2b/api/init.py:98), so about 20,000 concurrent reads were possible. Now ENVD_API_FILE_DOWNLOAD_HTTP_VERSION = "1.1" (e2b/envd/api.py:31) routes them to pools built with HTTPVersion.HTTP1 and the same max_connections_per_address=max_connections (client_async/init.py:131, client_sync/init.py:136). Each download holds a whole connection, and all sandboxes share the edge address. So the 201st concurrent non-stream read waits for a connection and fails once the 60s REQUEST_TIMEOUT whole-request deadline passes. A streamed read sends stream_timeout=None unless request_timeout is set (sandbox_async/filesystem/filesystem.py:243), so it waits until another caller closes its reader. README.md:71 still says files traffic shares the HTTP/2 pool.

Verification: envd/api.py:31 sets ENVD_API_FILE_DOWNLOAD_HTTP_VERSION = "1.1". That builds a separate SyncHTTPTransport with max_connections_per_address=max_connections (client_sync __init__.py:136) and http_version=HTTPVersion.HTTP1 (line 141). max_connections defaults to 200 (e2b/api/__init__.py:98). HTTP/1.1 carries one request per connection.

…indows

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Python SDK files.read of files over ~500 KB fails at random with too_many_data_frames (pyqwest h2)

0 participants