feat(transport): expose http2_max_concurrent_reset_streams on Endpoint by adriangb · Pull Request #2969 · grpc/grpc-rust

adriangb · 2026-10-28T15:27:34Z

Motivation

A long-lived client Channel that drops many streaming responses before their end (a cancelled query, a client that went away) resets each stream. The server usually still has DATA in flight for those streams, up to the stream window. h2 ignores DATA for a reset stream that it still remembers. By default it remembers at most 50 reset streams. DATA for a reset stream that h2 has forgotten makes it reset the stream again and count that toward max_local_error_reset_streams (1024). At the limit, h2 closes the connection with GOAWAY(ENHANCE_YOUR_CALM, "too_many_internal_resets"), and every in-flight RPC on it fails.

The fix is to have h2 remember more reset streams, so that late DATA is ignored and not counted. Go's x/net/http2, grpc-go and Netty all ignore late DATA this way. hyper's client builder has max_concurrent_reset_streams in every 1.x release, but Endpoint does not expose it. So today a client has to drop down to hyper's HTTP/2 builder and give up Endpoint/Channel to set it.

Solution

Add Endpoint::http2_max_concurrent_reset_streams(usize). When set, it is passed to hyper's client HTTP/2 builder. When unset, h2's default is unchanged, so this is non-breaking. The hyper requirement does not change.
Add tests/integration_tests/tests/http2_reset_streams.rs. A raw HTTP/2 server answers 100 requests with headers only. The client drops each response, and the server then sends 40 late 1 KiB DATA frames on every reset stream.
- With the default, the client closes the connection with GOAWAY("too_many_internal_resets"). This shows the test reproduces the failure. On Windows, the client's close can reset the connection before the server reads the GOAWAY, so there the test also accepts a reset.
- With http2_max_concurrent_reset_streams(10_000), the connection stays open.
- With the option left out of the connection builder, that test fails.

A follow-up PR, #2972, adds http2_reset_stream_duration and http2_max_local_error_reset_streams (the client counterpart of the Server option, from #2708). They need hyper 1.10 and 1.9, so that PR also raises the hyper requirement.

The branch is based on master. I'm happy to retarget it to v0.14.x if you'd prefer that.

Adds Endpoint::http2_max_concurrent_reset_streams and Endpoint::http2_reset_stream_duration, which set hyper's client options of the same name. A long-lived channel that drops many response streams with DATA still in flight can then make h2 remember those resets long enough to ignore the late DATA, rather than count it toward the local error reset limit and close the connection with GOAWAY(ENHANCE_YOUR_CALM). Also adds Endpoint::http2_max_local_error_reset_streams, the client-side counterpart of the Server option, so a caller can raise the limit or disable it with None. Unset keeps hyper's default of 1024. This follows grpc#2708. Requires hyper 1.10, which added the client reset_stream_duration option.

On Windows, a write after the client closed the connection resets it, and the reset discards a GOAWAY that the test server did not read yet. The server now writes the late DATA from a separate task and keeps reading.

A client that closes the socket with unread late DATA resets the connection, and Windows then discards the GOAWAY that the test server did not read yet. On Windows, the test that expects the connection to close also accepts that reset.

adriangb · 2026-10-28T17:46:31Z

 # transport
 h2 = {version = "0.4", optional = true}
-hyper = {version = "1", features = ["http1", "http2"], optional = true}
+hyper = {version = "1.10", features = ["http1", "http2"], optional = true}


This probably needs some discussion if we really need it

Moved the bump out of this PR. max_concurrent_reset_streams is in every hyper 1.x release, so this PR keeps hyper = "1". The two options that need hyper 1.9 and 1.10 are now in #2972, which also offers an optional-dependency feature so that only users who enable it need hyper 1.10.

…arate PR The client reset_stream_duration option needs hyper 1.10, and the client max_local_error_reset_streams option needs hyper 1.9. max_concurrent_reset_streams is in every hyper 1.x release, so this PR keeps only that option and the hyper requirement stays at 1.

Adds Endpoint::http2_max_concurrent_reset_streams, which sets hyper's client option of the same name. A long-lived channel that drops many response streams with DATA still in flight can then make h2 remember those resets and ignore the late DATA, rather than count it toward the local error reset limit and close the connection with GOAWAY(ENHANCE_YOUR_CALM). Upstream PR: grpc#2969

adriangb · 2026-10-29T02:52:04Z

Combined into #2972.

Replaces #2969, combined here as suggested. ## Motivation When a client resets many streams, a server can still send DATA for them. h2 remembers only 50 reset streams for 1 s by default. DATA for a forgotten stream counts toward the local error reset limit (1024), and reaching it closes the connection with `GOAWAY(ENHANCE_YOUR_CALM, "too_many_internal_resets")`. `Endpoint` exposes none of these h2 client options today, so a caller must build the connection with hyper directly to change them. ## Solution - Add `Endpoint::http2_max_concurrent_reset_streams(usize)`: how many reset streams h2 remembers. - Add `Endpoint::http2_reset_stream_duration(Duration)`: how long h2 remembers a reset stream. - Add `Endpoint::http2_max_local_error_reset_streams(impl Into<Option<usize>>)`, the client counterpart of the `Server` option, as in #2708 (closed). Unset keeps hyper's default of 1024, and `None` disables the limit. - Raise tonic's `hyper` requirement from `1` to `1.10`, the first release with the client `reset_stream_duration` option.

adriangb added 3 commits October 28, 2026 10:26

test(transport): read while the late DATA goes out

9f35041

On Windows, a write after the client closed the connection resets it, and the reset discards a GOAWAY that the test server did not read yet. The server now writes the late DATA from a separate task and keeps reading.

adriangb commented Oct 28, 2026

View reviewed changes

adriangb changed the title ~~feat(transport): expose h2 reset stream settings on Endpoint~~ feat(transport): expose http2_max_concurrent_reset_streams on Endpoint Oct 28, 2026

adriangb mentioned this pull request Oct 28, 2026

feat(transport): expose h2 reset stream settings on Endpoint #2972

Merged

adriangb closed this Oct 29, 2026

Navigation Menu

Repository navigation

feat(transport): expose http2_max_concurrent_reset_streams on Endpoint - #2969

feat(transport): expose http2_max_concurrent_reset_streams on Endpoint#2969
adriangb wants to merge 4 commits into
grpc:masterfrom
pydantic:client-h2-reset-stream-settings

adriangb commented Oct 28, 2026 •

edited

Loading

Uh oh!

adriangb Oct 28, 2026

Uh oh!

adriangb Oct 28, 2026

Uh oh!

adriangb commented Oct 29, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

adriangb commented Oct 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Solution

Uh oh!

adriangb Oct 28, 2026

Choose a reason for hiding this comment

Uh oh!

adriangb Oct 28, 2026

Choose a reason for hiding this comment

Uh oh!

adriangb commented Oct 29, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

adriangb commented Oct 28, 2026 •

edited

Loading