waltz: fix h2 and gRPC stream lifecycle and flow control

[esemeniuc]

A gRPC header timeout sends RST_STREAM and releases the stream. If the response HEADERS arrive afterwards, fd_h2 treats them as a connection error and drops every other request on the connection. Reviewing that path turned up related bugs in fd_h2 and fd_grpc_client, all fixed here.

fd_h2

1. Late HEADERS on a closed stream

   • Before: PROTOCOL_ERROR on a released local stream, dropping the whole connection. On a CLOSED stream the app kept in its map, the block reached the app, or caused a connection error if it carried END_STREAM.
   • After: the block is dropped. The new fd_hpack_skip still HPACK-decodes it incrementally across HEADERS and CONTINUATION, without keeping the decoded fields, and invalid HPACK is COMPRESSION_ERROR. No HPACK state is needed between blocks because fd_h2 advertises SETTINGS_HEADER_TABLE_SIZE=0. HEADERS on a local ID that was never opened is still PROTOCOL_ERROR.
   • As a server, late HEADERS on a released client stream are dropped the same way. fd_h2 cannot tell such an ID from one the client skipped, so HEADERS on a skipped ID are dropped too, where RFC 9113 §5.1.1 asks for PROTOCOL_ERROR.

2. CONTINUATION for a stream released or aborted mid-block (for example, fd_grpc_client rejecting the first fragment, or, since fix 15, a deadline firing between fragments)

   • Before: one RST_STREAM(INTERNAL_ERROR) per remaining CONTINUATION frame.
   • After: no RST_STREAM; the rest of the block is HPACK-validated and dropped. This also applies when the app aborts the stream with fd_h2_stream_error or fd_h2_stream_reset but keeps it in its map.

3. DATA on a stream that is not open

   • Before: DATA on a released or closed stream was not counted against the connection receive window, so the peer's connection send window shrank for good. Each chunk on a released local stream also got an RST_STREAM(STREAM_CLOSED).
   • After: all DATA counts against the connection window and is re-granted by WINDOW_UPDATE. DATA on a released, refused or implicitly closed stream is dropped without RST_STREAM.
   • DATA on a half-closed (remote) stream and a stream window overrun are stream errors (STREAM_CLOSED and FLOW_CONTROL_ERROR): one RST_STREAM, the stream is closed, and the app's rst_stream callback runs. An overrun used to send RST_STREAM only, leaving the stream open and counted. DATA on a stream the app keeps in IDLE state is a connection PROTOCOL_ERROR (before: RST_STREAM(STREAM_CLOSED)).

4. WINDOW_UPDATE and RST_STREAM on closed or never-opened streams

   • Before: WINDOW_UPDATE on a released stream got RST_STREAM(STREAM_CLOSED), or RST_STREAM(PROTOCOL_ERROR) for a zero increment. RST_STREAM on a CLOSED stream the app kept made the stream ILLEGAL and ran rst_stream again. Only IDs at or above the higher of the next local and next peer stream IDs counted as never opened.
   • After: both are ignored on a closed stream (RFC 9113 §5.1, §6.9). Stream IDs are classified per initiator, so DATA, WINDOW_UPDATE or RST_STREAM on an ID its initiator never opened is a connection PROTOCOL_ERROR (§5.1), even below the other side's next ID. A zero increment on an open stream is fix 10.

5. DATA padding

   • Before: only the data bytes counted against the receive windows. The peer also counts the Pad Length byte and the padding, so every padded frame shrank its send windows for good.
   • After: they count too, charged with the chunk that completes the frame's data. A peer that doesn't count padding now gets FLOW_CONTROL_ERROR when it overruns a window.

6. Handshake completion mask

   • Before: FD_H2_CONN_FLAGS_HANDSHAKING stayed 0xf0 when bit 7 became the WINDOW_UPDATE flag, so it included WINDOW_UPDATE and missed CLIENT_INITIAL. A WINDOW_UPDATE pending at the end of the handshake kept conn_established from firing, which blocked every fd_grpc_client request. A misbehaving server triggers it with one DATA frame before its SETTINGS ACK. When fd_h2 is the server, a conforming client does by sending more than about 19.6 KB of DATA before acknowledging SETTINGS.
   • After: the mask is built from the four handshake flags.

7. HPACK table size update with a multi-byte size

   • Before: fd_hpack_rd consumed a leading 0x3f update as one byte and decoded its continuation byte as a header field. For example, 0x3f 0x88 became :status: 200.
   • After: zero-size updates (0x20) are skipped and any other update is an HPACK error, which is all that is valid once the peer acknowledges SETTINGS_HEADER_TABLE_SIZE=0 (RFC 7541 §6.3).

8. END_STREAM on a CONTINUATION frame, where the flag is undefined

   • Before: it closed the receive side and reached the headers callback, and fd_grpc_client ended the request on it.
   • After: CONTINUATION flags other than END_HEADERS are ignored.

9. RST_STREAM on an already closed stream

   • Before: fd_h2_stream_error always sent RST_STREAM, so fd_grpc_client sent one on a closed stream when an oversized message or an unparsable header block arrived with END_STREAM after the request was fully sent.
   • After: RST_STREAM is sent only while the stream is open or half-closed.

10. Zero-increment WINDOW_UPDATE on an open stream

    • Before: RST_STREAM(PROTOCOL_ERROR), but the stream stayed open and the app was not told. The client could keep sending DATA on it, pass the rest of the response to the app, and later send a second RST_STREAM.
    • After: a stream error like a window overflow: one RST_STREAM(PROTOCOL_ERROR), the stream is closed, and rst_stream runs once. Later frames are dropped as for any released stream.

11. DATA on a stream ID that was never opened

    • Before: RST_STREAM(STREAM_CLOSED), which RFC 9113 §6.4 forbids for idle streams.
    • After: connection PROTOCOL_ERROR (§5.1) for local and peer-initiated IDs, as in fix 4. A refused peer stream now consumes its ID, so later frames on it are dropped rather than treated as idle.

12. HEADERS opening a server-initiated stream on a client

    • Before: passed to stream_create; fd_grpc_client refused it with RST_STREAM(REFUSED_STREAM) and kept the connection.
    • After: connection PROTOCOL_ERROR by default. RFC 9113 §5.1 lets HEADERS open a stream only when a client sends them or a server receives them, and fd_h2 disables server push. Setting conn->allow_server_requests restores the old nonstandard behavior (see the h2 README).

fd_grpc_client

13. END_STREAM on a HEADERS frame continued by CONTINUATION frames

    • Before: END_STREAM only reached the callback on the first frame, before END_HEADERS, so the request did not end when the block completed. Its stream lingered until a deadline, a server RST_STREAM or a reconnect.
    • After: the request ends on END_HEADERS with the stream in CLOSING_RX or CLOSED.

14. The server ends a request the client has not half-closed (a streaming request, or a body still waiting for flow control or TX space)

    • Before: the client called rx_end and freed its stream object without closing the HTTP/2 stream, leaking a concurrent stream slot, and ignored the server's later RST_STREAM. Once the leaks reached the server's MAX_CONCURRENT_STREAMS, every new request blocked.
    • After: the client closes the stream with RST_STREAM(NO_ERROR), as grpc-go does. That is one extra frame per such request (for example when the server ends the event client's StreamEvents); a unary request whose body was fully sent is unaffected. The RST_STREAM is queued before any app callback, in TX space fd_h2_rx reserves, so callbacks cannot crowd it out. Timeouts and oversized messages use the same order.

15. Stream deadlines while a connection flag is set

    • Before: fd_grpc_client_service_streams returned early on any connection flag. A server that sent HEADERS without END_HEADERS and then stalled, or a connection WINDOW_UPDATE stuck behind a TX buffer with under 128 bytes free, froze every deadline and stream WINDOW_UPDATE.
    • After: it returns early only when the connection is dead or sending GOAWAY. A stream that END_STREAM closed mid-block gets no WINDOW_UPDATE, and its deadline fires without RST_STREAM (only PRIORITY may be sent on a closed stream). Deadlines still wait while the TX buffer has no room for an RST_STREAM.

Other changes

• fd_h2_rx stops at a pending GOAWAY, not only once the connection is dead. It also drops the wrapped tail of a DATA chunk if the first callback raised a connection error.
• TX space for DATA responses is reserved before the frame header is consumed. Before, an empty DATA frame with END_STREAM was lost when TX was short.
• fd_h2 also HPACK-validates fragmented field blocks on live streams, so invalid HPACK there is a connection COMPRESSION_ERROR (RFC 9113 §4.3). So is, on a server, a discarded or fragmented block that uses the HPACK dynamic table before the client acknowledges SETTINGS_HEADER_TABLE_SIZE=0 (see "HPACK dynamic table" in the h2 README). Single-frame blocks are still parsed only by the app, and fd_grpc_client still answers an invalid one with a stream error.
• fd_h2_tx_rst_stream is no longer inline, which keeps its stack frame and stack protector canary out of the DATA receive path.
• fuzz_h2_actor, and fuzz_h2 on half of its client-side inputs, set allow_server_requests, so the fuzzers still reach server-initiated streams on a client.

Notes

• fd_grpc_client still parses each header fragment on its own (h2 README). A server that splits a field across frames fails that request at the first fragment, so fix 13 applies to blocks split on field boundaries.
• The bundle and event clients reconnect on any request deadline, so after a timeout fix 1 only matters for clients that keep the connection. The bundle client does reach fix 1 when fd_grpc_client kills a unary request over an oversized response: the connection stays up and the rest of the response arrives on the released stream.
• This conflicts with grpc: add server #11608 (grpc: add server), which adds an HPACK dynamic table under the same rx_hpack field name and hands discarded field blocks to the app.

Testing

Each numbered fix and the first two "Other changes" have a unit test that fails when only that change is reverted. The h2, padded DATA and gRPC unit tests, test_bundle_client and test_event_client pass with gcc, clang and clang ASan+UBSan (test_event_client's existing test-only leaks aside), and firedancer-dev builds with both compilers. fuzz_hpack_rd, fuzz_h2, fuzz_h2_actor, fuzz_grpc_client, fuzz_grpc_actor and fuzz_bundle_client found nothing under ASan+UBSan. fd_hpack_skip matched fd_hpack_rd and an independent RFC 7541 decoder on about 18M inputs, split at every byte boundary. In a microbenchmark of small DATA frames and gRPC messages, the gcc-built receive path runs 3 to 4 more instructions per frame, with no measurable change in cycles.

RFC 9113:

• §5.1: frames in flight after RST_STREAM are minimally processed and discarded, which an endpoint can do for any closed stream; only PRIORITY may be sent on a closed stream; frames other than HEADERS or PRIORITY on an idle stream are a connection error.
• §4.3: discarded field blocks are still decompressed; a decoding error is a connection COMPRESSION_ERROR.
• §6.9: all DATA counts toward the connection window; WINDOW_UPDATE on a closed stream is not an error; a zero increment on a stream is a stream error.
• §6.4: no RST_STREAM on an idle stream. §6.1: padding counts toward flow control. §4.1: undefined flags are ignored.

[github-actions]

 ┌─ ⚡ PERF · 2531717 vs main@48ae213 ─────────────────────────────────
 │ SUITE                               BASELINE          NEW         Δ
 │ replay tps, mainnet               24,568 tps   24,726 tps  ·  +0.64%
 │ bench tps, localnet              760,180 tps  761,652 tps  ·  +0.19%
 │ snapshot load, testnet                4.92 s       4.98 s  ·  +1.36%
 │ mem total, mainnet                159.96 GiB   159.96 GiB  ·   0.00%
 │ mem total, testnet                124.53 GiB   124.53 GiB  ·   0.00%
 │ mem total, ag mainnet             184.45 GiB   184.45 GiB  ·   0.00%
 │ clean compile, firedancer             3.23 s       3.21 s  ·  -0.40%
 │ binary size, firedancer             58.35 MB     58.35 MB  ·   0.00%
 ├─────────────────────────────────────────────────────────────────────
@@ 0 REGRESSIONS · 0 WARNINGS · 0 IMPROVED · 8 NOISE @@
 └─────────────────────────────────────────────────────────────────────

history · 4 pushes

 ┌─ HISTORY · Δ vs main, per push, newest first ──────────────────────────────────
 │ HEAD         TPS    BENCH     SNAP    MEM·M    MEM·T     AG·M  COMPILE   BINARY
 │ 2531717   +0.64%   +0.19%   +1.36%    0.00%    0.00%    0.00%   -0.40%    0.00%
 │ 2ac1f79   -0.74%   -0.05%   +2.41%    0.00%    0.00%    0.00%   +1.44%    0.00%
 │ 8b42589   +0.09%   -0.08%   +0.86%    0.00%    0.00%    0.00%   +1.68%    0.00%
 │ 889871e   +0.30%   +0.47%   +2.94%    0.00%    0.00%    0.00%   -0.16%    0.00%
 └────────────────────────────────────────────────────────────────────────────────

[Copilot]

Copilot review overview

🟢 Approval recommended

The changes are internally consistent and include focused regression coverage for the corrected edge cases.

Review effort: Balanced
Findings: None

What changed in this PR

Fixes HTTP/2 and gRPC stream lifecycle, flow-control accounting, and handshake edge cases.

Changes:

• Corrects handling of late frames, padding, continuations, and HPACK updates.
• Fixes handshake completion and gRPC stream termination/deadline behavior.
• Adds focused regression tests and documents released-stream header handling.

File	Description
src/​waltz/​h2/​test_h2_padded_data.c	Tests padded DATA flow-control accounting.
src/​waltz/​h2/​test_h2_conn.c	Tests late frames and handshake completion.
src/​waltz/​h2/​README.md	Documents released-stream header validation.
src/​waltz/​h2/​fd_hpack.c	Rejects nonzero HPACK table-size updates correctly.
src/​waltz/​h2/​fd_h2_conn.h	Corrects the handshake flag mask.
src/​waltz/​h2/​fd_h2_conn.c	Fixes late-frame handling and flow control.
src/​waltz/​grpc/​test_grpc_client.c	Tests deadlines and stream termination cases.
src/​waltz/​grpc/​fd_grpc_client.c	Fixes deadline servicing and early server termination.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Copilot review overview

🟡 Changes recommended

Stream-level DATA flow-control errors send a reset without closing the stream or releasing its quota.

Review effort: Balanced
Findings: 1

Open (1)

• Stream flow-control violations leave streams active and owners uninformed · New

[Copilot]

Copilot review overview

🔵 Needs a closer look

The broad cross-layer HTTP/2 state-machine and flow-control changes warrant final human protocol review.

Review effort: Balanced
Findings: 1

Open (1)

• Stream flow-control violations leave streams active and owners uninformed

[Copilot]

Copilot review overview

🟡 Changes recommended

Several behaviors implemented and asserted by tests materially contradict the PR description.

Review effort: Balanced
Findings: 2

Open (2)

• Align documentation with idle stream ID classification behavior · New
• Align documentation with CONTINUATION frame error handling · New

Resolved since last review (1)

• Stream flow-control violations leave streams active and owners uninformed

[Copilot]

Copilot review overview

🔵 Needs a closer look

The broad protocol state-machine and flow-control changes warrant final human validation despite comprehensive tests.

Review effort: Balanced
Findings: 2

Open (2)

• Align documentation with CONTINUATION frame error handling
• Align documentation with idle stream ID classification behavior

[Copilot]

Copilot review overview

🔵 Needs a closer look

The extensive protocol state-machine and flow-control changes warrant final human validation despite strong regression coverage.

Review effort: Balanced
Findings: 2

Open (2)

• Align documentation with CONTINUATION frame error handling
• Align documentation with idle stream ID classification behavior

[esemeniuc]

Copilot review overview

🔵 Needs a closer look

The broad protocol state-machine and flow-control changes warrant final human validation despite comprehensive tests.

Review effort: Balanced Findings: 2
Open (2)

* <img alt="Low severity" width="62" height="18" src="https://camo.githubusercontent.com/0ed7919ac116f938883e9a187291a1081d863ad712697b774e141f8e4192685e/68747470733a2f2f6769746875622e6769746875626173736574732e636f6d2f7374617469632f696d616765732f69636f6e732f636f70696c6f742d636f64652d7265766965772f6c6f772d76322d6c696768742e706e67"> [Align documentation with CONTINUATION frame error handling](#discussion_r4162171106)

* <img alt="Low severity" width="62" height="18" src="https://camo.githubusercontent.com/0ed7919ac116f938883e9a187291a1081d863ad712697b774e141f8e4192685e/68747470733a2f2f6769746875622e6769746875626173736574732e636f6d2f7374617469632f696d616765732f69636f6e732f636f70696c6f742d636f64652d7265766965772f6c6f772d76322d6c696768742e706e67"> [Align documentation with idle stream ID classification behavior](#discussion_r4162171074)

pr description is updated now