Fix race in HTTP/2 buffered data writes - #396
Open
kafkiansky wants to merge 1 commit into
Open
kafkiansky wants to merge 1 commit into
kafkiansky wants to merge 1 commit into
Conversation
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
writeBufferedData()updated the stream buffer and flow-control windows only afterwriteFrame(), which may suspend. A concurrentsendBufferedData()(deferred on everyWINDOW_UPDATE) could then resend data that was already partly written and complete the shareddeferredFutureearly. Trailers withEND_STREAMthen went out before the body was finished. Peers received duplicated, reordered or truncated bodies, or the stream stalled.The buffer and windows are now updated before writing, and all DATA frames of one call go out in a single
write(), so frames from concurrent writers cannot interleave.The new test
testConcurrentWindowUpdatesWithSuspendedWritesmakes every write suspend, sends connection and streamWINDOW_UPDATEframes together, and checks that the body arrives intact withEND_STREAMon the trailers. It fails without the fix.This overlaps with #387, which addresses the same race, but #387 fixes only part of it:
$delta >= $lengthbranch. In the$delta > 0branch,$stream->bufferis still advanced after the suspendingwriteFrame()calls. A concurrentsendBufferedData()can still see the untrimmed buffer and resend data. This is the path that truncated a gRPC server stream at ~720 KB in my case.This PR updates the state before writing in both branches and sends all DATA frames of a call in a single
write(). The test here makes every write suspend and sends connection and streamWINDOW_UPDATEframes together, so it covers the race end to end, not just the buffer invariant.How to reproduce: