Repository navigation
Check a later Gzip member's CRC before accepting a short trailer [patch] - #173
Merged
Merged
Conversation
To allow concatenated members, the Gzip end check accepted any input whose last four bytes read as a size below the output so far. In truncated input those bytes are payload, and stored blocks copy payload verbatim, so a cut just after a zero or small little-endian field "succeeded" with partial output on the span, stream and async paths. Accept a shorter trailer only when its CRC-32 matches the output it claims to cover: on the span path by checking that suffix directly, and on the stream paths by deriving the CRC of the output since each read began, because GZipStream returns from a read at every member's end. Fixes #171 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017vfJW671w9qNTjehHKvwnt
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Fixes #171
What was wrong
To allow concatenated members,
CompressedStreamEnd.Gzip.EndsWithaccepted any input whose last 4 bytes, read as ISIZE, were smaller than the output so far (|| trailerLength < length), and it never checked a CRC for that case. In truncated input those bytes are payload. Deflate copies stored blocks verbatim, so a cut just after a zero field or a small little-endian integer was reported as success with partial output on the span, stream and async paths. The repro in the issue reproduces onmain.Change
A trailer whose size is less than the whole output is now accepted only if its CRC-32 matches the output it claims to cover:
IsComplete): the whole output is available, so the CRC ofoutput[^trailerLength..]is computed and compared directly. This goes through a newEndsWith(inputTail, output)overload, which ZLib leaves at its default.Appendrecords where each read's output began, along with the CRC-32 of the output before it.GZipStreamreturns from a read at the end of each member, so a later member starts at one of these points. The suffix CRC is derived ascrc(all) ^ shift(crc(before), suffixLength), using zlib'scrc32_combinearithmetic (Multiply/XToThe), so no output has to be kept.A last member that is empty (trailer size 0) is no longer accepted. Accepting a zero size is what made eight zero bytes at a cut look like a valid trailer, and an empty trailing member is not something the providers produce. ZLib and Brotli are unchanged.
Tests (
TruncatedCompressionTests)Gzip_Rejects_A_Stored_Block_Payload_Cut_After_A_Zeroed_Field: the issue's repro, checked on all four entry pointsGzip_Rejects_A_Stored_Block_Payload_Cut_At_Every_Offset: random bytes with small fields followed by zero runs, cut at every offset, on all four entry pointsGzip_Rejects_Concatenated_Members_Whose_Last_Member_Is_Cut: every cut inside the last memberGzip_Accepts_Concatenated_Members_On_Every_Path: three members, the first larger than one 80 KiB read, so the CRC combine runs on the stream pathsThe three rejection tests fail on
mainand pass with the fix. The existingGzip_Accepts_Concatenated_Membersstill passes. Full suite: 1349 passed, 1 skipped.Essentials.CompressionProviders.Gzipbuilds for all its targets with-warnaserror.Generated by Claude Code