Add Transpose filter for fixed-size records - #245
Open
plokijuter wants to merge 8 commits into
Open
Conversation
Groups byte i of each R-byte record together so that homogeneous columns reach the next coder. Aimed at fixed-size records: sensor logs, struct arrays, PCM audio, numeric tables -- data where LZ and context models see interleaved, unrelated byte streams. Measured with PPMd:o=16 behind the filter: int32 ascending array 98450 -> 18411 (+81.2%) 16-byte struct records 68722 -> 22215 (+67.6%) fixed-record .dat file 76019 -> 25319 (+66.6%) 4x float32 sensor log 399919 -> 221368 (+44.6%) random data 205808 -> 205824 (identity) text 3862 -> 3878 (identity) R is auto-detected by default (-m0=Transpose); -m0=Transpose:15 forces it. Detection asks whether transposing HELPS, not whether the data is periodic: it compares the mean absolute difference between bytes R apart against that of adjacent bytes, and stays at R=1 (identity) unless a column is clearly more homogeneous. An autocorrelation-based detector was tried first and rejected -- it found periods everywhere (harmonic sidebands) and degraded 7 of 10 test files, one from 107607 to 137309 bytes. The block size is a fixed 64 KiB constant, deliberately independent of the caller's buffer size: 7-Zip does not use the same buffer sizes when compressing and decompressing, and a buffer-dependent block makes the transform irreversible. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018CeqaL4dXjGa9pUc6qRnof
The last partial block of a stream is never transposed: a filter is not told which call is the final one, so the tail is written through unfiltered. With a fixed 64 KiB block that tail cost a lot on small and medium files -- on a 450 KB test file, 56509 bytes (12.6%) reached PPMd as raw interleaved data and cost 7020 bytes, the whole gap against transposing the tail by hand. The block size is now chosen at encoding time from kExpectedDataSize (aim for at most 1/32 of the stream, clamped to 4 KiB..64 KiB) and written into the coder properties, so encoder and decoder agree regardless of their buffer sizes. Properties grow to 2 bytes; a 1-byte property is still read and implies the previous 64 KiB block. With PPMd:o=16 behind the filter: 16-byte struct records 22215 -> 3211 int32 ascending array 18411 -> 2442 fixed-record .dat file 25319 -> 20975 4x float32 sensor log 221368 -> 215478 48 MB sensor log unchanged (large streams keep the 64 KiB block) 65535-byte file 10305 -> 3055 (the filter now engages from 4 KiB) Zero-padding the last block through the AES-CBC path was considered and rejected: that protocol is only safe because a 16-byte block divides the FilterCoder buffer, so a full buffer never presents a partial block. A block that must be a multiple of R has no such property, and an exactly-full buffer would spin in CFilterCoder::Code. Verified: 60 real files, forced R from 2 to 256, sizes 0/1/100/4095/4096/ 65535/65536/200000, a 48 MB stream, and archives written by the previous 1-byte-property build -- all extract byte-identical. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018CeqaL4dXjGa9pUc6qRnof
The heuristic that picks R from the mean absolute difference between bytes R apart is not safe as a default. Measured over a 40-file corpus it degrades 13 files, the worst from 15026 to 352332 bytes (+2245%): a tiled RGB image whose tile repeats across the whole file, which a context model exploits directly and which block-wise transposition destroys. Adds an opt-in mode that measures instead of guessing: it transposes a sample with each of the most promising R values, compresses each, and keeps the winner. R=1 is always in the running, so measuring cannot pick worse than not filtering -- as far as the sample is representative. -m0=Transpose heuristic, fast -m0=Transpose:a=1 measure, probing with PPMd -m0=Transpose:a=2 measure, probing with LZMA The probe must be the coder that actually follows. LZMA and PPMd do not prefer the same R, and probing with the wrong one produces confident nonsense: on a 1.6 MB stereo sine, an LZMA probe picked R=4 (17374 bytes) where PPMd wanted R=1 (9420). Three cases fixed by probing correctly: 1.6 MB stereo sine 17374 -> 9436 mono sine, R=2 optimal 384966 -> 186871 256-byte repeating tile 16939 -> 885 The measurement sample also had to grow from 256 KB to 4 MB. The verdict flips with sample size, because the transposed form costs linearly in the data while the raw form stays nearly flat when it has long-range redundancy: on a 3.5 MB sawtooth, R=12 wins on any prefix up to 2 MB (7295 vs 9644) and loses on the whole file (19520 vs 10170). A short sample concludes backwards, with confidence. Known limitation: a filter never sees more than the FilterCoder buffer (2 MB), so on files a few MB and larger the sample is still not representative and the measure mode gets 3 of 15 test files wrong. Deciding reliably needs a pass over the whole input, which does not belong in a stream filter. Extrapolating the slope between two samples and requiring a margin was tried and rejected: tuned on 15 files it looked exact, and on the 40-file corpus it left +11.7% on the table at every threshold. Also adds mingw-shim/README.md: cross-compiling from Linux needs symlinks for the headers the source includes with Windows casing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018CeqaL4dXjGa9pUc6qRnof
…lter
A stream filter cannot choose R. It never sees more than the FilterCoder
buffer (2 MB), and the verdict flips with how much it sees: on a 3.5 MB
sawtooth, R=12 wins on every prefix up to 2 MB (7295 vs 9644 bytes) and loses
on the whole file (19520 vs 10170). The transposed form costs linearly in the
data while the raw form stays nearly flat when it has long-range redundancy,
so a short sample concludes backwards, with confidence.
Adds -m0=Transpose:a=3, resolved in Update.cpp before the coders are built,
where the input files are known. Four steps, cheapest first:
1. one reference compression at R=1, needed anyway. If the file already
compresses more than 10x, stop: measured over 55 files, no gain is ever
missed above that ratio, and every trap sits above it.
2. rank by mean absolute difference over all R in 2..256 -- integer
subtraction, essentially free. Not a grid: the optimum of one test file
is R=15 and of another R=21, and a grid that skips them costs 27% and
40x respectively.
3. rank by a fast LZMA pass over a curated grid, which sees repetitions the
statistical criterion is blind to.
4. compress for real at R=1 and at the (at most two) candidates, keep the
smallest.
Step 4 is what matters: R=1 is always in the race and the result is a minimum
over real compressed sizes, so degrading is impossible by construction, not
merely rare. Every earlier attempt tried to predict well; this one stops
predicting. When R=1 wins, the filter is dropped from the chain entirely, so
it does not even cost the 16-byte coder record.
Because degradation is impossible, the trigger thresholds can be generous: a
candidate too many costs time, never correctness. A tight 0.30 threshold left
85% and 139% of gain on the table on float sensor logs whose ratio is 0.52.
Probing with PPMd, whatever coder follows. The obvious rule -- probe with the
coder that will actually run -- was implemented and then reverted on
measurement: for an LZMA2 chain, an LZMA probe missed 5.8x on struct records
and 11x on an int32 array, while the PPMd probe found the optimum on 4 of 5.
A context model ranks column homogeneity better than a match finder does.
Measured with -m0=Transpose:a=3 -m1=PPMd:o=16, against the best R found by
exhaustive search:
fixed-record .dat 76019 -> 20975 optimum
16-byte struct records 68722 -> 3211 optimum
4x float32 sensor log 399919 -> 215478 optimum
periodic PCM (85% trap) 9420 -> 9420 correctly abstains
256-byte tile (19x trap) 869 -> 869 correctly abstains
RGB tile16 (13.4x trap) 2280 -> 2280 correctly abstains
70 files of assorted types: 0 degradations, 0 round-trip failures. Sizes
0/1/100/1000/100000, multiple files per archive, PPMd and LZMA2 chains all
round-trip byte-identical.
Above TRANSPOSE_FULL_LIMIT (64 MB) the decision is made on a prefix and the
guarantee no longer holds, so a clear margin is required before accepting the
transposition there.
The pass lives in Update.cpp, which compiles into the executables rather than
7z.dll, so UI/Console needs the encoder objects to link -- 7z.exe grows from
576 KB to 814 KB. 7zFM.exe and 7zG.exe are unchanged and therefore still
lack a=3; they keep the older in-filter modes a=1 and a=2.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018CeqaL4dXjGa9pUc6qRnof
The filter was only reachable by typing "0=Transpose:a=3 1=LZMA2" into the Parameters box, which nobody will do. Add it to the method drop-down of the Add-to-archive dialog instead, as a single entry named "anyz2". "anyz2" is not a coder: UpdateGUI expands the selection into the two stages the encoder actually needs (0 = Transpose:a=3, 1 = LZMA2) and re-prefixes the dictionary and word-size properties to coder 1, so the dialog's own sliders keep working. Everywhere the dialog reasons about LZMA2 -- dictionary defaults, solid block size, thread counts, memory estimate -- anyz2 follows the same path. It is pre-selected for 7z when the user has no saved method: a=3 measures the record stride over the whole input with R=1 in the race, so the filter cannot come out worse than LZMA2 alone. It cannot be the first entry of the list -- that slot is the "*" automatic entry, whose item data is -1 and which emits no method property at all. Also: the filter now falls back to a=1 instead of returning E_INVALIDARG when it receives a=3 directly, which happens with a stock 7zFM/7zG that has no whole-file pass. 7zG.exe had no GNU makefile; add one for mingw-w64 cross builds. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018CeqaL4dXjGa9pUc6qRnof
7zFM.exe had no makefile.gcc either. It needs no source change for the Transpose filter -- it shells out to 7zG.exe for every compression -- but an install cannot be left with a 24.09 file manager sitting next to a 26.02 7z.dll, so it has to be buildable from the same tree. MPR.dll is imported statically here rather than delay-loaded as the MSVC build does; it is present on every desktop Windows. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018CeqaL4dXjGa9pUc6qRnof
…rogress Reported from a real run: compressing a 4.2 GB folder of photos and video, the window said "Compressing" at 0% and sat there for minutes before any byte was written, with Cancel doing nothing. Transpose_ChooseR_Full ran the PPMd reference over the whole 64 MB buffer FIRST, then looked for candidate strides -- and found none, because H.264 and JPEG have no byte-column structure. All of that work was thrown away. The reference only exists to be compared against candidates; with no candidate there is nothing to compare, so it must not be paid for. Reordered: the mean-absolute-difference scan and the fast LZMA-1 ranking now run first, on a 4 MB sample rather than the whole buffer, and the function returns 1 before touching PPMd when neither detector proposes a stride. Measured on a 13-file corpus, old build against new, interleaved on the same machine: the chosen R is identical on all 13 -- no gain is lost. A 70 MB video prefix goes from 144.8 s to 6.1 s, 24 MB of random data from 44.7 s to 6.2 s. The one regression is highly compressible text, 54 ms to 1.0 s, which is the price of dropping the guard below. The >= 10x "already very compressible" guard is REMOVED. It could only ever lose gain -- the verdict is a minimum that always includes R=1, so degradation was already impossible without it. And re-measuring it on the fast coder made it actively harmful: c_int32.bin (consecutive integers) is repetitive enough that LZMA-1 clears 10x where PPMd did not, so the guard fired and returned R=1 instead of R=4. Caught by the corpus, not by reasoning. Also: the pass now reports progress and honours Cancel through IUpdateCallbackUI, so the window no longer looks frozen. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018CeqaL4dXjGa9pUc6qRnof
Owner
|
It's bad idea to use short id=0xC for third-party filter. |
Igor Pavlov pointed this out upstream, and he is right: Methods.txt reserves
ALL short IDs for the 7-Zip and xz developers. A third party that helps
itself to one will eventually collide with an official method, and the two
archives then become indistinguishable -- there is no version field to tell
them apart.
Methods.txt defines the convention for third parties:
3F ZZ ZZ ZZ ZZ ZZ MM MM
3F prefix for random IDs
ZZ ZZ ZZ ZZ ZZ developer ID, real random bytes
MM MM method number within that developer
Our developer ID was drawn from /dev/urandom: E2B7E19B8A. Transpose is
method 0001, so the full ID is 3F E2B7E19B8A 0001.
Methods.txt gains a "Third-party IDs" section and the entry leaves the short
ID list.
This BREAKS every archive written with 0x0C. That is the point of the fix,
and it is the right moment to take it -- the filter has only ever been used
locally.
Verified end to end with the rebuilt binaries: 600 000 -> 7 065 on a record
file, extraction byte-identical, "Method = Transpose LZMA2" listed correctly.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018CeqaL4dXjGa9pUc6qRnof
Author
|
Sorry about that, and thank you for catching it — you're right, I should not have taken a short ID. Fixed: the filter now uses Happy to switch to an allocated ID instead if you would prefer to assign one. |
plokijuter
changed the title
Owner
|
I can allocate new IDs: |
Owner
|
While your code is not final with that new ID, you can change and optimize encoding and properties. But you can store encoded Actually 7-Zip's code that calls filters was not optimized for cases with big buffers in filters, because all current filters are limited by 16-bytes remainder. Now I don't include any new filters to main code of 7-zip. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a byte-transposition filter for data made of fixed-size records: it groups byte i of each R-byte record together, so that homogeneous columns reach the next coder instead of interleaved, unrelated byte streams.
This is the one structure LZ and context models handle poorly today. Sensor logs, struct arrays, PCM audio and numeric tables all have it, and the existing Delta filter only covers the special case R = the stride of a single scalar series.
Measured, with
PPMd:o=16behind the filterOn the 48 MB sensor log that is also 39% smaller than
xz -9eand 66% smaller thanzstd -19.Choosing R
-m0=Transposedetects R automatically;-m0=Transpose:15forces it.Detection asks whether transposing helps, not whether the data is periodic. It compares the mean absolute difference between bytes R apart against that of adjacent bytes, and stays at R=1 (identity) unless a column is clearly more homogeneous, so data with no record structure is passed through untouched.
An autocorrelation detector was written first and rejected on measurement: it found periods everywhere (harmonic sidebands clear a 60%-of-peak threshold) and degraded 7 of 10 test files, one from 107607 to 137309 bytes. Periodicity of a signal says nothing about the homogeneity of its columns.
Block size
The transform is applied on fixed-size blocks whose size is deliberately independent of the caller's buffer, since 7-Zip does not use the same buffer sizes when compressing and decompressing. The size is chosen at encoding time from
kExpectedDataSize(at most 1/32 of the stream, clamped to 4 KiB..64 KiB) and recorded in the coder properties, so both sides agree whatever their buffers are.The reason it is not simply a constant: a filter is never told which call is the final one, so the last partial block of a stream is written through unfiltered. With a fixed 64 KiB block that tail reached 12.6% of a 450 KB file and cost 7020 bytes.
Zero-padding the final block through the AES-CBC path in
CFilterCoder::Codewas considered and rejected: that protocol is only safe because a 16-byte block divides the FilterCoder buffer, so a full buffer never presents a partial block. A block that must be a multiple of R has no such property, and an exactly-full buffer would spin.Interface
0C(free slot;0BRISCV is the last assigned,21is LZMA2)R-1thenlog2of the block size. A 1-byte property is still accepted and implies a 64 KiB block.C/Transpose.{c,h}plusCPP/7zip/Compress/TransposeFilter.cpp, registered withREGISTER_FILTER_E. No existing file is modified beyond the build hooks andDOC/Methods.txt.Verification
60 real files of assorted types, forced R from 2 to 256, sizes 0/1/100/4095/4096/65535/65536/200000, a 48 MB stream, and archives written by an earlier 1-byte-property build: all extract byte-identical. Compiles clean under
-Werror -Wall -Wextra.Happy to adjust the method ID, the property layout, or the detection heuristic if you would rather they were shaped differently.