Repository navigation
chunkers: review fixes (chunking_time precision, param checks, cleanups) - #10603
Merged
ThomasWaldmann merged 5 commits intoNov 9, 2026
Merged
Conversation
ChunkerBase.chunking_time was a C float. It adds up one small time increment per chunk, and a 32-bit float stops growing once an increment is smaller than half an ulp of the sum: with 50 us per chunk at 1024 s at the latest, with 20 us at 512 s. "Time spent in chunking" of a long backup of many small files thus came out too low. The fixed and failing chunkers already use Python floats (doubles). Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The CDC chunkers checked their constructor parameters with asserts, but their scan code relies on these checks to stay inside the buffer, and python -O drops asserts: e.g. an AES chunker created with chunk_min_exp 4 then read 48 bytes in front of its buffer. They raise ValueError now. A negative hash_window_size was not checked at all - not by the buzhash chunkers, where it became a huge size_t, and not by ChunkerParams, so that e.g. --chunker-params buzhash64,10,16,14,-5,0 crashed borg. Both reject it now. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
To stop a scan at the strict->loose mask transition of normalized chunking, process() computed last + normal_size and compared that as a pointer into the buffer. normal_size is a caller-given size_t: for a huge value the sum wrapped around, the "transition" ended up in front of the scan position and the scan span went negative, i.e. far out of the buffer (normal_size = 2^64 - 4096 crashed with a bus error). Compare lengths instead: the scan is capped if normal_size - chunk_len is less than what is left to scan. Cut points are unchanged. No borg command passes normal_size, this only affected API users. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The headers declare it as "const uint64_t *tables", the definitions said "const uint64_t tables[N * 256]". That is the same type, but gcc >= 11 warns about the mismatched bound (-Warray-parameter, part of -Wall). Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
- phte_core.h said that everything in it is static, but it also defines the three phte_kernel_*() functions with external linkage. - fc_kernel_select() / bz64_kernel_select() had a "compiler too old" branch for avx512 under #ifndef FC_KIND_512 / BZ_KIND_512, inside a block that always defines these macros. Remove it and say in the headers that *_KSEL_NOTBUILT is never returned by these two selectors. - ChunkerParams: there are three UHF-then-PRF chunkers, not "both". Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #10603 +/- ##
=======================================
Coverage 89.38% 89.39%
=======================================
Files 105 105
Lines 20336 20340 +4
Branches 3192 3194 +2
=======================================
+ Hits 18177 18182 +5
+ Misses 1503 1500 -3
- Partials 656 658 +2 ☔ View full report in Codecov by Harness. |
ThomasWaldmann
force-pushed
the
chunker-review-fixes
branch
from
November 9, 2026 21:22
9c28c10 to
94ac75b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes for what a review of the chunker code (
src/borg/chunkers/, C and Cython) turned up. None of them changes a cut point. One commit per issue:chunking_timewas a Cfloat. It adds up one small increment per chunk, and a 32-bit float stops growing once the increments are below half an ulp of the sum (with 50 us per chunk at 1024 s at the latest). "Time spent in chunking" came out too low for long backups of small files. Now adouble.asserts, but the scan code relies on them to stay inside its buffer andpython -Odrops asserts. They raiseValueErrornow. A negativewindow_sizewas not checked at all, neither by the buzhash chunkers nor byChunkerParams(--chunker-params buzhash64,10,16,14,-5,0crashed); both reject it now.last + normal_size) and compared as a pointer, which wraps for a hugenormal_sizeand sends the scan out of the buffer. It is a length comparison now. API only: no borg command passesnormal_size.-Warray-parameterwarnings forra_new/gl_new/tp_new: header and definition declared thetablesparameter differently.phte_core.hclaimed everything in it is static; theavx512"compiler too old" branches infc_kernel_select()/bz64_kernel_select()were unreachable; "both" UHF-then-PRF chunkers are three.Testing
ValueError(all CDC chunkers),ChunkerParamsrejects a negative window size, a hugenormal_sizegives the cuts of the strict mask alone.src/borg/testsuite/chunkersandhelpers/parseformat_test.pypass, also the archive / create / recreate / transfer / extract tests (macOS arm64).buzhash,buzhash64,fastcdcandtoeplitz-aeson several hundred random parameter/data cases, each via 6 input paths (BytesIO, regular file with and withoutfh, short-reading file object, pipe, sparse).-Wall -Wextra, and a standalone fuzz of all scan kernels against reference loops still passes on aarch64 (neon, blockwise, scalar, aes-arm64, evp) and on a Zen 4 (avx2, avx512, aes-ni, vaes, evp).There is no test for 1:
chunking_timeis read-only and a meaningful check would need minutes of accumulated time.🤖 Generated with Claude Code