Skip to content

chunkers: review fixes (chunking_time precision, param checks, cleanups) - #10603

Merged
ThomasWaldmann merged 5 commits into
borgbackup:masterfrom
ThomasWaldmann:chunker-review-fixes
Nov 9, 2026
Merged

ThomasWaldmann merged 5 commits into
borgbackup:masterfrom
ThomasWaldmann:chunker-review-fixes

Conversation

@ThomasWaldmann

@ThomasWaldmann ThomasWaldmann commented Nov 9, 2026 •

Copy link
Copy Markdown
Member

Fixes for what a review of the chunker code (src/borg/chunkers/, C and Cython) turned up. None of them changes a cut point. One commit per issue:

  1. chunking_time was a C float. It adds up one small increment per chunk, and a 32-bit float stops growing once the increments are below half an ulp of the sum (with 50 us per chunk at 1024 s at the latest). "Time spent in chunking" came out too low for long backups of small files. Now a double.
  2. Parameter checks were asserts, but the scan code relies on them to stay inside its buffer and python -O drops asserts. They raise ValueError now. A negative window_size was not checked at all, neither by the buzhash chunkers nor by ChunkerParams (--chunker-params buzhash64,10,16,14,-5,0 crashed); both reject it now.
  3. The normalized-chunking transition was computed as a position (last + normal_size) and compared as a pointer, which wraps for a huge normal_size and sends the scan out of the buffer. It is a length comparison now. API only: no borg command passes normal_size.
  4. gcc -Warray-parameter warnings for ra_new / gl_new / tp_new: header and definition declared the tables parameter differently.
  5. Stale comments and dead code: phte_core.h claimed everything in it is static; the avx512 "compiler too old" branches in fc_kernel_select() / bz64_kernel_select() were unreachable; "both" UHF-then-PRF chunkers are three.

Testing

  • New tests: constructors reject invalid params with ValueError (all CDC chunkers), ChunkerParams rejects a negative window size, a huge normal_size gives the cuts of the strict mask alone.
  • src/borg/testsuite/chunkers and helpers/parseformat_test.py pass, also the archive / create / recreate / transfer / extract tests (macOS arm64).
  • Cut points before and after are the same: a pure-Python model of the cut rules was compared with buzhash, buzhash64, fastcdc and toeplitz-aes on several hundred random parameter/data cases, each via 6 input paths (BytesIO, regular file with and without fh, short-reading file object, pipe, sparse).
  • The C sources build without warnings with gcc 14 -Wall -Wextra, and a standalone fuzz of all scan kernels against reference loops still passes on aarch64 (neon, blockwise, scalar, aes-arm64, evp) and on a Zen 4 (avx2, avx512, aes-ni, vaes, evp).

There is no test for 1: chunking_time is read-only and a meaningful check would need minutes of accumulated time.

🤖 Generated with Claude Code

ThomasWaldmann and others added 5 commits November 9, 2026 21:59
ChunkerBase.chunking_time was a C float. It adds up one small time
increment per chunk, and a 32-bit float stops growing once an increment
is smaller than half an ulp of the sum: with 50 us per chunk at 1024 s
at the latest, with 20 us at 512 s. "Time spent in chunking" of a long
backup of many small files thus came out too low.

The fixed and failing chunkers already use Python floats (doubles).

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The CDC chunkers checked their constructor parameters with asserts, but
their scan code relies on these checks to stay inside the buffer, and
python -O drops asserts: e.g. an AES chunker created with chunk_min_exp
4 then read 48 bytes in front of its buffer. They raise ValueError now.

A negative hash_window_size was not checked at all - not by the buzhash
chunkers, where it became a huge size_t, and not by ChunkerParams, so
that e.g. --chunker-params buzhash64,10,16,14,-5,0 crashed borg. Both
reject it now.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
To stop a scan at the strict->loose mask transition of normalized
chunking, process() computed last + normal_size and compared that as a
pointer into the buffer. normal_size is a caller-given size_t: for a
huge value the sum wrapped around, the "transition" ended up in front of
the scan position and the scan span went negative, i.e. far out of the
buffer (normal_size = 2^64 - 4096 crashed with a bus error).

Compare lengths instead: the scan is capped if normal_size - chunk_len
is less than what is left to scan. Cut points are unchanged.

No borg command passes normal_size, this only affected API users.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The headers declare it as "const uint64_t *tables", the definitions said
"const uint64_t tables[N * 256]". That is the same type, but gcc >= 11
warns about the mismatched bound (-Warray-parameter, part of -Wall).

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
- phte_core.h said that everything in it is static, but it also defines
  the three phte_kernel_*() functions with external linkage.
- fc_kernel_select() / bz64_kernel_select() had a "compiler too old"
  branch for avx512 under #ifndef FC_KIND_512 / BZ_KIND_512, inside a
  block that always defines these macros. Remove it and say in the
  headers that *_KSEL_NOTBUILT is never returned by these two selectors.
- ChunkerParams: there are three UHF-then-PRF chunkers, not "both".

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@codecov

codecov Bot commented Nov 9, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.39%. Comparing base (0a319a2) to head (94ac75b).
⚠️ Report is 3 commits behind head on master.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@           Coverage Diff           @@
##           master   #10603   +/-   ##
=======================================
  Coverage   89.38%   89.39%           
=======================================
  Files         105      105           
  Lines       20336    20340    +4     
  Branches     3192     3194    +2     
=======================================
+ Hits        18177    18182    +5     
+ Misses       1503     1500    -3     
- Partials      656      658    +2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

@ThomasWaldmann ThomasWaldmann changed the title chunkers: review fixes (chunking_time precision, param checks, EOF read, cleanups) chunkers: review fixes (chunking_time precision, param checks, cleanups) Nov 9, 2026
@ThomasWaldmann
ThomasWaldmann merged commit 13a3472 into borgbackup:master Nov 9, 2026
27 checks passed
@ThomasWaldmann
ThomasWaldmann deleted the chunker-review-fixes branch November 9, 2026 22:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant