Repository navigation
fixed chunker: a short read is taken for EOF, silently truncating --read-special FIFO input #10565
Description
BUG (silent data loss)
System information
borg versions affected: master, 1.4.x and 1.2.x. Seen on macOS.
fixed versions: borg1 >= 1.4.6, borg2 >= 2.0.0b26
no new release for 1.2.x, but i backported the fix to 1.2-maint branch for package maintainers to cherry-pick it from there.
Describe the problem you’re observing
ChunkerFixed.chunkify() in src/borg/chunker.pyx treats any read that returns less than the requested block size (got < wanted) as end of file and stops chunking the whole input.
A short read is not EOF, though. Reads from a pipe, FIFO or some character devices often return less data than requested. With borg create --read-special --chunker-params fixed,..., borg then archives only the first part of such an input and silently drops the rest. There is no warning or error, and the return code is 0.
The buzhash chunker and stdin / --content-from-command input are not affected, because their buffered readers always fill the request completely.
Reproducing the problem
import os, threading
from borg.chunker import ChunkerFixed
r, w = os.pipe()
data = os.urandom(3 * 1024 * 1024)
def writer():
with os.fdopen(w, 'wb') as f:
f.write(data)
threading.Thread(target=writer).start()
ch = ChunkerFixed(4 * 1024 * 1024, 0, sparse=False)
total = sum(c.meta['size'] for c in ch.chunkify(None, r))
print('wrote', len(data), 'chunked', total)
On macOS this prints wrote 3145728 chunked 65536. Only the first 64 KiB, one pipe buffer, ends up in chunks. The writer then blocks forever, because nobody reads the rest.
The same thing happens end to end when you back up a FIFO with borg create --read-special --chunker-params fixed,4194304 while another process writes into it: the archived file is truncated.
Fix
After a short read, keep reading until the block is full or a read returns no data (real EOF). Filling the block also keeps the chunk boundaries at fixed offsets. That matters for dedup: if short pieces became chunks, boundaries would depend on how the kernel split the reads.
- master: chunkers: do not re-read a file map that does not reach EOF #10166 (also contains the fix for this bug)
- 1.4-maint: ChunkerFixed: do not take a short read for EOF (1.4-maint) #10563
- 1.2-maint: ChunkerFixed: do not take a short read for EOF (1.2-maint) #10564
Activity
ThomasWaldmann commentedon Oct 29, 2026
Who is affected?
Only a small group of users: the silent data loss needs all of the following.
--chunker-params fixed,.... The default buzhash chunker uses a different code path and is not affected.borg create --read-specialreading a FIFO, including FIFOs given via--paths-from-stdin/--paths-from-command. A pipe returns at most its buffer size per read (64 KiB on Linux and macOS), so any block size above that triggers it. Character devices that return less than requested per read can trigger it too (e.g. tape devices with a smaller record size than the block size, serial devices), but that is rare.os.posix_fadvisethere). The reproducer above shows 3 MiB written into a pipe ending up as 64 KiB of chunks.posix_fadvise, so it should behave like macOS (not tested).posix_fadvise()on a pipe fails withESPIPE, so borg reports the FIFO as a read error and exits with a warning instead of archiving partial content. Character devices passposix_fadviseon Linux, so a short-reading character device could still be affected there.posix_fadvise, and FreeBSD documentsESPIPEfor pipes, so they should fail with an error like Linux (not tested).Not affected
read()returns the full block there, except at the real end of the file.borg create ... -) and--content-from-command: Python's buffered reader always fills the request completely.borg import-tarandborg recreate --chunker-params fixed,...: their readers also fill the request completely.Symptoms
borg createfinishes with rc 0 and no warning.EPIPE/SIGPIPE. Scripts that check the writer's exit status would have noticed a failure.How to check existing archives
List the sizes of the FIFO items, e.g.:
Items whose size is 64 KiB or less, but should be much bigger, are truncated.
ThomasWaldmann commentedon Oct 29, 2026
Note: all branches fixed, but keeping this open for visibility for a while...
tests: fixed chunker reading from a pipe / FIFO
ThomasWaldmann commentedon Oct 29, 2026
Fixes for the short read issue:
Fixes for the ESPIPE fadvise issue:
safe_fadvise.ChunkerFixed: ignore posix_fadvise() errors, e.g. ESPIPE for a FIFO
ChunkerFixed: ignore posix_fadvise() errors, e.g. ESPIPE for a FIFO