You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Every hardlink of a file is read and SHA-256-hashed in full, so a snapshot tree re-reads each unchanged file once per snapshot even when it has no duplicate #196
The size pre-filter from #147 (FileHasher.HashPossibleDuplicates, FileDeduplicator/FileHasher.cs:33-38) groups paths by size. N hardlinks of one file all share that size, so they pass as "possible duplicates" and HashFiles (FileHasher.cs:40-70) reads the same inode end to end N times. Only afterwards does Deduplicator.CollapseSameFile (Deduplicator.cs:78-84) recognise them as one file and conclude there is nothing to deduplicate.
Repro (HEAD 0708fa5): one 50 MB file hardlinked into 4 snapshot directories, no other file of that size:
200 MB read to conclude there are no duplicates; the correct cost is zero reads, because only one file has that size.
Why it matters
Hardlink-heavy trees (rsnapshot, cp -al, rsync --link-dest, Time Machine-style backups) are exactly the trees #132 and #190 are about. In such a tree almost every file is a multi-linked unchanged file, so the run reads the whole backup volume once per snapshot — on the cold disks, USB drives and network shares the #147 remarks say dominate run time — just to discover that the links are the same file.
Suggested fix
In HashPossibleDuplicates, read FileIdentity for each path and group by identity before the size check:
count distinct identities per size bucket when deciding whether a size is unique;
Priority: Low. The results are correct; only the run time suffers. It matters most on hardlink-heavy backup trees, where the run reads the whole volume once per snapshot.
Area / suggested assignee:FileHasher.HashPossibleDuplicates and HashFiles (FileDeduplicator/FileHasher.cs:33-70), next to Deduplicator.CollapseSameFile. Owner: @matt-edmondson
Activity
matt-edmondson commentedon Oct 29, 2026
Triage
FileHasher.HashPossibleDuplicatesandHashFiles(FileDeduplicator/FileHasher.cs:33-70), next toDeduplicator.CollapseSameFile. Owner: @matt-edmondsonFileIdentitybefore the size-uniqueness check, hash one representative per identity, and copy its hash to every linked path soFileHashesstays complete. Consider landing this alongside Deduplicate removes one name of a hardlinked duplicate and reports the space as reclaimed, but frees nothing and the next Scan finds the same duplicate again #190.Generated by Claude Code