Skip to content

Every hardlink of a file is read and SHA-256-hashed in full, so a snapshot tree re-reads each unchanged file once per snapshot even when it has no duplicate #196

Open

Description

@matt-edmondson

What happens

The size pre-filter from #147 (FileHasher.HashPossibleDuplicates, FileDeduplicator/FileHasher.cs:33-38) groups paths by size. N hardlinks of one file all share that size, so they pass as "possible duplicates" and HashFiles (FileHasher.cs:40-70) reads the same inode end to end N times. Only afterwards does Deduplicator.CollapseSameFile (Deduplicator.cs:78-84) recognise them as one file and conclude there is nothing to deduplicate.

Repro (HEAD 0708fa5): one 50 MB file hardlinked into 4 snapshot directories, no other file of that size:

Found 5 file(s).
Hashing files...
  Hashed: video.bin -> 9f854cf82d19...
  Hashed: video.bin -> 9f854cf82d19...
  Hashed: video.bin -> 9f854cf82d19...
  Hashed: video.bin -> 9f854cf82d19...
No duplicate files found.

200 MB read to conclude there are no duplicates; the correct cost is zero reads, because only one file has that size.

Why it matters

Hardlink-heavy trees (rsnapshot, cp -al, rsync --link-dest, Time Machine-style backups) are exactly the trees #132 and #190 are about. In such a tree almost every file is a multi-linked unchanged file, so the run reads the whole backup volume once per snapshot — on the cold disks, USB drives and network shares the #147 remarks say dominate run time — just to discover that the links are the same file.

Suggested fix

In HashPossibleDuplicates, read FileIdentity for each path and group by identity before the size check:

Acceptance criteria

Activity

  1. matt-edmondson commented on Oct 29, 2026

    @matt-edmondson
    ContributorAuthor

    Triage


    Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions