Deduplicate removes one name of a hardlinked duplicate and reports the space as reclaimed, but frees nothing and the next Scan finds the same duplicate again

What's wrong

CollapseSameFile reduces each set of paths sharing a file identity to one representative:

FileDeduplicator/FileDeduplicator/Deduplicator.cs

Lines 78 to 83 in 8de035e

    
           private static IEnumerable<AbsoluteFilePath> CollapseSameFile(IEnumerable<AbsoluteFilePath> paths) 
        
           { 
        
           	HashSet<FileIdentity> seen = []; 
        
           	return OrderByPreference(paths) 
        
           		.Where(path => !FileIdentity.TryGet(path.WeakString, out FileIdentity identity) || seen.Add(identity));

That is correct for paths that lead to the keeper. A duplicate that is not the keeper can also have several hardlinks, though. Its other names are then dropped from the group, so DeleteDuplicates deletes only the representative's name. It still adds the full file size to bytesReclaimed:

FileDeduplicator/FileDeduplicator/Deduplicator.cs

Lines 182 to 186 in 8de035e

    
           if (TryDelete(file, errors, out long fileSize)) 
        
           { 
        
           	deletedCount++; 
        
           	bytesReclaimed += fileSize; 
        
           }

DryRun's Space to reclaim makes the same claim:

FileDeduplicator/FileDeduplicator/DuplicateReport.cs

Line 60 in 8de035e

bytesReclaimable += group.FileSize;

Removing one link out of two unlinks no data blocks.

Reproduction (Linux, built from HEAD)

mkdir -p t/sub
head -c 100000 /dev/urandom > t/k.bin
cp t/k.bin t/sub/copy-long-name.bin
ln t/sub/copy-long-name.bin t/sub/copy-long-name-2.bin
echo y | FileDeduplicator Deduplicate -p t
FileDeduplicator Scan -p t

Deduplicate lists KEEP: t/k.bin and DELETE: t/sub/copy-long-name.bin, then prints Deleted 1 file(s). and Reclaimed 97.7 KB of disk space.. Afterwards t/sub/copy-long-name-2.bin still holds the same 100,000 bytes, so no space was freed. The following Scan reports the group again (k.bin [KEEP], copy-long-name-2.bin [DELETE], Total wasted space: 97.7 KB).

Why it matters

The reclaimed-space figure is wrong in the hardlink-heavy trees Hardlinked files are deduplicated like independent copies, risking silent data loss in snapshot/backup trees #132 was about (rsnapshot, cp -al, rsync --link-dest).
The user is told deduplication finished while the duplicate data is still on disk. It takes one more run per remaining link to actually free the space.
In a snapshot tree this is what Hardlinked files are deduplicated like independent copies, risking silent data loss in snapshot/backup trees #132 set out to prevent: a name is removed from one snapshot while the data stays referenced from another. It is the representative's name only because CollapseSameFile happened to prefer it.

Suggested fix

Treat a non-keeper duplicate as all of its paths rather than one. Pick one consistent policy:

Fail closed (in keeping with the triage on Hardlinked files are deduplicated like independent copies, risking silent data loss in snapshot/backup trees #132): when a non-keeper copy has other names, skip deleting it and record a SkippedFile such as "has N other hardlinks; deleting this name would free nothing". Leave it out of DryRun's Files to delete / Space to reclaim.
Delete every name: keep the collapsed-away paths alongside their representative in DuplicateGroup, delete all of them, and count the size once. List each name under DELETE: so the preview shows them.

Either way, bytesReclaimed and bytesReclaimable should only count a file's size when its last name is removed.

Acceptance criteria

In the reproduction above, Deduplicate does not report 97.7 KB reclaimed unless the data is actually freed.
After Deduplicate, Scan finds no duplicates, or Deduplicate's output explains why the remaining link was kept.
DryRun's totals match what Deduplicate then does.
A test in FileDeduplicator.Test/SameFileTests.cs covering a non-keeper copy with two hardlinks.

@matt-edmondson

Triage

Category: Bug
Priority: Medium. No data is lost, but Deduplicate reports space as reclaimed when none was freed, and in snapshot trees it removes a name from one snapshot while the data stays referenced elsewhere. That is the outcome Hardlinked files are deduplicated like independent copies, risking silent data loss in snapshot/backup trees #132 set out to prevent.
Area / suggested assignee: Deduplicator.CollapseSameFile / DeleteDuplicates and DuplicateReport (DryRun totals). Owner: @matt-edmondson
Duplicates / in progress: not a duplicate. It is closely related to Stats counts hardlinks as duplicate files and double-counts their size, so it reports "Duplicate files: 1, Duplicate groups: 0" where Scan finds no duplicates #189: both come from collapsing hardlinks in CollapseSameFile without carrying the extra names forward, so one change to how a group represents a file's names could fix both. No open PR in FileDeduplicator covers either.
Next step: choose the policy first. "Fail closed", which skips the file with a SkippedFile reason, matches the Hardlinked files are deduplicated like independent copies, risking silent data loss in snapshot/backup trees #132 triage. "Delete every name" is the alternative. Either way, count bytes only when the last name goes, keep DryRun and Deduplicate in agreement, and add the two-hardlink non-keeper test to SameFileTests.cs.

Generated by Claude Code

Triage

Depends on PR #191 (the fix for #189). Not actionable until that lands.

I started on this and stopped because of how it interacts with #191:

Count a hardlinked file once in Stats [patch] #191 makes Stats' Duplicate files come from the collapsed groups (Sum(g => g.Files.Count - 1)), and its test pins Stats == DryRun Files to delete for a copy plus a hardlink pair (k.bin, sub/copy-long-name.bin and its link). Today that count is 1.
Under the "fail closed" policy, DryRun's Files to delete drops to 0 for that same tree, because the non-keeper has another name. Stats then has to make the same exclusion, or the two disagree again. That change belongs on top of Count a hardlinked file once in Stats [patch] #191's Stats code. Doing it on a separate branch from main would either stack on Count a hardlinked file once in Stats [patch] #191 or conflict with it.

Plan once #191 is merged:

FindDuplicates records, per representative, how many other scanned paths share its FileIdentity. The collapse already computes this and then throws it away.
DeleteDuplicates skips a non-keeper that has other names, with a SkippedFile reason ("has N other hardlink(s); deleting this name would free nothing"), and adds nothing to bytesReclaimed.
DuplicateReport.PlanDeletions lists that copy as kept with the same reason, and leaves it out of Files to delete / Space to reclaim. Stats' Duplicate files uses the same rule.
Add the two-hardlink non-keeper test to SameFileTests.cs, and update Count a hardlinked file once in Stats [patch] #191's copy-plus-hardlink-pair assertion to the new count.

One limit to record: only links inside the scanned tree can be detected this way. The runtime's SystemNative_Stat structure has no link count, so a link outside the tree (an rsnapshot sibling, say) still needs a native st_nlink read on Unix. NumberOfLinks is already in ByHandleFileInformation on Windows. Decide whether that's in scope.

Generated by Claude Code

Navigation Menu

Repository navigation

Deduplicate removes one name of a hardlinked duplicate and reports the space as reclaimed, but frees nothing and the next Scan finds the same duplicate again #190

Triage

Triage

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	private static IEnumerable<AbsoluteFilePath> CollapseSameFile(IEnumerable<AbsoluteFilePath> paths)
	{
	HashSet<FileIdentity> seen = [];

	return OrderByPreference(paths)
	.Where(path => !FileIdentity.TryGet(path.WeakString, out FileIdentity identity) \|\| seen.Add(identity));

	if (TryDelete(file, errors, out long fileSize))
	{
	deletedCount++;
	bytesReclaimed += fileSize;
	}

Deduplicate removes one name of a hardlinked duplicate and reports the space as reclaimed, but frees nothing and the next Scan finds the same duplicate again #190

Description

Activity

matt-edmondson commented on Oct 19, 2026

Triage

matt-edmondson commented on Oct 19, 2026

Triage

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions