Repository navigation
Conversation
…layer dedup
Read-only audit of the build-cache bucket: groups {scope}/files/{hash}.tar by
content hash across scopes and reports cross-scope duplicate layers, wasted
bytes, and a delete-candidate manifest. Deletes nothing; physical deletion is
a separate operator-gated step that consumes the manifest.
Complements template-gc (e2b-dev#3742): that tool reclaims unreferenced builds by
liveness in the TEMPLATE bucket; this one reports content-duplicate layers in
the BUILD_CACHE bucket. Neither merges the other's data. Pairs with the
opt-in shared cache scope (e2b-dev#3748/e2b-dev#3749) that prevents new duplicates.
Refs e2b-dev#3742 e2b-dev#3748
AdaAibaby
requested review from
ValentaTomas,
dobrac and
jakubno
as code owners
October 26, 2026 09:04
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A read-only (mark-only) tool,
cmd/layer-dedup-audit, that finds historical cross-scope duplicate template layers in the build-cache bucket and reports wasted bytes + a delete-candidate manifest. It deletes nothing.Why
Layer caching is content-addressed (
cache.HashKeys= sha256 over base + keys, no team), but the storage path is prefixed bycacheScope, and the API historically setcacheScope = teamID(create_template.go,upload_template_layer_files.go). So two teams that build byte-identical layers each hold a private copy at{teamID}/files/{hash}.tar(+ the tiny{teamID}/index/{hash}). The content hash already proves the bytes are identical, so these copies are reclaimable in a single-trust-domain deployment.This tool quantifies that waste and emits candidates; it pairs with the opt-in shared cache scope (#3748 / #3749) which prevents new duplicates.
How
Lists the bucket, keeps
{scope}/files/{hash}.tar, groups by hash, flags any hash present under >1 scope. Keeps the oldest copy per hash as canonical, lists the rest (with their pairedindex/{hash}) inout/candidates.json. Prints scanned/total/wasted summary + top wasters. Self-contained S3 client (TOS/MinIO viaS3_USE_PATH_STYLE).Relation to #3742 (template-gc)
Orthogonal and complementary — both mark-only, each on a different bucket:
{buildID}/{scope}/files/{hash}.tartemplate-gc cannot dedup cross-team duplicates: teamA/teamB each have a live template pointing at their own buildID, so both buildIDs are live roots and both are kept. This tool cannot reclaim orphan build artifacts in the template bucket. They compose: shared scope (#3749) stops new duplicates, existing templates converge on rebuild, template-gc then reaps the orphaned old builds, and this tool marks the build-cache layer tars that will never be hit again.
Safety
Deletes nothing. Deletion is a separate operator-gated step over
candidates.json(quarantine then delete) — same posture as #3742: a cache is more dangerous than reading ground truth, a wrong delete is permanent corruption.Follow-up
Once #3742 merges, refactor onto its
storage.ObjectLister(adds an fs backend too) instead of the direct S3 client here.