tarfs: compact Entry, reconstructing tar.Header on demand - #2433
Draft
markusthoemmes wants to merge 5 commits into
Draft
markusthoemmes wants to merge 5 commits into
markusthoemmes wants to merge 5 commits into
Conversation
markusthoemmes
force-pushed
the
tarfs-compact
branch
from
August 25, 2026 10:13
add210d to
d2171c7
Compare
coreydaley-cg
added a commit
that referenced
this pull request
Sep 10, 2026
apk-tools records APK-TOOLS.checksum.SHA1 on nearly every regular file as 40 lowercase hex characters. Holding that as a PAX record costs the string, its header, and a slice element per file. Decode it into a fixed [20]byte on the entry and re-encode on materialization. Only the exact lowercase hex spelling is decoded, checked by re-encoding, so any other form round-trips verbatim. Measured on real apks: 446 -> 385 bytes per member on a 133-file package. The decoded-checksum idea is from #2433. Co-authored-by: Claude <noreply@anthropic.com>
A tarfs index retains ~1KB per file: the stored tar.Header costs three time.Times and several always-zero fields, every entry carries a PAX map allocated to hold a single well-known checksum record, and hdr.FileInfo() boxes another allocation. Since the package cache keeps a tarfs index alive per cached package, this multiplies across every file of every package in long-running processes. Store a compact form instead: packed scalar fields, the APK-TOOLS.checksum.SHA1 record hex-decoded into a fixed [20]byte with remaining PAX records (xattrs, mostly) spilled to a nil-by-default map, and *Entry implementing fs.FileInfo itself. Entry.Header becomes a method that reconstructs the header on demand, which is a breaking change for consumers of the Entry.Header field. On a wolfi go-1.25 package (6339 files) this shrinks the index from 987 to 324 bytes per entry (6.0 to 2.0 MB), over half of which comes from replacing the per-entry PAX map with the decoded checksum field. An image built from this branch is byte-identical to one built from main in every blob (layers, manifests, configs), differing only in the vcs-revision annotations.
Narrowing uid, gid and device numbers to int32 silently corrupts ids above 2^31, which tar can carry. Re-deriving the fs.FileMode by hand duplicates headerFileInfo.Mode and can drift from it, so compute it once from hdr.FileInfo() at index time and store it. Name() and Sys() now match archive/tar's FileInfo too: directories are cleaned before taking the base, and Sys returns a *tar.Header. Header() documents that access and change times are not retained.
hex.Decode accepts uppercase, but hex.EncodeToString always emits lowercase, so an uppercase APK-TOOLS.checksum.SHA1 record did not round-trip through Header(). Keep anything but the exact lowercase form apk-tools writes as a verbatim PAX record.
The package had no tests. Fixtures cover directories, setuid and xattr records, symlinks, hardlinks, long names, wide ids, device nodes, fifos, non-hex checksum spellings and format variants, and a seeded property test generates headers across the field space and compares every Header() and FileInfo method with what archive/tar reads back from the same bytes. Open, Stat, Readlink and WalkDir are exercised as well.
markusthoemmes
force-pushed
the
tarfs-compact
branch
from
September 14, 2026 06:48
d2171c7 to
b4b4851
Compare
Ordering fields by size removes 24 bytes of padding, and path.Dir(name) is a prefix slice of name that does not allocate, so storing it separately cost 16 bytes per entry for nothing. Entry goes from 208 to 176 bytes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A tarfs index retains ~1KB per file: the stored tar.Header costs three time.Times and several always-zero fields, every entry carries a PAX map allocated to hold a single well-known checksum record, and hdr.FileInfo() boxes another allocation. Since the package cache keeps a tarfs index alive per cached package, this multiplies across every file of every package in long-running processes.
Store a compact form instead: scalar fields at their original widths, the fs.FileMode computed once from hdr.FileInfo() so the FileInfo view cannot drift from archive/tar, the APK-TOOLS.checksum.SHA1 record hex-decoded into a fixed [20]byte when it is the lowercase form apk-tools writes (any other spelling stays a verbatim PAX record so it round-trips), and remaining PAX records (xattrs, mostly) spilled to a nil-by-default map. *Entry implements fs.FileInfo and fs.DirEntry itself, and Sys() returns a *tar.Header as archive/tar does. Entry.Header becomes a method that reconstructs the header on demand, which is a breaking change for consumers of the Entry.Header field. The only in-tree consumer is contents.Entries(), which already copies into []tar.Header. AccessTime and ChangeTime are not retained since apk-tools never writes them.
The package had no tests. Fixtures cover directories, setuid and xattr records, symlinks, hardlinks, PAX long names, ids above 2^31, device nodes, fifos, non-hex checksum spellings and format variants, and a seeded property test generates headers across the field space and compares every Header() and FileInfo method against what archive/tar reads back from the same bytes.
On a wolfi go-1.25 package (6354 files), measured with runtime.MemStats around tarfs.New and including the index and directory maps:
Over half of the saving comes from replacing the per-entry PAX map with the decoded checksum field. Before the rebase onto the PackageContents refactor, an image built from this branch was byte-identical to one built from main in every blob (layers, manifests, configs), differing only in the vcs-revision annotations.
Supersedes #2493, which has the same diagnosis and fix shape. Fixes #2492.