erofs: add 'apko erofs mount/umount' sub commands. - #2415
Conversation
Restores the `apko erofs mount` and `apko erofs umount` surface that 1fe041e descoped so chainguard-dev#2249 could land as a writer plus `ls`: the mount_linux.go orchestration, the kernel/fuse driver, the mount state file, the non-linux stubs, and their tests. Nothing here had callers outside the package, so it comes back private: the driver interface, newDriver, resolveMode, the mode type and its constants, mountState, stateSchemaVersion, statePath, writeState, loadState and removeState. Options is replaced by an exported MountOptions holding plain strings, which is all the CLI needs to say: Mount(ctx, src, dest string, opts MountOptions) error Mount returns only an error now. It used to hand back *MountState, but that type is private and the CLI discarded it anyway. Ls keeps the signature it has on main -- a plain arch string, not an Options bag -- and does not get its `--mode` flag back. That flag was documented as "accepted for symmetry with mount and ignored", which is not worth restoring. This commit deliberately restores behavior that is known to be wrong; the rest of the PR fixes it. In this state Unmount still passes whatever paths the state file names to umount as root without checking them, a partial unmount cannot be recovered by rerunning the command, and the default mount is writable and discards what was written on umount. The docs' overlayfs section is not restored: it was removed by e5778f1, the layering descope, and its recipe only works once multi-layer splitting returns. Refs chainguard-dev#2408 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Mount and Unmount built their driver internally, so the ~300 lines of orchestration around it -- cleanup LIFO, state file lifecycle, unmount policy -- could not be driven with a fake. driver_linux_test.go covered only the argv builders, and nothing anywhere executed the orchestration at all. Add a driverFactory type and unexported mountWith/unmountWith that take it; the exported Mount and Unmount now just pass newDriver. Tests pass a factory returning a fake that records calls. The kernel-vs-fuse umount choice moves into the driver as a new Unmount(ctx, mp) method, replacing unmountOne's switch on drv.Name(). That also puts unmountBlob behind the seam: it now calls the fuse driver's Unmount, which is already the kernel-umount-then-fusermount chain unmountBlob was open-coding, so a blob teardown no longer shells out from the orchestration layer. Behavior is unchanged. Six tests, all through a fake: mount order and recorded state for a three-layer image, lowerdir reversal, LIFO teardown with no state file left behind when overlay assembly fails, refusal to clobber an existing state file, the single-layer read-only short circuit skipping overlay entirely, unmount following the recorded order and cleaning up, and a stateless dest being treated as a blob mountpoint. This lands first because the fixes that follow -- state file containment, partial-unmount recovery, a read-only default -- are only testable through it. Refs chainguard-dev#2408 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Unmount read <dest>/.apko-erofs-mount.json and handed every path it
named to umount, in kernel mode as root. loadState checked only JSON
shape and schema version. The state file lives inside dest, so anyone
who can write there chooses its contents -- and the docs suggest shared
destinations like /mnt/apko-erofs. A planted file naming /home had root
unmount /home.
Validate in loadState, so every caller is covered:
- the recorded Dest must equal the dest being unmounted. Without
this, a file that is internally consistent about some other
directory still passes a containment check.
- every entry in Mounts must be absolute and must be a path a mount
actually creates under dest: <dest>/merged or <dest>/layers/NN.
The second check is deliberately a whitelist rather than plain
containment. Containment alone would still accept <dest>/upper or
<dest>/anything, and mounts are only ever made at those two shapes, so
nothing legitimate is rejected. It also subsumes traversal: a path that
climbs out with .. cannot match either shape.
An empty Mounts list is rejected too. Mount never writes one, and
Unmount removes the file rather than emptying it.
Tests cover eight rejected shapes -- absolute elsewhere, dest's parent,
traversal through dest and through layers/, a relative path, dest
itself, an unexpected subdir, a non-numeric layer name -- plus the dest
mismatch, the empty list, and a legitimate layout being accepted. The
one that states the actual guarantee is in mount_linux_test.go: a
planted state file makes Unmount fail without a single path reaching the
driver, rather than being noticed afterwards.
All of these fail against the previous code.
Refs chainguard-dev#2408
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
On EBUSY part-way through, Unmount told the user to rerun `apko erofs umount`. That rerun could not succeed. Mounts is [merged, layers/NN..00] and the state file was never rewritten, so the second attempt started again at merged -- already unmounted -- and umount(8) exits 32 on a path that is not a mountpoint. It failed there without ever reaching the layer that was actually busy. Recovery meant unmounting by hand and deleting the state file. Drop each entry as it comes down, and on failure rewrite the state file with what is left before returning. The file then describes reality, so a rerun resumes at the entry that was busy. writeState is CreateTemp+Rename in the same directory, so a rerun never sees a half-written file. The error also names the count still up and the exact command to repeat. The fake driver now refuses to unmount a path twice, mirroring umount's exit 32. Without that a rerun would pass simply because the fake was willing to take merged down again, which is the bug this commit fixes. The new test unmounts a three-layer image with layers/01 held busy, asserts merged and layers/02 came down, asserts the state file now lists exactly [layers/01, layers/00], then clears the error and reruns: the teardown completes and the state file is gone. It fails against the previous code at the state file assertion. Refs chainguard-dev#2408 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The default mount was writable, and umount then RemoveAll'd <dest>/upper with no flag and no warning, silently discarding everything written through it. Read-only fits what these commands are for -- looking at an image apko just built -- so writes become opt-in. MountOptions.ReadOnly becomes MountOptions.Writable, so the zero value is the safe one and a library caller that fills in nothing gets a read-only mount. The CLI's --read-only becomes --rw. upper and work are only created when writable; overlayfs takes a lowerdir-only stack otherwise. A single-layer image now skips overlayfs by default rather than only when asked, which is every image apko can currently produce. umount no longer removes upper at all. For a read-only mount there is nothing there; for a writable one it is the only copy of what was written, so it is left behind and its path logged. merged, work and layers still go. mountState gains a writable field so umount knows which case it is in. No schema bump: mount/umount has never been in a release, so there are no v1 state files to stay compatible with, and the field's zero value reads as read-only anyway. Two tests: a two-layer default mount asserts the overlay is assembled read-only, upper and work are absent, and the state file agrees; the --rw case asserts the overlay is writable, writes a file into upper, unmounts, and requires the file to still be there afterwards. Refs chainguard-dev#2408 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1b985c0 to
3820ef5
Compare
ociDirWithLayers returned an unused dir string (unparam); drop it. Simplify a filepath.Dir wrapper lambda to the function itself (gocritic).
b6c5162 to
a5264b2
Compare
The privileged CI job added in chainguard-dev#2414 exercised mount(8) directly, so nothing anywhere ran the orchestration in pkg/erofsmount against a real kernel -- the unit tests drive it through a fake driver, which cannot catch a wrong mount(8) invocation or a layout that overlayfs rejects. Extend the script with four sections after the existing comparison: - Read-only image mount: assert the single-layer short-circuit (the layer straight at merged, no layers/ or upper/), check the state file's mode, dest, writable and mounts, and diff the mounted tree against the `apko erofs ls` listing already computed above. - `--rw`: assert the overlay layout (layers/00 plus merged) and the LIFO order recorded in the state file, write a file through the mount, and check umount left it in upper/ rather than deleting it. - A raw blob, which carries no state file, so umount takes the single-mountpoint fall-back. - A tampered state file naming a decoy tmpfs outside DEST: umount must fail *and* leave the decoy mounted. Asserting on the message alone would pass against code that ran the umount anyway. Cleanup no longer tracks one mountpoint. It reads /proc/self/mounts and unmounts everything under the workdir, deepest first, which covers the mounts apko makes and a mount left behind by a failure mid-test; the rm -rf now runs under sudo since apko created part of the tree as root. The two normalization pipelines become functions so the mounted-tree comparison can be reused, and the mount sections run apko through sudo via an absolute path.
mattmoor
left a comment
There was a problem hiding this comment.
I verified all five sections of the PR body against the code, and they hold: the restored surface is private with Mount/Unmount/MountOptions as the only new exports, the seam is behavior-preserving (argv builders byte-identical to the pre-descope code), the resumability test is honest in exactly the way that matters (the fake's exit-32 mirroring means the rerun passes only because of the state rewrite; delete the rewrite and it goes red), read-only-by-default and upper preservation check out end to end, and the erofs CI job provably executed all four new script sections on this head, from a fork, including the behavioral decoy assertion rather than message-matching. The four bugs from the #2249 review are fixed as described.
I'm holding approval on one substantive finding, inline on state.go: the whitelist validates strings, and the attacker in its own threat model can reroute a root umount with a symlink instead of a tampered file. Disposition there (hardening or an explicitly documented residual) and I'm happy to approve; two independent verification passes converged on it, so I don't think it's a misread.
Smaller notes, none blocking, take or file with #2408:
validateaccepts duplicateMountsentries (e.g.mergedtwice). No escape, since both are inside dest, but the second umount exits 32 and wedges the teardown into the rewrite loop.unmountBlobroutes through the fuse driver, so on a box without fusermount the returned error is "neither fusermount3 nor fusermount found in PATH", swallowing the kernel umount's actual stderr (e.g. "not mounted"). The pre-descope inline code reported both.- The tampered-state script section accepts any nonzero exit from
apko erofs umount; the decoy-still-mounted assertion narrows it, but pinning the refusal (state file untouched, or the validate error text) would distinguish "refused" from "crashed first". - The fuse path still never executes against reality anywhere: CI runs as root, so erofsfuse, fuse-overlayfs, and the fusermount fallback chain are covered only by argv-shape tests. Fine to leave noted in #2408.
loadState's whitelist constrains the *string* each state file entry holds -- <dest>/merged or <dest>/layers/NN -- but umount(8) canonicalizes its argument before unmounting. So the escape it was meant to stop is still reachable one step out: plant <dest>/merged as a symlink to /home along with a state file that passes validate, and a root `apko erofs umount <dest>` takes down /home anyway. Write access inside dest is the precondition for both, so the symlink costs an attacker nothing. checkResolved requires each recorded mountpoint to resolve to the path its name claims, immediately before it is handed to the driver. It compares against dest with dest's *own* symlinks resolved, so a dest legitimately reached through one -- somewhere under /var/run, say -- keeps working and only the components below it have to be real. The check sits inside the LIFO loop rather than in a pre-pass so that a missing entry still fails exactly where umount(8) would have, leaving the earlier entries unmounted and the state file rewritten. The same symlink works on the mount side, where MkdirAll is satisfied by a link to an existing directory: <dest>/merged pointing at /etc would have the layer mounted over /etc, and with the above in place Unmount would then refuse to take it down. ensureDir now lstats and insists on a real directory. None of this closes the hole, and the comment and docs say so: the check and the umount it guards are separate steps, so someone who can write in dest can still swap a directory for a symlink in between. Closing it outright wants openat2(RESOLVE_NO_SYMLINKS) and umount -c against /proc/self/fd/N, which is a much larger change. The guidance stays "use a DEST only you can write to". hack/test-erofs.sh grows the symlinked-DEST/merged case beside the existing outside-DEST one, sharing a single tmpfs decoy through a new plant_state helper; both assert the decoy is still mounted afterwards. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
fuseDriver.Unmount tries kernel umount first and falls back to fusermount, but it discarded the kernel error. Unmount is also what unmountBlob uses, since a raw blob has no state file and so no recorded mode -- which makes the kernel attempt the one that failed for the interesting reason. A busy blob mount on a host without fusermount installed therefore reported "neither fusermount3 nor fusermount found in PATH" instead of "target is busy", pointing at a missing package rather than at whatever was holding the mount open. errors.Join on both failure paths, so the caller sees what each tool actually said. The test drives Unmount with PATH set to an empty directory and asserts the result names both halves. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Not deleting <dest>/upper on umount is right -- for a --rw mount it is the only copy of everything written through the overlay -- but it left the next mount at that dest to walk into it. MkdirAll is happy with an upper that already holds the previous session's writes and the overlayfs metadata they carry, possibly for an entirely different image, and stacks it over unrelated lowers. The umount after that hands both sessions back as one. Mount now refuses a --rw mount when <dest>/upper is not empty, naming the count and saying to move or remove it. Read-only mounts never touch upper, so they are unaffected. That alone would make every --rw dest single-use, so umount cleans up the empty case: os.Remove rather than os.RemoveAll, which is the whole point of the choice. os.Remove cannot delete a non-empty directory, so writes survive by construction and the guarantee cannot be undone by a later edit here; ENOTEMPTY is what logs where they were left. An upper nothing was written through goes away and the dest is immediately reusable. hack/test-erofs.sh asserts a second --rw mount over the sentinel is refused, and that a --rw round trip writing nothing leaves no upper and can be repeated. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every path in an overlayfs option string here is derived from the DEST the user named, and none of them were escaped. overlayfs splits the option string on "," and lowerdir on ":", and treats "\" as an escape -- ovl_next_opt and ovl_split_lowerdirs on the way in, ovl_unescape for upperdir and workdir. A "," fails loudly as an unrecognized option, but a ":" does not: it turns one lowerdir into two, which can compose a wrong stack rather than failing. escapeOverlayPath quotes all three, applied to each lowerdir entry and to upperdir and workdir, through an overlayOpts helper the kernel and fuse-overlayfs builders now share. The replacer runs in a single pass, so the backslash rule cannot re-escape the backslashes the other two introduce. The mountpoint is its own argv element rather than part of an option value, so it stays verbatim -- the test checks that too. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Per review: checkResolved narrows the symlink window but cannot close it, because umount(8) canonicalizes its argument. Whatever the check saw, the exec that follows resolves the path again, so a symlink swapped into <dest>/merged in between is still followed -- and in kernel mode that is a root umount of wherever it points. umount(2) with UMOUNT_NOFOLLOW refuses a symlinked final component in the same syscall that unmounts, so there is no window to race at all. kernelUnmount replaces every umount(8) exec: kernelDriver.Unmount, the teardown closures from MountLayer and AssembleOverlay, and the kernel attempt fuseDriver.Unmount makes first. buildKernelUmountArgs and its argv test go with them. The errno for the symlink case is EINVAL, which is also what "not a mountpoint" returns, so the refusal is named explicitly rather than surfaced as "invalid argument". That lstat is for the message only -- the flag is what provides the guarantee, so racing it weakens nothing. checkResolved stays, with a narrower job: UMOUNT_NOFOLLOW covers the final component, and a symlinked *parent* (<dest>/layers itself) is resolved during the kernel's own path walk, so that half is still a check followed by a syscall. Its comment and docs/erofs.md now record the split rather than claiming the whole thing is closed. The unit test can only pin the wording: unmounting a symlink to a directory that is not a mountpoint fails with or without the flag, so nothing unprivileged can tell them apart. hack/test-erofs.sh can, and now does -- a symlink pointed at a live tmpfs, passed in as DEST so nothing validates it first. Without the flag that tmpfs comes down. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
mountImage stat'd the state file and then, several steps later, wrote it. Calling that "refuse to clobber an existing mount" overstated it: two mounts aimed at one dest both see it free and both proceed, and the second overwrites the first's record, stranding its mounts with nothing left that names them. claimState creates the file empty with O_CREATE|O_EXCL before anything is mounted, and writeState's rename replaces it with the real record at the end. The claim is registered as a cleanup, so every error path releases it. O_EXCL also declines to follow a symlink parked at that path, which the plain create would have. That leaves a window where a killed mount leaves an empty file behind, so loadState now names it -- "a mount was interrupted before it finished" beats a JSON syntax error for someone deciding what to do about it. Also from review: validate rejects a mountpoint listed twice. A repeat is inside dest so it escapes nothing, but the second umount of it fails on a path that is no longer a mountpoint, wedging the teardown on an entry that is already done. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every failure out of the unmount loop got the same suffix: "rerun `apko erofs umount DEST` once they are no longer busy". That is the right advice for a busy mountpoint and useless for a checkResolved rejection, which no amount of waiting clears -- the path resolves somewhere else and will keep doing so until someone removes the symlink. Split the two. A rejection says what it refused and where the remaining mounts are recorded; only a real umount failure keeps the retry hint. The state rewrite both share moves into a small helper so it cannot drift between them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
From review: fakeDriver's factory discarded its mode argument, so the unit layer certified the mode plumbing without ever looking at it. unmountBlob's deliberate modeFuse choice -- the kernel-then-fusermount chain that covers a blob whose mode was never recorded -- could regress to kernel and every test here would still pass, breaking only for real fuse users. Same for unmountImage ignoring st.Mode. The factory records each mode it is handed. The blob fall-back test now asserts modeFuse, and a new test walks a mount and unmount, checking that the flag decides the first and the state file decides the second. Also fills the other hole in this file: nothing covered a Preflight failure, which is what reports a missing mount(8) or a kernel mount without root. The test asserts it stops before any mount, any overlay, and before the dest claim leaves a state file behind. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
hack/test-erofs.sh: - mountpoint(1) is what every assert_mounted call runs, but it was in neither the required-tool loop nor the Requires header. A box without util-linux failed at the first assertion instead of at the preflight. - The --rw teardown checked merged and layers were gone but not work, which umount removes alongside them. - Both tampered-state cases accepted any nonzero exit, which a crash before the check ever ran would also produce. Asserting the planted state file is still there -- and the symlink undisturbed -- is what says it was refused rather than that apko fell over. docs/erofs.md: the "unknown filesystem type 'erofs'" note appeared twice, once for `apko erofs mount` and once for the manual equivalents below it. The first covers both modes and names --mode=fuse; drop the second. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
kernelUnmount reports "umount <path>: ..." and both callers prefixed it again, so the EROFS job's log came back with umount /tmp/.../decoy-link: umount /tmp/.../decoy-link: refusing to follow a symlink Drop the outer prefix at both sites; the driver already names the path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Left over from the switch to umount(2): Preflight still refused to mount when umount(8) was missing from PATH, though nothing execs it any more. mount(8) is the only binary the kernel driver still needs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
mattmoor
left a comment
There was a problem hiding this comment.
Approving at 28acdfd. I re-verified the whole response round: the held finding is settled with three layers rather than the minimum ask. UMOUNT_NOFOLLOW via umount(2) closes the final-component race atomically in every path that consumes state-file entries (kernel driver, both teardown closures, the fuse driver's kernel attempt; buildKernelUmountArgs is gone), checkResolved covers the symlinked-parent half with the residual race documented honestly in code and docs, and the mount side refuses a symlinked mountpoint up front. The e2e script now proves all three refusals behaviorally, asserting the decoy tmpfs remains mounted and the state file survives, and the EROFS job ran those sections on this head.
All six secondary notes from the prior round are fixed as well, several beyond what was asked: the O_EXCL state-file claim kills the mount-side race while the partial-unmount rewrite correctly keeps the atomic CreateTemp+rename, duplicate mount entries are rejected, fuse umount failures are joined instead of swallowed, the fake driver now observes the mode it is handed with real assertions, the rerun hint is only offered when a rerun can help, and the upperdir and option-escaping hardening are both welcome. The last commit also cleared the stale umount(8) preflight requirement I was about to note.
Three take-or-leave nits inline; none blocking.
| ) | ||
|
|
||
| // driver wraps the externally-invoked mount and umount commands used by Mount. | ||
| // Two implementations exist on Linux: kernelDriver shells out to mount(8) and |
There was a problem hiding this comment.
Stale half of what 28acdfd fixed: kernelDriver no longer shells out to umount(8); unmounts go through umount(2) directly. Worth updating so the doc comment and Preflight tell the same story.
| return []string{"mount", "-t", "overlay", "-o", opts, "overlay", merged} | ||
| } | ||
|
|
||
| func buildFuseOverlayArgs(lowers []string, upper, work, merged string, readOnly bool) []string { |
There was a problem hiding this comment.
Take or leave: this feeds kernel-style escapes (\:, \,, \\) to fuse-overlayfs, whose own lowerdir colon-split has historically not honored \: the way the kernel does. A dest containing : was broken on the fuse leg before this PR too, so no regression there, but a dest containing a bare backslash is a narrow new fuse-leg-only regression (it previously passed verbatim). One comment acknowledging the parity assumption, or a quick check against the fuse-overlayfs option parser, would cover it.
| // interesting reason -- returning only the fusermount error turns "target is | ||
| // busy" into "neither fusermount3 nor fusermount found in PATH". | ||
| func (d *fuseDriver) Unmount(ctx context.Context, mp string) error { | ||
| kerr := kernelUnmount(ctx, mp) |
There was a problem hiding this comment.
Take or leave: when kernelUnmount fails with the symlink refusal, fusermount is still exec'd on the same path. It is safe (fusermount has carried its own internal UMOUNT_NOFOLLOW since the CVE-2010-3879 fix, and the CI decoy-link log shows it refusing), and state-file paths pass checkResolved first, so this is purely surface-shrinking: skipping the fusermount attempt when the kernel failure was the symlink refusal would also keep that refusal from being buried in a joined error.
The three take-or-leave nits from the review of #2415, all in `pkg/erofsmount/driver_linux.go`. One commit each. ### 1. The interface comment describes an unmount that no longer happens `28acdfd7` dropped the `umount(8)` requirement from `Preflight` but left the `driver` doc comment saying `kernelDriver` shells out to it. Text only. ### 2. Stop at a symlink refusal in fuse mode `fuseDriver.Unmount` tries the kernel unmount and falls back to `fusermount`, which is right when the mount was simply made by the other tool and wrong when the kernel refused because the mountpoint is a symlink. `fusermount` has carried its own `UMOUNT_NOFOLLOW` since the CVE-2010-3879 fix, so it refuses the same path again and the joined error buries the reason: ``` Error: umount .../decoy-link: refusing to follow a symlink /usr/bin/fusermount3 -u .../decoy-link: exit status 1: ... Invalid argument ``` That second line is noise once the first has decided the matter. `kernelUnmount` now wraps a sentinel for the symlink case so `Unmount` can return it as-is. State-file paths pass `checkResolved` before reaching a driver, so this shrinks surface rather than fixing an exposure. ### 3. Refuse paths fuse-overlayfs cannot round-trip `escapeOverlayPath` emits the kernel's escaping to both legs, and review asked whether fuse-overlayfs honours it. **Measured against 1.17 rather than assumed** — mounting a tree whose path contains each character, escaped and verbatim, against a clean-path control: | in `DEST` | escaped (today) | verbatim (before #2415) | | --- | --- | --- | | *(control)* | ok | ok | | `,` | **ok** | broken — split into separate options | | `:` | broken | broken | | `\` | broken | broken | Two things follow. The escaping is a straight win for the comma on this leg and cannot help with the other two. And it is **not** the regression it looked like: a bare backslash was already eaten before the escaping went in, so that dest fails identically before and after. What is left is a bad failure mode rather than a wrong mount — fuse-overlayfs reports `cannot resolve path .../dd`, naming a path the user never wrote. So `:` and `\` are refused up front on the fall-back leg, with the reason and a pointer at `--mode=kernel`, which handles them correctly. The kernel overlay attempted just before the fall-back is unaffected: only the tool that cannot cope declines. The probe script is not included — it needs a live `fuse-overlayfs` and the result is a fact about that tool, not about apko, so it is recorded as a table in the comment on `fuseOverlayUnsupported` instead. ### Verification `gofmt -l` clean, `golangci-lint run -n` reports 0 issues, `go build ./...` and `GOOS=darwin go build ./...` both succeed, and `SOURCE_DATE_EPOCH=0 go test ./...` passes. The fuse-mode symlink test was checked to fail without its change. Both new unit tests do run in CI, in the `go-tests` job on the non-root runner — where the symlink test passes via the EPERM+lstat path rather than EINVAL, since the kernel checks `may_mount()` before `path_mounted()`. What never runs anywhere is the **e2e** fuse leg: `hack/test-erofs.sh` goes through sudo, so `resolveMode` picks kernel. That is one of the two items already filed against #2408. ### Review follow-ups Four commits addressing the review, one concern each: - `06801ac9` — move `errSymlinkedMountpoint` above `kernelUnmount`'s doc block (it had captured the whole UMOUNT_NOFOLLOW rationale as its own godoc), and fix the two comments commit 2 made untrue: the lstat is no longer "for the message only", and "Both failures are reported" is now conditional. The lstat comment records the EPERM-vs-EINVAL detail, so a later cleanup keying it on EINVAL does not silently break the fuse leg. - `64166080` — run `checkFuseOverlayPaths` before the `fuse-overlayfs` `LookPath`, since the refusal is terminal and "not installed" is fixable. With that order an empty-PATH test can pin the wiring: raw paths, and read-only excluding upper/work. Mutation-checked both ways. - `cabc8693` — `--mode=kernel (as root)`; the user on this leg is non-root by construction, so the bare suggestion bounces off `kernelDriver.Preflight`. On the fifth comment: yes, the refusal is deliberately version-blind rather than a versioned check — "refuse until re-measured". There is no probe, and 1.17 is what the comment records. Refs #2408 --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
#2418) Section 2 of #2408: brings back the EROFS layer split, descoped in e5778f1 so #2249 could land as a single-layer writer plus `ls`, with the five bugs review found fixed on top. #2415 has since merged, and this is rebased on it. The code is still disjoint — that PR is `pkg/erofsmount` and the CLI, this one is `pkg/build` — they meet only in `hack/test-erofs.sh`. ### 1. Restore, unchanged `pkg/build/erofs_layers.go` and its test come back exactly as they were, and the `layering` + `format: erofs` rejection leaves `ImageConfiguration.Validate` and `buildLayers`. This commit deliberately restores the known-broken behavior; the four after it are the fix, and that delta is the point of the PR. ### 2. Package routing — the bug that inverted the whole thing The type assertion asked `info.Sys()` for a `Package()` method. tarfs, which is what a real build walks, returns a fresh `*tar.Header` from `Sys()` and hangs `Package()` off the FileInfo itself — the receiver `splitLayers` asserts on. So it never succeeded outside a fixture: every file landed in the top layer, each group layer held only ancestor directories plus a partial installed db (that case keys on the path string, so it still fired), and per-layer scanners read a db naming packages whose files were not there. Assert on `info`, and turn a `packageToWriter` miss into an error rather than a silent fall back to `top`. `TestSplitErofsLayers` could not catch this: it drove the split through `apkfs.NewMemFS()`, which implements `Package()` nowhere, so the fixture had zero package-owned files by construction and passed identically with and without routing. It is replaced by `erofs_layers_test.go`, which builds its fixture with `pkg/tarfs` and installs files through `WriteHeader` with an `*apk.Package`, the way an apk install does. Its routing assertions fail against the previous code. ### 3. Directory-only subtrees Directories were recorded during the walk and materialized only by `emitAncestors`, which runs for non-directory entries. A directory whose subtree holds no file was therefore never written anywhere: `/tmp`, `/run`, `/var/empty`, every empty dir and every mount point were absent from the merged view. Each directory is now emitted into its owning writer as it is walked, which is what both siblings already do. ### 4. Temp files and fds No error return out of `splitErofsLayers` closed or removed the per-layer temp files, and each go-erofs `Writer` also holds an unlinked spool fd that only `Close` releases. The CLI happened to be bounded by its `MkdirTemp`/`RemoveAll` wrapper; a library caller accumulated both until process exit. The per-writer state moves into an `erofsGroupWriter` type with `finish`/`discard`, and a deferred sweep discards every writer unless the function reaches its return. Still no upstream abort API, so `discard` writes an image it then removes. ### 5. Build time `newErofsGroupWriter` passed `WithBuildTime` only for a non-zero build time — the exact case `erofsBuildTime`'s comment describes, where go-erofs stamps `time.Now()` from `Close` and a library caller gets different layer digests every build. It now uses `erofsBuildTime`, same as `writeErofs`. apko's own CLI was never affected. ### 6. Drive it from `hack/test-erofs.sh` The privileged job from #2414, extended by #2415, only built and mounted a single layer, because that was all apko could produce. The script now builds the same config again with `layering` and adds, after the existing comparisons: - layer roles and mediaTypes across the manifest, and `digest` == the sha256 of each blob; - `fsck.erofs` on every layer, and a kernel mount of every layer stacked as read-only overlayfs lowerdirs; - `apko erofs ls` on the layered OCI directory diffed against that overlay mount — `Stack`'s merge only becomes reachable for apko's own output now that splitting is back, and this is the first thing to compare it with what the kernel assembles; - every non-final layer must hold a regular file other than the partial installed db, which is exactly the shape the routing bug produced; - the merged tree diffed against a **tar** build of the same config unpacked in layer order. The tar split is the reference for what splitting must preserve, and a directory that reaches no layer shows up here and nowhere else. Both builds resolve from one `apko lock` output so a package published between them cannot make them differ. The unpack uses `--numeric-owner`; without it GNU tar resolves each header's uname/gname against the runner's `/etc/passwd` and invents ids for `lp`, `mail`, `news` and `uucp`. The layered section runs last, after #2415's `apko erofs mount` / `apko erofs umount` checks, and reuses their helpers — `fail`, `assert_mounted`, `normalize_ls`, `tree_listing`. Their `cleanup` already unmounts everything under the workdir deepest-first, so the layer and overlay mounts need no separate bookkeeping. The one existing line this touches is the whitespace-in-paths guard, lifted into a `check_ls_whitespace` function so the layered listing gets it too. ### Not in this PR Whiteout support in `Stack` — the last item in section 2 — is really the layer-horizon fix from section 5, and is reader-side and independently testable. Left for its own PR. ### Verification `gofmt -l` clean, `golangci-lint run -n` reports 0 issues, `go build ./...` and `GOOS=darwin go build ./...` both succeed, and `SOURCE_DATE_EPOCH=0 go test ./...` passes. `shellcheck` is clean on the script. Each fix was checked to fail without its change: the four new tests are red against the restore commit and green after their own commit. Locally, a layered `wolfi-base` build produces five layers whose merged `apko erofs ls` listing is identical to the single-layer build's, and whose non-top layers are 0.7–7.4 MB rather than the directories-only skeletons the routing bug produced. Refs #2408 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Section 1 of #2408: brings back
apko erofs mount/apko erofs umount,descoped in 1fe041e so #2249 could land as a writer plus
ls, with the fourbugs that review found fixed on top — and, after review here, four more.
#2414 has landed, and
hack/test-erofs.shis extended to driveapko erofs mountandapko erofs umount— the first thing anywhere that executes thisorchestration for real.
1. Restore, with the package API private
Everything restored had no callers outside
pkg/erofsmount, so it comes backunexported: the driver interface,
newDriver,resolveMode, themodetypeand constants,
mountState,stateSchemaVersion,statePath,writeState,loadState,removeState.Optionsis replaced by an exportedMountOptionsof plain strings, which is all the CLI needs:Lskeeps main's signature — a plain arch string, not an options bag — anddoes not get its
--modeflag back; it was documented as "accepted forsymmetry with mount and ignored".
The first commit deliberately restores the known-broken behavior; the ones
after it fix it. Splitting it the other way would put the root-
umountbug onmainin between.2. A test seam
MountandUnmountbuilt their driver internally, so ~300 lines oforchestration could not be driven with a fake —
driver_linux_test.gocoveredonly the argv builders, and nothing anywhere executed the orchestration.
A
driverFactoryand unexportedmountWith/unmountWithtake it; theexported entry points pass
newDriver. The kernel-vs-fuse umount choice movesinto the driver as an
Unmount(ctx, mp)method. The fake records the mode itis asked for, so the tests observe rather than assume the two places the mode
comes from: the
--modeflag on the way up, the state file on the way down.3. Validate the state file before unmounting — the security one
Unmountread<dest>/.apko-erofs-mount.jsonand handed every path it namedto
umount, as root in kernel mode.loadStatechecked only JSON shape andschema version. The state file lives inside
dest, and the docs suggestshared destinations — so anyone who could write there could plant a file
naming
/homeand have root unmount/home.Three layers, because the first two are not enough on their own:
Mountsentry must be absolute anda path a mount actually creates:
<dest>/mergedor<dest>/layers/NN. Therecorded
Destmust equal the dest being unmounted, and no entry may repeat(a repeat is inside dest, but unmounting it twice wedges the teardown).
Containment alone would still accept
<dest>/upperor<dest>/anything,and a whitelist subsumes traversal, since a path that climbs out with
..matches neither shape.
umount(2)withUMOUNT_NOFOLLOW, notumount(8). The whitelistconstrains the path string;
umount(8)canonicalizes, soln -s /home <dest>/mergedbeside a well-formed state file redirects a root umount withnothing to validate against. No check-then-exec can close that — whatever
the check saw, the exec resolves again. The syscall flag refuses a symlinked
final component in the same call that unmounts, so there is no window.
UMOUNT_NOFOLLOWcovers the finalcomponent only; a symlinked parent (
<dest>/layersitself) is resolvedduring the kernel's own path walk.
checkResolvedrejects that, comparingagainst dest with dest's own symlinks resolved so a dest legitimately
reached through one still works.
The mount side had the same hole in reverse:
MkdirAllis satisfied by asymlink to an existing directory, so
<dest>/mergedpointing at/etcwouldhave the layer mounted over
/etc.ensureDirnow insists on a realdirectory.
The residual is stated rather than papered over, in the code comment and
in
docs/erofs.md: the parent-component check and the unmount it guards aretwo steps, so someone who can write inside
destcan still race them. Closingthat wants
openat2(RESOLVE_NO_SYMLINKS), which is a much larger change. Theguidance stays "use a
DESTonly you can write to".4. Make a partial unmount recoverable
On EBUSY part-way through, the error said to rerun
apko erofs umount. Thatcould never work:
Mountsis[merged, layers/NN…00]and the file was neverrewritten, so the rerun started at
merged— already gone — and failed therebefore ever reaching the busy layer.
Entries are now dropped as they come down, and the file is rewritten with
what's left before returning. A rejected path gets a different message: no
amount of waiting clears a symlink, so only a real umount failure carries the
retry hint.
5. Read-only by default
The default mount was writable, and
umountthenRemoveAll'd<dest>/upperwith no flag and no warning, silently discarding everythingwritten through it.
MountOptions.ReadOnlybecomesMountOptions.Writable, so the zero value isthe safe one.
--read-onlybecomes--rw.upperandworkare onlycreated when writable, and a single-layer image now skips overlayfs by
default — that is every image apko can currently produce.
umountusesos.Removeonupper, notos.RemoveAll, which is the pointof the choice: it cannot delete a non-empty directory, so writes survive by
construction rather than by remembering not to delete them. An
uppernothingwas written through goes away, so the dest stays reusable; one that holds
something stays, and a later
--rwmount refuses to start on top of it ratherthan stacking two sessions' writes — possibly from different images —
together.
6. Three more found in review here
from
DESTand none were escaped. overlayfs splits on,and lowerdir on:, honouring\as an escape. A,fails loudly; a:does not — itturns one lowerdir into two and can compose a wrong stack.
fuseDriver.Unmounttrieskernel umount then fusermount, and discarded the first error. Since it is
also the blob fall-back, a busy blob mount on a host without fusermount
reported "neither fusermount3 nor fusermount found in PATH" instead of the
real reason.
destwithO_CREATE|O_EXCLrather than stat-then-write, so"refuse to clobber an existing mount" is true rather than a race between two
mounts aimed at one dest.
7. Drive it from
hack/test-erofs.shUnit tests go through a fake driver, which cannot catch a wrong
mount(8)invocation or a layout overlayfs rejects. The privileged job from #2414 ran
mount -t erofsdirectly, so nothing anywhere ran this orchestration againsta real kernel. Added after the existing
ls-vs-kernel comparison:mode/dest/writable/mounts, and a diff of the mounted tree againstthe
apko erofs lslisting the script already computed.--rw— the overlay layout and LIFO order; a file written through themount must still be in
upper/afterumount, a second--rwmount atthat dest must be refused, and a round trip that writes nothing must leave
no
upperand be repeatable.umounttakes thesingle-mountpoint fall-back.
DEST, a symlinkedDEST/merged, and — the only thing that actually exercisesUMOUNT_NOFOLLOW— a symlink pointed straight at a live tmpfs and handed inas
DEST, so nothing validates it first. Without the flag that tmpfs comesdown. Each asserts the decoy is still mounted and the planted file
untouched; asserting on a nonzero exit alone would also pass for a crash.
Cleanup reads
/proc/self/mountsand unmounts everything under the workdir,deepest first, so a failure mid-test does not wedge the job.
Verification
gofmt -lclean,golangci-lint run -nreports 0 issues,go build ./...and
GOOS=darwin go build ./...both succeed (the non-Linux stubs still lineup),
SOURCE_DATE_EPOCH=0 go test ./...passes, andshellcheckis clean onthe script.
Each fix was checked to fail without its change, by reverting just the fixed
lines and re-running. The one thing a unit test cannot pin is
UMOUNT_NOFOLLOW— unmounting a symlink to a non-mountpoint fails with orwithout it — which is why the script covers it against a live mount instead.
The EROFS job is green with all of the above, and its sudo audit log shows
only two
umount(8)executions in the whole run, both the script's own: apkoitself no longer execs it.
Two items from review are deferred to #2408 rather than fixed here: the state
file goes stale during a succeeding teardown (distinguishing "not mounted"
from "busy" needs
/proc/self/mountinfo, since neither exit 32 norEINVALdoes), and the fuse path has never executed anywhere, because CI runs as root.
Refs #2408