Skip to content

erofs: add 'apko erofs mount/umount' sub commands. - #2415

Merged
smoser merged 18 commits into
chainguard-dev:mainfrom
smoser:erofs-mount-restore
Aug 21, 2026
Merged

smoser merged 18 commits into
chainguard-dev:mainfrom
smoser:erofs-mount-restore

Conversation

@smoser

@smoser smoser commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Section 1 of #2408: brings back apko erofs mount / apko erofs umount,
descoped in 1fe041e so #2249 could land as a writer plus ls, with the four
bugs that review found fixed on top — and, after review here, four more.

#2414 has landed, and hack/test-erofs.sh is extended to drive apko erofs mount and apko erofs umount — the first thing anywhere that executes this
orchestration for real.

1. Restore, with the package API private

Everything restored had no callers outside pkg/erofsmount, so it comes back
unexported: the driver interface, newDriver, resolveMode, the mode type
and constants, mountState, stateSchemaVersion, statePath, writeState,
loadState, removeState. Options is replaced by an exported
MountOptions of plain strings, which is all the CLI needs:

Mount(ctx, src, dest string, opts MountOptions) error

Ls keeps main's signature — a plain arch string, not an options bag — and
does not get its --mode flag back; it was documented as "accepted for
symmetry with mount and ignored".

The first commit deliberately restores the known-broken behavior; the ones
after it fix it. Splitting it the other way would put the root-umount bug on
main in between.

2. A test seam

Mount and Unmount built their driver internally, so ~300 lines of
orchestration could not be driven with a fake — driver_linux_test.go covered
only the argv builders, and nothing anywhere executed the orchestration.

A driverFactory and unexported mountWith/unmountWith take it; the
exported entry points pass newDriver. The kernel-vs-fuse umount choice moves
into the driver as an Unmount(ctx, mp) method. The fake records the mode it
is asked for, so the tests observe rather than assume the two places the mode
comes from: the --mode flag on the way up, the state file on the way down.

3. Validate the state file before unmounting — the security one

Unmount read <dest>/.apko-erofs-mount.json and handed every path it named
to umount, as root in kernel mode. loadState checked only JSON shape and
schema version. The state file lives inside dest, and the docs suggest
shared destinations — so anyone who could write there could plant a file
naming /home and have root unmount /home.

Three layers, because the first two are not enough on their own:

  • A whitelist, not containment. Every Mounts entry must be absolute and
    a path a mount actually creates: <dest>/merged or <dest>/layers/NN. The
    recorded Dest must equal the dest being unmounted, and no entry may repeat
    (a repeat is inside dest, but unmounting it twice wedges the teardown).
    Containment alone would still accept <dest>/upper or <dest>/anything,
    and a whitelist subsumes traversal, since a path that climbs out with ..
    matches neither shape.
  • umount(2) with UMOUNT_NOFOLLOW, not umount(8). The whitelist
    constrains the path string; umount(8) canonicalizes, so ln -s /home <dest>/merged beside a well-formed state file redirects a root umount with
    nothing to validate against. No check-then-exec can close that — whatever
    the check saw, the exec resolves again. The syscall flag refuses a symlinked
    final component in the same call that unmounts, so there is no window.
  • A resolution check for the rest. UMOUNT_NOFOLLOW covers the final
    component only; a symlinked parent (<dest>/layers itself) is resolved
    during the kernel's own path walk. checkResolved rejects that, comparing
    against dest with dest's own symlinks resolved so a dest legitimately
    reached through one still works.

The mount side had the same hole in reverse: MkdirAll is satisfied by a
symlink to an existing directory, so <dest>/merged pointing at /etc would
have the layer mounted over /etc. ensureDir now insists on a real
directory.

The residual is stated rather than papered over, in the code comment and
in docs/erofs.md: the parent-component check and the unmount it guards are
two steps, so someone who can write inside dest can still race them. Closing
that wants openat2(RESOLVE_NO_SYMLINKS), which is a much larger change. The
guidance stays "use a DEST only you can write to".

4. Make a partial unmount recoverable

On EBUSY part-way through, the error said to rerun apko erofs umount. That
could never work: Mounts is [merged, layers/NN…00] and the file was never
rewritten, so the rerun started at merged — already gone — and failed there
before ever reaching the busy layer.

Entries are now dropped as they come down, and the file is rewritten with
what's left before returning. A rejected path gets a different message: no
amount of waiting clears a symlink, so only a real umount failure carries the
retry hint.

5. Read-only by default

The default mount was writable, and umount then RemoveAll'd
<dest>/upper with no flag and no warning, silently discarding everything
written through it.

MountOptions.ReadOnly becomes MountOptions.Writable, so the zero value is
the safe one. --read-only becomes --rw. upper and work are only
created when writable, and a single-layer image now skips overlayfs by
default — that is every image apko can currently produce.

umount uses os.Remove on upper, not os.RemoveAll, which is the point
of the choice: it cannot delete a non-empty directory, so writes survive by
construction rather than by remembering not to delete them. An upper nothing
was written through goes away, so the dest stays reusable; one that holds
something stays, and a later --rw mount refuses to start on top of it rather
than stacking two sessions' writes — possibly from different images —
together.

6. Three more found in review here

  • Escape overlayfs option paths. Every path in the option string derives
    from DEST and none were escaped. overlayfs splits on , and lowerdir on
    :, honouring \ as an escape. A , fails loudly; a : does not — it
    turns one lowerdir into two and can compose a wrong stack.
  • Report both umount failures in fuse mode. fuseDriver.Unmount tries
    kernel umount then fusermount, and discarded the first error. Since it is
    also the blob fall-back, a busy blob mount on a host without fusermount
    reported "neither fusermount3 nor fusermount found in PATH" instead of the
    real reason.
  • Claim dest with O_CREATE|O_EXCL rather than stat-then-write, so
    "refuse to clobber an existing mount" is true rather than a race between two
    mounts aimed at one dest.

7. Drive it from hack/test-erofs.sh

Unit tests go through a fake driver, which cannot catch a wrong mount(8)
invocation or a layout overlayfs rejects. The privileged job from #2414 ran
mount -t erofs directly, so nothing anywhere ran this orchestration against
a real kernel. Added after the existing ls-vs-kernel comparison:

  • Read-only image mount — the single-layer short-circuit, the state file's
    mode/dest/writable/mounts, and a diff of the mounted tree against
    the apko erofs ls listing the script already computed.
  • --rw — the overlay layout and LIFO order; a file written through the
    mount must still be in upper/ after umount, a second --rw mount at
    that dest must be refused, and a round trip that writes nothing must leave
    no upper and be repeatable.
  • A raw blob, which carries no state file, so umount takes the
    single-mountpoint fall-back.
  • Three tampered-state cases: a decoy named outside DEST, a symlinked
    DEST/merged, and — the only thing that actually exercises
    UMOUNT_NOFOLLOW — a symlink pointed straight at a live tmpfs and handed in
    as DEST, so nothing validates it first. Without the flag that tmpfs comes
    down. Each asserts the decoy is still mounted and the planted file
    untouched; asserting on a nonzero exit alone would also pass for a crash.

Cleanup reads /proc/self/mounts and unmounts everything under the workdir,
deepest first, so a failure mid-test does not wedge the job.

Verification

gofmt -l clean, golangci-lint run -n reports 0 issues, go build ./...
and GOOS=darwin go build ./... both succeed (the non-Linux stubs still line
up), SOURCE_DATE_EPOCH=0 go test ./... passes, and shellcheck is clean on
the script.

Each fix was checked to fail without its change, by reverting just the fixed
lines and re-running. The one thing a unit test cannot pin is
UMOUNT_NOFOLLOW — unmounting a symlink to a non-mountpoint fails with or
without it — which is why the script covers it against a live mount instead.

The EROFS job is green with all of the above, and its sudo audit log shows
only two umount(8) executions in the whole run, both the script's own: apko
itself no longer execs it.

Two items from review are deferred to #2408 rather than fixed here: the state
file goes stale during a succeeding teardown (distinguishing "not mounted"
from "busy" needs /proc/self/mountinfo, since neither exit 32 nor EINVAL
does), and the fuse path has never executed anywhere, because CI runs as root.

Refs #2408

@smoser
smoser marked this pull request as ready for review August 19, 2026 22:31
smoser and others added 5 commits August 19, 2026 18:32
Restores the `apko erofs mount` and `apko erofs umount` surface that
1fe041e descoped so chainguard-dev#2249 could land as a writer plus `ls`: the
mount_linux.go orchestration, the kernel/fuse driver, the mount state
file, the non-linux stubs, and their tests.

Nothing here had callers outside the package, so it comes back private:
the driver interface, newDriver, resolveMode, the mode type and its
constants, mountState, stateSchemaVersion, statePath, writeState,
loadState and removeState.  Options is replaced by an exported
MountOptions holding plain strings, which is all the CLI needs to say:

    Mount(ctx, src, dest string, opts MountOptions) error

Mount returns only an error now.  It used to hand back *MountState,
but that type is private and the CLI discarded it anyway.

Ls keeps the signature it has on main -- a plain arch string, not an
Options bag -- and does not get its `--mode` flag back.  That flag was
documented as "accepted for symmetry with mount and ignored", which is
not worth restoring.

This commit deliberately restores behavior that is known to be wrong;
the rest of the PR fixes it.  In this state Unmount still passes
whatever paths the state file names to umount as root without checking
them, a partial unmount cannot be recovered by rerunning the command,
and the default mount is writable and discards what was written on
umount.

The docs' overlayfs section is not restored: it was removed by
e5778f1, the layering descope, and its recipe only works once
multi-layer splitting returns.

Refs chainguard-dev#2408

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Mount and Unmount built their driver internally, so the ~300 lines of
orchestration around it -- cleanup LIFO, state file lifecycle, unmount
policy -- could not be driven with a fake.  driver_linux_test.go
covered only the argv builders, and nothing anywhere executed the
orchestration at all.

Add a driverFactory type and unexported mountWith/unmountWith that take
it; the exported Mount and Unmount now just pass newDriver.  Tests pass
a factory returning a fake that records calls.

The kernel-vs-fuse umount choice moves into the driver as a new
Unmount(ctx, mp) method, replacing unmountOne's switch on drv.Name().
That also puts unmountBlob behind the seam: it now calls the fuse
driver's Unmount, which is already the kernel-umount-then-fusermount
chain unmountBlob was open-coding, so a blob teardown no longer shells
out from the orchestration layer.  Behavior is unchanged.

Six tests, all through a fake: mount order and recorded state for a
three-layer image, lowerdir reversal, LIFO teardown with no state file
left behind when overlay assembly fails, refusal to clobber an existing
state file, the single-layer read-only short circuit skipping overlay
entirely, unmount following the recorded order and cleaning up, and a
stateless dest being treated as a blob mountpoint.

This lands first because the fixes that follow -- state file
containment, partial-unmount recovery, a read-only default -- are only
testable through it.

Refs chainguard-dev#2408

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Unmount read <dest>/.apko-erofs-mount.json and handed every path it
named to umount, in kernel mode as root.  loadState checked only JSON
shape and schema version.  The state file lives inside dest, so anyone
who can write there chooses its contents -- and the docs suggest shared
destinations like /mnt/apko-erofs.  A planted file naming /home had root
unmount /home.

Validate in loadState, so every caller is covered:

  - the recorded Dest must equal the dest being unmounted.  Without
    this, a file that is internally consistent about some other
    directory still passes a containment check.
  - every entry in Mounts must be absolute and must be a path a mount
    actually creates under dest: <dest>/merged or <dest>/layers/NN.

The second check is deliberately a whitelist rather than plain
containment.  Containment alone would still accept <dest>/upper or
<dest>/anything, and mounts are only ever made at those two shapes, so
nothing legitimate is rejected.  It also subsumes traversal: a path that
climbs out with .. cannot match either shape.

An empty Mounts list is rejected too.  Mount never writes one, and
Unmount removes the file rather than emptying it.

Tests cover eight rejected shapes -- absolute elsewhere, dest's parent,
traversal through dest and through layers/, a relative path, dest
itself, an unexpected subdir, a non-numeric layer name -- plus the dest
mismatch, the empty list, and a legitimate layout being accepted.  The
one that states the actual guarantee is in mount_linux_test.go: a
planted state file makes Unmount fail without a single path reaching the
driver, rather than being noticed afterwards.

All of these fail against the previous code.

Refs chainguard-dev#2408

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
On EBUSY part-way through, Unmount told the user to rerun `apko erofs
umount`.  That rerun could not succeed.  Mounts is [merged,
layers/NN..00] and the state file was never rewritten, so the second
attempt started again at merged -- already unmounted -- and umount(8)
exits 32 on a path that is not a mountpoint.  It failed there without
ever reaching the layer that was actually busy.  Recovery meant
unmounting by hand and deleting the state file.

Drop each entry as it comes down, and on failure rewrite the state file
with what is left before returning.  The file then describes reality,
so a rerun resumes at the entry that was busy.  writeState is
CreateTemp+Rename in the same directory, so a rerun never sees a
half-written file.  The error also names the count still up and the
exact command to repeat.

The fake driver now refuses to unmount a path twice, mirroring umount's
exit 32.  Without that a rerun would pass simply because the fake was
willing to take merged down again, which is the bug this commit fixes.

The new test unmounts a three-layer image with layers/01 held busy,
asserts merged and layers/02 came down, asserts the state file now
lists exactly [layers/01, layers/00], then clears the error and reruns:
the teardown completes and the state file is gone.  It fails against
the previous code at the state file assertion.

Refs chainguard-dev#2408

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The default mount was writable, and umount then RemoveAll'd <dest>/upper
with no flag and no warning, silently discarding everything written
through it.  Read-only fits what these commands are for -- looking at an
image apko just built -- so writes become opt-in.

MountOptions.ReadOnly becomes MountOptions.Writable, so the zero value
is the safe one and a library caller that fills in nothing gets a
read-only mount.  The CLI's --read-only becomes --rw.  upper and work
are only created when writable; overlayfs takes a lowerdir-only stack
otherwise.  A single-layer image now skips overlayfs by default rather
than only when asked, which is every image apko can currently produce.

umount no longer removes upper at all.  For a read-only mount there is
nothing there; for a writable one it is the only copy of what was
written, so it is left behind and its path logged.  merged, work and
layers still go.  mountState gains a writable field so umount knows
which case it is in.  No schema bump: mount/umount has never been in a
release, so there are no v1 state files to stay compatible with, and
the field's zero value reads as read-only anyway.

Two tests: a two-layer default mount asserts the overlay is assembled
read-only, upper and work are absent, and the state file agrees; the
--rw case asserts the overlay is writable, writes a file into upper,
unmounts, and requires the file to still be there afterwards.

Refs chainguard-dev#2408

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@smoser
smoser force-pushed the erofs-mount-restore branch from 1b985c0 to 3820ef5 Compare August 19, 2026 22:33
ociDirWithLayers returned an unused dir string (unparam); drop it.
Simplify a filepath.Dir wrapper lambda to the function itself (gocritic).
@smoser
smoser force-pushed the erofs-mount-restore branch from b6c5162 to a5264b2 Compare August 19, 2026 22:38
@smoser
smoser marked this pull request as draft August 19, 2026 23:34
@smoser smoser changed the title erofs: restore mount/umount, with the review fixes erofs: add 'apko erofs mount/umount' sub commands. Aug 19, 2026
The privileged CI job added in chainguard-dev#2414 exercised mount(8) directly, so
nothing anywhere ran the orchestration in pkg/erofsmount against a real
kernel -- the unit tests drive it through a fake driver, which cannot
catch a wrong mount(8) invocation or a layout that overlayfs rejects.

Extend the script with four sections after the existing comparison:

- Read-only image mount: assert the single-layer short-circuit (the
  layer straight at merged, no layers/ or upper/), check the state
  file's mode, dest, writable and mounts, and diff the mounted tree
  against the `apko erofs ls` listing already computed above.
- `--rw`: assert the overlay layout (layers/00 plus merged) and the
  LIFO order recorded in the state file, write a file through the
  mount, and check umount left it in upper/ rather than deleting it.
- A raw blob, which carries no state file, so umount takes the
  single-mountpoint fall-back.
- A tampered state file naming a decoy tmpfs outside DEST: umount must
  fail *and* leave the decoy mounted. Asserting on the message alone
  would pass against code that ran the umount anyway.

Cleanup no longer tracks one mountpoint. It reads /proc/self/mounts and
unmounts everything under the workdir, deepest first, which covers the
mounts apko makes and a mount left behind by a failure mid-test; the
rm -rf now runs under sudo since apko created part of the tree as root.

The two normalization pipelines become functions so the mounted-tree
comparison can be reused, and the mount sections run apko through sudo
via an absolute path.

@mattmoor mattmoor left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I verified all five sections of the PR body against the code, and they hold: the restored surface is private with Mount/Unmount/MountOptions as the only new exports, the seam is behavior-preserving (argv builders byte-identical to the pre-descope code), the resumability test is honest in exactly the way that matters (the fake's exit-32 mirroring means the rerun passes only because of the state rewrite; delete the rewrite and it goes red), read-only-by-default and upper preservation check out end to end, and the erofs CI job provably executed all four new script sections on this head, from a fork, including the behavioral decoy assertion rather than message-matching. The four bugs from the #2249 review are fixed as described.

I'm holding approval on one substantive finding, inline on state.go: the whitelist validates strings, and the attacker in its own threat model can reroute a root umount with a symlink instead of a tampered file. Disposition there (hardening or an explicitly documented residual) and I'm happy to approve; two independent verification passes converged on it, so I don't think it's a misread.

Smaller notes, none blocking, take or file with #2408:

  • validate accepts duplicate Mounts entries (e.g. merged twice). No escape, since both are inside dest, but the second umount exits 32 and wedges the teardown into the rewrite loop.
  • unmountBlob routes through the fuse driver, so on a box without fusermount the returned error is "neither fusermount3 nor fusermount found in PATH", swallowing the kernel umount's actual stderr (e.g. "not mounted"). The pre-descope inline code reported both.
  • The tampered-state script section accepts any nonzero exit from apko erofs umount; the decoy-still-mounted assertion narrows it, but pinning the refusal (state file untouched, or the validate error text) would distinguish "refused" from "crashed first".
  • The fuse path still never executes against reality anywhere: CI runs as root, so erofsfuse, fuse-overlayfs, and the fusermount fallback chain are covered only by argv-shape tests. Fine to leave noted in #2408.

Comment thread pkg/erofsmount/state.go
Comment thread pkg/erofsmount/mount_linux.go
Comment thread pkg/erofsmount/mount_linux_test.go Outdated
Comment thread docs/erofs.md Outdated
smoser and others added 11 commits August 20, 2026 12:34
loadState's whitelist constrains the *string* each state file entry
holds -- <dest>/merged or <dest>/layers/NN -- but umount(8)
canonicalizes its argument before unmounting.  So the escape it was
meant to stop is still reachable one step out: plant <dest>/merged as a
symlink to /home along with a state file that passes validate, and a
root `apko erofs umount <dest>` takes down /home anyway.  Write access
inside dest is the precondition for both, so the symlink costs an
attacker nothing.

checkResolved requires each recorded mountpoint to resolve to the path
its name claims, immediately before it is handed to the driver.  It
compares against dest with dest's *own* symlinks resolved, so a dest
legitimately reached through one -- somewhere under /var/run, say --
keeps working and only the components below it have to be real.  The
check sits inside the LIFO loop rather than in a pre-pass so that a
missing entry still fails exactly where umount(8) would have, leaving
the earlier entries unmounted and the state file rewritten.

The same symlink works on the mount side, where MkdirAll is satisfied by
a link to an existing directory: <dest>/merged pointing at /etc would
have the layer mounted over /etc, and with the above in place Unmount
would then refuse to take it down.  ensureDir now lstats and insists on
a real directory.

None of this closes the hole, and the comment and docs say so: the check
and the umount it guards are separate steps, so someone who can write in
dest can still swap a directory for a symlink in between.  Closing it
outright wants openat2(RESOLVE_NO_SYMLINKS) and umount -c against
/proc/self/fd/N, which is a much larger change.  The guidance stays "use
a DEST only you can write to".

hack/test-erofs.sh grows the symlinked-DEST/merged case beside the
existing outside-DEST one, sharing a single tmpfs decoy through a new
plant_state helper; both assert the decoy is still mounted afterwards.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
fuseDriver.Unmount tries kernel umount first and falls back to
fusermount, but it discarded the kernel error.  Unmount is also what
unmountBlob uses, since a raw blob has no state file and so no recorded
mode -- which makes the kernel attempt the one that failed for the
interesting reason.  A busy blob mount on a host without fusermount
installed therefore reported "neither fusermount3 nor fusermount found
in PATH" instead of "target is busy", pointing at a missing package
rather than at whatever was holding the mount open.

errors.Join on both failure paths, so the caller sees what each tool
actually said.  The test drives Unmount with PATH set to an empty
directory and asserts the result names both halves.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Not deleting <dest>/upper on umount is right -- for a --rw mount it is
the only copy of everything written through the overlay -- but it left
the next mount at that dest to walk into it.  MkdirAll is happy with an
upper that already holds the previous session's writes and the overlayfs
metadata they carry, possibly for an entirely different image, and
stacks it over unrelated lowers.  The umount after that hands both
sessions back as one.

Mount now refuses a --rw mount when <dest>/upper is not empty, naming
the count and saying to move or remove it.  Read-only mounts never touch
upper, so they are unaffected.

That alone would make every --rw dest single-use, so umount cleans up
the empty case: os.Remove rather than os.RemoveAll, which is the whole
point of the choice.  os.Remove cannot delete a non-empty directory, so
writes survive by construction and the guarantee cannot be undone by a
later edit here; ENOTEMPTY is what logs where they were left.  An upper
nothing was written through goes away and the dest is immediately
reusable.

hack/test-erofs.sh asserts a second --rw mount over the sentinel is
refused, and that a --rw round trip writing nothing leaves no upper and
can be repeated.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every path in an overlayfs option string here is derived from the DEST
the user named, and none of them were escaped.  overlayfs splits the
option string on "," and lowerdir on ":", and treats "\" as an escape --
ovl_next_opt and ovl_split_lowerdirs on the way in, ovl_unescape for
upperdir and workdir.  A "," fails loudly as an unrecognized option, but
a ":" does not: it turns one lowerdir into two, which can compose a
wrong stack rather than failing.

escapeOverlayPath quotes all three, applied to each lowerdir entry and
to upperdir and workdir, through an overlayOpts helper the kernel and
fuse-overlayfs builders now share.  The replacer runs in a single pass,
so the backslash rule cannot re-escape the backslashes the other two
introduce.  The mountpoint is its own argv element rather than part of
an option value, so it stays verbatim -- the test checks that too.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Per review: checkResolved narrows the symlink window but cannot close
it, because umount(8) canonicalizes its argument.  Whatever the check
saw, the exec that follows resolves the path again, so a symlink swapped
into <dest>/merged in between is still followed -- and in kernel mode
that is a root umount of wherever it points.

umount(2) with UMOUNT_NOFOLLOW refuses a symlinked final component in
the same syscall that unmounts, so there is no window to race at all.
kernelUnmount replaces every umount(8) exec: kernelDriver.Unmount, the
teardown closures from MountLayer and AssembleOverlay, and the kernel
attempt fuseDriver.Unmount makes first.  buildKernelUmountArgs and its
argv test go with them.

The errno for the symlink case is EINVAL, which is also what "not a
mountpoint" returns, so the refusal is named explicitly rather than
surfaced as "invalid argument".  That lstat is for the message only --
the flag is what provides the guarantee, so racing it weakens nothing.

checkResolved stays, with a narrower job: UMOUNT_NOFOLLOW covers the
final component, and a symlinked *parent* (<dest>/layers itself) is
resolved during the kernel's own path walk, so that half is still a
check followed by a syscall.  Its comment and docs/erofs.md now record
the split rather than claiming the whole thing is closed.

The unit test can only pin the wording: unmounting a symlink to a
directory that is not a mountpoint fails with or without the flag, so
nothing unprivileged can tell them apart.  hack/test-erofs.sh can, and
now does -- a symlink pointed at a live tmpfs, passed in as DEST so
nothing validates it first.  Without the flag that tmpfs comes down.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
mountImage stat'd the state file and then, several steps later, wrote
it.  Calling that "refuse to clobber an existing mount" overstated it:
two mounts aimed at one dest both see it free and both proceed, and the
second overwrites the first's record, stranding its mounts with nothing
left that names them.

claimState creates the file empty with O_CREATE|O_EXCL before anything
is mounted, and writeState's rename replaces it with the real record at
the end.  The claim is registered as a cleanup, so every error path
releases it.  O_EXCL also declines to follow a symlink parked at that
path, which the plain create would have.

That leaves a window where a killed mount leaves an empty file behind,
so loadState now names it -- "a mount was interrupted before it
finished" beats a JSON syntax error for someone deciding what to do
about it.

Also from review: validate rejects a mountpoint listed twice.  A repeat
is inside dest so it escapes nothing, but the second umount of it fails
on a path that is no longer a mountpoint, wedging the teardown on an
entry that is already done.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every failure out of the unmount loop got the same suffix: "rerun `apko
erofs umount DEST` once they are no longer busy".  That is the right
advice for a busy mountpoint and useless for a checkResolved rejection,
which no amount of waiting clears -- the path resolves somewhere else
and will keep doing so until someone removes the symlink.

Split the two.  A rejection says what it refused and where the remaining
mounts are recorded; only a real umount failure keeps the retry hint.
The state rewrite both share moves into a small helper so it cannot
drift between them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
From review: fakeDriver's factory discarded its mode argument, so the
unit layer certified the mode plumbing without ever looking at it.
unmountBlob's deliberate modeFuse choice -- the kernel-then-fusermount
chain that covers a blob whose mode was never recorded -- could regress
to kernel and every test here would still pass, breaking only for real
fuse users.  Same for unmountImage ignoring st.Mode.

The factory records each mode it is handed.  The blob fall-back test now
asserts modeFuse, and a new test walks a mount and unmount, checking
that the flag decides the first and the state file decides the second.

Also fills the other hole in this file: nothing covered a Preflight
failure, which is what reports a missing mount(8) or a kernel mount
without root.  The test asserts it stops before any mount, any overlay,
and before the dest claim leaves a state file behind.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
hack/test-erofs.sh:

- mountpoint(1) is what every assert_mounted call runs, but it was in
  neither the required-tool loop nor the Requires header.  A box without
  util-linux failed at the first assertion instead of at the preflight.
- The --rw teardown checked merged and layers were gone but not work,
  which umount removes alongside them.
- Both tampered-state cases accepted any nonzero exit, which a crash
  before the check ever ran would also produce.  Asserting the planted
  state file is still there -- and the symlink undisturbed -- is what
  says it was refused rather than that apko fell over.

docs/erofs.md: the "unknown filesystem type 'erofs'" note appeared
twice, once for `apko erofs mount` and once for the manual equivalents
below it.  The first covers both modes and names --mode=fuse; drop the
second.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
kernelUnmount reports "umount <path>: ..." and both callers prefixed it
again, so the EROFS job's log came back with

  umount /tmp/.../decoy-link: umount /tmp/.../decoy-link: refusing to
  follow a symlink

Drop the outer prefix at both sites; the driver already names the path.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Left over from the switch to umount(2): Preflight still refused to
mount when umount(8) was missing from PATH, though nothing execs it
any more. mount(8) is the only binary the kernel driver still needs.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@smoser
smoser enabled auto-merge (squash) August 21, 2026 15:39

@mattmoor mattmoor left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at 28acdfd. I re-verified the whole response round: the held finding is settled with three layers rather than the minimum ask. UMOUNT_NOFOLLOW via umount(2) closes the final-component race atomically in every path that consumes state-file entries (kernel driver, both teardown closures, the fuse driver's kernel attempt; buildKernelUmountArgs is gone), checkResolved covers the symlinked-parent half with the residual race documented honestly in code and docs, and the mount side refuses a symlinked mountpoint up front. The e2e script now proves all three refusals behaviorally, asserting the decoy tmpfs remains mounted and the state file survives, and the EROFS job ran those sections on this head.

All six secondary notes from the prior round are fixed as well, several beyond what was asked: the O_EXCL state-file claim kills the mount-side race while the partial-unmount rewrite correctly keeps the atomic CreateTemp+rename, duplicate mount entries are rejected, fuse umount failures are joined instead of swallowed, the fake driver now observes the mode it is handed with real assertions, the rerun hint is only offered when a rerun can help, and the upperdir and option-escaping hardening are both welcome. The last commit also cleared the stale umount(8) preflight requirement I was about to note.

Three take-or-leave nits inline; none blocking.

)

// driver wraps the externally-invoked mount and umount commands used by Mount.
// Two implementations exist on Linux: kernelDriver shells out to mount(8) and

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale half of what 28acdfd fixed: kernelDriver no longer shells out to umount(8); unmounts go through umount(2) directly. Worth updating so the doc comment and Preflight tell the same story.

return []string{"mount", "-t", "overlay", "-o", opts, "overlay", merged}
}

func buildFuseOverlayArgs(lowers []string, upper, work, merged string, readOnly bool) []string {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Take or leave: this feeds kernel-style escapes (\:, \,, \\) to fuse-overlayfs, whose own lowerdir colon-split has historically not honored \: the way the kernel does. A dest containing : was broken on the fuse leg before this PR too, so no regression there, but a dest containing a bare backslash is a narrow new fuse-leg-only regression (it previously passed verbatim). One comment acknowledging the parity assumption, or a quick check against the fuse-overlayfs option parser, would cover it.

// interesting reason -- returning only the fusermount error turns "target is
// busy" into "neither fusermount3 nor fusermount found in PATH".
func (d *fuseDriver) Unmount(ctx context.Context, mp string) error {
kerr := kernelUnmount(ctx, mp)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Take or leave: when kernelUnmount fails with the symlink refusal, fusermount is still exec'd on the same path. It is safe (fusermount has carried its own internal UMOUNT_NOFOLLOW since the CVE-2010-3879 fix, and the CI decoy-link log shows it refusing), and state-file paths pass checkResolved first, so this is purely surface-shrinking: skipping the fusermount attempt when the kernel failure was the symlink refusal would also keep that refusal from being buried in a joined error.

@smoser
smoser merged commit d1b5d5e into chainguard-dev:main Aug 21, 2026
23 checks passed
smoser added a commit that referenced this pull request Sep 2, 2026
The three take-or-leave nits from the review of #2415, all in
`pkg/erofsmount/driver_linux.go`. One commit each.

### 1. The interface comment describes an unmount that no longer happens

`28acdfd7` dropped the `umount(8)` requirement from `Preflight` but left
the
`driver` doc comment saying `kernelDriver` shells out to it. Text only.

### 2. Stop at a symlink refusal in fuse mode

`fuseDriver.Unmount` tries the kernel unmount and falls back to
`fusermount`,
which is right when the mount was simply made by the other tool and
wrong when
the kernel refused because the mountpoint is a symlink. `fusermount` has
carried its own `UMOUNT_NOFOLLOW` since the CVE-2010-3879 fix, so it
refuses
the same path again and the joined error buries the reason:

```
Error: umount .../decoy-link: refusing to follow a symlink
/usr/bin/fusermount3 -u .../decoy-link: exit status 1: ... Invalid argument
```

That second line is noise once the first has decided the matter.
`kernelUnmount`
now wraps a sentinel for the symlink case so `Unmount` can return it
as-is.
State-file paths pass `checkResolved` before reaching a driver, so this
shrinks
surface rather than fixing an exposure.

### 3. Refuse paths fuse-overlayfs cannot round-trip

`escapeOverlayPath` emits the kernel's escaping to both legs, and review
asked
whether fuse-overlayfs honours it. **Measured against 1.17 rather than
assumed** — mounting a tree whose path contains each character, escaped
and
verbatim, against a clean-path control:

| in `DEST` | escaped (today) | verbatim (before #2415) |
| --- | --- | --- |
| *(control)* | ok | ok |
| `,` | **ok** | broken — split into separate options |
| `:` | broken | broken |
| `\` | broken | broken |

Two things follow. The escaping is a straight win for the comma on this
leg and
cannot help with the other two. And it is **not** the regression it
looked
like: a bare backslash was already eaten before the escaping went in, so
that
dest fails identically before and after.

What is left is a bad failure mode rather than a wrong mount —
fuse-overlayfs
reports `cannot resolve path .../dd`, naming a path the user never
wrote. So
`:` and `\` are refused up front on the fall-back leg, with the reason
and a
pointer at `--mode=kernel`, which handles them correctly. The kernel
overlay
attempted just before the fall-back is unaffected: only the tool that
cannot
cope declines.

The probe script is not included — it needs a live `fuse-overlayfs` and
the
result is a fact about that tool, not about apko, so it is recorded as a
table
in the comment on `fuseOverlayUnsupported` instead.

### Verification

`gofmt -l` clean, `golangci-lint run -n` reports 0 issues, `go build
./...` and
`GOOS=darwin go build ./...` both succeed, and `SOURCE_DATE_EPOCH=0 go
test
./...` passes. The fuse-mode symlink test was checked to fail without
its
change.

Both new unit tests do run in CI, in the `go-tests` job on the non-root
runner — where the symlink test passes via the EPERM+lstat path rather
than
EINVAL, since the kernel checks `may_mount()` before `path_mounted()`.
What
never runs anywhere is the **e2e** fuse leg: `hack/test-erofs.sh` goes
through
sudo, so `resolveMode` picks kernel. That is one of the two items
already
filed against #2408.

### Review follow-ups

Four commits addressing the review, one concern each:

- `06801ac9` — move `errSymlinkedMountpoint` above `kernelUnmount`'s doc
block
(it had captured the whole UMOUNT_NOFOLLOW rationale as its own godoc),
and
fix the two comments commit 2 made untrue: the lstat is no longer "for
the
message only", and "Both failures are reported" is now conditional. The
lstat comment records the EPERM-vs-EINVAL detail, so a later cleanup
keying
  it on EINVAL does not silently break the fuse leg.
- `64166080` — run `checkFuseOverlayPaths` before the `fuse-overlayfs`
`LookPath`, since the refusal is terminal and "not installed" is
fixable.
  With that order an empty-PATH test can pin the wiring: raw paths, and
  read-only excluding upper/work. Mutation-checked both ways.
- `cabc8693` — `--mode=kernel (as root)`; the user on this leg is
non-root by
construction, so the bare suggestion bounces off
`kernelDriver.Preflight`.

On the fifth comment: yes, the refusal is deliberately version-blind
rather
than a versioned check — "refuse until re-measured". There is no probe,
and
1.17 is what the comment records.

Refs #2408

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
smoser added a commit that referenced this pull request Sep 2, 2026
#2418)

Section 2 of #2408: brings back the EROFS layer split, descoped in e5778f1 so
#2249 could land as a single-layer writer plus `ls`, with the five bugs review
found fixed on top.

#2415 has since merged, and this is rebased on it. The code is still
disjoint — that PR is `pkg/erofsmount` and the CLI, this one is
`pkg/build` — they meet only in `hack/test-erofs.sh`.

### 1. Restore, unchanged

`pkg/build/erofs_layers.go` and its test come back exactly as they were,
and the `layering` + `format: erofs` rejection leaves
`ImageConfiguration.Validate` and `buildLayers`. This commit deliberately
restores the known-broken behavior; the four after it are the fix, and
that delta is the point of the PR.

### 2. Package routing — the bug that inverted the whole thing

The type assertion asked `info.Sys()` for a `Package()` method. tarfs,
which is what a real build walks, returns a fresh `*tar.Header` from
`Sys()` and hangs `Package()` off the FileInfo itself — the receiver
`splitLayers` asserts on. So it never succeeded outside a fixture: every
file landed in the top layer, each group layer held only ancestor
directories plus a partial installed db (that case keys on the path
string, so it still fired), and per-layer scanners read a db naming
packages whose files were not there.

Assert on `info`, and turn a `packageToWriter` miss into an error rather
than a silent fall back to `top`.

`TestSplitErofsLayers` could not catch this: it drove the split through
`apkfs.NewMemFS()`, which implements `Package()` nowhere, so the fixture
had zero package-owned files by construction and passed identically with
and without routing. It is replaced by `erofs_layers_test.go`, which
builds its fixture with `pkg/tarfs` and installs files through
`WriteHeader` with an `*apk.Package`, the way an apk install does. Its
routing assertions fail against the previous code.

### 3. Directory-only subtrees

Directories were recorded during the walk and materialized only by
`emitAncestors`, which runs for non-directory entries. A directory whose
subtree holds no file was therefore never written anywhere: `/tmp`,
`/run`, `/var/empty`, every empty dir and every mount point were absent
from the merged view. Each directory is now emitted into its owning
writer as it is walked, which is what both siblings already do.

### 4. Temp files and fds

No error return out of `splitErofsLayers` closed or removed the
per-layer
temp files, and each go-erofs `Writer` also holds an unlinked spool fd
that only `Close` releases. The CLI happened to be bounded by its
`MkdirTemp`/`RemoveAll` wrapper; a library caller accumulated both until
process exit. The per-writer state moves into an `erofsGroupWriter` type
with `finish`/`discard`, and a deferred sweep discards every writer
unless the function reaches its return. Still no upstream abort API, so
`discard` writes an image it then removes.

### 5. Build time

`newErofsGroupWriter` passed `WithBuildTime` only for a non-zero build
time — the exact case `erofsBuildTime`'s comment describes, where
go-erofs stamps `time.Now()` from `Close` and a library caller gets
different layer digests every build. It now uses `erofsBuildTime`, same
as `writeErofs`. apko's own CLI was never affected.

### 6. Drive it from `hack/test-erofs.sh`

The privileged job from #2414, extended by #2415, only built and mounted
a single layer, because that was all apko could produce. The script now
builds the same config again with `layering` and adds, after the
existing
comparisons:

- layer roles and mediaTypes across the manifest, and `digest` == the
  sha256 of each blob;
- `fsck.erofs` on every layer, and a kernel mount of every layer stacked
  as read-only overlayfs lowerdirs;
- `apko erofs ls` on the layered OCI directory diffed against that
  overlay mount — `Stack`'s merge only becomes reachable for apko's own
  output now that splitting is back, and this is the first thing to
  compare it with what the kernel assembles;
- every non-final layer must hold a regular file other than the partial
  installed db, which is exactly the shape the routing bug produced;
- the merged tree diffed against a **tar** build of the same config
  unpacked in layer order. The tar split is the reference for what
splitting must preserve, and a directory that reaches no layer shows up
  here and nowhere else. Both builds resolve from one `apko lock` output
so a package published between them cannot make them differ. The unpack
  uses `--numeric-owner`; without it GNU tar resolves each header's
  uname/gname against the runner's `/etc/passwd` and invents ids for
  `lp`, `mail`, `news` and `uucp`.

The layered section runs last, after #2415's `apko erofs mount` /
`apko erofs umount` checks, and reuses their helpers — `fail`,
`assert_mounted`, `normalize_ls`, `tree_listing`. Their `cleanup`
already
unmounts everything under the workdir deepest-first, so the layer and
overlay mounts need no separate bookkeeping. The one existing line this
touches is the whitespace-in-paths guard, lifted into a
`check_ls_whitespace` function so the layered listing gets it too.

### Not in this PR

Whiteout support in `Stack` — the last item in section 2 — is really the
layer-horizon fix from section 5, and is reader-side and independently
testable. Left for its own PR.

### Verification

`gofmt -l` clean, `golangci-lint run -n` reports 0 issues, `go build
./...` and `GOOS=darwin go build ./...` both succeed, and
`SOURCE_DATE_EPOCH=0 go test ./...` passes. `shellcheck` is clean on the
script.

Each fix was checked to fail without its change: the four new tests are
red against the restore commit and green after their own commit.

Locally, a layered `wolfi-base` build produces five layers whose merged
`apko erofs ls` listing is identical to the single-layer build's, and
whose non-top layers are 0.7–7.4 MB rather than the directories-only
skeletons the routing bug produced.

Refs #2408

🤖 Generated with [Claude Code](https://claude.com/claude-code)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants