Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 23 additions & 7 deletions docs/erofs.md
Original file line number Diff line number Diff line change
Expand Up @@ -174,25 +174,41 @@ The merge approximates what the kernel would assemble, and diverges in two corne

## Mount the layer

A layer blob is a complete filesystem image, so mounting it takes no apko-specific tooling — either the kernel `erofs` driver (needs root) or `erofsfuse` (unprivileged):
`apko erofs mount SOURCE DEST` mounts a raw EROFS blob or an OCI image directory at `DEST`. It chooses between a kernel mount (root) and `erofsfuse` (unprivileged) based on the effective UID; use `--mode=kernel|fuse|auto` to force a choice. `apko erofs umount DEST` tears it back down.

The mount is **read-only** unless you pass `--rw`. Read-only is what inspecting an image wants, and it lets a single-layer image skip overlayfs entirely — the lone layer is mounted straight at `DEST/merged`. With `--rw` you get an overlayfs upperdir at `DEST/upper`; `umount` removes it only if nothing was written through the mount, and otherwise leaves it where it is and logs the path. A later `--rw` mount at the same `DEST` then refuses to start until you move or remove it, rather than quietly stacking two sessions' writes — possibly from different images — on top of each other.

```sh
mkdir -p /mnt/apko-erofs
apko erofs mount out/blobs/sha256/$LAYER /mnt/apko-erofs
ls /mnt/apko-erofs/
file /mnt/apko-erofs/bin/sh
apko erofs umount /mnt/apko-erofs
```

If the kernel mount mode complains "unknown filesystem type 'erofs'", the kernel module is missing on your system; install it (e.g. `linux-modules-extra-$(uname -r)` on Ubuntu) or pass `--mode=fuse` to use `erofsfuse`, which does not require root and works inside CI containers that lack the kernel module.

For an OCI source, `umount` works from `DEST/.apko-erofs-mount.json`, which `mount` wrote; a raw blob has no enclosing directory to hold one, so `umount` falls back to unmounting `DEST` itself. It only accepts mountpoints that a mount creates under `DEST` — `DEST/merged` and `DEST/layers/NN` — so a tampered file cannot name a path outside `DEST`. Because that is a check on the path *string*, a symlink at `DEST/merged` would otherwise redirect it anyway; kernel mode unmounts with `umount(2)` and `UMOUNT_NOFOLLOW`, which refuses that in the same syscall, with no window. A symlinked *parent* (`DEST/layers` itself) is caught by a separate check, and there the check and the unmount are two steps — so **use a `DEST` only you can write to**: anywhere else, another user can race them and choose what your `umount` takes down.

If a mountpoint is busy, `umount` stops there and rewrites the state file to list only what is still mounted, so rerunning it once the mount is free finishes the teardown.

### Doing it manually

A layer blob is a complete filesystem image, so mounting it needs no apko-specific tooling. For reference, `apko erofs mount` is equivalent to one of:

```sh
# Kernel (root):
sudo mount -t erofs -o ro out/blobs/sha256/$LAYER /mnt/apko-erofs
ls /mnt/apko-erofs/
file /mnt/apko-erofs/bin/sh
# ...later:
sudo umount /mnt/apko-erofs

# FUSE (unprivileged):
erofsfuse out/blobs/sha256/$LAYER /mnt/apko-erofs
# ...later:
fusermount3 -u /mnt/apko-erofs # or `fusermount -u`
```

If `mount` reports "unknown filesystem type 'erofs'", the kernel module is missing on your system; install it (e.g. `linux-modules-extra-$(uname -r)` on Ubuntu) or use `erofsfuse`, which needs no root and works inside CI containers that lack the module.

`hack/test-erofs.sh` runs everything above in one go — build, `fsck.erofs`, kernel mount, and a comparison of `apko erofs ls` against the mounted tree — and is what the `EROFS` CI workflow executes.
`hack/test-erofs.sh` runs everything above in one go — build, `fsck.erofs`, kernel mount, a comparison of `apko erofs ls` against the mounted tree, and a round trip through `apko erofs mount` and `apko erofs umount` (read-only, `--rw`, a raw blob, and a tampered state file) — and is what the `EROFS` CI workflow executes.

## Pulling from a registry

Expand Down Expand Up @@ -248,7 +264,7 @@ _, layer, err := bc.ImageLayoutToLayer(ctx)

If you have a plain `fs.FS` and want an EROFS image, **use [go-erofs](https://github.com/erofs/go-erofs) directly** — apko doesn't expose its EROFS writer as a standalone library (and wrapping go-erofs wouldn't add meaningful value over its existing `Writer.CopyFrom(fs.FS)` API).

For inspection, apko *does* expose a focused leaf library — see `chainguard.dev/apko/pkg/erofsmount` — which provides `Stack` (layered `fs.FS` with overlay/whiteout semantics), `OpenLayers` (open an OCI EROFS image's blobs), `ReadOCILayers` (parse an OCI manifest with EROFS layers), and `Ls` (the `apko erofs ls` helper). All of it is cross-platform: go-erofs is pure Go and nothing here mounts anything.
For inspection, apko *does* expose a focused leaf library — see `chainguard.dev/apko/pkg/erofsmount` — which provides `Stack` (layered `fs.FS` with overlay/whiteout semantics), `OpenLayers` (open an OCI EROFS image's blobs), `ReadOCILayers` (parse an OCI manifest with EROFS layers), and `Mount`/`Unmount`/`Ls` (the CLI subcommand helpers, Linux-only for mount/umount; `Ls` is cross-platform).

## Current limitations

Expand Down
263 changes: 226 additions & 37 deletions hack/test-erofs.sh
Original file line number Diff line number Diff line change
Expand Up @@ -4,17 +4,19 @@
# SPDX-License-Identifier: Apache-2.0

# End-to-end check of an apko --format=erofs build: the layer blob is a real
# EROFS filesystem that erofs-utils accepts and the kernel will mount, and
# `apko erofs ls` reports the same tree the kernel does.
# EROFS filesystem that erofs-utils accepts and the kernel will mount,
# `apko erofs ls` reports the same tree the kernel does, and
# `apko erofs mount` / `apko erofs umount` drive that mount themselves.
#
# Usage: hack/test-erofs.sh <yaml>
#
# Example:
# hack/test-erofs.sh ./examples/wolfi-base.yaml
#
# Requires: jq, erofs-utils (fsck.erofs, dump.erofs), the kernel erofs driver,
# and root (or passwordless sudo) for the mount. Set APKO to use a binary
# other than ./apko.
# Requires: jq, mountpoint (util-linux), erofs-utils (fsck.erofs, dump.erofs),
# the kernel erofs driver, and root (or passwordless sudo) for the mount. Set APKO to use a binary
# other than ./apko. The --rw section needs a TMPDIR that overlayfs accepts
# as an upperdir, which rules out tmpfs on older kernels.

set -euo pipefail

Expand All @@ -26,13 +28,17 @@ fi
yaml="$1"
apko="${APKO:-./apko}"
name=$(basename "${yaml}" .yaml)
# Written inside DEST by `apko erofs mount`, read back by `apko erofs umount`.
state=".apko-erofs-mount.json"

if [ ! -x "${apko}" ]; then
echo "no apko binary at ${apko}; run 'make apko' first" >&2
exit 1
fi
# Resolved once: the mount sections run it through sudo.
apko=$(readlink -f "${apko}")

for tool in jq fsck.erofs dump.erofs; do
for tool in jq mountpoint fsck.erofs dump.erofs; do
command -v "${tool}" >/dev/null || {
echo "missing required tool: ${tool}" >&2
exit 1
Expand All @@ -51,15 +57,89 @@ mnt="${workdir}/mnt"
out="${workdir}/out"
mkdir -p "${mnt}" "${out}"

mounted=""
cleanup() {
# Unmount before removing anything, on the failure paths too: leaving a
# mount behind wedges the rest of the job.
[ -n "${mounted}" ] && "${sudo[@]}" umount "${mnt}" || true
rm -rf "${workdir}"
# mount behind wedges the rest of the job. Everything this script mounts
# lives under workdir, so take down whatever is still there, deepest first,
# rather than tracking each mount separately.
local mp
while read -r mp; do
"${sudo[@]}" umount "${mp}" || true
done < <(awk -v pfx="${workdir}/" 'index($2, pfx) == 1 { print $2 }' \
/proc/self/mounts | LC_ALL=C sort -r)
# apko ran under sudo, so parts of workdir are root-owned by now. Don't
# let a failure here mask the script's own exit status.
"${sudo[@]}" rm -rf "${workdir:?}" ||
echo "warning: ${workdir} not fully removed" >&2
}
trap cleanup EXIT

fail() {
echo "$*" >&2
exit 1
}

assert_mounted() {
mountpoint -q "$1" || fail "expected $1 to be a mountpoint"
}

assert_not_mounted() {
if mountpoint -q "$1"; then
fail "expected $1 not to be a mountpoint"
fi
}

assert_absent() {
[ ! -e "$1" ] || fail "expected $1 to be gone"
}

# assert_state DEST JQ-EXPR EXPECTED. The state file is written 0600 by root,
# so it is read back through sudo.
assert_state() {
local got
got=$("${sudo[@]}" jq -r "$2" "$1/${state}")
[ "${got}" = "$3" ] || fail "state of $1: $2 is ${got}, expected $3"
}

# plant_state DEST MOUNTPOINT. Writes the state file an attacker with write
# access to DEST would, naming MOUNTPOINT as the thing to take down.
plant_state() {
jq -n --arg dest "$1" --arg mp "$2" '{
schemaVersion: 1,
mode: "kernel",
source: "tampered",
dest: $dest,
created: "2026-01-01T00:00:00Z",
writable: false,
mounts: [$mp]
}' >"$1/${state}"
}

# Both listings below collapse to "mode uid/gid path [-> target]" so they can
# be diffed. `apko erofs ls` columns: mode uid/gid size date time path
# [-> target].
normalize_ls() {
awk '{
line = $1 " " $2 " " $6
if ($7 == "->") line = line " -> " $8
print line
}' | LC_ALL=C sort
}

# The walk runs privileged. The image intentionally contains mode-0700
# directories owned by other uids (root, usr/man, var/adm), which an
# unprivileged find cannot descend into; `apko erofs ls` reads the image
# directly and is not subject to that, so the two would disagree for a reason
# that has nothing to do with apko.
tree_listing() {
"${sudo[@]}" find "$1" -mindepth 1 -printf '%M\t%U/%G\t%P\t%y\t%l\n' |
awk -F'\t' '{
line = $1 " " $2 " " $3
if ($4 == "l") line = line " -> " $5
print line
}' | LC_ALL=C sort
}

echo "::group::build ${name} as erofs"
"${apko}" build "${yaml}" "${name}:build" "${out}/" --format=erofs --arch=host
echo "::endgroup::"
Expand Down Expand Up @@ -102,8 +182,7 @@ if ! grep -qw erofs /proc/filesystems; then
fi

"${sudo[@]}" mount -t erofs -o ro "${blob}" "${mnt}"
mounted=1
mountpoint -q "${mnt}"
assert_mounted "${mnt}"
echo "::endgroup::"

# Cross-check apko's own reader against the kernel's: same paths, same mode
Expand All @@ -120,37 +199,147 @@ if grep -qE '[[:space:]]$| +->' "${workdir}/ls.raw"; then
exit 1
fi

# ls columns: mode uid/gid size date time path [-> target]
awk '{
line = $1 " " $2 " " $6
if ($7 == "->") line = line " -> " $8
print line
}' "${workdir}/ls.raw" | LC_ALL=C sort >"${workdir}/from-apko"

# The walk runs privileged. The image intentionally contains mode-0700
# directories owned by other uids (root, usr/man, var/adm), which an
# unprivileged find cannot descend into; `apko erofs ls` reads the image
# directly and is not subject to that, so the two would disagree for a reason
# that has nothing to do with apko.
"${sudo[@]}" find "${mnt}" -mindepth 1 -printf '%M\t%U/%G\t%P\t%y\t%l\n' |
awk -F'\t' '{
line = $1 " " $2 " " $3
if ($4 == "l") line = line " -> " $5
print line
}' | LC_ALL=C sort >"${workdir}/from-kernel"
normalize_ls <"${workdir}/ls.raw" >"${workdir}/from-apko"
tree_listing "${mnt}" >"${workdir}/from-kernel"

if ! diff -u "${workdir}/from-kernel" "${workdir}/from-apko"; then
echo "'apko erofs ls' disagrees with the kernel about the layer contents" >&2
exit 1
fail "'apko erofs ls' disagrees with the kernel about the layer contents"
fi
echo "$(wc -l <"${workdir}/from-apko") entries agree"
echo "::endgroup::"

"${sudo[@]}" umount "${mnt}"
mounted=""
if mountpoint -q "${mnt}"; then
echo "${mnt} still mounted after umount" >&2
exit 1
assert_not_mounted "${mnt}"

# Everything above drives mount(8) directly. The rest drives `apko erofs
# mount` and `apko erofs umount`, which is the only place their orchestration
# -- layout, state file, teardown order -- runs against a real kernel.

echo "::group::apko erofs mount (read-only image)"
ro="${workdir}/ro"
"${sudo[@]}" "${apko}" erofs mount "${out}" "${ro}"

# One layer read-only: overlayfs is skipped and the layer is mounted straight
# at merged, so layers/, upper/ and work/ are never created.
assert_mounted "${ro}/merged"
assert_absent "${ro}/layers"
assert_absent "${ro}/upper"

assert_state "${ro}" .mode kernel
assert_state "${ro}" .dest "${ro}"
assert_state "${ro}" .writable false
assert_state "${ro}" '.mounts | join(",")' "${ro}/merged"

tree_listing "${ro}/merged" >"${workdir}/from-mount"
if ! diff -u "${workdir}/from-apko" "${workdir}/from-mount"; then
fail "'apko erofs mount' exposes a different tree than 'apko erofs ls'"
fi

"${sudo[@]}" "${apko}" erofs umount "${ro}"
assert_not_mounted "${ro}/merged"
assert_absent "${ro}/merged"
assert_absent "${ro}/${state}"
echo "::endgroup::"

echo "::group::apko erofs mount --rw (overlay, writes preserved)"
rw="${workdir}/rw"
"${sudo[@]}" "${apko}" erofs mount --rw "${out}" "${rw}"

# --rw always composes through overlayfs, single layer or not.
assert_mounted "${rw}/layers/00"
assert_mounted "${rw}/merged"
assert_state "${rw}" .writable true
# LIFO: merged comes down before the layer it is stacked on.
assert_state "${rw}" '.mounts | join(",")' "${rw}/merged,${rw}/layers/00"

echo "written through the mount" | "${sudo[@]}" tee "${rw}/merged/sentinel" >/dev/null
"${sudo[@]}" "${apko}" erofs umount "${rw}"
assert_not_mounted "${rw}/merged"
assert_absent "${rw}/merged"
assert_absent "${rw}/layers"
assert_absent "${rw}/work"
assert_absent "${rw}/${state}"
# upper is the one directory umount must leave alone once something has been
# written through it: removing it would silently discard those writes.
[ -f "${rw}/upper/sentinel" ] ||
fail "umount discarded the writes made through a --rw mount"
# And because it is still there, a second --rw mount at the same DEST has to
# refuse rather than stack this session's writes under the next one.
if "${sudo[@]}" "${apko}" erofs mount --rw "${out}" "${rw}"; then
fail "--rw mount reused an upper left behind by an earlier mount"
fi
assert_not_mounted "${rw}/merged"

# An upper nothing was written through is not worth keeping, so that round trip
# leaves DEST clean and immediately reusable.
rw2="${workdir}/rw2"
"${sudo[@]}" "${apko}" erofs mount --rw "${out}" "${rw2}"
"${sudo[@]}" "${apko}" erofs umount "${rw2}"
assert_absent "${rw2}/upper"
"${sudo[@]}" "${apko}" erofs mount --rw "${out}" "${rw2}"
"${sudo[@]}" "${apko}" erofs umount "${rw2}"
echo "::endgroup::"

echo "::group::apko erofs mount (raw blob)"
blobmnt="${workdir}/blobmnt"
"${sudo[@]}" "${apko}" erofs mount "${blob}" "${blobmnt}"
assert_mounted "${blobmnt}"
# A blob has no enclosing directory to hold state, so umount has to fall back
# to treating dest as a single mountpoint.
"${sudo[@]}" "${apko}" erofs umount "${blobmnt}"
assert_not_mounted "${blobmnt}"
echo "::endgroup::"

echo "::group::apko erofs umount rejects a tampered state file"
# The state file lives inside DEST, so whoever can write there decides what a
# root umount is asked to take down. Two shapes have to be refused: a
# mountpoint plainly outside DEST, and one named DEST/merged -- which the
# whitelist allows -- that is a symlink pointing out. umount(8) canonicalizes
# its argument, so following the second lands on the decoy just as surely as
# the first. Both check that the decoy is still mounted afterwards, which is
# the part a message-only check would miss.
decoy="${workdir}/decoy"
mkdir -p "${decoy}"
"${sudo[@]}" mount -t tmpfs -o size=1m tmpfs "${decoy}"
assert_mounted "${decoy}"

outside="${workdir}/tampered-outside"
mkdir -p "${outside}"
plant_state "${outside}" "${decoy}"
if "${sudo[@]}" "${apko}" erofs umount "${outside}"; then
fail "umount accepted a state file naming a mount outside DEST"
fi
assert_mounted "${decoy}"
# A nonzero exit on its own would also match a crash before the check ran.
# The state file surviving untouched is what says it was refused.
[ -f "${outside}/${state}" ] ||
fail "umount removed the state file it was supposed to refuse"

symlinked="${workdir}/tampered-symlink"
mkdir -p "${symlinked}"
ln -s "${decoy}" "${symlinked}/merged"
plant_state "${symlinked}" "${symlinked}/merged"
if "${sudo[@]}" "${apko}" erofs umount "${symlinked}"; then
fail "umount followed a symlinked DEST/merged out of DEST"
fi
assert_mounted "${decoy}"
[ -f "${symlinked}/${state}" ] ||
fail "umount removed the state file it was supposed to refuse"
[ -L "${symlinked}/merged" ] ||
fail "umount disturbed the symlink instead of refusing it"

# The two refusals above are both decided before umount(2) is reached, so
# neither exercises UMOUNT_NOFOLLOW. This does: a symlink pointing straight at
# a live mountpoint, handed in as DEST so nothing validates it first. Without
# the flag the kernel would resolve it and take the tmpfs down.
ln -s "${decoy}" "${workdir}/decoy-link"
if "${sudo[@]}" "${apko}" erofs umount "${workdir}/decoy-link"; then
fail "umount followed a symlink to a live mountpoint"
fi
assert_mounted "${decoy}"

"${sudo[@]}" umount "${decoy}"
echo "::endgroup::"

echo "PASS: ${name} erofs layer mounts and matches 'apko erofs ls'"
echo "PASS: ${name} erofs layer mounts, matches 'apko erofs ls', and round-trips"
echo " through 'apko erofs mount' / 'apko erofs umount'"
Loading