Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
86f50b6
build: add EROFS layer output format
smoser May 26, 2026
caff1a5
docs: add EROFS build and verification walkthrough
smoser May 26, 2026
1a74824
cli: add `apko erofs` mount/umount/ls subcommands
smoser May 27, 2026
d9c01a4
golangci-lint
smoser May 27, 2026
545f18d
erofs: make `ls` mount-less and cross-platform via layered fs.FS
smoser May 27, 2026
9805d50
build: rename writeERofs -> writeErofs for capitalization consistency
smoser May 28, 2026
8a51763
add doc on using akpo_build with erofs
smoser May 28, 2026
0e54c8e
docs(erofs): refresh stale notes and fix the manual-overlay snippet
smoser May 28, 2026
e244004
erofs: dedupe spec constants into pkg/build/types
smoser May 28, 2026
7914a1e
erofs(mount): drop -o loop from kernel layer mount
smoser May 28, 2026
be3f3bc
erofs(mount): fail Unmount fast instead of cascading EBUSY
smoser May 28, 2026
a1738d2
erofs(mount): expose --read-only and short-circuit single-layer mounts
smoser May 28, 2026
85de9d8
erofs: preserve setuid/setgid/sticky in the writer
smoser Aug 1, 2026
d0607cb
deps: bump github.com/erofs/go-erofs to v0.3.1
smoser Aug 1, 2026
752868f
erofs(ls): fix uid/gid, device columns and special mode bits
smoser Aug 1, 2026
ae26e8c
erofs: cover file capabilities and other xattrs with a test
smoser Aug 1, 2026
fdb74fa
fix(apkfs): seed character devices as devices, not empty files
smoser Aug 1, 2026
53d98e7
erofs(ls): print 0 as the size of a directory
smoser Aug 1, 2026
245e55a
test(erofs): guard the go-erofs metadata source erofs ls depends on
smoser Aug 1, 2026
50525af
build(erofs): close the image explicitly instead of syncing it
smoser Aug 14, 2026
a2fada1
erofs: stop discarding ListXattrs and short-write errors
smoser Aug 14, 2026
1c0535e
test(erofs): make the optional fsck.erofs cross-check explicit
smoser Aug 14, 2026
46ac139
erofsmount: explain erofs: vs oci:, drop an unbacked claim
smoser Aug 14, 2026
1fe041e
erofs: descope mount/umount to a follow-up
smoser Aug 18, 2026
e5778f1
erofs: descope multi-layer splitting to a follow-up
smoser Aug 18, 2026
460ec8d
erofsmount: use the spec's whiteout encoding, not tar's
smoser Aug 18, 2026
4aa34df
erofsmount: accept the role shapes the spec allows
smoser Aug 18, 2026
98ca640
build(erofs): always stamp the build time
smoser Aug 18, 2026
e41e8d8
oci: carry os.features onto the index platform descriptor
smoser Aug 18, 2026
7e42dc6
build(erofs): name the default output .erofs, not .tar.gz
smoser Aug 18, 2026
ca9a747
erofs: record the hardlink and go-erofs v0.3.1 caveats
smoser Aug 18, 2026
e028849
ci: install erofs-utils so the fsck cross-checks run
smoser Aug 18, 2026
32214b1
erofs: fix what installing erofs-utils in CI exposed
smoser Aug 18, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions .github/workflows/go-tests.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,8 @@ jobs:
alpinelinux.org:443
api.github.com:443
apk.cgr.dev:443
archive.ubuntu.com:80
azure.archive.ubuntu.com:80
dl-cdn.alpinelinux.org:443
dl.google.com:443
files.example.com:443
Expand All @@ -33,6 +35,7 @@ jobs:
objects.githubusercontent.com:443
proxy.golang.org:443
release-assets.githubusercontent.com:443
security.ubuntu.com:80
storage.googleapis.com:443
sum.golang.org:443

Expand All @@ -47,6 +50,21 @@ jobs:
go-version-file: 'go.mod'
check-latest: true

# erofs-utils supplies fsck.erofs, which several EROFS writer tests use as
# a second opinion from the C reference implementation. Those checks are
# optional by design so contributors without the package still get a green
# build -- which means that without this step they never run anywhere, and
# a misunderstanding shared by our writer and our reader would go
# unnoticed. Not conditional on changed paths: a mismatch can be
# introduced by a go-erofs bump as easily as by an apko change.
- name: Install erofs-utils
run: |
sudo apt-get update
sudo apt-get install -y --no-install-recommends erofs-utils
# Assert it landed on PATH: the tests degrade to a t.Log if it did
# not, so a silent install failure would look like a pass.
command -v fsck.erofs

- name: Test
run: |
# The tests golden fixtures all expect `SOURCE_DATE_EPOCH` to be `0`.
Expand Down
15 changes: 15 additions & 0 deletions docs/apko_file.md
Original file line number Diff line number Diff line change
Expand Up @@ -274,3 +274,18 @@ It contains the following children:
- `budget`: The number of additional layers apko will use for layering.

See [layering.md](layering.md) for more information.

### Format (experimental)

`format` selects the on-wire layer payload format:

- `tar` (default): gzip-compressed tar layers (`application/vnd.oci.image.layer.v1.tar+gzip`).
- `erofs`: EROFS filesystem images (`application/vnd.erofs`), per the draft [erofs/erofs-image-spec](https://github.com/erofs/erofs-image-spec). Written by a pure-Go writer with no internal compression.

EROFS layers advertise `erofs` in the image config's `os.features` so consumers that do not implement the spec can identify and skip them.

`format` may also be selected on the command line with `--format=erofs` on `apko build` and `apko publish`. The CLI flag overrides whatever is in the config file.

**Status:** EROFS support is experimental and tracks the draft spec at https://github.com/erofs/erofs-image-spec; media types and annotations may change before the spec reaches a stable release. `format: erofs` produces a single layer: combining it with `layering` is rejected, and compression and dm-verity are not implemented.

See [erofs.md](erofs.md) for a step-by-step guide to building, inspecting, mounting, and pulling EROFS images.
262 changes: 262 additions & 0 deletions docs/erofs.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,262 @@
# EROFS Output Format (experimental)

apko can emit image layers as [EROFS](https://erofs.docs.kernel.org/) filesystem images instead of the default gzip-compressed tar.
The format tracks [erofs/erofs-image-spec](https://github.com/erofs/erofs-image-spec); section numbers below refer to [`spec.md`](https://github.com/erofs/erofs-image-spec/blob/main/spec.md) on `main`.
The spec is still in its draft phase — there is no tagged release, and it says media-type strings, annotation keys, and the binary chunk-index layout are subject to change until the first stable one.
So the media types, annotations, and layer layout used here may change too.

## Why EROFS?

- **Mount, don't unpack.** A layer blob is a complete, kernel-mountable read-only filesystem.
You can `mount -t erofs` the layer directly and look at it, without extracting a tarball.
- **Random access.** Container runtimes that consume EROFS images can seek into a layer rather than streaming the whole tar.
- **Designed for sharing.** The spec defines `overlay-lower` and `overlay-data` roles that compose via the kernel's `overlayfs` exactly the way OCI tar layers do.

This document focuses on producing EROFS images and verifying they look legit using widely available tools.

## Prerequisites

To build and inspect EROFS images you need:

- apko built from a revision that contains EROFS support.
- The `erofs-utils` package, which provides `mkfs.erofs`, `fsck.erofs`, and `dump.erofs`.
apko ships a pure-Go writer (no CGO), so `mkfs.erofs` is not required for *producing* images — but `fsck.erofs` and `dump.erofs` are the easiest way to inspect what apko produced.
- To mount an EROFS layer: either the kernel `erofs` module (present in modern Linux distros) plus root for `mount(8)`, or the unprivileged `erofsfuse` binary from `erofs-utils-fuse`.

Install on Wolfi / Chainguard / Alpine:

```sh
sudo apk add erofs-utils # fsck.erofs, dump.erofs, mkfs.erofs
sudo apk add erofs-utils-fuse # erofsfuse (optional, for unprivileged mount)
```

Install on Debian / Ubuntu:

```sh
sudo apt install erofs-utils # ships mkfs.erofs, fsck.erofs, dump.erofs, and erofsfuse
```

## Single-layer build

The simplest case: opt into EROFS via the `--format=erofs` flag or `format: erofs` in apko.yaml.

`erofs-demo.yaml`:

```yaml
contents:
keyring:
- https://packages.wolfi.dev/os/wolfi-signing.rsa.pub
repositories:
- https://packages.wolfi.dev/os
packages:
- wolfi-base

cmd: /bin/sh -l
archs:
- host
```

Build into an OCI image layout directory:

```sh
mkdir -p out
apko build erofs-demo.yaml apko-erofs-demo:latest out/ --format=erofs --arch=host
```

The OCI layout under `out/` is a regular OCI image directory — the layer blob just happens to be an EROFS filesystem:

```
out/
├── blobs/sha256/
│ ├── <config-digest> # JSON image config
│ ├── <manifest-digest> # JSON image manifest
│ └── <layer-digest> # raw EROFS filesystem image
├── index.json
└── oci-layout
```

### Verify the manifest references EROFS

```sh
MANIFEST=$(jq -r '.manifests[0].digest | split(":")[1]' out/index.json)
jq . out/blobs/sha256/$MANIFEST
```

Expected (excerpt):

```json
{
"layers": [
{
"mediaType": "application/vnd.erofs",
"size": 16207872,
"digest": "sha256:8a2205cc..."
}
]
}
```

The image config records `erofs` in `os.features` per spec §5.4, signalling to tools that don't implement the spec that they should not attempt to apply the layer as a tar:

```sh
CONFIG=$(jq -r '.config.digest | split(":")[1]' out/blobs/sha256/$MANIFEST)
jq '.["os.features"]' out/blobs/sha256/$CONFIG
# → ["erofs"]
```

## Inspect the layer (no mount required)

The layer blob is a complete EROFS filesystem. You can validate and inspect it without mounting anything.

### Identify the file

```sh
LAYER=$(jq -r '.layers[0].digest | split(":")[1]' out/blobs/sha256/$MANIFEST)
file out/blobs/sha256/$LAYER
# → out/blobs/sha256/...: EROFS filesystem, blocksize=12, exslots=0, ...
```

### Integrity check

```sh
fsck.erofs -d3 out/blobs/sha256/$LAYER
# <I> erofs: No errors found
```

### Dump the superblock

```sh
dump.erofs out/blobs/sha256/$LAYER | head -15
```

This prints the on-disk metadata: block size, inode count, build time, UUID, feature flags.

### Extract without root

`fsck.erofs --extract` reads every inode and writes the resulting tree to a directory.
This is the strongest unprivileged validation you can run: if the image is malformed, extraction fails; if it succeeds, the file tree on disk is exactly what a kernel mount would expose.

```sh
mkdir extracted
fsck.erofs --extract=extracted --force out/blobs/sha256/$LAYER
ls extracted/
# bin dev etc home lib ...
cat extracted/etc/os-release
```

Flag availability varies between erofs-utils releases — `--xattrs`, for instance, is not in the version Debian and Ubuntu ship — so check `fsck.erofs --help` on your machine before reaching for anything beyond the above.

### List contents with `apko erofs ls`

For a quick `tar tvf`-style listing of any EROFS source (raw blob or OCI image directory), use `apko erofs ls`. It opens the EROFS blobs directly, walks the merged view in user space, and prints one line per entry — no mounts, no root or FUSE required, works on Linux/macOS/Windows.

Each line is mode, uid/gid, size, mtime and path; setuid/setgid/sticky show up in the mode string as `ls -l` renders them, devices print `major,minor` in place of a size, and directories print 0 as they do in `tar tv`.

```sh
apko erofs ls out/blobs/sha256/$LAYER | head
# lrwxrwxrwx 0/0 7 2026-04-17 19:17 bin -> usr/bin
# drwxr-xr-x 0/0 0 2026-04-17 19:17 dev
# crw-rw-rw- 0/0 1,3 2026-04-17 19:17 dev/null
# drwxrwxrwt 0/0 0 2026-04-17 19:17 tmp
# -rwsr-xr-x 0/0 178528 2026-04-17 19:17 usr/bin/sudo
# -rw-r--r-- 13/15 1183 2026-04-17 19:17 usr/share/man/whatis
# ...

apko erofs ls out/ # works against the whole OCI image too
```

For multi-layer images, `ls` applies overlay semantics in user space to present the merged view the kernel would assemble. It uses the overlayfs-native deletion encoding the spec mandates (§3.6) — a whiteout is a character device with rdev 0, an opaque directory sets `trusted.overlay.opaque="y"` — not the `.wh.` filename convention of tar layers, which §8.1 forbids in EROFS images. A single-layer image is listed as-is: the kernel would mount it directly, applying no overlay semantics, so neither does `ls`.

## Mount the layer

A layer blob is a complete filesystem image, so mounting it takes no apko-specific tooling — either the kernel `erofs` driver (needs root) or `erofsfuse` (unprivileged):

```sh
mkdir -p /mnt/apko-erofs

# Kernel (root):
sudo mount -t erofs -o ro out/blobs/sha256/$LAYER /mnt/apko-erofs
ls /mnt/apko-erofs/
file /mnt/apko-erofs/bin/sh
sudo umount /mnt/apko-erofs

# FUSE (unprivileged):
erofsfuse out/blobs/sha256/$LAYER /mnt/apko-erofs
fusermount3 -u /mnt/apko-erofs # or `fusermount -u`
```

If `mount` reports "unknown filesystem type 'erofs'", the kernel module is missing on your system; install it (e.g. `linux-modules-extra-$(uname -r)` on Ubuntu) or use `erofsfuse`, which needs no root and works inside CI containers that lack the module.

## Pulling from a registry

If you push the image with `apko publish` or `crane push`, the registry stores each blob unchanged — including the EROFS layer blob.
Most registry clients can extract layers by digest:

```sh
# Read the manifest and pull layer blobs.
crane manifest registry.example.com/apko-erofs-demo:latest > manifest.json
LAYER_DIGEST=$(jq -r '.layers[0].digest' manifest.json)
crane blob registry.example.com/apko-erofs-demo:latest@$LAYER_DIGEST > layer.erofs

file layer.erofs # EROFS filesystem...
fsck.erofs -d3 layer.erofs # <I> erofs: No errors found
```

Once you have the blob on disk you can inspect or mount it exactly as in the previous sections.

## Multi-layer builds

Not yet. `format: erofs` currently emits a single layer; combining it with apko's
[layering](layering.md) configuration is rejected at config-validation time rather
than silently producing a one-layer image.

Support for splitting a rootfs into one EROFS layer per package group (each tagged
`org.erofs.role=overlay-lower` per spec §3.8, with the final layer carrying no role)
is in progress.

## Using EROFS support as a Go library

### From apko-consuming projects

If you're already building apko images programmatically (`apko_build.New(ctx, fsys, opts...).BuildImage(...)`), EROFS is just a configuration choice — set the layer format on your `ImageConfiguration` and apko handles the rest:

```go
import (
apko_build "chainguard.dev/apko/pkg/build"
apko_types "chainguard.dev/apko/pkg/build/types"
)

imgConfig := apko_types.ImageConfiguration{
// ... your existing fields ...
Format: apko_types.LayerFormatErofs,
}

bc, err := apko_build.New(ctx, fsys, apko_build.WithImageConfiguration(imgConfig), ...)
if err != nil { /* ... */ }
if err := bc.BuildImage(ctx); err != nil { /* ... */ }
_, layer, err := bc.ImageLayoutToLayer(ctx)
```

### From projects that don't use apko

If you have a plain `fs.FS` and want an EROFS image, **use [go-erofs](https://github.com/erofs/go-erofs) directly** — apko doesn't expose its EROFS writer as a standalone library (and wrapping go-erofs wouldn't add meaningful value over its existing `Writer.CopyFrom(fs.FS)` API).

For inspection, apko *does* expose a focused leaf library — see `chainguard.dev/apko/pkg/erofsmount` — which provides `Stack` (layered `fs.FS` with overlay/whiteout semantics), `OpenLayers` (open an OCI EROFS image's blobs), `ReadOCILayers` (parse an OCI manifest with EROFS layers), and `Ls` (the `apko erofs ls` helper). All of it is cross-platform: go-erofs is pure Go and nothing here mounts anything.

## Current limitations

- **Single layer only.** `layering` and `format: erofs` cannot be combined yet; see [Multi-layer builds](#multi-layer-builds) above.
- **No compression.** apko emits raw `application/vnd.erofs` layers only. The draft spec defines `application/vnd.erofs+zstd` but neither apko's writer nor the underlying go-erofs library writes compressed images yet.
- **No dm-verity.** The spec's verified-mount path (§3.5) is not produced.
- **No chunk index.** Lazy-loading runtimes (per spec §3.4) won't get an index; reads are sequential.
- **No `overlay-data` or `device` roles.** apko emits one unannotated EROFS layer; `org.erofs.role` is never set.
- **Hardlinks become independent copies.** go-erofs has no API to point two names at one inode, so each link costs another full copy of the file's data (rounded up to the block size) and `st_nlink`/`st_ino` identity is lost. Spec §3.7 allows this — a producer must either materialize links or fail — but a hardlink-heavy image will be larger as EROFS than as tar, where extra links are zero-byte entries.
- **Spec is draft.** Media-type strings and annotation keys may change before the spec stabilizes. Treat any image built today as experimental.

If you need any of the above, please open an issue.

## See also

- [erofs/erofs-image-spec `spec.md`](https://github.com/erofs/erofs-image-spec/blob/main/spec.md) — the layer format spec apko tracks.
- [EROFS kernel documentation](https://erofs.docs.kernel.org/) — on-disk format reference.
- [Layering in apko](layering.md) — how the multi-layer strategy partitions packages into groups.
1 change: 1 addition & 0 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ require (
chainguard.dev/sdk v0.1.191
github.com/chainguard-dev/clog v1.8.1
github.com/charmbracelet/log v1.0.0
github.com/erofs/go-erofs v0.3.1
github.com/go-git/go-git/v5 v5.19.2
github.com/google/go-cmp v0.7.0
github.com/google/go-containerregistry v0.21.9
Expand Down
2 changes: 2 additions & 0 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,8 @@ github.com/elazarl/goproxy v1.7.2 h1:Y2o6urb7Eule09PjlhQRGNsqRfPmYI3KKQLFpCAV3+o
github.com/elazarl/goproxy v1.7.2/go.mod h1:82vkLNir0ALaW14Rc399OTTjyNREgmdL2cVoIbS6XaE=
github.com/emirpasic/gods v1.18.1 h1:FXtiHYKDGKCW2KzwZKx0iC0PQmdlorYgdFG9jPXJ1Bc=
github.com/emirpasic/gods v1.18.1/go.mod h1:8tpGGwCnJ5H4r6BWwaV6OrWmMoPhUl5jm/FMNAnJvWQ=
github.com/erofs/go-erofs v0.3.1 h1:Sux82Jq9yvyYhIoLgSHDp741p/+370HsOj9dAh1+VVs=
github.com/erofs/go-erofs v0.3.1/go.mod h1:XkSeN9MHszGd4+3gcEjadJLYHCQpWzJ7/8yznzMuzJs=
github.com/fatih/color v1.18.0 h1:S8gINlzdQ840/4pfAwic/ZE0djQEH3wM94VfqLTZcOM=
github.com/fatih/color v1.18.0/go.mod h1:4FelSpRwEGDpQ12mAdzqdOukCy4u8WUtOY6lkT/6HfU=
github.com/felixge/httpsnoop v1.1.0 h1:3YtUj32ZZkqZtt3sZZsClsymw/QDuVfpNhoA31zeORc=
Expand Down
3 changes: 3 additions & 0 deletions internal/cli/build.go
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,7 @@ func buildCmd() *cobra.Command {
var includePaths []string
var ignoreSignatures bool
var sizeLimits options.SizeLimits
var format string

cmd := &cobra.Command{
Use: "build",
Expand Down Expand Up @@ -119,6 +120,7 @@ Along the image, apko will generate SBOMs (software bill of materials) describin
build.WithIncludePaths(includePaths),
build.WithIgnoreSignatures(ignoreSignatures),
build.WithSizeLimits(sizeLimits),
build.WithFormat(format),
)
},
}
Expand All @@ -139,6 +141,7 @@ Along the image, apko will generate SBOMs (software bill of materials) describin
cmd.Flags().StringVar(&lockfile, "lockfile", "", "a path to .lock.json file (e.g. produced by apko lock) that constraints versions of packages to the listed ones (default '' means no additional constraints)")
cmd.Flags().StringSliceVar(&includePaths, "include-paths", []string{}, "Additional include paths where to look for input files (config, base image, etc.). By default apko will search for paths only in workdir. Include paths may be absolute, or relative. Relative paths are interpreted relative to workdir. For adding extra paths for packages, use --repository-append.")
cmd.Flags().BoolVar(&ignoreSignatures, "ignore-signatures", false, "ignore repository signature verification")
cmd.Flags().StringVar(&format, "format", "", "layer payload format: 'tar' (default) or 'erofs' (experimental, tracks erofs-image-spec draft)")
addClientLimitFlags(cmd, &sizeLimits)
return cmd
}
Expand Down
1 change: 1 addition & 0 deletions internal/cli/commands.go
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,7 @@ func New() *cobra.Command {
cmd.AddCommand(resolve())
cmd.AddCommand(installKeys())
cmd.AddCommand(cleanCmd())
cmd.AddCommand(erofsCmd())
cmd.AddCommand(version.Version())

cmd.PersistentFlags().StringVarP(&workDir, "workdir", "C", cwd, "working dir (default is current dir where executed)")
Expand Down
Loading