research: bootc-dev/bootc#2571 (operator request, cgwalters-forge/tracker#703)

A devspace run for the operator's request on bootc-dev/bootc#2571, relayed in cgwalters-forge/tracker#703. The run answers here, because the run profile takes an issue as its item.

Generated-by: https://github.com/cgwalters/#llms

✅ agent ended well. Result: #22 (comment)

Partial work. The run was stopped at its timeout before the task was done; this is what it handed back.

Research on bootc-dev/bootc#2571 at PR head 7b970f0a: why chmod 0555 /usr in the Dockerfile has no effect on the deployed system.

Short answer: reproduced. There are two separate bugs in the merge commit, and /usr/bin 0755 isn't a bootc problem.

What I ran (verified)

In a Fedora 44 KVM VM: I applied the chmod 0555 /usr Dockerfile patch posted above to the PR head and ran just build. The ELN grub2-efi-x64-cc download was stubbed out because that package isn't available right now; it's unrelated to this issue. The built image has bootc-202610151811.g7b970f0a39.

$ podman run --rm localhost/bootc stat -c '%a %n' / /usr /usr/bin
555 /
555 /usr
755 /usr/bin
# per-layer tar entries (7 layers): layer1 'drwxr-xr-x usr/' 'drwxr-xr-x usr/bin/', layer3/4 'drwxr-xr-x usr/', layer6 'dr-xr-xr-x usr/'
$ podman run --privileged ... localhost/bootc sh -c 'ostree init --repo=/repo --mode=bare; bootc internals ostree-container image pull /repo ostree-unverified-image:containers-storage:localhost/bootc'
$ ostree ls -X -d <merge> / /usr /usr/bin
d00755 0 0 { security.selinux root_t } /
d00755 0 0 { security.selinux usr_t } /usr
d00755 0 0 { security.selinux bin_t } /usr/bin
$ for each layer ref: ostree ls -d <layer> /usr  ->  d00755, d00555 (one layer), d00755, d00755
$ ostree ls -C -d <merge> / /usr  ->  root dirmeta 50773817... (same as every layer commit's root), /usr 9c359a8b...

I had to run setenforce 0 in the VM for the pull, because the image's SELinux labels aren't in the host policy.

Root cause

A later layer's directory metadata is dropped (an existing bug). write_merge_commit_impl does a union checkout of each layer (store.rs#L1252-L1285) with UNION_FILES. In ostree, when the directory already exists it sets did_exist, and fchmod, fchown, xattrs and mtime are all skipped (ostree-repo-checkout.c#L1096-L1110, L1200-L1229). So for directories the first layer wins. OCI and composefs-rs do the opposite: the last layer wins (generic_tree.rs#L561-L567). Verified: the layer commit holds /usr 0555, but the merge has 0755.
The PR's normalize_root_dirmeta does nothing. mt.walk(&["usr"], 0) (store.rs#L634) returns the root itself, because ostree_mutable_tree_walk returns self when start == len-1 (ostree-mutable-tree.c#L552-L556). So the root's checksum is copied onto the root. Verified: the merge root's dirmeta checksum equals the layer roots' and differs from /usr's. The new test can't catch this, because in the fixtures / and /usr already have the same dirmeta. Even if walk worked, copying the whole dirmeta checksum after labelling would give / the label usr_t.
/usr/bin 755 is in the image itself. It's 755 in layer 1 and in stock quay.io/centos-bootc/centos-bootc:stream10, while rpm -ql --dump filesystem says 040555. That comes from base image build tooling, not from the bootc import (verified 755 in the image; where it comes from is inferred).

Proposed patch (not compiled or tested: the run ran out of time before the build)

Reapply directory mode and owner from the layers in order (last wins) to the checkout. Then copy /usr's mode and owner to /, matching composefs-rs copy_root_metadata_from_usr. Both happen before write_dfd_to_mtree, so SELinux labelling still gives / root_t. No hardcoded 0o555.

On "use the image's root when present": --tar-autocreate-parents synthesizes a 0755 root for every layer commit, so at merge time ostree-ext can't tell whether the image had a ./ entry. composefs-rs' transform_for_oci copies /usr over the root unconditionally anyway. Honoring ./ would need filter_tar to record whether it saw one.

The test helper assert_root_dirmeta_matches_usr should also compare mode, uid and gid instead of the dirmeta checksum, and a fixture needs a derived layer that chmods /usr.

diff --git a/crates/ostree-ext/src/container/store.rs b/crates/ostree-ext/src/container/store.rs
index 64943c5b..59904e97 100644
--- a/crates/ostree-ext/src/container/store.rs
+++ b/crates/ostree-ext/src/container/store.rs
@@ -624,22 +624,107 @@ fn cleanup_root(root: &Dir) -> Result<()> {
     Ok(())
 }
 
-/// Copy `/usr` directory metadata to the root, matching the composefs-rs
-/// approach since OCI images don't reliably include root dir metadata.
+/// Mode, uid and gid of a directory.
+type DirOwnerMode = (u32, u32, u32);
+
+/// Record the mode and ownership of every directory in `dir` into `out`,
+/// keyed by path relative to the commit root.
+fn collect_dirmeta(
+    dir: &ostree::RepoFile,
+    path: &Utf8Path,
+    out: &mut HashMap<Utf8PathBuf, DirOwnerMode>,
+    cancellable: Option<&gio::Cancellable>,
+) -> Result<()> {
+    let queryflags = gio::FileQueryInfoFlags::NOFOLLOW_SYMLINKS;
+    let info = dir.query_info("unix::mode,unix::uid,unix::gid", queryflags, cancellable)?;
+    out.insert(
+        path.to_owned(),
+        (
+            info.attribute_uint32("unix::mode"),
+            info.attribute_uint32("unix::uid"),
+            info.attribute_uint32("unix::gid"),
+        ),
+    );
+    let e = dir.enumerate_children("standard::name,standard::type", queryflags, cancellable)?;
+    while let Some(child_info) = e.next_file(cancellable)? {
+        if child_info.file_type() != gio::FileType::Directory {
+            continue;
+        }
+        let name = child_info.name();
+        let name = name.to_str().expect("UTF-8 ostree name");
+        let child = e.child(&child_info).downcast::<ostree::RepoFile>().unwrap();
+        collect_dirmeta(&child, &path.join(name), out, cancellable)?;
+    }
+    Ok(())
+}
+
+/// Apply directory mode (and ownership, if `chown` is set) to `dir`.
+fn apply_dirmeta(
+    dir: impl rustix::fd::AsFd,
+    (mode, uid, gid): DirOwnerMode,
+    chown: bool,
+) -> Result<()> {
+    use rustix::fs::{Gid, Mode, Uid};
+    if chown {
+        rustix::fs::fchown(&dir, Some(Uid::from_raw(uid)), Some(Gid::from_raw(gid)))
+            .context("fchown")?;
+    }
+    rustix::fs::fchmod(&dir, Mode::from_raw_mode(mode & 0o7777)).context("fchmod")?;
+    Ok(())
+}
+
+/// A `UnionFiles` checkout leaves a directory that an earlier layer created
+/// untouched, so a later layer changing only a directory's metadata (e.g.
+/// `RUN chmod 0555 /usr`) would be lost. OCI semantics (and composefs-rs) are
+/// that the last layer wins, so reapply directory metadata from the layers
+/// in order once they have all been checked out.
+fn apply_layer_dirmeta(
+    repo: &ostree::Repo,
+    layer_commits: &[LayerRef],
+    root: &Dir,
+    chown: bool,
+    cancellable: Option<&gio::Cancellable>,
+) -> Result<()> {
+    let mut dirmeta = HashMap::new();
+    for lc in layer_commits {
+        let (commit_root, _) = repo.read_commit(&lc.commit, cancellable)?;
+        let commit_root = commit_root.downcast::<ostree::RepoFile>().unwrap();
+        collect_dirmeta(&commit_root, Utf8Path::new("."), &mut dirmeta, cancellable)
+            .with_context(|| format!("Reading directory metadata of {}", lc.commit))?;
+    }
+    for (path, meta) in dirmeta {
+        use rustix::fs::{Mode, OFlags, ResolveFlags};
+        let flags = OFlags::RDONLY | OFlags::DIRECTORY | OFlags::NOFOLLOW | OFlags::CLOEXEC;
+        let resolve = ResolveFlags::BENEATH | ResolveFlags::NO_SYMLINKS;
+        let fd = match rustix::fs::openat2(root, path.as_std_path(), flags, Mode::empty(), resolve)
+        {
+            Ok(fd) => fd,
+            // The directory was removed by a later layer, or it or a parent
+            // was replaced by a non-directory.
+            Err(rustix::io::Errno::NOENT | rustix::io::Errno::NOTDIR | rustix::io::Errno::LOOP) => {
+                continue;
+            }
+            Err(e) => return Err(e).with_context(|| format!("Opening {path}")),
+        };
+        apply_dirmeta(fd, meta, chown).with_context(|| format!("Updating {path}"))?;
+    }
+    Ok(())
+}
+
+/// Copy the `/usr` directory mode and ownership to the root, matching
+/// composefs-rs, since OCI images don't reliably include root dir metadata.
+/// This happens before the merged tree is committed, so the root still gets
+/// its own SELinux label.
 ///
 /// See:
 ///   - <https://github.com/coreos/fedora-coreos-tracker/issues/2238>
 ///   - <https://github.com/coreos/fedora-coreos-tracker/issues/2238#issuecomment-6082074286>
-fn normalize_root_dirmeta(mt: &ostree::MutableTree) -> Result<()> {
-    let usr = mt.walk(&["usr"], 0).context("looking up /usr in mtree")?;
-
-    let usr_meta_checksum = usr.metadata_checksum();
-    if usr_meta_checksum.is_empty() {
-        anyhow::bail!("/usr has no metadata checksum in mtree");
+fn normalize_root_dirmeta(root: &Dir, chown: bool) -> Result<()> {
+    let usr = root.symlink_metadata("usr").context("looking up /usr")?;
+    if !usr.is_dir() {
+        anyhow::bail!("/usr is not a directory");
     }
-
-    mt.set_metadata_checksum(&usr_meta_checksum);
-    Ok(())
+    apply_dirmeta(root, (usr.mode(), usr.uid(), usr.gid()), chown)
 }
 
 impl ImageImporter {
@@ -1315,6 +1400,11 @@ impl ImageImporter {
 
         cleanup_root(&root_dir)?;
 
+        let chown = checkout_mode == ostree::RepoCheckoutMode::None;
+        apply_layer_dirmeta(repo, layer_commits, &root_dir, chown, cancellable)
+            .context("Applying layer directory metadata")?;
+        normalize_root_dirmeta(&root_dir, chown).context("normalizing root dir metadata")?;
+
         let mt = ostree::MutableTree::new();
         repo.write_dfd_to_mtree(
             (*td).as_raw_fd(),
@@ -1325,8 +1415,6 @@ impl ImageImporter {
         )
         .context("Writing merged filesystem to mtree")?;
 
-        normalize_root_dirmeta(&mt).context("normalizing root dir metadata on mtree")?;
-
         let merged_root = repo
             .write_mtree(&mt, cancellable)
             .context("Writing mtree")?;

Generated by agent · ◷

Follow-up on bootc-dev/bootc#2571: I built and tested the patch proposed in the previous comment on PR head 7b970f0a.

Short answer: the patch as posted compiled but broke 15 of the 30 ostree-ext integration tests. After a one-line-scope fixup, all tests pass. The VM repro now gives / 0555 and /usr 0555 in the merge commit; unpatched bootc on the same image gives 0755 for both.

Environment: Fedora 44 Cloud VM under KVM, Fedora rust/cargo 1.99. Every result below was run and observed (verified) unless it is marked inferred.

Build and unit tests (verified)

Check	Result
`git apply` of the proposed patch on `7b970f0a`	applies cleanly
`cargo build -p ostree-ext --all-targets`	ok, no ostree-ext warnings
`cargo test -p ostree-ext --lib container::store`	2 passed (`test_ref_for_descriptor`, `test_cleanup_root`)
`cargo test -p ostree-ext --test it`, unpatched PR head (baseline)	29 passed, 1 ignored
`cargo test -p ostree-ext --test it`, patch as proposed	15 failed, 14 passed: `Importing: normalizing root dir metadata: fchmod: Bad file descriptor (os error 9)`
`cargo test -p ostree-ext --test it`, patch + fixup	29 passed, 1 ignored
`cargo clippy -p ostree-ext --all-targets`	exits 101 both with and without the patch, because of a pre-existing `disallowed_methods` error in `fixture.rs:274` (newer clippy). store.rs has the same 5 pre-existing lints before and after, so the patch adds none

The bug in the proposed patch. normalize_root_dirmeta called fchmod on root_dir, which comes from cap-std open_dir and is an O_PATH fd. fchmod/fchown on an O_PATH fd fail with EBADF. The fixup reopens . with O_RDONLY|O_DIRECTORY first. apply_layer_dirmeta already used openat2(O_RDONLY...), so it wasn't affected. (Verified that the error came from this call, and that the fixup makes it go away. That O_PATH is the cause is inferred from cap-std's behaviour and the EBADF.)

VM repro (verified)

The PR head plus the chmod 0555 /usr Dockerfile change, built with just build. As in the previous run, the ELN grub2-efi-x64-cc download was stubbed out (touch stub.rpm) because ELN doesn't have the package; it's only used when bootloader=grub-cc. The image's /usr/bin/bootc contains the patched code (the string Applying layer directory metadata is present). Then ostree init --mode=bare and bootc internals ostree-container image pull <repo> ostree-unverified-image:containers-storage:localhost/bootc as root (privileged, setenforce 0).

$ podman run --rm localhost/bootc stat -c '%a %u:%g %n' / /usr /usr/bin
555 0:0 /
555 0:0 /usr
755 0:0 /usr/bin

# patched bootc (from the image), merge 72b03d55...
$ ostree ls -X -d $M / /usr /usr/bin /usr/lib
d00555 0 0 0 { security.selinux system_u:object_r:root_t:s0 } /
d00555 0 0 0 { security.selinux system_u:object_r:usr_t:s0 } /usr
d00755 0 0 0 { security.selinux system_u:object_r:bin_t:s0 } /usr/bin
d00755 0 0 0 { security.selinux system_u:object_r:lib_t:s0 } /usr/lib
$ ostree ls -C -d $M / /usr      # dirmeta checksums differ, so / keeps root_t
d00555 ... dab7cf3e75352c4b... /
d00555 ... 35e347a8da902d00... /usr

# control: unpatched bootc built from 7b970f0a, same image, fresh bare repo, merge a1f8462e...
d00755 0 0 0 { ... root_t } /
d00755 0 0 0 { ... usr_t } /usr
d00755 0 0 0 { ... bin_t } /usr/bin
dirmeta: / 50773817e451...  /usr 9c359a8bcc95...   (same as the previous run)

# layer commits' /usr: 0755, 0755, 0755, 0555 (one layer); other layers have no /usr

The 0755 on /usr/bin matches the image (it's 755 in podman run too), so it isn't an import problem, as found before.

Not covered (gaps / inferred)

Only the bare repo path (root, chown = true) went through the VM pull. The it tests run unprivileged, so they presumably exercise only the chmod path (inferred, not instrumented).
No booted deployment: I checked the merge commit, not bootc install plus a boot, or just test-tmt readonly.
No new test was added. The existing assert_root_dirmeta_matches_usr still compares dirmeta checksums. That passes only because the fixtures' / and /usr are identical, and it can't hold for real images where / gets root_t and /usr gets usr_t. It should compare mode/uid/gid, and a fixture needs a derived layer that chmods /usr.
apply_layer_dirmeta carries mode/uid/gid, not xattrs, so a later layer that changes only a directory's xattrs is still lost (inferred from the code). I didn't measure the extra cost of walking every layer commit's directories; pulling this 7-layer, 1.7 GB image went through without trouble.

Final diff (proposed patch + fixup, `cargo fmt`)

diff --git a/crates/ostree-ext/src/container/store.rs b/crates/ostree-ext/src/container/store.rs
index 64943c5b..1dad077a 100644
--- a/crates/ostree-ext/src/container/store.rs
+++ b/crates/ostree-ext/src/container/store.rs
@@ -624,22 +624,112 @@ fn cleanup_root(root: &Dir) -> Result<()> {
     Ok(())
 }
 
-/// Copy `/usr` directory metadata to the root, matching the composefs-rs
-/// approach since OCI images don't reliably include root dir metadata.
+/// Mode, uid and gid of a directory.
+type DirOwnerMode = (u32, u32, u32);
+
+/// Record the mode and ownership of every directory in `dir` into `out`,
+/// keyed by path relative to the commit root.
+fn collect_dirmeta(
+    dir: &ostree::RepoFile,
+    path: &Utf8Path,
+    out: &mut HashMap<Utf8PathBuf, DirOwnerMode>,
+    cancellable: Option<&gio::Cancellable>,
+) -> Result<()> {
+    let queryflags = gio::FileQueryInfoFlags::NOFOLLOW_SYMLINKS;
+    let info = dir.query_info("unix::mode,unix::uid,unix::gid", queryflags, cancellable)?;
+    out.insert(
+        path.to_owned(),
+        (
+            info.attribute_uint32("unix::mode"),
+            info.attribute_uint32("unix::uid"),
+            info.attribute_uint32("unix::gid"),
+        ),
+    );
+    let e = dir.enumerate_children("standard::name,standard::type", queryflags, cancellable)?;
+    while let Some(child_info) = e.next_file(cancellable)? {
+        if child_info.file_type() != gio::FileType::Directory {
+            continue;
+        }
+        let name = child_info.name();
+        let name = name.to_str().expect("UTF-8 ostree name");
+        let child = e.child(&child_info).downcast::<ostree::RepoFile>().unwrap();
+        collect_dirmeta(&child, &path.join(name), out, cancellable)?;
+    }
+    Ok(())
+}
+
+/// Apply directory mode (and ownership, if `chown` is set) to `dir`.
+fn apply_dirmeta(
+    dir: impl rustix::fd::AsFd,
+    (mode, uid, gid): DirOwnerMode,
+    chown: bool,
+) -> Result<()> {
+    use rustix::fs::{Gid, Mode, Uid};
+    if chown {
+        rustix::fs::fchown(&dir, Some(Uid::from_raw(uid)), Some(Gid::from_raw(gid)))
+            .context("fchown")?;
+    }
+    rustix::fs::fchmod(&dir, Mode::from_raw_mode(mode & 0o7777)).context("fchmod")?;
+    Ok(())
+}
+
+/// A `UnionFiles` checkout leaves a directory that an earlier layer created
+/// untouched, so a later layer changing only a directory's metadata (e.g.
+/// `RUN chmod 0555 /usr`) would be lost. OCI semantics (and composefs-rs) are
+/// that the last layer wins, so reapply directory metadata from the layers
+/// in order once they have all been checked out.
+fn apply_layer_dirmeta(
+    repo: &ostree::Repo,
+    layer_commits: &[LayerRef],
+    root: &Dir,
+    chown: bool,
+    cancellable: Option<&gio::Cancellable>,
+) -> Result<()> {
+    let mut dirmeta = HashMap::new();
+    for lc in layer_commits {
+        let (commit_root, _) = repo.read_commit(&lc.commit, cancellable)?;
+        let commit_root = commit_root.downcast::<ostree::RepoFile>().unwrap();
+        collect_dirmeta(&commit_root, Utf8Path::new("."), &mut dirmeta, cancellable)
+            .with_context(|| format!("Reading directory metadata of {}", lc.commit))?;
+    }
+    for (path, meta) in dirmeta {
+        use rustix::fs::{Mode, OFlags, ResolveFlags};
+        let flags = OFlags::RDONLY | OFlags::DIRECTORY | OFlags::NOFOLLOW | OFlags::CLOEXEC;
+        let resolve = ResolveFlags::BENEATH | ResolveFlags::NO_SYMLINKS;
+        let fd = match rustix::fs::openat2(root, path.as_std_path(), flags, Mode::empty(), resolve)
+        {
+            Ok(fd) => fd,
+            // The directory was removed by a later layer, or it or a parent
+            // was replaced by a non-directory.
+            Err(rustix::io::Errno::NOENT | rustix::io::Errno::NOTDIR | rustix::io::Errno::LOOP) => {
+                continue;
+            }
+            Err(e) => return Err(e).with_context(|| format!("Opening {path}")),
+        };
+        apply_dirmeta(fd, meta, chown).with_context(|| format!("Updating {path}"))?;
+    }
+    Ok(())
+}
+
+/// Copy the `/usr` directory mode and ownership to the root, matching
+/// composefs-rs, since OCI images don't reliably include root dir metadata.
+/// This happens before the merged tree is committed, so the root still gets
+/// its own SELinux label.
 ///
 /// See:
 ///   - <https://github.com/coreos/fedora-coreos-tracker/issues/2238>
 ///   - <https://github.com/coreos/fedora-coreos-tracker/issues/2238#issuecomment-6082074286>
-fn normalize_root_dirmeta(mt: &ostree::MutableTree) -> Result<()> {
-    let usr = mt.walk(&["usr"], 0).context("looking up /usr in mtree")?;
-
-    let usr_meta_checksum = usr.metadata_checksum();
-    if usr_meta_checksum.is_empty() {
-        anyhow::bail!("/usr has no metadata checksum in mtree");
+fn normalize_root_dirmeta(root: &Dir, chown: bool) -> Result<()> {
+    let usr = root.symlink_metadata("usr").context("looking up /usr")?;
+    if !usr.is_dir() {
+        anyhow::bail!("/usr is not a directory");
     }
-
-    mt.set_metadata_checksum(&usr_meta_checksum);
-    Ok(())
+    // `root` may be an `O_PATH` fd, on which fchmod/fchown fail with EBADF.
+    let flags =
+        rustix::fs::OFlags::RDONLY | rustix::fs::OFlags::DIRECTORY | rustix::fs::OFlags::CLOEXEC;
+    let rootfd =
+        rustix::fs::openat(root, ".", flags, rustix::fs::Mode::empty()).context("opening root")?;
+    apply_dirmeta(rootfd, (usr.mode(), usr.uid(), usr.gid()), chown)
 }
 
 impl ImageImporter {
@@ -1315,6 +1405,11 @@ impl ImageImporter {
 
         cleanup_root(&root_dir)?;
 
+        let chown = checkout_mode == ostree::RepoCheckoutMode::None;
+        apply_layer_dirmeta(repo, layer_commits, &root_dir, chown, cancellable)
+            .context("Applying layer directory metadata")?;
+        normalize_root_dirmeta(&root_dir, chown).context("normalizing root dir metadata")?;
+
         let mt = ostree::MutableTree::new();
         repo.write_dfd_to_mtree(
             (*td).as_raw_fd(),
@@ -1325,8 +1420,6 @@ impl ImageImporter {
         )
         .context("Writing merged filesystem to mtree")?;
 
-        normalize_root_dirmeta(&mt).context("normalizing root dir metadata on mtree")?;
-
         let merged_root = repo
             .write_mtree(&mt, cancellable)
             .context("Writing mtree")?;

Generated-by: https://github.com/cgwalters/#llms

Generated by agent · ◷

Navigation Menu

research: bootc-dev/bootc#2571 (operator request, cgwalters-forge/tracker#703) #22

What I ran (verified)

Root cause

Proposed patch (not compiled or tested: the run ran out of time before the build)

Build and unit tests (verified)

VM repro (verified)

Not covered (gaps / inferred)

Final diff (proposed patch + fixup, `cargo fmt`)

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

research: bootc-dev/bootc#2571 (operator request, cgwalters-forge/tracker#703) #22

Description

Activity

cgwalters-bot commented on Oct 20, 2026

cgwalters-bot commented on Oct 20, 2026

What I ran (verified)

Root cause

Proposed patch (not compiled or tested: the run ran out of time before the build)

cgwalters-bot commented on Oct 20, 2026

Build and unit tests (verified)

VM repro (verified)

Not covered (gaps / inferred)

Final diff (proposed patch + fixup, cargo fmt)

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Final diff (proposed patch + fixup, `cargo fmt`)