research: bootc-dev/bootc#2569 (operator request, cgwalters-forge/tracker#709) #24

Activity

1. cgwalters-bot commented on Oct 20, 2026

   

   cgwalters-bot

   on Oct 20, 2026

   Last edited by cgwalters-bot

   CollaboratorAuthor

   More actions

   ✅ agent ended well. Result: #24 (comment)
2. cgwalters-bot commented on Oct 20, 2026

   

   cgwalters-bot

   on Oct 20, 2026

   CollaboratorAuthor

   More actions

   Result: the container detection is confirmed as the blocker, and there is a second one behind it. systemd's container detection is what makes bootctl install --variables=yes fail. Getting past it is not enough, though. bootctl then writes an entry with an empty device path, which the firmware can't load. Two things cause that: bootc's SYSTEMD_RELAX_ESP_CHECKS=1, and a second container check in systemd's ESP lookup. A small systemd patch plus a bootc change fixes both; I tested this end to end in an OVMF VM.

   Setup (verified)

   • Fedora 44 guest under QEMU/KVM with OVMF (Secure Boot off), systemd 259.10. bootc built from main (de7ae55) and from the PR head (5685e48).
   • Image: quay.io/fedora/fedora-bootc:44 with bootupd and shim removed, systemd-boot-unsigned added, and the bootc binary replaced.
   • Command: podman run --rm --privileged --pid=host -v /dev:/dev -v /var/lib/containers:/var/lib/containers --security-opt label=type:unconfined_t IMAGE bootc install to-disk --composefs-backend --bootloader systemd --filesystem xfs --wipe /dev/vdb
   • NVRAM is compared with efibootmgr before and after each run. Each new entry was boot-tested by booting the target disk alone with the resulting NVRAM and reading OVMF's BdsDxe: lines on the serial console.

   bootc	bootctl used by the install	install	firmware entry
   main	259 (image)	exit 0	none, NVRAM unchanged (#2557)
   PR	259 (image)	exit 1: Failed to determine current boot order: Operation not supported	none
   PR	259, run in a private mount ns with a tmpfs over /run (hides /run/.containerenv)	exit 0	Linux Boot Manager HD(0,GPT,00000000-0000-0000-0000-000000000000,0x0,0x0)/\EFI\systemd\systemd-bootx64.efi. OVMF: failed to load Boot000B ... Not Found, then falls back to EFI/BOOT/BOOTX64.EFI
   PR	same, plus SYSTEMD_RELAX_ESP_CHECKS unset	exit 0	HD(2,GPT,<ESP PARTUUID>,0x1000,0x400000)/.... OVMF loads and starts it, and the guest booted the installed system from it
   PR	systemd main (263~devel, d9ae3fa), unpatched	exit 1, same error	none
   PR	systemd main + patch below	exit 0	zeroed HD(0,...) entries (bootc still sets SYSTEMD_RELAX_ESP_CHECKS=1)
   PR	systemd main + patch below, SYSTEMD_RELAX_ESP_CHECKS unset	exit 0	HD(2,GPT,<ESP PARTUUID>,...), and OVMF boots it

   What it looks like inside the install container (--pid=host, efivarfs mounted rw):

   • bootctl status prints Not booted with EFI. With SYSTEMD_LOG_LEVEL=debug it prints Found container virtualization podman.
   • systemd-detect-virt --container prints podman, also with container= set, so the env container="" idea doesn't work.
   • bootctl --esp-path=... status shows the ESP with no PARTUUID. Once /run/.containerenv is hidden, it shows /dev/disk/by-partuuid/....
   • On the PR head, bootctl did write LoaderSystemToken from inside the container. So efivarfs writes work there, and only systemd's own checks stop the boot entry.

   The PR's timeout commit works: the menu showed "Boot in 5s" on the OVMF console.

   Why (code, systemd main d9ae3fa)

   • efivars.c:335-349: is_efi_boot() is false in a container. efi-api.c:446-455: efi_get_boot_order() then returns -EOPNOTSUPP.
   • systemd v260 already added an override in 1fa1e88: --variables=yes calls set_efi_boot(true). That only happens in touch_variables() (bootctl-util.c:18-38), which the status, random-seed and set-* verbs use. bootctl install decides in should_touch_install_variables() instead (bootctl-install.c:240-250), which never calls it. That is why v262 and main still fail.
   • find-esp.c:54-55: in a container, systemd skips probing the ESP's device. SYSTEMD_RELAX_ESP_CHECKS skips it too (L52). Either way the partition number and UUID come back as 0/null (L366-L372), and that produces the HD(0,GPT,000...) entry.
   • bootc sets SYSTEMD_RELAX_ESP_CHECKS=1 unconditionally (PR bootloader.rs:322, main bootloader.rs:307). It also passes --variables=yes for systemd >= 258 (PR bootloader.rs:309-316).

   Proposed fix

   1. systemd (upstream). The patch makes bootctl install --variables=yes behave like the other verbs since 1fa1e88, and makes it keep probing the ESP in that case. This is the explicit opt-in suggested above, but through the existing --variables=yes rather than a new --variables=always. I tested the CLI path as shown in the table; I did not test the varlink Install path.

   --- a/src/bootctl/bootctl-install.c
   +++ b/src/bootctl/bootctl-install.c
   @@ -139,6 +139,12 @@ static int install_context_from_cmdline(
            assert(operation >= 0);
            assert(operation < _INSTALL_OPERATION_MAX);
    
   +        /* Like touch_variables(): an explicit --variables=yes overrides the is_efi_boot() check (which
   +         * is always false in a container). Do this before looking for the ESP, so that its partition is
   +         * probed. */
   +        if (arg_touch_variables > 0)
   +                set_efi_boot(true);
   +
            _cleanup_(install_context_done) InstallContext b = INSTALL_CONTEXT_NULL;
   @@ -2170,6 +2176,9 @@ int vl_method_install(
            if (p.context.entry_token_type < 0)
                    p.context.entry_token_type = BOOT_ENTRY_TOKEN_AUTO;
    
   +        if (p.context.touch_variables > 0)
   +                set_efi_boot(true);
   +
            r = find_esp_and_warn_at_full(
   --- a/src/shared/find-esp.c
   +++ b/src/shared/find-esp.c
   @@ -15,6 +15,7 @@
    #include "devnum-util.h"
   +#include "efivars.h"
    #include "env-util.h"
   @@ -51,7 +52,9 @@ static VerifyESPFlags verify_esp_flags_init(int unprivileged_mode, const char *e
   -        if (detect_container() > 0)
   +        /* In a container, skip probing the backing device, unless EFI access was explicitly requested
   +         * (e.g. bootctl --variables=yes), since boot entries need the ESP's partition number and UUID. */
   +        if (detect_container() > 0 && !is_efi_boot())
                    flags |= VERIFY_ESP_SKIP_DEVICE_CHECK;

   2. bootc, for this PR now. Don't land the --variables=yes commit (69488fde). It turns "no entry" into a failed install, and no released systemd can create a working entry from the container. Keep the old skip and warn instead. I tested this in the same VM: exit 0, the warning is printed, and NVRAM is unchanged.

   --- a/crates/lib/src/bootloader.rs
   +++ b/crates/lib/src/bootloader.rs
   @@ -307,12 +307,14 @@ pub(crate) fn install_systemd_boot(
        } else {
   -        // Update firmware variables if not a generic image
   -        // This corresponds to --update-firmware CLI option for bootupd installs
   -        // This option was added in systemd v258
   -        if systemd_version()? >= 258 {
   -            bootctl_args.push("--variables=yes");
   -        }
   +        // bootctl can't create the firmware boot entry from inside a container:
   +        // it treats a container as not booted with EFI (so `--variables=yes`
   +        // fails with EOPNOTSUPP), and it skips probing the ESP's partition, so
   +        // any entry it wrote would have an empty device path. Leave a hint
   +        // instead of failing.
   +        crate::utils::medium_visibility_warning(
   +            "Not creating a UEFI boot entry for systemd-boot; the firmware will use EFI/BOOT/BOOTX64.EFI or an existing entry",
   +        );
        }

   3. bootc, once a systemd release has (1). Pass --variables=yes only from that version on, as is already done for --random-seed. Set SYSTEMD_RELAX_ESP_CHECKS=1 only for --generic-image. I checked that to-disk works without it here. I did not check the osbuild/to-filesystem case its comment mentions. The timeout commit doesn't depend on any of this and can land separately.

   Not verified

   • Runs without --pid=host.
   • Secure Boot.
   • The varlink install path.
   • The efibootmgr --create route (already reported working earlier in this thread).
   • The fedora-44 CI failures on this PR. I didn't look at them.

   Generated by agent · ◷