Staged deployment is silently lost at shutdown #2622

Summary

Staged deployment is silently lost at shutdown when /boot is a systemd
gpt-auto ESP automount that idle-expired and was then re-mounted through the
automount (by bootc switch itself) before staging.
ostree-finalize-staged.service ExecStop fails with:

ostree[...]: error: Remounting /boot read-write: Invalid argument

The system boots back into the old deployment. No staged or rollback entry
remains, and nothing is reported to the user.

Environment

• Fedora 44, kernel 6.19.14, systemd 259.9-1.fc44
• bootc 202610172021.gde7ae55a1c-1.fc44, ostree 2026.4-1.fc44
• /boot is mounted by systemd-gpt-auto-generator
  ("EFI System Partition Automount"), idle timeout ~2min
• VM-based host (originally seen in bootc-operator CI, reproduced standalone
  with plain bootc, no operator involved)

Steps to reproduce

In short: on a host where /boot is the gpt-auto ESP automount, let /boot
idle-expire, then bootc switch and reboot.

Exact steps used, with bink VMs (cluster
and update images from the
bootc-operator repo; the
operator itself is not involved, the node is not managed by it):

1. Start the cluster and push the update image (in a bootc-operator checkout):

   make start-bink build-update-image
   export CLUSTER=e2e
   export NODE_IMAGE=ghcr.io/bootc-dev/bink/node:v1.36-fedora-44-disk
   

2. Add a fresh node and SSH into it:

   bink node add repro1 --cluster-name $CLUSTER \
       --node-image $NODE_IMAGE \
       --target-imgref registry.cluster.local:5000/node:latest
   bink node ssh repro1 --cluster-name $CLUSTER
   sudo -i
   

3. On the node, check /boot is the automount and currently mounted:

   systemctl show boot.automount -p Where -p TimeoutIdleUSec   # /boot, ~2min
   systemctl is-active boot.mount                             # active
   

4. Do not access /boot (no ls /boot, findmnt /boot, bootc status)
   and wait for the idle expiry:

   until [ "$(systemctl is-active boot.mount)" = inactive ]; do sleep 10; done
   journalctl -u boot.mount --no-pager | tail -3   # "Unmounting boot.mount"
   

5. Stage an update (this re-triggers the automount) and reboot right away:

   bootc switch registry.cluster.local:5000/node:update
   systemctl reboot
   

6. After the node is back (~1 min), SSH in again and check:

   journalctl -b -1 -u ostree-finalize-staged --no-pager
   journalctl -b -1 --no-pager | grep -E "boot.automount|boot.mount|finalize"
   bootc status
   

Cleanup: bink node remove repro1 --force --cluster-name $CLUSTER

Expected

Finalized deployment; system boots into node:update.

Actual

Finalize fails; system boots into the old image; staged deployment is gone.

# bootc status
● Booted image: registry.cluster.local:5000/node:latest
        Digest: sha256:e2e2f54da669cfbb2728bf46cd8db42de26b49478e91be78544a84f282df3fc2 (amd64)
     Timestamp: 2026-10-21T09:18:30Z

In bootc-operator CI, all 4 failures had a /boot expiry followed by an
automount re-trigger before ostree-finalize-staged-hold.service started.
None of the ~40 successful finalizations had one.

Journal (standalone reproducer, previous boot)

09:28:02 systemd[1]: Set up automount boot.automount - EFI System Partition Automount.
09:28:03 systemd[1]: Mounted boot.mount - /boot.
09:31:41 systemd[1]: Unmounting boot.mount - /boot...           <- idle expiry
09:31:41 systemd[1]: Unmounted boot.mount - /boot.
09:32:24 systemd[1]: boot.automount: Got automount request for /boot, triggered by 3291 (tokio-rt-worker)   <- bootc switch
09:32:24 systemd[1]: Mounted boot.mount - /boot.
09:32:29 systemd[1]: Started ostree-finalize-staged-hold.service - Hold /boot Open for OSTree Finalize Staged Deployment.
09:32:29 systemd[1]: Finished ostree-finalize-staged.service - OSTree Finalize Staged Deployment.
--- systemctl reboot ---
09:32:32 systemd[1]: boot.automount: Deactivated successfully.  <- before finalize
09:32:32 systemd[1]: Unset automount boot.automount - EFI System Partition Automount.
09:32:32 systemd[1]: boot.mount: Deactivated successfully.      <- no "Unmounting" line
09:32:32 systemd[1]: Stopping ostree-finalize-staged.service - OSTree Finalize Staged Deployment...
09:32:32 ostree[3438]: error: Remounting /boot read-write: Invalid argument
09:32:32 systemd[1]: ostree-finalize-staged.service: Control process exited, code=exited, status=1/FAILURE
09:32:32 systemd[1]: ostree-finalize-staged.service: Failed with result 'exit-code'.

Journal (good shutdown for comparison, from CI, no expiry)

13:13:33 systemd[1]: Mounted boot.mount - /boot.
13:14:24 systemd[1]: Started ostree-finalize-staged-hold.service - Hold /boot Open for OSTree Finalize Staged Deployment.
--- reboot ---
13:14:48 systemd[1]: Stopping ostree-finalize-staged.service - OSTree Finalize Staged Deployment...
13:15:01 ostree[1651]: Finalized deployment
13:15:07 systemd[1]: Stopped ostree-finalize-staged.service - OSTree Finalize Staged Deployment.
13:15:07 systemd[1]: boot.automount: Deactivated successfully.  <- after finalize
13:15:07 systemd[1]: boot.mount: Deactivated successfully.

Observations

• Bad case: boot.automount and boot.mount are stopped before
  finalize's ExecStop. Good case: they are stopped after. The shutdown
  ordering between ostree-finalize-staged*.service and boot.mount seems
  to break once the mount has been re-established through the automount
  after an expiry.
• In the bad case boot.mount deactivates without an "Unmounting" line.
  This suggests a lazy unmount while ostree-finalize-staged-hold still
  holds /boot open; the following MS_REMOUNT then fails with EINVAL.
• Timing does not matter: here the reboot came 3s after staging and it still
  failed. The /boot expiry before staging is the only requirement.
• The failure is silent: users only notice that the update did not apply.

References

• CI job (bootc-operator): https://github.com/bootc-dev/bootc-operator/actions/runs/41061149430/job/125282552197?pr=251
• CI journals (artifact e2e-logs-1.36):
  https://github.com/bootc-dev/bootc-operator/actions/runs/41061149430/artifacts/13146204757
  • testmultinoderollout/journal-testmultinoderollout-f108e0.txt
  • testmidrolloutimagechange/journal-testmidrolloutimagechange-bc377b.txt