Skip to content

fix: conditional wipefs before systemd-makefs for false-positive signature detection - #947

Open
youwalther65 wants to merge 1 commit into
bottlerocket-os:developfrom
youwalther65:fix/wipefs-false-positive-atari
Open

fix: conditional wipefs before systemd-makefs for false-positive signature detection#947
youwalther65 wants to merge 1 commit into
bottlerocket-os:developfrom
youwalther65:fix/wipefs-false-positive-atari

Conversation

@youwalther65

Copy link
Copy Markdown

Summary

Fixes bottlerocket-os/bottlerocket#4855

On encrypted EBS volumes, pseudorandom data on the BOTTLEROCKET-DATA partition can match an Atari partition table signature (only 2 bytes at a specific offset), causing mkfs.xfs to refuse formatting (~1/1800 instance launches). This cascades to prevent all of /local, /var, /opt, /mnt from mounting — containerd and kubelet never start, the node never joins the EKS cluster.

This adds a conditional wipefs -a before systemd-makefs in prepare-local-fs.service that only runs when no valid filesystem is detected.

Implementation Details

  • wipefs -a erases all magic strings/signatures from a device (partition tables, filesystem superblocks, etc.)
  • blkid exits 0 when it finds a filesystem, 2 when it does not — the ! inverts this
  • systemd-makefs internally checks for existing filesystems and is a no-op if one exists
  • The - prefix on the ExecStart line makes the entire wipefs step non-fatal
  • The encrypted variant (prepare-local-fs-encrypted.conf) overrides DATA_PARTITION_BLOCK_DEVICE to /dev/mapper/BOTTLEROCKET-DATA — the same fix applies to both the plain and encrypted paths since the decrypted mapper device also contains random-looking data before first format

Test plan

  • Boot a Bottlerocket instance with encrypted EBS data volume and verify prepare-local-fs.service succeeds
  • Verify on an instance with an already-formatted data partition that wipefs is skipped (blkid returns 0, short-circuits)
  • Simulate the false positive by writing Atari signature bytes to a test block device and confirm wipefs clears them before mkfs runs

…sitive signature detection

On encrypted EBS volumes, pseudorandom data on the BOTTLEROCKET-DATA
partition can match an Atari partition table signature (2 bytes), causing
mkfs.xfs to refuse formatting. This happens in approximately 1/1800
instance launches and prevents the node from booting.

Add a conditional wipefs -a that only runs when blkid detects no existing
filesystem, clearing spurious signatures before systemd-makefs attempts
to create the filesystem.
@youwalther65

youwalther65 commented Jun 17, 2026

Copy link
Copy Markdown
Author

Did a test in an admin container sudo sheltie environment and it failed:

[root@admin]# sudo sheltie

bash-5.2# ls -l /dev/disk/by-partlabel/BOTTLEROCKET-DATA
lrwxrwxrwx. 1 root root 15 Jun 16 17:09 /dev/disk/by-partlabel/BOTTLEROCKET-DATA -> ../../nvme1n1p1
bash-5.2# ls -l /usr/sbin/blkid
-rwxr-xr-x. 1 root root 64976 May  1 04:20 /usr/sbin/blkid

bash-5.2# ls -l /usr/sbin/wipefs
-rwxr-xr-x. 1 root root 41440 May  1 04:20 /usr/sbin/wipefs

bash-5.2# /bin/sh -c '! /usr/sbin/blkid /dev/disk/by-partlabel/BOTTLEROCKET-DATA && /usr/sbin/wipefs -a /dev/disk/by-partlabel/BOTTLEROCKET-DATA'
expected first arg 'sh', not '/bin/sh'

bash-5.2# sh -c '! /usr/sbin/blkid /dev/disk/by-partlabel/BOTTLEROCKET-DATA && /usr/sbin/wipefs -a /dev/disk/by-partlabel/BOTTLEROCKET-DATA'     
'!' could not be found

Caused by these errors (recent errors listed first):
  1: failed to locate '!' in PATH
  2: cannot find binary path

So unfortunately this simple approach does not work because of the usage of underlying brush instead of a real bash like shell.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

prepare-local-fs.service fails intermittently on encrypted EBS volumes due to mkfs.xfs false-positive signature detection

1 participant