fix: conditional wipefs before systemd-makefs for false-positive signature detection - #947
Open
youwalther65 wants to merge 1 commit into
Open
Conversation
…sitive signature detection On encrypted EBS volumes, pseudorandom data on the BOTTLEROCKET-DATA partition can match an Atari partition table signature (2 bytes), causing mkfs.xfs to refuse formatting. This happens in approximately 1/1800 instance launches and prevents the node from booting. Add a conditional wipefs -a that only runs when blkid detects no existing filesystem, clearing spurious signatures before systemd-makefs attempts to create the filesystem.
Author
|
Did a test in an admin container [root@admin]# sudo sheltie
bash-5.2# ls -l /dev/disk/by-partlabel/BOTTLEROCKET-DATA
lrwxrwxrwx. 1 root root 15 Jun 16 17:09 /dev/disk/by-partlabel/BOTTLEROCKET-DATA -> ../../nvme1n1p1
bash-5.2# ls -l /usr/sbin/blkid
-rwxr-xr-x. 1 root root 64976 May 1 04:20 /usr/sbin/blkid
bash-5.2# ls -l /usr/sbin/wipefs
-rwxr-xr-x. 1 root root 41440 May 1 04:20 /usr/sbin/wipefs
bash-5.2# /bin/sh -c '! /usr/sbin/blkid /dev/disk/by-partlabel/BOTTLEROCKET-DATA && /usr/sbin/wipefs -a /dev/disk/by-partlabel/BOTTLEROCKET-DATA'
expected first arg 'sh', not '/bin/sh'
bash-5.2# sh -c '! /usr/sbin/blkid /dev/disk/by-partlabel/BOTTLEROCKET-DATA && /usr/sbin/wipefs -a /dev/disk/by-partlabel/BOTTLEROCKET-DATA'
'!' could not be found
Caused by these errors (recent errors listed first):
1: failed to locate '!' in PATH
2: cannot find binary pathSo unfortunately this simple approach does not work because of the usage of underlying |
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes bottlerocket-os/bottlerocket#4855
On encrypted EBS volumes, pseudorandom data on the
BOTTLEROCKET-DATApartition can match an Atari partition table signature (only 2 bytes at a specific offset), causingmkfs.xfsto refuse formatting (~1/1800 instance launches). This cascades to prevent all of/local,/var,/opt,/mntfrom mounting — containerd and kubelet never start, the node never joins the EKS cluster.This adds a conditional
wipefs -abeforesystemd-makefsinprepare-local-fs.servicethat only runs when no valid filesystem is detected.Implementation Details
wipefs -aerases all magic strings/signatures from a device (partition tables, filesystem superblocks, etc.)blkidexits 0 when it finds a filesystem, 2 when it does not — the!inverts thissystemd-makefsinternally checks for existing filesystems and is a no-op if one exists-prefix on the ExecStart line makes the entire wipefs step non-fatalprepare-local-fs-encrypted.conf) overridesDATA_PARTITION_BLOCK_DEVICEto/dev/mapper/BOTTLEROCKET-DATA— the same fix applies to both the plain and encrypted paths since the decrypted mapper device also contains random-looking data before first formatTest plan
prepare-local-fs.servicesucceedswipefsis skipped (blkid returns 0, short-circuits)