Skip to content

perf(convert): reduce rootfs copies from 3 to 1#110

Open
imlk0 wants to merge 36 commits into
masterfrom
fde-convert-slow
Open

perf(convert): reduce rootfs copies from 3 to 1#110
imlk0 wants to merge 36 commits into
masterfrom
fde-convert-slow

Conversation

@imlk0
Copy link
Copy Markdown
Collaborator

@imlk0 imlk0 commented Apr 29, 2026 •
edited
Loading

Summary

  • Replace the intermediate rootfs.img file and final qemu-img convert step with a qcow2 overlay/snapshot architecture, reducing full rootfs copies from 3 → 1
  • Source image is modified through a writable overlay (source-mod), then forked into two independent snapshots (source-read for verity hash, source-write for dracut initrd generation)
  • Eliminates two full rootfs copies (~17s dd + ~32s qemu-img convert ≈ ~49s savings)
  • Remove --device / operate_on_device support: deprecate in-place device conversion in favor of file-based (--in/--out) only
  • Fix RPM version detection: rpm -q outputs error to stdout on uninstalled packages, which was captured as the version string
  • Fix unbound boot_part variable when source has no separate boot partition
  • Remove --wipe-freed-space option (no longer needed with qcow2 overlay architecture)
  • Update docs: quick-start.md, quick-start_zh.md — replace --device examples with --in/--out

Test plan

  • Run conversion on aliyun_3_x64 image — all 11 steps complete successfully
  • Verify output qcow2 has correct partition layout (BIOS boot, EFI, boot ext4, rootfs LUKS2)
  • Verify verity hash is computed correctly
  • Verify dracut generates initrd successfully
  • Timing: ~5m24s total

@shankailun-aliyun
Copy link
Copy Markdown

shankailun-aliyun commented Apr 29, 2026

@imlk0 ，您好，您的请求已接收，请耐心等待结果。

@shankailun-aliyun
Copy link
Copy Markdown

shankailun-aliyun commented Apr 29, 2026

@imlk0 ，您好，未检测到有镜像需要构建，如需重新检测请评论 /start 。

@imlk0 imlk0 force-pushed the fde-convert-slow branch from aa6ba77 to ec85e19 Compare April 29, 2026 12:12
@shankailun-aliyun
Copy link
Copy Markdown

shankailun-aliyun commented Apr 29, 2026

@imlk0 ，您好，您的请求已接收，请耐心等待结果。

@shankailun-aliyun
Copy link
Copy Markdown

shankailun-aliyun commented Apr 29, 2026

@imlk0 ，您好，未检测到有镜像需要构建，如需重新检测请评论 /start 。

@shankailun-aliyun
Copy link
Copy Markdown

shankailun-aliyun commented Apr 29, 2026

@imlk0 ，您好，您的请求已接收，请耐心等待结果。

@shankailun-aliyun
Copy link
Copy Markdown

shankailun-aliyun commented Apr 29, 2026

@imlk0 ，您好，未检测到有镜像需要构建，如需重新检测请评论 /start 。

@imlk0 imlk0 force-pushed the fde-convert-slow branch from ec85e19 to 07fc888 Compare April 29, 2026 12:17
@shankailun-aliyun
Copy link
Copy Markdown

shankailun-aliyun commented Apr 29, 2026

@imlk0 ，您好，您的请求已接收，请耐心等待结果。

@shankailun-aliyun
Copy link
Copy Markdown

shankailun-aliyun commented Apr 29, 2026

@imlk0 ，您好，未检测到有镜像需要构建，如需重新检测请评论 /start 。

@shankailun-aliyun
Copy link
Copy Markdown

shankailun-aliyun commented Apr 29, 2026

@imlk0 ，您好，您的请求已接收，请耐心等待结果。

@shankailun-aliyun
Copy link
Copy Markdown

shankailun-aliyun commented Apr 29, 2026

@imlk0 ，您好，未检测到有镜像需要构建，如需重新检测请评论 /start 。

@shankailun-aliyun
Copy link
Copy Markdown

shankailun-aliyun commented Apr 29, 2026

@imlk0 ，您好，您的请求已接收，请耐心等待结果。

@shankailun-aliyun
Copy link
Copy Markdown

shankailun-aliyun commented Apr 29, 2026

@imlk0 ，您好，未检测到有镜像需要构建，如需重新检测请评论 /start 。

@shankailun-aliyun
Copy link
Copy Markdown

shankailun-aliyun commented Apr 29, 2026

@imlk0 ，您好，您的请求已接收，请耐心等待结果。

@shankailun-aliyun
Copy link
Copy Markdown

shankailun-aliyun commented Apr 29, 2026

@imlk0 ，您好，未检测到有镜像需要构建，如需重新检测请评论 /start 。

@imlk0 imlk0 force-pushed the fde-convert-slow branch from 9ef0a95 to 0c97e2d Compare April 29, 2026 13:09
@shankailun-aliyun
Copy link
Copy Markdown

shankailun-aliyun commented Apr 29, 2026

@imlk0 ，您好，您的请求已接收，请耐心等待结果。

@shankailun-aliyun
Copy link
Copy Markdown

shankailun-aliyun commented Apr 29, 2026

@imlk0 ，您好，未检测到有镜像需要构建，如需重新检测请评论 /start 。

@imlk0 imlk0 force-pushed the fde-convert-slow branch from 0c97e2d to 8a094cc Compare April 29, 2026 13:13
@shankailun-aliyun
Copy link
Copy Markdown

shankailun-aliyun commented Apr 29, 2026

@imlk0 ，您好，您的请求已接收，请耐心等待结果。

@shankailun-aliyun
Copy link
Copy Markdown

shankailun-aliyun commented Apr 29, 2026

@imlk0 ，您好，未检测到有镜像需要构建，如需重新检测请评论 /start 。

@shankailun-aliyun
Copy link
Copy Markdown

shankailun-aliyun commented Apr 30, 2026

@imlk0 ，您好，您的请求已接收，请耐心等待结果。

@shankailun-aliyun
Copy link
Copy Markdown

shankailun-aliyun commented Apr 30, 2026

@imlk0 ，您好，未检测到有镜像需要构建，如需重新检测请评论 /start 。

@shankailun-aliyun
Copy link
Copy Markdown

shankailun-aliyun commented Apr 30, 2026

@imlk0 ，您好，您的请求已接收，请耐心等待结果。

@shankailun-aliyun
Copy link
Copy Markdown

shankailun-aliyun commented Apr 30, 2026

@imlk0 ，您好，未检测到有镜像需要构建，如需重新检测请评论 /start 。

@shankailun-aliyun
Copy link
Copy Markdown

shankailun-aliyun commented May 27, 2026

@imlk0 ，您好，您的请求已接收，请耐心等待结果。

@shankailun-aliyun
Copy link
Copy Markdown

shankailun-aliyun commented May 27, 2026

@imlk0 ，您好，未检测到有镜像需要构建，如需重新检测请评论 /start 。

@shankailun-aliyun
Copy link
Copy Markdown

shankailun-aliyun commented May 27, 2026

@imlk0 ，您好，您的请求已接收，请耐心等待结果。

@shankailun-aliyun
Copy link
Copy Markdown

shankailun-aliyun commented May 27, 2026

@imlk0 ，您好，未检测到有镜像需要构建，如需重新检测请评论 /start 。

@shankailun-aliyun
Copy link
Copy Markdown

shankailun-aliyun commented Jun 4, 2026

@imlk0 ，您好，您的请求已接收，请耐心等待结果。

@shankailun-aliyun
Copy link
Copy Markdown

shankailun-aliyun commented Jun 4, 2026

@imlk0 ，您好，未检测到有镜像需要构建，如需重新检测请评论 /start 。

@shankailun-aliyun
Copy link
Copy Markdown

shankailun-aliyun commented Jun 4, 2026

@imlk0 ，您好，您的请求已接收，请耐心等待结果。

@shankailun-aliyun
Copy link
Copy Markdown

shankailun-aliyun commented Jun 4, 2026

@imlk0 ，您好，未检测到有镜像需要构建，如需重新检测请评论 /start 。

@imlk0
fix(convert): use correct Ubuntu package names for UKI grub tools
1b9b00a 
grub2-tools is the RHEL package name. On Ubuntu, the UKI grub tools
are in grub-efi-amd64-bin and sbsigntool. Also add these packages
to the CI test-convert dependencies.
@shankailun-aliyun
Copy link
Copy Markdown

shankailun-aliyun commented Jun 4, 2026

@imlk0 ，您好，您的请求已接收，请耐心等待结果。

@shankailun-aliyun
Copy link
Copy Markdown

shankailun-aliyun commented Jun 4, 2026

@imlk0 ，您好，未检测到有镜像需要构建，如需重新检测请评论 /start 。

@imlk0
fix(convert): parse virtual size from text output instead of JSON
70bb1ed 
qemu-img info --output=json failed to extract virtual-size
from backing-file qcow2 overlays. Parse the human-readable
text output instead: 'virtual size: 10 GiB (10737418240 bytes)'.
Also add validation before passing to qemu-img create.
@shankailun-aliyun
Copy link
Copy Markdown

shankailun-aliyun commented Jun 5, 2026

@imlk0 ，您好，您的请求已接收，请耐心等待结果。

@shankailun-aliyun
Copy link
Copy Markdown

shankailun-aliyun commented Jun 5, 2026

@imlk0 ，您好，未检测到有镜像需要构建，如需重新检测请评论 /start 。

@imlk0
fix(test): increase QEMU boot timeout and add Ubuntu login prompt det…
398ce8f 
…ection

- Increase timeout from 6 min to 15 min (GH runners without KVM
  boot ~10x slower)
- Add Ubuntu login prompt detection patterns: 'Welcome to Ubuntu'
  and 'ttyN login:' alongside existing 'on an x86_64'
@shankailun-aliyun
Copy link
Copy Markdown

shankailun-aliyun commented Jun 5, 2026

@imlk0 ，您好，您的请求已接收，请耐心等待结果。

@shankailun-aliyun
Copy link
Copy Markdown

shankailun-aliyun commented Jun 5, 2026

@imlk0 ，您好，未检测到有镜像需要构建，如需重新检测请评论 /start 。

@imlk0
fix(convert): handle cloud images with non-sequential GPT partition n…
53ae746 
…umbering

- UKI mode: check if EFI partition is already >= 250M before
  attempting resize (avoids unnecessary parted operations)
- Use sfdisk dump/modify/apply approach for UKI EFI resize instead
  of parted, which fails on cloud images with non-sequential
  partition numbering (e.g., Ubuntu cloud images with partitions
  1, 14, 15, 16)
- Add partprobe + udevadm settle before partition modifications
- Fall back to parted if sfdisk approach fails

fix(test): increase QEMU boot timeout and add Ubuntu login detection
- Timeout increased from 6 min to 15 min for KVM-less runners
- Added 'Welcome to Ubuntu' and 'ttyN login:' patterns
@shankailun-aliyun
Copy link
Copy Markdown

shankailun-aliyun commented Jun 5, 2026

@imlk0 ，您好，您的请求已接收，请耐心等待结果。

@shankailun-aliyun
Copy link
Copy Markdown

shankailun-aliyun commented Jun 5, 2026

@imlk0 ，您好，未检测到有镜像需要构建，如需重新检测请评论 /start 。

@imlk0
fix(convert): fix sed command for UKI EFI partition resize
2c975f3 
- The sed command had issues with device path containing '/'
  characters. Use '|' delimiter and escape device paths properly.
- Simplified the sed replacement to just modify start and size.
@shankailun-aliyun
Copy link
Copy Markdown

shankailun-aliyun commented Jun 5, 2026

@imlk0 ，您好，您的请求已接收，请耐心等待结果。

@shankailun-aliyun
Copy link
Copy Markdown

shankailun-aliyun commented Jun 5, 2026

@imlk0 ，您好，未检测到有镜像需要构建，如需重新检测请评论 /start 。

@imlk0
fix(test): skip QEMU boot test when KVM is not available
f1a3fc3 
GH Actions runners don't have /dev/kvm, so QEMU runs ~10x slower.
A full Ubuntu boot takes > 15 minutes without KVM, which exceeds
even our extended timeout. Skip the boot test when KVM is absent;
the conversion itself is still fully tested.
@shankailun-aliyun
Copy link
Copy Markdown

shankailun-aliyun commented Jun 5, 2026

@imlk0 ，您好，您的请求已接收，请耐心等待结果。

@shankailun-aliyun
Copy link
Copy Markdown

shankailun-aliyun commented Jun 5, 2026

@imlk0 ，您好，未检测到有镜像需要构建，如需重新检测请评论 /start 。

@shankailun-aliyun
Copy link
Copy Markdown

shankailun-aliyun commented Jun 5, 2026

@imlk0 ，您好，您的请求已接收，请耐心等待结果。

@shankailun-aliyun
Copy link
Copy Markdown

shankailun-aliyun commented Jun 5, 2026

@imlk0 ，您好，未检测到有镜像需要构建，如需重新检测请评论 /start 。

@shankailun-aliyun
Copy link
Copy Markdown

shankailun-aliyun commented Jun 5, 2026

@imlk0 ，您好，您的请求已接收，请耐心等待结果。

@shankailun-aliyun
Copy link
Copy Markdown

shankailun-aliyun commented Jun 5, 2026

@imlk0 ，您好，未检测到有镜像需要构建，如需重新检测请评论 /start 。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@imlk0 @shankailun-aliyun