Skip to content

build: Speed up RPM builds by avoiding redundant cargo rebuilds - #2609

Open
jeckersb wants to merge 4 commits into
bootc-dev:mainfrom
jeckersb:build-optimizations
Open

jeckersb wants to merge 4 commits into
bootc-dev:mainfrom
jeckersb:build-optimizations

Conversation

@jeckersb

Copy link
Copy Markdown
Collaborator

The Koji RPM build was spending a lot of time recompiling the same code
over and over. This series removes those duplicate builds:

  • Drop panic = "abort" from the release profile. Since Rust 1.81
    (we need at least 1.85), a panic that reaches an extern "C" boundary
    aborts on its own, so this setting is no longer needed for safety. The
    test harness can't use it, so cargo test --release / %cargo_test
    was rebuilding every dependency with a different panic setting.
  • Enable tokio test-util in the workspace dependency. Before, only
    a dev-dependency turned it on, so tokio and everything that depends on
    it was compiled once for build and again for test. With this change,
    cargo test --release after cargo build --release rebuilds 20 crates
    instead of 36. The binary grows by ~28KB (0.1%).
  • Don't rebuild during RPM %install. make install re-ran cargo
    three times (~12 minutes in Koji). These --release builds also wrote
    over the %cargo_build output through the target/release ->
    target/rpm symlink, so the RPM shipped binaries built with our
    release profile instead of Fedora's rpm profile. Add SKIP_BUILD=1
    to the Makefile and use it from the spec. Plain make install works
    the same as before.
  • Don't build bootc to generate man pages in the RPM. CI already
    checks that the generated sections of the man pages match the CLI, so
    re-syncing them in a release build changes nothing and costs a full
    extra build (~7 minutes). Add xtask manpages --no-sync /
    make manpages-no-sync and build xtask with the rpm profile and
    %{__cargo} so it doesn't trigger a rebuild in %cargo_test.
    The output is identical to make manpages.

Assisted-by: AI

Copilot AI balanced review requested due to automatic review settings October 30, 2026 22:02
@bootc-bot
bootc-bot Bot requested a review from henrywang October 30, 2026 22:02

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The non-container build’s new rpm Cargo profile assumption needs maintainer validation for older supported RHEL 9 rust-toolsets because the macro definitions needed to check legacy-branch compatibility are absent from this snapshot.

No open issues 🎉

What changed in this PR

This PR reduces redundant Cargo compilation during RPM packaging while preserving the binaries built with Fedora’s rpm profile and keeping ordinary make install behavior unchanged.

Changes:

  • Align release and test builds by removing panic = "abort" and enabling Tokio’s test-util feature at workspace level.
  • Add SKIP_BUILD=1 so RPM installation reuses existing build artifacts.
  • Generate man pages without rebuilding bootc, using the RPM Cargo environment and profile for xtask.
File Description
Makefile Adds build-skipping controls, configurable Cargo invocation, and a no-sync man-page target.
crates/​xtask/​src/​xtask.rs Adds and dispatches the manpages --no-sync option.
crates/​xtask/​src/​man.rs Adds man-page generation from existing Markdown without CLI synchronization.
crates/​lib/​Cargo.toml Removes the redundant Tokio dev-dependency configuration.
contrib/​packaging/​bootc.spec Uses no-sync man-page generation and skips rebuilding during installation.
Cargo.toml Removes release panic-abort behavior and enables Tokio test-util workspace-wide.

Review effort: Balanced

@jeckersb

Copy link
Copy Markdown
Collaborator Author

🔵 Needs a closer look

The non-container build’s new rpm Cargo profile assumption needs maintainer validation for older supported RHEL 9 rust-toolsets because the macro definitions needed to check legacy-branch compatibility are absent from this snapshot.

Good to consider that. Poked around a bit with an agent and came up with this:

Checked macros.rust-toolset from el9 rust-toolset 1.84.1, 1.92.0 and
1.99.1: all define [profile.rpm], create the target/release →
target/rpm symlink in %cargo_prep, and use --profile rpm in
%cargo_build/%cargo_test. 1.75.0 (9.4) predates this, but it is
already unsupported by main (MSRV 1.85, and the spec needs %cargo_prep
-N and %cargo_vendor_manifest).

So I think this is fine as-is

@jeckersb jeckersb added this to the 1.18 milestone Oct 30, 2026
@jeckersb

Copy link
Copy Markdown
Collaborator Author

ballpark comparison, over here the f44 package job took 31m20s whereas the same job here took 19m47s. But CI uses the "container" packaging flow which previously was already a bit better optimized than the "real" builds in koji. For me a local mockbuild went from about 15m down to 5m.

@jmarrero

Copy link
Copy Markdown
Contributor

@jmarrero-bot review this and explain how is the rpm produced with this change different from the current ones.

@jmarrero jmarrero self-assigned this Oct 31, 2026
@jmarrero-bot jmarrero-bot moved this to In Progress in Workstream Oct 31, 2026
This was added in 6792973 (Aug 2023) because a panic unwinding
across an `extern "C"` boundary was undefined behavior. Since Rust
1.81.0, panics reaching a non-unwind ABI boundary abort automatically,
so the profile setting is no longer needed for FFI safety. Our MSRV
is 1.85.0.

- https://blog.rust-lang.org/2024/09/05/Rust-1.81.0/
- https://rust-lang.github.io/rfcs/2945-c-unwind-abi.html

Keeping it has a real cost: the test harness requires unwinding, so
`cargo test --release` (and `%cargo_test` in the RPM build) rebuilds
every dependency with a different panic strategy instead of reusing
the release build. See
https://doc.rust-lang.org/cargo/reference/profiles.html#panic

Spawned threads/tasks have their join handles awaited, so a panic in
one is propagated as an error. The one exception is the hyper
connection driver task for the podman API client, where a panic would
surface as a failed request instead.

Assisted-by: AI
Signed-off-by: John Eckersberg <dev@eckersberg.com>
The `test-util` feature was only enabled via a dev-dependency in
bootc-lib (for the `start_paused` retry tests in deploy.rs). Cargo
unifies dev-dependency features when building tests, so tokio and
everything depending on it (hyper, tokio-util, composefs, etc.) was
compiled twice: once for `cargo build --release` and again for
`cargo test --release`.

Enabling it in the workspace dependency means both use the same tokio
build. After `cargo build --release`, `cargo test --release` now
rebuilds 20 crates (workspace members and dev-only deps) instead of 36.

The cost to the shipped binary is negligible: the stripped `bootc`
binary grows by ~28KB (0.1%), within normal build-to-build variance.

Assisted-by: AI
Signed-off-by: John Eckersberg <dev@eckersberg.com>
`make install` depends on `completion` -> `bin` -> `manpages`, so the
RPM %install phase re-ran cargo three times: rebuilding xtask, building
bootc with the docgen feature to dump the CLI, then building the
binaries again without docgen. In the Koji build log this accounted for
roughly 12 minutes.

Worse, `%cargo_prep` symlinks `target/release` to `target/rpm`, so
these `--release` builds overwrote the `%cargo_build` output, and the
RPM actually shipped binaries built with our `release` profile rather
than Fedora's `rpm` profile.

Add a `SKIP_BUILD=1` option to the Makefile that drops the build
dependency from `completion`, and use it from the spec's %install so
it only installs what %build produced. Plain `make install` is
unchanged.

Assisted-by: AI
Signed-off-by: John Eckersberg <dev@eckersberg.com>
`make manpages` builds bootc with the docgen feature, dumps its CLI as
JSON and syncs the generated options/subcommands sections of
docs/src/man/*.md before converting them to roff. The CI `validate`
job already enforces (via `update-generated from-code --check`) that
those committed sections match the CLI, so in a release build the sync
is a no-op that costs a full extra build of bootc (~7 minutes in Koji).

Add `xtask manpages --no-sync` and a `make manpages-no-sync` target
that only do the conversion, and use it from the spec. The output is
identical to `make manpages`.

xtask itself still needs to be built, and it must not disturb the
%cargo_build state that %cargo_test reuses. %cargo_prep symlinks
target/release to target/rpm, and the %cargo_* macros run cargo with
RUSTC_BOOTSTRAP=1, which some build scripts (proc-macro2, anyhow,
thiserror) track. Building xtask with plain `cargo run --release`
therefore caused %cargo_test to rebuild 100+ crates. Add XTASK_PROFILE
and CARGO variables to the Makefile, and have the spec build xtask with
the `rpm` profile and `%{__cargo}`.

Assisted-by: AI
Signed-off-by: John Eckersberg <dev@eckersberg.com>
@jeckersb
jeckersb force-pushed the build-optimizations branch from 4962df3 to a7f0421 Compare October 31, 2026 20:33
@jmarrero-bot

Copy link
Copy Markdown

@jmarrero The packaging changes look good to merge. Two things are worth a decision first: dropping panic = "abort", and where tokio test-util is enabled. Reviewed at a7f0421.

I built both RPMs Koji-style (%cargo_build, %check on) in fedora:44 on 16 cores, base 709dfe5 vs head:

  • rpmbuild takes 12m18s instead of 26m49s: %install drops from 9m12s to 21s, and %check rebuilds 20 crates instead of 358.
  • Confirmed: today's RPM ships the release-profile bootc that %install relinks through the target/release -> target/rpm symlink. On Fedora the codegen flags were effectively the same (%install gets the same RUSTFLAGS, and profile.rpm inherits release). What was missing is RUSTC_BOOTSTRAP=1, so anyhow/thiserror/cap-std were built without their nightly cfgs (error_generic_member_access, can_vector, write_all_vectored, io_error_more). Those only affect backtrace forwarding and vectored-I/O hints, so bootc behaves the same either way. On EL9, %install exports no RUSTFLAGS at all, so the old binary there used codegen-units=16.
  • Apart from the version string in LICENSE.dependencies, only the three binaries (and their debuginfo) differ. Man pages, units, completions, configs and Requires/Provides are byte-identical.
  • /usr/bin/bootc grows from 19.08 to 22.81 MB (+19.5%; the bootc RPM goes from 5.9 to 7.3 MB): about 2.6 MB of landing-pad code in .text plus 1.1 MB of unwind tables. A forced panic (bootc completion bash > /dev/full) now exits 101 with no core dump, instead of SIGABRT with a core.

Findings:

  • Cargo.toml:8: dropping panic = "abort" accounts for nearly all of the ~3.7 MB per image; together with test-util it saves about 1 minute of %check here (16 cores). The "~28KB" in the description covers only test-util. The FFI reasoning holds, so this is a size and crash-reporting trade-off.
  • Cargo.toml:77: with test-util on the workspace tokio dependency, cargo package writes it into the published bootc-internal-utils and bootc-internal-blockdev manifests, so bootupd would get it too. Enabling it on an unpublished crate's tokio dependency (e.g. crates/cli) should avoid the rebuild just as well.

Commands, logs, readelf output and tables: https://gist.github.com/jmarrero-bot/23ecd97acbd9cabb70d4f0c56f8092c9

Generated-by: https://github.com/jmarrero-bot#llms

@jmarrero-bot jmarrero-bot moved this from In Progress to Draft in Workstream Nov 1, 2026
@jmarrero-bot jmarrero-bot moved this from Draft to Done in Workstream Nov 1, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants