From 950207799e77661fb491101f67d9fc0b2b0e639d Mon Sep 17 00:00:00 2001 From: Sylvestre Ledru Date: Sun, 30 Aug 2026 16:42:33 +0200 Subject: [PATCH 1/2] build-pgo.sh: instrument with the same LTO settings as the final build The instrumented binary was built with LTO off and 16 codegen units, but the profile is consumed by a fat-LTO build. Inlining runs before instrumentation, so the counters describe a different call graph: the mismatched profile covers 4098 functions where the matched one covers 1883. The result was a profile that made several utils slower than no PGO at all: wc -w regressed 57% (+21.7% instructions, all in uu_wc::process_chunk). With the instrumented build matched to the final one, wc -w is back to parity and PGO gives cat -n -29%, uniq -c -27%, nl -26%, fold -22%, sort -18%, sort -n -17%, wc -15%, sort -k -11%. Build time goes from 1m56s to 2m58s. --- util/build-pgo.sh | 13 +++++-------- 1 file changed, 5 insertions(+), 8 deletions(-) diff --git a/util/build-pgo.sh b/util/build-pgo.sh index 7d88ab7aed5..04b3b475422 100755 --- a/util/build-pgo.sh +++ b/util/build-pgo.sh @@ -83,14 +83,11 @@ begin_step "Step 1: instrumented build" INSTR_DIR="${TARGET_DIR}/instrumented" rm -rf "$PROFILE_DIR" mkdir -p "$PROFILE_DIR" -# The instrumented binary is only ever run for training, so skip the expensive -# whole-program codegen. Profiles are keyed by function, not by LTO/CGU layout, -# so this does not affect the profile the final build consumes. -( - export CARGO_PROFILE_RELEASE_LTO=false - export CARGO_PROFILE_RELEASE_CODEGEN_UNITS=16 - cargo_build "$INSTR_DIR" "-Cprofile-generate=${PROFILE_DIR}" -) +# The instrumented build must use the same LTO and codegen-unit settings as the +# final one. Inlining happens before instrumentation, so building it with LTO +# off yields counters for functions that no longer exist once the final build +# runs whole-program codegen, and the profile is then largely wasted on it. +cargo_build "$INSTR_DIR" "-Cprofile-generate=${PROFILE_DIR}" BIN="${INSTR_DIR}/release/coreutils" [ -x "$BIN" ] || { echo "instrumented binary not found: ${BIN}" >&2; exit 1; } From 2195b97b0976696b49c3814ea79108df67d7ada8 Mon Sep 17 00:00:00 2001 From: Sylvestre Ledru Date: Sun, 30 Aug 2026 16:53:57 +0200 Subject: [PATCH 2/2] docs: document the PGO training/build settings requirement Record what the LTO mismatch cost and why the existing function-count check does not catch it, and add measured per-utility gains to the performance page. --- docs/src/packaging.md | 19 +++++++++++++++++-- docs/src/performance.md | 23 +++++++++++++++++++++++ 2 files changed, 40 insertions(+), 2 deletions(-) diff --git a/docs/src/packaging.md b/docs/src/packaging.md index 30487eb5714..222c8c5de78 100644 --- a/docs/src/packaging.md +++ b/docs/src/packaging.md @@ -144,12 +144,27 @@ Things to know before wiring it into a package build: input to the final build, so re-training on another machine can change the resulting code layout. Generate it once, ship it as a source artifact, and build with `-Cprofile-use=` instead of re-running the training. +- **Train with the same `lto` and `codegen-units` as the final build.** This is + the easy way to get a profile that makes things *slower*. Inlining happens + before instrumentation, so a training build with different settings records + counters for a call graph the final build no longer has. Our `[profile.release]` + uses `lto = "fat"` and `codegen-units = 1`, and the script trains with them; if + you use `--train-only` and then run your own `cargo build`, keep those two + values identical. When they did not match, `wc -w` came out **57% slower** than + a plain non-PGO release build, while the profile itself looked perfectly + healthy. - **A bad profile fails the build rather than silently degrading it**: the script refuses to continue if the merged profile covers fewer than 500 functions, which is what an environment where the training workloads did not - actually run looks like. + actually run looks like. Note that this only catches a profile that is + *missing*, not one that is *mismatched*: the mismatch above produced a profile + covering more functions (4098) than the correct one (1883), since a non-LTO + build still has all the symbols that whole-program codegen later merges away. + If you change the training or build settings, measure the result. - It costs a second full build plus the training run, so expect the package - build to take noticeably longer. + build to take noticeably longer. On a 24-core x86_64 machine the full script + takes about 3 minutes for the `unix` feature set, and the resulting binary is + ~0.8% larger. ## Additional artifacts diff --git a/docs/src/performance.md b/docs/src/performance.md index c492d81bc97..5542559d17a 100644 --- a/docs/src/performance.md +++ b/docs/src/performance.md @@ -164,3 +164,26 @@ rustup component add llvm-tools See [packaging](packaging.md#profile-guided-optimization-pgo) for the details and the options. + +How much it buys varies a lot by utility. Measured on x86_64 with `hyperfine` +against a plain `cargo build --release`, the text-processing utilities gain the +most, while utilities dominated by syscalls or by a single hand-tuned loop +barely move: + +| Workload | Change | +| -------- | ------ | +| `cat -n`, `uniq -c`, `nl`, `fold -w` | -22% to -29% | +| `sort`, `sort -n` | -17% | +| `wc`, `sort -k` | -11% to -15% | +| `ls -lR`, `head` | -4% to -8% | +| `cut`, `seq`, `sha256sum`, `base64`, process startup | no change | + +Two things to keep in mind when benchmarking a PGO build: + +- The workloads in `util/build-pgo.sh` are what the profile is trained on. A + utility or a mode that is not exercised there gets no benefit, and measuring + one tells you little about the utilities that are. +- Wall-clock and instruction counts can disagree, since PGO changes code layout + as well as the code itself. `wc` gets 15% faster while executing *more* + instructions. If you use `valgrind --tool=cachegrind` for a noise-free + comparison, confirm the result with `hyperfine` before believing it.