build-pgo.sh: instrument with the same LTO settings as the final build

docs/src/packaging.md

@@ -144,12 +144,27 @@ Things to know before wiring it into a package build:
   input to the final build, so re-training on another machine can change the
   resulting code layout. Generate it once, ship it as a source artifact, and
   build with `-Cprofile-use=<that file>` instead of re-running the training.
+- **Train with the same `lto` and `codegen-units` as the final build.** This is
+  the easy way to get a profile that makes things *slower*. Inlining happens
+  before instrumentation, so a training build with different settings records
+  counters for a call graph the final build no longer has. Our `[profile.release]`
+  uses `lto = "fat"` and `codegen-units = 1`, and the script trains with them; if
+  you use `--train-only` and then run your own `cargo build`, keep those two
+  values identical. When they did not match, `wc -w` came out **57% slower** than
+  a plain non-PGO release build, while the profile itself looked perfectly
+  healthy.
 - **A bad profile fails the build rather than silently degrading it**: the
   script refuses to continue if the merged profile covers fewer than 500
   functions, which is what an environment where the training workloads did not
-  actually run looks like.
+  actually run looks like. Note that this only catches a profile that is
+  *missing*, not one that is *mismatched*: the mismatch above produced a profile
+  covering more functions (4098) than the correct one (1883), since a non-LTO
+  build still has all the symbols that whole-program codegen later merges away.
+  If you change the training or build settings, measure the result.
 - It costs a second full build plus the training run, so expect the package
-  build to take noticeably longer.
+  build to take noticeably longer. On a 24-core x86_64 machine the full script
+  takes about 3 minutes for the `unix` feature set, and the resulting binary is
+  ~0.8% larger.
 
 ## Additional artifacts
 

docs/src/performance.md

@@ -164,3 +164,26 @@ rustup component add llvm-tools
 
 See [packaging](packaging.md#profile-guided-optimization-pgo) for the details
 and the options.
+
+How much it buys varies a lot by utility. Measured on x86_64 with `hyperfine`
+against a plain `cargo build --release`, the text-processing utilities gain the
+most, while utilities dominated by syscalls or by a single hand-tuned loop
+barely move:
+
+| Workload | Change |
+| -------- | ------ |
+| `cat -n`, `uniq -c`, `nl`, `fold -w` | -22% to -29% |
+| `sort`, `sort -n` | -17% |
+| `wc`, `sort -k` | -11% to -15% |
+| `ls -lR`, `head` | -4% to -8% |
+| `cut`, `seq`, `sha256sum`, `base64`, process startup | no change |
+
+Two things to keep in mind when benchmarking a PGO build:
+
+- The workloads in `util/build-pgo.sh` are what the profile is trained on. A
+  utility or a mode that is not exercised there gets no benefit, and measuring
+  one tells you little about the utilities that are.
+- Wall-clock and instruction counts can disagree, since PGO changes code layout
+  as well as the code itself. `wc` gets 15% faster while executing *more*
+  instructions. If you use `valgrind --tool=cachegrind` for a noise-free
+  comparison, confirm the result with `hyperfine` before believing it.

util/build-pgo.sh

@@ -83,14 +83,11 @@ begin_step "Step 1: instrumented build"
 INSTR_DIR="${TARGET_DIR}/instrumented"
 rm -rf "$PROFILE_DIR"
 mkdir -p "$PROFILE_DIR"
-# The instrumented binary is only ever run for training, so skip the expensive
-# whole-program codegen. Profiles are keyed by function, not by LTO/CGU layout,
-# so this does not affect the profile the final build consumes.
-(
-    export CARGO_PROFILE_RELEASE_LTO=false
-    export CARGO_PROFILE_RELEASE_CODEGEN_UNITS=16
-    cargo_build "$INSTR_DIR" "-Cprofile-generate=${PROFILE_DIR}"
-)
+# The instrumented build must use the same LTO and codegen-unit settings as the
+# final one. Inlining happens before instrumentation, so building it with LTO
+# off yields counters for functions that no longer exist once the final build
+# runs whole-program codegen, and the profile is then largely wasted on it.
+cargo_build "$INSTR_DIR" "-Cprofile-generate=${PROFILE_DIR}"
 
 BIN="${INSTR_DIR}/release/coreutils"
 [ -x "$BIN" ] || { echo "instrumented binary not found: ${BIN}" >&2; exit 1; }