Skip to content

fix(xtask): run Criterion benchmarks per target - #473

Merged
Mohamed Mansour (mohamedmansour) merged 1 commit into
mainfrom
mohamedmansour-fix-criterion-per-target-bench
Aug 25, 2026
Merged

fix(xtask): run Criterion benchmarks per target#473
Mohamed Mansour (mohamedmansour) merged 1 commit into
mainfrom
mohamedmansour-fix-criterion-per-target-bench

Conversation

@mohamedmansour

Copy link
Copy Markdown
Contributor

The failure on main

cargo xtask bench all mapped every Criterion target onto a single cargo bench --workspace. Cargo forwards everything after -- to every benchable target in the workspace, including the libtest unit-test harnesses of libraries and binaries. Those harnesses don't understand Criterion's baseline flags, so the run aborted on the first one it reached.

Reproduced on today's tip (e971c13a):

$ cargo xtask bench all --save-baseline a9-red-final
    Finished `bench` profile [optimized] target(s) in 1m 09s
     Running unittests src\main.rs (target\release\deps\demo_shell-ffd2d20b0bea4e63.exe)
error: Unrecognized option: 'save-baseline'
error: bench failed, to rerun pass `-p demo-shell --bin demo-shell`
bench failed: exit code 101
error: process didn't exit successfully: `xtask.exe bench all --save-baseline a9-red-final` (exit code: 1)

Baselines recorded: 0. The before/after workflow documented in BENCHMARKS.md could not work at all.

The fix

bench all now walks an explicit CRITERION_BENCHES table and invokes each target on its own, so baseline flags only ever reach a real Criterion binary:

cargo bench -p <package> --bench <target> -- [extra args] --save-baseline NAME

Declared targets (11):

package bench target
microsoft-webui-parser parser_bench
microsoft-webui-handler handler_bench
microsoft-webui-handler bootstrap_state_bench
microsoft-webui-handler streaming_hydration_bench
microsoft-webui-protocol protocol_bench
microsoft-webui-expressions expressions_bench
microsoft-webui-state state_bench
microsoft-webui-ffi protocol_bench
microsoft-webui contact_book_bench
microsoft-webui streaming_bench
microsoft-webui component_assets_bench

Each target is announced and run separately:

$ cargo xtask bench all --save-baseline a9-green-final
▸ microsoft-webui-parser / parser_bench
     Running benches\parser_bench.rs (...)
▸ microsoft-webui-handler / handler_bench
     Running benches\handler_bench.rs (...)
▸ microsoft-webui-handler / bootstrap_state_bench
     Running benches\bootstrap_state_bench.rs (...)
▸ microsoft-webui-handler / streaming_hydration_bench

Baseline directories recorded under target/criterion/: 0 on main → 77 with this change. That is the claim this PR makes.

The run stays fail-fast: a declared benchmark that is broken stops the run rather than being silently skipped.

New: cargo xtask bench lazy-hydration

Exposes the existing Playwright/CDP offscreen hydration matrix in examples/integration/streaming-browser-bench as a first-class bench target, matching how the other integration benches are surfaced. Baseline flags map onto WEBUI_LAZY_HYDRATION_SAVE / WEBUI_LAZY_HYDRATION_COMPARE, which is exactly what tests/lazy_hydration_matrix.spec.ts already reads and what the bench README already documents.

The shared driver/fixtures also now assert the Toggle/Delete bindings are wired to the right handlers via __benchToggleCount / __benchRemoveCount, instead of only counting interactions in aggregate, and TODO_TEMPLATE is typed as TemplateMeta so fixture drift is caught at typecheck time.

Two pre-existing main failures this surfaces — neither introduced here

This PR touches no file under crates/ or packages/. Both failures below reproduce on pristine main with every change in this PR stashed.

1. microsoft-webui-handler / streaming_hydration_bench

thread 'main' panicked at crates\webui-handler\benches\streaming_hydration_bench.rs:175:9:
legacy render failed: Rendering invariant error: Shadow style hook `bench-island` does not match the active component root

The old --workspace path died at the demo-shell harness before ever reaching this bench, so fixing the routing un-masks it. Left declared and fail-fast deliberately; skipping it would hide a real failure.

The other 7 declared targets were each run individually to confirm they route correctly and accept their baseline arguments — all exit 0.

2. cargo xtask bench lazy-hydration[WebUI] componentStyles is required.

Verified identical on pristine main via pnpm test:lazy-hydration (same assertion, lazy_hydration_matrix.spec.ts:312). Root cause: packages/webui-framework/src/template.ts:274 calls registerComponentStyles(parsed.componentStyles) unconditionally in loadWebUIDataBlock(), and requireComponentStyles throws when the value is undefined. The bench fixture's #webui-data block legitimately carries only {templates, state} — it has no component CSS. git log -L blames that line to #429.

Not patched here: the fix belongs in the runtime, and emitting dummy componentStyles from the fixture would paper over the regression. Both failures are being routed to separate correctness PRs.

The new command itself is verified correct — it resolves the bench directory, spawns playwright test -c playwright.lazy-hydration.config.ts, and faithfully propagates the child's exit code; cargo xtask bench lazy-hydration and a direct pnpm test:lazy-hydration produce identical output.

Validation

cargo xtask check:

✔ license-headers   ✔ fmt   ✔ clippy   ✔ proto (drift check)
✔ deny              ✔ test  ✔ build    ✔ build (wasm)
✔ build (examples)  ✔ bench (validate)
✘ docs

The docs failure is PROJ-C013: Adapter module graph is incomplete or inconsistent, an esbuild projection error. Confirmed pristine-main-equivalent by stashing this PR and running pnpm build in docs/ — byte-identical failure. Documented, not fixed; out of scope here.

Includes 7 new xtask unit tests covering per-target routing, ---separator placement, extra args ordering ahead of baseline flags, and CRITERION_BENCHES integrity (non-empty, no duplicate pairs, every declared harness exists on disk).

Scope

Four files: xtask/src/main.rs, BENCHMARKS.md, and the two lazy-hydration bench lib files.

Deliberately excluded: benchmark_report / bench report, ssr-frameworks and examples/integration/ssr-framework-bench, docs snapshot generation and published benchmark JSON, xtask/src/build_examples.rs, workspace member changes, and any lockfile, dependency, or product runtime change.

This is a tooling-correctness change. It makes no product performance claim and reports no framework speedup.

`cargo xtask bench all` mapped every Criterion target onto a single
`cargo bench --workspace`. Cargo forwards everything after `--` to every
benchable target in the workspace, including the libtest unit-test
harnesses of libraries and binaries. Those harnesses reject Criterion's
baseline flags, so the run aborted on the first one it reached and no
baseline was ever recorded.

Walk an explicit CRITERION_BENCHES table instead and invoke each target
on its own, so baseline flags only ever reach a real Criterion binary.
Keep the run fail-fast so a broken declared benchmark stays visible.

Add `cargo xtask bench lazy-hydration` for the existing offscreen
hydration matrix, mapping baseline flags onto the WEBUI_LAZY_HYDRATION_*
env vars its spec already consumes, and tighten the shared lazy bench
driver/fixtures to assert Toggle/Delete wiring per handler.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 2605df0b-c217-4047-9413-49be787b1786
@mohamedmansour
Mohamed Mansour (mohamedmansour) merged commit ca4ed0a into main Aug 25, 2026
35 checks passed
@mohamedmansour
Mohamed Mansour (mohamedmansour) deleted the mohamedmansour-fix-criterion-per-target-bench branch August 25, 2026 02:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants