fix(xtask): run Criterion benchmarks per target - #473
Merged
Mohamed Mansour (mohamedmansour) merged 1 commit intoAug 25, 2026
Merged
Conversation
`cargo xtask bench all` mapped every Criterion target onto a single `cargo bench --workspace`. Cargo forwards everything after `--` to every benchable target in the workspace, including the libtest unit-test harnesses of libraries and binaries. Those harnesses reject Criterion's baseline flags, so the run aborted on the first one it reached and no baseline was ever recorded. Walk an explicit CRITERION_BENCHES table instead and invoke each target on its own, so baseline flags only ever reach a real Criterion binary. Keep the run fail-fast so a broken declared benchmark stays visible. Add `cargo xtask bench lazy-hydration` for the existing offscreen hydration matrix, mapping baseline flags onto the WEBUI_LAZY_HYDRATION_* env vars its spec already consumes, and tighten the shared lazy bench driver/fixtures to assert Toggle/Delete wiring per handler. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2605df0b-c217-4047-9413-49be787b1786
Mohamed Mansour (mohamedmansour)
requested review from
Akrosh Gandhi (akroshg),
Jane Chu (janechu) and
mcritzjam
August 25, 2026 02:20
Jane Chu (janechu)
approved these changes
Aug 25, 2026
Mohamed Mansour (mohamedmansour)
deleted the
mohamedmansour-fix-criterion-per-target-bench
branch
August 25, 2026 02:56
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The failure on
maincargo xtask bench allmapped every Criterion target onto a singlecargo bench --workspace. Cargo forwards everything after--to every benchable target in the workspace, including the libtest unit-test harnesses of libraries and binaries. Those harnesses don't understand Criterion's baseline flags, so the run aborted on the first one it reached.Reproduced on today's tip (
e971c13a):Baselines recorded: 0. The before/after workflow documented in
BENCHMARKS.mdcould not work at all.The fix
bench allnow walks an explicitCRITERION_BENCHEStable and invokes each target on its own, so baseline flags only ever reach a real Criterion binary:Declared targets (11):
Each target is announced and run separately:
Baseline directories recorded under
target/criterion/: 0 onmain→ 77 with this change. That is the claim this PR makes.The run stays fail-fast: a declared benchmark that is broken stops the run rather than being silently skipped.
New:
cargo xtask bench lazy-hydrationExposes the existing Playwright/CDP offscreen hydration matrix in
examples/integration/streaming-browser-benchas a first-class bench target, matching how the other integration benches are surfaced. Baseline flags map ontoWEBUI_LAZY_HYDRATION_SAVE/WEBUI_LAZY_HYDRATION_COMPARE, which is exactly whattests/lazy_hydration_matrix.spec.tsalready reads and what the bench README already documents.The shared driver/fixtures also now assert the Toggle/Delete bindings are wired to the right handlers via
__benchToggleCount/__benchRemoveCount, instead of only counting interactions in aggregate, andTODO_TEMPLATEis typed asTemplateMetaso fixture drift is caught at typecheck time.Two pre-existing
mainfailures this surfaces — neither introduced hereThis PR touches no file under
crates/orpackages/. Both failures below reproduce on pristinemainwith every change in this PR stashed.1.
microsoft-webui-handler / streaming_hydration_benchThe old
--workspacepath died at thedemo-shellharness before ever reaching this bench, so fixing the routing un-masks it. Left declared and fail-fast deliberately; skipping it would hide a real failure.The other 7 declared targets were each run individually to confirm they route correctly and accept their baseline arguments — all exit 0.
2.
cargo xtask bench lazy-hydration→[WebUI] componentStyles is required.Verified identical on pristine
mainviapnpm test:lazy-hydration(same assertion,lazy_hydration_matrix.spec.ts:312). Root cause:packages/webui-framework/src/template.ts:274callsregisterComponentStyles(parsed.componentStyles)unconditionally inloadWebUIDataBlock(), andrequireComponentStylesthrows when the value isundefined. The bench fixture's#webui-datablock legitimately carries only{templates, state}— it has no component CSS.git log -Lblames that line to #429.Not patched here: the fix belongs in the runtime, and emitting dummy
componentStylesfrom the fixture would paper over the regression. Both failures are being routed to separate correctness PRs.The new command itself is verified correct — it resolves the bench directory, spawns
playwright test -c playwright.lazy-hydration.config.ts, and faithfully propagates the child's exit code;cargo xtask bench lazy-hydrationand a directpnpm test:lazy-hydrationproduce identical output.Validation
cargo xtask check:The
docsfailure isPROJ-C013: Adapter module graph is incomplete or inconsistent, an esbuild projection error. Confirmed pristine-main-equivalent by stashing this PR and runningpnpm buildindocs/— byte-identical failure. Documented, not fixed; out of scope here.Includes 7 new
xtaskunit tests covering per-target routing,---separator placement, extra args ordering ahead of baseline flags, andCRITERION_BENCHESintegrity (non-empty, no duplicate pairs, every declared harness exists on disk).Scope
Four files:
xtask/src/main.rs,BENCHMARKS.md, and the two lazy-hydration bench lib files.Deliberately excluded:
benchmark_report/bench report,ssr-frameworksandexamples/integration/ssr-framework-bench, docs snapshot generation and published benchmark JSON,xtask/src/build_examples.rs, workspace member changes, and any lockfile, dependency, or product runtime change.This is a tooling-correctness change. It makes no product performance claim and reports no framework speedup.