Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 42 additions & 0 deletions BENCHMARKS.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ how to compare results.
| `cargo xtask bench streaming-resource` | example | ~30 s | exact alloc count + bytes + getrusage CPU + RSS | proving zero-alloc claims; allocation regression hunting |
| `cargo xtask bench streaming-e2e-ttfb` | example | ~10 s | HTTP-level TTFB / TTLB through actix | confirming wire-level streaming win |
| `cargo xtask bench streaming-browser` | Playwright | ~30 s | real Chromium TTFB / FCP / LCP / DCL / load | proving user-perceived paint improvement |
| `cargo xtask bench lazy-hydration` | Playwright + CDP | ~1 min | hydration, heap, rendering, and trace metrics at 10/100/1000 rows | validating offscreen work reduction |
| `cargo xtask bench full` (= `streaming-all`) | suite | ~3 min | runs all four streaming-related benches in sequence | full streaming evidence pack for a PR |

## The before/after workflow
Expand All @@ -41,13 +42,35 @@ Baselines are stored at `target/bench-baselines/`:
* `streaming-resource-<name>.json` — alloc + RSS + CPU table
* `e2e-ttfb-<name>.json` — HTTP TTFB/TTLB table
* `browser-<name>.json` — browser metrics table
* `browser-lazy-hydration-<name>.json` — offscreen rendering/hydration matrix
* `node-addon-<name>.json` — Node/V8/N-API latency table
* `target/criterion/<bench>/<name>` — criterion's native baseline
directory tree

The compare phase prints a Δ%-table for every row. Negative Δ% =
improvement; positive = regression.

### Criterion baselines are recorded per target

`cargo xtask bench all` invokes each Criterion target separately —
`cargo bench -p <package> --bench <target> -- --save-baseline NAME` — rather
than a single `cargo bench --workspace`. A workspace-wide run forwards
Criterion's flags to *every* benchable target, including the libtest
unit-test harnesses of libraries and binaries, which abort with
`error: Unrecognized option: 'save-baseline'` before a single baseline is
written.

Because each target runs on its own, criterion writes one baseline directory
per bench:

```bash
cargo xtask bench all --save-baseline before
# target/criterion/<group>/<bench-id>/before/ for every Criterion target
```

The target list lives in `CRITERION_BENCHES` in `xtask/src/main.rs`. Add new
`benches/*.rs` harnesses there so `bench all` and its baselines pick them up.

### Threshold guidance

| Source | Treat as noise | Treat as signal |
Expand Down Expand Up @@ -155,6 +178,25 @@ the workload. The first-callback metric is in-process; it is not HTTP
TTFB. The runner verifies JSON-string, object-state, and streamed output are
byte-identical before collecting samples.

### `lazy-hydration` (offscreen hydration matrix)

`cargo xtask bench lazy-hydration` drives the Playwright + CDP lazy-hydration
matrix in `examples/integration/streaming-browser-bench` across the `eager`,
`lazy-hydrate`, and `lazy-render` modes at 10/100/1000 rows. It reports bundle
init cost, hydration CPU, hydrated-root and listener counts, JS heap, rendering
trace metrics, and validated interaction counts.

```bash
cargo xtask bench lazy-hydration --save-baseline before
# … change …
cargo xtask bench lazy-hydration --baseline before
```

The command maps its baseline flags onto the `WEBUI_LAZY_HYDRATION_SAVE` and
`WEBUI_LAZY_HYDRATION_COMPARE` env vars consumed by the spec, and writes
`target/bench-baselines/browser-lazy-hydration-<name>.json`. See the bench
README for run-count, mode-subset, and framework-source overrides.

## Recommended PR workflow

For any change touching `crates/webui/src/streaming.rs` or its
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,8 @@ export async function runLazyHydration(mode: LazyMode): Promise<LazyRunMetrics>
__benchListenerCount?: number;
__benchPeakHeap?: number;
__benchInteractionCount?: number;
__benchRemoveCount?: number;
__benchToggleCount?: number;
__defineBenchTodo(): void;
};

Expand Down Expand Up @@ -99,6 +101,8 @@ export async function runLazyHydration(mode: LazyMode): Promise<LazyRunMetrics>
win.__benchHydratedCount = 0;
win.__benchListenerCount = 0;
win.__benchInteractionCount = 0;
win.__benchRemoveCount = 0;
win.__benchToggleCount = 0;
const baseHeap = heapSize();
win.__benchPeakHeap = baseHeap ?? 0;
const roots = document.getElementsByTagName('bench-todo-item');
Expand Down Expand Up @@ -150,9 +154,10 @@ export async function runLazyHydration(mode: LazyMode): Promise<LazyRunMetrics>
}

const firstButton = roots[0]?.querySelector('button');
const firstDeleteButton = roots[0]?.querySelectorAll('button')[1];
const lastRoot = roots[roots.length - 1] as HTMLElement | undefined;
const lastButton = lastRoot?.querySelector('button');
if (!firstButton || !lastRoot || !lastButton) {
if (!firstButton || !firstDeleteButton || !lastRoot || !lastButton) {
throw new Error('lazy hydration benchmark roots are incomplete');
}
const dormantContentSkipped = typeof lastButton.checkVisibility === 'function'
Expand All @@ -168,6 +173,14 @@ export async function runLazyHydration(mode: LazyMode): Promise<LazyRunMetrics>
lastButton.click();
const dormantInteractionMs = performance.now() - dormantStarted;
const afterDormant = win.__benchHydratedCount ?? 0;
const interactionCount = win.__benchInteractionCount ?? 0;
firstDeleteButton.click();
if (
win.__benchToggleCount !== 2
|| win.__benchRemoveCount !== 1
) {
throw new Error('lazy hydration benchmark wired Toggle/Delete incorrectly');
}

return {
bundleInitMs: win.__benchBundleInitMs ?? 0,
Expand Down Expand Up @@ -196,7 +209,7 @@ export async function runLazyHydration(mode: LazyMode): Promise<LazyRunMetrics>
dormantInteractionMs,
dormantInteractionHydrated: afterDormant > beforeDormant,
dormantContentSkipped,
interactionCount: win.__benchInteractionCount ?? 0,
interactionCount,
liveRootCount: roots.length,
};
}
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ import { build } from 'esbuild';
import { gzipSync } from 'node:zlib';
import { dirname, resolve } from 'node:path';
import { fileURLToPath } from 'node:url';
import type { TemplateMeta } from '../../../../../packages/webui-framework/src/template-types.js';

const here = dirname(fileURLToPath(import.meta.url));
const FRAMEWORK_SRC = process.env.WEBUI_LAZY_HYDRATION_FRAMEWORK_SRC
Expand Down Expand Up @@ -32,7 +33,7 @@ export const ITEM_COUNTS = [10, 100, 1_000] as const;
*/
export type LazyBenchMode = 'eager' | 'lazy-hydrate' | 'lazy-render';

const TODO_TEMPLATE = {
const TODO_TEMPLATE: TemplateMeta = {
h: '<article><input type="checkbox"><span></span><small></small><strong></strong><time></time><button type="button">Toggle</button><button type="button">Delete</button></article>',
tr: ['title', 'description', 'priority', 'due'],
tx: [
Expand All @@ -47,7 +48,7 @@ const TODO_TEMPLATE = {
['remove', [], 8],
]],
],
} as const;
};

function entrySource(mode: LazyBenchMode): string {
const optionalImport = mode === 'eager'
Expand Down Expand Up @@ -86,10 +87,12 @@ class BenchTodoItem extends WebUIElement {

toggle() {
window.__benchInteractionCount = (window.__benchInteractionCount || 0) + 1;
window.__benchToggleCount = (window.__benchToggleCount || 0) + 1;
}

remove() {
window.__benchInteractionCount = (window.__benchInteractionCount || 0) + 1;
window.__benchRemoveCount = (window.__benchRemoveCount || 0) + 1;
}
}

Expand Down
Loading
Loading