diff --git a/domains/performance/knowledge/metrics-pipeline-design.md b/domains/performance/knowledge/metrics-pipeline-design.md new file mode 100644 index 00000000..bd2e2d65 --- /dev/null +++ b/domains/performance/knowledge/metrics-pipeline-design.md @@ -0,0 +1,67 @@ +--- +name: metrics-pipeline-design +domain: performance +description: Four-layer metric pipeline architecture for E2E benchmarks, with domain-specific statistical bounds and split reporting paths. +--- + +# Metrics Pipeline Design + +Architecture for adding metric types to an E2E benchmark suite. Separates collection, running, statistics, and reporting into independent layers. + +## Architecture + +``` +Collector → Runner → Statistics → Reporter +``` + +| Layer | Responsibility | +|-------|----------------| +| **Collector** | Extract raw metric from browser/extension per iteration | +| **Runner** | Per-iteration capture + aggregation orchestration | +| **Statistics** | Domain-specific filtering, outlier detection, percentiles | +| **Reporter** | Per-run spans (per-iteration granularity) + aggregated structured logs (for dashboards) | + +Flow files call the collector and return snapshots alongside timers. No flow file does statistics or reporting. + +## Adding a New Metric Type + +1. **Create collector** — function returning typed snapshot with nullable fields for unobserved metrics +2. **Define types** — per-run snapshot, aggregated (reuse `TimerStatistics` for numeric fields), summary +3. **Add domain-specific bounds** — each numeric field gets `{ min, max, allowZero }` +4. **Wire into runner** — collect alongside timers, call aggregation +5. **Add reporter** — per-run spans with `setMeasurement`, aggregated summary as structured log + +## Domain-Specific Statistical Bounds + +Generic timer bounds (1ms–120s, zero=invalid) silently discard valid data from other domains. + +```typescript +// WRONG: CLS values (0–1) all rejected by min=1ms floor +const result = filterBySanityChecks(clsValues); // → empty array + +// RIGHT: per-metric bounds +const BOUNDS = { + inp: { min: 1, max: 30_000, allowZero: false }, // ms + lcp: { min: 1, max: 60_000, allowZero: false }, // ms + cls: { min: 0, max: 10, allowZero: true }, // unitless ratio +}; +``` + +**Rule:** When adding a new metric type, verify whether existing `filterBySanityChecks` assumptions (ms units, zero=invalid) hold. If not, define metric-specific bounds. + +`allowZero` is the critical distinction: CLS=0 means perfect stability (valid); timer=0ms means measurement error (invalid). + +## Split Reporting Path + +| Data | Mechanism | Rationale | +|------|-----------|-----------| +| Aggregated statistics (mean, p75, p95) | Structured log | Low cardinality, dashboard-friendly | +| Per-run snapshots | Sentry spans + `setMeasurement` | Preserves per-iteration granularity for inspection. Sentry is not a store for experimental raw data, so a Mann-Whitney U quality gate needs the per-run samples persisted elsewhere | + +`tracesSampleRate: 1.0` required in CI so all per-run spans are captured. + +## SDK Isolation Pattern + +When CI benchmark scripts run in Node but the extension uses a browser SDK (e.g. `@sentry/node` vs `@sentry/browser`): these never share a process. The package manager resolves separate versions per dependency tree. No compatibility issue — they are fully isolated under different lockfile entries. + +Risk: a shared module accidentally importing from the wrong SDK at bundle time. Mitigation: keep the CI SDK as a devDependency excluded from extension builds. diff --git a/domains/performance/knowledge/web-vitals-production-vs-benchmarks.md b/domains/performance/knowledge/web-vitals-production-vs-benchmarks.md new file mode 100644 index 00000000..b6bfdf63 --- /dev/null +++ b/domains/performance/knowledge/web-vitals-production-vs-benchmarks.md @@ -0,0 +1,21 @@ +--- +name: web-vitals-production-vs-benchmarks +domain: performance +description: Web Vitals need different collection in production (web-vitals lib) vs benchmarks (PerformanceObserver); no TBT in prod +--- + +# Web Vitals — Production vs Benchmarks + +Collection differs by environment due to timing constraints. + +## Production — `web-vitals` library +- Reports on `visibilitychange` / `pagehide` +- Handles browser quirks, bfcache, session windowing +- Attribution build shows which element/script caused the metric +- Metrics: **INP, LCP, CLS** (not TBT) + +**Why no TBT in production:** TBT is cumulative and unbounded — it grows indefinitely over an open-ended session. INP is per-interaction → meaningful for real users. TBT fits bounded flows (benchmarks), not sessions. + +## Benchmarks — direct `PerformanceObserver` +- Query on demand (not dependent on page hide) +- Fits an existing `collectMetrics()` pattern diff --git a/domains/performance/knowledge/web-vitals-runtime-metrics.md b/domains/performance/knowledge/web-vitals-runtime-metrics.md new file mode 100644 index 00000000..028f9bbc --- /dev/null +++ b/domains/performance/knowledge/web-vitals-runtime-metrics.md @@ -0,0 +1,22 @@ +--- +name: web-vitals-runtime-metrics +domain: performance +description: Core Web Vitals (INP, TBT) are runtime responsiveness metrics, not just page-load, with measurement limits as CI benchmark gates +--- + +# Web Vitals as Runtime Metrics + +Core Web Vitals (INP, TBT) measure **runtime responsiveness**, not just page load. For a browser extension this distinction is critical. + +- **Page load is less relevant** — the popup opens fast; there's no traditional navigation. +- **Runtime interactions matter** — every button click, form submit, confirmation. INP and TBT measure responsiveness during interactions. As CI benchmark gates both have measurement limits: INP is quantized to 8 ms before any harness code sees it, and TBT and every long-task metric read exactly 0 in 60 of 60 Firefox runs. + +## Orthogonal to distributed tracing + +| | Web Vitals | Distributed Tracing | +|---|---|---| +| Question | "How did the user perceive it?" | "Which controller caused it?" | +| Scope | user perception | operation attribution | +| Granularity | per-interaction aggregate | per-operation breakdown | + +Use both — perception (web vitals) + attribution (tracing) — not one instead of the other. diff --git a/domains/testing/knowledge/benchmark-statistical-hygiene.md b/domains/testing/knowledge/benchmark-statistical-hygiene.md new file mode 100644 index 00000000..50c53101 --- /dev/null +++ b/domains/testing/knowledge/benchmark-statistical-hygiene.md @@ -0,0 +1,44 @@ +--- +name: benchmark-statistical-hygiene +domain: testing +description: Three patterns for defensible A/B benchmark results: one primary analysis with per-round sensitivity, fix-vector isolation, and artifact sort-order trap. +--- + +# Benchmark Statistical Hygiene + +Three patterns that prevent the most common classes of invalid benchmark conclusions. + +## Pattern: One Primary Analysis, Per-Round Sensitivity + +Later benchmark rounds accumulate system noise (background load, memory pressure, I/O contention). Pooling all rounds blindly treats noisy late-session data equally with clean early-session data. + +**Instead:** Fix one primary analysis per metric before the runs, including any rule for dropping a round, such as a stability check that does not look at the treatment effect. Compute per-round statistics as sensitivity analyses and report them with explicit round attribution. They cannot change the verdict. Choosing the round with the cleanest signal after seeing the data is cherry-picking a favorable time range. + +``` +Round 1 (clean): metric X → treatment wins, p=0.04, d=-1.7 +Round 2 (moderate): metric X → treatment wins, p=0.08, d=-0.9 +Round 3 (noisy): metric X → not resolvable at this n, p=0.90, d=+0.04 + +Pooled (all): metric X → not resolvable at this n, p=0.50, d=-0.2 + +Report, with pooled as the primary analysis and no round-drop rule fixed in advance: + "X is not resolvable at this n (pooled p=0.50, d=-0.2). + Sensitivity: Round 1 alone p=0.04, d=-1.7. Round 3 alone d=+0.04." +``` + +## Pattern: Isolate the Fix Vector + +Design each benchmark flow to exercise the optimization's specific input vector as its primary signal source. Incidental coverage produces fragile results where signal-to-noise depends on how much of the measured duration is optimization-affected. + +| | Weak | Strong | +|-|------|--------| +| Design | End-to-end flow that incidentally triggers target once among many other operations | Rapid sequence of actions each triggering the target with minimal other overhead | +| Optimization signal | ~5% of measured duration | ~80% of measured duration | + +## Pattern: Artifact Sort-Order Trap + +Unpadded iteration numbers in filenames break lexicographic sorting: `iteration-1, iteration-10, iteration-2, ...` interleaves data from different rounds when processed in glob order. + +**Rule:** When processing sequentially-numbered artifacts, extract the embedded timestamp or numeric value for sorting. Never rely on string sort order when numbers cross digit boundaries. + +**Diagnosis:** If pipeline results look implausible (p-values that are too perfect, round-level stats that don't match spot checks), print the actual file ordering the pipeline used. Check for lexicographic interleaving at digit boundaries. Re-sort by extracted timestamp or zero-padded key. diff --git a/domains/testing/skills/benchmark-design/skill.md b/domains/testing/skills/benchmark-design/skill.md new file mode 100644 index 00000000..72e59ed6 --- /dev/null +++ b/domains/testing/skills/benchmark-design/skill.md @@ -0,0 +1,71 @@ +--- +maturity: experimental +name: benchmark-design +description: Design, run, and analyze E2E performance benchmarks — session hygiene, per-round reporting, artifact grouping +--- + +# Benchmark Design + +## When To Use + +- Writing a new E2E benchmark flow +- Interpreting or presenting benchmark results +- Adding new metrics to existing benchmarks +- Diagnosing unexpected benchmark results + +## Do Not Use When + +- Adding unit, integration, or correctness E2E tests +- Profiling a single user-reported slowdown (use `selector-antipattern-scan`) +- Writing micro-benchmarks outside the E2E harness + +## Workflow + +1. **Design the flow** — target ONE optimization vector per benchmark. Maximize ratio of optimization-affected time to total measured time. +2. **Run reference benchmarks first** in any session — session state degrades over time. +3. **Compute per-round statistics** before pooling. Check each round for stability (CV < 0.3 is a reasonable threshold). +4. **Group artifacts by timestamp**, not filename sort order. +5. **Report the primary analysis** fixed before the runs. Show per-round results as sensitivity, with explicit round attribution. + +## Flow Design by Optimization Type + +| Optimization | Primary cascade vector | Recommended flow | +|---|---|---| +| Selector memoization | State mutations | Multi-confirmation queue | +| Context memoization | Any state update | Account switching cycle | +| HOC stabilization | Route changes | Rapid route cycling (8+ transitions) | +| Dead code removal | Navigation | Return-to-home timer | + +## Session Hygiene + +System state degrades over long sessions — background load and memory pressure inflate variance and can **invert** treatment effects. + +- Run reference/critical benchmarks first +- If a late round contradicts clean earlier rounds, suspect session degradation before re-running the full suite + +## Artifact Grouping + +Filenames use `{test}-iteration-{N}-{ISO-timestamp}.json`. Unpadded N produces incorrect lexicographic sort. + +```javascript +// Extract seconds-since-midnight for round assignment +const match = filename.match(/T(\d{2})-(\d{2})-(\d{2})/); +const secondsOfDay = +match[1] * 3600 + +match[2] * 60 + +match[3]; +// Group by time range — never by filename position or array index +``` + +## Adding Metrics + +Extend `collectMetrics()` in `test/e2e/webdriver/driver.js` and register the metric key in `test/e2e/benchmarks/utils/constants.ts` → `ALL_METRICS`. + +- **Performance API metrics** (paint, navigation timing): collect directly inside `collectMetrics()` via `window.performance.getEntriesByType(...)`. +- **Long Task / TBT metrics**: already wired — `collectMetrics()` reads `window.stateHooks.getLongTaskMetricsWithTBT()`. Adding new long-task-derived metrics requires extending the `stateHooks` observer, not the driver. Where the browser delivers no `longtask` entries, the observer in `ui/helpers/utils/performance-observers.ts` leaves every count and duration at its initial 0, so a 0 reads the same whether no task ran long or no entry was observed. TBT and every long-task metric read exactly 0 in 60 of 60 recorded Firefox runs. + +## Common Pitfalls + +| Mistake | Correct Approach | +|---------|-----------------| +| Pool all rounds before checking per-round stats | Per-round first — late-session noise can invert the treatment effect | +| Sort artifacts by filename | Extract ISO timestamp; sort by numeric time value | +| Benchmark flow that exercises multiple vectors | One vector per flow — mixed flows produce ambiguous signal | +| Choose the reported round after seeing which one shows the effect | Report the primary analysis fixed before the runs. Per-round results are sensitivity, with round attribution |