feat: add E2E benchmark design and statistics skills

domains/performance/knowledge/metrics-pipeline-design.md

@@ -0,0 +1,67 @@
+---
+name: metrics-pipeline-design
+domain: performance
+description: Four-layer metric pipeline architecture for E2E benchmarks, with domain-specific statistical bounds and split reporting paths.
+---
+
+# Metrics Pipeline Design
+
+Architecture for adding metric types to an E2E benchmark suite. Separates collection, running, statistics, and reporting into independent layers.
+
+## Architecture
+
+```
+Collector → Runner → Statistics → Reporter
+```
+
+| Layer | Responsibility |
+|-------|----------------|
+| **Collector** | Extract raw metric from browser/extension per iteration |
+| **Runner** | Per-iteration capture + aggregation orchestration |
+| **Statistics** | Domain-specific filtering, outlier detection, percentiles |
+| **Reporter** | Per-run spans (per-iteration granularity) + aggregated structured logs (for dashboards) |
+
+Flow files call the collector and return snapshots alongside timers. No flow file does statistics or reporting.
+
+## Adding a New Metric Type
+
+1. **Create collector** — function returning typed snapshot with nullable fields for unobserved metrics
+2. **Define types** — per-run snapshot, aggregated (reuse `TimerStatistics` for numeric fields), summary
+3. **Add domain-specific bounds** — each numeric field gets `{ min, max, allowZero }`
+4. **Wire into runner** — collect alongside timers, call aggregation
+5. **Add reporter** — per-run spans with `setMeasurement`, aggregated summary as structured log
+
+## Domain-Specific Statistical Bounds
+
+Generic timer bounds (1ms–120s, zero=invalid) silently discard valid data from other domains.
+
+```typescript
+// WRONG: CLS values (0–1) all rejected by min=1ms floor
+const result = filterBySanityChecks(clsValues); // → empty array
+
+// RIGHT: per-metric bounds
+const BOUNDS = {
+  inp: { min: 1, max: 30_000, allowZero: false },  // ms
+  lcp: { min: 1, max: 60_000, allowZero: false },  // ms
+  cls: { min: 0, max: 10,     allowZero: true  },  // unitless ratio
+};
+```
+
+**Rule:** When adding a new metric type, verify whether existing `filterBySanityChecks` assumptions (ms units, zero=invalid) hold. If not, define metric-specific bounds.
+
+`allowZero` is the critical distinction: CLS=0 means perfect stability (valid); timer=0ms means measurement error (invalid).
+
+## Split Reporting Path
+
+| Data | Mechanism | Rationale |
+|------|-----------|-----------|
+| Aggregated statistics (mean, p75, p95) | Structured log | Low cardinality, dashboard-friendly |
+| Per-run snapshots | Sentry spans + `setMeasurement` | Preserves per-iteration granularity for inspection. Sentry is not a store for experimental raw data, so a Mann-Whitney U quality gate needs the per-run samples persisted elsewhere |
+
+`tracesSampleRate: 1.0` required in CI so all per-run spans are captured.
+
+## SDK Isolation Pattern
+
+When CI benchmark scripts run in Node but the extension uses a browser SDK (e.g. `@sentry/node` vs `@sentry/browser`): these never share a process. The package manager resolves separate versions per dependency tree. No compatibility issue — they are fully isolated under different lockfile entries.
+
+Risk: a shared module accidentally importing from the wrong SDK at bundle time. Mitigation: keep the CI SDK as a devDependency excluded from extension builds.

domains/performance/knowledge/web-vitals-production-vs-benchmarks.md

@@ -0,0 +1,21 @@
+---
+name: web-vitals-production-vs-benchmarks
+domain: performance
+description: Web Vitals need different collection in production (web-vitals lib) vs benchmarks (PerformanceObserver); no TBT in prod
+---
+
+# Web Vitals — Production vs Benchmarks
+
+Collection differs by environment due to timing constraints.
+
+## Production — `web-vitals` library
+- Reports on `visibilitychange` / `pagehide`
+- Handles browser quirks, bfcache, session windowing
+- Attribution build shows which element/script caused the metric
+- Metrics: **INP, LCP, CLS** (not TBT)
+
+**Why no TBT in production:** TBT is cumulative and unbounded — it grows indefinitely over an open-ended session. INP is per-interaction → meaningful for real users. TBT fits bounded flows (benchmarks), not sessions.
+
+## Benchmarks — direct `PerformanceObserver`
+- Query on demand (not dependent on page hide)
+- Fits an existing `collectMetrics()` pattern

domains/performance/knowledge/web-vitals-runtime-metrics.md

@@ -0,0 +1,22 @@
+---
+name: web-vitals-runtime-metrics
+domain: performance
+description: Core Web Vitals (INP, TBT) are runtime responsiveness metrics, not just page-load, with measurement limits as CI benchmark gates
+---
+
+# Web Vitals as Runtime Metrics
+
+Core Web Vitals (INP, TBT) measure **runtime responsiveness**, not just page load. For a browser extension this distinction is critical.
+
+- **Page load is less relevant** — the popup opens fast; there's no traditional navigation.
+- **Runtime interactions matter** — every button click, form submit, confirmation. INP and TBT measure responsiveness during interactions. As CI benchmark gates both have measurement limits: INP is quantized to 8 ms before any harness code sees it, and TBT and every long-task metric read exactly 0 in 60 of 60 Firefox runs.
+
+## Orthogonal to distributed tracing
+
+| | Web Vitals | Distributed Tracing |
+|---|---|---|
+| Question | "How did the user perceive it?" | "Which controller caused it?" |
+| Scope | user perception | operation attribution |
+| Granularity | per-interaction aggregate | per-operation breakdown |
+
+Use both — perception (web vitals) + attribution (tracing) — not one instead of the other.

domains/testing/knowledge/benchmark-statistical-hygiene.md

@@ -0,0 +1,44 @@
+---
+name: benchmark-statistical-hygiene
+domain: testing
+description: Three patterns for defensible A/B benchmark results: one primary analysis with per-round sensitivity, fix-vector isolation, and artifact sort-order trap.
+---
+
+# Benchmark Statistical Hygiene
+
+Three patterns that prevent the most common classes of invalid benchmark conclusions.
+
+## Pattern: One Primary Analysis, Per-Round Sensitivity
+
+Later benchmark rounds accumulate system noise (background load, memory pressure, I/O contention). Pooling all rounds blindly treats noisy late-session data equally with clean early-session data.
+
+**Instead:** Fix one primary analysis per metric before the runs, including any rule for dropping a round, such as a stability check that does not look at the treatment effect. Compute per-round statistics as sensitivity analyses and report them with explicit round attribution. They cannot change the verdict. Choosing the round with the cleanest signal after seeing the data is cherry-picking a favorable time range.
+
+```
+Round 1 (clean):    metric X → treatment wins,           p=0.04, d=-1.7
+Round 2 (moderate): metric X → treatment wins,           p=0.08, d=-0.9
+Round 3 (noisy):    metric X → not resolvable at this n, p=0.90, d=+0.04
+
+Pooled (all):       metric X → not resolvable at this n, p=0.50, d=-0.2
+
+Report, with pooled as the primary analysis and no round-drop rule fixed in advance:
+  "X is not resolvable at this n (pooled p=0.50, d=-0.2).
+   Sensitivity: Round 1 alone p=0.04, d=-1.7. Round 3 alone d=+0.04."
+```
+
+## Pattern: Isolate the Fix Vector
+
+Design each benchmark flow to exercise the optimization's specific input vector as its primary signal source. Incidental coverage produces fragile results where signal-to-noise depends on how much of the measured duration is optimization-affected.
+
+| | Weak | Strong |
+|-|------|--------|
+| Design | End-to-end flow that incidentally triggers target once among many other operations | Rapid sequence of actions each triggering the target with minimal other overhead |
+| Optimization signal | ~5% of measured duration | ~80% of measured duration |
+
+## Pattern: Artifact Sort-Order Trap
+
+Unpadded iteration numbers in filenames break lexicographic sorting: `iteration-1, iteration-10, iteration-2, ...` interleaves data from different rounds when processed in glob order.
+
+**Rule:** When processing sequentially-numbered artifacts, extract the embedded timestamp or numeric value for sorting. Never rely on string sort order when numbers cross digit boundaries.
+
+**Diagnosis:** If pipeline results look implausible (p-values that are too perfect, round-level stats that don't match spot checks), print the actual file ordering the pipeline used. Check for lexicographic interleaving at digit boundaries. Re-sort by extracted timestamp or zero-padded key.

domains/testing/skills/benchmark-design/skill.md

@@ -0,0 +1,71 @@
+---
+maturity: experimental
+name: benchmark-design
+description: Design, run, and analyze E2E performance benchmarks — session hygiene, per-round reporting, artifact grouping
+---
+
+# Benchmark Design
+
+## When To Use
+
+- Writing a new E2E benchmark flow
+- Interpreting or presenting benchmark results
+- Adding new metrics to existing benchmarks
+- Diagnosing unexpected benchmark results
+
+## Do Not Use When
+
+- Adding unit, integration, or correctness E2E tests
+- Profiling a single user-reported slowdown (use `selector-antipattern-scan`)
+- Writing micro-benchmarks outside the E2E harness
+
+## Workflow
+
+1. **Design the flow** — target ONE optimization vector per benchmark. Maximize ratio of optimization-affected time to total measured time.
+2. **Run reference benchmarks first** in any session — session state degrades over time.
+3. **Compute per-round statistics** before pooling. Check each round for stability (CV < 0.3 is a reasonable threshold).
+4. **Group artifacts by timestamp**, not filename sort order.
+5. **Report the primary analysis** fixed before the runs. Show per-round results as sensitivity, with explicit round attribution.
+
+## Flow Design by Optimization Type
+
+| Optimization | Primary cascade vector | Recommended flow |
+|---|---|---|
+| Selector memoization | State mutations | Multi-confirmation queue |
+| Context memoization | Any state update | Account switching cycle |
+| HOC stabilization | Route changes | Rapid route cycling (8+ transitions) |
+| Dead code removal | Navigation | Return-to-home timer |
+
+## Session Hygiene
+
+System state degrades over long sessions — background load and memory pressure inflate variance and can **invert** treatment effects.
+
+- Run reference/critical benchmarks first
+- If a late round contradicts clean earlier rounds, suspect session degradation before re-running the full suite
+
+## Artifact Grouping
+
+Filenames use `{test}-iteration-{N}-{ISO-timestamp}.json`. Unpadded N produces incorrect lexicographic sort.
+
+```javascript
+// Extract seconds-since-midnight for round assignment
+const match = filename.match(/T(\d{2})-(\d{2})-(\d{2})/);
+const secondsOfDay = +match[1] * 3600 + +match[2] * 60 + +match[3];
+// Group by time range — never by filename position or array index
+```
+
+## Adding Metrics
+
+Extend `collectMetrics()` in `test/e2e/webdriver/driver.js` and register the metric key in `test/e2e/benchmarks/utils/constants.ts` → `ALL_METRICS`.
+
+- **Performance API metrics** (paint, navigation timing): collect directly inside `collectMetrics()` via `window.performance.getEntriesByType(...)`.
+- **Long Task / TBT metrics**: already wired — `collectMetrics()` reads `window.stateHooks.getLongTaskMetricsWithTBT()`. Adding new long-task-derived metrics requires extending the `stateHooks` observer, not the driver. Where the browser delivers no `longtask` entries, the observer in `ui/helpers/utils/performance-observers.ts` leaves every count and duration at its initial 0, so a 0 reads the same whether no task ran long or no entry was observed. TBT and every long-task metric read exactly 0 in 60 of 60 recorded Firefox runs.
+
+## Common Pitfalls
+
+| Mistake | Correct Approach |
+|---------|-----------------|
+| Pool all rounds before checking per-round stats | Per-round first — late-session noise can invert the treatment effect |
+| Sort artifacts by filename | Extract ISO timestamp; sort by numeric time value |
+| Benchmark flow that exercises multiple vectors | One vector per flow — mixed flows produce ambiguous signal |
+| Choose the reported round after seeing which one shows the effect | Report the primary analysis fixed before the runs. Per-round results are sensitivity, with round attribution |