From c87398232bcad56242d1cb0909780618e09ef13c Mon Sep 17 00:00:00 2001
From: Jongsun Suh <jongsun.suh@icloud.com>
Date: Tue, 1 Sep 2026 10:02:50 -0400
Subject: [PATCH 1/5] Add performance measurement skills: profiling, render
 deltas, benchmarks
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

`extension-profiling`, `react-render-delta`, `data-analysis` and
`benchmark-design`, split from #43 so the audit half reviews separately.

The existing `performance` skill is mobile-scoped and advisory — it says what
to change. Nothing in the repo says how to prove a change worked, and the
extension has no profiling skill at all.
---
 .../knowledge/metrics-pipeline-design.md      |  67 +++++++
 .../web-vitals-attribution-import.md          |  19 ++
 .../web-vitals-production-vs-benchmarks.md    |  21 +++
 .../knowledge/web-vitals-runtime-metrics.md   |  22 +++
 .../performance/skills/data-analysis/skill.md | 164 ++++++++++++++++++
 .../skills/extension-profiling/skill.md       |  65 +++++++
 .../skills/react-render-delta/skill.md        | 112 ++++++++++++
 .../benchmark-statistical-hygiene.md          |  45 +++++
 .../testing/skills/benchmark-design/skill.md  |  71 ++++++++
 9 files changed, 586 insertions(+)
 create mode 100644 domains/performance/knowledge/metrics-pipeline-design.md
 create mode 100644 domains/performance/knowledge/web-vitals-attribution-import.md
 create mode 100644 domains/performance/knowledge/web-vitals-production-vs-benchmarks.md
 create mode 100644 domains/performance/knowledge/web-vitals-runtime-metrics.md
 create mode 100644 domains/performance/skills/data-analysis/skill.md
 create mode 100644 domains/performance/skills/extension-profiling/skill.md
 create mode 100644 domains/performance/skills/react-render-delta/skill.md
 create mode 100644 domains/testing/knowledge/benchmark-statistical-hygiene.md
 create mode 100644 domains/testing/skills/benchmark-design/skill.md

diff --git a/domains/performance/knowledge/metrics-pipeline-design.md b/domains/performance/knowledge/metrics-pipeline-design.md
new file mode 100644
index 00000000..03821ef5
--- /dev/null
+++ b/domains/performance/knowledge/metrics-pipeline-design.md
@@ -0,0 +1,67 @@
+---
+name: metrics-pipeline-design
+domain: performance
+description: Four-layer metric pipeline architecture for E2E benchmarks, with domain-specific statistical bounds and split reporting paths.
+---
+
+# Metrics Pipeline Design
+
+Architecture for adding metric types to an E2E benchmark suite. Separates collection, running, statistics, and reporting into independent layers.
+
+## Architecture
+
+```
+Collector → Runner → Statistics → Reporter
+```
+
+| Layer | Responsibility |
+|-------|----------------|
+| **Collector** | Extract raw metric from browser/extension per iteration |
+| **Runner** | Per-iteration capture + aggregation orchestration |
+| **Statistics** | Domain-specific filtering, outlier detection, percentiles |
+| **Reporter** | Per-run spans (for quality gate comparison) + aggregated structured logs (for dashboards) |
+
+Flow files call the collector and return snapshots alongside timers. No flow file does statistics or reporting.
+
+## Adding a New Metric Type
+
+1. **Create collector** — function returning typed snapshot with nullable fields for unobserved metrics
+2. **Define types** — per-run snapshot, aggregated (reuse `TimerStatistics` for numeric fields), summary
+3. **Add domain-specific bounds** — each numeric field gets `{ min, max, allowZero }`
+4. **Wire into runner** — collect alongside timers, call aggregation
+5. **Add reporter** — per-run spans with `setMeasurement`, aggregated summary as structured log
+
+## Domain-Specific Statistical Bounds
+
+Generic timer bounds (1ms–120s, zero=invalid) silently discard valid data from other domains.
+
+```typescript
+// WRONG: CLS values (0–1) all rejected by min=1ms floor
+const result = filterBySanityChecks(clsValues); // → empty array
+
+// RIGHT: per-metric bounds
+const BOUNDS = {
+  inp: { min: 1, max: 30_000, allowZero: false },  // ms
+  lcp: { min: 1, max: 60_000, allowZero: false },  // ms
+  cls: { min: 0, max: 10,     allowZero: true  },  // unitless ratio
+};
+```
+
+**Rule:** When adding a new metric type, verify whether existing `filterBySanityChecks` assumptions (ms units, zero=invalid) hold. If not, define metric-specific bounds.
+
+`allowZero` is the critical distinction: CLS=0 means perfect stability (valid); timer=0ms means measurement error (invalid).
+
+## Split Reporting Path
+
+| Data | Mechanism | Rationale |
+|------|-----------|-----------|
+| Aggregated statistics (mean, p75, p95) | Structured log | Low cardinality, dashboard-friendly |
+| Per-run snapshots | Sentry spans + `setMeasurement` | Preserves granularity, enables quality gate comparison via Mann-Whitney U |
+
+`tracesSampleRate: 1.0` required in CI so all per-run spans are captured.
+
+## SDK Isolation Pattern
+
+When CI benchmark scripts run in Node but the extension uses a browser SDK (e.g. `@sentry/node` vs `@sentry/browser`): these never share a process. The package manager resolves separate versions per dependency tree. No compatibility issue — they are fully isolated under different lockfile entries.
+
+Risk: a shared module accidentally importing from the wrong SDK at bundle time. Mitigation: keep the CI SDK as a devDependency excluded from extension builds.
diff --git a/domains/performance/knowledge/web-vitals-attribution-import.md b/domains/performance/knowledge/web-vitals-attribution-import.md
new file mode 100644
index 00000000..98c0067b
--- /dev/null
+++ b/domains/performance/knowledge/web-vitals-attribution-import.md
@@ -0,0 +1,19 @@
+---
+name: web-vitals-attribution-import
+domain: performance
+description: web-vitals/attribution is a module import path, not a separate package — no meaningful bundle cost, gives the symptom→cause link
+---
+
+# Web Vitals Attribution Import
+
+`web-vitals/attribution` is a **module import path**, not a separate package. The attribution build:
+- Provides which script/element caused each metric
+- Does **not** meaningfully increase production bundle size (tree-shaking applies)
+
+Don't skip it for "bundle size" reasons — that's a misread.
+
+## Why it matters
+Attribution is the symptom→cause link:
+- INP spike of 500ms
+- Attribution: `eventTarget: '#confirm-swap-button'`, `eventType: 'click'`
+- Combined with tracing → identifies the controller that blocked
diff --git a/domains/performance/knowledge/web-vitals-production-vs-benchmarks.md b/domains/performance/knowledge/web-vitals-production-vs-benchmarks.md
new file mode 100644
index 00000000..b6bfdf63
--- /dev/null
+++ b/domains/performance/knowledge/web-vitals-production-vs-benchmarks.md
@@ -0,0 +1,21 @@
+---
+name: web-vitals-production-vs-benchmarks
+domain: performance
+description: Web Vitals need different collection in production (web-vitals lib) vs benchmarks (PerformanceObserver); no TBT in prod
+---
+
+# Web Vitals — Production vs Benchmarks
+
+Collection differs by environment due to timing constraints.
+
+## Production — `web-vitals` library
+- Reports on `visibilitychange` / `pagehide`
+- Handles browser quirks, bfcache, session windowing
+- Attribution build shows which element/script caused the metric
+- Metrics: **INP, LCP, CLS** (not TBT)
+
+**Why no TBT in production:** TBT is cumulative and unbounded — it grows indefinitely over an open-ended session. INP is per-interaction → meaningful for real users. TBT fits bounded flows (benchmarks), not sessions.
+
+## Benchmarks — direct `PerformanceObserver`
+- Query on demand (not dependent on page hide)
+- Fits an existing `collectMetrics()` pattern
diff --git a/domains/performance/knowledge/web-vitals-runtime-metrics.md b/domains/performance/knowledge/web-vitals-runtime-metrics.md
new file mode 100644
index 00000000..e05d7c55
--- /dev/null
+++ b/domains/performance/knowledge/web-vitals-runtime-metrics.md
@@ -0,0 +1,22 @@
+---
+name: web-vitals-runtime-metrics
+domain: performance
+description: Core Web Vitals (INP, TBT) are runtime responsiveness metrics, not just page-load — high-value for extension UX gates
+---
+
+# Web Vitals as Runtime Metrics
+
+Core Web Vitals (INP, TBT) measure **runtime responsiveness**, not just page load. For a browser extension this distinction is critical.
+
+- **Page load is less relevant** — the popup opens fast; there's no traditional navigation.
+- **Runtime interactions matter** — every button click, form submit, confirmation. INP and TBT measure responsiveness during interactions → high-value for extension UX quality gates.
+
+## Orthogonal to distributed tracing
+
+| | Web Vitals | Distributed Tracing |
+|---|---|---|
+| Question | "How did the user perceive it?" | "Which controller caused it?" |
+| Scope | user perception | operation attribution |
+| Granularity | per-interaction aggregate | per-operation breakdown |
+
+Use both — perception (web vitals) + attribution (tracing) — not one instead of the other.
diff --git a/domains/performance/skills/data-analysis/skill.md b/domains/performance/skills/data-analysis/skill.md
new file mode 100644
index 00000000..4566045d
--- /dev/null
+++ b/domains/performance/skills/data-analysis/skill.md
@@ -0,0 +1,164 @@
+---
+maturity: experimental
+name: data-analysis
+description: Structured approach for analyzing metrics, attributing changes, and communicating findings — five phases (collection → filtering → curation → questioning → synthesis), confidence assignment, audience-appropriate artifacts
+---
+
+# Data Analysis Skill
+
+Structured approach for analyzing metrics, attributing changes, and communicating findings.
+
+---
+
+## When to Use
+
+- Performance analysis from production metrics
+- Attribution of improvements/regressions to code changes
+- Creating executive summaries or stakeholder communications
+- Any analysis requiring correlation of changes to measured outcomes
+
+---
+
+## Quick Reference
+
+### Five Phases
+
+```
+Collection → Filtering → Curation → Questioning → Synthesis
+```
+
+| Phase       | Key Question                | Output                              |
+| ----------- | --------------------------- | ----------------------------------- |
+| Collection  | What are we measuring?      | Baseline, scope, change list        |
+| Filtering   | What's signal vs. noise?    | Categorized changes with confidence |
+| Curation    | What correlates with what?  | Attribution table                   |
+| Questioning | Do we KNOW or BELIEVE this? | Validated claims with caveats       |
+| Synthesis   | Who needs to know what?     | Audience-appropriate artifacts      |
+
+### Confidence Assignment
+
+| Level      | Use When                                                         |
+| ---------- | ---------------------------------------------------------------- |
+| **High**   | Clear mechanism + timing alignment + targets measured population |
+| **Medium** | Plausible mechanism but confounded by other changes              |
+| **Low**    | Speculative or enabling-only                                     |
+
+### Attribution Table Template
+
+| Change        | Evidence    | Release   | Metric            | Confidence   | Notes                 |
+| ------------- | ----------- | --------- | ----------------- | ------------ | --------------------- |
+| [Description] | [PR/commit] | [version] | [affected metric] | High/Med/Low | [mechanism or caveat] |
+
+---
+
+## Process
+
+### 1. Collection
+
+```markdown
+**Metrics:** [What are you measuring?]
+**Population:** [Who? All users, p75, specific cohort?]
+**Period:** [Measurement window - release tags or dates]
+**Source:** [APM, logs, synthetic benchmarks?]
+**Baseline:** [Starting values with methodology]
+```
+
+Enumerate ALL changes in scope:
+
+- Code changes (PRs, commits)
+- Config changes
+- External factors (traffic, user growth, infrastructure)
+
+### 2. Filtering
+
+Categorize each change:
+
+- **Direct:** Clear causal path to measured metric
+- **Indirect:** Enabling infrastructure (value materializes later)
+- **Unknown:** In scope but mechanism unclear
+- **Noise:** Unlikely to affect measured metrics
+
+### 3. Curation
+
+Build attribution table:
+
+1. Map changes to metric movements by release
+2. Note co-landed changes (shared attribution)
+3. Flag anomalies (improvement without cause, unexplained regression)
+4. Separate measured vs. post-cutoff work
+
+### 4. Questioning
+
+Challenge every attribution:
+
+- [ ] "Do we KNOW this, or do we BELIEVE this?"
+- [ ] "What would need to be true for this to be wrong?"
+- [ ] "Are there alternative explanations?"
+
+Document what's missing:
+
+- [ ] Unexplained improvements
+- [ ] Unexplained regressions
+- [ ] Work that SHOULD have helped but didn't
+- [ ] Metrics you wish you had
+
+### 5. Synthesis
+
+Create audience-appropriate artifacts:
+
+| Artifact              | Audience        | Focus                                 |
+| --------------------- | --------------- | ------------------------------------- |
+| Executive Summary     | Leadership      | Hard data, key wins, team recognition |
+| Attribution Catalogue | Engineering     | Detailed per-change analysis          |
+| Methodology Doc       | Future analysts | Process, assumptions, data sources    |
+| Communication Post    | Stakeholders    | Exciting but honest, caveats visible  |
+
+---
+
+## Communication Template
+
+```markdown
+**[Metric]: [Before] → [After] ([Change %])**
+
+Population: [Who this measures]
+Caveat: [Key limitation]
+What's NOT included: [Equally interesting gaps]
+
+Notable contributors:
+
+- [Change 1] — [mechanism]
+- [Change 2] — [mechanism]
+
+Bottom line: [One sentence impact statement]
+```
+
+---
+
+## Anti-Patterns
+
+| Don't                                    | Do Instead                                       |
+| ---------------------------------------- | ------------------------------------------------ |
+| Claim causation from correlation         | "Correlates with" or "plausible contributor"     |
+| Attribute release total to single change | Note multiple changes, unknown isolated impact   |
+| Bury caveats in footnotes                | Caveats are part of the story                    |
+| Use superlatives without data            | Let numbers speak                                |
+| Hide uncertainty                         | Use qualifiers: "likely," "plausible," "unknown" |
+
+---
+
+## Checklist
+
+Before finalizing:
+
+- [ ] Measurement methodology documented
+- [ ] Baseline values recorded with source
+- [ ] All changes in scope enumerated
+- [ ] Confidence levels assigned with justification
+- [ ] Unexplained anomalies noted
+- [ ] Limitations explicitly stated
+- [ ] What's NOT included documented
+- [ ] Uncertainty reflected in language
+- [ ] Links/references for all claims
+- [ ] Multiple artifacts for different audiences
+
+---
diff --git a/domains/performance/skills/extension-profiling/skill.md b/domains/performance/skills/extension-profiling/skill.md
new file mode 100644
index 00000000..c2a80272
--- /dev/null
+++ b/domains/performance/skills/extension-profiling/skill.md
@@ -0,0 +1,65 @@
+---
+maturity: experimental
+name: extension-profiling
+description: Compare browser extension performance between branches using WDYR, React DevTools Profiler, and E2E benchmarks with statistical rigor.
+---
+
+# Browser Extension Profiling
+
+Methodology for profiling and comparing extension performance across branches or commits.
+
+## When To Use
+
+- Validating that a refactor reduces unnecessary re-renders (needs before/after comparison)
+- Establishing baseline metrics for a performance initiative
+- Investigating a reported UI slowdown in the extension
+
+## Do Not Use When
+
+- Single-run comparisons — statistical significance requires ≥10 runs per scenario
+- The change touches only non-render paths (background scripts, network with no UI impact)
+- Target behavior is server-side latency, not UI rendering
+
+## Workflow
+
+1. **Build both branches** with `yarn build:test` on the same machine and Chrome version
+
+2. **WDYR profiling** (unnecessary re-render counts)
+   ```bash
+   ENABLE_WHY_DID_YOU_RENDER=true yarn start
+   ```
+   Flags to watch:
+   - `different objects that are equal by value` → object recreation
+   - `different functions with the same name` → callback recreation
+   - `props object itself changed but values equal` → parent cascade
+
+3. **React DevTools Profiler** for flame graphs and commit timings
+   ```bash
+   yarn devtools:react
+   ```
+
+4. **E2E benchmarks** for scenario durations
+   ```bash
+   yarn test:e2e:benchmark
+   ```
+
+5. **Collect ≥10 runs** per scenario. Discard top/bottom 10%. Report mean, median, stddev, p75, p95.
+
+6. **Statistical threshold:** Cohen's d > 0.5 for a meaningful difference.
+
+## Common Pitfalls
+
+| Mistake | Correct Approach |
+|---------|-----------------|
+| Running branches on different machines or Chrome versions | Same machine, same Chrome, no other apps running |
+| Pooling all runs including noisy late-session ones | Compute per-round stats first; report cleanest signal with explicit round attribution |
+| Reporting absolute re-render counts without scenario context | Normalize per-action; cascade fixes show multiplied impact at root |
+| Skipping cache and state reset between runs | Clear browser cache, reset extension state for each run |
+
+## Pre-Profiling Checklist
+
+- [ ] Both branches built with `yarn build:test`
+- [ ] Same machine, same Chrome version
+- [ ] No other tabs or applications running
+- [ ] WDYR enabled: `ENABLE_WHY_DID_YOU_RENDER=true`
+- [ ] Cache and extension state cleared between runs
diff --git a/domains/performance/skills/react-render-delta/skill.md b/domains/performance/skills/react-render-delta/skill.md
new file mode 100644
index 00000000..7b50c00b
--- /dev/null
+++ b/domains/performance/skills/react-render-delta/skill.md
@@ -0,0 +1,112 @@
+---
+name: react-render-delta
+description: Prove a React rendering or memoization change actually reduced work, with a delivery gate and a reported band. Covers re-render counts (why-did-you-render), selector recomputes (reselect's real `.recomputations()` API), and A/B arms toggled at a FIXED commit rather than across a merge boundary. The falsifier is an arm whose treatment never reached the built bundle — a null from undelivered treatment is indistinguishable from a null from a small effect and reports as the second. Triggers on /mms-react-render-delta, or when asked to prove a component stopped over-rendering, measure selector recomputation, validate a memoization/React Compiler change, run a render-count A/B, or interpret a re-render benchmark. Callable by `evidence` as its React render & selector proof engine.
+maturity: experimental
+---
+
+# /react-render-delta
+
+A render-count number is worthless until two things are true: the **treatment reached the
+artifact the browser executes**, and the number is reported as a **band** rather than a point.
+Most of this skill is those two checks. The measurement itself is easy; the failure mode is
+reporting a difference between arms that never differed.
+
+> **Falsifier.** An arm whose manipulation cannot be observed in the built bundle. If you
+> cannot point at output that differs *in kind* between arms — a symbol present in one and
+> absent in the other, a flag line in a log — the A/B is not designed yet, and any delta it
+> produces is noise with a story attached.
+
+## Method
+
+1. **Name the delivery check before building arms, and verify it emits.** State how the run
+   itself will show the arms differ, then confirm that output exists on one real build before
+   scaling to N repeats. This ordering is the whole skill. A measurement launched before the
+   instrument is proven emits a null that reads exactly like "no effect".
+
+2. **Derive the needle from output at the stage you will grep, not one stage upstream.**
+   Compiler and bundler output are not the same text. Worked failures, both real:
+   - React Compiler at `target: '17'` emits `react-compiler-runtime`; at `target: '19'` it
+     emits `react/compiler-runtime`. Grepping for the wrong one returns 0 in *both* arms and
+     fails the arm that actually got the treatment.
+   - `_c(` is the form **babel** emits. Metro transforms it further — in a real 119 MB React
+     Native bundle it scored 13 hits, every one a minified vendor identifier
+     (`function _c(e,t){return e|t}`), while the true compiler output went uncounted. The form
+     that survived metro was `memo_cache_sentinel` (4681 in the treated arm vs 166 in the
+     control). Same needle, two bundlers, two different answers.
+
+   Compile one real file through the project's own config and read the output. Ten minutes
+   here saves a whole run.
+
+3. **A name is not a witness — count what only exists when the module is included.** A bare
+   module specifier appears in bundled `package.json` dependency lists whether or not the
+   module was ever pulled in; a clean control arm scored exactly 1 that way and was wrongly
+   failed. Gate on artifacts that cannot appear otherwise (a runtime sentinel, a compiled call
+   site). Keep the specifier count as a diagnostic — 3081-vs-1 is informative, it just isn't a
+   boolean.
+
+4. **Use the library's real counter before injecting your own.** `reselect` exposes
+   **`.recomputations()`** on memoized selectors — a genuine API, not a patch. Read it (sample
+   on an interval if the count should visibly climb). An injected `console.log` you added to a
+   selector body is an authored claim, not an observation; reach for it only when no real API
+   exists, and say so when you do. *(Note: the evidence catalog's render-and-selector entry long claimed there
+   was "no built-in selector-call counter". There is.)*
+
+5. **Toggle at a fixed commit, not across a merge boundary.** Same tree in both arms, one
+   thing different. A commit boundary drags in unrelated change you will then be unable to
+   exclude. When the real commit bundles two changes (a scope change *and* a version bump),
+   reproduce only the one under test — moving both reintroduces the confound the fixed-commit
+   design exists to remove, and can silently flip your delivery needle mid-experiment.
+
+6. **Repeat the capture, not the build; report the band.** Counts vary run to run — one
+   baseline measured 153/164/224 across three runs. The build dominates cost (~6 min vs ~90 s
+   per capture), so repeats are nearly free. Publish one artifact; report every repeat's count.
+
+7. **When the delta is under the spread, say "not resolvable at this n" and give the MDE.**
+   Not "no effect". State the smallest detectable effect and what n would resolve the observed
+   difference. A real worked result: 112–128 vs 115–133, delta 4.6%, t=1.33 — with delivery
+   proven (1244 compiled sites vs 0), so the null was about effect size, not plumbing.
+
+8. **A check that finds nothing needs a positive control.** Before believing a zero, confirm
+   the same check finds something it should. A search that returned "0 references" looked like
+   confirmation until searching for a string known to be present *also* returned 0 — the index
+   didn't reach that content and the zero meant nothing.
+
+## Gates, in order
+
+| gate | asserts | on failure |
+|---|---|---|
+| source manipulation | the intended edit applied, and *only* it | abort the arm |
+| **delivery** | the change reached the built bundle | abort **before any capture** |
+| metric | the instrument emitted a non-zero count on capture 1 | abort before spending repeats |
+
+Each catches what the previous cannot. Source changing is not delivery; delivery is not the
+instrument working. Wire them as script-level aborts so a broken arm cannot report a number —
+"refusing to emit a render count from an arm whose treatment is unproven" is the correct
+output, and it is not a failure of the run.
+
+**Never relax a gate to make an arm pass.** When a gate fires, go read the artifact and find
+the mechanism first. Loosening is the work-reducing direction, which is exactly where scrutiny
+collapses. Demoting a needle from gate to diagnostic *after* proving it fires for an unrelated
+reason is legitimate; doing it because the arm failed is not.
+
+## What the count does and does not mean
+
+WDYR counts **every** re-render in the measured window, including boot settling — not only the
+cascade a given fix targeted. So an RCA predicting "→ 0 re-renders" for a specific cascade is
+not refuted by a non-zero WDYR total. Say which quantity you measured, and don't let a global
+counter stand in for a scoped claim.
+
+Global application does not imply a large effect: 1244 auto-memoized call sites moved one
+interaction's re-render count under 5%. Reach for a flow the change plausibly dominates, and
+treat a single flow as a lower bound on reach, not a summary of it.
+
+## Caveats to publish with the number
+
+Fixture parity (structurally matched vs byte-identical), arm ordering (randomized or not),
+what window the counter covers, and how many flows were measured. State them; they are cheap
+and their absence is what makes a number unfalsifiable.
+
+## Related
+
+- `evidence` — packages this skill's output as its [React render & selector proof category](https://github.com/MetaMask/skills/blob/main/domains/pr-workflow/skills/evidence/references/evidence-catalog.md).
+- `memory-leak`, `supply-chain-audit` — sibling engines behind other categories.
diff --git a/domains/testing/knowledge/benchmark-statistical-hygiene.md b/domains/testing/knowledge/benchmark-statistical-hygiene.md
new file mode 100644
index 00000000..7015bbc5
--- /dev/null
+++ b/domains/testing/knowledge/benchmark-statistical-hygiene.md
@@ -0,0 +1,45 @@
+---
+name: benchmark-statistical-hygiene
+domain: testing
+description: Three patterns for defensible A/B benchmark results: per-round best-subset reporting, fix-vector isolation, and artifact sort-order trap.
+---
+
+# Benchmark Statistical Hygiene
+
+Three patterns that prevent the most common classes of invalid benchmark conclusions.
+
+## Pattern: Per-Round Best-Subset Reporting
+
+Later benchmark rounds accumulate system noise (background load, memory pressure, I/O contention). Pooling all rounds blindly treats noisy late-session data equally with clean early-session data.
+
+**Instead:** Compute per-round statistics first, then report the cleanest signal per metric with explicit round attribution.
+
+```
+Round 1 (clean):    metric X → treatment wins, p=0.04, d=-1.7
+Round 2 (moderate): metric X → treatment wins, p=0.08, d=-0.9
+Round 3 (noisy):    metric X → no effect,       p=0.90, d=+0.04
+
+Pooled (all):       metric X → no effect,       p=0.50, d=-0.2   ← signal destroyed
+
+Correct report: "X improved 49% (Round 1, n=5, p=0.04, d=-1.7).
+                 Pooled n=20 loses significance due to Round 3 outliers."
+```
+
+A small N with large effect size (|d| > 1.5, p < 0.05) is more defensible than a large N where noise has diluted significance to nothing.
+
+## Pattern: Isolate the Fix Vector
+
+Design each benchmark flow to exercise the optimization's specific input vector as its primary signal source. Incidental coverage produces fragile results where signal-to-noise depends on how much of the measured duration is optimization-affected.
+
+| | Weak | Strong |
+|-|------|--------|
+| Design | End-to-end flow that incidentally triggers target once among many other operations | Rapid sequence of actions each triggering the target with minimal other overhead |
+| Optimization signal | ~5% of measured duration | ~80% of measured duration |
+
+## Pattern: Artifact Sort-Order Trap
+
+Unpadded iteration numbers in filenames break lexicographic sorting: `iteration-1, iteration-10, iteration-2, ...` interleaves data from different rounds when processed in glob order.
+
+**Rule:** When processing sequentially-numbered artifacts, extract the embedded timestamp or numeric value for sorting. Never rely on string sort order when numbers cross digit boundaries.
+
+**Diagnosis:** If pipeline results look implausible (p-values that are too perfect, round-level stats that don't match spot checks), print the actual file ordering the pipeline used. Check for lexicographic interleaving at digit boundaries. Re-sort by extracted timestamp or zero-padded key.
diff --git a/domains/testing/skills/benchmark-design/skill.md b/domains/testing/skills/benchmark-design/skill.md
new file mode 100644
index 00000000..5f343006
--- /dev/null
+++ b/domains/testing/skills/benchmark-design/skill.md
@@ -0,0 +1,71 @@
+---
+maturity: experimental
+name: benchmark-design
+description: Design, run, and analyze E2E performance benchmarks — session hygiene, per-round reporting, artifact grouping
+---
+
+# Benchmark Design
+
+## When To Use
+
+- Writing a new E2E benchmark flow
+- Interpreting or presenting benchmark results
+- Adding new metrics to existing benchmarks
+- Diagnosing unexpected benchmark results
+
+## Do Not Use When
+
+- Adding unit, integration, or correctness E2E tests
+- Profiling a single user-reported slowdown (use `selector-antipattern-scan`)
+- Writing micro-benchmarks outside the E2E harness
+
+## Workflow
+
+1. **Design the flow** — target ONE optimization vector per benchmark. Maximize ratio of optimization-affected time to total measured time.
+2. **Run reference benchmarks first** in any session — session state degrades over time.
+3. **Compute per-round statistics** before pooling. Check each round for stability (CV < 0.3 is a reasonable threshold).
+4. **Group artifacts by timestamp**, not filename sort order.
+5. **Report per-metric best subset** with explicit round attribution. Show pooled data as supplementary.
+
+## Flow Design by Optimization Type
+
+| Optimization | Primary cascade vector | Recommended flow |
+|---|---|---|
+| Selector memoization | State mutations | Multi-confirmation queue |
+| Context memoization | Any state update | Account switching cycle |
+| HOC stabilization | Route changes | Rapid route cycling (8+ transitions) |
+| Dead code removal | Navigation | Return-to-home timer |
+
+## Session Hygiene
+
+System state degrades over long sessions — background load and memory pressure inflate variance and can **invert** treatment effects.
+
+- Run reference/critical benchmarks first
+- If a late round contradicts clean earlier rounds, suspect session degradation before re-running the full suite
+
+## Artifact Grouping
+
+Filenames use `{test}-iteration-{N}-{ISO-timestamp}.json`. Unpadded N produces incorrect lexicographic sort.
+
+```javascript
+// Extract seconds-since-midnight for round assignment
+const match = filename.match(/T(\d{2})-(\d{2})-(\d{2})/);
+const secondsOfDay = +match[1] * 3600 + +match[2] * 60 + +match[3];
+// Group by time range — never by filename position or array index
+```
+
+## Adding Metrics
+
+Extend `collectMetrics()` in `test/e2e/webdriver/driver.js` and register the metric key in `test/e2e/benchmarks/utils/constants.ts` → `ALL_METRICS`.
+
+- **Performance API metrics** (paint, navigation timing): collect directly inside `collectMetrics()` via `window.performance.getEntriesByType(...)`.
+- **Long Task / TBT metrics**: already wired — `collectMetrics()` reads `window.stateHooks.getLongTaskMetricsWithTBT()`. Adding new long-task-derived metrics requires extending the `stateHooks` observer, not the driver.
+
+## Common Pitfalls
+
+| Mistake | Correct Approach |
+|---------|-----------------|
+| Pool all rounds before checking per-round stats | Per-round first — late-session noise can invert the treatment effect |
+| Sort artifacts by filename | Extract ISO timestamp; sort by numeric time value |
+| Benchmark flow that exercises multiple vectors | One vector per flow — mixed flows produce ambiguous signal |
+| Report pooled p-value as primary result | Report cleanest per-metric signal with round attribution; pooled is supplementary |

From 71a5dd072a2c26b2a809ccb981d79f720f2fe6c7 Mon Sep 17 00:00:00 2001
From: Jongsun Suh <jongsun.suh@icloud.com>
Date: Mon, 14 Sep 2026 08:03:23 -0400
Subject: [PATCH 2/5] Profile React on a development build, and report a null
 as unresolved

`yarn build:test` runs webpack with `--mode production`, which sets
`NODE_ENV=production`, and React's production build records no profiling
data. The statistics example labeled two non-significant results "no
effect", which an underpowered run and a true null cannot tell apart.
---
 domains/performance/skills/extension-profiling/skill.md   | 6 +++---
 .../testing/knowledge/benchmark-statistical-hygiene.md    | 8 ++++----
 2 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/domains/performance/skills/extension-profiling/skill.md b/domains/performance/skills/extension-profiling/skill.md
index c2a80272..35408128 100644
--- a/domains/performance/skills/extension-profiling/skill.md
+++ b/domains/performance/skills/extension-profiling/skill.md
@@ -22,7 +22,7 @@ Methodology for profiling and comparing extension performance across branches or
 
 ## Workflow
 
-1. **Build both branches** with `yarn build:test` on the same machine and Chrome version
+1. **Build both branches** on the same machine and Chrome version: `yarn build:test` for the E2E benchmarks, and a development build (`yarn start` or `yarn build:test:dev`) for the React DevTools Profiler
 
 2. **WDYR profiling** (unnecessary re-render counts)
    ```bash
@@ -33,7 +33,7 @@ Methodology for profiling and comparing extension performance across branches or
    - `different functions with the same name` → callback recreation
    - `props object itself changed but values equal` → parent cascade
 
-3. **React DevTools Profiler** for flame graphs and commit timings
+3. **React DevTools Profiler** for flame graphs and commit timings, on a development build. `yarn build:test` passes `--mode production`, which sets `NODE_ENV=production`, and React's production build records no profiling data.
    ```bash
    yarn devtools:react
    ```
@@ -58,7 +58,7 @@ Methodology for profiling and comparing extension performance across branches or
 
 ## Pre-Profiling Checklist
 
-- [ ] Both branches built with `yarn build:test`
+- [ ] Both branches built with `yarn build:test` (benchmarks) and a development build (React DevTools Profiler)
 - [ ] Same machine, same Chrome version
 - [ ] No other tabs or applications running
 - [ ] WDYR enabled: `ENABLE_WHY_DID_YOU_RENDER=true`
diff --git a/domains/testing/knowledge/benchmark-statistical-hygiene.md b/domains/testing/knowledge/benchmark-statistical-hygiene.md
index 7015bbc5..01476daa 100644
--- a/domains/testing/knowledge/benchmark-statistical-hygiene.md
+++ b/domains/testing/knowledge/benchmark-statistical-hygiene.md
@@ -15,11 +15,11 @@ Later benchmark rounds accumulate system noise (background load, memory pressure
 **Instead:** Compute per-round statistics first, then report the cleanest signal per metric with explicit round attribution.
 
 ```
-Round 1 (clean):    metric X → treatment wins, p=0.04, d=-1.7
-Round 2 (moderate): metric X → treatment wins, p=0.08, d=-0.9
-Round 3 (noisy):    metric X → no effect,       p=0.90, d=+0.04
+Round 1 (clean):    metric X → treatment wins,           p=0.04, d=-1.7
+Round 2 (moderate): metric X → treatment wins,           p=0.08, d=-0.9
+Round 3 (noisy):    metric X → not resolvable at this n, p=0.90, d=+0.04
 
-Pooled (all):       metric X → no effect,       p=0.50, d=-0.2   ← signal destroyed
+Pooled (all):       metric X → not resolvable at this n, p=0.50, d=-0.2   ← signal destroyed
 
 Correct report: "X improved 49% (Round 1, n=5, p=0.04, d=-1.7).
                  Pooled n=20 loses significance due to Round 3 outliers."

From a976fa8b21d89aae144fb9a0e48eae07dcf88576 Mon Sep 17 00:00:00 2001
From: Jongsun Suh <jongsun.suh@icloud.com>
Date: Mon, 14 Sep 2026 08:43:56 -0400
Subject: [PATCH 3/5] Move the E2E benchmark skill and knowledge to their own
 PR

`benchmark-design`, `benchmark-statistical-hygiene`,
`metrics-pipeline-design` and the two benchmark-facing Web Vitals files
now ship in MetaMask/skills#162 (add E2E benchmark design and statistics
skills). This PR keeps profiling, render-delta proof and data analysis.
---
 .../knowledge/metrics-pipeline-design.md      | 67 -----------------
 .../web-vitals-production-vs-benchmarks.md    | 21 ------
 .../knowledge/web-vitals-runtime-metrics.md   | 22 ------
 .../benchmark-statistical-hygiene.md          | 45 ------------
 .../testing/skills/benchmark-design/skill.md  | 71 -------------------
 5 files changed, 226 deletions(-)
 delete mode 100644 domains/performance/knowledge/metrics-pipeline-design.md
 delete mode 100644 domains/performance/knowledge/web-vitals-production-vs-benchmarks.md
 delete mode 100644 domains/performance/knowledge/web-vitals-runtime-metrics.md
 delete mode 100644 domains/testing/knowledge/benchmark-statistical-hygiene.md
 delete mode 100644 domains/testing/skills/benchmark-design/skill.md

diff --git a/domains/performance/knowledge/metrics-pipeline-design.md b/domains/performance/knowledge/metrics-pipeline-design.md
deleted file mode 100644
index 03821ef5..00000000
--- a/domains/performance/knowledge/metrics-pipeline-design.md
+++ /dev/null
@@ -1,67 +0,0 @@
----
-name: metrics-pipeline-design
-domain: performance
-description: Four-layer metric pipeline architecture for E2E benchmarks, with domain-specific statistical bounds and split reporting paths.
----
-
-# Metrics Pipeline Design
-
-Architecture for adding metric types to an E2E benchmark suite. Separates collection, running, statistics, and reporting into independent layers.
-
-## Architecture
-
-```
-Collector → Runner → Statistics → Reporter
-```
-
-| Layer | Responsibility |
-|-------|----------------|
-| **Collector** | Extract raw metric from browser/extension per iteration |
-| **Runner** | Per-iteration capture + aggregation orchestration |
-| **Statistics** | Domain-specific filtering, outlier detection, percentiles |
-| **Reporter** | Per-run spans (for quality gate comparison) + aggregated structured logs (for dashboards) |
-
-Flow files call the collector and return snapshots alongside timers. No flow file does statistics or reporting.
-
-## Adding a New Metric Type
-
-1. **Create collector** — function returning typed snapshot with nullable fields for unobserved metrics
-2. **Define types** — per-run snapshot, aggregated (reuse `TimerStatistics` for numeric fields), summary
-3. **Add domain-specific bounds** — each numeric field gets `{ min, max, allowZero }`
-4. **Wire into runner** — collect alongside timers, call aggregation
-5. **Add reporter** — per-run spans with `setMeasurement`, aggregated summary as structured log
-
-## Domain-Specific Statistical Bounds
-
-Generic timer bounds (1ms–120s, zero=invalid) silently discard valid data from other domains.
-
-```typescript
-// WRONG: CLS values (0–1) all rejected by min=1ms floor
-const result = filterBySanityChecks(clsValues); // → empty array
-
-// RIGHT: per-metric bounds
-const BOUNDS = {
-  inp: { min: 1, max: 30_000, allowZero: false },  // ms
-  lcp: { min: 1, max: 60_000, allowZero: false },  // ms
-  cls: { min: 0, max: 10,     allowZero: true  },  // unitless ratio
-};
-```
-
-**Rule:** When adding a new metric type, verify whether existing `filterBySanityChecks` assumptions (ms units, zero=invalid) hold. If not, define metric-specific bounds.
-
-`allowZero` is the critical distinction: CLS=0 means perfect stability (valid); timer=0ms means measurement error (invalid).
-
-## Split Reporting Path
-
-| Data | Mechanism | Rationale |
-|------|-----------|-----------|
-| Aggregated statistics (mean, p75, p95) | Structured log | Low cardinality, dashboard-friendly |
-| Per-run snapshots | Sentry spans + `setMeasurement` | Preserves granularity, enables quality gate comparison via Mann-Whitney U |
-
-`tracesSampleRate: 1.0` required in CI so all per-run spans are captured.
-
-## SDK Isolation Pattern
-
-When CI benchmark scripts run in Node but the extension uses a browser SDK (e.g. `@sentry/node` vs `@sentry/browser`): these never share a process. The package manager resolves separate versions per dependency tree. No compatibility issue — they are fully isolated under different lockfile entries.
-
-Risk: a shared module accidentally importing from the wrong SDK at bundle time. Mitigation: keep the CI SDK as a devDependency excluded from extension builds.
diff --git a/domains/performance/knowledge/web-vitals-production-vs-benchmarks.md b/domains/performance/knowledge/web-vitals-production-vs-benchmarks.md
deleted file mode 100644
index b6bfdf63..00000000
--- a/domains/performance/knowledge/web-vitals-production-vs-benchmarks.md
+++ /dev/null
@@ -1,21 +0,0 @@
----
-name: web-vitals-production-vs-benchmarks
-domain: performance
-description: Web Vitals need different collection in production (web-vitals lib) vs benchmarks (PerformanceObserver); no TBT in prod
----
-
-# Web Vitals — Production vs Benchmarks
-
-Collection differs by environment due to timing constraints.
-
-## Production — `web-vitals` library
-- Reports on `visibilitychange` / `pagehide`
-- Handles browser quirks, bfcache, session windowing
-- Attribution build shows which element/script caused the metric
-- Metrics: **INP, LCP, CLS** (not TBT)
-
-**Why no TBT in production:** TBT is cumulative and unbounded — it grows indefinitely over an open-ended session. INP is per-interaction → meaningful for real users. TBT fits bounded flows (benchmarks), not sessions.
-
-## Benchmarks — direct `PerformanceObserver`
-- Query on demand (not dependent on page hide)
-- Fits an existing `collectMetrics()` pattern
diff --git a/domains/performance/knowledge/web-vitals-runtime-metrics.md b/domains/performance/knowledge/web-vitals-runtime-metrics.md
deleted file mode 100644
index e05d7c55..00000000
--- a/domains/performance/knowledge/web-vitals-runtime-metrics.md
+++ /dev/null
@@ -1,22 +0,0 @@
----
-name: web-vitals-runtime-metrics
-domain: performance
-description: Core Web Vitals (INP, TBT) are runtime responsiveness metrics, not just page-load — high-value for extension UX gates
----
-
-# Web Vitals as Runtime Metrics
-
-Core Web Vitals (INP, TBT) measure **runtime responsiveness**, not just page load. For a browser extension this distinction is critical.
-
-- **Page load is less relevant** — the popup opens fast; there's no traditional navigation.
-- **Runtime interactions matter** — every button click, form submit, confirmation. INP and TBT measure responsiveness during interactions → high-value for extension UX quality gates.
-
-## Orthogonal to distributed tracing
-
-| | Web Vitals | Distributed Tracing |
-|---|---|---|
-| Question | "How did the user perceive it?" | "Which controller caused it?" |
-| Scope | user perception | operation attribution |
-| Granularity | per-interaction aggregate | per-operation breakdown |
-
-Use both — perception (web vitals) + attribution (tracing) — not one instead of the other.
diff --git a/domains/testing/knowledge/benchmark-statistical-hygiene.md b/domains/testing/knowledge/benchmark-statistical-hygiene.md
deleted file mode 100644
index 01476daa..00000000
--- a/domains/testing/knowledge/benchmark-statistical-hygiene.md
+++ /dev/null
@@ -1,45 +0,0 @@
----
-name: benchmark-statistical-hygiene
-domain: testing
-description: Three patterns for defensible A/B benchmark results: per-round best-subset reporting, fix-vector isolation, and artifact sort-order trap.
----
-
-# Benchmark Statistical Hygiene
-
-Three patterns that prevent the most common classes of invalid benchmark conclusions.
-
-## Pattern: Per-Round Best-Subset Reporting
-
-Later benchmark rounds accumulate system noise (background load, memory pressure, I/O contention). Pooling all rounds blindly treats noisy late-session data equally with clean early-session data.
-
-**Instead:** Compute per-round statistics first, then report the cleanest signal per metric with explicit round attribution.
-
-```
-Round 1 (clean):    metric X → treatment wins,           p=0.04, d=-1.7
-Round 2 (moderate): metric X → treatment wins,           p=0.08, d=-0.9
-Round 3 (noisy):    metric X → not resolvable at this n, p=0.90, d=+0.04
-
-Pooled (all):       metric X → not resolvable at this n, p=0.50, d=-0.2   ← signal destroyed
-
-Correct report: "X improved 49% (Round 1, n=5, p=0.04, d=-1.7).
-                 Pooled n=20 loses significance due to Round 3 outliers."
-```
-
-A small N with large effect size (|d| > 1.5, p < 0.05) is more defensible than a large N where noise has diluted significance to nothing.
-
-## Pattern: Isolate the Fix Vector
-
-Design each benchmark flow to exercise the optimization's specific input vector as its primary signal source. Incidental coverage produces fragile results where signal-to-noise depends on how much of the measured duration is optimization-affected.
-
-| | Weak | Strong |
-|-|------|--------|
-| Design | End-to-end flow that incidentally triggers target once among many other operations | Rapid sequence of actions each triggering the target with minimal other overhead |
-| Optimization signal | ~5% of measured duration | ~80% of measured duration |
-
-## Pattern: Artifact Sort-Order Trap
-
-Unpadded iteration numbers in filenames break lexicographic sorting: `iteration-1, iteration-10, iteration-2, ...` interleaves data from different rounds when processed in glob order.
-
-**Rule:** When processing sequentially-numbered artifacts, extract the embedded timestamp or numeric value for sorting. Never rely on string sort order when numbers cross digit boundaries.
-
-**Diagnosis:** If pipeline results look implausible (p-values that are too perfect, round-level stats that don't match spot checks), print the actual file ordering the pipeline used. Check for lexicographic interleaving at digit boundaries. Re-sort by extracted timestamp or zero-padded key.
diff --git a/domains/testing/skills/benchmark-design/skill.md b/domains/testing/skills/benchmark-design/skill.md
deleted file mode 100644
index 5f343006..00000000
--- a/domains/testing/skills/benchmark-design/skill.md
+++ /dev/null
@@ -1,71 +0,0 @@
----
-maturity: experimental
-name: benchmark-design
-description: Design, run, and analyze E2E performance benchmarks — session hygiene, per-round reporting, artifact grouping
----
-
-# Benchmark Design
-
-## When To Use
-
-- Writing a new E2E benchmark flow
-- Interpreting or presenting benchmark results
-- Adding new metrics to existing benchmarks
-- Diagnosing unexpected benchmark results
-
-## Do Not Use When
-
-- Adding unit, integration, or correctness E2E tests
-- Profiling a single user-reported slowdown (use `selector-antipattern-scan`)
-- Writing micro-benchmarks outside the E2E harness
-
-## Workflow
-
-1. **Design the flow** — target ONE optimization vector per benchmark. Maximize ratio of optimization-affected time to total measured time.
-2. **Run reference benchmarks first** in any session — session state degrades over time.
-3. **Compute per-round statistics** before pooling. Check each round for stability (CV < 0.3 is a reasonable threshold).
-4. **Group artifacts by timestamp**, not filename sort order.
-5. **Report per-metric best subset** with explicit round attribution. Show pooled data as supplementary.
-
-## Flow Design by Optimization Type
-
-| Optimization | Primary cascade vector | Recommended flow |
-|---|---|---|
-| Selector memoization | State mutations | Multi-confirmation queue |
-| Context memoization | Any state update | Account switching cycle |
-| HOC stabilization | Route changes | Rapid route cycling (8+ transitions) |
-| Dead code removal | Navigation | Return-to-home timer |
-
-## Session Hygiene
-
-System state degrades over long sessions — background load and memory pressure inflate variance and can **invert** treatment effects.
-
-- Run reference/critical benchmarks first
-- If a late round contradicts clean earlier rounds, suspect session degradation before re-running the full suite
-
-## Artifact Grouping
-
-Filenames use `{test}-iteration-{N}-{ISO-timestamp}.json`. Unpadded N produces incorrect lexicographic sort.
-
-```javascript
-// Extract seconds-since-midnight for round assignment
-const match = filename.match(/T(\d{2})-(\d{2})-(\d{2})/);
-const secondsOfDay = +match[1] * 3600 + +match[2] * 60 + +match[3];
-// Group by time range — never by filename position or array index
-```
-
-## Adding Metrics
-
-Extend `collectMetrics()` in `test/e2e/webdriver/driver.js` and register the metric key in `test/e2e/benchmarks/utils/constants.ts` → `ALL_METRICS`.
-
-- **Performance API metrics** (paint, navigation timing): collect directly inside `collectMetrics()` via `window.performance.getEntriesByType(...)`.
-- **Long Task / TBT metrics**: already wired — `collectMetrics()` reads `window.stateHooks.getLongTaskMetricsWithTBT()`. Adding new long-task-derived metrics requires extending the `stateHooks` observer, not the driver.
-
-## Common Pitfalls
-
-| Mistake | Correct Approach |
-|---------|-----------------|
-| Pool all rounds before checking per-round stats | Per-round first — late-session noise can invert the treatment effect |
-| Sort artifacts by filename | Extract ISO timestamp; sort by numeric time value |
-| Benchmark flow that exercises multiple vectors | One vector per flow — mixed flows produce ambiguous signal |
-| Report pooled p-value as primary result | Report cleanest per-metric signal with round attribution; pooled is supplementary |

From 069c0df30793ee4e3103355fe14ef01fc0044ab4 Mon Sep 17 00:00:00 2001
From: Jongsun Suh <jongsun.suh@icloud.com>
Date: Mon, 14 Sep 2026 08:44:24 -0400
Subject: [PATCH 4/5] Fix the primary analysis before the runs, and report
 effect size in units

Choosing the round with the cleanest signal after seeing the data is
cherry-picking a time range, so per-round results are sensitivity. A
Cohen's d cutoff sets meaningfulness from the benchmark's own spread, so
the difference worth acting on is fixed from product impact instead.
---
 domains/performance/skills/extension-profiling/skill.md | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/domains/performance/skills/extension-profiling/skill.md b/domains/performance/skills/extension-profiling/skill.md
index 35408128..39b85962 100644
--- a/domains/performance/skills/extension-profiling/skill.md
+++ b/domains/performance/skills/extension-profiling/skill.md
@@ -45,14 +45,14 @@ Methodology for profiling and comparing extension performance across branches or
 
 5. **Collect ≥10 runs** per scenario. Discard top/bottom 10%. Report mean, median, stddev, p75, p95.
 
-6. **Statistical threshold:** Cohen's d > 0.5 for a meaningful difference.
+6. **Effect size:** report the difference in the metric's own units alongside Cohen's d. Cohen's d is the difference divided by the benchmark's own spread, so the difference worth acting on comes from product impact, fixed before the runs.
 
 ## Common Pitfalls
 
 | Mistake | Correct Approach |
 |---------|-----------------|
 | Running branches on different machines or Chrome versions | Same machine, same Chrome, no other apps running |
-| Pooling all runs including noisy late-session ones | Compute per-round stats first; report cleanest signal with explicit round attribution |
+| Pooling all runs including noisy late-session ones | Fix the primary analysis, and any rule for dropping a round, before the runs. Report per-round stats as sensitivity, with explicit round attribution |
 | Reporting absolute re-render counts without scenario context | Normalize per-action; cascade fixes show multiplied impact at root |
 | Skipping cache and state reset between runs | Clear browser cache, reset extension state for each run |
 

From 91383194c0b36d2c69baeb4755e9f9050d6f5d88 Mon Sep 17 00:00:00 2001
From: Jongsun Suh <jongsun.suh@icloud.com>
Date: Mon, 14 Sep 2026 08:58:35 -0400
Subject: [PATCH 5/5] Measure exposure and release containment before reading
 an outcome

A percentile is not a cohort, and an outcome read without the fraction of
users on a build that contains the change is not valid. Each change is
confirmed in the release whose traffic is read, with pre- and post-change
clients kept apart. A PR's baseline is its merge-base, and a profiling
run names the wallet state it ran on.
---
 domains/performance/skills/data-analysis/skill.md       | 6 ++++--
 domains/performance/skills/extension-profiling/skill.md | 3 ++-
 2 files changed, 6 insertions(+), 3 deletions(-)

diff --git a/domains/performance/skills/data-analysis/skill.md b/domains/performance/skills/data-analysis/skill.md
index 4566045d..342d3ff1 100644
--- a/domains/performance/skills/data-analysis/skill.md
+++ b/domains/performance/skills/data-analysis/skill.md
@@ -57,7 +57,8 @@ Collection → Filtering → Curation → Questioning → Synthesis
 
 ```markdown
 **Metrics:** [What are you measuring?]
-**Population:** [Who? All users, p75, specific cohort?]
+**Population:** [Who? All users, p75, specific cohort? A percentile is not a cohort: "p75 = power users" is an assumption until verified against one]
+**Exposure:** [Fraction of the population running a build that contains the change, over the same window, as a number. If it is unknown, no outcome reading is valid]
 **Period:** [Measurement window - release tags or dates]
 **Source:** [APM, logs, synthetic benchmarks?]
 **Baseline:** [Starting values with methodology]
@@ -82,7 +83,7 @@ Categorize each change:
 
 Build attribution table:
 
-1. Map changes to metric movements by release
+1. Map changes to metric movements by release. Confirm each change's merge commit is contained in the release whose traffic you read, and segment by release so pre- and post-change clients are never pooled
 2. Note co-landed changes (shared attribution)
 3. Flag anomalies (improvement without cause, unexplained regression)
 4. Separate measured vs. post-cutoff work
@@ -94,6 +95,7 @@ Challenge every attribution:
 - [ ] "Do we KNOW this, or do we BELIEVE this?"
 - [ ] "What would need to be true for this to be wrong?"
 - [ ] "Are there alternative explanations?"
+- [ ] "Could a change in the population's composition explain it?" Decompose within-unit change from composition before reporting an aggregate
 
 Document what's missing:
 
diff --git a/domains/performance/skills/extension-profiling/skill.md b/domains/performance/skills/extension-profiling/skill.md
index 39b85962..23eecb70 100644
--- a/domains/performance/skills/extension-profiling/skill.md
+++ b/domains/performance/skills/extension-profiling/skill.md
@@ -22,7 +22,7 @@ Methodology for profiling and comparing extension performance across branches or
 
 ## Workflow
 
-1. **Build both branches** on the same machine and Chrome version: `yarn build:test` for the E2E benchmarks, and a development build (`yarn start` or `yarn build:test:dev`) for the React DevTools Profiler
+1. **Build both branches** on the same machine and Chrome version: `yarn build:test` for the E2E benchmarks, and a development build (`yarn start` or `yarn build:test:dev`) for the React DevTools Profiler. For a PR, build the PR's merge-base as the baseline, not main's tip. The merge-base is the build the PR actually changed.
 
 2. **WDYR profiling** (unnecessary re-render counts)
    ```bash
@@ -63,3 +63,4 @@ Methodology for profiling and comparing extension performance across branches or
 - [ ] No other tabs or applications running
 - [ ] WDYR enabled: `ENABLE_WHY_DID_YOU_RENDER=true`
 - [ ] Cache and extension state cleared between runs
+- [ ] Wallet state sized for the scenario: `yarn start:with-state` runs `yarn start` with a generated fixture wallet (30 accounts by default). A null on an empty wallet is a fact about the fixture, not the code