Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions domains/performance/knowledge/web-vitals-attribution-import.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
---
name: web-vitals-attribution-import
domain: performance
description: web-vitals/attribution is a module import path, not a separate package — no meaningful bundle cost, gives the symptom→cause link
---

# Web Vitals Attribution Import

`web-vitals/attribution` is a **module import path**, not a separate package. The attribution build:
- Provides which script/element caused each metric
- Does **not** meaningfully increase production bundle size (tree-shaking applies)

Don't skip it for "bundle size" reasons — that's a misread.

## Why it matters
Attribution is the symptom→cause link:
- INP spike of 500ms
- Attribution: `eventTarget: '#confirm-swap-button'`, `eventType: 'click'`
- Combined with tracing → identifies the controller that blocked
166 changes: 166 additions & 0 deletions domains/performance/skills/data-analysis/skill.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,166 @@
---
maturity: experimental
name: data-analysis
description: Structured approach for analyzing metrics, attributing changes, and communicating findings — five phases (collection → filtering → curation → questioning → synthesis), confidence assignment, audience-appropriate artifacts
---

# Data Analysis Skill

Structured approach for analyzing metrics, attributing changes, and communicating findings.

---

## When to Use

- Performance analysis from production metrics
- Attribution of improvements/regressions to code changes
- Creating executive summaries or stakeholder communications
- Any analysis requiring correlation of changes to measured outcomes

---

## Quick Reference

### Five Phases

```
Collection → Filtering → Curation → Questioning → Synthesis
```

| Phase | Key Question | Output |
| ----------- | --------------------------- | ----------------------------------- |
| Collection | What are we measuring? | Baseline, scope, change list |
| Filtering | What's signal vs. noise? | Categorized changes with confidence |
| Curation | What correlates with what? | Attribution table |
| Questioning | Do we KNOW or BELIEVE this? | Validated claims with caveats |
| Synthesis | Who needs to know what? | Audience-appropriate artifacts |

### Confidence Assignment

| Level | Use When |
| ---------- | ---------------------------------------------------------------- |
| **High** | Clear mechanism + timing alignment + targets measured population |
| **Medium** | Plausible mechanism but confounded by other changes |
| **Low** | Speculative or enabling-only |

### Attribution Table Template

| Change | Evidence | Release | Metric | Confidence | Notes |
| ------------- | ----------- | --------- | ----------------- | ------------ | --------------------- |
| [Description] | [PR/commit] | [version] | [affected metric] | High/Med/Low | [mechanism or caveat] |

---

## Process

### 1. Collection

```markdown
**Metrics:** [What are you measuring?]
**Population:** [Who? All users, p75, specific cohort? A percentile is not a cohort: "p75 = power users" is an assumption until verified against one]
**Exposure:** [Fraction of the population running a build that contains the change, over the same window, as a number. If it is unknown, no outcome reading is valid]
**Period:** [Measurement window - release tags or dates]
**Source:** [APM, logs, synthetic benchmarks?]
**Baseline:** [Starting values with methodology]
```

Enumerate ALL changes in scope:

- Code changes (PRs, commits)
- Config changes
- External factors (traffic, user growth, infrastructure)

### 2. Filtering

Categorize each change:

- **Direct:** Clear causal path to measured metric
- **Indirect:** Enabling infrastructure (value materializes later)
- **Unknown:** In scope but mechanism unclear
- **Noise:** Unlikely to affect measured metrics

### 3. Curation

Build attribution table:

1. Map changes to metric movements by release. Confirm each change's merge commit is contained in the release whose traffic you read, and segment by release so pre- and post-change clients are never pooled
2. Note co-landed changes (shared attribution)
3. Flag anomalies (improvement without cause, unexplained regression)
4. Separate measured vs. post-cutoff work

### 4. Questioning

Challenge every attribution:

- [ ] "Do we KNOW this, or do we BELIEVE this?"
- [ ] "What would need to be true for this to be wrong?"
- [ ] "Are there alternative explanations?"
- [ ] "Could a change in the population's composition explain it?" Decompose within-unit change from composition before reporting an aggregate

Document what's missing:

- [ ] Unexplained improvements
- [ ] Unexplained regressions
- [ ] Work that SHOULD have helped but didn't
- [ ] Metrics you wish you had

### 5. Synthesis

Create audience-appropriate artifacts:

| Artifact | Audience | Focus |
| --------------------- | --------------- | ------------------------------------- |
| Executive Summary | Leadership | Hard data, key wins, team recognition |
| Attribution Catalogue | Engineering | Detailed per-change analysis |
| Methodology Doc | Future analysts | Process, assumptions, data sources |
| Communication Post | Stakeholders | Exciting but honest, caveats visible |

---

## Communication Template

```markdown
**[Metric]: [Before] → [After] ([Change %])**

Population: [Who this measures]
Caveat: [Key limitation]
What's NOT included: [Equally interesting gaps]

Notable contributors:

- [Change 1] — [mechanism]
- [Change 2] — [mechanism]

Bottom line: [One sentence impact statement]
```

---

## Anti-Patterns

| Don't | Do Instead |
| ---------------------------------------- | ------------------------------------------------ |
| Claim causation from correlation | "Correlates with" or "plausible contributor" |
| Attribute release total to single change | Note multiple changes, unknown isolated impact |
| Bury caveats in footnotes | Caveats are part of the story |
| Use superlatives without data | Let numbers speak |
| Hide uncertainty | Use qualifiers: "likely," "plausible," "unknown" |

---

## Checklist

Before finalizing:

- [ ] Measurement methodology documented
- [ ] Baseline values recorded with source
- [ ] All changes in scope enumerated
- [ ] Confidence levels assigned with justification
- [ ] Unexplained anomalies noted
- [ ] Limitations explicitly stated
- [ ] What's NOT included documented
- [ ] Uncertainty reflected in language
- [ ] Links/references for all claims
- [ ] Multiple artifacts for different audiences

---
66 changes: 66 additions & 0 deletions domains/performance/skills/extension-profiling/skill.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
---
maturity: experimental
name: extension-profiling
description: Compare browser extension performance between branches using WDYR, React DevTools Profiler, and E2E benchmarks with statistical rigor.
---

# Browser Extension Profiling

Methodology for profiling and comparing extension performance across branches or commits.

## When To Use

- Validating that a refactor reduces unnecessary re-renders (needs before/after comparison)
- Establishing baseline metrics for a performance initiative
- Investigating a reported UI slowdown in the extension

## Do Not Use When

- Single-run comparisons — statistical significance requires ≥10 runs per scenario
- The change touches only non-render paths (background scripts, network with no UI impact)
- Target behavior is server-side latency, not UI rendering

## Workflow

1. **Build both branches** on the same machine and Chrome version: `yarn build:test` for the E2E benchmarks, and a development build (`yarn start` or `yarn build:test:dev`) for the React DevTools Profiler. For a PR, build the PR's merge-base as the baseline, not main's tip. The merge-base is the build the PR actually changed.

2. **WDYR profiling** (unnecessary re-render counts)
```bash
ENABLE_WHY_DID_YOU_RENDER=true yarn start
```
Flags to watch:
- `different objects that are equal by value` → object recreation
- `different functions with the same name` → callback recreation
- `props object itself changed but values equal` → parent cascade

3. **React DevTools Profiler** for flame graphs and commit timings, on a development build. `yarn build:test` passes `--mode production`, which sets `NODE_ENV=production`, and React's production build records no profiling data.
```bash
yarn devtools:react
```

4. **E2E benchmarks** for scenario durations
```bash
yarn test:e2e:benchmark
```

5. **Collect ≥10 runs** per scenario. Discard top/bottom 10%. Report mean, median, stddev, p75, p95.

6. **Effect size:** report the difference in the metric's own units alongside Cohen's d. Cohen's d is the difference divided by the benchmark's own spread, so the difference worth acting on comes from product impact, fixed before the runs.

## Common Pitfalls

| Mistake | Correct Approach |
|---------|-----------------|
| Running branches on different machines or Chrome versions | Same machine, same Chrome, no other apps running |
| Pooling all runs including noisy late-session ones | Fix the primary analysis, and any rule for dropping a round, before the runs. Report per-round stats as sensitivity, with explicit round attribution |
| Reporting absolute re-render counts without scenario context | Normalize per-action; cascade fixes show multiplied impact at root |
| Skipping cache and state reset between runs | Clear browser cache, reset extension state for each run |

## Pre-Profiling Checklist

- [ ] Both branches built with `yarn build:test` (benchmarks) and a development build (React DevTools Profiler)
- [ ] Same machine, same Chrome version
- [ ] No other tabs or applications running
- [ ] WDYR enabled: `ENABLE_WHY_DID_YOU_RENDER=true`
- [ ] Cache and extension state cleared between runs
- [ ] Wallet state sized for the scenario: `yarn start:with-state` runs `yarn start` with a generated fixture wallet (30 accounts by default). A null on an empty wallet is a fact about the fixture, not the code
112 changes: 112 additions & 0 deletions domains/performance/skills/react-render-delta/skill.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,112 @@
---
name: react-render-delta
description: Prove a React rendering or memoization change actually reduced work, with a delivery gate and a reported band. Covers re-render counts (why-did-you-render), selector recomputes (reselect's real `.recomputations()` API), and A/B arms toggled at a FIXED commit rather than across a merge boundary. The falsifier is an arm whose treatment never reached the built bundle — a null from undelivered treatment is indistinguishable from a null from a small effect and reports as the second. Triggers on /mms-react-render-delta, or when asked to prove a component stopped over-rendering, measure selector recomputation, validate a memoization/React Compiler change, run a render-count A/B, or interpret a re-render benchmark. Callable by `evidence` as its React render & selector proof engine.
maturity: experimental
---

# /react-render-delta

A render-count number is worthless until two things are true: the **treatment reached the
artifact the browser executes**, and the number is reported as a **band** rather than a point.
Most of this skill is those two checks. The measurement itself is easy; the failure mode is
reporting a difference between arms that never differed.

> **Falsifier.** An arm whose manipulation cannot be observed in the built bundle. If you
> cannot point at output that differs *in kind* between arms — a symbol present in one and
> absent in the other, a flag line in a log — the A/B is not designed yet, and any delta it
> produces is noise with a story attached.

## Method

1. **Name the delivery check before building arms, and verify it emits.** State how the run
itself will show the arms differ, then confirm that output exists on one real build before
scaling to N repeats. This ordering is the whole skill. A measurement launched before the
instrument is proven emits a null that reads exactly like "no effect".

2. **Derive the needle from output at the stage you will grep, not one stage upstream.**
Compiler and bundler output are not the same text. Worked failures, both real:
- React Compiler at `target: '17'` emits `react-compiler-runtime`; at `target: '19'` it
emits `react/compiler-runtime`. Grepping for the wrong one returns 0 in *both* arms and
fails the arm that actually got the treatment.
- `_c(` is the form **babel** emits. Metro transforms it further — in a real 119 MB React
Native bundle it scored 13 hits, every one a minified vendor identifier
(`function _c(e,t){return e|t}`), while the true compiler output went uncounted. The form
that survived metro was `memo_cache_sentinel` (4681 in the treated arm vs 166 in the
control). Same needle, two bundlers, two different answers.

Compile one real file through the project's own config and read the output. Ten minutes
here saves a whole run.

3. **A name is not a witness — count what only exists when the module is included.** A bare
module specifier appears in bundled `package.json` dependency lists whether or not the
module was ever pulled in; a clean control arm scored exactly 1 that way and was wrongly
failed. Gate on artifacts that cannot appear otherwise (a runtime sentinel, a compiled call
site). Keep the specifier count as a diagnostic — 3081-vs-1 is informative, it just isn't a
boolean.

4. **Use the library's real counter before injecting your own.** `reselect` exposes
**`.recomputations()`** on memoized selectors — a genuine API, not a patch. Read it (sample
on an interval if the count should visibly climb). An injected `console.log` you added to a
selector body is an authored claim, not an observation; reach for it only when no real API
exists, and say so when you do. *(Note: the evidence catalog's render-and-selector entry long claimed there
was "no built-in selector-call counter". There is.)*

5. **Toggle at a fixed commit, not across a merge boundary.** Same tree in both arms, one
thing different. A commit boundary drags in unrelated change you will then be unable to
exclude. When the real commit bundles two changes (a scope change *and* a version bump),
reproduce only the one under test — moving both reintroduces the confound the fixed-commit
design exists to remove, and can silently flip your delivery needle mid-experiment.

6. **Repeat the capture, not the build; report the band.** Counts vary run to run — one
baseline measured 153/164/224 across three runs. The build dominates cost (~6 min vs ~90 s
per capture), so repeats are nearly free. Publish one artifact; report every repeat's count.

7. **When the delta is under the spread, say "not resolvable at this n" and give the MDE.**
Not "no effect". State the smallest detectable effect and what n would resolve the observed
difference. A real worked result: 112–128 vs 115–133, delta 4.6%, t=1.33 — with delivery
proven (1244 compiled sites vs 0), so the null was about effect size, not plumbing.

8. **A check that finds nothing needs a positive control.** Before believing a zero, confirm
the same check finds something it should. A search that returned "0 references" looked like
confirmation until searching for a string known to be present *also* returned 0 — the index
didn't reach that content and the zero meant nothing.

## Gates, in order

| gate | asserts | on failure |
|---|---|---|
| source manipulation | the intended edit applied, and *only* it | abort the arm |
| **delivery** | the change reached the built bundle | abort **before any capture** |
| metric | the instrument emitted a non-zero count on capture 1 | abort before spending repeats |

Each catches what the previous cannot. Source changing is not delivery; delivery is not the
instrument working. Wire them as script-level aborts so a broken arm cannot report a number —
"refusing to emit a render count from an arm whose treatment is unproven" is the correct
output, and it is not a failure of the run.

**Never relax a gate to make an arm pass.** When a gate fires, go read the artifact and find
the mechanism first. Loosening is the work-reducing direction, which is exactly where scrutiny
collapses. Demoting a needle from gate to diagnostic *after* proving it fires for an unrelated
reason is legitimate; doing it because the arm failed is not.

## What the count does and does not mean

WDYR counts **every** re-render in the measured window, including boot settling — not only the
cascade a given fix targeted. So an RCA predicting "→ 0 re-renders" for a specific cascade is
not refuted by a non-zero WDYR total. Say which quantity you measured, and don't let a global
counter stand in for a scoped claim.

Global application does not imply a large effect: 1244 auto-memoized call sites moved one
interaction's re-render count under 5%. Reach for a flow the change plausibly dominates, and
treat a single flow as a lower bound on reach, not a summary of it.

## Caveats to publish with the number

Fixture parity (structurally matched vs byte-identical), arm ordering (randomized or not),
what window the counter covers, and how many flows were measured. State them; they are cheap
and their absence is what makes a number unfalsifiable.

## Related

- `evidence` — packages this skill's output as its [React render & selector proof category](https://github.com/MetaMask/skills/blob/main/domains/pr-workflow/skills/evidence/references/evidence-catalog.md).
- `memory-leak`, `supply-chain-audit` — sibling engines behind other categories.
Loading