diff --git a/domains/performance/knowledge/web-vitals-attribution-import.md b/domains/performance/knowledge/web-vitals-attribution-import.md new file mode 100644 index 00000000..98c0067b --- /dev/null +++ b/domains/performance/knowledge/web-vitals-attribution-import.md @@ -0,0 +1,19 @@ +--- +name: web-vitals-attribution-import +domain: performance +description: web-vitals/attribution is a module import path, not a separate package — no meaningful bundle cost, gives the symptom→cause link +--- + +# Web Vitals Attribution Import + +`web-vitals/attribution` is a **module import path**, not a separate package. The attribution build: +- Provides which script/element caused each metric +- Does **not** meaningfully increase production bundle size (tree-shaking applies) + +Don't skip it for "bundle size" reasons — that's a misread. + +## Why it matters +Attribution is the symptom→cause link: +- INP spike of 500ms +- Attribution: `eventTarget: '#confirm-swap-button'`, `eventType: 'click'` +- Combined with tracing → identifies the controller that blocked diff --git a/domains/performance/skills/data-analysis/skill.md b/domains/performance/skills/data-analysis/skill.md new file mode 100644 index 00000000..342d3ff1 --- /dev/null +++ b/domains/performance/skills/data-analysis/skill.md @@ -0,0 +1,166 @@ +--- +maturity: experimental +name: data-analysis +description: Structured approach for analyzing metrics, attributing changes, and communicating findings — five phases (collection → filtering → curation → questioning → synthesis), confidence assignment, audience-appropriate artifacts +--- + +# Data Analysis Skill + +Structured approach for analyzing metrics, attributing changes, and communicating findings. + +--- + +## When to Use + +- Performance analysis from production metrics +- Attribution of improvements/regressions to code changes +- Creating executive summaries or stakeholder communications +- Any analysis requiring correlation of changes to measured outcomes + +--- + +## Quick Reference + +### Five Phases + +``` +Collection → Filtering → Curation → Questioning → Synthesis +``` + +| Phase | Key Question | Output | +| ----------- | --------------------------- | ----------------------------------- | +| Collection | What are we measuring? | Baseline, scope, change list | +| Filtering | What's signal vs. noise? | Categorized changes with confidence | +| Curation | What correlates with what? | Attribution table | +| Questioning | Do we KNOW or BELIEVE this? | Validated claims with caveats | +| Synthesis | Who needs to know what? | Audience-appropriate artifacts | + +### Confidence Assignment + +| Level | Use When | +| ---------- | ---------------------------------------------------------------- | +| **High** | Clear mechanism + timing alignment + targets measured population | +| **Medium** | Plausible mechanism but confounded by other changes | +| **Low** | Speculative or enabling-only | + +### Attribution Table Template + +| Change | Evidence | Release | Metric | Confidence | Notes | +| ------------- | ----------- | --------- | ----------------- | ------------ | --------------------- | +| [Description] | [PR/commit] | [version] | [affected metric] | High/Med/Low | [mechanism or caveat] | + +--- + +## Process + +### 1. Collection + +```markdown +**Metrics:** [What are you measuring?] +**Population:** [Who? All users, p75, specific cohort? A percentile is not a cohort: "p75 = power users" is an assumption until verified against one] +**Exposure:** [Fraction of the population running a build that contains the change, over the same window, as a number. If it is unknown, no outcome reading is valid] +**Period:** [Measurement window - release tags or dates] +**Source:** [APM, logs, synthetic benchmarks?] +**Baseline:** [Starting values with methodology] +``` + +Enumerate ALL changes in scope: + +- Code changes (PRs, commits) +- Config changes +- External factors (traffic, user growth, infrastructure) + +### 2. Filtering + +Categorize each change: + +- **Direct:** Clear causal path to measured metric +- **Indirect:** Enabling infrastructure (value materializes later) +- **Unknown:** In scope but mechanism unclear +- **Noise:** Unlikely to affect measured metrics + +### 3. Curation + +Build attribution table: + +1. Map changes to metric movements by release. Confirm each change's merge commit is contained in the release whose traffic you read, and segment by release so pre- and post-change clients are never pooled +2. Note co-landed changes (shared attribution) +3. Flag anomalies (improvement without cause, unexplained regression) +4. Separate measured vs. post-cutoff work + +### 4. Questioning + +Challenge every attribution: + +- [ ] "Do we KNOW this, or do we BELIEVE this?" +- [ ] "What would need to be true for this to be wrong?" +- [ ] "Are there alternative explanations?" +- [ ] "Could a change in the population's composition explain it?" Decompose within-unit change from composition before reporting an aggregate + +Document what's missing: + +- [ ] Unexplained improvements +- [ ] Unexplained regressions +- [ ] Work that SHOULD have helped but didn't +- [ ] Metrics you wish you had + +### 5. Synthesis + +Create audience-appropriate artifacts: + +| Artifact | Audience | Focus | +| --------------------- | --------------- | ------------------------------------- | +| Executive Summary | Leadership | Hard data, key wins, team recognition | +| Attribution Catalogue | Engineering | Detailed per-change analysis | +| Methodology Doc | Future analysts | Process, assumptions, data sources | +| Communication Post | Stakeholders | Exciting but honest, caveats visible | + +--- + +## Communication Template + +```markdown +**[Metric]: [Before] → [After] ([Change %])** + +Population: [Who this measures] +Caveat: [Key limitation] +What's NOT included: [Equally interesting gaps] + +Notable contributors: + +- [Change 1] — [mechanism] +- [Change 2] — [mechanism] + +Bottom line: [One sentence impact statement] +``` + +--- + +## Anti-Patterns + +| Don't | Do Instead | +| ---------------------------------------- | ------------------------------------------------ | +| Claim causation from correlation | "Correlates with" or "plausible contributor" | +| Attribute release total to single change | Note multiple changes, unknown isolated impact | +| Bury caveats in footnotes | Caveats are part of the story | +| Use superlatives without data | Let numbers speak | +| Hide uncertainty | Use qualifiers: "likely," "plausible," "unknown" | + +--- + +## Checklist + +Before finalizing: + +- [ ] Measurement methodology documented +- [ ] Baseline values recorded with source +- [ ] All changes in scope enumerated +- [ ] Confidence levels assigned with justification +- [ ] Unexplained anomalies noted +- [ ] Limitations explicitly stated +- [ ] What's NOT included documented +- [ ] Uncertainty reflected in language +- [ ] Links/references for all claims +- [ ] Multiple artifacts for different audiences + +--- diff --git a/domains/performance/skills/extension-profiling/skill.md b/domains/performance/skills/extension-profiling/skill.md new file mode 100644 index 00000000..23eecb70 --- /dev/null +++ b/domains/performance/skills/extension-profiling/skill.md @@ -0,0 +1,66 @@ +--- +maturity: experimental +name: extension-profiling +description: Compare browser extension performance between branches using WDYR, React DevTools Profiler, and E2E benchmarks with statistical rigor. +--- + +# Browser Extension Profiling + +Methodology for profiling and comparing extension performance across branches or commits. + +## When To Use + +- Validating that a refactor reduces unnecessary re-renders (needs before/after comparison) +- Establishing baseline metrics for a performance initiative +- Investigating a reported UI slowdown in the extension + +## Do Not Use When + +- Single-run comparisons — statistical significance requires ≥10 runs per scenario +- The change touches only non-render paths (background scripts, network with no UI impact) +- Target behavior is server-side latency, not UI rendering + +## Workflow + +1. **Build both branches** on the same machine and Chrome version: `yarn build:test` for the E2E benchmarks, and a development build (`yarn start` or `yarn build:test:dev`) for the React DevTools Profiler. For a PR, build the PR's merge-base as the baseline, not main's tip. The merge-base is the build the PR actually changed. + +2. **WDYR profiling** (unnecessary re-render counts) + ```bash + ENABLE_WHY_DID_YOU_RENDER=true yarn start + ``` + Flags to watch: + - `different objects that are equal by value` → object recreation + - `different functions with the same name` → callback recreation + - `props object itself changed but values equal` → parent cascade + +3. **React DevTools Profiler** for flame graphs and commit timings, on a development build. `yarn build:test` passes `--mode production`, which sets `NODE_ENV=production`, and React's production build records no profiling data. + ```bash + yarn devtools:react + ``` + +4. **E2E benchmarks** for scenario durations + ```bash + yarn test:e2e:benchmark + ``` + +5. **Collect ≥10 runs** per scenario. Discard top/bottom 10%. Report mean, median, stddev, p75, p95. + +6. **Effect size:** report the difference in the metric's own units alongside Cohen's d. Cohen's d is the difference divided by the benchmark's own spread, so the difference worth acting on comes from product impact, fixed before the runs. + +## Common Pitfalls + +| Mistake | Correct Approach | +|---------|-----------------| +| Running branches on different machines or Chrome versions | Same machine, same Chrome, no other apps running | +| Pooling all runs including noisy late-session ones | Fix the primary analysis, and any rule for dropping a round, before the runs. Report per-round stats as sensitivity, with explicit round attribution | +| Reporting absolute re-render counts without scenario context | Normalize per-action; cascade fixes show multiplied impact at root | +| Skipping cache and state reset between runs | Clear browser cache, reset extension state for each run | + +## Pre-Profiling Checklist + +- [ ] Both branches built with `yarn build:test` (benchmarks) and a development build (React DevTools Profiler) +- [ ] Same machine, same Chrome version +- [ ] No other tabs or applications running +- [ ] WDYR enabled: `ENABLE_WHY_DID_YOU_RENDER=true` +- [ ] Cache and extension state cleared between runs +- [ ] Wallet state sized for the scenario: `yarn start:with-state` runs `yarn start` with a generated fixture wallet (30 accounts by default). A null on an empty wallet is a fact about the fixture, not the code diff --git a/domains/performance/skills/react-render-delta/skill.md b/domains/performance/skills/react-render-delta/skill.md new file mode 100644 index 00000000..7b50c00b --- /dev/null +++ b/domains/performance/skills/react-render-delta/skill.md @@ -0,0 +1,112 @@ +--- +name: react-render-delta +description: Prove a React rendering or memoization change actually reduced work, with a delivery gate and a reported band. Covers re-render counts (why-did-you-render), selector recomputes (reselect's real `.recomputations()` API), and A/B arms toggled at a FIXED commit rather than across a merge boundary. The falsifier is an arm whose treatment never reached the built bundle — a null from undelivered treatment is indistinguishable from a null from a small effect and reports as the second. Triggers on /mms-react-render-delta, or when asked to prove a component stopped over-rendering, measure selector recomputation, validate a memoization/React Compiler change, run a render-count A/B, or interpret a re-render benchmark. Callable by `evidence` as its React render & selector proof engine. +maturity: experimental +--- + +# /react-render-delta + +A render-count number is worthless until two things are true: the **treatment reached the +artifact the browser executes**, and the number is reported as a **band** rather than a point. +Most of this skill is those two checks. The measurement itself is easy; the failure mode is +reporting a difference between arms that never differed. + +> **Falsifier.** An arm whose manipulation cannot be observed in the built bundle. If you +> cannot point at output that differs *in kind* between arms — a symbol present in one and +> absent in the other, a flag line in a log — the A/B is not designed yet, and any delta it +> produces is noise with a story attached. + +## Method + +1. **Name the delivery check before building arms, and verify it emits.** State how the run + itself will show the arms differ, then confirm that output exists on one real build before + scaling to N repeats. This ordering is the whole skill. A measurement launched before the + instrument is proven emits a null that reads exactly like "no effect". + +2. **Derive the needle from output at the stage you will grep, not one stage upstream.** + Compiler and bundler output are not the same text. Worked failures, both real: + - React Compiler at `target: '17'` emits `react-compiler-runtime`; at `target: '19'` it + emits `react/compiler-runtime`. Grepping for the wrong one returns 0 in *both* arms and + fails the arm that actually got the treatment. + - `_c(` is the form **babel** emits. Metro transforms it further — in a real 119 MB React + Native bundle it scored 13 hits, every one a minified vendor identifier + (`function _c(e,t){return e|t}`), while the true compiler output went uncounted. The form + that survived metro was `memo_cache_sentinel` (4681 in the treated arm vs 166 in the + control). Same needle, two bundlers, two different answers. + + Compile one real file through the project's own config and read the output. Ten minutes + here saves a whole run. + +3. **A name is not a witness — count what only exists when the module is included.** A bare + module specifier appears in bundled `package.json` dependency lists whether or not the + module was ever pulled in; a clean control arm scored exactly 1 that way and was wrongly + failed. Gate on artifacts that cannot appear otherwise (a runtime sentinel, a compiled call + site). Keep the specifier count as a diagnostic — 3081-vs-1 is informative, it just isn't a + boolean. + +4. **Use the library's real counter before injecting your own.** `reselect` exposes + **`.recomputations()`** on memoized selectors — a genuine API, not a patch. Read it (sample + on an interval if the count should visibly climb). An injected `console.log` you added to a + selector body is an authored claim, not an observation; reach for it only when no real API + exists, and say so when you do. *(Note: the evidence catalog's render-and-selector entry long claimed there + was "no built-in selector-call counter". There is.)* + +5. **Toggle at a fixed commit, not across a merge boundary.** Same tree in both arms, one + thing different. A commit boundary drags in unrelated change you will then be unable to + exclude. When the real commit bundles two changes (a scope change *and* a version bump), + reproduce only the one under test — moving both reintroduces the confound the fixed-commit + design exists to remove, and can silently flip your delivery needle mid-experiment. + +6. **Repeat the capture, not the build; report the band.** Counts vary run to run — one + baseline measured 153/164/224 across three runs. The build dominates cost (~6 min vs ~90 s + per capture), so repeats are nearly free. Publish one artifact; report every repeat's count. + +7. **When the delta is under the spread, say "not resolvable at this n" and give the MDE.** + Not "no effect". State the smallest detectable effect and what n would resolve the observed + difference. A real worked result: 112–128 vs 115–133, delta 4.6%, t=1.33 — with delivery + proven (1244 compiled sites vs 0), so the null was about effect size, not plumbing. + +8. **A check that finds nothing needs a positive control.** Before believing a zero, confirm + the same check finds something it should. A search that returned "0 references" looked like + confirmation until searching for a string known to be present *also* returned 0 — the index + didn't reach that content and the zero meant nothing. + +## Gates, in order + +| gate | asserts | on failure | +|---|---|---| +| source manipulation | the intended edit applied, and *only* it | abort the arm | +| **delivery** | the change reached the built bundle | abort **before any capture** | +| metric | the instrument emitted a non-zero count on capture 1 | abort before spending repeats | + +Each catches what the previous cannot. Source changing is not delivery; delivery is not the +instrument working. Wire them as script-level aborts so a broken arm cannot report a number — +"refusing to emit a render count from an arm whose treatment is unproven" is the correct +output, and it is not a failure of the run. + +**Never relax a gate to make an arm pass.** When a gate fires, go read the artifact and find +the mechanism first. Loosening is the work-reducing direction, which is exactly where scrutiny +collapses. Demoting a needle from gate to diagnostic *after* proving it fires for an unrelated +reason is legitimate; doing it because the arm failed is not. + +## What the count does and does not mean + +WDYR counts **every** re-render in the measured window, including boot settling — not only the +cascade a given fix targeted. So an RCA predicting "→ 0 re-renders" for a specific cascade is +not refuted by a non-zero WDYR total. Say which quantity you measured, and don't let a global +counter stand in for a scoped claim. + +Global application does not imply a large effect: 1244 auto-memoized call sites moved one +interaction's re-render count under 5%. Reach for a flow the change plausibly dominates, and +treat a single flow as a lower bound on reach, not a summary of it. + +## Caveats to publish with the number + +Fixture parity (structurally matched vs byte-identical), arm ordering (randomized or not), +what window the counter covers, and how many flows were measured. State them; they are cheap +and their absence is what makes a number unfalsifiable. + +## Related + +- `evidence` — packages this skill's output as its [React render & selector proof category](https://github.com/MetaMask/skills/blob/main/domains/pr-workflow/skills/evidence/references/evidence-catalog.md). +- `memory-leak`, `supply-chain-audit` — sibling engines behind other categories.