Background
Every PR against Expensify/App runs the Reassure performance check. It measures the whole suite twice: once on main, once on the PR, and compares the two. Each measurement takes about four minutes, nearly all of it the measuring itself.
The check fails on render-count changes only. A duration change is reported but gates nothing unless that scenario's count moved too.
Separately, main is already measured on the rare push that rotates the Jest transform cache key, to warm that cache (#100428).
Problem
Two things are wrong with measuring main inside every PR run:
- It is the same measurement every time, re-derived per run. It could be measured once per push to
main and cached.
- The two sides are not the same
main. The PR is measured on the main it branched from, while the baseline is measured on main as it is right now. Anything that merged in between counts as a difference the PR introduced, so an unrelated render-count change turns the check red, and re-running it later can flip the verdict without a line changing.
Solution
.reassure/baseline.perf stops being measured inside every PR run and becomes a cache entry keyed on the main commit it was measured at.
- Seed side.
seedJestPerfCache.yml already runs npx reassure --baseline, but only on the push that rotates the transform cache key. Run it on every push, and restore the transform cache instead of looking it up so that measure is warm. Save the .reassure/baseline.perf under a cache key containing github.sha, and upload the same file as a run artifact named after the commit.
- PR side.
baseline-perf-tests restores that entry keyed on pull_request.base.sha. On a hit it uploads the restored file as the artifact validate-perf-tests already downloads, and measures nothing. Every step after the restore carries the miss guard, so a hit is a restore plus an upload: under 30s, against 220s today.
- Miss path. If no entry exists for
base.sha, the job measures the baseline itself, as it does today. The one difference is which commit it measures: today the job checks out the latest main, whereas here it checks out base.sha, so a miss still compares the two sides at the same commit. That checkout adds about 7s. A miss costs what a run costs today, and it cannot change the verdict.
Rollout: two independent PRs
- PR 1 — the seed saves the baseline to the cache. Nothing restores it yet. It runs on the same runner class as the measure jobs, so what it saves is comparable, and entries exist on
main by the time PR 2 opens, which is what makes PR 2 testable.
- PR 2 — the PR run restores the baseline instead of measuring it. With PR 1 already on
main, this lands with real entries to restore, so the hit is visible in its own PR.
Reported in Slack.
Background
Every PR against
Expensify/Appruns the Reassure performance check. It measures the whole suite twice: once onmain, once on the PR, and compares the two. Each measurement takes about four minutes, nearly all of it the measuring itself.The check fails on render-count changes only. A duration change is reported but gates nothing unless that scenario's count moved too.
Separately,
mainis already measured on the rare push that rotates the Jest transform cache key, to warm that cache (#100428).Problem
Two things are wrong with measuring
maininside every PR run:mainand cached.main. The PR is measured on themainit branched from, while the baseline is measured onmainas it is right now. Anything that merged in between counts as a difference the PR introduced, so an unrelated render-count change turns the check red, and re-running it later can flip the verdict without a line changing.Solution
.reassure/baseline.perfstops being measured inside every PR run and becomes a cache entry keyed on themaincommit it was measured at.seedJestPerfCache.ymlalready runsnpx reassure --baseline, but only on the push that rotates the transform cache key. Run it on every push, and restore the transform cache instead of looking it up so that measure is warm. Save the.reassure/baseline.perfunder a cache key containinggithub.sha, and upload the same file as a run artifact named after the commit.baseline-perf-testsrestores that entry keyed onpull_request.base.sha. On a hit it uploads the restored file as the artifactvalidate-perf-testsalready downloads, and measures nothing. Every step after the restore carries the miss guard, so a hit is a restore plus an upload: under 30s, against 220s today.base.sha, the job measures the baseline itself, as it does today. The one difference is which commit it measures: today the job checks out the latestmain, whereas here it checks outbase.sha, so a miss still compares the two sides at the same commit. That checkout adds about 7s. A miss costs what a run costs today, and it cannot change the verdict.Rollout: two independent PRs
mainby the time PR 2 opens, which is what makes PR 2 testable.main, this lands with real entries to restore, so the hit is visible in its own PR.Reported in Slack.