Skip to content

[Perf] Measure main's Reassure baseline once per push, not once per PR run #100726

Description

@melvin-bot

Background

Every PR against Expensify/App runs the Reassure performance check. It measures the whole suite twice: once on main, once on the PR, and compares the two. Each measurement takes about four minutes, nearly all of it the measuring itself.

The check fails on render-count changes only. A duration change is reported but gates nothing unless that scenario's count moved too.

Separately, main is already measured on the rare push that rotates the Jest transform cache key, to warm that cache (#100428).

Problem

Two things are wrong with measuring main inside every PR run:

  • It is the same measurement every time, re-derived per run. It could be measured once per push to main and cached.
  • The two sides are not the same main. The PR is measured on the main it branched from, while the baseline is measured on main as it is right now. Anything that merged in between counts as a difference the PR introduced, so an unrelated render-count change turns the check red, and re-running it later can flip the verdict without a line changing.

Solution

.reassure/baseline.perf stops being measured inside every PR run and becomes a cache entry keyed on the main commit it was measured at.

  1. Seed side. seedJestPerfCache.yml already runs npx reassure --baseline, but only on the push that rotates the transform cache key. Run it on every push, and restore the transform cache instead of looking it up so that measure is warm. Save the .reassure/baseline.perf under a cache key containing github.sha, and upload the same file as a run artifact named after the commit.
  2. PR side. baseline-perf-tests restores that entry keyed on pull_request.base.sha. On a hit it uploads the restored file as the artifact validate-perf-tests already downloads, and measures nothing. Every step after the restore carries the miss guard, so a hit is a restore plus an upload: under 30s, against 220s today.
  3. Miss path. If no entry exists for base.sha, the job measures the baseline itself, as it does today. The one difference is which commit it measures: today the job checks out the latest main, whereas here it checks out base.sha, so a miss still compares the two sides at the same commit. That checkout adds about 7s. A miss costs what a run costs today, and it cannot change the verdict.

Rollout: two independent PRs

  1. PR 1 — the seed saves the baseline to the cache. Nothing restores it yet. It runs on the same runner class as the measure jobs, so what it saves is comparable, and entries exist on main by the time PR 2 opens, which is what makes PR 2 testable.
  2. PR 2 — the PR run restores the baseline instead of measuring it. With PR 1 already on main, this lands with real entries to restore, so the hit is visible in its own PR.

Reported in Slack.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions