Add a tool for measuring fixture tolerances

[kurkle]

Follow-up to #992, which recalibrated every fixture tolerance from measurement. This adds the tool that produced those numbers, so the next browser update does not start the guessing cycle over.

The problem

Fixture tolerances have been maintained reactively: a browser update shifts rendering, a fixture fails, and its tolerance gets nudged above the one number in the failure report. That number is the delta in the environment that happened to fail — it says nothing about the margin needed, so the new value is a guess. Repeat a few times and the suite is loose everywhere and still failing somewhere.

Usage

npm run measure-tolerances             # report what would change
npm run measure-tolerances -- --apply  # write the derived values

Requires Docker. Refuses to start if test/fixtures is dirty, since it rewrites the fixtures and restores them with git checkout afterwards.

Method

Every fixture is forced to tolerance: 0, which makes each one report its real pixel delta rather than just passing or failing. That runs in four environments: the host's Chrome and Firefox, and Linux Chrome and Firefox in a container matching the CI image.

condition	tolerance
delta 0 in every environment	0
otherwise	max(1.5 × linux, 1.05 × host)

Linux gets a real margin because CI is what gates the build. The host gets only enough to keep local runs green — macOS renders text 3-4x further from the reference than Linux does, and sizing every tolerance for it is what made them loose.

Validation

Run against master's fixtures, the tool reproduces all 226 values from #992 exactly, including the total allowed-pixel budget (247,975 → 159,947).

The container itself was validated against three pixel counts observed in a real CI run:

fixture	browser	CI	container
point/crossShadow	Firefox	1565px	1565px
point/starShadow	Firefox	1476px	1476px
doughnutLabel/contentMultiline	Chrome	919px	919px

Notes for review

The container runs linux/amd64 under emulation even on Apple Silicon. That is deliberate and documented: Chrome is not published for linux/arm64, and native arm64 rendering genuinely differs — point/starShadow measures 1474px there against 1476px in CI, and line/labelShadowColors differs by 74%. Faster, but not CI-faithful.

Incomplete runs are rejected rather than measured. A browser that disconnects partway still yields deltas for the specs it reached, and deriving tolerances from those would silently produce values that are too tight — the same class of failure as a green check that never measured anything.

Emulated Chrome needs --shm-size=2g (without it it hangs rather than running slowly), raised karma timeouts, and --no-sandbox. The flag was confirmed not to affect rendering rather than assumed: contentMultiline still measures the same 919px with it.

🤖 Generated with Claude Code