Skip to content

render-smoke test flakes under load: a 150ms budget has to cover PTY + Node startup #2177

Description

@cliffhall

scripts/lib/render-smoke.test.mjs"a paint landing just under the deadline is not failed by the render timer" fails intermittently under load, which fails npm run test:scripts and therefore validate and the whole local:gate.

✖ a paint landing just under the deadline is not failed by the render timer (403.360125ms)
  AssertionError [ERR_ASSERTION]: expected pass, got: child did not render "MCP Servers" within 400ms

Hit twice in consecutive npm run local:gate runs on an unrelated branch; passes every time when npm run test:scripts is run on its own.

Cause

The test gives itself a 150ms budget that has to cover process startup, and startup alone nearly exhausts it:

const paintAt = 250;
const r = await runRenderSmoke({
  ...stub(`setTimeout(() => { console.log(MARKER); setInterval(() => {}, 1000); }, ${paintAt});`),
  timeoutMs: paintAt + 150,   // 400ms
  surviveMs: 400,
});

paintAt is measured from when the child's JS starts. renderTimer is armed in the parent, at spawn (render-smoke.mjs:324). So the 150ms has to absorb script(1) PTY allocation plus Node boot before the child's own 250ms timer even starts.

Measured on this machine, idle, 8 samples of spawn → first byte of child JS output through the same PTY wrapper:

66, 67, 68, 68, 68, 69, 71, 114 ms   (max 114, budget 150)

So the margin is ~36ms at best when nothing else is running. Inside local:gate — where this runs alongside the rest of the suite — it goes over, the render timer fires before the paint, and the run is reported as "did not render", which is precisely the misdiagnosis the test exists to guard against.

Not a wrong assertion — a too-tight fixture

The behavior under test is right and worth keeping: the render deadline must be able to expire during the survival window without settling the run as a render failure. That shape only needs

actual_paint < timeoutMs < actual_paint + surviveMs

and the current constants satisfy it only when startup is under 150ms.

Proposed fix

Widen both constants so the same shape holds with real margin on both sides — e.g. timeoutMs: 1200, surviveMs: 1500 with paintAt: 250. Worst-case observed paint (~365ms) still lands well before 1200ms, and 1200ms still falls inside the survival window (which would end between ~1750ms and ~1900ms), so the timer still fires during it. Costs roughly a second of runtime on one test.

Do not fix it by removing the overlap — the overlap is the bug being guarded (see the comment above the test).

Done when

  • The test does not depend on process startup fitting in a fixed sub-150ms budget
  • The render deadline still expires inside the survival window, so the guard still guards
  • npm run test:scripts passes repeatedly while the machine is loaded

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingv2Issues and PRs for v2

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions