Skip to content

fix(scripts): give the render-smoke overlap test room for process startup - #2178

Merged
cliffhall merged 2 commits into
v2/mainfrom
v2/fix/2177-render-smoke-startup-budget
Aug 28, 2026
Merged

fix(scripts): give the render-smoke overlap test room for process startup#2178
cliffhall merged 2 commits into
v2/mainfrom
v2/fix/2177-render-smoke-startup-budget

Conversation

@cliffhall

Copy link
Copy Markdown
Member

Closes #2177

scripts/lib/render-smoke.test.mjs → "a paint landing just under the deadline is not failed by the render timer" failed three consecutive npm run local:gate runs while passing every time standalone, which fails test:scriptsvalidate → the whole gate.

Cause

The test gave itself a 150ms budget that has to cover process startup:

const paintAt = 250;
timeoutMs: paintAt + 150,   // 400ms
surviveMs: 400,

paintAt is measured from the child's first JS tick; renderTimer is armed in the parent, at spawn (render-smoke.mjs:324). So those 150ms have to absorb script(1) PTY allocation plus Node boot before the child's own timer even starts.

Measured on this machine, idle, 8 samples of spawn → first byte of child JS output through the same PTY wrapper:

66, 67, 68, 68, 68, 69, 71, 114 ms   (max 114, budget 150)

~36ms of slack at best with nothing else running. Under gate load it goes over, the render timer fires before the paint, and the run is reported as did not render "MCP Servers" within 400ms — which is exactly the misdiagnosis this test exists to catch.

Fix

Widen the constants, not the assertion. The shape under test is an overlap, and only an overlap:

actual paint  <  timeoutMs  <  actual paint + surviveMs

timeoutMs: 1200 / surviveMs: 1500 keeps that with real margin on both sides — worst-case paint lands near 365ms (well under 1200), and 1200 still falls inside a survival window that closes no earlier than ~1750ms. The comment now spells out why both margins are wide and warns against tightening them back for speed.

Verified by mutation, not just by going green

A widened timing test can pass vacuously, so I checked it still fails for the right reason. Removing the clearTimeout(renderTimer) on first paint (render-smoke.mjs:267) — the line this test guards:

✖ a paint landing just under the deadline is not failed by the render timer (1204.191542ms)
ℹ pass 14
ℹ fail 1

It fails at 1204ms, i.e. the render timer still fires inside the survival window. Restored, 15/15 pass.

Verification

npm run local:gate green end to end, with test:scripts at 292/292.

Cost: about a second of extra runtime on one test.

🤖 Generated with Claude Code

https://claude.ai/code/session_013hzwzS8UvBsr4yZm7o7sev

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Expands timing margins in the render-smoke overlap regression test to prevent load-related flakes.

Changes:

  • Increases render timeout to 1200 ms.
  • Increases survival window to 1500 ms.
  • Documents the timing relationship being tested.

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

Comment thread scripts/lib/render-smoke.test.mjs Outdated
@cliffhall

Copy link
Copy Markdown
Member Author

Correct on both counts, and thank you — the fix was right for the wrong reason. Fixed in 30beabb.

You are right that this path never touches script(1): stub() spawns process.execPath directly, and resolvePtyWrapper lives in smoke-tui.mjs — the caller — not in runRenderSmoke. My 66-114ms measurement was taken through the PTY wrapper, so it was measuring a cost this fixture does not pay.

Re-measured the way the test actually spawns (plain process.execPath, 12 samples, idle):

46, 47, 48, 48, 48, 49, 49, 49, 54, 55, 56, 59 ms

So the old 150ms budget had roughly 90ms of slack idle, not the ~36ms I claimed. That is genuinely useful to know rather than a wording nit: it explains why this presented as an occasional flake instead of an obvious bug, and it means the cause is scheduling delay under load — hitting both the child's boot and the parent's timer callback during a full local:gate run — not a fixed startup cost.

The comment now says that, names stub()'s direct spawn explicitly so nobody infers PTY coverage from this fixture, and drops the bogus "worst case 365ms" figure in favour of the reason 1200ms is safe. The constants and the fix are unchanged.

Worth noting the fixture is still verified by mutation rather than by going green: removing the clearTimeout(renderTimer) on first paint (render-smoke.mjs:267) makes it fail at 1204ms, so it still fires inside the survival window and still catches its regression.

@cliffhall
cliffhall force-pushed the v2/fix/2177-render-smoke-startup-budget branch 2 times, most recently from bb7ecf0 to d446de5 Compare August 28, 2026 02:59
cliffhall and others added 2 commits August 27, 2026 23:51
…rtup

The test gave itself a 150ms budget that had to cover `script(1)` PTY
allocation plus Node boot: `paintAt` is measured from the child's first
JS tick, while `renderTimer` is armed in the parent at spawn. Measured
idle, that startup is 66-114ms — ~36ms of slack at best — and under the
load of a full `local:gate` run it goes over, so the render timer fires
before the paint and the run is reported as "did not render", which is
precisely the misdiagnosis this test exists to catch. It failed three
gate runs in a row while passing standalone every time.

The constants are widened rather than the assertion changed: the shape
under test is the overlap (paint < timeoutMs < paint + surviveMs), and
1200/1500 keeps it with real margin on both sides. Verified by mutation
— removing the `clearTimeout(renderTimer)` this test guards makes it
fail at 1204ms, i.e. the timer still fires inside the survival window.

Closes #2177

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013hzwzS8UvBsr4yZm7o7sev
Signed-off-by: cliffhall <cliff@futurescale.com>
…ment

Copilot round 1. The comment blamed `script(1)` PTY allocation, but this
test's `stub()` spawns `process.execPath` directly — the PTY wrapper
lives in `smoke-tui.mjs`, the caller — so the fixture never pays that
cost and the 66-114ms figure (measured through the wrapper) was not
this path's.

Re-measured without it: Node boot alone is 46-59ms idle, which fits the
old 150ms budget comfortably. That is *why* it read as a flake rather
than a bug — what actually exhausted the budget was scheduling delay
under a full gate run, on both the child's boot and the parent's timer.
The fix is unchanged; only the reasoning was wrong.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013hzwzS8UvBsr4yZm7o7sev
Signed-off-by: cliffhall <cliff@futurescale.com>
@cliffhall
cliffhall force-pushed the v2/fix/2177-render-smoke-startup-budget branch from 474c43c to 7083c6c Compare August 28, 2026 03:51
@cliffhall
cliffhall merged commit e2de318 into v2/main Aug 28, 2026
4 checks passed
@cliffhall
cliffhall deleted the v2/fix/2177-render-smoke-startup-budget branch August 28, 2026 04:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

v2 Issues and PRs for v2

Projects

None yet

Development

Successfully merging this pull request may close these issues.

render-smoke test flakes under load: a 150ms budget has to cover PTY + Node startup

2 participants