Skip to content

Commit 35fac6e

Browse files
committed
test(core): wait for the external child's own signal, not the clock (#3103)
Fixes #3103 Owner: Dev-1 Written-by: Dev-1 Bot: FlyBot Dev-1 Tier: T1 **What changed.** `core/test/room-external.test.js` assumed this box starts a child process in under 120 ms. On a pinned core with two CPU hogs the same file fails deterministically: `not ok 5` ("a stop is asked for on the protocol before the signal"), `timed out waiting for the child`, 4150 ms and 4143 ms — `120 ms` sleep plus the 4 s `waitFor` budget. The child was still booting when `cancel` wrote the stop line and `TERM_GRACE_MS = 50` ended the process, so the line never reached it. The test now waits for what the child already reports: before `cancel` it waits for `"type":"turn"` in `SEEN`, the same file the assertion reads. `#2634` and `#2689` had the stricter version of the same bet — a 300 ms and a 700 ms reply window that started while the process was still starting — so their inline children append `ready` to a marker file after attaching the stdin handler, and the test opens the seat and waits for that marker before it writes the turn. The wait condition is an event, never a bigger budget: the 4 s `waitFor` only bounds the failure, and its message now names the handshake (`timed out waiting for the child to read the turn line`), which is what made the original red anonymous. **What did not change.** No product code. `core/src/host/room-participant-external.js` is untouched, `TERM_GRACE_MS = 50` stays, no timeout is raised, and every inline child is the test's own `-e` script. Every assertion is kept, including `turn` landing before `stop` and the pid checks. **Baseline.** The mirrors under `patches/ds-bot/files/` follow from `generate` (only `files/test/room-external.test.js` changes); `check` reports the three `ok` lines and `canonical` is a no-op. **Evidence.** Pinned to one core with two hogs (`taskset -c 0`, two `while :` shells): base 2/2 red, both `not ok 5`, `timed out waiting for the child`, 4150 ms / 4143 ms; fixed file 3/3 green, 20/20 each, exit 0.
1 parent 5cb1692 commit 35fac6e

2 files changed

Lines changed: 126 additions & 70 deletions

File tree

‎core/test/room-external.test.js‎

Lines changed: 63 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -54,11 +54,8 @@ test('an external Bot whose process dies answers with a failure and lands in the
5454
const member = { id: 'ext', name: 'Runner', external: { driver: 'subprocess', command: [process.execPath, '-e', DIE] } }
5555
const reply = await run(member)
5656
assert.equal(reply?.failure?.reason, 'crashed')
57-
const entry = await new Promise((resolve) => {
58-
const started = Date.now()
59-
const tick = () => (lines.length > 0 || Date.now() - started > 4000 ? resolve(lines[0]) : setTimeout(tick, 10))
60-
tick()
61-
})
57+
await waitFor(() => lines.length > 0, 'the room line for the participant that died')
58+
const entry = lines[0]
6259
assert.ok(entry, 'the room carries the state of a participant that died')
6360
assert.equal(entry.roomId, 'ops')
6461
assert.equal(entry.botId, 'ext')
@@ -79,10 +76,10 @@ test('a crashed external Bot starts a fresh process on its next turn (#198)', as
7976
const { lines, run } = fixture(t)
8077
const member = { id: 'gone', name: 'Gone', external: { driver: 'subprocess', command: [process.execPath, '-e', DIE] } }
8178
assert.equal((await run(member))?.failure?.reason, 'crashed')
82-
await waitFor(() => lines.length >= 1)
79+
await waitFor(() => lines.length >= 1, 'the crash line in the room')
8380
// A turn after the crash is not a dead end: the next one spawns again, and this one dies too.
8481
assert.equal((await run(member, 'again'))?.failure?.reason, 'crashed')
85-
await waitFor(() => lines.length >= 2)
82+
await waitFor(() => lines.length >= 2, 'the second crash line in the room')
8683
assert.equal(lines.filter(line => line.botId === 'gone').length, 2)
8784
})
8885

@@ -108,10 +105,13 @@ test('a stop is asked for on the protocol before the signal, and a shell is not
108105
external: { driver: 'subprocess', command: [process.execPath, '-e', script], env: { SEEN: seen } },
109106
}
110107
const turn = run(member, 'work')
111-
await new Promise(resolve => setTimeout(resolve, 120))
108+
// The stop line is a courtesy the driver follows with SIGTERM after TERM_GRACE_MS, so the child
109+
// has to be reading before the stop is asked for: the turn it already read is that proof, and a
110+
// fixed sleep here was only a bet on how fast this box starts a process (#3103).
111+
await waitFor(() => readFileSync(seen, 'utf8').includes('"type":"turn"'), 'the child to read the turn line')
112112
assert.equal(await driver.cancel({ sessionId: 'external:ops:stop' }), true)
113113
assert.equal(await turn, null)
114-
await waitFor(() => readFileSync(seen, 'utf8').includes('"type":"stop"'))
114+
await waitFor(() => readFileSync(seen, 'utf8').includes('"type":"stop"'), 'the child to read the stop line')
115115
const lines = readFileSync(seen, 'utf8').split('\n').filter(Boolean)
116116
assert.match(lines[0], /"type":"turn"/)
117117
assert.match(lines[1], /"type":"stop"/)
@@ -133,7 +133,7 @@ setInterval(()=>fs.writeFileSync(process.env.DIR+'/f'+(n++),'x'),1)`
133133
const { driver, room } = fixture(t, dir)
134134
const member = { id: 'writes', name: 'Writes', external: { driver: 'subprocess', command: [process.execPath, '-e', LINGER], env: { DIR: dir } } }
135135
await driver.open({ room, member })
136-
await waitFor(() => readdirSync(dir).length > 300)
136+
await waitFor(() => readdirSync(dir).length > 300, 'the child to fill the temp dir')
137137
driver.close()
138138
await removeExternalDir(dir)
139139
assert.equal(existsSync(dir), false, 'a temp dir outlived the assertions that used it')
@@ -163,11 +163,8 @@ test('a turn that times out comes back as no answer, not as a model failure (#19
163163
// silence, so { failure } here would count a slow process as a failed model call.
164164
assert.deepEqual(reply, { missed: true })
165165
assert.equal(driver.snapshot('external:ops:slow').state, 'timeout')
166-
const entry = await new Promise((resolve) => {
167-
const started = Date.now()
168-
const tick = () => (lines.length > 0 || Date.now() - started > 4000 ? resolve(lines[0]) : setTimeout(tick, 10))
169-
tick()
170-
})
166+
await waitFor(() => lines.length > 0, 'the timeout line in the room')
167+
const entry = lines[0]
171168
assert.match(entry.text, /timeout/)
172169
assert.equal(entry.botId, 'slow')
173170
})
@@ -181,11 +178,11 @@ test('a child that ignores the stop is killed after the grace (#1164)', async (t
181178
const { driver, run } = fixture(t, dir)
182179
const member = { id: 'stubborn', name: 'Stubborn', external: { driver: 'subprocess', command: [process.execPath, '-e', STUBBORN], env: { SEEN: seen } } }
183180
const turn = run(member, 'work')
184-
await waitFor(() => /start \d+/.test(readFileSync(seen, 'utf8')))
181+
await waitFor(() => /start \d+/.test(readFileSync(seen, 'utf8')), 'the child to report its pid')
185182
const pid = Number(readFileSync(seen, 'utf8').match(/start (\d+)/)[1])
186183
assert.equal(driver.closeParticipant('ops', 'stubborn'), true)
187184
assert.equal(await turn, null)
188-
await waitFor(() => { try { process.kill(pid, 0); return false } catch { return true } }, 4000)
185+
await waitFor(() => { try { process.kill(pid, 0); return false } catch { return true } }, 'the killed child to leave', 4000)
189186
})
190187

191188
test('a timed-out turn never settles the next one with its late answer (#2634)', async (t) => {
@@ -217,29 +214,39 @@ process.stdin.on('data',c=>{
217214
queue.push(m);
218215
pump();
219216
}
220-
});`
217+
});
218+
fs.appendFileSync(process.env.READY,'ready\\n')`
221219
const dir = mkdtempSync(join(tmpdir(), 'flybot-external-'))
222220
const seen = join(dir, 'seen.txt')
223221
const mark = join(dir, 'mark')
222+
const ready = readyMarker(dir)
224223
writeFileSync(seen, '')
225-
const { rt, run } = fixture(t, dir)
224+
const { driver, room, rt, run } = fixture(t, dir)
226225
rt.roomReplyTimeoutMs = 300
227-
const member = { id: 'held2', name: 'Held2', external: { driver: 'subprocess', command: [process.execPath, '-e', HELD], env: { SEEN: seen, MARK: mark } } }
226+
const member = { id: 'held2', name: 'Held2', external: { driver: 'subprocess', command: [process.execPath, '-e', HELD], env: { SEEN: seen, MARK: mark, READY: ready } } }
228227
const rows = () => readFileSync(seen, 'utf8').split('\n').filter(Boolean)
229228
const seenTurns = () => rows().filter(row => row.startsWith('turn '))
230229
const starts = () => rows().filter(row => row.startsWith('start '))
230+
// The reply window is 300ms, so the child has to be reading before the turn is written: opening
231+
// first and waiting for its marker measures the answer, not the boot (#3103).
232+
await driver.open({ room, member })
233+
await waitReady(ready, 'held2')
231234
const first = run(member, 'one')
232-
await waitFor(() => seenTurns().length === 1)
235+
await waitFor(() => seenTurns().length === 1, 'the child to record the first turn')
233236
assert.deepEqual(await first, { missed: true })
234237
// The second turn goes to whatever the driver believes answers it; the marker then releases the
235-
// first turn's answer, which must not become the second turn's answer.
238+
// first turn's answer, which must not become the second turn's answer. The timed-out process is
239+
// gone, so this turn needs a fresh one: start it and let it read before the turn, or its 300ms
240+
// window expires on a process that is still booting (#3103).
241+
await driver.open({ room, member })
242+
await waitReady(ready, 'held2', 2)
236243
const second = run(member, 'two')
237-
await waitFor(() => seenTurns().length === 2)
244+
await waitFor(() => seenTurns().length === 2, 'the child to record the second turn')
238245
writeFileSync(mark, '1')
239246
const answer = await second
240247
assert.notDeepEqual(answer, 'answer to one', "the next turn was settled by the timed-out turn's late answer")
241248
if (typeof answer === 'string') assert.equal(answer, 'answer to two')
242-
await waitFor(() => starts().length === 2)
249+
await waitFor(() => starts().length === 2, 'a second process to start')
243250
assert.notEqual(starts()[1], starts()[0], 'a timed-out process must not answer the next turn')
244251
})
245252

@@ -255,12 +262,12 @@ test('a seat that was closed is never handed the next turn (#1164)', async (t) =
255262
const member = { id: 'seat2', name: 'Seat2', external: { driver: 'subprocess', command: [process.execPath, '-e', PID_LOG], env: { SEEN: seen } } }
256263
const turns = () => readFileSync(seen, 'utf8').split('\n').filter(row => row.startsWith('turn ')).map(row => row.slice(5))
257264
const stopped = run(member, 'one')
258-
await waitFor(() => turns().length === 1)
265+
await waitFor(() => turns().length === 1, 'the child to record the first turn')
259266
const first = turns()[0]
260267
assert.equal(await driver.cancel({ sessionId: 'external:ops:seat2' }), true)
261268
assert.equal(await stopped, null)
262269
const next = run(member, 'two')
263-
await waitFor(() => turns().length === 2)
270+
await waitFor(() => turns().length === 2, 'the child to record the second turn')
264271
const second = turns()[1]
265272
assert.notEqual(second, first, 'the closed process must not answer the next turn')
266273
await driver.cancel({ sessionId: 'external:ops:seat2' })
@@ -382,7 +389,7 @@ test('an answer line split past the 4 KB buffer is a failure, not a timeout, and
382389
test('bytes after an over-long line are dropped, never parsed as the answer (#2689)', async (t) => {
383390
// The tail after the overflow happens to be a well-formed answer object; it is the remainder of
384391
// the line that already overflowed, so it must not settle the turn with its text.
385-
const SPLIT_FRAGMENT = `let b='';
392+
const SPLIT_FRAGMENT = `const fs=require('node:fs');let b='';
386393
process.stdin.setEncoding('utf8');
387394
process.stdin.on('data',c=>{
388395
b+=c;
@@ -395,10 +402,17 @@ process.stdin.on('data',c=>{
395402
process.stdout.write('{"text":"'+'x'.repeat(4200));
396403
setTimeout(()=>process.stdout.write('{"text":"sneaky"}\\n'),60);
397404
}
398-
});`
399-
const { rt, run } = fixture(t)
405+
});
406+
fs.appendFileSync(process.env.READY,'ready\\n')`
407+
const dir = mkdtempSync(join(tmpdir(), 'flybot-external-'))
408+
const ready = readyMarker(dir)
409+
const { driver, room, rt, run } = fixture(t, dir)
400410
rt.roomReplyTimeoutMs = 700
401-
const member = { id: 'frag', name: 'Frag', external: { driver: 'subprocess', command: [process.execPath, '-e', SPLIT_FRAGMENT] } }
411+
const member = { id: 'frag', name: 'Frag', external: { driver: 'subprocess', command: [process.execPath, '-e', SPLIT_FRAGMENT], env: { READY: ready } } }
412+
// The 700ms window belongs to the answer, so the turn is written to a child that is already
413+
// reading rather than to a process that may still be starting (#3103).
414+
await driver.open({ room, member })
415+
await waitReady(ready, 'frag')
402416
const reply = await run(member, 'big')
403417
assert.deepEqual(reply, { failure: { reason: 'error', detail: 'answer exceeded 4096 bytes' } })
404418
})
@@ -429,7 +443,7 @@ process.stdin.on('data',c=>{
429443
const member = { id: 'noisy', name: 'Noisy', external: { driver: 'subprocess', command: [process.execPath, '-e', NOISY], env: { SEEN: seen } } }
430444
// `open` starts the process without a turn: the noise line is written while the seat is idle.
431445
await driver.open({ room, member })
432-
await waitFor(() => readFileSync(seen, 'utf8').includes('start '))
446+
await waitFor(() => readFileSync(seen, 'utf8').includes('start '), 'the child to report it started')
433447
// The turn opens after the noise line is closed: the noise never becomes its answer or line.
434448
await new Promise(resolve => setTimeout(resolve, 300))
435449
assert.equal(await run(member, 'hi'), 'ok: hi')
@@ -461,7 +475,7 @@ process.stdin.on('data',c=>{
461475
const { driver, room, lines, run } = fixture(t, dir)
462476
const member = { id: 'cross', name: 'Cross', external: { driver: 'subprocess', command: [process.execPath, '-e', CROSSING], env: { SEEN: seen } } }
463477
await driver.open({ room, member })
464-
await waitFor(() => readFileSync(seen, 'utf8').includes('start '))
478+
await waitFor(() => readFileSync(seen, 'utf8').includes('start '), 'the child to report it started')
465479
// The 6000-byte log line has overflowed while the seat is idle; the turn opens before its
466480
// newline lands at ~400 ms, and the child answers 200 ms after that.
467481
await new Promise(resolve => setTimeout(resolve, 300))
@@ -489,13 +503,27 @@ process.stdin.on('data',c=>{
489503
})
490504

491505
const waitForLine = async (lines, botId, ms = 4000) => {
492-
await waitFor(() => lines.some(line => line.botId === botId), ms)
506+
await waitFor(() => lines.some(line => line.botId === botId), `the room line for ${botId}`, ms)
493507
return lines.find(line => line.botId === botId)
494508
}
495-
const waitFor = async (done, ms = 4000) => {
509+
// The condition is the event and `what` names it: the budget only bounds the failure, so a red
510+
// says which handshake never arrived instead of an anonymous timeout (#3103).
511+
const waitFor = async (done, what = 'the child', ms = 4000) => {
496512
const started = Date.now()
497513
while (!done()) {
498-
if (Date.now() - started > ms) throw new Error('timed out waiting for the child')
514+
if (Date.now() - started > ms) throw new Error(`timed out waiting for ${what}`)
499515
await new Promise(resolve => setTimeout(resolve, 10))
500516
}
501517
}
518+
// A child announces that its stdin handler is attached, so a test never has to bet on how fast
519+
// this box starts a process: `run` waits for the marker, and the reply windows that follow
520+
// measure the answer rather than the boot (#3103).
521+
const readyMarker = (dir) => {
522+
const ready = join(dir, 'ready.txt')
523+
writeFileSync(ready, '')
524+
return ready
525+
}
526+
const waitReady = async (ready, id, count = 1) => {
527+
await waitFor(() => readFileSync(ready, 'utf8').split('\n').filter(row => row === 'ready').length >= count,
528+
`process ${count} for ${id} to attach its stdin handler`)
529+
}

0 commit comments

Comments
 (0)