Repository navigation
[Hackathon] fix(testlist): interrupted --wait drops acknowledged run IDs and completed results #484
Description
Verified against (commit SHA or released version)
988c035 (public main, package 0.14.0), Node.js v22.23.2, macOS. Rebuilt and reproduced on 2026-10-17.
Discord Username / User ID
luckyzd.
What does this improvement do?
Preserve the dispatch receipt and already-observed results when an ordinary, non-tunnel testlist run --wait is interrupted. Currently a real SIGINT/SIGTERM during polling exits 130/143 with empty stdout, losing every acknowledged run ID, including the verdict of a member already polled to completion. The JSON stderr error then says "runId is in the partial JSON on stdout", but that partial JSON does not exist.
This is a Season 4 hackathon issue-first submission. The evidence below uses the actual compiled CLI, real OS signals, and contract-shaped HTTP fixtures. It is an offline reproduction, not a claim that a hosted incident was observed. No real credentials, tests, or credits were used.
Details / implementation notes
Minimal trigger and impact
For a normal test list targeting a public deployed environment:
testsprite --output json testlist run LIST --wait --timeout 600 --max-concurrency 1
# Ctrl-C while one accepted run is still being polled.The executable fixture below deterministically performs this sequence:
POST /testlist/list_fixture/runreturns 202 with three accepted entries:test_a/run_a,test_b/run_b,test_c/run_c. There are no conflicts, deferred cases or invalid responses.GET /runs/run_areturns a valid terminalpassed(orfailed) result.GET /runs/run_bstarts a pending long poll.run_cis already accepted by the service but has not yet been polled due to--max-concurrency 1.- The parent sends real SIGINT or SIGTERM to the CLI. The fixture's pending fetch observes its AbortSignal and rejects normally.
- The CLI exits conventionally, but stdout is 0 bytes, and stderr contains none of
run_a,run_b,run_c.
Actual JSON stderr, abbreviated:
{"error":{"code":"INTERRUPTED","message":"Interrupted by SIGINT.","nextAction":"The server-side run (if any) keeps executing and billing. Re-attach with: testsprite test wait <runId>, or stop it with: testsprite test cancel <runId> (runId is in the partial JSON on stdout).","requestId":"local","details":{"signal":"SIGINT"}}}The dispatch succeeded and the run work is not cancelled by this ordinary detach. Automation cannot use the promised partial receipt to reattach/cancel the unfinished runs, and the already-observed terminal result is discarded. Recovery via other history/list mechanisms may remain possible: this is loss of the CLI's current-invocation output, not deletion of server data. Re-dispatching the list just to recover can start additional work.
Expected: before returning 130/143, print one parseable partial envelope preserving all acknowledged test/run pairs, keep terminal results already obtained, identify accepted-but-unfinished members honestly, and print concrete reattach/cancel commands for those unfinished IDs. Preserve existing detach behavior; do not auto-cancel ordinary public-target runs.
Seven executed cases and controls
| Scenario | Exit | stdout | Observed |
|---|---|---|---|
| Testlist completes normally | 0 | 640 bytes | All 3 run IDs and passed results retained |
| Same list with --no-wait | 0 | 378 bytes | All 3 accepted run IDs retained; no poll |
| SIGINT during first pending poll | 130 | 0 bytes | All 3 accepted IDs missing |
| SIGINT after first member passed | 130 | 0 bytes | Passed result and all accepted IDs missing |
| SIGTERM after first member failed | 143 | 0 bytes | Failed result and all accepted IDs missing |
| Same SIGINT in text mode, redirected streams | 130 | 0 bytes | No concrete recovery IDs in either stream |
test wait run_a run_b run_c, same interruption |
130 | 271 bytes | Passed result preserved; run_b/run_c running; concrete resume hint |
All assertions passed again after a fresh build. Each process exited normally without the fixture watchdog; no SIGKILL or mocked clock/sleep was needed. Text mode was tested with redirected/non-TTY streams, not claimed as interactive ticker testing.
Existing testlist.spec.ts, interrupt.test.ts, render-error.test.ts: 111 tests passed, 3 files, 5.94 seconds. Product source is unchanged. These green existing tests do not cover the testlist polling catch together with the installed signal handler.
Source diagnosis
- The testlist poll fan-out catch only calls
emitEnvironmentDispatchPartialwhendeps.environmentTunnelis present, then rethrows. The ordinary public-target path emits nothing. - The accepted response and settled
resultsare already in scope, so no new persistent journal is needed. - The poller marks InterruptError as a run-wait interruption. The top-level envelope consequently promises a partial JSON receipt that this call site never writes.
- The shared shutdown contract delegates partial output to the armed wait's catch. The existing
runTestWaitManycatch demonstrates the intended receipt behavior and serves as an executed control. - Responses match CliTestListRunResponse, with actual HttpClient runtime validation on dispatch and run reads left enabled.
A contained fix should handle the non-tunnel interrupted fan-out using the accepted response plus settled member results, retaining the conventional signal exit and avoiding duplicate output on the existing tunnel path. This report does not request persistence or change verdict/exit-code policy.
Duplicate check
Searched open and closed Issues and PRs for testlist/test-list with interrupt, SIGINT, detach, receipt, partial, run IDs and runTestlistRunCore; also read related reports and reviews.
- [Hackathon] Graceful Ctrl+C (SIGINT) handling #75 / merged PR feat(cli): graceful termination signals + broken-pipe guard #185 added process signal handling. Its review concerned flushing an already-written stderr hint before exit. Here the handler and flushing work: the command catch never constructs the stdout receipt.
- [Hackathon] Make Interrupted Batch Runs Recoverable (Run Journal + Reattach) #80's maintainer discussion explicitly scopes remaining work to a persistent journal after terminal loss. This case is a handled interrupt with the accepted response and settled results still in memory.
- [Hackathon] fix(test): batch retry sleep exits on signal before preserving acknowledged IDs #426 loses create-batch receipts in a disarmed outer retry sleep, so the process exits before a catch can run. Here the signal arrives inside an armed long poll, the catch runs, and its non-tunnel branch skips output. No retry or 429 is involved.
- [Hackathon] fix(rerun): later chunk failure discards already-accepted run IDs (51 tests, 50 hidden runs) #405 / PR fix(rerun): preserve acknowledged runs after a later dispatch failure #410 concern a later rerun dispatch chunk failing, before aggregation. This is testlist after its single dispatch succeeds, during the poll fan-out; different command/catch and trigger. [Hackathon] fix(rerun): batch
--waitexits 0 when refused conflicts leave the rerun incomplete #380 is an incomplete-batch exit-0 policy issue; this exits 130/143 correctly. - [Hackathon] fix(rerun): JUnit drops undispatched tests and keeps stale reports when no runs are accepted #476 is rerun JUnit omission with normal completion; this loses the basic dispatch receipt and completed results on interruption even with no report flags.
No matching report or merged fix was found for this testlist-specific catch.
Self-contained reproduction
After npm ci && npm run build, save as reproduce-testlist-interrupt-receipts.mjs and run:
node reproduce-testlist-interrupt-receipts.mjs /absolute/path/to/testsprite-cliUse Node 22 on macOS/Linux. The script creates and removes a disposable temp directory, isolates CLI home lookup to it, blocks all fetches except the in-process fixture hostname, sends signals only to the CLI subprocesses it started, and prints JSON evidence. The assertions intentionally confirm the current bug and its normal controls.
Executable seven-case reproduction
import assert from 'node:assert/strict';
import {spawn} from 'node:child_process';
import {mkdtemp,readFile,writeFile,rm} from 'node:fs/promises';
import {tmpdir} from 'node:os';
import {join,resolve,dirname} from 'node:path';
// Real compiled CLI subprocess + real OS signals. Only HTTP is replaced.
// No actual service, jobs, credentials or credits are involved.
const root=resolve(process.argv[2]??'work/testsprite-cli');
const dir=await mkdtemp(join(tmpdir(),'testsprite-testlist-interrupt-'));
const preload=join(dir,'fixture.mjs');
await writeFile(preload,`
import assert from 'node:assert/strict';
import {writeFileSync} from 'node:fs';
import os from 'node:os';
import {syncBuiltinESMExports} from 'node:module';
os.homedir=()=>process.env.REPRO_HOME;syncBuiltinESMExports();
const trace=[],s=JSON.parse(process.env.REPRO_CASE),ids=['a','b','c'];
const stamp='2026-10-17T00:00:00Z';
process.on('exit',()=>writeFileSync(process.env.REPRO_TRACE,JSON.stringify(trace)));
const json=(body,status=200)=>new Response(JSON.stringify(body),{status,headers:{'content-type':'application/json'}});
globalThis.fetch=async(input,init={})=>{
const u=new URL(typeof input==='string'?input:input.url??input.href),method=init.method??'GET';
assert.equal(u.hostname,'mock.example.com');
const path=u.pathname.replace('/api/cli/v1',''),row={method,path,at:Date.now()};trace.push(row);
if(path==='/testlist/list_fixture'&&method==='GET')return json({
id:'list_fixture',name:'Fixture list',orgId:'org_fixture',caseCount:3,
projectEnvironments:[{projectId:'project_fixture',environmentName:'prod'}],lastExecutionId:null,
createdAt:stamp,updatedAt:stamp,lastExecutionCreated:null,
stats:{total:3,passed:3,failed:0,blocked:0,running:0},
cases:ids.map(id=>({testId:'test_'+id,type:'frontend',projectId:'project_fixture',projectName:'Fixture project',title:id,category:null,priority:null,modified:null,status:'passed'}))});
if(path==='/projects/project_fixture/env')return json({environments:[{name:'prod',url:'https://example.com',isDefault:true,isTemporary:false}]});
if(path==='/testlist/list_fixture/run'&&method==='POST'){
const response={accepted:ids.map(id=>({testId:'test_'+id,runId:'run_'+id,enqueuedAt:stamp})),conflicts:[],deferred:[]};
row.body=JSON.parse(String(init.body));row.response=response;return json(response,202);
}
const match=/^\\/runs\\/run_([abc])$/.exec(path);
if(match&&method==='GET'){
const id=match[1];
if(s.signal&&id===(s.beforeAny?'a':'b')){
row.pending=true;assert.ok(init.signal,'long poll must carry AbortSignal');
process.send({checkpoint:'poll-pending',runId:'run_'+id});
return new Promise((resolve,reject)=>{
const timer=setTimeout(()=>reject(new Error('Fixture watchdog: no signal received')),10000);
const onAbort=()=>{clearTimeout(timer);row.aborted=true;reject(init.signal.reason??new DOMException('Aborted','AbortError'));};
if(init.signal.aborted)onAbort();else init.signal.addEventListener('abort',onAbort,{once:true});
});
}
const failed=id==='a'&&s.firstFailed;row.returnedStatus=failed?'failed':'passed';
return json({runId:'run_'+id,testId:'test_'+id,projectId:'project_fixture',userId:'fixture',status:row.returnedStatus,
source:'cli',createdAt:stamp,startedAt:stamp,finishedAt:'2026-10-17T00:00:01Z',codeVersion:'v1',targetUrl:'https://example.com',createdFrom:'cli',
failedStepIndex:failed?0:null,failureKind:failed?'assertion':null,error:failed?'Fixture assertion failed':null,videoUrl:null,
stepSummary:{total:1,completed:1,passedCount:failed?0:1,failedCount:failed?1:0}});
}
row.unexpected=true;throw Error('Unexpected fixture route '+method+' '+path);
};
`);
const scenarios=[
{name:'testlist-completes-control',expectedExit:0},
{name:'testlist-no-wait-receipt-control',noWait:true,expectedExit:0},
{name:'testlist-sigint-before-any-result',signal:'SIGINT',beforeAny:true,expectedExit:130},
{name:'testlist-sigint-after-passed-member',signal:'SIGINT',expectedExit:130},
{name:'testlist-sigterm-after-failed-member',signal:'SIGTERM',firstFailed:true,expectedExit:143},
{name:'testlist-text-sigint',signal:'SIGINT',text:true,expectedExit:130},
{name:'multi-wait-sigint-control',signal:'SIGINT',waitMany:true,expectedExit:130},
];
const results=[];
try{
for(const s of scenarios){
const tracePath=join(dir,s.name+'.json');
const args=['--output',s.text?'text':'json',...(s.waitMany?['test','wait','run_a','run_b','run_c']:['testlist','run','list_fixture']),
...(s.waitMany?[]:[s.noWait?'--no-wait':'--wait']),...(!s.noWait?['--timeout','10','--max-concurrency','1']:[])];
const start=performance.now();
const child=spawn(process.execPath,['--import',preload,join(root,'dist/index.js'),...args],{
cwd:dir,stdio:['ignore','pipe','pipe','ipc'],env:{PATH:dirname(process.execPath)+':/usr/bin:/bin',
TESTSPRITE_API_KEY:'sk-user-offline-fixture',TESTSPRITE_API_URL:'https://mock.example.com',TESTSPRITE_NO_TELEMETRY:'1',TESTSPRITE_NO_UPDATE_NOTIFIER:'1',
REPRO_HOME:dir,REPRO_TRACE:tracePath,REPRO_CASE:JSON.stringify(s)}});
let stdout='',stderr='',signalAt=null,checkpoint=null,watchdog=false;
child.stdout.on('data',b=>stdout+=b);child.stderr.on('data',b=>stderr+=b);
child.on('message',m=>{if(signalAt===null){checkpoint=m;setTimeout(()=>{signalAt=performance.now()-start;child.kill(s.signal);},20);}});
const timer=setTimeout(()=>{watchdog=true;child.kill('SIGKILL');},15000);
const ended=await new Promise((res,rej)=>{child.on('error',rej);child.on('close',(code,signal)=>res({code,signal}));});clearTimeout(timer);
const trace=JSON.parse(await readFile(tracePath,'utf8'));
assert.equal(watchdog,false,s.name);assert.equal(ended.signal,null,s.name);assert.equal(ended.code,s.expectedExit,stderr);
assert.ok(!trace.some(x=>x.unexpected),s.name+': unexpected HTTP');
const dispatch=trace.find(x=>x.path==='/testlist/list_fixture/run');
if(s.waitMany)assert.equal(dispatch,undefined);else assert.equal(dispatch.response.accepted.length,3);
let payload=null;if(stdout.trim()&&!s.text)payload=JSON.parse(stdout);
if(s.signal){
assert.ok(checkpoint&&signalAt!==null);assert.ok(trace.some(x=>x.aborted));
if(!s.beforeAny)assert.equal(trace.find(x=>x.path==='/runs/run_a').returnedStatus,s.firstFailed?'failed':'passed');
if(s.waitMany){
assert.deepEqual(payload.results.map(x=>x.runId),['run_a','run_b','run_c']);
assert.equal(payload.results[0].status,'passed');assert.equal(payload.results[1].status,'running');
assert.ok(stderr.includes('testsprite test wait run_b run_c'));
}else{
assert.equal(stdout,'','current bug: accepted IDs and partial result disappear');
for(const id of ['run_a','run_b','run_c'])assert.ok(!stderr.includes(id),'current bug: concrete recovery IDs absent');
}
}else{
assert.deepEqual(payload.accepted.map(x=>x.runId),['run_a','run_b','run_c']);
if(!s.noWait)assert.ok(payload.accepted.every(x=>x.status==='passed'));
}
results.push({name:s.name,args,...ended,elapsedMs:performance.now()-start,signalAt,checkpoint,watchdog,stdout,stderr,payload,trace});
}
console.log(JSON.stringify({baseline:'988c0355bb05631e9c6c1785e443937ff2a88c0e',node:process.version,platform:process.platform,scope:'offline compiled CLI subprocesses; actual SIGINT/SIGTERM; HTTP fixtures only',results},null,2));
}finally{await rm(dir,{recursive:true,force:true});}Confirmations
- I have searched existing issues and this is not a duplicate.
- I have provided my Discord identity above for reward coordination.
Activity
zeshi-du commentedon Oct 20, 2026
Thanks @luckyzd — confirmed: in v0.15.0 the non-tunnel poll catch in
testlist run --waitrethrows without printing, while the INTERRUPTED hint promises partial JSON on stdout. Accepted.Scope:
src/commands/testlist.ts, only forInterruptErroron the non-tunnel path. Leave the tunnel branch untouched so nothing prints twice.runTestRunAll's fan-out catch intest.ts: print one parseable envelope (settled results as-is,status: "running"for unfinished members, plusconflicts,deferredand the other normal-path fields), then rethrow. If runs are unfinished, also print a stderr line with their run IDs and the re-attach and cancel commands (write a small local equivalent of the unexportedinterruptDetachMessage).test.ts,poll.tsorinterrupt.ts, and don't refactor shared code.Comment /assign to claim it.
luckyzd commentedon Oct 20, 2026
/assign
testsprite-hob commentedon Oct 20, 2026
Assigned to @luckyzd. Thanks for taking this on. (2 more slots available.)
Your open assigned issues (1/3):
(none)