Skip to content

[Hackathon] fix(testlist): interrupted --wait drops acknowledged run IDs and completed results #484

Open

Description

@luckyzd

Verified against (commit SHA or released version)

988c035 (public main, package 0.14.0), Node.js v22.23.2, macOS. Rebuilt and reproduced on 2026-10-17.

Discord Username / User ID

luckyzd.

What does this improvement do?

Preserve the dispatch receipt and already-observed results when an ordinary, non-tunnel testlist run --wait is interrupted. Currently a real SIGINT/SIGTERM during polling exits 130/143 with empty stdout, losing every acknowledged run ID, including the verdict of a member already polled to completion. The JSON stderr error then says "runId is in the partial JSON on stdout", but that partial JSON does not exist.

This is a Season 4 hackathon issue-first submission. The evidence below uses the actual compiled CLI, real OS signals, and contract-shaped HTTP fixtures. It is an offline reproduction, not a claim that a hosted incident was observed. No real credentials, tests, or credits were used.

Details / implementation notes

Minimal trigger and impact

For a normal test list targeting a public deployed environment:

testsprite --output json testlist run LIST --wait --timeout 600 --max-concurrency 1
# Ctrl-C while one accepted run is still being polled.

The executable fixture below deterministically performs this sequence:

  1. POST /testlist/list_fixture/run returns 202 with three accepted entries: test_a/run_a, test_b/run_b, test_c/run_c. There are no conflicts, deferred cases or invalid responses.
  2. GET /runs/run_a returns a valid terminal passed (or failed) result.
  3. GET /runs/run_b starts a pending long poll. run_c is already accepted by the service but has not yet been polled due to --max-concurrency 1.
  4. The parent sends real SIGINT or SIGTERM to the CLI. The fixture's pending fetch observes its AbortSignal and rejects normally.
  5. The CLI exits conventionally, but stdout is 0 bytes, and stderr contains none of run_a, run_b, run_c.

Actual JSON stderr, abbreviated:

{"error":{"code":"INTERRUPTED","message":"Interrupted by SIGINT.","nextAction":"The server-side run (if any) keeps executing and billing. Re-attach with: testsprite test wait <runId>, or stop it with: testsprite test cancel <runId> (runId is in the partial JSON on stdout).","requestId":"local","details":{"signal":"SIGINT"}}}

The dispatch succeeded and the run work is not cancelled by this ordinary detach. Automation cannot use the promised partial receipt to reattach/cancel the unfinished runs, and the already-observed terminal result is discarded. Recovery via other history/list mechanisms may remain possible: this is loss of the CLI's current-invocation output, not deletion of server data. Re-dispatching the list just to recover can start additional work.

Expected: before returning 130/143, print one parseable partial envelope preserving all acknowledged test/run pairs, keep terminal results already obtained, identify accepted-but-unfinished members honestly, and print concrete reattach/cancel commands for those unfinished IDs. Preserve existing detach behavior; do not auto-cancel ordinary public-target runs.

Seven executed cases and controls

Scenario Exit stdout Observed
Testlist completes normally 0 640 bytes All 3 run IDs and passed results retained
Same list with --no-wait 0 378 bytes All 3 accepted run IDs retained; no poll
SIGINT during first pending poll 130 0 bytes All 3 accepted IDs missing
SIGINT after first member passed 130 0 bytes Passed result and all accepted IDs missing
SIGTERM after first member failed 143 0 bytes Failed result and all accepted IDs missing
Same SIGINT in text mode, redirected streams 130 0 bytes No concrete recovery IDs in either stream
test wait run_a run_b run_c, same interruption 130 271 bytes Passed result preserved; run_b/run_c running; concrete resume hint

All assertions passed again after a fresh build. Each process exited normally without the fixture watchdog; no SIGKILL or mocked clock/sleep was needed. Text mode was tested with redirected/non-TTY streams, not claimed as interactive ticker testing.

Existing testlist.spec.ts, interrupt.test.ts, render-error.test.ts: 111 tests passed, 3 files, 5.94 seconds. Product source is unchanged. These green existing tests do not cover the testlist polling catch together with the installed signal handler.

Source diagnosis

A contained fix should handle the non-tunnel interrupted fan-out using the accepted response plus settled member results, retaining the conventional signal exit and avoiding duplicate output on the existing tunnel path. This report does not request persistence or change verdict/exit-code policy.

Duplicate check

Searched open and closed Issues and PRs for testlist/test-list with interrupt, SIGINT, detach, receipt, partial, run IDs and runTestlistRunCore; also read related reports and reviews.

No matching report or merged fix was found for this testlist-specific catch.

Self-contained reproduction

After npm ci && npm run build, save as reproduce-testlist-interrupt-receipts.mjs and run:

node reproduce-testlist-interrupt-receipts.mjs /absolute/path/to/testsprite-cli

Use Node 22 on macOS/Linux. The script creates and removes a disposable temp directory, isolates CLI home lookup to it, blocks all fetches except the in-process fixture hostname, sends signals only to the CLI subprocesses it started, and prints JSON evidence. The assertions intentionally confirm the current bug and its normal controls.

Executable seven-case reproduction
import assert from 'node:assert/strict';
import {spawn} from 'node:child_process';
import {mkdtemp,readFile,writeFile,rm} from 'node:fs/promises';
import {tmpdir} from 'node:os';
import {join,resolve,dirname} from 'node:path';

// Real compiled CLI subprocess + real OS signals. Only HTTP is replaced.
// No actual service, jobs, credentials or credits are involved.
const root=resolve(process.argv[2]??'work/testsprite-cli');
const dir=await mkdtemp(join(tmpdir(),'testsprite-testlist-interrupt-'));
const preload=join(dir,'fixture.mjs');
await writeFile(preload,`
import assert from 'node:assert/strict';
import {writeFileSync} from 'node:fs';
import os from 'node:os';
import {syncBuiltinESMExports} from 'node:module';
os.homedir=()=>process.env.REPRO_HOME;syncBuiltinESMExports();
const trace=[],s=JSON.parse(process.env.REPRO_CASE),ids=['a','b','c'];
const stamp='2026-10-17T00:00:00Z';
process.on('exit',()=>writeFileSync(process.env.REPRO_TRACE,JSON.stringify(trace)));
const json=(body,status=200)=>new Response(JSON.stringify(body),{status,headers:{'content-type':'application/json'}});
globalThis.fetch=async(input,init={})=>{
 const u=new URL(typeof input==='string'?input:input.url??input.href),method=init.method??'GET';
 assert.equal(u.hostname,'mock.example.com');
 const path=u.pathname.replace('/api/cli/v1',''),row={method,path,at:Date.now()};trace.push(row);
 if(path==='/testlist/list_fixture'&&method==='GET')return json({
  id:'list_fixture',name:'Fixture list',orgId:'org_fixture',caseCount:3,
  projectEnvironments:[{projectId:'project_fixture',environmentName:'prod'}],lastExecutionId:null,
  createdAt:stamp,updatedAt:stamp,lastExecutionCreated:null,
  stats:{total:3,passed:3,failed:0,blocked:0,running:0},
  cases:ids.map(id=>({testId:'test_'+id,type:'frontend',projectId:'project_fixture',projectName:'Fixture project',title:id,category:null,priority:null,modified:null,status:'passed'}))});
 if(path==='/projects/project_fixture/env')return json({environments:[{name:'prod',url:'https://example.com',isDefault:true,isTemporary:false}]});
 if(path==='/testlist/list_fixture/run'&&method==='POST'){
  const response={accepted:ids.map(id=>({testId:'test_'+id,runId:'run_'+id,enqueuedAt:stamp})),conflicts:[],deferred:[]};
  row.body=JSON.parse(String(init.body));row.response=response;return json(response,202);
 }
 const match=/^\\/runs\\/run_([abc])$/.exec(path);
 if(match&&method==='GET'){
  const id=match[1];
  if(s.signal&&id===(s.beforeAny?'a':'b')){
   row.pending=true;assert.ok(init.signal,'long poll must carry AbortSignal');
   process.send({checkpoint:'poll-pending',runId:'run_'+id});
   return new Promise((resolve,reject)=>{
    const timer=setTimeout(()=>reject(new Error('Fixture watchdog: no signal received')),10000);
    const onAbort=()=>{clearTimeout(timer);row.aborted=true;reject(init.signal.reason??new DOMException('Aborted','AbortError'));};
    if(init.signal.aborted)onAbort();else init.signal.addEventListener('abort',onAbort,{once:true});
   });
  }
  const failed=id==='a'&&s.firstFailed;row.returnedStatus=failed?'failed':'passed';
  return json({runId:'run_'+id,testId:'test_'+id,projectId:'project_fixture',userId:'fixture',status:row.returnedStatus,
   source:'cli',createdAt:stamp,startedAt:stamp,finishedAt:'2026-10-17T00:00:01Z',codeVersion:'v1',targetUrl:'https://example.com',createdFrom:'cli',
   failedStepIndex:failed?0:null,failureKind:failed?'assertion':null,error:failed?'Fixture assertion failed':null,videoUrl:null,
   stepSummary:{total:1,completed:1,passedCount:failed?0:1,failedCount:failed?1:0}});
 }
 row.unexpected=true;throw Error('Unexpected fixture route '+method+' '+path);
};
`);

const scenarios=[
 {name:'testlist-completes-control',expectedExit:0},
 {name:'testlist-no-wait-receipt-control',noWait:true,expectedExit:0},
 {name:'testlist-sigint-before-any-result',signal:'SIGINT',beforeAny:true,expectedExit:130},
 {name:'testlist-sigint-after-passed-member',signal:'SIGINT',expectedExit:130},
 {name:'testlist-sigterm-after-failed-member',signal:'SIGTERM',firstFailed:true,expectedExit:143},
 {name:'testlist-text-sigint',signal:'SIGINT',text:true,expectedExit:130},
 {name:'multi-wait-sigint-control',signal:'SIGINT',waitMany:true,expectedExit:130},
];
const results=[];
try{
 for(const s of scenarios){
  const tracePath=join(dir,s.name+'.json');
  const args=['--output',s.text?'text':'json',...(s.waitMany?['test','wait','run_a','run_b','run_c']:['testlist','run','list_fixture']),
   ...(s.waitMany?[]:[s.noWait?'--no-wait':'--wait']),...(!s.noWait?['--timeout','10','--max-concurrency','1']:[])];
  const start=performance.now();
  const child=spawn(process.execPath,['--import',preload,join(root,'dist/index.js'),...args],{
   cwd:dir,stdio:['ignore','pipe','pipe','ipc'],env:{PATH:dirname(process.execPath)+':/usr/bin:/bin',
    TESTSPRITE_API_KEY:'sk-user-offline-fixture',TESTSPRITE_API_URL:'https://mock.example.com',TESTSPRITE_NO_TELEMETRY:'1',TESTSPRITE_NO_UPDATE_NOTIFIER:'1',
    REPRO_HOME:dir,REPRO_TRACE:tracePath,REPRO_CASE:JSON.stringify(s)}});
  let stdout='',stderr='',signalAt=null,checkpoint=null,watchdog=false;
  child.stdout.on('data',b=>stdout+=b);child.stderr.on('data',b=>stderr+=b);
  child.on('message',m=>{if(signalAt===null){checkpoint=m;setTimeout(()=>{signalAt=performance.now()-start;child.kill(s.signal);},20);}});
  const timer=setTimeout(()=>{watchdog=true;child.kill('SIGKILL');},15000);
  const ended=await new Promise((res,rej)=>{child.on('error',rej);child.on('close',(code,signal)=>res({code,signal}));});clearTimeout(timer);
  const trace=JSON.parse(await readFile(tracePath,'utf8'));
  assert.equal(watchdog,false,s.name);assert.equal(ended.signal,null,s.name);assert.equal(ended.code,s.expectedExit,stderr);
  assert.ok(!trace.some(x=>x.unexpected),s.name+': unexpected HTTP');
  const dispatch=trace.find(x=>x.path==='/testlist/list_fixture/run');
  if(s.waitMany)assert.equal(dispatch,undefined);else assert.equal(dispatch.response.accepted.length,3);
  let payload=null;if(stdout.trim()&&!s.text)payload=JSON.parse(stdout);
  if(s.signal){
   assert.ok(checkpoint&&signalAt!==null);assert.ok(trace.some(x=>x.aborted));
   if(!s.beforeAny)assert.equal(trace.find(x=>x.path==='/runs/run_a').returnedStatus,s.firstFailed?'failed':'passed');
   if(s.waitMany){
    assert.deepEqual(payload.results.map(x=>x.runId),['run_a','run_b','run_c']);
    assert.equal(payload.results[0].status,'passed');assert.equal(payload.results[1].status,'running');
    assert.ok(stderr.includes('testsprite test wait run_b run_c'));
   }else{
    assert.equal(stdout,'','current bug: accepted IDs and partial result disappear');
    for(const id of ['run_a','run_b','run_c'])assert.ok(!stderr.includes(id),'current bug: concrete recovery IDs absent');
   }
  }else{
   assert.deepEqual(payload.accepted.map(x=>x.runId),['run_a','run_b','run_c']);
   if(!s.noWait)assert.ok(payload.accepted.every(x=>x.status==='passed'));
  }
  results.push({name:s.name,args,...ended,elapsedMs:performance.now()-start,signalAt,checkpoint,watchdog,stdout,stderr,payload,trace});
 }
 console.log(JSON.stringify({baseline:'988c0355bb05631e9c6c1785e443937ff2a88c0e',node:process.version,platform:process.platform,scope:'offline compiled CLI subprocesses; actual SIGINT/SIGTERM; HTTP fixtures only',results},null,2));
}finally{await rm(dir,{recursive:true,force:true});}

Confirmations

  • I have searched existing issues and this is not a duplicate.
  • I have provided my Discord identity above for reward coordination.

Activity

  1. zeshi-du commented on Oct 20, 2026

    @zeshi-du
    Contributor

    Thanks @luckyzd — confirmed: in v0.15.0 the non-tunnel poll catch in testlist run --wait rethrows without printing, while the INTERRUPTED hint promises partial JSON on stdout. Accepted.

    Scope:

    • Change only the poll fan-out catch in src/commands/testlist.ts, only for InterruptError on the non-tunnel path. Leave the tunnel branch untouched so nothing prints twice.
    • Follow runTestRunAll's fan-out catch in test.ts: print one parseable envelope (settled results as-is, status: "running" for unfinished members, plus conflicts, deferred and the other normal-path fields), then rethrow. If runs are unfinished, also print a stderr line with their run IDs and the re-attach and cancel commands (write a small local equivalent of the unexported interruptDetachMessage).
    • Additive only: exit codes stay 130/143, no auto-cancel, no persistence.
    • Don't touch test.ts, poll.ts or interrupt.ts, and don't refactor shared code.
    • Tests: SIGINT and SIGTERM, settled/unsettled mix, unchanged tunnel output, and one that fails without your change.

    Comment /assign to claim it.

  2. added
    acceptedTriaged and accepted — a PR for this will be reviewed
    and removed
    needs-triageNew issue/PR awaiting first triage
    on Oct 20, 2026
  3. luckyzd commented on Oct 20, 2026

    @luckyzd
    Author

    /assign

  4. testsprite-hob commented on Oct 20, 2026

    @testsprite-hob

    Assigned to @luckyzd. Thanks for taking this on. (2 more slots available.)

    Your open assigned issues (1/3):
    (none)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

acceptedTriaged and accepted — a PR for this will be reviewedhackathonCLI hackathon submissionsin-progressAssigned and actively being worked on

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions