Skip to content

[Hackathon] fix(test): retain new run receipts when auto-resume metadata is interrupted #533

Open

Description

@walter-w-white

Batch --wait loses acknowledged run IDs when interrupted during auto-resume metadata reads

Verified against 988c035, CLI 0.14.0, Linux / Node 22.23.3. Contact @walter-w-white on GitHub; no Discord identity.

test run --all --wait can successfully dispatch a new run while another test is already running. If Ctrl-C arrives while the CLI reads that existing run's metadata, it exits130 with empty stdout. The new run keeps executing, but its acknowledged run ID appears in neither stdout nor stderr. The same interruption during the subsequent polling stage correctly prints a partial receipt.

This prevents an agent or script from attaching to or cancelling the new run using the response it already received. It is recoverable through other listing/history paths; this report does not claim permanent server data loss or measure hosted billing impact.

Minimal sequence

  1. POST /tests/batch/run acknowledges accepted:[{testId:"test_new",runId:"run_new",enqueuedAt:...}] and conflicts:[{testId:"test_busy",currentRunId:"run_busy",reason:"run_in_flight"}].
  2. The CLI starts its preliminary GET /runs/run_busy to resolve the auto-resumed run's timestamp/source. Keep that ordinary response pending.
  3. Send SIGINT after150ms.
  4. Expected: one partial stdout receipt retaining at least acknowledged run_new, with honest unfinished state and exit130; preserve adopted-run identity as appropriate. Actual: stdout0bytes; stderr says Dispatched 1 test (1 conflict). and offers a generic test wait <runId>, without run_new.

SIGTERM during a partial metadata body similarly exits143 with no receipt. Text mode also omits the new ID. No extra trigger or cancellation occurs.

Verification

The standalone script below uses only Node builtins, an isolated synthetic home and a loopback server. It never contacts TestSprite or consumes credits. From a built checkout, run:

node reproduce.mjs /absolute/path/to/testsprite-cli/dist/index.js ./receipt-evidence
# exit1 means the receipt regression reproduced; exit2 indicates fixture failure

It records six actual subprocesses with exact arguments, fixture responses, request/close/signal times, stdout/stderr and exit codes:

Scenario Result
Normal one accepted + one resumable conflict both IDs retained, exit0
SIGINT during metadata before headers empty stdout, exit130
SIGTERM during metadata partial body empty stdout, exit143
SIGINT after metadata, during actual polling correct partial IDs, exit130
Metadata SIGINT in text mode no accepted run ID, exit130
Later test wait run_new correct run result, exit0

All fixture checks pass; three receipt assertions fail. A separate16-scenario native CLI matrix also covers24 simultaneous metadata reads, max polling widths1/2, duplicates,403,503 replay and shared deadline. Those paths retain IDs and correct verdicts unless interrupted before polling.

Scope and prior work

The preliminary Promise.all in runTestRunAll (test.ts:10320) rethrows InterruptError before reaching the poll fanout's partial-output catch (:10613). The HTTP/signal handler works; this lifecycle stage lacks receipt handling. A focused fix can extend existing graceful-detach accounting to this stage without introducing new local state.

Compared current OPEN and CLOSED threads, including #130, #153, #154/#135, #178 and #185. Credit the existing graceful signal and partial-result work. #80's latest maintainer discussion explicitly concerns a persistent journal after terminal loss; this proposal only preserves the already-acknowledged response in the current process. Larger preliminary metadata fanout is not reported as a concurrency violation: --max-concurrency explicitly bounds terminal polls.

No production patch is included before triage/assignment. The regression should cover real signal propagation while metadata is pending, retain normal and actual-poll interrupt behavior, and avoid cancelling adopted runs. Hosted frequency and Windows behavior are unmeasured.

Standalone reproducer
// Node22. Usage: node reproduce.mjs /absolute/path/to/upstream/dist/index.js [output-dir]
// Synthetic loopback only. Exit1 = regression reproduced; exit2 = fixture failure.
import {createServer} from 'node:http';
import {spawn,spawnSync} from 'node:child_process';
import {mkdirSync,writeFileSync,readFileSync} from 'node:fs';
import {resolve,join,dirname} from 'node:path';
import {createHash} from 'node:crypto';
const binary=resolve(process.argv[2]??'upstream/dist/index.js');
const out=resolve(process.argv[3]??'candidates/batch-resume-interrupt-loses-receipts/results');mkdirSync(out,{recursive:true});
const save=(p,x)=>writeFileSync(p,typeof x==='string'?x:JSON.stringify(x,null,2)+'\n');
const date='2026-10-11T20:00:00Z';
const row=id=>({runId:id,testId:id==='run_new'?'test_new':'test_busy',projectId:'project_lab',userId:'lab',status:'passed',source:'cli',createdAt:date,startedAt:date,finishedAt:date,codeVersion:'v1',targetUrl:'https://fixture.invalid',createdFrom:'cli',failedStepIndex:null,failureKind:null,error:null,videoUrl:null,stepSummary:{total:1,completed:1,passedCount:1,failedCount:0}});
const scenarios=[
 {name:'normal-two-members'},
 {name:'metadata-before-headers-SIGINT',stage:'metadata',signal:'SIGINT',exit:130},
 {name:'metadata-partial-body-SIGTERM',stage:'metadata',signal:'SIGTERM',exit:143,partial:true},
 {name:'poll-stage-SIGINT-control',stage:'poll',signal:'SIGINT',exit:130},
 {name:'metadata-before-headers-text',stage:'metadata',signal:'SIGINT',exit:130,text:true},
 {name:'followup-wait-known-new-id',wait:true},
];
const results=[];
for(const s of scenarios){
 const dir=join(out,s.name);mkdirSync(dir,{recursive:true});const userDir=join(dir,'fixture-user');mkdirSync(userDir,{recursive:true});const preload=join(dir,'isolate.mjs');
 save(preload,"import os from 'node:os';import {syncBuiltinESMExports} from 'node:module';os.homedir=()=>process.env.LAB_USER;syncBuiltinESMExports();const f=globalThis.fetch;globalThis.fetch=(u,o)=>{const v=new URL(typeof u==='string'||u instanceof URL?u:u.url);if(v.origin!==process.env.LAB_API)throw Error('Non-fixture origin blocked');return f(u,o);};\n");
 const requests=[],events=[];const t0=performance.now();let child,timer,signalled=false;
 const time=()=>Math.round(performance.now()-t0);
 const batch={accepted:[{testId:'test_new',runId:'run_new',enqueuedAt:date}],conflicts:[{testId:'test_busy',currentRunId:'run_busy',reason:'run_in_flight'}],deferred:[],skippedFrontend:[],skippedIntegration:[]};
 const server=createServer(async(req,res)=>{
  const chunks=[];for await(const b of req)chunks.push(b);const raw=Buffer.concat(chunks).toString();const url=new URL(req.url,'http://127.0.0.1');const path=url.pathname.replace('/api/cli/v1','');const q={atMs:time(),method:req.method,path,query:Object.fromEntries(url.searchParams),body:raw?JSON.parse(raw):null};requests.push(q);res.once('close',()=>events.push({atMs:time(),event:'close',path,finished:res.writableFinished}));
  let body;
  if(path==='/projects/project_lab/env')body={environments:[]};
  else if(path==='/tests/batch/run')body=batch;
  else if(path.startsWith('/runs/')){
   const stage=url.searchParams.has('waitSeconds')?'poll':'metadata';
   if(s.stage===stage){if(s.partial){res.writeHead(200,{'content-type':'application/json'});res.write('{"runId":"run_busy"');}if(!signalled){signalled=true;timer=setTimeout(()=>{events.push({atMs:time(),event:'signal',signal:s.signal});child.kill(s.signal);},150);}return;}
   body=row(path.slice('/runs/'.length));
  }else{q.fixtureError='Unexpected route';res.writeHead(500);res.end('{}');return;}
  q.response=body;res.writeHead(200,{'content-type':'application/json'});res.end(JSON.stringify(body));
 });
 await new Promise(r=>server.listen(0,'127.0.0.1',r));const api=`http://127.0.0.1:${server.address().port}`;
 const args=['--import',preload,binary,...(s.wait?['test','wait','run_new']:['test','run','--all','--project','project_lab','--wait','--max-concurrency','1']),'--timeout','5','--output',s.text?'text':'json','--endpoint-url',api];
 const env={PATH:'/usr/bin:/bin',CI:'true',NO_COLOR:'1',TESTSPRITE_API_KEY:'sk-user-lab-placeholder-not-a-real-key',TESTSPRITE_NO_UPDATE_NOTIFIER:'1',TESTSPRITE_NO_TELEMETRY:'1',LAB_USER:userDir,LAB_API:api};for(const k of ['HOME','home','CODEX_HOME'])if(process.env[k]!==undefined)env[k]=process.env[k];
 save(join(dir,'input.json'),{scenario:s,batch});save(join(dir,'command.json'),{argv:[process.execPath,...args],cwd:dir,syntheticEnv:{...env,HOME:'inherited unchanged',home:'inherited if present',CODEX_HOME:'inherited if present'}});
 child=spawn(process.execPath,args,{env,cwd:dir,stdio:['ignore','pipe','pipe']});let stdout='',stderr='',watchdog=false;
 child.stdout.on('data',b=>stdout+=b);child.stderr.on('data',b=>stderr+=b);const killer=setTimeout(()=>{watchdog=true;child.kill('SIGKILL');},8000);
 const exit=await new Promise((r,j)=>{child.on('error',j);child.on('close',(code,signal)=>r({code,signal}));});clearTimeout(killer);clearTimeout(timer);events.push({event:'exit',atMs:time(),...exit});server.closeAllConnections();await new Promise(r=>server.close(r));
 let json;try{json=JSON.parse(stdout);}catch{}
 const receipt=s.text?stdout.includes('run_new'):(s.wait?json?.runId==='run_new':json?.accepted?.some(x=>x.runId==='run_new'));
 const fixturesPass=exit.code===(s.exit??0)&&!watchdog&&requests.every(x=>!x.fixtureError)&&requests.filter(x=>x.method==='POST').length===(s.wait?0:1)&&requests.every(x=>x.method!=='DELETE');
 const r={scenario:s.name,exit,watchdog,elapsedMs:time(),fixtureChecksPass:fixturesPass,expected:{exit:s.exit??0,newRunReceipt:true},actual:{receipt:Boolean(receipt),stdoutBytes:stdout.length,stderrContainsNewRunId:stderr.includes('run_new')}};
 for(const [name,data] of Object.entries({'stdout.txt':stdout,'stderr.txt':stderr,'requests.json':requests,'events.json':events,'result.json':r}))save(join(dir,name),data);
 results.push(r);console.log(JSON.stringify(r));
}
const fixtureFailures=results.filter(x=>!x.fixtureChecksPass),regressions=results.filter(x=>!x.actual.receipt);const summary={node:process.version,sha:spawnSync('git',['-C',dirname(dirname(binary)),'rev-parse','HEAD'],{encoding:'utf8'}).stdout.trim(),binarySha256:createHash('sha256').update(readFileSync(binary)).digest('hex'),cases:results.length,fixtureFailures,regressions,results};save(join(out,'summary.json'),summary);process.exitCode=fixtureFailures.length?2:regressions.length?1:0;

Current duplicate review

Compared the current open and closed reports again on 2026-10-19. #426 is the outer create-batch retry-sleep signal path; #484 is the testlist polling catch; #405/PR #410 concern later rerun dispatch chunks. This report is the preliminary auto-resume metadata read in runTestRunAll, after a fresh run has been acknowledged and before polling begins. The succeeding polling-interrupt control keeps its receipt.

Contribution contact

Please contact @walter-w-white on this issue. No Discord identity is provided. AI assistance was used for source investigation, offline reproduction and drafting. This issue reports one existing CLI behavior; no production fix or assignment is claimed.

Pre-submission revalidation

Revalidated on 2026-10-19 with six actual CLI subprocesses. All fixture checks passed; the three interrupted-metadata cases still lose the receipt, while the normal, polling-interrupt and later-wait controls retain it. Tested upstream remains 988c0355bb05631e9c6c1785e443937ff2a88c0e (CLI 0.14.0, Linux, Node 22.23.3). All reproduction traffic is local; no hosted testing credits were used.

I intend to implement this fix after maintainer acceptance and assignment.

Please contact @walter-w-white on this GitHub issue or PR; no Discord identity is provided.

Activity

  1. zeshi-du commented on Oct 20, 2026

    @zeshi-du
    Contributor

    Thanks @walter-w-white for the careful report and reproducer. Accepted; please /assign yourself to claim it.

    Scope: only the stage in runTestRunAll (src/commands/test.ts) that reads metadata for in-flight conflict runs before polling. On SIGINT/SIGTERM there, print the already-acknowledged run IDs. Cover both cases:

    • Request in flight: InterruptError is rethrown by the per-run catches, with no printing catch above.
    • None in flight: the scope isn't armed (see src/lib/interrupt.ts), so the first signal exits 130/143 directly and nothing is thrown.

    Arming a scope around the stage and printing from its catch covers both; printing the accepted receipt before the stage also works.

    Constraints:

    • Reuse the poll-stage catch's partial-output shape (accepted as running, conflicts, deferred, skipped); JSON stays additive, text mode included.
    • Keep exit codes 130/143, never cancel runs (new or adopted), and leave pre-read concurrency and the resume/refuse gate alone.
    • Add a real-signal regression test for both cases that fails before the fix.
    • Keep the PR to test run --all.
  2. added
    acceptedTriaged and accepted — a PR for this will be reviewed
    on Oct 20, 2026
  3. walter-w-white commented on Oct 20, 2026

    @walter-w-white
    Author

    /assign

  4. testsprite-hob commented on Oct 20, 2026

    @testsprite-hob

    Assigned to @walter-w-white. Thanks for taking this on. (2 more slots available.)

    Your open assigned issues (1/3):
    (none)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

acceptedTriaged and accepted — a PR for this will be reviewedin-progressAssigned and actively being worked on

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions