Repository navigation
[Hackathon] fix(test): retain new run receipts when auto-resume metadata is interrupted #533
Description
Batch --wait loses acknowledged run IDs when interrupted during auto-resume metadata reads
Verified against 988c035, CLI 0.14.0, Linux / Node 22.23.3. Contact @walter-w-white on GitHub; no Discord identity.
test run --all --wait can successfully dispatch a new run while another test is already running. If Ctrl-C arrives while the CLI reads that existing run's metadata, it exits130 with empty stdout. The new run keeps executing, but its acknowledged run ID appears in neither stdout nor stderr. The same interruption during the subsequent polling stage correctly prints a partial receipt.
This prevents an agent or script from attaching to or cancelling the new run using the response it already received. It is recoverable through other listing/history paths; this report does not claim permanent server data loss or measure hosted billing impact.
Minimal sequence
POST /tests/batch/runacknowledgesaccepted:[{testId:"test_new",runId:"run_new",enqueuedAt:...}]andconflicts:[{testId:"test_busy",currentRunId:"run_busy",reason:"run_in_flight"}].- The CLI starts its preliminary
GET /runs/run_busyto resolve the auto-resumed run's timestamp/source. Keep that ordinary response pending. - Send SIGINT after150ms.
- Expected: one partial stdout receipt retaining at least acknowledged
run_new, with honest unfinished state and exit130; preserve adopted-run identity as appropriate. Actual: stdout0bytes; stderr saysDispatched 1 test (1 conflict).and offers a generictest wait <runId>, withoutrun_new.
SIGTERM during a partial metadata body similarly exits143 with no receipt. Text mode also omits the new ID. No extra trigger or cancellation occurs.
Verification
The standalone script below uses only Node builtins, an isolated synthetic home and a loopback server. It never contacts TestSprite or consumes credits. From a built checkout, run:
node reproduce.mjs /absolute/path/to/testsprite-cli/dist/index.js ./receipt-evidence
# exit1 means the receipt regression reproduced; exit2 indicates fixture failureIt records six actual subprocesses with exact arguments, fixture responses, request/close/signal times, stdout/stderr and exit codes:
| Scenario | Result |
|---|---|
| Normal one accepted + one resumable conflict | both IDs retained, exit0 |
| SIGINT during metadata before headers | empty stdout, exit130 |
| SIGTERM during metadata partial body | empty stdout, exit143 |
| SIGINT after metadata, during actual polling | correct partial IDs, exit130 |
| Metadata SIGINT in text mode | no accepted run ID, exit130 |
Later test wait run_new |
correct run result, exit0 |
All fixture checks pass; three receipt assertions fail. A separate16-scenario native CLI matrix also covers24 simultaneous metadata reads, max polling widths1/2, duplicates,403,503 replay and shared deadline. Those paths retain IDs and correct verdicts unless interrupted before polling.
Scope and prior work
The preliminary Promise.all in runTestRunAll (test.ts:10320) rethrows InterruptError before reaching the poll fanout's partial-output catch (:10613). The HTTP/signal handler works; this lifecycle stage lacks receipt handling. A focused fix can extend existing graceful-detach accounting to this stage without introducing new local state.
Compared current OPEN and CLOSED threads, including #130, #153, #154/#135, #178 and #185. Credit the existing graceful signal and partial-result work. #80's latest maintainer discussion explicitly concerns a persistent journal after terminal loss; this proposal only preserves the already-acknowledged response in the current process. Larger preliminary metadata fanout is not reported as a concurrency violation: --max-concurrency explicitly bounds terminal polls.
No production patch is included before triage/assignment. The regression should cover real signal propagation while metadata is pending, retain normal and actual-poll interrupt behavior, and avoid cancelling adopted runs. Hosted frequency and Windows behavior are unmeasured.
Standalone reproducer
// Node22. Usage: node reproduce.mjs /absolute/path/to/upstream/dist/index.js [output-dir]
// Synthetic loopback only. Exit1 = regression reproduced; exit2 = fixture failure.
import {createServer} from 'node:http';
import {spawn,spawnSync} from 'node:child_process';
import {mkdirSync,writeFileSync,readFileSync} from 'node:fs';
import {resolve,join,dirname} from 'node:path';
import {createHash} from 'node:crypto';
const binary=resolve(process.argv[2]??'upstream/dist/index.js');
const out=resolve(process.argv[3]??'candidates/batch-resume-interrupt-loses-receipts/results');mkdirSync(out,{recursive:true});
const save=(p,x)=>writeFileSync(p,typeof x==='string'?x:JSON.stringify(x,null,2)+'\n');
const date='2026-10-11T20:00:00Z';
const row=id=>({runId:id,testId:id==='run_new'?'test_new':'test_busy',projectId:'project_lab',userId:'lab',status:'passed',source:'cli',createdAt:date,startedAt:date,finishedAt:date,codeVersion:'v1',targetUrl:'https://fixture.invalid',createdFrom:'cli',failedStepIndex:null,failureKind:null,error:null,videoUrl:null,stepSummary:{total:1,completed:1,passedCount:1,failedCount:0}});
const scenarios=[
{name:'normal-two-members'},
{name:'metadata-before-headers-SIGINT',stage:'metadata',signal:'SIGINT',exit:130},
{name:'metadata-partial-body-SIGTERM',stage:'metadata',signal:'SIGTERM',exit:143,partial:true},
{name:'poll-stage-SIGINT-control',stage:'poll',signal:'SIGINT',exit:130},
{name:'metadata-before-headers-text',stage:'metadata',signal:'SIGINT',exit:130,text:true},
{name:'followup-wait-known-new-id',wait:true},
];
const results=[];
for(const s of scenarios){
const dir=join(out,s.name);mkdirSync(dir,{recursive:true});const userDir=join(dir,'fixture-user');mkdirSync(userDir,{recursive:true});const preload=join(dir,'isolate.mjs');
save(preload,"import os from 'node:os';import {syncBuiltinESMExports} from 'node:module';os.homedir=()=>process.env.LAB_USER;syncBuiltinESMExports();const f=globalThis.fetch;globalThis.fetch=(u,o)=>{const v=new URL(typeof u==='string'||u instanceof URL?u:u.url);if(v.origin!==process.env.LAB_API)throw Error('Non-fixture origin blocked');return f(u,o);};\n");
const requests=[],events=[];const t0=performance.now();let child,timer,signalled=false;
const time=()=>Math.round(performance.now()-t0);
const batch={accepted:[{testId:'test_new',runId:'run_new',enqueuedAt:date}],conflicts:[{testId:'test_busy',currentRunId:'run_busy',reason:'run_in_flight'}],deferred:[],skippedFrontend:[],skippedIntegration:[]};
const server=createServer(async(req,res)=>{
const chunks=[];for await(const b of req)chunks.push(b);const raw=Buffer.concat(chunks).toString();const url=new URL(req.url,'http://127.0.0.1');const path=url.pathname.replace('/api/cli/v1','');const q={atMs:time(),method:req.method,path,query:Object.fromEntries(url.searchParams),body:raw?JSON.parse(raw):null};requests.push(q);res.once('close',()=>events.push({atMs:time(),event:'close',path,finished:res.writableFinished}));
let body;
if(path==='/projects/project_lab/env')body={environments:[]};
else if(path==='/tests/batch/run')body=batch;
else if(path.startsWith('/runs/')){
const stage=url.searchParams.has('waitSeconds')?'poll':'metadata';
if(s.stage===stage){if(s.partial){res.writeHead(200,{'content-type':'application/json'});res.write('{"runId":"run_busy"');}if(!signalled){signalled=true;timer=setTimeout(()=>{events.push({atMs:time(),event:'signal',signal:s.signal});child.kill(s.signal);},150);}return;}
body=row(path.slice('/runs/'.length));
}else{q.fixtureError='Unexpected route';res.writeHead(500);res.end('{}');return;}
q.response=body;res.writeHead(200,{'content-type':'application/json'});res.end(JSON.stringify(body));
});
await new Promise(r=>server.listen(0,'127.0.0.1',r));const api=`http://127.0.0.1:${server.address().port}`;
const args=['--import',preload,binary,...(s.wait?['test','wait','run_new']:['test','run','--all','--project','project_lab','--wait','--max-concurrency','1']),'--timeout','5','--output',s.text?'text':'json','--endpoint-url',api];
const env={PATH:'/usr/bin:/bin',CI:'true',NO_COLOR:'1',TESTSPRITE_API_KEY:'sk-user-lab-placeholder-not-a-real-key',TESTSPRITE_NO_UPDATE_NOTIFIER:'1',TESTSPRITE_NO_TELEMETRY:'1',LAB_USER:userDir,LAB_API:api};for(const k of ['HOME','home','CODEX_HOME'])if(process.env[k]!==undefined)env[k]=process.env[k];
save(join(dir,'input.json'),{scenario:s,batch});save(join(dir,'command.json'),{argv:[process.execPath,...args],cwd:dir,syntheticEnv:{...env,HOME:'inherited unchanged',home:'inherited if present',CODEX_HOME:'inherited if present'}});
child=spawn(process.execPath,args,{env,cwd:dir,stdio:['ignore','pipe','pipe']});let stdout='',stderr='',watchdog=false;
child.stdout.on('data',b=>stdout+=b);child.stderr.on('data',b=>stderr+=b);const killer=setTimeout(()=>{watchdog=true;child.kill('SIGKILL');},8000);
const exit=await new Promise((r,j)=>{child.on('error',j);child.on('close',(code,signal)=>r({code,signal}));});clearTimeout(killer);clearTimeout(timer);events.push({event:'exit',atMs:time(),...exit});server.closeAllConnections();await new Promise(r=>server.close(r));
let json;try{json=JSON.parse(stdout);}catch{}
const receipt=s.text?stdout.includes('run_new'):(s.wait?json?.runId==='run_new':json?.accepted?.some(x=>x.runId==='run_new'));
const fixturesPass=exit.code===(s.exit??0)&&!watchdog&&requests.every(x=>!x.fixtureError)&&requests.filter(x=>x.method==='POST').length===(s.wait?0:1)&&requests.every(x=>x.method!=='DELETE');
const r={scenario:s.name,exit,watchdog,elapsedMs:time(),fixtureChecksPass:fixturesPass,expected:{exit:s.exit??0,newRunReceipt:true},actual:{receipt:Boolean(receipt),stdoutBytes:stdout.length,stderrContainsNewRunId:stderr.includes('run_new')}};
for(const [name,data] of Object.entries({'stdout.txt':stdout,'stderr.txt':stderr,'requests.json':requests,'events.json':events,'result.json':r}))save(join(dir,name),data);
results.push(r);console.log(JSON.stringify(r));
}
const fixtureFailures=results.filter(x=>!x.fixtureChecksPass),regressions=results.filter(x=>!x.actual.receipt);const summary={node:process.version,sha:spawnSync('git',['-C',dirname(dirname(binary)),'rev-parse','HEAD'],{encoding:'utf8'}).stdout.trim(),binarySha256:createHash('sha256').update(readFileSync(binary)).digest('hex'),cases:results.length,fixtureFailures,regressions,results};save(join(out,'summary.json'),summary);process.exitCode=fixtureFailures.length?2:regressions.length?1:0;Current duplicate review
Compared the current open and closed reports again on 2026-10-19. #426 is the outer create-batch retry-sleep signal path; #484 is the testlist polling catch; #405/PR #410 concern later rerun dispatch chunks. This report is the preliminary auto-resume metadata read in runTestRunAll, after a fresh run has been acknowledged and before polling begins. The succeeding polling-interrupt control keeps its receipt.
Contribution contact
Please contact @walter-w-white on this issue. No Discord identity is provided. AI assistance was used for source investigation, offline reproduction and drafting. This issue reports one existing CLI behavior; no production fix or assignment is claimed.
Pre-submission revalidation
Revalidated on 2026-10-19 with six actual CLI subprocesses. All fixture checks passed; the three interrupted-metadata cases still lose the receipt, while the normal, polling-interrupt and later-wait controls retain it. Tested upstream remains 988c0355bb05631e9c6c1785e443937ff2a88c0e (CLI 0.14.0, Linux, Node 22.23.3). All reproduction traffic is local; no hosted testing credits were used.
I intend to implement this fix after maintainer acceptance and assignment.
Please contact @walter-w-white on this GitHub issue or PR; no Discord identity is provided.
Activity
zeshi-du commentedon Oct 20, 2026
Thanks @walter-w-white for the careful report and reproducer. Accepted; please /assign yourself to claim it.
Scope: only the stage in
runTestRunAll(src/commands/test.ts) that reads metadata for in-flight conflict runs before polling. On SIGINT/SIGTERM there, print the already-acknowledged run IDs. Cover both cases:InterruptErroris rethrown by the per-run catches, with no printing catch above.src/lib/interrupt.ts), so the first signal exits 130/143 directly and nothing is thrown.Arming a scope around the stage and printing from its catch covers both; printing the accepted receipt before the stage also works.
Constraints:
running, conflicts, deferred, skipped); JSON stays additive, text mode included.test run --all.walter-w-white commentedon Oct 20, 2026
/assign
testsprite-hob commentedon Oct 20, 2026
Assigned to @walter-w-white. Thanks for taking this on. (2 more slots available.)
Your open assigned issues (1/3):
(none)