Repository navigation
[Hackathon] fix(test): batch retry sleep exits on signal before preserving acknowledged IDs #426
Description
create-batch loses acknowledged IDs on a real signal during outer retry sleep
Verified against 988c035, CLI 0.14.0, Linux / Node 22.23.3. Contact @jesse978 on GitHub; no Discord identity.
After test create-batch --run --output json creates tests, a trigger can return a transient429 and put the batch in its outer Retry-After sleep. If no other HTTP request or poll is active when Ctrl-C arrives, the process exits130 with empty stdout, discarding the acknowledged test IDs and any already accepted sibling run IDs. The same happens with SIGTERM (143) when --wait has already finished polling the accepted sibling.
This impedes automation recovering the newly created tests and attaching to or cancelling known runs. It does not cancel those runs. It is not permanent loss: history/list or the emitted creation replay key can support recovery, but the promised in-process partial result is missing.
Minimal scenario:
- Supply two ordinary frontend plans.
POST /tests/batchacknowledgesT_receipt_0andT_receipt_1. - Trigger0 acknowledges
R_T_receipt_0. Trigger1 returns429 with a typed RATE_LIMITED envelope andRetry-After:10. - Send SIGINT350ms after that refusal, while no request is in flight.
- Expected: one JSON partial result preserving both created IDs and acknowledged
R_T_receipt_0, followed by exit130. Actual: exit130, stdout0bytes, only a generic stderr suggestion using placeholder<runId>.
The self-contained script below reproduces this with Node 22 and a built CLI checkout; no lab files, TestSprite service or credits are needed. Save it as minimal.mjs and run node minimal.mjs /absolute/path/to/dist/index.js receipt. It saves the exact argv, input JSONL, server responses and request ledger, stdout/stderr, real signal time and exit code. Exit 1 indicates the reproduced receipt failure; exit 2 indicates a fixture failure.
The actual CLI invocation is test create-batch --plans <file> --run --max-concurrency 2 --output json --endpoint-url <loopback>. The broader seven-case matrix below also used --wait --timeout 5 for wait cases, with an isolated home lookup and native fetch restricted to loopback.
Seven actual CLI cases:
| Scenario | Result |
|---|---|
| Two normal accepted triggers | IDs retained, exit0 |
| One created test, trigger429, SIGINT | empty JSON stdout, exit130 |
| One accepted sibling plus trigger429, SIGINT | both test IDs and acknowledged run absent from stdout, exit130 |
| Accepted sibling polled passed, other trigger429, SIGTERM | empty stdout even with --wait, exit143 |
| 503 HTTP-internal backoff, SIGINT | creation receipt retained |
| Other sibling actively polling when SIGINT arrives | partial IDs/run retained |
| Text mode, same outer429 sleep | earlier create table preserves test IDs |
Source diagnosis: runBatchRun awaits sleepUntilOrInterrupt outside an armed/critical scope. installSignalHandlers synchronously aborts then immediately process.exits when disarmed with no critical operations. The promise rejection therefore never reaches the existing partial-result catch or withCreatedReceipt. In contrast, HTTP-internal retry sleeps remain in the tracked request lifetime, and active polls arm the shutdown controller.
test.create-batch-run.spec.ts:1330 already tests a disarmed backoff but calls shutdown.interrupt directly; it bypasses the installed process handler's immediate-exit branch. The recovery test at1744 injects InterruptError, also bypassing the real signal handler. A regression should exercise their composition while retaining the existing bounded repeated-signal escape behavior.
Duplicate review refreshed on 2026-10-16 across all 417 open and closed Issue/PR records and the relevant live discussions. #80's current maintainer discussion narrows it to a persistent journal after terminal loss, whereas this needs only the current invocation's existing catch to execute. #185 added the signal guard and is credited; it does not cover this outer-sleep/partial-receipt integration. #403/#406 handle caller cancellation inside HttpClient retry backoff; their current patch changes only src/lib/http.ts and its tests, while this failure occurs in create-batch's outer sleep after the request has settled. #405/#410 preserve rerun receipts after a later chunk fails through an ordinary catch; they do not change this process-signal lifetime. #154 concerns timeout classification in other polling pools, #6 scheduler concurrency, #267 no-wait success status. No new journal, billing behavior or rate-limit algorithm is proposed here. The self-contained receipt script was rerun against the stated SHA on 2026-10-16: valid fixture, exit 130, empty stdout, no watchdog.
No production change is included. After acceptance and actual assignment, scope the fix to the batch shutdown lifetime and add a regression that fails before the fix and passes afterward, followed by the full required checks. Hosted frequency and Windows behavior remain unmeasured.
Self-contained minimal reproducer
Save the following as minimal.mjs and run node minimal.mjs /absolute/path/to/dist/index.js receipt. Use Node22. It creates only a disposable directory below the current working directory, blocks non-loopback fetches, and uses no real credentials. Exit1 means the contract regression was reproduced; exit2 means the fixture failed.
// Usage: node minimal-reproduction.mjs /absolute/path/dist/index.js rate|receipt [output-dir]
import {createServer} from 'node:http';
import {spawn} from 'node:child_process';
import {mkdirSync,mkdtempSync,writeFileSync} from 'node:fs';
import {resolve,join} from 'node:path';
const cli=resolve(process.argv[2]),kind=process.argv[3],n=kind==='rate'?26:2;
if(!['rate','receipt'].includes(kind))throw Error('Select rate or receipt');
const dir=process.argv[4]?resolve(process.argv[4]):mkdtempSync(resolve('testsprite-repro-'));
mkdirSync(dir,{recursive:true});mkdirSync(join(dir,'fixture-user'),{recursive:true});
const save=(name,x)=>writeFileSync(join(dir,name),typeof x==='string'?x:JSON.stringify(x,null,2)+'\n');
const ids=Array.from({length:n},(_,i)=>'fixture_'+i),attempts=new Map(),requests=[];
const plans=ids.map(name=>({projectId:'fixture-project',type:'frontend',name,planSteps:[{type:'action',description:'open fixture'}]}));
save('plans.jsonl',plans.map(x=>JSON.stringify(x)).join('\n')+'\n');
save('isolate.mjs',`import os from 'node:os';import {syncBuiltinESMExports} from 'node:module';os.homedir=()=>process.env.REPRO_USER;syncBuiltinESMExports();const f=globalThis.fetch;globalThis.fetch=(i,o)=>{const u=new URL(typeof i==='string'||i instanceof URL?i:i.url);if(u.origin!==process.env.REPRO_ENDPOINT)throw Error('Nonfixture network blocked');return f(i,o);};`);
let child,signalAt=null;const started=performance.now();
const server=createServer(async(req,res)=>{
const chunks=[];for await(const b of req)chunks.push(b);
const path=new URL(req.url,'http://127.0.0.1').pathname.replace('/api/cli/v1','');
const row={ms:performance.now()-started,method:req.method,path,body:chunks.length?JSON.parse(Buffer.concat(chunks)):null};requests.push(row);
const reply=(status,body,headers={})=>{row.status=status;row.response=body;res.writeHead(status,{'content-type':'application/json',...headers});res.end(JSON.stringify(body));};
if(path.endsWith('/env'))return reply(200,{environments:[]});
if(path==='/tests/batch')return reply(201,{results:ids.map((testId,specIndex)=>({testId,specIndex,status:'created'})),summary:{total:n,created:n,failed:0}});
const m=/^\/tests\/(fixture_\d+)\/runs$/.exec(path);
if(m){const id=m[1],count=(attempts.get(id)??0)+1;attempts.set(id,count);
if(kind==='rate'&&count===1)return reply(503,{error:{code:'UNAVAILABLE',message:'fixture outage',requestId:'fixture'}});
if(kind==='receipt'&&id===ids[1]){reply(429,{error:{code:'RATE_LIMITED',message:'Run trigger rate limit exceeded: 60 triggers per minute per key.',requestId:'fixture'}},{'retry-after':'10'});setTimeout(()=>{signalAt=performance.now()-started;child.kill('SIGINT');},350);return;}
return reply(200,{runId:'run_'+id,status:'queued',enqueuedAt:'2026-10-11T00:00:00Z',codeVersion:'v1',targetUrl:'https://fixture.invalid'});
}
reply(404,{error:{code:'NOT_FOUND',message:'unexpected route',requestId:'fixture'}});
});
await new Promise(r=>server.listen(0,'127.0.0.1',r));
const endpoint=`http://127.0.0.1:${server.address().port}`;
const argv=['--import',join(dir,'isolate.mjs'),cli,'test','create-batch','--plans',join(dir,'plans.jsonl'),'--run','--max-concurrency',String(n),'--output','json','--endpoint-url',endpoint];
const env={PATH:process.env.PATH,CI:'true',NO_COLOR:'1',TESTSPRITE_API_KEY:'sk-user-fixture-not-a-real-key',TESTSPRITE_NO_TELEMETRY:'1',TESTSPRITE_NO_UPDATE_NOTIFIER:'1',REPRO_USER:join(dir,'fixture-user'),REPRO_ENDPOINT:endpoint};
for(const k of ['HOME','home','CODEX_HOME'])if(process.env[k]!==undefined)env[k]=process.env[k];
save('command.json',{argv:[process.execPath,...argv],cwd:dir,syntheticEnvironment:true});
child=spawn(process.execPath,argv,{cwd:dir,env,stdio:['ignore','pipe','pipe']});let stdout='',stderr='',watchdog=false;
child.stdout.on('data',b=>stdout+=b);child.stderr.on('data',b=>stderr+=b);
const timer=setTimeout(()=>{watchdog=true;child.kill('SIGKILL');},15000);
const exit=await new Promise(r=>child.on('close',(code,signal)=>r({code,signal})));clearTimeout(timer);server.closeAllConnections();await new Promise(r=>server.close(r));
const triggers=requests.filter(q=>q.method==='POST'&&q.path.endsWith('/runs')),span=triggers.at(-1).ms-triggers[0].ms;
let preserved=false;try{const j=JSON.parse(stdout);preserved=j.results.length===n&&j.results.some(x=>x.runId==='run_fixture_0');}catch{}
const violated=kind==='rate'?triggers.length>50&&span<59500:!preserved;
const result={kind,exit,watchdog,signalAt,posts:triggers.length,spanMs:span,stdoutBytes:Buffer.byteLength(stdout),violated,fixtureValid:exit.code===(kind==='rate'?0:130)&&!watchdog&&attempts.size===n};
for(const [name,value] of Object.entries({'requests.json':requests,'stdout.txt':stdout,'stderr.txt':stderr,'result.json':result}))save(name,value);
console.log(JSON.stringify(result));process.exitCode=!result.fixtureValid?2:violated?1:0;I intend to implement this fix after maintainer acceptance and assignment.
Please contact @jesse978 on this GitHub issue or PR; no Discord identity is provided.
Activity
zeshi-du commentedon Oct 20, 2026
Thanks @jesse978, this is a real bug. An interrupt during the outer 429 backoff sleep in
create-batch --run(and early in the trigger fan-out) exits 130 without printing the created test ids or accepted run ids, because no interrupt scope is armed there. Impact is low, since the created tests are in the project and the ids can be recovered.A proper fix changes the interrupt lifecycle, which we will do in the 1.0 rework of
create-batch(it becomestest create --file). We are not taking a community PR for it now. This stays open.