[Hackathon] fix(test run): preserve skipped tests returned by deferred retries #477

Verified against (commit SHA or released version)

988c035 (current public main verified 2026-10-17; package version 0.14.0); Node.js v22.23.2 / macOS

Discord Username / User ID

luckyzd.

What does this improvement do?

Preserve skippedFrontend and skippedIntegration returned by a deferred retry in testsprite test run --all --wait.

The initial response's skipped arrays are preserved, but the retry loop merges only accepted/conflicted/deferred items. If a retry reports a previously deferred test as skipped, that test disappears from every category in the final JSON and from the handler's returned envelope. With another accepted test passing, the CLI exits 0 and says all previously deferred tests were accepted, even though the retry explicitly returned the missing test as skipped.

This is a result-accounting problem: a consumer checking the raw skipped arrays cannot distinguish a complete run from the reproduced partial run. It does not require --report junit.

I am submitting this through the Season 4 hackathon issue-first flow. This report demonstrates the CLI's handling of schema-valid API responses using offline fixtures; I have not observed this transition against the hosted service. Maintainers should confirm whether their backend can reclassify a deferred test into a skipped category between requests. I am not claiming a production incident or asking to change the intentional exit policy for known skipped tests.

Details / implementation notes

Reproduction sequence and actual output

1. The first POST /tests/batch/run accepts test_a with run_a, defers test_b, and returns empty skipped arrays.
2. The CLI waits its real 61-second retry interval, then sends testIds: ["test_b"].
3. The retry returns HTTP 200 with accepted: [], conflicts: [], deferred: [], skippedFrontend: ["test_b"], and skippedIntegration: [].
4. GET /runs/run_a returns a passed terminal run.

The final JSON is:

{
  "accepted": [{"testId": "test_a", "runId": "run_a", "status": "passed"}],
  "conflicts": [],
  "deferred": [],
  "skippedFrontend": [],
  "skippedIntegration": [],
  "summary": {"passed": 1, "failed": 0, "timedOut": 0, "deferred": 0, "conflicts": 0, "total": 1}
}

Unrelated accepted-run fields are omitted above. test_b is absent from the actual complete output. Exit code is 0. Stderr includes:

[deferred-retry] attempt 1 — retrying 1 deferred test in 61s
[deferred-retry] attempt 1: all previously-deferred tests accepted.
Batch run complete: 1/1 passed, 0 failed/blocked, 0 timed out

The full compiled CLI took 61.145 seconds for this case. No timers or clock were mocked in that reproduction. Returning skippedFrontend: ["test_b"] in the initial response instead preserves it correctly in the final JSON.

Contract and cause

• BatchRunFreshResponse documents the non-dispatched categories as the accounting for tests that did not run.
• runTestRunAll initializes the skipped arrays only from the first response.
• The retry merge never reads the retry's skipped arrays, but replaces deferred with the now-empty retry value.
• Final JSON and the returned envelope consequently retain only the original skipped arrays.

The raw arrays matter independently of the status summary. summarizeAcceptedPayload intentionally excludes these two categories and delegates partial-run policy to the consumer. For a concrete published consumer, the archived TestSprite Action reads the raw arrays and rejects skips unless allow-partial is enabled. The missing record changes that skip count from 1 to 0. This reference illustrates the output contract; I am not proposing changes to the archived Action or claiming a live workflow was run.

Expected: retain retry-returned skipped IDs in their correct final categories, including the handler return; preserve initial skipped IDs; deduplicate repeated IDs as appropriate; emit an accurate advisory. An empty deferred list alone does not mean every test was accepted. No change to the intentional summary/exit policy for already-reported skipped tests is required.

Validation and duplicate check

Six offline handler-level integration cases use the real HttpClient, schema validation, command handler, and artifact writers, with injected HTTP responses and retry sleep:

The standalone compiled CLI also reproduced the initial frontend-skip control and the real 61-second retry failure. Existing src/commands/test.run.spec.ts and src/lib/gh-output.test.ts: 246 tests passed, 2 files, 7.70 seconds. Product source was not modified for the reproductions.

Searched open and closed Issues/PRs for skippedFrontend, skippedIntegration, and combinations of deferred/retry/skip/lost; also checked relevant earlier PR #33 review comments. No matching report was found as of 2026-10-17. Closest reports:

• [Hackathon] fix(rerun): JUnit drops undispatched tests and keeps stale reports when no runs are accepted #476 concerns rerun JUnit accounting for non-dispatched tests, while its raw JSON retains the buckets. This report concerns fresh batch retry response merging and loses the raw data before any report consumer sees it.
• [Hackathon] Fresh batch --wait returns exit 7 instead of auth exit 3 when tests remain deferred #471 concerns the priority of retry authentication errors over remaining deferred work. Here both dispatch responses are HTTP 200 and schema-valid.
• [Hackathon] fix(rerun): later chunk failure discards already-accepted run IDs (51 tests, 50 hidden runs) #405 / fix(rerun): preserve acknowledged runs after a later dispatch failure #410 concern preserving acknowledged run IDs after a later rerun chunk fails. Here run_a is retained correctly; the lost item is the explicit non-dispatched outcome for test_b from a successful retry response.

Standalone offline reproduction

Build the pinned checkout with npm ci && npm run build. Save the script below outside the checkout, then run node /path/to/repro.mjs /absolute/path/to/testsprite-cli. It takes about 61 seconds, uses synthetic credentials and intercepted HTTP responses only, and does not contact TestSprite or consume credits. Assertions describe the current faulty behavior so the script completes on the affected revision.

import assert from 'node:assert/strict';
import {execFile} from 'node:child_process';
import {mkdtemp,readFile,writeFile,rm} from 'node:fs/promises';
import {tmpdir} from 'node:os';
import {join,resolve,dirname} from 'node:path';
import {promisify} from 'node:util';
const root=resolve(process.argv[2]??'work/testsprite-cli');
const dir=await mkdtemp(join(tmpdir(),'testsprite-retry-cli-'));
const preload=join(dir,'fixture.mjs');
await writeFile(preload,`
import assert from 'node:assert/strict';
import {writeFileSync} from 'node:fs';
import os from 'node:os';
import {syncBuiltinESMExports} from 'node:module';
os.homedir=()=>process.env.REPRO_HOME;syncBuiltinESMExports();
const initial=process.env.REPRO_INITIAL==='true',trace=[];
let posts=0;
process.on('exit',()=>writeFileSync(process.env.REPRO_TRACE,JSON.stringify(trace)));
const json=body=>new Response(JSON.stringify(body),{status:200,headers:{'content-type':'application/json'}});
globalThis.fetch=async(input,init={})=>{
 const u=new URL(typeof input==='string'?input:input.url??input.href),method=init.method??'GET';
 assert.equal(u.hostname,'mock.example.com');
 const path=u.pathname.replace('/api/cli/v1','');
 const row={at:Date.now(),method,path,...(init.body?{body:JSON.parse(String(init.body))}:{})};trace.push(row);
 if(path==='/projects/project_fixture/env')return json({environments:[]});
 if(path==='/tests')return json({items:[{id:'test_a',name:'test_a',projectId:'project_fixture',type:'backend',status:'passed',createdFrom:'cli',createdAt:'2026-10-17T00:00:00Z',updatedAt:'2026-10-17T00:00:00Z'}],nextToken:null});
 if(method==='POST'&&path==='/tests/batch/run'){
  posts++;const first=posts===1;
  const response={accepted:first?[{testId:'test_a',runId:'run_a',enqueuedAt:'2026-10-17T00:00:00Z'}]:[],
   deferred:first&&!initial?[{testId:'test_b'}]:[],conflicts:[],skippedFrontend:!first||initial?['test_b']:[],skippedIntegration:[]};
  row.response=response;return json(response);
 }
 if(path==='/runs/run_a')return json({runId:'run_a',testId:'test_a',projectId:'project_fixture',userId:'fixture',status:'passed',source:'cli',
  createdAt:'2026-10-17T00:00:00Z',startedAt:'2026-10-17T00:00:01Z',finishedAt:'2026-10-17T00:00:02Z',codeVersion:'v1',targetUrl:null,createdFrom:'fresh',
  failedStepIndex:null,failureKind:null,error:null,videoUrl:null,stepSummary:{total:1,completed:1,passedCount:1,failedCount:0}});
 throw Error('Unexpected route '+method+' '+path);
};
`);
const results=[];
try{
 for(const initial of [true,false]){
  const traceFile=join(dir,String(initial)+'.json');
  const started=performance.now();
  const args=['test','run','--all','--project','project_fixture','--wait','--timeout','180','--output','json'];
  const {stdout,stderr}=await promisify(execFile)(process.execPath,['--import',preload,join(root,'dist/index.js'),...args],{
   cwd:dir,timeout:75000,env:{PATH:dirname(process.execPath)+':/usr/bin:/bin',
    TESTSPRITE_API_KEY:'sk-user-offline-fixture',TESTSPRITE_API_URL:'https://mock.example.com',TESTSPRITE_NO_TELEMETRY:'1',TESTSPRITE_NO_UPDATE_NOTIFIER:'1',
    REPRO_HOME:dir,REPRO_TRACE:traceFile,REPRO_INITIAL:String(initial)}});
  const trace=JSON.parse(await readFile(traceFile,'utf8')),payload=JSON.parse(stdout);
  const posts=trace.filter(x=>x.method==='POST');
  assert.equal(posts.length,initial?1:2);
  assert.deepEqual(payload.accepted.map(x=>x.runId),['run_a']);
  assert.deepEqual(payload.skippedFrontend,initial?['test_b']:[]);
  assert.deepEqual(payload.deferred,[]);
  if(!initial){
   assert.deepEqual(posts[1].body.testIds,['test_b']);
   assert.ok(posts[1].at-posts[0].at>=60000,'real retry wait must occur');
   assert.deepEqual(posts[1].response.skippedFrontend,['test_b']);
  }
  results.push({name:initial?'initial-skip-control':'real-61s-retry-skip-lost',exitCode:0,elapsedMs:performance.now()-started,payload,trace,stderr});
 }
 console.log(JSON.stringify({node:process.version,results},null,2));
}finally{await rm(dir,{recursive:true,force:true});}

Confirmations

• I have searched existing issues and this is not a duplicate.
• I have provided my Discord identity above for reward coordination.