[Hackathon] fix(test): emit requested CI artifacts when a single-run wait times out #649

Verified against (commit SHA or released version)

Commit a18d37c8183c1cc39190cfdf2a7a82897e4e113c, package @testsprite/testsprite-cli 0.15.0, Node 24.19.0, Linux cloud container. Runtime verification used unmodified production source and lockfile. No Mac/Windows runtime verification or hosted TestSprite calls are claimed.

Discord Username / User ID

frankzhu94

What does this improvement do?

Write the requested CI artifacts when a successfully triggered single-test test run <id> --wait reaches its polling deadline.

The actual compiled CLI, against an injected running-run fixture:

# GITHUB_ACTIONS=true; GITHUB_STEP_SUMMARY is a fresh path in a valid writable directory
node dist/index.js test run test_xyz --wait --timeout 1 \
  --summary-file summary.json --gh-output --output json

Actual: exit 7 (correct), partial stdout names run_abc with status running, but summary.json is absent, no job-summary file/row is written, and neither output stream contains an error annotation.

Expected: preserve exit 7 and the truthful partial stdout, and also write a one-row reduced CI summary with status:'timeout', total:1, timedOut:1, and other counters zero, a job-summary row, and ::error title=TestSprite test_xyz::status=timeout. In JSON mode the annotation must go to stderr, preserving parseable stdout. A client wait deadline does not prove the server run ended.

Positive passed/failed controls write the requested artifacts. A failed terminal run exits 1 and correctly emits its annotation on stderr in JSON mode. The omission is therefore specific to the early catch path, not an invalid output path.

Impact: downstream CI steps cannot read the requested machine artifact or show the timeout row/annotation, even though the run ID is known and the CLI exits nonzero. No false-green exit, hosted incident or billing outcome is claimed.

Details / implementation notes

Contract and cause

The flag help explicitly promises --summary-file and --gh-output with --wait for single test or batch, including annotations for timed-out runs.

• The TimeoutError branch prints partial JSON and throws before reaching the CI output block.
• emitCiArtifacts for a terminal single run is reached only after polling returns.
• summarizeSingleRun already understands status:'timeout'. The existing emitter already handles best-effort writes and JSON-safe annotation routing.

DOCUMENTATION.md's CI section discusses nonzero-exit artifacts, but its introductory command list names test run --all; the single-test flag help is the clearest direct contract for this report.

Executed full-CLI controls

All output parents exist and are writable; the positive controls use the same sink setup. Exact partial timeout stdout is:

{"runId":"run_abc","status":"running","enqueuedAt":"2026-10-22T00:00:00Z","codeVersion":"v1","targetUrl":"https://app.example.invalid"}

All HTTP is intercepted: test/environment discovery, trigger POST /tests/test_xyz/runs with {"source":"cli"}, then GET /runs/run_abc. The timeout fixture returns running and advances Date.now by two seconds after the first poll. This mirrors the clock seam in the existing upstream TimeoutError unit test; no hosted server or real one-second wait is involved.

Standalone offline reproduction

Build the pinned checkout, save this script outside it, and run node repro-ci-timeout.mjs /absolute/path/to/testsprite-cli. This exact script passed in the cloud before filing.

It runs the real compiled CLI with disposable HOME/USERPROFILE/XDG, dummy API key, notices/telemetry disabled, fresh output directories and fail-closed fetch fixtures. Nothing is sent to a hosted service or real local app. It prints argv-equivalent scenario data, exact request bodies, exit, stdout/stderr and artifact contents/absence for passed/failed/timeout. Its assertions pin currently observed upstream behavior; successful reproduction does not mean the desired fix is present.

import assert from 'node:assert/strict';
import {spawnSync} from 'node:child_process';
import {mkdtempSync,writeFileSync,readFileSync,existsSync,mkdirSync,rmSync} from 'node:fs';
import {tmpdir} from 'node:os';
import {resolve,join} from 'node:path';
const root=resolve(process.argv[2]);
const temp=mkdtempSync(join(tmpdir(),'testsprite-ci-timeout-'));
try {
  const fixture=join(temp,'fixture.mjs');
  writeFileSync(fixture,`
import {writeFileSync} from 'node:fs';
const trace=[];process.on('exit',()=>writeFileSync(process.env.TRACE,JSON.stringify(trace)));
const realNow=Date.now;let offset=0;Date.now=()=>realNow()+offset;
const json=body=>new Response(JSON.stringify(body),{status:200,headers:{'content-type':'application/json'}});
globalThis.fetch=async(input,init={})=>{
  const u=new URL(typeof input==='string'?input:input.url??input.href);
  if(u.hostname!=='fixture.invalid')throw Error('Denied destination');
  const path=u.pathname.replace('/api/cli/v1',''),method=init.method??'GET';
  trace.push({path,method,body:init.body?JSON.parse(String(init.body)):null});
  if(path==='/tests/test_xyz'&&method==='GET')return json({type:'frontend',projectId:'project_1'});
  if(path==='/projects/project_1/env'&&method==='GET')return json({environments:[{name:'default',url:'https://app.example.invalid',isDefault:true}]});
  if(path==='/tests/test_xyz/runs'&&method==='POST')return json({runId:'run_abc',status:'queued',enqueuedAt:'2026-10-22T00:00:00Z',codeVersion:'v1',targetUrl:'https://app.example.invalid'});
  if(path==='/runs/run_abc'&&method==='GET'){
    const scenario=process.env.SCENARIO,status=scenario==='timeout'?'running':scenario;
    if(scenario==='timeout')offset=2000;
    return json({runId:'run_abc',testId:'test_xyz',projectId:'project_1',userId:'fixture',status,source:'cli',createdAt:'2026-10-22T00:00:00Z',startedAt:'2026-10-22T00:00:01Z',finishedAt:scenario==='timeout'?null:'2026-10-22T00:00:02Z',codeVersion:'v1',targetUrl:'https://app.example.invalid',createdFrom:'cli',failedStepIndex:null,failureKind:null,error:null,videoUrl:null,stepSummary:{total:1,completed:1,passedCount:status==='passed'?1:0,failedCount:status==='failed'?1:0}});
  }
  throw Error('Unexpected fixture route '+method+' '+path);
};
`);
  for(const scenario of ['passed','failed','timeout']) {
    const dir=join(temp,scenario);mkdirSync(dir);
    const summary=join(dir,'summary.json'),step=join(dir,'step.md'),trace=join(dir,'http.json');
    const args=['--import',fixture,join(root,'dist/index.js'),'test','run','test_xyz','--wait','--timeout','1','--summary-file',summary,'--gh-output','--output','json'];
    const p=spawnSync(process.execPath,args,{cwd:dir,encoding:'utf8',timeout:15000,env:{PATH:process.env.PATH,HOME:dir,USERPROFILE:dir,XDG_CONFIG_HOME:dir,XDG_CACHE_HOME:dir,CI:'1',DO_NOT_TRACK:'1',TESTSPRITE_NO_TELEMETRY:'1',TESTSPRITE_NO_UPDATE_NOTIFIER:'1',TESTSPRITE_NO_SKILL_WARNING:'1',TESTSPRITE_API_KEY:'sk-user-dummy',TESTSPRITE_API_URL:'https://fixture.invalid',GITHUB_ACTIONS:'true',GITHUB_STEP_SUMMARY:step,TRACE:trace,SCENARIO:scenario}});
    if(p.error)throw p.error;
    const summaryValue=existsSync(summary)?JSON.parse(readFileSync(summary,'utf8')):null;
    const stepValue=existsSync(step)?readFileSync(step,'utf8'):null;
    assert.equal(p.status,{passed:0,failed:1,timeout:7}[scenario]);
    if(scenario==='timeout'){assert.equal(summaryValue,null);assert.equal(stepValue,null);assert.ok(!p.stdout.includes('::error')&&!p.stderr.includes('::error'));assert.equal(JSON.parse(p.stdout).status,'running');}
    else {assert.equal(summaryValue.total,1);assert.ok(stepValue.includes('test_xyz'));if(scenario==='failed')assert.ok(p.stderr.includes('::error title=TestSprite test_xyz::status=failed'));}
    console.log(JSON.stringify({scenario,exit:p.status,summary:summaryValue,stepSummary:stepValue,http:JSON.parse(readFileSync(trace,'utf8')),stdout:p.stdout,stderr:p.stderr},null,2));
  }
}finally{rmSync(temp,{recursive:true,force:true});}

Focused change proposed

Add a best-effort single-run CI emission helper for the polling-deadline branch, using the known testId/runId and a reduced CI status timeout. Keep the existing exit 7 and partial stdout status running; keep emitter exceptions from changing the command's exit, and preserve tunnel cancellation/detach handling. No library change or dependency should be necessary for this narrow branch.

Other poll exceptions also skip artifacts in source/runtime checks, but interruption, rate limiting and request timeout have distinct semantics. This report does not propose labeling all of them timeout or changing their exit codes. They can be considered separately after the deadline case is agreed.

Existing test.run.spec.ts:3047… checks TimeoutError exit 7 and partial stdout but supplies no CI sinks/summary path, so it misses this omission. Add CI artifact assertions beside that case and retain terminal pass/fail controls and JSON stdout parseability.

Baseline on the pinned unchanged source passed build, 4,798 unit tests / 2 skipped, 110 e2e / 1 skipped, typecheck, lint (0 errors, 215 warnings) and formatting. Focused local-target/environment-tunnel/test.quickwins/test.run/gh-output suites passed 356 tests / 5 files.

Related reports and maintainer policy

Reviewed the 642-record open/closed issue/PR snapshot through #646, relevant discussion and the existing CI-output PR. GitHub issue searches were refreshed before filing; no matching single-test deadline-artifact report was found in the checked records.

• [Hackathon] fix(ci): a refused ci run (unknown --env, whole-batch 402/403/400) writes no step outputs, though the docs promise them "refusals included" #573 was declined for unknown environment / whole-batch refusal before dispatch, with a documentation correction preferred. Here a run has already triggered and its polling deadline expires; the single-test flag contract applies. This report does not revisit zero-valued refusal outputs.
• [Hackathon] fix(ci): ci run against a local app fails on a --max-concurrency / --local error the user never passed; past it, the gate writes no step outputs and ignores --fail-on-skipped #549 addresses ci run's local-target path, a different command/orchestration.
• [Hackathon] Validate --report-file and --summary-file paths before dispatching tests #455 / PR fix(test): check --report-file and --summary-file paths before dispatching a run #594 address invalid paths before dispatch; every path here is valid.
• [Hackathon] Add CI reports to test wait so resumed runs can produce artifacts without rerunning tests #456 concerns adding artifacts to the reattach command and was deferred around the 1.0 reporting rework. This proposal is on the existing supported single-run wait path; acceptance/patch timing is still the maintainers' decision.
• [Hackathon] Add "testsprite ci": one-shot run + gate + GitHub-native output #99 / PR feat(test): GitHub-native CI output for "test run --all" (--gh-output, --summary-file) #264 explain the output layer and why JSON-mode annotations belong on stderr, not stdout.

No categorical claim about private/internal duplicates. No production patch, PR or assignment is included; implementation is pending triage. AI disclosure: Codex assisted with source review, cloud-only execution, duplicate checking and this write-up.

Confirmations

• I have searched existing issues and this is not a duplicate.
• I have provided my Discord identity above for reward coordination.