[Hackathon] Batch rerun polls queued members after the shared wait deadline expires #503

Verified against (commit SHA or released version)

988c035 (v0.14.0), Node 24.19.0, Windows 11

Discord Username / User ID

resquwue

What does this improvement do?

Enforce the existing shared --timeout budget for queued members of a batch test rerun. The current poll fan-out starts a fresh one-second poll even when the shared deadline is already exhausted. With four running tests, --timeout 1 and --max-concurrency 1, the actual command took 4.06 seconds and started GETs for the second, third and fourth runs after the deadline.

This affects predictable CI completion and performs unnecessary polling after the requested wait has ended. All accepted IDs are retained and the observed exit is correctly 7; the report is about the overrun and extra requests. A parallel control took 1.01 seconds and an immediate-pass control exited 0 in milliseconds.

Details / implementation notes

Reproduction and controls

Built the pinned commit, then exercised its actual Commander command, HTTP client and poller using injected in-process HTTP responses and real timers. No live requests, real credentials, failed HTTP responses, deferred retries or clock mocking were used.

Command being exercised: testsprite test rerun a b c d --wait --timeout 1 --max-concurrency 1 --no-auto-heal.

Repeated the queued case before adding the Commander entry: 4,051 ms, GETs at 4/1,010/2,019/3,030 ms. All four run IDs remain in the final accepted array with timeout status, so this is NOT a lost-receipts or wrong-exit-code claim. The defect is granting new polls and wait windows after the shared budget has expired. On a large batch the overrun grows with the number of queued members.

Expected: after the shared deadline, queued members should be returned as timeout without a new GET; preserve every accepted run ID and any already-completed results. A genuinely completed member observed within the budget should retain its verdict.

Root cause / proposed fix

src/commands/test.ts sets the batch rerun deadline at line 12941. In pollAccepted at line 13291 it computes Math.max(1, Math.ceil((batchDeadlineMs - Date.now()) / 1000)). A queued member that starts after expiry is therefore granted a fresh one-second poll session. startNext keeps dequeuing members as each previous poll settles.

Propose an exhausted-budget guard before starting each member poll, using the existing timeout-shaped member result. Apply it to the batch rerun path while preserving output order/association, completed outcomes, timeout telemetry, interrupt handling and exit precedence. Do not change remote cancellation policy, issue new runs, or broaden the patch into the transport retry layer.

Related work and duplicate check

Searched open and closed issue/PR records through #502. PR #130 fixed this same pattern for the distinct fresh-run runTestRunAll path; the current rerun path still has the floor and is not changed by that merged PR. This proposal is explicit parity with that earlier fix, not a claim to have invented the guard.

#480 is premature create-batch timeout from rounding a still-positive budget down; #485 is a create-batch trigger dispatched after throttle expiry; #403/#406 concern HTTP retry sleeping through an abort. This reproduction requires none of those paths. #410/#405 preserve acknowledged reruns after dispatch failure; our dispatch succeeds and all receipts are retained. No current report/PR covering the exhausted-budget guard in batch rerun was found.

Regression plan: concurrency 1 with multiple queued running members; expired members receive no GET and retain IDs; completed outcomes remain intact; concurrency >1 and all-pass controls; timeout exit 7 plus existing auth/interrupt precedence. Add deterministic fake-timer regression coverage, and run lint/format/typecheck/tests/coverage >=80%/build/local e2e before a PR.

Existing source is unmodified. Implementation will wait for maintainer acceptance and assignment.

Self-contained offline probe

After npm ci and npm run build, save this script outside the checkout. Run node repro-rerun-deadline.mjs /absolute/path/to/testsprite-cli. It prints full result envelopes, timing traces and writes a local evidence JSON beside the script. The temporary credentials path is empty; the key string is a dummy fixture value. The script only injects responses through the repository's TestDeps seam.

// Offline command-chain reproduction; injected fetch, real clock and timers.
import { createRequire } from 'node:module';
import { mkdtemp, writeFile } from 'node:fs/promises';
import { join, resolve } from 'node:path';
import { tmpdir } from 'node:os';
import { pathToFileURL } from 'node:url';
const repo = resolve(process.argv[2] ?? 'work/testsprite-cli');
const { createTestCommand } = await import(pathToFileURL(join(repo, 'dist/commands/test.js')));
const temp = await mkdtemp(join(tmpdir(), 'testsprite-rerun-probe-'));
const ids = ['a', 'b', 'c', 'd'];
const { Command } = createRequire(join(repo, 'package.json'))('commander');
const json = body => new Response(JSON.stringify(body), { headers: { 'content-type': 'application/json' } });
const results = [];
for (const scenario of [
  { name: 'queued-members', concurrency: 1, terminal: false },
  { name: 'parallel-control', concurrency: 4, terminal: false },
  { name: 'passed-control', concurrency: 1, terminal: true },
]) {
  const stdout = [], trace = [];
  let dispatchAt = 0;
  const fetchImpl = async (input, options = {}) => {
    const url = new URL(String(input));
    if (url.pathname.endsWith('/env')) return json({ environments: [] });
    if (url.pathname.endsWith('/tests/batch/rerun')) {
      dispatchAt = performance.now();
      return json({ accepted: ids.map(testId => ({ testId, runId: 'run_' + testId, enqueuedAt: new Date().toISOString() })),
        deferred: [], conflicts: [], notFound: [], closure: { byProject: [] } });
    }
    const match = /\/runs\/run_([a-d])$/.exec(url.pathname);
    if (match) {
      const testId = match[1];
      trace.push({ runId: 'run_' + testId, elapsedMs: Math.round(performance.now() - dispatchAt) });
      return json({ runId: 'run_' + testId, testId, projectId: 'project_demo', userId: 'demo',
        status: scenario.terminal ? 'passed' : 'running', source: 'cli', createdAt: new Date().toISOString(),
        startedAt: null, finishedAt: scenario.terminal ? new Date().toISOString() : null,
        codeVersion: 'v1', targetUrl: 'https://example.test',
        stepSummary: { total: 1, completed: scenario.terminal ? 1 : 0, passedCount: scenario.terminal ? 1 : 0, failedCount: 0 },
        retryAfterSeconds: 1 });
    }
    const testMatch = /\/tests\/([a-d])$/.exec(url.pathname);
    if (testMatch) return json({ id: testMatch[1], projectId: 'project_demo', name: testMatch[1], type: 'frontend',
      status: 'failed', createdFrom: 'portal', createdAt: new Date().toISOString(), updatedAt: new Date().toISOString() });
    throw new Error('Unexpected offline fixture route: ' + url.pathname);
  };
  let exitCode = 0, error;
  const start = performance.now();
  try {
    const deps = { env: { TESTSPRITE_API_KEY: 'sk-user-offline-fixture', TESTSPRITE_API_URL: 'https://offline.example/api/cli/v1' },
      credentialsPath: join(temp, 'unused-credentials'), fetchImpl,
      stdout: line => stdout.push(line), stderr: () => {} };
    const root = new Command().option('--output <mode>', 'output', 'json');
    root.exitOverride();
    root.addCommand(createTestCommand(deps));
    await root.parseAsync(['test', 'rerun', ...ids, '--wait', '--timeout', '1',
      '--max-concurrency', String(scenario.concurrency), '--no-auto-heal',
      '--idempotency-key', 'offline-deadline-probe'], { from: 'user' });
  } catch (e) { exitCode = e.exitCode ?? 1; error = e.message; }
  const elapsedMs = Math.round(performance.now() - start);
  results.push({ ...scenario, elapsedMs, exitCode, error, trace, payload: stdout.map(x => JSON.parse(x)) });
}
await writeFile(new URL('./rerun-deadline-evidence.json', import.meta.url), JSON.stringify(results, null, 2));
console.log(JSON.stringify(results, null, 2));

Season 4 CLI Improvement Bonus

• Entering the Season 4 CLI Improvement Bonus; Discord: resquwue.

AI assistance: Codex assisted with source review, duplicate checking and offline reproduction.

Confirmations

• I have searched existing issues and this is not a duplicate.
• I have provided my Discord identity above for reward coordination.