diff --git a/DOCUMENTATION.md b/DOCUMENTATION.md index aac4beeb..4fc0aff4 100644 --- a/DOCUMENTATION.md +++ b/DOCUMENTATION.md @@ -889,7 +889,7 @@ Flags: - `--all` — rerun every test in the resolved project; requires `--project `. - `--wait`, `--timeout ` — block until terminal; same exit matrix as `test run --wait`. -- `--no-auto-heal` — ask for a verbatim replay instead of frontend AI heal-on-drift, which is on by default for frontend reruns. When healing engages, the agent re-authors the test and the new code replaces the stored version — a rerun asks for healing explicitly, so this includes code you wrote yourself; pass `--no-auto-heal` to replay it verbatim. Ignored for backend reruns. What a heal costs depends on which execution engine answers — on the current engine it is a full agent run, billed as the run with nothing extra; on legacy V2 accounts a verbatim-replay pass is free and only a heal engage costs credit. `testsprite usage` reports the figure that applies to your account. Unlike `test run --no-auto-heal`, a rerun gets no echo of the server's effective decision, so a dropped opt-out cannot be detected here — there is no advisory on this path. +- `--no-auto-heal` — ask for a verbatim replay instead of frontend AI heal-on-drift, which is on by default for frontend reruns. When healing engages, the agent re-authors the test and the new code replaces the stored version — a rerun asks for healing explicitly, so this includes code you wrote yourself; pass `--no-auto-heal` to replay it verbatim. Ignored for backend reruns. What a heal costs depends on which execution engine answers — on the current engine it is a full agent run, billed as the run with nothing extra; on legacy V2 accounts a verbatim-replay pass is free and only a heal engage costs credit. `testsprite usage` reports the figure that applies to your account. A rerun may include a server advisory when the execution engine does not honor the opt-out; the CLI prints that warning to stderr and preserves it in JSON output. Absence of an advisory does not prove that the opt-out was enforced. - `--skip-dependencies` — backend only: rerun just the named test without expanding the producer/teardown closure. - `--max-concurrency ` — with `--wait`, cap on in-flight polls during a batch rerun. - `--idempotency-key ` — auto-minted when omitted (the minted key is printed to stderr under `--output json`, `--verbose`, or `--debug`). @@ -899,6 +899,12 @@ Flags: A batch rerun returns `accepted[]` (one `runId` per dispatched test) plus `deferred[]` for any test shed by the per-key run-rate limit; under `--wait`, a non-empty `deferred[]` exits 7 with a `nextAction` you can retry with a fresh idempotency key. A batch where **every** requested id lands in `notFound` (no replayable run) exits **4** — nothing was queued, so the run must not pass; a mixed batch surfaces its `notFound` ids as `skipped` rows in the CI summary without failing an otherwise-passing batch. +If a later dispatch request fails after earlier chunks succeeded, the CLI exits non-zero and prints the acknowledged results first. JSON output includes `partial: true`, the earlier `accepted[]` run IDs, and `unconfirmedTestIds` for tests without an acknowledged outcome. It also distinguishes `failedChunkTestIds` (tests in the failed request whose dispatch is uncertain) from `notAttemptedTestIds` (tests in later requests this command never sent); both exclude tests already acknowledged by earlier responses. Text output names these groups separately. A failed request is not proof that the server did not dispatch its tests, so avoid blindly retrying that group. + +Acknowledged runs may still be executing; use `test wait ` to reattach or `test cancel ` to cancel them. For runs without an owned tunnel, dispatch failure stops the command before result polling, even with `--wait`. Any server `advisories` from acknowledged chunks are deduplicated by feature and message, preserved in partial JSON, and printed once each to stderr, including warnings that an auto-heal opt-out was not enforced. + +Tunnel reruns preserve these same partial fields, merged dependency `closure` data, and advisories. They also retain the legacy `notRunTestIds` field for tests without an acknowledged outcome; that field alone does not prove a test was never dispatched. A capacity-wait failure places the unsent current group in `notAttemptedTestIds`. The partial CI summary and JUnit report distinguish uncertain dispatches from skipped, unattempted tests. Before cleanup, stderr prints copyable `testsprite test wait ` and `testsprite test cancel ` commands covering all acknowledged runs. Use these to inspect the outcome or retry cancellation if automatic cancellation fails; polling does not restore a closed tunnel. Existing tunnel cleanup still requests cancellation of acknowledged active runs before closing the owned tunnel; runs already settled during capacity polling retain their terminal status. + #### `testsprite test flaky ` Detect a **flaky** test by replaying it several times and reporting how often it passes. Each attempt is a rerun with auto-heal **off** (a strict verbatim replay), so healed drift can't disguise a nondeterministic pass/fail — this measures the replay stability of the saved script against the configured URL. Each replay is billed as a rerun, same as a fresh run: 0.5 credits for a frontend replay, 0.2 credits for a backend replay (legacy V2 accounts: FE rerun remains free) — so `--runs N` costs roughly N×0.5 credits for a frontend test. A one-line stderr advisory is printed before a backend replay. diff --git a/src/commands/test.environment-tunnel.spec.ts b/src/commands/test.environment-tunnel.spec.ts index 728cab13..c78dd96f 100644 --- a/src/commands/test.environment-tunnel.spec.ts +++ b/src/commands/test.environment-tunnel.spec.ts @@ -30,6 +30,10 @@ vi.mock('../lib/local-target.js', async importOriginal => ({ const clientId = '11111111-2222-3333-4444-555555555555'; const savedUrl = 'http://127.0.0.1:5173/'; const common = { profile: 'default', output: 'json' as const, debug: false }; +/** + * Build an offline command fixture with configurable API responses and a fake tunnel. + * Captured requests, output, and tunnel lifecycle calls expose dispatch and cleanup ordering. + */ function fixture( options: { url?: string; @@ -38,6 +42,7 @@ function fixture( echo?: boolean; mintStatus?: number; triggerStatus?: number; + cancelStatus?: number; batchResponse?: (body: Record) => Record; envs?: Array>; envsByProject?: Record>>; @@ -85,6 +90,7 @@ function fixture( createdFrom: 'cli', updatedAt: '2026-10-01', }; + /** Serve configured API fixtures and record requests without contacting a live service. */ const fetchImpl = vi.fn(async (input: Parameters[0], init: RequestInit = {}) => { const parsed = new URL( typeof input === 'string' ? input : input instanceof URL ? input.href : input.url, @@ -144,7 +150,8 @@ function fixture( ); if (method === 'DELETE' && path === `/tunnel/${clientId}`) return new Response(null, { status: 204 }); - if (method === 'POST' && path.endsWith('/cancel')) + if (method === 'POST' && path.endsWith('/cancel')) { + if (options.cancelStatus) return error(options.cancelStatus, 'cancellation unavailable'); return json({ runId: path.split('/')[2], testId: path.split('/')[2]!.replace('run_', ''), @@ -165,6 +172,7 @@ function fixture( stepSummary: { total: 1, completed: 0, passedCount: 0, failedCount: 0 }, alreadyCancelled: false, }); + } if ( method === 'POST' && (path.endsWith('/runs') || path.endsWith('/rerun') || path.endsWith('/run')) @@ -1192,6 +1200,13 @@ describe('environment tunnel resolution', () => { accepted: [{ testId: 'a', runId: 'run_a' }], notRunTestIds: ['b'], }); + if (command === 'rerun') + expect(JSON.parse(f.stdout[0]!)).toMatchObject({ + partial: true, + failedChunkTestIds: [], + notAttemptedTestIds: ['b'], + unconfirmedTestIds: ['b'], + }); // eslint-disable-next-line security/detect-non-literal-fs-filename -- fixture-owned temporary file. expect(JSON.parse(readFileSync(path, 'utf8'))).toMatchObject({ total: 2, @@ -1205,6 +1220,162 @@ describe('environment tunnel resolution', () => { }, ); + it.each([ + { all: false, output: 'json' as const, failAt: 2, cancelStatus: undefined }, + { all: true, output: 'json' as const, failAt: 2, cancelStatus: undefined }, + { all: false, output: 'json' as const, failAt: 3, cancelStatus: undefined }, + { all: false, output: 'text' as const, failAt: 2, cancelStatus: undefined }, + { all: false, output: 'json' as const, failAt: 3, cancelStatus: 403 }, + ])( + 'preserves tunnel rerun dispatch details and recovery hints for all=$all output=$output failAt=$failAt cancelStatus=$cancelStatus', + async ({ all, output, failAt, cancelStatus }) => { + const ids = Array.from({ length: 21 }, (_, index) => `test_${index}`); + const advisory = { feature: 'autoHeal', message: 'The auto-heal opt-out was not enforced.' }; + let dispatches = 0; + const fixtureOptions = { + tests: ids.map(id => ({ + id, + name: id, + type: 'frontend', + projectId: 'P', + status: 'passed', + createdFrom: 'cli', + })), + triggerStatus: undefined as number | undefined, + cancelStatus, + onTrigger: () => { + if (++dispatches === failAt) fixtureOptions.triggerStatus = 404; + }, + batchResponse: (body: Record) => ({ + advisories: [advisory, advisory], + closure: { + byProject: [ + { + projectId: 'P', + testIds: body.testIds, + addedProducers: [], + addedTeardowns: [], + clearedCaptured: 1, + }, + ], + }, + }), + }; + const f = fixture(fixtureOptions); + const outputBeforeCleanup: string[][] = []; + const fetchImpl = f.deps.fetchImpl; + f.deps.fetchImpl = vi.fn(async (input, init) => { + const url = + typeof input === 'string' ? input : input instanceof URL ? input.href : input.url; + if (url.endsWith('/cancel') || init?.method === 'DELETE') + outputBeforeCleanup.push([...f.stderr]); + return fetchImpl(input, init); + }); + const summaryFile = join( + mkdtempSync(join(tmpdir(), 'env-dispatch-partial-')), + 'summary.json', + ); + const reportFile = `${summaryFile}.xml`; + const error = await runTestRerun( + { + ...common, + output, + testIds: all ? [] : ids, + all, + projectId: 'P', + wait: false, + timeoutSeconds: 60, + maxConcurrency: 10, + autoHeal: false, + autoHealExplicit: false, + skipDependencies: false, + summaryFile, + report: 'junit', + reportFile, + }, + f.deps, + ).catch(err => err); + expect(error).toMatchObject({ code: 'NOT_FOUND', exitCode: 4, requestId: 'r' }); + const posts = f.calls.filter(call => call.path === '/tests/batch/rerun'); + expect(posts).toHaveLength(failAt); + expect(posts.every(call => call.body.tunnelClientId === clientId)).toBe(true); + const acknowledged = ids.slice(0, (failAt - 1) * 10); + const failed = ids.slice((failAt - 1) * 10, failAt * 10); + const notAttempted = ids.slice(failAt * 10); + expect(f.stdout).toHaveLength(1); + if (output === 'json') { + const partial = JSON.parse(f.stdout[0]!); + expect(partial).toMatchObject({ + partial: true, + failedChunkTestIds: failed, + notAttemptedTestIds: notAttempted, + unconfirmedTestIds: [...failed, ...notAttempted], + notRunTestIds: [...failed, ...notAttempted], + advisories: [advisory], + closure: { + byProject: [ + { + projectId: 'P', + testIds: acknowledged, + addedProducers: [], + addedTeardowns: [], + clearedCaptured: failAt - 1, + }, + ], + }, + }); + expect(partial.accepted.map((row: { runId: string }) => row.runId)).toEqual( + acknowledged.map(id => `run_${id}`), + ); + } else { + expect(f.stdout[0]).toContain( + `Failed request tests (dispatch unconfirmed): ${failed.join(', ')}`, + ); + expect(f.stdout[0]).toContain(`Not attempted by this command: ${notAttempted.join(', ')}`); + expect(f.stdout[0]).not.toContain(advisory.message); + } + expect(f.stderr.filter(line => line === `[advisory] ${advisory.message}`)).toHaveLength(1); + const runIds = acknowledged.map(id => `run_${id}`).join(' '); + const recoveryHints = [ + `Inspect acknowledged runs with: testsprite test wait ${runIds}`, + `Cancel with: testsprite test cancel ${runIds}`, + ]; + expect(outputBeforeCleanup.length).toBeGreaterThan(0); + for (const output of outputBeforeCleanup) + expect(output).toEqual(expect.arrayContaining(recoveryHints)); + expect(f.stdout[0]).not.toContain('testsprite test wait'); + expect(f.stdout[0]).not.toContain('testsprite test cancel'); + const summary = JSON.parse(readFileSync(summaryFile, 'utf8')); + expect(summary).toMatchObject({ total: 21, failed: 10, skipped: notAttempted.length }); + for (const id of failed) + expect(summary.runs).toContainEqual({ + testId: id, + status: 'unknown', + error: 'dispatch unconfirmed after a failed request; inspect before retrying', + }); + const junit = readFileSync(reportFile, 'utf8'); + expect(junit).toContain('failures="10"'); + expect(junit).toContain(`skipped="${notAttempted.length}"`); + expect(junit).toContain( + '', + ); + const cancellations = f.calls.filter(call => call.path.endsWith('/cancel')); + // Capacity polling settled earlier runs; cleanup must still cancel every remaining active run. + expect(cancellations.map(call => call.path)).toEqual( + acknowledged + .slice(-10) + .slice(failed.length) + .map(id => `/runs/run_${id}/cancel`), + ); + if (cancelStatus) + expect(f.stderr.filter(line => line.includes('Could not cancel'))).toHaveLength( + cancellations.length, + ); + expect(f.calls.at(-1)).toMatchObject({ method: 'DELETE', path: `/tunnel/${clientId}` }); + expect(f.stop).toHaveBeenCalledTimes(1); + }, + ); + it('capacity polling preserves its NOT_FOUND error rather than claiming a rejected rerun dispatch', async () => { const f = fixture({ pollStatus: 404 }); await expect( diff --git a/src/commands/test.rerun.dispatch-error.spec.ts b/src/commands/test.rerun.dispatch-error.spec.ts new file mode 100644 index 00000000..5dc9a25d --- /dev/null +++ b/src/commands/test.rerun.dispatch-error.spec.ts @@ -0,0 +1,200 @@ +import { describe, expect, it } from 'vitest'; +import { ApiError } from '../lib/errors.js'; +import type { RerunAdvisory } from '../lib/runs.types.js'; +import { runTestRerun } from './test.js'; + +const testIds = Array.from({ length: 51 }, (_, i) => `test_${i}`); +/** Dispatch fake batches, fail the selected request and capture recovery output without polling. */ +async function exercise( + options: { + all?: boolean; + wait?: boolean; + text?: boolean; + failAt?: number; + errorCode?: 'NOT_FOUND' | 'AUTH_INVALID'; + count?: number; + advisories?: RerunAdvisory[]; + } = {}, +) { + const requestedIds = Array.from({ length: options.count ?? 51 }, (_, i) => `test_${i}`); + const tests = requestedIds.map(id => ({ + id, + name: id, + type: 'frontend' as const, + projectId: 'project_repro', + status: 'passed', + createdFrom: 'portal', + })); + const stdout: string[] = [], + stderr: string[] = [], + chunks: string[][] = []; + /** + * Serve test and environment data offline, acknowledging batches until the selected failure. + * Reject unexpected requests so recovery-output tests cannot silently start result polling. + */ + const fetchImpl: typeof fetch = async (input, init = {}) => { + const url = String(input); + let status = 200; + let body: unknown; + if (url.includes('/tests/batch/rerun')) { + const ids = (JSON.parse(String(init.body)) as { testIds: string[] }).testIds; + chunks.push(ids); + if (chunks.length === (options.failAt ?? 2)) { + const code = options.errorCode ?? 'NOT_FOUND'; + status = code === 'NOT_FOUND' ? 404 : 401; + body = { + error: { + code, + message: 'Dispatch refused', + requestId: 'req_failed_chunk', + details: { testIds: ids }, + nextAction: 'Check this request', + }, + }; + } else { + status = 202; + body = { + accepted: ids.map(testId => ({ + testId, + runId: `run_${testId}`, + enqueuedAt: '2026-10-16T00:00:00Z', + })), + deferred: [], + conflicts: [], + closure: { byProject: [] }, + ...(options.advisories ? { advisories: options.advisories } : {}), + }; + } + } else if (url.includes('/env')) body = { environments: [] }; + else if (url.includes('/tests?')) body = { items: tests, nextToken: null }; + else throw new Error(`Unexpected request: ${url}`); + return new Response(JSON.stringify(body), { + status, + headers: { 'content-type': 'application/json' }, + }); + }; + const error = await runTestRerun( + { + testIds: options.all ? [] : requestedIds, + all: options.all ?? false, + projectId: 'project_repro', + knownTests: new Map(tests.map(test => [test.id, test])), + wait: options.wait ?? false, + noWait: !options.wait, + timeoutSeconds: 60, + autoHeal: false, + autoHealExplicit: false, + skipDependencies: false, + maxConcurrency: 5, + profile: 'default', + output: options.text ? 'text' : 'json', + debug: false, + idempotencyKey: 'dispatch-repro', + }, + { + env: { TESTSPRITE_API_KEY: 'sk-user-test', TESTSPRITE_API_URL: 'https://api.example.com' }, + fetchImpl, + stdout: line => stdout.push(line), + stderr: line => stderr.push(line), + }, + ).catch((err: unknown) => err); + return { error, stdout, stderr, chunks }; +} + +describe('batch rerun dispatch failures', () => { + it.each([false, true])('preserves deduplicated advisories with text=%s', async text => { + const advisory = { + feature: 'autoHeal', + message: 'The requested auto-heal opt-out was not enforced.', + }; + const { error, stdout, stderr, chunks } = await exercise({ + count: 101, + failAt: 3, + text, + advisories: [advisory, advisory], + }); + expect(error).toMatchObject({ exitCode: 4, requestId: 'req_failed_chunk' }); + expect(chunks.map(chunk => chunk.length)).toEqual([50, 50, 1]); + expect(stderr.filter(line => line.startsWith('[advisory]'))).toEqual([ + `[advisory] ${advisory.message}`, + ]); + if (text) expect(stdout.join('\n')).not.toContain(advisory.message); + else expect(JSON.parse(stdout[0]!).advisories).toEqual([advisory]); + }); + + it.each([ + { all: false, wait: false }, + { all: false, wait: true }, + { all: true, wait: false }, + { all: true, wait: true }, + ])('separates failed and unattempted requests for all=$all wait=$wait', async options => { + const { error, stdout, chunks } = await exercise({ ...options, count: 101 }); + expect(chunks.map(chunk => chunk.length)).toEqual([50, 50]); + expect(error).toMatchObject({ exitCode: 4 }); + const payload = JSON.parse(stdout[0]!); + expect(payload.accepted).toHaveLength(50); + expect(payload.failedChunkTestIds).toEqual(chunks[1]); + expect(payload.notAttemptedTestIds).toEqual(['test_100']); + expect(payload.unconfirmedTestIds).toEqual([...chunks[1]!, 'test_100']); + expect(payload.advisories).toEqual([]); + }); + + it('distinguishes failed and unattempted requests in text output', async () => { + const { stdout, error } = await exercise({ text: true, count: 101 }); + expect(error).toMatchObject({ exitCode: 4 }); + expect(stdout.join('\n')).toContain('Failed request tests (dispatch unconfirmed): test_50,'); + expect(stdout.join('\n')).toContain('Not attempted by this command: test_100'); + }); + + it.each([ + { all: false, wait: false }, + { all: false, wait: true }, + { all: true, wait: false }, + { all: true, wait: true }, + ])('preserves earlier runs for all=$all wait=$wait', async options => { + const { error, stdout, stderr, chunks } = await exercise(options); + expect(chunks.map(chunk => chunk.length)).toEqual([50, 1]); + expect(error).toBeInstanceOf(ApiError); + expect(error).toMatchObject({ + exitCode: 4, + requestId: 'req_failed_chunk', + details: { testIds: ['test_50'] }, + }); + expect(stdout).toHaveLength(1); + const payload = JSON.parse(stdout[0]!); + expect(payload.partial).toBe(true); + expect(payload.accepted.map((row: { runId: string }) => row.runId)).toEqual( + testIds.slice(0, 50).map(id => `run_${id}`), + ); + expect(payload.unconfirmedTestIds).toEqual(['test_50']); + expect(payload.failedChunkTestIds).toEqual(['test_50']); + expect(payload.notAttemptedTestIds).toEqual([]); + expect(stderr.join('\n')).toContain('testsprite test wait run_test_0'); + expect(stderr.join('\n')).toContain('testsprite test cancel run_test_0'); + }); + + it('prints accepted IDs and recovery hints in text mode', async () => { + const { stdout, stderr, error } = await exercise({ text: true }); + expect(error).toMatchObject({ exitCode: 4 }); + for (const id of testIds.slice(0, 50)) expect(stdout.join('\n')).toContain(`run_${id} `); + expect(stderr.join('\n')).toContain('50 run(s) were already accepted'); + }); + + it('preserves authentication errors after emitting partial results', async () => { + const { stdout, error } = await exercise({ errorCode: 'AUTH_INVALID' }); + expect(error).toMatchObject({ + exitCode: 3, + code: 'AUTH_INVALID', + requestId: 'req_failed_chunk', + }); + expect(JSON.parse(stdout[0]!).accepted).toHaveLength(50); + }); + + it('does not invent accepted runs when the first chunk fails', async () => { + const { stdout, stderr, chunks, error } = await exercise({ failAt: 1 }); + expect(error).toMatchObject({ exitCode: 4 }); + expect(chunks).toHaveLength(1); + expect(stdout).toEqual([]); + expect(stderr.join('\n')).not.toContain('already accepted'); + }); +}); diff --git a/src/commands/test.ts b/src/commands/test.ts index 31f3b107..f7a31716 100644 --- a/src/commands/test.ts +++ b/src/commands/test.ts @@ -11832,6 +11832,11 @@ export async function runTestRerun( ); } +/** + * Resolve and dispatch reruns, then optionally wait for their results. + * If dispatch fails after an acknowledged batch, emit recovery data and + * advisories before propagating the original failure; do not start polling. + */ async function runTestRerunCore( opts: RunTestRerunOptions, deps: TestDeps = {}, @@ -12737,6 +12742,8 @@ async function runTestRerunCore( } let chunkResponses: BatchRerunResponse[] = []; let waitingForCapacity = false; + let dispatchingTestIds: string[] = []; + let dispatchingChunkIndex = -1; try { // Dispatch chunks one at a time, NOT via Promise.all. BE producer/ // teardown closure dedup happens per-request, server-side. Two chunks @@ -12748,6 +12755,8 @@ async function runTestRerunCore( chunkResponses = []; for (let idx = 0; idx < chunks.length; idx++) { const chunk = chunks[idx]!; + dispatchingTestIds = chunk; + dispatchingChunkIndex = idx; // Bound the per-chunk idempotency key to <=256 chars (mirrors the retry // path). A long base key plus the `:chunkN` suffix could otherwise exceed // the server cap and be rejected or truncated inconsistently. @@ -12788,30 +12797,86 @@ async function runTestRerunCore( ); } } catch (err) { - if (deps.environmentTunnel && chunkResponses.length) - await emitEnvironmentDispatchPartial( - chunkResponses, - testIds, - opts, - deps, - 'rerun', - err, - deps.environmentTunnel, - ); - // D2 (dogfood): the batch endpoint rejects the WHOLE request when any id is - // unresolvable (unknown, cross-tenant, or never ran cleanly), so one bad id - // aborts the batch with NOT_FOUND. Replace the bare exit-4 with an - // actionable hint. (Server-side partial-accept of unknown ids — a - // `notFound[]` in the batch response so good ids still run — is a tracked - // backend follow-up.) + if (chunkResponses.length > 0) { + // Earlier HTTP requests have already dispatched work. A later request + // failing does not roll those runs back; preserve their IDs before exit. + const accepted = chunkResponses.flatMap(response => response.accepted); + const deferred = chunkResponses.flatMap(response => response.deferred); + const conflicts = chunkResponses.flatMap(response => response.conflicts); + const notFound = chunkResponses.flatMap(response => response.notFound ?? []); + const acknowledged = new Set([ + ...accepted.map(row => row.testId), + ...deferred.map(row => row.testId), + ...conflicts.map(row => row.testId), + ...notFound, + ]); + const partial = { + partial: true, + accepted, + deferred, + conflicts, + notFound, + advisories: dedupeRerunAdvisories( + chunkResponses.flatMap(response => response.advisories ?? []), + ), + closure: { + byProject: mergeBatchRerunClosureByProject( + chunkResponses.flatMap(response => response.closure.byProject), + ), + }, + // An errored request may have reached the server; do not claim rollback. + unconfirmedTestIds: testIds.filter(id => !acknowledged.has(id)), + failedChunkTestIds: waitingForCapacity + ? [] + : dispatchingTestIds.filter(id => !acknowledged.has(id)), + notAttemptedTestIds: chunks + .slice(dispatchingChunkIndex + (waitingForCapacity ? 0 : 1)) + .flat() + .filter(id => !acknowledged.has(id)), + }; + if (deps.environmentTunnel) { + await emitEnvironmentDispatchPartial( + chunkResponses, + testIds, + opts, + deps, + 'rerun', + err, + deps.environmentTunnel, + partial, + ); + } else { + out.print(partial, () => + [ + 'Batch rerun partially dispatched; a later request failed.', + ...accepted.map(row => `${row.runId} ${row.testId} dispatched (not awaited)`), + `Unconfirmed tests: ${partial.unconfirmedTestIds.join(', ') || '(none)'}`, + `Failed request tests (dispatch unconfirmed): ${partial.failedChunkTestIds.join(', ') || '(none)'}`, + `Not attempted by this command: ${partial.notAttemptedTestIds.join(', ') || '(none)'}`, + ].join('\n'), + ); + emitRerunAdvisories(stderrFn, partial.advisories); + if (accepted.length > 0) { + const runIds = [...new Set(accepted.map(row => row.runId))]; + stderrFn(`${runIds.length} run(s) were already accepted and may still be running.`); + stderrFn(`Re-attach with: testsprite test wait ${runIds.join(' ')}`); + stderrFn(`Cancel with: testsprite test cancel ${runIds.join(' ')}`); + } + } + } + // A batch request can fail even after earlier chunks succeeded. if (!waitingForCapacity && err instanceof ApiError && err.code === 'NOT_FOUND') { throw ApiError.fromEnvelope({ error: { code: 'NOT_FOUND', - message: `Batch rerun aborted: one or more of the ${testIds.length} requested test${testIds.length !== 1 ? 's' : ''} has no replayable run (unknown/cross-tenant id, or never completed a clean run). The batch endpoint rejects the whole request when any id is unresolvable.`, + message: `Batch rerun ${chunkResponses.length ? 'partially dispatched; a later request failed' : 'aborted'}: one or more of the ${dispatchingTestIds.length} tests in the failed request has no replayable run (unknown/cross-tenant id, or never completed a clean run). Earlier accepted runs are not rolled back.`, nextAction: `Verify the test ids and drop any that have never run, or trigger fresh runs individually: testsprite test run `, requestId: err.requestId ?? 'local', - details: { reason: 'batch_contains_unreplayable', testIds }, + details: { + ...err.details, + reason: 'batch_contains_unreplayable', + testIds: dispatchingTestIds, + }, }, }); } diff --git a/src/lib/environment-tunnel.ts b/src/lib/environment-tunnel.ts index ddc7f442..8dfd50ce 100644 --- a/src/lib/environment-tunnel.ts +++ b/src/lib/environment-tunnel.ts @@ -10,7 +10,7 @@ import { TunnelClient, type TunnelClientOptions } from '../vendor/tunnel-client/ import { assertLocalPortListening } from './local-target.js'; import { recordTelemetryTunnelOpened } from './telemetry.js'; import type { HttpClient } from './http.js'; -import type { RunResponse } from './runs.types.js'; +import type { BatchRerunClosure, RerunAdvisory, RunResponse } from './runs.types.js'; import { Output } from './output.js'; import { emitCiArtifacts, summarizeAcceptedPayload } from './gh-output.js'; import { buildJUnitReport, writeJUnitReportFile } from './junit-report.js'; @@ -551,10 +551,19 @@ export function groupEnvironmentTargets( }); } +/** Quote an environment name as one POSIX-shell argument in tunnel recovery commands. */ export function quoteEnvironmentName(name: string): string { return "'" + name.replaceAll("'", "'\\''") + "'"; } +/** + * Emit acknowledged tunnel receipts and CI artifacts before propagating a failure. + * Optional rerun dispatch details distinguish uncertain requests from unsent work + * and retain merged closure data and deduplicated advisories. Tunnel ownership + * and cleanup remain with the caller; this function never cancels or closes it. + * Recovery commands go to stderr before cleanup so callers can inspect or cancel + * acknowledged runs even if automatic cancellation later fails. + */ export async function emitEnvironmentDispatchPartial( responses: Array<{ accepted: Array<{ testId: string; runId: string }>; @@ -574,6 +583,13 @@ export async function emitEnvironmentDispatchPartial( label: string, error: unknown, context: EnvironmentTunnelContext, + dispatchDetails?: { + unconfirmedTestIds: string[]; + failedChunkTestIds: string[]; + notAttemptedTestIds: string[]; + closure: BatchRerunClosure; + advisories: RerunAdvisory[]; + }, ): Promise { const message = error instanceof Error ? error.message : String(error); const timedOut = @@ -601,21 +617,51 @@ export async function emitEnvironmentDispatchPartial( ...(context.skipped?.length ? { skipped: context.skipped } : {}), notRunTestIds, ...(responses.some(response => response.notFound) ? { notFound } : {}), + ...(dispatchDetails + ? { + partial: true, + unconfirmedTestIds: dispatchDetails.unconfirmedTestIds, + failedChunkTestIds: dispatchDetails.failedChunkTestIds, + notAttemptedTestIds: dispatchDetails.notAttemptedTestIds, + closure: dispatchDetails.closure, + advisories: dispatchDetails.advisories, + } + : {}), }; const stderr = deps.stderr ?? (line => process.stderr.write(`${line}\n`)); new Output(opts.output, { stdout: deps.stdout, stderr: deps.stderr }).print(partial, () => - accepted.map(row => `${row.runId} ${row.status}`).join('\n'), + [ + ...accepted.map(row => `${row.runId} ${row.status}`), + ...(dispatchDetails + ? [ + `Unconfirmed tests: ${dispatchDetails.unconfirmedTestIds.join(', ') || '(none)'}`, + `Failed request tests (dispatch unconfirmed): ${dispatchDetails.failedChunkTestIds.join(', ') || '(none)'}`, + `Not attempted by this command: ${dispatchDetails.notAttemptedTestIds.join(', ') || '(none)'}`, + ] + : []), + ].join('\n'), ); + for (const advisory of dispatchDetails?.advisories ?? []) + stderr(`[advisory] ${advisory.message}`); + const runIds = [...new Set(accepted.map(row => row.runId))]; + if (runIds.length > 0) { + stderr(`Inspect acknowledged runs with: testsprite test wait ${runIds.join(' ')}`); + stderr(`Cancel with: testsprite test cancel ${runIds.join(' ')}`); + } const summary = summarizeAcceptedPayload(JSON.stringify(partial)); + const uncertain = new Set(dispatchDetails?.failedChunkTestIds ?? []); summary.runs.push( ...notRunTestIds.map(testId => ({ testId, - status: 'skipped', - error: 'not dispatched before the owned tunnel closed', + status: uncertain.has(testId) ? 'unknown' : 'skipped', + error: uncertain.has(testId) + ? 'dispatch unconfirmed after a failed request; inspect before retrying' + : 'not dispatched before the owned tunnel closed', })), ); summary.total += notRunTestIds.length; - summary.skipped += notRunTestIds.length; + summary.failed += uncertain.size; + summary.skipped += notRunTestIds.length - uncertain.size; emitCiArtifacts( summary, opts, @@ -637,7 +683,10 @@ export async function emitEnvironmentDispatchPartial( testId: row.testId, runId: row.runId, status: row.status, - error: { code: error instanceof ApiError ? error.code : 'UNSUPPORTED', message }, + error: { + code: error instanceof ApiError ? error.code : 'UNSUPPORTED', + message: row.error ?? message, + }, })), }), );