fix(rerun): preserve acknowledged runs after a later dispatch failure

DOCUMENTATION.md

@@ -889,7 +889,7 @@ Flags:
 
 - `--all` — rerun every test in the resolved project; requires `--project <id>`.
 - `--wait`, `--timeout <s>` — block until terminal; same exit matrix as `test run --wait`.
-- `--no-auto-heal` — ask for a verbatim replay instead of frontend AI heal-on-drift, which is on by default for frontend reruns. When healing engages, the agent re-authors the test and the new code replaces the stored version — a rerun asks for healing explicitly, so this includes code you wrote yourself; pass `--no-auto-heal` to replay it verbatim. Ignored for backend reruns. What a heal costs depends on which execution engine answers — on the current engine it is a full agent run, billed as the run with nothing extra; on legacy V2 accounts a verbatim-replay pass is free and only a heal engage costs credit. `testsprite usage` reports the figure that applies to your account. Unlike `test run --no-auto-heal`, a rerun gets no echo of the server's effective decision, so a dropped opt-out cannot be detected here — there is no advisory on this path.
+- `--no-auto-heal` — ask for a verbatim replay instead of frontend AI heal-on-drift, which is on by default for frontend reruns. When healing engages, the agent re-authors the test and the new code replaces the stored version — a rerun asks for healing explicitly, so this includes code you wrote yourself; pass `--no-auto-heal` to replay it verbatim. Ignored for backend reruns. What a heal costs depends on which execution engine answers — on the current engine it is a full agent run, billed as the run with nothing extra; on legacy V2 accounts a verbatim-replay pass is free and only a heal engage costs credit. `testsprite usage` reports the figure that applies to your account. A rerun may include a server advisory when the execution engine does not honor the opt-out; the CLI prints that warning to stderr and preserves it in JSON output. Absence of an advisory does not prove that the opt-out was enforced.
 - `--skip-dependencies` — backend only: rerun just the named test without expanding the producer/teardown closure.
 - `--max-concurrency <n>` — with `--wait`, cap on in-flight polls during a batch rerun.
 - `--idempotency-key <key>` — auto-minted when omitted (the minted key is printed to stderr under `--output json`, `--verbose`, or `--debug`).
@@ -899,6 +899,12 @@ Flags:
 
 A batch rerun returns `accepted[]` (one `runId` per dispatched test) plus `deferred[]` for any test shed by the per-key run-rate limit; under `--wait`, a non-empty `deferred[]` exits 7 with a `nextAction` you can retry with a fresh idempotency key. A batch where **every** requested id lands in `notFound` (no replayable run) exits **4** — nothing was queued, so the run must not pass; a mixed batch surfaces its `notFound` ids as `skipped` rows in the CI summary without failing an otherwise-passing batch.
 
+If a later dispatch request fails after earlier chunks succeeded, the CLI exits non-zero and prints the acknowledged results first. JSON output includes `partial: true`, the earlier `accepted[]` run IDs, and `unconfirmedTestIds` for tests without an acknowledged outcome. It also distinguishes `failedChunkTestIds` (tests in the failed request whose dispatch is uncertain) from `notAttemptedTestIds` (tests in later requests this command never sent); both exclude tests already acknowledged by earlier responses. Text output names these groups separately. A failed request is not proof that the server did not dispatch its tests, so avoid blindly retrying that group.
+
+Acknowledged runs may still be executing; use `test wait <run-id...>` to reattach or `test cancel <run-id...>` to cancel them. For runs without an owned tunnel, dispatch failure stops the command before result polling, even with `--wait`. Any server `advisories` from acknowledged chunks are deduplicated by feature and message, preserved in partial JSON, and printed once each to stderr, including warnings that an auto-heal opt-out was not enforced.
+
+Tunnel reruns preserve these same partial fields, merged dependency `closure` data, and advisories. They also retain the legacy `notRunTestIds` field for tests without an acknowledged outcome; that field alone does not prove a test was never dispatched. A capacity-wait failure places the unsent current group in `notAttemptedTestIds`. The partial CI summary and JUnit report distinguish uncertain dispatches from skipped, unattempted tests. Before cleanup, stderr prints copyable `testsprite test wait <run-id...>` and `testsprite test cancel <run-id...>` commands covering all acknowledged runs. Use these to inspect the outcome or retry cancellation if automatic cancellation fails; polling does not restore a closed tunnel. Existing tunnel cleanup still requests cancellation of acknowledged active runs before closing the owned tunnel; runs already settled during capacity polling retain their terminal status.
+
 #### `testsprite test flaky <test-id>`
 
 Detect a **flaky** test by replaying it several times and reporting how often it passes. Each attempt is a rerun with auto-heal **off** (a strict verbatim replay), so healed drift can't disguise a nondeterministic pass/fail — this measures the replay stability of the saved script against the configured URL. Each replay is billed as a rerun, same as a fresh run: 0.5 credits for a frontend replay, 0.2 credits for a backend replay (legacy V2 accounts: FE rerun remains free) — so `--runs N` costs roughly N×0.5 credits for a frontend test. A one-line stderr advisory is printed before a backend replay.

src/commands/test.environment-tunnel.spec.ts

@@ -30,6 +30,10 @@ vi.mock('../lib/local-target.js', async importOriginal => ({
 const clientId = '11111111-2222-3333-4444-555555555555';
 const savedUrl = 'http://127.0.0.1:5173/';
 const common = { profile: 'default', output: 'json' as const, debug: false };
+/**
+ * Build an offline command fixture with configurable API responses and a fake tunnel.
+ * Captured requests, output, and tunnel lifecycle calls expose dispatch and cleanup ordering.
+ */
 function fixture(
   options: {
     url?: string;
@@ -38,6 +42,7 @@ function fixture(
     echo?: boolean;
     mintStatus?: number;
     triggerStatus?: number;
+    cancelStatus?: number;
     batchResponse?: (body: Record<string, unknown>) => Record<string, unknown>;
     envs?: Array<Record<string, unknown>>;
     envsByProject?: Record<string, Array<Record<string, unknown>>>;
@@ -85,6 +90,7 @@ function fixture(
       createdFrom: 'cli',
       updatedAt: '2026-10-01',
     };
+  /** Serve configured API fixtures and record requests without contacting a live service. */
   const fetchImpl = vi.fn(async (input: Parameters<typeof fetch>[0], init: RequestInit = {}) => {
     const parsed = new URL(
       typeof input === 'string' ? input : input instanceof URL ? input.href : input.url,
@@ -144,7 +150,8 @@ function fixture(
           );
     if (method === 'DELETE' && path === `/tunnel/${clientId}`)
       return new Response(null, { status: 204 });
-    if (method === 'POST' && path.endsWith('/cancel'))
+    if (method === 'POST' && path.endsWith('/cancel')) {
+      if (options.cancelStatus) return error(options.cancelStatus, 'cancellation unavailable');
       return json({
         runId: path.split('/')[2],
         testId: path.split('/')[2]!.replace('run_', ''),
@@ -165,6 +172,7 @@ function fixture(
         stepSummary: { total: 1, completed: 0, passedCount: 0, failedCount: 0 },
         alreadyCancelled: false,
       });
+    }
     if (
       method === 'POST' &&
       (path.endsWith('/runs') || path.endsWith('/rerun') || path.endsWith('/run'))
@@ -1192,6 +1200,13 @@ describe('environment tunnel resolution', () => {
           accepted: [{ testId: 'a', runId: 'run_a' }],
           notRunTestIds: ['b'],
         });
+        if (command === 'rerun')
+          expect(JSON.parse(f.stdout[0]!)).toMatchObject({
+            partial: true,
+            failedChunkTestIds: [],
+            notAttemptedTestIds: ['b'],
+            unconfirmedTestIds: ['b'],
+          });
         // eslint-disable-next-line security/detect-non-literal-fs-filename -- fixture-owned temporary file.
         expect(JSON.parse(readFileSync(path, 'utf8'))).toMatchObject({
           total: 2,
@@ -1205,6 +1220,162 @@ describe('environment tunnel resolution', () => {
     },
   );
 
+  it.each([
+    { all: false, output: 'json' as const, failAt: 2, cancelStatus: undefined },
+    { all: true, output: 'json' as const, failAt: 2, cancelStatus: undefined },
+    { all: false, output: 'json' as const, failAt: 3, cancelStatus: undefined },
+    { all: false, output: 'text' as const, failAt: 2, cancelStatus: undefined },
+    { all: false, output: 'json' as const, failAt: 3, cancelStatus: 403 },
+  ])(
+    'preserves tunnel rerun dispatch details and recovery hints for all=$all output=$output failAt=$failAt cancelStatus=$cancelStatus',
+    async ({ all, output, failAt, cancelStatus }) => {
+      const ids = Array.from({ length: 21 }, (_, index) => `test_${index}`);
+      const advisory = { feature: 'autoHeal', message: 'The auto-heal opt-out was not enforced.' };
+      let dispatches = 0;
+      const fixtureOptions = {
+        tests: ids.map(id => ({
+          id,
+          name: id,
+          type: 'frontend',
+          projectId: 'P',
+          status: 'passed',
+          createdFrom: 'cli',
+        })),
+        triggerStatus: undefined as number | undefined,
+        cancelStatus,
+        onTrigger: () => {
+          if (++dispatches === failAt) fixtureOptions.triggerStatus = 404;
+        },
+        batchResponse: (body: Record<string, unknown>) => ({
+          advisories: [advisory, advisory],
+          closure: {
+            byProject: [
+              {
+                projectId: 'P',
+                testIds: body.testIds,
+                addedProducers: [],
+                addedTeardowns: [],
+                clearedCaptured: 1,
+              },
+            ],
+          },
+        }),
+      };
+      const f = fixture(fixtureOptions);
+      const outputBeforeCleanup: string[][] = [];
+      const fetchImpl = f.deps.fetchImpl;
+      f.deps.fetchImpl = vi.fn(async (input, init) => {
+        const url =
+          typeof input === 'string' ? input : input instanceof URL ? input.href : input.url;
+        if (url.endsWith('/cancel') || init?.method === 'DELETE')
+          outputBeforeCleanup.push([...f.stderr]);
+        return fetchImpl(input, init);
+      });
+      const summaryFile = join(
+        mkdtempSync(join(tmpdir(), 'env-dispatch-partial-')),
+        'summary.json',
+      );
+      const reportFile = `${summaryFile}.xml`;
+      const error = await runTestRerun(
+        {
+          ...common,
+          output,
+          testIds: all ? [] : ids,
+          all,
+          projectId: 'P',
+          wait: false,
+          timeoutSeconds: 60,
+          maxConcurrency: 10,
+          autoHeal: false,
+          autoHealExplicit: false,
+          skipDependencies: false,
+          summaryFile,
+          report: 'junit',
+          reportFile,
+        },
+        f.deps,
+      ).catch(err => err);
+      expect(error).toMatchObject({ code: 'NOT_FOUND', exitCode: 4, requestId: 'r' });
+      const posts = f.calls.filter(call => call.path === '/tests/batch/rerun');
+      expect(posts).toHaveLength(failAt);
+      expect(posts.every(call => call.body.tunnelClientId === clientId)).toBe(true);
+      const acknowledged = ids.slice(0, (failAt - 1) * 10);
+      const failed = ids.slice((failAt - 1) * 10, failAt * 10);
+      const notAttempted = ids.slice(failAt * 10);
+      expect(f.stdout).toHaveLength(1);
+      if (output === 'json') {
+        const partial = JSON.parse(f.stdout[0]!);
+        expect(partial).toMatchObject({
+          partial: true,
+          failedChunkTestIds: failed,
+          notAttemptedTestIds: notAttempted,
+          unconfirmedTestIds: [...failed, ...notAttempted],
+          notRunTestIds: [...failed, ...notAttempted],
+          advisories: [advisory],
+          closure: {
+            byProject: [
+              {
+                projectId: 'P',
+                testIds: acknowledged,
+                addedProducers: [],
+                addedTeardowns: [],
+                clearedCaptured: failAt - 1,
+              },
+            ],
+          },
+        });
+        expect(partial.accepted.map((row: { runId: string }) => row.runId)).toEqual(
+          acknowledged.map(id => `run_${id}`),
+        );
+      } else {
+        expect(f.stdout[0]).toContain(
+          `Failed request tests (dispatch unconfirmed): ${failed.join(', ')}`,
+        );
+        expect(f.stdout[0]).toContain(`Not attempted by this command: ${notAttempted.join(', ')}`);
+        expect(f.stdout[0]).not.toContain(advisory.message);
+      }
+      expect(f.stderr.filter(line => line === `[advisory] ${advisory.message}`)).toHaveLength(1);
+      const runIds = acknowledged.map(id => `run_${id}`).join(' ');
+      const recoveryHints = [
+        `Inspect acknowledged runs with: testsprite test wait ${runIds}`,
+        `Cancel with: testsprite test cancel ${runIds}`,
+      ];
+      expect(outputBeforeCleanup.length).toBeGreaterThan(0);
+      for (const output of outputBeforeCleanup)
+        expect(output).toEqual(expect.arrayContaining(recoveryHints));
+      expect(f.stdout[0]).not.toContain('testsprite test wait');
+      expect(f.stdout[0]).not.toContain('testsprite test cancel');
+      const summary = JSON.parse(readFileSync(summaryFile, 'utf8'));
+      expect(summary).toMatchObject({ total: 21, failed: 10, skipped: notAttempted.length });
+      for (const id of failed)
+        expect(summary.runs).toContainEqual({
+          testId: id,
+          status: 'unknown',
+          error: 'dispatch unconfirmed after a failed request; inspect before retrying',
+        });
+      const junit = readFileSync(reportFile, 'utf8');
+      expect(junit).toContain('failures="10"');
+      expect(junit).toContain(`skipped="${notAttempted.length}"`);
+      expect(junit).toContain(
+        '<failure message="dispatch unconfirmed after a failed request; inspect before retrying" type="unknown">',
+      );
+      const cancellations = f.calls.filter(call => call.path.endsWith('/cancel'));
+      // Capacity polling settled earlier runs; cleanup must still cancel every remaining active run.
+      expect(cancellations.map(call => call.path)).toEqual(
+        acknowledged
+          .slice(-10)
+          .slice(failed.length)
+          .map(id => `/runs/run_${id}/cancel`),
+      );
+      if (cancelStatus)
+        expect(f.stderr.filter(line => line.includes('Could not cancel'))).toHaveLength(
+          cancellations.length,
+        );
+      expect(f.calls.at(-1)).toMatchObject({ method: 'DELETE', path: `/tunnel/${clientId}` });
+      expect(f.stop).toHaveBeenCalledTimes(1);
+    },
+  );
+
   it('capacity polling preserves its NOT_FOUND error rather than claiming a rejected rerun dispatch', async () => {
     const f = fixture({ pollStatus: 404 });
     await expect(