From 4fb3d5f7467ea18e167849f22c5a87b14eb93fc6 Mon Sep 17 00:00:00 2001 From: mai1015 Date: Wed, 28 Oct 2026 23:54:25 -0400 Subject: [PATCH 1/3] docs: correct seven statements that disagree with v0.15.0 - --plan-from recipes passed --project/--type, which the plan-file path ignores; say to set projectId in the plan JSON instead. - The "verbatim replay" rerun example omitted --no-auto-heal (FE auto-heal is on by default); same for the README command table. - test result: the latest-result object has a summary string, runIdIfAvailable, verdict and executionStatus, not counts and runId. - Both finished CI workflow shapes used job id `gate`; ci init and the branch-protection guidance use `testsprite-tests`. - TESTSPRITE_NO_SKILL_WARNING: blank, 0, false and no keep the reminder. - ci run JUnit includes skipped tests as testcases. - Telemetry: interrupted commands send no event. Co-Authored-By: Claude Opus 5.5 --- DOCUMENTATION.md | 24 +++++++++++++----------- README.md | 10 ++++++---- 2 files changed, 19 insertions(+), 15 deletions(-) diff --git a/DOCUMENTATION.md b/DOCUMENTATION.md index da69607..b44e6f3 100644 --- a/DOCUMENTATION.md +++ b/DOCUMENTATION.md @@ -113,8 +113,9 @@ This is the loop a coding agent runs on its own once you've onboarded it with `t testsprite setup # 1 — describe the behavior you want to guarantee, run it, wait -testsprite test create --project proj_8f0f6 --type frontend \ - --plan-from ./checkout-flow.plan.json --run --wait --output json +# Set projectId in checkout-flow.plan.json to your project ID. +testsprite test create --plan-from ./checkout-flow.plan.json \ + --run --wait --output json # → exits 1: the run failed # 2 — pull ONE self-consistent failure bundle to ./.testsprite/failure/ @@ -293,7 +294,7 @@ Common flags: `--page-size`, `--starting-token`, `--max-items` — same shape as #### `testsprite test result ` -Get the latest result for a test — status, started / finished timestamps, video and failure-analysis URLs, summary counts (`passed / failed / skipped`), and correlation fields (`snapshotId`, `runId`, `codeVersion`). With `--include-analysis`, the response also carries an inline `analysis` block (root-cause hypothesis, recommended fix target, failure kind). Backend tests additionally surface the run's captured stdout (`apiOutput`) and Python traceback (`trace`): full content under `--output json` (and in `result.json` / `failure.json` inside failure bundles); text mode prints a bounded 20-line tail of each with a byte count. +Get the latest result for a test — `status`, `verdict`, `executionStatus`, started / finished timestamps, video and failure-analysis URLs, a `summary` string, and correlation fields (`snapshotId`, `runIdIfAvailable`, `codeVersion`). With `--include-analysis`, the response also carries an inline `analysis` block (root-cause hypothesis, recommended fix target, failure kind). Backend tests additionally surface the run's captured stdout (`apiOutput`) and Python traceback (`trace`): full content under `--output json` (and in `result.json` / `failure.json` inside failure bundles); text mode prints a bounded 20-line tail of each with a byte count. With `--history`, each row also carries the environment the run used: an **ENV** column shows the environment name, or `—` for rows that predate environments. A run-time `--target-url` can reuse an existing same-origin environment or use a temporary one; `--env` can supply settings while the browser visits the override URL. The `targetUrl:` detail line reports the run's target. `--env ` filters the history to runs on that environment (server-side; only with `--history`). The latest-result view prints an `environment:` line for the same reason. JSON consumers get `environment: { id, name } | null` on every run/result payload and `targetUrl`; `targetUrlSource` is `null` on a V3 row. @@ -479,7 +480,8 @@ testsprite project update proj_xxxxxxxx --test-id-attributes data-element,data-t ```bash testsprite project create --type frontend --name --local [--skip-preflight] -testsprite test create --plan-from ./checkout.plan.json --project +# Set projectId in checkout.plan.json to the project ID returned above. +testsprite test create --plan-from ./checkout.plan.json testsprite test run ``` @@ -874,7 +876,7 @@ Re-execute one or more tests as a **replay** — distinct from `test run`, which ```bash # Frontend test — verbatim replay -testsprite test rerun test_xxxxxxxx --wait --output json +testsprite test rerun test_xxxxxxxx --no-auto-heal --wait --output json # Backend test — reruns the dependency closure (producers + teardowns) testsprite test rerun test_be_xxxx --wait --output json @@ -1156,7 +1158,7 @@ permissions: contents: read pull-requests: read jobs: - gate: + testsprite-tests: concurrency: group: testsprite-${{ github.event.pull_request.number || github.event.deployment.sha }} cancel-in-progress: true @@ -1205,7 +1207,7 @@ permissions: contents: read pull-requests: read jobs: - gate: + testsprite-tests: concurrency: group: testsprite-${{ github.event.pull_request.number || github.event.deployment.sha }} cancel-in-progress: true @@ -1350,7 +1352,7 @@ Any field named like `password`, `token`, `secret`, `apiKey` / `api_key`, `autho | `TESTSPRITE_NO_UPDATE_NOTIFIER` | Any non-empty value disables the once-per-24h "new version available" notice | | `NO_COLOR` | Suppress ANSI escape sequences in ticker output ([no-color.org](https://no-color.org/)) | | `HTTPS_PROXY` / `HTTP_PROXY` / `NO_PROXY` | Standard proxy support - API traffic is routed through the configured proxy | -| `TESTSPRITE_NO_SKILL_WARNING` | Any non-empty value silences the "verify skill not installed" reminder (CI / manual use) | +| `TESTSPRITE_NO_SKILL_WARNING` | Any value except blank, `0`, `false`, or `no` (after trimming and ignoring case) silences the "verify skill not installed" reminder (CI / manual use) | | `TESTSPRITE_NO_TELEMETRY` / `DO_NOT_TRACK` | Any truthy value (not `0`/`false`/empty) disables usage telemetry (see Telemetry below) | | `TESTSPRITE_PORTAL_URL` | Override the Portal origin used for `dashboardUrl` links (non-prod environments) | | `TESTSPRITE_CLIENT` | Identifies a wrapper driving the CLI (`/`, e.g. `github-action/v1`); appended to the User-Agent and reported as telemetry `client`. Invalid values are ignored | @@ -1361,7 +1363,7 @@ Authenticated runs send one best-effort "command outcome" event per invocation to TestSprite (`POST /api/cli/v1/telemetry`) so we can measure which commands run and diagnose failures. Each event carries only: -- the command name (e.g. `test run`), the outcome (`success`/`error`/`abort`), +- the command name (e.g. `test run`), the outcome (`success`/`error`), the exit code, a machine error **code** (e.g. `VALIDATION_ERROR`), and the duration; - context: CLI version, OS, Node version, output mode, CI-vs-interactive (`ci`), @@ -1395,7 +1397,7 @@ It **never** sends: your API key, target URLs, free-form flag or argument values run ids (other than the `ci run` execution id above — the server's own UUID, which names nothing about you or your code), repository names, or error **messages**. The event is a fixed allowlist; timeout seconds and the local concurrency limit are bounded numeric controls. It is bounded to ~1s and fully best-effort — it never delays beyond that, never changes a command's behavior or exit code, and is skipped entirely when no -API key is configured or under `--dry-run`. +API key is configured, under `--dry-run`, or when the command is interrupted (Ctrl-C / SIGTERM). Opt out with `TESTSPRITE_NO_TELEMETRY=1` or the cross-tool `DO_NOT_TRACK=1` (any truthy value; `0`/`false`/empty do not opt out). @@ -1483,7 +1485,7 @@ testsprite ci run --project proj_xxxxxxxx \ ``` - **Waiting is automatic.** The command follows accepted runs to a terminal verdict (or `--timeout`, default 600 s). Skipped tests warn by default; add `--fail-on-skipped` for a strict gate. -- **`--report junit --report-file `** optionally writes a JUnit XML sidecar for dispatched runs. Skipped tests are currently omitted from JUnit; the unified summary includes them. +- **`--report junit --report-file `** optionally writes a JUnit XML sidecar. Skipped tests appear as `` testcases with their reasons; the unified summary includes them. - **`--summary-file `** writes a compact machine summary (`{ total, passed, failed, skipped, timedOut, runs[] }`) you can read from any step — `failed` counts only dispatched runs that failed; tests that never dispatched (deferred / conflicted / not found / skipped) count under `skipped`. **`--gh-output`** additionally emits a job-summary table and annotations (`::error::` for failures/timeouts, `::warning::` for never-dispatched tests); it auto-enables under `GITHUB_ACTIONS=true`, so you rarely pass it by hand. Archive any requested reports with a step that also runs after failure (`if: always()` in GitHub Actions). diff --git a/README.md b/README.md index ce69eb9..9133de0 100644 --- a/README.md +++ b/README.md @@ -79,8 +79,9 @@ From there, the loop runs on its own — an example session, typed by the coding ```bash # 1 — describe the behavior you want to guarantee, run it, wait -testsprite test create --project proj_8f0f6 --type frontend \ - --plan-from ./checkout-flow.plan.json --run --wait --output json +# Set projectId in checkout-flow.plan.json to your project ID. +testsprite test create --plan-from ./checkout-flow.plan.json \ + --run --wait --output json # → exits 1: the run failed # 2 — pull ONE self-consistent failure bundle @@ -135,7 +136,8 @@ Use an API key with `run:tunnel`; the CLI names the missing scope if the key can ```bash testsprite project create --type frontend --name "My local app" --local 3000 -testsprite test create --plan-from ./checkout.plan.json --project +# Set projectId in checkout.plan.json to the project ID returned above. +testsprite test create --plan-from ./checkout.plan.json testsprite test run # the saved loopback URL opens a tunnel and waits testsprite test run --max-concurrency 5 testsprite test run --all --project @@ -190,7 +192,7 @@ On an owned tunnel, Ctrl-C or a timeout cancels unfinished runs by default, and | | `test plan accept` | Turn staged proposals into real tests — all of them, or a subset with `--only` | | | `project docs upload` | Upload an API spec or PRD as a project source, so generation has something to read | | **Run** | `test run` | Trigger a fresh run; saved loopback environments open a tunnel and wait automatically; `--wait` blocks until terminal; `--all --project ` runs all tests in a project in wave order `--env ` runs against a named environment. | -| | `test rerun` | Replay one/many tests (FE verbatim; BE with deps); billed as a run (0.5 credits FE / 0.2 BE); `--all --project ` reruns all tests `--env ` replays against a named environment; loopback URLs open a tunnel automatically when the server supports it. | +| | `test rerun` | Replay one/many tests (FE auto-heal by default; `--no-auto-heal` for verbatim; BE with deps); billed as a run (0.5 credits FE / 0.2 BE); `--all --project ` reruns all tests `--env ` replays against a named environment; loopback URLs open a tunnel automatically when the server supports it. | | | `test flaky` | Replay a test several times (auto-heal off) and report a stability score | | | `test wait` | Block on one or more `runId`s until terminal | | | `test cancel` | Cancel one or more in-flight runs (Ctrl-C during `--wait` only detaches, `cancel` is the real stop — except a run with an owned tunnel, which Ctrl-C cancels by default) | From 543f3d267c4ebcf7114f4f07529fb8c66eea3af0 Mon Sep 17 00:00:00 2001 From: mai1015 Date: Thu, 29 Oct 2026 01:38:18 -0400 Subject: [PATCH 2/3] docs: drop the remaining plan-owned flags and name SIGHUP in the telemetry rule Co-Authored-By: Claude Opus 5.5 --- DOCUMENTATION.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/DOCUMENTATION.md b/DOCUMENTATION.md index b44e6f3..5d377b5 100644 --- a/DOCUMENTATION.md +++ b/DOCUMENTATION.md @@ -395,7 +395,7 @@ testsprite test create --project proj_xxxxxxxx --type frontend --name "Checkout" --code-file ./checkout.py --step-timeout 45000 # Frontend test from an agent-supplied plan-steps document; trigger + wait inline -testsprite test create --plan-from ./checkout.plan.json --type frontend \ +testsprite test create --plan-from ./checkout.plan.json \ --run --wait --timeout 600 --output json # Print the plan-file skeleton, edit it, then create from it @@ -487,7 +487,7 @@ testsprite test run `project create --local` stores `http://localhost:`. Alternatively, pass `--url http://localhost:`, `--url http://127.0.0.1:` or `--url http://[::1]:` (an optional trailing `/` is preserved). Loopback stored URLs must use HTTP, an explicit port and no path; HTTPS, missing ports, other `127.x` hosts, unspecified addresses, LAN/RFC1918, link-local and metadata addresses remain refused. `--local-host` still overrides the shorthand host, but is deprecated and hidden from project/environment help; prefer an explicit URL. Existing rows retain their host spelling. `project get` and `project list` expose `originMode: 'local'` in JSON, while text output shows `(Local)`. Loopback creation is frontend-only; `--local` and `--url` remain mutually exclusive. The port is probed before creation unless `--skip-preflight` is given. V2-only accounts receive exit 7 (`local-origin-requires-v3`). -Creation performs **no exploration or plan generation**. `test plan generate --project ` on a local project is refused **before charge**, with exit 6 and guidance to use `test create --plan-from … --project ` followed by `test run `. Author a plan using the [plan file format](#plan-file-format), capture the created test id, and run it through the tunnel. +Creation performs **no exploration or plan generation**. `test plan generate --project ` on a local project is refused **before charge**, with exit 6 and guidance to set `projectId` in the plan and use `test create --plan-from …` followed by `test run `. Author a plan using the [plan file format](#plan-file-format), capture the created test id, and run it through the tunnel. Portal runs of a local project are **BLOCKED for free** until you set a public URL: @@ -1397,7 +1397,7 @@ It **never** sends: your API key, target URLs, free-form flag or argument values run ids (other than the `ci run` execution id above — the server's own UUID, which names nothing about you or your code), repository names, or error **messages**. The event is a fixed allowlist; timeout seconds and the local concurrency limit are bounded numeric controls. It is bounded to ~1s and fully best-effort — it never delays beyond that, never changes a command's behavior or exit code, and is skipped entirely when no -API key is configured, under `--dry-run`, or when the command is interrupted (Ctrl-C / SIGTERM). +API key is configured, under `--dry-run`, or when the command is interrupted (Ctrl-C / SIGTERM / SIGHUP). Opt out with `TESTSPRITE_NO_TELEMETRY=1` or the cross-tool `DO_NOT_TRACK=1` (any truthy value; `0`/`false`/empty do not opt out). From 74c8f9e73d21ff433829ddb609869d438ea15978 Mon Sep 17 00:00:00 2001 From: mai1015 Date: Thu, 29 Oct 2026 14:38:33 -0400 Subject: [PATCH 3/3] docs: label the --no-auto-heal rerun example as a request Co-Authored-By: Claude Opus 5.5 --- DOCUMENTATION.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/DOCUMENTATION.md b/DOCUMENTATION.md index 5d377b5..7e976b2 100644 --- a/DOCUMENTATION.md +++ b/DOCUMENTATION.md @@ -875,7 +875,7 @@ A second `tunnel start` or process using the same credential takes over, and the Re-execute one or more tests as a **replay** — distinct from `test run`, which starts a fresh run. A frontend rerun uses the saved script (verbatim unless AI heal-on-drift engages); a backend rerun re-runs the named test together with its producer/teardown dependency closure. Frontend reruns request auto-heal by default; the server controls the default for fresh test runs. A rerun needs a saved run, and a backend rerun expands the dependency closure. A rerun is billed the same as a fresh run — 0.5 credits per FE rerun, 0.2 credits per BE rerun (legacy V2 accounts: FE rerun remains free). Without `--wait`, prints the queued run(s) and exits 0; with `--wait`, polls to terminal with the same exit-code matrix as `test run --wait`. ```bash -# Frontend test — verbatim replay +# Frontend test — ask for a verbatim replay testsprite test rerun test_xxxxxxxx --no-auto-heal --wait --output json # Backend test — reruns the dependency closure (producers + teardowns)