Skip to content

docs: correct seven statements that disagree with v0.15.0 #700

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mai1015 wants to merge 3 commits into
base: main
Choose a base branch
from
Open

docs: correct seven statements that disagree with v0.15.0 #700

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
4fb3d5f
docs: correct seven statements that disagree with v0.15.0
mai1015 Oct 29, 2026
543f3d2
docs: drop the remaining plan-owned flags and name SIGHUP in the tele…
mai1015 Oct 29, 2026
74c8f9e
docs: label the --no-auto-heal rerun example as a request
mai1015 Oct 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 16 additions & 14 deletions DOCUMENTATION.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -113,8 +113,9 @@ This is the loop a coding agent runs on its own once you've onboarded it with `t
testsprite setup

# 1 — describe the behavior you want to guarantee, run it, wait
testsprite test create --project proj_8f0f6 --type frontend \
--plan-from ./checkout-flow.plan.json --run --wait --output json
# Set projectId in checkout-flow.plan.json to your project ID.
testsprite test create --plan-from ./checkout-flow.plan.json \
--run --wait --output json
# → exits 1: the run failed

# 2 — pull ONE self-consistent failure bundle to ./.testsprite/failure/
Expand Down Expand Up @@ -293,7 +294,7 @@ Common flags: `--page-size`, `--starting-token`, `--max-items` — same shape as

#### `testsprite test result <test-id>`

Get the latest result for a test — status, started / finished timestamps, video and failure-analysis URLs, summary counts (`passed / failed / skipped`), and correlation fields (`snapshotId`, `runId`, `codeVersion`). With `--include-analysis`, the response also carries an inline `analysis` block (root-cause hypothesis, recommended fix target, failure kind). Backend tests additionally surface the run's captured stdout (`apiOutput`) and Python traceback (`trace`): full content under `--output json` (and in `result.json` / `failure.json` inside failure bundles); text mode prints a bounded 20-line tail of each with a byte count.
Get the latest result for a test — `status`, `verdict`, `executionStatus`, started / finished timestamps, video and failure-analysis URLs, a `summary` string, and correlation fields (`snapshotId`, `runIdIfAvailable`, `codeVersion`). With `--include-analysis`, the response also carries an inline `analysis` block (root-cause hypothesis, recommended fix target, failure kind). Backend tests additionally surface the run's captured stdout (`apiOutput`) and Python traceback (`trace`): full content under `--output json` (and in `result.json` / `failure.json` inside failure bundles); text mode prints a bounded 20-line tail of each with a byte count.

With `--history`, each row also carries the environment the run used: an **ENV** column shows the environment name, or `—` for rows that predate environments. A run-time `--target-url` can reuse an existing same-origin environment or use a temporary one; `--env` can supply settings while the browser visits the override URL. The `targetUrl:` detail line reports the run's target. `--env <name>` filters the history to runs on that environment (server-side; only with `--history`). The latest-result view prints an `environment:` line for the same reason. JSON consumers get `environment: { id, name } | null` on every run/result payload and `targetUrl`; `targetUrlSource` is `null` on a V3 row.

Expand Down Expand Up @@ -394,7 +395,7 @@ testsprite test create --project proj_xxxxxxxx --type frontend --name "Checkout"
--code-file ./checkout.py --step-timeout 45000

# Frontend test from an agent-supplied plan-steps document; trigger + wait inline
testsprite test create --plan-from ./checkout.plan.json --type frontend \
testsprite test create --plan-from ./checkout.plan.json \
--run --wait --timeout 600 --output json

# Print the plan-file skeleton, edit it, then create from it
Expand Down Expand Up @@ -479,13 +480,14 @@ testsprite project update proj_xxxxxxxx --test-id-attributes data-element,data-t

```bash
testsprite project create --type frontend --name <name> --local <port> [--skip-preflight]
testsprite test create --plan-from ./checkout.plan.json --project <project-id>
# Set projectId in checkout.plan.json to the project ID returned above.
testsprite test create --plan-from ./checkout.plan.json
testsprite test run <test-id>
```

`project create --local` stores `http://localhost:<port>`. Alternatively, pass `--url http://localhost:<port>`, `--url http://127.0.0.1:<port>` or `--url http://[::1]:<port>` (an optional trailing `/` is preserved). Loopback stored URLs must use HTTP, an explicit port and no path; HTTPS, missing ports, other `127.x` hosts, unspecified addresses, LAN/RFC1918, link-local and metadata addresses remain refused. `--local-host` still overrides the shorthand host, but is deprecated and hidden from project/environment help; prefer an explicit URL. Existing rows retain their host spelling. `project get` and `project list` expose `originMode: 'local'` in JSON, while text output shows `(Local)`. Loopback creation is frontend-only; `--local` and `--url` remain mutually exclusive. The port is probed before creation unless `--skip-preflight` is given. V2-only accounts receive exit 7 (`local-origin-requires-v3`).

Creation performs **no exploration or plan generation**. `test plan generate --project <project-id>` on a local project is refused **before charge**, with exit 6 and guidance to use `test create --plan-from … --project <id>` followed by `test run <test-id>`. Author a plan using the [plan file format](#plan-file-format), capture the created test id, and run it through the tunnel.
Creation performs **no exploration or plan generation**. `test plan generate --project <project-id>` on a local project is refused **before charge**, with exit 6 and guidance to set `projectId` in the plan and use `test create --plan-from …` followed by `test run <test-id>`. Author a plan using the [plan file format](#plan-file-format), capture the created test id, and run it through the tunnel.

Portal runs of a local project are **BLOCKED for free** until you set a public URL:

Expand Down Expand Up @@ -873,8 +875,8 @@ A second `tunnel start` or process using the same credential takes over, and the
Re-execute one or more tests as a **replay** — distinct from `test run`, which starts a fresh run. A frontend rerun uses the saved script (verbatim unless AI heal-on-drift engages); a backend rerun re-runs the named test together with its producer/teardown dependency closure. Frontend reruns request auto-heal by default; the server controls the default for fresh test runs. A rerun needs a saved run, and a backend rerun expands the dependency closure. A rerun is billed the same as a fresh run — 0.5 credits per FE rerun, 0.2 credits per BE rerun (legacy V2 accounts: FE rerun remains free). Without `--wait`, prints the queued run(s) and exits 0; with `--wait`, polls to terminal with the same exit-code matrix as `test run --wait`.

```bash
# Frontend test — verbatim replay
testsprite test rerun test_xxxxxxxx --wait --output json
# Frontend test — ask for a verbatim replay
testsprite test rerun test_xxxxxxxx --no-auto-heal --wait --output json

# Backend test — reruns the dependency closure (producers + teardowns)
testsprite test rerun test_be_xxxx --wait --output json
Expand Down Expand Up @@ -1156,7 +1158,7 @@ permissions:
contents: read
pull-requests: read
jobs:
gate:
testsprite-tests:
concurrency:
group: testsprite-${{ github.event.pull_request.number || github.event.deployment.sha }}
cancel-in-progress: true
Expand Down Expand Up @@ -1205,7 +1207,7 @@ permissions:
contents: read
pull-requests: read
jobs:
gate:
testsprite-tests:
concurrency:
group: testsprite-${{ github.event.pull_request.number || github.event.deployment.sha }}
cancel-in-progress: true
Expand Down Expand Up @@ -1350,7 +1352,7 @@ Any field named like `password`, `token`, `secret`, `apiKey` / `api_key`, `autho
| `TESTSPRITE_NO_UPDATE_NOTIFIER` | Any non-empty value disables the once-per-24h "new version available" notice |
| `NO_COLOR` | Suppress ANSI escape sequences in ticker output ([no-color.org](https://no-color.org/)) |
| `HTTPS_PROXY` / `HTTP_PROXY` / `NO_PROXY` | Standard proxy support - API traffic is routed through the configured proxy |
| `TESTSPRITE_NO_SKILL_WARNING` | Any non-empty value silences the "verify skill not installed" reminder (CI / manual use) |
| `TESTSPRITE_NO_SKILL_WARNING` | Any value except blank, `0`, `false`, or `no` (after trimming and ignoring case) silences the "verify skill not installed" reminder (CI / manual use) |
| `TESTSPRITE_NO_TELEMETRY` / `DO_NOT_TRACK` | Any truthy value (not `0`/`false`/empty) disables usage telemetry (see Telemetry below) |
| `TESTSPRITE_PORTAL_URL` | Override the Portal origin used for `dashboardUrl` links (non-prod environments) |
| `TESTSPRITE_CLIENT` | Identifies a wrapper driving the CLI (`<name>/<version>`, e.g. `github-action/v1`); appended to the User-Agent and reported as telemetry `client`. Invalid values are ignored |
Expand All @@ -1361,7 +1363,7 @@ Authenticated runs send one best-effort "command outcome" event per invocation
to TestSprite (`POST /api/cli/v1/telemetry`) so we can measure which commands
run and diagnose failures. Each event carries only:

- the command name (e.g. `test run`), the outcome (`success`/`error`/`abort`),
- the command name (e.g. `test run`), the outcome (`success`/`error`),
the exit code, a machine error **code** (e.g. `VALIDATION_ERROR`), and the
duration;
- context: CLI version, OS, Node version, output mode, CI-vs-interactive (`ci`),
Expand Down Expand Up @@ -1395,7 +1397,7 @@ It **never** sends: your API key, target URLs, free-form flag or argument values
run ids (other than the `ci run` execution id above — the server's own UUID, which names nothing about you or your code), repository names, or error **messages**. The event is a fixed
allowlist; timeout seconds and the local concurrency limit are bounded numeric controls. It is bounded to ~1s and fully best-effort — it never delays beyond that,
never changes a command's behavior or exit code, and is skipped entirely when no
API key is configured or under `--dry-run`.
API key is configured, under `--dry-run`, or when the command is interrupted (Ctrl-C / SIGTERM / SIGHUP).

Opt out with `TESTSPRITE_NO_TELEMETRY=1` or the cross-tool
`DO_NOT_TRACK=1` (any truthy value; `0`/`false`/empty do not opt out).
Expand Down Expand Up @@ -1483,7 +1485,7 @@ testsprite ci run --project proj_xxxxxxxx \
```

- **Waiting is automatic.** The command follows accepted runs to a terminal verdict (or `--timeout`, default 600 s). Skipped tests warn by default; add `--fail-on-skipped` for a strict gate.
- **`--report junit --report-file <path>`** optionally writes a JUnit XML sidecar for dispatched runs. Skipped tests are currently omitted from JUnit; the unified summary includes them.
- **`--report junit --report-file <path>`** optionally writes a JUnit XML sidecar. Skipped tests appear as `<skipped>` testcases with their reasons; the unified summary includes them.
- **`--summary-file <path>`** writes a compact machine summary (`{ total, passed, failed, skipped, timedOut, runs[] }`) you can read from any step — `failed` counts only dispatched runs that failed; tests that never dispatched (deferred / conflicted / not found / skipped) count under `skipped`. **`--gh-output`** additionally emits a job-summary table and annotations (`::error::` for failures/timeouts, `::warning::` for never-dispatched tests); it auto-enables under `GITHUB_ACTIONS=true`, so you rarely pass it by hand.

Archive any requested reports with a step that also runs after failure (`if: always()` in GitHub Actions).
Expand Down
10 changes: 6 additions & 4 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,8 +79,9 @@ From there, the loop runs on its own — an example session, typed by the coding

```bash
# 1 — describe the behavior you want to guarantee, run it, wait
testsprite test create --project proj_8f0f6 --type frontend \
--plan-from ./checkout-flow.plan.json --run --wait --output json
# Set projectId in checkout-flow.plan.json to your project ID.
testsprite test create --plan-from ./checkout-flow.plan.json \
--run --wait --output json
# → exits 1: the run failed

# 2 — pull ONE self-consistent failure bundle
Expand Down Expand Up @@ -135,7 +136,8 @@ Use an API key with `run:tunnel`; the CLI names the missing scope if the key can

```bash
testsprite project create --type frontend --name "My local app" --local 3000
testsprite test create --plan-from ./checkout.plan.json --project <project-id>
# Set projectId in checkout.plan.json to the project ID returned above.
testsprite test create --plan-from ./checkout.plan.json
testsprite test run <test-id> # the saved loopback URL opens a tunnel and waits
testsprite test run <id> <id> --max-concurrency 5
testsprite test run --all --project <project-id>
Expand Down Expand Up @@ -190,7 +192,7 @@ On an owned tunnel, Ctrl-C or a timeout cancels unfinished runs by default, and
| | `test plan accept` | Turn staged proposals into real tests — all of them, or a subset with `--only` |
| | `project docs upload` | Upload an API spec or PRD as a project source, so generation has something to read |
| **Run** | `test run` | Trigger a fresh run; saved loopback environments open a tunnel and wait automatically; `--wait` blocks until terminal; `--all --project <id>` runs all tests in a project in wave order `--env <name>` runs against a named environment. |
| | `test rerun` | Replay one/many tests (FE verbatim; BE with deps); billed as a run (0.5 credits FE / 0.2 BE); `--all --project <id>` reruns all tests `--env <name>` replays against a named environment; loopback URLs open a tunnel automatically when the server supports it. |
| | `test rerun` | Replay one/many tests (FE auto-heal by default; `--no-auto-heal` for verbatim; BE with deps); billed as a run (0.5 credits FE / 0.2 BE); `--all --project <id>` reruns all tests `--env <name>` replays against a named environment; loopback URLs open a tunnel automatically when the server supports it. |
| | `test flaky` | Replay a test several times (auto-heal off) and report a stability score |
| | `test wait` | Block on one or more `runId`s until terminal |
| | `test cancel` | Cancel one or more in-flight runs (Ctrl-C during `--wait` only detaches, `cancel` is the real stop — except a run with an owned tunnel, which Ctrl-C cancels by default) |
Expand Down
Loading