diff --git a/.agents/skills/kane-cli/SKILL.md b/.agents/skills/kane-cli/SKILL.md index 1937ed9..1aebb55 100644 --- a/.agents/skills/kane-cli/SKILL.md +++ b/.agents/skills/kane-cli/SKILL.md @@ -1,6 +1,6 @@ --- name: kane-cli -description: "End-to-end (E2E) test generation and execution with KaneCLI. Generate tests from a PRD, specification, or user-provided description via the assurance commands (use-cases, acceptance criteria, scenarios, runnable _test.md tests; never write test cases by hand). Execute them on desktop web (Chrome, the default), Android and iOS: emulator/simulator locally on macOS Apple Silicon or via testrun run --remote, and real cloud devices from any machine (--target android|ios). Prove coverage with evidence packs and kane-cli cover. Use for any task needing a real browser or mobile app: navigation, clicks, form filling, UI checks, screenshots, deploy verification. Parse NDJSON output, inspect logs, and save replayable _test.md files. Share the assurance store (.context/) with a team through a GitHub repository, S3-compatible bucket, or folder via kane-cli context sync, and resolve decisions requested during a sync rebase." +description: "End-to-end (E2E) test generation and execution with KaneCLI. Generate tests from a PRD, specification, or user-provided description via the assurance commands (use-cases, acceptance criteria, scenarios, runnable _test.md tests; never write test cases by hand). Execute them on desktop web (Chrome, the default), Android and iOS: emulator/simulator locally on macOS Apple Silicon or via testrun run --remote, and real cloud devices from any machine (--target android|ios). Prove coverage with evidence packs and kane-cli cover. Work in a team's shared Test Manager project from a linked folder: link it with --project, save sessions, run the designed tests on their Test Manager cases, and read the project's coverage. Use for any task needing a real browser or mobile app: navigation, clicks, form filling, UI checks, screenshots, deploy verification. Parse NDJSON output, inspect logs, and save replayable _test.md files. Share a local assurance store (.context/) through a GitHub repository, S3-compatible bucket, or folder via kane-cli context sync, and resolve decisions requested during a sync rebase." --- # Kane CLI — End-to-End Test Generation and Execution Skill @@ -11,6 +11,12 @@ KaneCLI is one E2E package with three pillars: 2. **Execute**: desktop web (Chrome, the default), Android emulator, iOS simulator, and real Android and iOS devices in the TestMu AI cloud. 3. **Prove**: a sealed evidence pack per run, and coverage (`kane-cli cover`) measured against acceptance criteria. +**Working with a team?** On kane-cli 0.10.0+ a team works in a **shared project**: one Test Manager project that its +folders are linked to. What one person saves there (use-cases, tests, Test Manager cases, coverage) the whole team sees. +Test files stay in the folder where they were saved until they're brought over (`references/shared-projects.md` §7). In +agent mode, a folder that isn't set up yet goes local unless you link it with `--project `. So decide where the work +goes before the first assurance command, and read `references/shared-projects.md` first. + Use `kane-cli` to validate complete user journeys and design requirement-linked, coverage-accounted suites from a PRD, specification, or user-provided description via assurance commands. Never write test cases by hand. Do NOT use Playwright, Puppeteer, or Selenium directly. Use `--agent` for `run` and `testmd run`. `testrun run` emits NDJSON when **stdin** is not a TTY (0.8.21+ also takes `--agent`, which selects the same stream), so every `testrun run` line you write ends in `< /dev/null` (bash and zsh: macOS, Linux, Git Bash; in cmd.exe write `< NUL`; from PowerShell run it through cmd: `cmd /c "kane-cli testrun run … < NUL"`). Check the first stdout line: on 0.8.17+ it is `{"type":"stream_start"…}`. A prose plan there instead means the CLI is in terminal mode: it shows the human view and, after a real run, opens an evidence table that waits until `q` or Esc, so the process never exits on its own; stop it and rerun with the redirect. An empty stdout with a message on stderr is a usage error: read it. Assurance conversational commands use `--mode agent`. **Authoring test cases or scenarios?** Never write them by hand: every test case comes from the **assurance** commands — Read `references/assurance.md` first. @@ -54,7 +60,7 @@ On Windows PowerShell: `$env:KANE_CLI_USER_AGENT=''; kane-cli run **Watch mode.** Use the person's saved preference (`references/agent-config.md`). With none saved, show the browser unless the preflight says there is no display, an SSH session, or CI: then add `--headless`. -**Keeping runs.** When the person's saved purpose is `suite` or `ask`, add `--name ` to every one-off `run`. A named run is recorded as a `_test.md` while it runs, so keeping it afterwards costs nothing, and a run launched without a name cannot be kept without running again. With `one-off`, leave the flag out. Details: `references/first-run.md` §4. +**Keeping runs.** When the person's saved purpose is `suite` or `ask`, add `--name ` to every one-off `run`. A named run is recorded as a `_test.md` while it runs, so keeping it afterwards costs nothing, and a run launched without a name cannot be kept without running again. With `one-off`, leave the flag out. Details: `references/first-run.md` §4. Run one-offs from a folder that isn't linked: a named run in a linked folder writes its test beside the project's and can mix two projects (`references/shared-projects.md` §9). Bash blocks until kane-cli exits, then hands you the complete stdout. Parse it, summarize what happened, and present the result card. Wait for process completion on `testmd run` too, but parse its own completion event, `test_md_done`. An intermediate `run_end` does not finish a saved test. @@ -144,7 +150,7 @@ For exit code 1 (or `status: "failed"` in `run_end`), present the failure card. The failing step's screenshot lives inside the run's evidence pack (the stderr hint names the pack path): extract it with `unzip "tests/*/steps/*/screenshot.png" -d `, Read it, and show it **under** the card. For the pack layout and deeper diagnosis, see `references/debug.md`. -Exit code 2 means nothing ran: that is a `🟡 Didn't start` card, not a failure (`references/cards.md` §4). +Exit code 2 with no progress events means nothing ran: that is a `🟡 Didn't start` card, not a failure (`references/cards.md` §4). If progress events came before it, steps ran and the run stopped without a result: say so, and use the stopped-early card (`references/cards.md` §5). ### 1.7 After the first result: three choices, then save @@ -163,19 +169,28 @@ When the user's request involves a browser — or writing test cases: - A problem that stops the run → offer the fix from the card; deeper setup lives in `references/setup-and-config.md` - Ready ↓ +**Where does the work go?** (before any assurance command) +- The folder holds its own `.context/` and no link → it works locally, as before (with a link too, kane-cli refuses it `BOTH_STORES`): `references/assurance.md` (to share that store with a team, `references/context-sync.md`). The rows below are for a folder without one +- The folder holds `.testmuai/assurance.json`, or the person works with a team or names a Test Manager project → a **shared project**: Read `references/shared-projects.md` first. In a linked folder never use `kane-cli run` for a first result; its tests go to another project (§9 there) +- The person works alone, or asks to keep the work on this machine → the local store: `references/assurance.md` +- Not clear → ask once: "Should this work go to your team's Test Manager project, or stay in this folder?" + **What does the user want?** +- Tests that exploration (HyperExecute) designed, or a teammate's tests, in this folder → Read `references/shared-projects.md` §7 +- Explore an app on HyperExecute, shape, stop or re-run an exploration, or check what it designed → Read `references/explore.md` first +- A command was refused, or you need the shared-project event and code list → Read `references/agent-contract.md` - A single one-shot browser task → build a `kane-cli run --agent` command (§3 + §4) - A test they want to save / re-run / commit → Read `references/testmd.md` first, then use `kane-cli testmd` - Run a suite of saved tests (several `_test.md` at once) → Read `references/testrun.md` first, then use `kane-cli testrun run … < /dev/null` - Need test cases from a description the user gives in chat, with no document → **don't hand-write them**; write the description to a requirements file and design from it (§6) - Has requirement documents (PRD/spec) and wants a designed suite, coverage accounting, or suite upkeep → Read `references/assurance.md` first — the assurance commands (`context`/`design`/`cover`/`maintain reconcile`, kane-cli 0.6.1+; several features need newer releases, up to 0.8.14+ — the reference marks each) - A **designed** test (its `_test.md` carries an `assurance:` block) failed a run → Read `references/assurance.md` §6.1 **before touching the file** — an edit is adopted as the next design version on the next run; classify the failure first (app bug → report it; requirement changed → reconcile; wording → redesign through the CLI; capability missing → stop), and edit a step of an existing designed test by hand only after reading `references/objectives-cookbook.md` -- Share the context store with a team, join a teammate's, keep two stores level, or resolve a sync conflict → Read `references/context-sync.md` first — `kane-cli context sync`, `kane-cli context push`, `kane-cli context pull` and `kane-cli context clone` (kane-cli 0.8.14+); the store is shared through a location, never by copying or git-merging `.context/` +- Share a **local** context store through a repository, a bucket or a folder, join one, keep two stores level, or resolve a sync conflict → Read `references/context-sync.md` first (kane-cli 0.8.14+). A team in Test Manager uses a shared project instead (`references/shared-projects.md`); a linked folder refuses `kane-cli context sync`, `kane-cli context push` and `kane-cli context clone` (`REMOTE_UNSUPPORTED`), and a location pull (a location name, `--rebase`, `--yes` or `--answer`: `PULL_LOCATION_LINKED`); a plain `kane-cli context pull --mode agent` there brings the project's tests (`references/shared-projects.md` §7); the store is shared through a location, never by copying or git-merging `.context/` - Multiple independent browser tasks → Read `references/parallel.md` first - View, share, or validate run evidence (`.evidence` packs) → Read `references/evidence.md` - Debug a failed run → Read `references/debug.md` - Configure kane-cli or check directory layout → Read `references/setup-and-config.md` -- Browse / create / pick a Test Manager project or folder, or interpret the auto-default event → Read `references/test-manager.md` +- Browse / create / pick a Test Manager project or folder for `kane-cli run`, or interpret the auto-default event → Read `references/test-manager.md`. Linking an assurance folder to a project is different: `references/shared-projects.md` §2–3 - The person wants results saved somewhere else ("change project") → Read `references/test-manager.md` §6. The change is global, and the question must say so - The person wants to change how runs behave ("kane preferences": watch or quiet, one-off or suite) → Read `references/agent-config.md` - The person asks what kane-cli can do, or for the tour again ("kane tour") → show the tour from `references/first-run.md` §2 @@ -204,7 +219,7 @@ kane-cli run "" --agent [options] | Flag | Purpose | Default | |------|---------|---------| | `--headless` | No visible browser window | Off | -| `--max-steps ` | Cap agent reasoning steps | 30 | +| `--max-steps ` | Cap agent reasoning steps | 50 | | `--timeout ` | Hard kill after N seconds | No limit | | `--url ` | Start URL for the run (overrides config `default_url`; bare domains get `https://`) | Config `default_url` | | `--variables ` | Inline variables JSON (for `{{key}}` in objective) | None | @@ -226,7 +241,7 @@ Other flags (`--global-context`, `--local-context`, `--cdp-endpoint`, `--allow-m 1. Collect the names with no value: the `warning`, a dry-run plan's `unresolved[]` rows, or a design run's `variables_declared` and `variables_summary` rows. Design rows list what design declared, not every missing value, and a dry run's `valid: true` says nothing about values: the dry run's `warning` is the check. 2. A value you already have, because the user said it or the requirement document states it, you write into that key in the file the event names (for `run`, `--variables '{"name":{"value":"…"}}'` also works), and you tell the user what you filled and where. 3. The rest you ask for once, in one message, using each variable's `description` when design gave one. Plain values (a URL, an email, a user name) the user gives you here or adds to the file, their choice. Secrets, which are any row with `secret: true`, any description that names a credential, and any name containing `password`, `secret`, `token` or `key`, the user fills in the file and tells you when done; you never ask for the value in chat, never echo it, and report names and file paths only. -4. Then a fresh `--dry-run` of the exact selection: a valid plan with no `warning` is the check. A member whose rows are all filled may run while the others wait. Never fill a placeholder just to silence the check; a throwaway value is right only when the description asks for one, such as a deliberately wrong password. A frontmatter declaration with an empty value also silences the check, so look at the values a step relies on, not only at the warning. A name still empty is typed into the page as written; if a run then failed at that step, say so. +4. Then the check: for saved tests, a fresh `kane-cli testrun run --dry-run --agent < /dev/null` of the exact selection (`run` and `testmd run` have no dry run; for them the `warning` arrives before the first step): a valid plan with no `warning` is the check. A member whose rows are all filled may run while the others wait. Never fill a placeholder just to silence the check; a throwaway value is right only when the description asks for one, such as a deliberately wrong password. A frontmatter declaration with an empty value also silences the check, so look at the values a step relies on, not only at the warning. A name still empty is typed into the page as written; if a run then failed at that step, say so. ### Examples @@ -330,6 +345,8 @@ For one-shot `run`, build post-run logic on `run_end` and process exit. Saved te For full event schemas (`bifurcation` flow fields, `child_agent_*`, `ask_user` semantics, `cancel`/`user_response` outbound events, complete `run_end` field list), Read `references/parsing.md`. +**Which agent flag.** `--agent`: `run`, `testmd run`, `testrun run` (or stdin not a TTY), `projects`, `folders`, `explore`, `config show`, `generate`, `devices`, `apps`. `--mode agent`: `context …`, `design …`, `maintain …`, `cover`. `run` and `testmd run` also accept `--mode agent`; `testrun`, `projects`, `folders` and `explore` refuse it, and the assurance commands refuse `--agent`. With `--agent`, an update notice can still print on stderr; it is not part of the stream. In a linked folder the streams, events and refusal codes are in `references/agent-contract.md`. + The assurance conversational commands (`context ingest`/`context extract`, `design tests`, `maintain reconcile`, `maintain evolve` on 0.8.21+, `cover`) do NOT take `--agent` — they take **`--mode agent`** and speak their own typed stream ending in `done` (open vocabulary — tolerate unknown event types; on 0.7.2+ the stream is strict — every stdout line parses, stderr silent — while on 0.7.1 a merged `context ingest` prints a few prose receipt lines BEFORE the stream — skip to the first `{` line, harmless on 0.7.2+ — and a landing-phase ingest failure ends with prose + exit 1/2 and no stream at all: a refusal, not a crash; on 0.7.2+ those failures ride the stream as `error` + `done`); **for those commands only, exit `3` means paused-and-resumable, not timeout** — schema in `references/assurance-parsing.md`, behavior in `references/assurance.md`. The context sync verbs (`kane-cli context sync`, `kane-cli context push`, `kane-cli context pull`, `kane-cli context clone`; kane-cli 0.8.14+) take `--mode agent` the same way and speak a `sync_*` family ending in `done`; there too exit `3` means a decision or a pull is needed, not a failure — `references/context-sync.md`. `kane-cli context sync setup` refuses without a terminal (`TTY_REQUIRED`): agents use `kane-cli context sync add` and `kane-cli context clone`. `kane-cli testrun run` also emits its own typed stream (`testrun_plan` … terminal `testrun_done`) — schema in `references/testrun.md`. `kane-cli testmd run` may additionally emit `test_md_evidence_ingest` (replay evidence published) and `test_md_bundle_sync` (test bundle synced) — informational; describe in plain language, never surface raw names. The post-run evidence hint (`` evidence: view locally with `kane-cli evidence serve ` ``) is a **stderr** text line, not a stdout event — don't try to parse it from the NDJSON stream; see `references/evidence.md` for how to act on it. @@ -341,7 +358,7 @@ The assurance conversational commands (`context ingest`/`context extract`, `desi When the user describes what to test in chat and has no document, use the description as the requirement. 1. Write the user's words to `requirements/.md` in the project, as given, one heading per feature, nothing invented. Tell the user the file exists and that the tests will cite it. -2. `kane-cli context ingest requirements/.md --mode agent`, review the extracted use-cases at the checkpoint, then `kane-cli design tests …`, exactly as `references/assurance.md` describes. +2. `kane-cli context ingest requirements/.md --mode agent`, review the extracted use-cases at the checkpoint, then `kane-cli design tests …`, exactly as `references/assurance.md` describes. In a shared project add `--project ` the first time, and follow the journey in `references/shared-projects.md` §4 (save each session; the one-question checkpoint). 3. Present the designed tests, gaps, warnings and the variables that need values (§3 **Fill the variables before any run**). The designed tests get their own review checkpoint before any run (`references/assurance.md`). Then hand the set to `kane-cli testrun run … < /dev/null` (`references/assurance.md`, the authoring bridge). A draft in chat cannot be run, and a hand-written `_test.md` carries no requirement link and no coverage accounting; the deliverable is the designed test. @@ -356,6 +373,10 @@ A draft in chat cannot be run, and a hand-written `_test.md` carries no requirem | Run a suite of saved `_test.md` tests as one batch | `references/testrun.md` | | Run a suite on the cloud grid (`--remote`), incl. mobile suites from any machine | `references/testrun.md` §Remote + `references/mobile.md` §Remote | | Run on a real Android/iOS device (`--target android\|ios`), its plan requirements, or a hub session error | `references/mobile.md` §Real devices | +| A team, a Test Manager project, a linked folder (`.testmuai/assurance.json`), a teammate's or exploration's tests | `references/shared-projects.md` | +| This folder's shared project and its id (for a teammate, or a dashboard) | `references/shared-projects.md` §2, "This folder's project id" | +| A shared-project refusal code, event or exit code | `references/agent-contract.md` | +| Explore an app on HyperExecute | `references/explore.md` | | User has requirement docs (PRD/spec) → designed suite, coverage, or suite upkeep | `references/assurance.md` | | Need the assurance NDJSON event schema (`--mode agent`) | `references/assurance-parsing.md` | | Share the context store with a team, join one, keep stores level, or resolve a sync conflict (kane-cli 0.8.14+) | `references/context-sync.md` | diff --git a/.agents/skills/kane-cli/references/agent-config.md b/.agents/skills/kane-cli/references/agent-config.md index fa35bb8..4ab53e7 100644 --- a/.agents/skills/kane-cli/references/agent-config.md +++ b/.agents/skills/kane-cli/references/agent-config.md @@ -34,7 +34,7 @@ They follow the person across agents and projects, and they survive a skill rein | Key | Values | Meaning | |---|---|---| | `preferences.watch` | `visible` · `quiet` · `results-only` | `visible`: no `--headless`. `quiet`, `results-only`: `--headless`. `results-only` also skips the progress summary | -| `preferences.purpose` | `one-off` · `suite` · `ask` | Whether to offer keeping passing runs as saved tests. With `suite` or `ask`, launch every one-off run with `--name ` so keeping it costs nothing (`references/first-run.md` §4) | +| `preferences.purpose` | `one-off` · `suite` · `ask` | Whether to offer keeping passing runs as saved tests. With `suite` or `ask`, launch every one-off run with `--name ` so keeping it costs nothing (`references/first-run.md` §4), except in a folder linked to a shared project (`shared-projects.md` §9) | | `preferences.narration` | `quiet` · `milestones` · `every-step` | How much of the run you recount afterwards. Default `milestones` | | `onboarding.asked` | list of `watch`, `results`, `purpose` | What was already asked. Never ask these again | | `onboarding.first_run_explained` | boolean | The tour was shown | @@ -90,4 +90,4 @@ Before you write, tell the person in one line what you are saving and where. The ## 5. Changing preferences later -When the person says "kane preferences" (or asks to change how runs behave), show the current values in plain words, ask what to change, and write the file again. To change where results go, use the flow in `references/test-manager.md`: that setting is global and belongs to kane-cli. +When the person says "kane preferences" (or asks to change how runs behave), show the current values in plain words, ask what to change, and write the file again. To change where results go, use the flow in `references/test-manager.md`: that setting is global and belongs to kane-cli (in a folder linked to a shared project, designed tests directly in its `.testmuai/tests/` run on that project's cases whatever it says: `shared-projects.md`). diff --git a/.agents/skills/kane-cli/references/agent-contract.md b/.agents/skills/kane-cli/references/agent-contract.md new file mode 100644 index 0000000..1feb643 --- /dev/null +++ b/.agents/skills/kane-cli/references/agent-contract.md @@ -0,0 +1,329 @@ + + +# Shared projects: Agent Contract + +A folder is **linked** when `.testmuai/assurance.json` names a TestMu AI Test Manager project and the folder has no `.context/`. In a linked folder the commands below work in the **shared project**, so the team sees what each person saves. Link a folder with `kane-cli context ingest --project --mode agent` (or `kane-cli context extract --project --mode agent`, or, where `kane-cli context pull --help` lists `--project`, `kane-cli context pull --project --mode agent`); find the id with `kane-cli projects list --search --agent`. In a folder with neither a link nor `.context/`, `--mode agent` without `--project` goes on locally without asking, as before: `context ingest` makes a local store, and `context extract` refuses `NO_STORE` until one exists. + +| Area | Commands | +|---|---| +| Projects | `kane-cli projects list`, `kane-cli projects create` | +| Sources and use-cases | `kane-cli context ingest`, `kane-cli context extract`, `kane-cli context list` | +| Sessions | `kane-cli context sessions`, `kane-cli context sessions save `, `kane-cli context sessions discard ` | +| Tests | `kane-cli design tests`, `kane-cli maintain evolve`, `kane-cli maintain reconcile`, `kane-cli context pull` (`shared-projects.md` §7) | +| Runs and coverage | `kane-cli testmd run _test.md`, `kane-cli testrun run`, `kane-cli cover` | + +Nothing reaches the project until `kane-cli context sessions save`. Other assurance commands are refused in a linked folder (`REMOTE_UNSUPPORTED`, §6). Local-store parsing stays in `references/assurance-parsing.md`. + +## 1. Which flag + +| Command | Agent flag | The wrong one does | +|---|---|---| +| `context ingest`, `context extract`, `context list`, `context sessions` (list, save, discard), `design tests`, `maintain evolve`, `maintain reconcile`, `cover` | `--mode agent` | `--agent` is an unknown option: prose on stderr, no NDJSON, non-zero exit | +| `testmd run` | `--agent` (`--mode agent` works too) | — | +| `testrun run` | `--agent`, or stdin not a terminal | `--mode agent` is refused: `error` `USAGE` + `done` (exit 2); nothing runs | +| `projects list`, `projects create` | `--agent` (or stdin/stdout not a terminal) | `--mode agent` is refused the same way | +| `install skill` | none: with stdin not a terminal it prints one JSON line | `--agent` is an unknown option (exit 1); `--mode agent` is `USAGE` (exit 2) | + +- **Stderr.** Only a command line holding `--mode agent` keeps stderr free of the "Skill update available" and "Update available" banners. Under `--agent` those lines can appear on stderr, and `testrun run` also writes its coverage lines and `kane-cli evidence serve` hints there. Parse stdout only; a stderr line is not a failure. +- **No flag, no terminal.** Linked `context extract` and `design tests` refuse with a stderr line (exit 2). Linked `context ingest` falls back to land-only: it uploads, prints text lines and leaves the session paused. Always pass the flag. +- **Modes to avoid.** `--mode ci` on linked extract, design or reconcile is refused as a stderr line (exit 2), even with `--json`. `--json` with `--mode agent` is refused as a stderr line (exit 2). `--mode override` answers every question with its recommended option: never use it for the person. +- **Usage errors.** With `--mode agent` anywhere on the command line, a parse error arrives as `error` `USAGE` + `done{refused, 2}`; `--help` and `--version` print only a `done` line. + +## 2. Stream shape + +**Assurance stream** (`context …`, `design tests`, `maintain …`, `cover`): one JSON object per line, envelope `{"type": "", "v": 1, "verb": "", …}`. No `stream_start`, and don't rely on a `ts` field here. Verbs: `sync` (`context pull` in a linked folder; refusals of its flags and at its entry carry `pull`), `extract` (ingest and extract; `reconcile` from the upload on when the ingest is a new version of a source the project holds), `design`, `evolve`, `reconcile`, `save`, `discard`, `cover`. Refusals of `context sessions` (list, show, clean) are stamped `save`, and those of `context list` are stamped `list`. + +**Run stream** (`testmd run`, `testrun run`): opens with `stream_start {cli_version, surface, pid}`; every line carries `v` and `ts`. Shapes: `references/testmd.md`, `references/testrun.md`; linked-folder additions in §3.7. + +| Command (agent flag) | First line | Last line | +|---|---|---| +| `context extract` | `folder_linked` (with `--project`), else `nothing_new`, `warning` or `remote_connect` | `done` | +| `context ingest ` | `folder_linked` (with `--project`), else `remote_fetch` (an address) or `remote_upload` (a file); a refusal is `error` | `done` | +| `design tests` | `remote_connect` (or `warning`, `error`) | `done` | +| `maintain evolve` | `remote_connect` | `done` (may carry `not_updated`) | +| `maintain reconcile` | `remote_upload` or `remote_fetch`, else `remote_connect` | `done` | +| `context sessions save ` | `session_save{phase:"check"}`; `phase:"saving"` for a session saved before; `phase:"saved"` with `saved: 0` when it added nothing (or `error`) | `done` | +| `context sessions discard ` | `session_discard{phase:"check"}` (or `error`) | `done` | +| `context pull` (linked, or `--project`) | `folder_linked` (with `--project`), else `pull_test` or `error`; a project with no tests sends `done` alone | `done` (`shared-projects.md` §7a) | +| `cover` | `coverage` | `done` | +| `testmd run `, stopped on its Test Manager case | `stream_start` | `error` (no `test_md_done`) | +| `testrun run`, refused | `stream_start`; then `error` before any plan for `--from-context` in a linked folder; else `testrun_plan` | `PROJECT_MISMATCH`: `error`, the last line outside a linked folder. Another invalid plan, or no tests: no `error`, the stream ends at `testrun_plan` (read the exit code). **In a linked folder only**, from the release that carries the fix, every one of these (`PROJECT_MISMATCH` too) ends `error` then `testrun_done{overall_status:"refused"}` (the `--from-context` refusal still ends on its `error`); handle both shapes. Everywhere else these shapes don't change | + +Outputs that are **not** the envelope and have **no `done`** (read the exit code): + +| Command | Stdout on success | +|---|---| +| `projects list --agent` | `{"id","name"}` per project, then `{"_meta":"page","limit","offset","returned","has_more"}`. Errors are plain text on stderr, with no code | +| `projects create "" --agent` | one line `{"id","name"}`. Errors on stderr. It does not link the folder | +| `context list --mode agent` | one bare object per row: `{id, label, title, trust, fresh, tests_fresh?, next?}` (`trust: "derived"` = proposed, not approved yet; `"trusted"` = approved: say *proposed* or *approved*, never the value); `--undesigned` adds `designed: false`; nothing at all when nothing matches | +| `context sessions --mode agent` | one line holding a JSON array of `{sid, verb, subject, title, state, added, updated?, last_activity_at, owner}` | +| `install skill` (stdin not a terminal) | one line `{"type":"skill_install_result","status":"installed"\|"already-installed"\|"failed","version","installed_targets","error?"}`; exit 1 only on `failed` | + +Refusals of `context list` and `context sessions` do use the envelope: `error` + `done`. + +Dispatch on `type`, never on position. **The vocabulary is open:** tolerate unknown event types and unknown fields. A stream that should end in `done` and doesn't means the process died: check `kane-cli context sessions --mode agent` before retrying anything paid. + +## 3. Event catalogue + +"Tell the person" is plain words. Never show event or field names. + +### 3.1 Linking and loading (every assurance verb) + +| Event | When | Fields that matter | Tell the person | +|---|---|---|---| +| `folder_linked` | `--project ` linked the folder in this run; the first line | `project_id`, `project` | "This folder is now linked to ; what you save goes there." If the run then ends `PERMISSION_REQUIRED` or `ASSURANCE_OFF`, or a `context pull --project` is refused at the start or can't read the project, the link was removed again | +| `folder_local` | `--local` gave the folder its own store; `extract` then ends `done` exit 0 (ingest goes on locally) | `gitignore`: `added`, `failed`, null | "This folder keeps its own local store." On `failed`: `.context/` wasn't added to `.gitignore` | +| `nothing_new` | a plain `extract` found every source already read; no session opened; exit 0 | `reason`: `every_source_read`, or `sources_in_a_session` with `open_sessions[{sid, owner, state, added}]`; `next` | "Nothing new to extract" / "every source is already in an open session"; offer `next` | +| `remote_connect` | `step:"connect"`, then `step:"open_session"` once the session exists | `sid`, `project` (on `open_session`) | "Connecting to ." Keep `sid` | +| `remote_load` | the project's history loads | `part` (`project`, `project_ready`), `percent` (may be null) | fold into one progress line | +| `remote_fetch` | an address or item is fetched on this machine, before upload | `ref`, `provider` (`web`, `jira`, `confluence`, `linear`, `linear-doc`) | "Reading " | +| `remote_upload` | a file goes up before the session opens: 0%, per chunk, 100% | `file`, `percent` | one line per file | +| `remote_retry` | waiting on the shared project | `attempt`, `next_in_s`, `give_up_in_s`, `cause` (`unreachable`, `not_ready`, `timeout`) | "The shared project isn't answering; retrying." Let it run | +| `remote_overlap` | a new session opened while others are open on the same subject | `sessions[{sid, owner: you\|someone_else\|null, state, added, next?}]` | name the other sessions; offer the resume `next` on the person's own; this run goes on | +| `run_start` | the run starts | `session` = the shared project's session id (not a support id); no log path | — | +| `warning` | a non-fatal condition | `message`, `code?` | relay it | + +### 3.2 Ingest, extract, review, evolve + +| Event | When | Fields that matter | Tell the person | +|---|---|---|---| +| `ingested` | each item lands in the session | `source_id`, `status` (`created`, `versioned`, `unchanged`), `behind?`, `unread?`, `media?`, `duplicate_of?`, `origin?`, `warnings?`; no `cid` | "Added to this session" / "a new version" / "already in , unchanged" | +| `commit` | the agent's items land in the **session** | counts, `minted[]` | "N use-cases proposed in this session, not saved yet" | +| `held` | items held back | `reason`: `duplicate_check_reduced`, `review` | "N items are held for review" | +| `usage` | each agent turn | `credits`, `total_credits` | report the final total | +| `review_cards` | a new version's review holds its cards, before a `review_pending` pause | `source`, `cards[{key, kind, use_case?, title, change, diff?, tests?}]` | show each change; ask approve, reject or defer | +| `card_verdict` / `card_refused` | an `--answer =` landed / didn't | `key`, `verdict`, `line`, `offer?{kind, key, title}` / `code`, `refusal` | relay `line` or `refusal`. `offer.kind: "evolve"` runs at once (`child_start` … `child_end`); `offer.kind: "design"` does **not** run: after the save, offer to design that use-case's tests (`kane-cli design tests --use-case --mode agent`, its id from `kane-cli context list --mode agent`) | +| `evolve_paused` | evolve asks once per use-case; exit 3 | `pending_questions[]` (id `evolve:`), `next` (update all, leave all out) | present the blast radius and ask (`references/assurance.md` §11) | +| `child_start` / `child_end` | a design run inside a review or an evolve | `child`, `subject` / `ok`, `exit_code`, `next?` | "Designing tests for … done" / "didn't finish" | +| `evolve_skipped` | a use-case evolve leaves alone | `use_case`, `reason` (`no-design`, `source_changed`, `up_to_date`, `archived`, `unreviewed`, `held`, `declined`), `held_by?`, `next?`, `remedy?` | the reason in words; offer `next` | +| `session_complete` | the run finished; **nothing is saved yet** | `sid`, `next` (save, then discard), `no_start_page?[{test}]` | "Done. Save it to ?" Name tests with no start page. No `next` = it added nothing and was discarded. On a resume of a finished run whose ending was never recorded, `next` is the save alone | +| `session_ended` | the session was saved or discarded in another terminal; exit 0 | `sid`, `reason`: `saved_elsewhere`, `discarded_elsewhere`, `ended_elsewhere` | say where it went; nothing to do | +| `error` | a refusal or failure | `code?`, `message`, `remedy?`, `next?`, `held_by?`, `not_added?[]`, `final?` | §6 | +| `done` | always last | `status`, `exit_code`, `next?`, `not_updated?` (evolve) | §4, §5 | + +### 3.3 `session_paused` (exit 3) + +Every pause has `sid`, `expires_at: null` and `pending_questions[]` (empty unless questions wait); most carry `resume` (the exact command). Branch on `reason`: + +| `reason` | Means | `next` | Do | +|---|---|---|---| +| none, questions pending | the agent asks the person | resume; resume with `--answer =` | the pause loop (`references/assurance.md` §2) | +| none, no questions | a forced stop (second Ctrl+C), or a stop during an evolve | resume | resume when the person wants | +| `held_back` | `--trust hold` held items | — | tell the person; nothing promises they come back on a resume | +| `connection_lost` | the connection dropped, or the ending couldn't be recorded | resume | resume once the network is back | +| `not_accepted` | the shared project refused a step | resume | resume once; then ask for help | +| `computer_slept` | this machine slept over 2 minutes | none | run `resume` when the person wants | +| `resumed_elsewhere` | the session was resumed in another terminal | none | leave it; it goes on there | +| `review_pending` | review cards wait (`pending_cards[]`) | resume with `--answer =approve` … | ask per card: `approve`, `reject` or `defer` | +| `review_decided` | every card was decided, then a stop | resume | resume to finish the review | +| several sources read (`not_named?[]`, no `resume`) | an extract that read 2+ sources paused | discard; start again | it can't be resumed or saved: tell the person, discard on their word, then extract one source at a time | + +### 3.4 Save and discard + +| Event | When | Fields that matter | Tell the person | +|---|---|---|---| +| `session_save` `check` | first: what the save would land | `sid`, `project`, `adds{use_cases, acceptance_criteria, scenarios, tests, open_questions, sources, style_notes}`, `changes?`/`approves?`/`archives?`/`removes?`, `held_changes?`, `replaces?`, `marks_up_to_date?`, `other_work?`, `not_updated?[]` | "Saving adds N use-cases and M tests to ." Without `--yes` the command stops here (`CONFIRM_REQUIRED`): show this preview and get the person's word | +| `session_save` `saving` | the save is sent | `sid` | — | +| `session_save` `saved` | landed | `saved` (records landed; `0` = nothing to save, and the session was discarded), `test_manager`, `test_files`, `next?` (evolve what is still out of date) | "Saved to ", then the two blocks below | +| `session_save` `behind` | a later save changed what this session touched; exit 2, no `error` | `conflict?{kind, id, title}` or `{kind:"link", from, to}`, `next` | "Another save changed ; nothing was saved." Offer `next` | +| `session_save` `failed` | the project refused the save; exit 1, no `error` | `reason:"not_accepted"`, `code?` (`SCOPE_RECORD_REFUSED`) | "The project refused the save." Don't retry; ask for help | +| `session_discard` | `check` (the same counts), then `discarded` | `sid`, `adds?` | "Discarded; nothing was saved." An already-discarded session gets `discarded` alone (exit 0) | + +`test_manager` (when the project's Test Manager takes the cases): `total`, `created`, `updated`, `already_there`, `waiting` with `waiting_on[{scenario, title, reason}]` (`changed_both_sides`, `not_pushable`, `archived`, `busy`, `not_accepted`), `next_run[{scenario, title}]` (written by the next `testmd run`), `not_covered?[]`, `error?`, `support_id?`, `next?`. Tell: "N test cases created and M updated in Test Manager"; name what waits and why. + +`test_files` (design, reconcile and evolve sessions): `written[]` (paths in `.testmuai/tests/`), `refused[{test, title, reason, file?}]` (`reason` is prose; a file edited by hand is kept), `replaced?[]` (files of tests this save replaced: they stay in the folder, won't run any more, and a bare `testrun run` still picks them up; tell the person, and offer to move them out of `.testmuai/tests/`), `variables_needed?{file, variables}`, `no_start_page?[]`, `next?`; or `{written: [], refused: [], error: "TEST_FILES_NOT_WRITTEN", next}`. Test files are written **only** by the save, only in the folder where it runs. Tell: "Wrote N test files." Fill `variables_needed` before any run. + +### 3.5 Design + +| Event | When | Fields that matter | Tell the person | +|---|---|---|---| +| `gate_refused` | design entry refused: unknown, removed or not-a-use-case ref; or the use-case already has tests, or changed since | `message`, `final: true`, `next?[{cmd, why, title}]`; **no `code`** | relay `message`; offer `next` (`Redesign` = `--force`) | +| `error` `UC_UNREVIEWED` / `UC_ARCHIVED` | the use-case is proposed / archived | `message`, `final`, `next?` | §6 | +| `commit`, `usage` | as §3.2 | | "Designed N tests in this session, not saved yet" | +| `variables_declared` | the tests need variables | `file`, `variables[]` | a to-do; the stubs are this folder's only | +| `note`, `panel_resolved` | a resume with `--answer` | `text` / `id` | confirmation only | +| `session_paused`, `session_complete` | as §3.2–3.3 | | | + +### 3.6 Cover + +| Event | When | Fields that matter | Tell the person | +|---|---|---|---| +| `coverage` | `cover --mode agent` | `source: "project"`, `pct`, `usecases`, `criteria`, `perUc`, `perAc`, `flags`, `recent_run?{may_not_count_yet: true, up_to_minutes: 5}` | "Coverage is N% across <project>'s K use-cases." With `recent_run`: "a run from the last 5 minutes may not count yet" | +| `coverage` (computing) | the project is still working it out; exit 0 | `{source: "project", state: "computing"}` | run `next` (`kane-cli cover --mode agent`) in a moment | +| `done` | last | `next?`: designs to make, `testrun run` while nothing has run; `kane-cli context pull --mode agent` first (and no run) when the project has designs and this folder has no designed test files yet | offer them, adding the agent flag (§4) | + +### 3.7 `testmd run` and `testrun run` in a linked folder + +A file directly in `.testmuai/tests/` with an `assurance:` header runs on its scenario's Test Manager case. Its first run authors (about 30–85 credits per test, approximate); later runs replay. + +| Event | When | Fields that matter | Tell the person | +|---|---|---|---| +| `error` (testmd) | the case lookup or attach stopped the run | `code`, `message`, `remedy`, `support_id?`; no `next`, no `test_md_done` | §6, runs of designed tests | +| `error` `TEST_RUN_NOT_SAVED` (testmd) | before `test_md_done`; the verdict stands | as above | report the verdict; exit 1 even when steps passed | +| `testrun_plan` | the plan | `members[{path, test_id, failure?}]`, `valid` | on 0.10.0 an invalid plan other than `PROJECT_MISMATCH`, or no tests at all (`members: []`), has **no `error`**: read each `failure` | +| `error` `NO_TESTS` | *(linked folder only, from the release that carries it)* no `*_test.md` selected | `message` "no tests here yet", `next` | `next` is `kane-cli context pull --mode agent`. A filter that matches nothing: the old message and no `next`. Outside a linked folder there is no such event: the stream ends at `testrun_plan` with `members: []` and exit 2 | +| `error` `PLAN_INVALID` | *(linked folder only, from the release that carries it)* an invalid plan other than `PROJECT_MISMATCH` | `offenders[{path, failure}]`, no `next` | fix those tests or leave them out. Outside a linked folder: read `testrun_plan.members[].failure` | +| `error` `PROJECT_MISMATCH` | after `testrun_plan` | `projects[{project_id, name, this_folder, tests}]`, `remedy`, `next` | §6 | +| `testrun_member_end` (`testrun_authored_member_end` for a test's first run) | each member | `status`, `failure?{message}`, `log_path` | a member stopped on its case shows only `failure.message`; its `error` (with `code` and `remedy`) is in the member's log at `log_path`, or inside `testrun_member_event` with `--stream-members` | +| `coverage` | after the evidence ingest | `status` (`written`, `none`, `skipped`, `failed`); `written`: `computed_by: "project"`, `pct`; `none`: `reason` (`on_grid`, `project_no_answer`) | "This run reached N use-cases." `none`: the project didn't answer in time; the run is fine | +| `testrun_done` | last (local run) | `overall_status` (`passed`, `failed`, `cancelled`; `refused` after a refusal, in a linked folder only, from the release that carries it), `coverage?` | the verdict | +| `remote_error` (`--remote`) | one per member that stops before dispatch | `code`, `detail` (`<path>: <message>`), `remedy?`, `support_id?`; no `next` | nothing was sent to HyperExecute; `testrun_done{overall_status:"failed"}` and `remote_done{status:"failed", exit:2}` follow | + +## 4. `next` + +Shape: `next: [{cmd, why}]`, one runnable command per item (design walls add `title`). Tolerate a plain string item. + +Where: `error`, `session_complete`, `session_paused`, `nothing_new`, `remote_overlap.sessions[]`, `gate_refused`, `evolve_paused`, `evolve_skipped`, `child_end`, `session_save` (`saved`, `behind`, and inside `test_manager` and `test_files`), a `context list` row whose tests are out of date, and `done` (it repeats a pause's `next`). + +- **Agent flag.** Each `cmd` already carries `--mode agent` (`--agent` for `testrun run`, `testmd run`, `projects`); end a `testrun run` line in `< /dev/null`. If a `cmd` has no agent flag, add the one that command takes (§1) before running it. +- **One at a time.** Run each `cmd` as its own command. Never join them with `&&`. +- **Placeholders.** A `cmd` holding `<files>` (from `nothing_new`, a `behind` save) is a template: put in the person's real paths. +- **Read it before running it.** Some spend credits (`context extract`, `context ingest`, `design tests`, `maintain evolve`, `testmd run`, `testrun run`). Some need the person's word: `--allow-unreviewed` (designs from a use-case nobody approved), `--force` (replaces a use-case's tests), `context sessions save --yes` (publishes to the team's project and Test Manager), `context sessions discard --yes` (throws the session's work away). A `next` built from a partial `--answer` set carries the answers already given. +- **No `next`?** Read `remedy`: prose that may name a command. Add the agent flag that command takes (§1) before running it. Refusals at a folder's entry (linking, sign-in, access, `REMOTE_UNSUPPORTED`) carry `remedy` only; §6 gives the command for each. For `projects`, read stderr. + +## 5. Exit codes + +`done.status` follows the exit (`0` complete, `2` refused, `3` paused, `130` interrupted, any other `error`), with two exceptions: `context ingest` and `maintain reconcile` refusals can end `done{status:"refused", exit_code:1}`, and a failed read of `context list`, `context sessions` or `cover` ends `done{status:"error", exit_code:1}`. Branch on `exit_code`. + +| Command | 0 | 1 | 2 | 3 | 130 | +|---|---|---|---|---|---| +| `context extract` | `session_complete`, `nothing_new`, `session_ended` | the run stopped (work kept in the session), `SESSION_OPEN_FAILED`, `SESSION_NOT_ENDED`, agent failures | entry, sign-in, access, resume and `--answer` refusals | paused | stopped before a session opened | +| `context ingest` | as extract | as extract, plus `UPLOAD_STALLED`, `UPLOAD_NOT_TAKEN`, `RESULT_REFUSED`, `FETCH_FAILED`, conversion failures | file, address, conversion and source-id refusals | paused | stopped during a fetch or upload | +| `design tests` | complete | as extract | `UC_*`, `gate_refused`, `USAGE`, `STRUCTURED_*` | paused | as extract | +| `maintain evolve` | complete | some use-cases not updated (`done.not_updated`, `next`) | entry refusals, `USAGE`, `MODE_USAGE` | paused | as extract | +| `maintain reconcile` | complete; nothing to review | `DESIGN_CHILD_FAILED` | `REMOTE_UNSUPPORTED`, `SOURCE_*`, `MODE_USAGE`, `STRUCTURED_TARGET_UNKNOWN` | `review_pending` | as ingest | +| `context sessions save` | saved; nothing to save | `failed` phase, `SAVE_FAILED`, `ASSURANCE_OFF` | `CONFIRM_REQUIRED`, `behind` phase, `SESSION_*`, `CONNECTION_FAILED`, sign-in | — | — | +| `context sessions discard` | discarded; already discarded | `DISCARD_FAILED` | every other refusal | — | — | +| `context sessions`, `context list` | the array / the rows | `LIST_FAILED` | `REMOTE_UNSUPPORTED`, `ASSURANCE_OFF`, sign-in, `USAGE`, `MODE_USAGE` | — | — | +| `cover` | `coverage` (also computing) | `COVERAGE_FAILED` | `ASSURANCE_OFF`, sign-in, `REMOTE_UNSUPPORTED` (`cover gaps`) | — | — | +| `testmd run` (designed test) | passed | failed, `TEST_MANAGER_OUTDATED`, `TEST_CASE_ATTACH_FAILED`, `TEST_RUN_NOT_SAVED` | case and lookup refusals | — | — | +| `testrun run` | all passed | a member failed, including one stopped on its case | `REMOTE_UNSUPPORTED`, `PROJECT_MISMATCH`, invalid plans, every `--remote` stop before dispatch | cancelled | — | + +On the assurance verbs, `3` is a **pause** (resumable), not a failure: run the pause loop. `130` is a forced stop: resumable only if a `session_paused` arrived. On `testrun run`, `3` means cancelled. + +## 6. Refusal codes + +`Do` says what to tell or ask the person and the exact command, so you can go on when the event has no `next`. "Ask the person:" marks a step only they (or an admin) can take. Never rerun a paid command (extract, ingest, design, evolve, testmd, testrun) blindly: an exit 1 after turns ran has spent credits. `<the command>` = the refused command as typed. + +| Code | Exit | When | Do | +|---|---|---|---| +| **Linking and the folder** | | | | +| `FOLDER_USAGE` | 2 | `--project` and `--local` together | ask the person: share the work (`<the command>` without `--local`) or keep it here (`<the command>` without `--project`) | +| `FOLDER_USAGE` | 2 | `--project` or `--local` with `--resume` | `<the command>` without the folder flag; a session stays where it started | +| `FOLDER_CHOSEN` | 2 | `--project` in a folder with its own store; `--project Q` in a folder linked to P; `--local` in a linked folder | `<the command>` without `--project` / `--local` (it works in the folder's own store or project). To change that, ask the person: move `.context/` out, or remove `.testmuai/assurance.json` | +| `BOTH_STORES` | 2 | the folder has `.context/` and a link | ask the person: work locally (remove `.testmuai/assurance.json`) or in the project (move `.context/` out of the folder) | +| `PROJECT_FILE_UNREADABLE` | 2 | `.testmuai/assurance.json` can't be read | ask the person: remove `.testmuai/assurance.json`. Then `kane-cli context extract --project <id> --mode agent` | +| `PROJECT_FILE_UNWRITABLE` | 2 | the link couldn't be written | ask the person: make the folder writable, or move a file named `.testmuai` out of it | +| `PROJECT_NO_ACCESS` | 2 | `--project <id>` names no project of theirs | `kane-cli projects list --search <name> --agent`; confirm the id with the person; run `<the command>` with it | +| `PROJECT_NO_ACCESS` | 2 | no access to the folder's project | ask the person: a Test Manager admin grants their account access | +| `PERMISSION_REQUIRED` | 2 | the project's first session needs permission to create test cases | ask the person: an admin grants it, or someone who has it starts the first session. A link made in this run was removed | +| `ASSURANCE_OFF` | 2 (1 from `sessions save`) | assurance isn't available for the project | ask the person; no command | +| `REMOTE_UNSUPPORTED` | 2 | `context review`, `context explain` | `kane-cli context list --mode agent` (the use-cases, their status, scenarios and tests) | +| `REMOTE_UNSUPPORTED` | 2 | `cover gaps` | `kane-cli cover --mode agent` | +| `REMOTE_UNSUPPORTED` | 2 | `context sessions show`, `context sessions clean` | `kane-cli context sessions --mode agent` | +| `REMOTE_UNSUPPORTED` | 2 | `--mode ci` on extract or design (a stderr line) | `<the command>` with `--mode agent` | +| `REMOTE_UNSUPPORTED` | 2 | `context ingest --plan` or `--mode override`; `context list --all`, or a `--type` other than `source`, `usecase`, `scenario`, `test` | `<the command>` without that flag, with `--mode agent` | +| `REMOTE_UNSUPPORTED` | 2 | `maintain reconcile --plan`, `--apply`, `--mode ci`, or no `--source-id`; `--from <address>` | `kane-cli maintain reconcile --from <file> --source-id <id> --mode agent`; for an address, `kane-cli context ingest <address> --mode agent` | +| `REMOTE_UNSUPPORTED` | 2 | `context sync`, `push`, `clone`, `fsck`, `view`, `retire`, `revert`, `rebuild`, `name`, `design explain` (and `context pull` where its help has no `--project`: `shared-projects.md` §7b) | no command in a linked folder; tell the person these work on a folder's own store. The project's tests come with `kane-cli context pull --mode agent` (`shared-projects.md` §7) | +| `PULL_LOCATION_LINKED` | 2 | `context pull` with a location name, `--rebase`, `--yes` or `--answer` in a linked folder | `kane-cli context pull --mode agent` | +| `REMOTE_UNSUPPORTED` | 2 | `testrun run --from-context` | run `next`, or `kane-cli testrun run <test files> --agent` | +| **Sign-in and access** | | | | +| `CREDENTIALS_MISSING` | 2 | nobody signed in | start sign-in (`kane-cli login --oauth`, `references/ready-check.md`), then `<the command>` | +| `CREDENTIALS_MISSING` | 2 | "couldn't confirm your login for shared projects" (a browser sign-in with no access key) | `<the command>` once more; if it refuses again, ask the person: run `kane-cli login --username <user> --access-key <key>` in their own terminal | +| `CREDENTIALS_REJECTED` | 2 | the shared project rejected the credentials | sign in again (`kane-cli login --oauth`), then `<the command>` | +| `CONNECTION_FAILED` | 2 | couldn't connect (or for 2 minutes); nothing was started | check the network; `<the command>` once more | +| `PREFLIGHT` | 2 | kane-cli's check before a session opens failed: unreachable, sign-in unchecked or rejected, or "assurance sessions aren't available here yet" | unreachable: `<the command>` once more later. Sign-in: `kane-cli login --oauth`. Not available: stop and pass on the support id in `remedy` | +| `VERSION_MISMATCH` | 2 | this kane-cli is too old, or the shared project doesn't support it yet | too old: ask the person to update (`npm install -g @testmuai/kane-cli@latest`). Not supported yet: stop and pass on the support id in `remedy` | +| `PROJECT_BUSY` | 2 | a live session holds the project, use-case or source: `held_by{sid, verb, use_case, mine, status}` | no `next`. When `remedy` names a resume (`kane-cli context extract --resume <sid> --mode agent`, or `kane-cli design tests --resume <sid> --mode agent`; only for the person's own session, `mine: true`), running it takes the session over and stops it in the other terminal, so ask the person first; otherwise wait for that run to finish. When `remedy` only says to wait (a resume refused this way, or a design while the person's own extract holds the project), wait. Someone else's: tell the person a teammate's session holds it; wait; `<the command>` later | +| **Sessions** | | | | +| `SESSION_NOT_FOUND` | 2 | no session with that id is theirs (a colleague's can't be resumed or saved) | `kane-cli context sessions --mode agent`; use a `sid` whose `owner` is `you` | +| `SESSION_FINISHED` | 2 | resume of a finished session | on the person's word: `kane-cli context sessions save <sid> --yes --mode agent` | +| `SESSION_NOT_FINISHED` | 2 | save of an unfinished session | resume first: `kane-cli context extract --resume <sid> --mode agent` (`design-…`, `evolve-…` ids: `kane-cli design tests --resume <sid> --mode agent`); save after `session_complete` | +| `SESSION_NOT_ENDED` | 1 | the run finished but the project didn't take its last step; the work is safe | run `next` (the resume command above): it records the ending; then save | +| `SESSION_FAILED` | 2 | the session ended with an error; it can't be resumed or saved | start again: `kane-cli context extract --mode agent`, or `kane-cli design tests --use-case <uc> --mode agent` | +| `CHECKPOINT_PROTOCOL`, `SESSION_KEPT_NOTHING`, `SESSION_MISSING` | 2 | a resume can't read the session's last step | tell the person it can't be resumed; start a new one: `kane-cli context extract --mode agent` or `kane-cli design tests --use-case <uc> --mode agent` | +| `SESSION_OPEN_FAILED` | 1 | the session couldn't be opened | `<the command>` once more; then ask for help | +| no code: "stopped before it finished" | 1 | the run stopped; the work so far is kept in the session | resume with the `sid` from `remote_connect`: `kane-cli context extract --resume <sid> --mode agent` (design: `kane-cli design tests --resume <sid> --mode agent`). Don't start over | +| `CONFIRM_REQUIRED` | 2 | save or discard without `--yes` | the `check` line before it is the preview: show it; on the person's word, `kane-cli context sessions save <sid> --yes --mode agent` (or `discard`) | +| `SESSION_ELSEWHERE` | 2 | discard while the session runs in another terminal, or is held | wait (up to 3 minutes) or stop the other run; then `kane-cli context sessions discard <sid> --yes --mode agent` | +| `ALREADY_SAVED`, `ALREADY_ENDED` | 2 | discard of a saved or ended session | nothing to discard; tell the person | +| `DISCARD_FAILED` | 1 | the discard failed | stop; ask for help | +| `LIST_FAILED` | 1 | the session list or `context list` couldn't load | `<the command>` once more; then ask for help | +| `STRUCTURED_FLAGS_USAGE` | 2 | `--answer` without `--resume <sid>` or `--mode agent`, or unreadable | `kane-cli context extract --resume <sid> --mode agent --answer <id>=<value>` (design: `kane-cli design tests --resume <sid> …`) | +| `STRUCTURED_TARGET_UNKNOWN` | 2 | `--answer` names no pending question or card, isn't a verdict, or repeats a card | use the ids from `pending_questions` or `review_cards` | +| `USAGE` | 2 | a parse error; save or discard with no `<sid>`; evolve with both or neither of `<ref>` and `--from-stale`; design with no `--use-case` | fix the command (`kane-cli <command> --help`); never resend it unchanged | +| `MODE_USAGE` | 2 | `--mode` isn't one the command takes | `<the command>` with `--mode agent` | +| **Extract and ingest inputs** | | | | +| `NO_PROPOSAL` | 1 | the agent returned no use-cases | retry `kane-cli context extract --mode agent` once (it reads the source again and costs credits again). If it happens twice, stop, and give the person the command and the message | +| `AS_SINGLE_SOURCE` | 2 | `--as` with several refs | run `next`, or `kane-cli context ingest <ref> --as <id> --mode agent` once per source | +| `SAME_SOURCE_ID` | 2 | two refs would share one source id | run `next`; with none, ask the person for distinct ids and run `kane-cli context ingest <ref> --as <id> --mode agent` per ref | +| `NEW_VERSION_ALONE` | 2 | a new version of a held source among other refs; nothing added | `kane-cli context ingest <the other refs> --mode agent`, then `kane-cli context ingest <new version> --mode agent` alone (it opens a review) | +| `SITE_MOVED`, `KIND_MISMATCH`, `WEB_URL_CHANGED`, `UUID_MOVED` | 2 | the fetched item can't become a new version of the source id it would replace | ask the person which they mean; run that `next`, or `kane-cli context ingest <address> --as <new id> --mode agent` for a source of its own | +| `UNSUPPORTED_URL` | 2 | the ref is neither a file nor a web address | ask the person for a file path or a full `https://` address | +| `FILE_NOT_FOUND`, `FILE_UNREADABLE`, `FILE_TOO_LARGE`, `UNSUPPORTED_MEDIA`, `MEDIA_SNIFF_MISMATCH`, `BAD_SOURCE_ID` | 2 | checked before anything was sent | fix the path or file (relay the message); `BAD_SOURCE_ID`: pass a valid `--as <id>` | +| `PDF_ENCRYPTED`, `PDF_NO_TEXT_LAYER`, `PDF_UNREADABLE`, `DOCX_UNREADABLE`, `DOCX_NOT_WORDPROCESSING`, `DOCX_NO_TEXT`, `DOCX_ENCRYPTED_OR_LEGACY`, `DOCX_PART_TOO_LARGE`, `DOCX_TOO_MANY_PARTS`, `DOCX_BOMB`, `PROJECTION_TOO_LARGE` | 2 | the file can't be converted on this machine | relay the message; ask the person for an unlocked, text-bearing or smaller copy | +| `SOURCE_ID_OWNED` | 2 | that source id is taken | ask the person for another id; `kane-cli context ingest <ref> --as <id> --mode agent` | +| `INGEST_FAILED`, other conversion failures | 1 | the conversion failed | report it; ask for help | +| `INVALID_URL`, `ADDRESS_NOT_ALLOWED`, `NOT_A_PAGE`, `PAGE_NOT_FOUND`, `PAGE_BLOCKED`, `PAGE_PAYWALLED`, `PAGE_TOO_LARGE`, `PAGE_TIMEOUT`, `PAGES_UNAVAILABLE`, `PAGES_NOT_AVAILABLE`, `ITEM_UNAVAILABLE`, `LINEAR_PAGE`, `LINEAR_LINK_MALFORMED`, `ATTACHMENT_TOO_LARGE` | 2 | the address or item can't be read | relay the message; ask the person for another address or a saved file | +| `NO_CONNECTION`, `NO_ACCESS`, `NO_ISSUE_ACCESS`, `NO_DOCUMENT_ACCESS`, `NO_CONFLUENCE_ACCESS`, `NO_COMMENTS_ACCESS`, `PROJECT_NOT_SELECTED`, `INTEGRATION_SETUP`, `INTEGRATION_UNAVAILABLE` | 2 | the Jira, Confluence or Linear connection is missing or can't see the item | ask the person: connect the integration or grant access (relay the message) | +| `INTEGRATION_BUSY` | 2 | the integration is busy | wait; `<the command>` once more | +| `OUT_OF_CREDITS` | 2 | no credits left for the fetch | tell the person (`kane-cli balance`); stop | +| `FETCH_FAILED` | 1 | the fetch failed | `<the command>` once more; then ask for help | +| `PLUGIN_NOT_CLAIMED`, `PLUGIN_SOURCE_ID_INVALID`, `PLUGIN_MISSING` | 2 | no source plugin takes the ref, or its id is invalid | ask the person: check the ref's form and that the plugin is installed | +| `PLUGIN_HOOK_FAILED`, `PLUGIN_HOOK_TIMEOUT` | 1 | the source plugin failed | report it; ask for help | +| `UPLOAD_STALLED`, `UPLOAD_EXPIRED` | 1 / 2 | the upload stopped, or its window ended | `<the command>` once more | +| `UPLOAD_NOT_TAKEN`, `RESULT_REFUSED` | 1 | the shared project didn't take the upload or its result | stop; ask for help | +| `LINKS_NOT_TAKEN` | 2 | the shared project can't take addresses | ask the person for a saved file of the page; `kane-cli context ingest <file> --mode agent` | +| `SOURCE_UNKNOWN`, `SOURCE_REMOVED` | 2 | reconcile: the project doesn't hold that `--source-id`, or removed it | `kane-cli context list --type source --mode agent`; a new source: `kane-cli context ingest <file> --mode agent` | +| `SOURCE_ORIGIN_UNKNOWN`, `SOURCE_REF_UNROUTED`, `SOURCE_ID_MISMATCH` | 2 | reconcile of a plugin source that doesn't line up | relay the message; check the `--source-id` with the person (`kane-cli context list --type source --mode agent`) | +| on `card_refused`: `CONCURRENT_APPEND`, `TARGET_RETIRED`, `SOURCE_MOVED`, `SOURCE_UNRESOLVED`, `ARCHIVE_UNRESOLVED`, `ALREADY_VERDICTED`, `CARD_GONE`, `HOLD_GONE` | — | a review verdict didn't land | relay `refusal`; `kane-cli context extract --resume <sid> --mode agent` shows the cards still open | +| `DESIGN_CHILD_FAILED` | 1 | an approved change's design run failed | later: `kane-cli design tests --use-case <uc> --mode agent` | +| `AGENT_FAILED`, `AGENT_CRASHED`, `REPAIR_FAILED`, `PROPOSAL_INVALID`, `COMMIT_FAILED` | 1 | the agent's run failed | report it; credits were spent; check `kane-cli context sessions --mode agent` before any retry | +| `TRUST_USAGE` | 2 | `--trust hold` in a terminal run | `<the command>` without `--trust`, with `--mode agent` | +| **Design** | | | | +| `UC_UNREVIEWED` | 2 | the use-case is still proposed; nobody approved it | show it (`kane-cli context list --type usecase --mode agent`); on the person's word, `kane-cli design tests --use-case <uc> --allow-unreviewed --mode agent` (one that already has tests also needs `--force`: the `gate_refused` row below). A person approves at the end of a terminal session | +| `UC_ARCHIVED` | 2 | the use-case is archived | tell the person; pick another use-case | +| `gate_refused` (no code) | 2 | unknown, removed or not-a-use-case ref | pick from the ids the message lists (`kane-cli context list --type usecase --mode agent`) | +| `gate_refused` (no code) | 2 | the use-case already has tests, or changed since they were designed | ask the person: redesign (`kane-cli design tests --use-case <uc> --force --mode agent`, replaces its tests) or update (`kane-cli maintain evolve <uc> --mode agent`, only what changed). A use-case that is also still **proposed** needs both flags on a redesign: `kane-cli design tests --use-case <uc> --force --allow-unreviewed --mode agent` (each alone refuses with the other wall); evolve refuses it (`UC_UNREVIEWED`), so use the redesign | +| `NOTHING_CHANGED` | 2 | evolve of a use-case unchanged since its tests were designed | ask the person why: `kane-cli maintain evolve <uc> --because "<reason>" --mode agent`; or run `next` | +| `NO_TESTS_YET` (`evolve_skipped` `no-design`) | 2 | evolve of a use-case with no tests | `kane-cli design tests --use-case <uc> --mode agent` | +| `evolve_skipped` `source_changed` | 2 | only its source changed | run `next`; a file source has none: `kane-cli maintain reconcile --from <file> --source-id <id> --mode agent` | +| **Save** | | | | +| `session_save` `phase: "behind"` (no `error` event; a terminal shows `PROJECT_CHANGED`) | 2 | another save changed what this session touched | tell the person what (`conflict`); on their word run one `next` (start again from the latest) | +| `session_save` `phase: "failed"` (`reason`, maybe `code: "SCOPE_RECORD_REFUSED"`; a terminal shows `SAVE_FAILED` with a support id) | 1 | the project refused the save | saving again won't help: stop, and give the person the `reason` and the session id | +| `CONNECTION_FAILED` | 2 | the save's answer was lost after sending | `<the command>` once more: nothing is saved twice | +| `TEST_MANAGER_TIMED_OUT`, `TEST_MANAGER_UNREACHABLE`, `TEST_MANAGER_FAILED` (`test_manager.error`) | 0 | the save landed; Test Manager wasn't fully updated | run `test_manager.next`, or `kane-cli context sessions save <sid> --mode agent` (saving again finishes it) | +| `PROJECT_NO_ACCESS`, `ASSURANCE_OFF` (`test_manager.error`) | 0 | the save landed; Test Manager refused the cases | ask the person: their Test Manager access, or assurance for the project | +| `TEST_FILES_NOT_WRITTEN` (`test_files.error`) | 0 | the save landed; no test files were written here | run `test_files.next`, or `kane-cli context sessions save <sid> --mode agent` | +| `test_files.refused[]` | 0 | a test file wasn't written; `reason` says why: edited by hand and kept, a duplicate, an unreadable or unresolved record, an unwritable folder, a missing import (some have no `file`) | tell the person the reason; follow `test_files.next` when it is there. Only for a file edited by hand: when `test_files.next` offers `kane-cli testmd run <file> --agent`, it sends their edit to its case; otherwise (or instead) they move the file out and save again | +| **Runs of designed tests** | | | | +| `TEST_CASE_MISSING` | 2 | the session that designed the test isn't saved | on the person's word: `kane-cli context sessions save <sid> --yes --mode agent` (`kane-cli context sessions --mode agent` lists the sids) | +| `TEST_CASE_GONE` | 2 | its Test Manager case was deleted or moved | show the person; on their word, `kane-cli context sessions save <sid> --yes --mode agent` with the sid `remedy` names (it makes a new case). A `remedy` with no sid: `kane-cli context sessions --mode agent` to find it | +| `TEST_CASE_ARCHIVED` | 2 | its case is archived in Test Manager | ask the person: restore the case, or leave the test out of the run | +| `TEST_NOT_IN_PROJECT` | 2 | the test isn't in the folder's project, its file is out of date, or the project has no work yet | run it in the folder linked to its project; or, on the person's word, design it again: `kane-cli design tests --use-case <uc> --mode agent` (a proposed use-case also needs `--allow-unreviewed`, one with tests `--force`: the Design rows), then save; a project with no work: `kane-cli context ingest <files> --mode agent` | +| `TEST_MANAGER_OUTDATED` | 1 (2 on `--remote`) | Test Manager can't add this run to the test's case: on a device target (it attaches desktop-browser runs only), or because Test Manager itself needs an update (or made a case of its own) | stop and tell the person. On a device target, offer the desktop browser; otherwise the run can only go again once Test Manager is updated. Pass on the support id | +| `TEST_CASE_ATTACH_FAILED` | 1 | attaching to the case failed | `<the command>` once more later; then pass on the support id | +| `TEST_RUN_NOT_SAVED` | 1 | the run ran but its version wasn't saved on the case | report the verdict as it stands; tell the person Test Manager has no record of this run; pass on the support id | +| `CREDENTIALS_MISSING`, `CREDENTIALS_REJECTED`, `PROJECT_NO_ACCESS`, `ASSURANCE_OFF`, `CONNECTION_FAILED` | 2 | a first run couldn't look up its case | as the sign-in and access rows above; a later run goes on without the lookup | +| `REMOTE_KANE_NOT_PUBLISHED` | 2 | `testrun run --remote` with designed tests from a kane-cli that isn't a published release | ask the person to install a published kane-cli; `remedy` names the flags that pick one | +| **testrun plans** | | | | +| `PROJECT_MISMATCH` | 2 | the selected tests belong to more than one Test Manager project | show `projects[]` (`this_folder: true` marks the folder's project); run `next`, or `kane-cli testrun run <that project's test paths> --agent`. Keep `kane-cli run` experiments in another folder | +| invalid plan (no `error`: outside a linked folder, or on 0.10.0) | 2 | `testrun_plan.valid` is false | read each `members[].failure`; fix those tests or leave them out | +| `PLAN_INVALID` | 2 | *(linked folder only, from the release that carries it)* the same, as an `error` with `offenders[{path, failure}]`, then `testrun_done{refused}` | fix the named tests or leave them out | +| no tests (`members: []`) | 2 | nothing selected (everywhere on 0.10.0; outside a linked folder on every release) | in a linked folder: get the project's tests (`shared-projects.md` §7); elsewhere design them (`kane-cli context list --mode agent`, then `kane-cli design tests --use-case <id> --mode agent`) | +| `NO_TESTS` | 2 | *(linked folder only, from the release that carries it)* "no tests here yet", then `testrun_done{refused}` | run its `next`: `kane-cli context pull --mode agent`. A filter matching nothing has no `next`: change the filter | +| **Coverage** | | | | +| `COVERAGE_FAILED` | 1 | the project's coverage couldn't be read | `kane-cli cover --mode agent` once more in a moment; then ask for help | +| `coverage` computing | 0 | still being worked out | `kane-cli cover --mode agent` in a moment | + +## 7. Support ids + +A support id looks like `as-YYYYMMDDTHHMM-xxxxxxxx`. It names one command's attempt for TestMu AI support. + +| Where | What shows | +|---|---| +| Terminal | under a refusal: `if this keeps failing, share this id with support: <id>` | +| Assurance stream | **not** on `error` events. Only inside `remedy` for `VERSION_MISMATCH` (not supported yet), `PREFLIGHT` (not available here) and `SAVE_FAILED`, and in `session_save.test_manager.support_id` (a test that couldn't be read) | +| Run stream | `testmd run`: `error.support_id`; `testrun run --remote`: `remote_error.support_id`; a member's `error` inside `testrun_member_event` (`--stream-members`) | + +When you ask the person to get help, give them the support id, the command, the code and the message. With no id on the stream, give the rest; never rerun a paid command just to get an id. With `KANE_TELEMETRY=0` the id still shows, but nothing is recorded under it: say so. diff --git a/.agents/skills/kane-cli/references/assurance-parsing.md b/.agents/skills/kane-cli/references/assurance-parsing.md index 0ab581c..66e4625 100644 --- a/.agents/skills/kane-cli/references/assurance-parsing.md +++ b/.agents/skills/kane-cli/references/assurance-parsing.md @@ -2,6 +2,11 @@ # Assurance NDJSON — Wire Contract +> In a folder linked to a shared project (kane-cli 0.10.0+) the same commands add their own events (for example +> `folder_linked`, `nothing_new`, `remote_*` progress, `session_save`, `session_discard`, `session_ended`, `pull_test`) +> and refusal codes, and `session_complete` there carries the save's `next`. They are in `agent-contract.md` §3 and §6; +> this page covers the local store. + `kane-cli context extract --mode agent`, `kane-cli context ingest … --mode agent`, and `kane-cli design tests --mode agent` speak a versioned NDJSON vocabulary on stdout — one JSON object per line, envelope `{"type": "<name>", "v": 1, "verb": "extract"|"design", ...}`. On 0.7.2+ the stream is **strict**: stdout carries only NDJSON and stderr stays silent (crash traces excepted); on 0.7.1, prose diagnostics could ride stderr. `maintain reconcile --mode agent` (verb `reconcile`) and `cover`/`cover gaps --mode agent` (verbs `cover`/`gaps`) share the envelope with their own event sets (below). **The vocabulary is open**: new event types and fields may appear in any release — tolerate unknowns, never fail on them. @@ -19,7 +24,7 @@ for each line: else → per-type handling below ``` -A stream that ends **without** `done` means the process crashed — outcome unknown; inspect `context sessions --json` and `context list` before retrying anything paid. One designed exception: `kane-cli context review --mode agent` keeps its `--json` shape — verdict and drift listings are one JSON object per line with no `done` (an empty listing prints nothing, exit `0`); its early refusals — no store, an open rebase, a verdict queue that needs a terminal (`TTY_REQUIRED`) — are typed events followed by `done`; its later refusals exit without `done`. Read its exit code instead. One version-scoped rule for merged-ingest landing failures (bad path, unsupported media, refused URL, a misused `--mode` or `--as`): on 0.7.2+ they ride the stream as `error` (codes `MODE_USAGE`, `AS_SINGLE_SOURCE`, `UNSUPPORTED_URL`, `INGEST_FAILED`) + `done` — the guarantee covers the whole invocation; on 0.7.1 they end with prose + exit `1`/`2` **before any NDJSON begins** — no stream at all is a refusal to fix, not a crash, and the guarantee starts with the extraction stream. +A stream that ends **without** `done` means the process crashed — outcome unknown; inspect `context sessions --json` and `context list` before retrying anything paid. No stream at all, with one line on stderr and exit `2`, is a command-line refusal before anything ran (for example `--json` with `--mode agent`): fix the command, don't treat it as a crash. One designed exception: `kane-cli context review --mode agent` keeps its `--json` shape — verdict and drift listings are one JSON object per line with no `done` (an empty listing prints nothing, exit `0`); its early refusals — no store, an open rebase, a verdict queue that needs a terminal (`TTY_REQUIRED`) — are typed events followed by `done`; its later refusals exit without `done`. Read its exit code instead. One version-scoped rule for merged-ingest landing failures (bad path, unsupported media, refused URL, a misused `--mode` or `--as`): on 0.7.2+ they ride the stream as `error` (codes `MODE_USAGE`, `AS_SINGLE_SOURCE`, `UNSUPPORTED_URL`, `INGEST_FAILED`) + `done` — the guarantee covers the whole invocation; on 0.7.1 they end with prose + exit `1`/`2` **before any NDJSON begins** — no stream at all is a refusal to fix, not a crash, and the guarantee starts with the extraction stream. ## Events (extract / design) @@ -93,12 +98,12 @@ Use `text` + `options[].label` + `recommended_index` + `rationale` to decide or | `child_start` / `child_end` | around the design run, after a proceed | `child: "design"`, `subject` / `ok`, `dur_s`, `exit_code?` | | *(the design run's events)* | between the two, stamped `verb: "design"` | every event of `design tests --mode agent` except its `done`; a paused design session arrives as `session_paused` with `sid`, `resume`, `next[]` | | `evolve_diff` | after a clean design run | `use_case`, `generation` (`before`, `after`), `superseded[]`, `minted[]`, `retained[]` (pairs as `scenario` + `test`), `moves[]` (`ac`, `before[]`, `after[]`) | -| `evolve_skipped` | a use-case not evolved | `use_case`, `reason`: `declined`, or `no-design` in a `--from-stale` batch | +| `evolve_skipped` | a use-case not evolved | `use_case`, `reason`: `declined`, `no-design` in a `--from-stale` batch, or `source_changed` (out of date only because its source changed; `next?` reviews that change with `kane-cli maintain reconcile`, `remedy?`) | | `error` | a refusal or failure | `message`, `code?`: `MODE_USAGE` (only `interactive` and `agent` are accepted), `ANSWER_USAGE` (an answer to a question the run does not ask, or a value that is neither option) | | `done` | last | `status`, `exit_code`, `next?`: `complete` 0, `refused` 2, `paused` 3, `error` with the design run's own code; `next[]` carries every paused session's resume, uncapped | - **Answering.** `--answer evolve:<use-case>=1` (or `=proceed`) and `--answer evolve:<use-case>=2` (or `=decline`), repeatable, with `--mode agent`. A refused answer is `error` + `done` with nothing before it. -- **Refusals.** An explicit target that is fresh and has no `--because`, or that was never designed, exits `2`. A `--from-stale` run with nothing stale exits `0`. +- **Refusals.** An explicit target that is fresh and has no `--because`, that was never designed, or that is out of date only because its source changed (`evolve_skipped` `source_changed`; `--because` lets it run), exits `2`. A `--from-stale` run with nothing stale exits `0`, and one whose every stale use-case is source-changed sends those `evolve_skipped` events and exits `0`. - **Batches.** Every use-case runs whatever the earlier ones did. The exit is the first design-run code that is neither 0 nor 3; otherwise 3 if any paused; otherwise 0. ## Sync events (verb `sync`) *(0.8.14+)* diff --git a/.agents/skills/kane-cli/references/assurance.md b/.agents/skills/kane-cli/references/assurance.md index 7295995..efbe6ea 100644 --- a/.agents/skills/kane-cli/references/assurance.md +++ b/.agents/skills/kane-cli/references/assurance.md @@ -6,7 +6,7 @@ When the user has **requirements** — a PRD, a spec, acceptance notes — and w - **Assurance** = tests derived from the actual documents, every claim cited, every test permanently tagged with the acceptance criteria it verifies, coverage measured against requirements. Use it whenever the user cares about "what exactly is covered, and how do we know?" -Everything here works over a local store (`.context/` in the project directory) that the commands create and manage themselves. +Everything here works over a local store (`.context/` in the project directory) that the commands create and manage themselves. **A folder linked to a team's shared project** (it holds `.testmuai/assurance.json`; kane-cli 0.10.0+) works in the project instead, and its journey differs: no `context review` checkpoint, every session is saved explicitly, and `cover gaps` is refused. Follow `shared-projects.md` §4 there; the sections below describe the local store: a linked folder's events and refusal codes are in `agent-contract.md`, and the options it refuses in `shared-projects.md` §8. In agent mode a folder that isn't set up yet goes local unless you pass `--project <id>`, so settle where the work goes first (`shared-projects.md` §1). **Version gate — check before improvising.** The assurance commands exist on kane-cli **0.6.1 and later**; flags and events marked with a release (**0.7.1+** … **0.8.14+**) below need at least that release. On an older CLI, `kane-cli context …` fails as an *unknown command* (exit 2 with a "did you mean" suggestion) — that error means the CLI is too old, not that you typed it wrong. Confirm with `kane-cli --version`, tell the user to update (`npm install -g @testmuai/kane-cli`, or `brew upgrade kane-cli`), and stop — do not try to reproduce the workflow with other commands. @@ -19,7 +19,7 @@ kane-cli design tests --use-case <uc-ref> --mode agent --max 8 # 3. design ACs kane-cli context review --verdicts <file> --json # 4. CHECKPOINT: user approves the design kane-cli testmd run .testmuai/tests/<t>_test.md --agent # 5. author each kept test once (real browser) kane-cli testrun run --match 't-' < /dev/null # 6. batch replays from then on -kane-cli cover gaps # 7. designed % × proven % + per-use-case debt +kane-cli cover gaps # 7. the coverage figure, use-cases with tests, per-use-case debt kane-cli maintain reconcile --from <new.md> --source-id <id> --mode agent # when a source changes (§11) ``` @@ -30,7 +30,7 @@ Two hard rules: 1. **The two checkpoints are the user's, not yours.** Everything the extract and design agents emit is *derived* (unreviewed) until a human decision promotes it — see §4. Do not start authoring (step 5) until the user has approved the design output or explicitly told you to proceed. 2. **One step at a time, one store-writer at a time.** Never run two store-mutating commands concurrently (`ingest`, `extract`, `review` with verdicts, `design`, `reconcile`, `evolve`) — the store is single-writer (§8), and extraction additionally holds a store-wide lock (a concurrent extract refuses `EXTRACT_LOCKED`, exit 2 — wait, never delete locks). -Extract, design, reconcile and evolve call the KaneAI service and consume credits; everything else is local and free. +Extract, design, reconcile and evolve call the KaneAI service and consume credits. URL sources are fetched server-side, and a web page's ingest is refused when the organisation is out of credits. Everything else is local and free. ## 2. The pause loop — exit 3 is a pause, NOT a failure @@ -49,7 +49,7 @@ Three ways to resume: kane-cli context extract --resume <sid> --mode agent --message "Account required — the update section supersedes the older text" # by id (0.7.1+) — for precise scripted answers; each echoes a panel_resolved event: -kane-cli context extract --resume <sid> --mode agent --answer q1=1 --answer q2="https://staging.example.com" +kane-cli context extract --resume <sid> --mode agent --answer q1=1 --answer q2="https://app.example.com" # land a new source instead of answering (0.7.1+) — the batch defers (ask_deferred), the agent # reads the source and re-asks only what's still open: @@ -57,7 +57,7 @@ kane-cli context extract --resume <sid> --mode agent --with-source ./addendum.md ``` - If the answer leaves any question standing, the run pauses again with refreshed questions — repeat. -- Sessions live 24 hours. Inspect without contending a live run: `kane-cli context sessions --json` (all resumable sessions + their resume commands) and `kane-cli context sessions show <sid> --json` (the pending questions in wire shape). A session written by a NEWER kane-cli than the installed one lists as unknown with no resume command (0.7.1+) — that means "upgrade to resume it", not corruption. If you abandon a session deliberately, remove it with `kane-cli context sessions clean <sid>` — bare `clean` only collects *expired* sessions, and `--all` is a purge that needs explicit user authorization. +- Sessions live 24 hours. Inspect without contending a live run: `kane-cli context sessions --json` (all resumable sessions + their resume commands) and `kane-cli context sessions show <sid> --json` (the pending questions in wire shape). A session written by a NEWER kane-cli than the installed one lists as unknown with no resume command (0.7.1+) — that means "upgrade to resume it", not corruption. If you abandon a session deliberately, remove it with `kane-cli context sessions clean <sid>` — bare `clean` only collects *expired* sessions, and `--all` is a purge that needs explicit user authorization. (A linked folder refuses `sessions show` and `sessions clean`: `kane-cli context sessions --mode agent` lists the project's sessions there.) Full event schema: `references/assurance-parsing.md`. @@ -86,11 +86,11 @@ Remote ids share ONE space (0.8.6+): a URL whose id is already backed by a diffe Re-ingesting changed bytes under the same id **versions** the source and marks everything derived from the old snapshot stale — but for a changed document, prefer `maintain reconcile` (§11), which does the version move AND triages the fallout. -Extract behavior to expect (0.7.1+): the sweep **continues past a per-source failure** (one line each; exit 1 at the end if anything failed — the failed source simply retries next run, no `--force` needed). `--trust hold` holds everything new for the user's review instead of committing (headless-only; `--mode ci` refuses the flag entirely; use it one source at a time — resuming held work from a multi-source run is not yet supported). The agent cites every claim (fabrication is rejected before write), asks when the source contradicts itself, and commits proposals as `derived`. Watch the `commit` event and say it in plain language ("5 use-cases extracted from the PRD"). +Extract behavior to expect (0.7.1+): the sweep **continues past a per-source failure** (one line each; exit 1 at the end if anything failed — the failed source is read again on the next extract, which costs credits again; no `--force` needed). `--trust hold` holds everything new for the user's review instead of committing (headless-only; `--mode ci` refuses the flag entirely; use it one source at a time — resuming held work from a multi-source run is not yet supported). The agent cites every claim (fabrication is rejected before write), asks when the source contradicts itself, and commits proposals as `derived`. Watch the `commit` event and say it in plain language ("5 use-cases extracted from the PRD"). ## 4. Review — trust is the user's decision -Promotion from `derived` to `trusted` **always requires explicit user confirmation** — even when the user asked for an "end-to-end" run. End-to-end authorizes completing the workflow, not making product-requirement judgements. +Promotion from `derived` to `trusted` **always requires explicit user confirmation** — even when the user asked for an "end-to-end" run. *(In a shared project there is no review command: `context review` and `context explain` are refused in a linked folder, and use cases an agent run saves stay proposed. Run the checkpoint as `shared-projects.md` §4 says: show them, ask once for the batch, then design with `--allow-unreviewed` on a yes.)* End-to-end authorizes completing the workflow, not making product-requirement judgements. **Present the material, not the counts.** The `commit` event carries counts and ids only — to run a checkpoint, enumerate what's waiting with `kane-cli context list --json --inferred` (one JSON row per unreviewed node) and pull any item's full content, citations, and history with `kane-cli context explain <ref> --json`. Present titles, descriptions, and the cited evidence; collect the user's decisions; build the verdicts from those same refs. The same recipe runs the design checkpoint. Then land the decisions atomically: @@ -150,20 +150,22 @@ So a failing authored run is never, by itself, a reason to rewrite the step. Cla ## 7. What's next — let the tool tell you ```bash -kane-cli cover # the pack audit: what it proved, plus the live-graph completeness worklist -kane-cli cover gaps # the coverage ribbon: one row per use-case, both axes +kane-cli cover # one coverage figure over the use-cases; --from <pack> reads one run's own figure +kane-cli cover gaps # the coverage ribbon: that figure, then one row per use-case (designed count, coverage, debt) kane-cli cover gaps <uc-id> # 0.8.2+: the dossier — every AC of that use-case + next actions -kane-cli cover gaps --json # the nested document (designed axis, proven axis, per-UC pending rows with ready_command) +kane-cli cover gaps --json # the nested document (the figure, designed counts, per-UC pending rows with ready_command) kane-cli cover gaps --mode agent # 0.7.1+: same data as ONE `gaps` event + done.next[] ready-commands ``` -On 0.8.2+ the default `cover gaps` output is the **coverage ribbon** — a high-level band-table: one severity-ordered row per use-case with designed/proven bars and per-class debt counts, deliberately carrying NO commands. To act, drill in: `cover gaps <uc-id>` (the dossier) renders every AC of that use-case plus a `next` actions block — evidence first; surface its warning that re-running an authored test on a broken app may heal the test around the failure — or read `--json`, whose per-UC pending rows keep `ready_command`. `--rollup lenient|strict` selects the proven-axis formula. BREAKING at 0.8.2: `--flat` and gaps' `--from` are REMOVED and refuse (pre-0.8.2 CLIs still have the flat worklist). With a `<uc-id>`, `--json` and agent mode close the document over that use-case. +**In a shared project** (a linked folder): `cover gaps` is refused. `kane-cli cover --mode agent` gives the project's own figure, one `coverage` event then `done`. A `coverage` event with `state: "computing"` and no `pct` means the project is still working its figure out: say so, and run `done.next` again in a minute. A run less than 5 minutes old may not count yet (`recent_run.may_not_count_yet: true`); say so and offer to check again. Its `done.next` commands carry their agent flag (`--mode agent`, or `--agent` for `testrun run`): run them as given, ending a `testrun run` line in `< /dev/null`. If a `next` command has no agent flag, add the one that command takes before running it. + +On 0.8.2+ the default `cover gaps` output is the **coverage ribbon** — a high-level band-table: one severity-ordered row per use-case with its designed count, its coverage and per-class debt counts, deliberately carrying NO commands. To act, drill in: `cover gaps <uc-id>` (the dossier) renders every AC of that use-case plus a `next` actions block — evidence first; surface its warning that re-running an authored test on a broken app may heal the test around the failure — or read `--json`, whose per-UC pending rows keep `ready_command`. `--rollup` no longer changes the figure: don't pass it (agent mode answers it with a `warning` `ROLLUP_IGNORED`). BREAKING at 0.8.2: `--flat` and gaps' `--from` are REMOVED and refuse (pre-0.8.2 CLIs still have the flat worklist). With a `<uc-id>`, `--json` and agent mode close the document over that use-case. ## 8. Store rules — don't corrupt the user's graph - `.context/` is **append-only and single-writer**. Never run two store-mutating commands concurrently (extract, review verdicts, design, ingest, reconcile, retire/revert/rebuild). On a lock error (`EXTRACT_LOCKED`, or reconcile's walk lock), another run is live: report it and wait — never delete lock files; a dead run's lock clears itself. - Never hand-edit anything under `.context/`, and never `git merge` it — two histories merge cleanly in git and brick the store on the next read. Keep it gitignored — kane-cli 0.8.14 adds `.context/` to the folder's `.gitignore` itself when it creates the store inside a git repository — `gitignore_updated` on the stream, or a `warning{message}` when the line could not be written (relay it and add the line yourself); `KANE_CONTEXT_GITIGNORE=0` keeps it out. To share it with a team, use a **location** (a GitHub repository, an S3-compatible bucket, or a folder) through `kane-cli context sync`, `kane-cli context push`, `kane-cli context pull` and `kane-cli context clone` (0.8.14+) — a Git repository used as a location is not the same as committing `.context/`. While a sync rebase is open the store refuses every other write with `SYNC_REBASE_PENDING` (the fence) — finish it or `kane-cli context sync doctor --abort`, never delete files. Rules, events and decisions: `references/context-sync.md`. -- Safe inspection any time: `kane-cli context list --json` (nodes with trust + freshness), `kane-cli context explain <ref>` (a node's full recorded history, no AI), `kane-cli context fsck` (integrity check), `kane-cli context view --no-open --out <path>` (writes a self-contained HTML graph snapshot to that path — never opens a browser, never touches the graph). +- Safe inspection any time (a linked folder refuses `explain`, `fsck` and `view`; there use `kane-cli context list --mode agent`): `kane-cli context list --json` (nodes with trust + freshness), `kane-cli context explain <ref>` (a node's full recorded history, no AI), `kane-cli context fsck` (integrity check), `kane-cli context view --no-open --out <path>` (writes a self-contained HTML graph snapshot to that path — never opens a browser, never touches the graph). - Destructive verbs (`context retire`, `revert`, `rebuild`, `name --backfill`) exist and take `--yes` headless — run them **only on an explicit user request**, never autonomously. ## 9. When things fail @@ -197,6 +199,8 @@ On 0.8.2+ the default `cover gaps` output is the **coverage ribbon** — a high- | `sync_error` code `SYNC_BEHIND` / `SYNC_DIVERGED` (exit 3, 0.8.14+) | the location has records this store has not pulled / both sides added work — a pull or a rebase is needed, not a failure | relay the `remedy` verbatim (`kane-cli context pull <name>`, or `--rebase` on the user's go-ahead) — `references/context-sync.md` §6 | | `sync_behind` event on an extract/design/reconcile stream (0.8.14+) | advisory: the location has moved past this store | one line ("origin has records you have not pulled"); the run continues | | `sync_error` code `SYNC_STORE_BUSY` (exit 2, 0.8.14+) | a sync verb met a live session or lock on this store | wait; `kane-cli context sync doctor` when it names an interrupted rebase | +| `error` code `NO_PROPOSAL` | the agent stopped before proposing use-cases for that source | re-run `kane-cli context extract --mode agent` once; it reads that source again and costs credits again. If it happens twice, stop, and give the person the command and the message | +| a shared-project code (`PROJECT_NO_ACCESS`, `FOLDER_CHOSEN`, `PROJECT_BUSY`, `REMOTE_UNSUPPORTED`, `CONFIRM_REQUIRED`, `PROJECT_MISMATCH`, `TEST_CASE_MISSING` …) | a linked folder's refusal | `agent-contract.md`: each code with the command that recovers it | | `error` + `done` with exit `1` | runtime failure (incl. a sweep where some sources failed) | report the message; **do not blindly re-run a paid command** | | auth/credit failure mid-run | token or balance problem | keep the `sid`, have the user fix auth/balance (`kane-cli whoami`, `kane-cli balance`), then resume | | stream ends with no `done` event | the process crashed — outcome unknown (the one designed exception: `kane-cli context review --mode agent` keeps its `--json` shape and ends its listings without `done`; read its exit code) | check `context sessions --json` and `context list` before any retry, to avoid duplicate paid work | @@ -219,6 +223,8 @@ A changed requirements document is **one command**, not a re-ingest: kane-cli maintain reconcile --from ./prd-v2.md --source-id prd --mode agent ``` +**In a linked folder** reconcile takes a file, always with `--source-id`: `kane-cli maintain reconcile --from <file> --source-id <id> --mode agent`. It opens a review session that you save like any other (`shared-projects.md` §4), not the `reconcile_*` stream below. `--plan`, `--apply`, `--mode ci`, a missing `--source-id` and a URL in `--from` are refused (`REMOTE_UNSUPPORTED`); for a page's new version run `kane-cli context ingest <address> --mode agent`. The rest of this section is the local store's. + Never `context ingest` the new version first — reconcile does its own re-ingest, records the version move, and triages the fallout into rows: ADD (new use-case → design it), MODIFY (content moved → update or re-design), ARCHIVE (evidence decayed — **never applied headless**, always waits for the user's interactive session). It emits its own event family (`reconcile_plan` → `reconcile_row_start`/`reconcile_row_end` per row → `reconcile_paused`? → `reconcile_summary` → `done`; see `references/assurance-parsing.md`). Modes: `agent` auto-applies the safe ADD/MODIFY tier and exits 3 with the stored plan when ARCHIVE rows need a human; `ci` fail-closes (plan stored, exit 2); `--plan` is a safe staged preview. Re-running the same command is idempotent — it resumes a pending plan rather than re-billing. 0.7.2+ additions: @@ -227,9 +233,9 @@ Never `context ingest` the new version first — reconcile does its own re-inges - **`--from` takes URLs.** The same remote URLs ingest takes (§3): Jira/Confluence (0.7.2+), web pages (0.8.3+), Linear issues and documents (0.8.6+). `--source-id` is then optional — the URL carries its own id, and a contradicting `--source-id` refuses. Kind continuity holds both ways: a URL can't version a file-backed source and a file can't version a remote one — each refusal names the correct `--from`. A source ingested under a custom id (`--as`) is maintained by re-running that ingest with the same `--as`. The first reconcile of a Jira issue last ingested pre-0.7.2 may report the one-time upgrade re-version (§3) — not a content edit. - **One stream.** The re-extract child's events ride the reconcile stream itself, stamped `verb: "reconcile"` — parse per `references/assurance-parsing.md`. -For staleness that arrived outside a reconcile, `kane-cli maintain evolve <ref>` re-designs the parent use-case of the entity you name (`--from-stale` takes every use-case with stale designs). Give a `<ref>` or `--from-stale`, never both. A fresh target exits `2` unless you pass `--because "<reason>"`, and a use-case that was never designed exits `2` too (run `design tests` first). In a terminal evolve states the blast radius and asks. +For staleness that arrived outside a reconcile, `kane-cli maintain evolve <ref>` re-designs the parent use-case of the entity you name (`--from-stale` takes every use-case with stale designs). Give a `<ref>` or `--from-stale`, never both. A fresh target exits `2` unless you pass `--because "<reason>"`, and a use-case that was never designed exits `2` too (run `design tests` first). A use-case out of date only because its source changed is not evolved: as a target it exits `2` (unless `--because` is passed) and a `--from-stale` batch leaves it out, each with `evolve_skipped` `reason: "source_changed"` (and a `next`, when kane-cli knows the source's file, that reviews its change with `kane-cli maintain reconcile …`): offer the reconcile instead. In a terminal evolve states the blast radius and asks. -*(0.8.21+)* `kane-cli maintain evolve <ref> --mode agent` puts that question on the stream. A call on a stale target with no answer ends paused, exit `3`: `evolve_target` carries the blast radius (`use_case`, the `pairs` it affects, `because?`), then `evolve_paused` carries `pending_questions` (one per use-case, `id: "evolve:<use-case>"`, options `proceed` / `decline`), `resume` and `next[]` (two ready commands: proceed all, decline all), then `done{paused}`. Present the blast radius with the question, as with any pause (the pause loop above). Answer on the next call with `--answer evolve:<use-case>=1` (or `=proceed`) or `--answer evolve:<use-case>=2` (or `=decline`), repeatable; `--answer` needs `--mode agent`, and the resume command keeps `--because`. Nothing runs until every question is answered. A partial set pauses again on the rest, and the `next` commands then carry the answers already given, so a "decline the rest" command still re-designs what was answered proceed: read a `next` command before running it. On proceed, the design run's own events arrive between `child_start` and `child_end` stamped `verb: "design"`. After a clean design run comes `evolve_diff` (superseded, minted, retained, criteria whose tests moved). A design session that pauses arrives as its own `session_paused`: resume that session, then run evolve again for the diff. `done.next[]` carries every paused session's resume, all of them, whatever the exit code. In a `--from-stale` batch every use-case runs whatever the earlier ones did, and a declined or never-designed one is reported as `evolve_skipped`. Schema: `references/assurance-parsing.md`. On a CLI before 0.8.21 evolve has no `--mode`: suggest the user run it in a terminal rather than scripting around it. +*(0.8.21+)* `kane-cli maintain evolve <ref> --mode agent` puts that question on the stream. A call on a stale target with no answer ends paused, exit `3`: `evolve_target` carries the blast radius (`use_case`, the `pairs` it affects, `because?`), then `evolve_paused` carries `pending_questions` (one per use-case, `id: "evolve:<use-case>"`, options `proceed` / `decline`), `resume` and `next[]` (two ready commands: proceed all, decline all), then `done{paused}`. Present the blast radius with the question, as with any pause (the pause loop above). Answer on the next call with `--answer evolve:<use-case>=1` (or `=proceed`) or `--answer evolve:<use-case>=2` (or `=decline`), repeatable; `--answer` needs `--mode agent`, and the resume command keeps `--because`. Nothing runs until every question is answered. A partial set pauses again on the rest, and the `next` commands then carry the answers already given, so a "decline the rest" command still re-designs what was answered proceed: read a `next` command before running it. On proceed, the design run's own events arrive between `child_start` and `child_end` stamped `verb: "design"`. After a clean design run comes `evolve_diff` (superseded, minted, retained, criteria whose tests moved). A design session that pauses arrives as its own `session_paused`: resume that session, then run evolve again for the diff. `done.next[]` carries every paused session's resume, all of them, whatever the exit code. In a `--from-stale` batch every use-case runs whatever the earlier ones did, and a declined, never-designed or source-changed one is reported as `evolve_skipped`. Schema: `references/assurance-parsing.md`. On a CLI before 0.8.21 evolve has no `--mode`: suggest the user run it in a terminal rather than scripting around it. ## Sharing a context store diff --git a/.agents/skills/kane-cli/references/cards.md b/.agents/skills/kane-cli/references/cards.md index b035268..c8e58f2 100644 --- a/.agents/skills/kane-cli/references/cards.md +++ b/.agents/skills/kane-cli/references/cards.md @@ -73,7 +73,7 @@ Exit code `1`, or `status: "failed"`. Show the failing step's screenshot under t ## 4. Didn't start -Exit code `2`: nothing ran and no credits were used. Causes include no start URL, sign-in or setup errors, a test file that does not parse, an invalid suite plan, a cloud grid refusal. +Exit code `2`: nothing ran and no credits were used. Causes include no start URL, sign-in or setup errors, a test file that does not parse, an invalid suite plan, a cloud grid refusal. A designed test in a shared project that stopped on its Test Manager case before any step (an `error` with a `TEST_*` code) is this card too, even when it exits `1`. ```markdown | | | @@ -173,3 +173,38 @@ On kane-cli older than 0.8.17 the end event has no reason: read it from the evid A test that comes back broken with zero steps on the grid was refused before it launched: say so, point to the job link, and suggest checking that the app id belongs to this account. An invalid plan is a `🟡 Didn't start` card (§4) with one line per rejected test. + +**In a shared project** a suite's card adds a coverage row from the run's `coverage` event, and a line for each first +run (`authored`): + +```markdown +| 📈 **Coverage** | <pct>% of "<project>" · this run reached <n> use cases · it can take up to 5 minutes to count | +| ✍️ **First runs** | <n> tests wrote their steps on this run; later runs replay them | +``` + +Only a `coverage` event with `status: "written"` has a `pct`. With any other status, the 📈 row says +`The project's coverage isn't ready yet · I can check again in a few minutes` (`kane-cli cover --mode agent`). + +A member that stopped on its Test Manager case shows `failed` with a reason such as "isn't in Test Manager yet": say what +it needs (`testrun.md` §Designed tests in a shared project), not "the test failed". + +## 9. Coverage (`kane-cli cover`) + +Fields: the `coverage` event's `pct`, `usecases` (`total`, `proven`, `partial`, `failing`, `not_run`, `not_designed`, +`no_criteria`), `note`, and `recent_run` when a run is under 5 minutes old; then `done.next`. A `coverage` event with +`state: "computing"` has none of these: say the project is still working its figure out, draw no card, and run +`done.next` again in a minute. + +```markdown +| | | +|---|---| +| 📈 **Coverage** | <pct>% across "<project>"'s <total> use cases | +| ✅ **Proven** | <proven> · ⚠️ partial <partial> · ❌ failing <failing> | +| ⏳ **Not yet** | <not_run> not run · <not_designed> without tests · <no_criteria> with no criteria yet | +| 🕒 **Recent run** | still counting: check again in a few minutes *(only with `recent_run.may_not_count_yet`)* | +| ➡️ **Next** | <the first `done.next` command in plain words, e.g. run every test · design tests for "<use case>"> | +``` + +`note: "nothing_run_yet"` with `total: 0` means no use case exists yet: offer `kane-cli context ingest <file>`, not a run. +`done.next` commands carry their agent flag; run them as given, ending a `testrun run` line in `< /dev/null`. If one +has no agent flag, add the one that command takes. diff --git a/.agents/skills/kane-cli/references/context-sync.md b/.agents/skills/kane-cli/references/context-sync.md index a9f19b0..671f3c9 100644 --- a/.agents/skills/kane-cli/references/context-sync.md +++ b/.agents/skills/kane-cli/references/context-sync.md @@ -2,6 +2,13 @@ # Context sync — Agent Surface +> **Not for shared projects.** Location sync shares a **local** `.context/` store through a Git repository, a bucket or a +> folder. A team that works in a Test Manager project uses a shared project instead (`shared-projects.md`). A folder +> linked to one refuses `kane-cli context sync`, `kane-cli context push` and `kane-cli context clone` +> (`REMOTE_UNSUPPORTED`), and a location pull (a location name, `--rebase`, `--yes` or `--answer`: +> `PULL_LOCATION_LINKED`); a plain `kane-cli context pull --mode agent` there brings the project's tests, when +> `kane-cli context pull --help` lists `--project` (`shared-projects.md` §7). Never use both in one folder. + The assurance store (`.context/`, `references/assurance.md` §8) is one person's. **Context sync** shares it with a team through a **location**: a GitHub repository, an S3-compatible bucket, or a folder. Each person keeps their own store and publishes to the location; nothing on a location is ever overwritten or deleted. Reach for this reference when the user wants to share their context store, join a teammate's, get two stores level, or resolve a sync conflict. Sync never calls the KaneAI service — no credits, no model. **Version gate.** `kane-cli context sync`, `kane-cli context push`, `kane-cli context pull` and `kane-cli context clone` exist on **kane-cli 0.8.14 and later**. On an older CLI they fail as an *unknown command* (exit 2, a "did you mean" line) — the CLI is too old, not a typo. Confirm with `kane-cli --version`, have the user update, stop. diff --git a/.agents/skills/kane-cli/references/debug.md b/.agents/skills/kane-cli/references/debug.md index 372d188..9a33997 100644 --- a/.agents/skills/kane-cli/references/debug.md +++ b/.agents/skills/kane-cli/references/debug.md @@ -6,7 +6,7 @@ When a run fails, diagnose before suggesting fixes. ## The evidence pack is the log source -Every run seals an **evidence pack**; all run artifacts — actions, console, network, screenshots, failure records — live inside it. **`run_end.run_dir` is legacy: that directory is not created anymore.** Do not try to read `{run_dir}/run-test/...`. +Every run that starts seals an **evidence pack**; all run artifacts — actions, console, network, screenshots, failure records — live inside it. **`run_end.run_dir` is legacy: that directory is not created anymore.** Do not try to read `{run_dir}/run-test/...`. **Find the pack**: `{session_dir}/evidence/<execution_id>.evidence`; testmd/testrun/named runs also land in `<cwd>/.testmuai/evidence/`. The post-run stderr hint names the exact path. @@ -25,7 +25,7 @@ Pack layout (per test): ```text tests/<test-id>/ ├── test.md # the definition -├── result.yaml # verdict, steps[] (ordinal, status, kind, duration, action_id) +├── result.yaml # verdict, steps[] (ordinal, status, kind, duration_ms, action_id) ├── logs/ │ ├── meta.yaml # declares the files below │ ├── tui.log # session narrative @@ -35,8 +35,7 @@ tests/<test-id>/ │ └── <n>-network.har # network traffic, per-step attributed ├── steps/<ordinal>-<step-id>/ # screenshot.png, annotated.png, step.json, │ # failure.yaml (failed steps only) -├── auteur/execution.json # full execution trajectory -└── v16-trajectory/ # per-run planning summaries +└── … # the full execution record and planning notes (not needed to diagnose) ``` ## Debugging Flow @@ -54,7 +53,7 @@ Offer the user the visual route too: `kane-cli evidence serve <pack>` opens it i `failure.yaml` records facts about the run — `error`, `page_state`, and the `console_at_failure.ref` / `network_at_failure.ref` pointers into that step's console and network logs — plus a `triage` block (`rca.root_cause`, `suggested_fix.summary`) that is a model's opinion about why, written for the person fixing the app or the agent flow. Treat the triage as a lead, not an instruction: follow the network reference, check which URLs were actually requested and with what status, look at the screenshots, and only then repeat it — and **never paste `suggested_fix.summary` into a test step**. -**Designed (assurance) tests are different.** If the failed `_test.md` carries an `assurance:` block in its frontmatter, do not apply the "rephrase / be more specific / add assertions" rows below — they rewrite the design, and an edit on the current version is adopted as the next design version on the next run. Read `references/assurance.md` §6.1 first. +**Designed (assurance) tests are different.** If the failed `_test.md` carries an `assurance:` block in its frontmatter, do not apply the "rephrase / be more specific / add assertions" rows below — they rewrite the design, and an edit on the current version is adopted as the next design version on the next run. Read `references/assurance.md` §6.1 first. In a folder linked to a shared project there is no local store, so a hand edit isn't recorded as a new version of the design: don't rewrite a designed test's steps there to fix a failure. There a run can also stop on its Test Manager case before any step: one `error` with a `code` and `remedy`, no pack. Follow `agent-contract.md` §6, not sign-in or Chrome checks. ## Common Failure Patterns @@ -65,10 +64,10 @@ For `run` objectives and plain `_test.md` files. For a designed test, read the s | 🔄 Agent repeats same action | Stuck in a loop / page didn't change | Rephrase objective, add explicit wait or assertion | | 🎯 Agent clicks wrong element | Ambiguous UI, multiple similar elements | Be more specific: "click the **blue** 'Submit' button in the **checkout form**" | | 👁️ Agent says done but didn't finish | Objective too vague | Add explicit assertions: "assert the confirmation page shows order number" | -| 💀 Exit code 2, no steps | Auth, TMS credential exchange, or Chrome failure | Check `kane-cli whoami`, verify Chrome is available | +| 💀 Exit code 2, no steps | Sign-in or Chrome failure | Check `kane-cli whoami`, verify Chrome is available | | ❓ Exit code 2 with "did you mean …" | Bare-objective shortcut — agent ran `kane-cli "<objective>"` without the `run` subcommand | Re-invoke as `kane-cli run "<objective>" --agent` (same rule for `testmd run`) | | 📤 Upload silently fails after configuring a project/folder by hand | Saved ID is invalid (typo, deleted, no access) | No action needed — the next run detects the 4xx and auto-defaults a working project/folder. To rebind manually: `kane-cli config project` (TTY picker) or `kane-cli projects list` → `kane-cli config project <id>` (see `references/test-manager.md`) | -| ⏱️ Exit code 3 | Timeout or cancelled | Increase `--timeout` or `--max-steps`, or split into smaller objectives | +| ⏱️ Exit code 3 (`run`; a saved test reports it as failed, exit 1) | Timeout or cancelled | Increase `--timeout` or `--max-steps`, or split into smaller objectives | | 🚫 "CDP endpoint not reachable" | Chrome not running | Let kane-cli manage Chrome (remove `--cdp-endpoint`) | ## Filing a bug report diff --git a/.agents/skills/kane-cli/references/evidence.md b/.agents/skills/kane-cli/references/evidence.md index 32ff71b..fbe6643 100644 --- a/.agents/skills/kane-cli/references/evidence.md +++ b/.agents/skills/kane-cli/references/evidence.md @@ -20,8 +20,7 @@ A `.evidence` file is a standard zip: `unzip -l <pack>` lists it, `unzip -p <pac │ # <n>-console.ndjson, <n>-network.har ├── steps/<ordinal>-<step-id>/ # screenshot.png, annotated.png, step.json │ # (+ failure.yaml on failed steps) - ├── auteur/execution.json # full execution trajectory - └── v16-trajectory/ # per-run planning summaries + └── … # the full execution record and planning notes (not needed to diagnose) ``` Reading and debugging from the pack (unzip recipes, failure records): `references/debug.md`. @@ -36,7 +35,7 @@ Reading and debugging from the pack (unzip recipes, failure records): `reference ## The stderr hint — and how to act on it -After a successful agent-mode run, kane-cli prints one hint line to **stderr** (it is NOT an NDJSON event; never look for it on stdout): +After an agent-mode run whose evidence pack sealed (passed or failed), kane-cli prints one hint line to **stderr** (it is NOT an NDJSON event; never look for it on stdout): ``` evidence: view locally with `kane-cli evidence serve <packPath>` @@ -70,7 +69,7 @@ Exit `0` valid · `1` invalid · `2` not found. Use it when a pack won't open in kane-cli evidence merge <targets...> --run-id <id> ``` -Combines packs (execution ids or paths, order-significant) into one, sealed by default at `.testmuai/evidence/<run-id>.evidence`. `--run-id` is required. Exit `0` merged · `1` policy abort (e.g. mixed projects, duplicate run ids) · `2` usage. Full policy detail is in the user guide — for anything beyond a simple merge, point the user there rather than improvising flags. +Combines packs (execution ids or paths, order-significant) into one, sealed by default at `.testmuai/evidence/<run-id>.evidence`. `--run-id` is required. Exit `0` merged · `1` policy abort (e.g. packs from different projects or organisations, or a pack that isn't sealed and valid; a pack whose run id repeats is skipped, not an abort) · `2` usage. Full policy detail is in the user guide — for anything beyond a simple merge, point the user there rather than improvising flags. ## Debugging with a pack @@ -78,6 +77,6 @@ The pack is the first place to look on a failure: the failed step's **failure re ## Evidence merge identity -For ordinary runs, default merge identity distinguishes the test and commit (`external_id.test_id`, `external_id.commit_id`) and the environment (`environment.os`, `environment.os_version`, `environment.browser`, `environment.browser_version`, `environment.device`, `environment.app`). Re-runs with the same identity nest as attempts; a different environment produces a separate sibling, so one test on two devices or two builds is two entries. On a real device, `environment.device` is the device the hub allocated (never the requested pattern) and `device_slug` is derived from it. +For ordinary runs, default merge identity distinguishes the test, commit and dataset row (`external_id.test_id`, `external_id.commit_id`, `external_id.dataset_row_id`) and the environment (`environment.os`, `environment.os_version`, `environment.browser`, `environment.browser_version`, `environment.device`, `environment.app`). Re-runs with the same identity nest as attempts; a different environment produces a separate sibling, so one test on two devices, or on two builds with different app identities, is two entries (two local builds with the same file name can group together). On a real device, `environment.device` is the device the hub allocated (never the requested pattern) and `device_slug` is derived from it. Explicit collision policies can change grouping; a custom `--rules` file replaces the default rules rather than extending them. Check the selected identity rules before interpreting two runs as retries of the same test. diff --git a/.agents/skills/kane-cli/references/execution-controls.md b/.agents/skills/kane-cli/references/execution-controls.md index 5cae455..3cb1086 100644 --- a/.agents/skills/kane-cli/references/execution-controls.md +++ b/.agents/skills/kane-cli/references/execution-controls.md @@ -4,7 +4,7 @@ `run` and `testmd run` support `--assertion-mode dom|visual` (default `dom` with vision fallback) and `--final-validation on|off` (default off). Persist with `config set-assertion-mode` and `config set-final-validation`. Final validation controls the synthesized `cp_final` checkpoint independently of action/testing mode; keep explicit terminal assertions in objectives. -`run --analyzer-only --condition "<condition>"` checks the current desktop browser page without an objective, action steps or saved test. Repeat `--condition` for multiple checks. Use `--agent` or non-TTY input. Results contain `condition_results: boolean[]`; exit `0` means all conditions were judged, **not** that all are true. Exit `1` means a result was missing, and `3` means cancelled. This mode rejects mobile target/app/device options and code-export/name options. +`run --analyzer-only --condition "<condition>"` checks the current desktop browser page without an objective, action steps or saved test. Repeat `--condition` for multiple checks. Use `--agent` or non-TTY input. Results contain `condition_results: (boolean | null)[]`, one per condition; `null` means that condition couldn't be judged, and `condition_reasons` says why. Exit `0` means all conditions were judged, **not** that all are true. Exit `1` means a result was missing or `null`, and `3` means cancelled. This mode rejects mobile target/app/device options and code-export/name options. Experimental `run --network-ws` and `run --network-sse` enable WebSocket and SSE capture; both default off. Persist with `config set-network-ws on|off` and `config set-network-sse on|off`. SSE capture is Chromium-only. Do not copy these run-only flags onto `testmd` or `testrun` commands. diff --git a/.agents/skills/kane-cli/references/explore.md b/.agents/skills/kane-cli/references/explore.md new file mode 100644 index 0000000..64199f5 --- /dev/null +++ b/.agents/skills/kane-cli/references/explore.md @@ -0,0 +1,217 @@ +<!-- kane-cli skill reference: exploration on HyperExecute, and the tests it designs into a shared project. Read when the person asks to explore an app, map its flows, shape, stop or re-run an exploration, or get and run the tests an exploration made. Requires kane-cli 0.10.0+. --> + +# Exploration: map an app on HyperExecute, and get the tests it designs + +`kane-cli explore` sends an explorer through the person's app on **HyperExecute**. It builds a map of the app (screens, +flows, findings, with evidence). While its **assurance context** is on (the default), every few minutes while it runs, +and when it ends, what it has mapped is turned into use cases, scenarios and tests in the Test Manager project the +exploration belongs to, with a Test Manager case per scenario. Exploration **designs and saves** those tests; it never +runs them. + +**Exploration and assurance.** The tests an exploration designs live in its project, as a document's do in a shared +project (`shared-projects.md`). To run them here, a folder has to hold them: a folder linked to that project gets them +with `kane-cli context pull --mode agent`, and a fresh folder links and gets them in one step with +`kane-cli context pull --project <id> --mode agent` (§5). A project nobody has shared yet becomes a shared project with +the exploration's first design job, as the person who started the run. + +Explore takes `--agent`, not `--mode agent`. Its messages can name internal parts: use them to work out what happened, +then tell the person in plain words. + +## 1. Start an exploration + +```bash +kane-cli explore run --project <id> --url <app url> --prompt "<what to explore>" --agent # add --policy read_only if the person said no changes +``` + +- `--project <id>` is the Test Manager project the exploration saves into: the team's shared project + (`shared-projects.md` §2 finds the id). One project, one exploration, one URL. **Always pass it**: explore doesn't + read the folder's link, so without `--project` it falls back to the project kane-cli is configured with, and a run + with none is refused. In a linked folder, pass the folder's project id (the preflight's `## link`). +- It queues the run on HyperExecute and exits; it doesn't wait for the exploration to finish. +- It uses the person's sign-in, turned into their username and access key. Only if it says it needs basic credentials, + ask the person to run `kane-cli login --username <user> --access-key <key>` in their own terminal. It stores the + sign-in and any secret variables in the person's HyperExecute Secrets, and the non-secret variables in Test Manager. + Never put values in chat. +- The default policy is `mutating`: the explorer may submit forms and change data on the sites it reaches, and in agent + mode kane-cli asks nobody. Before `explore run`, tell the person and ask whether it may change data in their app; if + not, pass `--policy read_only` (it keeps the explorer from changing data). A `mutating` run starts with a `warning` + (`code: "policy_mutating"`): relay it. +- Under `read_only`, the setup steps in `--deps` (a sign-in, for example) still run as written, so put in `--deps` only + what the person allows. +- An exploration spends credits while it runs, and so does each design job it starts (§4). Without `--max-leases` it + runs until it runs out of work. Tell the person before `explore run`, and offer a limit when credits matter (§2). +- A project that isn't shared yet is shared by the exploration's first design job, as the person who started the run. + If they may not create test cases in it, no tests are designed: `explore context status` shows `context.enabled: true` + with `context.off_reason: "mark_refused"`. Tell the person: "Ask a Test Manager admin for permission to create test + cases, or ask someone who has it to share the project." Once they have permission, turn context off and on (§4), only + on their word. + +**The stream:** `stream_start` (`surface: "explore"`), maybe `warning`, `explore_remote_started {exploration_id, run_id, +state, hx_job_id, job_link}`, then `done {state: "queued", exit_code: 0, exploration_id, run_id}`. Three plain lines on +stderr name the run, the job link and how to watch it. Give the person the job link. + +| Exit | Means | Do | +|---|---|---| +| 0 | queued | tell the person it runs on HyperExecute and can take a while; offer to check its status later | +| 1 | refused: `RUN_ACTIVE` (this exploration already has a run), or the launch failed | `RUN_ACTIVE`: wait, or ask the person whether to stop it (§3). A failed launch: try again in a minute; then tell the person | +| 2 | bad input (a different `--url` for the project's exploration, a `{{name}}` with no value, `--deps` with no `--allow-domain`), sign-in, a secret HyperExecute doesn't list (`SECRETS_MISSING`), or a message starting "could not push" or "could not read" | fix the input or the secret's name. If `done` names a run (`state: "stopped"`, `remark` `failed:secrets_missing`), report that run, fix the value, then start a new one. "Could not push / read": try again in a minute; then tell the person | + +Explore's `error` events carry a `message` only: no `code`, no `next`. Read the message. + +**On this machine** (`--local-orchestration`, below) the command streams the run and ends with it: `done` carries +`state` and `remark`. Exit 0 completed or reached its limit, 1 failed, 3 stopped or timed out, 2 couldn't start. + +## 2. Shape the run + +Add these to `explore run` only when the person's request needs them: + +| The person wants | Pass | +|---|---| +| An app behind a login | `--deps "Logged in as {{user}} / {{pass}}"`, with the values in a variables file (`.testmuai/variables/*.json`, or `--variables-file <path>`; a secret has `"secret": true`). `--deps` always needs `--allow-domain`: the app's own host, and every other host it signs in on (`--allow-domain app.example.com --allow-domain login.example.com`). Only the names the prompt and `--deps` use travel: secrets go to the person's HyperExecute Secrets, the rest to the organisation's Test Manager variables, where they replace a variable of the same name (tell the person). Names are letters, digits and `_`; `LT_USERNAME` and `LT_ACCESS_KEY` are taken. Check the values are real first: an empty secret stops the run `failed:secrets_missing`, and a sign-in that fails stops it `failed:setup_failed` | +| Only some sites | `--allow-domain <host>`, repeatable (the URL's host is always in, subdomains included) | +| A control it must never press | `--deny "<control name>"`, repeatable, under either policy | +| A `read_only` run of an app that reads its data through POST | `--allow-post <host/path>`, repeatable, only for an endpoint the person confirms only reads; its host must be the URL's or one passed with `--allow-domain` | +| Start from one screen of the map it already has | `--focus "<screen>"` (a later run; the screen's name or words from it). On this machine it also needs `--exploration-id <exploration-id>` | +| A limit on the explorer's work | `--max-leases <n>`: it stops after n pieces of work (leases), and the run ends `budget`. Every run also stops at built-in limits (an hour, 400 actions or 800 model calls), also `budget`. Neither limits the design jobs' credits: for a map with no new tests, see the next rows | +| More browsers at once | `--max-parallel <n>` (default 1; up to 20 cloud browsers, 4 with Chrome on this machine). Each may sign in once: keep 1 for an app that allows one login | +| A map, no tests | `--no-context`, on the project's first exploration only; for an existing one, `explore context off` (§4) | +| How deep it goes, what counts as slow, whether findings are checked again | `--depth-caps`, `--slow-budget`, `--no-recheck`: `kane-cli explore run --help` gives their values | +| An app on `localhost` or a private network | `--tunnel-name <name>`, through a LambdaTest tunnel on the person's machine (below) | +| To explore from this machine | `--local-orchestration` (Chrome here, `--headless` for no window; or `--browser remote` for cloud browsers). The command then waits for the run to end (§1), and Ctrl+C stops it | + +### An app on `localhost` or a private network + +The cloud's browsers can't reach it on their own: they reach it through a LambdaTest tunnel (LT) running on the person's +machine. kane-cli never starts or stops the tunnel; the person does, in their own terminal, so their key never comes +into chat. + +1. If `LT --version` doesn't run there, they download LT for their machine and put it on their PATH: + - Apple silicon Mac: `https://downloads.lambdatest.com/tunnel/v3/mac/arm64/LT_Mac.zip` + - Intel Mac: `https://downloads.lambdatest.com/tunnel/v3/mac/64bit/LT_Mac.zip` + - Linux: `https://downloads.lambdatest.com/tunnel/v3/linux/64bit/LT_Linux.zip` (`linux/arm64/LT_Linux.zip` on ARM) + - Windows: `https://downloads.lambdatest.com/tunnel/v3/windows/64bit/LT_Windows.zip`, then run `LT.exe` + + If a Mac blocks it from opening: `xattr -d com.apple.quarantine <path to LT>`. +2. They start it as the same LambdaTest user kane-cli is signed in as, and leave it running: + + ```bash + LT --user "$LT_USERNAME" --key "$LT_ACCESS_KEY" --env ht-prod --tunnelName <name> + ``` + + (`$LT_USERNAME` and `$LT_ACCESS_KEY` are their LambdaTest username and access key.) `--env ht-prod` is required. + Wait until LT says the tunnel is connected. +3. Run the exploration through it: + + ```bash + kane-cli explore run --project <id> --url http://localhost:3000 --prompt "<what to explore>" --tunnel-name <name> --agent + ``` + + `--tunnel-name` must match `--tunnelName` from step 2, and it turns `--tunnel` on; with only one tunnel running, + `--tunnel` alone is enough. A re-run keeps the exploration's URL but not its tunnel, so pass the tunnel flag again: + `kane-cli explore run --project <id> --tunnel-name <name> --agent`. + +- The tunnel **and that machine** must stay up until the run ends; the run warns about it when it starts. If the tunnel + drops mid-run, pages stop loading. +- A tunnel needs cloud browsers: with `--local-orchestration` and local Chrome, `--tunnel` is refused (local Chrome + already reaches the machine). + +## 3. Watch it, stop it, run it again + +```bash +kane-cli explore list --agent # the running explorations; --all lists every one +kane-cli explore sessions <exploration-id> --agent # its runs, newest first: state, remark, leases, findings, credits +kane-cli explore sessions <exploration-id> --run <run-id> --agent # one run in full +``` + +The person can watch a run live with `kane-cli explore view --exploration-id <exploration-id>` in their own terminal +(§4). + +A run is `queued` (waiting for its machine), `running` or `stopped`. A stopped run's `remark.code` says why: +`completed` (it ran out of work), `budget` (`--max-leases` or a built-in limit reached), `user_stop`, `stopped_before_start` (stopped while +queued), `heartbeat_lost` (it stopped reporting), `timeout`, or `failed:<reason>` (for example `failed:secrets_missing`, +`failed:setup_failed`: the sign-in in `--deps` didn't work, or `failed:vm_never_started`: HyperExecute never started it, +so run it again). Say it to the person in plain words. + +- **Stop it:** `kane-cli explore stop <exploration-id> --agent` stops a queued run at once, and asks a running one to + stop cleanly (its work in hand finishes and is kept). It waits for that, unless you pass `--no-wait`. It emits + `explore_stop {run_id, state, remark, stopping}`. When `stopping` is `true` the run hasn't stopped yet: check + `explore sessions` until it is `stopped` before you run it again or delete it. With nothing running it refuses + (`RUN_NOT_ACTIVE`, exit 1). +- **Run it again:** `kane-cli explore run --project <id> … --agent` once more. It goes on the same exploration, on its + URL, and adds to its map; a different `--url` is refused (exit 2). A run keeps only the URL: pass every option again + (`--policy read_only` above all, since the default is `mutating`, and the prompt, `--deps`, domains, variables, + limits and tunnel). +- **Explore another site:** use another project, so its tests don't mix with this site's. Or, only on the person's + word, `kane-cli explore delete <exploration-id> --yes --agent` (refused with `RUN_ACTIVE`, exit 1, while a run is + queued or running): no command shows that map again and the project can start a new exploration of any URL, but the + use cases and tests designed for the first site stay in the project; a pull brings them along with the new site's. +- `list`, `sessions`, `stop` and `delete` exit 0 when done, 1 when refused (`RUN_ACTIVE`, `RUN_NOT_ACTIVE`) or when the + service failed (read the message), and 2 for bad input, a missing `--yes`, an unknown exploration, or no sign-in. + +## 4. Check what it designed + +```bash +kane-cli explore context status <exploration-id> --agent +``` + +`explore context status` emits `explore_context {exploration_id, action, context}`. Read `context.enabled`, +`context.off_reason`, `context.running_job` and `context.pending_cut` (work waiting for the next job) as well as +`context.last_job`, the newest design job that has **ended**: `status` (`ok`, `noop`, `partial`, `failed`), +`use_cases.added[]` and `modified[]`, `tests.designed`, `tests.saved`, `tests.failed[]`, `tests.saved_cases[]` (each +test's Test Manager case), `held[]` (changes, such as an archive, the job left for a person to review), `credits` (that +job's own), `error`. A job runs after the exploration's run ends, and during a long run every few minutes when enough +changed. Before you call the designs ready, check that no job is still running or waiting. An entry of `saved_cases[]` +can have no case and a refusal even when the job says `ok`: tell the person which tests are saved but can't run yet, +and follow that refusal's remedy. + +```bash +kane-cli explore context sync <exploration-id> --agent # ask for a job now +``` + +It emits `explore_context_sync {exploration_id, state, …}`: `launched`, `launching`, `pending` (behind a running job), +`running` and `up_to_date` all end exit 0. A known refusal's `error` message starts with its code (other failures are +plain words: read the message and the exit code): + +| Refusal | Exit | Means | Do | +|---|---|---|---| +| `TOO_SOON` | 1 | the last design job started a few minutes ago | try again when it says | +| `NO_ENDED_RUN` | 1 | no run of this exploration has ended | wait for the run, then sync | +| `CONTEXT_OFF` | 1 | assurance context is off for this exploration | `kane-cli explore context on <exploration-id> --agent`, only with the person's word. If the message names a reason in brackets other than `user_off`, tell the person instead: turning it on helps only for `mark_refused`, once they have permission (§1) | +| `NOT_CONFIGURED` | 1 | design jobs aren't set up here yet; context stays on and runs once they are | tell the person; don't retry | +| a `failed` state | 1 | the job couldn't launch | try once more; then tell the person | +| sign-in or access | 2 | the sign-in was refused, or basic credentials are needed | as in §1: sign-in first, then once more | +| not found | 2 | no such exploration, or assurance context isn't available here yet | check the id (`kane-cli explore list --all --agent`); then stop | + +`kane-cli explore context off <exploration-id> --agent` stops new design jobs and drops the work waiting for one; +nothing already saved is deleted, and a job already running finishes (it may still save tests and use credits): check +the status and tell the person. `on` catches up from the newest ended run. Both only on the person's word. + +To show the map, ask the person to run `kane-cli explore view --exploration-id <exploration-id>` (or `--project <id>`) +in their own terminal: it opens the map in their browser and keeps serving it until they press Ctrl+C. There they can +look at the evidence, correct findings, leave notes, and mark screens or controls not to touch. Don't run it yourself: +it doesn't exit on its own. + +## 5. Get exploration's tests into a folder, and run them + +The use cases and tests are in the project, and their cases are in Test Manager, as soon as a job saves them. No +folder holds their files until it pulls them. + +- Read `shared-projects.md` §7a before the pull, and check first whether this kane-cli has it (`kane-cli context pull + --help` lists `--project`). With it: + - **a folder linked to the project:** `kane-cli context pull --mode agent`; + - **a fresh folder:** `kane-cli context pull --project <id> --mode agent` links it and brings the tests in one step. +- Read each `pull_test`'s `outcome`, and count the first runs from its `first_run: true` (a fresh folder's tests all + have it; a folder that ran them before may not). Fill the values `variables_needed` names: an app behind a login + needs its sign-in values in this folder too. A pull with no tests yet ends `done` exit 0: check + `explore context status <id>`. +- Without the pull (kane-cli 0.10.0), exploration's tests can be seen (`kane-cli context list --mode agent` in a linked + folder; Test Manager) but not run from a folder: no folder holds their files. +- The first run of each pulled test writes its steps in a real browser: minutes, and about 30–85 credits per test. Tell + the person the count and the likely cost, then run the files they choose: + `kane-cli testrun run <files> --agent < /dev/null`. +- Exploration's use cases are designed without review: in agent mode `context list` shows them as `trust: "derived"`. + Present them to the person as exploration's proposals. The source row's `id` is `explore-<exploration id>`. + `kane-cli explore list --all --agent` lists every exploration (without `--all`, only the running ones); a page's + `next_cursor` goes to `--cursor` for the next. +- After the first runs, `kane-cli cover --mode agent` in the linked folder shows the project's coverage, exploration's + use cases included. diff --git a/.agents/skills/kane-cli/references/fair-evaluation.md b/.agents/skills/kane-cli/references/fair-evaluation.md index 2b8f1e6..03908ef 100644 --- a/.agents/skills/kane-cli/references/fair-evaluation.md +++ b/.agents/skills/kane-cli/references/fair-evaluation.md @@ -9,7 +9,7 @@ When you compare kane-cli to anything else (an agent that generates Playwright s kane-cli has two phases with very different costs: - **Authoring** (first run, or after an edit): an AI agent works out each step → spends LLM tokens. One-time per step. -- **Replay** (every later run): executes the recorded action plan from local cache → **zero LLM cost**, runs in seconds. Cost is proportional to how often a test **changes**, not how often it **runs**. +- **Replay** (every later run, for each unchanged step that has a recording): executes the recorded action plan from local cache → **mostly avoids the authoring cost**, runs in seconds. A replay that heals re-authors the failing steps, so it costs more. Cost is proportional to how often a test **changes**, not how often it **runs**. (In a shared project, each folder that pulls a test authors it once, on its first run there: `shared-projects.md` §7a.) The most common evaluation error is pitting kane-cli **authoring** against the other approach's **execution**. That is apples-to-oranges. Compare the *same phase* on both sides: @@ -30,7 +30,7 @@ Compare kane **authoring** to the other approach's **creation effort** (the toke ## Mandatory corrections (real errors seen under time pressure) -- ❌ "kane spends LLM tokens on every test run." → **Wrong.** Tokens are spent only when a step is *authored* (first run or after an edit). Replay of unchanged steps is zero-LLM. State this before any cost claim. +- ❌ "kane spends LLM tokens on every test run." → **Wrong.** The authoring cost is spent when a step is *authored* (first run, after an edit, or when a replay heals). Replay of unchanged steps mostly avoids it. State this before any cost claim. - ❌ "The Playwright scripts already exist, so their creation cost is $0." → **Sunk-cost trap.** The creation effort was still spent (tokens or engineer hours). For an adoption/ROI decision, either count creation on **both** sides, or compare the **forward maintenance trajectory** — never zero out one side's creation while charging the other's. - ❌ "Free CI runs vs. tokens per test." → Phase mismatch (execution vs. authoring). Both execute cheaply; the real differentiators are creation effort and maintenance under UI churn. - ❌ Judging a single isolated run. → The right unit is the **suite over its lifetime** (many runs + UI changes). kane's advantage widens with run count and churn. diff --git a/.agents/skills/kane-cli/references/first-run.md b/.agents/skills/kane-cli/references/first-run.md index 0c4b960..aae2b93 100644 --- a/.agents/skills/kane-cli/references/first-run.md +++ b/.agents/skills/kane-cli/references/first-run.md @@ -2,6 +2,18 @@ # The first run +**In a folder linked to a shared project** (it holds `.testmuai/assurance.json`), or when the person's first request is +for a team's project, the first result is the shared-project journey, never a `kane-cli run`: `kane-cli run` saves to +the profile's project, and when that isn't the team's, its test then clashes with the team's tests +(`shared-projects.md` §9). The first result is +the list of use cases the first extract proposes; show it as the first result, then offer the same three choices. A +teammate joining a project whose documents are all read gets no proposal (`nothing_new`): show the project's +`context list` as the first result instead. In a linked folder, the "where results go" choice only changes where +`kane-cli run` saves; say so. Quick `kane-cli run` experiments go in a separate folder that isn't linked. + +**When the first request starts from requirements** (a PRD, a spec, a description) in a local folder, the first result +is likewise the use cases the extract proposes. + A person's first request should reach its first result with nothing standing in the way. The order is fixed: 1. Ready card (`references/ready-check.md`) @@ -73,6 +85,10 @@ Use the normal card from `references/cards.md`, and on this first run make two o From the second run on, the evidence viewer goes back to an offer. +When the first result is an extract's use cases (above), nothing has run yet, so there is no evidence and no test case: +show the use cases (in a linked folder, also whether the session is saved yet), with no 📁 or 🔗 row. Those rows come with the first test run. +Choice 1 below then reads "Runs open a browser window by default. Keep that, or run without a window?" + ## 4. Three choices, asked once, after the first result Ask these after the first payoff card. They read as tailoring, not as a toll gate, because the person has already seen a result. diff --git a/.agents/skills/kane-cli/references/live-strip.md b/.agents/skills/kane-cli/references/live-strip.md index cbe9dc2..0f5f2ee 100644 --- a/.agents/skills/kane-cli/references/live-strip.md +++ b/.agents/skills/kane-cli/references/live-strip.md @@ -27,8 +27,8 @@ If any of these is missing, do not offer the strip. Nothing else changes: the st - It **wraps the status line the person already has**: their line prints first, unchanged, and the kane line appears under it. - It appears **only while a run is live, and for five minutes after it ends**. The rest of the time the person sees exactly what they had before. -- It appears **only in the Claude Code session that started the run**. Other sessions show nothing, even when they are open in the same project. The reader tells sessions apart by checking that the run descends from the same session process it was started by. A run the person starts by hand in a terminal is not shown. On Windows, where that check is not available yet, every session open in the run's project shows it. -- It reads two things kane-cli writes on its own: a small pointer file for each live run, and that run's event log. It starts no process besides the person's original status line command, makes no network calls, and sends nothing anywhere. +- It appears **only in the Claude Code session that started the run**. Other sessions show nothing, even when they are open in the same project. The reader tells sessions apart by checking that the run descends from the same session process it was started by. A run the person starts by hand in a terminal is not shown. Where that check can't run (on Windows, or wherever the process list can't be read), every session open in the run's project shows it. +- It reads two things kane-cli writes on its own: a small pointer file for each live run, and that run's event log. It starts no process besides the person's original status line command and a local process-list check (`ps`), makes no network calls, and sends nothing anywhere. - **Typed text is never echoed.** A typing step shows as `typing in <field>`. - It refreshes every two seconds. It starts showing a run once kane-cli has created the session, which takes roughly 10 to 30 seconds after launch (the browser has to start first). Until then the person sees their normal status line. While a step is still working, the line shows the last finished action, marked `last:`. diff --git a/.agents/skills/kane-cli/references/mobile.md b/.agents/skills/kane-cli/references/mobile.md index f5f2e18..e790251 100644 --- a/.agents/skills/kane-cli/references/mobile.md +++ b/.agents/skills/kane-cli/references/mobile.md @@ -2,7 +2,7 @@ # Mobile testing (real devices from any machine; emulator/simulator locally on macOS Apple Silicon or on the cloud grid) -Desktop (the browser) is the **default** target and the primary use of kane-cli. Mobile is a **scoped addition**: kane-cli can also drive a native app on a **real Android or iOS device** in the TestMu AI cloud, or on a virtual Android or iOS device — on this machine, or on a LambdaTest HyperExecute macOS host via `testrun run --remote`. Nothing about web runs changes. A run with no mobile flags still drives Chrome exactly as before. +Desktop (the browser) is the **default** target and the primary use of kane-cli. Mobile is a **scoped addition**: kane-cli can also drive a native app on a **real Android or iOS device** in the TestMu AI cloud, or on a virtual Android or iOS device — on this machine, or on a LambdaTest HyperExecute macOS host via `testrun run --remote`. Nothing about web runs changes. A run with no mobile flags still drives Chrome exactly as before, unless a mobile target was saved for `kane-cli run` (`kane-cli config set-target`); a saved test follows its own `target:`. ## Availability (read first) @@ -12,7 +12,7 @@ Desktop (the browser) is the **default** target and the primary use of kane-cli. | **Local** (a simulator/emulator on this machine) | **macOS on Apple Silicon (arm64) only** — not Intel Macs, Linux, or Windows | `run --target …`, `testmd run`, `testrun run` | | **Cloud grid** (a virtual device on a HyperExecute macOS host) | **Any machine** — no Xcode / Android Studio needed. The account needs a HyperExecute plan with macOS runners | `testrun run <paths> --remote … < /dev/null` — see §Remote below | -- **Desktop stays the default.** The `--target` axis is what selects mobile. Leave it off and you get the browser. +- **Desktop stays the default.** The `--target` axis is what selects mobile. Leave it off and `kane-cli run` uses the saved target (`config show`), else the browser; a saved test uses its own `target:`, else the browser. Pass `--target desktop` when you mean the browser. - If the user is not on mac-arm64, local emulator/simulator is not an option — **offer a real device** (`--target android|ios`, works from any machine, plan permitting) or **the grid**: save the objective as a `_test.md` (`target: emulator|simulator` + `app:`) and run it with `kane-cli testrun run <path> --remote --device-name "<grid device>" --os-version <v> < /dev/null`. Do not tell them mobile is unavailable. ## The five targets @@ -60,7 +60,7 @@ Every mobile run needs an app. Provide it one of two ways: kane-cli installs that app on the device and runs the objective against it. `kane-cli apps list --target emulator|simulator|android|ios --agent` lists the account's uploads (NDJSON; the `app_id` field is what `--app`/`app:` take). There is no upload subcommand: any run with a local build — local or `--remote` — uploads it to the account (once per build; a per-machine cache skips a build the account already has) and prints the `APP…` id. Uploads belong to an organisation — `apps list` for the active profile is the authority. -**Not accepted:** a package/bundle id (e.g. `com.example.app`), a `.ipa` on a simulator (it is a device build — use `--target ios`), or a `.app` bundle. There is no default app: a mobile run without a valid build or `APP…` id cannot start. +**Not accepted:** a package/bundle id (e.g. `com.example.app`), a `.ipa` on a simulator (it is a device build — use `--target ios`), or a `.app` bundle. A mobile run without a valid build or `APP…` id (from `--app`, the file's `app:`, or one saved with `config set-app`) cannot start. ## Setup path @@ -143,7 +143,7 @@ Rules that differ from a local run (each violation is a `remote_error` before di | **Payload** | Members must live under the cwd being dispatched, and recordings must not be gitignored (un-ignore with `!output-*/`). Builds are not payload. | `member_outside_payload`, `gitignored_inputs` | | **Auth** | HyperExecute needs a LambdaTest username + access key; an OAuth profile is exchanged automatically, or pass `--username`/`--access-key`. Any `APP…` id must belong to the same organisation. | `no_basic_auth` | -What to present after `testrun_done`/`remote_done`: the suite rollup (per `references/testrun.md`), the device line (name + OS from `remote_device`; for simulators the pool, and each member's own pair), each build the laptop uploaded (`remote_app` path → `app_id`), and the job link (`remote_dispatched.job_url`) for the dashboard's stage logs. If a member comes back `broken` with zero steps, the grid-side run refused before launching — check the app id belongs to this account (`apps list` for the active profile) and the job's scenario log at the job link. +What to present after `testrun_done`/`remote_done`: the suite rollup (per `references/testrun.md`), the device line (name + OS from `remote_device`; for simulators the pool, and each member's own pair), each build the laptop uploaded (`remote_app` path → `app_id`), and the job link (`remote_dispatched.job_url`) for the job's step logs. If a member comes back `broken` with zero steps, the grid-side run refused before launching — check the app id belongs to this account (`apps list` for the active profile) and the job's scenario log at the job link. ## Real devices (`--target android|ios`) diff --git a/.agents/skills/kane-cli/references/objectives-cookbook.md b/.agents/skills/kane-cli/references/objectives-cookbook.md index a0a65de..94a935a 100644 --- a/.agents/skills/kane-cli/references/objectives-cookbook.md +++ b/.agents/skills/kane-cli/references/objectives-cookbook.md @@ -121,6 +121,8 @@ The agent automatically picks the right method based on phrasing. To get the met | **Title** | Browser tab `document.title` | "page title contains …", "title is …" | | **DevTools** | Things not visible on screen — network, console, performance, cookies, localStorage | see §3.2 below | +Under kane-cli's default assertion mode (`dom`), Visual and Textual reads both run DOM-first with a vision fallback; `--assertion-mode visual` keeps Visual reads on vision only (`references/execution-controls.md`). + ### 3.2 DevTools analyze methods Five subdomains. Each one is the right choice when the data you care about lives in the browser's internals rather than on the page. @@ -486,7 +488,7 @@ Positional assertions check where something is on the page: | "Test the checkout flow" | "Go to /cart, click Checkout, fill the address form with {{tester}}, click Pay, assert the order confirmation page loads" | "Test" has no end state — the agent doesn't know when to stop. | | "Add the item" | "Click the 'Add to Cart' button on the first product card" | Vague target — agent may click the wrong element. | | "Tell me the price" | "Store the cart total as 'total'" | Vague verbs don't extract — use "store" / "extract" / "get". | -| Hardcode credentials in the objective | Use `{{username}}` / `{{password}}` from `--variables-file` | Credentials in plain text leak into logs and TMS. | +| Hardcode credentials in the objective | Use `{{username}}` / `{{password}}` from `--variables-file` | Credentials in plain text leak into logs and Test Manager. | | Omit the URL | "Go to https://example.com/login first, then …" | Agent doesn't know where to start. | | Cram 25 operations into one objective | Split at logical boundaries (login, navigate, action, verify) | Long runs drift and stall. | | "Check the page is fast" | "Assert LCP is under 2500ms and CLS is below 0.1" | Use the explicit web-vital metric, not a vague "fast." | diff --git a/.agents/skills/kane-cli/references/parallel.md b/.agents/skills/kane-cli/references/parallel.md index 0866044..fdc63ca 100644 --- a/.agents/skills/kane-cli/references/parallel.md +++ b/.agents/skills/kane-cli/references/parallel.md @@ -4,7 +4,7 @@ For multiple independent browser tasks, decompose and run in parallel using the Agent tool. -> **Saved tests? Use testrun instead.** If the tasks are committed `_test.md` files, do NOT hand-roll parallelism — `kane-cli testrun run --parallel N < /dev/null` gives you isolated Chromes, a pooled scheduler, one rollup, and one evidence pack. Read `references/testrun.md`. This reference is for **ad-hoc `run` objectives** only. +> **Saved tests? Use testrun instead.** If the tasks are committed `_test.md` files, do NOT hand-roll parallelism — `kane-cli testrun run --parallel N < /dev/null` gives you isolated Chromes, a pooled scheduler, one rollup, and one evidence pack. Read `references/testrun.md`. This reference is for **ad-hoc `run` objectives** only. In a folder linked to a shared project, these runs go to the profile's Test Manager project, which may not be the shared one: run them from a folder that isn't linked (`shared-projects.md` §9). ## When to Split diff --git a/.agents/skills/kane-cli/references/parsing.md b/.agents/skills/kane-cli/references/parsing.md index 823702a..2b5a971 100644 --- a/.agents/skills/kane-cli/references/parsing.md +++ b/.agents/skills/kane-cli/references/parsing.md @@ -1,4 +1,4 @@ -<!-- Read this when you need the full NDJSON event schema for kane-cli --agent output, or when SKILL.md §5's summary is not enough. Owns event types (progress / bifurcation / child_agent_start|end / ask_user / error), parsing strategy, run_end terminal-event schema, and ask_user/cancel responses. --> +<!-- Read this when you need the full NDJSON event schema for kane-cli --agent output, or when SKILL.md §5's summary is not enough. Owns event types (progress / bifurcation / ask_user / error / coverage), parsing strategy, run_end terminal-event schema, and ask_user/cancel responses. --> # Parsing --agent Output @@ -21,14 +21,14 @@ The first line on every surface is an opening event: {"type":"stream_start","cli_version":"0.8.17","surface":"run","pid":16664,"v":1,"ts":"2026-09-21T08:47:26.889Z"} ``` -`surface` is `run`, `testmd` or `testrun`. Use `cli_version` to tell whether a newer event or flag is available. `session_dir` may also be present when a session already exists. +`surface` is `run`, `testmd`, `testrun` or `explore` (`references/explore.md`; it ends with `done`). Use `cli_version` to tell whether a newer event or flag is available. `session_dir` may also be present when a session already exists. Rules a parser must follow: - **Ignore unknown fields and unknown event types.** New ones can appear in any release without a `v` bump. - **Never assume the first line is a progress line**, and skip any line that is not JSON. - Step lines on `run` stay **typeless** (below). Do not look for `type: "step"`. -- The documented completion event is always the last line: `run_end` for `run`, `test_md_done` for `testmd run`, `testrun_done` for `testrun run` (then `remote_done` on cloud grid runs). +- The documented completion event is the last line of a run that went ahead: `run_end` for `run`, `test_md_done` for `testmd run`, `testrun_done` for `testrun run` (then `remote_done` on cloud grid runs). A run refused before it starts can end on an `error` with no completion event: read the exit code. **The same stream is also written to disk**, line by line as it happens: `<session_dir>/events.ndjson`, byte for byte what stdout printed. While a run is live, kane-cli keeps a small pointer file at `~/.testmuai/kaneai/sessions/active/<pid>.json` (`pid`, `cwd`, `surface`, `session_dir`, `started`, `cli_version`, `host_agent`) and removes it on exit. You normally need neither: one blocking call hands you the whole stdout. They exist for watchers such as the live strip (`references/live-strip.md`), and the log is where a suite keeps each test's own events (`references/testrun.md`). The log holds exactly what stdout held, so treat it with the same care. @@ -48,6 +48,7 @@ Rules a parser must follow: | `step` | number | Step index. It can run one ahead of the step the person would count (a `bifurcation` takes the first slot), so count completed `done`/`failed` lines for "steps taken" rather than reading the last index | | `status` | string | `"running"` at start; `"done"` or `"failed"` at completion | | `remark` | string | What the agent did or why it failed | +| `child_id` | string, optional | The step belongs to a child flow. Keep children apart when step indices repeat | These are **untyped** — they have no `type` field. Do **not** key on `event.type === 'step_start'` or `'step_end'`; those event types are not emitted. @@ -57,13 +58,14 @@ These are **untyped** — they have no `type` field. Do **not** key on `event.ty |-------|-----------|---------| | `project_folder_auto_defaulted` | resolved project + folder (id, name) | Run-startup gate auto-resolved a project/folder when none was configured (or the cached one was stale/invalid). Fires before any progress event on `run` / `testmd run`. Translate to plain language (see `references/test-manager.md`). | | `bifurcation` | `flows[]`, `count` | Agent split objective into sub-flows | -| `child_agent_start` | `child_id`, `objective`, `parent_step` | Child agent spawned | -| `child_agent_end` | `child_id`, `success`, `steps_taken`, `summary` | Child agent finished | | `ask_user` | `question`, `step_index`, `options?` | Agent needs user input | -| `error` | `message` | Error occurred | +| `error` | `message` | Error occurred. On `testmd run` / `testrun run` in a folder linked to a shared project it also carries a `code`, and `remedy`, `next` or `support_id` when kane-cli has one (`agent-contract.md` §3.7) | | `warning` | `code`, `message`, … | *(0.8.15+)* Before any progress event: `code: "unresolved_variables"` — a `{{name}}` had no value; the run continues. Schema below. | | `test_md_evidence_ingest` | `status: "ok"\|"failed"`, `evidence_id`, `stage?` (failure only) | `testmd run` only: a replay's evidence pack published to the dashboard. Informational. | | `test_md_bundle_sync` | `status: "ok"\|"failed"`, `commit_id`, `bytes?` (success) / `stage?` (failure) | `testmd run` / `testmd sync`: test bundle pushed to the cloud after an authored commit. Informational. | +| `coverage` | `status` (`written`, `none`, `skipped`, `failed`), `pct?`, `reason?`, `pack` | `run` / `testmd run`: one per sealed pack, just before the completion event, which repeats it as `coverage`. `skipped` (nothing to measure here, as in a folder with no assurance work): say nothing. In a linked folder: `agent-contract.md` §3.7 | +| `update_available` | `current`, `latest`, `severity` | `run` (also `generate --agent`), at any point before `run_end`, or not at all: a newer kane-cli exists. Mention it once, after the result | +| `skill_update_available` | `current`, `latest`, `install_cmd` | `run` only, like `update_available`: a newer skill exists. After the result, offer `kane-cli install skill` once (`references/ready-check.md` §4a) | | `testrun_*` family | see `references/testrun.md` | Emitted only by `kane-cli testrun run`; terminal event is `testrun_done`, not `run_end`. | **Note:** The `run` stream has no `run_start` event; startup metadata or errors can precede progress. @@ -88,7 +90,7 @@ Emitted by `run`, `testmd run` and (after `testrun_plan`) `testrun run` when an | `variables[].reason` | `value_missing` — the key exists in `variables[].file` with an empty value · `not_declared` — the key is in no variable file · `not_a_dataset_column` — a `${x}` that is no column of the Test Manager dataset the run was given (`variables[].dataset` names it) | | `variables[].file` | the pool file that holds the empty key (`value_missing` only); `--variables` when it came inline | | `variables[].used_by[]` | `{file, step}` — `file` is `objective` for `kane-cli run`, else the test file (flattened step index) | -| `suggested_file` | where to add a `not_declared` key: `.testmuai/variables/assurance.json` inside an assurance store, `variables.json` otherwise | +| `suggested_file` | where to add a `not_declared` key; use it as given: `.testmuai/variables/assurance.json` in a folder with its own assurance store (`.context/`), `.testmuai/variables/variables.json` otherwise (a linked folder too: `shared-projects.md` §5) | Report it with the run's outcome. If a later step failed on a literal placeholder, point at this. diff --git a/.agents/skills/kane-cli/references/ready-check.md b/.agents/skills/kane-cli/references/ready-check.md index cc5569b..c72becb 100644 --- a/.agents/skills/kane-cli/references/ready-check.md +++ b/.agents/skills/kane-cli/references/ready-check.md @@ -25,7 +25,12 @@ Add a flag only when the request needs it: | A local mobile run | `--mobile emulator` or `--mobile simulator` | `## mobile` (device tooling readiness) | | A cloud grid suite (`--remote`) | `--grid` | `## grid` (grid plugin readiness) | -The script only reads status. It changes nothing, takes about two seconds, and always exits `0`. If the script is missing (an older skill install), run `kane-cli whoami`, `kane-cli balance` and `kane-cli config show` yourself and build the same card. +The script only reads status (its sign-in check may renew a saved sign-in that is about to expire, as any kane-cli command does). It changes nothing else, takes about two seconds, and always exits `0`. + +If you can't run the script (it isn't there in an older skill install, or your host doesn't allow it), run the same +checks one at a time and build the same card: `kane-cli --version`, `kane-cli whoami`, `kane-cli balance`, +`kane-cli config show`, and look for `.testmuai/assurance.json` in the folder (only its `projectName` and `projectId`). +The preferences file is then unknown: treat the session as a first session only if the person says it is. ## 2. What the script prints @@ -34,9 +39,10 @@ Plain text in `## <section>` blocks, always in this order. Command blocks end wi | Section | Content | What you take from it | |---|---|---| | `## version` | kane-cli version, or `missing` | Installed or not. Compare with the minimum version this skill notes for a feature | -| `## whoami` | The sign-in box | `Authenticated` plus `User`, `Environment`. **Ignore `Expires`**: it is a short-lived token that renews itself, never show it | -| `## balance` | `Available credits` and `Total credits` | Credits left, rounded to a whole number | +| `## whoami` | The sign-in box: a header (`✓ Authenticated`, `✗ Not authenticated`, `⚠ Couldn't verify` or `⚠ Not logged in`), then rows such as `Profile`, `Method`, `User`, `Token` | The header and `User`. Never show the `Token` row | +| `## balance` | `Available credits` | Credits left, rounded to a whole number | | `## settings` | Settings as JSON | `project_name`, `folder_name`, `target`, `default_url` | +| `## link` | `project=<name>` and `project_id=<id>` when this folder is linked to a shared project, or `none` | Linked or not; see §3's 🔗 row and `shared-projects.md` | | `## agent-config` | The preferences file, or `none` | See `references/agent-config.md`. `none` or no `onboarding.completed_at` means this is a first session | | `## env` | `ci`, `ssh`, `display`, `os`, `arch`, `node` | Watch-mode default and whether a human is present | | `## chrome` | `found=<path>` and `override=<path>` | Chrome present for local browser runs | @@ -82,9 +88,14 @@ Row rules: - **🚀 Your app** appears only when the `app` section found a port. No row when nothing was found: never show a negative row for an optional finding. Ask for a URL only when the request lacks one. - **🌐 Chrome** appears only for local browser runs. Skip it for mobile and cloud grid requests. +- **🔗 Shared project**: when the preflight's `## link` section names a project, this folder is linked to it + (`shared-projects.md`). Add a row `🔗 **Shared project** | <name> (<id>) · designed tests run and count here`, and + word **Results go to** as "Quick `kane-cli run` checks save to <project> / <folder>; runs of the designed tests here count in <the shared project>". Read `shared-projects.md` before any assurance command. + A `link` of `unreadable` means kane-cli can't read the folder's link file, and a linked folder that also holds + `.context/` is refused too: run `kane-cli context list --mode agent` and relay its fix (`PROJECT_FILE_UNREADABLE`, `BOTH_STORES`) + before any assurance work. - **🗂️ Results go to** comes from `project_name` / `folder_name`. When they are empty, say `kane-cli will pick a default project on this run, and I'll tell you where it landed`. The offer to change it never stops the run. The change flow is in `references/test-manager.md`. - **👀 This run** states the watch mode you are about to use: the saved `preferences.watch`, or the detected default (see `references/first-run.md`). -- Name the environment (for example `stage`) only when it is not production. - For mobile or grid requests add a `📱 **Device tooling**` or `☁️ **Cloud grid**` row from the extra section. ## 4. Problems: which ones stop the run @@ -92,7 +103,8 @@ Row rules: | Problem | How you see it | Stops the run? | The fix the card offers | |---|---|---|---| | kane-cli not installed | `## version` is `missing` | Yes | Offer to run `npm install -g @testmuai/kane-cli` (or Homebrew) | -| Not signed in, or token not valid | `whoami` shows no `Authenticated`, or `exit` is not 0 | Yes | Sign-in flow below | +| Not signed in, or token not valid | `whoami` header is `✗ Not authenticated` or `⚠ Not logged in` (exit 1) | Yes | Sign-in flow below | +| Couldn't verify the sign-in | `whoami` header `⚠ Couldn't verify` (exit 3): TestMu AI didn't answer | Yes, for now | Signing in again won't help: check the connection and run the preflight again in a minute | | No credits left | Available credits is 0 | Yes | Point to https://www.testmuai.com/pricing/ to pick a plan | | Chrome missing | `found=` is empty, local browser run | Yes | Install hint for the platform, or `KANE_CLI_CHROME_PATH` for a custom location | | Low credits | Available credits under 100 | No | One warning line on the card | @@ -102,6 +114,28 @@ Row rules: When a problem stops the run, do not launch. Show the card, offer the fix, and wait. +## 4a. Get or update the skill + +The skill ships as its own package, separate from kane-cli. Most kane-cli commands (not `context`, `explore`, or anything +run with `--mode agent`) may print a +`Skill update available: <installed|not installed> → <version> — run \`kane-cli install skill\`` line on stderr. A +`kane-cli run --agent` stream also carries the notices as events on stdout: `skill_update_available` (`current`, +`latest`, `install_cmd`), so offer `kane-cli install skill`; `update_available` (`current`, `latest`, `severity`) is +kane-cli's own update (`cards.md` §1). + +```bash +kane-cli install skill # the newest skill this kane-cli supports +kane-cli install skill <version> # exactly that version, e.g. one your team's guide names +``` + +- It writes the skill for Claude Code, Codex CLI and Gemini CLI at once (`~/.claude/skills/kane-cli/`, + `~/.agents/skills/kane-cli/`, `~/.gemini/skills/kane-cli/`). Restart the agent to load it. +- With no stdin terminal it prints one JSON line: `{"type":"skill_install_result","status":"installed|already-installed|failed","version":…,"installed_targets":<n>,"error":…}`; exit 1 on `failed`. +- A pinned version installs even when it is older or newer than the default. A later bare `kane-cli install skill` never + downgrades, so it keeps a newer pinned version in place. +- `kane-cli install skill` doesn't write the preferences file `agent-config.md` reads, so the first session still counts + as a first session. + ## 5. Sign-in - **Default:** offer to open the sign-in page, then run `kane-cli login --oauth` yourself with a generous timeout. It opens the browser, waits for the person to finish, and returns. It works without a TTY. diff --git a/.agents/skills/kane-cli/references/setup-and-config.md b/.agents/skills/kane-cli/references/setup-and-config.md index 1800ccc..2e80974 100644 --- a/.agents/skills/kane-cli/references/setup-and-config.md +++ b/.agents/skills/kane-cli/references/setup-and-config.md @@ -4,6 +4,7 @@ ## Install and auth + Before first use, verify installation and auth. > **Local mobile setup (macOS Apple Silicon)** is separate: install Xcode 16+ (iOS) or Android Studio with one `arm64-v8a` AVD (Android), then `kane-cli login` and `kane-cli doctor --target emulator|simulator --install` (installs kane-cli's managed test tooling); `kane-cli doctor --target <kind>` checks readiness. Not needed for `testrun run --remote`, which runs mobile suites on the cloud grid from any machine (see §Remote execution plugin), nor for **real devices** (`--target android|ios`), which need only `kane-cli login` and a KaneAI Max or KaneAI credits + Real Device Plus Automation Cloud plan. Read `references/mobile.md`. @@ -22,7 +23,7 @@ The preflight script covers this along with credits and settings in one call (`r kane-cli whoami ``` -`whoami` prints a box, not JSON, even when piped. Its `Expires` line is a short-lived token that renews itself: never show it to the person. +`whoami` prints a box, not JSON, even when piped. Show the person only its header and the `User` row; never the `Token` row. If this shows "not configured" or errors, run login: @@ -35,7 +36,7 @@ kane-cli login --username <user> --access-key <key> This creates the default profile with basic auth, auto-selects the KaneAI project, and marks setup complete. Credentials come from the user's TestmuAI dashboard (Settings → Keys). Optional flag: -- `--profile <name>` — profile name (default: last selected profile check using `config show`) +- `--profile <name>` — profile name (default: `default`; pass a name to sign in to another profile) ### Login (OAuth) @@ -79,6 +80,10 @@ Variables parameterize objectives with reusable values and secrets. Use `{{key}} 2. `{cwd}/.testmuai/variables/*.json` (local project overrides) 3. `--variables-file <path>` 4. `--variables '{...}'` (inline JSON) +5. a saved test's own `variables:` frontmatter (`testmd run`, `testrun run`) +6. a step's own `variables:` (that step only) + +A Test Manager dataset row's `${col}` values also win over a pool entry of the same name. When a value you changed in a pool file doesn't reach the test, look for a higher layer that sets the same name. **Values (0.8.12+):** a string is a value when non-empty; a number is accepted and loaded as its string; a boolean, object or array is not a value. An empty `value` is a declared-but-unfilled key. @@ -128,9 +133,8 @@ kane-cli feedback --test-id <id> --feedback-type <positive|negative> --details " ├── config.json # Shared auth configuration ├── global-memory.md # Global agent context ├── chrome-profile/ # Default Chrome user profile -├── profiles/ # Stored credentials -│ └── {profile}/{env}/ -│ └── credentials +├── profiles/ # Stored credentials, one folder per profile +│ └── {profile}/ ├── sessions/ # Session history │ └── {session-id}/ │ ├── session.json # Metadata, run list, upload status diff --git a/.agents/skills/kane-cli/references/shared-projects.md b/.agents/skills/kane-cli/references/shared-projects.md new file mode 100644 index 0000000..24eb356 --- /dev/null +++ b/.agents/skills/kane-cli/references/shared-projects.md @@ -0,0 +1,281 @@ +<!-- kane-cli skill reference: shared projects (a folder linked to a team's Test Manager project). Read before any assurance command when the person works with a team, or when a folder holds .testmuai/assurance.json. Requires kane-cli 0.10.0+. --> + +# Shared projects: a team's work in one Test Manager project + +A **shared project** is a Test Manager project that a team's folders are **linked** to. In a linked folder the assurance +commands (`context`, `design`, `maintain`, `cover`) work in the project itself, not in a local `.context/` store. What one +person saves, everyone on the team sees: use-cases, criteria, scenarios, tests, Test Manager cases, coverage. + +Read this before the first assurance command of a session whenever: +- the person mentions a team, a teammate, a Test Manager project, or "our project"; +- the folder holds `.testmuai/assurance.json` (it is linked; the file names its `projectId` and `projectName`); +- a command prints `using project "<name>", shared with your team`, or emits `folder_linked`. + +The commands below are the agent forms. Everything in a linked folder runs from **the folder that holds +`.testmuai/assurance.json`**: a subfolder of a linked folder is not linked. + +## 1. Where the work goes: decide first + +In agent mode, a folder that isn't set up yet goes **local without asking** unless you pass `--project <id>`. So decide +before the first `context ingest`: + +| The person… | Do | +|---|---| +| works in a folder that already has its own `.context/` (and no link) | keep working locally, as before (`assurance.md`); to share that store, see below | +| works with a team, or names a Test Manager project | link this folder to that project (§2–3) | +| works alone, or asks to keep it on this machine | work locally: the journey in `assurance.md` §1 | +| isn't clear | ask once: "Should this work go to your team's Test Manager project, or stay in this folder?" | + +A folder is one or the other, never both: a folder with its own `.context/` refuses `--project` (`FOLDER_CHOSEN`), and a +folder with both refuses every assurance command (`BOTH_STORES`). To share an existing local store with a team through a +Git repository, a bucket or a folder instead, see `context-sync.md`; that is a different way of sharing and doesn't reach +Test Manager. + +## 2. Find or make the project + +```bash +kane-cli projects list --search "<part of the name>" --agent +kane-cli projects create "<name>" --description "<one line>" --agent +``` + +- `projects list` prints one JSON line per project, `{"id":"<project id>","name":"<name>"}`, then a page line + `{"_meta":"page",…,"has_more":true|false}`. There is no `type` and no `done`. `--limit` (1–200, default 10) and + `--offset` page through; `--search` matches any part of the name. +- `projects create` prints `{"id":…,"name":…}`. It does **not** link the folder and doesn't change where `kane-cli run` + saves. Create a project only when the person asks for a new one; otherwise use the team's. +- Errors from `projects` are plain text on stderr (exit 2 for usage or sign-in, 1 for a Test Manager refusal), not events. +- **When the shared path isn't open to the person:** if `projects list` is refused (exit 1: Test Manager refused or + failed the request), or it lists no project the person can work in and they don't want a new one, work locally + (`assurance.md` §1). Tell the person why, in plain words, for example "I couldn't reach your Test Manager projects, + so this work stays in this folder for now". Try the shared path once only; never retry it in a loop. +- A linked folder's own project id is in `.testmuai/assurance.json` (`projectId`). + +### This folder's project id + +A teammate's agent needs it to join (`kane-cli context pull --project <id>` where the pull exists, or +`kane-cli context extract --project <id>`), and so does anything that reads the project, such as a dashboard. Give the +person the name and the id together. Ways to get it: + +| Where | What | +|---|---| +| a linked folder's `.testmuai/assurance.json` | its `projectId` field (and `projectName`); read only those two fields | +| the skill's preflight | its `## link` section: `project=<name>`, `project_id=<id>` | +| the `folder_linked` event | `project_id` and `project`, when a command links the folder | +| `kane-cli projects list --search "<name>" --agent` | `{"id","name"}` for every project the person can open | + +## 3. Link this folder + +```bash +kane-cli context ingest <files> --project <id> --mode agent # link, then add the documents and find their use-cases +kane-cli context extract --project <id> --mode agent # link and read what the project hasn't read yet +``` + +- `context ingest`, `context extract` and, where its help lists it, `context pull` (§7a) take `--project`. The link is + made once; later commands in the folder need no flag. `--project` with the folder's own project is harmless; a + different id refuses `FOLDER_CHOSEN`. +- `{"type":"folder_linked","project_id":…,"project":…}` confirms the link. It can come before a check that still refuses + the run, so read the events after it. Tell the person: "This folder is now linked to <project> (<project id>); what you + save goes to the project." +- If the project refuses its first session (`PERMISSION_REQUIRED`: it needs someone allowed to create test cases), the link + is removed again; ask the person to get someone allowed to create test cases in that project to start the first + session, or to pick another project. After `PROJECT_NO_ACCESS` the folder **stays linked**, and every + assurance command refuses the same way until the person gets access or removes `.testmuai/assurance.json` (their call). + A `context pull --project` that is refused at the start, or can't read the project, removes the link it made: check + the folder again after one. +- A person at a terminal is asked instead: "where should this folder's work go?", then a project picker (Tab makes a new + project). Agents never see these questions. +- There is no unlink command. Linking a folder to another project means removing `.testmuai/assurance.json` first; that + is the person's call, never yours. + +## 4. The journey in a shared project + +| Step | Command | What to watch | +|---|---|---| +| 1. Add the documents and find their use-cases | `kane-cli context ingest <file> --project <id> --mode agent` (the first time), later `kane-cli context ingest <file> --mode agent` | one document per run; it ends `session_complete` with two `next` commands, save and discard (none when it added nothing) | +| 2. Save the session | `kane-cli context sessions save <sid> --yes --mode agent` | **nothing is in the project until this runs.** `session_save` `phase: "saved"` confirms it | +| 3. The use-case checkpoint | `kane-cli context list --mode agent` to show them; then ask once (below) | use-cases an agent run saves are `proposed`, not approved | +| 4. Design the tests | `kane-cli design tests --use-case <uc> --mode agent --allow-unreviewed` | only after the person's yes; ends `session_complete`. To cap the scenarios (and the cost), offer `--max <n>`; add it only on the person's word | +| 5. Save the design | `kane-cli context sessions save <sid> --yes --mode agent` | writes the test files here and makes one Test Manager case per scenario; `test_files.written[]` lists the files. Before going on, read `test_files.refused[]` and `test_files.error`, and `test_manager.error` and `waiting`: a save can succeed while a file or a case is still missing. Files listed in `test_files.replaced[]` stay but won't run any more: name only the `written[]` files on `testrun run`, and offer to move the replaced ones out | +| 6. Fill the variables, and the start pages | the file in `test_files.variables_needed.file` (usually `.testmuai/variables/assurance.json`); `test_files.no_start_page[]` names tests with no start page | SKILL.md §3, before any run. Use only values the person gives you, never invented ones; write them into the file (a script is fine) without printing them to the terminal or a log. Ask the person for the site address of a test with no start page | +| 7. Run the tests | `kane-cli testrun run <the files from step 5> --agent < /dev/null` | the first run of each test writes its steps: minutes, and about 30–85 credits per test; replays are cheap (`testrun.md` §Designed tests in a shared project) | +| 8. Coverage | `kane-cli cover --mode agent` | the project's figure; a new run can take up to 5 minutes to count | + + +**The use-case checkpoint.** A linked folder has no command that approves use-cases: +`context review` is refused there, and use-cases an agent run saves stay `proposed`. So `design tests` refuses +`UC_UNREVIEWED` until it gets `--allow-unreviewed`. Show the person the use-cases you found (titles and what they cover, +from `context list --mode agent`), then ask **once for the batch**: "I found these N use-cases in <document>. Shall I design tests for them? Nobody has +approved them yet, so tell me any to leave out or change first." Pass `--allow-unreviewed` only after a yes, and only for the use-cases they keep. A person at a terminal approves them by finishing the +session there. + +**What things cost.** These are approximate figures from real runs of small documents; the person's billing page is +the authority. On the assurance commands (`context ingest`, `context extract`, `design tests` …) the stream's +`usage.total_credits` gives the real figure. A local `testrun run` stream carries no credits total (not on `testrun_done`, +a member's end event, `testrun_summary` or `coverage`), so run `kane-cli balance` before and after the run and give the +difference: + +| Step | Credits (approximate) | +|---|---| +| `projects list`, `projects create`, `context list`, `context sessions`, `cover`, a save, `explore list`, `explore context status` | 0 | +| linking (`context extract --project` when every document is already read) and `context pull` | 0 | +| reading one document (`context ingest`, or `context extract` with something unread) | about 5 | +| designing one use-case's tests (`design tests`) | about 10–15 | +| a designed test's first run in a folder (it writes its steps; each input it checks is its own browser run) | about 30–85 per test | +| a replay | a few; a replay that has to heal, or whose steps changed since the recording, writes those steps again and can cost up to a first run | + +Tell the person the likely cost before a step that costs credits, let them choose, and give the total after. Never add +a flag that changes cost or behaviour (for example `--no-adaptive-heal`, `--max`, `--force`) on your own: offer it, and +use it only when the person asks. + +**One session at a time.** A live session holds its source or its use-case; another person's start on the same one is +refused `PROJECT_BUSY`, saying whether the session in the way is the person's own (`held_by.mine`) or a teammate's. Save or discard each session before the next one, and tell the person when a +teammate's session is in the way. `kane-cli context sessions --mode agent` lists the project's open sessions (one JSON +array; `owner` says whose). + +## 5. Where everything lives + +| What | Where | How to see it | +|---|---|---| +| sources, use-cases, criteria, scenarios, tests | in the shared project | `kane-cli context list --mode agent` lists sources, use-cases, scenarios and tests (one plain JSON row each: `id`, `label`, `title`, `trust`, `fresh`; no `done`); criteria aren't listed there. `trust: "derived"` is a proposal nobody has approved yet, `"trusted"` is approved: say *proposed* or *approved* to the person, never the field's value | +| test files (`*_test.md`) | `.testmuai/tests/` of **the folder where the session was saved**, and nowhere else | `session_save.test_files.written[]` | +| Test Manager cases | one per scenario, in the shared project, made or updated when a design is saved | `session_save.test_manager`: counts `created`, `updated`, `already_there`, `waiting`, and `next_run[]` (the scenarios whose case updates on their next run) | +| a test's runs | on its scenario's Test Manager case | Test Manager; the run's `testrun_summary` | +| coverage | worked out by the shared project from the counted runs | `kane-cli cover --mode agent` | +| variables | names: in the saved tests; empty stubs: this folder's `.testmuai/variables/assurance.json`; values: whatever the person writes there | `test_files.variables_needed` | +| a run's recording | `.testmuai/tests/output-<test>/` beside the test file, after a first run that finished | the next run replays it cheaply; a first run that stopped early leaves it incomplete, and the next run writes the steps again | + +Two rules follow: +- **Keep designed test files directly in `.testmuai/tests/`.** A designed test moved into a subfolder runs as an ordinary + test, not on its Test Manager case, and doesn't count for coverage. +- **Variable values reach the whole organisation.** Every run sends the non-secret values it loads to Test Manager: the + global and folder variable files, `--variables` and `--variables-file`, and the test's own `variables:`. Test Manager + keeps one value per name for the organisation, so a teammate's run with a different value replaces it. Never fill a + value just to silence a warning. Any `*.json` in `.testmuai/variables/` counts; prefer the file `variables_needed` names. + +## 6. A teammate joins + +From the teammate's own empty folder. Where `kane-cli context pull --help` lists `--project` (§7), join with the pull: +it links the folder, reads no documents, costs nothing and brings the project's tests (§7a): + +```bash +kane-cli context pull --project <id> --mode agent +``` + +Otherwise join with the extract: + +```bash +kane-cli context extract --project <id> --mode agent +``` + +It links the folder, then: +- every document already read: `nothing_new` (`reason: "every_source_read"`), exit 0, no credits. That needs the project + to answer within a few seconds; when it doesn't, the extract goes on and opens a session: save or discard it, as below; +- a document nobody has read yet: it reads it (credits, see §4) and ends `session_complete`. Save that session (§4 step 2), + or discard it on the person's word; while it is open it holds that document, and teammates get `PROJECT_BUSY`; +- another person's session already holds the unread document: `nothing_new` (`reason: "sources_in_a_session"`) with + `open_sessions[]`; nothing to do here. + +After the extract, `context list` and `cover` show the project. **No test files arrive with the extract's link**, so +`testrun run` has nothing to run yet: see §7. + +## 7. Get the team's tests into this folder + +Test files are written only in the folder where a design was saved. Exploration saves its tests into the project, but its +files never reach anyone's folder. + +**First check whether this kane-cli can pull them:** + +```bash +kane-cli context pull --help +``` + +If its options include `--project` and `--use-case`, use §7a. If not (kane-cli 0.10.0), it can't: `context pull` in a +linked folder is refused (`REMOTE_UNSUPPORTED`), and `--project` is an unknown option. Use §7b. + +### 7a. Pull them (when `context pull --help` lists `--project`) + +After exploration finishes, or after a teammate saves a design: + +```bash +kane-cli context pull --mode agent # in a linked folder: write the project's tests here +kane-cli context pull --project <id> --mode agent # in a folder that isn't set up yet: join (reads no documents), then pull +kane-cli context pull --use-case <uc> --mode agent # only that use-case's tests, with the tests they use +kane-cli context pull --overwrite --mode agent # replace test files changed here with the project's +``` + +- The pull's own events and refusals carry `verb: "sync"`; a refusal of the pull's flags (`PULL_LOCATION_LINKED`, `PULL_FLAGS_MIXED`, `PULL_FLAGS_LINKED_ONLY`) or at the folder's entry (sign-in, access, the link) carries `verb: "pull"`. Dispatch on `type`, as always. +- One `pull_test` event per test: `{test, title, file, outcome, first_run?, imported?, reason?}`, where `outcome` is + `new`, `updated`, `unchanged`, `kept` (changed here, left alone), `replaced` (with `--overwrite`) or `not_written`. +- `first_run: true` means the test hasn't run in this folder: its first run writes its steps in a real browser, which + takes a few minutes and costs credits (§4); later runs replay. Tell the person the count and the cost first. +- `variables_needed {file, variables[], no_start_page[]}` names the variables to fill; the pull writes no stubs. If the + file it names doesn't exist yet, it is a JSON object with one entry per name: + `{"<name>": {"value": "<value>", "secret": true|false}}`. +- A variable the pull can't classify is marked `secret: true` to be safe. Ask the person whether it really is a secret + (a message to type usually isn't). A secret's value is never asked for or shown in chat: the person writes it into the + file. A value the person has already given you in their own words isn't a secret to them: write it there yourself, + without printing it, and go on (§4 step 6). +- `done.next` names `kane-cli testrun run`. A project with no tests yet (or a `--use-case` with none) sends no + `pull_test` events and ends `done` exit 0 with `next` = `kane-cli cover --mode agent`. Exit 1 when any test wasn't + written or the project couldn't be read (`PULL_FAILED`); + 2 for refusals. + +| Code | When | Do | +|---|---|---| +| — | `context pull` with no flags in a folder that isn't linked: it is the location pull, as before. In a new folder its message is "no context store here" | to join the team's project and bring its tests, run `kane-cli context pull --project <id> --mode agent` (`kane-cli projects list --agent` for the id) | +| `PULL_LOCATION_LINKED` | a location flag (a location name, `--rebase`, `--yes`, `--answer`) in a linked folder | `kane-cli context pull --mode agent` | +| `PULL_FLAGS_MIXED` | project flags mixed with location flags | use one set | +| `PULL_FLAGS_LINKED_ONLY` | `--use-case` or `--overwrite` without `--project` in a folder that isn't linked (a new one, or one with its own local store) | link it first with `--project <id>`, or drop those flags | +| `USE_CASE_UNKNOWN` | the project has no such use-case | `kane-cli context list --mode agent` for the ids | +| `PULL_FAILED` (exit 1) | the project's tests couldn't be read; nothing was written | try once more; then give the person the command, the code and the message (agent mode shows no support id here) | + +### 7b. Until then: copy only what the tests need + +Ask the person to get, from the teammate who saved the design, **only**: +- `.testmuai/assurance.json` (the link), when this folder isn't linked yet; +- the test files `.testmuai/tests/*_test.md`; +- for each test, its `output-<test>/` folder beside it, if the teammate ran it (so the first run here is a replay). + +**Never** `.testmuai/variables/` (it can hold secret values) or `.testmuai/evidence/`, and never unzip over this folder's +own `.testmuai/`: it would replace this folder's variables and its link. The receiving folder must have no `.context/` +and no link to another project. Then fill this folder's own variables (§4 step 6) and run the files by path. +Exploration's tests have no folder to copy from: they can be seen (`context list`; Test Manager) but not run until the +pull is available. + +## 8. What changes in a linked folder + +Non-assurance commands (`run`, `testmd`, `testrun`, `projects`, `config`, `evidence` …) always run. These assurance +commands are refused with `REMOTE_UNSUPPORTED` (exit 2): + +| Refused | Use instead | +|---|---| +| `context review`, `context explain` | `kane-cli context list --mode agent`; the checkpoint in §4 | +| `cover gaps` | `kane-cli cover --mode agent` | +| `context sessions show`, `context sessions clean` | `kane-cli context sessions --mode agent`; `… sessions discard <sid> --yes --mode agent` | +| `context sync`, `push`, `clone` (and `context pull` where its help has no `--project`: §7b) | the shared project is the team's copy; `kane-cli context pull --mode agent` brings its tests (§7a). A location pull is refused `PULL_LOCATION_LINKED` | +| `context view`, `fsck`, `rebuild`, `retire`, `revert`, `name`, `design explain` | none in a linked folder | +| `--mode ci` on `extract`, `design tests` and `maintain reconcile` | `--mode agent` | +| `context ingest --plan`, `--mode override` | `kane-cli context ingest <file> --mode agent` | +| `context list --type` other than `source`, `usecase`, `scenario`, `test` | one of those four | +| `context list --all` | `kane-cli context list --mode agent` | +| `maintain reconcile --plan`, `--apply`, a URL in `--from` | `kane-cli maintain reconcile --from <file> --source-id <id> --mode agent`; for an address, `kane-cli context ingest <address> --mode agent` | +| `testrun run --from-context <ids>` | name the files: the refusal's `next` does | + +In a linked folder `extract --plan` and `--with-source` are ignored, and an extract that read two or more documents and +then paused can't be resumed (its `next` adds them again or discards). Give each `context ingest` one document. + +## 9. `kane-cli run` in a linked folder, and "two projects" + +`kane-cli run` saves its test to the **profile's** Test Manager project (`config project`), not the folder's, unless the +profile's project is this folder's project. With `--name` it also writes the test into this folder's `.testmuai/tests/`. +A `testrun run` that selects tests from two projects then refuses `PROJECT_MISMATCH`. + +- In a linked folder, never use `kane-cli run` for a first result: the first result is the journey in §4. +- Quick `run` experiments go in a separate folder that isn't linked. +- On `PROJECT_MISMATCH`: read `projects[]` (`project_id`, `name`, `this_folder`, `tests`). Its `next` runs one project's + tests, by path or with a `--match`: this folder's project's when it has any, else another's. Run it, and tell the person + which tests were left out and why. +- **The lasting fix**, with the person's word: move the tests listed under the project where `this_folder` is `false`, + each with its `output-<test>/` folder, out of `.testmuai/tests/` into a folder that isn't linked. Until then, name the + files on every `testrun run`. +- Recovery for every shared-project refusal: `agent-contract.md`. diff --git a/.agents/skills/kane-cli/references/test-manager.md b/.agents/skills/kane-cli/references/test-manager.md index 65e9030..f00a2d7 100644 --- a/.agents/skills/kane-cli/references/test-manager.md +++ b/.agents/skills/kane-cli/references/test-manager.md @@ -2,6 +2,11 @@ # Test Manager Reference — Agent Surface +> **Two different "projects".** This page is about the **profile's** project: where `kane-cli run`, `testmd` and +> `generate` save. An assurance folder **linked** to a team's shared project uses the link's project instead, and +> `projects create` or `config project` never change a folder's link (`shared-projects.md` §2–3). In a linked folder, +> `kane-cli run` still saves to the profile's project (`shared-projects.md` §9). + Every kane-cli session uploads to a TestmuAI **Test Manager (TMS)** test case. The session lands inside a **project** (required) and optionally a **folder** inside that project. This page is the agent-facing surface for everything that touches TMS: where the run ends up, how to browse and create projects/folders programmatically, and the event the run-startup gate emits when nothing is configured. Field names below are for parsing only — translate to plain language for the user, per the §5 rule in `SKILL.md`. @@ -10,13 +15,13 @@ Field names below are for parsing only — translate to plain language for the u ## 1. Where a run lands -After a successful upload, the terminal `run_end` event (`testmd_done` for `testmd run`) carries the dashboard link: +After a successful upload, the terminal `run_end` event of `kane-cli run` carries the dashboard link: ```json {"type":"run_end","status":"passed", "test_url":"https://test-manager.lambdatest.com/projects/<id>/test-cases/<id>", ...} ``` -For `testmd run`, the `test_md_summary` / `test_md_done` events also carry a `share_url` — a 7-day, no-login link suitable for CI artifacts. +For `testmd run`, the `test_md_summary` / `test_md_done` events carry no `test_url`; they carry a `share_url` when the upload made one — a 7-day, no-login link suitable for CI artifacts. Surface `test_url` (and `share_url` when present) as a "View in Test Manager" line per the §1.4 results table in `SKILL.md`. Never paste raw URLs into the middle of a summary — they belong in the results table. @@ -45,7 +50,7 @@ kane-cli folders list --project <id> [--search <q>] [--limit <n>] [--offset <n> | Flag | Purpose | |---|---| | `--search <q>` | Filter by name (substring match). | -| `--limit <n>` | Page size. Default is small (~10). Very large values are capped. | +| `--limit <n>` | Page size: a whole number from 1 to 200 (default 10). A value outside that is refused (exit 2). | | `--offset <n>` | Skip the first N rows. | | `--agent` | Force NDJSON. Auto-on when stdout is piped/redirected, but pass it explicitly anyway. | @@ -165,6 +170,5 @@ The results project and folder belong to kane-cli, not to the agent config. A ch | Code | Meaning | |---|---| | 0 | OK | -| 2 | Auth/setup error (missing or invalid credentials) or unknown subcommand | - -Other codes are run-specific (`run` / `testmd run`) and don't apply to `projects` / `folders`. +| 1 | Test Manager refused or failed the request (the message is on stderr) | +| 2 | Auth/setup error (missing or invalid credentials), a bad flag value, or unknown subcommand | diff --git a/.agents/skills/kane-cli/references/testmd.md b/.agents/skills/kane-cli/references/testmd.md index 62d35b9..919921b 100644 --- a/.agents/skills/kane-cli/references/testmd.md +++ b/.agents/skills/kane-cli/references/testmd.md @@ -2,7 +2,7 @@ # Saving & Replaying Tests with testmd -The §3 `run` command is the **primary** mode — one-shot, ephemeral. `testmd` is the **secondary** mode: tests live as `_test.md` files on disk, each step is cached on the first run, and every later run **replays from cache** with no LLM cost. +The §3 `run` command is the **primary** mode — one-shot, ephemeral. `testmd` is the **secondary** mode: tests live as `_test.md` files on disk, each step is cached on the first run, and every later run **replays from cache**, which mostly avoids the authoring cost. Use `testmd` whenever the user wants the test to persist. The decision is binary — once a test exists as a file, every later invocation is `testmd run`, never `run`. @@ -22,7 +22,7 @@ If unclear, ask: "Do you want me to save this test so you can re-run it later?" ## Quick start -Write the file (any path; filename must end in `_test.md`): +Write the file (any path; filename must end in `_test.md`). In a folder linked to a shared project, read `shared-projects.md` §9 first. ```markdown --- @@ -75,7 +75,7 @@ Click submit and verify the confirmation banner. | `mode` | root | `action` (halts on auth walls) or `testing` (default — pushes through so negative-test assertions can fire) | | `url` | root | Start URL for the first step (bare domains get `https://`). Overridden by `--url`; falls back to config `default_url`. | | `tags` | root | Labels for batch selection: YAML list, `[a, b]`, or bare `a, b`. Trimmed, lowercased, deduped. Selected with `testrun --tags` (`references/testrun.md`); shown by `testmd list`. Empty → parse error; per-step → parse error. | -| `max_steps` | root + step | Max agent reasoning steps. Engine fallback `30`; CLI default `50` (step config overrides). | +| `max_steps` | root + step | Max agent reasoning steps. Default `50`; a value in the file wins over `--max-steps`. | | `timeout` | root + step | Hard kill per step in seconds. | | `headless` | root | No browser window. | | `variables` | root + step | `{{name}}` params, same shape as §3, with `secret: true` for credentials. Counts as a value for the pre-run check (0.8.12+), even when `value` is empty. A `{{name}}` that is declared nowhere and has no pool value gets a warning before the run starts, and the run types it as written (SKILL.md §3) | @@ -86,7 +86,7 @@ Files ending in `_test.md` are tests (valid entry points). Any other `.md` is a ## The replay & cascade rule (CRITICAL) -On the **first** run of a test, the agent authors each step and saves a recording. On **every later run**, each step replays from its recording — no agent, no LLM cost, much faster. +On the **first** run of a test, the agent authors each step and saves a recording. On **every later run**, each unchanged step that has a recording replays from it — no authoring agent, so it mostly avoids the authoring cost and runs much faster. A step whose text changed, whose recording is missing, or that failed when recorded is authored again. A replay that heals (below) re-authors the failing steps, so it costs more. An authorable step replays only if **all** of these hold (replay-only marked steps always require their recording): - A recording for that step exists, @@ -112,10 +112,10 @@ Extract a repeating flow (login, setup, cookie banner dismissal) into a helper f ``` Rules: -- Helper filename **must not** end in `_test.md`. +- A step can import a helper (`.md`), another test (`_test.md`) or a module (`_module.md`). - Path resolves relative to the **importing file**, not the shell's cwd. - The step body must be exactly `@import <path>` — no mixed prose, no extra lines. -- The step's `yaml` block may contain **only** `optional`. Other keys are rejected. +- The step's `yaml` block may contain **only** `optional` and `variables` (values for `{{names}}` the imported file doesn't declare itself). Other keys are rejected. - `optional: true` on `@import` is allowed only at the root file, not on a nested import. Variables and context propagate into helpers. Chrome / `mode` / auth do not (root-only). @@ -129,8 +129,8 @@ Editing a helper re-authors that step in **every test that imports it**, plus ev | `kane-cli testmd run <path> --agent [flags]` | Run a test | | `kane-cli testmd list` | List `*_test.md` files under cwd (NDJSON when non-TTY; records include `tags`) | | `kane-cli testmd status <path>` | Test Manager identity + local-sync state | -| `kane-cli testmd export <path> [--language python\|javascript]` | Regenerate code export from existing recordings (no browser launch) | -| `kane-cli testmd delete <path>` | Local-only delete: removes source + `output-<stem>/`. Does NOT delete from Test Manager. | +| `kane-cli testmd export <path> [--language python\|javascript]` | Write code from existing recordings (no browser launch). An existing export is reused; `--force` writes it again | +| `kane-cli testmd delete <path> --yes` | Local-only delete, on the person's word: removes source + `output-<stem>/`. Does NOT delete from Test Manager. Without `--yes` it refuses (exit 2) | | `kane-cli testmd sync <path>` | Re-push the test bundle (test + imports + outputs) to the cloud. Rarely needed — it happens automatically after every authored commit; use only to recover a failed auto-sync. | **Running several tests at once?** Use `kane-cli testrun run` — one execution, one pack, isolated parallel Chromes. Read `references/testrun.md`. @@ -142,14 +142,14 @@ Editing a helper re-authors that step in **every test that imports it**, plus ev | `--url <url>` | frontmatter `url:` / config `default_url` | Start URL for the first step. Overrides the `url:` frontmatter key and config `default_url`. | | `--allow-missing-url` | off | Non-TTY only: start from the browser's current page instead of failing when the first step has no start URL. | | `--name <name>` | none | Persist the run under this name. Regex `[a-zA-Z0-9_-]+`. | -| `--on-lock-conflict <readonly\|fail\|wait>` | none | Behavior when another user holds the test's edit lock. `readonly` = replay-only / no upload, `fail` = exit 2, `wait` = block until released | +| `--on-lock-conflict <readonly\|fail\|wait>` | `readonly` | Behavior when another user holds the test's edit lock. `readonly` = runs, but saves no new version of the test (any re-authoring stays local), `fail` = exit 2, `wait` = retry every 30 s until released, with no time limit | | `--no-adaptive-heal` | healing enabled | Disable adaptive healing after replay failure | | `--author` | off | Force authorable steps to re-author; replay-only steps still replay | | `--bug-detection <off\|stop\|continue>` | config (`off`) | Flag suspected product bugs while **authoring** (`stop` halts on a confirmed bug; `continue` records it). Replay failures always investigate regardless. | Common flags include `--agent`, `--headless`, `--max-steps`, `--timeout`, and `--variables`. Command-specific flags differ; check `kane-cli testmd run --help`. -Flag wins over frontmatter for everything **except** `variables` — the file owns variables; you can add new keys via flags but cannot override file-defined ones. +For step settings such as `max_steps` and `timeout`, a value in the file (frontmatter or a step's `yaml`) wins over the flag; the flag fills only what the file leaves out. Browser flags (`--headless`, `--cdp-endpoint`, `--ws-endpoint`) and `--url` win over the file. The file owns `variables`: you can add new keys via flags but cannot override file-defined ones. ## Output: `output-<stem>/` and `Result.md` @@ -165,7 +165,7 @@ output-amazon/ **`output-<stem>/` is commit-safe and should be committed to git.** That's how teammates and CI replay the same recordings. -For tests using `@import`, helper recordings land next to the helper file in `helper-output-<helper>-<root>-<step>/` directories. Also commit-safe. +For tests using `@import`, each imported file's recordings land next to it in its own `output-<stem>/`. Also commit-safe. Older `helper-output-<helper>-<root>-<step>/` directories are still read, and removed once that file's own recordings are written. **`Result.md`** opens in any Markdown viewer. It contains: - Frontmatter — `status`, `started`, `duration_s`, `session_id` @@ -186,7 +186,7 @@ If the user runs an ad-hoc objective with §3 `run` and decides to keep it: kane-cli run "Search for noise-cancelling headphones on amazon.com" --name amazon-search ``` -On exit, kane-cli writes `<cwd>/.testmuai/tests/amazon-search_test.md`. Move that file into the user's repo and re-run it with `testmd run`. Without `--name`, an ad-hoc `run` is ephemeral and nothing is written. +On exit, kane-cli writes `<cwd>/.testmuai/tests/amazon-search_test.md` and its recordings in `output-amazon-search/` beside it. Move the file together with that folder into the user's repo and re-run it with `testmd run`; without the folder, the next run authors again. Without `--name`, an ad-hoc `run` is ephemeral and nothing is written. In a folder linked to a shared project, don't name runs: `shared-projects.md` §9. ## CI invocation @@ -199,7 +199,7 @@ kane-cli testmd run ./tests/checkout_test.md \ - `--agent` — NDJSON to stdout (auto-enabled when stdin is not a TTY; pass explicitly anyway). - `--headless` — no window. -- `--on-lock-conflict wait` — block instead of failing if a teammate is editing the same test. +- `--on-lock-conflict wait` — wait for a teammate's edit to finish instead of running without saving a new version. - Adaptive healing recovers replay failures by default; `--no-adaptive-heal` opts out. In non-TTY runs, interactive `ask_user` prompts are disabled — a step that would wait for input fails cleanly instead of hanging forever. Write CI test steps that don't depend on mid-run prompts. (Likewise, supply a start URL via the objective, `url:` frontmatter, or `--url`; a non-TTY run with no resolvable start URL fails unless you pass `--allow-missing-url`.) @@ -210,8 +210,10 @@ Exit codes: |------|---------| | 0 | ✅ Passed | | 1 | ❌ Failed | -| 2 | ⚠️ Error (auth, setup, infra) — for `testmd`, also includes parse errors and `--on-lock-conflict fail` | -| 3 | ⏱️ Timeout or cancelled — for `testmd`, also includes `--on-lock-conflict wait` timeout | +| 2 | ⚠️ Error (auth, setup, infra) — for `testmd`, also includes parse errors, `--on-lock-conflict fail`, and `wait` when the test's saved version moved on meanwhile | +| 3 | ⏱️ Timeout or cancelled (`run`). A `testmd run` that times out or is cancelled ends `failed`, exit 1 | + +In a folder linked to a shared project, a designed test's run can also stop before it starts, or exit 1 after passing: `agent-contract.md` §3.7. ## Parse errors (when writing a `_test.md`) @@ -222,8 +224,7 @@ Parse errors abort **before** any browser launch with exit `2`. Common ones and | `frontmatter is missing closing '---'` | Add the trailing `---` | | `invalid YAML in frontmatter` | Re-validate the YAML block | | `step body must be exactly one of prose / @import` | Split into two steps | -| `step config on @import may only contain 'optional'` | Remove other keys from the yaml block | -| `cannot @import a test file` | Imports may only reference helpers (not ending in `_test.md`) | +| `step config on @import may only contain 'optional' / 'variables'` | Remove other keys from the yaml block | | `cyclic reference` | Restructure helpers to break the loop | | `chrome config is global-only` | Move Chrome key to root frontmatter | | `'<key>' is run-level and cannot be set per-step` | Move `mode` / `tags` / `on_lock_conflict` to root frontmatter | @@ -237,7 +238,7 @@ When the user reports a parse error, fix the file before retrying — don't loop Adaptive healing is enabled by default: after a replay failure the runner tries up to three shrinking replay windows, then full re-authoring of authorable steps. `--no-adaptive-heal` disables this recovery. Retired `--retry` and `--retry-count` are accepted only to print a notice; they do not enable healing or change the fixed budget. -Headings marked `@db`, `@api`, `@js`, `@smartui`, `@network_query`, or `@network_assertion` are replay-only. They still replay under `--author`, downstream divergence and adaptive healing. Missing recordings cannot be re-authored through these markers: restore the recording or recreate it through the originating workflow. Do not delete their tapes to force authoring; adding a marker does not create an operation. +Headings marked `@db`, `@api`, `@js`, `@smartui`, `@network_query`, `@network_assertion`, or (on a mobile device) `@gps`, `@accessibility`, `@media_injection`, `@network_throttle`, `@deeplink` are replay-only. They still replay under `--author`, downstream divergence and adaptive healing. Missing recordings cannot be re-authored through these markers: restore the recording or recreate it through the originating workflow. Do not delete their tapes to force authoring; adding a marker does not create an operation. Structured control flow uses balanced heading markers: `@if`, `@elif`, `@else`, `@end-if`, `@while`, `@end-while`. An `@else` must be last in its conditional; end markers must match the opened block type. These are distinct from natural-language conditionals. Markers are excluded from the step body hash. Only one replay-only kind is allowed per step, and an import cannot also be marked replay-only. @@ -255,6 +256,7 @@ This stream is **not** the one-shot `run` stream. Every line is typed, and the f | `test_md_step_end` | `step_index`, `status`, `duration_s`, `failed_sub_step_index` | The step finished. `status` is `passed`, `failed` or `skipped` | | `test_md_evidence_ingest`, `test_md_bundle_sync` | `status` | Informational, before the summary | | `test_md_summary` | `overall_status`, `duration_s`, `steps: {total, passed, failed, skipped, replay_decisions, author_decisions}` | The numbers for the card. `replay_decisions` is how many steps replayed, `author_decisions` how many were authored | -| `test_md_done` | `overall_status`, `duration_s`, `session_id`, `share_url?` | Completion. Always the last line. `share_url` is absent on a pure replay | +| `coverage` | `status` (`written`, `none`, `skipped`, `failed`), `pct?`, `pack` | One per sealed pack, just before `test_md_done` (which repeats it as `coverage`). `skipped`: say nothing (`references/parsing.md`) | +| `test_md_done` | `overall_status`, `duration_s`, `session_id`, `share_url?`, `coverage?` | Completion. The last line of a run that started (a linked folder's stop before the start: `agent-contract.md` §3.7). `share_url` is absent on a pure replay | Most lines are inner `step_event`s (screenshots, reasoning, actions). Skip them unless you are diagnosing a failure: for the card you need only the step starts and ends, the summary and the completion event. On a failure, the failing step is the `test_md_step_end` with `status: "failed"`, its title comes from the matching `test_md_step_start`, and the last inner `step_end` or `step_event` before it says what went wrong. diff --git a/.agents/skills/kane-cli/references/testrun.md b/.agents/skills/kane-cli/references/testrun.md index e3d9f98..025e62a 100644 --- a/.agents/skills/kane-cli/references/testrun.md +++ b/.agents/skills/kane-cli/references/testrun.md @@ -26,7 +26,7 @@ kane-cli testrun run [paths...] [flags] < /dev/null # NDJSON when stdin is not | `--match <regex>` | Filter candidates by project-relative path regex | The path is as the OS writes it: `tests/app/` on macOS and Linux, `tests\app\` on Windows. Quote the regex with double quotes in cmd.exe; single quotes are literal there. | | `--tags <list>` | ANY-match on frontmatter `tags:` (repeatable or comma-separated, case-insensitive) | — | | `--parallel <n>` | Worker count; each desktop worker gets an isolated Chrome with a fresh temp profile | `1` | -| `--on-failure <mode>` | `continue` (run everything) \| `fail-fast` (stop dispatching new members after a failure) | `continue` | +| `--on-failure <mode>` | `continue` (run everything) \| `fail-fast` (stop dispatching new replays after a failure; tests with no recording yet are still all written, in the pass after the replays) | `continue` | | `--name <label>` | Run title in the dashboard | derived | | `--dry-run` | Print the plan (members + preflight failures) and exit; runs nothing | off | | `--no-adaptive-heal` | Disable default adaptive healing after replay failure | healing enabled | @@ -44,9 +44,63 @@ All members must share one org + project. *(0.8.4+)* Members need **not** be aut | Reason | Plain-language meaning | What to tell the user | |---|---|---| | `org_mismatch` | Different organisation than the other tests | "Check `kane-cli testmd status <path>` — it belongs to another org" | -| `project_mismatch` | Different project than the other tests | "Run it separately or per-project" | +| `project_mismatch` | Different project than the other tests | *(0.10.0+)* the run also emits `error` `PROJECT_MISMATCH` with `projects[]` (`project_id`, `name`, `this_folder`, `tests`) and a `next` that runs this folder's project's tests by path: run it, and tell the person which tests belong to the other project | -If **any** member fails preflight, the plan is invalid: nothing runs, exit `2`. Suggest `--dry-run` to preview the plan cheaply before a big run. *(0.8.15+)* Preflight also checks variables. A `{{name}}` with no value is a warning: the member stays in the plan (`valid` does not look at variables), and the run types the name as written unless a step sets it first. The check covers every step of the member, replayed ones included. `testrun run` has no `--variables` flag, so fill the pool file (`.testmuai/variables/*.json`) or the member's own `variables:` frontmatter first. +If **any** member fails preflight, the plan is invalid: nothing runs, exit `2`. *(In a folder linked to a shared project, from the release that carries it)* the run then emits `error` `PLAN_INVALID` +with `offenders[{path, failure}]`, and `testrun_done{overall_status:"refused"}`; `PROJECT_MISMATCH` is followed by the +same `testrun_done`; a folder with no tests gives `error` `NO_TESTS` ("no tests here yet") with `next` +`kane-cli context pull --mode agent`; and a `--match` or `--tags` that selects none of the folder's tests gives `NO_TESTS` +("no *_test.md files selected") with no `next`: fix the filter. Everywhere else, and on 0.10.0, an invalid plan ends +after `testrun_plan` (and, for a mix of projects, its `PROJECT_MISMATCH` `error` above) with no `testrun_done`: read the +exit code and each member's `failure`. Suggest `--dry-run` to preview the plan cheaply before a big run. *(0.8.15+)* Preflight also checks variables. A `{{name}}` with no value is a warning: the member stays in the plan (`valid` does not look at variables), and the run types the name as written unless a step sets it first. The check covers every step of the member, replayed ones included. `testrun run` has no `--variables` flag, so fill the pool file (`.testmuai/variables/*.json`) or the member's own `variables:` frontmatter first. + +## Designed tests in a shared project (kane-cli 0.10.0+) + +In a folder linked to a shared project (`shared-projects.md`), a designed test (its file sits directly in `.testmuai/tests/` +and carries an `assurance:` block) runs **on its scenario's Test Manager case**. Before the browser opens, kane-cli finds +that case from the test's `id` and `base`. + +| The test here | What its run does | Cost | +|---|---|---| +| no `output-<test>/` beside it (never run in this folder: just saved, or pulled) | a **first run**: it attaches a new KaneAI test to the case and writes every step in a real browser, then commits it | minutes, and about 30–85 credits per test (each input it checks is its own browser run); tell the person before running it | +| has `output-<test>/` from a first run here that finished | a **replay** of the recorded steps; a step whose text changed since the recording (edited here, or replaced by a save or a pull) is written again, as on a first run | a few credits; more for each step written again | +| a replayed step fails | **heal**: it rewrites from that step, up to 3 times, then the whole test | about a first run's cost | + +- Before a first run, tell the person how many tests will write their steps and the likely cost + (`shared-projects.md` §4), and let them choose. Don't add `--no-adaptive-heal`, `--author` or other flags that change + what the run does on your own (`--headless` from the watch preference is fine); heal is on by default. +- **Credits.** A local run's own stream carries no credits total: note `kane-cli balance` + before the run and again after it, and give the person the difference. (A `--remote` run's `testrun_summary.totals.creditsConsumed`, + when present, adds up what the grid's runs reported.) +- Run the files a save or a pull named: `kane-cli testrun run <files> --agent < /dev/null`. A bare `testrun run` takes + every `*_test.md` here, and a folder that also holds a `kane-cli run` test saved to another project (the profile's) + refuses `PROJECT_MISMATCH` (above). +- A designed test moved into a subfolder of `.testmuai/tests/` runs as an ordinary test, off its case, and doesn't count. +- **Counting.** A finished run counts toward the project's coverage once Test Manager has its evidence; `cover` can take + up to 5 minutes to show it. The run's own `coverage` event (`computed_by: "project"`) shows what this run reached + when its `status` is `written`; any other status has no figure yet (`cards.md` §8). +- **A test that can't run on its case** stops before the browser, and its member ends `failed`; in agent mode its + `failure.message` says why. That test's own log (`log_path`) holds the refusal as an `error` event with its `code` and + a `remedy`; for a missing or gone case the remedy names the session to save (add `--yes --mode agent`). When it names + none, run `kane-cli context sessions --mode agent` and find the person's session. The codes and what to do: + +| Code | Means | Do | +|---|---|---| +| `TEST_CASE_MISSING` | the design that made this test was never saved, so it has no case | save that session: `kane-cli context sessions save <sid> --yes --mode agent` (the person's own session) | +| `TEST_CASE_GONE` | its case was deleted or moved to another project in Test Manager | save the session that designed it again: `kane-cli context sessions save <sid> --yes --mode agent` (free; it makes a new case) | +| `TEST_CASE_ARCHIVED` | its case is archived in Test Manager | ask the person to restore the case in Test Manager, then run the test again | +| `TEST_NOT_IN_PROJECT` | the test isn't in this folder's project, the project has no work yet, or the design it came from can't be saved any more | do what the refusal's message says: run it in the folder linked to its own project, or design the use case again on the person's word (`kane-cli design tests --use-case <uc> --force --mode agent`; add `--allow-unreviewed` when the use-case is still proposed) | +| `TEST_MANAGER_OUTDATED` | Test Manager can't attach this run (for example a device target: it attaches web runs only), or it made a case of its own for the run | a device target: run it on desktop web. Otherwise tell the person Test Manager needs an update before this test can run, and give the support id | +| `TEST_CASE_ATTACH_FAILED`, `TEST_RUN_NOT_SAVED` | the run couldn't be recorded on the case (the verdict stands for the second) | try once more; then give the person the support id | +| `PROJECT_NO_ACCESS`, `ASSURANCE_OFF`, `CREDENTIALS_MISSING`, `CREDENTIALS_REJECTED`, `CONNECTION_FAILED` | the case lookup on a first run failed | `agent-contract.md` | + +- In a terminal the person sees only "failed" for such a member; the reason is in that member's own log (the + `log_path` of its end event, `testrun_member_end` or `testrun_authored_member_end`). Read it before you report the + failure. +- **With `--remote`**, every designed test's case is looked up on this machine before anything is sent (`--dry-run` + too). If any test would stop, nothing is sent: each such test arrives as one `remote_error` with the code above, + `detail` naming the file, and its `remedy`, then `testrun_done` failed and exit `2`. Fix those tests, or leave them out with `--match` + or `--tags`. ## Remote: the suite as one HyperExecute job (`--remote`) @@ -98,7 +152,7 @@ All typed; stdout; one JSON object per line. **Local completion: `testrun_done`. | `testrun_investigations_wait` | `count` | Failed replays left investigations running; the coordinator waits before sealing. Narrate as "investigating N failures". | | `testrun_evidence_ingest` | `status: "ok"\|"failed"`, `evidence_id`, `stage?` | Pack published to the dashboard. Absent when publish is skipped. | | `testrun_summary` | `totals: {tests, passed, failed, broken, skipped, authored}`, `duration_s`, `upload`, `cancelled`, `execution: {id, status}` | Build the rollup table from this. | -| `testrun_done` | `execution_id`, `overall_status: "passed"\|"failed"\|"cancelled"` | Local completion; remote runs continue through `remote_done`. | +| `testrun_done` | `execution_id`, `overall_status: "passed"\|"failed"\|"cancelled"` (`"refused"` for an invalid plan in a linked folder), `coverage?: {status, pct?}` | Local completion; remote runs continue through `remote_done`. `coverage` repeats the run's coverage figure (`pct` only when `status` is `written`). | ### Each test's own log, and `--stream-members` (0.8.17+) @@ -112,18 +166,18 @@ With `--remote`, the stream is wrapped in typed `remote_*` events (all on stdout | `type` | Payload | Notes | |---|---|---| -| `remote_start` | `backend`, `env`, *(0.8.17+)* `log_path` | Dispatch begins. `log_path` is the grid client's own log on this machine, useful when a dispatch fails before a job exists | +| `remote_start` | `backend`, *(0.8.17+)* `log_path`; ignore any other field | Dispatch begins. `log_path` is the grid client's own log on this machine, useful when a dispatch fails before a job exists | | `remote_device` | `platform`, `slug`, `name`, `os_version`, `avd_id?`, `pool?` | The resolved grid device (emulator/simulator only). Present it as the device line. | | `remote_member_device` | `path`, `kind: emulator\|simulator\|real-device`, `platform`, `name?`, `os_version?`, `name_source`, `os_version_source` | One per device member before dispatch. Sources: `flag`, `file`, `config` (saved device), `default`, `hub` (that half left to the hub, no value). Present as one device line per member. | | `remote_device_hint` | `reason: device_name_ignored\|catalog_stale\|saved_device_unavailable\|real_device_list_unavailable`, `detail` | Informational; `device_name_ignored` is emulator-only; `real_device_list_unavailable` = devices dispatched unchecked | | `remote_app` | `path`, `app_id`, `source: uploaded\|cache\|dry-run` | One per distinct local build uploaded from the laptop (mobile); `app_id` is empty on a dry run | | `remote_dispatched` | `job_id`, `job_url` | The HyperExecute job exists — give the user `job_url` | | *(0.8.17+)* member events on remote | `testrun_start`, then a start and end event per test, each with `post_hoc: true` | The grid reports per-test detail **after the job ends**, in plan order, just before `testrun_summary`. Their `ts` is the grid's own time. Same fields as local, including `log_path` and `failure`. `testrun_progress` is not emitted on remote. On 0.8.17+ `remote_dispatched` arrives as soon as the job exists, not at the end | -| `remote_error` | `code`, `detail` | Remote preflight refused (table above); expect `testrun_done` failed + exit 2 | +| `remote_error` | `code`, `detail`, `remedy?`, `support_id?` | Remote preflight refused (table above), or a designed test can't run on its case (§Designed tests in a shared project); expect `testrun_done` failed + exit 2. `code: "internal_error"` is a crash of the remote run itself: `testrun_done` and `remote_done` failed, exit 1 | | `remote_import_tape`, `remote_exec_sync`, `remote_coverage` | `status`, `reason`, `detail?` | Informational; sync/coverage are `skipped` when the project has no `.context` store | | `remote_done` | `status`, `exit`, `job_id`, `sessions_path` | Follows `testrun_done`; `sessions_path` holds the members' grid session logs | -`testrun_summary` additionally carries `remote: {backend, jobId, jobUrl, sessionsPath}`. A member `status: "broken"` with `execution: null` and `upload: "skipped"` means the grid-side kane-cli refused before launching — send the user to `job_url` for the scenario log and check the app id's environment. +`testrun_summary` additionally carries `remote: {backend, jobId, jobUrl, sessionsPath}`. A member `status: "broken"` with `execution: null` and `upload: "skipped"` means the grid-side kane-cli refused before launching — send the user to `job_url` for the scenario log and check that the app id belongs to this account (`kane-cli apps list --target <kind> --agent`). Parsing strategy: @@ -133,7 +187,7 @@ for each line: if type === "remote_done" → capture remote status, exit and sessions_path if type === "testrun_plan" && !valid → report offenders, expect exit 2 if type === "warning" && code === "unresolved_variables" → name the variables with no value (references/parsing.md); the run continues; ignore other warning codes - if type === "testrun_member_end" → note per-member outcome + if type === "testrun_member_end" or "testrun_authored_member_end" → note per-member outcome if type === "testrun_summary" → capture totals for the rollup else → informational; narrate sparingly ``` @@ -156,7 +210,7 @@ Don't list passing tests individually. If the pack published, mention the run is ## Exit codes -`0` all members passed · `1` at least one failed/broken · `2` usage / invalid plan / auth (nothing ran) · `3` cancelled (Ctrl-C — in-flight members finish, the pack still seals). +`0` all members passed · `1` at least one failed/broken · `2` usage / invalid plan / auth (nothing ran) · `3` cancelled (Ctrl-C — no new members start, running ones are stopped, and the pack still seals with what finished). ## Evidence & debugging @@ -164,14 +218,14 @@ The suite produces **one** sealed pack, created directly in `<cwd>/.testmuai/evi ## Execution constraints -Local suites containing any mobile member require `--parallel 1`; larger values are refused. Isolated Chrome workers apply to desktop members only. Remote suites support grid concurrency via `--parallel N`. +Local suites containing any emulator or simulator member require `--parallel 1`; larger values are refused. Real-device members each open their own session, so `--parallel` is allowed up to the plan's devices (§When to use). Isolated Chrome workers apply to desktop members only. Remote suites support grid concurrency via `--parallel N`. Healing is enabled by default (three shrinking replay windows, then re-authoring of authorable steps). `--no-adaptive-heal` disables it. Retired `--retry`/`--retry-count` only print a notice and have no effect. Replay-only recorded steps retain their recordings even during healing. NDJSON selection uses stdin, not stdout (on 0.8.21+ `--agent` selects the same stream): every `kane-cli testrun run` line ends in `< /dev/null` (bash and zsh on macOS, Linux and Git Bash; `< NUL` in cmd.exe; from PowerShell run it through cmd: `cmd /c "kane-cli testrun run … < NUL"`). Check the first stdout line: on 0.8.17+ it is `{"type":"stream_start"…}`. A prose plan there instead means the CLI is in terminal mode: it prints the human view and, after a real run, opens an evidence table that waits until `q` or Esc, so the process never exits on its own. Stop it and rerun with the redirect. An empty stdout with a message on stderr is a usage error: read it. Dry-run validates a plan, not runtime authentication or browser/device readiness. Always observe process exit, including paths without a normal completion event. -### Remote behavior still requiring verification - -The audited dispatch does not forward `--bug-detection` to member commands and does not map `--on-failure` into the job template. Do not rely on these flags for remote bug-detection or fail-fast behavior until implementation owners confirm or fix the mapping. `--name` is suite metadata, not a forwarded member flag. +### Remote flags that don't reach the tests -For dispatched runs, read through `remote_done` and process exit after `testrun_done`; retain the remote status and session-log path. Preflight refusal and dry-run may terminate without `remote_done`. +On `--remote`, `--bug-detection` and `--on-failure` don't reach the tests on the grid: don't rely on them there. For +dispatched runs, read through `remote_done` and the process exit after `testrun_done`, and keep the remote status and +the session-log path. A preflight refusal or a dry run can end without `remote_done`. diff --git a/.agents/skills/kane-cli/scripts/preflight.ps1 b/.agents/skills/kane-cli/scripts/preflight.ps1 index 85b797c..161c257 100644 --- a/.agents/skills/kane-cli/scripts/preflight.ps1 +++ b/.agents/skills/kane-cli/scripts/preflight.ps1 @@ -1,9 +1,10 @@ # kane-cli ready check (preflight). Windows PowerShell 5.1 and later. # # Prints, in one call, everything an agent needs to know before it starts a -# kane-cli run. It only reads status. It changes nothing: no sign-in, no -# config write, no install, no network call of its own. It never reads -# credential files. It always exits 0, and problems show up in the text. +# kane-cli run. It only reads status: no sign-in, no config write, no install, +# no network call of its own, and it never reads credential files itself. The +# kane-cli status commands it runs use the saved sign-in, and whoami may renew +# one that is about to expire. It always exits 0, and problems show up in the text. # This is the twin of preflight.sh: same sections, same keys, same order. # # Usage: powershell -File preflight.ps1 [--mobile emulator|simulator] [--grid] @@ -13,6 +14,7 @@ # whoami kane-cli whoami, then exit=<code> # balance kane-cli balance, then exit=<code> # settings kane-cli config show, then exit=<code> +# link project= and project_id= of a linked folder, "none", or "unreadable" # agent-config saved agent preferences, or the word "none" # env ci= ssh= display= os= arch= node= # chrome found=<path> and override=<KANE_CLI_CHROME_PATH> @@ -117,6 +119,23 @@ try { Write-Output '## settings' Write-CommandBlock -CliArgs @('config', 'show') + Write-Output '## link' + # A folder linked to a shared project holds .testmuai/assurance.json; print only its project's name and id, read as + # kane-cli reads them, or "unreadable" when kane-cli can't read the link either. + $linkFile = [IO.Path]::Combine((Get-Location).Path, '.testmuai', 'assurance.json') + if (Test-Path -LiteralPath $linkFile) { + $link = $null + try { $link = Get-Content -LiteralPath $linkFile -Raw -Encoding UTF8 -ErrorAction Stop | ConvertFrom-Json -ErrorAction Stop } catch { } + if (($link -is [System.Management.Automation.PSCustomObject]) -and ($link.projectId -is [string]) -and $link.projectId -and ($link.projectName -is [string])) { + Write-Output ("project=" + ($link.projectName -replace "[`r`n]+", ' ')) + Write-Output ("project_id=" + ($link.projectId -replace "[`r`n]+", ' ')) + } else { + Write-Output 'unreadable' + } + } else { + Write-Output 'none' + } + Write-Output '## agent-config' $agentConfig = [IO.Path]::Combine($HOME, '.testmuai', 'kaneai', 'agent-config', 'config.json') $agentConfigText = $null diff --git a/.agents/skills/kane-cli/scripts/preflight.sh b/.agents/skills/kane-cli/scripts/preflight.sh index 82dc3ec..232f69e 100755 --- a/.agents/skills/kane-cli/scripts/preflight.sh +++ b/.agents/skills/kane-cli/scripts/preflight.sh @@ -2,9 +2,10 @@ # kane-cli ready check (preflight). # # Prints, in one call, everything an agent needs to know before it starts a -# kane-cli run. It only reads status. It changes nothing: no sign-in, no -# config write, no install, no network call of its own. It never reads -# credential files. It always exits 0, and problems show up in the text. +# kane-cli run. It only reads status: no sign-in, no config write, no install, +# no network call of its own, and it never reads credential files itself. The +# kane-cli status commands it runs use the saved sign-in, and whoami may renew +# one that is about to expire. It always exits 0, and problems show up in the text. # # Usage: sh preflight.sh [--mobile emulator|simulator] [--grid] # @@ -13,6 +14,7 @@ # whoami kane-cli whoami, then exit=<code> # balance kane-cli balance, then exit=<code> # settings kane-cli config show, then exit=<code> +# link project= and project_id= of a linked folder, "none", or "unreadable" # agent-config saved agent preferences, or the word "none" # env ci= ssh= display= os= arch= node= # chrome found=<path> and override=<KANE_CLI_CHROME_PATH> @@ -126,6 +128,33 @@ emit_cmd balance kane-cli balance echo "## settings" emit_cmd settings kane-cli config show +echo "## link" +# A folder linked to a shared project holds .testmuai/assurance.json; print only its project's name and id, read as +# kane-cli reads them, or "unreadable" when kane-cli can't read the link either. +link_file=".testmuai/assurance.json" +if [ -e "$link_file" ]; then + link_out="" + if command -v node >/dev/null 2>&1; then + link_out=$(node -e ' + try { + const b = JSON.parse(require("fs").readFileSync(process.argv[1], "utf8")); + const one = (s) => s.replace(/[\r\n]+/g, " "); + if (b && typeof b === "object" && !Array.isArray(b) && typeof b.projectId === "string" && b.projectId !== "" && typeof b.projectName === "string") { + process.stdout.write("project=" + one(b.projectName) + "\nproject_id=" + one(b.projectId) + "\n"); + } + } catch (e) {} + ' "$link_file" 2>/dev/null </dev/null) + elif [ -f "$link_file" ] && [ -r "$link_file" ]; then + # No node (kane-cli itself needs it): a best-effort read of the two fields. + link_name=$(sed -n 's/.*"projectName"[[:space:]]*:[[:space:]]*"\([^"]*\)".*/\1/p' "$link_file" | head -n 1) + link_id=$(sed -n 's/.*"projectId"[[:space:]]*:[[:space:]]*"\([^"]*\)".*/\1/p' "$link_file" | head -n 1) + if [ -n "$link_id" ]; then link_out=$(printf 'project=%s\nproject_id=%s' "$link_name" "$link_id"); fi + fi + if [ -n "$link_out" ]; then printf '%s\n' "$link_out"; else echo "unreadable"; fi +else + echo "none" +fi + echo "## agent-config" agent_config="$HOME/.testmuai/kaneai/agent-config/config.json" if [ -f "$agent_config" ] && [ -r "$agent_config" ]; then diff --git a/.claude/skills/kane-cli/SKILL.md b/.claude/skills/kane-cli/SKILL.md index 1937ed9..1aebb55 100644 --- a/.claude/skills/kane-cli/SKILL.md +++ b/.claude/skills/kane-cli/SKILL.md @@ -1,6 +1,6 @@ --- name: kane-cli -description: "End-to-end (E2E) test generation and execution with KaneCLI. Generate tests from a PRD, specification, or user-provided description via the assurance commands (use-cases, acceptance criteria, scenarios, runnable _test.md tests; never write test cases by hand). Execute them on desktop web (Chrome, the default), Android and iOS: emulator/simulator locally on macOS Apple Silicon or via testrun run --remote, and real cloud devices from any machine (--target android|ios). Prove coverage with evidence packs and kane-cli cover. Use for any task needing a real browser or mobile app: navigation, clicks, form filling, UI checks, screenshots, deploy verification. Parse NDJSON output, inspect logs, and save replayable _test.md files. Share the assurance store (.context/) with a team through a GitHub repository, S3-compatible bucket, or folder via kane-cli context sync, and resolve decisions requested during a sync rebase." +description: "End-to-end (E2E) test generation and execution with KaneCLI. Generate tests from a PRD, specification, or user-provided description via the assurance commands (use-cases, acceptance criteria, scenarios, runnable _test.md tests; never write test cases by hand). Execute them on desktop web (Chrome, the default), Android and iOS: emulator/simulator locally on macOS Apple Silicon or via testrun run --remote, and real cloud devices from any machine (--target android|ios). Prove coverage with evidence packs and kane-cli cover. Work in a team's shared Test Manager project from a linked folder: link it with --project, save sessions, run the designed tests on their Test Manager cases, and read the project's coverage. Use for any task needing a real browser or mobile app: navigation, clicks, form filling, UI checks, screenshots, deploy verification. Parse NDJSON output, inspect logs, and save replayable _test.md files. Share a local assurance store (.context/) through a GitHub repository, S3-compatible bucket, or folder via kane-cli context sync, and resolve decisions requested during a sync rebase." --- # Kane CLI — End-to-End Test Generation and Execution Skill @@ -11,6 +11,12 @@ KaneCLI is one E2E package with three pillars: 2. **Execute**: desktop web (Chrome, the default), Android emulator, iOS simulator, and real Android and iOS devices in the TestMu AI cloud. 3. **Prove**: a sealed evidence pack per run, and coverage (`kane-cli cover`) measured against acceptance criteria. +**Working with a team?** On kane-cli 0.10.0+ a team works in a **shared project**: one Test Manager project that its +folders are linked to. What one person saves there (use-cases, tests, Test Manager cases, coverage) the whole team sees. +Test files stay in the folder where they were saved until they're brought over (`references/shared-projects.md` §7). In +agent mode, a folder that isn't set up yet goes local unless you link it with `--project <id>`. So decide where the work +goes before the first assurance command, and read `references/shared-projects.md` first. + Use `kane-cli` to validate complete user journeys and design requirement-linked, coverage-accounted suites from a PRD, specification, or user-provided description via assurance commands. Never write test cases by hand. Do NOT use Playwright, Puppeteer, or Selenium directly. Use `--agent` for `run` and `testmd run`. `testrun run` emits NDJSON when **stdin** is not a TTY (0.8.21+ also takes `--agent`, which selects the same stream), so every `testrun run` line you write ends in `< /dev/null` (bash and zsh: macOS, Linux, Git Bash; in cmd.exe write `< NUL`; from PowerShell run it through cmd: `cmd /c "kane-cli testrun run … < NUL"`). Check the first stdout line: on 0.8.17+ it is `{"type":"stream_start"…}`. A prose plan there instead means the CLI is in terminal mode: it shows the human view and, after a real run, opens an evidence table that waits until `q` or Esc, so the process never exits on its own; stop it and rerun with the redirect. An empty stdout with a message on stderr is a usage error: read it. Assurance conversational commands use `--mode agent`. **Authoring test cases or scenarios?** Never write them by hand: every test case comes from the **assurance** commands — Read `references/assurance.md` first. @@ -54,7 +60,7 @@ On Windows PowerShell: `$env:KANE_CLI_USER_AGENT='<your-runtime>'; kane-cli run **Watch mode.** Use the person's saved preference (`references/agent-config.md`). With none saved, show the browser unless the preflight says there is no display, an SSH session, or CI: then add `--headless`. -**Keeping runs.** When the person's saved purpose is `suite` or `ask`, add `--name <short-slug>` to every one-off `run`. A named run is recorded as a `_test.md` while it runs, so keeping it afterwards costs nothing, and a run launched without a name cannot be kept without running again. With `one-off`, leave the flag out. Details: `references/first-run.md` §4. +**Keeping runs.** When the person's saved purpose is `suite` or `ask`, add `--name <short-slug>` to every one-off `run`. A named run is recorded as a `_test.md` while it runs, so keeping it afterwards costs nothing, and a run launched without a name cannot be kept without running again. With `one-off`, leave the flag out. Details: `references/first-run.md` §4. Run one-offs from a folder that isn't linked: a named run in a linked folder writes its test beside the project's and can mix two projects (`references/shared-projects.md` §9). Bash blocks until kane-cli exits, then hands you the complete stdout. Parse it, summarize what happened, and present the result card. Wait for process completion on `testmd run` too, but parse its own completion event, `test_md_done`. An intermediate `run_end` does not finish a saved test. @@ -144,7 +150,7 @@ For exit code 1 (or `status: "failed"` in `run_end`), present the failure card. The failing step's screenshot lives inside the run's evidence pack (the stderr hint names the pack path): extract it with `unzip <pack> "tests/*/steps/*/screenshot.png" -d <tmpdir>`, Read it, and show it **under** the card. For the pack layout and deeper diagnosis, see `references/debug.md`. -Exit code 2 means nothing ran: that is a `🟡 Didn't start` card, not a failure (`references/cards.md` §4). +Exit code 2 with no progress events means nothing ran: that is a `🟡 Didn't start` card, not a failure (`references/cards.md` §4). If progress events came before it, steps ran and the run stopped without a result: say so, and use the stopped-early card (`references/cards.md` §5). ### 1.7 After the first result: three choices, then save @@ -163,19 +169,28 @@ When the user's request involves a browser — or writing test cases: - A problem that stops the run → offer the fix from the card; deeper setup lives in `references/setup-and-config.md` - Ready ↓ +**Where does the work go?** (before any assurance command) +- The folder holds its own `.context/` and no link → it works locally, as before (with a link too, kane-cli refuses it `BOTH_STORES`): `references/assurance.md` (to share that store with a team, `references/context-sync.md`). The rows below are for a folder without one +- The folder holds `.testmuai/assurance.json`, or the person works with a team or names a Test Manager project → a **shared project**: Read `references/shared-projects.md` first. In a linked folder never use `kane-cli run` for a first result; its tests go to another project (§9 there) +- The person works alone, or asks to keep the work on this machine → the local store: `references/assurance.md` +- Not clear → ask once: "Should this work go to your team's Test Manager project, or stay in this folder?" + **What does the user want?** +- Tests that exploration (HyperExecute) designed, or a teammate's tests, in this folder → Read `references/shared-projects.md` §7 +- Explore an app on HyperExecute, shape, stop or re-run an exploration, or check what it designed → Read `references/explore.md` first +- A command was refused, or you need the shared-project event and code list → Read `references/agent-contract.md` - A single one-shot browser task → build a `kane-cli run --agent` command (§3 + §4) - A test they want to save / re-run / commit → Read `references/testmd.md` first, then use `kane-cli testmd` - Run a suite of saved tests (several `_test.md` at once) → Read `references/testrun.md` first, then use `kane-cli testrun run … < /dev/null` - Need test cases from a description the user gives in chat, with no document → **don't hand-write them**; write the description to a requirements file and design from it (§6) - Has requirement documents (PRD/spec) and wants a designed suite, coverage accounting, or suite upkeep → Read `references/assurance.md` first — the assurance commands (`context`/`design`/`cover`/`maintain reconcile`, kane-cli 0.6.1+; several features need newer releases, up to 0.8.14+ — the reference marks each) - A **designed** test (its `_test.md` carries an `assurance:` block) failed a run → Read `references/assurance.md` §6.1 **before touching the file** — an edit is adopted as the next design version on the next run; classify the failure first (app bug → report it; requirement changed → reconcile; wording → redesign through the CLI; capability missing → stop), and edit a step of an existing designed test by hand only after reading `references/objectives-cookbook.md` -- Share the context store with a team, join a teammate's, keep two stores level, or resolve a sync conflict → Read `references/context-sync.md` first — `kane-cli context sync`, `kane-cli context push`, `kane-cli context pull` and `kane-cli context clone` (kane-cli 0.8.14+); the store is shared through a location, never by copying or git-merging `.context/` +- Share a **local** context store through a repository, a bucket or a folder, join one, keep two stores level, or resolve a sync conflict → Read `references/context-sync.md` first (kane-cli 0.8.14+). A team in Test Manager uses a shared project instead (`references/shared-projects.md`); a linked folder refuses `kane-cli context sync`, `kane-cli context push` and `kane-cli context clone` (`REMOTE_UNSUPPORTED`), and a location pull (a location name, `--rebase`, `--yes` or `--answer`: `PULL_LOCATION_LINKED`); a plain `kane-cli context pull --mode agent` there brings the project's tests (`references/shared-projects.md` §7); the store is shared through a location, never by copying or git-merging `.context/` - Multiple independent browser tasks → Read `references/parallel.md` first - View, share, or validate run evidence (`.evidence` packs) → Read `references/evidence.md` - Debug a failed run → Read `references/debug.md` - Configure kane-cli or check directory layout → Read `references/setup-and-config.md` -- Browse / create / pick a Test Manager project or folder, or interpret the auto-default event → Read `references/test-manager.md` +- Browse / create / pick a Test Manager project or folder for `kane-cli run`, or interpret the auto-default event → Read `references/test-manager.md`. Linking an assurance folder to a project is different: `references/shared-projects.md` §2–3 - The person wants results saved somewhere else ("change project") → Read `references/test-manager.md` §6. The change is global, and the question must say so - The person wants to change how runs behave ("kane preferences": watch or quiet, one-off or suite) → Read `references/agent-config.md` - The person asks what kane-cli can do, or for the tour again ("kane tour") → show the tour from `references/first-run.md` §2 @@ -204,7 +219,7 @@ kane-cli run "<objective>" --agent [options] | Flag | Purpose | Default | |------|---------|---------| | `--headless` | No visible browser window | Off | -| `--max-steps <n>` | Cap agent reasoning steps | 30 | +| `--max-steps <n>` | Cap agent reasoning steps | 50 | | `--timeout <s>` | Hard kill after N seconds | No limit | | `--url <url>` | Start URL for the run (overrides config `default_url`; bare domains get `https://`) | Config `default_url` | | `--variables <json>` | Inline variables JSON (for `{{key}}` in objective) | None | @@ -226,7 +241,7 @@ Other flags (`--global-context`, `--local-context`, `--cdp-endpoint`, `--allow-m 1. Collect the names with no value: the `warning`, a dry-run plan's `unresolved[]` rows, or a design run's `variables_declared` and `variables_summary` rows. Design rows list what design declared, not every missing value, and a dry run's `valid: true` says nothing about values: the dry run's `warning` is the check. 2. A value you already have, because the user said it or the requirement document states it, you write into that key in the file the event names (for `run`, `--variables '{"name":{"value":"…"}}'` also works), and you tell the user what you filled and where. 3. The rest you ask for once, in one message, using each variable's `description` when design gave one. Plain values (a URL, an email, a user name) the user gives you here or adds to the file, their choice. Secrets, which are any row with `secret: true`, any description that names a credential, and any name containing `password`, `secret`, `token` or `key`, the user fills in the file and tells you when done; you never ask for the value in chat, never echo it, and report names and file paths only. -4. Then a fresh `--dry-run` of the exact selection: a valid plan with no `warning` is the check. A member whose rows are all filled may run while the others wait. Never fill a placeholder just to silence the check; a throwaway value is right only when the description asks for one, such as a deliberately wrong password. A frontmatter declaration with an empty value also silences the check, so look at the values a step relies on, not only at the warning. A name still empty is typed into the page as written; if a run then failed at that step, say so. +4. Then the check: for saved tests, a fresh `kane-cli testrun run <files> --dry-run --agent < /dev/null` of the exact selection (`run` and `testmd run` have no dry run; for them the `warning` arrives before the first step): a valid plan with no `warning` is the check. A member whose rows are all filled may run while the others wait. Never fill a placeholder just to silence the check; a throwaway value is right only when the description asks for one, such as a deliberately wrong password. A frontmatter declaration with an empty value also silences the check, so look at the values a step relies on, not only at the warning. A name still empty is typed into the page as written; if a run then failed at that step, say so. ### Examples @@ -330,6 +345,8 @@ For one-shot `run`, build post-run logic on `run_end` and process exit. Saved te For full event schemas (`bifurcation` flow fields, `child_agent_*`, `ask_user` semantics, `cancel`/`user_response` outbound events, complete `run_end` field list), Read `references/parsing.md`. +**Which agent flag.** `--agent`: `run`, `testmd run`, `testrun run` (or stdin not a TTY), `projects`, `folders`, `explore`, `config show`, `generate`, `devices`, `apps`. `--mode agent`: `context …`, `design …`, `maintain …`, `cover`. `run` and `testmd run` also accept `--mode agent`; `testrun`, `projects`, `folders` and `explore` refuse it, and the assurance commands refuse `--agent`. With `--agent`, an update notice can still print on stderr; it is not part of the stream. In a linked folder the streams, events and refusal codes are in `references/agent-contract.md`. + The assurance conversational commands (`context ingest`/`context extract`, `design tests`, `maintain reconcile`, `maintain evolve` on 0.8.21+, `cover`) do NOT take `--agent` — they take **`--mode agent`** and speak their own typed stream ending in `done` (open vocabulary — tolerate unknown event types; on 0.7.2+ the stream is strict — every stdout line parses, stderr silent — while on 0.7.1 a merged `context ingest` prints a few prose receipt lines BEFORE the stream — skip to the first `{` line, harmless on 0.7.2+ — and a landing-phase ingest failure ends with prose + exit 1/2 and no stream at all: a refusal, not a crash; on 0.7.2+ those failures ride the stream as `error` + `done`); **for those commands only, exit `3` means paused-and-resumable, not timeout** — schema in `references/assurance-parsing.md`, behavior in `references/assurance.md`. The context sync verbs (`kane-cli context sync`, `kane-cli context push`, `kane-cli context pull`, `kane-cli context clone`; kane-cli 0.8.14+) take `--mode agent` the same way and speak a `sync_*` family ending in `done`; there too exit `3` means a decision or a pull is needed, not a failure — `references/context-sync.md`. `kane-cli context sync setup` refuses without a terminal (`TTY_REQUIRED`): agents use `kane-cli context sync add` and `kane-cli context clone`. `kane-cli testrun run` also emits its own typed stream (`testrun_plan` … terminal `testrun_done`) — schema in `references/testrun.md`. `kane-cli testmd run` may additionally emit `test_md_evidence_ingest` (replay evidence published) and `test_md_bundle_sync` (test bundle synced) — informational; describe in plain language, never surface raw names. The post-run evidence hint (`` evidence: view locally with `kane-cli evidence serve <path>` ``) is a **stderr** text line, not a stdout event — don't try to parse it from the NDJSON stream; see `references/evidence.md` for how to act on it. @@ -341,7 +358,7 @@ The assurance conversational commands (`context ingest`/`context extract`, `desi When the user describes what to test in chat and has no document, use the description as the requirement. 1. Write the user's words to `requirements/<feature>.md` in the project, as given, one heading per feature, nothing invented. Tell the user the file exists and that the tests will cite it. -2. `kane-cli context ingest requirements/<feature>.md --mode agent`, review the extracted use-cases at the checkpoint, then `kane-cli design tests …`, exactly as `references/assurance.md` describes. +2. `kane-cli context ingest requirements/<feature>.md --mode agent`, review the extracted use-cases at the checkpoint, then `kane-cli design tests …`, exactly as `references/assurance.md` describes. In a shared project add `--project <id>` the first time, and follow the journey in `references/shared-projects.md` §4 (save each session; the one-question checkpoint). 3. Present the designed tests, gaps, warnings and the variables that need values (§3 **Fill the variables before any run**). The designed tests get their own review checkpoint before any run (`references/assurance.md`). Then hand the set to `kane-cli testrun run … < /dev/null` (`references/assurance.md`, the authoring bridge). A draft in chat cannot be run, and a hand-written `_test.md` carries no requirement link and no coverage accounting; the deliverable is the designed test. @@ -356,6 +373,10 @@ A draft in chat cannot be run, and a hand-written `_test.md` carries no requirem | Run a suite of saved `_test.md` tests as one batch | `references/testrun.md` | | Run a suite on the cloud grid (`--remote`), incl. mobile suites from any machine | `references/testrun.md` §Remote + `references/mobile.md` §Remote | | Run on a real Android/iOS device (`--target android\|ios`), its plan requirements, or a hub session error | `references/mobile.md` §Real devices | +| A team, a Test Manager project, a linked folder (`.testmuai/assurance.json`), a teammate's or exploration's tests | `references/shared-projects.md` | +| This folder's shared project and its id (for a teammate, or a dashboard) | `references/shared-projects.md` §2, "This folder's project id" | +| A shared-project refusal code, event or exit code | `references/agent-contract.md` | +| Explore an app on HyperExecute | `references/explore.md` | | User has requirement docs (PRD/spec) → designed suite, coverage, or suite upkeep | `references/assurance.md` | | Need the assurance NDJSON event schema (`--mode agent`) | `references/assurance-parsing.md` | | Share the context store with a team, join one, keep stores level, or resolve a sync conflict (kane-cli 0.8.14+) | `references/context-sync.md` | diff --git a/.claude/skills/kane-cli/references/agent-config.md b/.claude/skills/kane-cli/references/agent-config.md index fa35bb8..4ab53e7 100644 --- a/.claude/skills/kane-cli/references/agent-config.md +++ b/.claude/skills/kane-cli/references/agent-config.md @@ -34,7 +34,7 @@ They follow the person across agents and projects, and they survive a skill rein | Key | Values | Meaning | |---|---|---| | `preferences.watch` | `visible` · `quiet` · `results-only` | `visible`: no `--headless`. `quiet`, `results-only`: `--headless`. `results-only` also skips the progress summary | -| `preferences.purpose` | `one-off` · `suite` · `ask` | Whether to offer keeping passing runs as saved tests. With `suite` or `ask`, launch every one-off run with `--name <short-slug>` so keeping it costs nothing (`references/first-run.md` §4) | +| `preferences.purpose` | `one-off` · `suite` · `ask` | Whether to offer keeping passing runs as saved tests. With `suite` or `ask`, launch every one-off run with `--name <short-slug>` so keeping it costs nothing (`references/first-run.md` §4), except in a folder linked to a shared project (`shared-projects.md` §9) | | `preferences.narration` | `quiet` · `milestones` · `every-step` | How much of the run you recount afterwards. Default `milestones` | | `onboarding.asked` | list of `watch`, `results`, `purpose` | What was already asked. Never ask these again | | `onboarding.first_run_explained` | boolean | The tour was shown | @@ -90,4 +90,4 @@ Before you write, tell the person in one line what you are saving and where. The ## 5. Changing preferences later -When the person says "kane preferences" (or asks to change how runs behave), show the current values in plain words, ask what to change, and write the file again. To change where results go, use the flow in `references/test-manager.md`: that setting is global and belongs to kane-cli. +When the person says "kane preferences" (or asks to change how runs behave), show the current values in plain words, ask what to change, and write the file again. To change where results go, use the flow in `references/test-manager.md`: that setting is global and belongs to kane-cli (in a folder linked to a shared project, designed tests directly in its `.testmuai/tests/` run on that project's cases whatever it says: `shared-projects.md`). diff --git a/.claude/skills/kane-cli/references/agent-contract.md b/.claude/skills/kane-cli/references/agent-contract.md new file mode 100644 index 0000000..1feb643 --- /dev/null +++ b/.claude/skills/kane-cli/references/agent-contract.md @@ -0,0 +1,329 @@ +<!-- kane-cli skill reference: the agent contract for a folder linked to a shared project (kane-cli 0.10.0+). Which agent flag each command takes, the NDJSON shapes, every event, `next`, exit codes, and every refusal code with what to do. Internal parsing reference: never show event or field names to the person. --> + +# Shared projects: Agent Contract + +A folder is **linked** when `.testmuai/assurance.json` names a TestMu AI Test Manager project and the folder has no `.context/`. In a linked folder the commands below work in the **shared project**, so the team sees what each person saves. Link a folder with `kane-cli context ingest <files> --project <id> --mode agent` (or `kane-cli context extract --project <id> --mode agent`, or, where `kane-cli context pull --help` lists `--project`, `kane-cli context pull --project <id> --mode agent`); find the id with `kane-cli projects list --search <name> --agent`. In a folder with neither a link nor `.context/`, `--mode agent` without `--project` goes on locally without asking, as before: `context ingest` makes a local store, and `context extract` refuses `NO_STORE` until one exists. + +| Area | Commands | +|---|---| +| Projects | `kane-cli projects list`, `kane-cli projects create` | +| Sources and use-cases | `kane-cli context ingest`, `kane-cli context extract`, `kane-cli context list` | +| Sessions | `kane-cli context sessions`, `kane-cli context sessions save <sid>`, `kane-cli context sessions discard <sid>` | +| Tests | `kane-cli design tests`, `kane-cli maintain evolve`, `kane-cli maintain reconcile`, `kane-cli context pull` (`shared-projects.md` §7) | +| Runs and coverage | `kane-cli testmd run <designed>_test.md`, `kane-cli testrun run`, `kane-cli cover` | + +Nothing reaches the project until `kane-cli context sessions save`. Other assurance commands are refused in a linked folder (`REMOTE_UNSUPPORTED`, §6). Local-store parsing stays in `references/assurance-parsing.md`. + +## 1. Which flag + +| Command | Agent flag | The wrong one does | +|---|---|---| +| `context ingest`, `context extract`, `context list`, `context sessions` (list, save, discard), `design tests`, `maintain evolve`, `maintain reconcile`, `cover` | `--mode agent` | `--agent` is an unknown option: prose on stderr, no NDJSON, non-zero exit | +| `testmd run` | `--agent` (`--mode agent` works too) | — | +| `testrun run` | `--agent`, or stdin not a terminal | `--mode agent` is refused: `error` `USAGE` + `done` (exit 2); nothing runs | +| `projects list`, `projects create` | `--agent` (or stdin/stdout not a terminal) | `--mode agent` is refused the same way | +| `install skill` | none: with stdin not a terminal it prints one JSON line | `--agent` is an unknown option (exit 1); `--mode agent` is `USAGE` (exit 2) | + +- **Stderr.** Only a command line holding `--mode agent` keeps stderr free of the "Skill update available" and "Update available" banners. Under `--agent` those lines can appear on stderr, and `testrun run` also writes its coverage lines and `kane-cli evidence serve` hints there. Parse stdout only; a stderr line is not a failure. +- **No flag, no terminal.** Linked `context extract` and `design tests` refuse with a stderr line (exit 2). Linked `context ingest` falls back to land-only: it uploads, prints text lines and leaves the session paused. Always pass the flag. +- **Modes to avoid.** `--mode ci` on linked extract, design or reconcile is refused as a stderr line (exit 2), even with `--json`. `--json` with `--mode agent` is refused as a stderr line (exit 2). `--mode override` answers every question with its recommended option: never use it for the person. +- **Usage errors.** With `--mode agent` anywhere on the command line, a parse error arrives as `error` `USAGE` + `done{refused, 2}`; `--help` and `--version` print only a `done` line. + +## 2. Stream shape + +**Assurance stream** (`context …`, `design tests`, `maintain …`, `cover`): one JSON object per line, envelope `{"type": "<name>", "v": 1, "verb": "<verb>", …}`. No `stream_start`, and don't rely on a `ts` field here. Verbs: `sync` (`context pull` in a linked folder; refusals of its flags and at its entry carry `pull`), `extract` (ingest and extract; `reconcile` from the upload on when the ingest is a new version of a source the project holds), `design`, `evolve`, `reconcile`, `save`, `discard`, `cover`. Refusals of `context sessions` (list, show, clean) are stamped `save`, and those of `context list` are stamped `list`. + +**Run stream** (`testmd run`, `testrun run`): opens with `stream_start {cli_version, surface, pid}`; every line carries `v` and `ts`. Shapes: `references/testmd.md`, `references/testrun.md`; linked-folder additions in §3.7. + +| Command (agent flag) | First line | Last line | +|---|---|---| +| `context extract` | `folder_linked` (with `--project`), else `nothing_new`, `warning` or `remote_connect` | `done` | +| `context ingest <refs>` | `folder_linked` (with `--project`), else `remote_fetch` (an address) or `remote_upload` (a file); a refusal is `error` | `done` | +| `design tests` | `remote_connect` (or `warning`, `error`) | `done` | +| `maintain evolve` | `remote_connect` | `done` (may carry `not_updated`) | +| `maintain reconcile` | `remote_upload` or `remote_fetch`, else `remote_connect` | `done` | +| `context sessions save <sid>` | `session_save{phase:"check"}`; `phase:"saving"` for a session saved before; `phase:"saved"` with `saved: 0` when it added nothing (or `error`) | `done` | +| `context sessions discard <sid>` | `session_discard{phase:"check"}` (or `error`) | `done` | +| `context pull` (linked, or `--project`) | `folder_linked` (with `--project`), else `pull_test` or `error`; a project with no tests sends `done` alone | `done` (`shared-projects.md` §7a) | +| `cover` | `coverage` | `done` | +| `testmd run <designed test>`, stopped on its Test Manager case | `stream_start` | `error` (no `test_md_done`) | +| `testrun run`, refused | `stream_start`; then `error` before any plan for `--from-context` in a linked folder; else `testrun_plan` | `PROJECT_MISMATCH`: `error`, the last line outside a linked folder. Another invalid plan, or no tests: no `error`, the stream ends at `testrun_plan` (read the exit code). **In a linked folder only**, from the release that carries the fix, every one of these (`PROJECT_MISMATCH` too) ends `error` then `testrun_done{overall_status:"refused"}` (the `--from-context` refusal still ends on its `error`); handle both shapes. Everywhere else these shapes don't change | + +Outputs that are **not** the envelope and have **no `done`** (read the exit code): + +| Command | Stdout on success | +|---|---| +| `projects list --agent` | `{"id","name"}` per project, then `{"_meta":"page","limit","offset","returned","has_more"}`. Errors are plain text on stderr, with no code | +| `projects create "<name>" --agent` | one line `{"id","name"}`. Errors on stderr. It does not link the folder | +| `context list --mode agent` | one bare object per row: `{id, label, title, trust, fresh, tests_fresh?, next?}` (`trust: "derived"` = proposed, not approved yet; `"trusted"` = approved: say *proposed* or *approved*, never the value); `--undesigned` adds `designed: false`; nothing at all when nothing matches | +| `context sessions --mode agent` | one line holding a JSON array of `{sid, verb, subject, title, state, added, updated?, last_activity_at, owner}` | +| `install skill` (stdin not a terminal) | one line `{"type":"skill_install_result","status":"installed"\|"already-installed"\|"failed","version","installed_targets","error?"}`; exit 1 only on `failed` | + +Refusals of `context list` and `context sessions` do use the envelope: `error` + `done`. + +Dispatch on `type`, never on position. **The vocabulary is open:** tolerate unknown event types and unknown fields. A stream that should end in `done` and doesn't means the process died: check `kane-cli context sessions --mode agent` before retrying anything paid. + +## 3. Event catalogue + +"Tell the person" is plain words. Never show event or field names. + +### 3.1 Linking and loading (every assurance verb) + +| Event | When | Fields that matter | Tell the person | +|---|---|---|---| +| `folder_linked` | `--project <id>` linked the folder in this run; the first line | `project_id`, `project` | "This folder is now linked to <project>; what you save goes there." If the run then ends `PERMISSION_REQUIRED` or `ASSURANCE_OFF`, or a `context pull --project` is refused at the start or can't read the project, the link was removed again | +| `folder_local` | `--local` gave the folder its own store; `extract` then ends `done` exit 0 (ingest goes on locally) | `gitignore`: `added`, `failed`, null | "This folder keeps its own local store." On `failed`: `.context/` wasn't added to `.gitignore` | +| `nothing_new` | a plain `extract` found every source already read; no session opened; exit 0 | `reason`: `every_source_read`, or `sources_in_a_session` with `open_sessions[{sid, owner, state, added}]`; `next` | "Nothing new to extract" / "every source is already in an open session"; offer `next` | +| `remote_connect` | `step:"connect"`, then `step:"open_session"` once the session exists | `sid`, `project` (on `open_session`) | "Connecting to <project>." Keep `sid` | +| `remote_load` | the project's history loads | `part` (`project`, `project_ready`), `percent` (may be null) | fold into one progress line | +| `remote_fetch` | an address or item is fetched on this machine, before upload | `ref`, `provider` (`web`, `jira`, `confluence`, `linear`, `linear-doc`) | "Reading <ref>" | +| `remote_upload` | a file goes up before the session opens: 0%, per chunk, 100% | `file`, `percent` | one line per file | +| `remote_retry` | waiting on the shared project | `attempt`, `next_in_s`, `give_up_in_s`, `cause` (`unreachable`, `not_ready`, `timeout`) | "The shared project isn't answering; retrying." Let it run | +| `remote_overlap` | a new session opened while others are open on the same subject | `sessions[{sid, owner: you\|someone_else\|null, state, added, next?}]` | name the other sessions; offer the resume `next` on the person's own; this run goes on | +| `run_start` | the run starts | `session` = the shared project's session id (not a support id); no log path | — | +| `warning` | a non-fatal condition | `message`, `code?` | relay it | + +### 3.2 Ingest, extract, review, evolve + +| Event | When | Fields that matter | Tell the person | +|---|---|---|---| +| `ingested` | each item lands in the session | `source_id`, `status` (`created`, `versioned`, `unchanged`), `behind?`, `unread?`, `media?`, `duplicate_of?`, `origin?`, `warnings?`; no `cid` | "Added <file> to this session" / "a new version" / "already in <project>, unchanged" | +| `commit` | the agent's items land in the **session** | counts, `minted[]` | "N use-cases proposed in this session, not saved yet" | +| `held` | items held back | `reason`: `duplicate_check_reduced`, `review` | "N items are held for review" | +| `usage` | each agent turn | `credits`, `total_credits` | report the final total | +| `review_cards` | a new version's review holds its cards, before a `review_pending` pause | `source`, `cards[{key, kind, use_case?, title, change, diff?, tests?}]` | show each change; ask approve, reject or defer | +| `card_verdict` / `card_refused` | an `--answer <key>=<verdict>` landed / didn't | `key`, `verdict`, `line`, `offer?{kind, key, title}` / `code`, `refusal` | relay `line` or `refusal`. `offer.kind: "evolve"` runs at once (`child_start` … `child_end`); `offer.kind: "design"` does **not** run: after the save, offer to design that use-case's tests (`kane-cli design tests --use-case <uc> --mode agent`, its id from `kane-cli context list --mode agent`) | +| `evolve_paused` | evolve asks once per use-case; exit 3 | `pending_questions[]` (id `evolve:<uc>`), `next` (update all, leave all out) | present the blast radius and ask (`references/assurance.md` §11) | +| `child_start` / `child_end` | a design run inside a review or an evolve | `child`, `subject` / `ok`, `exit_code`, `next?` | "Designing tests for <uc>… done" / "didn't finish" | +| `evolve_skipped` | a use-case evolve leaves alone | `use_case`, `reason` (`no-design`, `source_changed`, `up_to_date`, `archived`, `unreviewed`, `held`, `declined`), `held_by?`, `next?`, `remedy?` | the reason in words; offer `next` | +| `session_complete` | the run finished; **nothing is saved yet** | `sid`, `next` (save, then discard), `no_start_page?[{test}]` | "Done. Save it to <project>?" Name tests with no start page. No `next` = it added nothing and was discarded. On a resume of a finished run whose ending was never recorded, `next` is the save alone | +| `session_ended` | the session was saved or discarded in another terminal; exit 0 | `sid`, `reason`: `saved_elsewhere`, `discarded_elsewhere`, `ended_elsewhere` | say where it went; nothing to do | +| `error` | a refusal or failure | `code?`, `message`, `remedy?`, `next?`, `held_by?`, `not_added?[]`, `final?` | §6 | +| `done` | always last | `status`, `exit_code`, `next?`, `not_updated?` (evolve) | §4, §5 | + +### 3.3 `session_paused` (exit 3) + +Every pause has `sid`, `expires_at: null` and `pending_questions[]` (empty unless questions wait); most carry `resume` (the exact command). Branch on `reason`: + +| `reason` | Means | `next` | Do | +|---|---|---|---| +| none, questions pending | the agent asks the person | resume; resume with `--answer <qid>=<n>` | the pause loop (`references/assurance.md` §2) | +| none, no questions | a forced stop (second Ctrl+C), or a stop during an evolve | resume | resume when the person wants | +| `held_back` | `--trust hold` held items | — | tell the person; nothing promises they come back on a resume | +| `connection_lost` | the connection dropped, or the ending couldn't be recorded | resume | resume once the network is back | +| `not_accepted` | the shared project refused a step | resume | resume once; then ask for help | +| `computer_slept` | this machine slept over 2 minutes | none | run `resume` when the person wants | +| `resumed_elsewhere` | the session was resumed in another terminal | none | leave it; it goes on there | +| `review_pending` | review cards wait (`pending_cards[]`) | resume with `--answer <key>=approve` … | ask per card: `approve`, `reject` or `defer` | +| `review_decided` | every card was decided, then a stop | resume | resume to finish the review | +| several sources read (`not_named?[]`, no `resume`) | an extract that read 2+ sources paused | discard; start again | it can't be resumed or saved: tell the person, discard on their word, then extract one source at a time | + +### 3.4 Save and discard + +| Event | When | Fields that matter | Tell the person | +|---|---|---|---| +| `session_save` `check` | first: what the save would land | `sid`, `project`, `adds{use_cases, acceptance_criteria, scenarios, tests, open_questions, sources, style_notes}`, `changes?`/`approves?`/`archives?`/`removes?`, `held_changes?`, `replaces?`, `marks_up_to_date?`, `other_work?`, `not_updated?[]` | "Saving adds N use-cases and M tests to <project>." Without `--yes` the command stops here (`CONFIRM_REQUIRED`): show this preview and get the person's word | +| `session_save` `saving` | the save is sent | `sid` | — | +| `session_save` `saved` | landed | `saved` (records landed; `0` = nothing to save, and the session was discarded), `test_manager`, `test_files`, `next?` (evolve what is still out of date) | "Saved to <project>", then the two blocks below | +| `session_save` `behind` | a later save changed what this session touched; exit 2, no `error` | `conflict?{kind, id, title}` or `{kind:"link", from, to}`, `next` | "Another save changed <title>; nothing was saved." Offer `next` | +| `session_save` `failed` | the project refused the save; exit 1, no `error` | `reason:"not_accepted"`, `code?` (`SCOPE_RECORD_REFUSED`) | "The project refused the save." Don't retry; ask for help | +| `session_discard` | `check` (the same counts), then `discarded` | `sid`, `adds?` | "Discarded; nothing was saved." An already-discarded session gets `discarded` alone (exit 0) | + +`test_manager` (when the project's Test Manager takes the cases): `total`, `created`, `updated`, `already_there`, `waiting` with `waiting_on[{scenario, title, reason}]` (`changed_both_sides`, `not_pushable`, `archived`, `busy`, `not_accepted`), `next_run[{scenario, title}]` (written by the next `testmd run`), `not_covered?[]`, `error?`, `support_id?`, `next?`. Tell: "N test cases created and M updated in Test Manager"; name what waits and why. + +`test_files` (design, reconcile and evolve sessions): `written[]` (paths in `.testmuai/tests/`), `refused[{test, title, reason, file?}]` (`reason` is prose; a file edited by hand is kept), `replaced?[]` (files of tests this save replaced: they stay in the folder, won't run any more, and a bare `testrun run` still picks them up; tell the person, and offer to move them out of `.testmuai/tests/`), `variables_needed?{file, variables}`, `no_start_page?[]`, `next?`; or `{written: [], refused: [], error: "TEST_FILES_NOT_WRITTEN", next}`. Test files are written **only** by the save, only in the folder where it runs. Tell: "Wrote N test files." Fill `variables_needed` before any run. + +### 3.5 Design + +| Event | When | Fields that matter | Tell the person | +|---|---|---|---| +| `gate_refused` | design entry refused: unknown, removed or not-a-use-case ref; or the use-case already has tests, or changed since | `message`, `final: true`, `next?[{cmd, why, title}]`; **no `code`** | relay `message`; offer `next` (`Redesign` = `--force`) | +| `error` `UC_UNREVIEWED` / `UC_ARCHIVED` | the use-case is proposed / archived | `message`, `final`, `next?` | §6 | +| `commit`, `usage` | as §3.2 | | "Designed N tests in this session, not saved yet" | +| `variables_declared` | the tests need variables | `file`, `variables[]` | a to-do; the stubs are this folder's only | +| `note`, `panel_resolved` | a resume with `--answer` | `text` / `id` | confirmation only | +| `session_paused`, `session_complete` | as §3.2–3.3 | | | + +### 3.6 Cover + +| Event | When | Fields that matter | Tell the person | +|---|---|---|---| +| `coverage` | `cover --mode agent` | `source: "project"`, `pct`, `usecases`, `criteria`, `perUc`, `perAc`, `flags`, `recent_run?{may_not_count_yet: true, up_to_minutes: 5}` | "Coverage is N% across <project>'s K use-cases." With `recent_run`: "a run from the last 5 minutes may not count yet" | +| `coverage` (computing) | the project is still working it out; exit 0 | `{source: "project", state: "computing"}` | run `next` (`kane-cli cover --mode agent`) in a moment | +| `done` | last | `next?`: designs to make, `testrun run` while nothing has run; `kane-cli context pull --mode agent` first (and no run) when the project has designs and this folder has no designed test files yet | offer them, adding the agent flag (§4) | + +### 3.7 `testmd run` and `testrun run` in a linked folder + +A file directly in `.testmuai/tests/` with an `assurance:` header runs on its scenario's Test Manager case. Its first run authors (about 30–85 credits per test, approximate); later runs replay. + +| Event | When | Fields that matter | Tell the person | +|---|---|---|---| +| `error` (testmd) | the case lookup or attach stopped the run | `code`, `message`, `remedy`, `support_id?`; no `next`, no `test_md_done` | §6, runs of designed tests | +| `error` `TEST_RUN_NOT_SAVED` (testmd) | before `test_md_done`; the verdict stands | as above | report the verdict; exit 1 even when steps passed | +| `testrun_plan` | the plan | `members[{path, test_id, failure?}]`, `valid` | on 0.10.0 an invalid plan other than `PROJECT_MISMATCH`, or no tests at all (`members: []`), has **no `error`**: read each `failure` | +| `error` `NO_TESTS` | *(linked folder only, from the release that carries it)* no `*_test.md` selected | `message` "no tests here yet", `next` | `next` is `kane-cli context pull --mode agent`. A filter that matches nothing: the old message and no `next`. Outside a linked folder there is no such event: the stream ends at `testrun_plan` with `members: []` and exit 2 | +| `error` `PLAN_INVALID` | *(linked folder only, from the release that carries it)* an invalid plan other than `PROJECT_MISMATCH` | `offenders[{path, failure}]`, no `next` | fix those tests or leave them out. Outside a linked folder: read `testrun_plan.members[].failure` | +| `error` `PROJECT_MISMATCH` | after `testrun_plan` | `projects[{project_id, name, this_folder, tests}]`, `remedy`, `next` | §6 | +| `testrun_member_end` (`testrun_authored_member_end` for a test's first run) | each member | `status`, `failure?{message}`, `log_path` | a member stopped on its case shows only `failure.message`; its `error` (with `code` and `remedy`) is in the member's log at `log_path`, or inside `testrun_member_event` with `--stream-members` | +| `coverage` | after the evidence ingest | `status` (`written`, `none`, `skipped`, `failed`); `written`: `computed_by: "project"`, `pct`; `none`: `reason` (`on_grid`, `project_no_answer`) | "This run reached N use-cases." `none`: the project didn't answer in time; the run is fine | +| `testrun_done` | last (local run) | `overall_status` (`passed`, `failed`, `cancelled`; `refused` after a refusal, in a linked folder only, from the release that carries it), `coverage?` | the verdict | +| `remote_error` (`--remote`) | one per member that stops before dispatch | `code`, `detail` (`<path>: <message>`), `remedy?`, `support_id?`; no `next` | nothing was sent to HyperExecute; `testrun_done{overall_status:"failed"}` and `remote_done{status:"failed", exit:2}` follow | + +## 4. `next` + +Shape: `next: [{cmd, why}]`, one runnable command per item (design walls add `title`). Tolerate a plain string item. + +Where: `error`, `session_complete`, `session_paused`, `nothing_new`, `remote_overlap.sessions[]`, `gate_refused`, `evolve_paused`, `evolve_skipped`, `child_end`, `session_save` (`saved`, `behind`, and inside `test_manager` and `test_files`), a `context list` row whose tests are out of date, and `done` (it repeats a pause's `next`). + +- **Agent flag.** Each `cmd` already carries `--mode agent` (`--agent` for `testrun run`, `testmd run`, `projects`); end a `testrun run` line in `< /dev/null`. If a `cmd` has no agent flag, add the one that command takes (§1) before running it. +- **One at a time.** Run each `cmd` as its own command. Never join them with `&&`. +- **Placeholders.** A `cmd` holding `<files>` (from `nothing_new`, a `behind` save) is a template: put in the person's real paths. +- **Read it before running it.** Some spend credits (`context extract`, `context ingest`, `design tests`, `maintain evolve`, `testmd run`, `testrun run`). Some need the person's word: `--allow-unreviewed` (designs from a use-case nobody approved), `--force` (replaces a use-case's tests), `context sessions save --yes` (publishes to the team's project and Test Manager), `context sessions discard --yes` (throws the session's work away). A `next` built from a partial `--answer` set carries the answers already given. +- **No `next`?** Read `remedy`: prose that may name a command. Add the agent flag that command takes (§1) before running it. Refusals at a folder's entry (linking, sign-in, access, `REMOTE_UNSUPPORTED`) carry `remedy` only; §6 gives the command for each. For `projects`, read stderr. + +## 5. Exit codes + +`done.status` follows the exit (`0` complete, `2` refused, `3` paused, `130` interrupted, any other `error`), with two exceptions: `context ingest` and `maintain reconcile` refusals can end `done{status:"refused", exit_code:1}`, and a failed read of `context list`, `context sessions` or `cover` ends `done{status:"error", exit_code:1}`. Branch on `exit_code`. + +| Command | 0 | 1 | 2 | 3 | 130 | +|---|---|---|---|---|---| +| `context extract` | `session_complete`, `nothing_new`, `session_ended` | the run stopped (work kept in the session), `SESSION_OPEN_FAILED`, `SESSION_NOT_ENDED`, agent failures | entry, sign-in, access, resume and `--answer` refusals | paused | stopped before a session opened | +| `context ingest` | as extract | as extract, plus `UPLOAD_STALLED`, `UPLOAD_NOT_TAKEN`, `RESULT_REFUSED`, `FETCH_FAILED`, conversion failures | file, address, conversion and source-id refusals | paused | stopped during a fetch or upload | +| `design tests` | complete | as extract | `UC_*`, `gate_refused`, `USAGE`, `STRUCTURED_*` | paused | as extract | +| `maintain evolve` | complete | some use-cases not updated (`done.not_updated`, `next`) | entry refusals, `USAGE`, `MODE_USAGE` | paused | as extract | +| `maintain reconcile` | complete; nothing to review | `DESIGN_CHILD_FAILED` | `REMOTE_UNSUPPORTED`, `SOURCE_*`, `MODE_USAGE`, `STRUCTURED_TARGET_UNKNOWN` | `review_pending` | as ingest | +| `context sessions save` | saved; nothing to save | `failed` phase, `SAVE_FAILED`, `ASSURANCE_OFF` | `CONFIRM_REQUIRED`, `behind` phase, `SESSION_*`, `CONNECTION_FAILED`, sign-in | — | — | +| `context sessions discard` | discarded; already discarded | `DISCARD_FAILED` | every other refusal | — | — | +| `context sessions`, `context list` | the array / the rows | `LIST_FAILED` | `REMOTE_UNSUPPORTED`, `ASSURANCE_OFF`, sign-in, `USAGE`, `MODE_USAGE` | — | — | +| `cover` | `coverage` (also computing) | `COVERAGE_FAILED` | `ASSURANCE_OFF`, sign-in, `REMOTE_UNSUPPORTED` (`cover gaps`) | — | — | +| `testmd run` (designed test) | passed | failed, `TEST_MANAGER_OUTDATED`, `TEST_CASE_ATTACH_FAILED`, `TEST_RUN_NOT_SAVED` | case and lookup refusals | — | — | +| `testrun run` | all passed | a member failed, including one stopped on its case | `REMOTE_UNSUPPORTED`, `PROJECT_MISMATCH`, invalid plans, every `--remote` stop before dispatch | cancelled | — | + +On the assurance verbs, `3` is a **pause** (resumable), not a failure: run the pause loop. `130` is a forced stop: resumable only if a `session_paused` arrived. On `testrun run`, `3` means cancelled. + +## 6. Refusal codes + +`Do` says what to tell or ask the person and the exact command, so you can go on when the event has no `next`. "Ask the person:" marks a step only they (or an admin) can take. Never rerun a paid command (extract, ingest, design, evolve, testmd, testrun) blindly: an exit 1 after turns ran has spent credits. `<the command>` = the refused command as typed. + +| Code | Exit | When | Do | +|---|---|---|---| +| **Linking and the folder** | | | | +| `FOLDER_USAGE` | 2 | `--project` and `--local` together | ask the person: share the work (`<the command>` without `--local`) or keep it here (`<the command>` without `--project`) | +| `FOLDER_USAGE` | 2 | `--project` or `--local` with `--resume` | `<the command>` without the folder flag; a session stays where it started | +| `FOLDER_CHOSEN` | 2 | `--project` in a folder with its own store; `--project Q` in a folder linked to P; `--local` in a linked folder | `<the command>` without `--project` / `--local` (it works in the folder's own store or project). To change that, ask the person: move `.context/` out, or remove `.testmuai/assurance.json` | +| `BOTH_STORES` | 2 | the folder has `.context/` and a link | ask the person: work locally (remove `.testmuai/assurance.json`) or in the project (move `.context/` out of the folder) | +| `PROJECT_FILE_UNREADABLE` | 2 | `.testmuai/assurance.json` can't be read | ask the person: remove `.testmuai/assurance.json`. Then `kane-cli context extract --project <id> --mode agent` | +| `PROJECT_FILE_UNWRITABLE` | 2 | the link couldn't be written | ask the person: make the folder writable, or move a file named `.testmuai` out of it | +| `PROJECT_NO_ACCESS` | 2 | `--project <id>` names no project of theirs | `kane-cli projects list --search <name> --agent`; confirm the id with the person; run `<the command>` with it | +| `PROJECT_NO_ACCESS` | 2 | no access to the folder's project | ask the person: a Test Manager admin grants their account access | +| `PERMISSION_REQUIRED` | 2 | the project's first session needs permission to create test cases | ask the person: an admin grants it, or someone who has it starts the first session. A link made in this run was removed | +| `ASSURANCE_OFF` | 2 (1 from `sessions save`) | assurance isn't available for the project | ask the person; no command | +| `REMOTE_UNSUPPORTED` | 2 | `context review`, `context explain` | `kane-cli context list --mode agent` (the use-cases, their status, scenarios and tests) | +| `REMOTE_UNSUPPORTED` | 2 | `cover gaps` | `kane-cli cover --mode agent` | +| `REMOTE_UNSUPPORTED` | 2 | `context sessions show`, `context sessions clean` | `kane-cli context sessions --mode agent` | +| `REMOTE_UNSUPPORTED` | 2 | `--mode ci` on extract or design (a stderr line) | `<the command>` with `--mode agent` | +| `REMOTE_UNSUPPORTED` | 2 | `context ingest --plan` or `--mode override`; `context list --all`, or a `--type` other than `source`, `usecase`, `scenario`, `test` | `<the command>` without that flag, with `--mode agent` | +| `REMOTE_UNSUPPORTED` | 2 | `maintain reconcile --plan`, `--apply`, `--mode ci`, or no `--source-id`; `--from <address>` | `kane-cli maintain reconcile --from <file> --source-id <id> --mode agent`; for an address, `kane-cli context ingest <address> --mode agent` | +| `REMOTE_UNSUPPORTED` | 2 | `context sync`, `push`, `clone`, `fsck`, `view`, `retire`, `revert`, `rebuild`, `name`, `design explain` (and `context pull` where its help has no `--project`: `shared-projects.md` §7b) | no command in a linked folder; tell the person these work on a folder's own store. The project's tests come with `kane-cli context pull --mode agent` (`shared-projects.md` §7) | +| `PULL_LOCATION_LINKED` | 2 | `context pull` with a location name, `--rebase`, `--yes` or `--answer` in a linked folder | `kane-cli context pull --mode agent` | +| `REMOTE_UNSUPPORTED` | 2 | `testrun run --from-context` | run `next`, or `kane-cli testrun run <test files> --agent` | +| **Sign-in and access** | | | | +| `CREDENTIALS_MISSING` | 2 | nobody signed in | start sign-in (`kane-cli login --oauth`, `references/ready-check.md`), then `<the command>` | +| `CREDENTIALS_MISSING` | 2 | "couldn't confirm your login for shared projects" (a browser sign-in with no access key) | `<the command>` once more; if it refuses again, ask the person: run `kane-cli login --username <user> --access-key <key>` in their own terminal | +| `CREDENTIALS_REJECTED` | 2 | the shared project rejected the credentials | sign in again (`kane-cli login --oauth`), then `<the command>` | +| `CONNECTION_FAILED` | 2 | couldn't connect (or for 2 minutes); nothing was started | check the network; `<the command>` once more | +| `PREFLIGHT` | 2 | kane-cli's check before a session opens failed: unreachable, sign-in unchecked or rejected, or "assurance sessions aren't available here yet" | unreachable: `<the command>` once more later. Sign-in: `kane-cli login --oauth`. Not available: stop and pass on the support id in `remedy` | +| `VERSION_MISMATCH` | 2 | this kane-cli is too old, or the shared project doesn't support it yet | too old: ask the person to update (`npm install -g @testmuai/kane-cli@latest`). Not supported yet: stop and pass on the support id in `remedy` | +| `PROJECT_BUSY` | 2 | a live session holds the project, use-case or source: `held_by{sid, verb, use_case, mine, status}` | no `next`. When `remedy` names a resume (`kane-cli context extract --resume <sid> --mode agent`, or `kane-cli design tests --resume <sid> --mode agent`; only for the person's own session, `mine: true`), running it takes the session over and stops it in the other terminal, so ask the person first; otherwise wait for that run to finish. When `remedy` only says to wait (a resume refused this way, or a design while the person's own extract holds the project), wait. Someone else's: tell the person a teammate's session holds it; wait; `<the command>` later | +| **Sessions** | | | | +| `SESSION_NOT_FOUND` | 2 | no session with that id is theirs (a colleague's can't be resumed or saved) | `kane-cli context sessions --mode agent`; use a `sid` whose `owner` is `you` | +| `SESSION_FINISHED` | 2 | resume of a finished session | on the person's word: `kane-cli context sessions save <sid> --yes --mode agent` | +| `SESSION_NOT_FINISHED` | 2 | save of an unfinished session | resume first: `kane-cli context extract --resume <sid> --mode agent` (`design-…`, `evolve-…` ids: `kane-cli design tests --resume <sid> --mode agent`); save after `session_complete` | +| `SESSION_NOT_ENDED` | 1 | the run finished but the project didn't take its last step; the work is safe | run `next` (the resume command above): it records the ending; then save | +| `SESSION_FAILED` | 2 | the session ended with an error; it can't be resumed or saved | start again: `kane-cli context extract --mode agent`, or `kane-cli design tests --use-case <uc> --mode agent` | +| `CHECKPOINT_PROTOCOL`, `SESSION_KEPT_NOTHING`, `SESSION_MISSING` | 2 | a resume can't read the session's last step | tell the person it can't be resumed; start a new one: `kane-cli context extract --mode agent` or `kane-cli design tests --use-case <uc> --mode agent` | +| `SESSION_OPEN_FAILED` | 1 | the session couldn't be opened | `<the command>` once more; then ask for help | +| no code: "stopped before it finished" | 1 | the run stopped; the work so far is kept in the session | resume with the `sid` from `remote_connect`: `kane-cli context extract --resume <sid> --mode agent` (design: `kane-cli design tests --resume <sid> --mode agent`). Don't start over | +| `CONFIRM_REQUIRED` | 2 | save or discard without `--yes` | the `check` line before it is the preview: show it; on the person's word, `kane-cli context sessions save <sid> --yes --mode agent` (or `discard`) | +| `SESSION_ELSEWHERE` | 2 | discard while the session runs in another terminal, or is held | wait (up to 3 minutes) or stop the other run; then `kane-cli context sessions discard <sid> --yes --mode agent` | +| `ALREADY_SAVED`, `ALREADY_ENDED` | 2 | discard of a saved or ended session | nothing to discard; tell the person | +| `DISCARD_FAILED` | 1 | the discard failed | stop; ask for help | +| `LIST_FAILED` | 1 | the session list or `context list` couldn't load | `<the command>` once more; then ask for help | +| `STRUCTURED_FLAGS_USAGE` | 2 | `--answer` without `--resume <sid>` or `--mode agent`, or unreadable | `kane-cli context extract --resume <sid> --mode agent --answer <id>=<value>` (design: `kane-cli design tests --resume <sid> …`) | +| `STRUCTURED_TARGET_UNKNOWN` | 2 | `--answer` names no pending question or card, isn't a verdict, or repeats a card | use the ids from `pending_questions` or `review_cards` | +| `USAGE` | 2 | a parse error; save or discard with no `<sid>`; evolve with both or neither of `<ref>` and `--from-stale`; design with no `--use-case` | fix the command (`kane-cli <command> --help`); never resend it unchanged | +| `MODE_USAGE` | 2 | `--mode` isn't one the command takes | `<the command>` with `--mode agent` | +| **Extract and ingest inputs** | | | | +| `NO_PROPOSAL` | 1 | the agent returned no use-cases | retry `kane-cli context extract --mode agent` once (it reads the source again and costs credits again). If it happens twice, stop, and give the person the command and the message | +| `AS_SINGLE_SOURCE` | 2 | `--as` with several refs | run `next`, or `kane-cli context ingest <ref> --as <id> --mode agent` once per source | +| `SAME_SOURCE_ID` | 2 | two refs would share one source id | run `next`; with none, ask the person for distinct ids and run `kane-cli context ingest <ref> --as <id> --mode agent` per ref | +| `NEW_VERSION_ALONE` | 2 | a new version of a held source among other refs; nothing added | `kane-cli context ingest <the other refs> --mode agent`, then `kane-cli context ingest <new version> --mode agent` alone (it opens a review) | +| `SITE_MOVED`, `KIND_MISMATCH`, `WEB_URL_CHANGED`, `UUID_MOVED` | 2 | the fetched item can't become a new version of the source id it would replace | ask the person which they mean; run that `next`, or `kane-cli context ingest <address> --as <new id> --mode agent` for a source of its own | +| `UNSUPPORTED_URL` | 2 | the ref is neither a file nor a web address | ask the person for a file path or a full `https://` address | +| `FILE_NOT_FOUND`, `FILE_UNREADABLE`, `FILE_TOO_LARGE`, `UNSUPPORTED_MEDIA`, `MEDIA_SNIFF_MISMATCH`, `BAD_SOURCE_ID` | 2 | checked before anything was sent | fix the path or file (relay the message); `BAD_SOURCE_ID`: pass a valid `--as <id>` | +| `PDF_ENCRYPTED`, `PDF_NO_TEXT_LAYER`, `PDF_UNREADABLE`, `DOCX_UNREADABLE`, `DOCX_NOT_WORDPROCESSING`, `DOCX_NO_TEXT`, `DOCX_ENCRYPTED_OR_LEGACY`, `DOCX_PART_TOO_LARGE`, `DOCX_TOO_MANY_PARTS`, `DOCX_BOMB`, `PROJECTION_TOO_LARGE` | 2 | the file can't be converted on this machine | relay the message; ask the person for an unlocked, text-bearing or smaller copy | +| `SOURCE_ID_OWNED` | 2 | that source id is taken | ask the person for another id; `kane-cli context ingest <ref> --as <id> --mode agent` | +| `INGEST_FAILED`, other conversion failures | 1 | the conversion failed | report it; ask for help | +| `INVALID_URL`, `ADDRESS_NOT_ALLOWED`, `NOT_A_PAGE`, `PAGE_NOT_FOUND`, `PAGE_BLOCKED`, `PAGE_PAYWALLED`, `PAGE_TOO_LARGE`, `PAGE_TIMEOUT`, `PAGES_UNAVAILABLE`, `PAGES_NOT_AVAILABLE`, `ITEM_UNAVAILABLE`, `LINEAR_PAGE`, `LINEAR_LINK_MALFORMED`, `ATTACHMENT_TOO_LARGE` | 2 | the address or item can't be read | relay the message; ask the person for another address or a saved file | +| `NO_CONNECTION`, `NO_ACCESS`, `NO_ISSUE_ACCESS`, `NO_DOCUMENT_ACCESS`, `NO_CONFLUENCE_ACCESS`, `NO_COMMENTS_ACCESS`, `PROJECT_NOT_SELECTED`, `INTEGRATION_SETUP`, `INTEGRATION_UNAVAILABLE` | 2 | the Jira, Confluence or Linear connection is missing or can't see the item | ask the person: connect the integration or grant access (relay the message) | +| `INTEGRATION_BUSY` | 2 | the integration is busy | wait; `<the command>` once more | +| `OUT_OF_CREDITS` | 2 | no credits left for the fetch | tell the person (`kane-cli balance`); stop | +| `FETCH_FAILED` | 1 | the fetch failed | `<the command>` once more; then ask for help | +| `PLUGIN_NOT_CLAIMED`, `PLUGIN_SOURCE_ID_INVALID`, `PLUGIN_MISSING` | 2 | no source plugin takes the ref, or its id is invalid | ask the person: check the ref's form and that the plugin is installed | +| `PLUGIN_HOOK_FAILED`, `PLUGIN_HOOK_TIMEOUT` | 1 | the source plugin failed | report it; ask for help | +| `UPLOAD_STALLED`, `UPLOAD_EXPIRED` | 1 / 2 | the upload stopped, or its window ended | `<the command>` once more | +| `UPLOAD_NOT_TAKEN`, `RESULT_REFUSED` | 1 | the shared project didn't take the upload or its result | stop; ask for help | +| `LINKS_NOT_TAKEN` | 2 | the shared project can't take addresses | ask the person for a saved file of the page; `kane-cli context ingest <file> --mode agent` | +| `SOURCE_UNKNOWN`, `SOURCE_REMOVED` | 2 | reconcile: the project doesn't hold that `--source-id`, or removed it | `kane-cli context list --type source --mode agent`; a new source: `kane-cli context ingest <file> --mode agent` | +| `SOURCE_ORIGIN_UNKNOWN`, `SOURCE_REF_UNROUTED`, `SOURCE_ID_MISMATCH` | 2 | reconcile of a plugin source that doesn't line up | relay the message; check the `--source-id` with the person (`kane-cli context list --type source --mode agent`) | +| on `card_refused`: `CONCURRENT_APPEND`, `TARGET_RETIRED`, `SOURCE_MOVED`, `SOURCE_UNRESOLVED`, `ARCHIVE_UNRESOLVED`, `ALREADY_VERDICTED`, `CARD_GONE`, `HOLD_GONE` | — | a review verdict didn't land | relay `refusal`; `kane-cli context extract --resume <sid> --mode agent` shows the cards still open | +| `DESIGN_CHILD_FAILED` | 1 | an approved change's design run failed | later: `kane-cli design tests --use-case <uc> --mode agent` | +| `AGENT_FAILED`, `AGENT_CRASHED`, `REPAIR_FAILED`, `PROPOSAL_INVALID`, `COMMIT_FAILED` | 1 | the agent's run failed | report it; credits were spent; check `kane-cli context sessions --mode agent` before any retry | +| `TRUST_USAGE` | 2 | `--trust hold` in a terminal run | `<the command>` without `--trust`, with `--mode agent` | +| **Design** | | | | +| `UC_UNREVIEWED` | 2 | the use-case is still proposed; nobody approved it | show it (`kane-cli context list --type usecase --mode agent`); on the person's word, `kane-cli design tests --use-case <uc> --allow-unreviewed --mode agent` (one that already has tests also needs `--force`: the `gate_refused` row below). A person approves at the end of a terminal session | +| `UC_ARCHIVED` | 2 | the use-case is archived | tell the person; pick another use-case | +| `gate_refused` (no code) | 2 | unknown, removed or not-a-use-case ref | pick from the ids the message lists (`kane-cli context list --type usecase --mode agent`) | +| `gate_refused` (no code) | 2 | the use-case already has tests, or changed since they were designed | ask the person: redesign (`kane-cli design tests --use-case <uc> --force --mode agent`, replaces its tests) or update (`kane-cli maintain evolve <uc> --mode agent`, only what changed). A use-case that is also still **proposed** needs both flags on a redesign: `kane-cli design tests --use-case <uc> --force --allow-unreviewed --mode agent` (each alone refuses with the other wall); evolve refuses it (`UC_UNREVIEWED`), so use the redesign | +| `NOTHING_CHANGED` | 2 | evolve of a use-case unchanged since its tests were designed | ask the person why: `kane-cli maintain evolve <uc> --because "<reason>" --mode agent`; or run `next` | +| `NO_TESTS_YET` (`evolve_skipped` `no-design`) | 2 | evolve of a use-case with no tests | `kane-cli design tests --use-case <uc> --mode agent` | +| `evolve_skipped` `source_changed` | 2 | only its source changed | run `next`; a file source has none: `kane-cli maintain reconcile --from <file> --source-id <id> --mode agent` | +| **Save** | | | | +| `session_save` `phase: "behind"` (no `error` event; a terminal shows `PROJECT_CHANGED`) | 2 | another save changed what this session touched | tell the person what (`conflict`); on their word run one `next` (start again from the latest) | +| `session_save` `phase: "failed"` (`reason`, maybe `code: "SCOPE_RECORD_REFUSED"`; a terminal shows `SAVE_FAILED` with a support id) | 1 | the project refused the save | saving again won't help: stop, and give the person the `reason` and the session id | +| `CONNECTION_FAILED` | 2 | the save's answer was lost after sending | `<the command>` once more: nothing is saved twice | +| `TEST_MANAGER_TIMED_OUT`, `TEST_MANAGER_UNREACHABLE`, `TEST_MANAGER_FAILED` (`test_manager.error`) | 0 | the save landed; Test Manager wasn't fully updated | run `test_manager.next`, or `kane-cli context sessions save <sid> --mode agent` (saving again finishes it) | +| `PROJECT_NO_ACCESS`, `ASSURANCE_OFF` (`test_manager.error`) | 0 | the save landed; Test Manager refused the cases | ask the person: their Test Manager access, or assurance for the project | +| `TEST_FILES_NOT_WRITTEN` (`test_files.error`) | 0 | the save landed; no test files were written here | run `test_files.next`, or `kane-cli context sessions save <sid> --mode agent` | +| `test_files.refused[]` | 0 | a test file wasn't written; `reason` says why: edited by hand and kept, a duplicate, an unreadable or unresolved record, an unwritable folder, a missing import (some have no `file`) | tell the person the reason; follow `test_files.next` when it is there. Only for a file edited by hand: when `test_files.next` offers `kane-cli testmd run <file> --agent`, it sends their edit to its case; otherwise (or instead) they move the file out and save again | +| **Runs of designed tests** | | | | +| `TEST_CASE_MISSING` | 2 | the session that designed the test isn't saved | on the person's word: `kane-cli context sessions save <sid> --yes --mode agent` (`kane-cli context sessions --mode agent` lists the sids) | +| `TEST_CASE_GONE` | 2 | its Test Manager case was deleted or moved | show the person; on their word, `kane-cli context sessions save <sid> --yes --mode agent` with the sid `remedy` names (it makes a new case). A `remedy` with no sid: `kane-cli context sessions --mode agent` to find it | +| `TEST_CASE_ARCHIVED` | 2 | its case is archived in Test Manager | ask the person: restore the case, or leave the test out of the run | +| `TEST_NOT_IN_PROJECT` | 2 | the test isn't in the folder's project, its file is out of date, or the project has no work yet | run it in the folder linked to its project; or, on the person's word, design it again: `kane-cli design tests --use-case <uc> --mode agent` (a proposed use-case also needs `--allow-unreviewed`, one with tests `--force`: the Design rows), then save; a project with no work: `kane-cli context ingest <files> --mode agent` | +| `TEST_MANAGER_OUTDATED` | 1 (2 on `--remote`) | Test Manager can't add this run to the test's case: on a device target (it attaches desktop-browser runs only), or because Test Manager itself needs an update (or made a case of its own) | stop and tell the person. On a device target, offer the desktop browser; otherwise the run can only go again once Test Manager is updated. Pass on the support id | +| `TEST_CASE_ATTACH_FAILED` | 1 | attaching to the case failed | `<the command>` once more later; then pass on the support id | +| `TEST_RUN_NOT_SAVED` | 1 | the run ran but its version wasn't saved on the case | report the verdict as it stands; tell the person Test Manager has no record of this run; pass on the support id | +| `CREDENTIALS_MISSING`, `CREDENTIALS_REJECTED`, `PROJECT_NO_ACCESS`, `ASSURANCE_OFF`, `CONNECTION_FAILED` | 2 | a first run couldn't look up its case | as the sign-in and access rows above; a later run goes on without the lookup | +| `REMOTE_KANE_NOT_PUBLISHED` | 2 | `testrun run --remote` with designed tests from a kane-cli that isn't a published release | ask the person to install a published kane-cli; `remedy` names the flags that pick one | +| **testrun plans** | | | | +| `PROJECT_MISMATCH` | 2 | the selected tests belong to more than one Test Manager project | show `projects[]` (`this_folder: true` marks the folder's project); run `next`, or `kane-cli testrun run <that project's test paths> --agent`. Keep `kane-cli run` experiments in another folder | +| invalid plan (no `error`: outside a linked folder, or on 0.10.0) | 2 | `testrun_plan.valid` is false | read each `members[].failure`; fix those tests or leave them out | +| `PLAN_INVALID` | 2 | *(linked folder only, from the release that carries it)* the same, as an `error` with `offenders[{path, failure}]`, then `testrun_done{refused}` | fix the named tests or leave them out | +| no tests (`members: []`) | 2 | nothing selected (everywhere on 0.10.0; outside a linked folder on every release) | in a linked folder: get the project's tests (`shared-projects.md` §7); elsewhere design them (`kane-cli context list --mode agent`, then `kane-cli design tests --use-case <id> --mode agent`) | +| `NO_TESTS` | 2 | *(linked folder only, from the release that carries it)* "no tests here yet", then `testrun_done{refused}` | run its `next`: `kane-cli context pull --mode agent`. A filter matching nothing has no `next`: change the filter | +| **Coverage** | | | | +| `COVERAGE_FAILED` | 1 | the project's coverage couldn't be read | `kane-cli cover --mode agent` once more in a moment; then ask for help | +| `coverage` computing | 0 | still being worked out | `kane-cli cover --mode agent` in a moment | + +## 7. Support ids + +A support id looks like `as-YYYYMMDDTHHMM-xxxxxxxx`. It names one command's attempt for TestMu AI support. + +| Where | What shows | +|---|---| +| Terminal | under a refusal: `if this keeps failing, share this id with support: <id>` | +| Assurance stream | **not** on `error` events. Only inside `remedy` for `VERSION_MISMATCH` (not supported yet), `PREFLIGHT` (not available here) and `SAVE_FAILED`, and in `session_save.test_manager.support_id` (a test that couldn't be read) | +| Run stream | `testmd run`: `error.support_id`; `testrun run --remote`: `remote_error.support_id`; a member's `error` inside `testrun_member_event` (`--stream-members`) | + +When you ask the person to get help, give them the support id, the command, the code and the message. With no id on the stream, give the rest; never rerun a paid command just to get an id. With `KANE_TELEMETRY=0` the id still shows, but nothing is recorded under it: say so. diff --git a/.claude/skills/kane-cli/references/assurance-parsing.md b/.claude/skills/kane-cli/references/assurance-parsing.md index 0ab581c..66e4625 100644 --- a/.claude/skills/kane-cli/references/assurance-parsing.md +++ b/.claude/skills/kane-cli/references/assurance-parsing.md @@ -2,6 +2,11 @@ # Assurance NDJSON — Wire Contract +> In a folder linked to a shared project (kane-cli 0.10.0+) the same commands add their own events (for example +> `folder_linked`, `nothing_new`, `remote_*` progress, `session_save`, `session_discard`, `session_ended`, `pull_test`) +> and refusal codes, and `session_complete` there carries the save's `next`. They are in `agent-contract.md` §3 and §6; +> this page covers the local store. + `kane-cli context extract --mode agent`, `kane-cli context ingest … --mode agent`, and `kane-cli design tests --mode agent` speak a versioned NDJSON vocabulary on stdout — one JSON object per line, envelope `{"type": "<name>", "v": 1, "verb": "extract"|"design", ...}`. On 0.7.2+ the stream is **strict**: stdout carries only NDJSON and stderr stays silent (crash traces excepted); on 0.7.1, prose diagnostics could ride stderr. `maintain reconcile --mode agent` (verb `reconcile`) and `cover`/`cover gaps --mode agent` (verbs `cover`/`gaps`) share the envelope with their own event sets (below). **The vocabulary is open**: new event types and fields may appear in any release — tolerate unknowns, never fail on them. @@ -19,7 +24,7 @@ for each line: else → per-type handling below ``` -A stream that ends **without** `done` means the process crashed — outcome unknown; inspect `context sessions --json` and `context list` before retrying anything paid. One designed exception: `kane-cli context review --mode agent` keeps its `--json` shape — verdict and drift listings are one JSON object per line with no `done` (an empty listing prints nothing, exit `0`); its early refusals — no store, an open rebase, a verdict queue that needs a terminal (`TTY_REQUIRED`) — are typed events followed by `done`; its later refusals exit without `done`. Read its exit code instead. One version-scoped rule for merged-ingest landing failures (bad path, unsupported media, refused URL, a misused `--mode` or `--as`): on 0.7.2+ they ride the stream as `error` (codes `MODE_USAGE`, `AS_SINGLE_SOURCE`, `UNSUPPORTED_URL`, `INGEST_FAILED`) + `done` — the guarantee covers the whole invocation; on 0.7.1 they end with prose + exit `1`/`2` **before any NDJSON begins** — no stream at all is a refusal to fix, not a crash, and the guarantee starts with the extraction stream. +A stream that ends **without** `done` means the process crashed — outcome unknown; inspect `context sessions --json` and `context list` before retrying anything paid. No stream at all, with one line on stderr and exit `2`, is a command-line refusal before anything ran (for example `--json` with `--mode agent`): fix the command, don't treat it as a crash. One designed exception: `kane-cli context review --mode agent` keeps its `--json` shape — verdict and drift listings are one JSON object per line with no `done` (an empty listing prints nothing, exit `0`); its early refusals — no store, an open rebase, a verdict queue that needs a terminal (`TTY_REQUIRED`) — are typed events followed by `done`; its later refusals exit without `done`. Read its exit code instead. One version-scoped rule for merged-ingest landing failures (bad path, unsupported media, refused URL, a misused `--mode` or `--as`): on 0.7.2+ they ride the stream as `error` (codes `MODE_USAGE`, `AS_SINGLE_SOURCE`, `UNSUPPORTED_URL`, `INGEST_FAILED`) + `done` — the guarantee covers the whole invocation; on 0.7.1 they end with prose + exit `1`/`2` **before any NDJSON begins** — no stream at all is a refusal to fix, not a crash, and the guarantee starts with the extraction stream. ## Events (extract / design) @@ -93,12 +98,12 @@ Use `text` + `options[].label` + `recommended_index` + `rationale` to decide or | `child_start` / `child_end` | around the design run, after a proceed | `child: "design"`, `subject` / `ok`, `dur_s`, `exit_code?` | | *(the design run's events)* | between the two, stamped `verb: "design"` | every event of `design tests --mode agent` except its `done`; a paused design session arrives as `session_paused` with `sid`, `resume`, `next[]` | | `evolve_diff` | after a clean design run | `use_case`, `generation` (`before`, `after`), `superseded[]`, `minted[]`, `retained[]` (pairs as `scenario` + `test`), `moves[]` (`ac`, `before[]`, `after[]`) | -| `evolve_skipped` | a use-case not evolved | `use_case`, `reason`: `declined`, or `no-design` in a `--from-stale` batch | +| `evolve_skipped` | a use-case not evolved | `use_case`, `reason`: `declined`, `no-design` in a `--from-stale` batch, or `source_changed` (out of date only because its source changed; `next?` reviews that change with `kane-cli maintain reconcile`, `remedy?`) | | `error` | a refusal or failure | `message`, `code?`: `MODE_USAGE` (only `interactive` and `agent` are accepted), `ANSWER_USAGE` (an answer to a question the run does not ask, or a value that is neither option) | | `done` | last | `status`, `exit_code`, `next?`: `complete` 0, `refused` 2, `paused` 3, `error` with the design run's own code; `next[]` carries every paused session's resume, uncapped | - **Answering.** `--answer evolve:<use-case>=1` (or `=proceed`) and `--answer evolve:<use-case>=2` (or `=decline`), repeatable, with `--mode agent`. A refused answer is `error` + `done` with nothing before it. -- **Refusals.** An explicit target that is fresh and has no `--because`, or that was never designed, exits `2`. A `--from-stale` run with nothing stale exits `0`. +- **Refusals.** An explicit target that is fresh and has no `--because`, that was never designed, or that is out of date only because its source changed (`evolve_skipped` `source_changed`; `--because` lets it run), exits `2`. A `--from-stale` run with nothing stale exits `0`, and one whose every stale use-case is source-changed sends those `evolve_skipped` events and exits `0`. - **Batches.** Every use-case runs whatever the earlier ones did. The exit is the first design-run code that is neither 0 nor 3; otherwise 3 if any paused; otherwise 0. ## Sync events (verb `sync`) *(0.8.14+)* diff --git a/.claude/skills/kane-cli/references/assurance.md b/.claude/skills/kane-cli/references/assurance.md index 7295995..efbe6ea 100644 --- a/.claude/skills/kane-cli/references/assurance.md +++ b/.claude/skills/kane-cli/references/assurance.md @@ -6,7 +6,7 @@ When the user has **requirements** — a PRD, a spec, acceptance notes — and w - **Assurance** = tests derived from the actual documents, every claim cited, every test permanently tagged with the acceptance criteria it verifies, coverage measured against requirements. Use it whenever the user cares about "what exactly is covered, and how do we know?" -Everything here works over a local store (`.context/` in the project directory) that the commands create and manage themselves. +Everything here works over a local store (`.context/` in the project directory) that the commands create and manage themselves. **A folder linked to a team's shared project** (it holds `.testmuai/assurance.json`; kane-cli 0.10.0+) works in the project instead, and its journey differs: no `context review` checkpoint, every session is saved explicitly, and `cover gaps` is refused. Follow `shared-projects.md` §4 there; the sections below describe the local store: a linked folder's events and refusal codes are in `agent-contract.md`, and the options it refuses in `shared-projects.md` §8. In agent mode a folder that isn't set up yet goes local unless you pass `--project <id>`, so settle where the work goes first (`shared-projects.md` §1). **Version gate — check before improvising.** The assurance commands exist on kane-cli **0.6.1 and later**; flags and events marked with a release (**0.7.1+** … **0.8.14+**) below need at least that release. On an older CLI, `kane-cli context …` fails as an *unknown command* (exit 2 with a "did you mean" suggestion) — that error means the CLI is too old, not that you typed it wrong. Confirm with `kane-cli --version`, tell the user to update (`npm install -g @testmuai/kane-cli`, or `brew upgrade kane-cli`), and stop — do not try to reproduce the workflow with other commands. @@ -19,7 +19,7 @@ kane-cli design tests --use-case <uc-ref> --mode agent --max 8 # 3. design ACs kane-cli context review --verdicts <file> --json # 4. CHECKPOINT: user approves the design kane-cli testmd run .testmuai/tests/<t>_test.md --agent # 5. author each kept test once (real browser) kane-cli testrun run --match 't-' < /dev/null # 6. batch replays from then on -kane-cli cover gaps # 7. designed % × proven % + per-use-case debt +kane-cli cover gaps # 7. the coverage figure, use-cases with tests, per-use-case debt kane-cli maintain reconcile --from <new.md> --source-id <id> --mode agent # when a source changes (§11) ``` @@ -30,7 +30,7 @@ Two hard rules: 1. **The two checkpoints are the user's, not yours.** Everything the extract and design agents emit is *derived* (unreviewed) until a human decision promotes it — see §4. Do not start authoring (step 5) until the user has approved the design output or explicitly told you to proceed. 2. **One step at a time, one store-writer at a time.** Never run two store-mutating commands concurrently (`ingest`, `extract`, `review` with verdicts, `design`, `reconcile`, `evolve`) — the store is single-writer (§8), and extraction additionally holds a store-wide lock (a concurrent extract refuses `EXTRACT_LOCKED`, exit 2 — wait, never delete locks). -Extract, design, reconcile and evolve call the KaneAI service and consume credits; everything else is local and free. +Extract, design, reconcile and evolve call the KaneAI service and consume credits. URL sources are fetched server-side, and a web page's ingest is refused when the organisation is out of credits. Everything else is local and free. ## 2. The pause loop — exit 3 is a pause, NOT a failure @@ -49,7 +49,7 @@ Three ways to resume: kane-cli context extract --resume <sid> --mode agent --message "Account required — the update section supersedes the older text" # by id (0.7.1+) — for precise scripted answers; each echoes a panel_resolved event: -kane-cli context extract --resume <sid> --mode agent --answer q1=1 --answer q2="https://staging.example.com" +kane-cli context extract --resume <sid> --mode agent --answer q1=1 --answer q2="https://app.example.com" # land a new source instead of answering (0.7.1+) — the batch defers (ask_deferred), the agent # reads the source and re-asks only what's still open: @@ -57,7 +57,7 @@ kane-cli context extract --resume <sid> --mode agent --with-source ./addendum.md ``` - If the answer leaves any question standing, the run pauses again with refreshed questions — repeat. -- Sessions live 24 hours. Inspect without contending a live run: `kane-cli context sessions --json` (all resumable sessions + their resume commands) and `kane-cli context sessions show <sid> --json` (the pending questions in wire shape). A session written by a NEWER kane-cli than the installed one lists as unknown with no resume command (0.7.1+) — that means "upgrade to resume it", not corruption. If you abandon a session deliberately, remove it with `kane-cli context sessions clean <sid>` — bare `clean` only collects *expired* sessions, and `--all` is a purge that needs explicit user authorization. +- Sessions live 24 hours. Inspect without contending a live run: `kane-cli context sessions --json` (all resumable sessions + their resume commands) and `kane-cli context sessions show <sid> --json` (the pending questions in wire shape). A session written by a NEWER kane-cli than the installed one lists as unknown with no resume command (0.7.1+) — that means "upgrade to resume it", not corruption. If you abandon a session deliberately, remove it with `kane-cli context sessions clean <sid>` — bare `clean` only collects *expired* sessions, and `--all` is a purge that needs explicit user authorization. (A linked folder refuses `sessions show` and `sessions clean`: `kane-cli context sessions --mode agent` lists the project's sessions there.) Full event schema: `references/assurance-parsing.md`. @@ -86,11 +86,11 @@ Remote ids share ONE space (0.8.6+): a URL whose id is already backed by a diffe Re-ingesting changed bytes under the same id **versions** the source and marks everything derived from the old snapshot stale — but for a changed document, prefer `maintain reconcile` (§11), which does the version move AND triages the fallout. -Extract behavior to expect (0.7.1+): the sweep **continues past a per-source failure** (one line each; exit 1 at the end if anything failed — the failed source simply retries next run, no `--force` needed). `--trust hold` holds everything new for the user's review instead of committing (headless-only; `--mode ci` refuses the flag entirely; use it one source at a time — resuming held work from a multi-source run is not yet supported). The agent cites every claim (fabrication is rejected before write), asks when the source contradicts itself, and commits proposals as `derived`. Watch the `commit` event and say it in plain language ("5 use-cases extracted from the PRD"). +Extract behavior to expect (0.7.1+): the sweep **continues past a per-source failure** (one line each; exit 1 at the end if anything failed — the failed source is read again on the next extract, which costs credits again; no `--force` needed). `--trust hold` holds everything new for the user's review instead of committing (headless-only; `--mode ci` refuses the flag entirely; use it one source at a time — resuming held work from a multi-source run is not yet supported). The agent cites every claim (fabrication is rejected before write), asks when the source contradicts itself, and commits proposals as `derived`. Watch the `commit` event and say it in plain language ("5 use-cases extracted from the PRD"). ## 4. Review — trust is the user's decision -Promotion from `derived` to `trusted` **always requires explicit user confirmation** — even when the user asked for an "end-to-end" run. End-to-end authorizes completing the workflow, not making product-requirement judgements. +Promotion from `derived` to `trusted` **always requires explicit user confirmation** — even when the user asked for an "end-to-end" run. *(In a shared project there is no review command: `context review` and `context explain` are refused in a linked folder, and use cases an agent run saves stay proposed. Run the checkpoint as `shared-projects.md` §4 says: show them, ask once for the batch, then design with `--allow-unreviewed` on a yes.)* End-to-end authorizes completing the workflow, not making product-requirement judgements. **Present the material, not the counts.** The `commit` event carries counts and ids only — to run a checkpoint, enumerate what's waiting with `kane-cli context list --json --inferred` (one JSON row per unreviewed node) and pull any item's full content, citations, and history with `kane-cli context explain <ref> --json`. Present titles, descriptions, and the cited evidence; collect the user's decisions; build the verdicts from those same refs. The same recipe runs the design checkpoint. Then land the decisions atomically: @@ -150,20 +150,22 @@ So a failing authored run is never, by itself, a reason to rewrite the step. Cla ## 7. What's next — let the tool tell you ```bash -kane-cli cover # the pack audit: what it proved, plus the live-graph completeness worklist -kane-cli cover gaps # the coverage ribbon: one row per use-case, both axes +kane-cli cover # one coverage figure over the use-cases; --from <pack> reads one run's own figure +kane-cli cover gaps # the coverage ribbon: that figure, then one row per use-case (designed count, coverage, debt) kane-cli cover gaps <uc-id> # 0.8.2+: the dossier — every AC of that use-case + next actions -kane-cli cover gaps --json # the nested document (designed axis, proven axis, per-UC pending rows with ready_command) +kane-cli cover gaps --json # the nested document (the figure, designed counts, per-UC pending rows with ready_command) kane-cli cover gaps --mode agent # 0.7.1+: same data as ONE `gaps` event + done.next[] ready-commands ``` -On 0.8.2+ the default `cover gaps` output is the **coverage ribbon** — a high-level band-table: one severity-ordered row per use-case with designed/proven bars and per-class debt counts, deliberately carrying NO commands. To act, drill in: `cover gaps <uc-id>` (the dossier) renders every AC of that use-case plus a `next` actions block — evidence first; surface its warning that re-running an authored test on a broken app may heal the test around the failure — or read `--json`, whose per-UC pending rows keep `ready_command`. `--rollup lenient|strict` selects the proven-axis formula. BREAKING at 0.8.2: `--flat` and gaps' `--from` are REMOVED and refuse (pre-0.8.2 CLIs still have the flat worklist). With a `<uc-id>`, `--json` and agent mode close the document over that use-case. +**In a shared project** (a linked folder): `cover gaps` is refused. `kane-cli cover --mode agent` gives the project's own figure, one `coverage` event then `done`. A `coverage` event with `state: "computing"` and no `pct` means the project is still working its figure out: say so, and run `done.next` again in a minute. A run less than 5 minutes old may not count yet (`recent_run.may_not_count_yet: true`); say so and offer to check again. Its `done.next` commands carry their agent flag (`--mode agent`, or `--agent` for `testrun run`): run them as given, ending a `testrun run` line in `< /dev/null`. If a `next` command has no agent flag, add the one that command takes before running it. + +On 0.8.2+ the default `cover gaps` output is the **coverage ribbon** — a high-level band-table: one severity-ordered row per use-case with its designed count, its coverage and per-class debt counts, deliberately carrying NO commands. To act, drill in: `cover gaps <uc-id>` (the dossier) renders every AC of that use-case plus a `next` actions block — evidence first; surface its warning that re-running an authored test on a broken app may heal the test around the failure — or read `--json`, whose per-UC pending rows keep `ready_command`. `--rollup` no longer changes the figure: don't pass it (agent mode answers it with a `warning` `ROLLUP_IGNORED`). BREAKING at 0.8.2: `--flat` and gaps' `--from` are REMOVED and refuse (pre-0.8.2 CLIs still have the flat worklist). With a `<uc-id>`, `--json` and agent mode close the document over that use-case. ## 8. Store rules — don't corrupt the user's graph - `.context/` is **append-only and single-writer**. Never run two store-mutating commands concurrently (extract, review verdicts, design, ingest, reconcile, retire/revert/rebuild). On a lock error (`EXTRACT_LOCKED`, or reconcile's walk lock), another run is live: report it and wait — never delete lock files; a dead run's lock clears itself. - Never hand-edit anything under `.context/`, and never `git merge` it — two histories merge cleanly in git and brick the store on the next read. Keep it gitignored — kane-cli 0.8.14 adds `.context/` to the folder's `.gitignore` itself when it creates the store inside a git repository — `gitignore_updated` on the stream, or a `warning{message}` when the line could not be written (relay it and add the line yourself); `KANE_CONTEXT_GITIGNORE=0` keeps it out. To share it with a team, use a **location** (a GitHub repository, an S3-compatible bucket, or a folder) through `kane-cli context sync`, `kane-cli context push`, `kane-cli context pull` and `kane-cli context clone` (0.8.14+) — a Git repository used as a location is not the same as committing `.context/`. While a sync rebase is open the store refuses every other write with `SYNC_REBASE_PENDING` (the fence) — finish it or `kane-cli context sync doctor --abort`, never delete files. Rules, events and decisions: `references/context-sync.md`. -- Safe inspection any time: `kane-cli context list --json` (nodes with trust + freshness), `kane-cli context explain <ref>` (a node's full recorded history, no AI), `kane-cli context fsck` (integrity check), `kane-cli context view --no-open --out <path>` (writes a self-contained HTML graph snapshot to that path — never opens a browser, never touches the graph). +- Safe inspection any time (a linked folder refuses `explain`, `fsck` and `view`; there use `kane-cli context list --mode agent`): `kane-cli context list --json` (nodes with trust + freshness), `kane-cli context explain <ref>` (a node's full recorded history, no AI), `kane-cli context fsck` (integrity check), `kane-cli context view --no-open --out <path>` (writes a self-contained HTML graph snapshot to that path — never opens a browser, never touches the graph). - Destructive verbs (`context retire`, `revert`, `rebuild`, `name --backfill`) exist and take `--yes` headless — run them **only on an explicit user request**, never autonomously. ## 9. When things fail @@ -197,6 +199,8 @@ On 0.8.2+ the default `cover gaps` output is the **coverage ribbon** — a high- | `sync_error` code `SYNC_BEHIND` / `SYNC_DIVERGED` (exit 3, 0.8.14+) | the location has records this store has not pulled / both sides added work — a pull or a rebase is needed, not a failure | relay the `remedy` verbatim (`kane-cli context pull <name>`, or `--rebase` on the user's go-ahead) — `references/context-sync.md` §6 | | `sync_behind` event on an extract/design/reconcile stream (0.8.14+) | advisory: the location has moved past this store | one line ("origin has records you have not pulled"); the run continues | | `sync_error` code `SYNC_STORE_BUSY` (exit 2, 0.8.14+) | a sync verb met a live session or lock on this store | wait; `kane-cli context sync doctor` when it names an interrupted rebase | +| `error` code `NO_PROPOSAL` | the agent stopped before proposing use-cases for that source | re-run `kane-cli context extract --mode agent` once; it reads that source again and costs credits again. If it happens twice, stop, and give the person the command and the message | +| a shared-project code (`PROJECT_NO_ACCESS`, `FOLDER_CHOSEN`, `PROJECT_BUSY`, `REMOTE_UNSUPPORTED`, `CONFIRM_REQUIRED`, `PROJECT_MISMATCH`, `TEST_CASE_MISSING` …) | a linked folder's refusal | `agent-contract.md`: each code with the command that recovers it | | `error` + `done` with exit `1` | runtime failure (incl. a sweep where some sources failed) | report the message; **do not blindly re-run a paid command** | | auth/credit failure mid-run | token or balance problem | keep the `sid`, have the user fix auth/balance (`kane-cli whoami`, `kane-cli balance`), then resume | | stream ends with no `done` event | the process crashed — outcome unknown (the one designed exception: `kane-cli context review --mode agent` keeps its `--json` shape and ends its listings without `done`; read its exit code) | check `context sessions --json` and `context list` before any retry, to avoid duplicate paid work | @@ -219,6 +223,8 @@ A changed requirements document is **one command**, not a re-ingest: kane-cli maintain reconcile --from ./prd-v2.md --source-id prd --mode agent ``` +**In a linked folder** reconcile takes a file, always with `--source-id`: `kane-cli maintain reconcile --from <file> --source-id <id> --mode agent`. It opens a review session that you save like any other (`shared-projects.md` §4), not the `reconcile_*` stream below. `--plan`, `--apply`, `--mode ci`, a missing `--source-id` and a URL in `--from` are refused (`REMOTE_UNSUPPORTED`); for a page's new version run `kane-cli context ingest <address> --mode agent`. The rest of this section is the local store's. + Never `context ingest` the new version first — reconcile does its own re-ingest, records the version move, and triages the fallout into rows: ADD (new use-case → design it), MODIFY (content moved → update or re-design), ARCHIVE (evidence decayed — **never applied headless**, always waits for the user's interactive session). It emits its own event family (`reconcile_plan` → `reconcile_row_start`/`reconcile_row_end` per row → `reconcile_paused`? → `reconcile_summary` → `done`; see `references/assurance-parsing.md`). Modes: `agent` auto-applies the safe ADD/MODIFY tier and exits 3 with the stored plan when ARCHIVE rows need a human; `ci` fail-closes (plan stored, exit 2); `--plan` is a safe staged preview. Re-running the same command is idempotent — it resumes a pending plan rather than re-billing. 0.7.2+ additions: @@ -227,9 +233,9 @@ Never `context ingest` the new version first — reconcile does its own re-inges - **`--from` takes URLs.** The same remote URLs ingest takes (§3): Jira/Confluence (0.7.2+), web pages (0.8.3+), Linear issues and documents (0.8.6+). `--source-id` is then optional — the URL carries its own id, and a contradicting `--source-id` refuses. Kind continuity holds both ways: a URL can't version a file-backed source and a file can't version a remote one — each refusal names the correct `--from`. A source ingested under a custom id (`--as`) is maintained by re-running that ingest with the same `--as`. The first reconcile of a Jira issue last ingested pre-0.7.2 may report the one-time upgrade re-version (§3) — not a content edit. - **One stream.** The re-extract child's events ride the reconcile stream itself, stamped `verb: "reconcile"` — parse per `references/assurance-parsing.md`. -For staleness that arrived outside a reconcile, `kane-cli maintain evolve <ref>` re-designs the parent use-case of the entity you name (`--from-stale` takes every use-case with stale designs). Give a `<ref>` or `--from-stale`, never both. A fresh target exits `2` unless you pass `--because "<reason>"`, and a use-case that was never designed exits `2` too (run `design tests` first). In a terminal evolve states the blast radius and asks. +For staleness that arrived outside a reconcile, `kane-cli maintain evolve <ref>` re-designs the parent use-case of the entity you name (`--from-stale` takes every use-case with stale designs). Give a `<ref>` or `--from-stale`, never both. A fresh target exits `2` unless you pass `--because "<reason>"`, and a use-case that was never designed exits `2` too (run `design tests` first). A use-case out of date only because its source changed is not evolved: as a target it exits `2` (unless `--because` is passed) and a `--from-stale` batch leaves it out, each with `evolve_skipped` `reason: "source_changed"` (and a `next`, when kane-cli knows the source's file, that reviews its change with `kane-cli maintain reconcile …`): offer the reconcile instead. In a terminal evolve states the blast radius and asks. -*(0.8.21+)* `kane-cli maintain evolve <ref> --mode agent` puts that question on the stream. A call on a stale target with no answer ends paused, exit `3`: `evolve_target` carries the blast radius (`use_case`, the `pairs` it affects, `because?`), then `evolve_paused` carries `pending_questions` (one per use-case, `id: "evolve:<use-case>"`, options `proceed` / `decline`), `resume` and `next[]` (two ready commands: proceed all, decline all), then `done{paused}`. Present the blast radius with the question, as with any pause (the pause loop above). Answer on the next call with `--answer evolve:<use-case>=1` (or `=proceed`) or `--answer evolve:<use-case>=2` (or `=decline`), repeatable; `--answer` needs `--mode agent`, and the resume command keeps `--because`. Nothing runs until every question is answered. A partial set pauses again on the rest, and the `next` commands then carry the answers already given, so a "decline the rest" command still re-designs what was answered proceed: read a `next` command before running it. On proceed, the design run's own events arrive between `child_start` and `child_end` stamped `verb: "design"`. After a clean design run comes `evolve_diff` (superseded, minted, retained, criteria whose tests moved). A design session that pauses arrives as its own `session_paused`: resume that session, then run evolve again for the diff. `done.next[]` carries every paused session's resume, all of them, whatever the exit code. In a `--from-stale` batch every use-case runs whatever the earlier ones did, and a declined or never-designed one is reported as `evolve_skipped`. Schema: `references/assurance-parsing.md`. On a CLI before 0.8.21 evolve has no `--mode`: suggest the user run it in a terminal rather than scripting around it. +*(0.8.21+)* `kane-cli maintain evolve <ref> --mode agent` puts that question on the stream. A call on a stale target with no answer ends paused, exit `3`: `evolve_target` carries the blast radius (`use_case`, the `pairs` it affects, `because?`), then `evolve_paused` carries `pending_questions` (one per use-case, `id: "evolve:<use-case>"`, options `proceed` / `decline`), `resume` and `next[]` (two ready commands: proceed all, decline all), then `done{paused}`. Present the blast radius with the question, as with any pause (the pause loop above). Answer on the next call with `--answer evolve:<use-case>=1` (or `=proceed`) or `--answer evolve:<use-case>=2` (or `=decline`), repeatable; `--answer` needs `--mode agent`, and the resume command keeps `--because`. Nothing runs until every question is answered. A partial set pauses again on the rest, and the `next` commands then carry the answers already given, so a "decline the rest" command still re-designs what was answered proceed: read a `next` command before running it. On proceed, the design run's own events arrive between `child_start` and `child_end` stamped `verb: "design"`. After a clean design run comes `evolve_diff` (superseded, minted, retained, criteria whose tests moved). A design session that pauses arrives as its own `session_paused`: resume that session, then run evolve again for the diff. `done.next[]` carries every paused session's resume, all of them, whatever the exit code. In a `--from-stale` batch every use-case runs whatever the earlier ones did, and a declined, never-designed or source-changed one is reported as `evolve_skipped`. Schema: `references/assurance-parsing.md`. On a CLI before 0.8.21 evolve has no `--mode`: suggest the user run it in a terminal rather than scripting around it. ## Sharing a context store diff --git a/.claude/skills/kane-cli/references/cards.md b/.claude/skills/kane-cli/references/cards.md index b035268..c8e58f2 100644 --- a/.claude/skills/kane-cli/references/cards.md +++ b/.claude/skills/kane-cli/references/cards.md @@ -73,7 +73,7 @@ Exit code `1`, or `status: "failed"`. Show the failing step's screenshot under t ## 4. Didn't start -Exit code `2`: nothing ran and no credits were used. Causes include no start URL, sign-in or setup errors, a test file that does not parse, an invalid suite plan, a cloud grid refusal. +Exit code `2`: nothing ran and no credits were used. Causes include no start URL, sign-in or setup errors, a test file that does not parse, an invalid suite plan, a cloud grid refusal. A designed test in a shared project that stopped on its Test Manager case before any step (an `error` with a `TEST_*` code) is this card too, even when it exits `1`. ```markdown | | | @@ -173,3 +173,38 @@ On kane-cli older than 0.8.17 the end event has no reason: read it from the evid A test that comes back broken with zero steps on the grid was refused before it launched: say so, point to the job link, and suggest checking that the app id belongs to this account. An invalid plan is a `🟡 Didn't start` card (§4) with one line per rejected test. + +**In a shared project** a suite's card adds a coverage row from the run's `coverage` event, and a line for each first +run (`authored`): + +```markdown +| 📈 **Coverage** | <pct>% of "<project>" · this run reached <n> use cases · it can take up to 5 minutes to count | +| ✍️ **First runs** | <n> tests wrote their steps on this run; later runs replay them | +``` + +Only a `coverage` event with `status: "written"` has a `pct`. With any other status, the 📈 row says +`The project's coverage isn't ready yet · I can check again in a few minutes` (`kane-cli cover --mode agent`). + +A member that stopped on its Test Manager case shows `failed` with a reason such as "isn't in Test Manager yet": say what +it needs (`testrun.md` §Designed tests in a shared project), not "the test failed". + +## 9. Coverage (`kane-cli cover`) + +Fields: the `coverage` event's `pct`, `usecases` (`total`, `proven`, `partial`, `failing`, `not_run`, `not_designed`, +`no_criteria`), `note`, and `recent_run` when a run is under 5 minutes old; then `done.next`. A `coverage` event with +`state: "computing"` has none of these: say the project is still working its figure out, draw no card, and run +`done.next` again in a minute. + +```markdown +| | | +|---|---| +| 📈 **Coverage** | <pct>% across "<project>"'s <total> use cases | +| ✅ **Proven** | <proven> · ⚠️ partial <partial> · ❌ failing <failing> | +| ⏳ **Not yet** | <not_run> not run · <not_designed> without tests · <no_criteria> with no criteria yet | +| 🕒 **Recent run** | still counting: check again in a few minutes *(only with `recent_run.may_not_count_yet`)* | +| ➡️ **Next** | <the first `done.next` command in plain words, e.g. run every test · design tests for "<use case>"> | +``` + +`note: "nothing_run_yet"` with `total: 0` means no use case exists yet: offer `kane-cli context ingest <file>`, not a run. +`done.next` commands carry their agent flag; run them as given, ending a `testrun run` line in `< /dev/null`. If one +has no agent flag, add the one that command takes. diff --git a/.claude/skills/kane-cli/references/context-sync.md b/.claude/skills/kane-cli/references/context-sync.md index a9f19b0..671f3c9 100644 --- a/.claude/skills/kane-cli/references/context-sync.md +++ b/.claude/skills/kane-cli/references/context-sync.md @@ -2,6 +2,13 @@ # Context sync — Agent Surface +> **Not for shared projects.** Location sync shares a **local** `.context/` store through a Git repository, a bucket or a +> folder. A team that works in a Test Manager project uses a shared project instead (`shared-projects.md`). A folder +> linked to one refuses `kane-cli context sync`, `kane-cli context push` and `kane-cli context clone` +> (`REMOTE_UNSUPPORTED`), and a location pull (a location name, `--rebase`, `--yes` or `--answer`: +> `PULL_LOCATION_LINKED`); a plain `kane-cli context pull --mode agent` there brings the project's tests, when +> `kane-cli context pull --help` lists `--project` (`shared-projects.md` §7). Never use both in one folder. + The assurance store (`.context/`, `references/assurance.md` §8) is one person's. **Context sync** shares it with a team through a **location**: a GitHub repository, an S3-compatible bucket, or a folder. Each person keeps their own store and publishes to the location; nothing on a location is ever overwritten or deleted. Reach for this reference when the user wants to share their context store, join a teammate's, get two stores level, or resolve a sync conflict. Sync never calls the KaneAI service — no credits, no model. **Version gate.** `kane-cli context sync`, `kane-cli context push`, `kane-cli context pull` and `kane-cli context clone` exist on **kane-cli 0.8.14 and later**. On an older CLI they fail as an *unknown command* (exit 2, a "did you mean" line) — the CLI is too old, not a typo. Confirm with `kane-cli --version`, have the user update, stop. diff --git a/.claude/skills/kane-cli/references/debug.md b/.claude/skills/kane-cli/references/debug.md index 372d188..9a33997 100644 --- a/.claude/skills/kane-cli/references/debug.md +++ b/.claude/skills/kane-cli/references/debug.md @@ -6,7 +6,7 @@ When a run fails, diagnose before suggesting fixes. ## The evidence pack is the log source -Every run seals an **evidence pack**; all run artifacts — actions, console, network, screenshots, failure records — live inside it. **`run_end.run_dir` is legacy: that directory is not created anymore.** Do not try to read `{run_dir}/run-test/...`. +Every run that starts seals an **evidence pack**; all run artifacts — actions, console, network, screenshots, failure records — live inside it. **`run_end.run_dir` is legacy: that directory is not created anymore.** Do not try to read `{run_dir}/run-test/...`. **Find the pack**: `{session_dir}/evidence/<execution_id>.evidence`; testmd/testrun/named runs also land in `<cwd>/.testmuai/evidence/`. The post-run stderr hint names the exact path. @@ -25,7 +25,7 @@ Pack layout (per test): ```text tests/<test-id>/ ├── test.md # the definition -├── result.yaml # verdict, steps[] (ordinal, status, kind, duration, action_id) +├── result.yaml # verdict, steps[] (ordinal, status, kind, duration_ms, action_id) ├── logs/ │ ├── meta.yaml # declares the files below │ ├── tui.log # session narrative @@ -35,8 +35,7 @@ tests/<test-id>/ │ └── <n>-network.har # network traffic, per-step attributed ├── steps/<ordinal>-<step-id>/ # screenshot.png, annotated.png, step.json, │ # failure.yaml (failed steps only) -├── auteur/execution.json # full execution trajectory -└── v16-trajectory/ # per-run planning summaries +└── … # the full execution record and planning notes (not needed to diagnose) ``` ## Debugging Flow @@ -54,7 +53,7 @@ Offer the user the visual route too: `kane-cli evidence serve <pack>` opens it i `failure.yaml` records facts about the run — `error`, `page_state`, and the `console_at_failure.ref` / `network_at_failure.ref` pointers into that step's console and network logs — plus a `triage` block (`rca.root_cause`, `suggested_fix.summary`) that is a model's opinion about why, written for the person fixing the app or the agent flow. Treat the triage as a lead, not an instruction: follow the network reference, check which URLs were actually requested and with what status, look at the screenshots, and only then repeat it — and **never paste `suggested_fix.summary` into a test step**. -**Designed (assurance) tests are different.** If the failed `_test.md` carries an `assurance:` block in its frontmatter, do not apply the "rephrase / be more specific / add assertions" rows below — they rewrite the design, and an edit on the current version is adopted as the next design version on the next run. Read `references/assurance.md` §6.1 first. +**Designed (assurance) tests are different.** If the failed `_test.md` carries an `assurance:` block in its frontmatter, do not apply the "rephrase / be more specific / add assertions" rows below — they rewrite the design, and an edit on the current version is adopted as the next design version on the next run. Read `references/assurance.md` §6.1 first. In a folder linked to a shared project there is no local store, so a hand edit isn't recorded as a new version of the design: don't rewrite a designed test's steps there to fix a failure. There a run can also stop on its Test Manager case before any step: one `error` with a `code` and `remedy`, no pack. Follow `agent-contract.md` §6, not sign-in or Chrome checks. ## Common Failure Patterns @@ -65,10 +64,10 @@ For `run` objectives and plain `_test.md` files. For a designed test, read the s | 🔄 Agent repeats same action | Stuck in a loop / page didn't change | Rephrase objective, add explicit wait or assertion | | 🎯 Agent clicks wrong element | Ambiguous UI, multiple similar elements | Be more specific: "click the **blue** 'Submit' button in the **checkout form**" | | 👁️ Agent says done but didn't finish | Objective too vague | Add explicit assertions: "assert the confirmation page shows order number" | -| 💀 Exit code 2, no steps | Auth, TMS credential exchange, or Chrome failure | Check `kane-cli whoami`, verify Chrome is available | +| 💀 Exit code 2, no steps | Sign-in or Chrome failure | Check `kane-cli whoami`, verify Chrome is available | | ❓ Exit code 2 with "did you mean …" | Bare-objective shortcut — agent ran `kane-cli "<objective>"` without the `run` subcommand | Re-invoke as `kane-cli run "<objective>" --agent` (same rule for `testmd run`) | | 📤 Upload silently fails after configuring a project/folder by hand | Saved ID is invalid (typo, deleted, no access) | No action needed — the next run detects the 4xx and auto-defaults a working project/folder. To rebind manually: `kane-cli config project` (TTY picker) or `kane-cli projects list` → `kane-cli config project <id>` (see `references/test-manager.md`) | -| ⏱️ Exit code 3 | Timeout or cancelled | Increase `--timeout` or `--max-steps`, or split into smaller objectives | +| ⏱️ Exit code 3 (`run`; a saved test reports it as failed, exit 1) | Timeout or cancelled | Increase `--timeout` or `--max-steps`, or split into smaller objectives | | 🚫 "CDP endpoint not reachable" | Chrome not running | Let kane-cli manage Chrome (remove `--cdp-endpoint`) | ## Filing a bug report diff --git a/.claude/skills/kane-cli/references/evidence.md b/.claude/skills/kane-cli/references/evidence.md index 32ff71b..fbe6643 100644 --- a/.claude/skills/kane-cli/references/evidence.md +++ b/.claude/skills/kane-cli/references/evidence.md @@ -20,8 +20,7 @@ A `.evidence` file is a standard zip: `unzip -l <pack>` lists it, `unzip -p <pac │ # <n>-console.ndjson, <n>-network.har ├── steps/<ordinal>-<step-id>/ # screenshot.png, annotated.png, step.json │ # (+ failure.yaml on failed steps) - ├── auteur/execution.json # full execution trajectory - └── v16-trajectory/ # per-run planning summaries + └── … # the full execution record and planning notes (not needed to diagnose) ``` Reading and debugging from the pack (unzip recipes, failure records): `references/debug.md`. @@ -36,7 +35,7 @@ Reading and debugging from the pack (unzip recipes, failure records): `reference ## The stderr hint — and how to act on it -After a successful agent-mode run, kane-cli prints one hint line to **stderr** (it is NOT an NDJSON event; never look for it on stdout): +After an agent-mode run whose evidence pack sealed (passed or failed), kane-cli prints one hint line to **stderr** (it is NOT an NDJSON event; never look for it on stdout): ``` evidence: view locally with `kane-cli evidence serve <packPath>` @@ -70,7 +69,7 @@ Exit `0` valid · `1` invalid · `2` not found. Use it when a pack won't open in kane-cli evidence merge <targets...> --run-id <id> ``` -Combines packs (execution ids or paths, order-significant) into one, sealed by default at `.testmuai/evidence/<run-id>.evidence`. `--run-id` is required. Exit `0` merged · `1` policy abort (e.g. mixed projects, duplicate run ids) · `2` usage. Full policy detail is in the user guide — for anything beyond a simple merge, point the user there rather than improvising flags. +Combines packs (execution ids or paths, order-significant) into one, sealed by default at `.testmuai/evidence/<run-id>.evidence`. `--run-id` is required. Exit `0` merged · `1` policy abort (e.g. packs from different projects or organisations, or a pack that isn't sealed and valid; a pack whose run id repeats is skipped, not an abort) · `2` usage. Full policy detail is in the user guide — for anything beyond a simple merge, point the user there rather than improvising flags. ## Debugging with a pack @@ -78,6 +77,6 @@ The pack is the first place to look on a failure: the failed step's **failure re ## Evidence merge identity -For ordinary runs, default merge identity distinguishes the test and commit (`external_id.test_id`, `external_id.commit_id`) and the environment (`environment.os`, `environment.os_version`, `environment.browser`, `environment.browser_version`, `environment.device`, `environment.app`). Re-runs with the same identity nest as attempts; a different environment produces a separate sibling, so one test on two devices or two builds is two entries. On a real device, `environment.device` is the device the hub allocated (never the requested pattern) and `device_slug` is derived from it. +For ordinary runs, default merge identity distinguishes the test, commit and dataset row (`external_id.test_id`, `external_id.commit_id`, `external_id.dataset_row_id`) and the environment (`environment.os`, `environment.os_version`, `environment.browser`, `environment.browser_version`, `environment.device`, `environment.app`). Re-runs with the same identity nest as attempts; a different environment produces a separate sibling, so one test on two devices, or on two builds with different app identities, is two entries (two local builds with the same file name can group together). On a real device, `environment.device` is the device the hub allocated (never the requested pattern) and `device_slug` is derived from it. Explicit collision policies can change grouping; a custom `--rules` file replaces the default rules rather than extending them. Check the selected identity rules before interpreting two runs as retries of the same test. diff --git a/.claude/skills/kane-cli/references/execution-controls.md b/.claude/skills/kane-cli/references/execution-controls.md index 5cae455..3cb1086 100644 --- a/.claude/skills/kane-cli/references/execution-controls.md +++ b/.claude/skills/kane-cli/references/execution-controls.md @@ -4,7 +4,7 @@ `run` and `testmd run` support `--assertion-mode dom|visual` (default `dom` with vision fallback) and `--final-validation on|off` (default off). Persist with `config set-assertion-mode` and `config set-final-validation`. Final validation controls the synthesized `cp_final` checkpoint independently of action/testing mode; keep explicit terminal assertions in objectives. -`run --analyzer-only --condition "<condition>"` checks the current desktop browser page without an objective, action steps or saved test. Repeat `--condition` for multiple checks. Use `--agent` or non-TTY input. Results contain `condition_results: boolean[]`; exit `0` means all conditions were judged, **not** that all are true. Exit `1` means a result was missing, and `3` means cancelled. This mode rejects mobile target/app/device options and code-export/name options. +`run --analyzer-only --condition "<condition>"` checks the current desktop browser page without an objective, action steps or saved test. Repeat `--condition` for multiple checks. Use `--agent` or non-TTY input. Results contain `condition_results: (boolean | null)[]`, one per condition; `null` means that condition couldn't be judged, and `condition_reasons` says why. Exit `0` means all conditions were judged, **not** that all are true. Exit `1` means a result was missing or `null`, and `3` means cancelled. This mode rejects mobile target/app/device options and code-export/name options. Experimental `run --network-ws` and `run --network-sse` enable WebSocket and SSE capture; both default off. Persist with `config set-network-ws on|off` and `config set-network-sse on|off`. SSE capture is Chromium-only. Do not copy these run-only flags onto `testmd` or `testrun` commands. diff --git a/.claude/skills/kane-cli/references/explore.md b/.claude/skills/kane-cli/references/explore.md new file mode 100644 index 0000000..64199f5 --- /dev/null +++ b/.claude/skills/kane-cli/references/explore.md @@ -0,0 +1,217 @@ +<!-- kane-cli skill reference: exploration on HyperExecute, and the tests it designs into a shared project. Read when the person asks to explore an app, map its flows, shape, stop or re-run an exploration, or get and run the tests an exploration made. Requires kane-cli 0.10.0+. --> + +# Exploration: map an app on HyperExecute, and get the tests it designs + +`kane-cli explore` sends an explorer through the person's app on **HyperExecute**. It builds a map of the app (screens, +flows, findings, with evidence). While its **assurance context** is on (the default), every few minutes while it runs, +and when it ends, what it has mapped is turned into use cases, scenarios and tests in the Test Manager project the +exploration belongs to, with a Test Manager case per scenario. Exploration **designs and saves** those tests; it never +runs them. + +**Exploration and assurance.** The tests an exploration designs live in its project, as a document's do in a shared +project (`shared-projects.md`). To run them here, a folder has to hold them: a folder linked to that project gets them +with `kane-cli context pull --mode agent`, and a fresh folder links and gets them in one step with +`kane-cli context pull --project <id> --mode agent` (§5). A project nobody has shared yet becomes a shared project with +the exploration's first design job, as the person who started the run. + +Explore takes `--agent`, not `--mode agent`. Its messages can name internal parts: use them to work out what happened, +then tell the person in plain words. + +## 1. Start an exploration + +```bash +kane-cli explore run --project <id> --url <app url> --prompt "<what to explore>" --agent # add --policy read_only if the person said no changes +``` + +- `--project <id>` is the Test Manager project the exploration saves into: the team's shared project + (`shared-projects.md` §2 finds the id). One project, one exploration, one URL. **Always pass it**: explore doesn't + read the folder's link, so without `--project` it falls back to the project kane-cli is configured with, and a run + with none is refused. In a linked folder, pass the folder's project id (the preflight's `## link`). +- It queues the run on HyperExecute and exits; it doesn't wait for the exploration to finish. +- It uses the person's sign-in, turned into their username and access key. Only if it says it needs basic credentials, + ask the person to run `kane-cli login --username <user> --access-key <key>` in their own terminal. It stores the + sign-in and any secret variables in the person's HyperExecute Secrets, and the non-secret variables in Test Manager. + Never put values in chat. +- The default policy is `mutating`: the explorer may submit forms and change data on the sites it reaches, and in agent + mode kane-cli asks nobody. Before `explore run`, tell the person and ask whether it may change data in their app; if + not, pass `--policy read_only` (it keeps the explorer from changing data). A `mutating` run starts with a `warning` + (`code: "policy_mutating"`): relay it. +- Under `read_only`, the setup steps in `--deps` (a sign-in, for example) still run as written, so put in `--deps` only + what the person allows. +- An exploration spends credits while it runs, and so does each design job it starts (§4). Without `--max-leases` it + runs until it runs out of work. Tell the person before `explore run`, and offer a limit when credits matter (§2). +- A project that isn't shared yet is shared by the exploration's first design job, as the person who started the run. + If they may not create test cases in it, no tests are designed: `explore context status` shows `context.enabled: true` + with `context.off_reason: "mark_refused"`. Tell the person: "Ask a Test Manager admin for permission to create test + cases, or ask someone who has it to share the project." Once they have permission, turn context off and on (§4), only + on their word. + +**The stream:** `stream_start` (`surface: "explore"`), maybe `warning`, `explore_remote_started {exploration_id, run_id, +state, hx_job_id, job_link}`, then `done {state: "queued", exit_code: 0, exploration_id, run_id}`. Three plain lines on +stderr name the run, the job link and how to watch it. Give the person the job link. + +| Exit | Means | Do | +|---|---|---| +| 0 | queued | tell the person it runs on HyperExecute and can take a while; offer to check its status later | +| 1 | refused: `RUN_ACTIVE` (this exploration already has a run), or the launch failed | `RUN_ACTIVE`: wait, or ask the person whether to stop it (§3). A failed launch: try again in a minute; then tell the person | +| 2 | bad input (a different `--url` for the project's exploration, a `{{name}}` with no value, `--deps` with no `--allow-domain`), sign-in, a secret HyperExecute doesn't list (`SECRETS_MISSING`), or a message starting "could not push" or "could not read" | fix the input or the secret's name. If `done` names a run (`state: "stopped"`, `remark` `failed:secrets_missing`), report that run, fix the value, then start a new one. "Could not push / read": try again in a minute; then tell the person | + +Explore's `error` events carry a `message` only: no `code`, no `next`. Read the message. + +**On this machine** (`--local-orchestration`, below) the command streams the run and ends with it: `done` carries +`state` and `remark`. Exit 0 completed or reached its limit, 1 failed, 3 stopped or timed out, 2 couldn't start. + +## 2. Shape the run + +Add these to `explore run` only when the person's request needs them: + +| The person wants | Pass | +|---|---| +| An app behind a login | `--deps "Logged in as {{user}} / {{pass}}"`, with the values in a variables file (`.testmuai/variables/*.json`, or `--variables-file <path>`; a secret has `"secret": true`). `--deps` always needs `--allow-domain`: the app's own host, and every other host it signs in on (`--allow-domain app.example.com --allow-domain login.example.com`). Only the names the prompt and `--deps` use travel: secrets go to the person's HyperExecute Secrets, the rest to the organisation's Test Manager variables, where they replace a variable of the same name (tell the person). Names are letters, digits and `_`; `LT_USERNAME` and `LT_ACCESS_KEY` are taken. Check the values are real first: an empty secret stops the run `failed:secrets_missing`, and a sign-in that fails stops it `failed:setup_failed` | +| Only some sites | `--allow-domain <host>`, repeatable (the URL's host is always in, subdomains included) | +| A control it must never press | `--deny "<control name>"`, repeatable, under either policy | +| A `read_only` run of an app that reads its data through POST | `--allow-post <host/path>`, repeatable, only for an endpoint the person confirms only reads; its host must be the URL's or one passed with `--allow-domain` | +| Start from one screen of the map it already has | `--focus "<screen>"` (a later run; the screen's name or words from it). On this machine it also needs `--exploration-id <exploration-id>` | +| A limit on the explorer's work | `--max-leases <n>`: it stops after n pieces of work (leases), and the run ends `budget`. Every run also stops at built-in limits (an hour, 400 actions or 800 model calls), also `budget`. Neither limits the design jobs' credits: for a map with no new tests, see the next rows | +| More browsers at once | `--max-parallel <n>` (default 1; up to 20 cloud browsers, 4 with Chrome on this machine). Each may sign in once: keep 1 for an app that allows one login | +| A map, no tests | `--no-context`, on the project's first exploration only; for an existing one, `explore context off` (§4) | +| How deep it goes, what counts as slow, whether findings are checked again | `--depth-caps`, `--slow-budget`, `--no-recheck`: `kane-cli explore run --help` gives their values | +| An app on `localhost` or a private network | `--tunnel-name <name>`, through a LambdaTest tunnel on the person's machine (below) | +| To explore from this machine | `--local-orchestration` (Chrome here, `--headless` for no window; or `--browser remote` for cloud browsers). The command then waits for the run to end (§1), and Ctrl+C stops it | + +### An app on `localhost` or a private network + +The cloud's browsers can't reach it on their own: they reach it through a LambdaTest tunnel (LT) running on the person's +machine. kane-cli never starts or stops the tunnel; the person does, in their own terminal, so their key never comes +into chat. + +1. If `LT --version` doesn't run there, they download LT for their machine and put it on their PATH: + - Apple silicon Mac: `https://downloads.lambdatest.com/tunnel/v3/mac/arm64/LT_Mac.zip` + - Intel Mac: `https://downloads.lambdatest.com/tunnel/v3/mac/64bit/LT_Mac.zip` + - Linux: `https://downloads.lambdatest.com/tunnel/v3/linux/64bit/LT_Linux.zip` (`linux/arm64/LT_Linux.zip` on ARM) + - Windows: `https://downloads.lambdatest.com/tunnel/v3/windows/64bit/LT_Windows.zip`, then run `LT.exe` + + If a Mac blocks it from opening: `xattr -d com.apple.quarantine <path to LT>`. +2. They start it as the same LambdaTest user kane-cli is signed in as, and leave it running: + + ```bash + LT --user "$LT_USERNAME" --key "$LT_ACCESS_KEY" --env ht-prod --tunnelName <name> + ``` + + (`$LT_USERNAME` and `$LT_ACCESS_KEY` are their LambdaTest username and access key.) `--env ht-prod` is required. + Wait until LT says the tunnel is connected. +3. Run the exploration through it: + + ```bash + kane-cli explore run --project <id> --url http://localhost:3000 --prompt "<what to explore>" --tunnel-name <name> --agent + ``` + + `--tunnel-name` must match `--tunnelName` from step 2, and it turns `--tunnel` on; with only one tunnel running, + `--tunnel` alone is enough. A re-run keeps the exploration's URL but not its tunnel, so pass the tunnel flag again: + `kane-cli explore run --project <id> --tunnel-name <name> --agent`. + +- The tunnel **and that machine** must stay up until the run ends; the run warns about it when it starts. If the tunnel + drops mid-run, pages stop loading. +- A tunnel needs cloud browsers: with `--local-orchestration` and local Chrome, `--tunnel` is refused (local Chrome + already reaches the machine). + +## 3. Watch it, stop it, run it again + +```bash +kane-cli explore list --agent # the running explorations; --all lists every one +kane-cli explore sessions <exploration-id> --agent # its runs, newest first: state, remark, leases, findings, credits +kane-cli explore sessions <exploration-id> --run <run-id> --agent # one run in full +``` + +The person can watch a run live with `kane-cli explore view --exploration-id <exploration-id>` in their own terminal +(§4). + +A run is `queued` (waiting for its machine), `running` or `stopped`. A stopped run's `remark.code` says why: +`completed` (it ran out of work), `budget` (`--max-leases` or a built-in limit reached), `user_stop`, `stopped_before_start` (stopped while +queued), `heartbeat_lost` (it stopped reporting), `timeout`, or `failed:<reason>` (for example `failed:secrets_missing`, +`failed:setup_failed`: the sign-in in `--deps` didn't work, or `failed:vm_never_started`: HyperExecute never started it, +so run it again). Say it to the person in plain words. + +- **Stop it:** `kane-cli explore stop <exploration-id> --agent` stops a queued run at once, and asks a running one to + stop cleanly (its work in hand finishes and is kept). It waits for that, unless you pass `--no-wait`. It emits + `explore_stop {run_id, state, remark, stopping}`. When `stopping` is `true` the run hasn't stopped yet: check + `explore sessions` until it is `stopped` before you run it again or delete it. With nothing running it refuses + (`RUN_NOT_ACTIVE`, exit 1). +- **Run it again:** `kane-cli explore run --project <id> … --agent` once more. It goes on the same exploration, on its + URL, and adds to its map; a different `--url` is refused (exit 2). A run keeps only the URL: pass every option again + (`--policy read_only` above all, since the default is `mutating`, and the prompt, `--deps`, domains, variables, + limits and tunnel). +- **Explore another site:** use another project, so its tests don't mix with this site's. Or, only on the person's + word, `kane-cli explore delete <exploration-id> --yes --agent` (refused with `RUN_ACTIVE`, exit 1, while a run is + queued or running): no command shows that map again and the project can start a new exploration of any URL, but the + use cases and tests designed for the first site stay in the project; a pull brings them along with the new site's. +- `list`, `sessions`, `stop` and `delete` exit 0 when done, 1 when refused (`RUN_ACTIVE`, `RUN_NOT_ACTIVE`) or when the + service failed (read the message), and 2 for bad input, a missing `--yes`, an unknown exploration, or no sign-in. + +## 4. Check what it designed + +```bash +kane-cli explore context status <exploration-id> --agent +``` + +`explore context status` emits `explore_context {exploration_id, action, context}`. Read `context.enabled`, +`context.off_reason`, `context.running_job` and `context.pending_cut` (work waiting for the next job) as well as +`context.last_job`, the newest design job that has **ended**: `status` (`ok`, `noop`, `partial`, `failed`), +`use_cases.added[]` and `modified[]`, `tests.designed`, `tests.saved`, `tests.failed[]`, `tests.saved_cases[]` (each +test's Test Manager case), `held[]` (changes, such as an archive, the job left for a person to review), `credits` (that +job's own), `error`. A job runs after the exploration's run ends, and during a long run every few minutes when enough +changed. Before you call the designs ready, check that no job is still running or waiting. An entry of `saved_cases[]` +can have no case and a refusal even when the job says `ok`: tell the person which tests are saved but can't run yet, +and follow that refusal's remedy. + +```bash +kane-cli explore context sync <exploration-id> --agent # ask for a job now +``` + +It emits `explore_context_sync {exploration_id, state, …}`: `launched`, `launching`, `pending` (behind a running job), +`running` and `up_to_date` all end exit 0. A known refusal's `error` message starts with its code (other failures are +plain words: read the message and the exit code): + +| Refusal | Exit | Means | Do | +|---|---|---|---| +| `TOO_SOON` | 1 | the last design job started a few minutes ago | try again when it says | +| `NO_ENDED_RUN` | 1 | no run of this exploration has ended | wait for the run, then sync | +| `CONTEXT_OFF` | 1 | assurance context is off for this exploration | `kane-cli explore context on <exploration-id> --agent`, only with the person's word. If the message names a reason in brackets other than `user_off`, tell the person instead: turning it on helps only for `mark_refused`, once they have permission (§1) | +| `NOT_CONFIGURED` | 1 | design jobs aren't set up here yet; context stays on and runs once they are | tell the person; don't retry | +| a `failed` state | 1 | the job couldn't launch | try once more; then tell the person | +| sign-in or access | 2 | the sign-in was refused, or basic credentials are needed | as in §1: sign-in first, then once more | +| not found | 2 | no such exploration, or assurance context isn't available here yet | check the id (`kane-cli explore list --all --agent`); then stop | + +`kane-cli explore context off <exploration-id> --agent` stops new design jobs and drops the work waiting for one; +nothing already saved is deleted, and a job already running finishes (it may still save tests and use credits): check +the status and tell the person. `on` catches up from the newest ended run. Both only on the person's word. + +To show the map, ask the person to run `kane-cli explore view --exploration-id <exploration-id>` (or `--project <id>`) +in their own terminal: it opens the map in their browser and keeps serving it until they press Ctrl+C. There they can +look at the evidence, correct findings, leave notes, and mark screens or controls not to touch. Don't run it yourself: +it doesn't exit on its own. + +## 5. Get exploration's tests into a folder, and run them + +The use cases and tests are in the project, and their cases are in Test Manager, as soon as a job saves them. No +folder holds their files until it pulls them. + +- Read `shared-projects.md` §7a before the pull, and check first whether this kane-cli has it (`kane-cli context pull + --help` lists `--project`). With it: + - **a folder linked to the project:** `kane-cli context pull --mode agent`; + - **a fresh folder:** `kane-cli context pull --project <id> --mode agent` links it and brings the tests in one step. +- Read each `pull_test`'s `outcome`, and count the first runs from its `first_run: true` (a fresh folder's tests all + have it; a folder that ran them before may not). Fill the values `variables_needed` names: an app behind a login + needs its sign-in values in this folder too. A pull with no tests yet ends `done` exit 0: check + `explore context status <id>`. +- Without the pull (kane-cli 0.10.0), exploration's tests can be seen (`kane-cli context list --mode agent` in a linked + folder; Test Manager) but not run from a folder: no folder holds their files. +- The first run of each pulled test writes its steps in a real browser: minutes, and about 30–85 credits per test. Tell + the person the count and the likely cost, then run the files they choose: + `kane-cli testrun run <files> --agent < /dev/null`. +- Exploration's use cases are designed without review: in agent mode `context list` shows them as `trust: "derived"`. + Present them to the person as exploration's proposals. The source row's `id` is `explore-<exploration id>`. + `kane-cli explore list --all --agent` lists every exploration (without `--all`, only the running ones); a page's + `next_cursor` goes to `--cursor` for the next. +- After the first runs, `kane-cli cover --mode agent` in the linked folder shows the project's coverage, exploration's + use cases included. diff --git a/.claude/skills/kane-cli/references/fair-evaluation.md b/.claude/skills/kane-cli/references/fair-evaluation.md index 2b8f1e6..03908ef 100644 --- a/.claude/skills/kane-cli/references/fair-evaluation.md +++ b/.claude/skills/kane-cli/references/fair-evaluation.md @@ -9,7 +9,7 @@ When you compare kane-cli to anything else (an agent that generates Playwright s kane-cli has two phases with very different costs: - **Authoring** (first run, or after an edit): an AI agent works out each step → spends LLM tokens. One-time per step. -- **Replay** (every later run): executes the recorded action plan from local cache → **zero LLM cost**, runs in seconds. Cost is proportional to how often a test **changes**, not how often it **runs**. +- **Replay** (every later run, for each unchanged step that has a recording): executes the recorded action plan from local cache → **mostly avoids the authoring cost**, runs in seconds. A replay that heals re-authors the failing steps, so it costs more. Cost is proportional to how often a test **changes**, not how often it **runs**. (In a shared project, each folder that pulls a test authors it once, on its first run there: `shared-projects.md` §7a.) The most common evaluation error is pitting kane-cli **authoring** against the other approach's **execution**. That is apples-to-oranges. Compare the *same phase* on both sides: @@ -30,7 +30,7 @@ Compare kane **authoring** to the other approach's **creation effort** (the toke ## Mandatory corrections (real errors seen under time pressure) -- ❌ "kane spends LLM tokens on every test run." → **Wrong.** Tokens are spent only when a step is *authored* (first run or after an edit). Replay of unchanged steps is zero-LLM. State this before any cost claim. +- ❌ "kane spends LLM tokens on every test run." → **Wrong.** The authoring cost is spent when a step is *authored* (first run, after an edit, or when a replay heals). Replay of unchanged steps mostly avoids it. State this before any cost claim. - ❌ "The Playwright scripts already exist, so their creation cost is $0." → **Sunk-cost trap.** The creation effort was still spent (tokens or engineer hours). For an adoption/ROI decision, either count creation on **both** sides, or compare the **forward maintenance trajectory** — never zero out one side's creation while charging the other's. - ❌ "Free CI runs vs. tokens per test." → Phase mismatch (execution vs. authoring). Both execute cheaply; the real differentiators are creation effort and maintenance under UI churn. - ❌ Judging a single isolated run. → The right unit is the **suite over its lifetime** (many runs + UI changes). kane's advantage widens with run count and churn. diff --git a/.claude/skills/kane-cli/references/first-run.md b/.claude/skills/kane-cli/references/first-run.md index 0c4b960..aae2b93 100644 --- a/.claude/skills/kane-cli/references/first-run.md +++ b/.claude/skills/kane-cli/references/first-run.md @@ -2,6 +2,18 @@ # The first run +**In a folder linked to a shared project** (it holds `.testmuai/assurance.json`), or when the person's first request is +for a team's project, the first result is the shared-project journey, never a `kane-cli run`: `kane-cli run` saves to +the profile's project, and when that isn't the team's, its test then clashes with the team's tests +(`shared-projects.md` §9). The first result is +the list of use cases the first extract proposes; show it as the first result, then offer the same three choices. A +teammate joining a project whose documents are all read gets no proposal (`nothing_new`): show the project's +`context list` as the first result instead. In a linked folder, the "where results go" choice only changes where +`kane-cli run` saves; say so. Quick `kane-cli run` experiments go in a separate folder that isn't linked. + +**When the first request starts from requirements** (a PRD, a spec, a description) in a local folder, the first result +is likewise the use cases the extract proposes. + A person's first request should reach its first result with nothing standing in the way. The order is fixed: 1. Ready card (`references/ready-check.md`) @@ -73,6 +85,10 @@ Use the normal card from `references/cards.md`, and on this first run make two o From the second run on, the evidence viewer goes back to an offer. +When the first result is an extract's use cases (above), nothing has run yet, so there is no evidence and no test case: +show the use cases (in a linked folder, also whether the session is saved yet), with no 📁 or 🔗 row. Those rows come with the first test run. +Choice 1 below then reads "Runs open a browser window by default. Keep that, or run without a window?" + ## 4. Three choices, asked once, after the first result Ask these after the first payoff card. They read as tailoring, not as a toll gate, because the person has already seen a result. diff --git a/.claude/skills/kane-cli/references/live-strip.md b/.claude/skills/kane-cli/references/live-strip.md index cbe9dc2..0f5f2ee 100644 --- a/.claude/skills/kane-cli/references/live-strip.md +++ b/.claude/skills/kane-cli/references/live-strip.md @@ -27,8 +27,8 @@ If any of these is missing, do not offer the strip. Nothing else changes: the st - It **wraps the status line the person already has**: their line prints first, unchanged, and the kane line appears under it. - It appears **only while a run is live, and for five minutes after it ends**. The rest of the time the person sees exactly what they had before. -- It appears **only in the Claude Code session that started the run**. Other sessions show nothing, even when they are open in the same project. The reader tells sessions apart by checking that the run descends from the same session process it was started by. A run the person starts by hand in a terminal is not shown. On Windows, where that check is not available yet, every session open in the run's project shows it. -- It reads two things kane-cli writes on its own: a small pointer file for each live run, and that run's event log. It starts no process besides the person's original status line command, makes no network calls, and sends nothing anywhere. +- It appears **only in the Claude Code session that started the run**. Other sessions show nothing, even when they are open in the same project. The reader tells sessions apart by checking that the run descends from the same session process it was started by. A run the person starts by hand in a terminal is not shown. Where that check can't run (on Windows, or wherever the process list can't be read), every session open in the run's project shows it. +- It reads two things kane-cli writes on its own: a small pointer file for each live run, and that run's event log. It starts no process besides the person's original status line command and a local process-list check (`ps`), makes no network calls, and sends nothing anywhere. - **Typed text is never echoed.** A typing step shows as `typing in <field>`. - It refreshes every two seconds. It starts showing a run once kane-cli has created the session, which takes roughly 10 to 30 seconds after launch (the browser has to start first). Until then the person sees their normal status line. While a step is still working, the line shows the last finished action, marked `last:`. diff --git a/.claude/skills/kane-cli/references/mobile.md b/.claude/skills/kane-cli/references/mobile.md index f5f2e18..e790251 100644 --- a/.claude/skills/kane-cli/references/mobile.md +++ b/.claude/skills/kane-cli/references/mobile.md @@ -2,7 +2,7 @@ # Mobile testing (real devices from any machine; emulator/simulator locally on macOS Apple Silicon or on the cloud grid) -Desktop (the browser) is the **default** target and the primary use of kane-cli. Mobile is a **scoped addition**: kane-cli can also drive a native app on a **real Android or iOS device** in the TestMu AI cloud, or on a virtual Android or iOS device — on this machine, or on a LambdaTest HyperExecute macOS host via `testrun run --remote`. Nothing about web runs changes. A run with no mobile flags still drives Chrome exactly as before. +Desktop (the browser) is the **default** target and the primary use of kane-cli. Mobile is a **scoped addition**: kane-cli can also drive a native app on a **real Android or iOS device** in the TestMu AI cloud, or on a virtual Android or iOS device — on this machine, or on a LambdaTest HyperExecute macOS host via `testrun run --remote`. Nothing about web runs changes. A run with no mobile flags still drives Chrome exactly as before, unless a mobile target was saved for `kane-cli run` (`kane-cli config set-target`); a saved test follows its own `target:`. ## Availability (read first) @@ -12,7 +12,7 @@ Desktop (the browser) is the **default** target and the primary use of kane-cli. | **Local** (a simulator/emulator on this machine) | **macOS on Apple Silicon (arm64) only** — not Intel Macs, Linux, or Windows | `run --target …`, `testmd run`, `testrun run` | | **Cloud grid** (a virtual device on a HyperExecute macOS host) | **Any machine** — no Xcode / Android Studio needed. The account needs a HyperExecute plan with macOS runners | `testrun run <paths> --remote … < /dev/null` — see §Remote below | -- **Desktop stays the default.** The `--target` axis is what selects mobile. Leave it off and you get the browser. +- **Desktop stays the default.** The `--target` axis is what selects mobile. Leave it off and `kane-cli run` uses the saved target (`config show`), else the browser; a saved test uses its own `target:`, else the browser. Pass `--target desktop` when you mean the browser. - If the user is not on mac-arm64, local emulator/simulator is not an option — **offer a real device** (`--target android|ios`, works from any machine, plan permitting) or **the grid**: save the objective as a `_test.md` (`target: emulator|simulator` + `app:`) and run it with `kane-cli testrun run <path> --remote --device-name "<grid device>" --os-version <v> < /dev/null`. Do not tell them mobile is unavailable. ## The five targets @@ -60,7 +60,7 @@ Every mobile run needs an app. Provide it one of two ways: kane-cli installs that app on the device and runs the objective against it. `kane-cli apps list --target emulator|simulator|android|ios --agent` lists the account's uploads (NDJSON; the `app_id` field is what `--app`/`app:` take). There is no upload subcommand: any run with a local build — local or `--remote` — uploads it to the account (once per build; a per-machine cache skips a build the account already has) and prints the `APP…` id. Uploads belong to an organisation — `apps list` for the active profile is the authority. -**Not accepted:** a package/bundle id (e.g. `com.example.app`), a `.ipa` on a simulator (it is a device build — use `--target ios`), or a `.app` bundle. There is no default app: a mobile run without a valid build or `APP…` id cannot start. +**Not accepted:** a package/bundle id (e.g. `com.example.app`), a `.ipa` on a simulator (it is a device build — use `--target ios`), or a `.app` bundle. A mobile run without a valid build or `APP…` id (from `--app`, the file's `app:`, or one saved with `config set-app`) cannot start. ## Setup path @@ -143,7 +143,7 @@ Rules that differ from a local run (each violation is a `remote_error` before di | **Payload** | Members must live under the cwd being dispatched, and recordings must not be gitignored (un-ignore with `!output-*/`). Builds are not payload. | `member_outside_payload`, `gitignored_inputs` | | **Auth** | HyperExecute needs a LambdaTest username + access key; an OAuth profile is exchanged automatically, or pass `--username`/`--access-key`. Any `APP…` id must belong to the same organisation. | `no_basic_auth` | -What to present after `testrun_done`/`remote_done`: the suite rollup (per `references/testrun.md`), the device line (name + OS from `remote_device`; for simulators the pool, and each member's own pair), each build the laptop uploaded (`remote_app` path → `app_id`), and the job link (`remote_dispatched.job_url`) for the dashboard's stage logs. If a member comes back `broken` with zero steps, the grid-side run refused before launching — check the app id belongs to this account (`apps list` for the active profile) and the job's scenario log at the job link. +What to present after `testrun_done`/`remote_done`: the suite rollup (per `references/testrun.md`), the device line (name + OS from `remote_device`; for simulators the pool, and each member's own pair), each build the laptop uploaded (`remote_app` path → `app_id`), and the job link (`remote_dispatched.job_url`) for the job's step logs. If a member comes back `broken` with zero steps, the grid-side run refused before launching — check the app id belongs to this account (`apps list` for the active profile) and the job's scenario log at the job link. ## Real devices (`--target android|ios`) diff --git a/.claude/skills/kane-cli/references/objectives-cookbook.md b/.claude/skills/kane-cli/references/objectives-cookbook.md index a0a65de..94a935a 100644 --- a/.claude/skills/kane-cli/references/objectives-cookbook.md +++ b/.claude/skills/kane-cli/references/objectives-cookbook.md @@ -121,6 +121,8 @@ The agent automatically picks the right method based on phrasing. To get the met | **Title** | Browser tab `document.title` | "page title contains …", "title is …" | | **DevTools** | Things not visible on screen — network, console, performance, cookies, localStorage | see §3.2 below | +Under kane-cli's default assertion mode (`dom`), Visual and Textual reads both run DOM-first with a vision fallback; `--assertion-mode visual` keeps Visual reads on vision only (`references/execution-controls.md`). + ### 3.2 DevTools analyze methods Five subdomains. Each one is the right choice when the data you care about lives in the browser's internals rather than on the page. @@ -486,7 +488,7 @@ Positional assertions check where something is on the page: | "Test the checkout flow" | "Go to /cart, click Checkout, fill the address form with {{tester}}, click Pay, assert the order confirmation page loads" | "Test" has no end state — the agent doesn't know when to stop. | | "Add the item" | "Click the 'Add to Cart' button on the first product card" | Vague target — agent may click the wrong element. | | "Tell me the price" | "Store the cart total as 'total'" | Vague verbs don't extract — use "store" / "extract" / "get". | -| Hardcode credentials in the objective | Use `{{username}}` / `{{password}}` from `--variables-file` | Credentials in plain text leak into logs and TMS. | +| Hardcode credentials in the objective | Use `{{username}}` / `{{password}}` from `--variables-file` | Credentials in plain text leak into logs and Test Manager. | | Omit the URL | "Go to https://example.com/login first, then …" | Agent doesn't know where to start. | | Cram 25 operations into one objective | Split at logical boundaries (login, navigate, action, verify) | Long runs drift and stall. | | "Check the page is fast" | "Assert LCP is under 2500ms and CLS is below 0.1" | Use the explicit web-vital metric, not a vague "fast." | diff --git a/.claude/skills/kane-cli/references/parallel.md b/.claude/skills/kane-cli/references/parallel.md index 0866044..fdc63ca 100644 --- a/.claude/skills/kane-cli/references/parallel.md +++ b/.claude/skills/kane-cli/references/parallel.md @@ -4,7 +4,7 @@ For multiple independent browser tasks, decompose and run in parallel using the Agent tool. -> **Saved tests? Use testrun instead.** If the tasks are committed `_test.md` files, do NOT hand-roll parallelism — `kane-cli testrun run --parallel N < /dev/null` gives you isolated Chromes, a pooled scheduler, one rollup, and one evidence pack. Read `references/testrun.md`. This reference is for **ad-hoc `run` objectives** only. +> **Saved tests? Use testrun instead.** If the tasks are committed `_test.md` files, do NOT hand-roll parallelism — `kane-cli testrun run --parallel N < /dev/null` gives you isolated Chromes, a pooled scheduler, one rollup, and one evidence pack. Read `references/testrun.md`. This reference is for **ad-hoc `run` objectives** only. In a folder linked to a shared project, these runs go to the profile's Test Manager project, which may not be the shared one: run them from a folder that isn't linked (`shared-projects.md` §9). ## When to Split diff --git a/.claude/skills/kane-cli/references/parsing.md b/.claude/skills/kane-cli/references/parsing.md index 823702a..2b5a971 100644 --- a/.claude/skills/kane-cli/references/parsing.md +++ b/.claude/skills/kane-cli/references/parsing.md @@ -1,4 +1,4 @@ -<!-- Read this when you need the full NDJSON event schema for kane-cli --agent output, or when SKILL.md §5's summary is not enough. Owns event types (progress / bifurcation / child_agent_start|end / ask_user / error), parsing strategy, run_end terminal-event schema, and ask_user/cancel responses. --> +<!-- Read this when you need the full NDJSON event schema for kane-cli --agent output, or when SKILL.md §5's summary is not enough. Owns event types (progress / bifurcation / ask_user / error / coverage), parsing strategy, run_end terminal-event schema, and ask_user/cancel responses. --> # Parsing --agent Output @@ -21,14 +21,14 @@ The first line on every surface is an opening event: {"type":"stream_start","cli_version":"0.8.17","surface":"run","pid":16664,"v":1,"ts":"2026-09-21T08:47:26.889Z"} ``` -`surface` is `run`, `testmd` or `testrun`. Use `cli_version` to tell whether a newer event or flag is available. `session_dir` may also be present when a session already exists. +`surface` is `run`, `testmd`, `testrun` or `explore` (`references/explore.md`; it ends with `done`). Use `cli_version` to tell whether a newer event or flag is available. `session_dir` may also be present when a session already exists. Rules a parser must follow: - **Ignore unknown fields and unknown event types.** New ones can appear in any release without a `v` bump. - **Never assume the first line is a progress line**, and skip any line that is not JSON. - Step lines on `run` stay **typeless** (below). Do not look for `type: "step"`. -- The documented completion event is always the last line: `run_end` for `run`, `test_md_done` for `testmd run`, `testrun_done` for `testrun run` (then `remote_done` on cloud grid runs). +- The documented completion event is the last line of a run that went ahead: `run_end` for `run`, `test_md_done` for `testmd run`, `testrun_done` for `testrun run` (then `remote_done` on cloud grid runs). A run refused before it starts can end on an `error` with no completion event: read the exit code. **The same stream is also written to disk**, line by line as it happens: `<session_dir>/events.ndjson`, byte for byte what stdout printed. While a run is live, kane-cli keeps a small pointer file at `~/.testmuai/kaneai/sessions/active/<pid>.json` (`pid`, `cwd`, `surface`, `session_dir`, `started`, `cli_version`, `host_agent`) and removes it on exit. You normally need neither: one blocking call hands you the whole stdout. They exist for watchers such as the live strip (`references/live-strip.md`), and the log is where a suite keeps each test's own events (`references/testrun.md`). The log holds exactly what stdout held, so treat it with the same care. @@ -48,6 +48,7 @@ Rules a parser must follow: | `step` | number | Step index. It can run one ahead of the step the person would count (a `bifurcation` takes the first slot), so count completed `done`/`failed` lines for "steps taken" rather than reading the last index | | `status` | string | `"running"` at start; `"done"` or `"failed"` at completion | | `remark` | string | What the agent did or why it failed | +| `child_id` | string, optional | The step belongs to a child flow. Keep children apart when step indices repeat | These are **untyped** — they have no `type` field. Do **not** key on `event.type === 'step_start'` or `'step_end'`; those event types are not emitted. @@ -57,13 +58,14 @@ These are **untyped** — they have no `type` field. Do **not** key on `event.ty |-------|-----------|---------| | `project_folder_auto_defaulted` | resolved project + folder (id, name) | Run-startup gate auto-resolved a project/folder when none was configured (or the cached one was stale/invalid). Fires before any progress event on `run` / `testmd run`. Translate to plain language (see `references/test-manager.md`). | | `bifurcation` | `flows[]`, `count` | Agent split objective into sub-flows | -| `child_agent_start` | `child_id`, `objective`, `parent_step` | Child agent spawned | -| `child_agent_end` | `child_id`, `success`, `steps_taken`, `summary` | Child agent finished | | `ask_user` | `question`, `step_index`, `options?` | Agent needs user input | -| `error` | `message` | Error occurred | +| `error` | `message` | Error occurred. On `testmd run` / `testrun run` in a folder linked to a shared project it also carries a `code`, and `remedy`, `next` or `support_id` when kane-cli has one (`agent-contract.md` §3.7) | | `warning` | `code`, `message`, … | *(0.8.15+)* Before any progress event: `code: "unresolved_variables"` — a `{{name}}` had no value; the run continues. Schema below. | | `test_md_evidence_ingest` | `status: "ok"\|"failed"`, `evidence_id`, `stage?` (failure only) | `testmd run` only: a replay's evidence pack published to the dashboard. Informational. | | `test_md_bundle_sync` | `status: "ok"\|"failed"`, `commit_id`, `bytes?` (success) / `stage?` (failure) | `testmd run` / `testmd sync`: test bundle pushed to the cloud after an authored commit. Informational. | +| `coverage` | `status` (`written`, `none`, `skipped`, `failed`), `pct?`, `reason?`, `pack` | `run` / `testmd run`: one per sealed pack, just before the completion event, which repeats it as `coverage`. `skipped` (nothing to measure here, as in a folder with no assurance work): say nothing. In a linked folder: `agent-contract.md` §3.7 | +| `update_available` | `current`, `latest`, `severity` | `run` (also `generate --agent`), at any point before `run_end`, or not at all: a newer kane-cli exists. Mention it once, after the result | +| `skill_update_available` | `current`, `latest`, `install_cmd` | `run` only, like `update_available`: a newer skill exists. After the result, offer `kane-cli install skill` once (`references/ready-check.md` §4a) | | `testrun_*` family | see `references/testrun.md` | Emitted only by `kane-cli testrun run`; terminal event is `testrun_done`, not `run_end`. | **Note:** The `run` stream has no `run_start` event; startup metadata or errors can precede progress. @@ -88,7 +90,7 @@ Emitted by `run`, `testmd run` and (after `testrun_plan`) `testrun run` when an | `variables[].reason` | `value_missing` — the key exists in `variables[].file` with an empty value · `not_declared` — the key is in no variable file · `not_a_dataset_column` — a `${x}` that is no column of the Test Manager dataset the run was given (`variables[].dataset` names it) | | `variables[].file` | the pool file that holds the empty key (`value_missing` only); `--variables` when it came inline | | `variables[].used_by[]` | `{file, step}` — `file` is `objective` for `kane-cli run`, else the test file (flattened step index) | -| `suggested_file` | where to add a `not_declared` key: `.testmuai/variables/assurance.json` inside an assurance store, `variables.json` otherwise | +| `suggested_file` | where to add a `not_declared` key; use it as given: `.testmuai/variables/assurance.json` in a folder with its own assurance store (`.context/`), `.testmuai/variables/variables.json` otherwise (a linked folder too: `shared-projects.md` §5) | Report it with the run's outcome. If a later step failed on a literal placeholder, point at this. diff --git a/.claude/skills/kane-cli/references/ready-check.md b/.claude/skills/kane-cli/references/ready-check.md index cc5569b..c72becb 100644 --- a/.claude/skills/kane-cli/references/ready-check.md +++ b/.claude/skills/kane-cli/references/ready-check.md @@ -25,7 +25,12 @@ Add a flag only when the request needs it: | A local mobile run | `--mobile emulator` or `--mobile simulator` | `## mobile` (device tooling readiness) | | A cloud grid suite (`--remote`) | `--grid` | `## grid` (grid plugin readiness) | -The script only reads status. It changes nothing, takes about two seconds, and always exits `0`. If the script is missing (an older skill install), run `kane-cli whoami`, `kane-cli balance` and `kane-cli config show` yourself and build the same card. +The script only reads status (its sign-in check may renew a saved sign-in that is about to expire, as any kane-cli command does). It changes nothing else, takes about two seconds, and always exits `0`. + +If you can't run the script (it isn't there in an older skill install, or your host doesn't allow it), run the same +checks one at a time and build the same card: `kane-cli --version`, `kane-cli whoami`, `kane-cli balance`, +`kane-cli config show`, and look for `.testmuai/assurance.json` in the folder (only its `projectName` and `projectId`). +The preferences file is then unknown: treat the session as a first session only if the person says it is. ## 2. What the script prints @@ -34,9 +39,10 @@ Plain text in `## <section>` blocks, always in this order. Command blocks end wi | Section | Content | What you take from it | |---|---|---| | `## version` | kane-cli version, or `missing` | Installed or not. Compare with the minimum version this skill notes for a feature | -| `## whoami` | The sign-in box | `Authenticated` plus `User`, `Environment`. **Ignore `Expires`**: it is a short-lived token that renews itself, never show it | -| `## balance` | `Available credits` and `Total credits` | Credits left, rounded to a whole number | +| `## whoami` | The sign-in box: a header (`✓ Authenticated`, `✗ Not authenticated`, `⚠ Couldn't verify` or `⚠ Not logged in`), then rows such as `Profile`, `Method`, `User`, `Token` | The header and `User`. Never show the `Token` row | +| `## balance` | `Available credits` | Credits left, rounded to a whole number | | `## settings` | Settings as JSON | `project_name`, `folder_name`, `target`, `default_url` | +| `## link` | `project=<name>` and `project_id=<id>` when this folder is linked to a shared project, or `none` | Linked or not; see §3's 🔗 row and `shared-projects.md` | | `## agent-config` | The preferences file, or `none` | See `references/agent-config.md`. `none` or no `onboarding.completed_at` means this is a first session | | `## env` | `ci`, `ssh`, `display`, `os`, `arch`, `node` | Watch-mode default and whether a human is present | | `## chrome` | `found=<path>` and `override=<path>` | Chrome present for local browser runs | @@ -82,9 +88,14 @@ Row rules: - **🚀 Your app** appears only when the `app` section found a port. No row when nothing was found: never show a negative row for an optional finding. Ask for a URL only when the request lacks one. - **🌐 Chrome** appears only for local browser runs. Skip it for mobile and cloud grid requests. +- **🔗 Shared project**: when the preflight's `## link` section names a project, this folder is linked to it + (`shared-projects.md`). Add a row `🔗 **Shared project** | <name> (<id>) · designed tests run and count here`, and + word **Results go to** as "Quick `kane-cli run` checks save to <project> / <folder>; runs of the designed tests here count in <the shared project>". Read `shared-projects.md` before any assurance command. + A `link` of `unreadable` means kane-cli can't read the folder's link file, and a linked folder that also holds + `.context/` is refused too: run `kane-cli context list --mode agent` and relay its fix (`PROJECT_FILE_UNREADABLE`, `BOTH_STORES`) + before any assurance work. - **🗂️ Results go to** comes from `project_name` / `folder_name`. When they are empty, say `kane-cli will pick a default project on this run, and I'll tell you where it landed`. The offer to change it never stops the run. The change flow is in `references/test-manager.md`. - **👀 This run** states the watch mode you are about to use: the saved `preferences.watch`, or the detected default (see `references/first-run.md`). -- Name the environment (for example `stage`) only when it is not production. - For mobile or grid requests add a `📱 **Device tooling**` or `☁️ **Cloud grid**` row from the extra section. ## 4. Problems: which ones stop the run @@ -92,7 +103,8 @@ Row rules: | Problem | How you see it | Stops the run? | The fix the card offers | |---|---|---|---| | kane-cli not installed | `## version` is `missing` | Yes | Offer to run `npm install -g @testmuai/kane-cli` (or Homebrew) | -| Not signed in, or token not valid | `whoami` shows no `Authenticated`, or `exit` is not 0 | Yes | Sign-in flow below | +| Not signed in, or token not valid | `whoami` header is `✗ Not authenticated` or `⚠ Not logged in` (exit 1) | Yes | Sign-in flow below | +| Couldn't verify the sign-in | `whoami` header `⚠ Couldn't verify` (exit 3): TestMu AI didn't answer | Yes, for now | Signing in again won't help: check the connection and run the preflight again in a minute | | No credits left | Available credits is 0 | Yes | Point to https://www.testmuai.com/pricing/ to pick a plan | | Chrome missing | `found=` is empty, local browser run | Yes | Install hint for the platform, or `KANE_CLI_CHROME_PATH` for a custom location | | Low credits | Available credits under 100 | No | One warning line on the card | @@ -102,6 +114,28 @@ Row rules: When a problem stops the run, do not launch. Show the card, offer the fix, and wait. +## 4a. Get or update the skill + +The skill ships as its own package, separate from kane-cli. Most kane-cli commands (not `context`, `explore`, or anything +run with `--mode agent`) may print a +`Skill update available: <installed|not installed> → <version> — run \`kane-cli install skill\`` line on stderr. A +`kane-cli run --agent` stream also carries the notices as events on stdout: `skill_update_available` (`current`, +`latest`, `install_cmd`), so offer `kane-cli install skill`; `update_available` (`current`, `latest`, `severity`) is +kane-cli's own update (`cards.md` §1). + +```bash +kane-cli install skill # the newest skill this kane-cli supports +kane-cli install skill <version> # exactly that version, e.g. one your team's guide names +``` + +- It writes the skill for Claude Code, Codex CLI and Gemini CLI at once (`~/.claude/skills/kane-cli/`, + `~/.agents/skills/kane-cli/`, `~/.gemini/skills/kane-cli/`). Restart the agent to load it. +- With no stdin terminal it prints one JSON line: `{"type":"skill_install_result","status":"installed|already-installed|failed","version":…,"installed_targets":<n>,"error":…}`; exit 1 on `failed`. +- A pinned version installs even when it is older or newer than the default. A later bare `kane-cli install skill` never + downgrades, so it keeps a newer pinned version in place. +- `kane-cli install skill` doesn't write the preferences file `agent-config.md` reads, so the first session still counts + as a first session. + ## 5. Sign-in - **Default:** offer to open the sign-in page, then run `kane-cli login --oauth` yourself with a generous timeout. It opens the browser, waits for the person to finish, and returns. It works without a TTY. diff --git a/.claude/skills/kane-cli/references/setup-and-config.md b/.claude/skills/kane-cli/references/setup-and-config.md index 1800ccc..2e80974 100644 --- a/.claude/skills/kane-cli/references/setup-and-config.md +++ b/.claude/skills/kane-cli/references/setup-and-config.md @@ -4,6 +4,7 @@ ## Install and auth + Before first use, verify installation and auth. > **Local mobile setup (macOS Apple Silicon)** is separate: install Xcode 16+ (iOS) or Android Studio with one `arm64-v8a` AVD (Android), then `kane-cli login` and `kane-cli doctor --target emulator|simulator --install` (installs kane-cli's managed test tooling); `kane-cli doctor --target <kind>` checks readiness. Not needed for `testrun run --remote`, which runs mobile suites on the cloud grid from any machine (see §Remote execution plugin), nor for **real devices** (`--target android|ios`), which need only `kane-cli login` and a KaneAI Max or KaneAI credits + Real Device Plus Automation Cloud plan. Read `references/mobile.md`. @@ -22,7 +23,7 @@ The preflight script covers this along with credits and settings in one call (`r kane-cli whoami ``` -`whoami` prints a box, not JSON, even when piped. Its `Expires` line is a short-lived token that renews itself: never show it to the person. +`whoami` prints a box, not JSON, even when piped. Show the person only its header and the `User` row; never the `Token` row. If this shows "not configured" or errors, run login: @@ -35,7 +36,7 @@ kane-cli login --username <user> --access-key <key> This creates the default profile with basic auth, auto-selects the KaneAI project, and marks setup complete. Credentials come from the user's TestmuAI dashboard (Settings → Keys). Optional flag: -- `--profile <name>` — profile name (default: last selected profile check using `config show`) +- `--profile <name>` — profile name (default: `default`; pass a name to sign in to another profile) ### Login (OAuth) @@ -79,6 +80,10 @@ Variables parameterize objectives with reusable values and secrets. Use `{{key}} 2. `{cwd}/.testmuai/variables/*.json` (local project overrides) 3. `--variables-file <path>` 4. `--variables '{...}'` (inline JSON) +5. a saved test's own `variables:` frontmatter (`testmd run`, `testrun run`) +6. a step's own `variables:` (that step only) + +A Test Manager dataset row's `${col}` values also win over a pool entry of the same name. When a value you changed in a pool file doesn't reach the test, look for a higher layer that sets the same name. **Values (0.8.12+):** a string is a value when non-empty; a number is accepted and loaded as its string; a boolean, object or array is not a value. An empty `value` is a declared-but-unfilled key. @@ -128,9 +133,8 @@ kane-cli feedback --test-id <id> --feedback-type <positive|negative> --details " ├── config.json # Shared auth configuration ├── global-memory.md # Global agent context ├── chrome-profile/ # Default Chrome user profile -├── profiles/ # Stored credentials -│ └── {profile}/{env}/ -│ └── credentials +├── profiles/ # Stored credentials, one folder per profile +│ └── {profile}/ ├── sessions/ # Session history │ └── {session-id}/ │ ├── session.json # Metadata, run list, upload status diff --git a/.claude/skills/kane-cli/references/shared-projects.md b/.claude/skills/kane-cli/references/shared-projects.md new file mode 100644 index 0000000..24eb356 --- /dev/null +++ b/.claude/skills/kane-cli/references/shared-projects.md @@ -0,0 +1,281 @@ +<!-- kane-cli skill reference: shared projects (a folder linked to a team's Test Manager project). Read before any assurance command when the person works with a team, or when a folder holds .testmuai/assurance.json. Requires kane-cli 0.10.0+. --> + +# Shared projects: a team's work in one Test Manager project + +A **shared project** is a Test Manager project that a team's folders are **linked** to. In a linked folder the assurance +commands (`context`, `design`, `maintain`, `cover`) work in the project itself, not in a local `.context/` store. What one +person saves, everyone on the team sees: use-cases, criteria, scenarios, tests, Test Manager cases, coverage. + +Read this before the first assurance command of a session whenever: +- the person mentions a team, a teammate, a Test Manager project, or "our project"; +- the folder holds `.testmuai/assurance.json` (it is linked; the file names its `projectId` and `projectName`); +- a command prints `using project "<name>", shared with your team`, or emits `folder_linked`. + +The commands below are the agent forms. Everything in a linked folder runs from **the folder that holds +`.testmuai/assurance.json`**: a subfolder of a linked folder is not linked. + +## 1. Where the work goes: decide first + +In agent mode, a folder that isn't set up yet goes **local without asking** unless you pass `--project <id>`. So decide +before the first `context ingest`: + +| The person… | Do | +|---|---| +| works in a folder that already has its own `.context/` (and no link) | keep working locally, as before (`assurance.md`); to share that store, see below | +| works with a team, or names a Test Manager project | link this folder to that project (§2–3) | +| works alone, or asks to keep it on this machine | work locally: the journey in `assurance.md` §1 | +| isn't clear | ask once: "Should this work go to your team's Test Manager project, or stay in this folder?" | + +A folder is one or the other, never both: a folder with its own `.context/` refuses `--project` (`FOLDER_CHOSEN`), and a +folder with both refuses every assurance command (`BOTH_STORES`). To share an existing local store with a team through a +Git repository, a bucket or a folder instead, see `context-sync.md`; that is a different way of sharing and doesn't reach +Test Manager. + +## 2. Find or make the project + +```bash +kane-cli projects list --search "<part of the name>" --agent +kane-cli projects create "<name>" --description "<one line>" --agent +``` + +- `projects list` prints one JSON line per project, `{"id":"<project id>","name":"<name>"}`, then a page line + `{"_meta":"page",…,"has_more":true|false}`. There is no `type` and no `done`. `--limit` (1–200, default 10) and + `--offset` page through; `--search` matches any part of the name. +- `projects create` prints `{"id":…,"name":…}`. It does **not** link the folder and doesn't change where `kane-cli run` + saves. Create a project only when the person asks for a new one; otherwise use the team's. +- Errors from `projects` are plain text on stderr (exit 2 for usage or sign-in, 1 for a Test Manager refusal), not events. +- **When the shared path isn't open to the person:** if `projects list` is refused (exit 1: Test Manager refused or + failed the request), or it lists no project the person can work in and they don't want a new one, work locally + (`assurance.md` §1). Tell the person why, in plain words, for example "I couldn't reach your Test Manager projects, + so this work stays in this folder for now". Try the shared path once only; never retry it in a loop. +- A linked folder's own project id is in `.testmuai/assurance.json` (`projectId`). + +### This folder's project id + +A teammate's agent needs it to join (`kane-cli context pull --project <id>` where the pull exists, or +`kane-cli context extract --project <id>`), and so does anything that reads the project, such as a dashboard. Give the +person the name and the id together. Ways to get it: + +| Where | What | +|---|---| +| a linked folder's `.testmuai/assurance.json` | its `projectId` field (and `projectName`); read only those two fields | +| the skill's preflight | its `## link` section: `project=<name>`, `project_id=<id>` | +| the `folder_linked` event | `project_id` and `project`, when a command links the folder | +| `kane-cli projects list --search "<name>" --agent` | `{"id","name"}` for every project the person can open | + +## 3. Link this folder + +```bash +kane-cli context ingest <files> --project <id> --mode agent # link, then add the documents and find their use-cases +kane-cli context extract --project <id> --mode agent # link and read what the project hasn't read yet +``` + +- `context ingest`, `context extract` and, where its help lists it, `context pull` (§7a) take `--project`. The link is + made once; later commands in the folder need no flag. `--project` with the folder's own project is harmless; a + different id refuses `FOLDER_CHOSEN`. +- `{"type":"folder_linked","project_id":…,"project":…}` confirms the link. It can come before a check that still refuses + the run, so read the events after it. Tell the person: "This folder is now linked to <project> (<project id>); what you + save goes to the project." +- If the project refuses its first session (`PERMISSION_REQUIRED`: it needs someone allowed to create test cases), the link + is removed again; ask the person to get someone allowed to create test cases in that project to start the first + session, or to pick another project. After `PROJECT_NO_ACCESS` the folder **stays linked**, and every + assurance command refuses the same way until the person gets access or removes `.testmuai/assurance.json` (their call). + A `context pull --project` that is refused at the start, or can't read the project, removes the link it made: check + the folder again after one. +- A person at a terminal is asked instead: "where should this folder's work go?", then a project picker (Tab makes a new + project). Agents never see these questions. +- There is no unlink command. Linking a folder to another project means removing `.testmuai/assurance.json` first; that + is the person's call, never yours. + +## 4. The journey in a shared project + +| Step | Command | What to watch | +|---|---|---| +| 1. Add the documents and find their use-cases | `kane-cli context ingest <file> --project <id> --mode agent` (the first time), later `kane-cli context ingest <file> --mode agent` | one document per run; it ends `session_complete` with two `next` commands, save and discard (none when it added nothing) | +| 2. Save the session | `kane-cli context sessions save <sid> --yes --mode agent` | **nothing is in the project until this runs.** `session_save` `phase: "saved"` confirms it | +| 3. The use-case checkpoint | `kane-cli context list --mode agent` to show them; then ask once (below) | use-cases an agent run saves are `proposed`, not approved | +| 4. Design the tests | `kane-cli design tests --use-case <uc> --mode agent --allow-unreviewed` | only after the person's yes; ends `session_complete`. To cap the scenarios (and the cost), offer `--max <n>`; add it only on the person's word | +| 5. Save the design | `kane-cli context sessions save <sid> --yes --mode agent` | writes the test files here and makes one Test Manager case per scenario; `test_files.written[]` lists the files. Before going on, read `test_files.refused[]` and `test_files.error`, and `test_manager.error` and `waiting`: a save can succeed while a file or a case is still missing. Files listed in `test_files.replaced[]` stay but won't run any more: name only the `written[]` files on `testrun run`, and offer to move the replaced ones out | +| 6. Fill the variables, and the start pages | the file in `test_files.variables_needed.file` (usually `.testmuai/variables/assurance.json`); `test_files.no_start_page[]` names tests with no start page | SKILL.md §3, before any run. Use only values the person gives you, never invented ones; write them into the file (a script is fine) without printing them to the terminal or a log. Ask the person for the site address of a test with no start page | +| 7. Run the tests | `kane-cli testrun run <the files from step 5> --agent < /dev/null` | the first run of each test writes its steps: minutes, and about 30–85 credits per test; replays are cheap (`testrun.md` §Designed tests in a shared project) | +| 8. Coverage | `kane-cli cover --mode agent` | the project's figure; a new run can take up to 5 minutes to count | + + +**The use-case checkpoint.** A linked folder has no command that approves use-cases: +`context review` is refused there, and use-cases an agent run saves stay `proposed`. So `design tests` refuses +`UC_UNREVIEWED` until it gets `--allow-unreviewed`. Show the person the use-cases you found (titles and what they cover, +from `context list --mode agent`), then ask **once for the batch**: "I found these N use-cases in <document>. Shall I design tests for them? Nobody has +approved them yet, so tell me any to leave out or change first." Pass `--allow-unreviewed` only after a yes, and only for the use-cases they keep. A person at a terminal approves them by finishing the +session there. + +**What things cost.** These are approximate figures from real runs of small documents; the person's billing page is +the authority. On the assurance commands (`context ingest`, `context extract`, `design tests` …) the stream's +`usage.total_credits` gives the real figure. A local `testrun run` stream carries no credits total (not on `testrun_done`, +a member's end event, `testrun_summary` or `coverage`), so run `kane-cli balance` before and after the run and give the +difference: + +| Step | Credits (approximate) | +|---|---| +| `projects list`, `projects create`, `context list`, `context sessions`, `cover`, a save, `explore list`, `explore context status` | 0 | +| linking (`context extract --project` when every document is already read) and `context pull` | 0 | +| reading one document (`context ingest`, or `context extract` with something unread) | about 5 | +| designing one use-case's tests (`design tests`) | about 10–15 | +| a designed test's first run in a folder (it writes its steps; each input it checks is its own browser run) | about 30–85 per test | +| a replay | a few; a replay that has to heal, or whose steps changed since the recording, writes those steps again and can cost up to a first run | + +Tell the person the likely cost before a step that costs credits, let them choose, and give the total after. Never add +a flag that changes cost or behaviour (for example `--no-adaptive-heal`, `--max`, `--force`) on your own: offer it, and +use it only when the person asks. + +**One session at a time.** A live session holds its source or its use-case; another person's start on the same one is +refused `PROJECT_BUSY`, saying whether the session in the way is the person's own (`held_by.mine`) or a teammate's. Save or discard each session before the next one, and tell the person when a +teammate's session is in the way. `kane-cli context sessions --mode agent` lists the project's open sessions (one JSON +array; `owner` says whose). + +## 5. Where everything lives + +| What | Where | How to see it | +|---|---|---| +| sources, use-cases, criteria, scenarios, tests | in the shared project | `kane-cli context list --mode agent` lists sources, use-cases, scenarios and tests (one plain JSON row each: `id`, `label`, `title`, `trust`, `fresh`; no `done`); criteria aren't listed there. `trust: "derived"` is a proposal nobody has approved yet, `"trusted"` is approved: say *proposed* or *approved* to the person, never the field's value | +| test files (`*_test.md`) | `.testmuai/tests/` of **the folder where the session was saved**, and nowhere else | `session_save.test_files.written[]` | +| Test Manager cases | one per scenario, in the shared project, made or updated when a design is saved | `session_save.test_manager`: counts `created`, `updated`, `already_there`, `waiting`, and `next_run[]` (the scenarios whose case updates on their next run) | +| a test's runs | on its scenario's Test Manager case | Test Manager; the run's `testrun_summary` | +| coverage | worked out by the shared project from the counted runs | `kane-cli cover --mode agent` | +| variables | names: in the saved tests; empty stubs: this folder's `.testmuai/variables/assurance.json`; values: whatever the person writes there | `test_files.variables_needed` | +| a run's recording | `.testmuai/tests/output-<test>/` beside the test file, after a first run that finished | the next run replays it cheaply; a first run that stopped early leaves it incomplete, and the next run writes the steps again | + +Two rules follow: +- **Keep designed test files directly in `.testmuai/tests/`.** A designed test moved into a subfolder runs as an ordinary + test, not on its Test Manager case, and doesn't count for coverage. +- **Variable values reach the whole organisation.** Every run sends the non-secret values it loads to Test Manager: the + global and folder variable files, `--variables` and `--variables-file`, and the test's own `variables:`. Test Manager + keeps one value per name for the organisation, so a teammate's run with a different value replaces it. Never fill a + value just to silence a warning. Any `*.json` in `.testmuai/variables/` counts; prefer the file `variables_needed` names. + +## 6. A teammate joins + +From the teammate's own empty folder. Where `kane-cli context pull --help` lists `--project` (§7), join with the pull: +it links the folder, reads no documents, costs nothing and brings the project's tests (§7a): + +```bash +kane-cli context pull --project <id> --mode agent +``` + +Otherwise join with the extract: + +```bash +kane-cli context extract --project <id> --mode agent +``` + +It links the folder, then: +- every document already read: `nothing_new` (`reason: "every_source_read"`), exit 0, no credits. That needs the project + to answer within a few seconds; when it doesn't, the extract goes on and opens a session: save or discard it, as below; +- a document nobody has read yet: it reads it (credits, see §4) and ends `session_complete`. Save that session (§4 step 2), + or discard it on the person's word; while it is open it holds that document, and teammates get `PROJECT_BUSY`; +- another person's session already holds the unread document: `nothing_new` (`reason: "sources_in_a_session"`) with + `open_sessions[]`; nothing to do here. + +After the extract, `context list` and `cover` show the project. **No test files arrive with the extract's link**, so +`testrun run` has nothing to run yet: see §7. + +## 7. Get the team's tests into this folder + +Test files are written only in the folder where a design was saved. Exploration saves its tests into the project, but its +files never reach anyone's folder. + +**First check whether this kane-cli can pull them:** + +```bash +kane-cli context pull --help +``` + +If its options include `--project` and `--use-case`, use §7a. If not (kane-cli 0.10.0), it can't: `context pull` in a +linked folder is refused (`REMOTE_UNSUPPORTED`), and `--project` is an unknown option. Use §7b. + +### 7a. Pull them (when `context pull --help` lists `--project`) + +After exploration finishes, or after a teammate saves a design: + +```bash +kane-cli context pull --mode agent # in a linked folder: write the project's tests here +kane-cli context pull --project <id> --mode agent # in a folder that isn't set up yet: join (reads no documents), then pull +kane-cli context pull --use-case <uc> --mode agent # only that use-case's tests, with the tests they use +kane-cli context pull --overwrite --mode agent # replace test files changed here with the project's +``` + +- The pull's own events and refusals carry `verb: "sync"`; a refusal of the pull's flags (`PULL_LOCATION_LINKED`, `PULL_FLAGS_MIXED`, `PULL_FLAGS_LINKED_ONLY`) or at the folder's entry (sign-in, access, the link) carries `verb: "pull"`. Dispatch on `type`, as always. +- One `pull_test` event per test: `{test, title, file, outcome, first_run?, imported?, reason?}`, where `outcome` is + `new`, `updated`, `unchanged`, `kept` (changed here, left alone), `replaced` (with `--overwrite`) or `not_written`. +- `first_run: true` means the test hasn't run in this folder: its first run writes its steps in a real browser, which + takes a few minutes and costs credits (§4); later runs replay. Tell the person the count and the cost first. +- `variables_needed {file, variables[], no_start_page[]}` names the variables to fill; the pull writes no stubs. If the + file it names doesn't exist yet, it is a JSON object with one entry per name: + `{"<name>": {"value": "<value>", "secret": true|false}}`. +- A variable the pull can't classify is marked `secret: true` to be safe. Ask the person whether it really is a secret + (a message to type usually isn't). A secret's value is never asked for or shown in chat: the person writes it into the + file. A value the person has already given you in their own words isn't a secret to them: write it there yourself, + without printing it, and go on (§4 step 6). +- `done.next` names `kane-cli testrun run`. A project with no tests yet (or a `--use-case` with none) sends no + `pull_test` events and ends `done` exit 0 with `next` = `kane-cli cover --mode agent`. Exit 1 when any test wasn't + written or the project couldn't be read (`PULL_FAILED`); + 2 for refusals. + +| Code | When | Do | +|---|---|---| +| — | `context pull` with no flags in a folder that isn't linked: it is the location pull, as before. In a new folder its message is "no context store here" | to join the team's project and bring its tests, run `kane-cli context pull --project <id> --mode agent` (`kane-cli projects list --agent` for the id) | +| `PULL_LOCATION_LINKED` | a location flag (a location name, `--rebase`, `--yes`, `--answer`) in a linked folder | `kane-cli context pull --mode agent` | +| `PULL_FLAGS_MIXED` | project flags mixed with location flags | use one set | +| `PULL_FLAGS_LINKED_ONLY` | `--use-case` or `--overwrite` without `--project` in a folder that isn't linked (a new one, or one with its own local store) | link it first with `--project <id>`, or drop those flags | +| `USE_CASE_UNKNOWN` | the project has no such use-case | `kane-cli context list --mode agent` for the ids | +| `PULL_FAILED` (exit 1) | the project's tests couldn't be read; nothing was written | try once more; then give the person the command, the code and the message (agent mode shows no support id here) | + +### 7b. Until then: copy only what the tests need + +Ask the person to get, from the teammate who saved the design, **only**: +- `.testmuai/assurance.json` (the link), when this folder isn't linked yet; +- the test files `.testmuai/tests/*_test.md`; +- for each test, its `output-<test>/` folder beside it, if the teammate ran it (so the first run here is a replay). + +**Never** `.testmuai/variables/` (it can hold secret values) or `.testmuai/evidence/`, and never unzip over this folder's +own `.testmuai/`: it would replace this folder's variables and its link. The receiving folder must have no `.context/` +and no link to another project. Then fill this folder's own variables (§4 step 6) and run the files by path. +Exploration's tests have no folder to copy from: they can be seen (`context list`; Test Manager) but not run until the +pull is available. + +## 8. What changes in a linked folder + +Non-assurance commands (`run`, `testmd`, `testrun`, `projects`, `config`, `evidence` …) always run. These assurance +commands are refused with `REMOTE_UNSUPPORTED` (exit 2): + +| Refused | Use instead | +|---|---| +| `context review`, `context explain` | `kane-cli context list --mode agent`; the checkpoint in §4 | +| `cover gaps` | `kane-cli cover --mode agent` | +| `context sessions show`, `context sessions clean` | `kane-cli context sessions --mode agent`; `… sessions discard <sid> --yes --mode agent` | +| `context sync`, `push`, `clone` (and `context pull` where its help has no `--project`: §7b) | the shared project is the team's copy; `kane-cli context pull --mode agent` brings its tests (§7a). A location pull is refused `PULL_LOCATION_LINKED` | +| `context view`, `fsck`, `rebuild`, `retire`, `revert`, `name`, `design explain` | none in a linked folder | +| `--mode ci` on `extract`, `design tests` and `maintain reconcile` | `--mode agent` | +| `context ingest --plan`, `--mode override` | `kane-cli context ingest <file> --mode agent` | +| `context list --type` other than `source`, `usecase`, `scenario`, `test` | one of those four | +| `context list --all` | `kane-cli context list --mode agent` | +| `maintain reconcile --plan`, `--apply`, a URL in `--from` | `kane-cli maintain reconcile --from <file> --source-id <id> --mode agent`; for an address, `kane-cli context ingest <address> --mode agent` | +| `testrun run --from-context <ids>` | name the files: the refusal's `next` does | + +In a linked folder `extract --plan` and `--with-source` are ignored, and an extract that read two or more documents and +then paused can't be resumed (its `next` adds them again or discards). Give each `context ingest` one document. + +## 9. `kane-cli run` in a linked folder, and "two projects" + +`kane-cli run` saves its test to the **profile's** Test Manager project (`config project`), not the folder's, unless the +profile's project is this folder's project. With `--name` it also writes the test into this folder's `.testmuai/tests/`. +A `testrun run` that selects tests from two projects then refuses `PROJECT_MISMATCH`. + +- In a linked folder, never use `kane-cli run` for a first result: the first result is the journey in §4. +- Quick `run` experiments go in a separate folder that isn't linked. +- On `PROJECT_MISMATCH`: read `projects[]` (`project_id`, `name`, `this_folder`, `tests`). Its `next` runs one project's + tests, by path or with a `--match`: this folder's project's when it has any, else another's. Run it, and tell the person + which tests were left out and why. +- **The lasting fix**, with the person's word: move the tests listed under the project where `this_folder` is `false`, + each with its `output-<test>/` folder, out of `.testmuai/tests/` into a folder that isn't linked. Until then, name the + files on every `testrun run`. +- Recovery for every shared-project refusal: `agent-contract.md`. diff --git a/.claude/skills/kane-cli/references/test-manager.md b/.claude/skills/kane-cli/references/test-manager.md index 65e9030..f00a2d7 100644 --- a/.claude/skills/kane-cli/references/test-manager.md +++ b/.claude/skills/kane-cli/references/test-manager.md @@ -2,6 +2,11 @@ # Test Manager Reference — Agent Surface +> **Two different "projects".** This page is about the **profile's** project: where `kane-cli run`, `testmd` and +> `generate` save. An assurance folder **linked** to a team's shared project uses the link's project instead, and +> `projects create` or `config project` never change a folder's link (`shared-projects.md` §2–3). In a linked folder, +> `kane-cli run` still saves to the profile's project (`shared-projects.md` §9). + Every kane-cli session uploads to a TestmuAI **Test Manager (TMS)** test case. The session lands inside a **project** (required) and optionally a **folder** inside that project. This page is the agent-facing surface for everything that touches TMS: where the run ends up, how to browse and create projects/folders programmatically, and the event the run-startup gate emits when nothing is configured. Field names below are for parsing only — translate to plain language for the user, per the §5 rule in `SKILL.md`. @@ -10,13 +15,13 @@ Field names below are for parsing only — translate to plain language for the u ## 1. Where a run lands -After a successful upload, the terminal `run_end` event (`testmd_done` for `testmd run`) carries the dashboard link: +After a successful upload, the terminal `run_end` event of `kane-cli run` carries the dashboard link: ```json {"type":"run_end","status":"passed", "test_url":"https://test-manager.lambdatest.com/projects/<id>/test-cases/<id>", ...} ``` -For `testmd run`, the `test_md_summary` / `test_md_done` events also carry a `share_url` — a 7-day, no-login link suitable for CI artifacts. +For `testmd run`, the `test_md_summary` / `test_md_done` events carry no `test_url`; they carry a `share_url` when the upload made one — a 7-day, no-login link suitable for CI artifacts. Surface `test_url` (and `share_url` when present) as a "View in Test Manager" line per the §1.4 results table in `SKILL.md`. Never paste raw URLs into the middle of a summary — they belong in the results table. @@ -45,7 +50,7 @@ kane-cli folders list --project <id> [--search <q>] [--limit <n>] [--offset <n> | Flag | Purpose | |---|---| | `--search <q>` | Filter by name (substring match). | -| `--limit <n>` | Page size. Default is small (~10). Very large values are capped. | +| `--limit <n>` | Page size: a whole number from 1 to 200 (default 10). A value outside that is refused (exit 2). | | `--offset <n>` | Skip the first N rows. | | `--agent` | Force NDJSON. Auto-on when stdout is piped/redirected, but pass it explicitly anyway. | @@ -165,6 +170,5 @@ The results project and folder belong to kane-cli, not to the agent config. A ch | Code | Meaning | |---|---| | 0 | OK | -| 2 | Auth/setup error (missing or invalid credentials) or unknown subcommand | - -Other codes are run-specific (`run` / `testmd run`) and don't apply to `projects` / `folders`. +| 1 | Test Manager refused or failed the request (the message is on stderr) | +| 2 | Auth/setup error (missing or invalid credentials), a bad flag value, or unknown subcommand | diff --git a/.claude/skills/kane-cli/references/testmd.md b/.claude/skills/kane-cli/references/testmd.md index 62d35b9..919921b 100644 --- a/.claude/skills/kane-cli/references/testmd.md +++ b/.claude/skills/kane-cli/references/testmd.md @@ -2,7 +2,7 @@ # Saving & Replaying Tests with testmd -The §3 `run` command is the **primary** mode — one-shot, ephemeral. `testmd` is the **secondary** mode: tests live as `_test.md` files on disk, each step is cached on the first run, and every later run **replays from cache** with no LLM cost. +The §3 `run` command is the **primary** mode — one-shot, ephemeral. `testmd` is the **secondary** mode: tests live as `_test.md` files on disk, each step is cached on the first run, and every later run **replays from cache**, which mostly avoids the authoring cost. Use `testmd` whenever the user wants the test to persist. The decision is binary — once a test exists as a file, every later invocation is `testmd run`, never `run`. @@ -22,7 +22,7 @@ If unclear, ask: "Do you want me to save this test so you can re-run it later?" ## Quick start -Write the file (any path; filename must end in `_test.md`): +Write the file (any path; filename must end in `_test.md`). In a folder linked to a shared project, read `shared-projects.md` §9 first. ```markdown --- @@ -75,7 +75,7 @@ Click submit and verify the confirmation banner. | `mode` | root | `action` (halts on auth walls) or `testing` (default — pushes through so negative-test assertions can fire) | | `url` | root | Start URL for the first step (bare domains get `https://`). Overridden by `--url`; falls back to config `default_url`. | | `tags` | root | Labels for batch selection: YAML list, `[a, b]`, or bare `a, b`. Trimmed, lowercased, deduped. Selected with `testrun --tags` (`references/testrun.md`); shown by `testmd list`. Empty → parse error; per-step → parse error. | -| `max_steps` | root + step | Max agent reasoning steps. Engine fallback `30`; CLI default `50` (step config overrides). | +| `max_steps` | root + step | Max agent reasoning steps. Default `50`; a value in the file wins over `--max-steps`. | | `timeout` | root + step | Hard kill per step in seconds. | | `headless` | root | No browser window. | | `variables` | root + step | `{{name}}` params, same shape as §3, with `secret: true` for credentials. Counts as a value for the pre-run check (0.8.12+), even when `value` is empty. A `{{name}}` that is declared nowhere and has no pool value gets a warning before the run starts, and the run types it as written (SKILL.md §3) | @@ -86,7 +86,7 @@ Files ending in `_test.md` are tests (valid entry points). Any other `.md` is a ## The replay & cascade rule (CRITICAL) -On the **first** run of a test, the agent authors each step and saves a recording. On **every later run**, each step replays from its recording — no agent, no LLM cost, much faster. +On the **first** run of a test, the agent authors each step and saves a recording. On **every later run**, each unchanged step that has a recording replays from it — no authoring agent, so it mostly avoids the authoring cost and runs much faster. A step whose text changed, whose recording is missing, or that failed when recorded is authored again. A replay that heals (below) re-authors the failing steps, so it costs more. An authorable step replays only if **all** of these hold (replay-only marked steps always require their recording): - A recording for that step exists, @@ -112,10 +112,10 @@ Extract a repeating flow (login, setup, cookie banner dismissal) into a helper f ``` Rules: -- Helper filename **must not** end in `_test.md`. +- A step can import a helper (`.md`), another test (`_test.md`) or a module (`_module.md`). - Path resolves relative to the **importing file**, not the shell's cwd. - The step body must be exactly `@import <path>` — no mixed prose, no extra lines. -- The step's `yaml` block may contain **only** `optional`. Other keys are rejected. +- The step's `yaml` block may contain **only** `optional` and `variables` (values for `{{names}}` the imported file doesn't declare itself). Other keys are rejected. - `optional: true` on `@import` is allowed only at the root file, not on a nested import. Variables and context propagate into helpers. Chrome / `mode` / auth do not (root-only). @@ -129,8 +129,8 @@ Editing a helper re-authors that step in **every test that imports it**, plus ev | `kane-cli testmd run <path> --agent [flags]` | Run a test | | `kane-cli testmd list` | List `*_test.md` files under cwd (NDJSON when non-TTY; records include `tags`) | | `kane-cli testmd status <path>` | Test Manager identity + local-sync state | -| `kane-cli testmd export <path> [--language python\|javascript]` | Regenerate code export from existing recordings (no browser launch) | -| `kane-cli testmd delete <path>` | Local-only delete: removes source + `output-<stem>/`. Does NOT delete from Test Manager. | +| `kane-cli testmd export <path> [--language python\|javascript]` | Write code from existing recordings (no browser launch). An existing export is reused; `--force` writes it again | +| `kane-cli testmd delete <path> --yes` | Local-only delete, on the person's word: removes source + `output-<stem>/`. Does NOT delete from Test Manager. Without `--yes` it refuses (exit 2) | | `kane-cli testmd sync <path>` | Re-push the test bundle (test + imports + outputs) to the cloud. Rarely needed — it happens automatically after every authored commit; use only to recover a failed auto-sync. | **Running several tests at once?** Use `kane-cli testrun run` — one execution, one pack, isolated parallel Chromes. Read `references/testrun.md`. @@ -142,14 +142,14 @@ Editing a helper re-authors that step in **every test that imports it**, plus ev | `--url <url>` | frontmatter `url:` / config `default_url` | Start URL for the first step. Overrides the `url:` frontmatter key and config `default_url`. | | `--allow-missing-url` | off | Non-TTY only: start from the browser's current page instead of failing when the first step has no start URL. | | `--name <name>` | none | Persist the run under this name. Regex `[a-zA-Z0-9_-]+`. | -| `--on-lock-conflict <readonly\|fail\|wait>` | none | Behavior when another user holds the test's edit lock. `readonly` = replay-only / no upload, `fail` = exit 2, `wait` = block until released | +| `--on-lock-conflict <readonly\|fail\|wait>` | `readonly` | Behavior when another user holds the test's edit lock. `readonly` = runs, but saves no new version of the test (any re-authoring stays local), `fail` = exit 2, `wait` = retry every 30 s until released, with no time limit | | `--no-adaptive-heal` | healing enabled | Disable adaptive healing after replay failure | | `--author` | off | Force authorable steps to re-author; replay-only steps still replay | | `--bug-detection <off\|stop\|continue>` | config (`off`) | Flag suspected product bugs while **authoring** (`stop` halts on a confirmed bug; `continue` records it). Replay failures always investigate regardless. | Common flags include `--agent`, `--headless`, `--max-steps`, `--timeout`, and `--variables`. Command-specific flags differ; check `kane-cli testmd run --help`. -Flag wins over frontmatter for everything **except** `variables` — the file owns variables; you can add new keys via flags but cannot override file-defined ones. +For step settings such as `max_steps` and `timeout`, a value in the file (frontmatter or a step's `yaml`) wins over the flag; the flag fills only what the file leaves out. Browser flags (`--headless`, `--cdp-endpoint`, `--ws-endpoint`) and `--url` win over the file. The file owns `variables`: you can add new keys via flags but cannot override file-defined ones. ## Output: `output-<stem>/` and `Result.md` @@ -165,7 +165,7 @@ output-amazon/ **`output-<stem>/` is commit-safe and should be committed to git.** That's how teammates and CI replay the same recordings. -For tests using `@import`, helper recordings land next to the helper file in `helper-output-<helper>-<root>-<step>/` directories. Also commit-safe. +For tests using `@import`, each imported file's recordings land next to it in its own `output-<stem>/`. Also commit-safe. Older `helper-output-<helper>-<root>-<step>/` directories are still read, and removed once that file's own recordings are written. **`Result.md`** opens in any Markdown viewer. It contains: - Frontmatter — `status`, `started`, `duration_s`, `session_id` @@ -186,7 +186,7 @@ If the user runs an ad-hoc objective with §3 `run` and decides to keep it: kane-cli run "Search for noise-cancelling headphones on amazon.com" --name amazon-search ``` -On exit, kane-cli writes `<cwd>/.testmuai/tests/amazon-search_test.md`. Move that file into the user's repo and re-run it with `testmd run`. Without `--name`, an ad-hoc `run` is ephemeral and nothing is written. +On exit, kane-cli writes `<cwd>/.testmuai/tests/amazon-search_test.md` and its recordings in `output-amazon-search/` beside it. Move the file together with that folder into the user's repo and re-run it with `testmd run`; without the folder, the next run authors again. Without `--name`, an ad-hoc `run` is ephemeral and nothing is written. In a folder linked to a shared project, don't name runs: `shared-projects.md` §9. ## CI invocation @@ -199,7 +199,7 @@ kane-cli testmd run ./tests/checkout_test.md \ - `--agent` — NDJSON to stdout (auto-enabled when stdin is not a TTY; pass explicitly anyway). - `--headless` — no window. -- `--on-lock-conflict wait` — block instead of failing if a teammate is editing the same test. +- `--on-lock-conflict wait` — wait for a teammate's edit to finish instead of running without saving a new version. - Adaptive healing recovers replay failures by default; `--no-adaptive-heal` opts out. In non-TTY runs, interactive `ask_user` prompts are disabled — a step that would wait for input fails cleanly instead of hanging forever. Write CI test steps that don't depend on mid-run prompts. (Likewise, supply a start URL via the objective, `url:` frontmatter, or `--url`; a non-TTY run with no resolvable start URL fails unless you pass `--allow-missing-url`.) @@ -210,8 +210,10 @@ Exit codes: |------|---------| | 0 | ✅ Passed | | 1 | ❌ Failed | -| 2 | ⚠️ Error (auth, setup, infra) — for `testmd`, also includes parse errors and `--on-lock-conflict fail` | -| 3 | ⏱️ Timeout or cancelled — for `testmd`, also includes `--on-lock-conflict wait` timeout | +| 2 | ⚠️ Error (auth, setup, infra) — for `testmd`, also includes parse errors, `--on-lock-conflict fail`, and `wait` when the test's saved version moved on meanwhile | +| 3 | ⏱️ Timeout or cancelled (`run`). A `testmd run` that times out or is cancelled ends `failed`, exit 1 | + +In a folder linked to a shared project, a designed test's run can also stop before it starts, or exit 1 after passing: `agent-contract.md` §3.7. ## Parse errors (when writing a `_test.md`) @@ -222,8 +224,7 @@ Parse errors abort **before** any browser launch with exit `2`. Common ones and | `frontmatter is missing closing '---'` | Add the trailing `---` | | `invalid YAML in frontmatter` | Re-validate the YAML block | | `step body must be exactly one of prose / @import` | Split into two steps | -| `step config on @import may only contain 'optional'` | Remove other keys from the yaml block | -| `cannot @import a test file` | Imports may only reference helpers (not ending in `_test.md`) | +| `step config on @import may only contain 'optional' / 'variables'` | Remove other keys from the yaml block | | `cyclic reference` | Restructure helpers to break the loop | | `chrome config is global-only` | Move Chrome key to root frontmatter | | `'<key>' is run-level and cannot be set per-step` | Move `mode` / `tags` / `on_lock_conflict` to root frontmatter | @@ -237,7 +238,7 @@ When the user reports a parse error, fix the file before retrying — don't loop Adaptive healing is enabled by default: after a replay failure the runner tries up to three shrinking replay windows, then full re-authoring of authorable steps. `--no-adaptive-heal` disables this recovery. Retired `--retry` and `--retry-count` are accepted only to print a notice; they do not enable healing or change the fixed budget. -Headings marked `@db`, `@api`, `@js`, `@smartui`, `@network_query`, or `@network_assertion` are replay-only. They still replay under `--author`, downstream divergence and adaptive healing. Missing recordings cannot be re-authored through these markers: restore the recording or recreate it through the originating workflow. Do not delete their tapes to force authoring; adding a marker does not create an operation. +Headings marked `@db`, `@api`, `@js`, `@smartui`, `@network_query`, `@network_assertion`, or (on a mobile device) `@gps`, `@accessibility`, `@media_injection`, `@network_throttle`, `@deeplink` are replay-only. They still replay under `--author`, downstream divergence and adaptive healing. Missing recordings cannot be re-authored through these markers: restore the recording or recreate it through the originating workflow. Do not delete their tapes to force authoring; adding a marker does not create an operation. Structured control flow uses balanced heading markers: `@if`, `@elif`, `@else`, `@end-if`, `@while`, `@end-while`. An `@else` must be last in its conditional; end markers must match the opened block type. These are distinct from natural-language conditionals. Markers are excluded from the step body hash. Only one replay-only kind is allowed per step, and an import cannot also be marked replay-only. @@ -255,6 +256,7 @@ This stream is **not** the one-shot `run` stream. Every line is typed, and the f | `test_md_step_end` | `step_index`, `status`, `duration_s`, `failed_sub_step_index` | The step finished. `status` is `passed`, `failed` or `skipped` | | `test_md_evidence_ingest`, `test_md_bundle_sync` | `status` | Informational, before the summary | | `test_md_summary` | `overall_status`, `duration_s`, `steps: {total, passed, failed, skipped, replay_decisions, author_decisions}` | The numbers for the card. `replay_decisions` is how many steps replayed, `author_decisions` how many were authored | -| `test_md_done` | `overall_status`, `duration_s`, `session_id`, `share_url?` | Completion. Always the last line. `share_url` is absent on a pure replay | +| `coverage` | `status` (`written`, `none`, `skipped`, `failed`), `pct?`, `pack` | One per sealed pack, just before `test_md_done` (which repeats it as `coverage`). `skipped`: say nothing (`references/parsing.md`) | +| `test_md_done` | `overall_status`, `duration_s`, `session_id`, `share_url?`, `coverage?` | Completion. The last line of a run that started (a linked folder's stop before the start: `agent-contract.md` §3.7). `share_url` is absent on a pure replay | Most lines are inner `step_event`s (screenshots, reasoning, actions). Skip them unless you are diagnosing a failure: for the card you need only the step starts and ends, the summary and the completion event. On a failure, the failing step is the `test_md_step_end` with `status: "failed"`, its title comes from the matching `test_md_step_start`, and the last inner `step_end` or `step_event` before it says what went wrong. diff --git a/.claude/skills/kane-cli/references/testrun.md b/.claude/skills/kane-cli/references/testrun.md index e3d9f98..025e62a 100644 --- a/.claude/skills/kane-cli/references/testrun.md +++ b/.claude/skills/kane-cli/references/testrun.md @@ -26,7 +26,7 @@ kane-cli testrun run [paths...] [flags] < /dev/null # NDJSON when stdin is not | `--match <regex>` | Filter candidates by project-relative path regex | The path is as the OS writes it: `tests/app/` on macOS and Linux, `tests\app\` on Windows. Quote the regex with double quotes in cmd.exe; single quotes are literal there. | | `--tags <list>` | ANY-match on frontmatter `tags:` (repeatable or comma-separated, case-insensitive) | — | | `--parallel <n>` | Worker count; each desktop worker gets an isolated Chrome with a fresh temp profile | `1` | -| `--on-failure <mode>` | `continue` (run everything) \| `fail-fast` (stop dispatching new members after a failure) | `continue` | +| `--on-failure <mode>` | `continue` (run everything) \| `fail-fast` (stop dispatching new replays after a failure; tests with no recording yet are still all written, in the pass after the replays) | `continue` | | `--name <label>` | Run title in the dashboard | derived | | `--dry-run` | Print the plan (members + preflight failures) and exit; runs nothing | off | | `--no-adaptive-heal` | Disable default adaptive healing after replay failure | healing enabled | @@ -44,9 +44,63 @@ All members must share one org + project. *(0.8.4+)* Members need **not** be aut | Reason | Plain-language meaning | What to tell the user | |---|---|---| | `org_mismatch` | Different organisation than the other tests | "Check `kane-cli testmd status <path>` — it belongs to another org" | -| `project_mismatch` | Different project than the other tests | "Run it separately or per-project" | +| `project_mismatch` | Different project than the other tests | *(0.10.0+)* the run also emits `error` `PROJECT_MISMATCH` with `projects[]` (`project_id`, `name`, `this_folder`, `tests`) and a `next` that runs this folder's project's tests by path: run it, and tell the person which tests belong to the other project | -If **any** member fails preflight, the plan is invalid: nothing runs, exit `2`. Suggest `--dry-run` to preview the plan cheaply before a big run. *(0.8.15+)* Preflight also checks variables. A `{{name}}` with no value is a warning: the member stays in the plan (`valid` does not look at variables), and the run types the name as written unless a step sets it first. The check covers every step of the member, replayed ones included. `testrun run` has no `--variables` flag, so fill the pool file (`.testmuai/variables/*.json`) or the member's own `variables:` frontmatter first. +If **any** member fails preflight, the plan is invalid: nothing runs, exit `2`. *(In a folder linked to a shared project, from the release that carries it)* the run then emits `error` `PLAN_INVALID` +with `offenders[{path, failure}]`, and `testrun_done{overall_status:"refused"}`; `PROJECT_MISMATCH` is followed by the +same `testrun_done`; a folder with no tests gives `error` `NO_TESTS` ("no tests here yet") with `next` +`kane-cli context pull --mode agent`; and a `--match` or `--tags` that selects none of the folder's tests gives `NO_TESTS` +("no *_test.md files selected") with no `next`: fix the filter. Everywhere else, and on 0.10.0, an invalid plan ends +after `testrun_plan` (and, for a mix of projects, its `PROJECT_MISMATCH` `error` above) with no `testrun_done`: read the +exit code and each member's `failure`. Suggest `--dry-run` to preview the plan cheaply before a big run. *(0.8.15+)* Preflight also checks variables. A `{{name}}` with no value is a warning: the member stays in the plan (`valid` does not look at variables), and the run types the name as written unless a step sets it first. The check covers every step of the member, replayed ones included. `testrun run` has no `--variables` flag, so fill the pool file (`.testmuai/variables/*.json`) or the member's own `variables:` frontmatter first. + +## Designed tests in a shared project (kane-cli 0.10.0+) + +In a folder linked to a shared project (`shared-projects.md`), a designed test (its file sits directly in `.testmuai/tests/` +and carries an `assurance:` block) runs **on its scenario's Test Manager case**. Before the browser opens, kane-cli finds +that case from the test's `id` and `base`. + +| The test here | What its run does | Cost | +|---|---|---| +| no `output-<test>/` beside it (never run in this folder: just saved, or pulled) | a **first run**: it attaches a new KaneAI test to the case and writes every step in a real browser, then commits it | minutes, and about 30–85 credits per test (each input it checks is its own browser run); tell the person before running it | +| has `output-<test>/` from a first run here that finished | a **replay** of the recorded steps; a step whose text changed since the recording (edited here, or replaced by a save or a pull) is written again, as on a first run | a few credits; more for each step written again | +| a replayed step fails | **heal**: it rewrites from that step, up to 3 times, then the whole test | about a first run's cost | + +- Before a first run, tell the person how many tests will write their steps and the likely cost + (`shared-projects.md` §4), and let them choose. Don't add `--no-adaptive-heal`, `--author` or other flags that change + what the run does on your own (`--headless` from the watch preference is fine); heal is on by default. +- **Credits.** A local run's own stream carries no credits total: note `kane-cli balance` + before the run and again after it, and give the person the difference. (A `--remote` run's `testrun_summary.totals.creditsConsumed`, + when present, adds up what the grid's runs reported.) +- Run the files a save or a pull named: `kane-cli testrun run <files> --agent < /dev/null`. A bare `testrun run` takes + every `*_test.md` here, and a folder that also holds a `kane-cli run` test saved to another project (the profile's) + refuses `PROJECT_MISMATCH` (above). +- A designed test moved into a subfolder of `.testmuai/tests/` runs as an ordinary test, off its case, and doesn't count. +- **Counting.** A finished run counts toward the project's coverage once Test Manager has its evidence; `cover` can take + up to 5 minutes to show it. The run's own `coverage` event (`computed_by: "project"`) shows what this run reached + when its `status` is `written`; any other status has no figure yet (`cards.md` §8). +- **A test that can't run on its case** stops before the browser, and its member ends `failed`; in agent mode its + `failure.message` says why. That test's own log (`log_path`) holds the refusal as an `error` event with its `code` and + a `remedy`; for a missing or gone case the remedy names the session to save (add `--yes --mode agent`). When it names + none, run `kane-cli context sessions --mode agent` and find the person's session. The codes and what to do: + +| Code | Means | Do | +|---|---|---| +| `TEST_CASE_MISSING` | the design that made this test was never saved, so it has no case | save that session: `kane-cli context sessions save <sid> --yes --mode agent` (the person's own session) | +| `TEST_CASE_GONE` | its case was deleted or moved to another project in Test Manager | save the session that designed it again: `kane-cli context sessions save <sid> --yes --mode agent` (free; it makes a new case) | +| `TEST_CASE_ARCHIVED` | its case is archived in Test Manager | ask the person to restore the case in Test Manager, then run the test again | +| `TEST_NOT_IN_PROJECT` | the test isn't in this folder's project, the project has no work yet, or the design it came from can't be saved any more | do what the refusal's message says: run it in the folder linked to its own project, or design the use case again on the person's word (`kane-cli design tests --use-case <uc> --force --mode agent`; add `--allow-unreviewed` when the use-case is still proposed) | +| `TEST_MANAGER_OUTDATED` | Test Manager can't attach this run (for example a device target: it attaches web runs only), or it made a case of its own for the run | a device target: run it on desktop web. Otherwise tell the person Test Manager needs an update before this test can run, and give the support id | +| `TEST_CASE_ATTACH_FAILED`, `TEST_RUN_NOT_SAVED` | the run couldn't be recorded on the case (the verdict stands for the second) | try once more; then give the person the support id | +| `PROJECT_NO_ACCESS`, `ASSURANCE_OFF`, `CREDENTIALS_MISSING`, `CREDENTIALS_REJECTED`, `CONNECTION_FAILED` | the case lookup on a first run failed | `agent-contract.md` | + +- In a terminal the person sees only "failed" for such a member; the reason is in that member's own log (the + `log_path` of its end event, `testrun_member_end` or `testrun_authored_member_end`). Read it before you report the + failure. +- **With `--remote`**, every designed test's case is looked up on this machine before anything is sent (`--dry-run` + too). If any test would stop, nothing is sent: each such test arrives as one `remote_error` with the code above, + `detail` naming the file, and its `remedy`, then `testrun_done` failed and exit `2`. Fix those tests, or leave them out with `--match` + or `--tags`. ## Remote: the suite as one HyperExecute job (`--remote`) @@ -98,7 +152,7 @@ All typed; stdout; one JSON object per line. **Local completion: `testrun_done`. | `testrun_investigations_wait` | `count` | Failed replays left investigations running; the coordinator waits before sealing. Narrate as "investigating N failures". | | `testrun_evidence_ingest` | `status: "ok"\|"failed"`, `evidence_id`, `stage?` | Pack published to the dashboard. Absent when publish is skipped. | | `testrun_summary` | `totals: {tests, passed, failed, broken, skipped, authored}`, `duration_s`, `upload`, `cancelled`, `execution: {id, status}` | Build the rollup table from this. | -| `testrun_done` | `execution_id`, `overall_status: "passed"\|"failed"\|"cancelled"` | Local completion; remote runs continue through `remote_done`. | +| `testrun_done` | `execution_id`, `overall_status: "passed"\|"failed"\|"cancelled"` (`"refused"` for an invalid plan in a linked folder), `coverage?: {status, pct?}` | Local completion; remote runs continue through `remote_done`. `coverage` repeats the run's coverage figure (`pct` only when `status` is `written`). | ### Each test's own log, and `--stream-members` (0.8.17+) @@ -112,18 +166,18 @@ With `--remote`, the stream is wrapped in typed `remote_*` events (all on stdout | `type` | Payload | Notes | |---|---|---| -| `remote_start` | `backend`, `env`, *(0.8.17+)* `log_path` | Dispatch begins. `log_path` is the grid client's own log on this machine, useful when a dispatch fails before a job exists | +| `remote_start` | `backend`, *(0.8.17+)* `log_path`; ignore any other field | Dispatch begins. `log_path` is the grid client's own log on this machine, useful when a dispatch fails before a job exists | | `remote_device` | `platform`, `slug`, `name`, `os_version`, `avd_id?`, `pool?` | The resolved grid device (emulator/simulator only). Present it as the device line. | | `remote_member_device` | `path`, `kind: emulator\|simulator\|real-device`, `platform`, `name?`, `os_version?`, `name_source`, `os_version_source` | One per device member before dispatch. Sources: `flag`, `file`, `config` (saved device), `default`, `hub` (that half left to the hub, no value). Present as one device line per member. | | `remote_device_hint` | `reason: device_name_ignored\|catalog_stale\|saved_device_unavailable\|real_device_list_unavailable`, `detail` | Informational; `device_name_ignored` is emulator-only; `real_device_list_unavailable` = devices dispatched unchecked | | `remote_app` | `path`, `app_id`, `source: uploaded\|cache\|dry-run` | One per distinct local build uploaded from the laptop (mobile); `app_id` is empty on a dry run | | `remote_dispatched` | `job_id`, `job_url` | The HyperExecute job exists — give the user `job_url` | | *(0.8.17+)* member events on remote | `testrun_start`, then a start and end event per test, each with `post_hoc: true` | The grid reports per-test detail **after the job ends**, in plan order, just before `testrun_summary`. Their `ts` is the grid's own time. Same fields as local, including `log_path` and `failure`. `testrun_progress` is not emitted on remote. On 0.8.17+ `remote_dispatched` arrives as soon as the job exists, not at the end | -| `remote_error` | `code`, `detail` | Remote preflight refused (table above); expect `testrun_done` failed + exit 2 | +| `remote_error` | `code`, `detail`, `remedy?`, `support_id?` | Remote preflight refused (table above), or a designed test can't run on its case (§Designed tests in a shared project); expect `testrun_done` failed + exit 2. `code: "internal_error"` is a crash of the remote run itself: `testrun_done` and `remote_done` failed, exit 1 | | `remote_import_tape`, `remote_exec_sync`, `remote_coverage` | `status`, `reason`, `detail?` | Informational; sync/coverage are `skipped` when the project has no `.context` store | | `remote_done` | `status`, `exit`, `job_id`, `sessions_path` | Follows `testrun_done`; `sessions_path` holds the members' grid session logs | -`testrun_summary` additionally carries `remote: {backend, jobId, jobUrl, sessionsPath}`. A member `status: "broken"` with `execution: null` and `upload: "skipped"` means the grid-side kane-cli refused before launching — send the user to `job_url` for the scenario log and check the app id's environment. +`testrun_summary` additionally carries `remote: {backend, jobId, jobUrl, sessionsPath}`. A member `status: "broken"` with `execution: null` and `upload: "skipped"` means the grid-side kane-cli refused before launching — send the user to `job_url` for the scenario log and check that the app id belongs to this account (`kane-cli apps list --target <kind> --agent`). Parsing strategy: @@ -133,7 +187,7 @@ for each line: if type === "remote_done" → capture remote status, exit and sessions_path if type === "testrun_plan" && !valid → report offenders, expect exit 2 if type === "warning" && code === "unresolved_variables" → name the variables with no value (references/parsing.md); the run continues; ignore other warning codes - if type === "testrun_member_end" → note per-member outcome + if type === "testrun_member_end" or "testrun_authored_member_end" → note per-member outcome if type === "testrun_summary" → capture totals for the rollup else → informational; narrate sparingly ``` @@ -156,7 +210,7 @@ Don't list passing tests individually. If the pack published, mention the run is ## Exit codes -`0` all members passed · `1` at least one failed/broken · `2` usage / invalid plan / auth (nothing ran) · `3` cancelled (Ctrl-C — in-flight members finish, the pack still seals). +`0` all members passed · `1` at least one failed/broken · `2` usage / invalid plan / auth (nothing ran) · `3` cancelled (Ctrl-C — no new members start, running ones are stopped, and the pack still seals with what finished). ## Evidence & debugging @@ -164,14 +218,14 @@ The suite produces **one** sealed pack, created directly in `<cwd>/.testmuai/evi ## Execution constraints -Local suites containing any mobile member require `--parallel 1`; larger values are refused. Isolated Chrome workers apply to desktop members only. Remote suites support grid concurrency via `--parallel N`. +Local suites containing any emulator or simulator member require `--parallel 1`; larger values are refused. Real-device members each open their own session, so `--parallel` is allowed up to the plan's devices (§When to use). Isolated Chrome workers apply to desktop members only. Remote suites support grid concurrency via `--parallel N`. Healing is enabled by default (three shrinking replay windows, then re-authoring of authorable steps). `--no-adaptive-heal` disables it. Retired `--retry`/`--retry-count` only print a notice and have no effect. Replay-only recorded steps retain their recordings even during healing. NDJSON selection uses stdin, not stdout (on 0.8.21+ `--agent` selects the same stream): every `kane-cli testrun run` line ends in `< /dev/null` (bash and zsh on macOS, Linux and Git Bash; `< NUL` in cmd.exe; from PowerShell run it through cmd: `cmd /c "kane-cli testrun run … < NUL"`). Check the first stdout line: on 0.8.17+ it is `{"type":"stream_start"…}`. A prose plan there instead means the CLI is in terminal mode: it prints the human view and, after a real run, opens an evidence table that waits until `q` or Esc, so the process never exits on its own. Stop it and rerun with the redirect. An empty stdout with a message on stderr is a usage error: read it. Dry-run validates a plan, not runtime authentication or browser/device readiness. Always observe process exit, including paths without a normal completion event. -### Remote behavior still requiring verification - -The audited dispatch does not forward `--bug-detection` to member commands and does not map `--on-failure` into the job template. Do not rely on these flags for remote bug-detection or fail-fast behavior until implementation owners confirm or fix the mapping. `--name` is suite metadata, not a forwarded member flag. +### Remote flags that don't reach the tests -For dispatched runs, read through `remote_done` and process exit after `testrun_done`; retain the remote status and session-log path. Preflight refusal and dry-run may terminate without `remote_done`. +On `--remote`, `--bug-detection` and `--on-failure` don't reach the tests on the grid: don't rely on them there. For +dispatched runs, read through `remote_done` and the process exit after `testrun_done`, and keep the remote status and +the session-log path. A preflight refusal or a dry run can end without `remote_done`. diff --git a/.claude/skills/kane-cli/scripts/preflight.ps1 b/.claude/skills/kane-cli/scripts/preflight.ps1 index 85b797c..161c257 100644 --- a/.claude/skills/kane-cli/scripts/preflight.ps1 +++ b/.claude/skills/kane-cli/scripts/preflight.ps1 @@ -1,9 +1,10 @@ # kane-cli ready check (preflight). Windows PowerShell 5.1 and later. # # Prints, in one call, everything an agent needs to know before it starts a -# kane-cli run. It only reads status. It changes nothing: no sign-in, no -# config write, no install, no network call of its own. It never reads -# credential files. It always exits 0, and problems show up in the text. +# kane-cli run. It only reads status: no sign-in, no config write, no install, +# no network call of its own, and it never reads credential files itself. The +# kane-cli status commands it runs use the saved sign-in, and whoami may renew +# one that is about to expire. It always exits 0, and problems show up in the text. # This is the twin of preflight.sh: same sections, same keys, same order. # # Usage: powershell -File preflight.ps1 [--mobile emulator|simulator] [--grid] @@ -13,6 +14,7 @@ # whoami kane-cli whoami, then exit=<code> # balance kane-cli balance, then exit=<code> # settings kane-cli config show, then exit=<code> +# link project= and project_id= of a linked folder, "none", or "unreadable" # agent-config saved agent preferences, or the word "none" # env ci= ssh= display= os= arch= node= # chrome found=<path> and override=<KANE_CLI_CHROME_PATH> @@ -117,6 +119,23 @@ try { Write-Output '## settings' Write-CommandBlock -CliArgs @('config', 'show') + Write-Output '## link' + # A folder linked to a shared project holds .testmuai/assurance.json; print only its project's name and id, read as + # kane-cli reads them, or "unreadable" when kane-cli can't read the link either. + $linkFile = [IO.Path]::Combine((Get-Location).Path, '.testmuai', 'assurance.json') + if (Test-Path -LiteralPath $linkFile) { + $link = $null + try { $link = Get-Content -LiteralPath $linkFile -Raw -Encoding UTF8 -ErrorAction Stop | ConvertFrom-Json -ErrorAction Stop } catch { } + if (($link -is [System.Management.Automation.PSCustomObject]) -and ($link.projectId -is [string]) -and $link.projectId -and ($link.projectName -is [string])) { + Write-Output ("project=" + ($link.projectName -replace "[`r`n]+", ' ')) + Write-Output ("project_id=" + ($link.projectId -replace "[`r`n]+", ' ')) + } else { + Write-Output 'unreadable' + } + } else { + Write-Output 'none' + } + Write-Output '## agent-config' $agentConfig = [IO.Path]::Combine($HOME, '.testmuai', 'kaneai', 'agent-config', 'config.json') $agentConfigText = $null diff --git a/.claude/skills/kane-cli/scripts/preflight.sh b/.claude/skills/kane-cli/scripts/preflight.sh index 82dc3ec..232f69e 100755 --- a/.claude/skills/kane-cli/scripts/preflight.sh +++ b/.claude/skills/kane-cli/scripts/preflight.sh @@ -2,9 +2,10 @@ # kane-cli ready check (preflight). # # Prints, in one call, everything an agent needs to know before it starts a -# kane-cli run. It only reads status. It changes nothing: no sign-in, no -# config write, no install, no network call of its own. It never reads -# credential files. It always exits 0, and problems show up in the text. +# kane-cli run. It only reads status: no sign-in, no config write, no install, +# no network call of its own, and it never reads credential files itself. The +# kane-cli status commands it runs use the saved sign-in, and whoami may renew +# one that is about to expire. It always exits 0, and problems show up in the text. # # Usage: sh preflight.sh [--mobile emulator|simulator] [--grid] # @@ -13,6 +14,7 @@ # whoami kane-cli whoami, then exit=<code> # balance kane-cli balance, then exit=<code> # settings kane-cli config show, then exit=<code> +# link project= and project_id= of a linked folder, "none", or "unreadable" # agent-config saved agent preferences, or the word "none" # env ci= ssh= display= os= arch= node= # chrome found=<path> and override=<KANE_CLI_CHROME_PATH> @@ -126,6 +128,33 @@ emit_cmd balance kane-cli balance echo "## settings" emit_cmd settings kane-cli config show +echo "## link" +# A folder linked to a shared project holds .testmuai/assurance.json; print only its project's name and id, read as +# kane-cli reads them, or "unreadable" when kane-cli can't read the link either. +link_file=".testmuai/assurance.json" +if [ -e "$link_file" ]; then + link_out="" + if command -v node >/dev/null 2>&1; then + link_out=$(node -e ' + try { + const b = JSON.parse(require("fs").readFileSync(process.argv[1], "utf8")); + const one = (s) => s.replace(/[\r\n]+/g, " "); + if (b && typeof b === "object" && !Array.isArray(b) && typeof b.projectId === "string" && b.projectId !== "" && typeof b.projectName === "string") { + process.stdout.write("project=" + one(b.projectName) + "\nproject_id=" + one(b.projectId) + "\n"); + } + } catch (e) {} + ' "$link_file" 2>/dev/null </dev/null) + elif [ -f "$link_file" ] && [ -r "$link_file" ]; then + # No node (kane-cli itself needs it): a best-effort read of the two fields. + link_name=$(sed -n 's/.*"projectName"[[:space:]]*:[[:space:]]*"\([^"]*\)".*/\1/p' "$link_file" | head -n 1) + link_id=$(sed -n 's/.*"projectId"[[:space:]]*:[[:space:]]*"\([^"]*\)".*/\1/p' "$link_file" | head -n 1) + if [ -n "$link_id" ]; then link_out=$(printf 'project=%s\nproject_id=%s' "$link_name" "$link_id"); fi + fi + if [ -n "$link_out" ]; then printf '%s\n' "$link_out"; else echo "unreadable"; fi +else + echo "none" +fi + echo "## agent-config" agent_config="$HOME/.testmuai/kaneai/agent-config/config.json" if [ -f "$agent_config" ] && [ -r "$agent_config" ]; then diff --git a/docs/user-guide/README.md b/docs/user-guide/README.md index 8685daf..6357064 100644 --- a/docs/user-guide/README.md +++ b/docs/user-guide/README.md @@ -25,10 +25,13 @@ Every page below is standalone — start wherever your job starts. ## 1. Generate tests - **[The assurance lifecycle](./assurance/overview.md) — start here**: the journey, the vocabulary, and when to use it instead of `generate`. +<!-- release: needs context pull --> +- **[Shared projects](./assurance/shared-projects.md)** *(0.10.0)*: work in your team's Test Manager project. Link a folder, join a teammate's project, bring the project's tests into your folder, and see the project's coverage. - [Building the context graph](./assurance/context.md) — ingest sources, extract use-cases, review what's trusted. - [Designing tests](./assurance/design.md) — acceptance criteria, scenarios, and one runnable test per scenario. - [Generate test cases](./generate-test-cases/overview.md) — scenarios and cases from a plain-language description ([workflow](./generate-test-cases/workflow.md)). -- [Sharing the context graph with your team](./assurance/sharing.md) — one shared location (GitHub, S3-compatible, or a folder); publishing, taking your teammates' records, cloning, and decisions when two people change the same thing. +- **[Exploration](./exploration.md)**: no requirements to start from? An explorer walks your app on HyperExecute, maps its screens, flows and findings, and designs tests from what it found into your Test Manager project. +- [Sharing a local store](./assurance/sharing.md): for a local store only. One shared location (GitHub, S3-compatible, or a folder); publishing, taking your teammates' records, cloning, and decisions when two people change the same thing. For a team in Test Manager, use [Shared projects](./assurance/shared-projects.md). - [Automation](./assurance/automation.md) — the headless contract: modes, NDJSON streams, exit codes, CI shapes. ## 2. Execute tests diff --git a/docs/user-guide/agents.md b/docs/user-guide/agents.md index 8065fd3..6c5538e 100644 --- a/docs/user-guide/agents.md +++ b/docs/user-guide/agents.md @@ -20,6 +20,8 @@ Then open your agent in a project and ask for something that needs a browser: check that the home page loads on my app ``` +Already have kane-cli? `kane-cli install skill` installs the newest skill that fits your kane-cli, for Claude Code, Codex CLI and Gemini CLI. Pass a version to pin one: `kane-cli install skill <version>`. When kane-cli tells you a skill update is available, this is the command to run. For a first install, use `npx @testmuai/kane-cli-skill`: it also creates the folder that holds your agent preferences. + ## What happens on your first run 1. **A ready check.** The agent runs one short script and shows a card: who is signed in, credits left, Chrome found, your app if it is already running locally, and where results will be saved. If something is missing, such as a sign-in, the card says so and offers the fix. The agent can open the sign-in page for you. It never asks you to paste an access key into the chat. @@ -57,6 +59,21 @@ Every run is saved as a test case in a Test Manager project and folder ([Test Ma This setting belongs to kane-cli, not to the agent, so a change applies to **every later kane-cli session for your sign-in**: every project folder, every agent, and the terminal. Tests you already ran stay in their original project. +## Working in your team's shared project + +If your team keeps its tests in a Test Manager project, tell your agent which one the first time you work in a folder: + +```text +use our "Checkout revamp" project in Test Manager, and design tests from prd.md +``` + +The agent finds the project's id and links the folder to it (`--project <id>`), so what it designs and saves goes to the project, where your team sees it. Name the project the first time: in a folder that isn't set up yet, kane-cli asks "shared or local?" only at a terminal, so an agent that isn't told works **locally**, in the folder alone. Once a folder is linked, every later request in it uses the same project. See [Shared projects](./assurance/shared-projects.md). + +Two things your agent will check with you in a shared project: + +- **The use-cases it found.** When an agent extracts use-cases, they stay proposed until a person checks them. The agent shows you the list and asks before it designs tests from them. +- **Before a first run.** The first run of each designed test writes its steps in a real browser, which takes a few minutes and uses more credits than later runs. + ## Watch runs live (Claude Code) In Claude Code you can turn on a live status strip. While a run works, one line in your status bar names the current step: @@ -96,6 +113,17 @@ Driving kane-cli through an agent changes nothing about what kane-cli uploads. B In CI, or with a cloud agent that cannot ask you anything, the agent skips the tour and the questions, runs headless with default settings, and does not write the preferences file. It still stops and reports clearly if kane-cli is not installed, not signed in, or out of credits. For pipelines, calling kane-cli directly is usually simpler: see [CI/CD recipes](./cicd.md). +## For agent authors: the two agent flags + +| Commands | Agent flag | +|---|---| +| `run`, `testmd`, `testrun`, `projects`, `folders`, `explore`, `config show`, `generate`, `devices`, `apps` | `--agent` | +| `context`, `design`, `maintain`, `cover` | `--mode agent` | + +Both print JSON lines on stdout. `run` and `testmd` also take `--mode agent`. Every other command refuses the spelling it doesn't take, as an unknown option. + +The assurance commands' events and codes are in [Automation](./assurance/automation.md). The codes a shared project adds, and what to do about each, are in [Troubleshooting: shared projects](./troubleshooting.md#shared-projects). + ## Next steps - [Getting started](./getting-started.md): the same first run, typed by hand. diff --git a/docs/user-guide/assurance/context.md b/docs/user-guide/assurance/context.md index a5de8a6..65a3544 100644 --- a/docs/user-guide/assurance/context.md +++ b/docs/user-guide/assurance/context.md @@ -133,15 +133,64 @@ When the agent needs an answer you're not there to give (or you Ctrl+C, or the p kane-cli context sessions # list resumable sessions + their resume commands kane-cli context extract --resume <sid> # re-presents only what's still unanswered kane-cli context extract --resume <sid> --message "Account required — the update supersedes the old section" -kane-cli context extract --resume <sid> --mode agent --answer q1=2 --answer q3="staging URL" +kane-cli context extract --resume <sid> --mode agent --answer q1=2 --answer q3="https://app.example.com" kane-cli context extract --resume <sid> --with-source ./addendum.md ``` `--message` answers **in plain words** — the agent maps your statement to its own pending questions; a statement that answers nothing pending is treated as steering. `--answer` *(0.7.1)* answers **by id** for scripted resumes (`--resume --mode agent` only). Resume is honest about what happened in between: questions answered elsewhere are never re-asked, defaults assumed in your absence are reported (`N default(s) were assumed while you were away`), and your earlier answers pre-fill rather than auto-submit. +## In a shared project + +In a folder linked to a [shared project](./shared-projects.md), `context ingest` and `context extract` add documents and use-cases to the project instead of to a local store. The first `ingest` or `extract` in a new folder is also how you [link it](./shared-projects.md#link-a-folder-to-the-project). + +```bash +kane-cli context ingest prd.md # at a terminal: upload, then the extract chat +kane-cli context ingest prd.md --mode agent # an agent: upload and extract on one stream +``` + +**Ingest** uploads each document to the project, then goes straight into the extract chat in the same session. It takes the same kinds of source as a local store: files, and web, Jira, Confluence and Linear addresses, which are fetched on your machine with your own access. It says what it added: + +```text +✔ added prd.md to this session +``` + +or, for a document already in the project, `✔ prd.md is already in "Checkout revamp", unchanged`. A **new version** of a document the project already has opens a review of what changed, one card per change, instead of a new extract. Add a new version on its own: with other documents in the same command, nothing is added. + +With no terminal and no `--mode` (CI, a pipe), ingest only uploads, and leaves the session for later: + +```text +added prd.md to session <sid>, not saved yet +extract: kane-cli context extract --resume <sid> +``` + +**Extract** proposes use-cases from the documents the project hasn't read yet. When there's nothing left to read, it opens no session: + +```text +nothing new to extract: every source is already read +next: to add more, run: + kane-cli context ingest <files> +``` + +**Finish and save.** In the chat, `looks good — finish the session` approves the use-cases this session proposed, then asks `save this session to "<project>"?`. Choose `save now`, and the use-cases reach the project. In agent mode the session ends with `session_complete` and saves nothing: run its `next`, `kane-cli context sessions save <sid> --yes --mode agent`. + +**Approving use-cases.** `context review` isn't available in a linked folder, so finishing the chat is how use-cases are approved. An extract in agent mode leaves its use-cases **proposed**, and designing one then needs `--allow-unreviewed`: see [Designing tests](./design.md#in-a-shared-project). + +**Pausing.** Press `Ctrl+C`, close the lid, or lose the connection, and the session pauses: `your work is safe — resume it from any machine`. Go on with `kane-cli context extract --resume <sid>`. An extract that read two or more documents and then paused can't be resumed or saved. It says so, and prints the commands that discard it and start again with the same documents. To keep every extract resumable, add one document at a time. + +**What's different in a linked folder:** + +- `--mode ci` isn't available on `context extract`. Use `--mode agent`. +- `context ingest --plan` and `--mode override` aren't available. +- `context extract --plan` and `--with-source` don't apply. +- `context list` shows the project's sources, use-cases, scenarios and tests. `--type` takes `source`, `usecase`, `scenario` or `test`; `--all` isn't available. +- `context review`, `context explain`, `context view` and the housekeeping commands (`retire`, `name`, `revert`, `fsck`, `rebuild`) work on a local store only. +- Each agent turn of the extract uses credits, as it does locally. + <a name="review"></a> ## `context review` — review outside the extract session +In a folder linked to a shared project, `context review` isn't available: see [In a shared project](#in-a-shared-project). + ```bash kane-cli context review [--queue derived|skipped|archived|drift] [--verdicts <file>] [--json] kane-cli context review --approve <refs...> | --skip <refs...> | --defer <refs...> # 0.7.1 diff --git a/docs/user-guide/assurance/coverage.md b/docs/user-guide/assurance/coverage.md index ba23f22..c8473c5 100644 --- a/docs/user-guide/assurance/coverage.md +++ b/docs/user-guide/assurance/coverage.md @@ -36,8 +36,49 @@ The panel always shows **both** axes: the pack's proven depth, then the live-gra - Coverage reflects **this run**: sealed packs cover only what the run touched. Project-wide coverage lives in the graph axis, unaffected by any single pack. - `--json` emits the full panel as structured data. +## In a shared project + +In a folder linked to a [shared project](./shared-projects.md), `kane-cli cover` shows the **project's** coverage: every use-case in the project, measured from everyone's runs on its test cases in Test Manager, not only yours. + +```bash +kane-cli cover +``` + +```text +using project "Checkout revamp", shared with your team + +coverage: 40% across "Checkout revamp"'s 5 use-cases + 2 proven · 1 partial · 1 not run · 1 not designed +``` + +A state with no use-cases in it is left out of that line. Then comes one row per use-case, with its coverage and what's proven, failing or not run yet, and the next steps: + +```text +next: for UC-5 "Apply a coupon", run: + kane-cli design tests --use-case uc-5 +``` + +| State | Meaning | +|---|---| +| `proven` | Every acceptance criterion's newest counted run passed | +| `partial` | Some criteria passed and none failed; the rest haven't run, or have no tests | +| `failing` | At least one criterion's newest counted run failed | +| `not run` | It has tests, but no run of them counts yet | +| `not designed` | No tests designed for it yet | +| `no criteria` | It has no acceptance criteria yet | + +- **A new run takes a few minutes to count.** When a run in this folder ended in the last 5 minutes, `cover` adds `a new run can take up to 5 minutes to count`. Run it again a little later. +- **Still working it out.** `coverage: "Checkout revamp" is still working it out; run the command again in a moment` means just that. +- **A project with no runs yet** reads 0% with `nothing run yet`, and its `next` is `kane-cli testrun run`. +- **`cover --from <pack>`** still reads one evidence pack's own figure, on this machine. +- **`cover gaps`** isn't available in a linked folder. `cover` shows each use-case's state. + +In agent mode, `kane-cli cover --mode agent` prints one `coverage` event (`source: "project"`, with `pct`, `usecases` and `criteria`, and `recent_run` when a new run may not count yet), then `done` with a `next` list. + ## `cover gaps` — the coverage ribbon *(0.8.2)* +Not available in a folder linked to a shared project: use `kane-cli cover`. + The default output is one continuous band-table over two axes, entirely graph-fed — no pack is opened or needed: - **designed** — how much of the live requirement set has a live test verifying it (risk-weighted: high-risk criteria weigh more). Freshness never moves the number — stale criteria surface as `STALE` debt instead. diff --git a/docs/user-guide/assurance/design.md b/docs/user-guide/assurance/design.md index 49184cf..00b9db3 100644 --- a/docs/user-guide/assurance/design.md +++ b/docs/user-guide/assurance/design.md @@ -112,6 +112,55 @@ Ask it "why does this test exist?" six months later and the answer is the one re The design engine ships with an embedded catalog of test-design techniques and surface profiles. Drop replacement or additional YAML under `.context/design/` (same id overrides, new ids append) to extend it per store — no upgrade needed. +## In a shared project + +In a folder linked to a [shared project](./shared-projects.md), `kane-cli design tests --use-case <id>` designs as it does locally, with three differences: it works in a session that you save, the test files appear only when you save it, and saving makes the test cases in Test Manager. + +```bash +kane-cli context list # the project's use-cases and their ids +kane-cli design tests --use-case uc-1 +``` + +**The use-case must be approved.** A use-case is approved when its extract was finished at a terminal with `looks good — finish the session`. An extract in agent mode leaves its use-cases proposed, and a linked folder has no review command, so design stops with `"<use-case>" isn't approved yet` and names the way on: + +```bash +kane-cli design tests --use-case uc-1 --allow-unreviewed +``` + +Use `--allow-unreviewed` once the person has checked the use-case. + +**Save the session.** At a terminal, the design ends with `save this session to "<project>"?`: choose `save now`. In agent mode, run the `next` from `session_complete`: + +```bash +kane-cli context sessions save <sid> --yes --mode agent +``` + +Until you save, the design lives only in its session: your team can't see it, no test case exists for it, and there are no test files. See [Sessions](./shared-projects.md#sessions-nothing-reaches-the-project-until-you-save). + +### What the save makes + +| What | Where | +|---|---| +| The criteria, scenarios and tests | in the shared project, where your team sees them | +| A test case per scenario | in Test Manager, in the project, titled with the scenario's title and tagged `assurance` and the use-case's id. The save prints `✔ 3 test cases created in Test Manager` | +| A test file per test | `.testmuai/tests/<title>_test.md` in the folder where you saved. The save prints `⚒ wrote 3 test files, one for each saved test` | +| Empty variable stubs | `.testmuai/variables/assurance.json`, for the values the tests need. They're written when the design ends, before the save | + +Each test file starts with an `assurance:` header naming the test. That header is how a run finds the test's case, so leave it as it is. + +- **A test file you edited by hand is kept.** A later save of the same test doesn't overwrite it, and says so. Running the edited file with `kane-cli testmd run <file>` sends your change to its case. +<!-- release: needs context pull --> +- **Only the saving folder gets files.** A teammate gets them with [`kane-cli context pull`](./shared-projects.md#bringing-the-projects-tests-into-your-folder). +- **Run the tests** with `kane-cli testrun run`. Their first run writes their steps, so it takes longer and uses more credits. See [Designed tests in a shared project](../testrun.md#designed-tests-in-a-shared-project). + +In agent mode the save's last `session_save` event (`phase: "saved"`) has `test_manager` (with `created` and `updated` counts) and `test_files.written`, the paths of the files it wrote. + +### Also different in a linked folder + +- `--mode ci` isn't available. Use `--mode agent`. +- One design of a use-case at a time: while someone is designing it, or an extract is running in the project, a second design can't start until it finishes. kane-cli says whether it's yours or someone else's. +- `kane-cli design explain` isn't available. `kane-cli context list` shows what was designed. + ## Exit codes | Code | Meaning | @@ -126,3 +175,4 @@ The design engine ships with an embedded catalog of test-design techniques and s - [Coverage](./coverage.md) — measure what the designed tests prove. - [Maintaining the suite](./maintain.md) — reconcile the suite when requirements change. - [Automation](./automation.md) — headless design in CI or from an agent. +- [Shared projects](./shared-projects.md) — design with your team, in Test Manager. diff --git a/docs/user-guide/assurance/maintain.md b/docs/user-guide/assurance/maintain.md index 840db2c..5cc8f28 100644 --- a/docs/user-guide/assurance/maintain.md +++ b/docs/user-guide/assurance/maintain.md @@ -161,6 +161,30 @@ kane-cli maintain evolve t-4 --mode agent --answer evolve:uc-3=1 # 1 = proce The resume command keeps `--because`. With `--from-stale` there is one question per use-case and nothing runs until every one is answered; a partial set pauses again on the rest and keeps the answers given. A design session that pauses arrives as its own `session_paused`, and `done` carries its resume command: resume it, then run evolve again for the diff. After a partial answer the suggested commands carry the answers already given, so declining the rest still re-designs what you answered proceed. `--mode` accepts `interactive` and `agent` only (`MODE_USAGE` otherwise); an answer to a question the run does not ask refuses `ANSWER_USAGE`. The event table is in [Automation](./automation.md#evolves-stream). +## In a shared project + +In a folder linked to a [shared project](./shared-projects.md), both commands work in a session, like a design, and nothing reaches the project until you save it with `kane-cli context sessions save <sid>`. The save writes the updated test files into this folder's `.testmuai/tests/` and updates the scenarios' test cases in Test Manager. + +**`maintain reconcile`** takes the changed document as a file, and the id of the document it replaces: + +```bash +kane-cli maintain reconcile --from prd-v2.md --source-id prd +``` + +- `--plan`, `--apply` and `--mode ci` aren't available. Use the chat at a terminal, or `--mode agent`. +- `--from` takes a file, not a web address. To review a web page's new version, add it with `kane-cli context ingest <address>`: a new version of a document the project already has opens the same review. +- Run on the same file again, and with nothing to review it says the file is `already in "<project>", unchanged`. + +**`maintain evolve`** updates the tests of a use-case whose document changed after its tests were designed: + +```bash +kane-cli maintain evolve uc-3 +kane-cli maintain evolve --from-stale --mode agent +``` + +- A use-case that hasn't changed stops with `hasn't changed since its tests were designed`, and a use-case with no tests yet with `has no tests yet`. Each names the `kane-cli design tests --use-case <id>` to run instead. +- While someone else is designing a use-case, `--from-stale` skips it and goes on with the rest. + ## Exit codes | Code | Meaning | diff --git a/docs/user-guide/assurance/overview.md b/docs/user-guide/assurance/overview.md index 0bd7a71..f2de8bd 100644 --- a/docs/user-guide/assurance/overview.md +++ b/docs/user-guide/assurance/overview.md @@ -2,7 +2,21 @@ kane-cli began as a way to author and replay browser tests. The **assurance** commands take on the step before and after: describe what your product must do, and kane-cli designs the tests that prove it — each one permanently linked to the requirement it verifies. Run them, and coverage stops being a guess: every run reports exactly what it proved and what it still owes. And as your product changes, the suite is reconciled instead of quietly rotting. -> Requires kane-cli **0.6.1 or later** (`kane-cli --version`). On 0.6.0 these commands fail after a fresh install — upgrade. This guide describes **0.8.14**; version-specific behavior is marked with the release that introduced it — *(0.7.1)* through *(0.8.14)* — on its page. +> Requires kane-cli **0.6.1 or later** (`kane-cli --version`). This guide describes **0.10.0**; version-specific behaviour is marked with the release that introduced it. Shared projects need **0.10.0**. + +## Where the work goes: a shared project or a local store + +Each folder you work in keeps its work in one of two places. You choose once, the first time you add documents to it. + +| | A [shared project](./shared-projects.md) *(0.10.0)* | A local store | +|---|---|---| +| Where the work lives | in a Test Manager project, where your team sees it | in `.context/` in this folder, on this machine | +| Test cases | one per scenario in Test Manager, made when you save | none | +| Coverage | the project's figure, from everyone's runs | from the runs recorded in your local store | +| Sharing | teammates join the project from their own folders | [`context sync`](./sharing.md) through a location | +| Reviewing use-cases | when you finish the extract chat | `context review` | + +If your team works in Test Manager, use a shared project, and start at [Shared projects](./shared-projects.md). The loop below is the same in both, except where the shared-projects page says otherwise. The biggest difference: in a shared project, each step's work reaches the project only when you save its session. ## The loop @@ -62,6 +76,8 @@ If you have a PRD and care about coverage accounting, start with assurance. If y ## The store: `.context/` +In a folder linked to a [shared project](./shared-projects.md), there's no `.context/`: the work lives in the project. Everything below is about a local store. + The assurance commands work over a local store in your project directory, created on first `ingest`: - It is **append-only** in everyday use: no command that writes to it deletes or rewrites a record. Edits create new versions; mistakes are reverted with compensation records. `kane-cli context explain` can replay the full history of any node. The one exception is a sync rebase *(0.8.14)*, which replaces your records after the last shared one with the location's and reapplies your own on top — every replaced record is kept in a backup. diff --git a/docs/user-guide/assurance/shared-projects.md b/docs/user-guide/assurance/shared-projects.md new file mode 100644 index 0000000..2b97c07 --- /dev/null +++ b/docs/user-guide/assurance/shared-projects.md @@ -0,0 +1,286 @@ +# Shared projects: working with your team in Test Manager + +A **shared project** is a Test Manager project your team works in together. Link a folder to one, and what you save there goes to the project, where your teammates see it: your documents, use-cases, acceptance criteria, scenarios and tests. Each scenario also gets its own test case in Test Manager. Runs count toward one coverage figure for the whole project, whoever ran them. + +> Requires kane-cli **0.10.0 or later** (`kane-cli --version`). + +## Two places your work can go + +Each folder works one way or the other, and you choose once, the first time you add documents to it. + +| | A shared project | A local store | +|---|---|---| +| Where the work lives | in the project, where everyone with access to it sees it | in `.context/` in this folder, on this machine | +| Test cases | one per scenario, in the project's Test Manager | none: your runs go to your profile's Test Manager project | +| Coverage | the project's figure, from everyone's runs | from the runs recorded in this folder's store | +| Sharing with a teammate | they join the project from their own folder | [`context sync`](./sharing.md), through a repository, a bucket or a folder | + +If your team keeps its tests in Test Manager, use a shared project. The rest of this page is about shared projects. For a local store, start at [Building the context graph](./context.md). + +## Before you start + +Sign in, then find the project you'll work in, or make one: + +```bash +kane-cli login +kane-cli projects list --search "Checkout" +kane-cli projects create "Checkout revamp" +``` + +`projects list` prints each project's id beside its name: + +```text +ID NAME +-------------------------- ---- +01HZX3K9QW8T5M2C7V4N6B1R0D Checkout revamp +``` + +`projects create` ends by suggesting `kane-cli config project <id>`. That sets your **profile's** project, which `kane-cli run` uploads to. You don't need it for a shared project: linking a folder is a separate step, below. + +## Link a folder to the project + +Only two commands can link a folder: `kane-cli context ingest` and `kane-cli context extract`. + +**At a terminal**, run either one in a folder that isn't set up yet: + +```bash +kane-cli context ingest prd.md +``` + +kane-cli asks where the folder's work goes: + +```text +where should this folder's work go? +you choose once; later runs in this folder use it + work in a shared project — your team sees what you save + work locally — it stays in this folder, on this machine +``` + +Pick the shared project, then the project itself from your list (type part of a name to search, or press `Tab` to make a new one). Once the session opens: + +```text +✓ Project: "Checkout revamp" + this folder is now linked to it; what you save goes to the project +``` + +From then on, each chat session in this folder, and `kane-cli cover`, starts with `using project "Checkout revamp", shared with your team`. + +**From a script or an agent**, name the project with `--project`: + +```bash +kane-cli context ingest prd.md --project 01HZX3K9QW8T5M2C7V4N6B1R0D --mode agent +``` + +Without `--project`, a folder that isn't set up yet works **locally**, without asking, whenever there's no terminal to ask in (`--mode agent`, CI, a pipe). So an agent should always pass `--project` the first time. + +### The link + +The link is one small file, `.testmuai/assurance.json`, in the folder. It names the project. + +- It covers this folder only. A subfolder isn't linked: kane-cli treats it as a folder that isn't set up yet. +- To unlink, delete the file. There's no unlink command. +- If the folder is a git repository, you can commit the file. A teammate who clones the repository then works in the same project without being asked. +- Don't confuse it with `.testmuai/variables/assurance.json`, which holds your variables. + +A folder can't be both: a folder with its own local store (`.context/`) and a link refuses every assurance command until you remove one of them. + +## Sessions: nothing reaches the project until you save + +In a shared project, `context extract`, `design tests` and `maintain` each work in a **session**. A session holds its work apart from the project until you save it. Your teammates see it in the project only after the save. + +At a terminal, the session ends by asking `save this session to "Checkout revamp"?`, with three choices: + +| Choice | What happens | +|---|---| +| `save now` | The work goes to the project, and your team sees it | +| `keep for later` | Nothing is saved yet. You can save it later, from any machine | +| `adjust — tell me what to change` | Back to the chat | + +Keep it for later, and kane-cli prints the commands to finish it: + +```text +save: kane-cli context sessions save <sid> +discard: kane-cli context sessions discard <sid> +``` + +In agent mode a session ends with a `session_complete` event that saves nothing. Its `next` names the save: + +```bash +kane-cli context sessions save <sid> --yes --mode agent +``` + +| Command | What it does | +|---|---| +| `kane-cli context sessions` | Lists the project's open sessions, yours first, with the command that saves or resumes each of yours | +| `kane-cli context sessions save <sid>` | Saves your session to the project | +| `kane-cli context sessions discard <sid>` | Throws your session away; nothing reaches the project | +| `kane-cli context extract --resume <sid>` | Goes on with a paused extract | +| `kane-cli design tests --resume <sid>` | Goes on with a paused design | + +You can save, resume or discard only your own sessions. Anyone else's reads `no session <sid> of yours`. + +**One session at a time per use-case.** While a session is designing a use-case's tests, a second design of it can't start, and while an extract runs, no design can start. kane-cli says whether the session in the way is yours or someone else's (`someone else is designing tests for "Add to cart" right now, so this design can't start`), and you run the command again once it finishes. Finish and save a session before you start the next one, and your teammates won't wait on you. + +## From documents to coverage + +```bash +kane-cli context ingest prd.md --project <id> # add the document, find its use-cases, save +kane-cli design tests --use-case uc-1 # design the tests for one use-case, save +kane-cli testrun run # run the tests +kane-cli cover # the project's coverage +``` + +1. **Add your documents.** `context ingest` uploads each document and goes straight into the extract chat, where the agent proposes use-cases. Choose `looks good — finish the session`, then `save now`. Finishing that way **approves** the session's proposed use-cases. See [Building the context graph](./context.md#in-a-shared-project). +2. **Design.** `design tests --use-case uc-1` designs one use-case's acceptance criteria, scenarios and tests. Save it, and kane-cli writes the test files into `.testmuai/tests/` and makes a test case in Test Manager for each scenario. See [Designing tests](./design.md#in-a-shared-project). +3. **Run.** `testrun run` runs each designed test on its scenario's test case. The first run of a test writes its steps in a real browser; later runs replay them. See [Batch runs](../testrun.md#designed-tests-in-a-shared-project). +4. **Measure.** `cover` shows the project's coverage, from everyone's runs. See [Coverage](./coverage.md#in-a-shared-project). + +To see what's in the project at any point, run `kane-cli context list`. + +## Where everything lives + +<!-- release: needs context pull --> +| What | Where | +|---|---| +| The folder's link to its project | `.testmuai/assurance.json` in the folder | +| Documents, use-cases, criteria, scenarios and tests | in the shared project. `kane-cli context list` shows them | +| Test files (`*_test.md`) | `.testmuai/tests/` in the folder where you saved the session, or where you ran `kane-cli context pull` | +| A test's recorded steps | `.testmuai/tests/output-<name>/` beside the test, made by its first run in this folder | +| Evidence packs | `.testmuai/evidence/` | +| Test cases | in Test Manager, in the project: one per scenario, made or updated when a session is saved | +| Runs | in Test Manager, on each scenario's test case | +| Coverage | in the shared project. `kane-cli cover` shows it | +| Variable values | `.testmuai/variables/` in your folder. See [Variables in a shared project](../variables-and-context.md#variables-in-a-shared-project) | +| Your sign-in, and your profile's project | `~/.testmuai/` | + +<!-- release: needs context pull --> +## Joining a teammate's project + +Ask your teammate for the project's name, find its id, then join from a new folder of your own: + +```bash +kane-cli projects list --search "Checkout" +mkdir checkout && cd checkout +kane-cli context pull --project 01HZX3K9QW8T5M2C7V4N6B1R0D +``` + +`context pull --project` links the folder, reads none of the project's documents, and brings the project's tests here: + +```text +✓ Project: "Checkout revamp" + this folder is now linked to it; what you save goes to the project +✓ "Checkout revamp": 3 tests here — 3 new + new .testmuai/tests/add_one_item_to_the_cart_test.md + new .testmuai/tests/remove_the_last_item_test.md + new .testmuai/tests/cart_total_with_a_discount_test.md +3 of these tests haven't run in this folder yet: the first run of each writes its steps in a real browser, which takes a few minutes and uses more credits; later runs replay them +next: to run them, run: + kane-cli testrun run +``` + +From then on the folder works like your teammate's: `kane-cli context list` shows the project's work, `kane-cli context sessions` its open sessions, and `kane-cli cover` its coverage. To add documents of your own, run `kane-cli context ingest <files>`. + +`kane-cli context extract --project <id>` also joins. But if the project has a document nobody has extracted yet, it starts extracting it, which uses credits and holds that document until the session ends. To join without reading anything, use `context pull --project`. + +<!-- release: needs context pull --> +## Bringing the project's tests into your folder + +In a linked folder, `kane-cli context pull` writes the project's tests as `*_test.md` files in `.testmuai/tests/`. They're the same files a save writes, including the tests your teammates designed and the ones an exploration designed. + +```bash +kane-cli context pull +``` + +```text +✓ "Checkout revamp": 5 tests here — 2 new, 1 updated, 2 already up to date + new .testmuai/tests/apply_an_expired_coupon_test.md + new .testmuai/tests/checkout_as_a_guest_test.md + updated .testmuai/tests/add_one_item_to_the_cart_test.md +2 of these tests haven't run in this folder yet: the first run of each writes its steps in a real browser, which takes a few minutes and uses more credits; later runs replay them +next: to run them, run: + kane-cli testrun run +``` + +| Option | What it does | +|---|---| +| `--use-case <id>` | Only this use-case's tests, with the tests they use | +| `--overwrite` | Replace the test files you changed here with the project's | +| `--project <id>` | In a folder that isn't set up yet: join this shared project, reading no documents, then bring its tests | + +- **A file you changed is kept.** The pull lists it as `kept <file> — you changed it here` and names `kane-cli context pull --overwrite` to replace it. `--overwrite` lists what it replaced. +- **A file kane-cli wrote earlier and you haven't touched** is brought up to date (`updated`). +- **Tests archived in the project** aren't pulled. +- **Variables.** If the tests need variables your folder has no value for, the pull names them and the file to add them to. See [Variables in a shared project](../variables-and-context.md#variables-in-a-shared-project). +- **Nothing in the project changes.** A pull only reads. + +**The first run of a pulled test costs more.** A test's recorded steps stay with the folder that ran it, in `output-<name>/`, so they don't come with the pull. Each pulled test's first run in your folder writes its steps in a real browser, which uses more credits, and later runs replay them for far fewer. + +In agent mode, `kane-cli context pull --mode agent` prints one `pull_test` event per test, with its `file` and `outcome` (`new`, `updated`, `unchanged`, `kept`, `replaced` or `not_written`), then `done` with a `next` that runs them. + +### What an exploration built + +An exploration of your app (`kane-cli explore run --project <id> --url <app>`) designs tests and saves them into the project as it goes. Their test cases are in Test Manager, and `kane-cli context list` shows their use-cases, scenarios and tests, but their files stay where the exploration ran. To get them, run `kane-cli context pull` in a folder linked to that project, or `kane-cli context pull --project <id>` in a new folder, which links it and gets them in one step. They arrive as `new`, and none has run yet, so each one's first run writes its steps. + +Pass the project with `--project` on every `kane-cli explore run`: explore doesn't read a folder's link. If nobody has shared the project yet, the exploration's first design job shares it, as you; if you can't create test cases in that project, it designs no tests until someone who can shares it. [Exploration](../exploration.md) covers starting, shaping and watching an exploration. + +## What a linked folder doesn't do + +Some commands work only on a local store. In a linked folder they stop with `isn't available yet in a folder linked to a shared project`, and nothing changes: + +<!-- release: needs context pull --> +| Refused in a linked folder | Use instead | +|---|---| +| `kane-cli context review`, `kane-cli context explain`, `kane-cli context view` | `kane-cli context list` | +| `kane-cli cover gaps` | `kane-cli cover` | +| `kane-cli context sessions show`, `kane-cli context sessions clean` | `kane-cli context sessions` | +| `kane-cli context sync`, `push`, `clone`, and `pull` with a location name | a shared project is already shared: your teammates join it | +| `kane-cli context fsck`, `rebuild`, `retire`, `revert`, `name`, `kane-cli design explain` | none: these work on a local store only | +| `--mode ci` on `context extract` and `design tests` | `--mode agent` | +| `kane-cli context ingest --plan`, `--mode override` | `kane-cli context ingest <files>` | +| `kane-cli maintain reconcile --plan`, `--apply`, `--mode ci` | `kane-cli maintain reconcile --from <file> --source-id <id>` | +| `kane-cli testrun run --from-context <ids>` | `kane-cli testrun run <files>`: the refusal names the files | + +Everything that isn't an assurance command runs as usual in a linked folder: `kane-cli run`, `testmd run`, `testrun run`, `login`, `config`, `projects`, `explore` and the rest. + +## `kane-cli run` in a linked folder + +`kane-cli run "<objective>"` uploads to your **profile's** project (the one `kane-cli config project` sets), not to the folder's shared project, and it saves its test in `.testmuai/tests/` like any other. The next `kane-cli testrun run` over the whole folder then finds tests from two projects, and stops: + +```text +error: these tests are in different Test Manager projects, and a run counts for one project only + this folder's project, "Checkout revamp": + .testmuai/tests/add_one_item_to_the_cart_test.md + "KaneAI Generated": + .testmuai/tests/search_for_headphones_test.md +next: to run the tests of "Checkout revamp", this folder's project, run: + kane-cli testrun run .testmuai/tests/add_one_item_to_the_cart_test.md +``` + +Run the command it names, or keep `kane-cli run` experiments in a different folder. + +## For agents and scripts + +The same journey with no terminal. Each step's last event carries a `next` with the command to run after it. + +```bash +kane-cli projects list --search "Checkout" --agent +kane-cli context ingest prd.md --project <id> --mode agent +kane-cli context sessions save <sid> --yes --mode agent +kane-cli context list --mode agent +kane-cli design tests --use-case uc-1 --allow-unreviewed --mode agent +kane-cli context sessions save <sid> --yes --mode agent +kane-cli testrun run .testmuai/tests/<file>_test.md --agent +kane-cli cover --mode agent +``` + +- `projects`, `run`, `testmd` and `testrun` take `--agent`. `context`, `design`, `maintain` and `cover` take `--mode agent`. +- An agent-mode extract leaves its use-cases **proposed**, and a linked folder has no review command. So `design tests` refuses them with `isn't approved yet`, and its `next` adds `--allow-unreviewed`. Show the person the proposed use-cases (`kane-cli context list --mode agent`) and get their go-ahead before you design with it. +- Every code these commands can stop with, and what to do about it, is in [Troubleshooting: shared projects](../troubleshooting.md#shared-projects). + +## Next steps + +- [Building the context graph](./context.md#in-a-shared-project): ingest and extract in a linked folder. +- [Designing tests](./design.md#in-a-shared-project): design and save. +- [Batch runs](../testrun.md#designed-tests-in-a-shared-project): first runs, replays and test cases. +- [Coverage](./coverage.md#in-a-shared-project): the project's figure. +- [Troubleshooting: shared projects](../troubleshooting.md#shared-projects): every refusal and what to do. diff --git a/docs/user-guide/assurance/sharing.md b/docs/user-guide/assurance/sharing.md index b9a4a36..a72c7c3 100644 --- a/docs/user-guide/assurance/sharing.md +++ b/docs/user-guide/assurance/sharing.md @@ -11,6 +11,9 @@ kane-cli context sync origin # every day after that: take the team's new reco Two rules stand behind every command here: **never merge `.context/` with git**, and never edit it by hand. Sharing goes through a location, and only through the commands on this page. +<!-- release: needs context pull --> +> **Is your team in Test Manager?** Use a [shared project](./shared-projects.md) instead *(0.10.0)*: everyone works in one Test Manager project, and nothing needs a location. This page is about sharing a **local store** (`.context/`) through a location. A folder linked to a shared project refuses `context sync`, `context push` and `context clone`. There, `context pull` brings the project's tests, and takes no location name: see [Bringing the project's tests into your folder](./shared-projects.md#bringing-the-projects-tests-into-your-folder). + <a name="locations"></a> ## Three kinds of location diff --git a/docs/user-guide/configuration.md b/docs/user-guide/configuration.md index 019deda..e16f87a 100644 --- a/docs/user-guide/configuration.md +++ b/docs/user-guide/configuration.md @@ -111,6 +111,8 @@ In a non-interactive shell (CI, pipes), pass an explicit `<project-id>`. To disc If you don't configure a project at all, kane-cli auto-resolves a sensible default when the first run starts — see "Auto-default on first run" in [test-manager-integration.md](./test-manager-integration.md). +The project set here is your **profile's** project: `kane-cli run` uploads to it, in every folder. A folder linked to a [shared project](./assurance/shared-projects.md) keeps its own project in `.testmuai/assurance.json`, and `kane-cli config project` never changes it. See [Shared projects: a folder's own project](./test-manager-integration.md#shared-projects-a-folders-own-project). + ### TMS folder ```bash diff --git a/docs/user-guide/exploration.md b/docs/user-guide/exploration.md new file mode 100644 index 0000000..a0b2c09 --- /dev/null +++ b/docs/user-guide/exploration.md @@ -0,0 +1,339 @@ +# Exploration: map your app and design tests from it + +`kane-cli explore` sends an explorer through your app. It clicks, types and follows links the way a curious tester +would, and builds a **map** of what it finds: the app's screens, the flows between them, and **findings**: things that +look broken, and places it couldn't get past, each with its evidence. While it explores, it also **designs tests** +from what it mapped, into your Test Manager project: the same kind of tests [assurance](./assurance/overview.md) +designs from a document. + +``` +your app ──► explorer (on HyperExecute) ──► map: screens · flows · findings + └─► design jobs ──► use-cases · scenarios · tests in your Test Manager project + │ + kane-cli context pull ◄──────┘ then run them like any designed test +``` + +Use exploration when you have an app but no written requirements, or to find what your documents missed. Use +[assurance](./assurance/overview.md) when you have a PRD or a spec. Both put their tests in the same project, so you +can do both. + +Each project has **one exploration of one URL**. Every run adds to that exploration's map. + +## Before you start + +| You need | Why | How to check | +|---|---|---| +| kane-cli, signed in ([Installation](./installation.md), [Authentication](./authentication.md)) | the explorer runs as you | `kane-cli whoami` | +| A Test Manager project for the exploration | the map and the tests belong to one project | `kane-cli projects list --search <name>`, or `kane-cli projects create "<name>"` | +| Permission to create test cases in that project | the design jobs save tests and cases there | if you can't, the map still builds, but no tests are designed ([below](#no-tests-were-designed)) | +| Credits | an exploration and each design job spend credits | `kane-cli balance` | + +Exploration runs on **HyperExecute** by default: kane-cli queues the run there, and the run goes on without your +machine (except for a tunnel, [below](#an-app-on-localhost-or-a-private-network)). + +## Your first exploration + +```bash +kane-cli explore run --project <project-id> \ + --url https://staging.example.com \ + --prompt "Explore sign-up, the dashboard and the settings pages" \ + --policy read_only \ + --max-leases 40 +``` + +- `--project` is the project the exploration belongs to. **Always pass it.** `explore` doesn't read a folder's link to + a shared project; without `--project` it uses the project kane-cli is configured with, which may not be your + team's. +- `--url` is where it starts. The exploration keeps this URL. +- `--prompt` says what to explore and why, in your words. The explorer works within it; the hard limits are + `--allow-domain` and `--deny` ([below](#keeping-it-in-bounds)). +- `--policy read_only` blocks write requests and never presses controls such as Delete, Sign out and Pay. The default + can change data: see [Can it change my data?](#can-it-change-my-data) before you leave it out. +- `--max-leases 40` is a good first limit: it stops after 40 pieces of work (kane-cli calls them **leases**). + +**On a terminal**, kane-cli shows a review of your settings (or a setup form, if something is missing). Launch it, and +you see `● queued on HyperExecute · run … · <job link>`, then a live view: the run's state and policy, the time so far, +and `leases · screens · flows · findings · credits` as they grow. **Ctrl+C detaches**: the run keeps going on +HyperExecute. The lines printed when you detach give the full exploration id and the commands to watch, map or +stop it; when the run ends, they give the map command. + +**Your exploration id.** Most commands below take it. It's in those lines; or run plain `kane-cli explore`, pick the +project, and use the exploration's **card** (its map, its last run, and run again, watch, stop, map, sessions, +assurance context, delete). `kane-cli explore view --project <project-id>` opens the map with no id at all. + +> **Every run takes its options again.** A run keeps the exploration's URL and map, and nothing else you passed: +> policy, prompt, sign-in steps, domains, variables, limits and tunnel all start from their defaults. The policy's +> default is `mutating`, so pass `--policy read_only` again on every run that mustn't change data. + +## Can it change my data? + +Yes, by default. `--policy` decides: + +| | `read_only` | `mutating` (the default) | +|---|---|---| +| Write requests (POST, PUT, PATCH, DELETE) | blocked, to any host | allowed, everywhere it reaches | +| Delete, sign out, pay and similar controls | never pressed | pressed like any other control | +| `--deny "<control name>"` | added to that list | the only controls it won't press | + +- Every `mutating` run says so before it starts. On a terminal, your first `mutating` run in a project also asks you + once. +- `read_only` has three exceptions: the setup steps you give in `--deps` (a sign-in, for example) run as written, so + put in `--deps` only what you're happy for it to do; the POST endpoints you let through with `--allow-post`; and + the one click that ends an app session it opened. It can't stop an app that changes data on an ordinary page load. +- Some apps read their data with POST requests (GraphQL, for example), so under `read_only` they show no data. Let + through only endpoints you know only read: `--allow-post api.example.com/graphql` (repeatable). Its host must be the + `--url`'s or one you add with `--allow-domain`, so here also pass `--allow-domain api.example.com`. + +Use `mutating` on a test or staging copy of your app, never on data you care about. + +## An app behind a login + +Tell the explorer how to get in with `--deps`, and keep the values in a variables file: + +```bash +kane-cli explore run --project <project-id> --url https://app.example.com \ + --prompt "Explore the account area" --policy read_only \ + --deps "Logged in as {{username}} / {{password}}" \ + --allow-domain login.example.com \ + --variables-file ./explore-vars.json +``` + +```json +{ + "username": { "value": "qa@example.com" }, + "password": { "value": "…", "secret": true } +} +``` + +- `--deps` always needs `--allow-domain`: name every host it signs in or writes on, the `--url`'s own host too if + that's where it signs in (`--allow-domain app.example.com`). Use a project of its own: one project explores one URL. +- You can also keep the values in `.testmuai/variables/*.json` (see [Variables & context](./variables-and-context.md)). + Only the names your `--prompt` and `--deps` use are sent. Names are letters, digits and `_` (`login_email`, not + `login-email`); `LT_USERNAME` and `LT_ACCESS_KEY` are taken. +- **Secrets** (`"secret": true`) go to your **HyperExecute Secrets**. Other values go to your organisation's Test + Manager variables, and **replace any variable of the same name there**, so pick names your team can share. A value + never travels in the request itself. kane-cli also keeps your LambdaTest username and access key in HyperExecute + Secrets, for the run to sign in to LambdaTest with. +- A `{{name}}` no variables file defines stops the run before it starts. An empty secret stops it as + `failed:secrets_missing`, so fill in your values first. +- Keep `--max-parallel` at 1 (the default) if the app allows one session per account: each extra browser signs in on + its own. +- If the sign-in doesn't work, the run stops as `failed:setup_failed`: check the values, and every sign-in host in + `--allow-domain`. + +## An app on localhost or a private network + +The cloud's browsers can't reach `localhost` or your private network on their own. They reach it through a LambdaTest +tunnel (LT) that runs on **your machine**: + +``` +HyperExecute browsers ── LT tunnel ── your machine ── your app +``` + +1. If `LT --version` doesn't work, download LT and put it on your PATH: + + | Machine | Download | + |---|---| + | Apple silicon Mac | https://downloads.lambdatest.com/tunnel/v3/mac/arm64/LT_Mac.zip | + | Intel Mac | https://downloads.lambdatest.com/tunnel/v3/mac/64bit/LT_Mac.zip | + | Linux | https://downloads.lambdatest.com/tunnel/v3/linux/64bit/LT_Linux.zip (`linux/arm64/LT_Linux.zip` on ARM) | + | Windows | https://downloads.lambdatest.com/tunnel/v3/windows/64bit/LT_Windows.zip, then run `LT.exe` | + + For example, on an Apple silicon Mac: + + ```bash + curl -fLo /tmp/LT_Mac.zip https://downloads.lambdatest.com/tunnel/v3/mac/arm64/LT_Mac.zip + unzip -o /tmp/LT_Mac.zip -d /tmp/lt + sudo mkdir -p /usr/local/bin && sudo mv /tmp/lt/LT /usr/local/bin/LT && sudo chmod +x /usr/local/bin/LT + LT --version + ``` + + If macOS blocks it from opening: `xattr -d com.apple.quarantine /usr/local/bin/LT`. +2. Start the tunnel as the same LambdaTest user kane-cli is signed in as, and leave it running: + + ```bash + LT --user "$LT_USERNAME" --key "$LT_ACCESS_KEY" --env ht-prod --tunnelName dev + ``` + + (`$LT_USERNAME` and `$LT_ACCESS_KEY` are your LambdaTest username and access key: set them in your shell first. + [Where to find your access key](./authentication.md#where-to-find-your-access-key).) `--env ht-prod` is required. + Wait until LT says the tunnel is connected. +3. In another terminal, explore through it: + + ```bash + kane-cli explore run --project <project-id> --url http://localhost:3000 \ + --prompt "Explore checkout" --policy read_only --tunnel-name dev + ``` + + `--tunnel-name` must match `--tunnelName` from step 2, and it turns the tunnel on. With only one tunnel running, + `--tunnel` alone is enough. Use a project of its own: one project explores one URL. + +- Keep the tunnel **and your machine** up until the run ends; kane-cli reminds you when the run starts. If the tunnel + drops, pages stop loading. +- Pass the tunnel flag again on every run, with your other options (see the box under + [Your first exploration](#your-first-exploration)). +- A tunnel needs cloud browsers: with `--local-orchestration` and your own Chrome, `--tunnel` is refused (your Chrome + already reaches your machine). + +## Keeping it in bounds + +| You want | Pass | +|---|---| +| More sites than the `--url`'s host | `--allow-domain <host>` (repeatable). The `--url`'s host, and its subdomains, are always in | +| A control it must never press | `--deny "<control name>"` (repeatable), under either policy, for example `--deny "Archive"` | +| A limit on how much it explores | `--max-leases <n>`: it stops after n leases. Every run also stops at built-in limits (an hour, 400 actions or 800 model calls). Either way it ends `budget`. Neither limits the design jobs ([below](#the-tests-it-designs)) | +| More browsers at once | `--max-parallel <n>` (default 1): up to 20 cloud browsers | +| To start from one screen of the map it already has | `--focus "<screen>"` (a later run): the screen's name, or words from its name or URL | +| To explore from your machine | `--local-orchestration`: your Chrome (up to 4 at once; `--headless` for no window), or `--browser remote` for cloud browsers (up to 20). The command waits until the run ends, Ctrl+C stops it, and `--focus` also needs `--exploration-id <id>` | + +How many controls it tries per screen (`--depth-caps`), what counts as slow (`--slow-budget`) and whether new findings +are checked again (`--no-recheck`) are in `kane-cli explore run --help`, with their values and defaults. + +## Watching, stopping, running again + +**The map.** Open it in your browser: + +```bash +kane-cli explore view --project <project-id> # or --exploration-id <id> +``` + +It serves the map on your machine until you press Ctrl+C, and follows a live run from anywhere you're signed in. + +1. **Find your way:** zoom from **areas** of the app, to **screens** and the flows between them, to each screen's + **details** and controls. Press `/` to search by name or URL. +2. **Follow a flow:** open it to see its steps, screen by screen. +3. **Check a finding:** open it for its evidence: the screenshots, the steps that led there, and the network and + console logs. **Promote** it to a bug, or mark it **resolved**. +4. **Teach the next run:** leave **labels** and **notes**, mark a flow or a finding correct or incorrect, and mark a + screen or a control **do not touch**. The explorer reads them when its next run starts, so they're read-only + while a run is going. + +The map also shows what each flow cost to find, and what each run spent. + +**Runs.** + +```bash +kane-cli explore list # explorations with a run going; --all for every one +kane-cli explore sessions <id> # its runs, newest first: state, why it stopped, findings, credits +kane-cli explore sessions <id> --run <run-id> # one run in full +``` + +| A stopped run's reason | Means | +|---|---| +| `completed` | it ran out of things to explore | +| `budget` | it reached `--max-leases` or a built-in limit | +| `user_stop` | someone stopped it | +| `stopped_before_start` | it was stopped while still queued | +| `heartbeat_lost` | it stopped reporting, so it was stopped | +| `failed:setup_failed` | the sign-in in `--deps` didn't work | +| `failed:secrets_missing` | a secret it needs wasn't in HyperExecute Secrets, or was empty | +| `failed:vm_never_started` | HyperExecute never started it: run it again | +| `failed:<other>` | it failed; `kane-cli explore sessions <id> --run <run-id>` shows more | + +**Stop a run:** `kane-cli explore stop <id>`. A queued run stops at once. A running one finishes the work in hand, +keeps it, then stops; the command waits for that unless you pass `--no-wait`. + +**Run it again:** `kane-cli explore run --project <project-id>` with your options again (policy, prompt, sign-in, +domains, variables, limits, tunnel). It goes on the same exploration and its URL, and adds to the map. Only one run at +a time: while one is queued or running, another is refused (`RUN_ACTIVE`). + +**Explore another site:** use another project. A project's one exploration explores one URL, so a different `--url` +is refused. If you'd rather reuse the project, `kane-cli explore delete <id>` (it asks first; `--yes` when piped; +refused while a run is queued or running) frees it for a new exploration of any URL. Its map is gone from every +command, but the use-cases and tests designed for the first site stay in the project, and a pull brings them along +with the new site's. + +## The tests it designs + +While an exploration's **assurance context** is on (the default), **design jobs** (kane-cli calls them assurance +jobs) turn what it mapped into use-cases, scenarios and tests in its project, with a Test Manager case for each +scenario. A job runs when a run ends, and every few minutes during a long run when enough has changed. Exploration +designs and saves tests; it never runs them. + +```bash +kane-cli explore context status <id> # on or off, a job running or waiting, and what the last job did +kane-cli explore context sync <id> # ask for a job now +kane-cli explore context off <id> # no more design jobs (nothing saved is deleted) +kane-cli explore context on <id> # turn it back on; it catches up from the newest ended run +``` + +- `explore context status` shows how many use-cases the last job added and changed, how many tests it designed and + saved (and any that failed), what it left for you to review, and the credits it used. (With `--agent` it also lists + each saved test's Test Manager case, or why it has none yet.) +- `kane-cli explore run … --no-context` starts a **new** exploration with design jobs off, for a map only. For an + existing exploration, use `explore context off`. Turning it off doesn't stop a job that's already running. +- The use-cases it designs were not reviewed by a person, so they show as proposed. Their source is + `explore-<exploration id>`: in a folder linked to the project, `kane-cli context list` shows it. + +### No tests were designed + +If nobody had shared the project yet, the exploration's first design job shares it, as you. If you're not allowed to +create test cases there, it can't, and `explore context status` says the context is **blocked**. Ask a Test Manager +admin for permission to create test cases, or ask someone who has it to share the project. Once you have permission, +turn the context off and on again (`explore context off <id>`, then `on`). + +### Get the tests into a folder and run them + +<!-- release: needs context pull --> +The tests are in the project as soon as a design job saves them (`explore context status <id>` says so). To run them +on your machine, bring them into a folder: + +```bash +mkdir explore-tests && cd explore-tests +kane-cli context pull --project <project-id> # links this folder and brings the tests in one step +``` + +In a folder that's already linked to the project, `kane-cli context pull` is enough. + +- The pull lists each test it wrote, and the variables those tests need: a start-page URL, or an app's sign-in + values, for example. Put them in `.testmuai/variables/assurance.json` (the format above). If the project has no + tests yet, the pull says so: wait for a design job. +- Each test arrives with no recorded steps yet, so its **first run** writes them in a real browser. That takes + minutes and uses more credits than a replay. Later runs replay the steps for far fewer. +- These tests run in Google Chrome on your machine, so install it first (if kane-cli can't find it, see + [Chrome failed to launch](./troubleshooting.md#chrome-failed-to-launch)). +- Open a test to see what it checks, then run the ones you want: `kane-cli testrun run .testmuai/tests/<name>_test.md`. + `kane-cli cover` then shows the project's coverage, the exploration's use-cases included. + +See [Shared projects](./assurance/shared-projects.md#bringing-the-projects-tests-into-your-folder) for what a pull +writes and how to update tests that changed. + +## Exit codes + +`explore run` on a terminal: `0` if you detach with Ctrl+C. If you watch until the run ends, it exits with the run's +own code: `0` completed or `budget`, `1` failed, `3` stopped. + +| `explore run` | With `--agent` or in a pipe (HyperExecute) | From your machine (`--local-orchestration`) | +|---|---|---| +| `0` | the run is queued | it ended `completed` or `budget` | +| `1` | refused: a run is already going (`RUN_ACTIVE`), or the launch failed | it ended `failed:…` | +| `2` | bad input, sign-in, or a secret problem (the run may be recorded as stopped: check `explore sessions`) | it couldn't start | +| `3` | | it was stopped | + +`explore list`, `sessions`, `stop` and `delete`: `0` done; `1` refused (for example `RUN_NOT_ACTIVE` on a stop when +nothing is running, or "No" to a delete) or the service failed; `2` bad input, a missing `--yes`, an unknown +exploration, or no sign-in. + +## Troubleshooting + +| What you see | Why | What to do | +|---|---|---| +| `required option '--url <url>' not specified` | the first run of a project's exploration needs a URL | pass `--url` | +| `a remote run needs a project …` | no `--project`, and none configured | pass `--project <project-id>` | +| `RUN_ACTIVE: run … is queued/running` | the exploration already has a run going | wait for it, or `kane-cli explore stop <id>` | +| `--url … is not exploration <id>'s URL` | a project explores one URL | use another project, or `explore delete` this one | +| `--deps needs --allow-domain …` | a sign-in needs the hosts it signs in on | add `--allow-domain` for each host it signs in on, the `--url`'s own host included | +| `host … is outside the run's domains` on `--allow-post` | the endpoint's host isn't one the run may visit | add it with `--allow-domain` first | +| `--deps uses {{name}} but no variable "name" is defined` | a variable your `--prompt` or `--deps` uses isn't in any variables file | add it to `--variables-file` or `.testmuai/variables/` | +| `HyperExecute Secrets does not list …`, or a run stopped `failed:secrets_missing` | a secret didn't reach HyperExecute, or it's empty | check its name and value in your variables file, mark it `"secret": true`, run again | +| a run stopped `failed:setup_failed` | the sign-in didn't work | check the values, and every sign-in host in `--allow-domain` | +| `could not push …`, `could not read your HyperExecute secrets`, or `HyperExecute refused the job` | a service problem | try again in a minute, or explore from your machine with `--local-orchestration` | +| `no running LT tunnel found …` | the tunnel isn't running, the name doesn't match, or it was started without `--env ht-prod` | start LT as in [step 2](#an-app-on-localhost-or-a-private-network), and pass the same `--tunnel-name` | +| `--tunnel needs remote browsers` | your own Chrome already reaches your machine | drop `--tunnel`, or use `--browser remote` | +| a `read_only` run shows empty pages | the app reads its data with POST | `--allow-post <host/path>` for the endpoints that only read | +| no tests in the project | the context is off or blocked, or no design job has ended yet | `kane-cli explore context status <id>`; see [No tests were designed](#no-tests-were-designed) | +| `TOO_SOON` on `explore context sync` | a design job started a few minutes ago | try again when it says | +| `NO_ENDED_RUN` on `explore context sync` | no run has ended yet | wait for the run to end | + +Driving exploration from an AI coding agent: every `explore` command takes `--agent` ([Using kane-cli from an AI +coding agent](./agents.md)). diff --git a/docs/user-guide/getting-started.md b/docs/user-guide/getting-started.md index 5db211e..9d81192 100644 --- a/docs/user-guide/getting-started.md +++ b/docs/user-guide/getting-started.md @@ -72,12 +72,31 @@ By default each run starts on the KaneAI playground site. The agent navigates fr When the session ends, kane-cli uploads the run to TestmuAI Test Manager and prints a share link. The run is also captured as a sealed [evidence pack](./evidence.md) — screenshots, logs, and results in one file you can open in the browser viewer when kane-cli offers it. For details on the upload, the share-link experience, and the run mode toggle, see [test-manager-integration](./test-manager-integration.md). To change settings like window size, Chrome profile, or the active project and folder, see [configuration](./configuration.md). +## With your team + +If your team keeps its tests in a Test Manager project, work in it as a **shared project**: what you design there, your teammates see and run. From an empty folder: + +```bash +kane-cli projects list --search "Checkout" # find the project's id +kane-cli context ingest prd.md --project <id> # link this folder, add the document, find its use-cases +kane-cli design tests --use-case uc-1 # design one use-case's tests, then save +kane-cli testrun run # run them +kane-cli cover # the project's coverage +``` + +<!-- release: needs context pull --> +To join a project your teammate already started, run `kane-cli context pull --project <id>` in a new folder: it links the folder and brings the project's tests. See [Shared projects](./assurance/shared-projects.md). + +Keep quick `kane-cli run` experiments in a different folder: `kane-cli run` uploads to your profile's project, not to the shared one. + ## Next steps - [Running tests](./running-tests.md) - [Evidence packs](./evidence.md) - [Configuration](./configuration.md) - [Authentication](./authentication.md) +- [Shared projects](./assurance/shared-projects.md) — design and run tests with your team. +- [Exploration](./exploration.md) — map your app and design tests from what it finds, without a requirements document. ## Reviews diff --git a/docs/user-guide/test-manager-integration.md b/docs/user-guide/test-manager-integration.md index dc14219..e99e901 100644 --- a/docs/user-guide/test-manager-integration.md +++ b/docs/user-guide/test-manager-integration.md @@ -83,6 +83,30 @@ The same auto-default kicks in if a previously-configured project or folder has After a successful upload, kane-cli prints two links to the terminal (see [Share links at session exit](#share-links-at-session-exit)). Both lead into the TestmuAI Test Manager UI for the test case that was just created. From there you can browse the project and folder you configured (or that were auto-defaulted), view the run summary, and dig into individual steps and screenshots. +## Shared projects: a folder's own project + +A folder linked to a [shared project](./assurance/shared-projects.md) has its own Test Manager project, separate from your profile's. + +| | Your profile's project | A folder's shared project | +|---|---|---| +| Set with | `kane-cli config project <id>`, or the first-run default | `--project <id>` on the folder's first `kane-cli context ingest` or `kane-cli context extract`, or the question those ask at a terminal | +| Kept in | your profile, under `~/.testmuai/` | `.testmuai/assurance.json` in the folder | +| Used by | `kane-cli run`, and tests that aren't designed in a shared project | the assurance commands, and designed tests' runs, in that folder | +| Shown by | `kane-cli config show` | `using project "<name>", shared with your team` at the start of each chat session in the folder, and of `kane-cli cover` | + +`kane-cli projects create` and `kane-cli config project` never link a folder. A `kane-cli run` in a linked folder still uploads to your profile's project. + +### Test cases from a design + +Saving a design session makes one test case per scenario in the folder's project, or updates the one it made before: + +- its title is the scenario's title, and its description is the scenario's intent and the criteria it verifies; +- it's tagged `assurance` and with its use-case's id; +- its steps are the test's steps; +- it's made at the project's top level, in no folder. + +A designed test's first run adds its KaneAI test to that case, and every later run is recorded on it. Coverage counts those runs. If you edit a case's steps in Test Manager, the next save doesn't overwrite them: it lists the case as one it couldn't update. + ## Code export Code export converts a completed test case into runnable Playwright code that you can check into your repository or extend by hand. diff --git a/docs/user-guide/testrun.md b/docs/user-guide/testrun.md index e2eaa2a..6e0b04e 100644 --- a/docs/user-guide/testrun.md +++ b/docs/user-guide/testrun.md @@ -23,6 +23,26 @@ Duplicates are removed and the final list runs in a stable order. kane-cli testrun run --match 'tests/e2e/.*' --tags smoke ``` +### When nothing is selected + +When no `_test.md` file is selected, nothing runs (exit `2`): + +```text +error: no *_test.md files selected. +``` + +Check the paths, or widen a `--match` or `--tags` filter; `--dry-run` shows what a selection picks. + +<!-- release: needs the testrun fix --> +<!-- release: needs context pull --> +In a folder linked to a [shared project](./assurance/shared-projects.md) that holds no tests at all, kane-cli says where to get them instead: + +```text +error: no tests here yet +next: to bring the project's tests into this folder, run: + kane-cli context pull +``` + ## Preflight Before anything runs, every member is checked: @@ -35,7 +55,7 @@ A member can fail preflight for these reasons: | Reason | Meaning | Fix | |---|---|---| | `org_mismatch` | Belongs to a different organisation than the rest | Check with `kane-cli testmd status <path>` | -| `project_mismatch` | Belongs to a different project than the rest | Check with `kane-cli testmd status <path>`; run project-by-project | +| `project_mismatch` | Belongs to a different project than the rest | The refusal lists each project's tests and names the command that runs one project's: see [Tests from two projects](#tests-from-two-projects) | If any member fails preflight, the plan is invalid and **nothing runs** (exit `2`). The offenders print to stderr: @@ -131,6 +151,58 @@ In a terminal, kane-cli offers to open the pack in the [evidence viewer](./evide | `2` | Usage error, invalid plan (preflight failures), or auth error. Nothing ran. | | `3` | Cancelled (Ctrl-C). | +## Designed tests in a shared project + +In a folder linked to a [shared project](./assurance/shared-projects.md), each designed test runs on its scenario's **test case** in Test Manager, so every run is recorded on that case and counts toward the project's coverage. + +A test runs on its case when its file sits directly in `.testmuai/tests/` and starts with the `assurance:` header the save writes. A designed file moved into a subfolder (`.testmuai/tests/smoke/`) or anywhere else runs as an ordinary test, under your profile's project, and doesn't count. + +### First run, replay and heal + +| Run | When | What it does | Cost | +|---|---|---|---| +| **First run** | The test has no `output-<name>/` beside it in this folder: it was just saved, pulled or copied here | Adds a KaneAI test to the scenario's test case, then writes each step in a real browser | More credits, and a few minutes | +| **Replay** | `output-<name>/` holds its recorded steps | Replays them | Far fewer credits | +| **Heal** | A replayed step no longer matches the page | Writes that step and the ones after it again, then goes on | Like a first run, for the steps it writes | + +A test's recorded steps stay in the folder that ran it. So a teammate's copy of the same test makes its own first run in their folder, on the same test case. + +A designed test's first run is on desktop web: Test Manager adds runs to a test case from web runs only, so a first run on a device (the file's `target:`, or `--target` on `kane-cli testmd run`) stops with `Test Manager can't add this test's runs to its test case yet, so the run didn't start`. + +### How runs count + +At the end of a run, kane-cli shows what it proved against the project: + +```text +coverage: 40% across "Checkout revamp"'s 5 use-cases, from this run's tests + this run reached 2 use-cases: 6 criteria passed, 1 failed +``` + +`kane-cli cover` then shows the project's figure from everyone's runs. A new run can take up to 5 minutes to count there. + +A test that stops before it starts (its session was never saved, or its test case was deleted) shows only as **failed** in the run's live view. Run it on its own with `kane-cli testmd run <file>` to see why, and see [Troubleshooting: shared projects](./troubleshooting.md#running-designed-tests). + +### Tests from two projects + +One `testrun run` is one execution in one Test Manager project. A linked folder can still end up holding tests from two projects. The usual way: `kane-cli run` uploads to your profile's project, not the folder's, and saves its test in `.testmuai/tests/` beside the designed ones. A whole-folder run then stops before anything runs: + +```text +error: these tests are in different Test Manager projects, and a run counts for one project only + this folder's project, "Checkout revamp": + .testmuai/tests/add_one_item_to_the_cart_test.md + .testmuai/tests/remove_the_last_item_test.md + "KaneAI Generated": + .testmuai/tests/search_for_headphones_test.md +next: to run the tests of "Checkout revamp", this folder's project, run: + kane-cli testrun run .testmuai/tests/add_one_item_to_the_cart_test.md .testmuai/tests/remove_the_last_item_test.md +``` + +Run the command it names. It lists the tests' paths. Past five, it gives a `--match` instead when one selects exactly those tests. To keep the folder clean, run `kane-cli run` experiments in a different folder. + +The same refusal comes in any folder whose tests are in two projects. + +In agent mode the refusal is an `error` event with `code: "PROJECT_MISMATCH"`, a `projects` list (each with `project_id`, `name`, `this_folder` and `tests`), and a `next` that runs the folder's project's tests. + ## Remote runs Add `--remote` and the same selection runs as one **HyperExecute** job: Chrome on a grid macOS runner for web suites, a grid emulator or simulator for mobile suites. Each member runs as its own headless `testmd run` on the grid, `--parallel N` spreads the members across N grid runners, and the job link, the recordings, and the evidence pack come back to your project. Requires a LambdaTest plan with HyperExecute (macOS runners) and the `remote-execution` plugin — nothing else locally, not even Chrome. diff --git a/docs/user-guide/troubleshooting.md b/docs/user-guide/troubleshooting.md index f698769..95fcda3 100644 --- a/docs/user-guide/troubleshooting.md +++ b/docs/user-guide/troubleshooting.md @@ -17,6 +17,8 @@ This page lists common problems you may hit while using kane-cli, what causes th - [Debugging a failed run with its evidence pack](#debugging-a-failed-run-with-its-evidence-pack) - [A `--remote` run refused, failed, or came back empty](#a---remote-run-refused-failed-or-came-back-empty) - [testrun says "plan invalid" or skips members](#testrun-says-plan-invalid-or-skips-members) +- [Shared projects: a command stopped](#shared-projects) — linking a folder, sessions, designs, runs on a test case, and the pull +- [Exploration: a run refused or designed no tests](./exploration.md#troubleshooting) — the URL, a login, secrets, the tunnel, design jobs - [Context sync: Git not found or too old](#context-sync-git-not-found-or-too-old) - [Context sync: a location cannot be reached or refuses you](#context-sync-a-location-cannot-be-reached-or-refuses-you) - [Context sync: "a rebase is open" — every change refused](#context-sync-a-rebase-is-open--every-change-refused) @@ -229,10 +231,107 @@ Full reference: [Remote runs on the cloud grid](./remote-execution.md). | `missing_meta` | The test has no recorded output directory next to it. | Run it once: `kane-cli testmd run <path>`. | | `not_authored` | The test ran but never committed (no test id). | Run it to completion so it commits. | | `org_mismatch` | The test belongs to a different organisation than the other members. | Check identities with `kane-cli testmd status <path>`. | -| `project_mismatch` | The test belongs to a different project than the other members. | Same check; run project-by-project, or re-home the test. | +| `project_mismatch` | The selected tests are in more than one Test Manager project. The refusal says so in words, lists each project's tests, and names the command that runs one project's: see [Batch runs](./testrun.md#tests-from-two-projects). | Run the command it names, which runs one project's tests by path. | + +<!-- release: needs the testrun fix --> +<!-- release: needs context pull --> +**No tests selected.** `error: no *_test.md files selected.` means no `_test.md` file matched: check the paths, or widen a `--match` or `--tags` filter. In a folder linked to a [shared project](./assurance/shared-projects.md) that holds no tests at all, kane-cli says `error: no tests here yet` instead, and its `next` is `kane-cli context pull`, which brings the project's tests into the folder. Use `kane-cli testrun run --dry-run …` to see the full plan and every offender without executing anything. See [Batch runs with testrun](./testrun.md#preflight). +## Shared projects + +In a folder linked to a [shared project](./assurance/shared-projects.md), a command that can't go on prints an `error:` line and a `next:` line, and changes nothing. In agent mode the same stop is an `error` event with a `code`: run its `next` when it has one, and when it has none, read its `remedy`, which says the same as the terminal's `next:` line. + +### Linking a folder + +| You see | What it means | What to do | +|---|---|---| +| `you don't have a project with the id "<id>"` (`PROJECT_NO_ACCESS`) | No project of yours has that id, or you can't open it | Find the id with `kane-cli projects list --search <name>` | +| `you don't have access to "<project>", the project this folder is linked to` (`PROJECT_NO_ACCESS`) | Your account can't open the folder's project | Ask a Test Manager admin to grant your account access, then run the command again | +| `this folder works locally, so --project can't link it to a project` (`FOLDER_CHOSEN`) | The folder already has its own local store (`.context/`) | Run it without `--project`. To share instead, move `.context` out of the folder first | +| `this folder is linked to "<project>", not to the project given` (`FOLDER_CHOSEN`) | The folder is already linked to another project | Run it without `--project`. To link it to another project, remove `.testmuai/assurance.json` first | +| `this folder is linked to "<project>", so --local can't keep it local` (`FOLDER_CHOSEN`) | `--local` in a linked folder | Run it without `--local`. To work locally, remove `.testmuai/assurance.json` first | +| `--project and --local can't be used together` (`FOLDER_USAGE`) | Both flags at once | Use one: `--project <id>` to share the folder's work, or `--local` to keep it here | +| `--project can't be used with --resume; a session stays where it started` (`FOLDER_USAGE`) | A session always goes on in the project it started in | Run it again without `--project` | +| `this folder has its own context store and is also linked to "<project>"` (`BOTH_STORES`) | The folder has both `.context/` and a link | To work locally, remove `.testmuai/assurance.json`. To work in the project, move `.context` out of the folder | +| `this folder's link to its project could not be read` (`PROJECT_FILE_UNREADABLE`) | `.testmuai/assurance.json` was edited or damaged | Remove it, then choose the project again with `kane-cli context extract` | +| `you can't start the first session in "<project>": it needs permission to create test cases` (`PERMISSION_REQUIRED`) | The project's first session makes test cases, and your account can't | Ask a Test Manager admin for that permission, or someone who has it to start the first session | +| `assurance isn't available for "<project>"` (`ASSURANCE_OFF`) | Shared projects are switched off for this project | Run the command in a folder that isn't linked to a project | +| `you're not logged in`, `couldn't confirm your login for shared projects` or `your credentials were rejected` (`CREDENTIALS_MISSING`, `CREDENTIALS_REJECTED`) | Shared projects need your sign-in | Run `kane-cli login` in a terminal. If the second one keeps happening, sign in with your username and access key: `kane-cli login --username <user> --access-key <key>` | +| `couldn't connect, so nothing was started`, or `couldn't connect for 2 minutes, so nothing was started` (`CONNECTION_FAILED`, `PREFLIGHT`) | The shared project, or the service that runs the agent, couldn't be reached | Check your connection, then run the command again. Nothing was started, so it's safe | + +### Commands a linked folder refuses + +`kane-cli <command> isn't available yet in a folder linked to a shared project` (`REMOTE_UNSUPPORTED`): the command works on a local store only. [What a linked folder doesn't do](./assurance/shared-projects.md#what-a-linked-folder-doesnt-do) lists each one and what to use instead. The most common: + +| Refused | Use instead | +|---|---| +| `kane-cli context review`, `kane-cli context explain` | `kane-cli context list` | +| `kane-cli cover gaps` | `kane-cli cover` | +| `--mode ci` | `--mode agent` | +| `kane-cli testrun run --from-context <ids>` | the `kane-cli testrun run <files>` command the refusal names | + +### Sessions + +| You see | What it means | What to do | +|---|---|---| +| `someone else is designing tests for "<use-case>" right now, so this design can't start` (`PROJECT_BUSY`) | One session at a time designs a use-case, and this one was refused, not queued | Wait for it to finish, then run the command again. `kane-cli context sessions` lists the project's sessions | +| `someone else's extract is running in "<project>", so a design can't start until it finishes` (`PROJECT_BUSY`) | A design waits for an extract to end | Wait for it to finish, then run the command again | +| `you're already designing tests for "<use-case>" in another session` (`PROJECT_BUSY`) | Your own session holds it | Go on with it here with the `kane-cli design tests --resume <sid>` it names; the other terminal then stops | +| `no session <sid> of yours` (`SESSION_NOT_FOUND`) | The id is wrong, or the session is a teammate's | See your sessions with `kane-cli context sessions`. Only the person who started a session can save it | +| `refusing to save this session without --yes (no terminal to confirm)` (`CONFIRM_REQUIRED`) | There's no terminal to confirm the save in | Add `--yes`: `kane-cli context sessions save <sid> --yes` | +| `session <sid> is finished, so it can't be resumed` (`SESSION_FINISHED`) | It's done and waiting to be saved | Save it: `kane-cli context sessions save <sid>` | +| `not saved — "<project>" changed after this session started` (`PROJECT_CHANGED`) | A teammate saved a change to the same work first | The `next` lines say how to start again: design again, or add the session's documents again | +| `this session read 2 sources, so it can't be resumed or saved` | An extract that read two or more documents and then stopped can't go on | Run the commands it prints: discard the session, then start again with its documents. To keep extracts resumable, add one document at a time | +| `nothing to save — this session added nothing` | Not an error: the session found nothing new, so it was discarded | Nothing to do | +| `nothing new to extract: every source is already read` | Not an error: every document in the project is already read | To add more, run `kane-cli context ingest <files>` | + +### Designing + +| You see | What it means | What to do | +|---|---|---| +| `"<use-case>" isn't approved yet` (`UC_UNREVIEWED`) | The use-case is still proposed. An extract in agent mode leaves its use-cases proposed, and a linked folder has no review command | Check the use-case with the person (`kane-cli context list`), then run the `next`: `kane-cli design tests --use-case <id> --allow-unreviewed`. At a terminal, finishing the extract chat with `looks good — finish the session` approves them | +| `"<use-case>" already has tests; --force designs them again, replacing the ones it has` | It's designed already (agent mode) | To design it again, add `--force` | +| `error: agent ended without a proposal for <source>` (`NO_PROPOSAL`) | The agent stopped before it proposed use-cases for that document | Run the extract again once: it reads the document again and uses credits again. If it happens twice, [report it](#reporting-bugs) | + +### Running designed tests + +A designed test in a linked folder runs on its scenario's test case in Test Manager. + +`these tests are in different Test Manager projects, and a run counts for one project only` (`PROJECT_MISMATCH`) stops the whole `testrun run` before anything runs. The folder holds tests from two projects, usually a `kane-cli run` test (your profile's project) beside designed tests (the folder's project). Run the command it names, which runs one project's tests by path, and keep `kane-cli run` experiments in another folder. See [Tests from two projects](./testrun.md#tests-from-two-projects). + +The stops below are each test's own. In `kane-cli testrun run` at a terminal, a test that stops for one of them shows only as **failed**. To see why, run it on its own: `kane-cli testmd run <file>`. + +| You see | What it means | What to do | +|---|---|---| +| `this test isn't in Test Manager yet, so it can't run` (`TEST_CASE_MISSING`) | Its design session was never saved, so its scenario has no test case | Save the session that designed it: the `next` names `kane-cli context sessions save <sid>` | +| `this test's test case was deleted or moved to another project in Test Manager, so it can't run` (`TEST_CASE_GONE`) | Someone deleted or moved the case | Save the session that designed it again; that makes a new test case | +| `this test's test case is archived in Test Manager, so it can't run` (`TEST_CASE_ARCHIVED`) | The case is archived | Restore it in Test Manager, then run the test again | +| `this test isn't in "<project>", and the session that designed it can't be saved anymore` (`TEST_NOT_IN_PROJECT`) | The file came from another project, or from a design that was never saved | Design the test again and save that session: `kane-cli design tests --use-case <id>` | +| `this test isn't in "<project>", which has no use cases or tests yet` (`TEST_NOT_IN_PROJECT`) | The file was designed in another project | Run it in the folder linked to the project it was designed in | +| `Test Manager couldn't save this run, so <case> doesn't have it yet; the test itself passed` (`TEST_RUN_NOT_SAVED`) | The run finished, but Test Manager didn't record it | The test's next run tries again. To try now: `kane-cli testmd run <file>` | + +### Coverage + +| You see | What it means | What to do | +|---|---|---| +| `a new run can take up to 5 minutes to count` | Coverage reads Test Manager's runs, and a new run takes a few minutes to show | Run `kane-cli cover` again in a few minutes | +| `coverage: "<project>" is still working it out; run the command again in a moment` | The project is working out its figure | Run `kane-cli cover` again | +| `couldn't load "<project>"'s coverage` (`COVERAGE_FAILED`) | The figure couldn't be read | Run the command again. Share the support id it prints if it keeps happening | + +<!-- release: needs context pull --> +### Bringing the project's tests here + +| You see | What it means | What to do | +|---|---|---| +| ``error: no context store here (run `kane-cli context ingest <files>` first)``, from `kane-cli cover`, `context list` or `context pull` in a new folder | The folder isn't linked to a shared project and has no local store yet | To join your team's shared project and bring its tests here, run `kane-cli context pull --project <id>` (`kane-cli projects list` shows the ids). To start with your own documents, run `kane-cli context ingest <files>` | +| `kept <file> — you changed it here` | Not an error: your edits are kept | To replace them with the project's version, run `kane-cli context pull --overwrite` | +| `"<project>" has no tests yet` | Nothing has been designed in the project yet | `kane-cli cover` shows which use-cases need tests designed | +| `this folder takes its tests from "<project>", not from a location` | A location name was given in a linked folder | Run `kane-cli context pull` with no name | +| `"<project>" has no use-case <id>` | The `--use-case` id isn't in the project | See its use-cases with `kane-cli context list` | +| `the tests of "<project>" couldn't be read, so none was written` | The project couldn't be read; nothing changed here | Run `kane-cli context pull` again | + ## Context sync: Git not found or too old A GitHub location (`kane-cli context sync add`, `kane-cli context clone`, or `kane-cli context sync setup`) needs Git **2.31 or newer** on the machine that runs kane-cli. Without it the command refuses before touching anything: diff --git a/docs/user-guide/variables-and-context.md b/docs/user-guide/variables-and-context.md index 728513e..d9f2bf1 100644 --- a/docs/user-guide/variables-and-context.md +++ b/docs/user-guide/variables-and-context.md @@ -191,6 +191,23 @@ For each name it declares, `kane-cli` writes an **empty stub** into `.testmuai/v Fill `value`. `kane-cli` never overwrites a key that already exists in any of your variable files, so a stub cannot replace a value you set. A value you type in chat stays a literal in the step; a variable is created only when the value is unknown. +### Variables in a shared project + +In a folder linked to a [shared project](./assurance/shared-projects.md), the tests are shared but the values aren't. The project holds each test's `{{name}}` references, never a value. Each person fills the values in their own folder. + +- **Design writes the stubs where it ran.** When a design ends, and again when it's saved, kane-cli writes an empty stub for each value the tests need into `.testmuai/variables/assurance.json` in that folder, as it does locally. +<!-- release: needs context pull --> +- **A teammate's folder has no values.** When the project's tests come to another folder through [`kane-cli context pull`](./assurance/shared-projects.md#bringing-the-projects-tests-into-your-folder), the pull writes no stubs. It names the values the tests need, and where to add them: + + ```text + ⚒ 2 variables need values before these tests run: login_email and login_password + add them to .testmuai/variables/assurance.json + ``` + +- **A run sends its values to Test Manager.** Every run sends the non-secret values it loads to Test Manager: those in your global and folder variable files, in `--variables` and `--variables-file`, and in the test's own `variables:`. Test Manager keeps one value per name for your whole organisation, so each run's value replaces the one before it, for everyone. An empty stub is sent as an empty value. Values marked `"secret": true` go to your account's secrets instead, never to Test Manager's variables. + +So fill real values before a run, keep test accounts and passwords `secret`, and never fill a value only to quiet the warning. + ### Before a run: unresolved variables `kane-cli run`, `kane-cli testmd run` and `kane-cli testrun run` check every `{{name}}` an authored step references **before anything starts**. A name with no value is a warning: the run goes ahead and types the name as written, unless a step sets it first: diff --git a/skill-installer/package.json b/skill-installer/package.json index fa40fae..b16c172 100644 --- a/skill-installer/package.json +++ b/skill-installer/package.json @@ -1,6 +1,6 @@ { "name": "@testmuai/kane-cli-skill", - "version": "0.1.3", + "version": "0.2.0", "description": "End-to-end (E2E) test generation and execution: generate use-cases, acceptance criteria, scenarios and tests from requirements, then run them on desktop web, Android and iOS (emulators, simulators and real devices) with execution evidence and coverage.", "type": "module", "bin": "./cli.js", diff --git a/skill-installer/skills/SKILL.md b/skill-installer/skills/SKILL.md index 1937ed9..1aebb55 100644 --- a/skill-installer/skills/SKILL.md +++ b/skill-installer/skills/SKILL.md @@ -1,6 +1,6 @@ --- name: kane-cli -description: "End-to-end (E2E) test generation and execution with KaneCLI. Generate tests from a PRD, specification, or user-provided description via the assurance commands (use-cases, acceptance criteria, scenarios, runnable _test.md tests; never write test cases by hand). Execute them on desktop web (Chrome, the default), Android and iOS: emulator/simulator locally on macOS Apple Silicon or via testrun run --remote, and real cloud devices from any machine (--target android|ios). Prove coverage with evidence packs and kane-cli cover. Use for any task needing a real browser or mobile app: navigation, clicks, form filling, UI checks, screenshots, deploy verification. Parse NDJSON output, inspect logs, and save replayable _test.md files. Share the assurance store (.context/) with a team through a GitHub repository, S3-compatible bucket, or folder via kane-cli context sync, and resolve decisions requested during a sync rebase." +description: "End-to-end (E2E) test generation and execution with KaneCLI. Generate tests from a PRD, specification, or user-provided description via the assurance commands (use-cases, acceptance criteria, scenarios, runnable _test.md tests; never write test cases by hand). Execute them on desktop web (Chrome, the default), Android and iOS: emulator/simulator locally on macOS Apple Silicon or via testrun run --remote, and real cloud devices from any machine (--target android|ios). Prove coverage with evidence packs and kane-cli cover. Work in a team's shared Test Manager project from a linked folder: link it with --project, save sessions, run the designed tests on their Test Manager cases, and read the project's coverage. Use for any task needing a real browser or mobile app: navigation, clicks, form filling, UI checks, screenshots, deploy verification. Parse NDJSON output, inspect logs, and save replayable _test.md files. Share a local assurance store (.context/) through a GitHub repository, S3-compatible bucket, or folder via kane-cli context sync, and resolve decisions requested during a sync rebase." --- # Kane CLI — End-to-End Test Generation and Execution Skill @@ -11,6 +11,12 @@ KaneCLI is one E2E package with three pillars: 2. **Execute**: desktop web (Chrome, the default), Android emulator, iOS simulator, and real Android and iOS devices in the TestMu AI cloud. 3. **Prove**: a sealed evidence pack per run, and coverage (`kane-cli cover`) measured against acceptance criteria. +**Working with a team?** On kane-cli 0.10.0+ a team works in a **shared project**: one Test Manager project that its +folders are linked to. What one person saves there (use-cases, tests, Test Manager cases, coverage) the whole team sees. +Test files stay in the folder where they were saved until they're brought over (`references/shared-projects.md` §7). In +agent mode, a folder that isn't set up yet goes local unless you link it with `--project <id>`. So decide where the work +goes before the first assurance command, and read `references/shared-projects.md` first. + Use `kane-cli` to validate complete user journeys and design requirement-linked, coverage-accounted suites from a PRD, specification, or user-provided description via assurance commands. Never write test cases by hand. Do NOT use Playwright, Puppeteer, or Selenium directly. Use `--agent` for `run` and `testmd run`. `testrun run` emits NDJSON when **stdin** is not a TTY (0.8.21+ also takes `--agent`, which selects the same stream), so every `testrun run` line you write ends in `< /dev/null` (bash and zsh: macOS, Linux, Git Bash; in cmd.exe write `< NUL`; from PowerShell run it through cmd: `cmd /c "kane-cli testrun run … < NUL"`). Check the first stdout line: on 0.8.17+ it is `{"type":"stream_start"…}`. A prose plan there instead means the CLI is in terminal mode: it shows the human view and, after a real run, opens an evidence table that waits until `q` or Esc, so the process never exits on its own; stop it and rerun with the redirect. An empty stdout with a message on stderr is a usage error: read it. Assurance conversational commands use `--mode agent`. **Authoring test cases or scenarios?** Never write them by hand: every test case comes from the **assurance** commands — Read `references/assurance.md` first. @@ -54,7 +60,7 @@ On Windows PowerShell: `$env:KANE_CLI_USER_AGENT='<your-runtime>'; kane-cli run **Watch mode.** Use the person's saved preference (`references/agent-config.md`). With none saved, show the browser unless the preflight says there is no display, an SSH session, or CI: then add `--headless`. -**Keeping runs.** When the person's saved purpose is `suite` or `ask`, add `--name <short-slug>` to every one-off `run`. A named run is recorded as a `_test.md` while it runs, so keeping it afterwards costs nothing, and a run launched without a name cannot be kept without running again. With `one-off`, leave the flag out. Details: `references/first-run.md` §4. +**Keeping runs.** When the person's saved purpose is `suite` or `ask`, add `--name <short-slug>` to every one-off `run`. A named run is recorded as a `_test.md` while it runs, so keeping it afterwards costs nothing, and a run launched without a name cannot be kept without running again. With `one-off`, leave the flag out. Details: `references/first-run.md` §4. Run one-offs from a folder that isn't linked: a named run in a linked folder writes its test beside the project's and can mix two projects (`references/shared-projects.md` §9). Bash blocks until kane-cli exits, then hands you the complete stdout. Parse it, summarize what happened, and present the result card. Wait for process completion on `testmd run` too, but parse its own completion event, `test_md_done`. An intermediate `run_end` does not finish a saved test. @@ -144,7 +150,7 @@ For exit code 1 (or `status: "failed"` in `run_end`), present the failure card. The failing step's screenshot lives inside the run's evidence pack (the stderr hint names the pack path): extract it with `unzip <pack> "tests/*/steps/*/screenshot.png" -d <tmpdir>`, Read it, and show it **under** the card. For the pack layout and deeper diagnosis, see `references/debug.md`. -Exit code 2 means nothing ran: that is a `🟡 Didn't start` card, not a failure (`references/cards.md` §4). +Exit code 2 with no progress events means nothing ran: that is a `🟡 Didn't start` card, not a failure (`references/cards.md` §4). If progress events came before it, steps ran and the run stopped without a result: say so, and use the stopped-early card (`references/cards.md` §5). ### 1.7 After the first result: three choices, then save @@ -163,19 +169,28 @@ When the user's request involves a browser — or writing test cases: - A problem that stops the run → offer the fix from the card; deeper setup lives in `references/setup-and-config.md` - Ready ↓ +**Where does the work go?** (before any assurance command) +- The folder holds its own `.context/` and no link → it works locally, as before (with a link too, kane-cli refuses it `BOTH_STORES`): `references/assurance.md` (to share that store with a team, `references/context-sync.md`). The rows below are for a folder without one +- The folder holds `.testmuai/assurance.json`, or the person works with a team or names a Test Manager project → a **shared project**: Read `references/shared-projects.md` first. In a linked folder never use `kane-cli run` for a first result; its tests go to another project (§9 there) +- The person works alone, or asks to keep the work on this machine → the local store: `references/assurance.md` +- Not clear → ask once: "Should this work go to your team's Test Manager project, or stay in this folder?" + **What does the user want?** +- Tests that exploration (HyperExecute) designed, or a teammate's tests, in this folder → Read `references/shared-projects.md` §7 +- Explore an app on HyperExecute, shape, stop or re-run an exploration, or check what it designed → Read `references/explore.md` first +- A command was refused, or you need the shared-project event and code list → Read `references/agent-contract.md` - A single one-shot browser task → build a `kane-cli run --agent` command (§3 + §4) - A test they want to save / re-run / commit → Read `references/testmd.md` first, then use `kane-cli testmd` - Run a suite of saved tests (several `_test.md` at once) → Read `references/testrun.md` first, then use `kane-cli testrun run … < /dev/null` - Need test cases from a description the user gives in chat, with no document → **don't hand-write them**; write the description to a requirements file and design from it (§6) - Has requirement documents (PRD/spec) and wants a designed suite, coverage accounting, or suite upkeep → Read `references/assurance.md` first — the assurance commands (`context`/`design`/`cover`/`maintain reconcile`, kane-cli 0.6.1+; several features need newer releases, up to 0.8.14+ — the reference marks each) - A **designed** test (its `_test.md` carries an `assurance:` block) failed a run → Read `references/assurance.md` §6.1 **before touching the file** — an edit is adopted as the next design version on the next run; classify the failure first (app bug → report it; requirement changed → reconcile; wording → redesign through the CLI; capability missing → stop), and edit a step of an existing designed test by hand only after reading `references/objectives-cookbook.md` -- Share the context store with a team, join a teammate's, keep two stores level, or resolve a sync conflict → Read `references/context-sync.md` first — `kane-cli context sync`, `kane-cli context push`, `kane-cli context pull` and `kane-cli context clone` (kane-cli 0.8.14+); the store is shared through a location, never by copying or git-merging `.context/` +- Share a **local** context store through a repository, a bucket or a folder, join one, keep two stores level, or resolve a sync conflict → Read `references/context-sync.md` first (kane-cli 0.8.14+). A team in Test Manager uses a shared project instead (`references/shared-projects.md`); a linked folder refuses `kane-cli context sync`, `kane-cli context push` and `kane-cli context clone` (`REMOTE_UNSUPPORTED`), and a location pull (a location name, `--rebase`, `--yes` or `--answer`: `PULL_LOCATION_LINKED`); a plain `kane-cli context pull --mode agent` there brings the project's tests (`references/shared-projects.md` §7); the store is shared through a location, never by copying or git-merging `.context/` - Multiple independent browser tasks → Read `references/parallel.md` first - View, share, or validate run evidence (`.evidence` packs) → Read `references/evidence.md` - Debug a failed run → Read `references/debug.md` - Configure kane-cli or check directory layout → Read `references/setup-and-config.md` -- Browse / create / pick a Test Manager project or folder, or interpret the auto-default event → Read `references/test-manager.md` +- Browse / create / pick a Test Manager project or folder for `kane-cli run`, or interpret the auto-default event → Read `references/test-manager.md`. Linking an assurance folder to a project is different: `references/shared-projects.md` §2–3 - The person wants results saved somewhere else ("change project") → Read `references/test-manager.md` §6. The change is global, and the question must say so - The person wants to change how runs behave ("kane preferences": watch or quiet, one-off or suite) → Read `references/agent-config.md` - The person asks what kane-cli can do, or for the tour again ("kane tour") → show the tour from `references/first-run.md` §2 @@ -204,7 +219,7 @@ kane-cli run "<objective>" --agent [options] | Flag | Purpose | Default | |------|---------|---------| | `--headless` | No visible browser window | Off | -| `--max-steps <n>` | Cap agent reasoning steps | 30 | +| `--max-steps <n>` | Cap agent reasoning steps | 50 | | `--timeout <s>` | Hard kill after N seconds | No limit | | `--url <url>` | Start URL for the run (overrides config `default_url`; bare domains get `https://`) | Config `default_url` | | `--variables <json>` | Inline variables JSON (for `{{key}}` in objective) | None | @@ -226,7 +241,7 @@ Other flags (`--global-context`, `--local-context`, `--cdp-endpoint`, `--allow-m 1. Collect the names with no value: the `warning`, a dry-run plan's `unresolved[]` rows, or a design run's `variables_declared` and `variables_summary` rows. Design rows list what design declared, not every missing value, and a dry run's `valid: true` says nothing about values: the dry run's `warning` is the check. 2. A value you already have, because the user said it or the requirement document states it, you write into that key in the file the event names (for `run`, `--variables '{"name":{"value":"…"}}'` also works), and you tell the user what you filled and where. 3. The rest you ask for once, in one message, using each variable's `description` when design gave one. Plain values (a URL, an email, a user name) the user gives you here or adds to the file, their choice. Secrets, which are any row with `secret: true`, any description that names a credential, and any name containing `password`, `secret`, `token` or `key`, the user fills in the file and tells you when done; you never ask for the value in chat, never echo it, and report names and file paths only. -4. Then a fresh `--dry-run` of the exact selection: a valid plan with no `warning` is the check. A member whose rows are all filled may run while the others wait. Never fill a placeholder just to silence the check; a throwaway value is right only when the description asks for one, such as a deliberately wrong password. A frontmatter declaration with an empty value also silences the check, so look at the values a step relies on, not only at the warning. A name still empty is typed into the page as written; if a run then failed at that step, say so. +4. Then the check: for saved tests, a fresh `kane-cli testrun run <files> --dry-run --agent < /dev/null` of the exact selection (`run` and `testmd run` have no dry run; for them the `warning` arrives before the first step): a valid plan with no `warning` is the check. A member whose rows are all filled may run while the others wait. Never fill a placeholder just to silence the check; a throwaway value is right only when the description asks for one, such as a deliberately wrong password. A frontmatter declaration with an empty value also silences the check, so look at the values a step relies on, not only at the warning. A name still empty is typed into the page as written; if a run then failed at that step, say so. ### Examples @@ -330,6 +345,8 @@ For one-shot `run`, build post-run logic on `run_end` and process exit. Saved te For full event schemas (`bifurcation` flow fields, `child_agent_*`, `ask_user` semantics, `cancel`/`user_response` outbound events, complete `run_end` field list), Read `references/parsing.md`. +**Which agent flag.** `--agent`: `run`, `testmd run`, `testrun run` (or stdin not a TTY), `projects`, `folders`, `explore`, `config show`, `generate`, `devices`, `apps`. `--mode agent`: `context …`, `design …`, `maintain …`, `cover`. `run` and `testmd run` also accept `--mode agent`; `testrun`, `projects`, `folders` and `explore` refuse it, and the assurance commands refuse `--agent`. With `--agent`, an update notice can still print on stderr; it is not part of the stream. In a linked folder the streams, events and refusal codes are in `references/agent-contract.md`. + The assurance conversational commands (`context ingest`/`context extract`, `design tests`, `maintain reconcile`, `maintain evolve` on 0.8.21+, `cover`) do NOT take `--agent` — they take **`--mode agent`** and speak their own typed stream ending in `done` (open vocabulary — tolerate unknown event types; on 0.7.2+ the stream is strict — every stdout line parses, stderr silent — while on 0.7.1 a merged `context ingest` prints a few prose receipt lines BEFORE the stream — skip to the first `{` line, harmless on 0.7.2+ — and a landing-phase ingest failure ends with prose + exit 1/2 and no stream at all: a refusal, not a crash; on 0.7.2+ those failures ride the stream as `error` + `done`); **for those commands only, exit `3` means paused-and-resumable, not timeout** — schema in `references/assurance-parsing.md`, behavior in `references/assurance.md`. The context sync verbs (`kane-cli context sync`, `kane-cli context push`, `kane-cli context pull`, `kane-cli context clone`; kane-cli 0.8.14+) take `--mode agent` the same way and speak a `sync_*` family ending in `done`; there too exit `3` means a decision or a pull is needed, not a failure — `references/context-sync.md`. `kane-cli context sync setup` refuses without a terminal (`TTY_REQUIRED`): agents use `kane-cli context sync add` and `kane-cli context clone`. `kane-cli testrun run` also emits its own typed stream (`testrun_plan` … terminal `testrun_done`) — schema in `references/testrun.md`. `kane-cli testmd run` may additionally emit `test_md_evidence_ingest` (replay evidence published) and `test_md_bundle_sync` (test bundle synced) — informational; describe in plain language, never surface raw names. The post-run evidence hint (`` evidence: view locally with `kane-cli evidence serve <path>` ``) is a **stderr** text line, not a stdout event — don't try to parse it from the NDJSON stream; see `references/evidence.md` for how to act on it. @@ -341,7 +358,7 @@ The assurance conversational commands (`context ingest`/`context extract`, `desi When the user describes what to test in chat and has no document, use the description as the requirement. 1. Write the user's words to `requirements/<feature>.md` in the project, as given, one heading per feature, nothing invented. Tell the user the file exists and that the tests will cite it. -2. `kane-cli context ingest requirements/<feature>.md --mode agent`, review the extracted use-cases at the checkpoint, then `kane-cli design tests …`, exactly as `references/assurance.md` describes. +2. `kane-cli context ingest requirements/<feature>.md --mode agent`, review the extracted use-cases at the checkpoint, then `kane-cli design tests …`, exactly as `references/assurance.md` describes. In a shared project add `--project <id>` the first time, and follow the journey in `references/shared-projects.md` §4 (save each session; the one-question checkpoint). 3. Present the designed tests, gaps, warnings and the variables that need values (§3 **Fill the variables before any run**). The designed tests get their own review checkpoint before any run (`references/assurance.md`). Then hand the set to `kane-cli testrun run … < /dev/null` (`references/assurance.md`, the authoring bridge). A draft in chat cannot be run, and a hand-written `_test.md` carries no requirement link and no coverage accounting; the deliverable is the designed test. @@ -356,6 +373,10 @@ A draft in chat cannot be run, and a hand-written `_test.md` carries no requirem | Run a suite of saved `_test.md` tests as one batch | `references/testrun.md` | | Run a suite on the cloud grid (`--remote`), incl. mobile suites from any machine | `references/testrun.md` §Remote + `references/mobile.md` §Remote | | Run on a real Android/iOS device (`--target android\|ios`), its plan requirements, or a hub session error | `references/mobile.md` §Real devices | +| A team, a Test Manager project, a linked folder (`.testmuai/assurance.json`), a teammate's or exploration's tests | `references/shared-projects.md` | +| This folder's shared project and its id (for a teammate, or a dashboard) | `references/shared-projects.md` §2, "This folder's project id" | +| A shared-project refusal code, event or exit code | `references/agent-contract.md` | +| Explore an app on HyperExecute | `references/explore.md` | | User has requirement docs (PRD/spec) → designed suite, coverage, or suite upkeep | `references/assurance.md` | | Need the assurance NDJSON event schema (`--mode agent`) | `references/assurance-parsing.md` | | Share the context store with a team, join one, keep stores level, or resolve a sync conflict (kane-cli 0.8.14+) | `references/context-sync.md` | diff --git a/skill-installer/skills/references/agent-config.md b/skill-installer/skills/references/agent-config.md index fa35bb8..4ab53e7 100644 --- a/skill-installer/skills/references/agent-config.md +++ b/skill-installer/skills/references/agent-config.md @@ -34,7 +34,7 @@ They follow the person across agents and projects, and they survive a skill rein | Key | Values | Meaning | |---|---|---| | `preferences.watch` | `visible` · `quiet` · `results-only` | `visible`: no `--headless`. `quiet`, `results-only`: `--headless`. `results-only` also skips the progress summary | -| `preferences.purpose` | `one-off` · `suite` · `ask` | Whether to offer keeping passing runs as saved tests. With `suite` or `ask`, launch every one-off run with `--name <short-slug>` so keeping it costs nothing (`references/first-run.md` §4) | +| `preferences.purpose` | `one-off` · `suite` · `ask` | Whether to offer keeping passing runs as saved tests. With `suite` or `ask`, launch every one-off run with `--name <short-slug>` so keeping it costs nothing (`references/first-run.md` §4), except in a folder linked to a shared project (`shared-projects.md` §9) | | `preferences.narration` | `quiet` · `milestones` · `every-step` | How much of the run you recount afterwards. Default `milestones` | | `onboarding.asked` | list of `watch`, `results`, `purpose` | What was already asked. Never ask these again | | `onboarding.first_run_explained` | boolean | The tour was shown | @@ -90,4 +90,4 @@ Before you write, tell the person in one line what you are saving and where. The ## 5. Changing preferences later -When the person says "kane preferences" (or asks to change how runs behave), show the current values in plain words, ask what to change, and write the file again. To change where results go, use the flow in `references/test-manager.md`: that setting is global and belongs to kane-cli. +When the person says "kane preferences" (or asks to change how runs behave), show the current values in plain words, ask what to change, and write the file again. To change where results go, use the flow in `references/test-manager.md`: that setting is global and belongs to kane-cli (in a folder linked to a shared project, designed tests directly in its `.testmuai/tests/` run on that project's cases whatever it says: `shared-projects.md`). diff --git a/skill-installer/skills/references/agent-contract.md b/skill-installer/skills/references/agent-contract.md new file mode 100644 index 0000000..1feb643 --- /dev/null +++ b/skill-installer/skills/references/agent-contract.md @@ -0,0 +1,329 @@ +<!-- kane-cli skill reference: the agent contract for a folder linked to a shared project (kane-cli 0.10.0+). Which agent flag each command takes, the NDJSON shapes, every event, `next`, exit codes, and every refusal code with what to do. Internal parsing reference: never show event or field names to the person. --> + +# Shared projects: Agent Contract + +A folder is **linked** when `.testmuai/assurance.json` names a TestMu AI Test Manager project and the folder has no `.context/`. In a linked folder the commands below work in the **shared project**, so the team sees what each person saves. Link a folder with `kane-cli context ingest <files> --project <id> --mode agent` (or `kane-cli context extract --project <id> --mode agent`, or, where `kane-cli context pull --help` lists `--project`, `kane-cli context pull --project <id> --mode agent`); find the id with `kane-cli projects list --search <name> --agent`. In a folder with neither a link nor `.context/`, `--mode agent` without `--project` goes on locally without asking, as before: `context ingest` makes a local store, and `context extract` refuses `NO_STORE` until one exists. + +| Area | Commands | +|---|---| +| Projects | `kane-cli projects list`, `kane-cli projects create` | +| Sources and use-cases | `kane-cli context ingest`, `kane-cli context extract`, `kane-cli context list` | +| Sessions | `kane-cli context sessions`, `kane-cli context sessions save <sid>`, `kane-cli context sessions discard <sid>` | +| Tests | `kane-cli design tests`, `kane-cli maintain evolve`, `kane-cli maintain reconcile`, `kane-cli context pull` (`shared-projects.md` §7) | +| Runs and coverage | `kane-cli testmd run <designed>_test.md`, `kane-cli testrun run`, `kane-cli cover` | + +Nothing reaches the project until `kane-cli context sessions save`. Other assurance commands are refused in a linked folder (`REMOTE_UNSUPPORTED`, §6). Local-store parsing stays in `references/assurance-parsing.md`. + +## 1. Which flag + +| Command | Agent flag | The wrong one does | +|---|---|---| +| `context ingest`, `context extract`, `context list`, `context sessions` (list, save, discard), `design tests`, `maintain evolve`, `maintain reconcile`, `cover` | `--mode agent` | `--agent` is an unknown option: prose on stderr, no NDJSON, non-zero exit | +| `testmd run` | `--agent` (`--mode agent` works too) | — | +| `testrun run` | `--agent`, or stdin not a terminal | `--mode agent` is refused: `error` `USAGE` + `done` (exit 2); nothing runs | +| `projects list`, `projects create` | `--agent` (or stdin/stdout not a terminal) | `--mode agent` is refused the same way | +| `install skill` | none: with stdin not a terminal it prints one JSON line | `--agent` is an unknown option (exit 1); `--mode agent` is `USAGE` (exit 2) | + +- **Stderr.** Only a command line holding `--mode agent` keeps stderr free of the "Skill update available" and "Update available" banners. Under `--agent` those lines can appear on stderr, and `testrun run` also writes its coverage lines and `kane-cli evidence serve` hints there. Parse stdout only; a stderr line is not a failure. +- **No flag, no terminal.** Linked `context extract` and `design tests` refuse with a stderr line (exit 2). Linked `context ingest` falls back to land-only: it uploads, prints text lines and leaves the session paused. Always pass the flag. +- **Modes to avoid.** `--mode ci` on linked extract, design or reconcile is refused as a stderr line (exit 2), even with `--json`. `--json` with `--mode agent` is refused as a stderr line (exit 2). `--mode override` answers every question with its recommended option: never use it for the person. +- **Usage errors.** With `--mode agent` anywhere on the command line, a parse error arrives as `error` `USAGE` + `done{refused, 2}`; `--help` and `--version` print only a `done` line. + +## 2. Stream shape + +**Assurance stream** (`context …`, `design tests`, `maintain …`, `cover`): one JSON object per line, envelope `{"type": "<name>", "v": 1, "verb": "<verb>", …}`. No `stream_start`, and don't rely on a `ts` field here. Verbs: `sync` (`context pull` in a linked folder; refusals of its flags and at its entry carry `pull`), `extract` (ingest and extract; `reconcile` from the upload on when the ingest is a new version of a source the project holds), `design`, `evolve`, `reconcile`, `save`, `discard`, `cover`. Refusals of `context sessions` (list, show, clean) are stamped `save`, and those of `context list` are stamped `list`. + +**Run stream** (`testmd run`, `testrun run`): opens with `stream_start {cli_version, surface, pid}`; every line carries `v` and `ts`. Shapes: `references/testmd.md`, `references/testrun.md`; linked-folder additions in §3.7. + +| Command (agent flag) | First line | Last line | +|---|---|---| +| `context extract` | `folder_linked` (with `--project`), else `nothing_new`, `warning` or `remote_connect` | `done` | +| `context ingest <refs>` | `folder_linked` (with `--project`), else `remote_fetch` (an address) or `remote_upload` (a file); a refusal is `error` | `done` | +| `design tests` | `remote_connect` (or `warning`, `error`) | `done` | +| `maintain evolve` | `remote_connect` | `done` (may carry `not_updated`) | +| `maintain reconcile` | `remote_upload` or `remote_fetch`, else `remote_connect` | `done` | +| `context sessions save <sid>` | `session_save{phase:"check"}`; `phase:"saving"` for a session saved before; `phase:"saved"` with `saved: 0` when it added nothing (or `error`) | `done` | +| `context sessions discard <sid>` | `session_discard{phase:"check"}` (or `error`) | `done` | +| `context pull` (linked, or `--project`) | `folder_linked` (with `--project`), else `pull_test` or `error`; a project with no tests sends `done` alone | `done` (`shared-projects.md` §7a) | +| `cover` | `coverage` | `done` | +| `testmd run <designed test>`, stopped on its Test Manager case | `stream_start` | `error` (no `test_md_done`) | +| `testrun run`, refused | `stream_start`; then `error` before any plan for `--from-context` in a linked folder; else `testrun_plan` | `PROJECT_MISMATCH`: `error`, the last line outside a linked folder. Another invalid plan, or no tests: no `error`, the stream ends at `testrun_plan` (read the exit code). **In a linked folder only**, from the release that carries the fix, every one of these (`PROJECT_MISMATCH` too) ends `error` then `testrun_done{overall_status:"refused"}` (the `--from-context` refusal still ends on its `error`); handle both shapes. Everywhere else these shapes don't change | + +Outputs that are **not** the envelope and have **no `done`** (read the exit code): + +| Command | Stdout on success | +|---|---| +| `projects list --agent` | `{"id","name"}` per project, then `{"_meta":"page","limit","offset","returned","has_more"}`. Errors are plain text on stderr, with no code | +| `projects create "<name>" --agent` | one line `{"id","name"}`. Errors on stderr. It does not link the folder | +| `context list --mode agent` | one bare object per row: `{id, label, title, trust, fresh, tests_fresh?, next?}` (`trust: "derived"` = proposed, not approved yet; `"trusted"` = approved: say *proposed* or *approved*, never the value); `--undesigned` adds `designed: false`; nothing at all when nothing matches | +| `context sessions --mode agent` | one line holding a JSON array of `{sid, verb, subject, title, state, added, updated?, last_activity_at, owner}` | +| `install skill` (stdin not a terminal) | one line `{"type":"skill_install_result","status":"installed"\|"already-installed"\|"failed","version","installed_targets","error?"}`; exit 1 only on `failed` | + +Refusals of `context list` and `context sessions` do use the envelope: `error` + `done`. + +Dispatch on `type`, never on position. **The vocabulary is open:** tolerate unknown event types and unknown fields. A stream that should end in `done` and doesn't means the process died: check `kane-cli context sessions --mode agent` before retrying anything paid. + +## 3. Event catalogue + +"Tell the person" is plain words. Never show event or field names. + +### 3.1 Linking and loading (every assurance verb) + +| Event | When | Fields that matter | Tell the person | +|---|---|---|---| +| `folder_linked` | `--project <id>` linked the folder in this run; the first line | `project_id`, `project` | "This folder is now linked to <project>; what you save goes there." If the run then ends `PERMISSION_REQUIRED` or `ASSURANCE_OFF`, or a `context pull --project` is refused at the start or can't read the project, the link was removed again | +| `folder_local` | `--local` gave the folder its own store; `extract` then ends `done` exit 0 (ingest goes on locally) | `gitignore`: `added`, `failed`, null | "This folder keeps its own local store." On `failed`: `.context/` wasn't added to `.gitignore` | +| `nothing_new` | a plain `extract` found every source already read; no session opened; exit 0 | `reason`: `every_source_read`, or `sources_in_a_session` with `open_sessions[{sid, owner, state, added}]`; `next` | "Nothing new to extract" / "every source is already in an open session"; offer `next` | +| `remote_connect` | `step:"connect"`, then `step:"open_session"` once the session exists | `sid`, `project` (on `open_session`) | "Connecting to <project>." Keep `sid` | +| `remote_load` | the project's history loads | `part` (`project`, `project_ready`), `percent` (may be null) | fold into one progress line | +| `remote_fetch` | an address or item is fetched on this machine, before upload | `ref`, `provider` (`web`, `jira`, `confluence`, `linear`, `linear-doc`) | "Reading <ref>" | +| `remote_upload` | a file goes up before the session opens: 0%, per chunk, 100% | `file`, `percent` | one line per file | +| `remote_retry` | waiting on the shared project | `attempt`, `next_in_s`, `give_up_in_s`, `cause` (`unreachable`, `not_ready`, `timeout`) | "The shared project isn't answering; retrying." Let it run | +| `remote_overlap` | a new session opened while others are open on the same subject | `sessions[{sid, owner: you\|someone_else\|null, state, added, next?}]` | name the other sessions; offer the resume `next` on the person's own; this run goes on | +| `run_start` | the run starts | `session` = the shared project's session id (not a support id); no log path | — | +| `warning` | a non-fatal condition | `message`, `code?` | relay it | + +### 3.2 Ingest, extract, review, evolve + +| Event | When | Fields that matter | Tell the person | +|---|---|---|---| +| `ingested` | each item lands in the session | `source_id`, `status` (`created`, `versioned`, `unchanged`), `behind?`, `unread?`, `media?`, `duplicate_of?`, `origin?`, `warnings?`; no `cid` | "Added <file> to this session" / "a new version" / "already in <project>, unchanged" | +| `commit` | the agent's items land in the **session** | counts, `minted[]` | "N use-cases proposed in this session, not saved yet" | +| `held` | items held back | `reason`: `duplicate_check_reduced`, `review` | "N items are held for review" | +| `usage` | each agent turn | `credits`, `total_credits` | report the final total | +| `review_cards` | a new version's review holds its cards, before a `review_pending` pause | `source`, `cards[{key, kind, use_case?, title, change, diff?, tests?}]` | show each change; ask approve, reject or defer | +| `card_verdict` / `card_refused` | an `--answer <key>=<verdict>` landed / didn't | `key`, `verdict`, `line`, `offer?{kind, key, title}` / `code`, `refusal` | relay `line` or `refusal`. `offer.kind: "evolve"` runs at once (`child_start` … `child_end`); `offer.kind: "design"` does **not** run: after the save, offer to design that use-case's tests (`kane-cli design tests --use-case <uc> --mode agent`, its id from `kane-cli context list --mode agent`) | +| `evolve_paused` | evolve asks once per use-case; exit 3 | `pending_questions[]` (id `evolve:<uc>`), `next` (update all, leave all out) | present the blast radius and ask (`references/assurance.md` §11) | +| `child_start` / `child_end` | a design run inside a review or an evolve | `child`, `subject` / `ok`, `exit_code`, `next?` | "Designing tests for <uc>… done" / "didn't finish" | +| `evolve_skipped` | a use-case evolve leaves alone | `use_case`, `reason` (`no-design`, `source_changed`, `up_to_date`, `archived`, `unreviewed`, `held`, `declined`), `held_by?`, `next?`, `remedy?` | the reason in words; offer `next` | +| `session_complete` | the run finished; **nothing is saved yet** | `sid`, `next` (save, then discard), `no_start_page?[{test}]` | "Done. Save it to <project>?" Name tests with no start page. No `next` = it added nothing and was discarded. On a resume of a finished run whose ending was never recorded, `next` is the save alone | +| `session_ended` | the session was saved or discarded in another terminal; exit 0 | `sid`, `reason`: `saved_elsewhere`, `discarded_elsewhere`, `ended_elsewhere` | say where it went; nothing to do | +| `error` | a refusal or failure | `code?`, `message`, `remedy?`, `next?`, `held_by?`, `not_added?[]`, `final?` | §6 | +| `done` | always last | `status`, `exit_code`, `next?`, `not_updated?` (evolve) | §4, §5 | + +### 3.3 `session_paused` (exit 3) + +Every pause has `sid`, `expires_at: null` and `pending_questions[]` (empty unless questions wait); most carry `resume` (the exact command). Branch on `reason`: + +| `reason` | Means | `next` | Do | +|---|---|---|---| +| none, questions pending | the agent asks the person | resume; resume with `--answer <qid>=<n>` | the pause loop (`references/assurance.md` §2) | +| none, no questions | a forced stop (second Ctrl+C), or a stop during an evolve | resume | resume when the person wants | +| `held_back` | `--trust hold` held items | — | tell the person; nothing promises they come back on a resume | +| `connection_lost` | the connection dropped, or the ending couldn't be recorded | resume | resume once the network is back | +| `not_accepted` | the shared project refused a step | resume | resume once; then ask for help | +| `computer_slept` | this machine slept over 2 minutes | none | run `resume` when the person wants | +| `resumed_elsewhere` | the session was resumed in another terminal | none | leave it; it goes on there | +| `review_pending` | review cards wait (`pending_cards[]`) | resume with `--answer <key>=approve` … | ask per card: `approve`, `reject` or `defer` | +| `review_decided` | every card was decided, then a stop | resume | resume to finish the review | +| several sources read (`not_named?[]`, no `resume`) | an extract that read 2+ sources paused | discard; start again | it can't be resumed or saved: tell the person, discard on their word, then extract one source at a time | + +### 3.4 Save and discard + +| Event | When | Fields that matter | Tell the person | +|---|---|---|---| +| `session_save` `check` | first: what the save would land | `sid`, `project`, `adds{use_cases, acceptance_criteria, scenarios, tests, open_questions, sources, style_notes}`, `changes?`/`approves?`/`archives?`/`removes?`, `held_changes?`, `replaces?`, `marks_up_to_date?`, `other_work?`, `not_updated?[]` | "Saving adds N use-cases and M tests to <project>." Without `--yes` the command stops here (`CONFIRM_REQUIRED`): show this preview and get the person's word | +| `session_save` `saving` | the save is sent | `sid` | — | +| `session_save` `saved` | landed | `saved` (records landed; `0` = nothing to save, and the session was discarded), `test_manager`, `test_files`, `next?` (evolve what is still out of date) | "Saved to <project>", then the two blocks below | +| `session_save` `behind` | a later save changed what this session touched; exit 2, no `error` | `conflict?{kind, id, title}` or `{kind:"link", from, to}`, `next` | "Another save changed <title>; nothing was saved." Offer `next` | +| `session_save` `failed` | the project refused the save; exit 1, no `error` | `reason:"not_accepted"`, `code?` (`SCOPE_RECORD_REFUSED`) | "The project refused the save." Don't retry; ask for help | +| `session_discard` | `check` (the same counts), then `discarded` | `sid`, `adds?` | "Discarded; nothing was saved." An already-discarded session gets `discarded` alone (exit 0) | + +`test_manager` (when the project's Test Manager takes the cases): `total`, `created`, `updated`, `already_there`, `waiting` with `waiting_on[{scenario, title, reason}]` (`changed_both_sides`, `not_pushable`, `archived`, `busy`, `not_accepted`), `next_run[{scenario, title}]` (written by the next `testmd run`), `not_covered?[]`, `error?`, `support_id?`, `next?`. Tell: "N test cases created and M updated in Test Manager"; name what waits and why. + +`test_files` (design, reconcile and evolve sessions): `written[]` (paths in `.testmuai/tests/`), `refused[{test, title, reason, file?}]` (`reason` is prose; a file edited by hand is kept), `replaced?[]` (files of tests this save replaced: they stay in the folder, won't run any more, and a bare `testrun run` still picks them up; tell the person, and offer to move them out of `.testmuai/tests/`), `variables_needed?{file, variables}`, `no_start_page?[]`, `next?`; or `{written: [], refused: [], error: "TEST_FILES_NOT_WRITTEN", next}`. Test files are written **only** by the save, only in the folder where it runs. Tell: "Wrote N test files." Fill `variables_needed` before any run. + +### 3.5 Design + +| Event | When | Fields that matter | Tell the person | +|---|---|---|---| +| `gate_refused` | design entry refused: unknown, removed or not-a-use-case ref; or the use-case already has tests, or changed since | `message`, `final: true`, `next?[{cmd, why, title}]`; **no `code`** | relay `message`; offer `next` (`Redesign` = `--force`) | +| `error` `UC_UNREVIEWED` / `UC_ARCHIVED` | the use-case is proposed / archived | `message`, `final`, `next?` | §6 | +| `commit`, `usage` | as §3.2 | | "Designed N tests in this session, not saved yet" | +| `variables_declared` | the tests need variables | `file`, `variables[]` | a to-do; the stubs are this folder's only | +| `note`, `panel_resolved` | a resume with `--answer` | `text` / `id` | confirmation only | +| `session_paused`, `session_complete` | as §3.2–3.3 | | | + +### 3.6 Cover + +| Event | When | Fields that matter | Tell the person | +|---|---|---|---| +| `coverage` | `cover --mode agent` | `source: "project"`, `pct`, `usecases`, `criteria`, `perUc`, `perAc`, `flags`, `recent_run?{may_not_count_yet: true, up_to_minutes: 5}` | "Coverage is N% across <project>'s K use-cases." With `recent_run`: "a run from the last 5 minutes may not count yet" | +| `coverage` (computing) | the project is still working it out; exit 0 | `{source: "project", state: "computing"}` | run `next` (`kane-cli cover --mode agent`) in a moment | +| `done` | last | `next?`: designs to make, `testrun run` while nothing has run; `kane-cli context pull --mode agent` first (and no run) when the project has designs and this folder has no designed test files yet | offer them, adding the agent flag (§4) | + +### 3.7 `testmd run` and `testrun run` in a linked folder + +A file directly in `.testmuai/tests/` with an `assurance:` header runs on its scenario's Test Manager case. Its first run authors (about 30–85 credits per test, approximate); later runs replay. + +| Event | When | Fields that matter | Tell the person | +|---|---|---|---| +| `error` (testmd) | the case lookup or attach stopped the run | `code`, `message`, `remedy`, `support_id?`; no `next`, no `test_md_done` | §6, runs of designed tests | +| `error` `TEST_RUN_NOT_SAVED` (testmd) | before `test_md_done`; the verdict stands | as above | report the verdict; exit 1 even when steps passed | +| `testrun_plan` | the plan | `members[{path, test_id, failure?}]`, `valid` | on 0.10.0 an invalid plan other than `PROJECT_MISMATCH`, or no tests at all (`members: []`), has **no `error`**: read each `failure` | +| `error` `NO_TESTS` | *(linked folder only, from the release that carries it)* no `*_test.md` selected | `message` "no tests here yet", `next` | `next` is `kane-cli context pull --mode agent`. A filter that matches nothing: the old message and no `next`. Outside a linked folder there is no such event: the stream ends at `testrun_plan` with `members: []` and exit 2 | +| `error` `PLAN_INVALID` | *(linked folder only, from the release that carries it)* an invalid plan other than `PROJECT_MISMATCH` | `offenders[{path, failure}]`, no `next` | fix those tests or leave them out. Outside a linked folder: read `testrun_plan.members[].failure` | +| `error` `PROJECT_MISMATCH` | after `testrun_plan` | `projects[{project_id, name, this_folder, tests}]`, `remedy`, `next` | §6 | +| `testrun_member_end` (`testrun_authored_member_end` for a test's first run) | each member | `status`, `failure?{message}`, `log_path` | a member stopped on its case shows only `failure.message`; its `error` (with `code` and `remedy`) is in the member's log at `log_path`, or inside `testrun_member_event` with `--stream-members` | +| `coverage` | after the evidence ingest | `status` (`written`, `none`, `skipped`, `failed`); `written`: `computed_by: "project"`, `pct`; `none`: `reason` (`on_grid`, `project_no_answer`) | "This run reached N use-cases." `none`: the project didn't answer in time; the run is fine | +| `testrun_done` | last (local run) | `overall_status` (`passed`, `failed`, `cancelled`; `refused` after a refusal, in a linked folder only, from the release that carries it), `coverage?` | the verdict | +| `remote_error` (`--remote`) | one per member that stops before dispatch | `code`, `detail` (`<path>: <message>`), `remedy?`, `support_id?`; no `next` | nothing was sent to HyperExecute; `testrun_done{overall_status:"failed"}` and `remote_done{status:"failed", exit:2}` follow | + +## 4. `next` + +Shape: `next: [{cmd, why}]`, one runnable command per item (design walls add `title`). Tolerate a plain string item. + +Where: `error`, `session_complete`, `session_paused`, `nothing_new`, `remote_overlap.sessions[]`, `gate_refused`, `evolve_paused`, `evolve_skipped`, `child_end`, `session_save` (`saved`, `behind`, and inside `test_manager` and `test_files`), a `context list` row whose tests are out of date, and `done` (it repeats a pause's `next`). + +- **Agent flag.** Each `cmd` already carries `--mode agent` (`--agent` for `testrun run`, `testmd run`, `projects`); end a `testrun run` line in `< /dev/null`. If a `cmd` has no agent flag, add the one that command takes (§1) before running it. +- **One at a time.** Run each `cmd` as its own command. Never join them with `&&`. +- **Placeholders.** A `cmd` holding `<files>` (from `nothing_new`, a `behind` save) is a template: put in the person's real paths. +- **Read it before running it.** Some spend credits (`context extract`, `context ingest`, `design tests`, `maintain evolve`, `testmd run`, `testrun run`). Some need the person's word: `--allow-unreviewed` (designs from a use-case nobody approved), `--force` (replaces a use-case's tests), `context sessions save --yes` (publishes to the team's project and Test Manager), `context sessions discard --yes` (throws the session's work away). A `next` built from a partial `--answer` set carries the answers already given. +- **No `next`?** Read `remedy`: prose that may name a command. Add the agent flag that command takes (§1) before running it. Refusals at a folder's entry (linking, sign-in, access, `REMOTE_UNSUPPORTED`) carry `remedy` only; §6 gives the command for each. For `projects`, read stderr. + +## 5. Exit codes + +`done.status` follows the exit (`0` complete, `2` refused, `3` paused, `130` interrupted, any other `error`), with two exceptions: `context ingest` and `maintain reconcile` refusals can end `done{status:"refused", exit_code:1}`, and a failed read of `context list`, `context sessions` or `cover` ends `done{status:"error", exit_code:1}`. Branch on `exit_code`. + +| Command | 0 | 1 | 2 | 3 | 130 | +|---|---|---|---|---|---| +| `context extract` | `session_complete`, `nothing_new`, `session_ended` | the run stopped (work kept in the session), `SESSION_OPEN_FAILED`, `SESSION_NOT_ENDED`, agent failures | entry, sign-in, access, resume and `--answer` refusals | paused | stopped before a session opened | +| `context ingest` | as extract | as extract, plus `UPLOAD_STALLED`, `UPLOAD_NOT_TAKEN`, `RESULT_REFUSED`, `FETCH_FAILED`, conversion failures | file, address, conversion and source-id refusals | paused | stopped during a fetch or upload | +| `design tests` | complete | as extract | `UC_*`, `gate_refused`, `USAGE`, `STRUCTURED_*` | paused | as extract | +| `maintain evolve` | complete | some use-cases not updated (`done.not_updated`, `next`) | entry refusals, `USAGE`, `MODE_USAGE` | paused | as extract | +| `maintain reconcile` | complete; nothing to review | `DESIGN_CHILD_FAILED` | `REMOTE_UNSUPPORTED`, `SOURCE_*`, `MODE_USAGE`, `STRUCTURED_TARGET_UNKNOWN` | `review_pending` | as ingest | +| `context sessions save` | saved; nothing to save | `failed` phase, `SAVE_FAILED`, `ASSURANCE_OFF` | `CONFIRM_REQUIRED`, `behind` phase, `SESSION_*`, `CONNECTION_FAILED`, sign-in | — | — | +| `context sessions discard` | discarded; already discarded | `DISCARD_FAILED` | every other refusal | — | — | +| `context sessions`, `context list` | the array / the rows | `LIST_FAILED` | `REMOTE_UNSUPPORTED`, `ASSURANCE_OFF`, sign-in, `USAGE`, `MODE_USAGE` | — | — | +| `cover` | `coverage` (also computing) | `COVERAGE_FAILED` | `ASSURANCE_OFF`, sign-in, `REMOTE_UNSUPPORTED` (`cover gaps`) | — | — | +| `testmd run` (designed test) | passed | failed, `TEST_MANAGER_OUTDATED`, `TEST_CASE_ATTACH_FAILED`, `TEST_RUN_NOT_SAVED` | case and lookup refusals | — | — | +| `testrun run` | all passed | a member failed, including one stopped on its case | `REMOTE_UNSUPPORTED`, `PROJECT_MISMATCH`, invalid plans, every `--remote` stop before dispatch | cancelled | — | + +On the assurance verbs, `3` is a **pause** (resumable), not a failure: run the pause loop. `130` is a forced stop: resumable only if a `session_paused` arrived. On `testrun run`, `3` means cancelled. + +## 6. Refusal codes + +`Do` says what to tell or ask the person and the exact command, so you can go on when the event has no `next`. "Ask the person:" marks a step only they (or an admin) can take. Never rerun a paid command (extract, ingest, design, evolve, testmd, testrun) blindly: an exit 1 after turns ran has spent credits. `<the command>` = the refused command as typed. + +| Code | Exit | When | Do | +|---|---|---|---| +| **Linking and the folder** | | | | +| `FOLDER_USAGE` | 2 | `--project` and `--local` together | ask the person: share the work (`<the command>` without `--local`) or keep it here (`<the command>` without `--project`) | +| `FOLDER_USAGE` | 2 | `--project` or `--local` with `--resume` | `<the command>` without the folder flag; a session stays where it started | +| `FOLDER_CHOSEN` | 2 | `--project` in a folder with its own store; `--project Q` in a folder linked to P; `--local` in a linked folder | `<the command>` without `--project` / `--local` (it works in the folder's own store or project). To change that, ask the person: move `.context/` out, or remove `.testmuai/assurance.json` | +| `BOTH_STORES` | 2 | the folder has `.context/` and a link | ask the person: work locally (remove `.testmuai/assurance.json`) or in the project (move `.context/` out of the folder) | +| `PROJECT_FILE_UNREADABLE` | 2 | `.testmuai/assurance.json` can't be read | ask the person: remove `.testmuai/assurance.json`. Then `kane-cli context extract --project <id> --mode agent` | +| `PROJECT_FILE_UNWRITABLE` | 2 | the link couldn't be written | ask the person: make the folder writable, or move a file named `.testmuai` out of it | +| `PROJECT_NO_ACCESS` | 2 | `--project <id>` names no project of theirs | `kane-cli projects list --search <name> --agent`; confirm the id with the person; run `<the command>` with it | +| `PROJECT_NO_ACCESS` | 2 | no access to the folder's project | ask the person: a Test Manager admin grants their account access | +| `PERMISSION_REQUIRED` | 2 | the project's first session needs permission to create test cases | ask the person: an admin grants it, or someone who has it starts the first session. A link made in this run was removed | +| `ASSURANCE_OFF` | 2 (1 from `sessions save`) | assurance isn't available for the project | ask the person; no command | +| `REMOTE_UNSUPPORTED` | 2 | `context review`, `context explain` | `kane-cli context list --mode agent` (the use-cases, their status, scenarios and tests) | +| `REMOTE_UNSUPPORTED` | 2 | `cover gaps` | `kane-cli cover --mode agent` | +| `REMOTE_UNSUPPORTED` | 2 | `context sessions show`, `context sessions clean` | `kane-cli context sessions --mode agent` | +| `REMOTE_UNSUPPORTED` | 2 | `--mode ci` on extract or design (a stderr line) | `<the command>` with `--mode agent` | +| `REMOTE_UNSUPPORTED` | 2 | `context ingest --plan` or `--mode override`; `context list --all`, or a `--type` other than `source`, `usecase`, `scenario`, `test` | `<the command>` without that flag, with `--mode agent` | +| `REMOTE_UNSUPPORTED` | 2 | `maintain reconcile --plan`, `--apply`, `--mode ci`, or no `--source-id`; `--from <address>` | `kane-cli maintain reconcile --from <file> --source-id <id> --mode agent`; for an address, `kane-cli context ingest <address> --mode agent` | +| `REMOTE_UNSUPPORTED` | 2 | `context sync`, `push`, `clone`, `fsck`, `view`, `retire`, `revert`, `rebuild`, `name`, `design explain` (and `context pull` where its help has no `--project`: `shared-projects.md` §7b) | no command in a linked folder; tell the person these work on a folder's own store. The project's tests come with `kane-cli context pull --mode agent` (`shared-projects.md` §7) | +| `PULL_LOCATION_LINKED` | 2 | `context pull` with a location name, `--rebase`, `--yes` or `--answer` in a linked folder | `kane-cli context pull --mode agent` | +| `REMOTE_UNSUPPORTED` | 2 | `testrun run --from-context` | run `next`, or `kane-cli testrun run <test files> --agent` | +| **Sign-in and access** | | | | +| `CREDENTIALS_MISSING` | 2 | nobody signed in | start sign-in (`kane-cli login --oauth`, `references/ready-check.md`), then `<the command>` | +| `CREDENTIALS_MISSING` | 2 | "couldn't confirm your login for shared projects" (a browser sign-in with no access key) | `<the command>` once more; if it refuses again, ask the person: run `kane-cli login --username <user> --access-key <key>` in their own terminal | +| `CREDENTIALS_REJECTED` | 2 | the shared project rejected the credentials | sign in again (`kane-cli login --oauth`), then `<the command>` | +| `CONNECTION_FAILED` | 2 | couldn't connect (or for 2 minutes); nothing was started | check the network; `<the command>` once more | +| `PREFLIGHT` | 2 | kane-cli's check before a session opens failed: unreachable, sign-in unchecked or rejected, or "assurance sessions aren't available here yet" | unreachable: `<the command>` once more later. Sign-in: `kane-cli login --oauth`. Not available: stop and pass on the support id in `remedy` | +| `VERSION_MISMATCH` | 2 | this kane-cli is too old, or the shared project doesn't support it yet | too old: ask the person to update (`npm install -g @testmuai/kane-cli@latest`). Not supported yet: stop and pass on the support id in `remedy` | +| `PROJECT_BUSY` | 2 | a live session holds the project, use-case or source: `held_by{sid, verb, use_case, mine, status}` | no `next`. When `remedy` names a resume (`kane-cli context extract --resume <sid> --mode agent`, or `kane-cli design tests --resume <sid> --mode agent`; only for the person's own session, `mine: true`), running it takes the session over and stops it in the other terminal, so ask the person first; otherwise wait for that run to finish. When `remedy` only says to wait (a resume refused this way, or a design while the person's own extract holds the project), wait. Someone else's: tell the person a teammate's session holds it; wait; `<the command>` later | +| **Sessions** | | | | +| `SESSION_NOT_FOUND` | 2 | no session with that id is theirs (a colleague's can't be resumed or saved) | `kane-cli context sessions --mode agent`; use a `sid` whose `owner` is `you` | +| `SESSION_FINISHED` | 2 | resume of a finished session | on the person's word: `kane-cli context sessions save <sid> --yes --mode agent` | +| `SESSION_NOT_FINISHED` | 2 | save of an unfinished session | resume first: `kane-cli context extract --resume <sid> --mode agent` (`design-…`, `evolve-…` ids: `kane-cli design tests --resume <sid> --mode agent`); save after `session_complete` | +| `SESSION_NOT_ENDED` | 1 | the run finished but the project didn't take its last step; the work is safe | run `next` (the resume command above): it records the ending; then save | +| `SESSION_FAILED` | 2 | the session ended with an error; it can't be resumed or saved | start again: `kane-cli context extract --mode agent`, or `kane-cli design tests --use-case <uc> --mode agent` | +| `CHECKPOINT_PROTOCOL`, `SESSION_KEPT_NOTHING`, `SESSION_MISSING` | 2 | a resume can't read the session's last step | tell the person it can't be resumed; start a new one: `kane-cli context extract --mode agent` or `kane-cli design tests --use-case <uc> --mode agent` | +| `SESSION_OPEN_FAILED` | 1 | the session couldn't be opened | `<the command>` once more; then ask for help | +| no code: "stopped before it finished" | 1 | the run stopped; the work so far is kept in the session | resume with the `sid` from `remote_connect`: `kane-cli context extract --resume <sid> --mode agent` (design: `kane-cli design tests --resume <sid> --mode agent`). Don't start over | +| `CONFIRM_REQUIRED` | 2 | save or discard without `--yes` | the `check` line before it is the preview: show it; on the person's word, `kane-cli context sessions save <sid> --yes --mode agent` (or `discard`) | +| `SESSION_ELSEWHERE` | 2 | discard while the session runs in another terminal, or is held | wait (up to 3 minutes) or stop the other run; then `kane-cli context sessions discard <sid> --yes --mode agent` | +| `ALREADY_SAVED`, `ALREADY_ENDED` | 2 | discard of a saved or ended session | nothing to discard; tell the person | +| `DISCARD_FAILED` | 1 | the discard failed | stop; ask for help | +| `LIST_FAILED` | 1 | the session list or `context list` couldn't load | `<the command>` once more; then ask for help | +| `STRUCTURED_FLAGS_USAGE` | 2 | `--answer` without `--resume <sid>` or `--mode agent`, or unreadable | `kane-cli context extract --resume <sid> --mode agent --answer <id>=<value>` (design: `kane-cli design tests --resume <sid> …`) | +| `STRUCTURED_TARGET_UNKNOWN` | 2 | `--answer` names no pending question or card, isn't a verdict, or repeats a card | use the ids from `pending_questions` or `review_cards` | +| `USAGE` | 2 | a parse error; save or discard with no `<sid>`; evolve with both or neither of `<ref>` and `--from-stale`; design with no `--use-case` | fix the command (`kane-cli <command> --help`); never resend it unchanged | +| `MODE_USAGE` | 2 | `--mode` isn't one the command takes | `<the command>` with `--mode agent` | +| **Extract and ingest inputs** | | | | +| `NO_PROPOSAL` | 1 | the agent returned no use-cases | retry `kane-cli context extract --mode agent` once (it reads the source again and costs credits again). If it happens twice, stop, and give the person the command and the message | +| `AS_SINGLE_SOURCE` | 2 | `--as` with several refs | run `next`, or `kane-cli context ingest <ref> --as <id> --mode agent` once per source | +| `SAME_SOURCE_ID` | 2 | two refs would share one source id | run `next`; with none, ask the person for distinct ids and run `kane-cli context ingest <ref> --as <id> --mode agent` per ref | +| `NEW_VERSION_ALONE` | 2 | a new version of a held source among other refs; nothing added | `kane-cli context ingest <the other refs> --mode agent`, then `kane-cli context ingest <new version> --mode agent` alone (it opens a review) | +| `SITE_MOVED`, `KIND_MISMATCH`, `WEB_URL_CHANGED`, `UUID_MOVED` | 2 | the fetched item can't become a new version of the source id it would replace | ask the person which they mean; run that `next`, or `kane-cli context ingest <address> --as <new id> --mode agent` for a source of its own | +| `UNSUPPORTED_URL` | 2 | the ref is neither a file nor a web address | ask the person for a file path or a full `https://` address | +| `FILE_NOT_FOUND`, `FILE_UNREADABLE`, `FILE_TOO_LARGE`, `UNSUPPORTED_MEDIA`, `MEDIA_SNIFF_MISMATCH`, `BAD_SOURCE_ID` | 2 | checked before anything was sent | fix the path or file (relay the message); `BAD_SOURCE_ID`: pass a valid `--as <id>` | +| `PDF_ENCRYPTED`, `PDF_NO_TEXT_LAYER`, `PDF_UNREADABLE`, `DOCX_UNREADABLE`, `DOCX_NOT_WORDPROCESSING`, `DOCX_NO_TEXT`, `DOCX_ENCRYPTED_OR_LEGACY`, `DOCX_PART_TOO_LARGE`, `DOCX_TOO_MANY_PARTS`, `DOCX_BOMB`, `PROJECTION_TOO_LARGE` | 2 | the file can't be converted on this machine | relay the message; ask the person for an unlocked, text-bearing or smaller copy | +| `SOURCE_ID_OWNED` | 2 | that source id is taken | ask the person for another id; `kane-cli context ingest <ref> --as <id> --mode agent` | +| `INGEST_FAILED`, other conversion failures | 1 | the conversion failed | report it; ask for help | +| `INVALID_URL`, `ADDRESS_NOT_ALLOWED`, `NOT_A_PAGE`, `PAGE_NOT_FOUND`, `PAGE_BLOCKED`, `PAGE_PAYWALLED`, `PAGE_TOO_LARGE`, `PAGE_TIMEOUT`, `PAGES_UNAVAILABLE`, `PAGES_NOT_AVAILABLE`, `ITEM_UNAVAILABLE`, `LINEAR_PAGE`, `LINEAR_LINK_MALFORMED`, `ATTACHMENT_TOO_LARGE` | 2 | the address or item can't be read | relay the message; ask the person for another address or a saved file | +| `NO_CONNECTION`, `NO_ACCESS`, `NO_ISSUE_ACCESS`, `NO_DOCUMENT_ACCESS`, `NO_CONFLUENCE_ACCESS`, `NO_COMMENTS_ACCESS`, `PROJECT_NOT_SELECTED`, `INTEGRATION_SETUP`, `INTEGRATION_UNAVAILABLE` | 2 | the Jira, Confluence or Linear connection is missing or can't see the item | ask the person: connect the integration or grant access (relay the message) | +| `INTEGRATION_BUSY` | 2 | the integration is busy | wait; `<the command>` once more | +| `OUT_OF_CREDITS` | 2 | no credits left for the fetch | tell the person (`kane-cli balance`); stop | +| `FETCH_FAILED` | 1 | the fetch failed | `<the command>` once more; then ask for help | +| `PLUGIN_NOT_CLAIMED`, `PLUGIN_SOURCE_ID_INVALID`, `PLUGIN_MISSING` | 2 | no source plugin takes the ref, or its id is invalid | ask the person: check the ref's form and that the plugin is installed | +| `PLUGIN_HOOK_FAILED`, `PLUGIN_HOOK_TIMEOUT` | 1 | the source plugin failed | report it; ask for help | +| `UPLOAD_STALLED`, `UPLOAD_EXPIRED` | 1 / 2 | the upload stopped, or its window ended | `<the command>` once more | +| `UPLOAD_NOT_TAKEN`, `RESULT_REFUSED` | 1 | the shared project didn't take the upload or its result | stop; ask for help | +| `LINKS_NOT_TAKEN` | 2 | the shared project can't take addresses | ask the person for a saved file of the page; `kane-cli context ingest <file> --mode agent` | +| `SOURCE_UNKNOWN`, `SOURCE_REMOVED` | 2 | reconcile: the project doesn't hold that `--source-id`, or removed it | `kane-cli context list --type source --mode agent`; a new source: `kane-cli context ingest <file> --mode agent` | +| `SOURCE_ORIGIN_UNKNOWN`, `SOURCE_REF_UNROUTED`, `SOURCE_ID_MISMATCH` | 2 | reconcile of a plugin source that doesn't line up | relay the message; check the `--source-id` with the person (`kane-cli context list --type source --mode agent`) | +| on `card_refused`: `CONCURRENT_APPEND`, `TARGET_RETIRED`, `SOURCE_MOVED`, `SOURCE_UNRESOLVED`, `ARCHIVE_UNRESOLVED`, `ALREADY_VERDICTED`, `CARD_GONE`, `HOLD_GONE` | — | a review verdict didn't land | relay `refusal`; `kane-cli context extract --resume <sid> --mode agent` shows the cards still open | +| `DESIGN_CHILD_FAILED` | 1 | an approved change's design run failed | later: `kane-cli design tests --use-case <uc> --mode agent` | +| `AGENT_FAILED`, `AGENT_CRASHED`, `REPAIR_FAILED`, `PROPOSAL_INVALID`, `COMMIT_FAILED` | 1 | the agent's run failed | report it; credits were spent; check `kane-cli context sessions --mode agent` before any retry | +| `TRUST_USAGE` | 2 | `--trust hold` in a terminal run | `<the command>` without `--trust`, with `--mode agent` | +| **Design** | | | | +| `UC_UNREVIEWED` | 2 | the use-case is still proposed; nobody approved it | show it (`kane-cli context list --type usecase --mode agent`); on the person's word, `kane-cli design tests --use-case <uc> --allow-unreviewed --mode agent` (one that already has tests also needs `--force`: the `gate_refused` row below). A person approves at the end of a terminal session | +| `UC_ARCHIVED` | 2 | the use-case is archived | tell the person; pick another use-case | +| `gate_refused` (no code) | 2 | unknown, removed or not-a-use-case ref | pick from the ids the message lists (`kane-cli context list --type usecase --mode agent`) | +| `gate_refused` (no code) | 2 | the use-case already has tests, or changed since they were designed | ask the person: redesign (`kane-cli design tests --use-case <uc> --force --mode agent`, replaces its tests) or update (`kane-cli maintain evolve <uc> --mode agent`, only what changed). A use-case that is also still **proposed** needs both flags on a redesign: `kane-cli design tests --use-case <uc> --force --allow-unreviewed --mode agent` (each alone refuses with the other wall); evolve refuses it (`UC_UNREVIEWED`), so use the redesign | +| `NOTHING_CHANGED` | 2 | evolve of a use-case unchanged since its tests were designed | ask the person why: `kane-cli maintain evolve <uc> --because "<reason>" --mode agent`; or run `next` | +| `NO_TESTS_YET` (`evolve_skipped` `no-design`) | 2 | evolve of a use-case with no tests | `kane-cli design tests --use-case <uc> --mode agent` | +| `evolve_skipped` `source_changed` | 2 | only its source changed | run `next`; a file source has none: `kane-cli maintain reconcile --from <file> --source-id <id> --mode agent` | +| **Save** | | | | +| `session_save` `phase: "behind"` (no `error` event; a terminal shows `PROJECT_CHANGED`) | 2 | another save changed what this session touched | tell the person what (`conflict`); on their word run one `next` (start again from the latest) | +| `session_save` `phase: "failed"` (`reason`, maybe `code: "SCOPE_RECORD_REFUSED"`; a terminal shows `SAVE_FAILED` with a support id) | 1 | the project refused the save | saving again won't help: stop, and give the person the `reason` and the session id | +| `CONNECTION_FAILED` | 2 | the save's answer was lost after sending | `<the command>` once more: nothing is saved twice | +| `TEST_MANAGER_TIMED_OUT`, `TEST_MANAGER_UNREACHABLE`, `TEST_MANAGER_FAILED` (`test_manager.error`) | 0 | the save landed; Test Manager wasn't fully updated | run `test_manager.next`, or `kane-cli context sessions save <sid> --mode agent` (saving again finishes it) | +| `PROJECT_NO_ACCESS`, `ASSURANCE_OFF` (`test_manager.error`) | 0 | the save landed; Test Manager refused the cases | ask the person: their Test Manager access, or assurance for the project | +| `TEST_FILES_NOT_WRITTEN` (`test_files.error`) | 0 | the save landed; no test files were written here | run `test_files.next`, or `kane-cli context sessions save <sid> --mode agent` | +| `test_files.refused[]` | 0 | a test file wasn't written; `reason` says why: edited by hand and kept, a duplicate, an unreadable or unresolved record, an unwritable folder, a missing import (some have no `file`) | tell the person the reason; follow `test_files.next` when it is there. Only for a file edited by hand: when `test_files.next` offers `kane-cli testmd run <file> --agent`, it sends their edit to its case; otherwise (or instead) they move the file out and save again | +| **Runs of designed tests** | | | | +| `TEST_CASE_MISSING` | 2 | the session that designed the test isn't saved | on the person's word: `kane-cli context sessions save <sid> --yes --mode agent` (`kane-cli context sessions --mode agent` lists the sids) | +| `TEST_CASE_GONE` | 2 | its Test Manager case was deleted or moved | show the person; on their word, `kane-cli context sessions save <sid> --yes --mode agent` with the sid `remedy` names (it makes a new case). A `remedy` with no sid: `kane-cli context sessions --mode agent` to find it | +| `TEST_CASE_ARCHIVED` | 2 | its case is archived in Test Manager | ask the person: restore the case, or leave the test out of the run | +| `TEST_NOT_IN_PROJECT` | 2 | the test isn't in the folder's project, its file is out of date, or the project has no work yet | run it in the folder linked to its project; or, on the person's word, design it again: `kane-cli design tests --use-case <uc> --mode agent` (a proposed use-case also needs `--allow-unreviewed`, one with tests `--force`: the Design rows), then save; a project with no work: `kane-cli context ingest <files> --mode agent` | +| `TEST_MANAGER_OUTDATED` | 1 (2 on `--remote`) | Test Manager can't add this run to the test's case: on a device target (it attaches desktop-browser runs only), or because Test Manager itself needs an update (or made a case of its own) | stop and tell the person. On a device target, offer the desktop browser; otherwise the run can only go again once Test Manager is updated. Pass on the support id | +| `TEST_CASE_ATTACH_FAILED` | 1 | attaching to the case failed | `<the command>` once more later; then pass on the support id | +| `TEST_RUN_NOT_SAVED` | 1 | the run ran but its version wasn't saved on the case | report the verdict as it stands; tell the person Test Manager has no record of this run; pass on the support id | +| `CREDENTIALS_MISSING`, `CREDENTIALS_REJECTED`, `PROJECT_NO_ACCESS`, `ASSURANCE_OFF`, `CONNECTION_FAILED` | 2 | a first run couldn't look up its case | as the sign-in and access rows above; a later run goes on without the lookup | +| `REMOTE_KANE_NOT_PUBLISHED` | 2 | `testrun run --remote` with designed tests from a kane-cli that isn't a published release | ask the person to install a published kane-cli; `remedy` names the flags that pick one | +| **testrun plans** | | | | +| `PROJECT_MISMATCH` | 2 | the selected tests belong to more than one Test Manager project | show `projects[]` (`this_folder: true` marks the folder's project); run `next`, or `kane-cli testrun run <that project's test paths> --agent`. Keep `kane-cli run` experiments in another folder | +| invalid plan (no `error`: outside a linked folder, or on 0.10.0) | 2 | `testrun_plan.valid` is false | read each `members[].failure`; fix those tests or leave them out | +| `PLAN_INVALID` | 2 | *(linked folder only, from the release that carries it)* the same, as an `error` with `offenders[{path, failure}]`, then `testrun_done{refused}` | fix the named tests or leave them out | +| no tests (`members: []`) | 2 | nothing selected (everywhere on 0.10.0; outside a linked folder on every release) | in a linked folder: get the project's tests (`shared-projects.md` §7); elsewhere design them (`kane-cli context list --mode agent`, then `kane-cli design tests --use-case <id> --mode agent`) | +| `NO_TESTS` | 2 | *(linked folder only, from the release that carries it)* "no tests here yet", then `testrun_done{refused}` | run its `next`: `kane-cli context pull --mode agent`. A filter matching nothing has no `next`: change the filter | +| **Coverage** | | | | +| `COVERAGE_FAILED` | 1 | the project's coverage couldn't be read | `kane-cli cover --mode agent` once more in a moment; then ask for help | +| `coverage` computing | 0 | still being worked out | `kane-cli cover --mode agent` in a moment | + +## 7. Support ids + +A support id looks like `as-YYYYMMDDTHHMM-xxxxxxxx`. It names one command's attempt for TestMu AI support. + +| Where | What shows | +|---|---| +| Terminal | under a refusal: `if this keeps failing, share this id with support: <id>` | +| Assurance stream | **not** on `error` events. Only inside `remedy` for `VERSION_MISMATCH` (not supported yet), `PREFLIGHT` (not available here) and `SAVE_FAILED`, and in `session_save.test_manager.support_id` (a test that couldn't be read) | +| Run stream | `testmd run`: `error.support_id`; `testrun run --remote`: `remote_error.support_id`; a member's `error` inside `testrun_member_event` (`--stream-members`) | + +When you ask the person to get help, give them the support id, the command, the code and the message. With no id on the stream, give the rest; never rerun a paid command just to get an id. With `KANE_TELEMETRY=0` the id still shows, but nothing is recorded under it: say so. diff --git a/skill-installer/skills/references/assurance-parsing.md b/skill-installer/skills/references/assurance-parsing.md index 0ab581c..66e4625 100644 --- a/skill-installer/skills/references/assurance-parsing.md +++ b/skill-installer/skills/references/assurance-parsing.md @@ -2,6 +2,11 @@ # Assurance NDJSON — Wire Contract +> In a folder linked to a shared project (kane-cli 0.10.0+) the same commands add their own events (for example +> `folder_linked`, `nothing_new`, `remote_*` progress, `session_save`, `session_discard`, `session_ended`, `pull_test`) +> and refusal codes, and `session_complete` there carries the save's `next`. They are in `agent-contract.md` §3 and §6; +> this page covers the local store. + `kane-cli context extract --mode agent`, `kane-cli context ingest … --mode agent`, and `kane-cli design tests --mode agent` speak a versioned NDJSON vocabulary on stdout — one JSON object per line, envelope `{"type": "<name>", "v": 1, "verb": "extract"|"design", ...}`. On 0.7.2+ the stream is **strict**: stdout carries only NDJSON and stderr stays silent (crash traces excepted); on 0.7.1, prose diagnostics could ride stderr. `maintain reconcile --mode agent` (verb `reconcile`) and `cover`/`cover gaps --mode agent` (verbs `cover`/`gaps`) share the envelope with their own event sets (below). **The vocabulary is open**: new event types and fields may appear in any release — tolerate unknowns, never fail on them. @@ -19,7 +24,7 @@ for each line: else → per-type handling below ``` -A stream that ends **without** `done` means the process crashed — outcome unknown; inspect `context sessions --json` and `context list` before retrying anything paid. One designed exception: `kane-cli context review --mode agent` keeps its `--json` shape — verdict and drift listings are one JSON object per line with no `done` (an empty listing prints nothing, exit `0`); its early refusals — no store, an open rebase, a verdict queue that needs a terminal (`TTY_REQUIRED`) — are typed events followed by `done`; its later refusals exit without `done`. Read its exit code instead. One version-scoped rule for merged-ingest landing failures (bad path, unsupported media, refused URL, a misused `--mode` or `--as`): on 0.7.2+ they ride the stream as `error` (codes `MODE_USAGE`, `AS_SINGLE_SOURCE`, `UNSUPPORTED_URL`, `INGEST_FAILED`) + `done` — the guarantee covers the whole invocation; on 0.7.1 they end with prose + exit `1`/`2` **before any NDJSON begins** — no stream at all is a refusal to fix, not a crash, and the guarantee starts with the extraction stream. +A stream that ends **without** `done` means the process crashed — outcome unknown; inspect `context sessions --json` and `context list` before retrying anything paid. No stream at all, with one line on stderr and exit `2`, is a command-line refusal before anything ran (for example `--json` with `--mode agent`): fix the command, don't treat it as a crash. One designed exception: `kane-cli context review --mode agent` keeps its `--json` shape — verdict and drift listings are one JSON object per line with no `done` (an empty listing prints nothing, exit `0`); its early refusals — no store, an open rebase, a verdict queue that needs a terminal (`TTY_REQUIRED`) — are typed events followed by `done`; its later refusals exit without `done`. Read its exit code instead. One version-scoped rule for merged-ingest landing failures (bad path, unsupported media, refused URL, a misused `--mode` or `--as`): on 0.7.2+ they ride the stream as `error` (codes `MODE_USAGE`, `AS_SINGLE_SOURCE`, `UNSUPPORTED_URL`, `INGEST_FAILED`) + `done` — the guarantee covers the whole invocation; on 0.7.1 they end with prose + exit `1`/`2` **before any NDJSON begins** — no stream at all is a refusal to fix, not a crash, and the guarantee starts with the extraction stream. ## Events (extract / design) @@ -93,12 +98,12 @@ Use `text` + `options[].label` + `recommended_index` + `rationale` to decide or | `child_start` / `child_end` | around the design run, after a proceed | `child: "design"`, `subject` / `ok`, `dur_s`, `exit_code?` | | *(the design run's events)* | between the two, stamped `verb: "design"` | every event of `design tests --mode agent` except its `done`; a paused design session arrives as `session_paused` with `sid`, `resume`, `next[]` | | `evolve_diff` | after a clean design run | `use_case`, `generation` (`before`, `after`), `superseded[]`, `minted[]`, `retained[]` (pairs as `scenario` + `test`), `moves[]` (`ac`, `before[]`, `after[]`) | -| `evolve_skipped` | a use-case not evolved | `use_case`, `reason`: `declined`, or `no-design` in a `--from-stale` batch | +| `evolve_skipped` | a use-case not evolved | `use_case`, `reason`: `declined`, `no-design` in a `--from-stale` batch, or `source_changed` (out of date only because its source changed; `next?` reviews that change with `kane-cli maintain reconcile`, `remedy?`) | | `error` | a refusal or failure | `message`, `code?`: `MODE_USAGE` (only `interactive` and `agent` are accepted), `ANSWER_USAGE` (an answer to a question the run does not ask, or a value that is neither option) | | `done` | last | `status`, `exit_code`, `next?`: `complete` 0, `refused` 2, `paused` 3, `error` with the design run's own code; `next[]` carries every paused session's resume, uncapped | - **Answering.** `--answer evolve:<use-case>=1` (or `=proceed`) and `--answer evolve:<use-case>=2` (or `=decline`), repeatable, with `--mode agent`. A refused answer is `error` + `done` with nothing before it. -- **Refusals.** An explicit target that is fresh and has no `--because`, or that was never designed, exits `2`. A `--from-stale` run with nothing stale exits `0`. +- **Refusals.** An explicit target that is fresh and has no `--because`, that was never designed, or that is out of date only because its source changed (`evolve_skipped` `source_changed`; `--because` lets it run), exits `2`. A `--from-stale` run with nothing stale exits `0`, and one whose every stale use-case is source-changed sends those `evolve_skipped` events and exits `0`. - **Batches.** Every use-case runs whatever the earlier ones did. The exit is the first design-run code that is neither 0 nor 3; otherwise 3 if any paused; otherwise 0. ## Sync events (verb `sync`) *(0.8.14+)* diff --git a/skill-installer/skills/references/assurance.md b/skill-installer/skills/references/assurance.md index 7295995..efbe6ea 100644 --- a/skill-installer/skills/references/assurance.md +++ b/skill-installer/skills/references/assurance.md @@ -6,7 +6,7 @@ When the user has **requirements** — a PRD, a spec, acceptance notes — and w - **Assurance** = tests derived from the actual documents, every claim cited, every test permanently tagged with the acceptance criteria it verifies, coverage measured against requirements. Use it whenever the user cares about "what exactly is covered, and how do we know?" -Everything here works over a local store (`.context/` in the project directory) that the commands create and manage themselves. +Everything here works over a local store (`.context/` in the project directory) that the commands create and manage themselves. **A folder linked to a team's shared project** (it holds `.testmuai/assurance.json`; kane-cli 0.10.0+) works in the project instead, and its journey differs: no `context review` checkpoint, every session is saved explicitly, and `cover gaps` is refused. Follow `shared-projects.md` §4 there; the sections below describe the local store: a linked folder's events and refusal codes are in `agent-contract.md`, and the options it refuses in `shared-projects.md` §8. In agent mode a folder that isn't set up yet goes local unless you pass `--project <id>`, so settle where the work goes first (`shared-projects.md` §1). **Version gate — check before improvising.** The assurance commands exist on kane-cli **0.6.1 and later**; flags and events marked with a release (**0.7.1+** … **0.8.14+**) below need at least that release. On an older CLI, `kane-cli context …` fails as an *unknown command* (exit 2 with a "did you mean" suggestion) — that error means the CLI is too old, not that you typed it wrong. Confirm with `kane-cli --version`, tell the user to update (`npm install -g @testmuai/kane-cli`, or `brew upgrade kane-cli`), and stop — do not try to reproduce the workflow with other commands. @@ -19,7 +19,7 @@ kane-cli design tests --use-case <uc-ref> --mode agent --max 8 # 3. design ACs kane-cli context review --verdicts <file> --json # 4. CHECKPOINT: user approves the design kane-cli testmd run .testmuai/tests/<t>_test.md --agent # 5. author each kept test once (real browser) kane-cli testrun run --match 't-' < /dev/null # 6. batch replays from then on -kane-cli cover gaps # 7. designed % × proven % + per-use-case debt +kane-cli cover gaps # 7. the coverage figure, use-cases with tests, per-use-case debt kane-cli maintain reconcile --from <new.md> --source-id <id> --mode agent # when a source changes (§11) ``` @@ -30,7 +30,7 @@ Two hard rules: 1. **The two checkpoints are the user's, not yours.** Everything the extract and design agents emit is *derived* (unreviewed) until a human decision promotes it — see §4. Do not start authoring (step 5) until the user has approved the design output or explicitly told you to proceed. 2. **One step at a time, one store-writer at a time.** Never run two store-mutating commands concurrently (`ingest`, `extract`, `review` with verdicts, `design`, `reconcile`, `evolve`) — the store is single-writer (§8), and extraction additionally holds a store-wide lock (a concurrent extract refuses `EXTRACT_LOCKED`, exit 2 — wait, never delete locks). -Extract, design, reconcile and evolve call the KaneAI service and consume credits; everything else is local and free. +Extract, design, reconcile and evolve call the KaneAI service and consume credits. URL sources are fetched server-side, and a web page's ingest is refused when the organisation is out of credits. Everything else is local and free. ## 2. The pause loop — exit 3 is a pause, NOT a failure @@ -49,7 +49,7 @@ Three ways to resume: kane-cli context extract --resume <sid> --mode agent --message "Account required — the update section supersedes the older text" # by id (0.7.1+) — for precise scripted answers; each echoes a panel_resolved event: -kane-cli context extract --resume <sid> --mode agent --answer q1=1 --answer q2="https://staging.example.com" +kane-cli context extract --resume <sid> --mode agent --answer q1=1 --answer q2="https://app.example.com" # land a new source instead of answering (0.7.1+) — the batch defers (ask_deferred), the agent # reads the source and re-asks only what's still open: @@ -57,7 +57,7 @@ kane-cli context extract --resume <sid> --mode agent --with-source ./addendum.md ``` - If the answer leaves any question standing, the run pauses again with refreshed questions — repeat. -- Sessions live 24 hours. Inspect without contending a live run: `kane-cli context sessions --json` (all resumable sessions + their resume commands) and `kane-cli context sessions show <sid> --json` (the pending questions in wire shape). A session written by a NEWER kane-cli than the installed one lists as unknown with no resume command (0.7.1+) — that means "upgrade to resume it", not corruption. If you abandon a session deliberately, remove it with `kane-cli context sessions clean <sid>` — bare `clean` only collects *expired* sessions, and `--all` is a purge that needs explicit user authorization. +- Sessions live 24 hours. Inspect without contending a live run: `kane-cli context sessions --json` (all resumable sessions + their resume commands) and `kane-cli context sessions show <sid> --json` (the pending questions in wire shape). A session written by a NEWER kane-cli than the installed one lists as unknown with no resume command (0.7.1+) — that means "upgrade to resume it", not corruption. If you abandon a session deliberately, remove it with `kane-cli context sessions clean <sid>` — bare `clean` only collects *expired* sessions, and `--all` is a purge that needs explicit user authorization. (A linked folder refuses `sessions show` and `sessions clean`: `kane-cli context sessions --mode agent` lists the project's sessions there.) Full event schema: `references/assurance-parsing.md`. @@ -86,11 +86,11 @@ Remote ids share ONE space (0.8.6+): a URL whose id is already backed by a diffe Re-ingesting changed bytes under the same id **versions** the source and marks everything derived from the old snapshot stale — but for a changed document, prefer `maintain reconcile` (§11), which does the version move AND triages the fallout. -Extract behavior to expect (0.7.1+): the sweep **continues past a per-source failure** (one line each; exit 1 at the end if anything failed — the failed source simply retries next run, no `--force` needed). `--trust hold` holds everything new for the user's review instead of committing (headless-only; `--mode ci` refuses the flag entirely; use it one source at a time — resuming held work from a multi-source run is not yet supported). The agent cites every claim (fabrication is rejected before write), asks when the source contradicts itself, and commits proposals as `derived`. Watch the `commit` event and say it in plain language ("5 use-cases extracted from the PRD"). +Extract behavior to expect (0.7.1+): the sweep **continues past a per-source failure** (one line each; exit 1 at the end if anything failed — the failed source is read again on the next extract, which costs credits again; no `--force` needed). `--trust hold` holds everything new for the user's review instead of committing (headless-only; `--mode ci` refuses the flag entirely; use it one source at a time — resuming held work from a multi-source run is not yet supported). The agent cites every claim (fabrication is rejected before write), asks when the source contradicts itself, and commits proposals as `derived`. Watch the `commit` event and say it in plain language ("5 use-cases extracted from the PRD"). ## 4. Review — trust is the user's decision -Promotion from `derived` to `trusted` **always requires explicit user confirmation** — even when the user asked for an "end-to-end" run. End-to-end authorizes completing the workflow, not making product-requirement judgements. +Promotion from `derived` to `trusted` **always requires explicit user confirmation** — even when the user asked for an "end-to-end" run. *(In a shared project there is no review command: `context review` and `context explain` are refused in a linked folder, and use cases an agent run saves stay proposed. Run the checkpoint as `shared-projects.md` §4 says: show them, ask once for the batch, then design with `--allow-unreviewed` on a yes.)* End-to-end authorizes completing the workflow, not making product-requirement judgements. **Present the material, not the counts.** The `commit` event carries counts and ids only — to run a checkpoint, enumerate what's waiting with `kane-cli context list --json --inferred` (one JSON row per unreviewed node) and pull any item's full content, citations, and history with `kane-cli context explain <ref> --json`. Present titles, descriptions, and the cited evidence; collect the user's decisions; build the verdicts from those same refs. The same recipe runs the design checkpoint. Then land the decisions atomically: @@ -150,20 +150,22 @@ So a failing authored run is never, by itself, a reason to rewrite the step. Cla ## 7. What's next — let the tool tell you ```bash -kane-cli cover # the pack audit: what it proved, plus the live-graph completeness worklist -kane-cli cover gaps # the coverage ribbon: one row per use-case, both axes +kane-cli cover # one coverage figure over the use-cases; --from <pack> reads one run's own figure +kane-cli cover gaps # the coverage ribbon: that figure, then one row per use-case (designed count, coverage, debt) kane-cli cover gaps <uc-id> # 0.8.2+: the dossier — every AC of that use-case + next actions -kane-cli cover gaps --json # the nested document (designed axis, proven axis, per-UC pending rows with ready_command) +kane-cli cover gaps --json # the nested document (the figure, designed counts, per-UC pending rows with ready_command) kane-cli cover gaps --mode agent # 0.7.1+: same data as ONE `gaps` event + done.next[] ready-commands ``` -On 0.8.2+ the default `cover gaps` output is the **coverage ribbon** — a high-level band-table: one severity-ordered row per use-case with designed/proven bars and per-class debt counts, deliberately carrying NO commands. To act, drill in: `cover gaps <uc-id>` (the dossier) renders every AC of that use-case plus a `next` actions block — evidence first; surface its warning that re-running an authored test on a broken app may heal the test around the failure — or read `--json`, whose per-UC pending rows keep `ready_command`. `--rollup lenient|strict` selects the proven-axis formula. BREAKING at 0.8.2: `--flat` and gaps' `--from` are REMOVED and refuse (pre-0.8.2 CLIs still have the flat worklist). With a `<uc-id>`, `--json` and agent mode close the document over that use-case. +**In a shared project** (a linked folder): `cover gaps` is refused. `kane-cli cover --mode agent` gives the project's own figure, one `coverage` event then `done`. A `coverage` event with `state: "computing"` and no `pct` means the project is still working its figure out: say so, and run `done.next` again in a minute. A run less than 5 minutes old may not count yet (`recent_run.may_not_count_yet: true`); say so and offer to check again. Its `done.next` commands carry their agent flag (`--mode agent`, or `--agent` for `testrun run`): run them as given, ending a `testrun run` line in `< /dev/null`. If a `next` command has no agent flag, add the one that command takes before running it. + +On 0.8.2+ the default `cover gaps` output is the **coverage ribbon** — a high-level band-table: one severity-ordered row per use-case with its designed count, its coverage and per-class debt counts, deliberately carrying NO commands. To act, drill in: `cover gaps <uc-id>` (the dossier) renders every AC of that use-case plus a `next` actions block — evidence first; surface its warning that re-running an authored test on a broken app may heal the test around the failure — or read `--json`, whose per-UC pending rows keep `ready_command`. `--rollup` no longer changes the figure: don't pass it (agent mode answers it with a `warning` `ROLLUP_IGNORED`). BREAKING at 0.8.2: `--flat` and gaps' `--from` are REMOVED and refuse (pre-0.8.2 CLIs still have the flat worklist). With a `<uc-id>`, `--json` and agent mode close the document over that use-case. ## 8. Store rules — don't corrupt the user's graph - `.context/` is **append-only and single-writer**. Never run two store-mutating commands concurrently (extract, review verdicts, design, ingest, reconcile, retire/revert/rebuild). On a lock error (`EXTRACT_LOCKED`, or reconcile's walk lock), another run is live: report it and wait — never delete lock files; a dead run's lock clears itself. - Never hand-edit anything under `.context/`, and never `git merge` it — two histories merge cleanly in git and brick the store on the next read. Keep it gitignored — kane-cli 0.8.14 adds `.context/` to the folder's `.gitignore` itself when it creates the store inside a git repository — `gitignore_updated` on the stream, or a `warning{message}` when the line could not be written (relay it and add the line yourself); `KANE_CONTEXT_GITIGNORE=0` keeps it out. To share it with a team, use a **location** (a GitHub repository, an S3-compatible bucket, or a folder) through `kane-cli context sync`, `kane-cli context push`, `kane-cli context pull` and `kane-cli context clone` (0.8.14+) — a Git repository used as a location is not the same as committing `.context/`. While a sync rebase is open the store refuses every other write with `SYNC_REBASE_PENDING` (the fence) — finish it or `kane-cli context sync doctor --abort`, never delete files. Rules, events and decisions: `references/context-sync.md`. -- Safe inspection any time: `kane-cli context list --json` (nodes with trust + freshness), `kane-cli context explain <ref>` (a node's full recorded history, no AI), `kane-cli context fsck` (integrity check), `kane-cli context view --no-open --out <path>` (writes a self-contained HTML graph snapshot to that path — never opens a browser, never touches the graph). +- Safe inspection any time (a linked folder refuses `explain`, `fsck` and `view`; there use `kane-cli context list --mode agent`): `kane-cli context list --json` (nodes with trust + freshness), `kane-cli context explain <ref>` (a node's full recorded history, no AI), `kane-cli context fsck` (integrity check), `kane-cli context view --no-open --out <path>` (writes a self-contained HTML graph snapshot to that path — never opens a browser, never touches the graph). - Destructive verbs (`context retire`, `revert`, `rebuild`, `name --backfill`) exist and take `--yes` headless — run them **only on an explicit user request**, never autonomously. ## 9. When things fail @@ -197,6 +199,8 @@ On 0.8.2+ the default `cover gaps` output is the **coverage ribbon** — a high- | `sync_error` code `SYNC_BEHIND` / `SYNC_DIVERGED` (exit 3, 0.8.14+) | the location has records this store has not pulled / both sides added work — a pull or a rebase is needed, not a failure | relay the `remedy` verbatim (`kane-cli context pull <name>`, or `--rebase` on the user's go-ahead) — `references/context-sync.md` §6 | | `sync_behind` event on an extract/design/reconcile stream (0.8.14+) | advisory: the location has moved past this store | one line ("origin has records you have not pulled"); the run continues | | `sync_error` code `SYNC_STORE_BUSY` (exit 2, 0.8.14+) | a sync verb met a live session or lock on this store | wait; `kane-cli context sync doctor` when it names an interrupted rebase | +| `error` code `NO_PROPOSAL` | the agent stopped before proposing use-cases for that source | re-run `kane-cli context extract --mode agent` once; it reads that source again and costs credits again. If it happens twice, stop, and give the person the command and the message | +| a shared-project code (`PROJECT_NO_ACCESS`, `FOLDER_CHOSEN`, `PROJECT_BUSY`, `REMOTE_UNSUPPORTED`, `CONFIRM_REQUIRED`, `PROJECT_MISMATCH`, `TEST_CASE_MISSING` …) | a linked folder's refusal | `agent-contract.md`: each code with the command that recovers it | | `error` + `done` with exit `1` | runtime failure (incl. a sweep where some sources failed) | report the message; **do not blindly re-run a paid command** | | auth/credit failure mid-run | token or balance problem | keep the `sid`, have the user fix auth/balance (`kane-cli whoami`, `kane-cli balance`), then resume | | stream ends with no `done` event | the process crashed — outcome unknown (the one designed exception: `kane-cli context review --mode agent` keeps its `--json` shape and ends its listings without `done`; read its exit code) | check `context sessions --json` and `context list` before any retry, to avoid duplicate paid work | @@ -219,6 +223,8 @@ A changed requirements document is **one command**, not a re-ingest: kane-cli maintain reconcile --from ./prd-v2.md --source-id prd --mode agent ``` +**In a linked folder** reconcile takes a file, always with `--source-id`: `kane-cli maintain reconcile --from <file> --source-id <id> --mode agent`. It opens a review session that you save like any other (`shared-projects.md` §4), not the `reconcile_*` stream below. `--plan`, `--apply`, `--mode ci`, a missing `--source-id` and a URL in `--from` are refused (`REMOTE_UNSUPPORTED`); for a page's new version run `kane-cli context ingest <address> --mode agent`. The rest of this section is the local store's. + Never `context ingest` the new version first — reconcile does its own re-ingest, records the version move, and triages the fallout into rows: ADD (new use-case → design it), MODIFY (content moved → update or re-design), ARCHIVE (evidence decayed — **never applied headless**, always waits for the user's interactive session). It emits its own event family (`reconcile_plan` → `reconcile_row_start`/`reconcile_row_end` per row → `reconcile_paused`? → `reconcile_summary` → `done`; see `references/assurance-parsing.md`). Modes: `agent` auto-applies the safe ADD/MODIFY tier and exits 3 with the stored plan when ARCHIVE rows need a human; `ci` fail-closes (plan stored, exit 2); `--plan` is a safe staged preview. Re-running the same command is idempotent — it resumes a pending plan rather than re-billing. 0.7.2+ additions: @@ -227,9 +233,9 @@ Never `context ingest` the new version first — reconcile does its own re-inges - **`--from` takes URLs.** The same remote URLs ingest takes (§3): Jira/Confluence (0.7.2+), web pages (0.8.3+), Linear issues and documents (0.8.6+). `--source-id` is then optional — the URL carries its own id, and a contradicting `--source-id` refuses. Kind continuity holds both ways: a URL can't version a file-backed source and a file can't version a remote one — each refusal names the correct `--from`. A source ingested under a custom id (`--as`) is maintained by re-running that ingest with the same `--as`. The first reconcile of a Jira issue last ingested pre-0.7.2 may report the one-time upgrade re-version (§3) — not a content edit. - **One stream.** The re-extract child's events ride the reconcile stream itself, stamped `verb: "reconcile"` — parse per `references/assurance-parsing.md`. -For staleness that arrived outside a reconcile, `kane-cli maintain evolve <ref>` re-designs the parent use-case of the entity you name (`--from-stale` takes every use-case with stale designs). Give a `<ref>` or `--from-stale`, never both. A fresh target exits `2` unless you pass `--because "<reason>"`, and a use-case that was never designed exits `2` too (run `design tests` first). In a terminal evolve states the blast radius and asks. +For staleness that arrived outside a reconcile, `kane-cli maintain evolve <ref>` re-designs the parent use-case of the entity you name (`--from-stale` takes every use-case with stale designs). Give a `<ref>` or `--from-stale`, never both. A fresh target exits `2` unless you pass `--because "<reason>"`, and a use-case that was never designed exits `2` too (run `design tests` first). A use-case out of date only because its source changed is not evolved: as a target it exits `2` (unless `--because` is passed) and a `--from-stale` batch leaves it out, each with `evolve_skipped` `reason: "source_changed"` (and a `next`, when kane-cli knows the source's file, that reviews its change with `kane-cli maintain reconcile …`): offer the reconcile instead. In a terminal evolve states the blast radius and asks. -*(0.8.21+)* `kane-cli maintain evolve <ref> --mode agent` puts that question on the stream. A call on a stale target with no answer ends paused, exit `3`: `evolve_target` carries the blast radius (`use_case`, the `pairs` it affects, `because?`), then `evolve_paused` carries `pending_questions` (one per use-case, `id: "evolve:<use-case>"`, options `proceed` / `decline`), `resume` and `next[]` (two ready commands: proceed all, decline all), then `done{paused}`. Present the blast radius with the question, as with any pause (the pause loop above). Answer on the next call with `--answer evolve:<use-case>=1` (or `=proceed`) or `--answer evolve:<use-case>=2` (or `=decline`), repeatable; `--answer` needs `--mode agent`, and the resume command keeps `--because`. Nothing runs until every question is answered. A partial set pauses again on the rest, and the `next` commands then carry the answers already given, so a "decline the rest" command still re-designs what was answered proceed: read a `next` command before running it. On proceed, the design run's own events arrive between `child_start` and `child_end` stamped `verb: "design"`. After a clean design run comes `evolve_diff` (superseded, minted, retained, criteria whose tests moved). A design session that pauses arrives as its own `session_paused`: resume that session, then run evolve again for the diff. `done.next[]` carries every paused session's resume, all of them, whatever the exit code. In a `--from-stale` batch every use-case runs whatever the earlier ones did, and a declined or never-designed one is reported as `evolve_skipped`. Schema: `references/assurance-parsing.md`. On a CLI before 0.8.21 evolve has no `--mode`: suggest the user run it in a terminal rather than scripting around it. +*(0.8.21+)* `kane-cli maintain evolve <ref> --mode agent` puts that question on the stream. A call on a stale target with no answer ends paused, exit `3`: `evolve_target` carries the blast radius (`use_case`, the `pairs` it affects, `because?`), then `evolve_paused` carries `pending_questions` (one per use-case, `id: "evolve:<use-case>"`, options `proceed` / `decline`), `resume` and `next[]` (two ready commands: proceed all, decline all), then `done{paused}`. Present the blast radius with the question, as with any pause (the pause loop above). Answer on the next call with `--answer evolve:<use-case>=1` (or `=proceed`) or `--answer evolve:<use-case>=2` (or `=decline`), repeatable; `--answer` needs `--mode agent`, and the resume command keeps `--because`. Nothing runs until every question is answered. A partial set pauses again on the rest, and the `next` commands then carry the answers already given, so a "decline the rest" command still re-designs what was answered proceed: read a `next` command before running it. On proceed, the design run's own events arrive between `child_start` and `child_end` stamped `verb: "design"`. After a clean design run comes `evolve_diff` (superseded, minted, retained, criteria whose tests moved). A design session that pauses arrives as its own `session_paused`: resume that session, then run evolve again for the diff. `done.next[]` carries every paused session's resume, all of them, whatever the exit code. In a `--from-stale` batch every use-case runs whatever the earlier ones did, and a declined, never-designed or source-changed one is reported as `evolve_skipped`. Schema: `references/assurance-parsing.md`. On a CLI before 0.8.21 evolve has no `--mode`: suggest the user run it in a terminal rather than scripting around it. ## Sharing a context store diff --git a/skill-installer/skills/references/cards.md b/skill-installer/skills/references/cards.md index b035268..c8e58f2 100644 --- a/skill-installer/skills/references/cards.md +++ b/skill-installer/skills/references/cards.md @@ -73,7 +73,7 @@ Exit code `1`, or `status: "failed"`. Show the failing step's screenshot under t ## 4. Didn't start -Exit code `2`: nothing ran and no credits were used. Causes include no start URL, sign-in or setup errors, a test file that does not parse, an invalid suite plan, a cloud grid refusal. +Exit code `2`: nothing ran and no credits were used. Causes include no start URL, sign-in or setup errors, a test file that does not parse, an invalid suite plan, a cloud grid refusal. A designed test in a shared project that stopped on its Test Manager case before any step (an `error` with a `TEST_*` code) is this card too, even when it exits `1`. ```markdown | | | @@ -173,3 +173,38 @@ On kane-cli older than 0.8.17 the end event has no reason: read it from the evid A test that comes back broken with zero steps on the grid was refused before it launched: say so, point to the job link, and suggest checking that the app id belongs to this account. An invalid plan is a `🟡 Didn't start` card (§4) with one line per rejected test. + +**In a shared project** a suite's card adds a coverage row from the run's `coverage` event, and a line for each first +run (`authored`): + +```markdown +| 📈 **Coverage** | <pct>% of "<project>" · this run reached <n> use cases · it can take up to 5 minutes to count | +| ✍️ **First runs** | <n> tests wrote their steps on this run; later runs replay them | +``` + +Only a `coverage` event with `status: "written"` has a `pct`. With any other status, the 📈 row says +`The project's coverage isn't ready yet · I can check again in a few minutes` (`kane-cli cover --mode agent`). + +A member that stopped on its Test Manager case shows `failed` with a reason such as "isn't in Test Manager yet": say what +it needs (`testrun.md` §Designed tests in a shared project), not "the test failed". + +## 9. Coverage (`kane-cli cover`) + +Fields: the `coverage` event's `pct`, `usecases` (`total`, `proven`, `partial`, `failing`, `not_run`, `not_designed`, +`no_criteria`), `note`, and `recent_run` when a run is under 5 minutes old; then `done.next`. A `coverage` event with +`state: "computing"` has none of these: say the project is still working its figure out, draw no card, and run +`done.next` again in a minute. + +```markdown +| | | +|---|---| +| 📈 **Coverage** | <pct>% across "<project>"'s <total> use cases | +| ✅ **Proven** | <proven> · ⚠️ partial <partial> · ❌ failing <failing> | +| ⏳ **Not yet** | <not_run> not run · <not_designed> without tests · <no_criteria> with no criteria yet | +| 🕒 **Recent run** | still counting: check again in a few minutes *(only with `recent_run.may_not_count_yet`)* | +| ➡️ **Next** | <the first `done.next` command in plain words, e.g. run every test · design tests for "<use case>"> | +``` + +`note: "nothing_run_yet"` with `total: 0` means no use case exists yet: offer `kane-cli context ingest <file>`, not a run. +`done.next` commands carry their agent flag; run them as given, ending a `testrun run` line in `< /dev/null`. If one +has no agent flag, add the one that command takes. diff --git a/skill-installer/skills/references/context-sync.md b/skill-installer/skills/references/context-sync.md index a9f19b0..671f3c9 100644 --- a/skill-installer/skills/references/context-sync.md +++ b/skill-installer/skills/references/context-sync.md @@ -2,6 +2,13 @@ # Context sync — Agent Surface +> **Not for shared projects.** Location sync shares a **local** `.context/` store through a Git repository, a bucket or a +> folder. A team that works in a Test Manager project uses a shared project instead (`shared-projects.md`). A folder +> linked to one refuses `kane-cli context sync`, `kane-cli context push` and `kane-cli context clone` +> (`REMOTE_UNSUPPORTED`), and a location pull (a location name, `--rebase`, `--yes` or `--answer`: +> `PULL_LOCATION_LINKED`); a plain `kane-cli context pull --mode agent` there brings the project's tests, when +> `kane-cli context pull --help` lists `--project` (`shared-projects.md` §7). Never use both in one folder. + The assurance store (`.context/`, `references/assurance.md` §8) is one person's. **Context sync** shares it with a team through a **location**: a GitHub repository, an S3-compatible bucket, or a folder. Each person keeps their own store and publishes to the location; nothing on a location is ever overwritten or deleted. Reach for this reference when the user wants to share their context store, join a teammate's, get two stores level, or resolve a sync conflict. Sync never calls the KaneAI service — no credits, no model. **Version gate.** `kane-cli context sync`, `kane-cli context push`, `kane-cli context pull` and `kane-cli context clone` exist on **kane-cli 0.8.14 and later**. On an older CLI they fail as an *unknown command* (exit 2, a "did you mean" line) — the CLI is too old, not a typo. Confirm with `kane-cli --version`, have the user update, stop. diff --git a/skill-installer/skills/references/debug.md b/skill-installer/skills/references/debug.md index 372d188..9a33997 100644 --- a/skill-installer/skills/references/debug.md +++ b/skill-installer/skills/references/debug.md @@ -6,7 +6,7 @@ When a run fails, diagnose before suggesting fixes. ## The evidence pack is the log source -Every run seals an **evidence pack**; all run artifacts — actions, console, network, screenshots, failure records — live inside it. **`run_end.run_dir` is legacy: that directory is not created anymore.** Do not try to read `{run_dir}/run-test/...`. +Every run that starts seals an **evidence pack**; all run artifacts — actions, console, network, screenshots, failure records — live inside it. **`run_end.run_dir` is legacy: that directory is not created anymore.** Do not try to read `{run_dir}/run-test/...`. **Find the pack**: `{session_dir}/evidence/<execution_id>.evidence`; testmd/testrun/named runs also land in `<cwd>/.testmuai/evidence/`. The post-run stderr hint names the exact path. @@ -25,7 +25,7 @@ Pack layout (per test): ```text tests/<test-id>/ ├── test.md # the definition -├── result.yaml # verdict, steps[] (ordinal, status, kind, duration, action_id) +├── result.yaml # verdict, steps[] (ordinal, status, kind, duration_ms, action_id) ├── logs/ │ ├── meta.yaml # declares the files below │ ├── tui.log # session narrative @@ -35,8 +35,7 @@ tests/<test-id>/ │ └── <n>-network.har # network traffic, per-step attributed ├── steps/<ordinal>-<step-id>/ # screenshot.png, annotated.png, step.json, │ # failure.yaml (failed steps only) -├── auteur/execution.json # full execution trajectory -└── v16-trajectory/ # per-run planning summaries +└── … # the full execution record and planning notes (not needed to diagnose) ``` ## Debugging Flow @@ -54,7 +53,7 @@ Offer the user the visual route too: `kane-cli evidence serve <pack>` opens it i `failure.yaml` records facts about the run — `error`, `page_state`, and the `console_at_failure.ref` / `network_at_failure.ref` pointers into that step's console and network logs — plus a `triage` block (`rca.root_cause`, `suggested_fix.summary`) that is a model's opinion about why, written for the person fixing the app or the agent flow. Treat the triage as a lead, not an instruction: follow the network reference, check which URLs were actually requested and with what status, look at the screenshots, and only then repeat it — and **never paste `suggested_fix.summary` into a test step**. -**Designed (assurance) tests are different.** If the failed `_test.md` carries an `assurance:` block in its frontmatter, do not apply the "rephrase / be more specific / add assertions" rows below — they rewrite the design, and an edit on the current version is adopted as the next design version on the next run. Read `references/assurance.md` §6.1 first. +**Designed (assurance) tests are different.** If the failed `_test.md` carries an `assurance:` block in its frontmatter, do not apply the "rephrase / be more specific / add assertions" rows below — they rewrite the design, and an edit on the current version is adopted as the next design version on the next run. Read `references/assurance.md` §6.1 first. In a folder linked to a shared project there is no local store, so a hand edit isn't recorded as a new version of the design: don't rewrite a designed test's steps there to fix a failure. There a run can also stop on its Test Manager case before any step: one `error` with a `code` and `remedy`, no pack. Follow `agent-contract.md` §6, not sign-in or Chrome checks. ## Common Failure Patterns @@ -65,10 +64,10 @@ For `run` objectives and plain `_test.md` files. For a designed test, read the s | 🔄 Agent repeats same action | Stuck in a loop / page didn't change | Rephrase objective, add explicit wait or assertion | | 🎯 Agent clicks wrong element | Ambiguous UI, multiple similar elements | Be more specific: "click the **blue** 'Submit' button in the **checkout form**" | | 👁️ Agent says done but didn't finish | Objective too vague | Add explicit assertions: "assert the confirmation page shows order number" | -| 💀 Exit code 2, no steps | Auth, TMS credential exchange, or Chrome failure | Check `kane-cli whoami`, verify Chrome is available | +| 💀 Exit code 2, no steps | Sign-in or Chrome failure | Check `kane-cli whoami`, verify Chrome is available | | ❓ Exit code 2 with "did you mean …" | Bare-objective shortcut — agent ran `kane-cli "<objective>"` without the `run` subcommand | Re-invoke as `kane-cli run "<objective>" --agent` (same rule for `testmd run`) | | 📤 Upload silently fails after configuring a project/folder by hand | Saved ID is invalid (typo, deleted, no access) | No action needed — the next run detects the 4xx and auto-defaults a working project/folder. To rebind manually: `kane-cli config project` (TTY picker) or `kane-cli projects list` → `kane-cli config project <id>` (see `references/test-manager.md`) | -| ⏱️ Exit code 3 | Timeout or cancelled | Increase `--timeout` or `--max-steps`, or split into smaller objectives | +| ⏱️ Exit code 3 (`run`; a saved test reports it as failed, exit 1) | Timeout or cancelled | Increase `--timeout` or `--max-steps`, or split into smaller objectives | | 🚫 "CDP endpoint not reachable" | Chrome not running | Let kane-cli manage Chrome (remove `--cdp-endpoint`) | ## Filing a bug report diff --git a/skill-installer/skills/references/evidence.md b/skill-installer/skills/references/evidence.md index 32ff71b..fbe6643 100644 --- a/skill-installer/skills/references/evidence.md +++ b/skill-installer/skills/references/evidence.md @@ -20,8 +20,7 @@ A `.evidence` file is a standard zip: `unzip -l <pack>` lists it, `unzip -p <pac │ # <n>-console.ndjson, <n>-network.har ├── steps/<ordinal>-<step-id>/ # screenshot.png, annotated.png, step.json │ # (+ failure.yaml on failed steps) - ├── auteur/execution.json # full execution trajectory - └── v16-trajectory/ # per-run planning summaries + └── … # the full execution record and planning notes (not needed to diagnose) ``` Reading and debugging from the pack (unzip recipes, failure records): `references/debug.md`. @@ -36,7 +35,7 @@ Reading and debugging from the pack (unzip recipes, failure records): `reference ## The stderr hint — and how to act on it -After a successful agent-mode run, kane-cli prints one hint line to **stderr** (it is NOT an NDJSON event; never look for it on stdout): +After an agent-mode run whose evidence pack sealed (passed or failed), kane-cli prints one hint line to **stderr** (it is NOT an NDJSON event; never look for it on stdout): ``` evidence: view locally with `kane-cli evidence serve <packPath>` @@ -70,7 +69,7 @@ Exit `0` valid · `1` invalid · `2` not found. Use it when a pack won't open in kane-cli evidence merge <targets...> --run-id <id> ``` -Combines packs (execution ids or paths, order-significant) into one, sealed by default at `.testmuai/evidence/<run-id>.evidence`. `--run-id` is required. Exit `0` merged · `1` policy abort (e.g. mixed projects, duplicate run ids) · `2` usage. Full policy detail is in the user guide — for anything beyond a simple merge, point the user there rather than improvising flags. +Combines packs (execution ids or paths, order-significant) into one, sealed by default at `.testmuai/evidence/<run-id>.evidence`. `--run-id` is required. Exit `0` merged · `1` policy abort (e.g. packs from different projects or organisations, or a pack that isn't sealed and valid; a pack whose run id repeats is skipped, not an abort) · `2` usage. Full policy detail is in the user guide — for anything beyond a simple merge, point the user there rather than improvising flags. ## Debugging with a pack @@ -78,6 +77,6 @@ The pack is the first place to look on a failure: the failed step's **failure re ## Evidence merge identity -For ordinary runs, default merge identity distinguishes the test and commit (`external_id.test_id`, `external_id.commit_id`) and the environment (`environment.os`, `environment.os_version`, `environment.browser`, `environment.browser_version`, `environment.device`, `environment.app`). Re-runs with the same identity nest as attempts; a different environment produces a separate sibling, so one test on two devices or two builds is two entries. On a real device, `environment.device` is the device the hub allocated (never the requested pattern) and `device_slug` is derived from it. +For ordinary runs, default merge identity distinguishes the test, commit and dataset row (`external_id.test_id`, `external_id.commit_id`, `external_id.dataset_row_id`) and the environment (`environment.os`, `environment.os_version`, `environment.browser`, `environment.browser_version`, `environment.device`, `environment.app`). Re-runs with the same identity nest as attempts; a different environment produces a separate sibling, so one test on two devices, or on two builds with different app identities, is two entries (two local builds with the same file name can group together). On a real device, `environment.device` is the device the hub allocated (never the requested pattern) and `device_slug` is derived from it. Explicit collision policies can change grouping; a custom `--rules` file replaces the default rules rather than extending them. Check the selected identity rules before interpreting two runs as retries of the same test. diff --git a/skill-installer/skills/references/execution-controls.md b/skill-installer/skills/references/execution-controls.md index 5cae455..3cb1086 100644 --- a/skill-installer/skills/references/execution-controls.md +++ b/skill-installer/skills/references/execution-controls.md @@ -4,7 +4,7 @@ `run` and `testmd run` support `--assertion-mode dom|visual` (default `dom` with vision fallback) and `--final-validation on|off` (default off). Persist with `config set-assertion-mode` and `config set-final-validation`. Final validation controls the synthesized `cp_final` checkpoint independently of action/testing mode; keep explicit terminal assertions in objectives. -`run --analyzer-only --condition "<condition>"` checks the current desktop browser page without an objective, action steps or saved test. Repeat `--condition` for multiple checks. Use `--agent` or non-TTY input. Results contain `condition_results: boolean[]`; exit `0` means all conditions were judged, **not** that all are true. Exit `1` means a result was missing, and `3` means cancelled. This mode rejects mobile target/app/device options and code-export/name options. +`run --analyzer-only --condition "<condition>"` checks the current desktop browser page without an objective, action steps or saved test. Repeat `--condition` for multiple checks. Use `--agent` or non-TTY input. Results contain `condition_results: (boolean | null)[]`, one per condition; `null` means that condition couldn't be judged, and `condition_reasons` says why. Exit `0` means all conditions were judged, **not** that all are true. Exit `1` means a result was missing or `null`, and `3` means cancelled. This mode rejects mobile target/app/device options and code-export/name options. Experimental `run --network-ws` and `run --network-sse` enable WebSocket and SSE capture; both default off. Persist with `config set-network-ws on|off` and `config set-network-sse on|off`. SSE capture is Chromium-only. Do not copy these run-only flags onto `testmd` or `testrun` commands. diff --git a/skill-installer/skills/references/explore.md b/skill-installer/skills/references/explore.md new file mode 100644 index 0000000..64199f5 --- /dev/null +++ b/skill-installer/skills/references/explore.md @@ -0,0 +1,217 @@ +<!-- kane-cli skill reference: exploration on HyperExecute, and the tests it designs into a shared project. Read when the person asks to explore an app, map its flows, shape, stop or re-run an exploration, or get and run the tests an exploration made. Requires kane-cli 0.10.0+. --> + +# Exploration: map an app on HyperExecute, and get the tests it designs + +`kane-cli explore` sends an explorer through the person's app on **HyperExecute**. It builds a map of the app (screens, +flows, findings, with evidence). While its **assurance context** is on (the default), every few minutes while it runs, +and when it ends, what it has mapped is turned into use cases, scenarios and tests in the Test Manager project the +exploration belongs to, with a Test Manager case per scenario. Exploration **designs and saves** those tests; it never +runs them. + +**Exploration and assurance.** The tests an exploration designs live in its project, as a document's do in a shared +project (`shared-projects.md`). To run them here, a folder has to hold them: a folder linked to that project gets them +with `kane-cli context pull --mode agent`, and a fresh folder links and gets them in one step with +`kane-cli context pull --project <id> --mode agent` (§5). A project nobody has shared yet becomes a shared project with +the exploration's first design job, as the person who started the run. + +Explore takes `--agent`, not `--mode agent`. Its messages can name internal parts: use them to work out what happened, +then tell the person in plain words. + +## 1. Start an exploration + +```bash +kane-cli explore run --project <id> --url <app url> --prompt "<what to explore>" --agent # add --policy read_only if the person said no changes +``` + +- `--project <id>` is the Test Manager project the exploration saves into: the team's shared project + (`shared-projects.md` §2 finds the id). One project, one exploration, one URL. **Always pass it**: explore doesn't + read the folder's link, so without `--project` it falls back to the project kane-cli is configured with, and a run + with none is refused. In a linked folder, pass the folder's project id (the preflight's `## link`). +- It queues the run on HyperExecute and exits; it doesn't wait for the exploration to finish. +- It uses the person's sign-in, turned into their username and access key. Only if it says it needs basic credentials, + ask the person to run `kane-cli login --username <user> --access-key <key>` in their own terminal. It stores the + sign-in and any secret variables in the person's HyperExecute Secrets, and the non-secret variables in Test Manager. + Never put values in chat. +- The default policy is `mutating`: the explorer may submit forms and change data on the sites it reaches, and in agent + mode kane-cli asks nobody. Before `explore run`, tell the person and ask whether it may change data in their app; if + not, pass `--policy read_only` (it keeps the explorer from changing data). A `mutating` run starts with a `warning` + (`code: "policy_mutating"`): relay it. +- Under `read_only`, the setup steps in `--deps` (a sign-in, for example) still run as written, so put in `--deps` only + what the person allows. +- An exploration spends credits while it runs, and so does each design job it starts (§4). Without `--max-leases` it + runs until it runs out of work. Tell the person before `explore run`, and offer a limit when credits matter (§2). +- A project that isn't shared yet is shared by the exploration's first design job, as the person who started the run. + If they may not create test cases in it, no tests are designed: `explore context status` shows `context.enabled: true` + with `context.off_reason: "mark_refused"`. Tell the person: "Ask a Test Manager admin for permission to create test + cases, or ask someone who has it to share the project." Once they have permission, turn context off and on (§4), only + on their word. + +**The stream:** `stream_start` (`surface: "explore"`), maybe `warning`, `explore_remote_started {exploration_id, run_id, +state, hx_job_id, job_link}`, then `done {state: "queued", exit_code: 0, exploration_id, run_id}`. Three plain lines on +stderr name the run, the job link and how to watch it. Give the person the job link. + +| Exit | Means | Do | +|---|---|---| +| 0 | queued | tell the person it runs on HyperExecute and can take a while; offer to check its status later | +| 1 | refused: `RUN_ACTIVE` (this exploration already has a run), or the launch failed | `RUN_ACTIVE`: wait, or ask the person whether to stop it (§3). A failed launch: try again in a minute; then tell the person | +| 2 | bad input (a different `--url` for the project's exploration, a `{{name}}` with no value, `--deps` with no `--allow-domain`), sign-in, a secret HyperExecute doesn't list (`SECRETS_MISSING`), or a message starting "could not push" or "could not read" | fix the input or the secret's name. If `done` names a run (`state: "stopped"`, `remark` `failed:secrets_missing`), report that run, fix the value, then start a new one. "Could not push / read": try again in a minute; then tell the person | + +Explore's `error` events carry a `message` only: no `code`, no `next`. Read the message. + +**On this machine** (`--local-orchestration`, below) the command streams the run and ends with it: `done` carries +`state` and `remark`. Exit 0 completed or reached its limit, 1 failed, 3 stopped or timed out, 2 couldn't start. + +## 2. Shape the run + +Add these to `explore run` only when the person's request needs them: + +| The person wants | Pass | +|---|---| +| An app behind a login | `--deps "Logged in as {{user}} / {{pass}}"`, with the values in a variables file (`.testmuai/variables/*.json`, or `--variables-file <path>`; a secret has `"secret": true`). `--deps` always needs `--allow-domain`: the app's own host, and every other host it signs in on (`--allow-domain app.example.com --allow-domain login.example.com`). Only the names the prompt and `--deps` use travel: secrets go to the person's HyperExecute Secrets, the rest to the organisation's Test Manager variables, where they replace a variable of the same name (tell the person). Names are letters, digits and `_`; `LT_USERNAME` and `LT_ACCESS_KEY` are taken. Check the values are real first: an empty secret stops the run `failed:secrets_missing`, and a sign-in that fails stops it `failed:setup_failed` | +| Only some sites | `--allow-domain <host>`, repeatable (the URL's host is always in, subdomains included) | +| A control it must never press | `--deny "<control name>"`, repeatable, under either policy | +| A `read_only` run of an app that reads its data through POST | `--allow-post <host/path>`, repeatable, only for an endpoint the person confirms only reads; its host must be the URL's or one passed with `--allow-domain` | +| Start from one screen of the map it already has | `--focus "<screen>"` (a later run; the screen's name or words from it). On this machine it also needs `--exploration-id <exploration-id>` | +| A limit on the explorer's work | `--max-leases <n>`: it stops after n pieces of work (leases), and the run ends `budget`. Every run also stops at built-in limits (an hour, 400 actions or 800 model calls), also `budget`. Neither limits the design jobs' credits: for a map with no new tests, see the next rows | +| More browsers at once | `--max-parallel <n>` (default 1; up to 20 cloud browsers, 4 with Chrome on this machine). Each may sign in once: keep 1 for an app that allows one login | +| A map, no tests | `--no-context`, on the project's first exploration only; for an existing one, `explore context off` (§4) | +| How deep it goes, what counts as slow, whether findings are checked again | `--depth-caps`, `--slow-budget`, `--no-recheck`: `kane-cli explore run --help` gives their values | +| An app on `localhost` or a private network | `--tunnel-name <name>`, through a LambdaTest tunnel on the person's machine (below) | +| To explore from this machine | `--local-orchestration` (Chrome here, `--headless` for no window; or `--browser remote` for cloud browsers). The command then waits for the run to end (§1), and Ctrl+C stops it | + +### An app on `localhost` or a private network + +The cloud's browsers can't reach it on their own: they reach it through a LambdaTest tunnel (LT) running on the person's +machine. kane-cli never starts or stops the tunnel; the person does, in their own terminal, so their key never comes +into chat. + +1. If `LT --version` doesn't run there, they download LT for their machine and put it on their PATH: + - Apple silicon Mac: `https://downloads.lambdatest.com/tunnel/v3/mac/arm64/LT_Mac.zip` + - Intel Mac: `https://downloads.lambdatest.com/tunnel/v3/mac/64bit/LT_Mac.zip` + - Linux: `https://downloads.lambdatest.com/tunnel/v3/linux/64bit/LT_Linux.zip` (`linux/arm64/LT_Linux.zip` on ARM) + - Windows: `https://downloads.lambdatest.com/tunnel/v3/windows/64bit/LT_Windows.zip`, then run `LT.exe` + + If a Mac blocks it from opening: `xattr -d com.apple.quarantine <path to LT>`. +2. They start it as the same LambdaTest user kane-cli is signed in as, and leave it running: + + ```bash + LT --user "$LT_USERNAME" --key "$LT_ACCESS_KEY" --env ht-prod --tunnelName <name> + ``` + + (`$LT_USERNAME` and `$LT_ACCESS_KEY` are their LambdaTest username and access key.) `--env ht-prod` is required. + Wait until LT says the tunnel is connected. +3. Run the exploration through it: + + ```bash + kane-cli explore run --project <id> --url http://localhost:3000 --prompt "<what to explore>" --tunnel-name <name> --agent + ``` + + `--tunnel-name` must match `--tunnelName` from step 2, and it turns `--tunnel` on; with only one tunnel running, + `--tunnel` alone is enough. A re-run keeps the exploration's URL but not its tunnel, so pass the tunnel flag again: + `kane-cli explore run --project <id> --tunnel-name <name> --agent`. + +- The tunnel **and that machine** must stay up until the run ends; the run warns about it when it starts. If the tunnel + drops mid-run, pages stop loading. +- A tunnel needs cloud browsers: with `--local-orchestration` and local Chrome, `--tunnel` is refused (local Chrome + already reaches the machine). + +## 3. Watch it, stop it, run it again + +```bash +kane-cli explore list --agent # the running explorations; --all lists every one +kane-cli explore sessions <exploration-id> --agent # its runs, newest first: state, remark, leases, findings, credits +kane-cli explore sessions <exploration-id> --run <run-id> --agent # one run in full +``` + +The person can watch a run live with `kane-cli explore view --exploration-id <exploration-id>` in their own terminal +(§4). + +A run is `queued` (waiting for its machine), `running` or `stopped`. A stopped run's `remark.code` says why: +`completed` (it ran out of work), `budget` (`--max-leases` or a built-in limit reached), `user_stop`, `stopped_before_start` (stopped while +queued), `heartbeat_lost` (it stopped reporting), `timeout`, or `failed:<reason>` (for example `failed:secrets_missing`, +`failed:setup_failed`: the sign-in in `--deps` didn't work, or `failed:vm_never_started`: HyperExecute never started it, +so run it again). Say it to the person in plain words. + +- **Stop it:** `kane-cli explore stop <exploration-id> --agent` stops a queued run at once, and asks a running one to + stop cleanly (its work in hand finishes and is kept). It waits for that, unless you pass `--no-wait`. It emits + `explore_stop {run_id, state, remark, stopping}`. When `stopping` is `true` the run hasn't stopped yet: check + `explore sessions` until it is `stopped` before you run it again or delete it. With nothing running it refuses + (`RUN_NOT_ACTIVE`, exit 1). +- **Run it again:** `kane-cli explore run --project <id> … --agent` once more. It goes on the same exploration, on its + URL, and adds to its map; a different `--url` is refused (exit 2). A run keeps only the URL: pass every option again + (`--policy read_only` above all, since the default is `mutating`, and the prompt, `--deps`, domains, variables, + limits and tunnel). +- **Explore another site:** use another project, so its tests don't mix with this site's. Or, only on the person's + word, `kane-cli explore delete <exploration-id> --yes --agent` (refused with `RUN_ACTIVE`, exit 1, while a run is + queued or running): no command shows that map again and the project can start a new exploration of any URL, but the + use cases and tests designed for the first site stay in the project; a pull brings them along with the new site's. +- `list`, `sessions`, `stop` and `delete` exit 0 when done, 1 when refused (`RUN_ACTIVE`, `RUN_NOT_ACTIVE`) or when the + service failed (read the message), and 2 for bad input, a missing `--yes`, an unknown exploration, or no sign-in. + +## 4. Check what it designed + +```bash +kane-cli explore context status <exploration-id> --agent +``` + +`explore context status` emits `explore_context {exploration_id, action, context}`. Read `context.enabled`, +`context.off_reason`, `context.running_job` and `context.pending_cut` (work waiting for the next job) as well as +`context.last_job`, the newest design job that has **ended**: `status` (`ok`, `noop`, `partial`, `failed`), +`use_cases.added[]` and `modified[]`, `tests.designed`, `tests.saved`, `tests.failed[]`, `tests.saved_cases[]` (each +test's Test Manager case), `held[]` (changes, such as an archive, the job left for a person to review), `credits` (that +job's own), `error`. A job runs after the exploration's run ends, and during a long run every few minutes when enough +changed. Before you call the designs ready, check that no job is still running or waiting. An entry of `saved_cases[]` +can have no case and a refusal even when the job says `ok`: tell the person which tests are saved but can't run yet, +and follow that refusal's remedy. + +```bash +kane-cli explore context sync <exploration-id> --agent # ask for a job now +``` + +It emits `explore_context_sync {exploration_id, state, …}`: `launched`, `launching`, `pending` (behind a running job), +`running` and `up_to_date` all end exit 0. A known refusal's `error` message starts with its code (other failures are +plain words: read the message and the exit code): + +| Refusal | Exit | Means | Do | +|---|---|---|---| +| `TOO_SOON` | 1 | the last design job started a few minutes ago | try again when it says | +| `NO_ENDED_RUN` | 1 | no run of this exploration has ended | wait for the run, then sync | +| `CONTEXT_OFF` | 1 | assurance context is off for this exploration | `kane-cli explore context on <exploration-id> --agent`, only with the person's word. If the message names a reason in brackets other than `user_off`, tell the person instead: turning it on helps only for `mark_refused`, once they have permission (§1) | +| `NOT_CONFIGURED` | 1 | design jobs aren't set up here yet; context stays on and runs once they are | tell the person; don't retry | +| a `failed` state | 1 | the job couldn't launch | try once more; then tell the person | +| sign-in or access | 2 | the sign-in was refused, or basic credentials are needed | as in §1: sign-in first, then once more | +| not found | 2 | no such exploration, or assurance context isn't available here yet | check the id (`kane-cli explore list --all --agent`); then stop | + +`kane-cli explore context off <exploration-id> --agent` stops new design jobs and drops the work waiting for one; +nothing already saved is deleted, and a job already running finishes (it may still save tests and use credits): check +the status and tell the person. `on` catches up from the newest ended run. Both only on the person's word. + +To show the map, ask the person to run `kane-cli explore view --exploration-id <exploration-id>` (or `--project <id>`) +in their own terminal: it opens the map in their browser and keeps serving it until they press Ctrl+C. There they can +look at the evidence, correct findings, leave notes, and mark screens or controls not to touch. Don't run it yourself: +it doesn't exit on its own. + +## 5. Get exploration's tests into a folder, and run them + +The use cases and tests are in the project, and their cases are in Test Manager, as soon as a job saves them. No +folder holds their files until it pulls them. + +- Read `shared-projects.md` §7a before the pull, and check first whether this kane-cli has it (`kane-cli context pull + --help` lists `--project`). With it: + - **a folder linked to the project:** `kane-cli context pull --mode agent`; + - **a fresh folder:** `kane-cli context pull --project <id> --mode agent` links it and brings the tests in one step. +- Read each `pull_test`'s `outcome`, and count the first runs from its `first_run: true` (a fresh folder's tests all + have it; a folder that ran them before may not). Fill the values `variables_needed` names: an app behind a login + needs its sign-in values in this folder too. A pull with no tests yet ends `done` exit 0: check + `explore context status <id>`. +- Without the pull (kane-cli 0.10.0), exploration's tests can be seen (`kane-cli context list --mode agent` in a linked + folder; Test Manager) but not run from a folder: no folder holds their files. +- The first run of each pulled test writes its steps in a real browser: minutes, and about 30–85 credits per test. Tell + the person the count and the likely cost, then run the files they choose: + `kane-cli testrun run <files> --agent < /dev/null`. +- Exploration's use cases are designed without review: in agent mode `context list` shows them as `trust: "derived"`. + Present them to the person as exploration's proposals. The source row's `id` is `explore-<exploration id>`. + `kane-cli explore list --all --agent` lists every exploration (without `--all`, only the running ones); a page's + `next_cursor` goes to `--cursor` for the next. +- After the first runs, `kane-cli cover --mode agent` in the linked folder shows the project's coverage, exploration's + use cases included. diff --git a/skill-installer/skills/references/fair-evaluation.md b/skill-installer/skills/references/fair-evaluation.md index 2b8f1e6..03908ef 100644 --- a/skill-installer/skills/references/fair-evaluation.md +++ b/skill-installer/skills/references/fair-evaluation.md @@ -9,7 +9,7 @@ When you compare kane-cli to anything else (an agent that generates Playwright s kane-cli has two phases with very different costs: - **Authoring** (first run, or after an edit): an AI agent works out each step → spends LLM tokens. One-time per step. -- **Replay** (every later run): executes the recorded action plan from local cache → **zero LLM cost**, runs in seconds. Cost is proportional to how often a test **changes**, not how often it **runs**. +- **Replay** (every later run, for each unchanged step that has a recording): executes the recorded action plan from local cache → **mostly avoids the authoring cost**, runs in seconds. A replay that heals re-authors the failing steps, so it costs more. Cost is proportional to how often a test **changes**, not how often it **runs**. (In a shared project, each folder that pulls a test authors it once, on its first run there: `shared-projects.md` §7a.) The most common evaluation error is pitting kane-cli **authoring** against the other approach's **execution**. That is apples-to-oranges. Compare the *same phase* on both sides: @@ -30,7 +30,7 @@ Compare kane **authoring** to the other approach's **creation effort** (the toke ## Mandatory corrections (real errors seen under time pressure) -- ❌ "kane spends LLM tokens on every test run." → **Wrong.** Tokens are spent only when a step is *authored* (first run or after an edit). Replay of unchanged steps is zero-LLM. State this before any cost claim. +- ❌ "kane spends LLM tokens on every test run." → **Wrong.** The authoring cost is spent when a step is *authored* (first run, after an edit, or when a replay heals). Replay of unchanged steps mostly avoids it. State this before any cost claim. - ❌ "The Playwright scripts already exist, so their creation cost is $0." → **Sunk-cost trap.** The creation effort was still spent (tokens or engineer hours). For an adoption/ROI decision, either count creation on **both** sides, or compare the **forward maintenance trajectory** — never zero out one side's creation while charging the other's. - ❌ "Free CI runs vs. tokens per test." → Phase mismatch (execution vs. authoring). Both execute cheaply; the real differentiators are creation effort and maintenance under UI churn. - ❌ Judging a single isolated run. → The right unit is the **suite over its lifetime** (many runs + UI changes). kane's advantage widens with run count and churn. diff --git a/skill-installer/skills/references/first-run.md b/skill-installer/skills/references/first-run.md index 0c4b960..aae2b93 100644 --- a/skill-installer/skills/references/first-run.md +++ b/skill-installer/skills/references/first-run.md @@ -2,6 +2,18 @@ # The first run +**In a folder linked to a shared project** (it holds `.testmuai/assurance.json`), or when the person's first request is +for a team's project, the first result is the shared-project journey, never a `kane-cli run`: `kane-cli run` saves to +the profile's project, and when that isn't the team's, its test then clashes with the team's tests +(`shared-projects.md` §9). The first result is +the list of use cases the first extract proposes; show it as the first result, then offer the same three choices. A +teammate joining a project whose documents are all read gets no proposal (`nothing_new`): show the project's +`context list` as the first result instead. In a linked folder, the "where results go" choice only changes where +`kane-cli run` saves; say so. Quick `kane-cli run` experiments go in a separate folder that isn't linked. + +**When the first request starts from requirements** (a PRD, a spec, a description) in a local folder, the first result +is likewise the use cases the extract proposes. + A person's first request should reach its first result with nothing standing in the way. The order is fixed: 1. Ready card (`references/ready-check.md`) @@ -73,6 +85,10 @@ Use the normal card from `references/cards.md`, and on this first run make two o From the second run on, the evidence viewer goes back to an offer. +When the first result is an extract's use cases (above), nothing has run yet, so there is no evidence and no test case: +show the use cases (in a linked folder, also whether the session is saved yet), with no 📁 or 🔗 row. Those rows come with the first test run. +Choice 1 below then reads "Runs open a browser window by default. Keep that, or run without a window?" + ## 4. Three choices, asked once, after the first result Ask these after the first payoff card. They read as tailoring, not as a toll gate, because the person has already seen a result. diff --git a/skill-installer/skills/references/live-strip.md b/skill-installer/skills/references/live-strip.md index cbe9dc2..0f5f2ee 100644 --- a/skill-installer/skills/references/live-strip.md +++ b/skill-installer/skills/references/live-strip.md @@ -27,8 +27,8 @@ If any of these is missing, do not offer the strip. Nothing else changes: the st - It **wraps the status line the person already has**: their line prints first, unchanged, and the kane line appears under it. - It appears **only while a run is live, and for five minutes after it ends**. The rest of the time the person sees exactly what they had before. -- It appears **only in the Claude Code session that started the run**. Other sessions show nothing, even when they are open in the same project. The reader tells sessions apart by checking that the run descends from the same session process it was started by. A run the person starts by hand in a terminal is not shown. On Windows, where that check is not available yet, every session open in the run's project shows it. -- It reads two things kane-cli writes on its own: a small pointer file for each live run, and that run's event log. It starts no process besides the person's original status line command, makes no network calls, and sends nothing anywhere. +- It appears **only in the Claude Code session that started the run**. Other sessions show nothing, even when they are open in the same project. The reader tells sessions apart by checking that the run descends from the same session process it was started by. A run the person starts by hand in a terminal is not shown. Where that check can't run (on Windows, or wherever the process list can't be read), every session open in the run's project shows it. +- It reads two things kane-cli writes on its own: a small pointer file for each live run, and that run's event log. It starts no process besides the person's original status line command and a local process-list check (`ps`), makes no network calls, and sends nothing anywhere. - **Typed text is never echoed.** A typing step shows as `typing in <field>`. - It refreshes every two seconds. It starts showing a run once kane-cli has created the session, which takes roughly 10 to 30 seconds after launch (the browser has to start first). Until then the person sees their normal status line. While a step is still working, the line shows the last finished action, marked `last:`. diff --git a/skill-installer/skills/references/mobile.md b/skill-installer/skills/references/mobile.md index f5f2e18..e790251 100644 --- a/skill-installer/skills/references/mobile.md +++ b/skill-installer/skills/references/mobile.md @@ -2,7 +2,7 @@ # Mobile testing (real devices from any machine; emulator/simulator locally on macOS Apple Silicon or on the cloud grid) -Desktop (the browser) is the **default** target and the primary use of kane-cli. Mobile is a **scoped addition**: kane-cli can also drive a native app on a **real Android or iOS device** in the TestMu AI cloud, or on a virtual Android or iOS device — on this machine, or on a LambdaTest HyperExecute macOS host via `testrun run --remote`. Nothing about web runs changes. A run with no mobile flags still drives Chrome exactly as before. +Desktop (the browser) is the **default** target and the primary use of kane-cli. Mobile is a **scoped addition**: kane-cli can also drive a native app on a **real Android or iOS device** in the TestMu AI cloud, or on a virtual Android or iOS device — on this machine, or on a LambdaTest HyperExecute macOS host via `testrun run --remote`. Nothing about web runs changes. A run with no mobile flags still drives Chrome exactly as before, unless a mobile target was saved for `kane-cli run` (`kane-cli config set-target`); a saved test follows its own `target:`. ## Availability (read first) @@ -12,7 +12,7 @@ Desktop (the browser) is the **default** target and the primary use of kane-cli. | **Local** (a simulator/emulator on this machine) | **macOS on Apple Silicon (arm64) only** — not Intel Macs, Linux, or Windows | `run --target …`, `testmd run`, `testrun run` | | **Cloud grid** (a virtual device on a HyperExecute macOS host) | **Any machine** — no Xcode / Android Studio needed. The account needs a HyperExecute plan with macOS runners | `testrun run <paths> --remote … < /dev/null` — see §Remote below | -- **Desktop stays the default.** The `--target` axis is what selects mobile. Leave it off and you get the browser. +- **Desktop stays the default.** The `--target` axis is what selects mobile. Leave it off and `kane-cli run` uses the saved target (`config show`), else the browser; a saved test uses its own `target:`, else the browser. Pass `--target desktop` when you mean the browser. - If the user is not on mac-arm64, local emulator/simulator is not an option — **offer a real device** (`--target android|ios`, works from any machine, plan permitting) or **the grid**: save the objective as a `_test.md` (`target: emulator|simulator` + `app:`) and run it with `kane-cli testrun run <path> --remote --device-name "<grid device>" --os-version <v> < /dev/null`. Do not tell them mobile is unavailable. ## The five targets @@ -60,7 +60,7 @@ Every mobile run needs an app. Provide it one of two ways: kane-cli installs that app on the device and runs the objective against it. `kane-cli apps list --target emulator|simulator|android|ios --agent` lists the account's uploads (NDJSON; the `app_id` field is what `--app`/`app:` take). There is no upload subcommand: any run with a local build — local or `--remote` — uploads it to the account (once per build; a per-machine cache skips a build the account already has) and prints the `APP…` id. Uploads belong to an organisation — `apps list` for the active profile is the authority. -**Not accepted:** a package/bundle id (e.g. `com.example.app`), a `.ipa` on a simulator (it is a device build — use `--target ios`), or a `.app` bundle. There is no default app: a mobile run without a valid build or `APP…` id cannot start. +**Not accepted:** a package/bundle id (e.g. `com.example.app`), a `.ipa` on a simulator (it is a device build — use `--target ios`), or a `.app` bundle. A mobile run without a valid build or `APP…` id (from `--app`, the file's `app:`, or one saved with `config set-app`) cannot start. ## Setup path @@ -143,7 +143,7 @@ Rules that differ from a local run (each violation is a `remote_error` before di | **Payload** | Members must live under the cwd being dispatched, and recordings must not be gitignored (un-ignore with `!output-*/`). Builds are not payload. | `member_outside_payload`, `gitignored_inputs` | | **Auth** | HyperExecute needs a LambdaTest username + access key; an OAuth profile is exchanged automatically, or pass `--username`/`--access-key`. Any `APP…` id must belong to the same organisation. | `no_basic_auth` | -What to present after `testrun_done`/`remote_done`: the suite rollup (per `references/testrun.md`), the device line (name + OS from `remote_device`; for simulators the pool, and each member's own pair), each build the laptop uploaded (`remote_app` path → `app_id`), and the job link (`remote_dispatched.job_url`) for the dashboard's stage logs. If a member comes back `broken` with zero steps, the grid-side run refused before launching — check the app id belongs to this account (`apps list` for the active profile) and the job's scenario log at the job link. +What to present after `testrun_done`/`remote_done`: the suite rollup (per `references/testrun.md`), the device line (name + OS from `remote_device`; for simulators the pool, and each member's own pair), each build the laptop uploaded (`remote_app` path → `app_id`), and the job link (`remote_dispatched.job_url`) for the job's step logs. If a member comes back `broken` with zero steps, the grid-side run refused before launching — check the app id belongs to this account (`apps list` for the active profile) and the job's scenario log at the job link. ## Real devices (`--target android|ios`) diff --git a/skill-installer/skills/references/objectives-cookbook.md b/skill-installer/skills/references/objectives-cookbook.md index a0a65de..94a935a 100644 --- a/skill-installer/skills/references/objectives-cookbook.md +++ b/skill-installer/skills/references/objectives-cookbook.md @@ -121,6 +121,8 @@ The agent automatically picks the right method based on phrasing. To get the met | **Title** | Browser tab `document.title` | "page title contains …", "title is …" | | **DevTools** | Things not visible on screen — network, console, performance, cookies, localStorage | see §3.2 below | +Under kane-cli's default assertion mode (`dom`), Visual and Textual reads both run DOM-first with a vision fallback; `--assertion-mode visual` keeps Visual reads on vision only (`references/execution-controls.md`). + ### 3.2 DevTools analyze methods Five subdomains. Each one is the right choice when the data you care about lives in the browser's internals rather than on the page. @@ -486,7 +488,7 @@ Positional assertions check where something is on the page: | "Test the checkout flow" | "Go to /cart, click Checkout, fill the address form with {{tester}}, click Pay, assert the order confirmation page loads" | "Test" has no end state — the agent doesn't know when to stop. | | "Add the item" | "Click the 'Add to Cart' button on the first product card" | Vague target — agent may click the wrong element. | | "Tell me the price" | "Store the cart total as 'total'" | Vague verbs don't extract — use "store" / "extract" / "get". | -| Hardcode credentials in the objective | Use `{{username}}` / `{{password}}` from `--variables-file` | Credentials in plain text leak into logs and TMS. | +| Hardcode credentials in the objective | Use `{{username}}` / `{{password}}` from `--variables-file` | Credentials in plain text leak into logs and Test Manager. | | Omit the URL | "Go to https://example.com/login first, then …" | Agent doesn't know where to start. | | Cram 25 operations into one objective | Split at logical boundaries (login, navigate, action, verify) | Long runs drift and stall. | | "Check the page is fast" | "Assert LCP is under 2500ms and CLS is below 0.1" | Use the explicit web-vital metric, not a vague "fast." | diff --git a/skill-installer/skills/references/parallel.md b/skill-installer/skills/references/parallel.md index 0866044..fdc63ca 100644 --- a/skill-installer/skills/references/parallel.md +++ b/skill-installer/skills/references/parallel.md @@ -4,7 +4,7 @@ For multiple independent browser tasks, decompose and run in parallel using the Agent tool. -> **Saved tests? Use testrun instead.** If the tasks are committed `_test.md` files, do NOT hand-roll parallelism — `kane-cli testrun run --parallel N < /dev/null` gives you isolated Chromes, a pooled scheduler, one rollup, and one evidence pack. Read `references/testrun.md`. This reference is for **ad-hoc `run` objectives** only. +> **Saved tests? Use testrun instead.** If the tasks are committed `_test.md` files, do NOT hand-roll parallelism — `kane-cli testrun run --parallel N < /dev/null` gives you isolated Chromes, a pooled scheduler, one rollup, and one evidence pack. Read `references/testrun.md`. This reference is for **ad-hoc `run` objectives** only. In a folder linked to a shared project, these runs go to the profile's Test Manager project, which may not be the shared one: run them from a folder that isn't linked (`shared-projects.md` §9). ## When to Split diff --git a/skill-installer/skills/references/parsing.md b/skill-installer/skills/references/parsing.md index 823702a..2b5a971 100644 --- a/skill-installer/skills/references/parsing.md +++ b/skill-installer/skills/references/parsing.md @@ -1,4 +1,4 @@ -<!-- Read this when you need the full NDJSON event schema for kane-cli --agent output, or when SKILL.md §5's summary is not enough. Owns event types (progress / bifurcation / child_agent_start|end / ask_user / error), parsing strategy, run_end terminal-event schema, and ask_user/cancel responses. --> +<!-- Read this when you need the full NDJSON event schema for kane-cli --agent output, or when SKILL.md §5's summary is not enough. Owns event types (progress / bifurcation / ask_user / error / coverage), parsing strategy, run_end terminal-event schema, and ask_user/cancel responses. --> # Parsing --agent Output @@ -21,14 +21,14 @@ The first line on every surface is an opening event: {"type":"stream_start","cli_version":"0.8.17","surface":"run","pid":16664,"v":1,"ts":"2026-09-21T08:47:26.889Z"} ``` -`surface` is `run`, `testmd` or `testrun`. Use `cli_version` to tell whether a newer event or flag is available. `session_dir` may also be present when a session already exists. +`surface` is `run`, `testmd`, `testrun` or `explore` (`references/explore.md`; it ends with `done`). Use `cli_version` to tell whether a newer event or flag is available. `session_dir` may also be present when a session already exists. Rules a parser must follow: - **Ignore unknown fields and unknown event types.** New ones can appear in any release without a `v` bump. - **Never assume the first line is a progress line**, and skip any line that is not JSON. - Step lines on `run` stay **typeless** (below). Do not look for `type: "step"`. -- The documented completion event is always the last line: `run_end` for `run`, `test_md_done` for `testmd run`, `testrun_done` for `testrun run` (then `remote_done` on cloud grid runs). +- The documented completion event is the last line of a run that went ahead: `run_end` for `run`, `test_md_done` for `testmd run`, `testrun_done` for `testrun run` (then `remote_done` on cloud grid runs). A run refused before it starts can end on an `error` with no completion event: read the exit code. **The same stream is also written to disk**, line by line as it happens: `<session_dir>/events.ndjson`, byte for byte what stdout printed. While a run is live, kane-cli keeps a small pointer file at `~/.testmuai/kaneai/sessions/active/<pid>.json` (`pid`, `cwd`, `surface`, `session_dir`, `started`, `cli_version`, `host_agent`) and removes it on exit. You normally need neither: one blocking call hands you the whole stdout. They exist for watchers such as the live strip (`references/live-strip.md`), and the log is where a suite keeps each test's own events (`references/testrun.md`). The log holds exactly what stdout held, so treat it with the same care. @@ -48,6 +48,7 @@ Rules a parser must follow: | `step` | number | Step index. It can run one ahead of the step the person would count (a `bifurcation` takes the first slot), so count completed `done`/`failed` lines for "steps taken" rather than reading the last index | | `status` | string | `"running"` at start; `"done"` or `"failed"` at completion | | `remark` | string | What the agent did or why it failed | +| `child_id` | string, optional | The step belongs to a child flow. Keep children apart when step indices repeat | These are **untyped** — they have no `type` field. Do **not** key on `event.type === 'step_start'` or `'step_end'`; those event types are not emitted. @@ -57,13 +58,14 @@ These are **untyped** — they have no `type` field. Do **not** key on `event.ty |-------|-----------|---------| | `project_folder_auto_defaulted` | resolved project + folder (id, name) | Run-startup gate auto-resolved a project/folder when none was configured (or the cached one was stale/invalid). Fires before any progress event on `run` / `testmd run`. Translate to plain language (see `references/test-manager.md`). | | `bifurcation` | `flows[]`, `count` | Agent split objective into sub-flows | -| `child_agent_start` | `child_id`, `objective`, `parent_step` | Child agent spawned | -| `child_agent_end` | `child_id`, `success`, `steps_taken`, `summary` | Child agent finished | | `ask_user` | `question`, `step_index`, `options?` | Agent needs user input | -| `error` | `message` | Error occurred | +| `error` | `message` | Error occurred. On `testmd run` / `testrun run` in a folder linked to a shared project it also carries a `code`, and `remedy`, `next` or `support_id` when kane-cli has one (`agent-contract.md` §3.7) | | `warning` | `code`, `message`, … | *(0.8.15+)* Before any progress event: `code: "unresolved_variables"` — a `{{name}}` had no value; the run continues. Schema below. | | `test_md_evidence_ingest` | `status: "ok"\|"failed"`, `evidence_id`, `stage?` (failure only) | `testmd run` only: a replay's evidence pack published to the dashboard. Informational. | | `test_md_bundle_sync` | `status: "ok"\|"failed"`, `commit_id`, `bytes?` (success) / `stage?` (failure) | `testmd run` / `testmd sync`: test bundle pushed to the cloud after an authored commit. Informational. | +| `coverage` | `status` (`written`, `none`, `skipped`, `failed`), `pct?`, `reason?`, `pack` | `run` / `testmd run`: one per sealed pack, just before the completion event, which repeats it as `coverage`. `skipped` (nothing to measure here, as in a folder with no assurance work): say nothing. In a linked folder: `agent-contract.md` §3.7 | +| `update_available` | `current`, `latest`, `severity` | `run` (also `generate --agent`), at any point before `run_end`, or not at all: a newer kane-cli exists. Mention it once, after the result | +| `skill_update_available` | `current`, `latest`, `install_cmd` | `run` only, like `update_available`: a newer skill exists. After the result, offer `kane-cli install skill` once (`references/ready-check.md` §4a) | | `testrun_*` family | see `references/testrun.md` | Emitted only by `kane-cli testrun run`; terminal event is `testrun_done`, not `run_end`. | **Note:** The `run` stream has no `run_start` event; startup metadata or errors can precede progress. @@ -88,7 +90,7 @@ Emitted by `run`, `testmd run` and (after `testrun_plan`) `testrun run` when an | `variables[].reason` | `value_missing` — the key exists in `variables[].file` with an empty value · `not_declared` — the key is in no variable file · `not_a_dataset_column` — a `${x}` that is no column of the Test Manager dataset the run was given (`variables[].dataset` names it) | | `variables[].file` | the pool file that holds the empty key (`value_missing` only); `--variables` when it came inline | | `variables[].used_by[]` | `{file, step}` — `file` is `objective` for `kane-cli run`, else the test file (flattened step index) | -| `suggested_file` | where to add a `not_declared` key: `.testmuai/variables/assurance.json` inside an assurance store, `variables.json` otherwise | +| `suggested_file` | where to add a `not_declared` key; use it as given: `.testmuai/variables/assurance.json` in a folder with its own assurance store (`.context/`), `.testmuai/variables/variables.json` otherwise (a linked folder too: `shared-projects.md` §5) | Report it with the run's outcome. If a later step failed on a literal placeholder, point at this. diff --git a/skill-installer/skills/references/ready-check.md b/skill-installer/skills/references/ready-check.md index cc5569b..c72becb 100644 --- a/skill-installer/skills/references/ready-check.md +++ b/skill-installer/skills/references/ready-check.md @@ -25,7 +25,12 @@ Add a flag only when the request needs it: | A local mobile run | `--mobile emulator` or `--mobile simulator` | `## mobile` (device tooling readiness) | | A cloud grid suite (`--remote`) | `--grid` | `## grid` (grid plugin readiness) | -The script only reads status. It changes nothing, takes about two seconds, and always exits `0`. If the script is missing (an older skill install), run `kane-cli whoami`, `kane-cli balance` and `kane-cli config show` yourself and build the same card. +The script only reads status (its sign-in check may renew a saved sign-in that is about to expire, as any kane-cli command does). It changes nothing else, takes about two seconds, and always exits `0`. + +If you can't run the script (it isn't there in an older skill install, or your host doesn't allow it), run the same +checks one at a time and build the same card: `kane-cli --version`, `kane-cli whoami`, `kane-cli balance`, +`kane-cli config show`, and look for `.testmuai/assurance.json` in the folder (only its `projectName` and `projectId`). +The preferences file is then unknown: treat the session as a first session only if the person says it is. ## 2. What the script prints @@ -34,9 +39,10 @@ Plain text in `## <section>` blocks, always in this order. Command blocks end wi | Section | Content | What you take from it | |---|---|---| | `## version` | kane-cli version, or `missing` | Installed or not. Compare with the minimum version this skill notes for a feature | -| `## whoami` | The sign-in box | `Authenticated` plus `User`, `Environment`. **Ignore `Expires`**: it is a short-lived token that renews itself, never show it | -| `## balance` | `Available credits` and `Total credits` | Credits left, rounded to a whole number | +| `## whoami` | The sign-in box: a header (`✓ Authenticated`, `✗ Not authenticated`, `⚠ Couldn't verify` or `⚠ Not logged in`), then rows such as `Profile`, `Method`, `User`, `Token` | The header and `User`. Never show the `Token` row | +| `## balance` | `Available credits` | Credits left, rounded to a whole number | | `## settings` | Settings as JSON | `project_name`, `folder_name`, `target`, `default_url` | +| `## link` | `project=<name>` and `project_id=<id>` when this folder is linked to a shared project, or `none` | Linked or not; see §3's 🔗 row and `shared-projects.md` | | `## agent-config` | The preferences file, or `none` | See `references/agent-config.md`. `none` or no `onboarding.completed_at` means this is a first session | | `## env` | `ci`, `ssh`, `display`, `os`, `arch`, `node` | Watch-mode default and whether a human is present | | `## chrome` | `found=<path>` and `override=<path>` | Chrome present for local browser runs | @@ -82,9 +88,14 @@ Row rules: - **🚀 Your app** appears only when the `app` section found a port. No row when nothing was found: never show a negative row for an optional finding. Ask for a URL only when the request lacks one. - **🌐 Chrome** appears only for local browser runs. Skip it for mobile and cloud grid requests. +- **🔗 Shared project**: when the preflight's `## link` section names a project, this folder is linked to it + (`shared-projects.md`). Add a row `🔗 **Shared project** | <name> (<id>) · designed tests run and count here`, and + word **Results go to** as "Quick `kane-cli run` checks save to <project> / <folder>; runs of the designed tests here count in <the shared project>". Read `shared-projects.md` before any assurance command. + A `link` of `unreadable` means kane-cli can't read the folder's link file, and a linked folder that also holds + `.context/` is refused too: run `kane-cli context list --mode agent` and relay its fix (`PROJECT_FILE_UNREADABLE`, `BOTH_STORES`) + before any assurance work. - **🗂️ Results go to** comes from `project_name` / `folder_name`. When they are empty, say `kane-cli will pick a default project on this run, and I'll tell you where it landed`. The offer to change it never stops the run. The change flow is in `references/test-manager.md`. - **👀 This run** states the watch mode you are about to use: the saved `preferences.watch`, or the detected default (see `references/first-run.md`). -- Name the environment (for example `stage`) only when it is not production. - For mobile or grid requests add a `📱 **Device tooling**` or `☁️ **Cloud grid**` row from the extra section. ## 4. Problems: which ones stop the run @@ -92,7 +103,8 @@ Row rules: | Problem | How you see it | Stops the run? | The fix the card offers | |---|---|---|---| | kane-cli not installed | `## version` is `missing` | Yes | Offer to run `npm install -g @testmuai/kane-cli` (or Homebrew) | -| Not signed in, or token not valid | `whoami` shows no `Authenticated`, or `exit` is not 0 | Yes | Sign-in flow below | +| Not signed in, or token not valid | `whoami` header is `✗ Not authenticated` or `⚠ Not logged in` (exit 1) | Yes | Sign-in flow below | +| Couldn't verify the sign-in | `whoami` header `⚠ Couldn't verify` (exit 3): TestMu AI didn't answer | Yes, for now | Signing in again won't help: check the connection and run the preflight again in a minute | | No credits left | Available credits is 0 | Yes | Point to https://www.testmuai.com/pricing/ to pick a plan | | Chrome missing | `found=` is empty, local browser run | Yes | Install hint for the platform, or `KANE_CLI_CHROME_PATH` for a custom location | | Low credits | Available credits under 100 | No | One warning line on the card | @@ -102,6 +114,28 @@ Row rules: When a problem stops the run, do not launch. Show the card, offer the fix, and wait. +## 4a. Get or update the skill + +The skill ships as its own package, separate from kane-cli. Most kane-cli commands (not `context`, `explore`, or anything +run with `--mode agent`) may print a +`Skill update available: <installed|not installed> → <version> — run \`kane-cli install skill\`` line on stderr. A +`kane-cli run --agent` stream also carries the notices as events on stdout: `skill_update_available` (`current`, +`latest`, `install_cmd`), so offer `kane-cli install skill`; `update_available` (`current`, `latest`, `severity`) is +kane-cli's own update (`cards.md` §1). + +```bash +kane-cli install skill # the newest skill this kane-cli supports +kane-cli install skill <version> # exactly that version, e.g. one your team's guide names +``` + +- It writes the skill for Claude Code, Codex CLI and Gemini CLI at once (`~/.claude/skills/kane-cli/`, + `~/.agents/skills/kane-cli/`, `~/.gemini/skills/kane-cli/`). Restart the agent to load it. +- With no stdin terminal it prints one JSON line: `{"type":"skill_install_result","status":"installed|already-installed|failed","version":…,"installed_targets":<n>,"error":…}`; exit 1 on `failed`. +- A pinned version installs even when it is older or newer than the default. A later bare `kane-cli install skill` never + downgrades, so it keeps a newer pinned version in place. +- `kane-cli install skill` doesn't write the preferences file `agent-config.md` reads, so the first session still counts + as a first session. + ## 5. Sign-in - **Default:** offer to open the sign-in page, then run `kane-cli login --oauth` yourself with a generous timeout. It opens the browser, waits for the person to finish, and returns. It works without a TTY. diff --git a/skill-installer/skills/references/setup-and-config.md b/skill-installer/skills/references/setup-and-config.md index 1800ccc..2e80974 100644 --- a/skill-installer/skills/references/setup-and-config.md +++ b/skill-installer/skills/references/setup-and-config.md @@ -4,6 +4,7 @@ ## Install and auth + Before first use, verify installation and auth. > **Local mobile setup (macOS Apple Silicon)** is separate: install Xcode 16+ (iOS) or Android Studio with one `arm64-v8a` AVD (Android), then `kane-cli login` and `kane-cli doctor --target emulator|simulator --install` (installs kane-cli's managed test tooling); `kane-cli doctor --target <kind>` checks readiness. Not needed for `testrun run --remote`, which runs mobile suites on the cloud grid from any machine (see §Remote execution plugin), nor for **real devices** (`--target android|ios`), which need only `kane-cli login` and a KaneAI Max or KaneAI credits + Real Device Plus Automation Cloud plan. Read `references/mobile.md`. @@ -22,7 +23,7 @@ The preflight script covers this along with credits and settings in one call (`r kane-cli whoami ``` -`whoami` prints a box, not JSON, even when piped. Its `Expires` line is a short-lived token that renews itself: never show it to the person. +`whoami` prints a box, not JSON, even when piped. Show the person only its header and the `User` row; never the `Token` row. If this shows "not configured" or errors, run login: @@ -35,7 +36,7 @@ kane-cli login --username <user> --access-key <key> This creates the default profile with basic auth, auto-selects the KaneAI project, and marks setup complete. Credentials come from the user's TestmuAI dashboard (Settings → Keys). Optional flag: -- `--profile <name>` — profile name (default: last selected profile check using `config show`) +- `--profile <name>` — profile name (default: `default`; pass a name to sign in to another profile) ### Login (OAuth) @@ -79,6 +80,10 @@ Variables parameterize objectives with reusable values and secrets. Use `{{key}} 2. `{cwd}/.testmuai/variables/*.json` (local project overrides) 3. `--variables-file <path>` 4. `--variables '{...}'` (inline JSON) +5. a saved test's own `variables:` frontmatter (`testmd run`, `testrun run`) +6. a step's own `variables:` (that step only) + +A Test Manager dataset row's `${col}` values also win over a pool entry of the same name. When a value you changed in a pool file doesn't reach the test, look for a higher layer that sets the same name. **Values (0.8.12+):** a string is a value when non-empty; a number is accepted and loaded as its string; a boolean, object or array is not a value. An empty `value` is a declared-but-unfilled key. @@ -128,9 +133,8 @@ kane-cli feedback --test-id <id> --feedback-type <positive|negative> --details " ├── config.json # Shared auth configuration ├── global-memory.md # Global agent context ├── chrome-profile/ # Default Chrome user profile -├── profiles/ # Stored credentials -│ └── {profile}/{env}/ -│ └── credentials +├── profiles/ # Stored credentials, one folder per profile +│ └── {profile}/ ├── sessions/ # Session history │ └── {session-id}/ │ ├── session.json # Metadata, run list, upload status diff --git a/skill-installer/skills/references/shared-projects.md b/skill-installer/skills/references/shared-projects.md new file mode 100644 index 0000000..24eb356 --- /dev/null +++ b/skill-installer/skills/references/shared-projects.md @@ -0,0 +1,281 @@ +<!-- kane-cli skill reference: shared projects (a folder linked to a team's Test Manager project). Read before any assurance command when the person works with a team, or when a folder holds .testmuai/assurance.json. Requires kane-cli 0.10.0+. --> + +# Shared projects: a team's work in one Test Manager project + +A **shared project** is a Test Manager project that a team's folders are **linked** to. In a linked folder the assurance +commands (`context`, `design`, `maintain`, `cover`) work in the project itself, not in a local `.context/` store. What one +person saves, everyone on the team sees: use-cases, criteria, scenarios, tests, Test Manager cases, coverage. + +Read this before the first assurance command of a session whenever: +- the person mentions a team, a teammate, a Test Manager project, or "our project"; +- the folder holds `.testmuai/assurance.json` (it is linked; the file names its `projectId` and `projectName`); +- a command prints `using project "<name>", shared with your team`, or emits `folder_linked`. + +The commands below are the agent forms. Everything in a linked folder runs from **the folder that holds +`.testmuai/assurance.json`**: a subfolder of a linked folder is not linked. + +## 1. Where the work goes: decide first + +In agent mode, a folder that isn't set up yet goes **local without asking** unless you pass `--project <id>`. So decide +before the first `context ingest`: + +| The person… | Do | +|---|---| +| works in a folder that already has its own `.context/` (and no link) | keep working locally, as before (`assurance.md`); to share that store, see below | +| works with a team, or names a Test Manager project | link this folder to that project (§2–3) | +| works alone, or asks to keep it on this machine | work locally: the journey in `assurance.md` §1 | +| isn't clear | ask once: "Should this work go to your team's Test Manager project, or stay in this folder?" | + +A folder is one or the other, never both: a folder with its own `.context/` refuses `--project` (`FOLDER_CHOSEN`), and a +folder with both refuses every assurance command (`BOTH_STORES`). To share an existing local store with a team through a +Git repository, a bucket or a folder instead, see `context-sync.md`; that is a different way of sharing and doesn't reach +Test Manager. + +## 2. Find or make the project + +```bash +kane-cli projects list --search "<part of the name>" --agent +kane-cli projects create "<name>" --description "<one line>" --agent +``` + +- `projects list` prints one JSON line per project, `{"id":"<project id>","name":"<name>"}`, then a page line + `{"_meta":"page",…,"has_more":true|false}`. There is no `type` and no `done`. `--limit` (1–200, default 10) and + `--offset` page through; `--search` matches any part of the name. +- `projects create` prints `{"id":…,"name":…}`. It does **not** link the folder and doesn't change where `kane-cli run` + saves. Create a project only when the person asks for a new one; otherwise use the team's. +- Errors from `projects` are plain text on stderr (exit 2 for usage or sign-in, 1 for a Test Manager refusal), not events. +- **When the shared path isn't open to the person:** if `projects list` is refused (exit 1: Test Manager refused or + failed the request), or it lists no project the person can work in and they don't want a new one, work locally + (`assurance.md` §1). Tell the person why, in plain words, for example "I couldn't reach your Test Manager projects, + so this work stays in this folder for now". Try the shared path once only; never retry it in a loop. +- A linked folder's own project id is in `.testmuai/assurance.json` (`projectId`). + +### This folder's project id + +A teammate's agent needs it to join (`kane-cli context pull --project <id>` where the pull exists, or +`kane-cli context extract --project <id>`), and so does anything that reads the project, such as a dashboard. Give the +person the name and the id together. Ways to get it: + +| Where | What | +|---|---| +| a linked folder's `.testmuai/assurance.json` | its `projectId` field (and `projectName`); read only those two fields | +| the skill's preflight | its `## link` section: `project=<name>`, `project_id=<id>` | +| the `folder_linked` event | `project_id` and `project`, when a command links the folder | +| `kane-cli projects list --search "<name>" --agent` | `{"id","name"}` for every project the person can open | + +## 3. Link this folder + +```bash +kane-cli context ingest <files> --project <id> --mode agent # link, then add the documents and find their use-cases +kane-cli context extract --project <id> --mode agent # link and read what the project hasn't read yet +``` + +- `context ingest`, `context extract` and, where its help lists it, `context pull` (§7a) take `--project`. The link is + made once; later commands in the folder need no flag. `--project` with the folder's own project is harmless; a + different id refuses `FOLDER_CHOSEN`. +- `{"type":"folder_linked","project_id":…,"project":…}` confirms the link. It can come before a check that still refuses + the run, so read the events after it. Tell the person: "This folder is now linked to <project> (<project id>); what you + save goes to the project." +- If the project refuses its first session (`PERMISSION_REQUIRED`: it needs someone allowed to create test cases), the link + is removed again; ask the person to get someone allowed to create test cases in that project to start the first + session, or to pick another project. After `PROJECT_NO_ACCESS` the folder **stays linked**, and every + assurance command refuses the same way until the person gets access or removes `.testmuai/assurance.json` (their call). + A `context pull --project` that is refused at the start, or can't read the project, removes the link it made: check + the folder again after one. +- A person at a terminal is asked instead: "where should this folder's work go?", then a project picker (Tab makes a new + project). Agents never see these questions. +- There is no unlink command. Linking a folder to another project means removing `.testmuai/assurance.json` first; that + is the person's call, never yours. + +## 4. The journey in a shared project + +| Step | Command | What to watch | +|---|---|---| +| 1. Add the documents and find their use-cases | `kane-cli context ingest <file> --project <id> --mode agent` (the first time), later `kane-cli context ingest <file> --mode agent` | one document per run; it ends `session_complete` with two `next` commands, save and discard (none when it added nothing) | +| 2. Save the session | `kane-cli context sessions save <sid> --yes --mode agent` | **nothing is in the project until this runs.** `session_save` `phase: "saved"` confirms it | +| 3. The use-case checkpoint | `kane-cli context list --mode agent` to show them; then ask once (below) | use-cases an agent run saves are `proposed`, not approved | +| 4. Design the tests | `kane-cli design tests --use-case <uc> --mode agent --allow-unreviewed` | only after the person's yes; ends `session_complete`. To cap the scenarios (and the cost), offer `--max <n>`; add it only on the person's word | +| 5. Save the design | `kane-cli context sessions save <sid> --yes --mode agent` | writes the test files here and makes one Test Manager case per scenario; `test_files.written[]` lists the files. Before going on, read `test_files.refused[]` and `test_files.error`, and `test_manager.error` and `waiting`: a save can succeed while a file or a case is still missing. Files listed in `test_files.replaced[]` stay but won't run any more: name only the `written[]` files on `testrun run`, and offer to move the replaced ones out | +| 6. Fill the variables, and the start pages | the file in `test_files.variables_needed.file` (usually `.testmuai/variables/assurance.json`); `test_files.no_start_page[]` names tests with no start page | SKILL.md §3, before any run. Use only values the person gives you, never invented ones; write them into the file (a script is fine) without printing them to the terminal or a log. Ask the person for the site address of a test with no start page | +| 7. Run the tests | `kane-cli testrun run <the files from step 5> --agent < /dev/null` | the first run of each test writes its steps: minutes, and about 30–85 credits per test; replays are cheap (`testrun.md` §Designed tests in a shared project) | +| 8. Coverage | `kane-cli cover --mode agent` | the project's figure; a new run can take up to 5 minutes to count | + + +**The use-case checkpoint.** A linked folder has no command that approves use-cases: +`context review` is refused there, and use-cases an agent run saves stay `proposed`. So `design tests` refuses +`UC_UNREVIEWED` until it gets `--allow-unreviewed`. Show the person the use-cases you found (titles and what they cover, +from `context list --mode agent`), then ask **once for the batch**: "I found these N use-cases in <document>. Shall I design tests for them? Nobody has +approved them yet, so tell me any to leave out or change first." Pass `--allow-unreviewed` only after a yes, and only for the use-cases they keep. A person at a terminal approves them by finishing the +session there. + +**What things cost.** These are approximate figures from real runs of small documents; the person's billing page is +the authority. On the assurance commands (`context ingest`, `context extract`, `design tests` …) the stream's +`usage.total_credits` gives the real figure. A local `testrun run` stream carries no credits total (not on `testrun_done`, +a member's end event, `testrun_summary` or `coverage`), so run `kane-cli balance` before and after the run and give the +difference: + +| Step | Credits (approximate) | +|---|---| +| `projects list`, `projects create`, `context list`, `context sessions`, `cover`, a save, `explore list`, `explore context status` | 0 | +| linking (`context extract --project` when every document is already read) and `context pull` | 0 | +| reading one document (`context ingest`, or `context extract` with something unread) | about 5 | +| designing one use-case's tests (`design tests`) | about 10–15 | +| a designed test's first run in a folder (it writes its steps; each input it checks is its own browser run) | about 30–85 per test | +| a replay | a few; a replay that has to heal, or whose steps changed since the recording, writes those steps again and can cost up to a first run | + +Tell the person the likely cost before a step that costs credits, let them choose, and give the total after. Never add +a flag that changes cost or behaviour (for example `--no-adaptive-heal`, `--max`, `--force`) on your own: offer it, and +use it only when the person asks. + +**One session at a time.** A live session holds its source or its use-case; another person's start on the same one is +refused `PROJECT_BUSY`, saying whether the session in the way is the person's own (`held_by.mine`) or a teammate's. Save or discard each session before the next one, and tell the person when a +teammate's session is in the way. `kane-cli context sessions --mode agent` lists the project's open sessions (one JSON +array; `owner` says whose). + +## 5. Where everything lives + +| What | Where | How to see it | +|---|---|---| +| sources, use-cases, criteria, scenarios, tests | in the shared project | `kane-cli context list --mode agent` lists sources, use-cases, scenarios and tests (one plain JSON row each: `id`, `label`, `title`, `trust`, `fresh`; no `done`); criteria aren't listed there. `trust: "derived"` is a proposal nobody has approved yet, `"trusted"` is approved: say *proposed* or *approved* to the person, never the field's value | +| test files (`*_test.md`) | `.testmuai/tests/` of **the folder where the session was saved**, and nowhere else | `session_save.test_files.written[]` | +| Test Manager cases | one per scenario, in the shared project, made or updated when a design is saved | `session_save.test_manager`: counts `created`, `updated`, `already_there`, `waiting`, and `next_run[]` (the scenarios whose case updates on their next run) | +| a test's runs | on its scenario's Test Manager case | Test Manager; the run's `testrun_summary` | +| coverage | worked out by the shared project from the counted runs | `kane-cli cover --mode agent` | +| variables | names: in the saved tests; empty stubs: this folder's `.testmuai/variables/assurance.json`; values: whatever the person writes there | `test_files.variables_needed` | +| a run's recording | `.testmuai/tests/output-<test>/` beside the test file, after a first run that finished | the next run replays it cheaply; a first run that stopped early leaves it incomplete, and the next run writes the steps again | + +Two rules follow: +- **Keep designed test files directly in `.testmuai/tests/`.** A designed test moved into a subfolder runs as an ordinary + test, not on its Test Manager case, and doesn't count for coverage. +- **Variable values reach the whole organisation.** Every run sends the non-secret values it loads to Test Manager: the + global and folder variable files, `--variables` and `--variables-file`, and the test's own `variables:`. Test Manager + keeps one value per name for the organisation, so a teammate's run with a different value replaces it. Never fill a + value just to silence a warning. Any `*.json` in `.testmuai/variables/` counts; prefer the file `variables_needed` names. + +## 6. A teammate joins + +From the teammate's own empty folder. Where `kane-cli context pull --help` lists `--project` (§7), join with the pull: +it links the folder, reads no documents, costs nothing and brings the project's tests (§7a): + +```bash +kane-cli context pull --project <id> --mode agent +``` + +Otherwise join with the extract: + +```bash +kane-cli context extract --project <id> --mode agent +``` + +It links the folder, then: +- every document already read: `nothing_new` (`reason: "every_source_read"`), exit 0, no credits. That needs the project + to answer within a few seconds; when it doesn't, the extract goes on and opens a session: save or discard it, as below; +- a document nobody has read yet: it reads it (credits, see §4) and ends `session_complete`. Save that session (§4 step 2), + or discard it on the person's word; while it is open it holds that document, and teammates get `PROJECT_BUSY`; +- another person's session already holds the unread document: `nothing_new` (`reason: "sources_in_a_session"`) with + `open_sessions[]`; nothing to do here. + +After the extract, `context list` and `cover` show the project. **No test files arrive with the extract's link**, so +`testrun run` has nothing to run yet: see §7. + +## 7. Get the team's tests into this folder + +Test files are written only in the folder where a design was saved. Exploration saves its tests into the project, but its +files never reach anyone's folder. + +**First check whether this kane-cli can pull them:** + +```bash +kane-cli context pull --help +``` + +If its options include `--project` and `--use-case`, use §7a. If not (kane-cli 0.10.0), it can't: `context pull` in a +linked folder is refused (`REMOTE_UNSUPPORTED`), and `--project` is an unknown option. Use §7b. + +### 7a. Pull them (when `context pull --help` lists `--project`) + +After exploration finishes, or after a teammate saves a design: + +```bash +kane-cli context pull --mode agent # in a linked folder: write the project's tests here +kane-cli context pull --project <id> --mode agent # in a folder that isn't set up yet: join (reads no documents), then pull +kane-cli context pull --use-case <uc> --mode agent # only that use-case's tests, with the tests they use +kane-cli context pull --overwrite --mode agent # replace test files changed here with the project's +``` + +- The pull's own events and refusals carry `verb: "sync"`; a refusal of the pull's flags (`PULL_LOCATION_LINKED`, `PULL_FLAGS_MIXED`, `PULL_FLAGS_LINKED_ONLY`) or at the folder's entry (sign-in, access, the link) carries `verb: "pull"`. Dispatch on `type`, as always. +- One `pull_test` event per test: `{test, title, file, outcome, first_run?, imported?, reason?}`, where `outcome` is + `new`, `updated`, `unchanged`, `kept` (changed here, left alone), `replaced` (with `--overwrite`) or `not_written`. +- `first_run: true` means the test hasn't run in this folder: its first run writes its steps in a real browser, which + takes a few minutes and costs credits (§4); later runs replay. Tell the person the count and the cost first. +- `variables_needed {file, variables[], no_start_page[]}` names the variables to fill; the pull writes no stubs. If the + file it names doesn't exist yet, it is a JSON object with one entry per name: + `{"<name>": {"value": "<value>", "secret": true|false}}`. +- A variable the pull can't classify is marked `secret: true` to be safe. Ask the person whether it really is a secret + (a message to type usually isn't). A secret's value is never asked for or shown in chat: the person writes it into the + file. A value the person has already given you in their own words isn't a secret to them: write it there yourself, + without printing it, and go on (§4 step 6). +- `done.next` names `kane-cli testrun run`. A project with no tests yet (or a `--use-case` with none) sends no + `pull_test` events and ends `done` exit 0 with `next` = `kane-cli cover --mode agent`. Exit 1 when any test wasn't + written or the project couldn't be read (`PULL_FAILED`); + 2 for refusals. + +| Code | When | Do | +|---|---|---| +| — | `context pull` with no flags in a folder that isn't linked: it is the location pull, as before. In a new folder its message is "no context store here" | to join the team's project and bring its tests, run `kane-cli context pull --project <id> --mode agent` (`kane-cli projects list --agent` for the id) | +| `PULL_LOCATION_LINKED` | a location flag (a location name, `--rebase`, `--yes`, `--answer`) in a linked folder | `kane-cli context pull --mode agent` | +| `PULL_FLAGS_MIXED` | project flags mixed with location flags | use one set | +| `PULL_FLAGS_LINKED_ONLY` | `--use-case` or `--overwrite` without `--project` in a folder that isn't linked (a new one, or one with its own local store) | link it first with `--project <id>`, or drop those flags | +| `USE_CASE_UNKNOWN` | the project has no such use-case | `kane-cli context list --mode agent` for the ids | +| `PULL_FAILED` (exit 1) | the project's tests couldn't be read; nothing was written | try once more; then give the person the command, the code and the message (agent mode shows no support id here) | + +### 7b. Until then: copy only what the tests need + +Ask the person to get, from the teammate who saved the design, **only**: +- `.testmuai/assurance.json` (the link), when this folder isn't linked yet; +- the test files `.testmuai/tests/*_test.md`; +- for each test, its `output-<test>/` folder beside it, if the teammate ran it (so the first run here is a replay). + +**Never** `.testmuai/variables/` (it can hold secret values) or `.testmuai/evidence/`, and never unzip over this folder's +own `.testmuai/`: it would replace this folder's variables and its link. The receiving folder must have no `.context/` +and no link to another project. Then fill this folder's own variables (§4 step 6) and run the files by path. +Exploration's tests have no folder to copy from: they can be seen (`context list`; Test Manager) but not run until the +pull is available. + +## 8. What changes in a linked folder + +Non-assurance commands (`run`, `testmd`, `testrun`, `projects`, `config`, `evidence` …) always run. These assurance +commands are refused with `REMOTE_UNSUPPORTED` (exit 2): + +| Refused | Use instead | +|---|---| +| `context review`, `context explain` | `kane-cli context list --mode agent`; the checkpoint in §4 | +| `cover gaps` | `kane-cli cover --mode agent` | +| `context sessions show`, `context sessions clean` | `kane-cli context sessions --mode agent`; `… sessions discard <sid> --yes --mode agent` | +| `context sync`, `push`, `clone` (and `context pull` where its help has no `--project`: §7b) | the shared project is the team's copy; `kane-cli context pull --mode agent` brings its tests (§7a). A location pull is refused `PULL_LOCATION_LINKED` | +| `context view`, `fsck`, `rebuild`, `retire`, `revert`, `name`, `design explain` | none in a linked folder | +| `--mode ci` on `extract`, `design tests` and `maintain reconcile` | `--mode agent` | +| `context ingest --plan`, `--mode override` | `kane-cli context ingest <file> --mode agent` | +| `context list --type` other than `source`, `usecase`, `scenario`, `test` | one of those four | +| `context list --all` | `kane-cli context list --mode agent` | +| `maintain reconcile --plan`, `--apply`, a URL in `--from` | `kane-cli maintain reconcile --from <file> --source-id <id> --mode agent`; for an address, `kane-cli context ingest <address> --mode agent` | +| `testrun run --from-context <ids>` | name the files: the refusal's `next` does | + +In a linked folder `extract --plan` and `--with-source` are ignored, and an extract that read two or more documents and +then paused can't be resumed (its `next` adds them again or discards). Give each `context ingest` one document. + +## 9. `kane-cli run` in a linked folder, and "two projects" + +`kane-cli run` saves its test to the **profile's** Test Manager project (`config project`), not the folder's, unless the +profile's project is this folder's project. With `--name` it also writes the test into this folder's `.testmuai/tests/`. +A `testrun run` that selects tests from two projects then refuses `PROJECT_MISMATCH`. + +- In a linked folder, never use `kane-cli run` for a first result: the first result is the journey in §4. +- Quick `run` experiments go in a separate folder that isn't linked. +- On `PROJECT_MISMATCH`: read `projects[]` (`project_id`, `name`, `this_folder`, `tests`). Its `next` runs one project's + tests, by path or with a `--match`: this folder's project's when it has any, else another's. Run it, and tell the person + which tests were left out and why. +- **The lasting fix**, with the person's word: move the tests listed under the project where `this_folder` is `false`, + each with its `output-<test>/` folder, out of `.testmuai/tests/` into a folder that isn't linked. Until then, name the + files on every `testrun run`. +- Recovery for every shared-project refusal: `agent-contract.md`. diff --git a/skill-installer/skills/references/test-manager.md b/skill-installer/skills/references/test-manager.md index 65e9030..f00a2d7 100644 --- a/skill-installer/skills/references/test-manager.md +++ b/skill-installer/skills/references/test-manager.md @@ -2,6 +2,11 @@ # Test Manager Reference — Agent Surface +> **Two different "projects".** This page is about the **profile's** project: where `kane-cli run`, `testmd` and +> `generate` save. An assurance folder **linked** to a team's shared project uses the link's project instead, and +> `projects create` or `config project` never change a folder's link (`shared-projects.md` §2–3). In a linked folder, +> `kane-cli run` still saves to the profile's project (`shared-projects.md` §9). + Every kane-cli session uploads to a TestmuAI **Test Manager (TMS)** test case. The session lands inside a **project** (required) and optionally a **folder** inside that project. This page is the agent-facing surface for everything that touches TMS: where the run ends up, how to browse and create projects/folders programmatically, and the event the run-startup gate emits when nothing is configured. Field names below are for parsing only — translate to plain language for the user, per the §5 rule in `SKILL.md`. @@ -10,13 +15,13 @@ Field names below are for parsing only — translate to plain language for the u ## 1. Where a run lands -After a successful upload, the terminal `run_end` event (`testmd_done` for `testmd run`) carries the dashboard link: +After a successful upload, the terminal `run_end` event of `kane-cli run` carries the dashboard link: ```json {"type":"run_end","status":"passed", "test_url":"https://test-manager.lambdatest.com/projects/<id>/test-cases/<id>", ...} ``` -For `testmd run`, the `test_md_summary` / `test_md_done` events also carry a `share_url` — a 7-day, no-login link suitable for CI artifacts. +For `testmd run`, the `test_md_summary` / `test_md_done` events carry no `test_url`; they carry a `share_url` when the upload made one — a 7-day, no-login link suitable for CI artifacts. Surface `test_url` (and `share_url` when present) as a "View in Test Manager" line per the §1.4 results table in `SKILL.md`. Never paste raw URLs into the middle of a summary — they belong in the results table. @@ -45,7 +50,7 @@ kane-cli folders list --project <id> [--search <q>] [--limit <n>] [--offset <n> | Flag | Purpose | |---|---| | `--search <q>` | Filter by name (substring match). | -| `--limit <n>` | Page size. Default is small (~10). Very large values are capped. | +| `--limit <n>` | Page size: a whole number from 1 to 200 (default 10). A value outside that is refused (exit 2). | | `--offset <n>` | Skip the first N rows. | | `--agent` | Force NDJSON. Auto-on when stdout is piped/redirected, but pass it explicitly anyway. | @@ -165,6 +170,5 @@ The results project and folder belong to kane-cli, not to the agent config. A ch | Code | Meaning | |---|---| | 0 | OK | -| 2 | Auth/setup error (missing or invalid credentials) or unknown subcommand | - -Other codes are run-specific (`run` / `testmd run`) and don't apply to `projects` / `folders`. +| 1 | Test Manager refused or failed the request (the message is on stderr) | +| 2 | Auth/setup error (missing or invalid credentials), a bad flag value, or unknown subcommand | diff --git a/skill-installer/skills/references/testmd.md b/skill-installer/skills/references/testmd.md index 62d35b9..919921b 100644 --- a/skill-installer/skills/references/testmd.md +++ b/skill-installer/skills/references/testmd.md @@ -2,7 +2,7 @@ # Saving & Replaying Tests with testmd -The §3 `run` command is the **primary** mode — one-shot, ephemeral. `testmd` is the **secondary** mode: tests live as `_test.md` files on disk, each step is cached on the first run, and every later run **replays from cache** with no LLM cost. +The §3 `run` command is the **primary** mode — one-shot, ephemeral. `testmd` is the **secondary** mode: tests live as `_test.md` files on disk, each step is cached on the first run, and every later run **replays from cache**, which mostly avoids the authoring cost. Use `testmd` whenever the user wants the test to persist. The decision is binary — once a test exists as a file, every later invocation is `testmd run`, never `run`. @@ -22,7 +22,7 @@ If unclear, ask: "Do you want me to save this test so you can re-run it later?" ## Quick start -Write the file (any path; filename must end in `_test.md`): +Write the file (any path; filename must end in `_test.md`). In a folder linked to a shared project, read `shared-projects.md` §9 first. ```markdown --- @@ -75,7 +75,7 @@ Click submit and verify the confirmation banner. | `mode` | root | `action` (halts on auth walls) or `testing` (default — pushes through so negative-test assertions can fire) | | `url` | root | Start URL for the first step (bare domains get `https://`). Overridden by `--url`; falls back to config `default_url`. | | `tags` | root | Labels for batch selection: YAML list, `[a, b]`, or bare `a, b`. Trimmed, lowercased, deduped. Selected with `testrun --tags` (`references/testrun.md`); shown by `testmd list`. Empty → parse error; per-step → parse error. | -| `max_steps` | root + step | Max agent reasoning steps. Engine fallback `30`; CLI default `50` (step config overrides). | +| `max_steps` | root + step | Max agent reasoning steps. Default `50`; a value in the file wins over `--max-steps`. | | `timeout` | root + step | Hard kill per step in seconds. | | `headless` | root | No browser window. | | `variables` | root + step | `{{name}}` params, same shape as §3, with `secret: true` for credentials. Counts as a value for the pre-run check (0.8.12+), even when `value` is empty. A `{{name}}` that is declared nowhere and has no pool value gets a warning before the run starts, and the run types it as written (SKILL.md §3) | @@ -86,7 +86,7 @@ Files ending in `_test.md` are tests (valid entry points). Any other `.md` is a ## The replay & cascade rule (CRITICAL) -On the **first** run of a test, the agent authors each step and saves a recording. On **every later run**, each step replays from its recording — no agent, no LLM cost, much faster. +On the **first** run of a test, the agent authors each step and saves a recording. On **every later run**, each unchanged step that has a recording replays from it — no authoring agent, so it mostly avoids the authoring cost and runs much faster. A step whose text changed, whose recording is missing, or that failed when recorded is authored again. A replay that heals (below) re-authors the failing steps, so it costs more. An authorable step replays only if **all** of these hold (replay-only marked steps always require their recording): - A recording for that step exists, @@ -112,10 +112,10 @@ Extract a repeating flow (login, setup, cookie banner dismissal) into a helper f ``` Rules: -- Helper filename **must not** end in `_test.md`. +- A step can import a helper (`.md`), another test (`_test.md`) or a module (`_module.md`). - Path resolves relative to the **importing file**, not the shell's cwd. - The step body must be exactly `@import <path>` — no mixed prose, no extra lines. -- The step's `yaml` block may contain **only** `optional`. Other keys are rejected. +- The step's `yaml` block may contain **only** `optional` and `variables` (values for `{{names}}` the imported file doesn't declare itself). Other keys are rejected. - `optional: true` on `@import` is allowed only at the root file, not on a nested import. Variables and context propagate into helpers. Chrome / `mode` / auth do not (root-only). @@ -129,8 +129,8 @@ Editing a helper re-authors that step in **every test that imports it**, plus ev | `kane-cli testmd run <path> --agent [flags]` | Run a test | | `kane-cli testmd list` | List `*_test.md` files under cwd (NDJSON when non-TTY; records include `tags`) | | `kane-cli testmd status <path>` | Test Manager identity + local-sync state | -| `kane-cli testmd export <path> [--language python\|javascript]` | Regenerate code export from existing recordings (no browser launch) | -| `kane-cli testmd delete <path>` | Local-only delete: removes source + `output-<stem>/`. Does NOT delete from Test Manager. | +| `kane-cli testmd export <path> [--language python\|javascript]` | Write code from existing recordings (no browser launch). An existing export is reused; `--force` writes it again | +| `kane-cli testmd delete <path> --yes` | Local-only delete, on the person's word: removes source + `output-<stem>/`. Does NOT delete from Test Manager. Without `--yes` it refuses (exit 2) | | `kane-cli testmd sync <path>` | Re-push the test bundle (test + imports + outputs) to the cloud. Rarely needed — it happens automatically after every authored commit; use only to recover a failed auto-sync. | **Running several tests at once?** Use `kane-cli testrun run` — one execution, one pack, isolated parallel Chromes. Read `references/testrun.md`. @@ -142,14 +142,14 @@ Editing a helper re-authors that step in **every test that imports it**, plus ev | `--url <url>` | frontmatter `url:` / config `default_url` | Start URL for the first step. Overrides the `url:` frontmatter key and config `default_url`. | | `--allow-missing-url` | off | Non-TTY only: start from the browser's current page instead of failing when the first step has no start URL. | | `--name <name>` | none | Persist the run under this name. Regex `[a-zA-Z0-9_-]+`. | -| `--on-lock-conflict <readonly\|fail\|wait>` | none | Behavior when another user holds the test's edit lock. `readonly` = replay-only / no upload, `fail` = exit 2, `wait` = block until released | +| `--on-lock-conflict <readonly\|fail\|wait>` | `readonly` | Behavior when another user holds the test's edit lock. `readonly` = runs, but saves no new version of the test (any re-authoring stays local), `fail` = exit 2, `wait` = retry every 30 s until released, with no time limit | | `--no-adaptive-heal` | healing enabled | Disable adaptive healing after replay failure | | `--author` | off | Force authorable steps to re-author; replay-only steps still replay | | `--bug-detection <off\|stop\|continue>` | config (`off`) | Flag suspected product bugs while **authoring** (`stop` halts on a confirmed bug; `continue` records it). Replay failures always investigate regardless. | Common flags include `--agent`, `--headless`, `--max-steps`, `--timeout`, and `--variables`. Command-specific flags differ; check `kane-cli testmd run --help`. -Flag wins over frontmatter for everything **except** `variables` — the file owns variables; you can add new keys via flags but cannot override file-defined ones. +For step settings such as `max_steps` and `timeout`, a value in the file (frontmatter or a step's `yaml`) wins over the flag; the flag fills only what the file leaves out. Browser flags (`--headless`, `--cdp-endpoint`, `--ws-endpoint`) and `--url` win over the file. The file owns `variables`: you can add new keys via flags but cannot override file-defined ones. ## Output: `output-<stem>/` and `Result.md` @@ -165,7 +165,7 @@ output-amazon/ **`output-<stem>/` is commit-safe and should be committed to git.** That's how teammates and CI replay the same recordings. -For tests using `@import`, helper recordings land next to the helper file in `helper-output-<helper>-<root>-<step>/` directories. Also commit-safe. +For tests using `@import`, each imported file's recordings land next to it in its own `output-<stem>/`. Also commit-safe. Older `helper-output-<helper>-<root>-<step>/` directories are still read, and removed once that file's own recordings are written. **`Result.md`** opens in any Markdown viewer. It contains: - Frontmatter — `status`, `started`, `duration_s`, `session_id` @@ -186,7 +186,7 @@ If the user runs an ad-hoc objective with §3 `run` and decides to keep it: kane-cli run "Search for noise-cancelling headphones on amazon.com" --name amazon-search ``` -On exit, kane-cli writes `<cwd>/.testmuai/tests/amazon-search_test.md`. Move that file into the user's repo and re-run it with `testmd run`. Without `--name`, an ad-hoc `run` is ephemeral and nothing is written. +On exit, kane-cli writes `<cwd>/.testmuai/tests/amazon-search_test.md` and its recordings in `output-amazon-search/` beside it. Move the file together with that folder into the user's repo and re-run it with `testmd run`; without the folder, the next run authors again. Without `--name`, an ad-hoc `run` is ephemeral and nothing is written. In a folder linked to a shared project, don't name runs: `shared-projects.md` §9. ## CI invocation @@ -199,7 +199,7 @@ kane-cli testmd run ./tests/checkout_test.md \ - `--agent` — NDJSON to stdout (auto-enabled when stdin is not a TTY; pass explicitly anyway). - `--headless` — no window. -- `--on-lock-conflict wait` — block instead of failing if a teammate is editing the same test. +- `--on-lock-conflict wait` — wait for a teammate's edit to finish instead of running without saving a new version. - Adaptive healing recovers replay failures by default; `--no-adaptive-heal` opts out. In non-TTY runs, interactive `ask_user` prompts are disabled — a step that would wait for input fails cleanly instead of hanging forever. Write CI test steps that don't depend on mid-run prompts. (Likewise, supply a start URL via the objective, `url:` frontmatter, or `--url`; a non-TTY run with no resolvable start URL fails unless you pass `--allow-missing-url`.) @@ -210,8 +210,10 @@ Exit codes: |------|---------| | 0 | ✅ Passed | | 1 | ❌ Failed | -| 2 | ⚠️ Error (auth, setup, infra) — for `testmd`, also includes parse errors and `--on-lock-conflict fail` | -| 3 | ⏱️ Timeout or cancelled — for `testmd`, also includes `--on-lock-conflict wait` timeout | +| 2 | ⚠️ Error (auth, setup, infra) — for `testmd`, also includes parse errors, `--on-lock-conflict fail`, and `wait` when the test's saved version moved on meanwhile | +| 3 | ⏱️ Timeout or cancelled (`run`). A `testmd run` that times out or is cancelled ends `failed`, exit 1 | + +In a folder linked to a shared project, a designed test's run can also stop before it starts, or exit 1 after passing: `agent-contract.md` §3.7. ## Parse errors (when writing a `_test.md`) @@ -222,8 +224,7 @@ Parse errors abort **before** any browser launch with exit `2`. Common ones and | `frontmatter is missing closing '---'` | Add the trailing `---` | | `invalid YAML in frontmatter` | Re-validate the YAML block | | `step body must be exactly one of prose / @import` | Split into two steps | -| `step config on @import may only contain 'optional'` | Remove other keys from the yaml block | -| `cannot @import a test file` | Imports may only reference helpers (not ending in `_test.md`) | +| `step config on @import may only contain 'optional' / 'variables'` | Remove other keys from the yaml block | | `cyclic reference` | Restructure helpers to break the loop | | `chrome config is global-only` | Move Chrome key to root frontmatter | | `'<key>' is run-level and cannot be set per-step` | Move `mode` / `tags` / `on_lock_conflict` to root frontmatter | @@ -237,7 +238,7 @@ When the user reports a parse error, fix the file before retrying — don't loop Adaptive healing is enabled by default: after a replay failure the runner tries up to three shrinking replay windows, then full re-authoring of authorable steps. `--no-adaptive-heal` disables this recovery. Retired `--retry` and `--retry-count` are accepted only to print a notice; they do not enable healing or change the fixed budget. -Headings marked `@db`, `@api`, `@js`, `@smartui`, `@network_query`, or `@network_assertion` are replay-only. They still replay under `--author`, downstream divergence and adaptive healing. Missing recordings cannot be re-authored through these markers: restore the recording or recreate it through the originating workflow. Do not delete their tapes to force authoring; adding a marker does not create an operation. +Headings marked `@db`, `@api`, `@js`, `@smartui`, `@network_query`, `@network_assertion`, or (on a mobile device) `@gps`, `@accessibility`, `@media_injection`, `@network_throttle`, `@deeplink` are replay-only. They still replay under `--author`, downstream divergence and adaptive healing. Missing recordings cannot be re-authored through these markers: restore the recording or recreate it through the originating workflow. Do not delete their tapes to force authoring; adding a marker does not create an operation. Structured control flow uses balanced heading markers: `@if`, `@elif`, `@else`, `@end-if`, `@while`, `@end-while`. An `@else` must be last in its conditional; end markers must match the opened block type. These are distinct from natural-language conditionals. Markers are excluded from the step body hash. Only one replay-only kind is allowed per step, and an import cannot also be marked replay-only. @@ -255,6 +256,7 @@ This stream is **not** the one-shot `run` stream. Every line is typed, and the f | `test_md_step_end` | `step_index`, `status`, `duration_s`, `failed_sub_step_index` | The step finished. `status` is `passed`, `failed` or `skipped` | | `test_md_evidence_ingest`, `test_md_bundle_sync` | `status` | Informational, before the summary | | `test_md_summary` | `overall_status`, `duration_s`, `steps: {total, passed, failed, skipped, replay_decisions, author_decisions}` | The numbers for the card. `replay_decisions` is how many steps replayed, `author_decisions` how many were authored | -| `test_md_done` | `overall_status`, `duration_s`, `session_id`, `share_url?` | Completion. Always the last line. `share_url` is absent on a pure replay | +| `coverage` | `status` (`written`, `none`, `skipped`, `failed`), `pct?`, `pack` | One per sealed pack, just before `test_md_done` (which repeats it as `coverage`). `skipped`: say nothing (`references/parsing.md`) | +| `test_md_done` | `overall_status`, `duration_s`, `session_id`, `share_url?`, `coverage?` | Completion. The last line of a run that started (a linked folder's stop before the start: `agent-contract.md` §3.7). `share_url` is absent on a pure replay | Most lines are inner `step_event`s (screenshots, reasoning, actions). Skip them unless you are diagnosing a failure: for the card you need only the step starts and ends, the summary and the completion event. On a failure, the failing step is the `test_md_step_end` with `status: "failed"`, its title comes from the matching `test_md_step_start`, and the last inner `step_end` or `step_event` before it says what went wrong. diff --git a/skill-installer/skills/references/testrun.md b/skill-installer/skills/references/testrun.md index e3d9f98..025e62a 100644 --- a/skill-installer/skills/references/testrun.md +++ b/skill-installer/skills/references/testrun.md @@ -26,7 +26,7 @@ kane-cli testrun run [paths...] [flags] < /dev/null # NDJSON when stdin is not | `--match <regex>` | Filter candidates by project-relative path regex | The path is as the OS writes it: `tests/app/` on macOS and Linux, `tests\app\` on Windows. Quote the regex with double quotes in cmd.exe; single quotes are literal there. | | `--tags <list>` | ANY-match on frontmatter `tags:` (repeatable or comma-separated, case-insensitive) | — | | `--parallel <n>` | Worker count; each desktop worker gets an isolated Chrome with a fresh temp profile | `1` | -| `--on-failure <mode>` | `continue` (run everything) \| `fail-fast` (stop dispatching new members after a failure) | `continue` | +| `--on-failure <mode>` | `continue` (run everything) \| `fail-fast` (stop dispatching new replays after a failure; tests with no recording yet are still all written, in the pass after the replays) | `continue` | | `--name <label>` | Run title in the dashboard | derived | | `--dry-run` | Print the plan (members + preflight failures) and exit; runs nothing | off | | `--no-adaptive-heal` | Disable default adaptive healing after replay failure | healing enabled | @@ -44,9 +44,63 @@ All members must share one org + project. *(0.8.4+)* Members need **not** be aut | Reason | Plain-language meaning | What to tell the user | |---|---|---| | `org_mismatch` | Different organisation than the other tests | "Check `kane-cli testmd status <path>` — it belongs to another org" | -| `project_mismatch` | Different project than the other tests | "Run it separately or per-project" | +| `project_mismatch` | Different project than the other tests | *(0.10.0+)* the run also emits `error` `PROJECT_MISMATCH` with `projects[]` (`project_id`, `name`, `this_folder`, `tests`) and a `next` that runs this folder's project's tests by path: run it, and tell the person which tests belong to the other project | -If **any** member fails preflight, the plan is invalid: nothing runs, exit `2`. Suggest `--dry-run` to preview the plan cheaply before a big run. *(0.8.15+)* Preflight also checks variables. A `{{name}}` with no value is a warning: the member stays in the plan (`valid` does not look at variables), and the run types the name as written unless a step sets it first. The check covers every step of the member, replayed ones included. `testrun run` has no `--variables` flag, so fill the pool file (`.testmuai/variables/*.json`) or the member's own `variables:` frontmatter first. +If **any** member fails preflight, the plan is invalid: nothing runs, exit `2`. *(In a folder linked to a shared project, from the release that carries it)* the run then emits `error` `PLAN_INVALID` +with `offenders[{path, failure}]`, and `testrun_done{overall_status:"refused"}`; `PROJECT_MISMATCH` is followed by the +same `testrun_done`; a folder with no tests gives `error` `NO_TESTS` ("no tests here yet") with `next` +`kane-cli context pull --mode agent`; and a `--match` or `--tags` that selects none of the folder's tests gives `NO_TESTS` +("no *_test.md files selected") with no `next`: fix the filter. Everywhere else, and on 0.10.0, an invalid plan ends +after `testrun_plan` (and, for a mix of projects, its `PROJECT_MISMATCH` `error` above) with no `testrun_done`: read the +exit code and each member's `failure`. Suggest `--dry-run` to preview the plan cheaply before a big run. *(0.8.15+)* Preflight also checks variables. A `{{name}}` with no value is a warning: the member stays in the plan (`valid` does not look at variables), and the run types the name as written unless a step sets it first. The check covers every step of the member, replayed ones included. `testrun run` has no `--variables` flag, so fill the pool file (`.testmuai/variables/*.json`) or the member's own `variables:` frontmatter first. + +## Designed tests in a shared project (kane-cli 0.10.0+) + +In a folder linked to a shared project (`shared-projects.md`), a designed test (its file sits directly in `.testmuai/tests/` +and carries an `assurance:` block) runs **on its scenario's Test Manager case**. Before the browser opens, kane-cli finds +that case from the test's `id` and `base`. + +| The test here | What its run does | Cost | +|---|---|---| +| no `output-<test>/` beside it (never run in this folder: just saved, or pulled) | a **first run**: it attaches a new KaneAI test to the case and writes every step in a real browser, then commits it | minutes, and about 30–85 credits per test (each input it checks is its own browser run); tell the person before running it | +| has `output-<test>/` from a first run here that finished | a **replay** of the recorded steps; a step whose text changed since the recording (edited here, or replaced by a save or a pull) is written again, as on a first run | a few credits; more for each step written again | +| a replayed step fails | **heal**: it rewrites from that step, up to 3 times, then the whole test | about a first run's cost | + +- Before a first run, tell the person how many tests will write their steps and the likely cost + (`shared-projects.md` §4), and let them choose. Don't add `--no-adaptive-heal`, `--author` or other flags that change + what the run does on your own (`--headless` from the watch preference is fine); heal is on by default. +- **Credits.** A local run's own stream carries no credits total: note `kane-cli balance` + before the run and again after it, and give the person the difference. (A `--remote` run's `testrun_summary.totals.creditsConsumed`, + when present, adds up what the grid's runs reported.) +- Run the files a save or a pull named: `kane-cli testrun run <files> --agent < /dev/null`. A bare `testrun run` takes + every `*_test.md` here, and a folder that also holds a `kane-cli run` test saved to another project (the profile's) + refuses `PROJECT_MISMATCH` (above). +- A designed test moved into a subfolder of `.testmuai/tests/` runs as an ordinary test, off its case, and doesn't count. +- **Counting.** A finished run counts toward the project's coverage once Test Manager has its evidence; `cover` can take + up to 5 minutes to show it. The run's own `coverage` event (`computed_by: "project"`) shows what this run reached + when its `status` is `written`; any other status has no figure yet (`cards.md` §8). +- **A test that can't run on its case** stops before the browser, and its member ends `failed`; in agent mode its + `failure.message` says why. That test's own log (`log_path`) holds the refusal as an `error` event with its `code` and + a `remedy`; for a missing or gone case the remedy names the session to save (add `--yes --mode agent`). When it names + none, run `kane-cli context sessions --mode agent` and find the person's session. The codes and what to do: + +| Code | Means | Do | +|---|---|---| +| `TEST_CASE_MISSING` | the design that made this test was never saved, so it has no case | save that session: `kane-cli context sessions save <sid> --yes --mode agent` (the person's own session) | +| `TEST_CASE_GONE` | its case was deleted or moved to another project in Test Manager | save the session that designed it again: `kane-cli context sessions save <sid> --yes --mode agent` (free; it makes a new case) | +| `TEST_CASE_ARCHIVED` | its case is archived in Test Manager | ask the person to restore the case in Test Manager, then run the test again | +| `TEST_NOT_IN_PROJECT` | the test isn't in this folder's project, the project has no work yet, or the design it came from can't be saved any more | do what the refusal's message says: run it in the folder linked to its own project, or design the use case again on the person's word (`kane-cli design tests --use-case <uc> --force --mode agent`; add `--allow-unreviewed` when the use-case is still proposed) | +| `TEST_MANAGER_OUTDATED` | Test Manager can't attach this run (for example a device target: it attaches web runs only), or it made a case of its own for the run | a device target: run it on desktop web. Otherwise tell the person Test Manager needs an update before this test can run, and give the support id | +| `TEST_CASE_ATTACH_FAILED`, `TEST_RUN_NOT_SAVED` | the run couldn't be recorded on the case (the verdict stands for the second) | try once more; then give the person the support id | +| `PROJECT_NO_ACCESS`, `ASSURANCE_OFF`, `CREDENTIALS_MISSING`, `CREDENTIALS_REJECTED`, `CONNECTION_FAILED` | the case lookup on a first run failed | `agent-contract.md` | + +- In a terminal the person sees only "failed" for such a member; the reason is in that member's own log (the + `log_path` of its end event, `testrun_member_end` or `testrun_authored_member_end`). Read it before you report the + failure. +- **With `--remote`**, every designed test's case is looked up on this machine before anything is sent (`--dry-run` + too). If any test would stop, nothing is sent: each such test arrives as one `remote_error` with the code above, + `detail` naming the file, and its `remedy`, then `testrun_done` failed and exit `2`. Fix those tests, or leave them out with `--match` + or `--tags`. ## Remote: the suite as one HyperExecute job (`--remote`) @@ -98,7 +152,7 @@ All typed; stdout; one JSON object per line. **Local completion: `testrun_done`. | `testrun_investigations_wait` | `count` | Failed replays left investigations running; the coordinator waits before sealing. Narrate as "investigating N failures". | | `testrun_evidence_ingest` | `status: "ok"\|"failed"`, `evidence_id`, `stage?` | Pack published to the dashboard. Absent when publish is skipped. | | `testrun_summary` | `totals: {tests, passed, failed, broken, skipped, authored}`, `duration_s`, `upload`, `cancelled`, `execution: {id, status}` | Build the rollup table from this. | -| `testrun_done` | `execution_id`, `overall_status: "passed"\|"failed"\|"cancelled"` | Local completion; remote runs continue through `remote_done`. | +| `testrun_done` | `execution_id`, `overall_status: "passed"\|"failed"\|"cancelled"` (`"refused"` for an invalid plan in a linked folder), `coverage?: {status, pct?}` | Local completion; remote runs continue through `remote_done`. `coverage` repeats the run's coverage figure (`pct` only when `status` is `written`). | ### Each test's own log, and `--stream-members` (0.8.17+) @@ -112,18 +166,18 @@ With `--remote`, the stream is wrapped in typed `remote_*` events (all on stdout | `type` | Payload | Notes | |---|---|---| -| `remote_start` | `backend`, `env`, *(0.8.17+)* `log_path` | Dispatch begins. `log_path` is the grid client's own log on this machine, useful when a dispatch fails before a job exists | +| `remote_start` | `backend`, *(0.8.17+)* `log_path`; ignore any other field | Dispatch begins. `log_path` is the grid client's own log on this machine, useful when a dispatch fails before a job exists | | `remote_device` | `platform`, `slug`, `name`, `os_version`, `avd_id?`, `pool?` | The resolved grid device (emulator/simulator only). Present it as the device line. | | `remote_member_device` | `path`, `kind: emulator\|simulator\|real-device`, `platform`, `name?`, `os_version?`, `name_source`, `os_version_source` | One per device member before dispatch. Sources: `flag`, `file`, `config` (saved device), `default`, `hub` (that half left to the hub, no value). Present as one device line per member. | | `remote_device_hint` | `reason: device_name_ignored\|catalog_stale\|saved_device_unavailable\|real_device_list_unavailable`, `detail` | Informational; `device_name_ignored` is emulator-only; `real_device_list_unavailable` = devices dispatched unchecked | | `remote_app` | `path`, `app_id`, `source: uploaded\|cache\|dry-run` | One per distinct local build uploaded from the laptop (mobile); `app_id` is empty on a dry run | | `remote_dispatched` | `job_id`, `job_url` | The HyperExecute job exists — give the user `job_url` | | *(0.8.17+)* member events on remote | `testrun_start`, then a start and end event per test, each with `post_hoc: true` | The grid reports per-test detail **after the job ends**, in plan order, just before `testrun_summary`. Their `ts` is the grid's own time. Same fields as local, including `log_path` and `failure`. `testrun_progress` is not emitted on remote. On 0.8.17+ `remote_dispatched` arrives as soon as the job exists, not at the end | -| `remote_error` | `code`, `detail` | Remote preflight refused (table above); expect `testrun_done` failed + exit 2 | +| `remote_error` | `code`, `detail`, `remedy?`, `support_id?` | Remote preflight refused (table above), or a designed test can't run on its case (§Designed tests in a shared project); expect `testrun_done` failed + exit 2. `code: "internal_error"` is a crash of the remote run itself: `testrun_done` and `remote_done` failed, exit 1 | | `remote_import_tape`, `remote_exec_sync`, `remote_coverage` | `status`, `reason`, `detail?` | Informational; sync/coverage are `skipped` when the project has no `.context` store | | `remote_done` | `status`, `exit`, `job_id`, `sessions_path` | Follows `testrun_done`; `sessions_path` holds the members' grid session logs | -`testrun_summary` additionally carries `remote: {backend, jobId, jobUrl, sessionsPath}`. A member `status: "broken"` with `execution: null` and `upload: "skipped"` means the grid-side kane-cli refused before launching — send the user to `job_url` for the scenario log and check the app id's environment. +`testrun_summary` additionally carries `remote: {backend, jobId, jobUrl, sessionsPath}`. A member `status: "broken"` with `execution: null` and `upload: "skipped"` means the grid-side kane-cli refused before launching — send the user to `job_url` for the scenario log and check that the app id belongs to this account (`kane-cli apps list --target <kind> --agent`). Parsing strategy: @@ -133,7 +187,7 @@ for each line: if type === "remote_done" → capture remote status, exit and sessions_path if type === "testrun_plan" && !valid → report offenders, expect exit 2 if type === "warning" && code === "unresolved_variables" → name the variables with no value (references/parsing.md); the run continues; ignore other warning codes - if type === "testrun_member_end" → note per-member outcome + if type === "testrun_member_end" or "testrun_authored_member_end" → note per-member outcome if type === "testrun_summary" → capture totals for the rollup else → informational; narrate sparingly ``` @@ -156,7 +210,7 @@ Don't list passing tests individually. If the pack published, mention the run is ## Exit codes -`0` all members passed · `1` at least one failed/broken · `2` usage / invalid plan / auth (nothing ran) · `3` cancelled (Ctrl-C — in-flight members finish, the pack still seals). +`0` all members passed · `1` at least one failed/broken · `2` usage / invalid plan / auth (nothing ran) · `3` cancelled (Ctrl-C — no new members start, running ones are stopped, and the pack still seals with what finished). ## Evidence & debugging @@ -164,14 +218,14 @@ The suite produces **one** sealed pack, created directly in `<cwd>/.testmuai/evi ## Execution constraints -Local suites containing any mobile member require `--parallel 1`; larger values are refused. Isolated Chrome workers apply to desktop members only. Remote suites support grid concurrency via `--parallel N`. +Local suites containing any emulator or simulator member require `--parallel 1`; larger values are refused. Real-device members each open their own session, so `--parallel` is allowed up to the plan's devices (§When to use). Isolated Chrome workers apply to desktop members only. Remote suites support grid concurrency via `--parallel N`. Healing is enabled by default (three shrinking replay windows, then re-authoring of authorable steps). `--no-adaptive-heal` disables it. Retired `--retry`/`--retry-count` only print a notice and have no effect. Replay-only recorded steps retain their recordings even during healing. NDJSON selection uses stdin, not stdout (on 0.8.21+ `--agent` selects the same stream): every `kane-cli testrun run` line ends in `< /dev/null` (bash and zsh on macOS, Linux and Git Bash; `< NUL` in cmd.exe; from PowerShell run it through cmd: `cmd /c "kane-cli testrun run … < NUL"`). Check the first stdout line: on 0.8.17+ it is `{"type":"stream_start"…}`. A prose plan there instead means the CLI is in terminal mode: it prints the human view and, after a real run, opens an evidence table that waits until `q` or Esc, so the process never exits on its own. Stop it and rerun with the redirect. An empty stdout with a message on stderr is a usage error: read it. Dry-run validates a plan, not runtime authentication or browser/device readiness. Always observe process exit, including paths without a normal completion event. -### Remote behavior still requiring verification - -The audited dispatch does not forward `--bug-detection` to member commands and does not map `--on-failure` into the job template. Do not rely on these flags for remote bug-detection or fail-fast behavior until implementation owners confirm or fix the mapping. `--name` is suite metadata, not a forwarded member flag. +### Remote flags that don't reach the tests -For dispatched runs, read through `remote_done` and process exit after `testrun_done`; retain the remote status and session-log path. Preflight refusal and dry-run may terminate without `remote_done`. +On `--remote`, `--bug-detection` and `--on-failure` don't reach the tests on the grid: don't rely on them there. For +dispatched runs, read through `remote_done` and the process exit after `testrun_done`, and keep the remote status and +the session-log path. A preflight refusal or a dry run can end without `remote_done`. diff --git a/skill-installer/skills/scripts/preflight.ps1 b/skill-installer/skills/scripts/preflight.ps1 index 85b797c..161c257 100644 --- a/skill-installer/skills/scripts/preflight.ps1 +++ b/skill-installer/skills/scripts/preflight.ps1 @@ -1,9 +1,10 @@ # kane-cli ready check (preflight). Windows PowerShell 5.1 and later. # # Prints, in one call, everything an agent needs to know before it starts a -# kane-cli run. It only reads status. It changes nothing: no sign-in, no -# config write, no install, no network call of its own. It never reads -# credential files. It always exits 0, and problems show up in the text. +# kane-cli run. It only reads status: no sign-in, no config write, no install, +# no network call of its own, and it never reads credential files itself. The +# kane-cli status commands it runs use the saved sign-in, and whoami may renew +# one that is about to expire. It always exits 0, and problems show up in the text. # This is the twin of preflight.sh: same sections, same keys, same order. # # Usage: powershell -File preflight.ps1 [--mobile emulator|simulator] [--grid] @@ -13,6 +14,7 @@ # whoami kane-cli whoami, then exit=<code> # balance kane-cli balance, then exit=<code> # settings kane-cli config show, then exit=<code> +# link project= and project_id= of a linked folder, "none", or "unreadable" # agent-config saved agent preferences, or the word "none" # env ci= ssh= display= os= arch= node= # chrome found=<path> and override=<KANE_CLI_CHROME_PATH> @@ -117,6 +119,23 @@ try { Write-Output '## settings' Write-CommandBlock -CliArgs @('config', 'show') + Write-Output '## link' + # A folder linked to a shared project holds .testmuai/assurance.json; print only its project's name and id, read as + # kane-cli reads them, or "unreadable" when kane-cli can't read the link either. + $linkFile = [IO.Path]::Combine((Get-Location).Path, '.testmuai', 'assurance.json') + if (Test-Path -LiteralPath $linkFile) { + $link = $null + try { $link = Get-Content -LiteralPath $linkFile -Raw -Encoding UTF8 -ErrorAction Stop | ConvertFrom-Json -ErrorAction Stop } catch { } + if (($link -is [System.Management.Automation.PSCustomObject]) -and ($link.projectId -is [string]) -and $link.projectId -and ($link.projectName -is [string])) { + Write-Output ("project=" + ($link.projectName -replace "[`r`n]+", ' ')) + Write-Output ("project_id=" + ($link.projectId -replace "[`r`n]+", ' ')) + } else { + Write-Output 'unreadable' + } + } else { + Write-Output 'none' + } + Write-Output '## agent-config' $agentConfig = [IO.Path]::Combine($HOME, '.testmuai', 'kaneai', 'agent-config', 'config.json') $agentConfigText = $null diff --git a/skill-installer/skills/scripts/preflight.sh b/skill-installer/skills/scripts/preflight.sh index 82dc3ec..232f69e 100755 --- a/skill-installer/skills/scripts/preflight.sh +++ b/skill-installer/skills/scripts/preflight.sh @@ -2,9 +2,10 @@ # kane-cli ready check (preflight). # # Prints, in one call, everything an agent needs to know before it starts a -# kane-cli run. It only reads status. It changes nothing: no sign-in, no -# config write, no install, no network call of its own. It never reads -# credential files. It always exits 0, and problems show up in the text. +# kane-cli run. It only reads status: no sign-in, no config write, no install, +# no network call of its own, and it never reads credential files itself. The +# kane-cli status commands it runs use the saved sign-in, and whoami may renew +# one that is about to expire. It always exits 0, and problems show up in the text. # # Usage: sh preflight.sh [--mobile emulator|simulator] [--grid] # @@ -13,6 +14,7 @@ # whoami kane-cli whoami, then exit=<code> # balance kane-cli balance, then exit=<code> # settings kane-cli config show, then exit=<code> +# link project= and project_id= of a linked folder, "none", or "unreadable" # agent-config saved agent preferences, or the word "none" # env ci= ssh= display= os= arch= node= # chrome found=<path> and override=<KANE_CLI_CHROME_PATH> @@ -126,6 +128,33 @@ emit_cmd balance kane-cli balance echo "## settings" emit_cmd settings kane-cli config show +echo "## link" +# A folder linked to a shared project holds .testmuai/assurance.json; print only its project's name and id, read as +# kane-cli reads them, or "unreadable" when kane-cli can't read the link either. +link_file=".testmuai/assurance.json" +if [ -e "$link_file" ]; then + link_out="" + if command -v node >/dev/null 2>&1; then + link_out=$(node -e ' + try { + const b = JSON.parse(require("fs").readFileSync(process.argv[1], "utf8")); + const one = (s) => s.replace(/[\r\n]+/g, " "); + if (b && typeof b === "object" && !Array.isArray(b) && typeof b.projectId === "string" && b.projectId !== "" && typeof b.projectName === "string") { + process.stdout.write("project=" + one(b.projectName) + "\nproject_id=" + one(b.projectId) + "\n"); + } + } catch (e) {} + ' "$link_file" 2>/dev/null </dev/null) + elif [ -f "$link_file" ] && [ -r "$link_file" ]; then + # No node (kane-cli itself needs it): a best-effort read of the two fields. + link_name=$(sed -n 's/.*"projectName"[[:space:]]*:[[:space:]]*"\([^"]*\)".*/\1/p' "$link_file" | head -n 1) + link_id=$(sed -n 's/.*"projectId"[[:space:]]*:[[:space:]]*"\([^"]*\)".*/\1/p' "$link_file" | head -n 1) + if [ -n "$link_id" ]; then link_out=$(printf 'project=%s\nproject_id=%s' "$link_name" "$link_id"); fi + fi + if [ -n "$link_out" ]; then printf '%s\n' "$link_out"; else echo "unreadable"; fi +else + echo "none" +fi + echo "## agent-config" agent_config="$HOME/.testmuai/kaneai/agent-config/config.json" if [ -f "$agent_config" ] && [ -r "$agent_config" ]; then diff --git a/skill-installer/test/preflight-ps1.test.mjs b/skill-installer/test/preflight-ps1.test.mjs index 851f1b8..aae3109 100644 --- a/skill-installer/test/preflight-ps1.test.mjs +++ b/skill-installer/test/preflight-ps1.test.mjs @@ -11,7 +11,7 @@ import { fileURLToPath } from 'node:url'; const here = path.dirname(fileURLToPath(import.meta.url)); const SCRIPT = path.resolve(here, '..', 'skills', 'scripts', 'preflight.ps1'); const win = process.platform === 'win32' ? test : test.skip; -const BASE_SECTIONS = ['version', 'whoami', 'balance', 'settings', 'agent-config', 'env', 'chrome', 'app', 'tests']; +const BASE_SECTIONS = ['version', 'whoami', 'balance', 'settings', 'link', 'agent-config', 'env', 'chrome', 'app', 'tests']; // A stand-in kane-cli for Windows: a .cmd that answers the calls preflight makes. const FAKE_CLI = [ @@ -76,6 +76,20 @@ win('preflight.ps1 prints the nine base sections in order and exits 0', (t) => { assert.match(body(res.stdout, 'tests'), /count=0/); }); +win('preflight.ps1 prints the link as none, the project, or unreadable', (t) => { + const sb = sandbox(t); + assert.equal(body(run(sb).stdout, 'link'), 'none'); + const dir = path.join(sb.cwd, '.testmuai'); + fs.mkdirSync(dir, { recursive: true }); + const file = path.join(dir, 'assurance.json'); + fs.writeFileSync(file, JSON.stringify({ projectId: '01ID', projectName: 'The "big" one', writtenBy: 'kane-cli' }, null, 2)); + assert.equal(body(run(sb).stdout, 'link'), 'project=The "big" one\nproject_id=01ID'); + fs.writeFileSync(file, '{"projectId": "01ID", "projectName": "Half'); + assert.equal(body(run(sb).stdout, 'link'), 'unreadable'); + fs.writeFileSync(file, JSON.stringify({ projectName: 'No id' })); + assert.equal(body(run(sb).stdout, 'link'), 'unreadable'); +}); + win('preflight.ps1 reports a missing kane-cli and still prints every section', (t) => { const res = run(sandbox(t, { withCli: false })); assert.equal(res.status, 0, res.stderr); diff --git a/skill-installer/test/preflight.test.mjs b/skill-installer/test/preflight.test.mjs index c8e1d83..34f13e7 100644 --- a/skill-installer/test/preflight.test.mjs +++ b/skill-installer/test/preflight.test.mjs @@ -17,7 +17,7 @@ const posix = process.platform === 'win32' ? test.skip : test; const SCRIPT = path.resolve(here, '..', 'skills', 'scripts', 'preflight.sh'); const SYSTEM_PATH = '/usr/bin:/bin:/usr/sbin:/sbin'; const BASE_SECTIONS = [ - 'version', 'whoami', 'balance', 'settings', 'agent-config', + 'version', 'whoami', 'balance', 'settings', 'link', 'agent-config', 'env', 'chrome', 'app', 'tests', ]; const DEV_PORTS = [3000, 3001, 4200, 4321, 5173, 5174, 8000, 8080, 8888]; @@ -96,7 +96,7 @@ function listen(port) { }); } -posix('prints the nine base sections in order, with raw output and exit codes', (t) => { +posix('prints the ten base sections in order, with raw output and exit codes', (t) => { const sb = sandbox(t); const res = run(sb); assert.equal(res.status, 0, res.stderr); @@ -182,6 +182,46 @@ posix('agent-config prints none when absent and the file content when present', assert.deepEqual(JSON.parse(body['agent-config'].join('\n')), config); }); +posix('link prints none in a plain folder, and the project name and id in a linked one', (t) => { + const sb = sandbox(t); + assert.deepEqual(parse(run(sb).stdout).body.link, ['none']); + + const dir = path.join(sb.cwd, '.testmuai'); + fs.mkdirSync(dir, { recursive: true }); + const link = { projectId: '01DUMMYPROJECTID', projectName: 'Checkout revamp', writtenBy: 'kane-cli' }; + fs.writeFileSync(path.join(dir, 'assurance.json'), `${JSON.stringify(link, null, 2)}\n`); + const { order, body } = parse(run(sb).stdout); + assert.deepEqual(order, BASE_SECTIONS); + assert.deepEqual(body.link, ['project=Checkout revamp', 'project_id=01DUMMYPROJECTID']); +}); + +posix('link reads the file as JSON with node, and says unreadable where kane-cli could not read it', (t) => { + const sb = sandbox(t); + // node on PATH (kane-cli needs it), through a folder that holds node alone + const nodeBin = path.join(sb.root, 'nodebin'); + fs.mkdirSync(nodeBin); + fs.symlinkSync(process.execPath, path.join(nodeBin, 'node')); + const withNode = { env: { PATH: `${sb.bin}:${nodeBin}:${SYSTEM_PATH}` } }; + const dir = path.join(sb.cwd, '.testmuai'); + fs.mkdirSync(dir, { recursive: true }); + const file = path.join(dir, 'assurance.json'); + const linkOf = (text) => { + fs.writeFileSync(file, text); + return parse(run(sb, [], withNode).stdout).body.link; + }; + + assert.deepEqual(linkOf(JSON.stringify({ projectId: '01ID', projectName: 'The "big" one' })), + ['project=The "big" one', 'project_id=01ID']); + assert.deepEqual(linkOf('{"projectId": "01ID", "projectName": "Half'), ['unreadable']); + assert.deepEqual(linkOf(JSON.stringify({ projectName: 'No id' })), ['unreadable']); + assert.deepEqual(linkOf(JSON.stringify({ projectId: '', projectName: 'Empty id' })), ['unreadable']); + assert.deepEqual(linkOf(JSON.stringify([{ projectId: '01ID', projectName: 'A list' }])), ['unreadable']); + + // without node, the best-effort read still tells a link it can't read + fs.writeFileSync(file, 'not json'); + assert.deepEqual(parse(run(sb).stdout).body.link, ['unreadable']); +}); + posix('whoami, balance and settings run in parallel', (t) => { const sb = sandbox(t); const res = run(sb);