docs(runs): teach an agent to read a run's results and its trace

.changeset/run-results-skill-reference.md

@@ -0,0 +1,5 @@
+---
+"@qawolf/cli": minor
+---
+
+The `qawolf-cli` skill has a new reference file, `references/run-results.md`, for reading what `qawolf run get` returns. It explains the fields that a passing run does not show, such as a flow's failure diagnosis; the rules for the artifact URLs, which expire and can give a 404; and how to read the Playwright trace that `traceUrl` downloads. A trace is newline-delimited JSON, so an agent in a shell can pair each call with its result and find the failure without the trace viewer. The field list is generated from the contract, so it cannot drift from the installed version. Command help is unchanged.

scripts/genSkillMd.ts

@@ -4,15 +4,29 @@
 import { readFileSync, writeFileSync } from "node:fs";
 import { join } from "node:path";
 
+import { publicContractsV1 } from "@qawolf/api-contracts/v1";
+
 import { createProgram } from "~/commands/program.js";
 import { renderCommandsTable, spliceCommandsTable } from "~/commands/skill.js";
+import {
+  renderResponseFields,
+  spliceResponseFields,
+} from "~/commands/skillRunResults.js";
 import { makeNoopSignals } from "~/shell/signals/createSignalRegistry.fixtures.js";
 
 const skillMdPath = join(import.meta.dirname, "../skills/qawolf-cli/SKILL.md");
 const skillTemplatePath = join(
   import.meta.dirname,
   "../src/commands/qawolfCliSkill.template.md",
 );
+const runResultsMdPath = join(
+  import.meta.dirname,
+  "../skills/qawolf-cli/references/run-results.md",
+);
+const runResultsTemplatePath = join(
+  import.meta.dirname,
+  "../src/commands/qawolfCliRunResults.template.md",
+);
 
 const skillTemplate = readFileSync(skillTemplatePath, "utf8");
 const table = renderCommandsTable(
@@ -24,3 +38,12 @@ if (skillMd !== currentSkillMd) {
   writeFileSync(skillMdPath, skillMd);
   console.log("Updated skills/qawolf-cli/SKILL.md");
 }
+
+const runResultsMd = spliceResponseFields(
+  readFileSync(runResultsTemplatePath, "utf8"),
+  renderResponseFields(publicContractsV1.run.get),
+);
+if (runResultsMd !== readFileSync(runResultsMdPath, "utf8")) {
+  writeFileSync(runResultsMdPath, runResultsMd);
+  console.log("Updated skills/qawolf-cli/references/run-results.md");
+}

skills/qawolf-cli/SKILL.md

@@ -72,6 +72,15 @@ each printed line from the payload alone to the whole envelope (`sequence`,
 `recordedAt`, `payload`). Both are JSON. Pass it when you want to page by
 sequence, omit it when you want the payloads themselves.
 
+A `--json` response shows you most of its own shape, so read it first.
+
+`qawolf run get` is the exception worth reading about before you use it. Its
+artifact URLs expire, its failure fields are absent from a passing run, and its
+`traceUrl` downloads a Playwright trace that you can read as JSON without
+opening the trace viewer. **Read
+[`references/run-results.md`](references/run-results.md) before reporting on a
+run's outcome or opening its trace.**
+
 ## Safety: reads vs writes
 
 Read commands do not change team data, but some have operational effects noted

skills/qawolf-cli/references/run-results.md

@@ -0,0 +1,169 @@
+# Reading a run's results
+
+How to use what `qawolf run get --run-id <id> --json` returns, and how to read
+the Playwright trace it links to.
+
+Call `run get` with `--json` and you see most of the response immediately. Read
+this file for the parts a single response cannot show you: fields that appear
+only when something fails, rules about the artifact URLs, and how to read a
+trace without opening the trace viewer.
+
+## The shape
+
+A run holds flows, a flow holds attempts, and artifacts hang off an attempt:
+
+```text
+run
+└── flows[]
+    ├── failure          only when the flow failed
+    └── attempts[]       oldest first
+        ├── logsUrl
+        ├── traceUrl
+        └── videoUrl
+```
+
+`runId` in the response is canonical and can differ from the id you asked for.
+Use the returned value for follow-up calls.
+
+Poll `status` until it reaches `passed`, `failed` or `canceled`. The other
+values mean the run is still going.
+
+## Fields a passing run does not show you
+
+- `flows[].failure` exists only when a flow failed. Every flow passing means
+  there is no failure object at all, so its diagnosis and issue id are invisible
+  until something breaks. Do not conclude the field does not exist.
+- `git` is populated only when a deploy notification started the run. A run
+  started manually or with `run create` has an empty object here.
+- An attempt's `kind` and `status` select which other fields it has. Only
+  automated attempts that reached a verdict carry artifact URLs; canceled
+  attempts and manual Wolf Browser attempts carry none.
+- A flow that passed after a retry still lists its failed attempts. Read the
+  last attempt for the outcome, and the earlier ones to see what went wrong.
+
+## Artifact URLs
+
+Each automated attempt links `logsUrl` (execution logs), `videoUrl` (screen
+recording) and `traceUrl` (a Playwright `trace.zip`).
+
+1. They are signed URLs with a limited life. The contract guarantees at least a
+   day. Call `run get` again for fresh ones instead of storing them; a stored
+   URL becomes a dead link.
+2. A URL can return 404 when that attempt did not produce that artifact. Handle
+   the 404 rather than treating the URL's presence as a guarantee of content.
+3. Download with a plain HTTP GET. The signature is in the URL, so no
+   authentication header is needed and no QA Wolf credentials are involved.
+
+```bash
+qawolf run get --run-id "$RUN_ID" --json \
+  | jq -r '.flows[].attempts[-1].traceUrl // empty' \
+  | head -1 \
+  | xargs -r curl -sS -o trace.zip
+```
+
+## Reading the Playwright trace
+
+The usual advice is `npx playwright show-trace trace.zip`, which opens a
+browser window. That is useless in a shell and unnecessary: the zip holds
+newline-delimited JSON files, and reading them directly is faster than
+downloading a viewer.
+
+The zip holds `trace.trace` (the events), `trace.network` (one request and
+response per line) and a `resources/` directory of screencast frames. The
+frames are most of the size, so extract only what you need.
+
+### The event types
+
+Every line of `trace.trace` is one JSON object with a `type`:
+
+- `before` — a call started. Carries `callId`, `startTime`, `class`, `method`
+  and `params`. `params.selector` or `params.url` is usually the target.
+- `after` — that call finished. Matched to its `before` by `callId`. Carries
+  `endTime` and `result`, and an `error` when the call failed.
+- `console` — a browser console message, with `messageType` and `text`.
+- `log` — Playwright's own progress notes for a call.
+- `screencast-frame`, `frame-snapshot` — the filmstrip and DOM snapshots the
+  viewer renders. Usually not worth reading directly.
+
+Two details cost time if you miss them:
+
+- **Times are monotonic milliseconds, not seconds.** A `goto` whose `startTime`
+  and `endTime` differ by `161.6` took 161 milliseconds. Subtract the smallest
+  `startTime` to get an offset from the start of the trace.
+- **Return values use a serialized envelope.** `{"value":{"s":"passed"}}` is
+  the string `passed`, `{"n":640}` is the number `640`, and `{"o":[...]}` is an
+  object as a list of key and value pairs.
+
+### A worked example
+
+Pairing `before` with `after` gives an action timeline with durations and
+failures:
+
+```python
+import json, sys, zipfile
+
+with zipfile.ZipFile(sys.argv[1]) as z:
+    events = [json.loads(line) for line in z.read("trace.trace").decode().splitlines()]
+
+starts = {e["callId"]: e for e in events if e["type"] == "before"}
+ends = {e["callId"]: e for e in events if e["type"] == "after"}
+t0 = min(e["startTime"] for e in starts.values())
+
+for call_id, before in starts.items():
+    after = ends.get(call_id, {})
+    params = before.get("params", {})
+    target = params.get("selector") or params.get("url") or ""
+    error = after.get("error")
+    print(
+        f'{(before["startTime"] - t0) / 1000:7.2f}s'
+        f' {after.get("endTime", before["startTime"]) - before["startTime"]:7.1f}ms'
+        f'  {before["class"]}.{before["method"]:<18} {target[:40]}'
+        f'{"  FAILED: " + json.dumps(error)[:60] if error else ""}'
+    )
+
+for e in events:
+    if e["type"] == "console" and e["messageType"] == "error":
+        print(f'console error: {e["text"][:70]}')
+```
+
+It prints one line per call, in order:
+
+```text
+   0.00s   161.6ms  Frame.goto               https://example.com/
+   0.17s    28.3ms  Frame.waitForSelector    #screen
+   0.20s     4.2ms  Frame.innerText          #fps_stats
+console error: Failed to load resource: the server responded with a status of 404 ()
+```
+
+To find why an attempt failed, read the last `after` that carries an `error`,
+then the `console` errors near it in time. To see what the page did, read
+`trace.network`.
+
+## Response fields
+
+Every documented field of the `run.get` response. `[]` marks an array, so
+`flows[].attempts[].traceUrl` is the trace URL of one attempt of one flow.
+
+<!-- fields:start — generated by `bun run generate`, do not edit -->
+
+- `completedAt` — When the run finished executing. Absent while queued or running, and also absent for a terminal run that never completed execution (e.g. every flow was canceled or skipped).
+- `git` — The branch and commit under test. The fields are present when a deploy notification started the run, and absent for runs started another way, for example manually or with run.create.
+- `git.commitUrl` — Link to the commit on the code host.
+- `runId` — The run this response describes. Treat it as canonical: it can differ from the id you asked for. A deploy notification returns a run id before the run exists, and if a second notification for the same commit is folded into an earlier run, that id resolves to the earlier run instead.
+- `status` — One of: queued, running, passed, failed, canceled
+- `flows` — The run's flows, ordered alphabetically by name.
+- `flows[].attempts` — The flow's finished execution attempts, oldest first, including manual Wolf Browser attempts. Present once at least one attempt has finished, so a flow that passed after retries also lists its failed attempts. Artifact URLs appear only on automated attempts that reached a verdict, stay valid for at least a day (call run.get again for fresh ones), and can return 404 when the attempt did not produce that artifact.
+- `flows[].attempts[].logsUrl` — Signed URL for the attempt's execution logs.
+- `flows[].attempts[].traceUrl` — Signed URL for the attempt's Playwright trace (a trace.zip; open it with `npx playwright show-trace`).
+- `flows[].attempts[].videoUrl` — Signed URL for the attempt's screen recording.
+- `flows[].attempts[].kind` — One of: automated, manual
+- `flows[].attempts[].startedAt` — Absent when the attempt failed before it could start.
+- `flows[].attempts[].status` — One of: passed, failed, canceled
+- `flows[].failure.diagnosis` — QA Wolf's investigation verdict for the failure: `bug` means the application is broken, `maintenance` means the test needed an update and the failure does not indicate an application problem. Absent until the investigation reaches a verdict. Pass issueId to issue.get for details.
+- `flows[].failure.diagnosis.issueId` — The id of the issue.
+- `flows[].failure.diagnosis.type` — One of: bug, maintenance
+- `flows[].flowId` — The id of the flow.
+- `flows[].status` — One of: failed, queued, running, passed, canceled
+- `url` — Absolute URL of the run page.
+
+<!-- fields:end -->

src/commands/qawolfCliRunResults.template.md

@@ -0,0 +1,148 @@
+# Reading a run's results
+
+How to use what `qawolf run get --run-id <id> --json` returns, and how to read
+the Playwright trace it links to.
+
+Call `run get` with `--json` and you see most of the response immediately. Read
+this file for the parts a single response cannot show you: fields that appear
+only when something fails, rules about the artifact URLs, and how to read a
+trace without opening the trace viewer.
+
+## The shape
+
+A run holds flows, a flow holds attempts, and artifacts hang off an attempt:
+
+```text
+run
+└── flows[]
+    ├── failure          only when the flow failed
+    └── attempts[]       oldest first
+        ├── logsUrl
+        ├── traceUrl
+        └── videoUrl
+```
+
+`runId` in the response is canonical and can differ from the id you asked for.
+Use the returned value for follow-up calls.
+
+Poll `status` until it reaches `passed`, `failed` or `canceled`. The other
+values mean the run is still going.
+
+## Fields a passing run does not show you
+
+- `flows[].failure` exists only when a flow failed. Every flow passing means
+  there is no failure object at all, so its diagnosis and issue id are invisible
+  until something breaks. Do not conclude the field does not exist.
+- `git` is populated only when a deploy notification started the run. A run
+  started manually or with `run create` has an empty object here.
+- An attempt's `kind` and `status` select which other fields it has. Only
+  automated attempts that reached a verdict carry artifact URLs; canceled
+  attempts and manual Wolf Browser attempts carry none.
+- A flow that passed after a retry still lists its failed attempts. Read the
+  last attempt for the outcome, and the earlier ones to see what went wrong.
+
+## Artifact URLs
+
+Each automated attempt links `logsUrl` (execution logs), `videoUrl` (screen
+recording) and `traceUrl` (a Playwright `trace.zip`).
+
+1. They are signed URLs with a limited life. The contract guarantees at least a
+   day. Call `run get` again for fresh ones instead of storing them; a stored
+   URL becomes a dead link.
+2. A URL can return 404 when that attempt did not produce that artifact. Handle
+   the 404 rather than treating the URL's presence as a guarantee of content.
+3. Download with a plain HTTP GET. The signature is in the URL, so no
+   authentication header is needed and no QA Wolf credentials are involved.
+
+```bash
+qawolf run get --run-id "$RUN_ID" --json \
+  | jq -r '.flows[].attempts[-1].traceUrl // empty' \
+  | head -1 \
+  | xargs -r curl -sS -o trace.zip
+```
+
+## Reading the Playwright trace
+
+The usual advice is `npx playwright show-trace trace.zip`, which opens a
+browser window. That is useless in a shell and unnecessary: the zip holds
+newline-delimited JSON files, and reading them directly is faster than
+downloading a viewer.
+
+The zip holds `trace.trace` (the events), `trace.network` (one request and
+response per line) and a `resources/` directory of screencast frames. The
+frames are most of the size, so extract only what you need.
+
+### The event types
+
+Every line of `trace.trace` is one JSON object with a `type`:
+
+- `before` — a call started. Carries `callId`, `startTime`, `class`, `method`
+  and `params`. `params.selector` or `params.url` is usually the target.
+- `after` — that call finished. Matched to its `before` by `callId`. Carries
+  `endTime` and `result`, and an `error` when the call failed.
+- `console` — a browser console message, with `messageType` and `text`.
+- `log` — Playwright's own progress notes for a call.
+- `screencast-frame`, `frame-snapshot` — the filmstrip and DOM snapshots the
+  viewer renders. Usually not worth reading directly.
+
+Two details cost time if you miss them:
+
+- **Times are monotonic milliseconds, not seconds.** A `goto` whose `startTime`
+  and `endTime` differ by `161.6` took 161 milliseconds. Subtract the smallest
+  `startTime` to get an offset from the start of the trace.
+- **Return values use a serialized envelope.** `{"value":{"s":"passed"}}` is
+  the string `passed`, `{"n":640}` is the number `640`, and `{"o":[...]}` is an
+  object as a list of key and value pairs.
+
+### A worked example
+
+Pairing `before` with `after` gives an action timeline with durations and
+failures:
+
+```python
+import json, sys, zipfile
+
+with zipfile.ZipFile(sys.argv[1]) as z:
+    events = [json.loads(line) for line in z.read("trace.trace").decode().splitlines()]
+
+starts = {e["callId"]: e for e in events if e["type"] == "before"}
+ends = {e["callId"]: e for e in events if e["type"] == "after"}
+t0 = min(e["startTime"] for e in starts.values())
+
+for call_id, before in starts.items():
+    after = ends.get(call_id, {})
+    params = before.get("params", {})
+    target = params.get("selector") or params.get("url") or ""
+    error = after.get("error")
+    print(
+        f'{(before["startTime"] - t0) / 1000:7.2f}s'
+        f' {after.get("endTime", before["startTime"]) - before["startTime"]:7.1f}ms'
+        f'  {before["class"]}.{before["method"]:<18} {target[:40]}'
+        f'{"  FAILED: " + json.dumps(error)[:60] if error else ""}'
+    )
+
+for e in events:
+    if e["type"] == "console" and e["messageType"] == "error":
+        print(f'console error: {e["text"][:70]}')
+```
+
+It prints one line per call, in order:
+
+```text
+   0.00s   161.6ms  Frame.goto               https://example.com/
+   0.17s    28.3ms  Frame.waitForSelector    #screen
+   0.20s     4.2ms  Frame.innerText          #fps_stats
+console error: Failed to load resource: the server responded with a status of 404 ()
+```
+
+To find why an attempt failed, read the last `after` that carries an `error`,
+then the `console` errors near it in time. To see what the page did, read
+`trace.network`.
+
+## Response fields
+
+Every documented field of the `run.get` response. `[]` marks an array, so
+`flows[].attempts[].traceUrl` is the trace URL of one attempt of one flow.
+
+<!-- fields:start — generated by `bun run generate`, do not edit -->
+<!-- fields:end -->

src/commands/qawolfCliSkill.template.md

@@ -72,6 +72,15 @@ each printed line from the payload alone to the whole envelope (`sequence`,
 `recordedAt`, `payload`). Both are JSON. Pass it when you want to page by
 sequence, omit it when you want the payloads themselves.
 
+A `--json` response shows you most of its own shape, so read it first.
+
+`qawolf run get` is the exception worth reading about before you use it. Its
+artifact URLs expire, its failure fields are absent from a passing run, and its
+`traceUrl` downloads a Playwright trace that you can read as JSON without
+opening the trace viewer. **Read
+[`references/run-results.md`](references/run-results.md) before reporting on a
+run's outcome or opening its trace.**
+
 ## Safety: reads vs writes
 
 Read commands do not change team data, but some have operational effects noted