From 85de7e39f140ab6472323db33757d41dcd2af9a3 Mon Sep 17 00:00:00 2001 From: Paul DeVito Date: Sun, 27 Sep 2026 15:56:06 +0300 Subject: [PATCH] feat(skills): add verify-backwave verification skill Add a project skill that launches Sample.Api on a private port with its own SQLite file or Postgres container. The skill drives the dashboard in a private headless Chrome and keeps the evidence in .verify-evidence/. Cleanup stops only what the run started. Seed the feature map with five dashboard features: jobs and job detail, failures and requeue, queue pause and resume, workflows, and the tag filter. --- .agents/skills/verify-backwave/SKILL.md | 184 +++++++ .../skills/verify-backwave/features/README.md | 51 ++ .../features/failures-and-requeue.md | 41 ++ .../features/jobs-and-job-detail.md | 42 ++ .../features/queue-pause-resume.md | 44 ++ .../verify-backwave/features/tag-filter.md | 41 ++ .../verify-backwave/features/workflows.md | 41 ++ .agents/skills/verify-backwave/scripts/bwv | 450 ++++++++++++++++++ .claude/skills | 1 + .gitignore | 3 + 10 files changed, 898 insertions(+) create mode 100644 .agents/skills/verify-backwave/SKILL.md create mode 100644 .agents/skills/verify-backwave/features/README.md create mode 100644 .agents/skills/verify-backwave/features/failures-and-requeue.md create mode 100644 .agents/skills/verify-backwave/features/jobs-and-job-detail.md create mode 100644 .agents/skills/verify-backwave/features/queue-pause-resume.md create mode 100644 .agents/skills/verify-backwave/features/tag-filter.md create mode 100644 .agents/skills/verify-backwave/features/workflows.md create mode 100755 .agents/skills/verify-backwave/scripts/bwv create mode 120000 .claude/skills diff --git a/.agents/skills/verify-backwave/SKILL.md b/.agents/skills/verify-backwave/SKILL.md new file mode 100644 index 0000000..0fd02ea --- /dev/null +++ b/.agents/skills/verify-backwave/SKILL.md @@ -0,0 +1,184 @@ +--- +name: verify-backwave +description: Launch the real BackWave Sample.Api (the sample host with the BackWave dashboard at /backwave) on a private port with its own SQLite file or Postgres container, create real jobs through the sample's HTTP endpoints, drive the dashboard in a private headless Chrome, capture screenshot, accessibility, API, and database evidence, and clean up. Use to prove a BackWave change works end to end, to reproduce a bug that a user or operator would see, or whenever you need to see or screenshot the dashboard running. +--- + +# Verify BackWave + +Everything goes through one helper, `.agents/skills/verify-backwave/scripts/bwv`. Run it from anywhere in the worktree. The examples below use this alias: + +```bash +bwv() { "$(git rev-parse --show-toplevel)/.agents/skills/verify-backwave/scripts/bwv" "$@"; } +``` + +**Primary surface:** the BackWave dashboard at `/backwave` on `samples/BackWave.Sample.Api`, driven with `chrome-devtools-axi`. The jobs on the dashboard are real: the sample's own HTTP endpoints enqueue them, and the sample's real workers run them. + +**Secondary surfaces:** these surfaces have no drive recipe in the feature map. `bwv api` reaches the first two on the same run. + +- The Sample HTTP API: `/jobs/*`, `/workflows/*`, `/schedules/*`, `/ops/*`, `/monitor/*`, `/tx`, and Swagger at `/swagger`. +- The Pro MCP server at `/backwave-mcp` on the same host. +- `samples/BackWave.Demo`: a SQLite host with the dashboard at `/`, deployed to demo.backwave.app. +- The library API itself (`BackWaveClient`, `BackWaveOperator`, `BackWaveMonitor`) and the test projects under `tests/`. + +Before you drive a feature, read [`features/README.md`](features/README.md) and the matching feature file. The map lists every entry point. A proof that uses one convenient entry point is incomplete when the map lists others. + +## Isolation contract + +Each run owns these resources. All of them use the same run id (`YYYYMMDD-HHMMSS-`): + +| Resource | Name | How it stays private | +|---|---|---| +| Sample.Api process | `dotnet /app/BackWave.Sample.Api.dll` | A private copy of the build output. `bwv` identifies it by its PID and by this path, never by a process name. | +| HTTP port | `http://127.0.0.1:` | A free port from the OS. `bwv` never uses the default `5283` from `appsettings.json`. | +| Scratch directory | `${TMPDIR}/bw-verify-/` | Holds the app copy and the SQLite file `backwave.db`. | +| Postgres (only with `--store postgres`) | container `backwave-verify-`, label `backwave.verify.run=` | `postgres:17-alpine` on a free loopback port. It never uses the compose containers on `5398` or `14331`. | +| Browser | `CHROME_DEVTOOLS_AXI_SESSION=bw-verify-` | A private bridge and an isolated headless Chrome profile. `bwv` never uses the `default` session. | +| Evidence | `.verify-evidence//` at the worktree root | Gitignored. Cleanup never deletes it. | + +Other agents can run tests, `docker compose`, or their own `bwv` run at the same time. Never stop a process, a container, or a browser session that this run did not start. Do not run `docker compose down` for this skill: the compose file owns the shared test databases. + +Parallel runs in one worktree share `bin/` and `obj/`. For this reason, `bwv launch` builds under a lock at `.verify-evidence/.build.lock`, and then copies the output to the run's scratch directory. A later build by another run cannot change the files under a live app. + +## Launch + +```bash +bwv launch # SQLite co-resident store (the default) +bwv launch --store postgres # a private Postgres container +bwv launch --store inmemory # no database, fastest start +``` + +This command does these steps: + +1. Records the run in `.verify-evidence//run.env` and makes it the current run. It records each resource name before it creates the resource. +2. Runs `dotnet build` on `samples/BackWave.Sample.Api` under the build lock, and copies `bin/Debug/net10.0` to the scratch directory. +3. For `--store postgres`, starts the run's container and waits until Postgres accepts TCP connections. +4. Starts the app copy with `--urls http://127.0.0.1:` and the store flags. It sets `DOTNET_hostBuilder__reloadConfigOnChange=false` (see [Gotchas](#gotchas)). +5. Waits until `/backwave` returns 200, then makes sure that `app.log` names the expected store. + +Ready means that the last line is `bwv: READY run= store= pid= url=http://127.0.0.1:/backwave`. A warm launch takes 6-8 s. A cold build takes about one minute. + +The Sample.Api enqueues nothing at startup. The dashboard is empty until you call a `/jobs/*` or `/workflows/*` endpoint. For a busy dashboard, call `bwv api POST /demo/seed seed` (about 1,500 jobs). + +After you change product code, run `bwv cleanup`, then `bwv launch`. A running app never rebuilds itself. + +Teardown is `bwv cleanup` (see [Cleanup](#cleanup)). + +## Doctor + +```bash +bwv doctor +``` + +This is a read-only check. Run it after launch, before every proof, and first when anything looks wrong. Every line must read `ok`: + +- The run is `active` (not cleaned). +- The recorded PID still runs this run's `/app/BackWave.Sample.Api.dll`. +- That PID holds the listening socket on the run's port. +- `/backwave` serves the page with the title `Overview · BackWave`. +- `app.log` names the store: `SqliteCoResident`, `Postgres`, or `InMemory`. +- The build matches the worktree. The fingerprint covers the committed files, the uncommitted edits, and the untracked files. It ignores docs, `*.md`, `.agents`, and `.claude`. If you edit product code after the launch, this check fails. +- The store exists: the SQLite file, or the Postgres container with the run's label. +- The state of the run's browser session. + +The exit code is not zero when a check fails. Do not drive a run that fails doctor. Correct the cause, or run `bwv cleanup` and then `bwv launch`. + +## Drive + +The dashboard is server-rendered HTML with live refresh over Server-Sent Events (SSE). Operator actions are HTML forms that POST with an antiforgery token and redirect with a 303. For this reason, the only valid way to do an operator action is a click in the browser. Do not POST to `/backwave/...` with curl. + +Drive the browser with `bwv browser`. It passes every argument to `chrome-devtools-axi` in the run's own session. For `open` and `newpage`, a path that starts with `/` expands to the run's base URL: + +```bash +bwv api POST '/jobs/flaky?label=proof' flaky # arrange: a real job that dead-letters +bwv browser open /backwave/failures # prints the page snapshot with uids +bwv shot failures-before # evidence of the start state +bwv browser snapshot | grep -B2 -A12 'link "' # find the row's Requeue button uid +bwv browser click @ # the real click on "Requeue" +bwv shot job-after-requeue # the 303 lands on the job detail page +``` + +Rules for handles: + +- A `uid` (for example `@g3:3_42`) is valid only for the snapshot that printed it. Take a new `snapshot` after each navigation or click, and read the uid from it. +- `bwv shot` also takes a snapshot, so it makes all earlier uids stale. A click on a stale uid fails with `STALE_REF` and the page does not change. Take the snapshot, read the uid, and click at once. Do not put a `shot` between them: + + ```bash + uid=$(bwv browser snapshot | grep -A6 'link "critical"' | grep 'button "Pause"' | grep -o 'g[0-9]*:[0-9_]*') + bwv browser click "@$uid" + ``` + +- Do not hide the output of `click`. Read it, and make sure that it has no error. +- Find an element by its accessible name in the snapshot: headings (`heading "Failures"`), links (`link "critical"`), buttons (`button "Requeue"`), and comboboxes (`combobox "State"`). +- Many rows have a button with the same name, for example one `Pause` button per queue. Find the row's link first (`link "critical"`), then use the button that comes after it in the snapshot. +- To list the form actions of a page without a click, run `bwv browser eval "() => [...document.querySelectorAll('form')].map(f => f.getAttribute('action'))"`. The actions name the target: `/backwave/queues/critical/pause`, `/backwave/jobs//requeue`, and `/backwave/jobs//cancel`. +- Navigation links and filter links are plain URLs. You can `open` their URL directly, for example `/backwave/jobs?tk=tenant&tv=acme`. This is the same request as a click on the link. + +Dashboard pages: `/backwave` (Overview), `/backwave/executing`, `/backwave/jobs`, `/backwave/jobs/`, `/backwave/queues`, `/backwave/failures` (`?tab=quarantine`), `/backwave/workflows`, `/backwave/workflows/`, `/backwave/observers`, and `/backwave/schedules`. + +To wait for a job to reach a state, run `bwv wait-job [timeout-s]`. The states are `Scheduled`, `AwaitingParent`, `Leased`, `Succeeded`, `Cancelled`, `DeadLettered`, and `Quarantined`. + +## Evidence + +Everything goes to `.verify-evidence//`: + +| Path | Content | +|---|---| +| `api/.json`, `api/requests.log` | `bwv api METHOD /path [name] [json]`: the response body, and one log line per call with the HTTP status | +| `api/job--.json` | `bwv wait-job`: the job as `/monitor/jobs/` returned it in that state | +| `ui/.png`, `ui/.txt` | `bwv shot `: the screenshot and the accessibility snapshot of the current page | +| `db/.txt` | `bwv sql ""`: a read-only query on the run's own store | +| `db/final-backwave.db` | SQLite runs only: the database file as it was when cleanup stopped the app | +| `app.log` | The app's console output: ASP.NET Core logs and OpenTelemetry console export | +| `run.env`, `build.log`, `cleanup.log` | What ran, and what cleanup removed | + +`bwv api`, `bwv shot`, and `bwv sql` never overwrite a file. Give each capture a new name. + +Proof standards: + +- **Use the real user path.** Create jobs through the sample's HTTP endpoints, as a client application does. Do operator actions with a click on the dashboard. Do not write rows to the database, and do not add test-only endpoints. +- **Capture the action and the resulting state.** Take a `shot` before the key click and after it. The shot after the click must show the page that the 303 redirect opened. +- **Verify side effects on the other side of the boundary.** After an operator action, read the store with `bwv sql`. For example, read the new row in `backwave_operator_audit` (`backwave.operator_audit` on Postgres), and the job's `state` and `attempt` in `backwave_jobs`. Read the job again with `bwv api GET /monitor/jobs/`. When the feature logs, quote the `app.log` line. +- **Use mocks only at existing production boundaries.** The sample has no mocks. Its Slack observer writes a `slack-observer:` line to `app.log` and sends nothing. Do not replace a store, a worker, or a clock to make a proof pass. +- **Observe what a dry run skips.** A 303 response or a green toast does not prove the action. Make sure that the stored state changed: the audit row exists, the job state changed, and a paused queue no longer claims jobs. + +Keep evidence out of git. `.verify-evidence/` is in `.gitignore`. Never force-add it. + +## Cleanup + +```bash +bwv cleanup # the current run +BWV_RUN= bwv cleanup # a run that is not current (see: bwv runs) +``` + +Cleanup does these steps, in this order: + +1. Stops the run's browser session. An open dashboard page holds an SSE request, and the host waits for open requests before it stops. +2. Sends SIGTERM to the app only if the recorded PID still runs this run's app path. After 20 s, it sends SIGKILL. If the PID now runs a different command, cleanup prints `REFUSED` and does not stop it. +3. Removes the container and its volumes only if the container has the label `backwave.verify.run=`. Otherwise it prints `REFUSED`. +4. Copies the SQLite file to `db/final-backwave.db`, then deletes the scratch directory `${TMPDIR}/bw-verify-/`. +5. Sets the run to `cleaned` in `run.env`, writes `cleanup.log`, and prints the evidence path and its file count. + +Cleanup never stops anything by a process name. If a launch died before it recorded the PID, cleanup finds the app by the run's own app path. That path holds the run id. + +Run cleanup after every failed iteration too. `bwv runs` lists every run in this worktree with its status and whether its app is live. Clean every run that is still `active` when you finish. + +## Helpers + +| Helper | Invocation | +|---|---| +| `scripts/bwv` | `bwv launch [--store sqlite\|postgres\|inmemory] \| doctor \| url [path] \| api [name] [json] \| wait-job [timeout-s] \| browser \| shot \| sql \| cleanup \| runs`. Run `bwv` with no arguments to see the usage. | +| `chrome-devtools-axi` | Through `bwv browser` only, so that the run's session is always set. Run `chrome-devtools-axi --help` for the full command list. | +| Swagger UI | `bwv url /swagger` prints the URL. The page lists every sample endpoint and its parameters. | + +## Gotchas + +- **A launch that hangs before `READY` with an empty `app.log`** can be a stalled `sync()`. The appsettings FileSystemWatcher calls `sync()` on macOS. A stuck Time Machine backup to a network volume blocks `sync()` for all processes. `bwv` sets `DOTNET_hostBuilder__reloadConfigOnChange=false` to prevent this. If you start the sample yourself, set it too. +- **The first `bwv browser` call takes 20-40 s.** The bridge starts `npx chrome-devtools-mcp@latest`. `bwv` sets `CHROME_DEVTOOLS_AXI_BRIDGE_TIMEOUT_MS=120000`. The default deadline of 30 s fails on a cold start. +- **`app.log` grows by about 800 KB per minute.** The OpenTelemetry console exporter writes a span for each idle poll. To find a log line, `grep` for the message text or for `LogRecord.FormattedMessage`. +- **Every page shows the Pro banner** "BackWave Pro is running without a license key." It is a `status` region at the top of `main`. It is correct behavior, not a fault. +- **`/jobs/tagged-report` needs `priority=true`**, not `priority=high`. The parameter is a bool, and a wrong value returns 400. +- **`/jobs/flaky` dead-letters after 3 attempts in about 3 s.** It runs in the `Weighted` group on the `low` queue. +- **The Overview counts only queues that hold jobs.** The Queues page also lists `limited` (concurrency limit 1, set at startup by the actor `startup`). +- **SQL Server is not wired into `bwv`.** The sample supports `--BackWave:Store=SqlServer`, but `bwv` does not start a private SQL Server container. Do not point a run at the compose container on port `14331`: other test runs share it. +- **`sqlite3 -readonly` fails on this WAL database.** `bwv sql` uses `PRAGMA query_only=1` instead. +- **macOS has no `timeout` command.** Use `bwv wait-job` or a loop with `sleep` to wait. diff --git a/.agents/skills/verify-backwave/features/README.md b/.agents/skills/verify-backwave/features/README.md new file mode 100644 index 0000000..7de14ce --- /dev/null +++ b/.agents/skills/verify-backwave/features/README.md @@ -0,0 +1,51 @@ +# BackWave verification map + +This directory is the maintained source for verifying the user-facing behavior of BackWave, as an operator sees it on the dashboard. Read the index before you drive the dashboard. Then use the matching feature file as the recipe. Every command below is `bwv`, the helper in `../scripts/bwv` (see `../SKILL.md`). + +## Baseline preconditions + +- `bwv launch` printed `READY`, and `bwv doctor` passes every check on the run that you drive. +- The run's Sample.Api starts with no jobs. Each recipe creates its own jobs through the sample's HTTP endpoints with `bwv api`. +- The run's store is its own SQLite file (the default) or its own Postgres container (`--store postgres`). The queues are `critical`, `bulk`, `low`, and `limited` (concurrency limit 1). +- The operator actor on the dashboard and on `/ops/*` is `sample-operator`. The startup limit on `limited` has the actor `startup`. +- Never drive an app, a container, or a browser session that this run did not start. + +## Driving conventions + +- Arrange state through the sample's HTTP API (`bwv api POST /jobs/...`). Do every operator action with a click on the dashboard. +- Open a dashboard page with `bwv browser open /backwave/...`. The command prints the page snapshot with uids. +- Read each uid from the latest snapshot. A uid is not valid after a navigation or a click. +- Find elements by their accessible name in the snapshot: `heading "Queues"`, `link "critical"`, `button "Pause"`, `combobox "State"`. When many rows have a button with the same name, find the row's link first, then use the button that comes after it. +- Filter and navigation links are plain URLs. `bwv browser open ` on the link's URL is the same request as a click on it. +- Use `bwv wait-job ` to wait for a job. Do not use a fixed `sleep`. +- A run keeps its state until cleanup. If a recipe needs a clean state, use a new run. + +## Proof and skip reporting + +- Capture the user action and the resulting state: `bwv shot ` before the key click and after it. +- UI proof is the `ui/.png` screenshot and its `ui/.txt` snapshot. The snapshot must show the page heading. +- For an operator action, also read the store with `bwv sql` into `db/.txt`, and read the job or queue with `bwv api GET /monitor/...`. The page alone does not prove the action. +- Report every artifact with the run id, the feature ID, and the entry point, for example `.verify-evidence//ui/pause-after.png` for `queue-pause` from the Queues page. +- If you did not reach an entry point, report the command, its output, and the unmet precondition. Do not report it as verified through a different entry point. +- `bwv cleanup` keeps all evidence. Name the evidence path in the report. + +## Feature entry contract + +Each feature file starts with an H1 title and one paragraph that describes the user-visible behavior. It then uses exactly four H2 sections in this order. + +1. `Sub-features` lists short IDs with one line for each behavior. +2. `How to get to it (user POV)` lists every user entry point. +3. `Driving it with bwv` starts with `Preconditions:` and uses labeled bullets. Each bullet pairs a user action with an exact `bwv` command and the observable result. +4. `Gotchas` lists traps that can waste or invalidate a verification run. + +Keep implementation details out of the map. Name only user paths, stable handles, required state, commands, and observable proof. + +## Features + +- [Jobs and job detail](./jobs-and-job-detail.md) covers the Jobs list and its filters, the job detail page (payload, Transition Log, Failure Detail), and Cancel of a scheduled job. +- [Failures and Requeue](./failures-and-requeue.md) covers the Dead-Lettered and Quarantined tabs, Requeue from the Failures page and from the Jobs list, and the audit row. +- [Queue pause and resume](./queue-pause-resume.md) covers Pause and Resume on the Queues page, the Overview "Paused Queues" count, the stalled claims, and the audit rows. +- [Workflows](./workflows.md) covers the Workflows list, the member graph of `order-fulfillment`, member inspection, and a failed workflow. +- [Tag filter](./tag-filter.md) covers the tag pills on jobs, the Label and key-value filters, the combined filter, and the "Top labels" facet. + +Secondary surfaces have no recipe yet: the Sample HTTP API and Swagger as a surface of their own, the MCP server at `/backwave-mcp`, the `BackWave.Demo` host, Recurring Schedules, Observers, and Executing now. Report them as not verified with this skill. diff --git a/.agents/skills/verify-backwave/features/failures-and-requeue.md b/.agents/skills/verify-backwave/features/failures-and-requeue.md new file mode 100644 index 0000000..47c332a --- /dev/null +++ b/.agents/skills/verify-backwave/features/failures-and-requeue.md @@ -0,0 +1,41 @@ +# Failures and Requeue + +A job that fails every attempt goes to Dead-Lettered, and a job that the store refuses to run goes to Quarantined. The Failures page lists both, in two tabs. An operator chooses `Requeue` to give a failed job a new attempt. BackWave writes an audit row for the action, and the job runs again. + +## Sub-features + +- `failures-dead-lettered` lists the Dead-Lettered jobs with their count on the `Dead-Lettered` tab. +- `failures-quarantined` lists the Quarantined jobs on the `Quarantined` tab. +- `failures-requeue` requeues one job from its row, and opens the job detail page. +- `failures-requeue-from-jobs` requeues the same kind of job from its row on the Jobs list. +- `failures-overview` shows the failed jobs in the Overview "Needs attention" table and the `DEAD-LETTERED` count. + +## How to get to it (user POV) + +- Choose `Failures` in the sidebar. `Quarantined` is the second tab, at `/backwave/failures?tab=quarantine`. +- Choose `ALL FAILURES →` on the Overview "Needs attention" table. +- Choose `Jobs`, then filter `State` to `Dead-Lettered`. Each row has a `Requeue` button. + +## Driving it with bwv + +Preconditions: + +- `bwv doctor` passes. +- No earlier recipe in this run used the names `flaky`, `flaky-2`, or `failures-*`. + +- **Create a failure.** Enqueue a job that always fails. Run `bwv api POST '/jobs/flaky?label=proof' flaky`, then `bwv wait-job DeadLettered`. The job reaches `DeadLettered` in about 3 s, after 3 attempts. +- **Open Failures.** Choose `Failures`. Run `bwv browser open /backwave/failures`, then `bwv shot failures-before`. The page has `heading "Failures"`, `link "Dead-Lettered1"`, `link "Quarantined0"`, and a row with the job id prefix, `flaky`, `low`, `Dead-Lettered`, `3`, and `button "Requeue"`. +- **Requeue.** Choose `Requeue` in the job's row. Run `bwv browser snapshot | grep -A12 'link "'` to find the button uid, then `bwv browser click @`. The browser lands on `/backwave/jobs/`. +- **See the new attempts.** Wait for the job to fail again. Run `bwv wait-job DeadLettered`, then `bwv browser open /backwave/jobs/` and `bwv shot failures-after`. The Transition Log shows `14 RECORDED`, newest first. The first 7 rows end at `Dead-Lettered` attempt 3. Then the requeue adds `Scheduled` attempt 0 and 3 new attempts. +- **Requeue from Jobs.** Create a second failure and requeue it from the Jobs list. Run `bwv api POST '/jobs/flaky?label=two' flaky-2`, `bwv wait-job DeadLettered`, and `bwv browser open '/backwave/jobs?state=Dead-Lettered'`. Click the `Requeue` button in that row. The browser lands on the job detail page. +- **Proof.** Read the audit and the job. Run `bwv sql failures-audit "select actor, action, target from backwave_operator_audit"` (`backwave.operator_audit` on Postgres). There is one row for each click, with actor `sample-operator`, action 1 (Requeue), and the job id. Run `bwv api GET /monitor/jobs/ failures-job`. The job is back in state 5 (Dead-Lettered), because the handler always fails. + +## Gotchas + +- A requeued `flaky` job fails again. Prove the requeue with the audit row and the new rows in the Transition Log, not with the final state. +- The `Requeue` redirect goes to the job detail page, not back to the Failures page. +- Run `wait-job` only after the click landed on the job page. Before the requeue, the job is already `DeadLettered`, so `wait-job` returns at once. +- A requeue sets the attempt back to 0. Each attempt gets its own `Failure Detail` row in the Transition Log. +- The sample has no endpoint that makes a Quarantined job on purpose. The `Quarantined` tab stays at 0 unless you seed with `bwv api POST /demo/seed seed`. +- `POST /ops/jobs//requeue` does the same action through the API. It is not a dashboard proof. +- The `Dead-Lettered` tab count updates only when the page loads again. diff --git a/.agents/skills/verify-backwave/features/jobs-and-job-detail.md b/.agents/skills/verify-backwave/features/jobs-and-job-detail.md new file mode 100644 index 0000000..ec3a176 --- /dev/null +++ b/.agents/skills/verify-backwave/features/jobs-and-job-detail.md @@ -0,0 +1,42 @@ +# Jobs and job detail + +An operator finds a job on the Jobs page, filters the list by state, queue, wire name, or job id, and opens the job detail page. The detail page shows the job's fields, its JSON payload, its Transition Log, and for a failed job the Failure Detail. An operator can also cancel a job that is still Scheduled. + +## Sub-features + +- `jobs-list` lists jobs with job id, wire name, queue, state, attempt, due time, terminal time, tags, and actions. +- `jobs-filter` narrows the list with the `State`, `Queue`, `Wire Name`, `Job ID`, and `Rows` controls. +- `job-detail` shows the fields, the payload card, and the Transition Log of one job. +- `job-failure-detail` shows `TERMINAL CAUSE` and the exception text of a failed job. +- `job-cancel` cancels a Scheduled job from its row on the Jobs list. + +## How to get to it (user POV) + +- Choose `Jobs` in the sidebar, then a job id link in the `JOB` column. +- Choose a queue link on the Queues page. It opens the Jobs list with `?queue=`. +- Choose a job id link on the Overview "Needs attention" table or on the Failures page. +- Choose `Open full job page` on a workflow member. +- Open `/backwave/jobs/` from a link that the sample API returns. + +## Driving it with bwv + +Preconditions: + +- `bwv doctor` passes. +- No earlier recipe in this run used the names `greet`, `later`, or `flaky`. + +- **Enqueue a job.** Enqueue a greeting. Run `bwv api POST '/jobs/enqueue?name=Ada' greet`, then `bwv wait-job Succeeded`. The response has `jobId` and `"queue": "critical"`, and the job reaches `Succeeded` in about 1 s. +- **Open the list.** Choose `Jobs`. Run `bwv browser open /backwave/jobs`. The snapshot has `heading "Jobs"`, the comboboxes `State`, `Queue`, `Wire Name`, `Rows`, the textbox `Job ID`, and a row with `link "…"` and `greet`. +- **Filter the list.** Filter by queue and state. Run `bwv browser open '/backwave/jobs?state=Succeeded&queue=critical'`. The `State` combobox shows `Succeeded`, the `Queue` combobox shows `critical`, and only matching rows show. +- **Open job detail.** Choose the job id link. Run `bwv browser click @`, then `bwv shot job-detail`. The page has `heading "Job "`, `WIRE NAME greet`, `QUEUE critical`, `STATE Succeeded`, `ATTEMPT 1`, a `Payload` card with `"Name": "Ada"`, and `Transition Log` with `3 RECORDED` rows: `Scheduled`, `Leased`, `Succeeded`. +- **See a failure.** Enqueue a job that always fails. Run `bwv api POST '/jobs/flaky?label=x' flaky`, `bwv wait-job DeadLettered`, then `bwv browser open /backwave/jobs/`. The page shows `STATE Dead-Lettered`, `ATTEMPT 3`, `TERMINAL CAUSE flaky 'x' always fails (attempt 3)`, and a `Failure Detail` section with `System.InvalidOperationException`. +- **Cancel a scheduled job.** Enqueue a job due in 10 minutes. Run `bwv api POST '/jobs/delayed?name=Later&seconds=600' later`, then `bwv browser open '/backwave/jobs?state=Scheduled'`. Find the `Cancel` button in that job's row and run `bwv browser click @`. The browser lands on `/backwave/jobs/` with `STATE Cancelled`. +- **Proof.** Read the store. Run `bwv sql jobs "select job_id, state, attempt from backwave_jobs"` (`backwave.jobs` on Postgres), and `bwv sql cancel-audit "select actor, action, target from backwave_operator_audit"`. The states are 3 (Succeeded), 5 (Dead-Lettered), and 4 (Cancelled). The audit table has one row with actor `sample-operator`, action 0 (Cancel), and the job id as target. + +## Gotchas + +- The list shows the first 8 characters of the job id and `…`. Match the link on that prefix. +- The `Wire Name` combobox lists every registered job type, also types with no jobs yet. +- Only the Jobs list rows have the `Cancel` and `Requeue` buttons. The job detail page has no action button. +- A Scheduled job with a due time in the past runs at once. Use `seconds=600` so that the job stays Scheduled while you click. +- The Transition Log lists the newest transition first. diff --git a/.agents/skills/verify-backwave/features/queue-pause-resume.md b/.agents/skills/verify-backwave/features/queue-pause-resume.md new file mode 100644 index 0000000..57bf1ca --- /dev/null +++ b/.agents/skills/verify-backwave/features/queue-pause-resume.md @@ -0,0 +1,44 @@ +# Queue pause and resume + +An operator pauses a queue from the Queues page. A paused queue keeps its jobs but no worker claims them, and the Overview shows the queue under "Paused Queues". Resume starts the claims again, and the waiting jobs run. BackWave writes an audit row for each action. + +## Sub-features + +- `queue-list` shows each queue with its state (`Claiming` or `Paused`), the in-use slots and the cap, the depth, and one action button. +- `queue-pause` pauses one queue and redirects to the Overview. +- `queue-paused-stalls` keeps new jobs on a paused queue in `Scheduled`. +- `queue-resume` resumes the queue, and the waiting jobs run. +- `queue-audit` records one audit row for each pause and each resume. + +## How to get to it (user POV) + +- Choose `Queues` in the sidebar, then `Pause` or `Resume` in the queue's row. +- Choose ` QUEUES →` above the Overview "Queue depths" table. The count is the number of queues that hold jobs. +- The Overview card `PAUSED QUEUES` shows the count and the names of the paused queues. + +## Driving it with bwv + +Preconditions: + +- `bwv doctor` passes. +- The `critical` queue is `Claiming`. A new run starts with every queue claiming. +- The `critical` queue holds at least one job. On a new run, the Queues page lists only `limited`. The first step adds a job. +- No earlier recipe in this run used the names `pause-*`, `warmup`, `stalled`, or `resumed`. + +- **Put a job on critical.** Enqueue a greeting. Run `bwv api POST '/jobs/enqueue?name=Warmup' warmup`, then `bwv wait-job Succeeded`. The job reaches `Succeeded` in about 1 s. +- **Open Queues.** Choose `Queues`. Run `bwv browser open /backwave/queues`, then `bwv shot pause-before`. The page has `heading "Queues"` and the rows `critical` and `limited`, each `Claiming` with `button "Pause"`. +- **Pause critical.** Choose `Pause` in the `critical` row. Run `uid=$(bwv browser snapshot | grep -A6 'link "critical"' | grep 'button "Pause"' | grep -o 'g[0-9]*:[0-9_]*')`, then `bwv browser click "@$uid"` at once. The browser lands on the Overview. +- **See the paused state.** Capture the Overview. Run `bwv shot pause-overview`. The `PAUSED QUEUES` card shows `1` and `critical`. +- **Stall a job.** Enqueue a job on the paused queue. Run `bwv api POST '/jobs/enqueue?name=Paused' stalled`, then `bwv api GET /monitor/jobs/ pause-stalled-job`. After 5 s or more, the job still has `"state": 0` (Scheduled). +- **Resume critical.** Choose `Resume` in the `critical` row. Run `bwv browser open /backwave/queues` and `bwv shot pause-queues-paused`. The `critical` row is `Paused` with `button "Resume"`. Take a new snapshot, read the uid of `button "Resume"` after `link "critical"`, and click it. The browser lands on the Overview, and `PAUSED QUEUES` shows `0`. +- **See the job run.** Wait for the stalled job. Run `bwv wait-job Succeeded`. The job reaches `Succeeded` within a few seconds of the resume. +- **Proof.** Read the audit. Run `bwv sql pause-audit "select actor, action, target from backwave_operator_audit order by sequence"` (`backwave.operator_audit` on Postgres). It has two rows with actor `sample-operator` and target `critical`: action 3 (Pause), then action 4 (Resume). The first row is `startup`, `5`, `limited`. Run `bwv shot pause-after` on `/backwave/queues`. The `critical` row is `Claiming` again. + +## Gotchas + +- Every row has a button with the same name. Find the button after the queue's own link, or read the form actions (`/backwave/queues//pause`) with `bwv browser eval`. +- Pause and Resume redirect to the Overview, not back to the Queues page. +- The Overview "Queue depths" table and the Queues page list only the queues that hold jobs. The exception is `limited`, which has a limit from startup. The sample also has `bulk` and `low`. +- `bwv shot` takes a snapshot, so it makes the uid from the earlier snapshot stale. A click on that uid fails with `STALE_REF` and the page stays on Queues. Take the snapshot and click at once. +- A paused `critical` queue also stalls workflow members on `critical`, for example `validate-order`. Resume the queue before you drive the Workflows recipe on the same run. +- `limited` has the cap `1` and an audit row from the actor `startup` (action 5). Filter the audit query by actor or target if the run has that row. diff --git a/.agents/skills/verify-backwave/features/tag-filter.md b/.agents/skills/verify-backwave/features/tag-filter.md new file mode 100644 index 0000000..9d41582 --- /dev/null +++ b/.agents/skills/verify-backwave/features/tag-filter.md @@ -0,0 +1,41 @@ +# Tag filter + +A job carries Tags: Labels (a single word, for example `billing`) and key-value tags (for example `tenant:acme`). The Jobs list shows the tags as pills. An operator chooses a pill or a "Top labels" entry to filter the list, and can combine filters. The `Filter by Tag` box suggests Labels and keys as the operator types. + +## Sub-features + +- `tags-pills` shows each job's tags as links in the `TAGS` column. +- `tags-filter-label` filters the list by one Label (`?tl=