diff --git a/content/docs/best-practices/dreaming.mdx b/content/docs/best-practices/dreaming.mdx new file mode 100644 index 0000000..cb76f0b --- /dev/null +++ b/content/docs/best-practices/dreaming.mdx @@ -0,0 +1,45 @@ +--- +icon: Lightbulb +title: When to use Dreaming +description: Dreaming pays off when the same lessons keep coming up across runs — here's when to reach for it, and when to leave it off. +--- + +[Dreaming](/docs/extending/dreaming) turns the manual +[analyzing-failures](/docs/best-practices/analyzing-failures) loop into reviewed +suggestions. It's powerful, but it's not something to leave running mindlessly on +every project. Here's when it earns its keep. + +## Reach for it when… + +- **You keep fixing the same kind of thing by hand.** If [analyzing + failures](/docs/best-practices/analyzing-failures) has you adding the *same* rule + again and again — "dismiss the cookie banner first", "wait for the spinner" — + that's exactly the signal Dreaming learns from. Let it propose the rule so you + stop re-typing it. +- **You run a suite repeatedly.** Nightly regression and other suites you run often + are where the lessons compound: each run is more evidence, and a recurring problem + stands out from a one-off (Dreaming shows "4 of 5 runs" so you can tell the + difference). +- **The app under test drifts.** When the UI changes in small ways over time, + Dreaming spots the new obstacles from real runs and keeps your rules current — + instead of you chasing each drift manually. + +## Leave it off when… + +- **The suite is brand new or changing fast.** With only a run or two, there isn't + enough signal yet — write the first rules yourself, then turn Dreaming on to + maintain them. +- **A run failed for an external reason** — a flaky environment, a device that + dropped, an SDK/tooling problem. That's not a testware fix, and Dreaming will say + so (it raises an [advisory](/docs/extending/dreaming) pointing you at support + rather than inventing a rule). Sort the environment first. +- **You want a specific one-off tweak.** For a single deliberate change, just edit + the file — Dreaming is for the patterns that recur, not individual edits. + +## How it fits your workflow + +Treat Dreaming as the **maintenance loop that runs after** you've analyzed +failures, not a replacement for reading a report. You still decide what's true: +Dreaming only *proposes*, you approve each change, and every accepted edit is a +normal change in your project you can review and revert. See +[Dreaming](/docs/extending/dreaming) for how to turn it on and what it can change. diff --git a/content/docs/best-practices/meta.json b/content/docs/best-practices/meta.json index c125804..be63703 100644 --- a/content/docs/best-practices/meta.json +++ b/content/docs/best-practices/meta.json @@ -1 +1 @@ -{ "title": "Best Practices", "icon": "Lightbulb", "pages": ["writing-good-tests", "analyzing-failures", "security", "prompting-best-practices", "agent-behavior"] } +{ "title": "Best Practices", "icon": "Lightbulb", "pages": ["writing-good-tests", "analyzing-failures", "dreaming", "security", "prompting-best-practices", "agent-behavior"] } diff --git a/content/docs/extending/dreaming.mdx b/content/docs/extending/dreaming.mdx new file mode 100644 index 0000000..1b62875 --- /dev/null +++ b/content/docs/extending/dreaming.mdx @@ -0,0 +1,145 @@ +--- +icon: Sparkles +title: Dreaming +description: Let your suite learn from its own runs. Dreaming reviews past runs and proposes small, reviewed improvements to your rules and tests, so reliability compounds the more you run. +--- + +**Dreaming** reviews your recent test runs and proposes small, reviewed +improvements to your tests' rules — turning the manual +[analyzing-failures](/docs/best-practices/analyzing-failures) loop (read the report, +find the cause, fix it in the right place) into suggestions you simply approve. It +runs in the background between runs, so your suite gets more reliable the more you +run it. You stay in control: Dreaming only *proposes* changes; nothing is edited +until you approve it. + +## How it works + + + +### A prompt appears on the Dashboard +After some runs finish, you'll see **"N test runs to analyze"** with **Analyze +now** and **Remind me later**. (You can also start a pass from **Extending → +Dreaming**.) + + +### Analyze +Click **Analyze now**. Dreaming studies each run's conversation together with +whether it passed or failed — the outcome is what makes the suggestions +trustworthy rather than guesswork. A progress bar shows it working, with an +estimated time remaining. No device is needed. + + +### Review +When it finishes, the Dashboard shows **"Dreaming results ready for review"**, +along with how many runs it analyzed and what the analysis cost. Click **Review +now** to open the proposals. + + +### Accept or reject +Each proposal is shown as a diff — the exact line it would add to a `rules.md` or +change in a test — with a short reason and the runs it came from. Accept the ones +you like (individually or **Accept all**), reject the rest, then click **Finish**. +Accepted changes are written to your project files; rejected ones leave your files +untouched. + + + +Once you've reviewed a test's proposals, those runs won't be proposed again — so +each run is only ever analyzed once, unless the test regresses on later runs. + +## What Dreaming changes + +- **Rules** (`rules.md`) — the guidance your agent follows for a test or folder. + Most proposals are here: a new rule learned from a failure, or a small + refinement to an existing one. See [agent behavior](/docs/best-practices/agent-behavior) + for how rules shape a run. +- **App knowledge** (`ui.md`) — facts about the app under test. +- **Setup / teardown** (`setup.md`, `teardown.md`) — steps run before and after a test. +- **Procedures** (`procedures/*`) — your reusable, shared steps. +- **Test definitions** — occasionally, a proposed tweak to a test's own steps. + +Dreaming only ever **appends** a line or **refines** an existing one — it never +rewrites your files wholesale, and it never bakes in brittle pixel coordinates. It +does **not** change your device information — that lives with your +[device profiles](/docs/devices), not in a prose file. + +## When it's not your test — advisories + +Sometimes the trouble isn't your test at all. A tool (including your own +[custom tools](/docs/extending/custom-tools)), the AskUI SDK, or the environment +around the run can be at fault — and no change to your rules or tests would fix +that. When Dreaming spots something like that, it doesn't invent a change. Instead +it adds a short **advisory** to your review: a plain-language description of what +looked wrong, and a prominent pointer to email an AskUI solution engineer at +**support@askui.com**. + +You **acknowledge** or **dismiss** an advisory — nothing is edited either way. +It's simply Dreaming telling you "this one needs a human, not a test change." + +## It's experimental — turn it on deliberately + +Dreaming is an **experimental** feature. You find it under **Extending → +Dreaming**. When you switch it on, a warning appears explaining that learned +changes can, over time, drift and steer your tests in the wrong direction. You +tick a box to confirm you understand, then press **Turn on Dreaming** (the button +stays disabled until you tick the box). Before you start, **make your own backup +of your project** — copy the whole project folder somewhere safe, or zip it up. +Every change is saved to Git too, but a backup is the simplest safety net. + +## You need Git + +Dreaming works only when your project is a **Git repository**, and the enable +toggle stays disabled until Git is initialized. Every edit Dreaming applies is +committed to Git as its own commit (authored by "AskUI Dreaming"), so you can see +it in your history and revert it in one step if it turns out wrong. It only ever +commits the files it edits — never your other changes, your secrets, or your run +artifacts. + +## The Extending → Dreaming tab + +Everything about Dreaming lives on one tab — **Extending → Dreaming**: + +- **Turn it on** (the experimental warning and Git requirement above). +- **Choose the analysis model.** On the AskUI hub, we recommend — and default to — + the **best available** model. Dreaming runs only occasionally, so stronger reasoning + over your runs is worth far more than saving a few cents; you can instead pick "same + as your test runs" or a specific model. If you **bring your own model**, Dreaming + uses the same model as your test runs by default — untick "Same as for my test + runs" to configure a **separate** model for Dreaming (endpoint, model, API key), + for example a cheaper one, since it only reads run transcripts. +- **See what it costs.** After each pass, the tab shows the **cost of the last + analysis** and **how many analysis sessions** you've run. The same cost also + appears on the review banner and at the top of the review page, so you always + know what a round of Dreaming cost you. +- **Run it on demand.** If there are new runs to analyze, an **Analyze now** button + starts a pass right there — the same one the Dashboard offers. + +## Good to know + + +Dreaming never edits a file on its own. Every change waits for your approval, and +every applied change lands as its own Git commit you can revert. + + +- **It starts fresh from when you turn it on.** Dreaming only learns from runs you + make *after* enabling it — turning it on never trawls your entire history, so you + won't come back to hundreds of old runs waiting to be analyzed. +- **It respects edits you make yourself.** If you changed a test or rule after a + run and only then run Dreaming, it works from your *current* version — it won't + undo your change or re-suggest something you've already fixed. +- **It won't change things for the sake of it.** If your tests keep passing, + Dreaming proposes nothing — that's the expected, healthy outcome. +- **It shows how confident it is.** Each suggestion says how many of your recent + runs back it up (e.g. "4 of 5 runs"), so a recurring problem stands out from a + one-off. +- **It checks whether its past changes helped.** From your real runs since a + change, Dreaming works out whether it helped — and if a change made things + worse, it offers to undo it (you still approve the undo). +- **It remembers what it changed**, so it won't flip a file back and forth from + one session to the next. +- **It pays off most on suites you run repeatedly** (for example nightly + regression) — that's where the accumulated lessons compound. +- **It runs in the background** and never interferes with or slows down an actual + test run. +- If analysis can't finish (for example the model provider is unreachable), the + banner tells you, and your files are left untouched. diff --git a/content/docs/extending/meta.json b/content/docs/extending/meta.json index b88a44d..0ce24e6 100644 --- a/content/docs/extending/meta.json +++ b/content/docs/extending/meta.json @@ -1 +1 @@ -{ "title": "Extending", "icon": "Puzzle", "pages": ["tools", "custom-tools", "mcp", "secrets", "report-format", "model-providers"] } +{ "title": "Extending", "icon": "Puzzle", "pages": ["tools", "custom-tools", "mcp", "secrets", "report-format", "model-providers", "dreaming"] }