Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 45 additions & 0 deletions content/docs/best-practices/dreaming.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
---
icon: Lightbulb
title: When to use Dreaming
description: Dreaming pays off when the same lessons keep coming up across runs β€” here's when to reach for it, and when to leave it off.
---

[Dreaming](/docs/extending/dreaming) turns the manual
[analyzing-failures](/docs/best-practices/analyzing-failures) loop into reviewed
suggestions. It's powerful, but it's not something to leave running mindlessly on
every project. Here's when it earns its keep.

## Reach for it when…

- **You keep fixing the same kind of thing by hand.** If [analyzing
failures](/docs/best-practices/analyzing-failures) has you adding the *same* rule
again and again β€” "dismiss the cookie banner first", "wait for the spinner" β€”
that's exactly the signal Dreaming learns from. Let it propose the rule so you
stop re-typing it.
- **You run a suite repeatedly.** Nightly regression and other suites you run often
are where the lessons compound: each run is more evidence, and a recurring problem
stands out from a one-off (Dreaming shows "4 of 5 runs" so you can tell the
difference).
- **The app under test drifts.** When the UI changes in small ways over time,
Dreaming spots the new obstacles from real runs and keeps your rules current β€”
instead of you chasing each drift manually.

## Leave it off when…

- **The suite is brand new or changing fast.** With only a run or two, there isn't
enough signal yet β€” write the first rules yourself, then turn Dreaming on to
maintain them.
- **A run failed for an external reason** β€” a flaky environment, a device that
dropped, an SDK/tooling problem. That's not a testware fix, and Dreaming will say
so (it raises an [advisory](/docs/extending/dreaming) pointing you at support
rather than inventing a rule). Sort the environment first.
- **You want a specific one-off tweak.** For a single deliberate change, just edit
the file β€” Dreaming is for the patterns that recur, not individual edits.

## How it fits your workflow

Treat Dreaming as the **maintenance loop that runs after** you've analyzed
failures, not a replacement for reading a report. You still decide what's true:
Dreaming only *proposes*, you approve each change, and every accepted edit is a
normal change in your project you can review and revert. See
[Dreaming](/docs/extending/dreaming) for how to turn it on and what it can change.
2 changes: 1 addition & 1 deletion content/docs/best-practices/meta.json
Original file line number Diff line number Diff line change
@@ -1 +1 @@
{ "title": "Best Practices", "icon": "Lightbulb", "pages": ["writing-good-tests", "analyzing-failures", "security", "prompting-best-practices", "agent-behavior"] }
{ "title": "Best Practices", "icon": "Lightbulb", "pages": ["writing-good-tests", "analyzing-failures", "dreaming", "security", "prompting-best-practices", "agent-behavior"] }
145 changes: 145 additions & 0 deletions content/docs/extending/dreaming.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,145 @@
---
icon: Sparkles
title: Dreaming
description: Let your suite learn from its own runs. Dreaming reviews past runs and proposes small, reviewed improvements to your rules and tests, so reliability compounds the more you run.
---

**Dreaming** reviews your recent test runs and proposes small, reviewed
improvements to your tests' rules β€” turning the manual
[analyzing-failures](/docs/best-practices/analyzing-failures) loop (read the report,
find the cause, fix it in the right place) into suggestions you simply approve. It
runs in the background between runs, so your suite gets more reliable the more you
run it. You stay in control: Dreaming only *proposes* changes; nothing is edited
until you approve it.

## How it works

<Steps>
<Step>
### A prompt appears on the Dashboard
After some runs finish, you'll see **"N test runs to analyze"** with **Analyze
now** and **Remind me later**. (You can also start a pass from **Extending β†’
Dreaming**.)
</Step>
<Step>
### Analyze
Click **Analyze now**. Dreaming studies each run's conversation together with
whether it passed or failed β€” the outcome is what makes the suggestions
trustworthy rather than guesswork. A progress bar shows it working, with an
estimated time remaining. No device is needed.
</Step>
<Step>
### Review
When it finishes, the Dashboard shows **"Dreaming results ready for review"**,
along with how many runs it analyzed and what the analysis cost. Click **Review
now** to open the proposals.
</Step>
<Step>
### Accept or reject
Each proposal is shown as a diff β€” the exact line it would add to a `rules.md` or
change in a test β€” with a short reason and the runs it came from. Accept the ones
you like (individually or **Accept all**), reject the rest, then click **Finish**.
Accepted changes are written to your project files; rejected ones leave your files
untouched.
</Step>
</Steps>

Once you've reviewed a test's proposals, those runs won't be proposed again β€” so
each run is only ever analyzed once, unless the test regresses on later runs.

## What Dreaming changes

- **Rules** (`rules.md`) β€” the guidance your agent follows for a test or folder.
Most proposals are here: a new rule learned from a failure, or a small
refinement to an existing one. See [agent behavior](/docs/best-practices/agent-behavior)
for how rules shape a run.
- **App knowledge** (`ui.md`) β€” facts about the app under test.
- **Setup / teardown** (`setup.md`, `teardown.md`) β€” steps run before and after a test.
- **Procedures** (`procedures/*`) β€” your reusable, shared steps.
- **Test definitions** β€” occasionally, a proposed tweak to a test's own steps.

Dreaming only ever **appends** a line or **refines** an existing one β€” it never
rewrites your files wholesale, and it never bakes in brittle pixel coordinates. It
does **not** change your device information β€” that lives with your
[device profiles](/docs/devices), not in a prose file.

## When it's not your test β€” advisories

Sometimes the trouble isn't your test at all. A tool (including your own
[custom tools](/docs/extending/custom-tools)), the AskUI SDK, or the environment
around the run can be at fault β€” and no change to your rules or tests would fix
that. When Dreaming spots something like that, it doesn't invent a change. Instead
it adds a short **advisory** to your review: a plain-language description of what
looked wrong, and a prominent pointer to email an AskUI solution engineer at
**support@askui.com**.

You **acknowledge** or **dismiss** an advisory β€” nothing is edited either way.
It's simply Dreaming telling you "this one needs a human, not a test change."

## It's experimental β€” turn it on deliberately

Dreaming is an **experimental** feature. You find it under **Extending β†’
Dreaming**. When you switch it on, a warning appears explaining that learned
changes can, over time, drift and steer your tests in the wrong direction. You
tick a box to confirm you understand, then press **Turn on Dreaming** (the button
stays disabled until you tick the box). Before you start, **make your own backup
of your project** β€” copy the whole project folder somewhere safe, or zip it up.
Every change is saved to Git too, but a backup is the simplest safety net.

## You need Git

Dreaming works only when your project is a **Git repository**, and the enable
toggle stays disabled until Git is initialized. Every edit Dreaming applies is
committed to Git as its own commit (authored by "AskUI Dreaming"), so you can see
it in your history and revert it in one step if it turns out wrong. It only ever
commits the files it edits β€” never your other changes, your secrets, or your run
artifacts.

## The Extending β†’ Dreaming tab

Everything about Dreaming lives on one tab β€” **Extending β†’ Dreaming**:

- **Turn it on** (the experimental warning and Git requirement above).
- **Choose the analysis model.** On the AskUI hub, we recommend β€” and default to β€”
the **best available** model. Dreaming runs only occasionally, so stronger reasoning
over your runs is worth far more than saving a few cents; you can instead pick "same
as your test runs" or a specific model. If you **bring your own model**, Dreaming
uses the same model as your test runs by default β€” untick "Same as for my test
runs" to configure a **separate** model for Dreaming (endpoint, model, API key),
for example a cheaper one, since it only reads run transcripts.
- **See what it costs.** After each pass, the tab shows the **cost of the last
analysis** and **how many analysis sessions** you've run. The same cost also
appears on the review banner and at the top of the review page, so you always
know what a round of Dreaming cost you.
- **Run it on demand.** If there are new runs to analyze, an **Analyze now** button
starts a pass right there β€” the same one the Dashboard offers.

## Good to know

<Callout type="info">
Dreaming never edits a file on its own. Every change waits for your approval, and
every applied change lands as its own Git commit you can revert.
</Callout>

- **It starts fresh from when you turn it on.** Dreaming only learns from runs you
make *after* enabling it β€” turning it on never trawls your entire history, so you
won't come back to hundreds of old runs waiting to be analyzed.
- **It respects edits you make yourself.** If you changed a test or rule after a
run and only then run Dreaming, it works from your *current* version β€” it won't
undo your change or re-suggest something you've already fixed.
- **It won't change things for the sake of it.** If your tests keep passing,
Dreaming proposes nothing β€” that's the expected, healthy outcome.
- **It shows how confident it is.** Each suggestion says how many of your recent
runs back it up (e.g. "4 of 5 runs"), so a recurring problem stands out from a
one-off.
- **It checks whether its past changes helped.** From your real runs since a
change, Dreaming works out whether it helped β€” and if a change made things
worse, it offers to undo it (you still approve the undo).
- **It remembers what it changed**, so it won't flip a file back and forth from
one session to the next.
- **It pays off most on suites you run repeatedly** (for example nightly
regression) β€” that's where the accumulated lessons compound.
- **It runs in the background** and never interferes with or slows down an actual
test run.
- If analysis can't finish (for example the model provider is unreachable), the
banner tells you, and your files are left untouched.
2 changes: 1 addition & 1 deletion content/docs/extending/meta.json
Original file line number Diff line number Diff line change
@@ -1 +1 @@
{ "title": "Extending", "icon": "Puzzle", "pages": ["tools", "custom-tools", "mcp", "secrets", "report-format", "model-providers"] }
{ "title": "Extending", "icon": "Puzzle", "pages": ["tools", "custom-tools", "mcp", "secrets", "report-format", "model-providers", "dreaming"] }