From 42267e3c774d01dd03d991518960e348f65ad732 Mon Sep 17 00:00:00 2001 From: philipph-askui Date: Fri, 7 Aug 2026 06:58:13 -0700 Subject: [PATCH 01/14] docs(best-practices): add Dreaming page MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Document the Dreaming feature — the app analyzes past runs (grounded in pass/fail) and proposes reviewed edits to rules.md / test definitions, on the Dashboard. Mirrors docs/product/dreaming.md in the app repo. Companion to askui/integrated-task-platform#203 — merge after the app PR. Co-Authored-By: Claude Opus 4.8 (1M context) --- content/docs/best-practices/dreaming.mdx | 74 ++++++++++++++++++++++++ content/docs/best-practices/meta.json | 2 +- 2 files changed, 75 insertions(+), 1 deletion(-) create mode 100644 content/docs/best-practices/dreaming.mdx diff --git a/content/docs/best-practices/dreaming.mdx b/content/docs/best-practices/dreaming.mdx new file mode 100644 index 0000000..6b7fe01 --- /dev/null +++ b/content/docs/best-practices/dreaming.mdx @@ -0,0 +1,74 @@ +--- +icon: Sparkles +title: Dreaming +description: Let your suite learn from its own runs. Dreaming reviews past runs and proposes small, reviewed improvements to your rules and tests, so reliability compounds the more you run. +--- + +Where [analyzing failures](/docs/best-practices/analyzing-failures) is the manual +loop — read the report, find the cause, fix it in the right place — **Dreaming** +runs that loop for you in the background and brings you the proposed fixes to +approve. It reviews your recent runs and suggests small improvements to your +tests' rules, so your suite gets more reliable the more you run it. You stay in +control: Dreaming only *proposes* changes; nothing is edited until you approve it. + +## How it works + + + +### A prompt appears on the Dashboard +After some runs finish, you'll see **"N test runs to analyze"** with **Analyze +now** and **Remind me later**. + + +### Analyze +Click **Analyze now**. Dreaming studies each run's conversation together with +whether it passed or failed — the outcome is what makes the suggestions +trustworthy rather than guesswork. A progress bar shows it working. This uses the +model provider you configured in [Settings](/docs/extending/model-providers); no +device is needed. + + +### Review +When it finishes, the Dashboard shows **"Dreaming results ready for review"**. +Click **Review now** to open the proposals. + + +### Accept or reject +Each proposal is shown as a diff — the exact line it would add to a `rules.md` or +change in a test — with a short reason and the runs it came from. Accept the ones +you like (individually or **Accept all**), reject the rest. Accepted changes are +written to your project files; rejected ones leave your files untouched. + + + +Once you've reviewed a test's proposals, those runs won't be proposed again — so +each run is only ever analyzed once, unless the test regresses on later runs. + +## What Dreaming changes + +- **Rules** (`rules.md`) — the guidance your agent follows for a test or folder. + Most proposals are here: a new rule learned from a failure, or a small + refinement to an existing one. See [agent behavior](/docs/best-practices/agent-behavior) + for how rules shape a run. +- **Test definitions** — occasionally, a proposed tweak to a test's own steps. + +Dreaming only ever **appends** a rule or **refines** an existing line — it never +rewrites your files wholesale. + +## Good to know + + +Dreaming never edits a file on its own. Every change waits for your approval. + + +- **It pays off most on suites you run repeatedly** (for example nightly + regression) — that's where the accumulated lessons compound. +- **It runs in the background** and never interferes with or slows down an actual + test run. +- If analysis can't finish (for example the model provider is unreachable), the + banner tells you, and your files are left untouched. + +## Settings it uses + +Dreaming uses the **model provider** from +[Settings](/docs/extending/model-providers). No extra configuration is required. diff --git a/content/docs/best-practices/meta.json b/content/docs/best-practices/meta.json index c125804..be63703 100644 --- a/content/docs/best-practices/meta.json +++ b/content/docs/best-practices/meta.json @@ -1 +1 @@ -{ "title": "Best Practices", "icon": "Lightbulb", "pages": ["writing-good-tests", "analyzing-failures", "security", "prompting-best-practices", "agent-behavior"] } +{ "title": "Best Practices", "icon": "Lightbulb", "pages": ["writing-good-tests", "analyzing-failures", "dreaming", "security", "prompting-best-practices", "agent-behavior"] } From c23dcedb895453566a9334c358bc9553c9673dec Mon Sep 17 00:00:00 2001 From: philipph-askui Date: Fri, 7 Aug 2026 10:02:27 -0700 Subject: [PATCH 02/14] docs(best-practices): Dreaming edits all run-shaping prose files Broaden the list beyond rules + tests to ui.md, setup.md, teardown.md and procedures/*; note device info stays with device profiles. Co-Authored-By: Claude Opus 4.8 (1M context) --- content/docs/best-practices/dreaming.mdx | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/content/docs/best-practices/dreaming.mdx b/content/docs/best-practices/dreaming.mdx index 6b7fe01..0aa4691 100644 --- a/content/docs/best-practices/dreaming.mdx +++ b/content/docs/best-practices/dreaming.mdx @@ -50,10 +50,14 @@ each run is only ever analyzed once, unless the test regresses on later runs. Most proposals are here: a new rule learned from a failure, or a small refinement to an existing one. See [agent behavior](/docs/best-practices/agent-behavior) for how rules shape a run. +- **App knowledge** (`ui.md`) — facts about the app under test. +- **Setup / teardown** (`setup.md`, `teardown.md`) — steps run before and after a test. +- **Procedures** (`procedures/*`) — your reusable, shared steps. - **Test definitions** — occasionally, a proposed tweak to a test's own steps. -Dreaming only ever **appends** a rule or **refines** an existing line — it never -rewrites your files wholesale. +Dreaming only ever **appends** a line or **refines** an existing one — it never +rewrites your files wholesale. It does **not** change your device information — +that lives with your [device profiles](/docs/devices), not in a prose file. ## Good to know From eae18a0694dd3f6b2b7def2d3bf8d7e58fd2401d Mon Sep 17 00:00:00 2001 From: philipph-askui Date: Fri, 7 Aug 2026 11:00:26 -0700 Subject: [PATCH 03/14] docs(best-practices): Dreaming requires Git (audit trail) Note the Git requirement + per-edit commit so changes are revertable. Co-Authored-By: Claude Opus 4.8 (1M context) --- content/docs/best-practices/dreaming.mdx | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/content/docs/best-practices/dreaming.mdx b/content/docs/best-practices/dreaming.mdx index 0aa4691..497d62a 100644 --- a/content/docs/best-practices/dreaming.mdx +++ b/content/docs/best-practices/dreaming.mdx @@ -59,10 +59,20 @@ Dreaming only ever **appends** a line or **refines** an existing one — it neve rewrites your files wholesale. It does **not** change your device information — that lives with your [device profiles](/docs/devices), not in a prose file. +## You need Git + +Dreaming works only when your project is a **Git repository**, and you turn it on +in **Settings** — the toggle stays disabled until Git is initialized. Every edit +Dreaming applies is committed to Git as its own commit (authored by +"AskUI Dreaming"), so you can see it in your history and revert it in one step if +it turns out wrong. It only ever commits the files it edits — never your other +changes, your secrets, or your run artifacts. + ## Good to know -Dreaming never edits a file on its own. Every change waits for your approval. +Dreaming never edits a file on its own. Every change waits for your approval, and +every applied change lands as its own Git commit you can revert. - **It pays off most on suites you run repeatedly** (for example nightly From 9559c4b91774ec7afccff2dab773e3760ca5d6f5 Mon Sep 17 00:00:00 2001 From: philipph-askui Date: Fri, 7 Aug 2026 12:10:14 -0700 Subject: [PATCH 04/14] =?UTF-8?q?docs(best-practices):=20Dreaming=20is=20e?= =?UTF-8?q?xperimental=20=E2=80=94=20double-confirm=20+=20backup?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Note the experimental opt-in (warning + acknowledgement), the manual-backup advice, that it won't change things when tests pass, and the progress ETA. Co-Authored-By: Claude Opus 4.8 (1M context) --- content/docs/best-practices/dreaming.mdx | 22 ++++++++++++++++++---- 1 file changed, 18 insertions(+), 4 deletions(-) diff --git a/content/docs/best-practices/dreaming.mdx b/content/docs/best-practices/dreaming.mdx index 497d62a..14edcc4 100644 --- a/content/docs/best-practices/dreaming.mdx +++ b/content/docs/best-practices/dreaming.mdx @@ -59,10 +59,20 @@ Dreaming only ever **appends** a line or **refines** an existing one — it neve rewrites your files wholesale. It does **not** change your device information — that lives with your [device profiles](/docs/devices), not in a prose file. +## It's experimental — turn it on deliberately + +Dreaming is an **experimental** feature. When you switch it on in **Settings**, a +warning appears explaining that learned changes can, over time, drift and steer +your tests in the wrong direction. You tick a box to confirm you understand, then +press **Turn on Dreaming** (the button stays disabled until you tick the box). +Before you start, **make your own backup of your project** — copy the whole +project folder somewhere safe, or zip it up. Every change is saved to Git too, but +a backup is the simplest safety net. + ## You need Git -Dreaming works only when your project is a **Git repository**, and you turn it on -in **Settings** — the toggle stays disabled until Git is initialized. Every edit +Dreaming works only when your project is a **Git repository**, and the toggle in +**Settings** stays disabled until Git is initialized. Every edit Dreaming applies is committed to Git as its own commit (authored by "AskUI Dreaming"), so you can see it in your history and revert it in one step if it turns out wrong. It only ever commits the files it edits — never your other @@ -75,10 +85,14 @@ Dreaming never edits a file on its own. Every change waits for your approval, an every applied change lands as its own Git commit you can revert. +- **It won't change things for the sake of it.** If your tests keep passing, + Dreaming proposes nothing — that's the expected, healthy outcome. +- **It remembers what it changed**, so it won't undo itself or flip a file back + and forth from one session to the next. - **It pays off most on suites you run repeatedly** (for example nightly regression) — that's where the accumulated lessons compound. -- **It runs in the background** and never interferes with or slows down an actual - test run. +- **It runs in the background** (with a progress bar and estimated time remaining) + and never interferes with or slows down an actual test run. - If analysis can't finish (for example the model provider is unreachable), the banner tells you, and your files are left untouched. From b5ffaf3b30f463848745298b10bbecdf23e12bcb Mon Sep 17 00:00:00 2001 From: philipph-askui Date: Thu, 13 Aug 2026 14:06:14 -0700 Subject: [PATCH 05/14] docs(best-practices): Dreaming shows confidence + self-checks past changes Note the 'N of M runs' confidence and the retrospective helped/hurt check with a human-approved undo. Co-Authored-By: Claude Opus 4.8 (1M context) --- content/docs/best-practices/dreaming.mdx | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/content/docs/best-practices/dreaming.mdx b/content/docs/best-practices/dreaming.mdx index 14edcc4..c5bc66f 100644 --- a/content/docs/best-practices/dreaming.mdx +++ b/content/docs/best-practices/dreaming.mdx @@ -87,8 +87,14 @@ every applied change lands as its own Git commit you can revert. - **It won't change things for the sake of it.** If your tests keep passing, Dreaming proposes nothing — that's the expected, healthy outcome. -- **It remembers what it changed**, so it won't undo itself or flip a file back - and forth from one session to the next. +- **It shows how confident it is.** Each suggestion says how many of your recent + runs back it up (e.g. "4 of 5 runs"), so a recurring problem stands out from a + one-off. +- **It checks whether its past changes helped.** From your real runs since a + change, Dreaming works out whether it helped — and if a change made things + worse, it offers to undo it (you still approve the undo). +- **It remembers what it changed**, so it won't flip a file back and forth from + one session to the next. - **It pays off most on suites you run repeatedly** (for example nightly regression) — that's where the accumulated lessons compound. - **It runs in the background** (with a progress bar and estimated time remaining) From be7115e6e9112271d29fe950f396b54035335ad8 Mon Sep 17 00:00:00 2001 From: philipph-askui Date: Thu, 13 Aug 2026 21:18:32 -0700 Subject: [PATCH 06/14] =?UTF-8?q?docs(dreaming):=20cost,=20Extending=20?= =?UTF-8?q?=E2=86=92=20Dreaming=20tab,=20best=20model,=20activation=20floo?= =?UTF-8?q?r,=20advisories?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- content/docs/best-practices/dreaming.mdx | 96 ++++++++++++++++-------- 1 file changed, 66 insertions(+), 30 deletions(-) diff --git a/content/docs/best-practices/dreaming.mdx b/content/docs/best-practices/dreaming.mdx index c5bc66f..260ba1a 100644 --- a/content/docs/best-practices/dreaming.mdx +++ b/content/docs/best-practices/dreaming.mdx @@ -17,27 +17,29 @@ control: Dreaming only *proposes* changes; nothing is edited until you approve i ### A prompt appears on the Dashboard After some runs finish, you'll see **"N test runs to analyze"** with **Analyze -now** and **Remind me later**. +now** and **Remind me later**. (You can also start a pass from **Extending → +Dreaming**.) ### Analyze Click **Analyze now**. Dreaming studies each run's conversation together with whether it passed or failed — the outcome is what makes the suggestions -trustworthy rather than guesswork. A progress bar shows it working. This uses the -model provider you configured in [Settings](/docs/extending/model-providers); no -device is needed. +trustworthy rather than guesswork. A progress bar shows it working, with an +estimated time remaining. No device is needed. ### Review -When it finishes, the Dashboard shows **"Dreaming results ready for review"**. -Click **Review now** to open the proposals. +When it finishes, the Dashboard shows **"Dreaming results ready for review"**, +along with how many runs it analyzed and what the analysis cost. Click **Review +now** to open the proposals. ### Accept or reject Each proposal is shown as a diff — the exact line it would add to a `rules.md` or change in a test — with a short reason and the runs it came from. Accept the ones -you like (individually or **Accept all**), reject the rest. Accepted changes are -written to your project files; rejected ones leave your files untouched. +you like (individually or **Accept all**), reject the rest, then click **Finish**. +Accepted changes are written to your project files; rejected ones leave your files +untouched. @@ -56,27 +58,60 @@ each run is only ever analyzed once, unless the test regresses on later runs. - **Test definitions** — occasionally, a proposed tweak to a test's own steps. Dreaming only ever **appends** a line or **refines** an existing one — it never -rewrites your files wholesale. It does **not** change your device information — -that lives with your [device profiles](/docs/devices), not in a prose file. +rewrites your files wholesale, and it never bakes in brittle pixel coordinates. It +does **not** change your device information — that lives with your +[device profiles](/docs/devices), not in a prose file. + +## When it's not your test — advisories + +Sometimes the trouble isn't your test at all. A tool (including your own +[custom tools](/docs/extending/custom-tools)), the AskUI SDK, or the environment +around the run can be at fault — and no change to your rules or tests would fix +that. When Dreaming spots something like that, it doesn't invent a change. Instead +it adds a short **advisory** to your review: + +- a plain-language description of what looked wrong, +- any relevant screenshots from the run to look at, and +- a pointer to email an AskUI solution engineer at **support@askui.com**. + +You **acknowledge** or **dismiss** an advisory — nothing is edited either way. +It's simply Dreaming telling you "this one needs a human, not a test change." ## It's experimental — turn it on deliberately -Dreaming is an **experimental** feature. When you switch it on in **Settings**, a -warning appears explaining that learned changes can, over time, drift and steer -your tests in the wrong direction. You tick a box to confirm you understand, then -press **Turn on Dreaming** (the button stays disabled until you tick the box). -Before you start, **make your own backup of your project** — copy the whole -project folder somewhere safe, or zip it up. Every change is saved to Git too, but -a backup is the simplest safety net. +Dreaming is an **experimental** feature. You find it under **Extending → +Dreaming**. When you switch it on, a warning appears explaining that learned +changes can, over time, drift and steer your tests in the wrong direction. You +tick a box to confirm you understand, then press **Turn on Dreaming** (the button +stays disabled until you tick the box). Before you start, **make your own backup +of your project** — copy the whole project folder somewhere safe, or zip it up. +Every change is saved to Git too, but a backup is the simplest safety net. ## You need Git -Dreaming works only when your project is a **Git repository**, and the toggle in -**Settings** stays disabled until Git is initialized. Every edit -Dreaming applies is committed to Git as its own commit (authored by -"AskUI Dreaming"), so you can see it in your history and revert it in one step if -it turns out wrong. It only ever commits the files it edits — never your other -changes, your secrets, or your run artifacts. +Dreaming works only when your project is a **Git repository**, and the enable +toggle stays disabled until Git is initialized. Every edit Dreaming applies is +committed to Git as its own commit (authored by "AskUI Dreaming"), so you can see +it in your history and revert it in one step if it turns out wrong. It only ever +commits the files it edits — never your other changes, your secrets, or your run +artifacts. + +## The Extending → Dreaming tab + +Everything about Dreaming lives on one tab — **Extending → Dreaming**: + +- **Turn it on** (the experimental warning and Git requirement above). +- **Choose the analysis model.** We recommend — and default to — the **best + available** model. Dreaming runs only occasionally, so stronger reasoning over + your runs is worth far more than saving a few cents. You can instead pick "same + as your test runs" or a specific model. (This applies on the AskUI hub; a + bring-your-own-model setup always analyzes on your configured model.) +- **See what it costs.** After each pass, the tab shows the **cost of the last + analysis** and **how many analysis sessions** you've run. The same cost also + appears on the review banner and at the top of the review page, so you always + know what a round of Dreaming cost you. +- **Run it on demand.** If there are new runs to analyze, an **Analyze now** button + starts a pass right there — the same one the Dashboard offers. ## Good to know @@ -85,6 +120,12 @@ Dreaming never edits a file on its own. Every change waits for your approval, an every applied change lands as its own Git commit you can revert. +- **It starts fresh from when you turn it on.** Dreaming only learns from runs you + make *after* enabling it — turning it on never trawls your entire history, so you + won't come back to hundreds of old runs waiting to be analyzed. +- **It respects edits you make yourself.** If you changed a test or rule after a + run and only then run Dreaming, it works from your *current* version — it won't + undo your change or re-suggest something you've already fixed. - **It won't change things for the sake of it.** If your tests keep passing, Dreaming proposes nothing — that's the expected, healthy outcome. - **It shows how confident it is.** Each suggestion says how many of your recent @@ -97,12 +138,7 @@ every applied change lands as its own Git commit you can revert. one session to the next. - **It pays off most on suites you run repeatedly** (for example nightly regression) — that's where the accumulated lessons compound. -- **It runs in the background** (with a progress bar and estimated time remaining) - and never interferes with or slows down an actual test run. +- **It runs in the background** and never interferes with or slows down an actual + test run. - If analysis can't finish (for example the model provider is unreachable), the banner tells you, and your files are left untouched. - -## Settings it uses - -Dreaming uses the **model provider** from -[Settings](/docs/extending/model-providers). No extra configuration is required. From 47cbbcf522697a8c160af68433b42a1e87b71de3 Mon Sep 17 00:00:00 2001 From: philipph-askui Date: Thu, 13 Aug 2026 21:55:58 -0700 Subject: [PATCH 07/14] docs(dreaming): advisories are text-only (no screenshots) --- content/docs/best-practices/dreaming.mdx | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/content/docs/best-practices/dreaming.mdx b/content/docs/best-practices/dreaming.mdx index 260ba1a..b547d01 100644 --- a/content/docs/best-practices/dreaming.mdx +++ b/content/docs/best-practices/dreaming.mdx @@ -68,11 +68,9 @@ Sometimes the trouble isn't your test at all. A tool (including your own [custom tools](/docs/extending/custom-tools)), the AskUI SDK, or the environment around the run can be at fault — and no change to your rules or tests would fix that. When Dreaming spots something like that, it doesn't invent a change. Instead -it adds a short **advisory** to your review: - -- a plain-language description of what looked wrong, -- any relevant screenshots from the run to look at, and -- a pointer to email an AskUI solution engineer at **support@askui.com**. +it adds a short **advisory** to your review: a plain-language description of what +looked wrong, and a prominent pointer to email an AskUI solution engineer at +**support@askui.com**. You **acknowledge** or **dismiss** an advisory — nothing is edited either way. It's simply Dreaming telling you "this one needs a human, not a test change." From b942a98a5fcca61607a6c4f57ae4bd2228756d37 Mon Sep 17 00:00:00 2001 From: philipph-askui Date: Thu, 13 Aug 2026 22:09:00 -0700 Subject: [PATCH 08/14] docs(dreaming): BYOM analysis-model option (same-as-runs or separate) --- content/docs/best-practices/dreaming.mdx | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/content/docs/best-practices/dreaming.mdx b/content/docs/best-practices/dreaming.mdx index b547d01..237d037 100644 --- a/content/docs/best-practices/dreaming.mdx +++ b/content/docs/best-practices/dreaming.mdx @@ -99,11 +99,13 @@ artifacts. Everything about Dreaming lives on one tab — **Extending → Dreaming**: - **Turn it on** (the experimental warning and Git requirement above). -- **Choose the analysis model.** We recommend — and default to — the **best - available** model. Dreaming runs only occasionally, so stronger reasoning over - your runs is worth far more than saving a few cents. You can instead pick "same - as your test runs" or a specific model. (This applies on the AskUI hub; a - bring-your-own-model setup always analyzes on your configured model.) +- **Choose the analysis model.** On the AskUI hub, we recommend — and default to — + the **best available** model. Dreaming runs only occasionally, so stronger reasoning + over your runs is worth far more than saving a few cents; you can instead pick "same + as your test runs" or a specific model. If you **bring your own model**, Dreaming + uses the same model as your test runs by default — untick "Same as for my test + runs" to configure a **separate** model for Dreaming (endpoint, model, API key), + for example a cheaper one, since it only reads run transcripts. - **See what it costs.** After each pass, the tab shows the **cost of the last analysis** and **how many analysis sessions** you've run. The same cost also appears on the review banner and at the top of the review page, so you always From b3723ccb4700d8760dc9bfc754bb10820386be35 Mon Sep 17 00:00:00 2001 From: philipph-askui Date: Tue, 18 Aug 2026 10:04:10 -0700 Subject: [PATCH 09/14] docs(dreaming): move Dreaming page to Extending (functional docs) per review --- content/docs/extending/dreaming.mdx | 145 ++++++++++++++++++++++++++++ 1 file changed, 145 insertions(+) create mode 100644 content/docs/extending/dreaming.mdx diff --git a/content/docs/extending/dreaming.mdx b/content/docs/extending/dreaming.mdx new file mode 100644 index 0000000..1b62875 --- /dev/null +++ b/content/docs/extending/dreaming.mdx @@ -0,0 +1,145 @@ +--- +icon: Sparkles +title: Dreaming +description: Let your suite learn from its own runs. Dreaming reviews past runs and proposes small, reviewed improvements to your rules and tests, so reliability compounds the more you run. +--- + +**Dreaming** reviews your recent test runs and proposes small, reviewed +improvements to your tests' rules — turning the manual +[analyzing-failures](/docs/best-practices/analyzing-failures) loop (read the report, +find the cause, fix it in the right place) into suggestions you simply approve. It +runs in the background between runs, so your suite gets more reliable the more you +run it. You stay in control: Dreaming only *proposes* changes; nothing is edited +until you approve it. + +## How it works + + + +### A prompt appears on the Dashboard +After some runs finish, you'll see **"N test runs to analyze"** with **Analyze +now** and **Remind me later**. (You can also start a pass from **Extending → +Dreaming**.) + + +### Analyze +Click **Analyze now**. Dreaming studies each run's conversation together with +whether it passed or failed — the outcome is what makes the suggestions +trustworthy rather than guesswork. A progress bar shows it working, with an +estimated time remaining. No device is needed. + + +### Review +When it finishes, the Dashboard shows **"Dreaming results ready for review"**, +along with how many runs it analyzed and what the analysis cost. Click **Review +now** to open the proposals. + + +### Accept or reject +Each proposal is shown as a diff — the exact line it would add to a `rules.md` or +change in a test — with a short reason and the runs it came from. Accept the ones +you like (individually or **Accept all**), reject the rest, then click **Finish**. +Accepted changes are written to your project files; rejected ones leave your files +untouched. + + + +Once you've reviewed a test's proposals, those runs won't be proposed again — so +each run is only ever analyzed once, unless the test regresses on later runs. + +## What Dreaming changes + +- **Rules** (`rules.md`) — the guidance your agent follows for a test or folder. + Most proposals are here: a new rule learned from a failure, or a small + refinement to an existing one. See [agent behavior](/docs/best-practices/agent-behavior) + for how rules shape a run. +- **App knowledge** (`ui.md`) — facts about the app under test. +- **Setup / teardown** (`setup.md`, `teardown.md`) — steps run before and after a test. +- **Procedures** (`procedures/*`) — your reusable, shared steps. +- **Test definitions** — occasionally, a proposed tweak to a test's own steps. + +Dreaming only ever **appends** a line or **refines** an existing one — it never +rewrites your files wholesale, and it never bakes in brittle pixel coordinates. It +does **not** change your device information — that lives with your +[device profiles](/docs/devices), not in a prose file. + +## When it's not your test — advisories + +Sometimes the trouble isn't your test at all. A tool (including your own +[custom tools](/docs/extending/custom-tools)), the AskUI SDK, or the environment +around the run can be at fault — and no change to your rules or tests would fix +that. When Dreaming spots something like that, it doesn't invent a change. Instead +it adds a short **advisory** to your review: a plain-language description of what +looked wrong, and a prominent pointer to email an AskUI solution engineer at +**support@askui.com**. + +You **acknowledge** or **dismiss** an advisory — nothing is edited either way. +It's simply Dreaming telling you "this one needs a human, not a test change." + +## It's experimental — turn it on deliberately + +Dreaming is an **experimental** feature. You find it under **Extending → +Dreaming**. When you switch it on, a warning appears explaining that learned +changes can, over time, drift and steer your tests in the wrong direction. You +tick a box to confirm you understand, then press **Turn on Dreaming** (the button +stays disabled until you tick the box). Before you start, **make your own backup +of your project** — copy the whole project folder somewhere safe, or zip it up. +Every change is saved to Git too, but a backup is the simplest safety net. + +## You need Git + +Dreaming works only when your project is a **Git repository**, and the enable +toggle stays disabled until Git is initialized. Every edit Dreaming applies is +committed to Git as its own commit (authored by "AskUI Dreaming"), so you can see +it in your history and revert it in one step if it turns out wrong. It only ever +commits the files it edits — never your other changes, your secrets, or your run +artifacts. + +## The Extending → Dreaming tab + +Everything about Dreaming lives on one tab — **Extending → Dreaming**: + +- **Turn it on** (the experimental warning and Git requirement above). +- **Choose the analysis model.** On the AskUI hub, we recommend — and default to — + the **best available** model. Dreaming runs only occasionally, so stronger reasoning + over your runs is worth far more than saving a few cents; you can instead pick "same + as your test runs" or a specific model. If you **bring your own model**, Dreaming + uses the same model as your test runs by default — untick "Same as for my test + runs" to configure a **separate** model for Dreaming (endpoint, model, API key), + for example a cheaper one, since it only reads run transcripts. +- **See what it costs.** After each pass, the tab shows the **cost of the last + analysis** and **how many analysis sessions** you've run. The same cost also + appears on the review banner and at the top of the review page, so you always + know what a round of Dreaming cost you. +- **Run it on demand.** If there are new runs to analyze, an **Analyze now** button + starts a pass right there — the same one the Dashboard offers. + +## Good to know + + +Dreaming never edits a file on its own. Every change waits for your approval, and +every applied change lands as its own Git commit you can revert. + + +- **It starts fresh from when you turn it on.** Dreaming only learns from runs you + make *after* enabling it — turning it on never trawls your entire history, so you + won't come back to hundreds of old runs waiting to be analyzed. +- **It respects edits you make yourself.** If you changed a test or rule after a + run and only then run Dreaming, it works from your *current* version — it won't + undo your change or re-suggest something you've already fixed. +- **It won't change things for the sake of it.** If your tests keep passing, + Dreaming proposes nothing — that's the expected, healthy outcome. +- **It shows how confident it is.** Each suggestion says how many of your recent + runs back it up (e.g. "4 of 5 runs"), so a recurring problem stands out from a + one-off. +- **It checks whether its past changes helped.** From your real runs since a + change, Dreaming works out whether it helped — and if a change made things + worse, it offers to undo it (you still approve the undo). +- **It remembers what it changed**, so it won't flip a file back and forth from + one session to the next. +- **It pays off most on suites you run repeatedly** (for example nightly + regression) — that's where the accumulated lessons compound. +- **It runs in the background** and never interferes with or slows down an actual + test run. +- If analysis can't finish (for example the model provider is unreachable), the + banner tells you, and your files are left untouched. From 20e3b82f50bc272047a076b5470ea6cce3efa8e6 Mon Sep 17 00:00:00 2001 From: philipph-askui Date: Tue, 18 Aug 2026 10:04:11 -0700 Subject: [PATCH 10/14] docs(extending): add dreaming to the section index --- content/docs/extending/meta.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/docs/extending/meta.json b/content/docs/extending/meta.json index b88a44d..0ce24e6 100644 --- a/content/docs/extending/meta.json +++ b/content/docs/extending/meta.json @@ -1 +1 @@ -{ "title": "Extending", "icon": "Puzzle", "pages": ["tools", "custom-tools", "mcp", "secrets", "report-format", "model-providers"] } +{ "title": "Extending", "icon": "Puzzle", "pages": ["tools", "custom-tools", "mcp", "secrets", "report-format", "model-providers", "dreaming"] } From 99b47b88834a677e630a2d77758b445a4ca4b5af Mon Sep 17 00:00:00 2001 From: philipph-askui Date: Tue, 18 Aug 2026 10:04:12 -0700 Subject: [PATCH 11/14] docs(best-practices): drop dreaming (moved to Extending) --- content/docs/best-practices/meta.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/docs/best-practices/meta.json b/content/docs/best-practices/meta.json index be63703..c125804 100644 --- a/content/docs/best-practices/meta.json +++ b/content/docs/best-practices/meta.json @@ -1 +1 @@ -{ "title": "Best Practices", "icon": "Lightbulb", "pages": ["writing-good-tests", "analyzing-failures", "dreaming", "security", "prompting-best-practices", "agent-behavior"] } +{ "title": "Best Practices", "icon": "Lightbulb", "pages": ["writing-good-tests", "analyzing-failures", "security", "prompting-best-practices", "agent-behavior"] } From 2cbb0a0de7a9eff1fa3ee910260bd3ffd8526719 Mon Sep 17 00:00:00 2001 From: philipph-askui Date: Tue, 18 Aug 2026 10:04:14 -0700 Subject: [PATCH 12/14] docs(best-practices): remove dreaming page (moved to Extending) --- content/docs/best-practices/dreaming.mdx | 144 ----------------------- 1 file changed, 144 deletions(-) delete mode 100644 content/docs/best-practices/dreaming.mdx diff --git a/content/docs/best-practices/dreaming.mdx b/content/docs/best-practices/dreaming.mdx deleted file mode 100644 index 237d037..0000000 --- a/content/docs/best-practices/dreaming.mdx +++ /dev/null @@ -1,144 +0,0 @@ ---- -icon: Sparkles -title: Dreaming -description: Let your suite learn from its own runs. Dreaming reviews past runs and proposes small, reviewed improvements to your rules and tests, so reliability compounds the more you run. ---- - -Where [analyzing failures](/docs/best-practices/analyzing-failures) is the manual -loop — read the report, find the cause, fix it in the right place — **Dreaming** -runs that loop for you in the background and brings you the proposed fixes to -approve. It reviews your recent runs and suggests small improvements to your -tests' rules, so your suite gets more reliable the more you run it. You stay in -control: Dreaming only *proposes* changes; nothing is edited until you approve it. - -## How it works - - - -### A prompt appears on the Dashboard -After some runs finish, you'll see **"N test runs to analyze"** with **Analyze -now** and **Remind me later**. (You can also start a pass from **Extending → -Dreaming**.) - - -### Analyze -Click **Analyze now**. Dreaming studies each run's conversation together with -whether it passed or failed — the outcome is what makes the suggestions -trustworthy rather than guesswork. A progress bar shows it working, with an -estimated time remaining. No device is needed. - - -### Review -When it finishes, the Dashboard shows **"Dreaming results ready for review"**, -along with how many runs it analyzed and what the analysis cost. Click **Review -now** to open the proposals. - - -### Accept or reject -Each proposal is shown as a diff — the exact line it would add to a `rules.md` or -change in a test — with a short reason and the runs it came from. Accept the ones -you like (individually or **Accept all**), reject the rest, then click **Finish**. -Accepted changes are written to your project files; rejected ones leave your files -untouched. - - - -Once you've reviewed a test's proposals, those runs won't be proposed again — so -each run is only ever analyzed once, unless the test regresses on later runs. - -## What Dreaming changes - -- **Rules** (`rules.md`) — the guidance your agent follows for a test or folder. - Most proposals are here: a new rule learned from a failure, or a small - refinement to an existing one. See [agent behavior](/docs/best-practices/agent-behavior) - for how rules shape a run. -- **App knowledge** (`ui.md`) — facts about the app under test. -- **Setup / teardown** (`setup.md`, `teardown.md`) — steps run before and after a test. -- **Procedures** (`procedures/*`) — your reusable, shared steps. -- **Test definitions** — occasionally, a proposed tweak to a test's own steps. - -Dreaming only ever **appends** a line or **refines** an existing one — it never -rewrites your files wholesale, and it never bakes in brittle pixel coordinates. It -does **not** change your device information — that lives with your -[device profiles](/docs/devices), not in a prose file. - -## When it's not your test — advisories - -Sometimes the trouble isn't your test at all. A tool (including your own -[custom tools](/docs/extending/custom-tools)), the AskUI SDK, or the environment -around the run can be at fault — and no change to your rules or tests would fix -that. When Dreaming spots something like that, it doesn't invent a change. Instead -it adds a short **advisory** to your review: a plain-language description of what -looked wrong, and a prominent pointer to email an AskUI solution engineer at -**support@askui.com**. - -You **acknowledge** or **dismiss** an advisory — nothing is edited either way. -It's simply Dreaming telling you "this one needs a human, not a test change." - -## It's experimental — turn it on deliberately - -Dreaming is an **experimental** feature. You find it under **Extending → -Dreaming**. When you switch it on, a warning appears explaining that learned -changes can, over time, drift and steer your tests in the wrong direction. You -tick a box to confirm you understand, then press **Turn on Dreaming** (the button -stays disabled until you tick the box). Before you start, **make your own backup -of your project** — copy the whole project folder somewhere safe, or zip it up. -Every change is saved to Git too, but a backup is the simplest safety net. - -## You need Git - -Dreaming works only when your project is a **Git repository**, and the enable -toggle stays disabled until Git is initialized. Every edit Dreaming applies is -committed to Git as its own commit (authored by "AskUI Dreaming"), so you can see -it in your history and revert it in one step if it turns out wrong. It only ever -commits the files it edits — never your other changes, your secrets, or your run -artifacts. - -## The Extending → Dreaming tab - -Everything about Dreaming lives on one tab — **Extending → Dreaming**: - -- **Turn it on** (the experimental warning and Git requirement above). -- **Choose the analysis model.** On the AskUI hub, we recommend — and default to — - the **best available** model. Dreaming runs only occasionally, so stronger reasoning - over your runs is worth far more than saving a few cents; you can instead pick "same - as your test runs" or a specific model. If you **bring your own model**, Dreaming - uses the same model as your test runs by default — untick "Same as for my test - runs" to configure a **separate** model for Dreaming (endpoint, model, API key), - for example a cheaper one, since it only reads run transcripts. -- **See what it costs.** After each pass, the tab shows the **cost of the last - analysis** and **how many analysis sessions** you've run. The same cost also - appears on the review banner and at the top of the review page, so you always - know what a round of Dreaming cost you. -- **Run it on demand.** If there are new runs to analyze, an **Analyze now** button - starts a pass right there — the same one the Dashboard offers. - -## Good to know - - -Dreaming never edits a file on its own. Every change waits for your approval, and -every applied change lands as its own Git commit you can revert. - - -- **It starts fresh from when you turn it on.** Dreaming only learns from runs you - make *after* enabling it — turning it on never trawls your entire history, so you - won't come back to hundreds of old runs waiting to be analyzed. -- **It respects edits you make yourself.** If you changed a test or rule after a - run and only then run Dreaming, it works from your *current* version — it won't - undo your change or re-suggest something you've already fixed. -- **It won't change things for the sake of it.** If your tests keep passing, - Dreaming proposes nothing — that's the expected, healthy outcome. -- **It shows how confident it is.** Each suggestion says how many of your recent - runs back it up (e.g. "4 of 5 runs"), so a recurring problem stands out from a - one-off. -- **It checks whether its past changes helped.** From your real runs since a - change, Dreaming works out whether it helped — and if a change made things - worse, it offers to undo it (you still approve the undo). -- **It remembers what it changed**, so it won't flip a file back and forth from - one session to the next. -- **It pays off most on suites you run repeatedly** (for example nightly - regression) — that's where the accumulated lessons compound. -- **It runs in the background** and never interferes with or slows down an actual - test run. -- If analysis can't finish (for example the model provider is unreachable), the - banner tells you, and your files are left untouched. From 33773b905458a0784d9ee45728a15e1fbb041fda Mon Sep 17 00:00:00 2001 From: philipph-askui Date: Tue, 18 Aug 2026 10:22:46 -0700 Subject: [PATCH 13/14] docs(best-practices): add 'When to use Dreaming' page --- content/docs/best-practices/dreaming.mdx | 45 ++++++++++++++++++++++++ 1 file changed, 45 insertions(+) create mode 100644 content/docs/best-practices/dreaming.mdx diff --git a/content/docs/best-practices/dreaming.mdx b/content/docs/best-practices/dreaming.mdx new file mode 100644 index 0000000..cb76f0b --- /dev/null +++ b/content/docs/best-practices/dreaming.mdx @@ -0,0 +1,45 @@ +--- +icon: Lightbulb +title: When to use Dreaming +description: Dreaming pays off when the same lessons keep coming up across runs — here's when to reach for it, and when to leave it off. +--- + +[Dreaming](/docs/extending/dreaming) turns the manual +[analyzing-failures](/docs/best-practices/analyzing-failures) loop into reviewed +suggestions. It's powerful, but it's not something to leave running mindlessly on +every project. Here's when it earns its keep. + +## Reach for it when… + +- **You keep fixing the same kind of thing by hand.** If [analyzing + failures](/docs/best-practices/analyzing-failures) has you adding the *same* rule + again and again — "dismiss the cookie banner first", "wait for the spinner" — + that's exactly the signal Dreaming learns from. Let it propose the rule so you + stop re-typing it. +- **You run a suite repeatedly.** Nightly regression and other suites you run often + are where the lessons compound: each run is more evidence, and a recurring problem + stands out from a one-off (Dreaming shows "4 of 5 runs" so you can tell the + difference). +- **The app under test drifts.** When the UI changes in small ways over time, + Dreaming spots the new obstacles from real runs and keeps your rules current — + instead of you chasing each drift manually. + +## Leave it off when… + +- **The suite is brand new or changing fast.** With only a run or two, there isn't + enough signal yet — write the first rules yourself, then turn Dreaming on to + maintain them. +- **A run failed for an external reason** — a flaky environment, a device that + dropped, an SDK/tooling problem. That's not a testware fix, and Dreaming will say + so (it raises an [advisory](/docs/extending/dreaming) pointing you at support + rather than inventing a rule). Sort the environment first. +- **You want a specific one-off tweak.** For a single deliberate change, just edit + the file — Dreaming is for the patterns that recur, not individual edits. + +## How it fits your workflow + +Treat Dreaming as the **maintenance loop that runs after** you've analyzed +failures, not a replacement for reading a report. You still decide what's true: +Dreaming only *proposes*, you approve each change, and every accepted edit is a +normal change in your project you can review and revert. See +[Dreaming](/docs/extending/dreaming) for how to turn it on and what it can change. From 5b1183b1b351ff3c084b90a70ca8b5474fe7232e Mon Sep 17 00:00:00 2001 From: philipph-askui Date: Tue, 18 Aug 2026 10:22:47 -0700 Subject: [PATCH 14/14] docs(best-practices): index the When-to-use Dreaming page --- content/docs/best-practices/meta.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/docs/best-practices/meta.json b/content/docs/best-practices/meta.json index c125804..be63703 100644 --- a/content/docs/best-practices/meta.json +++ b/content/docs/best-practices/meta.json @@ -1 +1 @@ -{ "title": "Best Practices", "icon": "Lightbulb", "pages": ["writing-good-tests", "analyzing-failures", "security", "prompting-best-practices", "agent-behavior"] } +{ "title": "Best Practices", "icon": "Lightbulb", "pages": ["writing-good-tests", "analyzing-failures", "dreaming", "security", "prompting-best-practices", "agent-behavior"] }