Streamline launch documentation

docs/README.md

@@ -0,0 +1,27 @@
+# Documentation
+
+Start with the [quickstart](quickstart.md), then use a recipe that matches your
+data:
+
+- [Logs, chat archives, and SQLite](recipes.md)
+- [Embed Droste in a Python application](embedding.md)
+- [Architecture overview](architecture-overview.md)
+- [Benchmark results](benchmarks.md)
+
+## Technical reference
+
+These documents specify contracts for hosts, provider authors, and contributors.
+You do not need them to use the command line tool.
+
+- [Architecture and runner protocol](architecture.md)
+- [Budgets](budgets.md)
+- [Provider manifests](provider-manifests.md)
+- [MCP over stdio](mcp-stdio.md)
+- [MCP over Streamable HTTP](mcp-http.md)
+- [Prompt packs](prompt-packs.md)
+- [RLM skills](rlm-skills.md)
+- [Scaffold manifest](scaffold-manifest.md)
+- [Trace ABI](trace-abi.md)
+- [Verifiers harness](verifiers-harness.md)
+
+Project design notes and brand assets live under [internal](internal/).

docs/architecture-overview.md

@@ -0,0 +1,31 @@
+# Architecture overview
+
+Droste keeps the source data in an execution environment instead of placing it
+in the root model prompt. On each iteration, the root model writes Python. The
+environment runs that code and returns only its printed output.
+
+```text
+question -> root model -> Python -> execution environment -> printed output
+               ^                                              |
+               +---------------- next iteration ---------------+
+```
+
+The generated program can use ordinary Python for exact work and call a
+smaller model for bounded interpretation:
+
+- `llm_query(prompt)` makes one model call.
+- `llm_query_batched(prompts)` evaluates an ordered batch concurrently.
+- `llm_batch_json(prompts, schema, ...)` validates structured batch results.
+
+The host decides which data and capabilities the program receives. Built-in
+sources cover text files and SQLite. Provider manifests and the capability
+broker allow hosts to expose additional sources without giving generated code
+direct access to transport clients.
+
+One run-scoped ledger authorizes tokens, subcalls, depth, wall time,
+iterations, and output ceilings. A separate sandbox configuration limits local
+execution and captured output. Every run can emit a versioned event stream for
+live progress, auditing, and replay.
+
+See the [technical architecture](architecture.md) for the complete broker,
+runner, provider, budget, sandbox, and trace contracts.

docs/benchmarks.md

@@ -0,0 +1,27 @@
+# Benchmark results
+
+The published suite compares direct long-context calls with Droste runs that
+use separate root and subcall models. Scores and costs below come from runs on
+July 17 and 18, 2026.
+
+| Benchmark | Scope | Best direct score | Droste score | Droste cost |
+|---|---|---:|---:|---:|
+| OOLONG | 131K tokens, 50 tasks | 0.6020 | **0.6432** | $10.16 |
+| S-NIAH | 32K tokens, 50 tasks | **1.00** | **1.00** | $0.66 |
+| LongBench-v2 CodeQA | 20 task sample | **0.75** | 0.65 | $3.79 |
+| OOLONG-Pairs | 32K tokens, 20 tasks | 0.034 | **0.80** | $2.14 |
+| BrowseComp-Plus | 6.0M to 11.1M tokens, 150 tasks | Could not run | **0.9400** | $24.54 |
+
+Droste was strongest when answers required aggregation across scattered or
+oversized context. Direct calls remained competitive on smaller lookup tasks,
+and the direct arm scored higher on the CodeQA sample.
+
+These are individual runs. Provider sampling, model changes behind an
+identifier, and trajectory variance can change the result. The OOLONG score
+ordering was not statistically separated at 95 percent confidence. Costs are
+recorded measurements from the selected price snapshot, not estimates of
+future runs.
+
+The repository's [benchmark guide](../benchmarks/README.md) contains the model
+identifiers used by each historical run, scoring rules, manifests, raw
+artifacts, reproduction commands, and complete caveats.

docs/embedding.md

@@ -0,0 +1,87 @@
+# Embed Droste
+
+The `droste` package has no runtime dependencies. An embedded run needs a root
+model client, a subcall client, an execution environment, and one shared
+execution context.
+
+```bash
+uv add droste
+```
+
+This example uses an OpenAI-compatible endpoint and a string as the dataset:
+
+```python
+from droste import (
+    Budget,
+    EnvironmentConfig,
+    OpenAICompatClient,
+    OpenAICompatSubcallClient,
+    RLMConfig,
+    create_environment,
+    create_environment_context,
+    run_rlm,
+)
+
+model = "YOUR_MODEL_ID"
+environment_config = EnvironmentConfig(
+    kind="native",
+    budget=Budget(subcalls=50, depth=1),
+)
+context = create_environment_context(environment_config)
+
+root = OpenAICompatClient(model=model)
+subcalls = OpenAICompatSubcallClient(model=model, context=context)
+environment = create_environment(
+    environment_config,
+    context="your source data",
+    registry=None,
+    subcalls=subcalls,
+    execution_context=context,
+)
+
+result = run_rlm(
+    "What happened?",
+    environment=environment,
+    root_llm=root,
+    subcalls=subcalls,
+    config=RLMConfig(root_model=model),
+    context=context,
+)
+print(result.answer)
+```
+
+`OpenAICompatClient` reads `OPENAI_API_KEY` and `OPENAI_BASE_URL`. Explicit
+constructor arguments take precedence. The package also provides
+`AnthropicClient` and `AnthropicSubcallClient`.
+
+## Data providers
+
+Pass plain data through `context`, or bind a `ProviderRegistry` for named data
+sources. Droste includes filesystem text and read-only SQLite providers. See
+[provider manifests](provider-manifests.md) when exposing an application data
+source or MCP server.
+
+## Budgets and concurrency
+
+`EnvironmentConfig` and its execution context must share one `Budget`. The
+same ledger reserves and settles root calls, subcalls, child runs, and wall
+time.
+
+Subcall clients default to five concurrent batch items. If you change
+`OpenAICompatSubcallClient(max_parallel=...)`, set the same value in
+`RLMConfig.rollout`. A mismatch fails before inference.
+
+See [budgets](budgets.md) for the full authorization model.
+
+## Policy and traces
+
+Droste does not infer enforcement rules from the wording of a question. Pass
+explicit `PolicyHints` when a product requires semantic subcalls or another
+answer contract.
+
+Attach `on_event` or configure trace retention through the execution context
+to consume structured events. The [Trace ABI](trace-abi.md) defines the event
+and retention contracts.
+
+Hosts that run Droste through HTTP or Pyodide should use the runner protocol in
+the [technical architecture](architecture.md#the-runner-protocol-embedding).

docs/internal/README.md

@@ -0,0 +1,6 @@
+# Internal project notes
+
+These documents guide maintainers and are not part of the user documentation.
+
+- [Architecture principles](architecture-principles.md)
+- [Brand and assets](brand.md)

docs/design/principles.md → docs/internal/architecture-principles.md

@@ -1,6 +1,6 @@
 # Architecture principles
 
-*The north star for droste's evolution. `architecture.md` describes what is; this
+*The north star for droste's evolution. `../architecture.md` describes what is; this
 describes what everything should converge toward, and the test every new piece of
 structure must pass.*
 

docs/brand.md → docs/internal/brand.md

@@ -38,8 +38,9 @@ Everything else: monospace, letter-spaced caps for labels. No italics.
 
 ## Assets
 
-- `docs/assets/droste.svg` / `droste-dark.svg` — the mark (light/dark)
-- `docs/assets/og.png` — 1200×630 social card (paper). Upload via GitHub
+- `../assets/droste.svg` / `../assets/droste-dark.svg` is the mark in light and
+  dark variants.
+- `../assets/og.png` is the 1200×630 social card. Upload it through GitHub
   Settings → Social preview. Regenerate: the card is plain HTML + a headless
   Chrome screenshot at 1200×630; composition = mark left, DROSTE / tagline /
   bordered `uvx` command / metadata line right, 5px inner border (frame zero

docs/prompt-packs.md

@@ -112,7 +112,7 @@ result = run_rlm(
     environment=environment,
     root_llm=root,
     subcalls=subcalls,
-    config=RLMConfig(root_model="gpt-5", prompt_profile="full"),
+    config=RLMConfig(root_model="YOUR_MODEL_ID", prompt_profile="full"),
     prompt_pack=pack,
 )
 print(result.prompt_pack.as_dict())

docs/quickstart.md

@@ -0,0 +1,62 @@
+# Quickstart
+
+Droste answers questions about files, directories, SQLite databases, and piped
+text.
+
+## Install and sign in
+
+```bash
+uv tool install droste
+droste login
+```
+
+`droste login` can use ModelRelay credits or store your own key. For scripts
+and CI, set a provider key directly:
+
+```bash
+export OPENAI_API_KEY=sk-...
+# Or: export ANTHROPIC_API_KEY=sk-ant-...
+```
+
+Set `OPENAI_BASE_URL` when using another OpenAI-compatible endpoint. You can
+also pass `--api-key` and `--base-url` for one run.
+
+For a one-off run without installation:
+
+```bash
+uvx droste "summarize the failures" server.log
+```
+
+## Ask a question
+
+```bash
+droste "what changed between these?" report.txt logs.txt
+droste "which customers churned last month?" app.db
+droste "how does authentication work?" ./src
+tail -5000 app.log | droste "why did it crash?"
+```
+
+Arguments that exist on disk are input data. The quoted argument that does not
+exist is the question. With no data argument, Droste reads the current
+directory. SQLite files are detected automatically.
+
+Directory reads skip dotfiles, binary files, and common generated directories
+such as `.git` and `node_modules`. Droste reports what it loaded and skipped
+before the run starts.
+
+## Useful options
+
+```bash
+droste "..." ./data --verbose
+droste "..." ./data --json
+droste "..." ./data --model MODEL --subcall-model SMALLER_MODEL
+droste "..." ./data --budget-subcalls 25 --max-iterations 15
+```
+
+`--verbose` streams progress and generated code to stderr. `--trace` includes
+the complete structured execution trace. Run `droste --help` for every budget,
+input, and output option.
+
+The generated Python code can read the data you provide. Database access is
+opened read-only and policy-gated, but process isolation and operating-system
+permissions remain the security boundary.

docs/recipes.md

@@ -1,35 +1,31 @@
 # Recipes
 
-Three copy-paste starting points. Every recipe works with any
-OpenAI-compatible endpoint — set `OPENAI_API_KEY` (and `OPENAI_BASE_URL` for
-non-OpenAI providers; see the README's BYOK section).
+These examples work with the account saved by `droste login` or any
+OpenAI-compatible endpoint configured with `OPENAI_API_KEY` and, when needed,
+`OPENAI_BASE_URL`.
 
 ## 1. Interrogate a huge log file
 
 ```bash
-droste "Group the errors by service and root cause. Which failure started
-   first, and did it cascade? Give me counts, not vibes." server.log
+droste "Group errors by service and cause. Which failure started first, and
+did it cascade? Include exact counts." server.log
 ```
 
-Multi-megabyte files are fine: the model is told the file's name and size,
-then pulls slices in via code. Counting, grouping, and time-ordering happen
-in Python, so the numbers are exact. Add `--verbose` to watch it work.
+The model inspects slices of the file through code. Python handles counting,
+grouping, and time ordering. Add `--verbose` to watch the run.
 
 ## 2. Ask questions across a chat/export archive
 
-Point it at an export directory's files — Slack, WhatsApp, or any
-line-oriented dump:
+Point it at Slack, WhatsApp, or another line-oriented export:
 
 ```bash
 droste export/channel-eng.txt export/channel-support.txt \
   "What did we promise customers about the migration, and did engineering's
    internal discussion match what support was telling people?"
 ```
 
-This is the shape retrieval can't do: the answer requires reading *both*
-files, aligning them in time, and judging consistency — the model narrows
-mechanically, then fans out `llm_query_batched` over the sections that need
-actual reading.
+Droste searches both files, aligns relevant passages in time, and uses model
+subcalls where judging consistency requires close reading.
 
 ## 3. Analyze a SQLite database
 
@@ -39,9 +35,7 @@ droste app.db \
    MRR walked out the door?"
 ```
 
-`--db` exposes the database as a read-only, policy-gated source (SELECT-only,
-single statement, bounded rows). The model reads the schema, iterates on
-queries, and computes in code. The policy is a guardrail, not a security
-boundary — the file is opened read-only, and OS permissions are the real
-control. Cheap subcalls under a stronger root: add
-`--subcall-model gpt-5.2-mini` (or any cheaper model on your endpoint).
+Droste detects SQLite files, reads the schema, writes bounded read-only
+queries, and computes over the results. Operating-system permissions remain
+the security boundary. To use a cheaper model for subcalls, add
+`--subcall-model MODEL`.