diff --git a/README.md b/README.md index ec8f5ad..733fc77 100644 --- a/README.md +++ b/README.md @@ -1,17 +1,16 @@ - Droste — nested frames converging on the answer + Droste # Droste -**A recursive analysis engine for data too large for a context window.** +**Ask questions about data too large for a context window.** -Droste implements the Recursive Language Model (RLM) technique. Rather than -placing an entire corpus in the root model's context, it exposes the corpus -through a sandboxed Python REPL. The model writes programs over that data and -delegates bounded semantic judgments through `llm_query` and -`llm_query_batched`. +Droste is an open source Recursive Language Model (RLM) engine. It gives a +model a Python REPL over your data, so the model can search, filter, count, and +delegate small pieces of interpretation without loading the full corpus into +its context. ```bash uvx droste "which customer had a failed charge, and why?" server.log @@ -21,377 +20,82 @@ uvx droste "how do the authentication flows differ?" ./docs ![droste answering a two-part question over a 444 kB server log, streaming its code as it works](docs/assets/demo.gif) -The first example runs against a 444 kB log: +## Why Droste -``` -$ droste "Which customer had a failed charge, for what amount, and why? - How many timeout errors are there, and which upstream do they blame?" server.log - -1. **Failed Charge Details**: - - **Customer**: `cus_9982` - - **Amount**: 1499 (USD, which is $14.99) - - **Reason**: The card was declined due to insufficient funds - (`reason=card_declined decline_code=insufficient_funds`). - -2. **Timeout Errors**: - - **Count**: There are exactly 66 timeout errors in the log. - - **Upstream blamed**: They blame `payments-v2` (`upstream=payments-v2`). -``` - -The counts are exact because the model *counted them in Python* — it never -read 3,400 log lines through its attention. In `--db` mode the model -introspects your schema, writes read-only SQL, and computes over the rows; -in the demo above it noticed the free plan makes refund-rate-vs-MRR -undefined and answered for the paid plans instead. - -## How it works - -Mechanical work stays mechanical: regex and SQL find *where*, model subcalls -interpret *what*, and code combines the results. The root model can inspect -the shape of the corpus, narrow it without model calls, and fan out only when -a step requires semantic judgment. +Code handles work that should be exact, such as searching, counting, joining, +and aggregating. Model calls handle the parts that require interpretation. +Droste combines both while keeping the full dataset outside the model's +context window. -This is a different data path from a general coding or tool agent. Agents -choose actions across open-ended tasks, and every observation they make — -file reads, search results, subagent reports — returns through the model's -conversation context. Droste has a narrower job: the corpus lives as a -variable in the REPL, the model sees only what its code chooses to print, and -selected slices go to bounded subcalls instead of accumulating in the -transcript. Code locates and aggregates; subcalls interpret bounded inputs; -the root model assembles the answer. - -Execution is bounded by explicit iteration, subcall, and output limits. Root -and subcall models can be configured independently. These controls make the -work observable and limitable; they are not a promise of a particular answer -quality, latency, or price, which depend on the data, models, and endpoint. - -## When to reach for it - -Each of these follows from the mechanism above, not from benchmark claims: - -- **Exact answers over large mechanical data.** Counts, aggregates, and joins - over logs, exports, or SQLite — where attention over thousands of lines - approximates, but `len()` and `GROUP BY` do not. -- **Semantic judgment at scale.** "Classify/judge each of these N records": - code selects the slices, `llm_query_batched` fans out bounded subcalls, - and code tallies the results. The full corpus does not have to pass - through the root model's context window. -- **Mixed questions.** Answers that need exact computation *and* actual - reading — "which plan has the highest refund rate, and what do those - customers complain about?" — the case neither pure SQL nor pure - long-context reading handles alone. -- **Embedding a bounded question-answering primitive.** Product features that - answer questions over user data behind hard compute budgets and audit - traces. Generated code gets no open-ended tool selection — only the data - bindings the host configures — though execution isolation remains the - host's job (see [Embed it](#embed-it)). - -When the data fits comfortably in a context window, or the task is -open-ended multi-step work rather than a question with an answer, use a -general agent — droste is deliberately not one. Three worked starting points -live in [docs/recipes.md](docs/recipes.md) (logs, chat archives, SQLite). +The root model sees only what its code prints and the bounded results of any +subcalls it makes. Runs have explicit limits for iterations, subcalls, time, +and model output. ## Use it -Ask questions over files, folders, and SQLite from the terminal. The -contract: **args that exist are data, the one that doesn't is the question, -no args means the current directory, pipes are data too — and it always -prints one line saying what it read.** +Install the command and sign in once: ```bash -uvx droste "…" ./docs # zero-install, npx-style -uv tool install droste # or keep the binary around -pipx install droste # the older equivalent +uv tool install droste +droste login ``` +Then ask a question about files, directories, SQLite databases, or piped text: + ```bash -droste login # one-time setup: free credits, or your own key droste "what changed between these?" report.txt logs.txt droste "which customers churned last month?" app.db -droste "how does auth work here?" ./docs -cd ~/notes && droste "what did I decide about pricing?" tail -5000 app.log | droste "why did it crash?" ``` -SQLite files are recognized by their magic bytes — no flag needed (`--db` -remains as an explicit override). Directory walks skip binaries, dotfiles, -and the usual junk (`.git`, `node_modules`, …) and cap sizes -(`--max-file-bytes`, `--max-bytes`); every skip is counted in the report -line. `droste ask …` still works as an alias. - -Files are materialized as the sandbox's `context` variable — the model is -told each file's name and size (not its contents) and pulls data in via -code, so multi-MB files are fine. What the model reads is whatever its code -chooses to print. `--db` uses the engine's local-mode SQL data source (read-only -policy as a guardrail, not a boundary; OS permissions are the boundary). - -Engine knobs mirror `RLMConfig`: `--subcall-model`, -`--subcall-max-output-tokens` (default 2048), `--reasoning-effort`, -`--max-iterations`, `--max-subcalls`. `--json` prints a result object for -scripting; `--verbose` streams one-line progress to stderr (watch it think); -`--trace` renders the full structured event stream — generated code, execution -output with per-iteration sub-call counts and answer state, LLM responses, -execution errors. Exit code 0 means a confirmed (or extracted-with-note) -answer. - -Droste is the open execution engine. Compatible hosted gateways and control -planes can add authentication, server-enforced policy and cost limits, and -audit around it; those services are integrations, not part of the engine. -Use `--base-url` to select a compatible endpoint. - -## Embed it - -The same wheel is the engine as a library — zero runtime dependencies, -`urllib`-only. Add it to your app and point the loop at your own data -sources: - -```bash -uv add droste # or: pip install droste -``` - -Using is asking over *your* data; embedding is building RLM answers into a -product for *your users*. - -### BYOK: compatible endpoints +For a one-off run, use `uvx droste` instead of installing it. The command also +supports your own OpenAI-compatible or Anthropic endpoint. See the +[quickstart](docs/quickstart.md) for credentials, directories, SQLite, and +useful flags. -The engine includes an OpenAI-compatible client and an Anthropic Messages -client. Configure the corresponding API key and model identifier; an explicit -base URL selects a compatible endpoint. Bring your own key — no hosted account -required. The CLI detects the protocol from credential and endpoint -configuration, and an explicit `--base-url`/`OPENAI_BASE_URL` always wins. +## Good fits -```bash -export ANTHROPIC_API_KEY=sk-ant-... -droste "why did it crash?" ./logs --model claude-opus-4-8 -``` +- Exact counts, aggregates, and joins over logs, exports, or databases. +- Classification or review across many records using bounded model batches. +- Questions that need both computation and close reading. -```python -from droste import ( - Budget, - EnvironmentConfig, - OpenAICompatClient, - OpenAICompatSubcallClient, - SandboxLimits, - create_environment, - create_environment_context, - run_rlm, -) - -environment_config = EnvironmentConfig( - kind="native", - budget=Budget(subcalls=50, depth=1), - sandbox=SandboxLimits(output_chars=25_000), -) -context = create_environment_context(environment_config) -root = OpenAICompatClient(model="gpt-5.2-mini") # OPENAI_API_KEY / OPENAI_BASE_URL from env -subcalls = OpenAICompatSubcallClient( - model="gpt-5.2-mini", - context=context, # shared call/token accounting - max_output_tokens=2048, # per-subcall output bound (cost control) -) - -env = create_environment( - environment_config, - context=data, - registry=registry, - subcalls=subcalls, - execution_context=context, -) -result = run_rlm(question, environment=env, root_llm=root, subcalls=subcalls, context=context) -``` +If the data already fits comfortably in a context window, or the task is +open-ended agent work rather than a question about a corpus, use a general +agent instead. The [recipes](docs/recipes.md) cover logs, chat archives, and +SQLite. -Explicit `base_url=` / `api_key=` constructor args win over the environment -variables. Subcall batches use the immutable rollout concurrency (default 5), -and every subcall's usage block is added to `result.tokens_used`. When choosing -a non-default value in-process, pass the same value as the built-in subcall -client's `max_parallel` and `RolloutConfiguration.concurrency`; a mismatch -fails before inference. +## Results -`reasoning_effort` and `extra_body` pass through to the endpoint as-is. -Disabling thinking per-subcall is a gateway capability: a compatible gateway -may enforce it server-side, while raw endpoints may ignore a client-side -disable. +Published runs show the largest gains on tasks that require aggregation across +scattered or oversized context. -### Runner architecture (droste_runner) +| Benchmark | Direct baseline | Droste | Recorded cost | +|---|---:|---:|---:| +| OOLONG, 131K tokens | 0.6020 | **0.6432** | $10.16 vs $26.18 | +| OOLONG-Pairs, 32K tokens | 0.034 | **0.80** | $2.14 vs $2.50 | +| BrowseComp-Plus, 6.0M to 11.1M tokens | Could not run | **0.9400** | $24.54 | -The `droste_runner` package is a thin orchestration layer that wires `droste` to -HTTP-backed root LLM calls and subcalls. It is shared across hosted and -in-process embedders so the loop logic stays in one place. For custom environments, -set `adapter_module` in the runner request to delegate to an adapter module's -`run(request)` function. +These are individual published runs, not universal quality or cost guarantees. +See the [results and caveats](docs/benchmarks.md), or open the repository's +[benchmark guide](benchmarks/README.md) for artifacts and reproduction steps. -```mermaid -flowchart LR - Host[Host App] --> Runner[droste_runner] - Runner --> Core[droste run_rlm] - Runner --> Env[RunnerEnvironment] - Env --> Sandbox[Python REPL execute] +## Use it as a library - Core --> RootLLM[LLMClient responses_create] - RootLLM --> Responses[Host /responses] - - Core --> Subcalls[SubcallClient llm_query llm_batch] - Subcalls --> SubcallAPI[Host /rlm/subcall] -``` - -**Runner inputs** - -- `protocol_version`: **required** on every request (currently `10`) — a - missing or mismatched version gets a structured refusal, so hosts detect - incompatibility instead of failing on a missing field. -- `budget`: **required** complete seven-field compute authorization object. -- `root_endpoint` + `subcall_endpoint` + `token`: required for HTTP-backed - runs. -- `operation`: `run` (default) or `preflight`; preflight resolves and checks - the content-free scaffold without model/provider calls or endpoint - credentials. -- Optional: `subcall_concurrency` (default `5`), `root_reasoning_effort` - (sent unchanged on every root callback), `adapter_module` (delegate the - runner to a custom module's `run(request)`). - -Refusal envelopes, operation semantics, and the compatibility window live in -[docs/architecture.md](docs/architecture.md) ("The runner protocol"); -per-release embedder migration notes are in [UPGRADING.md](UPGRADING.md). - -### Core concepts - -Implement these protocols to integrate with your infrastructure: - -- **`RLMEnvironment`** - Sandboxed Python REPL with data access -- **`LLMClient`** - Chat completion interface for the root LLM -- **`SubcallClient`** - Provides `llm_query()` and `llm_batch()` for sub-LLM calls -- **`SubcallOutputTokenLimitProvider`** - Optional companion protocol exposing a - read-only `output_token_limit`: a positive per-call token ceiling or `None` - when deliberately unbounded. Clients that omit it remain compatible and are - reported to the root model as having an unknown limit. -- **`ProviderManifest`** - Immutable data-operation metadata - -Data access is descriptor-driven: a reusable `ProviderManifest` declares each -provider's operations, schemas, and pagination; a host binds it with its own -side-effect classifications and policy through an explicit `ProviderCatalog`. -The bundled local providers are SQLite and `filesystem_text` (bounded -`list_files`, `read`, literal `grep`, index-free `search`, and `stat` over an -explicitly configured directory). Trusted hosts may also acquire MCP servers -as the same provider abstraction; generated code still receives only -descriptor-generated broker bindings. See -[Provider manifests](docs/provider-manifests.md) for the value model and -ownership boundaries, plus the [local stdio](docs/mcp-stdio.md) and -[Streamable HTTP](docs/mcp-http.md) MCP transport contracts. - -Every run emits a strict, versioned structured event stream — the -[Trace ABI](docs/trace-abi.md) gives each event one run identity, sequence, -and retention class. Retaining replay content and authorizing training use -are separate, default-denied decisions. The wheel includes the exact -cross-runtime conformance corpus for embedders. - -### Configuration - -```python -RLMConfig( - budget=Budget( - tokens=500_000, - subcalls=50, - depth=1, - wall_ms=300_000, - max_iterations=30, - root_output_tokens=4_096, - subcall_output_tokens=2_048, - ), - sandbox=SandboxLimits(output_chars=25_000), - prompt_profile="full", # Versioned prompt-pack profile (full/minimal/none) - policy_hints=PolicyHints(semantic=True), # Optional explicit contract -) -``` - -Compute authorization is one immutable vector, reconciled by one run-scoped -ledger. See [Budgets](docs/budgets.md). Sandbox output and execution guardrails -are separate because they describe the local REPL, not model/provider spend. - -Harness prompts resolve once per run from immutable, versioned data. See -[Prompt packs](docs/prompt-packs.md) for the stable five-slot contract, custom -pack loading, deterministic fallback order, and provenance records. - -Droste does not infer semantic intent from the question. When a caller supplies -`PolicyHints(semantic=True)`, at least one semantic subcall must succeed and any -incomplete `llm_batch_json` result blocks confirmation. Only an error-free -repeat with the exact prompts, contexts, schema, and validator object resolves -that partial evidence. Omit the hint to retain purely prompt-driven behavior. - -Embedders with product-specific metadata contracts may pass -`ready_metadata_validator` to `run_rlm`. It receives a detached copy of -validated `answer["metadata"]` only after Droste's core ready checks pass. -Return an empty sequence to accept it, or content-free violation strings to -revoke readiness and send those violations through the existing root repair -path. The validator is synchronous trusted host logic; it must not dispatch -model calls or mutate external state. - -### Result - -```python -RLMResult( - answer="...", # Final answer from answer["content"] - ready=True, # Whether answer["ready"] was set - iterations=3, # Iterations used - tokens_used=1500, # Total tokens consumed - sub_calls_made=12, # Total llm_query/llm_batch calls - trajectory=[...], # Full execution history - extracted=False, # True if the answer came from the post-exhaustion - # extract pass (best-effort, not confirmed) - prompt_pack=..., # Frozen resolved pack identity + provenance -) -``` - -## Benchmarks - -The repository ships a [versioned benchmark suite](benchmarks/README.md) with -per-task artifacts and offline reports. BrowseComp-Plus uses its official -model-judged semantic-equivalence methodology; the other reported scorers are -deterministic. Scores and measured costs below are from the published -2026-07-17 and 2026-07-18 runs; each result cell is score / cost. - -| Benchmark | Scope | Direct Sol | Direct Terra | Droste Terra + Luna | Outcome | -|---|---|---:|---:|---:|---| -| [OOLONG](benchmarks/README.md#oolong-trec_coarse) | 131K tokens, 50 tasks | 0.6020 / $26.18 | 0.5668 / $12.47 | **0.6432 / $10.16** | Best score at 2.6× lower cost than the best direct arm | -| [S-NIAH](benchmarks/README.md#s-niah) | 32K tokens, 50 tasks | 0.84 / $7.79 | **1.00 / $3.90** | **1.00 / $0.66** | Ties best accuracy at 5.9× lower cost | -| [LongBench-v2 CodeQA](benchmarks/README.md#longbench-v2-codeqa) | Cost-bounded 20-of-50 sample | **0.75 / $19.60** | 0.65 / $9.10 | 0.65 / $3.79 | Mixed: ties Terra, trails Sol by 0.10, and costs 5.2× less than Sol | -| [OOLONG-Pairs](benchmarks/README.md#oolong-pairs) | 32K tokens, 20 tasks | 0.00 / $0 | 0.034 / $2.50 | **0.80 / $2.14** | Strongest result: direct approaches structurally fail; Droste reaches 0.80 F1 at lower recorded cost | -| [BrowseComp-Plus](benchmarks/README.md#browsecomp-plus) | 6.0M–11.1M tokens, 150 tasks | N/A / $0 | N/A / $0 | **0.9400 / $24.54** | Direct approaches cannot attempt the task at any cost; Droste completes 148/150 at 0.9400 judged accuracy | - -Across the published suite, Droste wins or ties on accuracy in all but CodeQA -and is dramatically more cost-efficient wherever a direct comparison can run. -Its clearest wins are on tasks that require aggregation across scattered -context (OOLONG and OOLONG-Pairs) and on BrowseComp-Plus, where the raw context -is 6–10× beyond any model window. On smaller lookup-shaped tasks (S-NIAH and -CodeQA), direct approaches with sufficient context remain competitive on -accuracy. - -[Full results, methodology, provenance, and caveats for every published -family](benchmarks/README.md#results) are documented in the benchmark guide. +The same package is a dependency-free Python library. You can supply your own +models and data providers, set hard compute budgets, and consume structured +traces. Start with the [embedding guide](docs/embedding.md). Protocol and host +implementers can use the [technical reference](docs/README.md#technical-reference). ## Development ```bash -uv sync --extra verifiers # Install the full test surface when supported -uv run pytest # Verifiers tests skip when its extra is unavailable -uv build # Build wheel +uv sync --extra verifiers +uv run pytest +uv build ``` -## The name - -The [Droste effect](https://en.wikipedia.org/wiki/Droste_effect) is the -picture that contains itself. M.C. Escher's *Print Gallery* pushed it to its -limit — a man in a gallery viewing a print that contains the gallery he is -standing in — and Escher left the center of the spiral famously blank, -signed but uncompleted, where the recursion outran his hand. Fifty years -later, mathematicians completed it; their project was titled *"The -Mathematics Behind the Droste Effect."* - -The answer at the center of the spiral — the part the picture couldn't hold -— is what recursion computes. +See [CONTRIBUTING.md](CONTRIBUTING.md) for the contribution workflow. ## License -Apache-2.0. See [LICENSE](LICENSE). Contributions welcome — -[CONTRIBUTING.md](CONTRIBUTING.md). Versioning is semver; the runner -protocol and source-registry contract carry an explicit compatibility -window (see [docs/architecture.md](docs/architecture.md)). +Apache-2.0. See [LICENSE](LICENSE). diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000..52dcbbd --- /dev/null +++ b/docs/README.md @@ -0,0 +1,27 @@ +# Documentation + +Start with the [quickstart](quickstart.md), then use a recipe that matches your +data: + +- [Logs, chat archives, and SQLite](recipes.md) +- [Embed Droste in a Python application](embedding.md) +- [Architecture overview](architecture-overview.md) +- [Benchmark results](benchmarks.md) + +## Technical reference + +These documents specify contracts for hosts, provider authors, and contributors. +You do not need them to use the command line tool. + +- [Architecture and runner protocol](architecture.md) +- [Budgets](budgets.md) +- [Provider manifests](provider-manifests.md) +- [MCP over stdio](mcp-stdio.md) +- [MCP over Streamable HTTP](mcp-http.md) +- [Prompt packs](prompt-packs.md) +- [RLM skills](rlm-skills.md) +- [Scaffold manifest](scaffold-manifest.md) +- [Trace ABI](trace-abi.md) +- [Verifiers harness](verifiers-harness.md) + +Project design notes and brand assets live under [internal](internal/). diff --git a/docs/architecture-overview.md b/docs/architecture-overview.md new file mode 100644 index 0000000..da1029f --- /dev/null +++ b/docs/architecture-overview.md @@ -0,0 +1,31 @@ +# Architecture overview + +Droste keeps the source data in an execution environment instead of placing it +in the root model prompt. On each iteration, the root model writes Python. The +environment runs that code and returns only its printed output. + +```text +question -> root model -> Python -> execution environment -> printed output + ^ | + +---------------- next iteration ---------------+ +``` + +The generated program can use ordinary Python for exact work and call a +smaller model for bounded interpretation: + +- `llm_query(prompt)` makes one model call. +- `llm_query_batched(prompts)` evaluates an ordered batch concurrently. +- `llm_batch_json(prompts, schema, ...)` validates structured batch results. + +The host decides which data and capabilities the program receives. Built-in +sources cover text files and SQLite. Provider manifests and the capability +broker allow hosts to expose additional sources without giving generated code +direct access to transport clients. + +One run-scoped ledger authorizes tokens, subcalls, depth, wall time, +iterations, and output ceilings. A separate sandbox configuration limits local +execution and captured output. Every run can emit a versioned event stream for +live progress, auditing, and replay. + +See the [technical architecture](architecture.md) for the complete broker, +runner, provider, budget, sandbox, and trace contracts. diff --git a/docs/benchmarks.md b/docs/benchmarks.md new file mode 100644 index 0000000..e1ace13 --- /dev/null +++ b/docs/benchmarks.md @@ -0,0 +1,27 @@ +# Benchmark results + +The published suite compares direct long-context calls with Droste runs that +use separate root and subcall models. Scores and costs below come from runs on +July 17 and 18, 2026. + +| Benchmark | Scope | Best direct score | Droste score | Droste cost | +|---|---|---:|---:|---:| +| OOLONG | 131K tokens, 50 tasks | 0.6020 | **0.6432** | $10.16 | +| S-NIAH | 32K tokens, 50 tasks | **1.00** | **1.00** | $0.66 | +| LongBench-v2 CodeQA | 20 task sample | **0.75** | 0.65 | $3.79 | +| OOLONG-Pairs | 32K tokens, 20 tasks | 0.034 | **0.80** | $2.14 | +| BrowseComp-Plus | 6.0M to 11.1M tokens, 150 tasks | Could not run | **0.9400** | $24.54 | + +Droste was strongest when answers required aggregation across scattered or +oversized context. Direct calls remained competitive on smaller lookup tasks, +and the direct arm scored higher on the CodeQA sample. + +These are individual runs. Provider sampling, model changes behind an +identifier, and trajectory variance can change the result. The OOLONG score +ordering was not statistically separated at 95 percent confidence. Costs are +recorded measurements from the selected price snapshot, not estimates of +future runs. + +The repository's [benchmark guide](../benchmarks/README.md) contains the model +identifiers used by each historical run, scoring rules, manifests, raw +artifacts, reproduction commands, and complete caveats. diff --git a/docs/embedding.md b/docs/embedding.md new file mode 100644 index 0000000..6f3593e --- /dev/null +++ b/docs/embedding.md @@ -0,0 +1,87 @@ +# Embed Droste + +The `droste` package has no runtime dependencies. An embedded run needs a root +model client, a subcall client, an execution environment, and one shared +execution context. + +```bash +uv add droste +``` + +This example uses an OpenAI-compatible endpoint and a string as the dataset: + +```python +from droste import ( + Budget, + EnvironmentConfig, + OpenAICompatClient, + OpenAICompatSubcallClient, + RLMConfig, + create_environment, + create_environment_context, + run_rlm, +) + +model = "YOUR_MODEL_ID" +environment_config = EnvironmentConfig( + kind="native", + budget=Budget(subcalls=50, depth=1), +) +context = create_environment_context(environment_config) + +root = OpenAICompatClient(model=model) +subcalls = OpenAICompatSubcallClient(model=model, context=context) +environment = create_environment( + environment_config, + context="your source data", + registry=None, + subcalls=subcalls, + execution_context=context, +) + +result = run_rlm( + "What happened?", + environment=environment, + root_llm=root, + subcalls=subcalls, + config=RLMConfig(root_model=model), + context=context, +) +print(result.answer) +``` + +`OpenAICompatClient` reads `OPENAI_API_KEY` and `OPENAI_BASE_URL`. Explicit +constructor arguments take precedence. The package also provides +`AnthropicClient` and `AnthropicSubcallClient`. + +## Data providers + +Pass plain data through `context`, or bind a `ProviderRegistry` for named data +sources. Droste includes filesystem text and read-only SQLite providers. See +[provider manifests](provider-manifests.md) when exposing an application data +source or MCP server. + +## Budgets and concurrency + +`EnvironmentConfig` and its execution context must share one `Budget`. The +same ledger reserves and settles root calls, subcalls, child runs, and wall +time. + +Subcall clients default to five concurrent batch items. If you change +`OpenAICompatSubcallClient(max_parallel=...)`, set the same value in +`RLMConfig.rollout`. A mismatch fails before inference. + +See [budgets](budgets.md) for the full authorization model. + +## Policy and traces + +Droste does not infer enforcement rules from the wording of a question. Pass +explicit `PolicyHints` when a product requires semantic subcalls or another +answer contract. + +Attach `on_event` or configure trace retention through the execution context +to consume structured events. The [Trace ABI](trace-abi.md) defines the event +and retention contracts. + +Hosts that run Droste through HTTP or Pyodide should use the runner protocol in +the [technical architecture](architecture.md#the-runner-protocol-embedding). diff --git a/docs/internal/README.md b/docs/internal/README.md new file mode 100644 index 0000000..d6aef48 --- /dev/null +++ b/docs/internal/README.md @@ -0,0 +1,6 @@ +# Internal project notes + +These documents guide maintainers and are not part of the user documentation. + +- [Architecture principles](architecture-principles.md) +- [Brand and assets](brand.md) diff --git a/docs/design/principles.md b/docs/internal/architecture-principles.md similarity index 99% rename from docs/design/principles.md rename to docs/internal/architecture-principles.md index c417bac..e56ce3f 100644 --- a/docs/design/principles.md +++ b/docs/internal/architecture-principles.md @@ -1,6 +1,6 @@ # Architecture principles -*The north star for droste's evolution. `architecture.md` describes what is; this +*The north star for droste's evolution. `../architecture.md` describes what is; this describes what everything should converge toward, and the test every new piece of structure must pass.* diff --git a/docs/brand.md b/docs/internal/brand.md similarity index 90% rename from docs/brand.md rename to docs/internal/brand.md index e7f22fb..dc9b556 100644 --- a/docs/brand.md +++ b/docs/internal/brand.md @@ -38,8 +38,9 @@ Everything else: monospace, letter-spaced caps for labels. No italics. ## Assets -- `docs/assets/droste.svg` / `droste-dark.svg` — the mark (light/dark) -- `docs/assets/og.png` — 1200×630 social card (paper). Upload via GitHub +- `../assets/droste.svg` / `../assets/droste-dark.svg` is the mark in light and + dark variants. +- `../assets/og.png` is the 1200×630 social card. Upload it through GitHub Settings → Social preview. Regenerate: the card is plain HTML + a headless Chrome screenshot at 1200×630; composition = mark left, DROSTE / tagline / bordered `uvx` command / metadata line right, 5px inner border (frame zero diff --git a/docs/prompt-packs.md b/docs/prompt-packs.md index d6238c5..860e2f5 100644 --- a/docs/prompt-packs.md +++ b/docs/prompt-packs.md @@ -112,7 +112,7 @@ result = run_rlm( environment=environment, root_llm=root, subcalls=subcalls, - config=RLMConfig(root_model="gpt-5", prompt_profile="full"), + config=RLMConfig(root_model="YOUR_MODEL_ID", prompt_profile="full"), prompt_pack=pack, ) print(result.prompt_pack.as_dict()) diff --git a/docs/quickstart.md b/docs/quickstart.md new file mode 100644 index 0000000..c09150e --- /dev/null +++ b/docs/quickstart.md @@ -0,0 +1,62 @@ +# Quickstart + +Droste answers questions about files, directories, SQLite databases, and piped +text. + +## Install and sign in + +```bash +uv tool install droste +droste login +``` + +`droste login` can use ModelRelay credits or store your own key. For scripts +and CI, set a provider key directly: + +```bash +export OPENAI_API_KEY=sk-... +# Or: export ANTHROPIC_API_KEY=sk-ant-... +``` + +Set `OPENAI_BASE_URL` when using another OpenAI-compatible endpoint. You can +also pass `--api-key` and `--base-url` for one run. + +For a one-off run without installation: + +```bash +uvx droste "summarize the failures" server.log +``` + +## Ask a question + +```bash +droste "what changed between these?" report.txt logs.txt +droste "which customers churned last month?" app.db +droste "how does authentication work?" ./src +tail -5000 app.log | droste "why did it crash?" +``` + +Arguments that exist on disk are input data. The quoted argument that does not +exist is the question. With no data argument, Droste reads the current +directory. SQLite files are detected automatically. + +Directory reads skip dotfiles, binary files, and common generated directories +such as `.git` and `node_modules`. Droste reports what it loaded and skipped +before the run starts. + +## Useful options + +```bash +droste "..." ./data --verbose +droste "..." ./data --json +droste "..." ./data --model MODEL --subcall-model SMALLER_MODEL +droste "..." ./data --budget-subcalls 25 --max-iterations 15 +``` + +`--verbose` streams progress and generated code to stderr. `--trace` includes +the complete structured execution trace. Run `droste --help` for every budget, +input, and output option. + +The generated Python code can read the data you provide. Database access is +opened read-only and policy-gated, but process isolation and operating-system +permissions remain the security boundary. diff --git a/docs/recipes.md b/docs/recipes.md index 8f22052..5d201cf 100644 --- a/docs/recipes.md +++ b/docs/recipes.md @@ -1,24 +1,22 @@ # Recipes -Three copy-paste starting points. Every recipe works with any -OpenAI-compatible endpoint — set `OPENAI_API_KEY` (and `OPENAI_BASE_URL` for -non-OpenAI providers; see the README's BYOK section). +These examples work with the account saved by `droste login` or any +OpenAI-compatible endpoint configured with `OPENAI_API_KEY` and, when needed, +`OPENAI_BASE_URL`. ## 1. Interrogate a huge log file ```bash -droste "Group the errors by service and root cause. Which failure started - first, and did it cascade? Give me counts, not vibes." server.log +droste "Group errors by service and cause. Which failure started first, and +did it cascade? Include exact counts." server.log ``` -Multi-megabyte files are fine: the model is told the file's name and size, -then pulls slices in via code. Counting, grouping, and time-ordering happen -in Python, so the numbers are exact. Add `--verbose` to watch it work. +The model inspects slices of the file through code. Python handles counting, +grouping, and time ordering. Add `--verbose` to watch the run. ## 2. Ask questions across a chat/export archive -Point it at an export directory's files — Slack, WhatsApp, or any -line-oriented dump: +Point it at Slack, WhatsApp, or another line-oriented export: ```bash droste export/channel-eng.txt export/channel-support.txt \ @@ -26,10 +24,8 @@ droste export/channel-eng.txt export/channel-support.txt \ internal discussion match what support was telling people?" ``` -This is the shape retrieval can't do: the answer requires reading *both* -files, aligning them in time, and judging consistency — the model narrows -mechanically, then fans out `llm_query_batched` over the sections that need -actual reading. +Droste searches both files, aligns relevant passages in time, and uses model +subcalls where judging consistency requires close reading. ## 3. Analyze a SQLite database @@ -39,9 +35,7 @@ droste app.db \ MRR walked out the door?" ``` -`--db` exposes the database as a read-only, policy-gated source (SELECT-only, -single statement, bounded rows). The model reads the schema, iterates on -queries, and computes in code. The policy is a guardrail, not a security -boundary — the file is opened read-only, and OS permissions are the real -control. Cheap subcalls under a stronger root: add -`--subcall-model gpt-5.2-mini` (or any cheaper model on your endpoint). +Droste detects SQLite files, reads the schema, writes bounded read-only +queries, and computes over the results. Operating-system permissions remain +the security boundary. To use a cheaper model for subcalls, add +`--subcall-model MODEL`. diff --git a/pyodide/README.md b/pyodide/README.md index fea959b..9d565a1 100644 --- a/pyodide/README.md +++ b/pyodide/README.md @@ -1,45 +1,18 @@ # Deno + Pyodide RLM substrate -The Deno + Pyodide (CPython-on-WASM) substrate runs droste's RLM engine under -one Deno binary + an offline Pyodide runtime — no `Python.framework`, no -wheelhouse, no per-`.so` code signing. It started as a feasibility spike (see -History, below) and now ships as the default substrate inside a production -macOS `.app` bundle. - -This substrate has two layers, and the split matters if you're adopting droste -for your own host: - -- **The substrate itself** — droste-general, reusable, ModelRelay-shaped but - host-agnostic: `relay.ts`, `event_channel.ts`, `broker.ts`, `events.ts`, - `stream.ts`, and the - Python adapters in `droste.substrates.pyodide` (which ship inside `pip install - droste`). Plus the A′-1 / A′-2 security machinery, whose droste-side half - (`droste.sources.bridge`) is also part of the installable package. **None of - this knows anything about any particular host's data layer** — `relay.ts` - included. It's fully adapter-agnostic: it takes the name of a host adapter - module as a CLI argument and drives it through a fixed two-function contract - (below), never referencing any specific host. -- **A host adapter** — one small Python module per host that wires the - substrate up to that host's own data source and request/response shape. This - is *not* part of the droste package; each host writes its own. Two exist: - - `examples/pyodide-host/pyodide_host_adapter.py` — droste's own minimal - reference adapter, built entirely from droste-general pieces, with zero - third-party or product dependencies. It ships in this repo as the worked - example and the CI proof that `relay.ts` actually works end to end. - Copy the *shape* of this file, substituting your own data source and result - format. - - A production host's adapter — backed by the host's own data-layer package - instead of `droste.sources.sql_local`. It lives in the host's repo, **not - here** — droste's repo has no host-specific code anywhere under `pyodide/`. - Such an adapter is analogous in shape to the example adapter, just with the - host's own request contract and data layer. - -If you're reading this to embed droste elsewhere: everything in "the substrate" -is yours to reuse as-is; write one adapter module of your own and point -`relay.ts` at it. - -**Getting the Deno half:** the relay ships **inside the droste wheel** (#33) -as package data under `droste/substrates/_relay/` — `relay.ts`, +The Deno + Pyodide substrate runs Droste in CPython on WASM with an offline +Pyodide runtime. It has two parts: + +- The reusable relay and Python substrate included in the `droste` wheel. +- A host-owned adapter that supplies data sources and maps the host's request + and response types. + +The relay has no host-specific data logic. It loads the adapter module named by +the trusted process launcher and calls its fixed two-function contract. See +`examples/pyodide-host/pyodide_host_adapter.py` for a working adapter. + +The relay ships inside the wheel as package data under +`droste/substrates/_relay/`: `relay.ts`, `event_channel.ts`, `stream.ts`, `broker.ts`, `events.ts`, `offline-probe.ts`, and `deps.ts` (the single Pyodide version pin). Stage it in your build from the installed package: @@ -48,15 +21,11 @@ installed package: cp "$relay_dir"/*.ts / (Or `python -c 'from droste.substrates import relay_dir; print(relay_dir())'`.) -The wheel is already the one pinned, hash-verified artifact in your lockfile, -so relay and engine are version-locked by construction — no separate tarball -download, no sha pin to keep in sync. On an admitted run, the relay prefixes +The relay and engine therefore come from the same pinned artifact. On an +admitted run, the relay prefixes the first canonical event with a `startup` event -(`{engine_version, runner_protocol, provider_protocol}`) so contract adoption -is a structured signal, not a changelog audit. Preflight and pre-admission -refusal leave the event descriptor empty. (Tagged releases -still attach `droste-relay-vX.Y.Z.tar.gz` as a convenience for non-Python -consumers; it is no longer the embedder path.) +(`{engine_version, runner_protocol, provider_protocol}`). Preflight and +pre-admission refusal leave the event descriptor empty. In this repo the relay sources live at `src/droste/substrates/_relay/`; this directory (`pyodide/`) keeps the substrate's tests, spikes, and this README. @@ -284,35 +253,11 @@ needs. (An earlier version of `relay.ts` hardcoded one host's field names into its own Python template; making `meta` opaque is what removed the last host-specific knowledge from the relay.) -### Gotcha 1 — JsNull normalization (already handled for you) - -A JS `null` passed across the Pyodide FFI via `globals.set(name, null)` does -**not** arrive as Python's `None`. It arrives as a `JsProxy`-wrapped `JsNull` -sentinel, which passes an `is not None` check and then blows up the first time -an adapter tries to use it (e.g. `'JsNull' object is not callable`). **You do -not need to defend against this** — `relay.ts` normalizes it inside its own -embedded Python template, at both boundary crossings, before your adapter is -called: +### `sql_local` timeouts -- `bridge_call` → `_bridge_call = bridge_call if callable(bridge_call) else None` -- `duplex_bridge_call` → `_duplex_bridge_call = duplex_bridge_call if callable(duplex_bridge_call) else None` -- `contacts_db_path` → `_contacts_db_path = svc_contacts_db_path if isinstance(svc_contacts_db_path, str) else None` - -So the contract your adapter sees is a clean "real Python `None`, or a real -value." This is called out here only so nobody re-discovers and re-solves it in -their own adapter — the relay owns it. - -### Gotcha 2 — `sql_local` timeouts under Pyodide (handled, but know the limit) - -`LocalSqlRuntime.query()` enforces its per-query timeout with -`threading.Timer`, which needs real OS threads — unavailable under -Pyodide/WASM. Since the #8 fix, the source degrades gracefully there: the -timer is skipped with a one-time `RuntimeWarning`, and queries run normally -under the **default** policy. The thing to know is what that means: the -policy's `timeout_ms` is simply **not enforced** in this substrate — the -host's own wall-clock kill (Deno's process timeout) is the real enforcement, -exactly as the Pyodide environment factory requires for exec timeouts. Make -sure your host actually has one. +Pyodide has no OS threads, so `LocalSqlRuntime.query()` cannot enforce its +`threading.Timer` timeout. It emits one warning and continues. The host must +enforce the wall-clock deadline and terminate the process when needed. ## Configuration flags @@ -470,19 +415,3 @@ Roughly three tiers of coverage: silently falling back to raw loop output, but there's no data yet on how often that call actually fails or why. No retry has been added — that's a decision for once real failure data exists, not before. - -## History - -This substrate started as a feasibility spike: proving droste imports cleanly -under Pyodide, that a real corpus returns byte-identical results across sqlite -engines (native 3.53.1 vs Pyodide's bundled 3.39.0) against a large real-world -benchmark corpus, and that packaging a Deno binary + an offline -`--cached-only` `DENO_DIR` (~14MB) beats shipping a signed `Python.framework` + -wheelhouse per architecture. That work — plus the two security hardening passes -(A′-1 and A′-2, with A′-2 now the sole database-backed path) and the -adapter-agnostic split that made `relay.ts` fully adapter-agnostic and gave -droste its own example host — is in -this repo's git/PR history, not duplicated here. A few standalone investigation -scripts from that era (`spike_topology.ts`, `probe_dual_sqlite.ts`, -`verify_16_threading.ts`) still live in this directory as reference, outside -the `deno test` suite.