diff --git a/README.md b/README.md index 0edfb12..63cb8f0 100644 --- a/README.md +++ b/README.md @@ -1,446 +1,112 @@ # whittle -**Carves your agent's tool outputs down to what matters. Never cuts what doesn't come back.** +**Carve your AI agent's context down to what matters β€” and route every request to the cheapest model that can handle it. Local, fail-open, and every claim in this README is verifiable from a clone.** [![CI](https://github.com/firstops-dev/whittle/actions/workflows/ci.yml/badge.svg)](https://github.com/firstops-dev/whittle/actions/workflows/ci.yml) [![Release](https://img.shields.io/github/v/release/firstops-dev/whittle)](https://github.com/firstops-dev/whittle/releases) [![Go Reference](https://pkg.go.dev/badge/github.com/firstops-dev/whittle.svg)](https://pkg.go.dev/github.com/firstops-dev/whittle) [![License](https://img.shields.io/badge/license-Apache--2.0-blue)](LICENSE) -![whittle stats: tokens carved across real agent sessions](demo/stats.gif) - -Whittle is a content-aware compressor for the text AI agents read: tool outputs, -file reads, logs, JSON, terminal streams. Long agent sessions drown in tokens - -but most compressors buy their ratio by silently destroying things agents need: -array rows vanish, file reads get gutted, identifiers come back mangled. - -Whittle holds one hard line: **structural compression is lossless or clearly -marked, code never reaches a lossy model, and every anomaly fails open to the -original bytes.** The reduction number it reports is calibrated to real -tokenizer counts - not byte counts that overstate savings by up to 4Γ—. - -**How it plugs in:** `whittle setup` installs a Claude Code **PostToolUse hook** -backed by a small, always-on local daemon. Every tool output your agent produces -is compressed the moment it's created - on your machine, before it enters the -conversation, off the model-request path, with zero credentials leaving your -box. Claude Code is the supported agent today (Cursor, Codex, OpenCode adapters -are on the roadmap); the same daemon also exposes a Go library, an HTTP API, and -an MCP retrieval tool. See [Why whittle](#why-whittle---compress-at-write-time-not-read-time) -and [Architecture](#architecture) for the reasoning and the diagram. - -Whittle has a **second, independent surface: an opt-in model router** -(`whittle route`). It runs a local proxy on `ANTHROPIC_BASE_URL` that sends each -request to the cheapest model tier that can still handle it β€” hard reasoning -stays on your strongest model, trivial edits drop to the cheapest, the broad -middle rides a capable default. Decisions come from one auditable policy file -you own, driven by real signals: a trained subject classifier, a contrastive -difficulty score, and your own keywords. It's off by default, rewrites the -*model* and never your conversation history, forwards your credentials -untouched, and fails open on every path. See -[Model routing](#model-routing-opt-in) and the full design in -[docs/ROUTER.md](docs/ROUTER.md). +![whittle live: routing each request to the right model while carving tool outputs](demo/hero.gif) -## See it +Long agent sessions drown in tokens β€” and most compressors buy their ratio by silently destroying what agents need: array rows vanish, file reads get gutted, identifiers come back mangled. Whittle holds one hard line: **lossless or clearly marked, code never touched, every anomaly fails open to the original bytes.** -![whittle compressing a noisy build log](demo/compress.gif) +## Highlights -``` -$ tail -n 6 build.log | whittle compress -stats -ERROR migrate failed: relation "users" does not exist -... [118 lines omitted] -2026-07-04 INFO worker drained cleanly -ERROR shutdown: connection reset by peer - -whittle: action=compressed detected=log strategy=log_compressor tokens=1904->47 -``` - -Errors and the summary survive; 118 lines of INFO noise become one honest -marker. JSON reshapes losslessly, code passes through untouched, terminal -progress bars collapse to their final frame - see [Benchmarks](#benchmarks). +- πŸͺ“ **Write-time compression** β€” a Claude Code PostToolUse hook whittles each tool output *before* it enters context, so the savings repeat on every later turn. History is never mutated, so prompt caches stay intact. +- πŸ”’ **Lossless or marked** β€” JSON reshapes byte-exact (rows are never dropped), logs keep every error plus an honest `[N lines omitted]`, source code passes through untouched. +- 🎯 **Token-honest numbers** β€” savings measured against calibrated tokenizer counts, not byte counts that overstate by up to 4Γ—. +- 🧭 **Opt-in model router** β€” hard reasoning stays on your strongest model, trivia drops to the cheapest, per one auditable policy file driven by trained-classifier signals ([how it works](docs/ROUTER.md)). +- πŸ›Ÿ **Fail-open everywhere** β€” whittle down means your agent runs on originals; a rejected rewrite means your original request, retried. Never blocked, never corrupted. +- 🏠 **Runs entirely on your machine** β€” zero credentials leave your box; the Go binary has zero external dependencies. +- 🧾 **Executable claims** β€” `make test` and `go run ./bench` regenerate every number below from a clone. ## Install -``` +```sh go install github.com/firstops-dev/whittle/cmd/whittle@latest -whittle setup +whittle setup # hook + local daemon + optional ML sidecar β€” one command ``` -> `go install` places the binary in `~/go/bin`, which may not be on your `PATH`. -> If `whittle` isn't found, add it: `echo 'export PATH="$HOME/go/bin:$PATH"' >> ~/.zshrc && source ~/.zshrc`. -> (Homebrew users can skip this - `brew install firstops-dev/tap/whittle` puts it on `PATH` automatically.) +Tool outputs are whittled from now on; `whittle stats` shows what you're saving. (Homebrew: `brew install firstops-dev/tap/whittle`. If `go install`'s binary isn't found, add `~/go/bin` to your PATH. Linux runs the daemon under systemd β€” [notes](docs/compression.md).) -That's the whole thing. `setup`: +**Optional β€” turn on model routing:** -- installs the **Claude Code PostToolUse hook** - tool outputs your agent reads - are whittled from now on (Claude Code is the supported agent today; - Cursor, Codex and OpenCode adapters are on the roadmap); -- materializes the ML prose sidecar (embedded in the binary) into `~/.whittle`, - builds its venv, and uses your GPU automatically (CUDA > Apple MPS > CPU) - - if `python3` is missing, whittle simply runs deterministic-only; -- registers a **launchd agent** (macOS) so the service starts at login and is - kept alive. - -On Linux (launchd is macOS-only), run the daemon under systemd: - -``` -# ~/.config/systemd/user/whittle.service -[Unit] -Description=whittle daemon -[Service] -ExecStart=%h/go/bin/whittle daemon -Restart=always -[Install] -WantedBy=default.target +```sh +whittle policy init # calibrated policy, your model ids auto-detected +whittle route -install # background service (or `whittle route` in a terminal) +export ANTHROPIC_BASE_URL=http://127.0.0.1:45873 ``` -Manage it with `whittle status`, `whittle stop`, and `whittle cleanup` (stops -the service and removes the hook). Everything fails open: if whittle is down, -your agent sees original outputs, never an error. +## See it -## Use +![whittle compressing a noisy build log](demo/compress.gif) ``` -whittle compress output.json # compressed to stdout -cat build.log | whittle compress -stats # stats to stderr -whittle serve -addr :45871 # HTTP: POST /v1/compress -whittle route -addr 127.0.0.1:45873 # opt-in model router (ANTHROPIC_BASE_URL) -``` - -As a library: +$ tail -n 6 build.log | whittle compress -stats +ERROR migrate failed: relation "users" does not exist +... [118 lines omitted] +2026-07-04 INFO worker drained cleanly +ERROR shutdown: connection reset by peer -```go -eng := whittle.New(whittle.Options{}) -res := eng.Compress(ctx, toolOutput) -// res.Output, res.Action ("compressed"|"skipped"), res.SkipReason, -// res.Strategy, res.Detected +whittle: action=compressed detected=log strategy=log_compressor tokens=1904->47 ``` -## What it does per content type - -| detected | strategy | guarantee | -|---|---|---| -| JSON | minify + columnar reshape (union schema, typed CSV, nested flattening, constant factoring) | **lossless** - reconstructs byte-exact; rows are never dropped | -| logs / build output | keep errors, warnings, stack traces, summaries | lossy but **marked** - `... [N lines omitted]`, exact accounting | -| terminal | ANSI strip + CR-overwrite collapse (progress bars β†’ final frame) | what the terminal actually displayed; rune-safe | -| markdown file reads | structure-aware: prose compressed by the model, **code fences / tables / lists / headings passed through byte-exact** | code never reaches the model | -| source code | untouched | routed away from every lossy path | -| prose | extractive model (optional) with fidelity guards: entity protection, whole-token deletion, negation preservation | fails open on any guard trip | +Errors and the summary survive; 118 lines of INFO noise become one honest marker. Across **5,000 real agent sessions**: **22% tool-output reduction at zero measured information loss** β€” mechanically lossless on 15,846/15,846 items, and a blinded 4-judge panel found 0/120 material loss on the lossy prose path. Full receipts, including an honest side-by-side against headroom: [bench/](bench/README.md). -Every path is wrapped in fail-open guardrails: empty-output, expansion (both -byte- and token-honest), panic recovery. The worst case is always "not -compressed", never "corrupted". +## Compression: what happens to each content type -## The ML prose path (installed by `whittle setup`, optional by design) +JSON is reshaped **losslessly** (byte-exact reconstruction). Logs and terminal streams are cut lossily but **marked and exactly accounted**. Markdown file reads keep code fences, tables, and headings byte-exact while prose is compressed by an optional local model with fidelity guards. Source code is **never touched**. Every path is wrapped in fail-open guardrails β€” the worst case is always *not compressed*, never *corrupted*. -`whittle setup` installs and supervises the prose sidecar automatically: the -Python source ships inside the binary, setup builds its venv, and the daemon -keeps it running (GPU auto-selected: CUDA > Apple MPS > CPU). If `python3` is -missing, setup says so and whittle runs deterministic-only - nothing breaks. - -"Optional" means the deterministic strategies never depend on it. Only if you -use whittle as a bare library or `whittle serve` **without** running setup do -you wire it manually (LLMLingua-2 + whittle's fidelity guards - see -[model/](model/)): - -``` -cd model && python -m venv .venv && .venv/bin/pip install -r requirements.txt -.venv/bin/uvicorn app:app --port 45872 -export WHITTLE_MODEL_URL=http://127.0.0.1:45872 -``` +The full per-type contract, ML prose path, architecture, and performance tables: [docs/compression.md](docs/compression.md) Β· what each guarantee is pinned by: [GUARANTEES.md](GUARANTEES.md). ## Model routing (opt-in) -Whittle's second surface is a **local model router** - separate from the -compression hook, off by default, turned on deliberately. `whittle route` runs a -proxy on `ANTHROPIC_BASE_URL` that inspects each request and routes it to the -cheapest model tier that can still serve it, per a policy you author -(`~/.whittle/router.json`): - -``` -whittle policy init # write a starter policy to ~/.whittle/router.json -whittle route # proxy on 127.0.0.1:45873 -export ANTHROPIC_BASE_URL=http://127.0.0.1:45873 -``` - -- **Start from the built-in policy.** `whittle policy init` writes the calibrated - `default` policy with the model ids your account actually uses (auto-detected); - `whittle policy validate ` checks your edits against the real loader. - What it does and how to customize it: [router/policies/default.md](router/policies/default.md). -- **One auditable policy file.** Tiers, first-match routes over a boolean - condition tree, and a default - you can read the file and predict every - decision, and the per-request log names the rule that fired. The full grammar, - the signal math, and the design rationale live in - [docs/ROUTER.md](docs/ROUTER.md). -- **Multi-signal, not keyword-matching.** With smart mode on - (`WHITTLE_ROUTER_MODEL_URL` β†’ the sidecar), routes can test a trained - 14-subject classifier (thresholding its full probability mass, so an - *uncertain* classification falls to your default tier - never up to the - expensive one), a contrastive hard/easy difficulty score, and semantic - similarity to your own example phrases - alongside whole-word keywords and - context-size heuristics. Smart mode off β†’ the heuristics still route, - everything else rides the default. -- **It rewrites the model, not your history.** The router swaps the target model - and reconciles capabilities for it (stripping betas and features the cheaper - model rejects). It never mutates conversation history, so prompt-cache - prefixes stay intact - the failure mode that makes read-time proxies painful. -- **Your credentials pass through untouched.** The proxy forwards your own auth - to Anthropic; whittle never sees or stores a key. -- **Fail-open by construction.** A missing/invalid policy, a parse error, a dead - classifier, or a rewrite the upstream rejects all fall back to your original - request. If the router is down, unset `ANTHROPIC_BASE_URL` and you're direct - again - nothing is trapped behind it. -- **Savings you can measure.** Every request logs the requested model, the model - that actually served it, and the response's real token usage - enough to - compute exactly what routing saved you. - -The classifier and embedding models (and the scoring math behind the difficulty -signal) come from [vLLM Semantic Router](https://github.com/vllm-project/semantic-router) -([whitepaper](https://vllm-semantic-router.com/white-paper)); the architecture -around them - the single-file policy, the probability-mass thresholding, the -fail-open model - is whittle's own, and [docs/ROUTER.md](docs/ROUTER.md) -credits the line between the two precisely. Configure the upstream with -`WHITTLE_ROUTER_UPSTREAM` (defaults to `api.anthropic.com`; point it at a -corporate Anthropic gateway if you have one). - -Run it in the foreground, or install it as a background service (opt-in, separate -from the compress daemon): +Whittle's second surface: a local proxy on `ANTHROPIC_BASE_URL` that sends each request to the cheapest model tier that can still handle it β€” per a policy you can read in one screen. -``` -whittle route -install # register a launchd agent (macOS), starts at login -whittle status # shows model-router health alongside compress + sidecar -whittle route -uninstall # stop + unregister -``` - -## Configuration - -| env | default | meaning | -|---|---|---| -| `WHITTLE_MODEL_URL` | *(unset - prose off)* | model sidecar URL | -| `WHITTLE_MAX_CHARS` | 262144 | global size ceiling (skip before classify) | -| `WHITTLE_PROSE_MAX_CHARS` | 100000 | prose-path latency ceiling (lower on CPU-only machines) | -| `WHITTLE_ROUTER_UPSTREAM` | `api.anthropic.com` | model router upstream (opt-in `whittle route`) | -| `WHITTLE_ROUTER_MODEL_URL` | *(unset - smart off)* | router classifier sidecar URL (enables ML `intent`/`classify` routing) | - - - -## Benchmarks - -Three tiers, in increasing order of realism - every number regenerable from -this repo (`go run ./bench` for the deterministic rows; the prose row needs the -model sidecar), reductions on an estimated-token basis (labeled). - -### 1. Synthetic corpus (ours - headline per content class) - -Authored fixtures in [`bench/corpus/`](bench/corpus/), designed to exercise each -strategy and its guarantees. Full table: [`bench/REPORT.md`](bench/REPORT.md). - -| class | representative result | -|---|---| -| JSON (uniform/sparse/nested) | 57% - lossless, byte-exact reconstruction | -| repetitive logs | 97% - omissions marked and exactly accounted | -| terminal progress streams | 99% - final frame, rune-safe | -| code / config (py, go, yaml) | **0% by design - skipped, never touched** | -| prose | 30-40% extractive, fidelity-guarded (needs the model sidecar; not part of the deterministic `go run ./bench` output) | - -### 2. Side-by-side on headroom's data - -Inputs frozen from [headroom](https://github.com/headroomlabs-ai/headroom)'s own -benchmark generators (Apache-2.0; pinned commit, seed 42 - they check in no -corpora, so we froze what their numbers are computed on; `bench/corpus_headroom/`, -PROVENANCE.md). Both tools ran on identical bytes, defaults only, measured with -the same tokenizer. Full table + methodology: [`bench/SIDEBYSIDE.md`](bench/SIDEBYSIDE.md). - -| | headroom-ai 0.30.0 | whittle 0.2.1 | -|---|---|---| -| aggregate token reduction (10 files, 116.5k tokens) | **41.8%** | 36.5% | -| - conversation / agent-transcript JSON (3 files) | 2.1% | **5.4%** | -| - bulk data arrays (7 files) | **48.3%** | 41.6% | -| fidelity of that reduction | includes lossy row-dropping (recoverable via headroom's resident runtime) | **byte-exact lossless** on every file | -| median latency, in-process (same files) | 2.93 ms | 2.36 ms | - -Read it straight: on the aggregate, headroom-ai's defaults compress ~5 points -more - by dropping rows whittle refuses to drop. The category split shows where -each position pays: on conversation-shaped content (the shape agent tool -outputs actually take) whittle leads while staying lossless; on bulk data -arrays headroom-ai's lossy sampling buys its margin. Latency is near parity. -Which trade you want is the whole point of this project. - -### 3. Real-world: customer-service agents (two independent evaluations) - -Customer-service agents are whittle's strong-fit workload - their tool outputs -are structured JSON on essentially every call, so the compressor engages on -**100% of tool outputs**, all losslessly. Two evaluations, corroborating at -different scales: - -**Breadth** - [Salesforce APIGen-MT-5k](https://huggingface.co/datasets/Salesforce/APIGen-MT-5k), -5,000 verified multi-turn sessions: **22% tool-output reduction (o200k) at zero -measured information loss**, verified two ways - mechanically lossless on -**15,846 / 15,846** compressed items, and a blinded 4-judge panel finding -**0 / 120** material loss on the lossy prose path (honeypot-validated, Gwet AC2 -0.96-1.00). [`bench/datasets/salesforce_customer_support/`](bench/datasets/salesforce_customer_support/) - -**Depth** - Sierra's [Ο„-bench](bench/datasets/tau_bench/) (retail + airline), -counterfactual replay of reference trajectories: **22.0% / 23.3% reduction, every -record reconstructed field-for-field** (all 16 flight-search results recovered -exactly). This eval isolates whittle's real structural value-add - on -**multi-record results the columnar re-encoding goes beyond compact JSON: +24% -on small flight searches, +45% on an 80-row result**, and the advantage grows -with result size. [`bench/datasets/tau_bench/`](bench/datasets/tau_bench/) - -The product metric is **token/context reduction**: fewer tool-output tokens, -removed from every later turn's context, compounding as the session grows. -Dollar impact is a separate, caching-dependent question both reports publish in -full: under prompt caching the same cut is ~3-5% of session cost (cheap -cache-reads dominate the bill), so token savings and dollar savings are not the -same thing. - -*(Both measured on the shared compression engine `content-aware-router` v0.1.0; -whittle's `json_crusher` is lossless-only. See each report's provenance note.)* - -## Why whittle - compress at write-time, not read-time - -Context compressors typically integrate with coding agents as **request-path -proxies**: your agent's base URL is redirected through a local server that -rewrites conversation history at *read time*, on every LLM call. That position -forces hard problems - prompt-cache stabilization (mutating history invalidates -cached prefixes), per-call re-compression, terminating your API traffic (keys, -system prompts and all), and a resident runtime that must stay up or your agent -goes down with it. It also makes lossy compression the default, backed by a -retrieval loop: the runtime is guaranteed present, so dropped content can be -served back on demand. - -Whittle takes the other position: it is a **PostToolUse hook**. Each tool output -is compressed **once, at the moment it is born**, before it ever enters -conversation history. Everything else follows from that choice: - -- **Savings compound.** A tool output lives in context for every subsequent - turn. Tokens removed at write-time are removed from *every* later call - - no per-call rework, no cache surgery, because history is never mutated. -- **No trust expansion.** A hook sees one tool output at a time, locally, with - zero credentials. Nothing terminates your API traffic. -- **Failure is free.** The hook fails open; if whittle is down or declines, the - agent proceeds with the original output. A gateway outage is an agent outage. -- **The loss budget is honest.** A read-time proxy can afford recoverable lossy - compression - its resident runtime serves dropped content back when the model - asks. Whittle keeps no runtime in your request path; reduced outputs carry a tiny - retrieval pointer served by the local daemon (`whittle_get`), and lossless - transforms carry nothing at all - lossless-or-marked stays the construction, - recovery is the safety net, never the license. - -The hook is whittle's default surface, not its only one: **library** -(`whittle.New`) β†’ **HTTP service** (`whittle serve`) β†’ **hook adapters** -(Claude Code PostToolUse today; Cursor, Codex, OpenCode adapters on the -roadmap) - and the same library embeds in gateways or pipelines if that is -where you need it. The position is the point: compression happens where output -is born, whatever surface delivers it there. - -**This argument is about compression, not about routing.** Whittle's opt-in -[model router](#model-routing-opt-in) *is* a request-path proxy - but it exists -for a different job (send each request to the cheapest capable tier), and it is -deliberately the minimal kind. It rewrites only the model field and reconciles -capabilities; it never rewrites conversation history, so it inherits none of the -prompt-cache surgery above. It forwards your credentials untouched rather than -terminating them, and it fails open to your original request, so an outage is a -passthrough, not an agent outage. The read-time-compression proxy is the pattern -whittle rejects; a minimal, history-preserving, fail-open router is a different -tool that happens to share the address bar. - -## Architecture - -``` -Claude Code ──PostToolUse (HTTP)──▢ whittle daemon (:45871, launchd-kept) - β”‚ - β”œβ”€ dispatch β–Ά json Β· log Β· terminal Β· markdown - β”‚ (deterministic, in-process) - └─ prose ─▢ model sidecar (:45872, optional GPU) - β”‚ - whittle_get(id) ◀──MCP toolβ”€β”€β”€β”€β”€β”€β”€β”€β”˜ reduced originals, retrievable on demand - - ── opt-in, separate process, off by default ── -Claude Code ──ANTHROPIC_BASE_URL──▢ whittle route (:45873) ──▢ Anthropic API - policy-based model-tier routing; - rewrites the model, not history; fail-open -``` +- **Calibrated out of the box** β€” `whittle policy init` writes a conservative default (hard reasoning β†’ strongest tier, confident chit-chat β†’ cheapest, *everything else untouched*) with your account's real model ids auto-detected. [What it does & how to customize](router/policies/default.md). +- **Multi-signal, not keyword-matching** β€” a trained 14-subject classifier (probability-mass thresholded, so an *uncertain* classification never escalates), a contrastive difficulty score, and your own keywords. Every log line shows each signal's value against its gate. +- **Rewrites the model, never your history** β€” prompt-cache prefixes survive; capabilities the cheaper model rejects are stripped automatically; credentials pass through untouched. +- **Fail-open by construction** β€” bad policy, dead classifier, or a rejected rewrite all fall back to your original request. Unset the env var and you're direct again. +- **Savings you can measure** β€” every request logs requested model, served model, and real token usage. -The compression surfaces are one resident daemon, three ways in (hook Β· HTTP Β· -MCP) - compression happens where output is born, off the model-request path -entirely. The model router is a **separate, opt-in** process on the request path; -you run it only when you want tier routing. +The router β€” engine, policy design, signal composition β€” is whittle's own; two pretrained models power its ML signals (the 14-subject `domain` classifier and the text embedder, both from [vLLM Semantic Router](https://github.com/vllm-project/semantic-router)). Architecture, signal math, and precise credits: [docs/ROUTER.md](docs/ROUTER.md). -## Performance +## Why write-time? -Deterministic strategies are pure CPU, single static binary, zero allocatable -model state (Apple M-series, `go test -bench`): +Most context compressors are read-time proxies: they rewrite your conversation history on every LLM call β€” which breaks prompt caches, terminates your API traffic, and makes lossy compression the default. Whittle compresses each output **once, at the moment it's born**, before it enters history: savings compound across every later turn, nothing sits in your request path, and failure costs nothing. The full argument: [docs/why-write-time.md](docs/why-write-time.md). -| input | size | latency | -|---|---|---| -| JSON array, 200 rows, pretty-printed | ~21 KB | ~1.0 ms | -| terminal progress stream | ~12 KB | ~3.9 ms | -| build log, 800 lines | ~56 KB | ~21 ms | +## FAQ -The hook runs after the tool call completes, so this cost is **off the LLM -request path entirely** - model-call latency is unchanged. (These are absolute -in-path budgets on whittle's own corpus; for tool-vs-tool latency on identical -inputs see the Benchmarks side-by-side above and `bench/SIDEBYSIDE.md`.) The optional ML -prose path is capped by a fail-open budget (default 1.5 s) and never blocks -beyond it. +**Will it break my agent?** No β€” that's the core design constraint. Every path fails open: if whittle is down, declines, or errors, your agent sees original bytes. The router likewise: worst case is your request untouched. -## Design principles +**Does it need Python?** No. The deterministic compressors (JSON, logs, terminal, markdown structure) are pure Go. Python powers the *optional* prose model and router classifiers; `whittle setup` installs it if `python3` exists, and everything else works without it. -1. **Fail open.** A compressor that breaks your agent is worse than no compressor. -2. **Never silent loss.** Lossy paths mark what they removed and account for it exactly. -3. **Code is sacred.** File reads, fences, identifiers: byte-exact or untouched. -4. **Token-honest.** Accept gates and reported savings use calibrated token - estimates (MAE ~8% vs `o200k_base`; regenerate with `bench/calibrate_tokens.py`), - not bytes. -5. **Adversarially tested.** The invariants above are pinned by reconstruction - fuzzing, per-language routing suites, and fail-open contract tests. +**Are token savings dollar savings?** Not 1:1 β€” under prompt caching, cheap cache-reads dominate the bill, so a 22% token cut is roughly 3–5% of session cost. We publish both numbers rather than pretending otherwise: [bench/](bench/README.md). +**How does it compare to headroom?** On identical bytes, headroom's defaults compress ~5 points more β€” by dropping rows whittle refuses to drop. On conversation-shaped content whittle leads *while staying lossless*. Which trade you want is the whole point: [bench/SIDEBYSIDE.md](bench/SIDEBYSIDE.md). -## Acknowledgments +**Where does my data go?** Nowhere. Hook, daemon, models, and router all run on your machine. The router forwards your own credentials to Anthropic and logs token *counts*, never prompt text. -Whittle's log-selection strategy, several content-detection heuristics, and the -tabular parser were adapted from [Headroom](https://github.com/headroomlabs-ai/headroom) -(Apache-2.0) - adapted portions are marked in source comments, and we think -their compaction work is excellent. Whittle exists because we wanted the other -position: a write-time PostToolUse hook instead of a read-time request-path -proxy, with the stricter fidelity contract that position requires. See NOTICE. +**Which agents?** Claude Code today (hook + router). Cursor, Codex, and OpenCode adapters are on the roadmap β€” the compression engine is also a plain Go library and HTTP service. ## Verify it yourself -Every claim here is checkable from a clone - that is the point. +Every claim here is checkable from a clone β€” that is the point. +```sh +make test # guarantees as executable tests (GUARANTEES.md) +go run ./bench # corpus reductions + fidelity, SHA-pinned, CI-gated +python bench/calibrate_tokens.py # reproduces the token-estimator MAE ``` -make test # guarantees as executable tests (see GUARANTEES.md) -go run ./bench # corpus reductions + fidelity, SHA-pinned, CI-gated -python bench/calibrate_tokens.py # reproduces the token-estimator MAE (needs tiktoken) -``` - -[GUARANTEES.md](GUARANTEES.md) maps each fidelity promise to the test that pins it. -## Known limitations +## Contributing -- **Replacements must match the tool's output shape.** Claude Code schema-validates - `updatedToolOutput` and silently keeps the original on mismatch; whittle rebuilds - the tool's own response shape around the compressed text (verified live on Claude - Code 2.1.203 β€” see [docs/hook-output-cap.md](docs/hook-output-cap.md), which also - documents why the once-assumed 10k output cap does NOT apply to replacements). -- **Prose needs the sidecar.** Without it, prose and markdown docs pass through - unchanged; deterministic strategies are unaffected. -- **launchd is macOS-only.** Linux runs the daemon under systemd (unit above). -- **Prose latency ceiling** (default 100000 chars) trades a hard cap for predictable - in-path latency, sized for GPU/MPS inference (measured ~0.07s + 0.04s/KB on Apple - silicon: 100KB β‰ˆ 4s, within the 8s prose timeout). CPU-only machines run ~8x - slower (~0.3s/KB) and should lower `WHITTLE_PROSE_MAX_CHARS` to ~12000, or large - prose burns the timeout and passes through unchanged. +The bar: guarantees are executable β€” see [CONTRIBUTING.md](CONTRIBUTING.md). Fidelity bugs (whittle changing an output's meaning) are treated as urgent. Good first issues: agent adapters, Linux packaging, detection corpus cases. -## Contributing +## Acknowledgments -Whittle's bar is that guarantees are executable - see [CONTRIBUTING.md](CONTRIBUTING.md). -The highest-severity issue class is **fidelity**: if whittle ever changed the -meaning of an output, that is a bug we treat as urgent (use the fidelity issue -template). Good first issues: agent adapters (Cursor/Codex/OpenCode), Linux -packaging, detection corpus cases. +Whittle's log-selection strategy, several detection heuristics, and the tabular parser were adapted from [Headroom](https://github.com/headroomlabs-ai/headroom) (Apache-2.0) β€” their compaction work is excellent; we wanted the write-time position and the stricter fidelity contract it demands. The router's two pretrained models β€” the `domain` classifier and the text embedder behind the similarity signals β€” come from [vLLM Semantic Router](https://github.com/vllm-project/semantic-router) ([whitepaper](https://vllm-semantic-router.com/white-paper)); the routing engine and policy design are whittle's own. See [NOTICE](NOTICE). ## License diff --git a/bench/README.md b/bench/README.md new file mode 100644 index 0000000..0b5e6b2 --- /dev/null +++ b/bench/README.md @@ -0,0 +1,76 @@ +# whittle benchmarks + +The receipts behind the README's numbers. Every figure is regenerable from this repo. + +Three tiers, in increasing order of realism - every number regenerable from +this repo (`go run ./bench` for the deterministic rows; the prose row needs the +model sidecar), reductions on an estimated-token basis (labeled). + +### 1. Synthetic corpus (ours - headline per content class) + +Authored fixtures in [`corpus/`](corpus/), designed to exercise each +strategy and its guarantees. Full table: [`REPORT.md`](REPORT.md). + +| class | representative result | +|---|---| +| JSON (uniform/sparse/nested) | 57% - lossless, byte-exact reconstruction | +| repetitive logs | 97% - omissions marked and exactly accounted | +| terminal progress streams | 99% - final frame, rune-safe | +| code / config (py, go, yaml) | **0% by design - skipped, never touched** | +| prose | 30-40% extractive, fidelity-guarded (needs the model sidecar; not part of the deterministic `go run ./bench` output) | + +### 2. Side-by-side on headroom's data + +Inputs frozen from [headroom](https://github.com/headroomlabs-ai/headroom)'s own +benchmark generators (Apache-2.0; pinned commit, seed 42 - they check in no +corpora, so we froze what their numbers are computed on; `corpus_headroom/`, +PROVENANCE.md). Both tools ran on identical bytes, defaults only, measured with +the same tokenizer. Full table + methodology: [`SIDEBYSIDE.md`](SIDEBYSIDE.md). + +| | headroom-ai 0.30.0 | whittle 0.2.1 | +|---|---|---| +| aggregate token reduction (10 files, 116.5k tokens) | **41.8%** | 36.5% | +| - conversation / agent-transcript JSON (3 files) | 2.1% | **5.4%** | +| - bulk data arrays (7 files) | **48.3%** | 41.6% | +| fidelity of that reduction | includes lossy row-dropping (recoverable via headroom's resident runtime) | **byte-exact lossless** on every file | +| median latency, in-process (same files) | 2.93 ms | 2.36 ms | + +Read it straight: on the aggregate, headroom-ai's defaults compress ~5 points +more - by dropping rows whittle refuses to drop. The category split shows where +each position pays: on conversation-shaped content (the shape agent tool +outputs actually take) whittle leads while staying lossless; on bulk data +arrays headroom-ai's lossy sampling buys its margin. Latency is near parity. +Which trade you want is the whole point of this project. + +### 3. Real-world: customer-service agents (two independent evaluations) + +Customer-service agents are whittle's strong-fit workload - their tool outputs +are structured JSON on essentially every call, so the compressor engages on +**100% of tool outputs**, all losslessly. Two evaluations, corroborating at +different scales: + +**Breadth** - [Salesforce APIGen-MT-5k](https://huggingface.co/datasets/Salesforce/APIGen-MT-5k), +5,000 verified multi-turn sessions: **22% tool-output reduction (o200k) at zero +measured information loss**, verified two ways - mechanically lossless on +**15,846 / 15,846** compressed items, and a blinded 4-judge panel finding +**0 / 120** material loss on the lossy prose path (honeypot-validated, Gwet AC2 +0.96-1.00). [`datasets/salesforce_customer_support/`](datasets/salesforce_customer_support/) + +**Depth** - Sierra's [Ο„-bench](datasets/tau_bench/) (retail + airline), +counterfactual replay of reference trajectories: **22.0% / 23.3% reduction, every +record reconstructed field-for-field** (all 16 flight-search results recovered +exactly). This eval isolates whittle's real structural value-add - on +**multi-record results the columnar re-encoding goes beyond compact JSON: +24% +on small flight searches, +45% on an 80-row result**, and the advantage grows +with result size. [`datasets/tau_bench/`](datasets/tau_bench/) + +The product metric is **token/context reduction**: fewer tool-output tokens, +removed from every later turn's context, compounding as the session grows. +Dollar impact is a separate, caching-dependent question both reports publish in +full: under prompt caching the same cut is ~3-5% of session cost (cheap +cache-reads dominate the bill), so token savings and dollar savings are not the +same thing. + +*(Both measured on the shared compression engine `content-aware-router` v0.1.0; +whittle's `json_crusher` is lossless-only. See each report's provenance note.)* + diff --git a/cmd/whittle/main.go b/cmd/whittle/main.go index 8d0904c..2e8caea 100644 --- a/cmd/whittle/main.go +++ b/cmd/whittle/main.go @@ -95,7 +95,8 @@ route flags: -install register the router as a background launchd agent (opt-in), then exit -uninstall stop + unregister the router launchd agent, then exit env WHITTLE_ROUTER_UPSTREAM upstream API (default api.anthropic.com) - env WHITTLE_ROUTER_MODEL_URL classifier sidecar URL (unset β†’ smart mode off) + env WHITTLE_ROUTER_MODEL_URL classifier sidecar (default http://127.0.0.1:45872, + auto-detected; set to "off" to disable ML signals) env: WHITTLE_MODEL_URL enable the ML prose path (model sidecar URL) diff --git a/cmd/whittle/setup.go b/cmd/whittle/setup.go index 86c4517..7b82bad 100644 --- a/cmd/whittle/setup.go +++ b/cmd/whittle/setup.go @@ -255,22 +255,19 @@ func routerInstall() { fmt.Println("whittle:", err) return } - // Point smart mode at the compress sidecar (it also serves /v1/route/*). If the - // sidecar is down, the router fails open to heuristics-only β€” never blocks. + // Smart mode is automatic (the router probes the standard sidecar address and + // degrades to heuristics if absent) β€” no env plumbing needed here. plist := fmt.Sprintf(` Label%s ProgramArguments%sroute - EnvironmentVariables - WHITTLE_ROUTER_MODEL_URLhttp://%s - RunAtLoad KeepAlive StandardOutPath%s/logs/router.log StandardErrorPath%s/logs/router.log -`, routerAgentLabel, self, modelAddr, dir, dir) +`, routerAgentLabel, self, dir, dir) if err := os.WriteFile(routerPlistPath(), []byte(plist), 0o644); err != nil { fmt.Println("whittle: router agent registration failed:", err) return diff --git a/demo/hero-feed.sh b/demo/hero-feed.sh new file mode 100755 index 0000000..96f09e9 --- /dev/null +++ b/demo/hero-feed.sh @@ -0,0 +1,90 @@ +#!/bin/sh +# hero-feed.sh β€” whittle live-watch choreography for the README hero GIF. +# +# Emits a pre-sized, append-only ANSI stream (no scrolling, one clear) that +# replays a real captured session: four agent turns routed to the right model, +# tool outputs carved losslessly, then a `whittle stats`-style session summary. +# +# All color comes from this script's stdout (VHS `Type` can't emit ANSI). +# Reproducible from a clone: `sh demo/hero-feed.sh`. +# +# Palette (flat 16-color): reset \033[0m Β· bold \033[1m Β· dim \033[2m Β· +# dim-italic \033[2;3m Β· green \033[32m Β· cyan \033[36m Β· blue \033[34m Β· +# red \033[31m. Percent signs are doubled (%%) for printf. + +# ── stream ────────────────────────────────────────────────────────────────── +sleep 0.6 +printf '\033[2mπŸͺ“ whittle Β· live β€” carving outputs, routing requests Β· local Β· fail-open\033[0m\n' + +sleep 0.8 +printf '\n' + +# turn 1 β€” trivia drops to the cheapest model +sleep 0.3 +printf '\033[2mΒ»\033[0m "bigest country in europe?"\n' +sleep 0.5 +printf ' \033[32mβ–Έ\033[0m casual-easy \033[2mopus\033[0m\033[1;32mβ†’haiku\033[0m\033[2m cplx βˆ’0.27 Β· 52,848-tok ctx\033[0m\n' + +sleep 1.7 +printf '\n' + +# turn 2 β€” hard reasoning stays on the strongest model; its outputs get carved +sleep 0.3 +printf '\033[2mΒ»\033[0m "What is in embed.go file?"\n' +sleep 0.6 +printf ' \033[34mβ–Έ\033[0m requested \033[1;34mopusβ†’opus\033[0m \033[1;34mkept\033[0m\033[2m dom computer-science 0.98\033[0m\n' + +sleep 1.8 +printf '\033[2m πŸͺ“ \033[0m\033[1membed.go\033[0m\033[2m source Β· \033[0m\033[32muntouched\033[0m\033[2m β€” code is never cut\033[0m\n' +sleep 0.5 +printf '\033[2m πŸͺ“ \033[0m\033[1mbuild.log\033[0m\033[2m 144 lines, 2 errors buried\033[0m\n' +sleep 0.4 +printf '\033[31m ERROR migrate failed: relation "users" does not exist\033[0m\n' +sleep 0.4 +printf '\033[2;3m … [118 lines omitted]\033[0m\n' +sleep 0.4 +printf '\033[31m ERROR shutdown: connection reset by peer\033[0m\n' +sleep 0.4 +printf ' \033[2m1,904\033[0m \033[1;32mβ†’ 47\033[0m\033[2m tok Β· \033[0m\033[1;32mβˆ’97%%\033[0m\n' +sleep 0.8 +printf '\033[2m πŸͺ“ \033[0m\033[1mtest-run.log\033[0m\033[2m 670 lines Β· 20,595\033[0m \033[1;32mβ†’ 472\033[0m\033[2m tok Β· \033[0m\033[1;32mβˆ’98%%\033[0m\n' + +sleep 1.8 +printf '\n' + +# turn 3 β€” another trivial ask drops to haiku +sleep 0.2 +printf '\033[2mΒ»\033[0m "share command to rename a file"\n' +sleep 0.5 +printf ' \033[32mβ–Έ\033[0m casual-easy \033[2mopus\033[0m\033[1;32mβ†’haiku\033[0m\033[2m cplx βˆ’0.45\033[0m\n' + +sleep 1.2 +printf '\n' + +# turn 4 β€” mid-complexity drafting lands on sonnet +sleep 0.2 +printf '\033[2mΒ»\033[0m "draft a slack message for this content"\n' +sleep 0.5 +printf ' \033[36mβ–Έ\033[0m casual-medium \033[2mopus\033[0m\033[1;36mβ†’sonnet\033[0m\033[2m dom other 1.00\033[0m\n' + +# hold the full stream, then one clear into the session summary +sleep 1.6 +sleep 1.2 +clear + +# ── closer (session summary; `whittle stats` house style) ──────────────────── +sleep 0.3 +printf '\033[1mπŸͺ“ whittle\033[0m \033[2mΒ· session summary\033[0m\n' +sleep 0.6 +printf '\n' +printf ' \033[2mcontext\033[0m \033[1;32m21,980\033[0m tokens carved \033[2mlossless or marked Β· code untouched\033[0m\n' +sleep 0.6 +printf ' \033[2mrouting\033[0m 4 requests \033[32m●\033[0m haiku 2 \033[36m●\033[0m sonnet 1 \033[34m●\033[0m opus 1 held\n' +sleep 0.6 +printf ' \033[2msavings\033[0m ~$0.05 measured \033[2mon billed tokens Β· this session\033[0m\n' +sleep 0.6 +printf '\n' +printf ' \033[2mnever cuts what doesn'\''t come back Β· github.com/firstops-dev/whittle\033[0m\n' + +# loop-pause hold, then the GIF loops +sleep 4.0 diff --git a/demo/hero.gif b/demo/hero.gif new file mode 100644 index 0000000..241a95b Binary files /dev/null and b/demo/hero.gif differ diff --git a/demo/hero.tape b/demo/hero.tape new file mode 100644 index 0000000..cdb4321 --- /dev/null +++ b/demo/hero.tape @@ -0,0 +1,20 @@ +Output demo/hero.gif +Set Theme "Catppuccin Mocha" +Set FontSize 18 +Set Width 1000 +Set Height 600 +Set Padding 24 +Set TypingSpeed 45ms +Set CursorBlink false + +# Define whittle as this session's replay, hidden, so the visible command reads +# as the real subcommand (`whittle watch`) driving whittle's own captured output. +Hide +Type "whittle() { sh demo/hero-feed.sh; }; clear" +Enter +Show + +Sleep 500ms +Type "whittle watch" +Enter +Sleep 24s diff --git a/docs/ROUTER.md b/docs/ROUTER.md index df39bd8..572acd8 100644 --- a/docs/ROUTER.md +++ b/docs/ROUTER.md @@ -166,9 +166,14 @@ prints errors and cost-lint warnings. ## 5. Signals Two small models power three signals. Both run in the sidecar; the Go engine -holds only thresholds. If the sidecar is down or smart mode is off, ML leaves -evaluate false and the keyword/context heuristics still route (a route whose -match *depended* on an errored signal never fires β€” so a `not` over an +holds only thresholds. Smart mode is automatic: the router probes the standard +sidecar address (`127.0.0.1:45872`, installed by `whittle setup`) β€” no +configuration. An absent sidecar reads as smart-off (quiet; heuristics still +route, and signals activate the moment it appears); a present-but-broken one is +surfaced loudly (`err` traces + `ml-degraded`). Set +`WHITTLE_ROUTER_MODEL_URL=off` to disable ML entirely, or point it elsewhere to +override. When ML is unavailable, leaves evaluate false (a route whose match +*depended* on an errored signal never fires β€” so a `not` over an unavailable signal can't invert fail-open; the log gains an `ml-degraded` tag). ### 5.1 `domain` β€” subject classification with mass thresholding diff --git a/docs/compression.md b/docs/compression.md new file mode 100644 index 0000000..b1bbb6f --- /dev/null +++ b/docs/compression.md @@ -0,0 +1,97 @@ +# Compression: reference + +The full per-content-type contract, the ML prose path, architecture, performance, and limitations. + +## What it does per content type + +| detected | strategy | guarantee | +|---|---|---| +| JSON | minify + columnar reshape (union schema, typed CSV, nested flattening, constant factoring) | **lossless** - reconstructs byte-exact; rows are never dropped | +| logs / build output | keep errors, warnings, stack traces, summaries | lossy but **marked** - `... [N lines omitted]`, exact accounting | +| terminal | ANSI strip + CR-overwrite collapse (progress bars β†’ final frame) | what the terminal actually displayed; rune-safe | +| markdown file reads | structure-aware: prose compressed by the model, **code fences / tables / lists / headings passed through byte-exact** | code never reaches the model | +| source code | untouched | routed away from every lossy path | +| prose | extractive model (optional) with fidelity guards: entity protection, whole-token deletion, negation preservation | fails open on any guard trip | + +Every path is wrapped in fail-open guardrails: empty-output, expansion (both +byte- and token-honest), panic recovery. The worst case is always "not +compressed", never "corrupted". + + +## The ML prose path (installed by `whittle setup`, optional by design) + +`whittle setup` installs and supervises the prose sidecar automatically: the +Python source ships inside the binary, setup builds its venv, and the daemon +keeps it running (GPU auto-selected: CUDA > Apple MPS > CPU). If `python3` is +missing, setup says so and whittle runs deterministic-only - nothing breaks. + +"Optional" means the deterministic strategies never depend on it. Only if you +use whittle as a bare library or `whittle serve` **without** running setup do +you wire it manually (LLMLingua-2 + whittle's fidelity guards - see +[model/](../model/)): + +``` +cd model && python -m venv .venv && .venv/bin/pip install -r requirements.txt +.venv/bin/uvicorn app:app --port 45872 +export WHITTLE_MODEL_URL=http://127.0.0.1:45872 +``` + + +## Architecture + +``` +Claude Code ──PostToolUse (HTTP)──▢ whittle daemon (:45871, launchd-kept) + β”‚ + β”œβ”€ dispatch β–Ά json Β· log Β· terminal Β· markdown + β”‚ (deterministic, in-process) + └─ prose ─▢ model sidecar (:45872, optional GPU) + β”‚ + whittle_get(id) ◀──MCP toolβ”€β”€β”€β”€β”€β”€β”€β”€β”˜ reduced originals, retrievable on demand + + ── opt-in, separate process, off by default ── +Claude Code ──ANTHROPIC_BASE_URL──▢ whittle route (:45873) ──▢ Anthropic API + policy-based model-tier routing; + rewrites the model, not history; fail-open +``` + +The compression surfaces are one resident daemon, three ways in (hook Β· HTTP Β· +MCP) - compression happens where output is born, off the model-request path +entirely. The model router is a **separate, opt-in** process on the request path; +you run it only when you want tier routing. + + +## Performance + +Deterministic strategies are pure CPU, single static binary, zero allocatable +model state (Apple M-series, `go test -bench`): + +| input | size | latency | +|---|---|---| +| JSON array, 200 rows, pretty-printed | ~21 KB | ~1.0 ms | +| terminal progress stream | ~12 KB | ~3.9 ms | +| build log, 800 lines | ~56 KB | ~21 ms | + +The hook runs after the tool call completes, so this cost is **off the LLM +request path entirely** - model-call latency is unchanged. (These are absolute +in-path budgets on whittle's own corpus; for tool-vs-tool latency on identical +inputs see the Benchmarks side-by-side above and [`bench/SIDEBYSIDE.md`](../bench/SIDEBYSIDE.md).) The optional ML +prose path is capped by a fail-open budget (default 1.5 s) and never blocks +beyond it. + + +## Known limitations + +- **Replacements must match the tool's output shape.** Claude Code schema-validates + `updatedToolOutput` and silently keeps the original on mismatch; whittle rebuilds + the tool's own response shape around the compressed text (verified live on Claude + Code 2.1.203 β€” see [docs/hook-output-cap.md](hook-output-cap.md), which also + documents why the once-assumed 10k output cap does NOT apply to replacements). +- **Prose needs the sidecar.** Without it, prose and markdown docs pass through + unchanged; deterministic strategies are unaffected. +- **launchd is macOS-only.** Linux runs the daemon under systemd (unit in the README install notes). +- **Prose latency ceiling** (default 100000 chars) trades a hard cap for predictable + in-path latency, sized for GPU/MPS inference (measured ~0.07s + 0.04s/KB on Apple + silicon: 100KB β‰ˆ 4s, within the 8s prose timeout). CPU-only machines run ~8x + slower (~0.3s/KB) and should lower `WHITTLE_PROSE_MAX_CHARS` to ~12000, or large + prose burns the timeout and passes through unchanged. + diff --git a/docs/why-write-time.md b/docs/why-write-time.md new file mode 100644 index 0000000..d4860a4 --- /dev/null +++ b/docs/why-write-time.md @@ -0,0 +1,50 @@ +# Why write-time compression, not read-time + +Whittle's core position, in full. (The [README](../README.md) carries the short version.) + +Context compressors typically integrate with coding agents as **request-path +proxies**: your agent's base URL is redirected through a local server that +rewrites conversation history at *read time*, on every LLM call. That position +forces hard problems - prompt-cache stabilization (mutating history invalidates +cached prefixes), per-call re-compression, terminating your API traffic (keys, +system prompts and all), and a resident runtime that must stay up or your agent +goes down with it. It also makes lossy compression the default, backed by a +retrieval loop: the runtime is guaranteed present, so dropped content can be +served back on demand. + +Whittle takes the other position: it is a **PostToolUse hook**. Each tool output +is compressed **once, at the moment it is born**, before it ever enters +conversation history. Everything else follows from that choice: + +- **Savings compound.** A tool output lives in context for every subsequent + turn. Tokens removed at write-time are removed from *every* later call - + no per-call rework, no cache surgery, because history is never mutated. +- **No trust expansion.** A hook sees one tool output at a time, locally, with + zero credentials. Nothing terminates your API traffic. +- **Failure is free.** The hook fails open; if whittle is down or declines, the + agent proceeds with the original output. A gateway outage is an agent outage. +- **The loss budget is honest.** A read-time proxy can afford recoverable lossy + compression - its resident runtime serves dropped content back when the model + asks. Whittle keeps no runtime in your request path; reduced outputs carry a tiny + retrieval pointer served by the local daemon (`whittle_get`), and lossless + transforms carry nothing at all - lossless-or-marked stays the construction, + recovery is the safety net, never the license. + +The hook is whittle's default surface, not its only one: **library** +(`whittle.New`) β†’ **HTTP service** (`whittle serve`) β†’ **hook adapters** +(Claude Code PostToolUse today; Cursor, Codex, OpenCode adapters on the +roadmap) - and the same library embeds in gateways or pipelines if that is +where you need it. The position is the point: compression happens where output +is born, whatever surface delivers it there. + +**This argument is about compression, not about routing.** Whittle's opt-in +[model router](../README.md#model-routing-opt-in) *is* a request-path proxy - but it exists +for a different job (send each request to the cheapest capable tier), and it is +deliberately the minimal kind. It rewrites only the model field and reconciles +capabilities; it never rewrites conversation history, so it inherits none of the +prompt-cache surgery above. It forwards your credentials untouched rather than +terminating them, and it fails open to your original request, so an outage is a +passthrough, not an agent outage. The read-time-compression proxy is the pattern +whittle rejects; a minimal, history-preserving, fail-open router is a different +tool that happens to share the address bar. + diff --git a/router/quiet_test.go b/router/quiet_test.go new file mode 100644 index 0000000..a1a945c --- /dev/null +++ b/router/quiet_test.go @@ -0,0 +1,46 @@ +package router + +import ( + "errors" + "net/http" + "net/http/httptest" + "strings" + "testing" + + "github.com/firstops-dev/whittle/router/ml" +) + +// The zero-config contract: an ABSENT sidecar (connection refused) reads as +// smart-mode-off (ErrMLDisabled β€” quiet, no ml-degraded), while a PRESENT-but- +// broken sidecar keeps surfacing as a real error (loud). Absence is expected; +// breakage is not. +func TestQuietClassifier_AbsentVsBroken(t *testing.T) { + // Absent: nothing listening β†’ refused β†’ ErrMLDisabled. + q := quietClassifier{ml.New("http://127.0.0.1:1")} + if _, err := q.EmbeddingScore("x", nil); !errors.Is(err, ErrMLDisabled) { + t.Fatalf("refused connection must read as ErrMLDisabled, got %v", err) + } + // Broken: listening but 500 β†’ a real error, NOT ErrMLDisabled. + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.WriteHeader(http.StatusInternalServerError) + })) + defer srv.Close() + qb := quietClassifier{ml.New(srv.URL)} + if _, err := qb.EmbeddingScore("x", nil); err == nil || errors.Is(err, ErrMLDisabled) { + t.Fatalf("a broken sidecar must stay a real error, got %v", err) + } +} + +// End-to-end consequence: with the quiet wrapper, a down sidecar routes to the +// default WITHOUT the ml-degraded tag (it is not a degradation, it's absence). +func TestQuietClassifier_AbsenceIsNotDegraded(t *testing.T) { + p, _ := mustLoad(t, fullPolicy) + d := Decide(Signals{RecentText: "just a normal ask", ContextTokens: 100, SessionID: "s"}, + p, quietClassifier{ml.New("http://127.0.0.1:1")}, NewMemSessionStore(), "") + if d.Tier != "main" { + t.Fatalf("must fall to default, got %q (%s)", d.Tier, d.Reason) + } + if strings.Contains(d.Reason, "ml-degraded") { + t.Fatalf("absence must not tag ml-degraded: %q", d.Reason) + } +} diff --git a/router/server.go b/router/server.go index f606aea..7d05cee 100644 --- a/router/server.go +++ b/router/server.go @@ -1,6 +1,7 @@ package router import ( + "errors" "net/http" "os" "os/signal" @@ -10,10 +11,55 @@ import ( "github.com/firstops-dev/whittle/router/ml" ) +// quietClassifier wraps the sidecar client so an ABSENT sidecar (connection +// refused β€” never installed, or not up yet) reads as smart-mode-off +// (ErrMLDisabled: signals trace "off", no ml-degraded noise, routing falls to +// heuristics/default), while a PRESENT-but-broken sidecar (timeout, 5xx, +// malformed) keeps surfacing as a real error. The distinction keeps the +// zero-config default honest: absence is expected, breakage is loud. The check +// is per-request and costs ~nothing (localhost refusal is immediate), so the +// moment `whittle setup` brings the sidecar up, signals activate with no restart. +type quietClassifier struct{ c *ml.Client } + +func quiet(err error) error { + if err != nil && errors.Is(err, syscall.ECONNREFUSED) { + return ErrMLDisabled + } + return err +} + +func (q quietClassifier) Domain(text string) (string, float64, map[string]float64, error) { + l, c, p, err := q.c.Domain(text) + return l, c, p, quiet(err) +} +func (q quietClassifier) EmbeddingScore(text string, cands []string) (float64, error) { + s, err := q.c.EmbeddingScore(text, cands) + return s, quiet(err) +} +func (q quietClassifier) ComplexityMargin(text string, hard, easy []string) (float64, error) { + m, err := q.c.ComplexityMargin(text, hard, easy) + return m, quiet(err) +} + +// sidecarUp is a one-shot startup probe purely for an honest log line. +func sidecarUp(base string) bool { + c := http.Client{Timeout: 800 * time.Millisecond} + r, err := c.Get(base + "/health") + if err != nil { + return false + } + r.Body.Close() + return r.StatusCode == 200 +} + // ml.Client must satisfy Classifier β€” asserted here (at the import site) rather // than in package ml, which would import router back and form a cycle. var _ Classifier = (*ml.Client)(nil) +// DefaultModelURL is the classifier sidecar `whittle setup` installs β€” smart +// mode probes it automatically; no configuration required. +const DefaultModelURL = "http://127.0.0.1:45872" + // DefaultAddr is the router daemon's listen address. It is distinct from the // compress service (:45871) and the prose model sidecar (:45872): the router is a // separate front door that Claude Code points ANTHROPIC_BASE_URL at, while the @@ -41,18 +87,29 @@ func ListenAndServe(addr, policyPath string, lg Logger) error { lg.Printf("router: policy warning: %s", w) } - // Smart mode is opt-in: with a sidecar URL, ML intent/classify leaves resolve - // against the model; without it, the noop classifier makes them evaluate to - // heuristics-only / static default (Decide handles nil as noop). + // Smart mode is AUTOMATIC: the router defaults to the standard sidecar + // address (the one `whittle setup` installs and supervises) and degrades + // gracefully when it isn't there β€” a refused connection reads as "smart mode + // off" (quiet), not as a per-request error. WHITTLE_ROUTER_MODEL_URL overrides + // the address; "off" disables ML entirely. var cl Classifier - if u := os.Getenv("WHITTLE_ROUTER_MODEL_URL"); u != "" { - cl = ml.New(u) - lg.Printf("router: smart mode ON (classifier sidecar = %s)", u) - } else { - lg.Printf("router: smart mode OFF β€” ML signals (domain/embedding/complexity) are inert; set WHITTLE_ROUTER_MODEL_URL=http://127.0.0.1:45872 to enable") + modelURL := os.Getenv("WHITTLE_ROUTER_MODEL_URL") + if modelURL == "" { + modelURL = DefaultModelURL + } + switch modelURL { + case "off", "0", "none": + lg.Printf("router: smart mode disabled by WHITTLE_ROUTER_MODEL_URL=%s β€” heuristic signals only", modelURL) if pol != nil && policyUsesML(pol) { lg.Printf("router: WARNING: the policy references ML signals, which will never fire in this mode β€” requests they guard fall to the default") } + default: + cl = quietClassifier{ml.New(modelURL)} + if sidecarUp(modelURL) { + lg.Printf("router: smart mode ON (classifier sidecar = %s)", modelURL) + } else { + lg.Printf("router: smart mode AUTO β€” sidecar not responding at %s yet; routing on heuristics until it appears (run `whittle setup` if it was never installed)", modelURL) + } } px := NewProxy(pol, cl, NewMemSessionStore(), lg)