Long agent sessions drown in tokens, and most compressors buy their ratio by silently destroying what agents need: array rows vanish, file reads get gutted, identifiers come back mangled. Whittle holds one line: **lossless or clearly marked, code never touched, every anomaly fails open to the original bytes.**
@@ -77,7 +77,7 @@ The full per-type contract, ML prose path, architecture, and performance tables:
Whittle's second surface: a local proxy on `ANTHROPIC_BASE_URL` that sends each request to the cheapest model tier that can still handle it, per a policy you can read in one screen.
-- **Calibrated out of the box**: `whittle policy init` writes a conservative default (hard reasoning → strongest tier, confident chit-chat → cheapest, *everything else untouched*) with your account's real model ids auto-detected. [What it does & how to customize](router/policies/default.md).
+- **Calibrated out of the box**: `whittle policy init` writes a conservative default (hard reasoning → strongest tier, confident chit-chat → cheapest, *everything else untouched*) with your account's real model ids auto-detected. [What the default does](router/policies/default.md) · [how to write your own policy](docs/POLICY.md).
- **Multi-signal, not keyword-matching**: a trained 14-subject classifier (probability-mass thresholded, so an *uncertain* classification never escalates), a contrastive difficulty score, and your own keywords. Every log line shows each signal's value against its gate.
- **Rewrites the model, never your history**: prompt-cache prefixes survive; capabilities the cheaper model rejects are stripped automatically; credentials pass through untouched.
- **Never blocks you**: bad policy, dead classifier, or a rejected rewrite all fall back to your original request. Unset the env var and you're direct again.
@@ -97,7 +97,7 @@ Most context compressors are read-time proxies: they rewrite your conversation h
| loss model | lossless or marked | lossy by default, recover on demand |
| cost of failure | original bytes | a broken or blocked call |
-The full argument: [docs/why-write-time.md](docs/why-write-time.md).
+The full argument, and why routing is a different kind of proxy: [docs/why-whittle.md](docs/why-whittle.md).
## FAQ
diff --git a/docs/POLICY.md b/docs/POLICY.md
new file mode 100644
index 0000000..a4d3631
--- /dev/null
+++ b/docs/POLICY.md
@@ -0,0 +1,773 @@
+# Writing a whittle routing policy
+
+This is the task guide for authoring `~/.whittle/router.json`: the mental model,
+every condition you can write with its exact JSON, the signals and how to tune
+them, copy-paste recipes, and how to debug a policy from the router log. For the
+architecture and the signal math behind all of this, see
+[docs/ROUTER.md](ROUTER.md); this page is how to write the file, that page is how
+and why it works.
+
+Every JSON example below is validated against the real loader
+(`whittle policy validate`). If an example errors, that is a bug in this doc.
+
+## Mental model in ten lines
+
+1. You define **tiers**: named models, ordered cheap to capable.
+2. Every request is matched against an ordered list of **routes**.
+3. Each route has a `when` condition and a `to` tier. **First match wins.**
+4. If no route matches, the **`default`** applies.
+5. `default: "requested"` means "keep the model the client asked for" (a no-op).
+ A tier name as the default down-routes everything unmatched to that tier.
+6. A route condition is a boolean tree of **leaves** (keywords, token counts,
+ ML signals) combined with `all` / `any` / `not`.
+7. Cheap heuristic leaves (keywords, counts) cost nothing. **ML leaves**
+ (`domain`, `complexity`, `embedding`) run a local model.
+8. An ML signal that cannot answer (sidecar down, smart mode off) makes its leaf
+ **false**, and its route will not fire. Routing always falls open.
+9. You reload a changed policy with `SIGHUP`; a bad edit keeps the live policy.
+10. Every request logs which route fired and every signal value, so you can see
+ exactly why it routed.
+
+## The file at a glance
+
+```json
+{
+ "version": 1,
+ "tiers": [
+ {"name": "fast", "model": "claude-haiku-4-5-20251001"},
+ {"name": "main", "model": "claude-sonnet-4-5-20250929"},
+ {"name": "smart", "model": "claude-opus-4-8"}
+ ],
+ "default": "requested",
+ "inspect": {"scope": "recent_turns", "turns": 3},
+ "signals": { "domains": [], "embeddings": [], "complexity": [] },
+ "routes": [
+ {"name": "example", "when": {"keywords": ["refactor"]}, "to": "smart"}
+ ],
+ "session": {"sticky": false},
+ "overrides": {"pin_header": "x-whittle-route"}
+}
+```
+
+| field | required | what it is |
+|---|---|---|
+| `version` | yes | must be `1`. |
+| `tiers` | yes | at least one `{name, model}`, ordered cheap to capable. |
+| `default` | yes | a tier name, or the reserved `"requested"`. |
+| `inspect` | yes | `{scope, turns}`; bounds what text signals see (below). |
+| `routes` | no | the ordered waterfall; an empty list means "always default". |
+| `signals` | no | named ML signals your routes reference by name. |
+| `session` | no | stickiness; off by default. |
+| `overrides` | no | the pin header name. |
+
+Strict decoding: an unknown or misspelled key is a load error, never silently
+ignored. The smallest valid policy is:
+
+```json
+{
+ "version": 1,
+ "tiers": [{"name": "main", "model": "claude-sonnet-4-5-20250929"}],
+ "default": "requested",
+ "inspect": {"scope": "last_user_turn"}
+}
+```
+
+That policy routes nothing (it always keeps the requested model). It is the safe
+starting skeleton you add routes to.
+
+## Tiers and the default
+
+`tiers` order is meaningful: index 0 is the cheapest band, the last is the most
+capable. Session stickiness and the context-window guard use this rank, so keep
+the list ordered cheap to capable even if you never use those features.
+
+Use the **full dated model id** your account accepts (`claude-sonnet-4-5-20250929`,
+not `claude-sonnet-4-5`). Bare aliases are often rejected upstream with a 4xx.
+`whittle policy init` fills these in by reading the ids your Claude Code config
+actually sends; verify them before trusting a hand-edited id.
+
+`default` is where a request goes when no route claims it:
+
+- **`"requested"`** (the shipped posture): keep the model the client asked for,
+ untouched. With zero evidence about a request, whittle does not rewrite it, so
+ every model change (up or down) traces to a route you wrote. This also protects
+ mixed-model clients: Claude Code's cheap background requests are never
+ up-routed by a fixed default.
+- **A tier name** (for example `"main"`): down-route everything unmatched to that
+ tier. This is the single biggest cost lever, and a quality trade: traffic you
+ did not explicitly route now rides that tier. Pair it with guard routes that
+ push hard or sensitive work back up (see the aggressive-savings recipe).
+
+## Routes: the waterfall and the condition grammar
+
+`routes` is an ordered list. For each request the engine tries routes top to
+bottom and the **first** whose `when` matches decides the tier. Order your routes
+cheap-first and specific-first: put fast keyword routes above ML routes so a
+keyword match never pays for a classifier, and put narrow routes above broad ones.
+
+Each route is `{name, when, to}`:
+
+- **`name`**: appears in the log as `route:`. Name every route; an unnamed
+ route is logged by index and validation warns.
+- **`to`**: a tier name, or the reserved **`"keep"`** (below).
+- **`when`**: a condition tree.
+
+### The condition tree
+
+A `when` node is **either** one combinator **or** one leaf, never both, never two
+of either. The combinators:
+
+| combinator | JSON | matches when |
+|---|---|---|
+| `all` | `{"all": [A, B]}` | every child matches (AND) |
+| `any` | `{"any": [A, B]}` | at least one child matches (OR) |
+| `not` | `{"not": A}` | the single child does **not** match |
+
+`not` takes a **single** node, not a list. `all` and `any` take a list.
+
+Nesting is allowed up to 6 levels deep. There is **no implicit AND**: two leaves
+in one node is an error, wrap them in `all`.
+
+```json
+{"all": [
+ {"domain": "casual"},
+ {"not": {"has_tools": true}},
+ {"any": [
+ {"complexity": "reasoning:easy"},
+ {"keywords": ["typo", "rephrase"]}
+ ]}
+]}
+```
+
+A gotcha with `not` and ML signals: if the child leaf's signal is unavailable
+(sidecar down, smart mode off), the child is false, but the route still will not
+fire on the inverted result. An unavailable signal never causes a route to fire,
+directly or through `not`. This is deliberate fail-open behavior (see ROUTER.md
+§5), so you cannot accidentally up-route every request when the sidecar is down.
+
+### `to: "keep"`
+
+`keep` holds the session's current tier instead of naming a fixed one. It is only
+meaningful with session tracking: on the first request of a session there is
+nothing to keep, so `keep` falls to the `default`. `keep` never changes the stored
+tier. It is a niche tool for "do not move this session"; most policies do not need
+it. Stickiness in v1 is minimal (see ROUTER.md §11), so treat `keep` as advanced.
+
+## Leaf reference
+
+Every leaf, its exact JSON, its matching rule, and when to reach for it. A leaf is
+one key on a `when` node.
+
+### `keywords` (literal, whole-word)
+
+```json
+{"keywords": ["refactor", "race condition", "c++"]}
+```
+
+- **Matches**: case-insensitive, whole-word or whole-phrase, over the `inspect`
+ text window. An occurrence inside a larger alphanumeric run does not match:
+ `migration` does not fire on `immigration`, `refactor` does not fire on
+ `refactored`. List the variants you want. Word boundaries are any
+ non-alphanumeric character or the string edge, so `c++` matches in `use c++`.
+- **Use when**: you want a zero-latency, deterministic route for terms specific to
+ your work. Keywords are also the fallback when smart mode is off, so keyword
+ routes keep working with no sidecar.
+
+### `keywords_regex` (RE2, opt-in)
+
+```json
+{"keywords_regex": ["\\bTODO\\b", "fix ?me"]}
+```
+
+- **Matches**: Go's RE2 (linear-time, no catastrophic backtracking) against the
+ `inspect` window text. Patterns are capped at 512 characters; an invalid
+ pattern is a load error.
+- **Use when**: whole-word `keywords` is not expressive enough. Prefer `keywords`
+ for the common case; regex is the escape hatch.
+
+### `context_tokens` (numeric band)
+
+```json
+{"context_tokens": {"gte": 120000}}
+```
+
+- **Matches**: the estimated context size of the whole request (raw body bytes
+ divided by 4). This is the cost and prompt-cache scale; a floor around 20k is
+ normal even for a short prompt because the system prompt and tools count.
+- **Shape**: a bounds object with any of `gt`, `gte`, `lt`, `lte` (combine a
+ lower and an upper for a range), or a bare number as a shorthand for equality
+ (`{"context_tokens": 60000}` means exactly 60000, rarely what you want here).
+ `eq` cannot combine with the bounds. A quoted string is an error.
+- **Use when**: you want large requests on a stronger tier without any ML. This is
+ a heuristic proxy for difficulty (bigger context is often harder), cheap and
+ deterministic.
+
+### `message_count` (numeric band)
+
+```json
+{"message_count": {"lte": 2}}
+```
+
+- **Matches**: the number of messages in the request. Same band shape as
+ `context_tokens`; the scalar shorthand is handy here (`{"message_count": 1}`
+ means the very first turn).
+- **Use when**: you want to treat the opening turn differently from a deep
+ session, or gate on conversation depth.
+
+### `tool_loop` (boolean shape)
+
+```json
+{"tool_loop": true}
+```
+
+- **Matches**: the last message is `role:user` and carries a `tool_result` block,
+ meaning this request is an agent continuing a tool loop (not a fresh human
+ turn). `{"tool_loop": false}` matches when it is not. This is a precise
+ predicate, not "any tool was ever used".
+- **Use when**: you want tool-loop iterations to ride a different tier than
+ human-authored turns.
+
+### `has_tools` (boolean shape)
+
+```json
+{"has_tools": false}
+```
+
+- **Matches**: the request declares tools. `{"has_tools": false}` matches a
+ request with no tools (often plain chat). The boolean is compared for equality,
+ so `false` is a real predicate, not "ignore this leaf".
+- **Use when**: tool-enabled requests and plain-chat requests should route
+ differently.
+
+### `requested_model` (membership)
+
+```json
+{"requested_model": ["claude-opus-4-8"]}
+```
+
+- **Matches**: the model the client asked for is in the list. Both sides are
+ canonicalized (a trailing date snapshot and `-latest` are stripped), so
+ `claude-opus-4-8` matches an incoming `claude-opus-4-8-20260101`.
+- **Use when**: you want behavior that depends on what the client requested, for
+ example "only rewrite requests that came in on opus".
+
+### `domain` (ML: subject classification)
+
+```json
+{"domain": "quantitative"}
+```
+
+- **Matches**: the named domain signal (defined under `signals.domains`) fires.
+ See the signals section for the definition and its `min_mass` semantics.
+- **Use when**: you want to route by subject (math, law, health, casual chit-chat)
+ rather than by wording.
+
+### `complexity` (ML: difficulty level)
+
+```json
+{"complexity": "reasoning:hard"}
+```
+
+- **Matches**: the named complexity signal resolves to the given level. The ref is
+ `signal_name:level` where level is `hard`, `easy`, or `medium`. `medium` is a
+ real, matchable level (the band between hard and easy), not a fallback.
+- **Use when**: you want to route by how hard the request reads, independent of
+ its subject.
+
+### `embedding` (ML: similarity to your examples)
+
+```json
+{"embedding": "design-ask"}
+```
+
+- **Matches**: the named embedding signal's similarity score against your
+ candidate phrases clears its threshold.
+- **Use when**: you have one specific, high-value request shape you can describe
+ with a handful of example sentences.
+
+## Signals: define once, reference by name
+
+ML leaves (`domain`, `complexity`, `embedding`) do not carry their own parameters.
+You define a named signal under `signals`, then reference it by name from any
+number of leaves. Each signal is computed at most once per request, and only if a
+route actually reaches it.
+
+Two small models power all three signals, both in the sidecar `whittle setup`
+installs at `127.0.0.1:45872`. The engine holds only your thresholds. If the
+sidecar is absent, smart mode is simply off and ML leaves evaluate false; if it is
+present but broken, the request is tagged `ml-degraded` in the log.
+
+### `domains`: subject classification with mass thresholding
+
+```json
+"signals": {
+ "domains": [
+ {"name": "quantitative", "categories": ["math", "physics", "chemistry"], "min_mass": 0.7},
+ {"name": "casual", "categories": ["other"], "min_mass": 0.85}
+ ]
+}
+```
+
+- **`categories`**: one or more MMLU-Pro categories. The classifier only ever
+ emits these 14 labels: `biology`, `business`, `chemistry`, `computer science`,
+ `economics`, `engineering`, `health`, `history`, `law`, `math`, `other`,
+ `philosophy`, `physics`, `psychology`. A category outside this set is inert
+ (validation warns: the classifier will never emit it).
+- **`min_mass`** (0 to 1): the leaf fires when the **total** softmax probability
+ the classifier assigns to your categories is at least `min_mass`. Mass is
+ invariant to which in-set category won (math 0.4 + physics 0.4 clears a 0.7 gate
+ with no top-2 special case). An ambiguous, flat distribution never concentrates
+ enough mass, so it fails the gate and the request falls to the default. This is
+ why uncertainty lands on the middle tier and never escalates.
+- **Without `min_mass`**: the leaf falls back to argmax membership (fires if the
+ single top label is in your set). Prefer `min_mass`; it is the confident form.
+
+**Tuning `min_mass`.** Higher is stricter (more confidence required to fire).
+Probe the real number for a phrase before guessing:
+
+```sh
+curl -s localhost:45872/v1/route/domain \
+ -d '{"text": "prove that the sum of two odd numbers is even"}'
+# -> {"label":"math","confidence":0.91,"probs":{"math":0.91,"physics":0.03,...}}
+```
+
+Sum the `probs` over your categories: that number is what `min_mass` is compared
+against. Set the gate a little below the mass your target phrases produce and a
+little above what your false positives produce. The router log shows the evaluated
+mass on every request (`dom:quantitative=0.94/0.7`), so you can watch real traffic
+and adjust.
+
+### `complexity`: contrastive difficulty
+
+```json
+"signals": {
+ "complexity": [
+ {"name": "reasoning", "threshold": 0.15,
+ "hard": [
+ "analyze the root cause of this failure",
+ "weigh the tradeoffs and recommend an approach",
+ "design the architecture for this system"
+ ],
+ "easy": [
+ "fix this typo",
+ "rephrase this sentence",
+ "summarize this in one line"
+ ]}
+ ]
+}
+```
+
+- **`hard` / `easy`**: banks of example phrasings. The signal embeds the request
+ and both banks, scores similarity to each, and takes the margin
+ `score(hard) - score(easy)`.
+- **`threshold`** (a symmetric margin band, at least 0): margin above `+threshold`
+ is `hard`, below `-threshold` is `easy`, in between is `medium`. A larger
+ threshold widens the medium band, so fewer requests are classified either way.
+- Populate the banks with phrasings **shaped like your real requests**. The signal
+ scores similarity to these exemplars, so generic examples score generically.
+ Bank size is soft-capped at 32 phrases (a warning above that), hard-capped at
+ 256.
+
+**Tuning `threshold`.** Probe the margin for a phrase:
+
+```sh
+curl -s localhost:45872/v1/route/complexity \
+ -d '{"text": "debug this intermittent race condition",
+ "hard": ["analyze the root cause of this failure"],
+ "easy": ["fix this typo"]}'
+# -> {"margin": 0.184}
+```
+
+A margin of 0.184 is `hard` at threshold 0.15, `medium` at threshold 0.2. Probe
+a handful of your hard and easy phrasings, then set the threshold between the two
+clusters. The log shows the margin per request (`cplx:reasoning=+0.184`).
+
+### `embeddings`: similarity to your examples
+
+```json
+"signals": {
+ "embeddings": [
+ {"name": "design-ask", "threshold": 0.55,
+ "candidates": [
+ "design the architecture for this system",
+ "propose a system design for this feature",
+ "what are the tradeoffs between these approaches"
+ ]}
+ ]
+}
+```
+
+- **`candidates`**: example phrases. The signal scores the request's similarity to
+ this bank (same 0.75 best + 0.25 mean-top-2 blend as complexity), and the leaf
+ fires when the score is at least `threshold`. Same 32 soft / 256 hard caps.
+- **`threshold`**: the embedding space has a high similarity floor (unrelated
+ sentences score around 0.35 to 0.4), so useful thresholds live in a narrow band
+ above that floor. Do not trust a new candidate list without probing.
+
+**Tuning.** Probe the score:
+
+```sh
+curl -s localhost:45872/v1/route/embedding \
+ -d '{"text": "how should I structure the service layer",
+ "candidates": ["design the architecture for this system"]}'
+# -> {"score": 0.58}
+```
+
+Set the threshold above what unrelated requests score and below what your target
+phrasings score. The log shows the score per request (`emb:design-ask=+0.58`).
+
+## `inspect`: what text the signals see
+
+```json
+"inspect": {"scope": "recent_turns", "turns": 3}
+```
+
+`inspect` bounds which user text is considered, keeping the classifier off the
+giant system prompt and off tool output. Only user-authored text blocks count;
+tool results, model thinking, and framework-injected `` blocks
+are excluded from both keyword and ML input.
+
+| `scope` | text considered |
+|---|---|
+| `last_user_turn` | only the most recent user message. |
+| `recent_turns` | the last `turns` user messages, joined. Requires `turns` > 0. |
+| `full` | all user messages, joined. |
+
+One important asymmetry: `inspect` sets the **keyword** window (a keyword two
+turns back still protects, which is the point). The **ML signals always classify
+only the latest user turn** regardless of scope, because averaging turns dilutes a
+classifier (a turn scoring hard alone falls to medium once joined with two trivial
+turns, silently suppressing escalation). So `recent_turns: 3` gives keywords a
+three-turn memory while `domain` and `complexity` still judge just the latest turn.
+
+## Session stickiness and the pin header
+
+```json
+"session": {"sticky": false},
+"overrides": {"pin_header": "x-whittle-route"}
+```
+
+**Stickiness** damps a downgrade on the fuzzy path (the static default) so a
+session does not flap between tiers. It only damps **default** decisions, never an
+explicit route or a pin, and only **downgrades** (upgrades are always free). A
+downgrade is held unless it jumps at least `min_band_jump` bands. With only three
+tiers a one-band downgrade is the common case, so `min_band_jump` must be at least
+2 to damp anything; `sticky: true` with a smaller jump does nothing and validation
+warns. The shipped presets keep it off.
+
+**Pin header** is the per-request escape hatch. With `pin_header` set, a request
+carrying that header pins the tier and bypasses all routing:
+
+```sh
+curl ... -H 'x-whittle-route: smart'
+```
+
+An unknown tier in the header is ignored (the request falls through to normal
+routing), never a 400. A pin never writes the session's tracked tier.
+
+## The authoring loop
+
+1. **Start from a preset.** `whittle policy init` writes the calibrated `default`
+ policy to `~/.whittle/router.json` with your model ids detected. Run
+ `whittle policy list` to see available presets.
+
+2. **Edit** `~/.whittle/router.json`.
+
+3. **Validate before reloading.** Validation is the authoring contract: it runs
+ the real loader and prints errors and cost-lint warnings.
+
+ ```sh
+ whittle policy validate ~/.whittle/router.json
+ # VALID ✓ tiers=3 routes=3 default=requested
+ # warn: routes[0] (escalate): references an ML signal ... place cheap routes above it
+ ```
+
+ Fix every error. Read the warnings; they flag inert config (a route that never
+ fires, an ML route above a cheap one) that loads fine but is probably not what
+ you meant.
+
+4. **Reload with SIGHUP.** The running router hot-reloads on `SIGHUP`. A reload
+ that fails validation is rejected and the live policy stays, so a bad edit never
+ bricks routing.
+
+ ```sh
+ # launchd (installed with `whittle route -install`):
+ launchctl kill HUP gui/$(id -u)/dev.firstops.whittle.router
+
+ # foreground (`whittle route` in a terminal):
+ kill -HUP
+ ```
+
+ The startup and reload log lines confirm the policy loaded and whether smart
+ mode is on.
+
+5. **Watch the log.** The router writes one JSON line per request. Under launchd
+ it goes to `~/.whittle/logs/router.log`; in the foreground it goes to stderr.
+
+ ```sh
+ tail -f ~/.whittle/logs/router.log
+ ```
+
+## Reading the log's `signals` field
+
+Every request logs one line. The `reason` field names the rule that fired; the
+`signals` field shows every ML value computed, against its gate:
+
+```json
+{"tier":"fast","requested":"claude-opus-4-8","model":"claude-haiku-4-5-20251001",
+ "reason":"route:casual-easy","signals":"dom=other@1.000 dom:casual=0.899/0.85 cplx:reasoning=-0.471",
+ "status":200,"latency_ms":41,"ctx_tokens":2140,"in_tokens":1974,"out_tokens":16,"session":"c2f9659c"}
+```
+
+- `requested` vs `model` is what routing changed. Equal means no rewrite happened.
+- `reason` names the exact path: `route:`, `default`, `default:requested`,
+ `pin:`, plus suffixes like ` stripped:`, ` sticky:kept`,
+ ` ml-degraded`, and ` mode-b:...` on a retry.
+- `signals` is the ML trace, space-separated. It is empty when a heuristic leaf
+ decided first (no ML was reached).
+
+The `signals` vocabulary:
+
+| token | meaning |
+|---|---|
+| `dom=