Router: out-of-the-box policies, battle-tested reconciliation, calibrated signals, one design doc

README.md

@@ -29,11 +29,16 @@ an MCP retrieval tool. See [Why whittle](#why-whittle---compress-at-write-time-n
 and [Architecture](#architecture) for the reasoning and the diagram.
 
 Whittle has a **second, independent surface: an opt-in model router**
-(`whittle route`). It runs a local proxy on `ANTHROPIC_BASE_URL` that routes each
-request to the cheapest model tier that can still handle it, per a policy you
-write. It's off by default, rewrites the *model* and never your conversation
-history, forwards your own credentials untouched, and fails open (down → your
-request passes straight through). See [Model routing](#model-routing-opt-in).
+(`whittle route`). It runs a local proxy on `ANTHROPIC_BASE_URL` that sends each
+request to the cheapest model tier that can still handle it — hard reasoning
+stays on your strongest model, trivial edits drop to the cheapest, the broad
+middle rides a capable default. Decisions come from one auditable policy file
+you own, driven by real signals: a trained subject classifier, a contrastive
+difficulty score, and your own keywords. It's off by default, rewrites the
+*model* and never your conversation history, forwards your credentials
+untouched, and fails open on every path. See
+[Model routing](#model-routing-opt-in) and the full design in
+[docs/ROUTER.md](docs/ROUTER.md).
 
 ## See it
 
@@ -152,33 +157,50 @@ cheapest model tier that can still serve it, per a policy you author
 (`~/.whittle/router.json`):
 
 ```
+whittle policy init                            # write a starter policy to ~/.whittle/router.json
 whittle route                                  # proxy on 127.0.0.1:45873
 export ANTHROPIC_BASE_URL=http://127.0.0.1:45873
 ```
 
-- **You write the policy.** Tiers, first-match routes (keywords, context size,
-  tool-loop, requested model), and a default - a precedence ladder of
-  pin → routes → classify → default. See
-  [docs/ROUTER_POLICY_SCHEMA.md](docs/ROUTER_POLICY_SCHEMA.md).
+- **Start from the built-in policy.** `whittle policy init` writes the calibrated
+  `default` policy with the model ids your account actually uses (auto-detected);
+  `whittle policy validate <file>` checks your edits against the real loader.
+  What it does and how to customize it: [router/policies/default.md](router/policies/default.md).
+- **One auditable policy file.** Tiers, first-match routes over a boolean
+  condition tree, and a default - you can read the file and predict every
+  decision, and the per-request log names the rule that fired. The full grammar,
+  the signal math, and the design rationale live in
+  [docs/ROUTER.md](docs/ROUTER.md).
+- **Multi-signal, not keyword-matching.** With smart mode on
+  (`WHITTLE_ROUTER_MODEL_URL` → the sidecar), routes can test a trained
+  14-subject classifier (thresholding its full probability mass, so an
+  *uncertain* classification falls to your default tier - never up to the
+  expensive one), a contrastive hard/easy difficulty score, and semantic
+  similarity to your own example phrases - alongside whole-word keywords and
+  context-size heuristics. Smart mode off → the heuristics still route,
+  everything else rides the default.
 - **It rewrites the model, not your history.** The router swaps the target model
-  and reconciles capabilities for it (dropping features the cheaper model can't
-  serve). It never mutates conversation history, so prompt-cache prefixes stay
-  intact - the failure mode that makes read-time proxies painful.
+  and reconciles capabilities for it (stripping betas and features the cheaper
+  model rejects). It never mutates conversation history, so prompt-cache
+  prefixes stay intact - the failure mode that makes read-time proxies painful.
 - **Your credentials pass through untouched.** The proxy forwards your own auth
   to Anthropic; whittle never sees or stores a key.
-- **Fail-open by construction.** A missing/invalid policy, a parse error, or a
-  rewrite the upstream rejects all fall back to your original request. If the
-  router is down, unset `ANTHROPIC_BASE_URL` and you're direct again - nothing is
-  trapped behind it.
-
-The router is **new** and not on the battle-tested footing the compression hook
-is - run it when you want tier routing, leave it off and nothing changes. Today
-it routes on deterministic signals; opt-in **smart routing** (an ML intent
-classifier and few-shot classify, hosted in the same sidecar) turns on when you
-set `WHITTLE_ROUTER_MODEL_URL` to the classifier sidecar - unset, routing is
-heuristics-only and the ML paths cleanly fall through to your default. Configure
-the upstream with `WHITTLE_ROUTER_UPSTREAM` (defaults to `api.anthropic.com`;
-point it at a corporate Anthropic gateway if you have one).
+- **Fail-open by construction.** A missing/invalid policy, a parse error, a dead
+  classifier, or a rewrite the upstream rejects all fall back to your original
+  request. If the router is down, unset `ANTHROPIC_BASE_URL` and you're direct
+  again - nothing is trapped behind it.
+- **Savings you can measure.** Every request logs the requested model, the model
+  that actually served it, and the response's real token usage - enough to
+  compute exactly what routing saved you.
+
+The classifier and embedding models (and the scoring math behind the difficulty
+signal) come from [vLLM Semantic Router](https://github.com/vllm-project/semantic-router)
+([whitepaper](https://vllm-semantic-router.com/white-paper)); the architecture
+around them - the single-file policy, the probability-mass thresholding, the
+fail-open model - is whittle's own, and [docs/ROUTER.md](docs/ROUTER.md)
+credits the line between the two precisely. Configure the upstream with
+`WHITTLE_ROUTER_UPSTREAM` (defaults to `api.anthropic.com`; point it at a
+corporate Anthropic gateway if you have one).
 
 Run it in the foreground, or install it as a background service (opt-in, separate
 from the compress daemon):

cmd/whittle/main.go

@@ -37,6 +37,8 @@ func main() {
 		cmdServe(os.Args[2:])
 	case "route":
 		cmdRoute(os.Args[2:])
+	case "policy":
+		cmdPolicy(os.Args[2:])
 	case "setup":
 		cmdSetup(os.Args[2:])
 	case "daemon":
@@ -75,6 +77,7 @@ usage:
   whittle serve    [flags]          run the compress HTTP API in the foreground
   whittle route    [flags]          run the model-router daemon (opt-in) on
                                     ANTHROPIC_BASE_URL
+  whittle policy   <cmd>            manage router policies (list/show/init/validate)
   whittle version
 
 compress flags:

cmd/whittle/policy.go

@@ -0,0 +1,229 @@
+package main
+
+import (
+	"bytes"
+	"encoding/json"
+	"flag"
+	"fmt"
+	"os"
+	"path/filepath"
+	"regexp"
+	"strings"
+
+	"github.com/firstops-dev/whittle/router"
+)
+
+var (
+	anyModelID   = regexp.MustCompile(`claude-(?:haiku|sonnet|opus)-\d[0-9a-z-]*`)
+	modelValueID = regexp.MustCompile(`"model"\s*:\s*"(claude-(?:haiku|sonnet|opus)-\d[0-9a-z-]*)"`)
+	datedID      = regexp.MustCompile(`-\d{8}$`)
+)
+
+func modelFamily(id string) string {
+	switch {
+	case strings.Contains(id, "haiku"):
+		return "haiku"
+	case strings.Contains(id, "sonnet"):
+		return "sonnet"
+	case strings.Contains(id, "opus"):
+		return "opus"
+	}
+	return ""
+}
+
+// detectClaudeModels scans Claude Code's config for the model id per family
+// (haiku/sonnet/opus) the account actually uses, so `policy init` fills the tiers
+// with ids Anthropic will accept — not the guessable placeholders that 4xx on
+// every rewrite. Ranking: an id used as a "model": value (proven-valid, what
+// Claude Code sends) beats a dated id beats a bare/lower one.
+func detectClaudeModels() map[string]string {
+	home, _ := os.UserHomeDir()
+	files := []string{filepath.Join(home, ".claude.json"), filepath.Join(home, ".claude", "settings.json")}
+
+	type cand struct {
+		id      string
+		isValue bool
+	}
+	best := map[string]cand{}
+	consider := func(id string, isValue bool) {
+		fam := modelFamily(id)
+		if fam == "" {
+			return
+		}
+		cur, ok := best[fam]
+		if !ok || betterModel(id, isValue, cur.id, cur.isValue) {
+			best[fam] = cand{id, isValue}
+		}
+	}
+	for _, f := range files {
+		b, err := os.ReadFile(f)
+		if err != nil {
+			continue
+		}
+		s := string(b)
+		for _, m := range modelValueID.FindAllStringSubmatch(s, -1) {
+			consider(m[1], true)
+		}
+		for _, id := range anyModelID.FindAllString(s, -1) {
+			consider(id, false)
+		}
+	}
+	out := map[string]string{}
+	for fam, c := range best {
+		out[fam] = c.id
+	}
+	return out
+}
+
+func betterModel(aID string, aVal bool, bID string, bVal bool) bool {
+	// A full dated id first (bare aliases often 404). Then the highest version /
+	// newest date — the config's "model" field proved STALE (it named opus-4-7
+	// while Claude Code actually sent opus-4-8), so version outranks it; a "model":
+	// occurrence is only a final tiebreak between otherwise-identical ids.
+	if ad, bd := datedID.MatchString(aID), datedID.MatchString(bID); ad != bd {
+		return ad
+	}
+	if aID != bID {
+		return aID > bID
+	}
+	return aVal && !bVal
+}
+
+// fillModels substitutes the model ids detected from Claude Code's config.
+func fillModels(preset []byte) ([]byte, []string) {
+	return fillModelsWith(preset, detectClaudeModels())
+}
+
+// fillModelsWith substitutes the given family→id map into a preset's tiers, by
+// family. Returns the (possibly unchanged) JSON and human-readable notes.
+func fillModelsWith(preset []byte, detected map[string]string) ([]byte, []string) {
+	var doc struct {
+		Tiers []struct{ Name, Model string } `json:"tiers"`
+	}
+	if json.Unmarshal(preset, &doc) != nil {
+		return preset, nil
+	}
+	out := preset
+	var notes []string
+	for _, t := range doc.Tiers {
+		if id, ok := detected[modelFamily(t.Model)]; ok && id != t.Model {
+			out = bytes.ReplaceAll(out, []byte(`"`+t.Model+`"`), []byte(`"`+id+`"`))
+			notes = append(notes, fmt.Sprintf("%s: %s → %s", t.Name, t.Model, id))
+		}
+	}
+	return out, notes
+}
+
+// cmdPolicy manages router policies: list/show built-in presets, write one to
+// ~/.whittle/router.json, or validate a policy file with the real loader.
+func cmdPolicy(args []string) {
+	if len(args) == 0 {
+		policyUsage()
+		os.Exit(2)
+	}
+	switch args[0] {
+	case "list":
+		for _, n := range router.PresetNames() {
+			fmt.Printf("  %-10s %s\n", n, router.PresetDescription(n))
+		}
+	case "show":
+		policyShow(args[1:])
+	case "init":
+		policyInit(args[1:])
+	case "validate":
+		policyValidate(args[1:])
+	default:
+		policyUsage()
+		os.Exit(2)
+	}
+}
+
+func policyShow(args []string) {
+	name := "default"
+	if len(args) > 0 {
+		name = args[0]
+	}
+	b, err := router.Preset(name)
+	if err != nil {
+		fmt.Fprintln(os.Stderr, "whittle:", err)
+		os.Exit(1)
+	}
+	os.Stdout.Write(b)
+}
+
+func policyInit(args []string) {
+	fs := flag.NewFlagSet("policy init", flag.ExitOnError)
+	force := fs.Bool("force", false, "overwrite an existing policy file")
+	out := fs.String("o", filepath.Join(whittleHome(), "router.json"), "output path")
+	_ = fs.Parse(args)
+	name := "default"
+	if fs.NArg() > 0 {
+		name = fs.Arg(0)
+	}
+
+	b, err := router.Preset(name)
+	if err != nil {
+		fmt.Fprintln(os.Stderr, "whittle:", err)
+		os.Exit(1)
+	}
+	// Fill the tiers with the real model ids the account uses (from Claude Code's
+	// config) so the policy works on first request instead of 4xx-ing every rewrite.
+	b, notes := fillModels(b)
+	// Defensive: the result must still load (TestPresets_AllValid guards the base).
+	if _, _, verr := router.Load(b); verr != nil {
+		fmt.Fprintln(os.Stderr, "whittle: policy failed to validate (bug):", verr)
+		os.Exit(1)
+	}
+	if _, err := os.Stat(*out); err == nil && !*force {
+		fmt.Printf("whittle: %s already exists — pass -force to overwrite, or -o <path> to write elsewhere\n", *out)
+		os.Exit(1)
+	}
+	must(os.MkdirAll(filepath.Dir(*out), 0o755))
+	must(os.WriteFile(*out, b, 0o644))
+	fmt.Printf("  ✓ wrote %q policy → %s\n", name, *out)
+	if len(notes) > 0 {
+		fmt.Println("  set tier models from your Claude Code config (verify these are right):")
+		for _, n := range notes {
+			fmt.Println("      " + n)
+		}
+	} else {
+		fmt.Println("  ⚠ could not detect your model IDs — the tiers use PLACEHOLDER IDs that")
+		fmt.Println("    Anthropic will reject. Edit the tier \"model\" values to real IDs (the full")
+		fmt.Println("    dated form, e.g. claude-sonnet-4-5-20250929), then `whittle policy validate`.")
+	}
+	fmt.Println("  Then start the router:")
+	fmt.Printf("    whittle route -install    # or `whittle route` in the foreground\n")
+	fmt.Printf("    export ANTHROPIC_BASE_URL=http://%s\n", router.DefaultAddr)
+}
+
+func policyValidate(args []string) {
+	if len(args) == 0 {
+		fmt.Fprintln(os.Stderr, "usage: whittle policy validate <file>")
+		os.Exit(2)
+	}
+	b, err := os.ReadFile(args[0])
+	if err != nil {
+		fmt.Fprintln(os.Stderr, "whittle:", err)
+		os.Exit(1)
+	}
+	p, warns, err := router.Load(b)
+	if err != nil {
+		fmt.Println("INVALID:", err)
+		os.Exit(1)
+	}
+	fmt.Printf("VALID ✓  tiers=%d routes=%d default=%s\n", len(p.Tiers), len(p.Routes), p.Default)
+	for _, w := range warns {
+		fmt.Println("  warn:", w)
+	}
+}
+
+func policyUsage() {
+	fmt.Fprintln(os.Stderr, `whittle policy — router policy management
+
+  whittle policy list                       list the built-in policies
+  whittle policy show [name]                print a built-in policy (default: default)
+  whittle policy init [name] [-force] [-o path]
+                                            write a built-in policy to ~/.whittle/router.json
+                                            (see router/policies/default.md for customization)
+  whittle policy validate <file>            validate a policy file (loader errors + warnings)`)
+}