diff --git a/README.md b/README.md index ad06c1f..0edfb12 100644 --- a/README.md +++ b/README.md @@ -29,11 +29,16 @@ an MCP retrieval tool. See [Why whittle](#why-whittle---compress-at-write-time-n and [Architecture](#architecture) for the reasoning and the diagram. Whittle has a **second, independent surface: an opt-in model router** -(`whittle route`). It runs a local proxy on `ANTHROPIC_BASE_URL` that routes each -request to the cheapest model tier that can still handle it, per a policy you -write. It's off by default, rewrites the *model* and never your conversation -history, forwards your own credentials untouched, and fails open (down → your -request passes straight through). See [Model routing](#model-routing-opt-in). +(`whittle route`). It runs a local proxy on `ANTHROPIC_BASE_URL` that sends each +request to the cheapest model tier that can still handle it — hard reasoning +stays on your strongest model, trivial edits drop to the cheapest, the broad +middle rides a capable default. Decisions come from one auditable policy file +you own, driven by real signals: a trained subject classifier, a contrastive +difficulty score, and your own keywords. It's off by default, rewrites the +*model* and never your conversation history, forwards your credentials +untouched, and fails open on every path. See +[Model routing](#model-routing-opt-in) and the full design in +[docs/ROUTER.md](docs/ROUTER.md). ## See it @@ -152,33 +157,50 @@ cheapest model tier that can still serve it, per a policy you author (`~/.whittle/router.json`): ``` +whittle policy init # write a starter policy to ~/.whittle/router.json whittle route # proxy on 127.0.0.1:45873 export ANTHROPIC_BASE_URL=http://127.0.0.1:45873 ``` -- **You write the policy.** Tiers, first-match routes (keywords, context size, - tool-loop, requested model), and a default - a precedence ladder of - pin → routes → classify → default. See - [docs/ROUTER_POLICY_SCHEMA.md](docs/ROUTER_POLICY_SCHEMA.md). +- **Start from the built-in policy.** `whittle policy init` writes the calibrated + `default` policy with the model ids your account actually uses (auto-detected); + `whittle policy validate ` checks your edits against the real loader. + What it does and how to customize it: [router/policies/default.md](router/policies/default.md). +- **One auditable policy file.** Tiers, first-match routes over a boolean + condition tree, and a default - you can read the file and predict every + decision, and the per-request log names the rule that fired. The full grammar, + the signal math, and the design rationale live in + [docs/ROUTER.md](docs/ROUTER.md). +- **Multi-signal, not keyword-matching.** With smart mode on + (`WHITTLE_ROUTER_MODEL_URL` → the sidecar), routes can test a trained + 14-subject classifier (thresholding its full probability mass, so an + *uncertain* classification falls to your default tier - never up to the + expensive one), a contrastive hard/easy difficulty score, and semantic + similarity to your own example phrases - alongside whole-word keywords and + context-size heuristics. Smart mode off → the heuristics still route, + everything else rides the default. - **It rewrites the model, not your history.** The router swaps the target model - and reconciles capabilities for it (dropping features the cheaper model can't - serve). It never mutates conversation history, so prompt-cache prefixes stay - intact - the failure mode that makes read-time proxies painful. + and reconciles capabilities for it (stripping betas and features the cheaper + model rejects). It never mutates conversation history, so prompt-cache + prefixes stay intact - the failure mode that makes read-time proxies painful. - **Your credentials pass through untouched.** The proxy forwards your own auth to Anthropic; whittle never sees or stores a key. -- **Fail-open by construction.** A missing/invalid policy, a parse error, or a - rewrite the upstream rejects all fall back to your original request. If the - router is down, unset `ANTHROPIC_BASE_URL` and you're direct again - nothing is - trapped behind it. - -The router is **new** and not on the battle-tested footing the compression hook -is - run it when you want tier routing, leave it off and nothing changes. Today -it routes on deterministic signals; opt-in **smart routing** (an ML intent -classifier and few-shot classify, hosted in the same sidecar) turns on when you -set `WHITTLE_ROUTER_MODEL_URL` to the classifier sidecar - unset, routing is -heuristics-only and the ML paths cleanly fall through to your default. Configure -the upstream with `WHITTLE_ROUTER_UPSTREAM` (defaults to `api.anthropic.com`; -point it at a corporate Anthropic gateway if you have one). +- **Fail-open by construction.** A missing/invalid policy, a parse error, a dead + classifier, or a rewrite the upstream rejects all fall back to your original + request. If the router is down, unset `ANTHROPIC_BASE_URL` and you're direct + again - nothing is trapped behind it. +- **Savings you can measure.** Every request logs the requested model, the model + that actually served it, and the response's real token usage - enough to + compute exactly what routing saved you. + +The classifier and embedding models (and the scoring math behind the difficulty +signal) come from [vLLM Semantic Router](https://github.com/vllm-project/semantic-router) +([whitepaper](https://vllm-semantic-router.com/white-paper)); the architecture +around them - the single-file policy, the probability-mass thresholding, the +fail-open model - is whittle's own, and [docs/ROUTER.md](docs/ROUTER.md) +credits the line between the two precisely. Configure the upstream with +`WHITTLE_ROUTER_UPSTREAM` (defaults to `api.anthropic.com`; point it at a +corporate Anthropic gateway if you have one). Run it in the foreground, or install it as a background service (opt-in, separate from the compress daemon): diff --git a/cmd/whittle/main.go b/cmd/whittle/main.go index 0b83896..8d0904c 100644 --- a/cmd/whittle/main.go +++ b/cmd/whittle/main.go @@ -37,6 +37,8 @@ func main() { cmdServe(os.Args[2:]) case "route": cmdRoute(os.Args[2:]) + case "policy": + cmdPolicy(os.Args[2:]) case "setup": cmdSetup(os.Args[2:]) case "daemon": @@ -75,6 +77,7 @@ usage: whittle serve [flags] run the compress HTTP API in the foreground whittle route [flags] run the model-router daemon (opt-in) on ANTHROPIC_BASE_URL + whittle policy manage router policies (list/show/init/validate) whittle version compress flags: diff --git a/cmd/whittle/policy.go b/cmd/whittle/policy.go new file mode 100644 index 0000000..4d34f8e --- /dev/null +++ b/cmd/whittle/policy.go @@ -0,0 +1,229 @@ +package main + +import ( + "bytes" + "encoding/json" + "flag" + "fmt" + "os" + "path/filepath" + "regexp" + "strings" + + "github.com/firstops-dev/whittle/router" +) + +var ( + anyModelID = regexp.MustCompile(`claude-(?:haiku|sonnet|opus)-\d[0-9a-z-]*`) + modelValueID = regexp.MustCompile(`"model"\s*:\s*"(claude-(?:haiku|sonnet|opus)-\d[0-9a-z-]*)"`) + datedID = regexp.MustCompile(`-\d{8}$`) +) + +func modelFamily(id string) string { + switch { + case strings.Contains(id, "haiku"): + return "haiku" + case strings.Contains(id, "sonnet"): + return "sonnet" + case strings.Contains(id, "opus"): + return "opus" + } + return "" +} + +// detectClaudeModels scans Claude Code's config for the model id per family +// (haiku/sonnet/opus) the account actually uses, so `policy init` fills the tiers +// with ids Anthropic will accept — not the guessable placeholders that 4xx on +// every rewrite. Ranking: an id used as a "model": value (proven-valid, what +// Claude Code sends) beats a dated id beats a bare/lower one. +func detectClaudeModels() map[string]string { + home, _ := os.UserHomeDir() + files := []string{filepath.Join(home, ".claude.json"), filepath.Join(home, ".claude", "settings.json")} + + type cand struct { + id string + isValue bool + } + best := map[string]cand{} + consider := func(id string, isValue bool) { + fam := modelFamily(id) + if fam == "" { + return + } + cur, ok := best[fam] + if !ok || betterModel(id, isValue, cur.id, cur.isValue) { + best[fam] = cand{id, isValue} + } + } + for _, f := range files { + b, err := os.ReadFile(f) + if err != nil { + continue + } + s := string(b) + for _, m := range modelValueID.FindAllStringSubmatch(s, -1) { + consider(m[1], true) + } + for _, id := range anyModelID.FindAllString(s, -1) { + consider(id, false) + } + } + out := map[string]string{} + for fam, c := range best { + out[fam] = c.id + } + return out +} + +func betterModel(aID string, aVal bool, bID string, bVal bool) bool { + // A full dated id first (bare aliases often 404). Then the highest version / + // newest date — the config's "model" field proved STALE (it named opus-4-7 + // while Claude Code actually sent opus-4-8), so version outranks it; a "model": + // occurrence is only a final tiebreak between otherwise-identical ids. + if ad, bd := datedID.MatchString(aID), datedID.MatchString(bID); ad != bd { + return ad + } + if aID != bID { + return aID > bID + } + return aVal && !bVal +} + +// fillModels substitutes the model ids detected from Claude Code's config. +func fillModels(preset []byte) ([]byte, []string) { + return fillModelsWith(preset, detectClaudeModels()) +} + +// fillModelsWith substitutes the given family→id map into a preset's tiers, by +// family. Returns the (possibly unchanged) JSON and human-readable notes. +func fillModelsWith(preset []byte, detected map[string]string) ([]byte, []string) { + var doc struct { + Tiers []struct{ Name, Model string } `json:"tiers"` + } + if json.Unmarshal(preset, &doc) != nil { + return preset, nil + } + out := preset + var notes []string + for _, t := range doc.Tiers { + if id, ok := detected[modelFamily(t.Model)]; ok && id != t.Model { + out = bytes.ReplaceAll(out, []byte(`"`+t.Model+`"`), []byte(`"`+id+`"`)) + notes = append(notes, fmt.Sprintf("%s: %s → %s", t.Name, t.Model, id)) + } + } + return out, notes +} + +// cmdPolicy manages router policies: list/show built-in presets, write one to +// ~/.whittle/router.json, or validate a policy file with the real loader. +func cmdPolicy(args []string) { + if len(args) == 0 { + policyUsage() + os.Exit(2) + } + switch args[0] { + case "list": + for _, n := range router.PresetNames() { + fmt.Printf(" %-10s %s\n", n, router.PresetDescription(n)) + } + case "show": + policyShow(args[1:]) + case "init": + policyInit(args[1:]) + case "validate": + policyValidate(args[1:]) + default: + policyUsage() + os.Exit(2) + } +} + +func policyShow(args []string) { + name := "default" + if len(args) > 0 { + name = args[0] + } + b, err := router.Preset(name) + if err != nil { + fmt.Fprintln(os.Stderr, "whittle:", err) + os.Exit(1) + } + os.Stdout.Write(b) +} + +func policyInit(args []string) { + fs := flag.NewFlagSet("policy init", flag.ExitOnError) + force := fs.Bool("force", false, "overwrite an existing policy file") + out := fs.String("o", filepath.Join(whittleHome(), "router.json"), "output path") + _ = fs.Parse(args) + name := "default" + if fs.NArg() > 0 { + name = fs.Arg(0) + } + + b, err := router.Preset(name) + if err != nil { + fmt.Fprintln(os.Stderr, "whittle:", err) + os.Exit(1) + } + // Fill the tiers with the real model ids the account uses (from Claude Code's + // config) so the policy works on first request instead of 4xx-ing every rewrite. + b, notes := fillModels(b) + // Defensive: the result must still load (TestPresets_AllValid guards the base). + if _, _, verr := router.Load(b); verr != nil { + fmt.Fprintln(os.Stderr, "whittle: policy failed to validate (bug):", verr) + os.Exit(1) + } + if _, err := os.Stat(*out); err == nil && !*force { + fmt.Printf("whittle: %s already exists — pass -force to overwrite, or -o to write elsewhere\n", *out) + os.Exit(1) + } + must(os.MkdirAll(filepath.Dir(*out), 0o755)) + must(os.WriteFile(*out, b, 0o644)) + fmt.Printf(" ✓ wrote %q policy → %s\n", name, *out) + if len(notes) > 0 { + fmt.Println(" set tier models from your Claude Code config (verify these are right):") + for _, n := range notes { + fmt.Println(" " + n) + } + } else { + fmt.Println(" ⚠ could not detect your model IDs — the tiers use PLACEHOLDER IDs that") + fmt.Println(" Anthropic will reject. Edit the tier \"model\" values to real IDs (the full") + fmt.Println(" dated form, e.g. claude-sonnet-4-5-20250929), then `whittle policy validate`.") + } + fmt.Println(" Then start the router:") + fmt.Printf(" whittle route -install # or `whittle route` in the foreground\n") + fmt.Printf(" export ANTHROPIC_BASE_URL=http://%s\n", router.DefaultAddr) +} + +func policyValidate(args []string) { + if len(args) == 0 { + fmt.Fprintln(os.Stderr, "usage: whittle policy validate ") + os.Exit(2) + } + b, err := os.ReadFile(args[0]) + if err != nil { + fmt.Fprintln(os.Stderr, "whittle:", err) + os.Exit(1) + } + p, warns, err := router.Load(b) + if err != nil { + fmt.Println("INVALID:", err) + os.Exit(1) + } + fmt.Printf("VALID ✓ tiers=%d routes=%d default=%s\n", len(p.Tiers), len(p.Routes), p.Default) + for _, w := range warns { + fmt.Println(" warn:", w) + } +} + +func policyUsage() { + fmt.Fprintln(os.Stderr, `whittle policy — router policy management + + whittle policy list list the built-in policies + whittle policy show [name] print a built-in policy (default: default) + whittle policy init [name] [-force] [-o path] + write a built-in policy to ~/.whittle/router.json + (see router/policies/default.md for customization) + whittle policy validate validate a policy file (loader errors + warnings)`) +} diff --git a/cmd/whittle/policy_test.go b/cmd/whittle/policy_test.go new file mode 100644 index 0000000..baf4623 --- /dev/null +++ b/cmd/whittle/policy_test.go @@ -0,0 +1,68 @@ +package main + +import ( + "strings" + "testing" + + "github.com/firstops-dev/whittle/router" +) + +func TestModelFamily(t *testing.T) { + cases := map[string]string{ + "claude-haiku-4-5-20251001": "haiku", + "claude-sonnet-4-5": "sonnet", + "claude-opus-4-7": "opus", + "gpt-5": "", + } + for in, want := range cases { + if got := modelFamily(in); got != want { + t.Errorf("modelFamily(%q) = %q, want %q", in, got, want) + } + } +} + +func TestBetterModel(t *testing.T) { + // A dated id beats a bare one (real haiku/sonnet ids are dated; bare 404s). + if !betterModel("claude-sonnet-4-5-20250929", false, "claude-sonnet-4-5", false) { + t.Error("a dated id should win over a bare id") + } + // The highest version wins even over a lower-versioned config "model" value — + // the config field proved stale (named opus-4-7 while CC sent opus-4-8). + if !betterModel("claude-opus-4-8", false, "claude-opus-4-7", true) { + t.Error("higher opus version should win over a lower-versioned model-value") + } + // Same id: a "model": occurrence is the tiebreak. + if !betterModel("claude-opus-4-8", true, "claude-opus-4-8", false) { + t.Error("model-value should be the tiebreak for identical ids") + } +} + +// fillModelsWith substitutes by family and leaves the result a valid policy. +func TestFillModelsWith(t *testing.T) { + preset, err := router.Preset("default") + if err != nil { + t.Fatal(err) + } + detected := map[string]string{ + "haiku": "claude-haiku-4-5-20251001", + "sonnet": "claude-sonnet-4-5-20250929", + "opus": "claude-opus-4-7", + } + out, notes := fillModelsWith(preset, detected) + if len(notes) != 3 { + t.Fatalf("expected 3 substitutions, got %d: %v", len(notes), notes) + } + for _, id := range detected { + if !strings.Contains(string(out), id) { + t.Errorf("filled policy missing detected id %q", id) + } + } + if _, _, err := router.Load(out); err != nil { + t.Fatalf("filled policy no longer loads: %v", err) + } + // No detection → unchanged, no notes. + same, n := fillModelsWith(preset, nil) + if len(n) != 0 || string(same) != string(preset) { + t.Errorf("empty detection should leave the preset unchanged") + } +} diff --git a/docs/ROUTER.md b/docs/ROUTER.md new file mode 100644 index 0000000..df39bd8 --- /dev/null +++ b/docs/ROUTER.md @@ -0,0 +1,362 @@ +# The whittle model router + +*One document: architecture, policy reference, and design rationale. If you're +looking for how to turn it on, see the [README](../README.md#model-routing-opt-in); +this is how and why it works.* + +--- + +## 1. What this is + +`whittle route` is a local proxy that sits on `ANTHROPIC_BASE_URL` and rewrites +each Claude Code request to the **cheapest model tier that can still handle it**, +per a policy you author in one JSON file. Hard reasoning stays on your strongest +model; trivial edits drop to the cheapest; the broad middle rides a capable +default. It is opt-in, fails open on every path, and never touches your +credentials or conversation history — only the `model` field and the features the +target model can't serve. + +**Non-goals.** It is not an API gateway (no auth, no quotas, no multi-tenant +anything), not a prompt rewriter (history is never mutated, so prompt-cache +prefixes survive), and not a hosted service (everything runs on your machine; the +one log line per request never contains prompt text). + +## 2. Design principles + +- **Fail open, always.** A router that breaks your agent is worse than no router. + Every failure — bad policy, unparseable request, rejected rewrite, dead + upstream, dead classifier — degrades to "your original request, untouched" or + the safe middle tier. The worst case is *not routed*, never *broken*. +- **One policy file.** All routing behavior lives in `~/.whittle/router.json` — + no DSL, no database, no learned state at runtime. You can read the file and + predict every decision; the per-request log tells you which rule fired and why. +- **Compute where the data lives.** ML models stay in the Python sidecar + (whittle's Go binary has zero external dependencies); the sidecar returns raw + scores and distributions; the **policy owns every threshold**. Swapping a model + never changes the decision logic, and the decision logic never hides in a model. +- **No evidence → no rewrite.** When no rule matches — or a signal can't tell + what a request is — whittle leaves the request exactly as the client sent it + (`default: "requested"`), and an uncertain classification never *fires* a rule + (§5.1 shows how that falls out of the math). A request is only ever rewritten + because a rule you wrote affirmatively matched. Authors who prefer aggressive + savings can set a tier (e.g. `"main"`) as the default and accept the trade. + +## 3. Request lifecycle + +``` +Claude Code ──ANTHROPIC_BASE_URL──▶ whittle route (127.0.0.1:45873) + │ + ┌──────────────────┼──────────────────────────────┐ + │ 1 EXTRACT body → signals (tokens, tool-loop, │ + │ requested model, recent user text) │ + │ 2 DECIDE pin header → route waterfall → │ + │ default (ML leaves lazy, §5) │ + │ 3 RECONCILE rewrite model + strip features the │ + │ target can't serve (§6) │ + │ 4 FORWARD stream response back, per-chunk │ + │ flush; classify failures (§7) │ + └──────────────────┬──────────────────────────────┘ + ▼ + api.anthropic.com (your credentials, + forwarded untouched) + ML signals, on demand: sidecar :45872 /v1/route/{domain,embedding,complexity} +``` + +Walkthrough of one request: + +1. **Extract.** The body is parsed once into `Signals`: estimated context tokens + (whole body ÷ 4), message count, tool-loop shape, the requested model + (canonicalized — dated snapshots match bare ids), and the user text (tool + results and the system prompt are never signal input). Two text scopes are + kept deliberately: **keywords scan the `inspect` window** (a hard keyword two + turns back keeps protecting), while **ML signals classify only the latest + user turn** — averaging turns dilutes a classifier (measured live: a turn + scoring complexity +0.17 alone fell to +0.03 joined with two trivial turns, + silently suppressing mid-session escalation). +2. **Decide.** A pin header (if configured) wins outright. Otherwise the routes + run top-down, **first match wins**; if nothing matches, the static `default` + tier applies. Inside a route's condition tree, cheap heuristic leaves evaluate + before ML leaves and short-circuit — a keyword match never pays for a + classifier call. Each ML signal is computed at most once per request. +3. **No-op check.** If the chosen tier's model *is* the requested model, the + request is forwarded **byte-for-byte** — no rewrite, no reconciliation, no + risk. Guards also keep the original when the target is entitlement-blocked or + its context window can't fit the request. +4. **Reconcile** (§6) and **forward**, streaming the response with per-chunk + flush (SSE reaches the client immediately; `Accept-Encoding: identity` + upstream avoids the gzip-SSE framing hang). +5. **One structured log line** (§8) — verdict, models, token usage. Never prompt + text. + +## 4. The policy file + +Strict JSON (unknown keys are load errors, never silently ignored), validated +with path-precise messages. A missing or invalid policy puts the router in +transparent passthrough — it never bricks Claude Code. `SIGHUP` hot-reloads; a +bad edit keeps the running policy. + +```json +{ + "version": 1, + "tiers": [ + {"name": "fast", "model": "claude-haiku-4-5-20251001"}, + {"name": "main", "model": "claude-sonnet-4-5-20250929"}, + {"name": "smart", "model": "claude-opus-4-8"} + ], + "default": "requested", + "inspect": {"scope": "recent_turns", "turns": 3}, + "signals": { + "domains": [ + {"name": "quantitative", "categories": ["math", "physics", "chemistry"], "min_mass": 0.7}, + {"name": "casual", "categories": ["other"], "min_mass": 0.9} + ], + "complexity": [ + {"name": "reasoning", "threshold": 0.15, + "hard": ["analyze the root cause of this failure", "..."], + "easy": ["fix this typo", "..."]} + ] + }, + "routes": [ + {"name": "escalate", "when": {"any": [ + {"complexity": "reasoning:hard"}, + {"domain": "quantitative"}]}, "to": "smart"}, + {"name": "casual-easy", "when": {"all": [ + {"domain": "casual"}, {"complexity": "reasoning:easy"}]}, "to": "fast"}, + {"name": "casual-medium", "when": {"all": [ + {"domain": "casual"}, {"complexity": "reasoning:medium"}]}, "to": "main"} + ], + "session": {"sticky": false}, + "overrides": {"pin_header": "x-whittle-route"} +} +``` + +- **`tiers`** — ordered cheap → capable; the order defines band rank. Use the + full dated model ids your account accepts (`whittle policy init` auto-detects + them from your Claude Code config; bare ids often 404). +- **`default`** — where traffic goes when no route claims it: a tier name, or + the reserved **`"requested"`** — keep the model the client asked for, as a + guaranteed no-op. `"requested"` is the shipped posture: with zero evidence + about a request, whittle does not rewrite it, so **every model change (up or + down) traces to a rule you wrote**. It also protects mixed-model clients — + Claude Code's cheap-model background requests are never up-routed by a fixed + default. Set a tier (e.g. `"main"`) instead when you want aggressive savings: + unmatched traffic then down-routes to that tier. +- **`routes`** — the ordered waterfall. Each `when` is a recursive boolean tree: + combinators `all` / `any` / `not` (one per node), leaves one-per-node: + + | leaf | fires when | + |---|---| + | `context_tokens`, `message_count` | numeric band: scalar (`= n`) or `{gt,gte,lt,lte}` | + | `tool_loop`, `has_tools` | request shape booleans | + | `keywords` | literal, case-insensitive, **whole-word/phrase** (§5.4) | + | `keywords_regex` | explicit RE2, opt-in | + | `requested_model` | membership, canonicalized both sides | + | `domain` / `embedding` / `complexity` | named ML signals (§5) | + + `to` is a tier name, or `keep` (hold the session's current tier). +- **`signals`** — named ML tests, defined once, referenced by any number of + leaves, computed at most once per request. +- **`overrides.pin_header`** — a per-request escape hatch: send + `x-whittle-route: smart` and routing is bypassed. + +Author loop: `whittle policy init [name]` writes a built-in preset with your +detected model ids; `whittle policy validate ` runs the real loader and +prints errors and cost-lint warnings. + +## 5. Signals + +Two small models power three signals. Both run in the sidecar; the Go engine +holds only thresholds. If the sidecar is down or smart mode is off, ML leaves +evaluate false and the keyword/context heuristics still route (a route whose +match *depended* on an errored signal never fires — so a `not` over an +unavailable signal can't invert fail-open; the log gains an `ml-degraded` tag). + +### 5.1 `domain` — subject classification with mass thresholding + +A 14-category classifier (MMLU-Pro taxonomy: math, physics, law, health, +computer science, other, …) returns its **full softmax distribution**, and the +leaf fires iff the **total probability mass on the signal's categories** clears +`min_mass`: + +``` +fire ⟺ Σ p(c) for c in categories ≥ min_mass +``` + +One scalar does the work that otherwise takes an uncertainty ladder: + +- *Confident in-set* (`p(math)=0.95`) → passes. +- *Split across in-set* (`p(math)=0.4, p(physics)=0.4`) → 0.8 passes, with no + top-2 special case — mass is invariant to which in-set category won. +- *Ambiguous* (flat distribution) → mass ≈ |set|/14 → fails → the request rides + the default tier. **Uncertainty lands in the middle, never on the expensive + tier** — the cost-first safety default is a property of the arithmetic, not a + rule. + +Without `min_mass`, the leaf falls back to argmax membership (also the graceful +path when the sidecar predates distributions). + +### 5.2 `complexity` — contrastive difficulty + +The policy provides two banks of exemplar phrasings. The signal embeds the +request text and both banks, scores each bank as +`0.75·best-cosine + 0.25·mean(top-2)`, and takes the margin: + +``` +margin = score(hard bank) − score(easy bank) +margin > t → "hard" margin < −t → "easy" else "medium" +``` + +A leaf names a level: `"reasoning:hard"`. Medium fires neither direction — the +broad middle rides the default. Empirically, `domain` and `complexity` have +complementary failure modes (a theorem proof reads "medium" to the exemplars but +classifies math at 0.98; a misclassified equation still scores complexity-hard), +which is why the shipped default ORs them for escalation. + +### 5.3 `embedding` — similarity to your own examples + +Bank score of the request text against a candidate phrase list, thresholded. +Useful for one specific, high-value shape (e.g. routing architecture/design asks +to the strong tier). Note the embedding space has a high similarity floor +(unrelated sentences score ~0.35–0.4), so thresholds live in a narrow band — +probe before trusting a new candidate list. + +### 5.4 `keywords` — whole-word literals + +Case-insensitive whole-word/phrase matching: an occurrence embedded in a larger +alphanumeric run does not match (`migration` never fires on *immigration*, +`refactor` never on *refactored* — list the variants you want). Boundaries are +non-alphanumeric runes or string edges, so `c++` works. Keywords are the +smart-off fallback and the zero-latency fast path; the ML signals carry the +nuance. + +## 6. Capability reconciliation + +Down-routing is only safe if the cheaper model can accept the request. Requests +from a stronger model routinely carry features a cheaper one rejects with a 400: +long-context betas, extended-thinking config **and** its dependent beta tokens +**and** `context_management` edits that require thinking, effort parameters, +mid-conversation `system` messages, thinking blocks in history. + +The reconciler is a **blocklist**: forward everything, strip only what the +target is known to reject. Unknown model *families* are assumed fully capable +(forward untouched, let the retry safety net catch a real 400); unknown +*versions* of a known family get the family's conservative floor — over-stripping +on a down-route is harmless, under-stripping is a guaranteed 400. Stripping +whole messages can break user/assistant alternation, so a repair pass coalesces +adjacent same-role messages afterward. Every strip is named in the log +(`stripped:context-1m+thinking+…`). + +## 7. Failure model + +Three failure modes, deliberately distinct: + +| mode | trigger | behavior | +|---|---|---| +| **A** | *our* error (unparseable body, reconcile bug) | forward the **original** request untouched | +| **B** | upstream rejects our **rewrite** (400/403) | retry the **original** once, relay that; a genuine 403 `permission_error` blocks the tier (TTL-bounded); the log names the rejected model and upstream error | +| **C** | transport failure | synthetic 502 — never a hang | + +Plus the cheap safety paths: **no-op** (target = requested → byte passthrough), +oversized bodies stream through unbuffered and untouched, non-`/v1/messages` +traffic passes through blind, and `GET /health` answers locally. The commit +point is held until the upstream status is known, so a Mode-B retry never +double-writes the client. + +The design bias: Mode B means a reconciliation gap costs one extra round-trip, +not a user-visible failure — and the log line makes the gap loud instead of +silent. + +## 8. Observability + +Exactly one JSON line per request, never prompt text: + +```json +{"tier":"main", "requested":"claude-opus-4-8", "model":"claude-sonnet-4-5-20250929", + "reason":"default stripped:context-1m+thinking", "status":200, "latency_ms":2060, + "ctx_tokens":25640, "in_tokens":1974, "out_tokens":16, "session":"c2f9659c"} +``` + +`requested` vs `model` is what routing changed; `in_tokens`/`out_tokens` are the +response's real usage — enough to compute per-request savings offline: +`cost(requested, in, out) − cost(model, in, out)`. `reason` names the exact rule +that fired, every stripped feature, and any retry (`mode-b:…(rewrote→X got 400 +invalid_request_error: …)`), so misroutes and capability gaps are one-line +diagnoses. The same verdict rides response headers (`x-whittle-route`, +`x-whittle-reason`). + +## 9. Codemap + +All router code lives in `router/` (Go, stdlib only). The ML lives in `model/` +(Python sidecar, shared with the compression hook). + +| file | role | +|---|---| +| `router/policy.go` | policy types, signal catalog, canonicalization, load | +| `router/rule.go` | the recursive condition-tree grammar | +| `router/validate.go` | structural + referential validation, cost lints | +| `router/signals.go` | request body → `Signals` extraction | +| `router/engine.go` | `Decide`: waterfall, cheap-first eval, memoized ML leaves, keyword matching | +| `router/caps.go` | model capability table + family floors | +| `router/reconcile.go` | feature strips + alternation repair | +| `router/proxy.go` | HTTP handler: lifecycle, modes A/B/C, streaming, log line | +| `router/server.go` | daemon entrypoint, hot-reload, smart-mode wiring | +| `router/ml/client.go` | HTTP client to the sidecar (fail-open) | +| `router/policies/` | the calibrated `default` policy + its companion doc | +| `model/route.py` | the two models + exact scoring math (pure functions, stub-tested) | +| `cmd/whittle/route.go`, `policy.go` | CLI: `route`, `policy init/validate/…` | + +Invariants worth knowing before changing anything: the router package has **no +external Go dependencies**; handlers never see credentials as data (headers are +forwarded, not read); prompt text never reaches a log; every ML call is +fail-open; validation rejects what it doesn't understand rather than ignoring it. + +## 10. Prior art & credits + +The ML layer stands on [vLLM Semantic Router](https://github.com/vllm-project/semantic-router) +(Apache-2.0; their design is described in the whitepaper +[*Signal-driven decision routing for mixture-of-modality models*](https://vllm-semantic-router.com/white-paper)), +and we want to be precise about the debt: + +- **Models**: we use their two trained models directly — + `llm-semantic-router/mmbert32k-intent-classifier-merged` (the MMLU-Pro domain + classifier) and `llm-semantic-router/mmbert-embed-32k-2d-matryoshka` (the 32k + text embedder). +- **Math**: the bank-score blend (`0.75·best + 0.25·mean(top-2)`) and the + contrastive hard/easy margin are their mechanisms, replicated exactly. +- **Ideas**: the signal taxonomy (domain / embedding-similarity / complexity as + composable routing inputs, per their + [whitepaper](https://vllm-semantic-router.com/white-paper)) and the insight + that per-category routing should be grounded in measured accuracy-vs-cost + (their reasoning benchmark) shaped this design. + +The architecture around those pieces is our own and intentionally much smaller: +a single-file boolean policy instead of a category→model-score projection +system, probability-mass thresholding instead of an entropy decision ladder +(§5.1 — including inverting their uncertain→escalate default to +uncertain→middle, which is the right direction for a cost-first router), +capability reconciliation and the A/B/C fail-open model for rewriting real +Anthropic traffic, and per-request cost observability. Where we diverge, it's +deliberate; where we borrowed, it's named here. + +## 11. Honest limitations & roadmap + +- **Phrasing is not the task.** Every prompt-side signal judges the *wording*. + "Improve this" over a 10-line file and over a distributed system read + identically. Artifact-aware signals (actual context/diff size — already + visible to the proxy) are the designed next step. +- **Category is not difficulty.** A confidently-classified trivial question in + an escalating domain ("what is the boiling point of water" → chemistry 0.94) + over-escalates. Accepted for now: bounded by `min_mass`, rare in real traffic, + and the same trade vSR's measured config makes for math. +- **The parameters are hand-authored.** The mechanisms are trained models; the + thresholds, prototype banks, and category sets were chosen by probing, not + calibrated against labeled data. The roadmap is a **tier-sufficiency dataset**: + label real traffic with the *minimal sufficient tier* (replay through tiers, + judge blindly), then tune the category map, `min_mass`, complexity threshold, + and prototype banks against measured precision/recall — optimizing dollars + saved subject to a quality floor. Notably, no public benchmark measures + *coding* difficulty (vSR's reasoning bench is QA/math/science only), so the + coding half of that dataset has to come from real sessions. +- **Session stickiness is minimal.** With three tiers, band-jump damping is + nearly inert; the shipped presets disable it. A "hold the strong tier for N + turns after a hard trigger" latch is the likely replacement. diff --git a/docs/ROUTER_DESIGN.md b/docs/ROUTER_DESIGN.md deleted file mode 100644 index 062bb24..0000000 --- a/docs/ROUTER_DESIGN.md +++ /dev/null @@ -1,494 +0,0 @@ -# Whittle Router — design (DRAFT for review) - -Status: **proposal, no code yet.** This is the design to shred before we build. -Scope: a local model-router for Claude Code, folded into (or shipped beside) -whittle. Companion to the compressor: compress intercepts tool **outputs**; -route intercepts LLM **requests**. - ---- - -## 0. The conflict we must resolve first (read this before the rest) - -`PLAN.md` publishes these **non-goals**: *"No proxy mode · no model hosting · -no telemetry."* The router violates the first two: - -- **Proxy mode.** Today whittle is a *hook* — Claude Code calls -`POST /hook` (PostToolUse) and whittle is never in the network path. Routing -LLM calls **requires** being on `ANTHROPIC_BASE_URL` as a real proxy. There -is no hook that can redirect a model call; the interception point is the HTTP -request itself. -- **Model hosting.** The opt-in `smart` mode hosts two ONNX models -(~200-300MB). The compressor deliberately keeps models server-side -(the Python sidecar), not in the daemon. - -This is not just packaging/branding — it is a **trust escalation** and a -**resource-exclusivity** change (review C1/O2): - -- **Credential + prompt exposure.** The hook only ever saw tool *outputs* -post-hoc. A proxy on `ANTHROPIC_BASE_URL` sees **every credential** (the -user's OAuth token / API key) and **every token of every prompt**. A bug or -a compromised dependency now leaks secrets. The compressor's threat model -never included this. -- `**ANTHROPIC_BASE_URL` is a single exclusive slot** and it is **global** — it -captures *all* the user's Anthropic traffic, not just Claude Code, and it -cannot coexist with any other tool that also wants that slot (other routers, -observability proxies). whittle-as-hook had no such conflict. - -So this is not a feature addition — it changes whittle's deployment model from -"invisible hook" to "network proxy." Three ways forward, **founder's call** -(evaluate each against the trust + exclusivity facts above, not just branding): - -1. **Separate binary / mode** (`whittle route`), clearly partitioned from the - published hook surface; non-goals amended to "no proxy mode *for - compression*." -2. **Sibling project** entirely (a `whittle-router` repo) that shares the - compress library. -3. **Revise the non-goals** and make whittle a two-headed daemon (hook + - proxy). - -The rest of this doc assumes we build the router; it does **not** assume where -it ships. Everything below is written so the routing core is a library -(`route/`, `adapter/`) that any of the three packagings can wrap. - ---- - -## 1. High-level design - -``` -Claude Code ──(Anthropic Messages, streaming)──▶ whittle proxy (localhost) - │ - ┌──────────────────────────┼───────────────────────────┐ - │ 1. EXTRACT (cheap, always) │ - │ request body ─▶ Signals{tokens, tool_loop, │ - │ requested_model, last_user_text, session_key, …} │ - │ heuristic only — NO model call here │ - ├───────────────────────────────────────────────────────┤ - │ 2. DECIDE (cheap → expensive, short-circuits) │ - │ a. pin-header override ─▶ done │ - │ b. guards waterfall (heuristic) ─▶ maybe done │ - │ c. [smart] lazy ML only if unresolved: │ - │ intent classifier · text embedding (signals) │ - │ d. session stickiness gate │ - │ e. default tier │ - │ ⇒ Decision{tier, model, reason} │ - ├───────────────────────────────────────────────────────┤ - │ 3. ROUTE (adapter: factory + strategy) │ - │ rewrite body.model ─▶ forward to upstream │ - │ stream response back (v1: byte passthrough) │ - ├───────────────────────────────────────────────────────┤ - │ 4. OBSERVE │ - │ x-whittle-route header + one structured log line │ - └───────────────────────────────────────────────────────┘ - │ client's own auth, untouched - ▼ - api.anthropic.com -``` - -**The load-bearing property: cheap-first, lazy-ML.** Heuristic guards run before -any model call, so "huge context → smart" or "background → fast" never pays for -an embedding. The ML models are consulted only on the residual traffic no guard -resolved. A naive "extract-all-signals → score → decide" pipeline would run the -models on every request and defeat the entire point. **This is why "score -evaluator" is not a separate top-level stage** — scoring/ML is a sub-step *inside* -Decide, reached only when cheap rules don't settle it. - -**Package layout (mirrors `compress/`):** - - -| package | mirrors | responsibility | -| ------------------------------ | ----------------------- | ---------------------------------------------------------------- | -| `route/` | `compress/` | the brain: extract → decide. Library, no HTTP. | -| `route/` `signals.go` | `compress/router.go` | heuristic signal extraction (precompiled regex, hot-path) | -| `route/` `ml.go` | (new) | opt-in ONNX classifier + embedding, behind a `Classifier` iface | -| `route/` `policy.go` | `compress/types.go` | the fixed YAML struct + validation | -| `route/` `engine.go` | `compress/pipeline.go` | ties extract+decide; holds no mutable state except session store | -| `route/` `session.go` | (new) | stickiness state, TTL/LRU evicted | -| `adapter/` | `compress/compressors/` | provider strategies (factory), Anthropic passthrough in v1 | -| `proxy/` (or extend `server/`) | `server/hook.go` | the HTTP proxy; "we are a real proxy now" lives here | - - ---- - -## 2. Low-level design, per component - -### 2.1 Signal Extractor — `route/signals.go` - -**Responsibility:** parse the request body **once**, emit heuristic `Signals`. -No ML, no allocation of the full body where avoidable (bodies are 20k+ tokens -of system prompt + tool defs; work on offsets/slices). - -```go -type Signals struct { - RequestedModel string // body.model, canonicalized (see §2.6) - ContextTokens int // ESTIMATE (bytes/4); banding only, not billing - LastUserText string // per policy.Inspect.Scope - RecentText string // recent-turns window - ToolLoop bool // PRECISE: last message has role:user AND contains ≥1 tool_result block - MessageCount int // count of top-level messages[] entries - HasTools bool - SessionID string // = X-Claude-Code-Session-Id header (§2.5); no derivation - - // ML fields are NOT filled here — Decide fills them lazily. - Intent string - IntentConf float64 -} - -func Extract(body []byte, hdr http.Header, pol *Policy) (Signals, error) -``` - -- Parse strategy: bounded `gjson`-style peek for `model`, `messages[]` roles, -`tools` presence — never a full `json.Unmarshal` of the whole body on the hot -path (matches compress's `extractContentFast` discipline). -- **`ContextTokens` counts the WHOLE request body** (system + tools + messages) - ÷ 4 — because that is what actually drives cost and cache size. **Consequence - (R5): the floor is ~20k on a trivial first turn** (CC always sends a big - system prompt + tool defs), so author thresholds against the *whole-request* - scale — a `{ lt: 4000 }` band can never match; use realistic numbers - (e.g. `gt: 60000`). Every example threshold in the schema is calibrated to - this definition. -- **`ToolLoop` is exactly:** the last `messages[]` entry has `role: "user"` and - its content contains at least one `tool_result` block. NOT "any assistant turn - ever emitted a tool_use" (that is true from turn ~2 of every agent session and - would pin every session via `to: keep` — R4). -- **`LastUserText` / `RecentText` extraction (R6):** flatten only **`text` - blocks of `role:"user"` messages** within the `inspect` window; **exclude - `tool_result`, `tool_use`, `thinking`, and `image` blocks**. When the last - message is a `tool_result` (an agent step, not human intent), walk back to the - most recent genuine user-text turn for classification. These are the ONLY text - the ML sees — feeding it raw tool output re-creates the noise problem this - field exists to avoid. -- **`RequestedModel` canonicalization (R7) is required, not deferred:** strip - date suffixes and resolve `-latest` on BOTH the incoming id and the `tiers` - model strings before any membership match or no-op comparison — else CC's - dated ids (`claude-opus-4-…-`) silently never match `requested_model` - routes and routing quietly disables itself. - -### 2.2 Policy — `route/policy.go` - -The fixed struct the YAML unmarshals into. **No DSL**: every condition is typed -data; the only external grammar is regex (Go's `regexp`, not ours). Validates -with JSON-Schema, not a compiler. - -```go -type Policy struct { - Version int - Tiers []Tier // ORDERED cheap→capable; index = band rank - Default string - Inspect InspectCfg - Signals *SignalSet // named ML signals referenced by route leaves - Routes []Route // ordered waterfall; When is a recursive bool tree - Session SessionCfg - Overrides OverrideCfg -} -// A route's When is a recursive Rule tree (all/any/not + one leaf per node). Leaves -// are the heuristic ones (context_tokens, message_count, tool_loop, has_tools, -// keywords, keywords_regex, requested_model) PLUS three ML SIGNAL leaves that name -// an entry in Signals: -// Domain string // domain signal: classifier label ∈ its categories -// Embedding string // embedding signal: bank score ≥ its threshold -// Complexity string // "name:hard|easy|medium" -type SignalSet struct { // vSR-aligned; exact JSON in ROUTER_POLICY_SCHEMA.md - Domains []DomainSignal // {name, categories} — intent classifier - Embeddings []EmbeddingSignal // {name, threshold, candidates} — embedding model - Complexity []ComplexitySignal // {name, threshold, hard, easy} — embedding (contrastive) -} -``` - -Loaded behind `atomic.Pointer[Policy]` for SIGHUP hot-reload: parse → validate → -atomic swap. Invalid reload keeps the old policy and logs. Hot path reads the -pointer once per request. (Signal embeddings are NOT precomputed at load — the -sidecar caches candidate embeddings by content hash on first use; §2.4.) - -### 2.3 Decider — `route/engine.go` - -**Responsibility:** `Signals + Policy → Decision`. Owns the `Classifier`. Pure -except for the session store. - -```go -type Decision struct { - Tier, Model, Reason string // Reason: "route:hard-work" | "default" | "… ml-degraded" -} -func Decide(s Signals, p *Policy, cl Classifier, sess SessionStore, pin string) Decision -``` - -Ordered algorithm: - -1. **pin override** (`Overrides.PinHeader`) → done. -2. **routes, top-down.** First route whose `When` tree holds wins. A signal leaf - (domain/embedding/complexity) triggers a **lazy** classifier call, memoized per - request; cheap heuristic leaves in the same node evaluate first, so a matched - sibling skips the model entirely. A route fires only on a **definitive** match: - if a signal it consulted errored (sidecar down), the route is **skipped** - (fail-open) — so a `not` over an unavailable signal never fires, and the reason - is tagged `ml-degraded`. - ⚠️ **Cost lint (§4.4):** a route with a signal leaf runs a model on every - request reaching it; validation warns. Put cheap routes above it. -3. **stickiness** (§2.5): damp a fuzzy (default) downgrade from the session's - current tier when band-jump `< MinBandJump`. -4. **default** — the static fallback. There is **no separate "classify" stage**: - the ML lives inside route conditions as signal leaves, matching vSR (the - default is just the last rung). If weighted scoring is ever added it is another - signal leaf that thresholds inside a route, never a competing stage. - -**Failure mode = fail-open to the ORIGINAL request (corrected, review B3).** -Any error (bad signals, ML panic, timeout, body too large to parse, malformed -JSON) → **forward the client's request untouched, honoring its `requested_model`** -— never block, and never silently substitute `Default`. This matches the -compressor's convention (`pipeline.go` / `hook.go` fail-open to the *original*, -not to a chosen alternative). `Default` is reserved for the ONE case where -routing *succeeds* but no route matched — it is a routing outcome, -not an error path. The distinction matters: "route to Default on error" would -silently downgrade every request to a weaker tier on any bug, and is literally -impossible on the paths where we can't read the body at all. - -### 2.4 Classifier — `router/ml` (HTTP client to the sidecar) - -```go -type Classifier interface { - Domain(text string) (label string, conf float64, err error) // intent classifier - EmbeddingScore(text string, candidates []string) (score float64, err error) // bank score - ComplexityMargin(text string, hard, easy []string) (margin float64, err error) // contrastive -} -``` - -- Impls: `noop` (smart off → `ErrMLDisabled`) and the `router/ml` HTTP client to - the Python sidecar. **The models live in the sidecar** (vSR's two mmBERT models — - a ModernBERT intent classifier + a 32k text embedder); the Go engine carries no - model runtime and owns the **thresholds** (score ≥ T; margin band → level). -- **vSR's exact scoring, sidecar-side.** Bank score = `0.75·best + 0.25·mean(top-2)` - over per-candidate cosine (candidates are the policy's `candidates`/`hard`/`easy` - lists). `EmbeddingScore` returns the bank score; `ComplexityMargin` returns - `score(hard) − score(easy)`. Compute where the data lives. -- **Cache is sidecar-side, keyed by (candidate text hash + model version).** The - engine passes candidates on every call; the sidecar embeds them once and reuses. - No Go-side centroid/matrix/emb-cache — the design is simpler than the earlier - few-shot plan, which precomputed per-tier example matrices at load. The model - version in the key guards against silent v1-vs-v2 embedding garbage. -- **Per-list candidate cap** (schema validation): soft ~32, hard 256 — quality + - cold-start cost, not runtime. -- **Fail-open, two levels.** A classifier error makes the signal leaf evaluate - false AND marks the route non-firing (§2.3), so an unavailable signal never - routes anywhere — routing falls to `default`, never blocks, never panics. A - request-level parse/transport error is still Mode A at the proxy (forward the - ORIGINAL untouched, §2.3); the engine itself only decides. - -### 2.5 Session store — `route/session.go` (DE-RISKED by GATE-2 experiment) - -Stickiness needs "what tier is this session on." **The GATE-2 experiment -(2026-07-09) refuted the premise that we must derive a key.** Claude Code sends -a real, stable session id header: - -``` -X-Claude-Code-Session-Id: 911c4d7e-… (UUID) -``` - -Observed byte-stable across all requests of a session (including the internal -title-generation sub-request) and distinct between sessions. **Use it directly** -— the entire "derive a key by hashing the system prefix" design is deleted. The -experiment also proved that derivation would have *failed*: two distinct sessions -in the same cwd had **byte-identical** system blocks (no wall-clock/git-status in -the block), so a content hash would collide; and the positionally-last -`cache_control` breakpoint lives in the *messages* (moves every turn), so -"hash to the last breakpoint" would churn per turn. The header sidesteps both. - -```go -type SessionStore struct { /* sharded map keyed by session-id + TTL + LRU cap */ } -func (s *SessionStore) Current(id string) (tier string, ok bool) -func (s *SessionStore) Set(id, tier string) -``` - -- Lifecycle: TTL eviction (~30 min idle) + max-entries LRU. -- **Still to verify** (not testable headlessly): does the header stay stable - across `--resume` and `/compact`, and do **parallel subagents inherit the - parent's id or get their own**? The thrash question now hinges entirely on - that last behavior — if subagents share the parent id, concurrent subagent - requests contend on one store entry. Test before shipping `keep`/stickiness. - -### 2.6 Adapter — `adapter/` - -```go -type Adapter interface { - Name() string - // Reconcile sets the target model AND strips target-incompatible features - // across BODY and HEADERS together (atomic — see ROUTER_RECONCILIATION.md). - // Called ONLY on a genuine route; Decide detects the no-op (resolved == - // requested model) and byte-passthroughs WITHOUT calling Reconcile (R11/M5). - // Operates on a COPY — the caller retains the pristine original for Mode B - // retry (B2). `stripped` lists features removed, for the log line. - Reconcile(req *Request, target string) (stripped []string, err error) - Upstream(hdr http.Header) (baseURL string, out http.Header) - // v2 only: TranslateRequest / TranslateResponse for cross-protocol -} -func For(dest Destination) Adapter // factory -``` - -- **v1: `anthropicPassthrough`.** Auth headers pass through untouched (GATE-0 -confirmed OAuth/subscription works through a transparent proxy — NOT API-key- -only); **response is a byte stream passthrough** — GATE-1 confirmed Claude Code -does **not** validate the echoed `message_start.message.model`, so no SSE -parsing/rewrite is needed. This is the entire reason v1 is small, and it is now -empirically validated, not assumed. -- **⚠️ Rewrite is capability RECONCILIATION, not just `body.model`** — the real -work of v1, now fully specified in **`ROUTER_RECONCILIATION.md`**. Summary: a -blocklist-by-capability (forward everything, strip only features the target is -known to reject — never allowlist), strip-only/down-direction, body+headers -atomic, with Mode B (§3) as the self-healing safety net for unknown new betas. -Reconciliation and Decide share the capability table so oversized contexts are -never down-routed below a target's window (routing guard, not a body edit). -- **v2: cross-provider** (Claude Code → GPT/local). Do **not** reinvent SSE -translation — vllm-sr is Apache-2.0; port its `sse_out.go` (~650 LOC, three -production bugs already paid for). - -### 2.7 Proxy — `proxy/` (or `server/`) - -- Hand-rolled over `net/http` (not `httputil.ReverseProxy` — we must read the -body for signals before forwarding). Request body is **buffered** (bounded -read, e.g. 32MB like the hook); response is **streamed** (`io.Copy` + flush, -or `Flusher`). -- Content-length recomputed after model rewrite. -- **⚠️ SSE framing is a real hazard (GATE-1 reproduced a client hang).** - Anthropic returns **gzip-encoded `text/event-stream`**. Forwarding gzip - verbatim while re-chunking hung Claude Code. Fix that worked: send - `Accept-Encoding: identity` upstream, then stream the body with immediate - per-write flush. Get this exact or the client times out. -- **Unknown paths pass through untouched** — Claude Code also hits -`/v1/messages/count_tokens` and others; the proxy must not assume every -request is a routable `/v1/messages`. -- **Relay upstream status verbatim (review O1).** Unlike the hook's -unconditional `200` (a hook must never fail a tool call), a proxy MUST pass -upstream 4xx/5xx (401/403/429/400/529) through unchanged — collapsing them -breaks Claude Code's error handling and retry logic. "Never 500 *from our own -bug*" is a different rule from "always 200." -- **Retries/fallback:** v1 leaves retries to Claude Code (it already retries); -the proxy surfaces upstream errors verbatim. A fallback-tier chain is a -deliberate v2 decision (§4.7) to avoid double-retry storms. -- Never 500 on a routing bug: any internal error → **forward the original -request untouched** (B3), never a substituted model. - ---- - -## 3. Cross-cutting - -- **Config hot-reload:** SIGHUP → parse → validate → atomic swap. Invalid → keep -old + log. (No precompute step — the sidecar caches candidate embeddings lazily.) -- **Observability (the 90% that matters):** `x-whittle-route: ` + -`x-whittle-reason` response headers, and one structured JSON log line per -request. The log line carries **token counts, tier, reason, matched-rule -name, latency — NEVER prompt text** (review C1: the compressor's stats log -deliberately never persists content; the router must hold the same line, or -it becomes user-prompt-content-on-disk). This still answers "why did my -request go to Haiku" via the reason + matched-rule. No remote telemetry. -- **"Fail-open" is THREE distinct behaviors — never conflate them (the root - cause of the R1 fail-loop).** Name and separate: - - **Mode A — process-error fallthrough.** *Our* internal error (body parse - fails, ML classifier panics/errors, invalid signals). → **Forward the - ORIGINAL request untouched** (original model, unmodified body/headers). Safe - because nothing has been written to the client yet. Never substitute - `default`. (This is what B3, §2.3, and the §2.4 classifier path all mean — - §2.4 corrected below.) - - **Mode B — upstream 4xx/5xx on a request we forwarded.** (R1 **resolved**.) - If we did **not** rewrite (no-op, R11) → **relay verbatim**: it's the - client's own request, retrying would loop. If we **did** rewrite and get a - **400/403** → **retry the buffered ORIGINAL body+model once**, then relay if - it still fails (safe: 400/403 arrive as a full body before any SSE flush, so - nothing is on the wire). A **403 entitlement** failure additionally marks - that tier **blocked for the session** so Decide stops routing there. - Genuine client/quota statuses (429, 529) → **relay verbatim** regardless. - Full mechanism in `ROUTER_RECONCILIATION.md` ("Interaction with Mode B"). - - **Mode C — transport failure** (upstream unreachable/reset). Forwarding the - original can't help (the upstream is down). → **return a synthetic - Anthropic-shaped 502 + log**; do NOT call this "fail-open." -- **Fail-open never 500s from our own bug** (Mode A). The proxy is in the - critical path of every Claude Code call. - ---- - -## 4. Build gate — RESULTS (experiment 2026-07-09) + pre-code must-fixes - -The verification experiment (transparent proxy + real headless `claude -p`, -`experiment/proxy.py`) returned **architecture GO**: - -- **GATE-0 — OAuth/subscription through a proxy: PASS.** Not API-key-only; the - client's own auth forwards untouched. (Folded into §2.6.) -- **GATE-1 — model echo: PASS.** CC does not validate the echoed model → - byte-passthrough viable, no SSE rewrite. BUT surfaced the capability- - reconciliation work (§2.6) and the gzip-SSE hazard (§2.7). -- **GATE-2 — session key: PASS, better than designed.** `X-Claude-Code-Session-Id` - header replaces the risky content hash (§2.5). - -Still unverified (not testable headlessly), gate before shipping stickiness: -session-id behavior across `--resume`/`/compact`/**subagents**; interactive -`/cost` display; and the net-cost model on real multi-session traffic (per-model -cold-cache loss). - -**Two must-fixes before code (holistic review, 2026-07-09):** -1. **Tier ordering.** `min_band_jump` stickiness assumes tiers have rank, but - `Tiers` is an unordered `map` → the parameter is currently undefined. Make - tiers an ordered list / add explicit rank. (Schema §2.) -2. **~~`classify.strategy` discriminator~~ — SUPERSEDED (v5).** The `classify` - block was removed entirely; ML now lives in route conditions as named signal - leaves (domain/embedding/complexity), matching vSR. A future weighted scorer is - just another signal leaf, additive by construction. (Schema §7.) - -## 5. Ranked findings from review (fixes folded in above) - - -| # | sev | issue | status | -| ----- | ------- | --------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ | -| B1 | blocker | model echo | → GATE-1 | -| B2 | blocker | session key + thrash economics | → GATE-2; key now hashes stable prefix (§2.5) | -| B3 | blocker | fail-open was "route to Default" — WRONG | **fixed**: forward original untouched (§2.3, §2.7) | -| C1 | high | proxy sees creds; log leaked prompt text | **fixed**: §0 trust note; log excludes content (§3) | -| C2 | high | routing to a model the user's plan can't access → 400/403 + CC-retry fail-loop | **open** — needs entitlement handling; B3 at least breaks the loop | -| C3 | med | cheap-first under-specified: intra-`MatchCfg` field order, two separate ML models, warn-scope too narrow | **open** — spec exact eval order; cost-warn ANY intent guard | -| C4 | med | centroids must live *inside* the swapped Policy snapshot; `Current→Set` not atomic per key | **moot** — no Go-side centroids/matrix (sidecar caches embeddings); session store locks per op | -| C5 | med | user regexes = ReDoS on hot path; is `Any` recursive? | **open** — bound/timeout regexes; declare `Any` single-level | -| C6 | med | `count_tokens` passes through on original model, but request runs on routed model → tokenizer mismatch in CC's context math | **open** — tied to GATE-1 | -| O1 | obs | relay upstream status, not always-200 | **fixed** (§2.7) | -| O2 | obs | base-URL is one exclusive global slot | **fixed** (§0) | -| O3 | obs | ML latency is on synchronous TTFT path | **open** — set a latency budget | -| O4/O5 | obs | request gzip/chunked encoding; model-id canonicalization | **open** | - - -**Guard vs. stickiness precedence** and **token-estimate accuracy** remain open -design choices (not blockers): proposal is guards win over stickiness, and -bytes/4 is acceptable for banding. Revisit if GATE-2 shows band-edge misroutes -matter. - -## 6. Execution-readiness audit (2026-07-09) — resolved defaults + open work - -Signal-extraction precision (R4/R5/R6/R7) folded into §2.1; three-mode fail-open -(R1/R8/R23) into §3/§2.4. Remaining items: - -**Resolved as stated defaults (build to these):** -| id | decision | -|----|----------| -| R9 | Streaming: connect + idle timeout only; **no total-response deadline** on the streamed body (LLM gens run minutes). Upstream drop mid-stream → propagate close. | -| R10 | Smart off / models absent: a signal leaf (domain/embedding/complexity) evaluates **false** and its route is skipped → routing falls to `default`. Smart-off is the expected heuristics-only mode (NOT tagged); a real sidecar failure is surfaced per-request as an `ml-degraded` reason. | -| R11 | **No-op short-circuit:** resolved model == canonicalized requested model → byte-passthrough, NO rewrite, NO reconciliation. Protects the common (happy-path) case. | -| R12 | Each signal owns its own threshold on its own scale: `domain` is pure argmax membership (no floor), `embedding` a cosine-ish bank-score threshold, `complexity` a symmetric margin band — set per-signal in the policy, never a single global knob. | -| R13 | Nearest-example tie → **cheaper tier wins** (lowest band rank), deterministic. | -| R14 | `X-Claude-Code-Session-Id` absent → **skip stickiness** (stateless decide); never bucket session-less requests under `""`. | -| R15 | `min_band_jump` damps **downgrades only**; upgrades are free. Stored "current" = tier actually used. | -| R16 | Pin header naming unknown tier → **ignore + log**, fall through to routes; pin is **override-only** (does not write session current). | -| R17 | `Sticky:false` forced switch **does** update session current. | -| R18 | Body over 32MB cap → **stream-passthrough unbuffered** (can't route; forward untouched), not 413. | -| R19 | Config file deleted at runtime → keep last-good policy + log (same as invalid reload). | -| R20 | emb-cache version component = **model file content hash**, not a manual version string. | -| R21 | Models → download-once to `~/.whittle/route/models/`, pinned version + SHA256 verify; corrupt/failed load → smart disabled + warn-and-degrade, never crash. | -| R22 | `x-whittle-route`/`-reason` headers + one JSON log line on **all** paths with distinct reasons (`route:`, `default`, `fail-open:parse`, `passthrough:unroutable-path`, `mode-b:retried-original`, `… ml-degraded`). No prompt text. | -| R24 | `count_tokens` tokenizer skew vs routed model: **accepted for v1** (documented, not fixed). | - -**Resolved this pass:** -1. **[R1] Rewrite-caused 400/403 → retry-original-once, then relay** (§3 Mode B); - 403 entitlement marks the tier session-blocked. Distinguishes our-fault from - client-fault by "did we rewrite." -2. **[R2] Capability reconciliation** — fully specified in - **`ROUTER_RECONCILIATION.md`** (capability blocklist, strip-only, atomic - body+headers, context-length routing guard, Mode B self-healing, maintenance). - Three items still marked *validate-on-real-traffic* in that doc (mid-conv - `role:"system"` transform, thinking-history residue, full per-model cap probe). -3. **[R3] Cold-start with no/invalid config → start in transparent - passthrough + loud warning** (NOT refuse-to-start, which would brick all of - Claude Code given the exclusive global base-URL, §0). Runtime config delete - → keep last-good (R19). \ No newline at end of file diff --git a/docs/ROUTER_IMPLEMENTATION_PLAN.md b/docs/ROUTER_IMPLEMENTATION_PLAN.md deleted file mode 100644 index 88fd700..0000000 --- a/docs/ROUTER_IMPLEMENTATION_PLAN.md +++ /dev/null @@ -1,86 +0,0 @@ -# Whittle Router — implementation plan - -Execution breakdown for the design in `ROUTER_DESIGN.md`, `ROUTER_POLICY_SCHEMA.md`, -`ROUTER_RECONCILIATION.md`. Principles, in priority order: **simplicity, -correctness, extensibility, performance.** Each milestone ends with a -code-reviewer pass (opus) and a testing-engineer gap pass (opus). - -## Ground rules (founder, 2026-07-09) -- **§0 RESOLVED:** whittle is a PostToolUse hook **AND** an opt-in router. README - will be updated to drop the "no proxy" non-goal. The proxy milestone is no - longer gated. -- **All router code lives in ONE folder: `router/`. No leakage outside it.** A - single Go package `router` (files below), mirroring how `compress/` is one - folder. Only heavy/optional deps (ONNX) get quarantined in a `router/ml/` - subpackage (the `compress/compressors` pattern). Nothing is added at repo root. -- **Zero-dependency, still:** the policy loader uses stdlib `encoding/json` with - strict unknown-key rejection (isolated to one file); YAML front-end is a - drop-in later if/when we accept the dep. - -## Sequencing rationale -Build inside-out: the **pure decision core** first (no I/O, no ML, no network — -fully testable), then adapter/reconciliation, then proxy/daemon, then opt-in ML. -Matches the reviewer's "routes + static default is the true MVP" and lets each -layer be verified before the next depends on it. All files are `router/.go`. - ---- - -## Milestone 1 — Decision core (`route/`, pure, no deps beyond yaml loader) - -### T1.1 — Policy types + load + validation (`route/policy.go`, `route/validate.go`) -Types: `Policy, Route, Rule (recursive), NumBand, ClassifyCfg, SessionCfg, -Tier (ordered), InspectCfg`. Load from YAML; **validation is the product** here. -- **AC1** Parses every scenario in POLICY_SCHEMA §5.1–5.7 into the expected tree. -- **AC2** Rejects every §5.8 invalid case with a *specific* error naming the node. -- **AC3** Unknown keys rejected at every node (strict); typo'd leaf → error, not silent drop. -- **AC4** `NumBand` accepts scalar (`message_count: 1`⇒Eq) and mapping; rejects empty band / `gt≥lt` / `eq`+other. -- **AC5** One-shape-per-node enforced recursively (leaf XOR all XOR any XOR not); empty group rejected; single-child group warns. -- **AC6** Tiers are ordered (band rank defined); `keep` reserved; referential integrity (routes/default/classify keys → tiers). -- **AC7** `classify.strategy` discriminator present; only `few_shot` valid in v1; per-tier example cap (warn 32 / reject 256). -- **AC8** Loader isolates the YAML dep to one file (swap-to-JSON is one function). - -### T1.2 — Signal extraction (`route/signals.go`) -Parse the Anthropic request body once → `Signals`. -- **AC1** `RequestedModel` canonicalized (strip date suffix, resolve `-latest`). -- **AC2** `ContextTokens` = whole-body bytes/4 (documented scale). -- **AC3** `ToolLoop` = last message role:user AND has ≥1 tool_result block — exact predicate, not "any tool_use ever". -- **AC4** `LastUserText`/`RecentText`: only user-role `text` blocks in the `inspect` window; excludes tool_result/tool_use/thinking/image; walks back past a trailing tool_result turn. -- **AC5** `SessionID` from `X-Claude-Code-Session-Id`; absent → empty (caller skips stickiness). -- **AC6** Bounded parse, no full-body allocation on the hot path; never panics on malformed JSON (returns err → Mode A upstream). -- **AC7** Verified against the real captured request shapes in `exp_sem_routing/experiment/captures_*`. - -### T1.3 — Condition eval + decision engine (`route/engine.go`, `route/decide.go`) -The precedence ladder + boolean tree evaluator + stickiness. -- **AC1** Boolean eval: all/any/not correct; short-circuit; cheap-first child ordering (ML leaves last). -- **AC2** Precedence: pin → routes(first-match) → classify(smart default) → static default. -- **AC3** Classifier behind an interface; `noop` impl → ML leaves eval false, classify → default (smart-off path), surfaced in reason. -- **AC4** Stickiness: downgrade-only damping by band rank; `min_band_jump`; `keep`; account-level entitlement blocklist consulted. -- **AC5** Decision carries `{tier, model, reason, stripped?}`; reason distinguishes every path (`pin`, `route:`, `classify:@`, `sticky:kept`, `default`, `skipped:no-ml`). -- **AC6** No-op detection (resolved == canonicalized requested) → signals byte-passthrough (no reconcile). -- **AC7** Fail-open discipline: any internal error → caller forwards original (Mode A); engine never panics. - -**M1 exit:** `go build ./route/...` clean; core self-tests pass; code-reviewer(opus) + testing-engineer(opus) run. - ---- - -## Milestone 2 — Adapter + capability reconciliation (`adapter/`) -Per `ROUTER_RECONCILIATION.md`. Capability table, strip transforms, atomic -body+headers, context-length guard sharing the caps table, unknown-model -fully-capable sentinel. -- AC: down-route request → target-incompatible features stripped, result is valid Anthropic JSON; up-route/no-op untouched; unknown model → forward-all; each strip transform unit-tested incl. degenerate cases (empty beta header, emptied assistant msg, adjacent-role merge). - -## Milestone 3 — Proxy/daemon (§0-GATED) -HTTP proxy, streaming passthrough (gzip→identity, flush), three fail-open modes, -Mode B retry-original-once + commit-point invariant, session store, hot-reload, -observability. **Needs the §0 packaging decision before it ships.** - -## Milestone 4 — Smart mode (opt-in ONNX) -Intent classifier + embedding, precompute/cache with model-version key, -few-shot nearest-example, install-time provisioning. - ---- - -## Review cadence -End of each milestone: **code-reviewer (opus)** for correctness/integration, -**testing-engineer (opus)** to find coverage gaps. Fold findings before the next -milestone depends on the code. diff --git a/docs/ROUTER_POLICY_SCHEMA.md b/docs/ROUTER_POLICY_SCHEMA.md deleted file mode 100644 index 0f7c62a..0000000 --- a/docs/ROUTER_POLICY_SCHEMA.md +++ /dev/null @@ -1,409 +0,0 @@ -# Whittle Router — policy schema (v4, vSR-aligned) - -Status: **implemented.** The routing policy: `signals` (named ML signals) + -`routes` (explicit boolean rules referencing them) + condition grammar. v4 -(2026-07-09) replaces the invented `classify` "smart default" tier-picker with -vLLM Semantic Router's signal model — the ML lives *inside* route conditions as -boolean leaves, not a separate fallback stage (founder call: "use what vSR uses, -don't invent"). v3 renamed `guards`→`routes`; v2 folded the code-review. The -"Revision log" at the end maps findings to resolutions. Extends `ROUTER_DESIGN.md` -§2.2/§2.3. - ---- - -## 0. Decision precedence — how `routes` and `signals` relate - -There is **one ordered resolution**; each rung is the fallback for the one -above. This is the answer to "which wins": - -``` -1. pin header (overrides.pin_header) explicit user override → always wins -2. routes (ordered list) your explicit rules → FIRST MATCH wins -3. default (static tier) the STATIC fallback → when NO route matched - (session stickiness then damps a fuzzy default downgrade; see §8) -``` - -**There is no separate "classify" stage.** `signals` are not a routing rung — -they are *inputs* the routes evaluate. A `domain`/`embedding`/`complexity` leaf -inside a route's `when` is a boolean test computed from an ML model; the route -fires (or not) like any other condition. So the ML intelligence lives *in the -waterfall*, composable with heuristic leaves (`context_tokens`, `tool_loop`, -`keywords`), not bolted on as a competing fallback. This mirrors vSR exactly: its -`decisions` (our routes) reference typed signals; the "default" is just the last -rule. (Earlier drafts had a `classify` block that embedded per-tier examples and -argmax-picked a tier — that conflated topic with difficulty and could not compose -with other conditions; it was removed. See the Revision log.) - -**Weighted scoring, if ever added, is a signal too** — a scored leaf that -thresholds inside a route (`when score(X) ≥ t → to Y`). A route stays boolean; a -boolean can only *threshold* a score, never route to the highest-scoring tier. -Nothing is reserved on `Route`. - ---- - -## 1. The shape decision - -A condition is a **recursive boolean tree**. Each node is EITHER a **leaf** -(one predicate on one signal) or a **combinator** (`all` / `any` / `not`). -Operator-as-key (the combinator is a YAML map key), not positional: - -```yaml -any: - - all: - - context_tokens: { gt: 60000 } - - tool_loop: true - - keywords: [architecture, migrate] -``` - -**v2 decision — no implicit AND.** A leaf node holds **exactly one predicate**; -all conjunction is written explicitly with `all`. v1 allowed several leaf keys -in one node to mean AND, which created two ways to write AND and — combined with -"a one-element `any` is legal" — let a single mis-indentation silently flip an -OR into an AND with no error (review B2/H5). Explicit-only makes a two-key leaf -node an immediate validation error, so that trap becomes loud instead of silent. -More verbose, uniform, teachable. - -**Naming (v3):** the rule list is `routes:` (was `guards:` — "guard" reads as -blocking, wrong for a router), and each route's target field is `to:` (was -`route:`, which collided with the list name). Reads as plain English: *"when -context_tokens > 60k, route `to: smart`."* The **route waterfall** is ordered, -first-match-wins (like nginx location blocks); each route's `when` is a condition -tree. - -```go -type Route struct { - Name string `yaml:"name"` - When Rule `yaml:"when"` // the condition tree (§2) - To string `yaml:"to"` // tier name | "keep" (reserved keyword) -} -``` -(A per-route `sticky` override is **deferred to the stickiness milestone** — in -v1 explicit routes always win and are never damped, so it would be a -load-accepted field that does nothing. A stray `sticky:` on a route is rejected, -not silently ignored.) - ---- - -## 2. Go representation - -```go -// Rule is one node. EXACTLY ONE of: a single leaf predicate, All, Any, or Not. -type Rule struct { - All []Rule `yaml:"all,omitempty"` - Any []Rule `yaml:"any,omitempty"` - Not *Rule `yaml:"not,omitempty"` - - // ---- leaf predicates: a valid leaf sets EXACTLY ONE of these ---- - ContextTokens *NumBand `yaml:"context_tokens,omitempty"` - MessageCount *NumBand `yaml:"message_count,omitempty"` - ToolLoop *bool `yaml:"tool_loop,omitempty"` - HasTools *bool `yaml:"has_tools,omitempty"` - Keywords []string `yaml:"keywords,omitempty"` // LITERAL substring, case-insensitive - KeywordsRegex []string `yaml:"keywords_regex,omitempty"` // explicit regex, when you mean it - RequestedModel []string `yaml:"requested_model,omitempty"` // membership (canonicalized both sides) - - // ---- ML signal leaves: each NAMES a signal defined in Policy.Signals (§7), - // evaluated lazily via the classifier, memoized once per request ---- - Domain string `yaml:"domain,omitempty"` // domain signal fires: classifier label ∈ its categories - Embedding string `yaml:"embedding,omitempty"` // embedding signal fires: bank score ≥ its threshold - Complexity string `yaml:"complexity,omitempty"` // complexity level "name:hard|easy|medium" -} - -// NumBand: a numeric predicate. Custom UnmarshalYAML accepts EITHER a bare -// scalar (message_count: 1 ⇒ Eq=1) OR a mapping (context_tokens: {gte: 60000}). -// This is the ONLY custom unmarshaler; Rule itself stays plain-struct. -type NumBand struct{ Eq, Gt, Gte, Lt, Lte *int } -``` - -- **Three ML signal leaves, two models (vSR-aligned).** `domain` uses the intent - classifier (one MMLU-Pro label); `embedding` and `complexity` use the text - embedding model. A leaf holds the signal's *name* (defined in §7), not inline - data, so one signal is authored once and referenced by many routes and computed - at most once per request. `complexity` additionally carries the level after a - colon (`needs_reasoning:hard`). Replaces the old single `intent:[labels]` leaf. -- **`keywords` is literal + case-insensitive by default** (review H6): a coder - typing `["c++"]` or `["a.b"]` must not hit a regex-metachar explosion or a - silent over-match. Regex is opt-in via `keywords_regex`. -- **No `Score` field.** v1 reserved a `score:` leaf; removed (review H4). - Weighted scoring arrives as a `WeightedScorer` *classify strategy* (a decision - mode that argmax-routes by score), per `ROUTER_DESIGN.md` §2.3 — NOT as a - boolean route leaf, because a boolean tree can only threshold a score, never - route by highest. Reserving nothing beats reserving the wrong seam (§7). - ---- - -## 3. Evaluation semantics - -- **Boolean only** — a route's condition tree returns match / no-match; no confidence - propagates up the tree (confidence lives in the classify/score layer). Avoids - vllm-sr's confidence-averaging complexity. -- **`not` is unary.** Empty `all`/`any` rejected. -- **Cost is data-dependent — NOT a guarantee (corrected, review H1/H2).** - Predicates are pure, so the evaluator reorders each node's children - cheap-heuristics-first and short-circuits (`any` on first true, `all` on first - false). This *reduces* classifier calls but does **not** guarantee zero: an - `all` whose cheap child is usually-true, or an `any` whose cheap child is - usually-false, still descends into an ML child. And reordering is **intra-tree - only** — the route *list* is author-ordered and first-match-wins, so a bare - `intent` route at the top of the waterfall runs the classifier on every - request. The honest rule: cheap-first is a best-effort cost optimization; the - cost lint (§4) is what actually protects you. - ---- - -## 4. Validation (pure structural checks, applied RECURSIVELY at every node) - -1. **Strict keys.** Loader uses `yaml.Decoder.KnownFields(true)` / - `additionalProperties:false`. **Any unknown key is rejected** (review B1) — - this is what catches a typo'd leaf (`keywrods:`) or signal that would - otherwise be silently dropped and misroute. -2. **One shape per node.** Exactly one of: one leaf predicate, `all`, `any`, - `not`. Zero keys → error. Two leaf keys → error (this is what makes the - mis-indent B2 loud). Combinator + leaf → error. Two combinators → error. - Checked at **every** node, including `all`/`any` elements and `not`'s child. -3. `all: []` / `any: []` → error. Single-element `all`/`any` → **warn** (usually - a mis-indent, §5). -4. **NumBand sanity** (review M1): ≥1 bound set; `eq` exclusive of other bounds; - `gt < lt`, `gte ≤ lte`; reject impossible/empty bands. -5. **Regex safety** (review C5): `keywords_regex` entries compile and are - length/complexity-bounded (ReDoS guard) — they run per-request on user text. - `keywords` (literal) are auto-quoted, never a ReDoS vector. -6. **Referential integrity**: every `route` tier and `default` exists in - `tiers`; every `domain`/`embedding`/`complexity` leaf names a signal defined - in `signals`; a `complexity` leaf's level is `hard|easy|medium`. `keep` is a - **reserved keyword** — rejected as a tier name (review M5). Signal names are - unique per kind. -7. **Depth cap = 6, hard** (review H3/L2): also serves as the recursion bound - the parent's C5 asked for — this recursive tree **supersedes** C5's - "single-level `Any`." -8. **Cost lint** (restores parent C3): warn on ANY route whose `when` references - an ML signal leaf (`domain`/`embedding`/`complexity`) — it runs a model on - every request that reaches it; place cheap routes above it. -9. Helpful errors for the two highest-frequency author mistakes: scalar where a - list is expected (`keywords: architecture` → "wrap in `[ ]`"), and scalar - where a NumBand mapping is expected. -10. **Signal candidate-list cap**: each `candidates`/`hard`/`easy` list rejects - > **256** (hard) and warns > **32** (soft). Candidates are embedded once and - cached (sidecar, by content hash); the cap bounds cold-start cost. Domain - `categories` warn if not in the MMLU-Pro set; complexity `threshold` must be - ≥ 0 (a symmetric margin band). Duplicate candidates within a list → warn. - ---- - -## 5. Scenarios - -### 5.1 Trivial — single leaf -```yaml -routes: - - name: huge-context - when: { context_tokens: { gt: 60000 } } - to: smart -``` - -### 5.2 AND (explicit — the only way in v2) -```yaml - - name: big-coding-task - when: - all: - - intent: [coding] - - context_tokens: { gt: 30000 } - to: smart -``` - -### 5.3 OR -```yaml - - name: obviously-hard - when: - any: - - keywords: ["race condition", deadlock, "root cause"] - - context_tokens: { gt: 80000 } - to: smart -``` - -### 5.4 Nested (the OR[ leaf, AND[ leaf, OR[ leaf, leaf ]]] case) -```yaml - - name: nested - when: - any: - - requested_model: [claude-opus-4-8] - - all: - - tool_loop: false - - any: - - keywords: [architecture, migrate] - - context_tokens: { gt: 100000 } - to: smart -``` - -### 5.5 NOT + literal keywords (note `keywords` is literal here — safe for `def `, backticks) -```yaml - - name: cheap-unless-code - when: - all: - - context_tokens: { lt: 4000 } - - not: { keywords: ["```", "def ", "func ", "class "] } - to: fast -``` - -### 5.6 Realistic full policy -```yaml -version: 1 -tiers: # ORDERED cheap→capable; the order - - { name: fast, model: claude-haiku-4-5 } # IS the band rank, so min_band_jump - - { name: main, model: claude-sonnet-5 } # has defined meaning (was an unordered - - { name: smart, model: claude-opus-4-8 } # map → min_band_jump was undefined) -default: main -inspect: { scope: recent_turns, turns: 3 } # keyword/signal text scans THIS window - -signals: # named ML signals, referenced by routes - domains: # intent classifier → MMLU-Pro label ∈ set - - { name: coding, categories: [computer science, engineering] } - - { name: math, categories: [math] } - embeddings: # bank score ≥ threshold → fires - - name: architecture_design - threshold: 0.66 - candidates: - - design a scalable architecture for this system - - plan a migration strategy for a distributed system - - compare tradeoffs between system designs and recommend one - complexity: # contrastive margin → hard|easy|medium - - name: needs_reasoning - threshold: 0.15 - hard: ["debug this race condition", "analyze the root cause of this failure", "solve this step by step"] - easy: ["rename this variable", "fix this typo", "answer briefly"] - -routes: - - name: pinned-opus - when: { requested_model: [claude-opus-4-8] } - to: smart - - name: background - when: { requested_model: [claude-haiku-4-5] } - to: fast - - name: mid-tool-loop - when: { tool_loop: true } - to: keep - - name: hard-work # ML leaves reached last, cost-linted - when: - any: - - keywords: [migrate, "race condition", "root cause", architecture, refactor] - - context_tokens: { gt: 60000 } - - complexity: needs_reasoning:hard # contrastive difficulty signal - - embedding: architecture_design # semantic similarity signal - to: smart - - name: coding # domain (subject) signal - when: { any: [ { domain: coding }, { domain: math } ] } - to: main - -session: { sticky: true, min_band_jump: 2 } -overrides: { pin_header: x-whittle-route } -``` - -### 5.7 First-turn equality via scalar shorthand -```yaml - - name: first-turn - when: { message_count: 1 } # scalar ⇒ Eq=1 - to: main -``` - -### 5.8 Invalid — validation must reject (expanded) -```yaml -when: { any: [ {tool_loop: true} ], context_tokens: {gt: 10} } # combinator + leaf -when: { all: [...], any: [...] } # two combinators -when: { any: [] } # empty group -when: {} # empty node -when: { not: [ {tool_loop: true}, {has_tools: true} ] } # not is unary -when: { keywords: [a], context_tokens: {gt: 5} } # two leaf keys (no implicit AND) -when: { keywrods: [a] } # unknown key (typo) — B1 catch -when: { context_tokens: {} } # empty NumBand -when: { context_tokens: {gt: 100, lt: 50} } # impossible range -when: { any: [ {}, {tool_loop: true} ] } # empty child (recursive check) -``` - ---- - -## 6. Route-list OR vs in-route `any` — choose deliberately (review M7) - -They are NOT interchangeable style: -- **Separate routes** produce distinct `matched-rule` names in the log (§3 obs), - can each carry a different `Sticky`, and are **priority-ordered** - (first-match-wins). -- **`any` branches** collapse to one route name, share one stickiness, and are - unordered (reorderable for cost). - -Use separate routes when you want different destinations, labels, or stickiness -per condition; use `any` when one destination is chosen by a compound OR. - ---- - -## 7. The `signals` catalog (vSR-aligned) - -Signals are named ML tests defined once and referenced by route leaves (§2). -Two models back them (hosted in the Python sidecar; the Go engine owns the -thresholds — *compute where the data lives*): - -```yaml -signals: - domains: [{ name, categories: [] }] # intent classifier - embeddings: [{ name, threshold, candidates: [] }] # embedding model - complexity: [{ name, threshold, hard: [], easy: [] }] # embedding model -``` - -- **`domain`** — the intent classifier (ModernBERT, 14 MMLU-Pro categories: - biology, business, chemistry, computer science, economics, engineering, health, - history, law, math, other, philosophy, physics, psychology) predicts one label; - the leaf fires when that label ∈ `categories`. Unknown categories warn (a - swapped model could differ), never error. -- **`embedding`** — bank score of the query against `candidates`, fires when - `score ≥ threshold`. The bank score is vSR's exact blend: - `0.75·best + 0.25·mean(top-2)` over per-candidate cosine similarities. Cosine - lands ~0.3–0.5 for related phrasing, so a `threshold` near 0.4–0.66 is typical. -- **`complexity`** — contrastive difficulty. `margin = bank_score(hard) − - bank_score(easy)`; the leaf `name:hard` fires when `margin > threshold`, - `name:easy` when `margin < −threshold`, `name:medium` otherwise. This isolates - *difficulty* from *topic* (unlike a plain nearest-example scheme), which is the - whole point of tier routing. - -Each signal is computed **at most once per request**, lazily (only if a route -actually reaches its leaf — cheap heuristic siblings evaluate first), and the -sidecar caches candidate embeddings across requests by content hash + model -version, so re-embedding the static `candidates`/`hard`/`easy` is amortized. - -**Future — weighted scoring** stays a signal, not a route mode: a scored leaf -that thresholds inside a route (`when score(X) ≥ t`). Routes remain boolean; a -boolean can only threshold a score, never route by *highest*. No field is -reserved on `Route`. - ---- - -## 8. Revision log (v1 review → v2 resolution) - -| finding | v2 | -|---|---| -| B1 unknown-key silent drop; `domain` not in struct | §4.1 `KnownFields(true)`; removed `domain`, `intent` is the one classifier signal | -| B2/H5 mis-indent OR→AND via implicit-AND | §1 dropped implicit AND; §4.2 two-leaf node = error; §4.3 warn single-child group | -| H1/H2 cheap-first overclaimed | §3 rewritten: data-dependent, intra-tree; §4.8 restores C3 top-of-waterfall warn | -| H3 recursive tree vs C5 single-level `Any` | §4.7 explicit supersession; depth cap = recursion bound | -| H4 Score seam bakes wrong assumptions | §2 removed `Score`; §7 scoring is a classify strategy | -| H6 keywords = regex, `c++` breaks | §2 `keywords` literal+case-insensitive; `keywords_regex` opt-in | -| H7 no equality/scalar shorthand | §2 NumBand `Eq` + scalar-⇒-Eq unmarshal; §5.7 | -| M1 NumBand nonsense | §4.4 sanity checks | -| M2 validation recursion unstated | §4 "applied recursively at every node" | -| M3 keyword window/case unstated | §2 case-insensitive; §5.6 scans `inspect` window | -| M4 requested_model canonicalization | §2 canonicalized both sides (gated on parent O5) | -| M5 `keep` magic string | §4.6 reserved keyword; no-session → `default` (still gated on GATE-2) | -| M6 scalar-not-list hard error | §4.9 helpful hint | -| M7 route-list vs `any` "cosmetic" | §6 documents the real differences | -| L1 `not:[list]` cryptic error | §4.9 helpful hint | - -**Open for founder/next turn:** -- (a) **Flatten the tree?** (holistic review, judgment call) Real hand-authored - policies are shallow; IAM/Envoy/k8s/claude-code-router all use flat - implicit-AND + OR-via-multiple-rules, no deep nesting — and §6 already argues - separate routes often beat a nested `any`. Option: a route's `when` is a flat - implicit-AND list + a single non-nestable `any` for OR, dropping the recursive - tree (or keeping it only as a capped escape hatch). Simpler for the 95% case; - cost = reopens the v2 "no implicit AND" decision and can't express a deeply - nested route as one node. **Founder's call.** -- (b) **Does `classify` ship in v1 at all,** or do `routes` + static `default` - ship first (the true MVP), with `classify` sequenced behind GATE-1/2 passing? -- (c) `keep`'s dependence on the session store — ship only after the - subagent/`--resume`/`/compact` session-id behavior is verified. diff --git a/docs/ROUTER_RECONCILIATION.md b/docs/ROUTER_RECONCILIATION.md deleted file mode 100644 index f3a81c9..0000000 --- a/docs/ROUTER_RECONCILIATION.md +++ /dev/null @@ -1,243 +0,0 @@ -# Whittle Router — capability reconciliation (R2 spec, DRAFT) - -Status: **proposal, no code.** Closes the execution-audit's #2 gap ("the real -work of v1, unspecified"). Companion to `ROUTER_DESIGN.md` §2.6. - -## Why this exists - -Claude Code tailors each request to the **requested** model's capabilities. When -the router sends that request to a **different** model, the target rejects -features it doesn't support with **hard 400s**. Observed in the GATE-1 -experiment (down-routing Opus→Haiku): - -| observed 400 | where the feature lives | -|---|---| -| "long context beta not available…" | header `anthropic-beta: context-1m-…` | -| "does not support the effort parameter" | body `output_config.effort` + header `effort-…` beta | -| "adaptive thinking is not supported…" | body `thinking` config (+ `thinking` blocks in history) | -| "role 'system' is not supported…" | a `messages[]` entry with `role:"system"` (+ its beta) | - -Reconciliation is the layer that makes a rewritten request acceptable to its new -target. **It only runs when the model actually changes.** Decide detects the -no-op (resolved model == canonicalized requested model) and **skips Reconcile -entirely** — happy-path requests are byte-passthrough (R11); Reconcile is called -only on a genuine route, and when called it always changes at least the model -(review M5). - -**Scope (review M3): reconciliation handles hard-400 mismatches only.** It makes -a rewritten request *accepted* (no 400) — it does **not** make it *equivalent*. -A feature the target accepts but silently ignores or treats differently (a -valid-but-no-op param, a quietly-different default) causes **silent behavioral -drift** that no 400 and no Mode B can detect. Down-routing is not -behavior-preserving; that is out of scope and undetectable by this mechanism — -named here so no reader assumes otherwise. - -## Governing principle — forward everything, strip selectively - -Per the codebase's own rule ("forward all, block the few you must; never -allowlist what to keep — you'll miss something"), reconciliation is a -**blocklist by capability, not an allowlist**: - -- Forward the request **as-is** by default. Only strip a feature when the target - model is **known** to reject it. -- **Strip-only, down-direction.** A more-capable target is a superset — up-routing - forwards unchanged (we are never obligated to *add* features). Only - down-routing (or cross-routing to a narrower model) strips. -- **Mode B is the safety net** (Design §3): if reconciliation is incomplete and a - 400 still occurs, retry-original-once recovers the user. So an *unknown* new - beta is forwarded (works if supported; if not, Mode B catches it) — vastly - better than an allowlist that would strip every new Anthropic feature the day - it ships. - -Allowlist was rejected precisely because it breaks on every Anthropic release; -blocklist degrades gracefully and self-heals via Mode B. - -## Data model — capabilities, not pairwise model×feature - -```go -// Compiled-in baseline (NOT user config — the transform is code, tied to -// knowing how to edit each feature). Overridable via an escape-hatch config -// key for emergencies (a new model ships before we cut a release). -type ModelCaps struct { - Supports map[Capability]bool // on a KNOWN model, an absent capability ⇒ unsupported - MaxContextTokens int -} - -// capsFor MUST NOT return the zero value on a lookup miss (review B1). An -// unknown model id (Anthropic ships claude-opus-5; a user edits Tiers: to an id -// we have no entry for) is treated as FULLY CAPABLE + UNBOUNDED context — -// strip nothing, forward as-is, let Mode B catch any real 400. This is the -// governing principle ("forward everything") applied where it matters most; the -// zero struct would do the opposite (over-strip / MaxContextTokens==0 makes the -// tier unroutable — silent). Two distinct defaults, do not conflate: -// unknown CAPABILITY on a KNOWN model ⇒ unsupported (safe: strip) -// entirely UNKNOWN model ⇒ fully capable (safe: forward) -func capsFor(model string) ModelCaps // miss ⇒ {all-true, MaxContextTokens: ∞} -type Feature struct { - Needs Capability - Detect func(req *Request) bool // is the feature present? - Strip func(req *Request) // remove it from BOTH body and headers, atomically -} -``` - -Reconcile algorithm (per rewritten request, target model `m`): -``` -for f in features: - if f.Detect(req) and not caps[m].Supports[f.Needs]: - f.Strip(req) # atomic across body+headers -``` -Adding a **model** = declare its capability set. Adding a **feature** = declare -its capability + detector + strip transform. No N×M table. - -## The feature table (v1 baseline) - -| feature | capability | detect | strip transform (atomic) | -|---|---|---|---| -| 1M context | `long_context` | `anthropic-beta` contains `context-1m-*` | remove that token from the beta header. **Coupled to routing** — see below. | -| effort | `effort_param` | body `output_config.effort` present | delete the body field **and** remove the `effort-*` beta token — both or the 400 persists. | -| adaptive thinking | `thinking` | body `thinking` config present | delete `thinking` config; **also strip `thinking` blocks from prior assistant `messages[]`** (a non-thinking target rejects thinking content in history — the subtle second half). | -| mid-conversation system role | `midconv_system` | any `messages[i].role == "system"` | **see the hard case below** + remove its beta token. | - -## The hard case — mid-conversation `role:"system"` (needs validation) - -Dropping vs converting vs merging each changes prompt semantics differently: -- **Drop the message** — loses the instruction entirely; changes behavior. -- **Convert to `role:"user"`** — keeps the text, in place, in a role every model - accepts. Least-lossy; the instruction stays at its intended conversational - position. -- **Merge into top-level `system`** — reorders the instruction to "global," - losing its positional intent (it was meant to apply from that turn on). - -**Recommended: convert to `role:"user"`** (least semantically destructive, -avoids the role-not-supported 400), logged as a lossy transform. **But this can -itself 400 (review B3):** converting a mid-array `system` message to `user` -often produces **two adjacent `user` messages** (`[…, system, assistant]` → -`[…, user, assistant]` is fine, but `[user, system, …]` → `[user, user, …]`), -and the Messages API may reject non-alternating roles. So the transform is: -convert to `user`, and if that yields adjacent same-role messages, **merge the -text into the neighboring same-role message** rather than inserting a new turn. -**Validation must test the adjacent-role 400 specifically** (not just behavioral -quality) — if convert-or-merge still 400s, Mode B recovers the user but pays a -guaranteed 400 + double-latency *every* such turn, so this must be gotten right, -not left to the safety net. - -## Context-length coupling (a routing constraint, not just a strip) - -Stripping the `context-1m` beta is not enough if the live conversation actually -exceeds the target's normal window — the request then 400s on context length. -So: **do not down-route a request whose `ContextTokens` exceeds -`caps[target].MaxContextTokens × 0.9`.** This is a routing guard, enforced in -Decide (resolved tier rejected → next-capable tier chosen), not a body edit. - -- **Safety margin (review M1):** `ContextTokens` is `bytes/4` (§2.1), which - **under**-counts dense JSON/code — the dangerous direction (we think it fits, - it doesn't). The `× 0.9` margin absorbs the estimate bias; without it, - band-edge requests 400 on length → Mode B. -- **Terminal case — no tier fits (review H3):** if NO tier's window holds the - context, the fallback is **keep-original (no-op, R11), NEVER `Default`** — - routing a 240k context to a 200k Default just 400s every turn. The guard uses - the **pre-reconciliation** token count (conservative — reconciliation can only - *shrink* context by stripping thinking history, O2). -- **Table values lean conservative (review M4):** a wrong `MaxContextTokens` is - asymmetric — under-stating starves a tier (never chosen, safe-ish), - over-stating routes oversized contexts that 400 (per-turn Mode B). Pending the - systematic per-model probe (open #3), **under-state windows.** - -## Atomicity — the Adapter interface must own body + headers together - -The audit flagged that `Rewrite(body)` and `Upstream()→headers` being separate -invites "stripped the body field, forgot the paired header → still a 400." Fix: -one method reconciles the whole request. - -```go -type Adapter interface { - Name() string - // Reconcile mutates BODY and HEADERS together for the chosen target model, - // and returns whether anything changed (false ⇒ no-op short-circuit, R11). - Reconcile(req *Request, target string) (changed bool, err error) - Upstream(hdr http.Header) (baseURL string, out http.Header) -} -``` -`Request` carries both parsed body and headers so a `Feature.Strip` edits both -in one place. - -## Strip mechanics — the degenerate cases that must stay valid JSON (review M2) - -Each transform must leave a **valid Anthropic request**; the failure cases: -- **Empty `anthropic-beta` after removing the only token** → remove the header - entirely, don't send an empty value. Comma-list edits must not leave - leading/trailing/double commas. -- **Assistant message emptied by stripping `thinking` blocks** → if content - becomes `[]`, drop the whole message (a degenerate empty turn is itself - invalid). -- **Content-length is recomputed after ALL mutations**, not just the model - rewrite (§2.7's "after model rewrite" is too narrow — reconciliation edits - body fields too). - -**Body handling: parse → strip → re-serialize** (correctness over cache -preservation). This means the reserialized body loses ALL `cache_control` prefix -alignment on a **routed** request → a prompt-cache miss + cost regression on that -request. **Accepted for v1** because only routed requests pay it (the happy path -is byte-passthrough, untouched, R11) — and surgical byte-editing to preserve -cache positions risks emitting invalid JSON, a worse failure. This supersedes -the earlier "prefer position-preserving transforms" idea: under re-serialization -that is moot. Revisit only if measured cost regression on routed traffic is -material. - -## Maintenance & self-healing - -- **Static baseline** (the table above), compiled in, updated per release. -- **Mode B feedback loop:** an unrecognized feature is forwarded; if the target - 400s, Mode B retry-original recovers the user AND the log line records - `model + rejected-feature + error` — the signal to add a blocklist entry. -- **Honesty:** the static-baseline loop is self-*diagnosing*, human-*healing* — - it recovers one request and logs; a human must read the log and ship a new - baseline (days). Without the runtime learning below, the same unknown feature - 400s + retries **every turn** until then (≈2× TTFT per turn). -- **v1 — generic beta-token runtime-strip (review H4, pulled into v1).** Most new - Anthropic capabilities ship as `anthropic-beta` header tokens. When a - rewrite-caused 4xx names a beta token, remember "target `m` rejects beta `X`" - (account-scoped, alongside the entitlement blocklist) and strip token `X` - proactively thereafter. Low-complexity (betas are comma-list header tokens; - the generic strip already exists) and it kills the per-turn double-request for - the common case. **In v1.** -- **v1.1 — body-structural learning:** an unknown *body* param is genuinely hard - (we don't know how to strip a field we've never seen safely) → deferred. - -## Interaction with Mode B / entitlement (folds R1) - -- **Capture the pristine original BEFORE Reconcile runs (review B2).** Reconcile - mutates body+headers in place, so the retry source must be a snapshot taken - before reconciliation — the original body, model id, and headers — held until - the upstream **status code** is known. Peak memory ≈ 2×body per in-flight - routed request (≤32MB each, R18); fine for a single-user local proxy (O3). - Equivalently, Reconcile operates on a copy and the proxy keeps the original. -- **The commit-point invariant (review H1).** The proxy must NOT write the - status line or any body byte to the client until it has read the upstream - **HTTP** status and decided retry-vs-relay (hold the response head; do not - begin `io.Copy` early). Mode B fires **only on an upstream HTTP 4xx received - before that commit point.** A 200 stream that later carries an `event: error` - mid-flight (e.g. `overloaded_error`) is **relayed verbatim, never retried** — - bytes are already on the wire. "Retry on any error" would be unsound; retry is - keyed on pre-commit HTTP status only. -- On a pre-commit 4xx after a rewrite → **retry the snapshotted original once**, - then relay if it still fails. -- Distinguishing our-fault from client-fault: **only retry if we actually - rewrote**. A 4xx on a no-op (R11) request is the client's own → relay, never - retry (same request would loop). Accepted double-latency (O1): if we rewrote - but the original was itself malformed, we retry → it 400s → relay (correct, 2× - latency). -- **Entitlement failures are ACCOUNT-global, classified by `error.type`, not - status code (review H2).** Classify by the Anthropic error body's - `error.type` (`permission_error` = entitlement; `invalid_request_error` = - capability), not by 403-vs-400 (Anthropic is not 1:1). On an entitlement - error: retry original once, AND **record the tier as blocked at the - daemon/account level (persisted), not per-session** — a plan's tier access is - invariant across sessions, and a session-less request (R14) has nowhere to - store a per-session block, which would reopen the per-turn 403 loop (C2). - Decide consults this account-level blocklist and skips blocked tiers. - -## Open / validate before build -1. Mid-conversation `role:"system"` transform (convert-to-user) — validate on real traffic. -2. Whether stripping `thinking` blocks from history is sufficient, or the target also rejects other residue (tool-use signatures, etc.) — needs a down-route-with-history capture. -3. The full capability set of each current Anthropic model — the table above is seeded from 4 observed 400s; a systematic probe (send each feature to each tier, record 400s) would complete it before GA. diff --git a/embed.go b/embed.go index 1f76f55..b324362 100644 --- a/embed.go +++ b/embed.go @@ -6,5 +6,5 @@ import "embed" // `go install .../cmd/whittle@latest` is a complete installation: `whittle // setup` materializes these files to ~/.whittle/model and builds a venv there. // -//go:embed model/app.py model/gate.py model/fidelity.py model/requirements.txt +//go:embed model/app.py model/gate.py model/fidelity.py model/route.py model/requirements.txt var SidecarFS embed.FS diff --git a/embed_test.go b/embed_test.go new file mode 100644 index 0000000..84aa74f --- /dev/null +++ b/embed_test.go @@ -0,0 +1,23 @@ +package whittle + +import ( + "io/fs" + "testing" +) + +// Every Python module app.py imports must be embedded — a missing one breaks the +// whole sidecar on `whittle setup` (app.py fails at `import route`), taking +// compression down with it. This guards the //go:embed list against drift. +func TestSidecarFS_HasAllModules(t *testing.T) { + for _, f := range []string{ + "model/app.py", + "model/route.py", + "model/gate.py", + "model/fidelity.py", + "model/requirements.txt", + } { + if _, err := fs.Stat(SidecarFS, f); err != nil { + t.Errorf("SidecarFS missing %s (setup would install a broken sidecar): %v", f, err) + } + } +} diff --git a/model/app.py b/model/app.py index e1d4b2a..55e2665 100644 --- a/model/app.py +++ b/model/app.py @@ -11,7 +11,7 @@ import os import threading import time -from typing import List, Optional +from typing import Dict, List, Optional import tiktoken from fastapi import FastAPI, HTTPException @@ -206,6 +206,11 @@ class RouteDomainRequest(BaseModel): class RouteDomainResponse(BaseModel): label: str confidence: float + # Full softmax over the classifier's categories. The router thresholds + # probability MASS over policy-defined category sets — a scalar that subsumes + # entropy handling (a flat/ambiguous distribution simply fails the threshold + # and falls to the safe middle tier). argmax kept for logging/back-compat. + probs: Dict[str, float] = Field(default_factory=dict) class RouteEmbeddingRequest(BaseModel): @@ -229,8 +234,8 @@ class RouteComplexityResponse(BaseModel): @app.post("/v1/route/domain", response_model=RouteDomainResponse) def route_domain(req: RouteDomainRequest): - label, conf = _get_route_domain().classify(req.text) - return RouteDomainResponse(label=label, confidence=conf) + label, conf, probs = _get_route_domain().classify(req.text) + return RouteDomainResponse(label=label, confidence=conf, probs=probs) @app.post("/v1/route/embedding", response_model=RouteEmbeddingResponse) diff --git a/model/route.py b/model/route.py index cc68713..a611ed6 100644 --- a/model/route.py +++ b/model/route.py @@ -92,16 +92,27 @@ def complexity_margin(query_vec: Sequence[float], - bank_score(query_vec, easy_vecs, best_weight, top_m)) -def domain_label(logits: Sequence[float], labels: Sequence[str]) -> Tuple[str, float]: - """argmax label + its softmax probability. Pure (no torch) so the label-selection - logic is testable from a plain list of logits. Empty logits -> ("", 0.0).""" +def domain_distribution(logits: Sequence[float], labels: Sequence[str]) -> Dict[str, float]: + """Full softmax distribution as {label: prob}. Pure (no torch). The router + thresholds PROBABILITY MASS over a policy-defined category set — a single + scalar that subsumes entropy laddering: mass concentrates only when the + classifier is confidently in-set, and an ambiguous/flat distribution simply + fails the threshold (fail to the safe middle tier, cost-first). Empty -> {}.""" if not logits: - return "", 0.0 + return {} hi = max(logits) exps = [math.exp(x - hi) for x in logits] # shift for numerical stability total = sum(exps) - idx = max(range(len(logits)), key=logits.__getitem__) - return labels[idx], exps[idx] / total + return {labels[i]: exps[i] / total for i in range(len(logits))} + + +def domain_label(logits: Sequence[float], labels: Sequence[str]) -> Tuple[str, float]: + """argmax label + its softmax probability. Empty logits -> ("", 0.0).""" + probs = domain_distribution(logits, labels) + if not probs: + return "", 0.0 + label = max(probs, key=probs.get) + return label, probs[label] # ---- lazy real models (injected in tests) ----------------------------------- @@ -169,12 +180,16 @@ def _predict_real(self, text: str) -> Tuple[List[float], List[str]]: labels = [id2label.get(i, id2label.get(str(i), "")) for i in range(logits.shape[-1])] return logits.tolist(), labels - def classify(self, text: str) -> Tuple[str, float]: + def classify(self, text: str) -> Tuple[str, float, Dict[str, float]]: + """(argmax label, its prob, the FULL distribution). The distribution is the + payload — the router thresholds mass over category sets; argmax is kept for + logging/back-compat.""" if not text: - return "", 0.0 + return "", 0.0, {} predict = self._predict or self._predict_real logits, labels = predict(text) - return domain_label(logits, labels) + label, conf = domain_label(logits, labels) + return label, conf, domain_distribution(logits, labels) # ---- embedding-signal engine (embed + cache + score) ------------------------ diff --git a/model/test_route.py b/model/test_route.py index 8276460..8153a24 100644 --- a/model/test_route.py +++ b/model/test_route.py @@ -12,7 +12,7 @@ from pytest import approx -from route import (Scorer, bank_score, complexity_margin, domain_label, +from route import (Scorer, bank_score, complexity_margin, domain_distribution, domain_label, embedding_score, DomainClassifier) @@ -127,7 +127,7 @@ def test_domain_classifier_uses_injected_predict(): # Inject a predict fn so no transformers/torch is needed; classify wires argmax. predict = lambda text: ([0.0, 5.0], ["easy_domain", "hard_domain"]) clf = DomainClassifier(predict=predict) - label, conf = clf.classify("anything") + label, conf, probs = clf.classify("anything") assert label == "hard_domain" assert conf > 0.9 @@ -135,7 +135,7 @@ def test_domain_classifier_uses_injected_predict(): def test_domain_classifier_empty_text_short_circuits(): called = [] clf = DomainClassifier(predict=lambda t: called.append(t) or ([1.0], ["x"])) - assert clf.classify("") == ("", 0.0) + assert clf.classify("") == ("", 0.0, {}) assert called == [] # model is never consulted for empty text @@ -203,3 +203,24 @@ def test_complexity_one_sided_bank_still_scores(): # Only an easy bank present → hard bank scores 0.0 → margin is negative. margin = scorer.score_complexity("say hi", hard=[], easy=["say hi"]) assert margin < 0.0 + + +def test_domain_distribution_sums_to_one_and_matches_argmax(): + labels = ["biology", "law", "math"] + probs = domain_distribution([0.1, 3.0, 0.2], labels) + assert abs(sum(probs.values()) - 1.0) < 1e-9 + label, conf = domain_label([0.1, 3.0, 0.2], labels) + assert max(probs, key=probs.get) == label == "law" + assert probs["law"] == conf + + +def test_domain_distribution_empty(): + assert domain_distribution([], []) == {} + + +def test_classify_returns_full_distribution(): + predict = lambda text: ([0.0, 1.0, 2.0], ["a", "b", "c"]) + clf = DomainClassifier(predict=predict) + label, conf, probs = clf.classify("x") + assert label == "c" and set(probs) == {"a", "b", "c"} + assert abs(sum(probs.values()) - 1.0) < 1e-9 diff --git a/router/caps.go b/router/caps.go index a145782..e05dbca 100644 --- a/router/caps.go +++ b/router/caps.go @@ -1,9 +1,11 @@ package router +import "strings" + // Capability is a model feature that a request may use and a target model may or // may not support. Reconciliation strips a feature only when the target is // KNOWN to lack the capability (blocklist, not allowlist — see -// docs/ROUTER_RECONCILIATION.md). +// docs/ROUTER.md §6). type Capability string const ( @@ -56,22 +58,50 @@ var baselineCaps = map[string]ModelCaps{ } // fullyCapable is the unknown-model sentinel: every capability supported, window -// unbounded. +// unbounded. Reached only when a model matches no known FAMILY either. var fullyCapable = ModelCaps{allSupported: true, MaxContextTokens: unboundedContext} -// capsFor returns the capabilities of a model. The critical rule (review B1): a -// lookup MISS (an unknown model — Anthropic ships a new id, or a user pins one -// we don't know) returns the FULLY-CAPABLE, UNBOUNDED sentinel — strip nothing, -// forward as-is, let Mode B catch any real 400. The zero ModelCaps value would -// do the opposite (support nothing, MaxContextTokens 0 ⇒ unroutable), so we must -// never return it on a miss. Two distinct safe defaults, never conflated: +// familyCaps is the conservative per-FAMILY fallback for a model not in +// baselineCaps — Anthropic ships versioned ids faster than we enumerate, and the +// common case is DOWN-routing to a cheaper same-family model. Ordered +// most-specific first. +// +// The floor is deliberately AGGRESSIVE: an unrecognized haiku/sonnet supports +// NOTHING optional, so every request rewritten to it is a plain request every +// model accepts. Over-stripping on a down-route is safe (the request runs, just +// without the betas — which the cheaper tier may not honor anyway); UNDER- +// stripping 400s (confirmed live: an unknown sonnet rejected both the context-1m +// beta AND adaptive thinking). Only opus — the top tier, reached by UP-routing, +// where the source had fewer features to begin with — keeps the full set. A model +// matching no family at all still gets the fully-capable sentinel (forward, let +// Mode B catch a genuine 400). +var familyCaps = []struct { + sub string + caps ModelCaps +}{ + {"haiku", ModelCaps{Supports: map[Capability]bool{}, MaxContextTokens: 200_000}}, + {"sonnet", ModelCaps{Supports: map[Capability]bool{}, MaxContextTokens: 200_000}}, + {"opus", ModelCaps{Supports: map[Capability]bool{CapLongContext: true, CapEffortParam: true, CapThinking: true, CapMidConvSystem: true}, MaxContextTokens: 200_000}}, +} + +// capsFor returns the capabilities of a model. Resolution order: +// +// exact canonical id in baselineCaps → its caps +// else a known FAMILY (haiku/sonnet/opus) → conservative family floor (strip) +// else entirely unknown → fully capable (forward, let Mode B catch a real 400) // -// unknown CAPABILITY on a KNOWN model → false (strip) -// entirely UNKNOWN model → fully capable (forward) +// The family step is what fixes the silent-down-route-400: an unrecognized but +// same-family model no longer masquerades as fully capable. func capsFor(model string) ModelCaps { - if c, ok := baselineCaps[canonicalModel(model)]; ok { + canon := canonicalModel(model) + if c, ok := baselineCaps[canon]; ok { return c } + for _, f := range familyCaps { + if strings.Contains(canon, f.sub) { + return f.caps + } + } return fullyCapable } diff --git a/router/caps_family_test.go b/router/caps_family_test.go new file mode 100644 index 0000000..324b329 --- /dev/null +++ b/router/caps_family_test.go @@ -0,0 +1,29 @@ +package router + +import "testing" + +// The family fallback is what fixes the silent down-route 400: a model not in +// baselineCaps but of a known family gets the conservative family floor, so +// reconciliation still strips features the target rejects. +func TestCapsFor_FamilyFallback(t *testing.T) { + // An unrecognized sonnet must claim NOTHING optional — an opus[1m] request + // carries the context-1m beta AND adaptive thinking, both of which an older + // sonnet rejects; the aggressive floor strips them so the down-route succeeds. + for _, cap := range []Capability{CapLongContext, CapThinking, CapEffortParam, CapMidConvSystem} { + if capsFor("claude-sonnet-4-5-20250929").supports(cap) { + t.Errorf("unknown sonnet must not claim %s (aggressive family floor)", cap) + } + } + // An unrecognized opus keeps long-context. + if !capsFor("claude-opus-4-7").supports(CapLongContext) { + t.Error("opus family should support CapLongContext") + } + // Haiku supports nothing. + if capsFor("claude-haiku-4-5-20251001").supports(CapThinking) { + t.Error("haiku supports nothing") + } + // A genuinely unknown family stays fully capable (forward, let Mode B catch). + if !capsFor("some-vendor-model-x").supports(CapLongContext) { + t.Error("unknown family should be fully capable") + } +} diff --git a/router/doc.go b/router/doc.go index 7ed8af1..cf67946 100644 --- a/router/doc.go +++ b/router/doc.go @@ -4,7 +4,7 @@ // compress package (compress intercepts tool OUTPUTS; router intercepts LLM // REQUESTS) and is fully self-contained under this folder. // -// Layering, inside-out (see docs/ROUTER_IMPLEMENTATION_PLAN.md): +// Layering, inside-out (see docs/ROUTER.md §9): // - policy/rule/validate — the policy schema, loading, and validation // - signals — heuristic extraction from a request body // - engine/decide — the precedence ladder that turns signals into a Decision diff --git a/router/engine.go b/router/engine.go index 15f3925..b3ae90b 100644 --- a/router/engine.go +++ b/router/engine.go @@ -20,8 +20,11 @@ var ErrMLDisabled = errors.New("router: ML classifier disabled") // thresholds/membership; the classifier only computes raw scores from text + // candidates (compute where the data lives). type Classifier interface { - // Domain returns the intent classifier's category label for text + confidence. - Domain(text string) (label string, conf float64, err error) + // Domain returns the intent classifier's argmax label + confidence AND the + // full softmax distribution over its categories. The engine thresholds + // probability MASS over a policy-defined category set (matchDomain); nil/empty + // probs (older sidecar, degraded mode) falls back to argmax membership. + Domain(text string) (label string, conf float64, probs map[string]float64, err error) // EmbeddingScore returns the bank score of text against candidates // (0.75·best + 0.25·mean(top-2) cosine). EmbeddingScore(text string, candidates []string) (score float64, err error) @@ -32,8 +35,8 @@ type Classifier interface { // noopClassifier is the smart-off implementation: every call reports disabled. type noopClassifier struct{} -func (noopClassifier) Domain(string) (string, float64, error) { - return "", 0, ErrMLDisabled +func (noopClassifier) Domain(string) (string, float64, map[string]float64, error) { + return "", 0, nil, ErrMLDisabled } func (noopClassifier) EmbeddingScore(string, []string) (float64, error) { return 0, ErrMLDisabled @@ -54,12 +57,20 @@ type Decision struct { Model string Reason string Stripped []string + // MLTrace records every ML signal actually computed for this request, with + // its raw value ("cplx:reasoning=-0.471 dom=other@1.00"), or "off"/"err" when + // unavailable — so the log shows WHY a route did or didn't fire. Empty when no + // ML leaf was reached (heuristics decided first). + MLTrace string } // IsNoOp reports whether the decision routes to the model the client already // requested (compared canonicalized). The proxy uses this to byte-passthrough // the request unchanged — no model rewrite, no capability reconciliation (R11). func IsNoOp(d Decision, s Signals) bool { + if d.Model == "" { + return true // no rewrite target (e.g. default:requested with no model field) + } return s.RequestedModel != "" && canonicalModel(d.Model) == s.RequestedModel } @@ -73,7 +84,7 @@ const ( srcDefault ) -// Decide runs the precedence ladder (docs/ROUTER_POLICY_SCHEMA.md §0): +// Decide runs the precedence ladder (docs/ROUTER.md §4): // // pin → routes (first match, ML signals evaluated in-condition) → static default // then session stickiness damps a fuzzy (default) downgrade. @@ -118,7 +129,12 @@ func Decide(s Signals, p *Policy, cl Classifier, sess SessionStore, pin string) // 3. Static default. The ML now lives inside route conditions (signal leaves), // not a separate "smart default" step — matching vSR, where the default is - // just the last rule. + // just the last rule. `default: "requested"` short-circuits to a guaranteed + // no-op: no evidence → no rewrite (stickiness and session writes are skipped — + // there is no tier to remember). + if p.Default == requestedDefault { + return st.annotate(Decision{Tier: "-", Model: s.RequestedModel, Reason: "default:requested"}) + } return st.annotate(p.decide(p.Default, "default", srcDefault, s, sess)) } @@ -171,12 +187,17 @@ type evalState struct { // domain classification: computed once (the classifier emits one label). domainLabel string + domainProbs map[string]float64 domainDone bool domainErr error // embedding/complexity results memoized by signal name. embed map[string]mlResult complex map[string]mlResult + // trace accumulates computed signal values for observability (Decision.MLTrace). + trace []string + tracedDom map[string]bool // gate-mass traced once per domain signal + // mlErr is set when an ML leaf actually consulted for the CURRENT route // errored (reset per route). A route whose match depended on an unavailable // signal must NOT fire — otherwise a `not` over a failed leaf inverts fail-open @@ -209,9 +230,23 @@ func (st *evalState) annotate(d Decision) Decision { if st.anyMLErr { d.Reason += " ml-degraded" } + d.MLTrace = strings.Join(st.trace, " ") return d } +// traceVal formats one computed signal for the trace ("off" = smart mode +// disabled, "err" = a real sidecar failure — the reason also gets ml-degraded). +func (st *evalState) traceVal(prefix string, val float64, err error) { + switch { + case errors.Is(err, ErrMLDisabled): + st.trace = append(st.trace, prefix+"=off") + case err != nil: + st.trace = append(st.trace, prefix+"=err") + default: + st.trace = append(st.trace, fmt.Sprintf("%s=%+.3f", prefix, val)) + } +} + // mlResult memoizes one signal computation (a score/margin and its error) so a // signal referenced by several leaves in one request is computed only once. type mlResult struct { @@ -221,7 +256,7 @@ type mlResult struct { // eval evaluates a condition tree to a boolean. Pure predicates + short-circuit // + cheap-first child ordering keep the classifier off requests a cheap sibling -// already decided (docs ROUTER_DESIGN §2.3). +// already decided (docs/ROUTER.md §5). func (st *evalState) eval(r *Rule) bool { switch { case r.All != nil: @@ -294,18 +329,43 @@ func (st *evalState) recentLower() string { return st.loweredRecent } -// matchLiteralKeywords: case-insensitive substring OR over the inspect-window -// text. Literal (not regex) — a coder's "c++" never explodes. +// matchLiteralKeywords: case-insensitive WHOLE-WORD/PHRASE OR over the +// inspect-window text. Literal (not regex) — a coder's "c++" never explodes. +// Whole-word means the occurrence is not embedded inside a larger alphanumeric +// run: "migration" no longer matches "immigration", "refactor" no longer matches +// "refactored" (list the variants you want). A boundary is any non-alphanumeric +// rune or the string edge, so "c++" still matches in "use c++ here". func (st *evalState) matchLiteralKeywords(kws []string) bool { hay := st.recentLower() for _, kw := range kws { - if kw != "" && strings.Contains(hay, strings.ToLower(kw)) { + if kw != "" && containsWord(hay, strings.ToLower(kw)) { return true } } return false } +// containsWord reports whether needle occurs in hay with non-alphanumeric (or +// edge) boundaries on both sides. Both inputs must already be lowercased. +func containsWord(hay, needle string) bool { + for from := 0; ; { + i := strings.Index(hay[from:], needle) + if i < 0 { + return false + } + start := from + i + end := start + len(needle) + if (start == 0 || !isAlnum(hay[start-1])) && (end == len(hay) || !isAlnum(hay[end])) { + return true + } + from = start + 1 + } +} + +func isAlnum(b byte) bool { + return b >= 'a' && b <= 'z' || b >= 'A' && b <= 'Z' || b >= '0' && b <= '9' +} + func (st *evalState) matchRegexKeywords(pats []string) bool { for _, pat := range pats { if re := compiledRegex(pat); re != nil && re.MatchString(st.s.RecentText) { @@ -315,6 +375,20 @@ func (st *evalState) matchRegexKeywords(pats []string) bool { return false } +// mlText is what the ML signals classify: the LATEST user turn, not the joined +// inspect window. Averaging turns dilutes the classifiers — measured live: a turn +// scoring complexity +0.171 (hard) alone fell to +0.025 (medium) once joined with +// two earlier trivial turns, silently suppressing mid-session escalation. Keywords +// deliberately KEEP the window (matchLiteralKeywords) — persistence is a feature +// there (a hard keyword two turns back keeps protecting); for a classifier it is +// noise. Falls back to the window text when the last turn has no user text. +func (st *evalState) mlText() string { + if st.s.LastUserText != "" { + return st.s.LastUserText + } + return st.s.RecentText +} + // matchDomain fires when the classifier's predicted label is in the named // domain's category set. The classification is computed once per request; on // ML-disabled/error the leaf is false (the route simply won't fire). @@ -324,12 +398,47 @@ func (st *evalState) matchDomain(name string) bool { return false // validation guarantees the reference resolves } if !st.domainDone { - st.domainLabel, _, st.domainErr = st.cl.Domain(st.s.RecentText) + var conf float64 + st.domainLabel, conf, st.domainProbs, st.domainErr = st.cl.Domain(st.mlText()) st.domainDone = true + switch { + case errors.Is(st.domainErr, ErrMLDisabled): + st.trace = append(st.trace, "dom=off") + case st.domainErr != nil: + st.trace = append(st.trace, "dom=err") + default: + st.trace = append(st.trace, fmt.Sprintf("dom=%s@%.3f", st.domainLabel, conf)) + } } if st.domainErr != nil { return st.mlFailed(st.domainErr) } + // Mass thresholding (preferred): fire iff the total probability the classifier + // assigns to the signal's categories clears MinMass. One scalar subsumes + // entropy handling — mass concentrates only on a CONFIDENT in-set + // classification; an ambiguous/flat distribution fails the threshold, so an + // uncertain classification falls to the policy default (the safe middle tier), + // never up. Invariant to which in-set category won (math 0.4 + physics 0.4 + // passes 0.7 without a special top-2 case). + if sig.MinMass > 0 && len(st.domainProbs) > 0 { + mass := 0.0 + for _, c := range sig.Categories { + mass += st.domainProbs[c] + } + // Trace the EVALUATED gate mass at full precision (once per signal): a + // mass of 0.897 against a 0.9 gate must never display as a self- + // contradictory "0.90 didn't fire" (caught live). + if !st.tracedDom[name] { + if st.tracedDom == nil { + st.tracedDom = map[string]bool{} + } + st.tracedDom[name] = true + st.trace = append(st.trace, fmt.Sprintf("dom:%s=%.3f/%.2f", name, mass, sig.MinMass)) + } + return mass >= sig.MinMass + } + // Argmax membership fallback: MinMass unset, or the sidecar didn't return a + // distribution (older sidecar) — degrade gracefully rather than never firing. for _, c := range sig.Categories { if c == st.domainLabel { return true @@ -387,8 +496,9 @@ func (st *evalState) embedScore(name string, candidates []string) (float64, erro if r, ok := st.embed[name]; ok { return r.val, r.err } - score, err := st.cl.EmbeddingScore(st.s.RecentText, candidates) + score, err := st.cl.EmbeddingScore(st.mlText(), candidates) st.embed[name] = mlResult{score, err} + st.traceVal("emb:"+name, score, err) return score, err } @@ -400,8 +510,9 @@ func (st *evalState) complexMargin(name string, hard, easy []string) (float64, e if r, ok := st.complex[name]; ok { return r.val, r.err } - margin, err := st.cl.ComplexityMargin(st.s.RecentText, hard, easy) + margin, err := st.cl.ComplexityMargin(st.mlText(), hard, easy) st.complex[name] = mlResult{margin, err} + st.traceVal("cplx:"+name, margin, err) return margin, err } diff --git a/router/engine_gap_test.go b/router/engine_gap_test.go index 489fad0..af4c8de 100644 --- a/router/engine_gap_test.go +++ b/router/engine_gap_test.go @@ -47,7 +47,9 @@ func loadWhenSignals(t *testing.T, when string) *Policy { // fixedDomain is a deterministic classifier: Domain always returns `label`. type fixedDomain struct{ label string } -func (f fixedDomain) Domain(string) (string, float64, error) { return f.label, 1, nil } +func (f fixedDomain) Domain(string) (string, float64, map[string]float64, error) { + return f.label, 1, nil, nil +} func (f fixedDomain) EmbeddingScore(string, []string) (float64, error) { return 0, nil } @@ -93,7 +95,7 @@ func naiveLeaf(r *Rule, s *Signals, cl Classifier, pol *Policy) bool { case r.Keywords != nil: hay := strings.ToLower(s.RecentText) for _, kw := range r.Keywords { - if kw != "" && strings.Contains(hay, strings.ToLower(kw)) { + if kw != "" && containsWord(hay, strings.ToLower(kw)) { return true } } @@ -113,7 +115,7 @@ func naiveLeaf(r *Rule, s *Signals, cl Classifier, pol *Policy) bool { } return false case r.Domain != "": - label, _, err := cl.Domain(s.RecentText) + label, _, _, err := cl.Domain(s.RecentText) if err != nil { return false } diff --git a/router/engine_test.go b/router/engine_test.go index 2a3c8ec..64a5eb7 100644 --- a/router/engine_test.go +++ b/router/engine_test.go @@ -10,6 +10,7 @@ import ( // already decided the node) and per-request memoization. type spyClassifier struct { domainLabel string + domainProbs map[string]float64 embedScore float64 complexMargin float64 domainCalls int @@ -17,9 +18,9 @@ type spyClassifier struct { complexCalls int } -func (c *spyClassifier) Domain(string) (string, float64, error) { +func (c *spyClassifier) Domain(string) (string, float64, map[string]float64, error) { c.domainCalls++ - return c.domainLabel, 1, nil + return c.domainLabel, 1, c.domainProbs, nil } func (c *spyClassifier) EmbeddingScore(string, []string) (float64, error) { c.embedCalls++ diff --git a/router/injected_block_test.go b/router/injected_block_test.go new file mode 100644 index 0000000..9da131c --- /dev/null +++ b/router/injected_block_test.go @@ -0,0 +1,44 @@ +package router + +import "testing" + +// Claude Code injects text blocks INSIDE user messages +// (captured live). They are framework context, not user-authored — signal text +// must exclude them, or the classifier scores the reminder instead of the human +// question and mid-session routing silently degrades to the default. +func TestSignals_SystemReminderBlocksExcluded(t *testing.T) { + // The exact structure captured from real Claude Code traffic. + body := `{"model":"claude-opus-4-8","messages":[ + {"role":"user","content":[ + {"type":"text","text":"\nAs you answer, you can use the following context:\n# userEmail\nuser@example.com\n# currentDate\n2026-07-10\n"}, + {"type":"text","text":"what day is today?"} + ]} + ]}` + s, err := Extract([]byte(body), "sess", InspectCfg{Scope: "recent_turns", Turns: 3}) + if err != nil { + t.Fatal(err) + } + if s.LastUserText != "what day is today?" { + t.Fatalf("LastUserText must be the human text only, got %q", s.LastUserText) + } + if s.RecentText != "what day is today?" { + t.Fatalf("RecentText must exclude injected blocks (keyword input too), got %q", s.RecentText) + } +} + +// A user message that is ONLY a reminder contributes no user text (the turn is +// skipped, like a tool_result-only turn). +func TestSignals_ReminderOnlyTurnSkipped(t *testing.T) { + body := `{"model":"m","messages":[ + {"role":"user","content":[{"type":"text","text":"real question"}]}, + {"role":"assistant","content":"answer"}, + {"role":"user","content":[{"type":"text","text":"nudge"}]} + ]}` + s, err := Extract([]byte(body), "sess", InspectCfg{Scope: "full"}) + if err != nil { + t.Fatal(err) + } + if s.LastUserText != "real question" { + t.Fatalf("reminder-only turn must be skipped, got %q", s.LastUserText) + } +} diff --git a/router/mass_word_test.go b/router/mass_word_test.go new file mode 100644 index 0000000..53d68dd --- /dev/null +++ b/router/mass_word_test.go @@ -0,0 +1,98 @@ +package router + +import "testing" + +// ---- domain mass thresholding (the entropy-subsuming scalar) ---------------- + +func massPolicy(minMass float64) string { + return `{ + "version":1, + "tiers":[{"name":"main","model":"claude-sonnet-4-5"},{"name":"smart","model":"claude-opus-4-8"}], + "default":"main","inspect":{"scope":"full"}, + "signals":{"domains":[{"name":"quant","categories":["math","physics","chemistry"],"min_mass":` + + trimFloat(minMass) + `}]}, + "routes":[{"name":"up","when":{"domain":"quant"},"to":"smart"}] + }` +} + +func trimFloat(f float64) string { + if f == 0.7 { + return "0.7" + } + return "0" +} + +func decideWithProbs(t *testing.T, minMass float64, label string, probs map[string]float64) Decision { + t.Helper() + p, _ := mustLoad(t, massPolicy(minMass)) + cl := &spyClassifier{domainLabel: label, domainProbs: probs} + return Decide(Signals{RecentText: "q", SessionID: "s"}, p, cl, nil, "") +} + +// Confident in-set classification clears the threshold. +func TestDomainMass_ConfidentInSetFires(t *testing.T) { + d := decideWithProbs(t, 0.7, "math", map[string]float64{"math": 0.95, "other": 0.05}) + if d.Tier != "smart" { + t.Fatalf("mass 0.95 on quant must escalate, got %s (%s)", d.Tier, d.Reason) + } +} + +// Mass is invariant to WHICH in-set category won — split across two hard +// categories still clears (no top-2 special case needed). +func TestDomainMass_SplitAcrossInSetFires(t *testing.T) { + d := decideWithProbs(t, 0.7, "math", map[string]float64{"math": 0.4, "physics": 0.4, "other": 0.2}) + if d.Tier != "smart" { + t.Fatalf("math0.4+physics0.4=0.8 must clear 0.7, got %s (%s)", d.Tier, d.Reason) + } +} + +// An ambiguous distribution fails the threshold even when the ARGMAX is in-set: +// uncertainty falls to the default (cost-first), never up. +func TestDomainMass_AmbiguousArgmaxDoesNotFire(t *testing.T) { + d := decideWithProbs(t, 0.7, "math", map[string]float64{"math": 0.4, "other": 0.35, "history": 0.25}) + if d.Tier != "main" { + t.Fatalf("mass 0.4 < 0.7 must fall to default even though argmax is math, got %s (%s)", d.Tier, d.Reason) + } +} + +// min_mass unset → legacy argmax membership. +func TestDomainMass_UnsetFallsBackToArgmax(t *testing.T) { + d := decideWithProbs(t, 0, "math", map[string]float64{"math": 0.4, "other": 0.35, "history": 0.25}) + if d.Tier != "smart" { + t.Fatalf("min_mass unset should use argmax membership, got %s (%s)", d.Tier, d.Reason) + } +} + +// Sidecar without a distribution (probs nil) degrades to argmax membership even +// when min_mass is set — never silently disables the signal. +func TestDomainMass_NilProbsDegradesToArgmax(t *testing.T) { + d := decideWithProbs(t, 0.7, "math", nil) + if d.Tier != "smart" { + t.Fatalf("nil probs should degrade to argmax membership, got %s (%s)", d.Tier, d.Reason) + } +} + +// ---- word-boundary keyword matching ------------------------------------------ + +func TestContainsWord(t *testing.T) { + cases := []struct { + hay, needle string + want bool + }{ + {"we need a db migration now", "migration", true}, + {"my immigration paperwork", "migration", false}, // inside a word + {"refactored the module", "refactor", false}, // suffix growth + {"please refactor the module", "refactor", true}, + {"use c++ for this", "c++", true}, // trailing non-alnum ok + {"c++x is not a language", "c++", false}, // embedded + {"a race condition appeared", "race condition", true}, // phrase + {"racecondition", "race condition", false}, + {"migration", "migration", true}, // exact / edges + {"reformat this financial model", "reformat this", true}, // phrase-in-context still matches (inherent to keywords) + } + for _, c := range cases { + if got := containsWord(c.hay, c.needle); got != c.want { + t.Errorf("containsWord(%q, %q) = %v, want %v", c.hay, c.needle, got, c.want) + } + } +} diff --git a/router/ml/client.go b/router/ml/client.go index 840b357..7e5b0cf 100644 --- a/router/ml/client.go +++ b/router/ml/client.go @@ -3,7 +3,7 @@ // sidecar URL is configured; when it isn't, the engine uses the noop classifier // and this package is never touched. // -// The classifier lives behind an HTTP boundary on purpose (docs/ROUTER_DESIGN.md): +// The classifier lives behind an HTTP boundary on purpose (docs/ROUTER.md §5): // the models (~200-300MB of ONNX) stay in the Python sidecar where the compressor // already hosts model weight, so the Go daemon carries no model runtime and no // heavy dependency — this package is stdlib-only. @@ -51,15 +51,16 @@ func New(baseURL string) *Client { // Intent classifies text into a single category label with a confidence. On any // sidecar error the engine's signal leaf simply evaluates false (the route won't // fire) — so the error is returned verbatim, not swallowed. -func (c *Client) Domain(text string) (string, float64, error) { +func (c *Client) Domain(text string) (string, float64, map[string]float64, error) { var out struct { - Label string `json:"label"` - Confidence float64 `json:"confidence"` + Label string `json:"label"` + Confidence float64 `json:"confidence"` + Probs map[string]float64 `json:"probs"` } if err := c.post("/v1/route/domain", domainReq{Text: text}, &out); err != nil { - return "", 0, err + return "", 0, nil, err } - return out.Label, out.Confidence, nil + return out.Label, out.Confidence, out.Probs, nil } // EmbeddingScore returns the query's bank score against candidates (the sidecar diff --git a/router/ml/client_gap_test.go b/router/ml/client_gap_test.go index 596a81f..eb81d49 100644 --- a/router/ml/client_gap_test.go +++ b/router/ml/client_gap_test.go @@ -52,7 +52,7 @@ func TestClient_EmptyOrMissingFieldsAreNotError(t *testing.T) { c := fakeSidecar(t, func(w http.ResponseWriter, r *http.Request) { io.WriteString(w, `{}`) }) - label, conf, err := c.Domain("x") + label, conf, _, err := c.Domain("x") if err != nil { t.Fatalf("empty domain reply must not error: %v", err) } @@ -179,7 +179,7 @@ func TestClient_DomainRequestWireContract(t *testing.T) { _ = json.Unmarshal(body, &wire) io.WriteString(w, `{"label":"math","confidence":0.9}`) }) - label, conf, err := c.Domain("prove this theorem") + label, conf, _, err := c.Domain("prove this theorem") if err != nil { t.Fatal(err) } diff --git a/router/ml/client_test.go b/router/ml/client_test.go index f9c484c..49fa34a 100644 --- a/router/ml/client_test.go +++ b/router/ml/client_test.go @@ -29,7 +29,7 @@ func TestClient_DomainSuccess(t *testing.T) { } io.WriteString(w, `{"label":"math","confidence":0.94}`) }) - label, conf, err := c.Domain("prove every finite integral domain is a field") + label, conf, _, err := c.Domain("prove every finite integral domain is a field") if err != nil { t.Fatal(err) } @@ -95,14 +95,14 @@ func TestClient_MalformedJSONIsError(t *testing.T) { c := fakeSidecar(t, func(w http.ResponseWriter, r *http.Request) { io.WriteString(w, `{"label":`) // truncated }) - if _, _, err := c.Domain("x"); err == nil { + if _, _, _, err := c.Domain("x"); err == nil { t.Fatal("a malformed reply must be an error") } } func TestClient_TransportErrorIsError(t *testing.T) { c := New("http://127.0.0.1:1") // nothing listening - if _, _, err := c.Domain("x"); err == nil { + if _, _, _, err := c.Domain("x"); err == nil { t.Fatal("a transport failure must be an error") } } diff --git a/router/mltext_test.go b/router/mltext_test.go new file mode 100644 index 0000000..d39d31f --- /dev/null +++ b/router/mltext_test.go @@ -0,0 +1,72 @@ +package router + +import "testing" + +// ML signals must classify the LATEST user turn, not the joined inspect window — +// averaging turns dilutes the classifier (measured live: a hard turn's margin +// fell from +0.171 to +0.025 when joined with two earlier trivial turns, +// suppressing mid-session escalation). Keywords keep the window on purpose. +func TestMLSignals_ClassifyLatestTurnNotWindow(t *testing.T) { + p, _ := mustLoad(t, `{ + "version":1, + "tiers":[{"name":"main","model":"claude-sonnet-4-5"},{"name":"smart","model":"claude-opus-4-8"}], + "default":"main","inspect":{"scope":"recent_turns","turns":3}, + "signals":{"complexity":[{"name":"r","threshold":0.15,"hard":["x"],"easy":["y"]}]}, + "routes":[{"name":"up","when":{"complexity":"r:hard"},"to":"smart"}] + }`) + cl := &textSpyClassifier{margin: 0.5} + sig := Signals{ + LastUserText: "dive deep and synthesize the architecture", + RecentText: "what is the date. explain this file. dive deep and synthesize the architecture", + SessionID: "s", + } + Decide(sig, p, cl, nil, "") + if cl.sawText != sig.LastUserText { + t.Fatalf("ML signal must classify the latest user turn, got %q", cl.sawText) + } + + // Fallback: a turn with no user text (tool-result-only) uses the window. + cl2 := &textSpyClassifier{margin: 0.5} + Decide(Signals{LastUserText: "", RecentText: "window text", SessionID: "s"}, p, cl2, nil, "") + if cl2.sawText != "window text" { + t.Fatalf("empty last turn must fall back to the window, got %q", cl2.sawText) + } +} + +// Keywords, by contrast, keep scanning the whole window (persistence is the point). +func TestKeywords_StillScanWindow(t *testing.T) { + p, _ := mustLoad(t, `{ + "version":1, + "tiers":[{"name":"main","model":"claude-sonnet-4-5"},{"name":"smart","model":"claude-opus-4-8"}], + "default":"main","inspect":{"scope":"recent_turns","turns":3}, + "routes":[{"name":"up","when":{"keywords":["deadlock"]},"to":"smart"}] + }`) + sig := Signals{ + LastUserText: "keep going please", + RecentText: "debug the deadlock in the pool. keep going please", + SessionID: "s", + } + d := Decide(sig, p, nil, nil, "") + if d.Tier != "smart" { + t.Fatalf("keyword from an earlier window turn must still protect, got %s (%s)", d.Tier, d.Reason) + } +} + +// textSpyClassifier records the text it was asked to classify. +type textSpyClassifier struct { + sawText string + margin float64 +} + +func (c *textSpyClassifier) Domain(text string) (string, float64, map[string]float64, error) { + c.sawText = text + return "", 0, nil, ErrMLDisabled +} +func (c *textSpyClassifier) EmbeddingScore(text string, _ []string) (float64, error) { + c.sawText = text + return 0, ErrMLDisabled +} +func (c *textSpyClassifier) ComplexityMargin(text string, _, _ []string) (float64, error) { + c.sawText = text + return c.margin, nil +} diff --git a/router/mltrace_test.go b/router/mltrace_test.go new file mode 100644 index 0000000..c8f66ad --- /dev/null +++ b/router/mltrace_test.go @@ -0,0 +1,34 @@ +package router + +import ( + "strings" + "testing" +) + +// The decision carries the computed signal values (for the log line): real values +// when smart mode is on, an explicit "off" marker when it isn't — so a policy +// silently running heuristics-only is visible on every line, not just at startup. +func TestMLTrace(t *testing.T) { + pol := `{ + "version":1, + "tiers":[{"name":"fast","model":"claude-haiku-4-5"},{"name":"smart","model":"claude-opus-4-8"}], + "default":"requested","inspect":{"scope":"full"}, + "signals":{"complexity":[{"name":"r","threshold":0.15,"hard":["x"],"easy":["y"]}]}, + "routes":[{"name":"up","when":{"complexity":"r:hard"},"to":"smart"}] + }` + p, _ := mustLoad(t, pol) + + // smart on: the trace carries the raw margin. + d := Decide(Signals{RequestedModel: "claude-haiku-4-5", RecentText: "q", SessionID: "s"}, + p, &spyClassifier{complexMargin: 0.42}, nil, "") + if !strings.Contains(d.MLTrace, "cplx:r=+0.420") { + t.Errorf("trace should carry the computed margin, got %q", d.MLTrace) + } + + // smart off: the trace says so explicitly. + d = Decide(Signals{RequestedModel: "claude-haiku-4-5", RecentText: "q", SessionID: "s"}, + p, nil, nil, "") + if !strings.Contains(d.MLTrace, "cplx:r=off") { + t.Errorf("smart-off must be visible in the trace, got %q", d.MLTrace) + } +} diff --git a/router/policies.go b/router/policies.go new file mode 100644 index 0000000..810fd3b --- /dev/null +++ b/router/policies.go @@ -0,0 +1,45 @@ +package router + +import ( + "embed" + "fmt" + "io/fs" + "sort" + "strings" +) + +// presetFS carries the built-in example policies inside the binary so +// `whittle policy init` works from a bare `go install` with no extra files. +// +//go:embed policies/*.json +var presetFS embed.FS + +// presetDescriptions is the one-line summary shown by `whittle policy list`, +// keyed by preset name. Every embedded preset must have an entry (enforced by +// TestPresets_AllValid). +var presetDescriptions = map[string]string{ + "default": "Mixed-use, conservative: hard reasoning or confident quantitative → opus; confidently-casual chit-chat → haiku (easy) or sonnet (medium); EVERYTHING else keeps the model you asked for. Down-routing requires two concordant signals. See policies/default.md.", +} + +// PresetNames returns the built-in example-policy names, sorted. +func PresetNames() []string { + entries, _ := fs.ReadDir(presetFS, "policies") + names := make([]string, 0, len(entries)) + for _, e := range entries { + names = append(names, strings.TrimSuffix(e.Name(), ".json")) + } + sort.Strings(names) + return names +} + +// PresetDescription returns the one-line summary for a preset ("" if unknown). +func PresetDescription(name string) string { return presetDescriptions[name] } + +// Preset returns the raw JSON of a built-in policy by name. +func Preset(name string) ([]byte, error) { + b, err := presetFS.ReadFile("policies/" + name + ".json") + if err != nil { + return nil, fmt.Errorf("no built-in policy %q (see `whittle policy list`)", name) + } + return b, nil +} diff --git a/router/policies/default.json b/router/policies/default.json new file mode 100644 index 0000000..d1a6777 --- /dev/null +++ b/router/policies/default.json @@ -0,0 +1,125 @@ +{ + "version": 1, + "tiers": [ + { + "name": "fast", + "model": "claude-haiku-4-5" + }, + { + "name": "main", + "model": "claude-sonnet-4-5" + }, + { + "name": "smart", + "model": "claude-opus-4-8" + } + ], + "default": "requested", + "inspect": { + "scope": "recent_turns", + "turns": 3 + }, + "signals": { + "domains": [ + { + "name": "quantitative", + "categories": [ + "math", + "physics", + "chemistry" + ], + "min_mass": 0.7 + }, + { + "name": "casual", + "categories": [ + "other" + ], + "min_mass": 0.85 + } + ], + "complexity": [ + { + "name": "reasoning", + "threshold": 0.15, + "hard": [ + "analyze the root cause of this failure", + "reason step by step through this multi-part problem", + "weigh the tradeoffs and recommend an approach", + "debug this intermittent race condition", + "critique this argument and find the flaws", + "assess the risks in this contract or decision", + "design the architecture for this system", + "synthesize these sources into a coherent brief" + ], + "easy": [ + "fix this typo", + "rephrase this sentence", + "summarize this in one line", + "what does this word mean", + "reformat this list", + "give me a quick yes or no", + "translate this short phrase", + "make this a little more concise", + "what is the date today", + "what time is it right now", + "what is the capital of that country", + "how do you spell this word", + "where is that place located", + "who wrote that famous book", + "when did that event happen" + ] + } + ] + }, + "routes": [ + { + "name": "escalate", + "when": { + "any": [ + { + "complexity": "reasoning:hard" + }, + { + "domain": "quantitative" + } + ] + }, + "to": "smart" + }, + { + "name": "casual-easy", + "when": { + "all": [ + { + "domain": "casual" + }, + { + "complexity": "reasoning:easy" + } + ] + }, + "to": "fast" + }, + { + "name": "casual-medium", + "when": { + "all": [ + { + "domain": "casual" + }, + { + "complexity": "reasoning:medium" + } + ] + }, + "to": "main" + } + ], + "session": { + "sticky": false + }, + "overrides": { + "pin_header": "x-whittle-route" + } +} \ No newline at end of file diff --git a/router/policies/default.md b/router/policies/default.md new file mode 100644 index 0000000..d94ada8 --- /dev/null +++ b/router/policies/default.md @@ -0,0 +1,38 @@ +# The `default` routing policy + +The calibrated out-of-the-box policy (`whittle policy init` writes it with the +model ids your account actually uses). Design: **conservative** — whittle only +changes your model when a rule affirmatively matches; on any doubt, your request +runs untouched on the model you asked for. + +| when | routed to | +|---|---| +| hard reasoning (contrastive complexity > 0.15) **or** confidently quantitative (math/physics/chemistry mass ≥ 0.7) | `smart` (opus) | +| confidently casual (non-academic mass ≥ 0.85) **and** trivially easy | `fast` (haiku) | +| confidently casual **and** medium (drafts, plans, brainstorms) | `main` (sonnet) | +| **everything else** — all coding traffic included | untouched (`default: "requested"`) | + +A down-route needs **two signals agreeing**; an up-route needs one — misroute +risk is priced by direction. The thresholds were calibrated against live traffic +(e.g. the 0.85 casual gate tolerates the ~0.05–0.10 confidence loss real typos +cause). + +## Customizing (`~/.whittle/router.json`) + +- **Your models**: edit the `tiers` — use full dated ids (`claude-sonnet-4-5-20250929`); + bare ids are often rejected upstream. +- **More savings**: set `"default": "main"` — unmatched traffic then rides sonnet + instead of your requested model (quality trade, biggest cost lever). +- **Route your own patterns**: add a route with whole-word `keywords`, or extend + the `hard`/`easy` example banks — the complexity signal scores similarity to + those phrasings, so add examples *shaped like your real requests*. +- **Tune the gates**: `min_mass` (domain confidence) and `threshold` (complexity + dead-band). Watch the `"signals"` field in the router log — it shows every + computed value against its gate (`dom:casual=0.899/0.85`), so you can see + exactly why a request routed before touching anything. +- **Escape hatch**: send header `x-whittle-route: ` to pin any request. + +After editing: `whittle policy validate ~/.whittle/router.json`, then `kill -HUP` +the router (or restart) — a bad edit keeps the running policy. + +Full architecture and signal math: [docs/ROUTER.md](../../docs/ROUTER.md). diff --git a/router/policies_test.go b/router/policies_test.go new file mode 100644 index 0000000..70d4082 --- /dev/null +++ b/router/policies_test.go @@ -0,0 +1,31 @@ +package router + +import "testing" + +// Every built-in example policy MUST load cleanly and have a description — a +// shipped preset that fails validation would break `whittle policy init`. +func TestPresets_AllValid(t *testing.T) { + names := PresetNames() + if len(names) == 0 { + t.Fatal("no example policies embedded") + } + for _, n := range names { + b, err := Preset(n) + if err != nil { + t.Fatalf("Preset(%q): %v", n, err) + } + if _, _, err := Load(b); err != nil { + t.Errorf("built-in policy %q does not load: %v", n, err) + } + if PresetDescription(n) == "" { + t.Errorf("built-in policy %q has no description", n) + } + } +} + +// Preset on an unknown name is a clear error, not a panic. +func TestPreset_UnknownName(t *testing.T) { + if _, err := Preset("does-not-exist"); err == nil { + t.Fatal("expected an error for an unknown preset name") + } +} diff --git a/router/policy.go b/router/policy.go index 0617d6e..74376a8 100644 --- a/router/policy.go +++ b/router/policy.go @@ -59,19 +59,28 @@ type InspectCfg struct { // route actually reaches it — cheap-first evaluation keeps the models off any // request a heuristic sibling already decided. Mirrors vLLM Semantic Router's // signal model: `domain` from the intent classifier, `embedding` + `complexity` -// from the text embedding model (docs/ROUTER_POLICY_SCHEMA.md). +// from the text embedding model (docs/ROUTER.md). type SignalSet struct { Domains []DomainSignal `json:"domains,omitempty"` Embeddings []EmbeddingSignal `json:"embeddings,omitempty"` Complexity []ComplexitySignal `json:"complexity,omitempty"` } -// DomainSignal fires when the intent classifier's predicted MMLU-Pro category is -// one of Categories. It groups raw classifier labels under a policy-friendly name -// (coding = {computer science, engineering}). +// DomainSignal fires on the intent classifier's output over Categories. +// +// With MinMass set (0 < m ≤ 1): fires iff the TOTAL softmax probability mass the +// classifier assigns to Categories is ≥ MinMass. Mass thresholding is the +// preferred form — it only passes on a confident in-set classification, is +// invariant to which in-set category won, and an ambiguous (high-entropy) +// distribution simply fails the threshold so routing falls to the policy default +// (cost-first: uncertainty lands on the middle tier, never escalates). +// +// With MinMass unset: fires iff the argmax label ∈ Categories (legacy behavior, +// also the graceful fallback when the sidecar returns no distribution). type DomainSignal struct { Name string `json:"name"` Categories []string `json:"categories"` + MinMass float64 `json:"min_mass,omitempty"` } // EmbeddingSignal fires when the query's bank score against Candidates @@ -109,6 +118,15 @@ const ( // model." It is rejected as a tier name so the two never collide. keepTier = "keep" + // requestedDefault is the reserved `default:` value meaning "no route matched → + // keep the model the client asked for, untouched" (a guaranteed no-op + // passthrough). This is the fail-open posture applied to routing itself: with + // zero evidence about a request, whittle does not rewrite it — EVERY model + // change, up or down, must come from a rule the author wrote. It also protects + // mixed-model clients (Claude Code sends cheap-model background requests; a + // fixed-tier default would silently up-route them). + requestedDefault = "requested" + // Candidate-list caps for embedding/complexity signals: the cap is about // prototype quality + cold-start embedding cost, not runtime. candidatesSoftCap = 32 @@ -177,6 +195,17 @@ func (p *Policy) complexitySignal(name string) *ComplexitySignal { return nil } +// policyUsesML reports whether any route references an ML signal leaf — used to +// warn loudly when such a policy runs with smart mode off. +func policyUsesML(p *Policy) bool { + for i := range p.Routes { + if ruleUsesML(&p.Routes[i].When) { + return true + } + } + return false +} + // dateSuffix matches an 8-digit date snapshot suffix on a model id // (claude-opus-4-8-20260101). It must NOT eat the version hyphens (…-4-8), so it // is anchored to a trailing -YYYYMMDD only. diff --git a/router/proxy.go b/router/proxy.go index c7c81ac..6ee5e06 100644 --- a/router/proxy.go +++ b/router/proxy.go @@ -3,6 +3,7 @@ package router import ( "bytes" "encoding/json" + "fmt" "io" "net/http" "strconv" @@ -106,9 +107,14 @@ func (p *Proxy) ServeHTTP(w http.ResponseWriter, r *http.Request) { p.serve(rec, r) // One structured line per request — routing verdict + outcome, NEVER prompt // text (review C1: the router must not persist request content to disk). - p.log.Printf(`{"tier":%q,"reason":%q,"status":%d,"latency_ms":%d,"ctx_tokens":%d,"session":%q}`, - rec.Header().Get("X-Whittle-Route"), rec.Header().Get("X-Whittle-Reason"), - rec.status, time.Since(start).Milliseconds(), rec.ctxTokens, + used := rec.used + if used == "" { + used = rec.requested // no-op/passthrough/mode-b-retry ran the requested model + } + p.log.Printf(`{"tier":%q,"requested":%q,"model":%q,"reason":%q,"signals":%q,"status":%d,"latency_ms":%d,"ctx_tokens":%d,"in_tokens":%d,"out_tokens":%d,"session":%q}`, + rec.Header().Get("X-Whittle-Route"), orDash(rec.requested), orDash(used), + rec.Header().Get("X-Whittle-Reason"), rec.mlTrace, rec.status, + time.Since(start).Milliseconds(), rec.ctxTokens, rec.inTokens, rec.outTokens, shortSession(r.Header.Get("X-Claude-Code-Session-Id"))) } @@ -146,6 +152,7 @@ func (p *Proxy) serve(rec *statusRecorder, r *http.Request) { } sig, err := Extract(body, r.Header.Get("X-Claude-Code-Session-Id"), pol.Inspect) + rec.requested = sig.RequestedModel // for the log line (model id is not prompt text) if err != nil { // Our parse error → Mode A: forward the ORIGINAL untouched. p.passthrough(w, r, body, "fail-open:parse") @@ -153,6 +160,7 @@ func (p *Proxy) serve(rec *statusRecorder, r *http.Request) { } dec := Decide(sig, pol, p.cl, p.sess, pinFromHeader(pol, r.Header)) + rec.mlTrace = dec.MLTrace // No-op: the resolved model is what the client already asked for → byte // passthrough, no rewrite, no reconciliation (R11). @@ -236,6 +244,12 @@ func (p *Proxy) forward(w http.ResponseWriter, r *http.Request, originalBody, re return } defer resp.Body.Close() + // The routed rewrite was accepted → the tier's model served this response. + // (No-op / passthrough / Mode-B-retry paths leave `used` empty, so the log + // falls back to the requested model — which is what actually ran there.) + if rec := recorderOf(w); rec != nil { + rec.used = dec.Model + } p.relay(w, resp, dec.Tier, reason) } @@ -261,14 +275,21 @@ func (p *Proxy) modeBRetry(w http.ResponseWriter, r *http.Request, originalBody reason += " entitlement-blocked" } + // Surface WHICH rewrite the upstream rejected — otherwise a bad tier model id + // (e.g. an invalid/undated model) fails on every request and is silently bailed + // out by the retry, with no clue in the log. The model id is not prompt text, so + // it is safe to record. + detail := fmt.Sprintf("(rewrote→%s got %d %s: %s)", dec.Model, status, + orDash(parseErrorType(errBody)), truncate(parseErrorMessage(errBody), 160)) + retry, err := p.sendUpstream(r, originalBody, r.Header) if err != nil { // Retry can't even be sent → relay the buffered original 4xx verbatim. - p.relayBytes(w, status, hdr, errBody, dec.Tier, reason+" mode-b:relay(retry-failed)") + p.relayBytes(w, status, hdr, errBody, dec.Tier, reason+" mode-b:relay-original"+detail) return } defer retry.Body.Close() - p.relay(w, retry, dec.Tier, reason+" mode-b:retried-original") + p.relay(w, retry, dec.Tier, reason+" mode-b:retried-original"+detail) } func (p *Proxy) modeC(w http.ResponseWriter, err error) { @@ -295,6 +316,28 @@ func parseErrorType(b []byte) string { return e.Error.Type } +// parseErrorMessage extracts error.message from an Anthropic error body. It is an +// API-structure message (which feature/model was rejected), never prompt text, so +// it is safe to log — and it turns a silent down-route 400 into a one-line +// diagnosis ("model X does not support the context-1m beta"). +func parseErrorMessage(b []byte) string { + var e struct { + Error struct { + Message string `json:"message"` + } `json:"error"` + } + _ = json.Unmarshal(b, &e) + return e.Error.Message +} + +// truncate bounds a string for a log field, appending an ellipsis if cut. +func truncate(s string, n int) string { + if len(s) <= n { + return s + } + return s[:n] + "…" +} + // sendUpstream builds and executes the upstream request: client headers minus // hop-by-hop, Host + Content-Length set, Accept-Encoding forced to identity so // the SSE stream is plaintext (the gzip-SSE framing hazard, GATE-1). @@ -318,7 +361,13 @@ func (p *Proxy) relay(w http.ResponseWriter, resp *http.Response, tier, reason s copyDownstreamHeaders(w.Header(), resp.Header) setVerdict(w.Header(), tier, reason) w.WriteHeader(resp.StatusCode) - streamFlush(w, resp.Body) + // Tee the response through a usage scanner so the log line can carry the actual + // input/output token counts (for cost-savings math) without buffering the stream. + var scan usageScanner + streamFlush(w, resp.Body, &scan) + if rec := recorderOf(w); rec != nil { + rec.inTokens, rec.outTokens = scan.in, scan.out + } } // relayBytes relays a fully-buffered response (status + headers + body). Used @@ -450,13 +499,17 @@ func pinFromHeader(pol *Policy, h http.Header) string { } // streamFlush copies src→dst flushing after every chunk so SSE events reach the -// client immediately (the gzip/flush framing the experiment proved). -func streamFlush(w http.ResponseWriter, src io.Reader) { +// client immediately (the gzip/flush framing the experiment proved). It also feeds +// each chunk to scan (if non-nil) so token usage is captured without buffering. +func streamFlush(w http.ResponseWriter, src io.Reader, scan *usageScanner) { rc := http.NewResponseController(w) buf := make([]byte, 8<<10) for { n, err := src.Read(buf) if n > 0 { + if scan != nil { + scan.feed(buf[:n]) + } if _, werr := w.Write(buf[:n]); werr != nil { return } @@ -478,6 +531,11 @@ func (discardLogger) Printf(string, ...any) {} type statusRecorder struct { http.ResponseWriter status int + requested string // the client's requested model (canonicalized) + mlTrace string // computed ML signal values (Decision.MLTrace) + used string // the model that actually served the response ("" ⇒ = requested) + inTokens int // response usage.input_tokens (0 if not captured) + outTokens int // response usage.output_tokens ctxTokens int wrote bool } @@ -497,6 +555,81 @@ func (s *statusRecorder) Write(b []byte) (int, error) { func (s *statusRecorder) Unwrap() http.ResponseWriter { return s.ResponseWriter } +// orDash returns "-" for an empty string, so log fields are never blank. +func orDash(s string) string { + if s == "" { + return "-" + } + return s +} + +// recorderOf returns the underlying *statusRecorder if w is one (always, on the +// router's own paths) — for stashing per-request observability (used model, usage). +func recorderOf(w http.ResponseWriter) *statusRecorder { + rec, _ := w.(*statusRecorder) + return rec +} + +// usageScanner extracts input_tokens / output_tokens from a streamed Anthropic +// response (SSE or JSON). It scans each chunk (plus a small carry, so a number +// split across a chunk boundary is still caught) for the token fields and keeps +// the max — output_tokens accumulates across message_delta events, input is +// constant. Best-effort by design: a miss yields 0. It never buffers or blocks the +// stream, and it only reads token COUNTS, never response content. +type usageScanner struct { + tail []byte + in, out int +} + +func (s *usageScanner) feed(b []byte) { + data := b + if len(s.tail) > 0 { + data = append(append([]byte(nil), s.tail...), b...) + } + if v := maxIntField(data, "input_tokens"); v > s.in { + s.in = v + } + if v := maxIntField(data, "output_tokens"); v > s.out { + s.out = v + } + const carry = 64 + if len(data) > carry { + s.tail = append([]byte(nil), data[len(data)-carry:]...) + } else { + s.tail = append([]byte(nil), data...) + } +} + +// maxIntField returns the largest integer value of any `"field": ` occurrence +// in b (0 if none). +func maxIntField(b []byte, field string) int { + key := []byte(`"` + field + `":`) + best := 0 + for i := 0; i < len(b); { + j := bytes.Index(b[i:], key) + if j < 0 { + break + } + k := i + j + len(key) + for k < len(b) && b[k] == ' ' { + k++ + } + n, adv := 0, 0 + for k+adv < len(b) && b[k+adv] >= '0' && b[k+adv] <= '9' { + n = n*10 + int(b[k+adv]-'0') + adv++ + } + if adv > 0 && n > best { + best = n + } + if adv == 0 { + adv = 1 + } + i = k + adv + } + return best +} + // shortSession returns a non-sensitive prefix of the session UUID for the log // (enough to correlate a session's requests; not the full id, no prompt text). func shortSession(id string) string { diff --git a/router/proxy_modeb_gap_test.go b/router/proxy_modeb_gap_test.go index 971a681..99fc52d 100644 --- a/router/proxy_modeb_gap_test.go +++ b/router/proxy_modeb_gap_test.go @@ -95,7 +95,7 @@ func TestProxy_ModeB_RetryTransportFailure_RelaysBufferedOriginal(t *testing.T) if !strings.Contains(string(body), "ROUTED_400_BUFFERED") { t.Errorf("client should receive the buffered original 400 body, got %q", string(body)) } - if r := resp.Header.Get("X-Whittle-Reason"); !strings.Contains(r, "mode-b:relay(retry-failed)") { + if r := resp.Header.Get("X-Whittle-Reason"); !strings.Contains(r, "mode-b:relay-original") { t.Errorf("reason should record the retry-failed relay: %q", r) } } diff --git a/router/proxy_usage_test.go b/router/proxy_usage_test.go new file mode 100644 index 0000000..37b420e --- /dev/null +++ b/router/proxy_usage_test.go @@ -0,0 +1,38 @@ +package router + +import "testing" + +// The usage scanner pulls input/output token counts out of a streamed Anthropic +// SSE response for the cost-savings log fields. +func TestUsageScanner_SSE(t *testing.T) { + var s usageScanner + // message_start carries input + a starting output; message_delta accumulates. + s.feed([]byte(`event: message_start\ndata: {"type":"message_start","message":{"usage":{"input_tokens":1200,"output_tokens":1}}}`)) + s.feed([]byte(`event: message_delta\ndata: {"type":"message_delta","usage":{"output_tokens":47}}`)) + if s.in != 1200 || s.out != 47 { + t.Errorf("got in=%d out=%d, want in=1200 out=47", s.in, s.out) + } +} + +// A token count split across a chunk boundary is still captured (carry buffer). +func TestUsageScanner_SplitChunk(t *testing.T) { + var s usageScanner + s.feed([]byte(`{"usage":{"output_to`)) + s.feed([]byte(`kens":12345}}`)) + if s.out != 12345 { + t.Errorf("split output_tokens not captured: got %d", s.out) + } +} + +func TestMaxIntField(t *testing.T) { + b := []byte(`{"input_tokens":10,"nested":{"input_tokens": 999},"output_tokens":0}`) + if v := maxIntField(b, "input_tokens"); v != 999 { // max across occurrences + t.Errorf("maxIntField(input_tokens) = %d, want 999", v) + } + if v := maxIntField(b, "output_tokens"); v != 0 { + t.Errorf("maxIntField(output_tokens) = %d, want 0", v) + } + if v := maxIntField(b, "absent"); v != 0 { + t.Errorf("absent field should be 0, got %d", v) + } +} diff --git a/router/reconcile.go b/router/reconcile.go index c966ef7..2d81257 100644 --- a/router/reconcile.go +++ b/router/reconcile.go @@ -1,5 +1,7 @@ package router +import "strings" + // feature is one reconcilable capability: how to detect its use in a request and // how to strip it (across body AND headers, atomically). Reconcile applies a // strip only when the target model lacks the capability. @@ -46,17 +48,20 @@ var reconcileFeatures = []feature{ if _, has := r.Body["thinking"]; has { return true } - return r.hasBetaPrefix("interleaved-thinking") || r.hasBetaPrefix("thinking-token-count") + return r.hasBetaContaining("thinking") || hasThinkingContextEdit(r) }, strip: func(r *Request) { // Strip the config, the thinking blocks already in history (or a - // non-thinking target rejects the history), AND the thinking beta - // tokens — a beta token alone causes a 400 on a non-supporting model - // (proven for context-1m; applied by analogy, validate-on-traffic). + // non-thinking target rejects the history), AND every thinking beta. + // Not just known prefixes: dependent betas are an OPEN-ENDED family + // (interleaved-thinking, thinking-token-count, clear_thinking_… — the + // last one confirmed live), and a lone thinking beta whose feature we + // removed 400s ("requires thinking to be enabled"). Any beta naming + // "thinking" must go when thinking is disabled. delete(r.Body, "thinking") stripThinkingFromHistory(r) - r.removeBetaPrefix("interleaved-thinking") - r.removeBetaPrefix("thinking-token-count") + r.removeBetaContaining("thinking") + stripThinkingContextEdits(r) }, }, { @@ -109,6 +114,59 @@ func Reconcile(req *Request, target string) []string { return stripped } +// hasThinkingContextEdit reports whether context_management carries an edit whose +// type references thinking (e.g. clear_thinking_20251015) — such an edit REQUIRES +// thinking to be enabled, so it must be removed when thinking is disabled. +func hasThinkingContextEdit(r *Request) bool { + for _, e := range contextEdits(r) { + if m, ok := e.(map[string]any); ok { + if t, _ := m["type"].(string); strings.Contains(strings.ToLower(t), "thinking") { + return true + } + } + } + return false +} + +// stripThinkingContextEdits drops context_management.edits that require thinking +// (see hasThinkingContextEdit). A leftover such edit 400s ("requires thinking to +// be enabled") once thinking is stripped. If edits becomes empty, context_management +// is removed entirely so an empty edits array can't itself be rejected. +func stripThinkingContextEdits(r *Request) { + edits := contextEdits(r) + if edits == nil { + return + } + kept := edits[:0:0] + for _, e := range edits { + if m, ok := e.(map[string]any); ok { + if t, _ := m["type"].(string); strings.Contains(strings.ToLower(t), "thinking") { + continue + } + } + kept = append(kept, e) + } + if len(kept) == len(edits) { + return + } + cm, _ := r.Body["context_management"].(map[string]any) + if len(kept) == 0 { + delete(r.Body, "context_management") + } else { + cm["edits"] = kept + } +} + +// contextEdits returns body.context_management.edits as a slice, or nil. +func contextEdits(r *Request) []any { + cm, ok := r.Body["context_management"].(map[string]any) + if !ok { + return nil + } + edits, _ := cm["edits"].([]any) + return edits +} + // stripThinkingFromHistory removes thinking / redacted_thinking blocks from every // message's array content. A message emptied by this (content becomes []) is // dropped entirely — a degenerate empty turn is itself invalid. Any adjacency diff --git a/router/reconcile_test.go b/router/reconcile_test.go index a409358..eb6c2bb 100644 --- a/router/reconcile_test.go +++ b/router/reconcile_test.go @@ -29,6 +29,51 @@ func has(ss []string, s string) bool { return false } +// When thinking is disabled on the target, EVERY thinking beta must go — not just +// the known prefixes. A lone dependent beta (clear_thinking_…, confirmed live) +// 400s with "requires thinking to be enabled". +func TestReconcile_StripsOpenEndedThinkingBetas(t *testing.T) { + r := mkReq(t, `{"model":"claude-opus-4-8","thinking":{"type":"enabled"},"messages":[{"role":"user","content":"hi"}]}`, + "context-1m-2025-08-07,clear_thinking_20251015,interleaved-thinking-2025-05-14,thinking-token-count-2025-01-01") + Reconcile(r, "claude-sonnet-4-5-20250929") // sonnet family floor: thinking unsupported + if beta := r.Headers.Get("anthropic-beta"); strings.Contains(strings.ToLower(beta), "thinking") { + t.Errorf("every thinking beta must be stripped, got %q", beta) + } + if _, has := r.Body["thinking"]; has { + t.Error("thinking config must be removed") + } +} + +// A context_management edit that REQUIRES thinking (clear_thinking_…, confirmed +// live via headless claude) must be dropped when thinking is disabled — a leftover +// one 400s. Non-thinking edits stay. +func TestReconcile_StripsThinkingContextEdit(t *testing.T) { + r := mkReq(t, `{"model":"claude-opus-4-8","thinking":{"type":"enabled"}, + "context_management":{"edits":[{"type":"clear_thinking_20251015"},{"type":"clear_tool_uses_20250919"}]}, + "messages":[{"role":"user","content":"hi"}]}`, "") + Reconcile(r, "claude-sonnet-4-5-20250929") // sonnet floor: thinking off + cm, ok := r.Body["context_management"].(map[string]any) + if !ok { + t.Fatal("a surviving non-thinking edit should keep context_management") + } + edits := cm["edits"].([]any) + if len(edits) != 1 || edits[0].(map[string]any)["type"] != "clear_tool_uses_20250919" { + t.Errorf("only the thinking edit should be dropped, got %v", edits) + } +} + +// If every edit required thinking, context_management is removed entirely (an +// empty edits array could itself be rejected). +func TestReconcile_DropsContextMgmtWhenAllThinking(t *testing.T) { + r := mkReq(t, `{"model":"claude-opus-4-8","thinking":{"type":"enabled"}, + "context_management":{"edits":[{"type":"clear_thinking_20251015"}]}, + "messages":[{"role":"user","content":"hi"}]}`, "") + Reconcile(r, "claude-sonnet-4-5-20250929") + if _, has := r.Body["context_management"]; has { + t.Error("context_management with only thinking edits should be removed") + } +} + // B1: an UNKNOWN model is fully capable — Reconcile strips nothing, only sets // the model. The zero-value trap (strip everything / unroutable) must not happen. func TestReconcile_UnknownModelStripsNothing(t *testing.T) { diff --git a/router/request.go b/router/request.go index da66839..1346326 100644 --- a/router/request.go +++ b/router/request.go @@ -12,7 +12,7 @@ import ( // reconciliation. Body is the parsed JSON as a generic map so strip transforms // can edit arbitrary fields and messages; Headers is the outbound header set // (anthropic-beta lives here). The proxy serializes Body back to bytes and -// recomputes Content-Length AFTER reconciliation (docs/ROUTER_RECONCILIATION.md +// recomputes Content-Length AFTER reconciliation (docs/ROUTER.md §6 // "Strip mechanics": parse → strip → re-serialize, accepting the prompt-cache // miss on routed requests only). type Request struct { @@ -113,6 +113,48 @@ func (r *Request) removeBetaPrefix(prefix string) bool { return true } +// hasBetaContaining reports whether any beta token contains substr (case-insensitive). +func (r *Request) hasBetaContaining(substr string) bool { + substr = strings.ToLower(substr) + for _, t := range r.betaTokens() { + if strings.Contains(strings.ToLower(t), substr) { + return true + } + } + return false +} + +// removeBetaContaining drops every beta token containing substr (case-insensitive) +// and removes the header if empty. Used to strip an OPEN-ENDED family of dependent +// betas — e.g. every "thinking" beta (interleaved-thinking, thinking-token-count, +// clear_thinking_…) when the thinking capability is disabled — since a lone +// dependent beta whose feature we removed 400s ("requires thinking to be enabled"). +func (r *Request) removeBetaContaining(substr string) bool { + toks := r.betaTokens() + if len(toks) == 0 { + return false + } + substr = strings.ToLower(substr) + kept := toks[:0:0] + removed := false + for _, t := range toks { + if strings.Contains(strings.ToLower(t), substr) { + removed = true + continue + } + kept = append(kept, t) + } + if !removed { + return false + } + if len(kept) == 0 { + r.Headers.Del(betaHeader) + } else { + r.Headers.Set(betaHeader, strings.Join(kept, ",")) + } + return true +} + // ---- message/content helpers (generic map[string]any shape) ---- // messages returns the mutable []any of message maps, or nil. diff --git a/router/requested_default_test.go b/router/requested_default_test.go new file mode 100644 index 0000000..3669da9 --- /dev/null +++ b/router/requested_default_test.go @@ -0,0 +1,58 @@ +package router + +import ( + "strings" + "testing" +) + +const requestedPolicy = `{ + "version":1, + "tiers":[{"name":"fast","model":"claude-haiku-4-5"},{"name":"smart","model":"claude-opus-4-8"}], + "default":"requested","inspect":{"scope":"full"}, + "routes":[{"name":"down","when":{"keywords":["trivial"]},"to":"fast"}] +}` + +// default:"requested" — no route matched → keep the client's model, a guaranteed +// no-op. Every rewrite must come from an explicit rule (fail-open applied to +// routing itself). +func TestRequestedDefault_NoMatchIsNoOp(t *testing.T) { + p, _ := mustLoad(t, requestedPolicy) + d := Decide(Signals{RequestedModel: "claude-opus-4-8", RecentText: "hard novel work", SessionID: "s"}, + p, nil, NewMemSessionStore(), "") + if !IsNoOp(d, Signals{RequestedModel: "claude-opus-4-8"}) { + t.Fatalf("unmatched traffic must be a no-op, got tier=%q model=%q", d.Tier, d.Model) + } + if !strings.Contains(d.Reason, "default:requested") { + t.Errorf("reason should say default:requested, got %q", d.Reason) + } +} + +// The critical mixed-model property: a cheap-model request (Claude Code +// background/title tasks) is NOT up-routed by the default — it stays cheap. +func TestRequestedDefault_NeverUpRoutesBackgroundTraffic(t *testing.T) { + p, _ := mustLoad(t, requestedPolicy) + sig := Signals{RequestedModel: "claude-haiku-4-5", RecentText: "novel work", SessionID: "s"} + d := Decide(sig, p, nil, NewMemSessionStore(), "") + if !IsNoOp(d, sig) { + t.Fatalf("a haiku request with no matching rule must stay haiku, got model=%q", d.Model) + } +} + +// Explicit rules still fire in both directions. +func TestRequestedDefault_RulesStillRoute(t *testing.T) { + p, _ := mustLoad(t, requestedPolicy) + d := Decide(Signals{RequestedModel: "claude-opus-4-8", RecentText: "a trivial ask", SessionID: "s"}, + p, nil, NewMemSessionStore(), "") + if d.Tier != "fast" { + t.Fatalf("explicit rule must still down-route, got %q (%s)", d.Tier, d.Reason) + } +} + +// Validation: "requested" is reserved — legal as default, illegal as a tier name. +func TestRequestedDefault_Validation(t *testing.T) { + if _, _, err := Load([]byte(`{"version":1, + "tiers":[{"name":"requested","model":"m"}],"default":"requested", + "inspect":{"scope":"full"},"routes":[]}`)); err == nil { + t.Fatal("a tier named 'requested' must be rejected") + } +} diff --git a/router/rule.go b/router/rule.go index 05c8642..46e17fe 100644 --- a/router/rule.go +++ b/router/rule.go @@ -12,7 +12,7 @@ import ( // predicates in one node is invalid — enforced recursively by validate(), not // by the type, because "operator-as-key unmarshals 1:1 into an all-optional // struct" is the entire reason this grammar needs no custom parser (see -// docs/ROUTER_POLICY_SCHEMA.md §1-2). +// docs/ROUTER.md §4). // // Strict-key decoding (json.Decoder.DisallowUnknownFields) turns a typo'd leaf // key ("keywrods") into a load error rather than a silently-dropped predicate @@ -112,7 +112,7 @@ func (r *Rule) combinatorCount() int { // isMLLeaf reports whether this node's single leaf needs a model call. Used by // the evaluator to order cheap heuristic children before ML children so an -// already-decided node never pays for a classifier (docs ROUTER_DESIGN §2.3). +// already-decided node never pays for a classifier (docs/ROUTER.md §5). func (r *Rule) isMLLeaf() bool { ls := r.leaves() if len(ls) != 1 { diff --git a/router/server.go b/router/server.go index 2f59972..f606aea 100644 --- a/router/server.go +++ b/router/server.go @@ -48,6 +48,11 @@ func ListenAndServe(addr, policyPath string, lg Logger) error { if u := os.Getenv("WHITTLE_ROUTER_MODEL_URL"); u != "" { cl = ml.New(u) lg.Printf("router: smart mode ON (classifier sidecar = %s)", u) + } else { + lg.Printf("router: smart mode OFF — ML signals (domain/embedding/complexity) are inert; set WHITTLE_ROUTER_MODEL_URL=http://127.0.0.1:45872 to enable") + if pol != nil && policyUsesML(pol) { + lg.Printf("router: WARNING: the policy references ML signals, which will never fire in this mode — requests they guard fall to the default") + } } px := NewProxy(pol, cl, NewMemSessionStore(), lg) diff --git a/router/signals.go b/router/signals.go index 1a34f93..52cb6ad 100644 --- a/router/signals.go +++ b/router/signals.go @@ -147,13 +147,24 @@ func userTextOf(content json.RawMessage) string { } var parts []string for _, b := range decodeBlocks(content) { - if b.Type == "text" && b.Text != "" { + if b.Type == "text" && b.Text != "" && !isInjectedBlock(b.Text) { parts = append(parts, b.Text) } } return strings.Join(parts, "\n") } +// isInjectedBlock reports whether a text block is framework-injected rather than +// user-authored. Claude Code prepends blocks (context, memory, +// task nudges) INSIDE user messages; captured live, they dominate the joined text +// and make the classifier score the reminder instead of the human's question +// ("what day is today?" read as computer-science@0.99 because the block above it +// mentioned emails and dates). The Signals contract is user-authored text only, +// so these blocks are excluded from BOTH classifier and keyword input. +func isInjectedBlock(text string) bool { + return strings.HasPrefix(strings.TrimSpace(text), "") +} + // decodeString returns the string value of a RawMessage if it is a JSON string. func decodeString(raw json.RawMessage) (string, bool) { t := bytes.TrimSpace(raw) diff --git a/router/validate.go b/router/validate.go index 145f6ed..4726f71 100644 --- a/router/validate.go +++ b/router/validate.go @@ -7,7 +7,7 @@ import ( ) // validate checks a decoded Policy against the schema rules -// (docs/ROUTER_POLICY_SCHEMA.md §4). It returns non-fatal warnings and fatal +// (docs/ROUTER.md §4). It returns non-fatal warnings and fatal // errors separately; Load turns any error into a failed load. Every problem // names its location so a hand-author can find it. // @@ -51,8 +51,8 @@ func (v *validator) validateTiers() { switch { case t.Name == "": v.errf("tiers[%d]: missing name", i) - case t.Name == keepTier: - v.errf("tiers[%d]: %q is a reserved keyword and cannot be a tier name", i, keepTier) + case t.Name == keepTier || t.Name == requestedDefault: + v.errf("tiers[%d]: %q is a reserved keyword and cannot be a tier name", i, t.Name) case seen[t.Name]: v.errf("tiers[%d]: duplicate tier name %q", i, t.Name) } @@ -69,11 +69,14 @@ func (v *validator) validateDefault() { return } if v.p.Default == keepTier { - v.errf("default: cannot be %q — the terminal fallback must be a real tier", keepTier) + v.errf("default: cannot be %q — use %q to keep the client's model, or a real tier", keepTier, requestedDefault) return } + if v.p.Default == requestedDefault { + return // reserved: no route matched → keep the requested model (no-op) + } if v.p.tierRank(v.p.Default) < 0 { - v.errf("default: %q is not a defined tier", v.p.Default) + v.errf("default: %q is not a defined tier (or %q)", v.p.Default, requestedDefault) } } @@ -293,6 +296,9 @@ func (v *validator) validateSignals() { if len(d.Categories) == 0 { v.errf("%s (%s): no categories", loc, d.Name) } + if d.MinMass < 0 || d.MinMass > 1 { + v.errf("%s (%s): min_mass must be in [0,1] (got %g)", loc, d.Name, d.MinMass) + } for _, c := range d.Categories { if !mmluCategories[c] { v.warnf("%s (%s): %q is not a known MMLU-Pro category — the classifier will never emit it, so this is inert", loc, d.Name, c)