Router: out-of-the-box policies, battle-tested reconciliation, calibrated signals, one design doc - #3
Merged
Merged
Conversation
Out-of-the-box policies so users don't start from a blank file: - Embedded presets (go:embed router/policies/*.json): `coding` (balanced — hard work → opus, quick edits → haiku, else sonnet; uses ML signals when smart mode is on, degrades to keywords/context otherwise) and `heuristic` (same tiering, no ML signals — works with no sidecar). - `whittle policy` command: `list`, `show [name]`, `init [name] [-force] [-o path]` (writes ~/.whittle/router.json, refuses to clobber without -force), and `validate <file>` (runs the real loader — surfaces errors + cost-lint warnings, high-value for authoring). - TestPresets_AllValid loads every embedded preset, so a shipped policy can never be invalid. README documents the flow. Full CI sequence (gofmt/vet/build/test/bench) green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… IDs The shipped presets hardcoded bare model IDs (claude-sonnet-4-5) that Anthropic rejects — it needs the full dated form (claude-sonnet-4-5-20250929) — so every rewrite 4xx'd and Mode B silently retried the original: correct answers, zero routing, ~2x latency, no clue in the log. Two fixes: 1. `whittle policy init` now auto-detects the real model IDs the account uses from Claude Code's config (~/.claude.json) and substitutes them into the preset tiers by family. Ranking: an id used as a "model": value (proven-valid, what Claude Code sends) > a dated id > a bare/lower one. If nothing is detected, it writes placeholders and warns LOUDLY to edit them. So the coding preset works on first request instead of failing. (Verified: picks haiku-4-5-20251001 / sonnet-4-5-20250929 / opus-4-7 — correctly choosing the active 4-7 over 4-8.) 2. Failures are no longer silent: the per-request log line now includes the client's requested model, and a Mode-B retry records exactly which rewrite the upstream rejected — e.g. reason `... mode-b:retried-original(rewrote→ claude-sonnet-4-5 got 404:not_found_error)`. Model IDs are not prompt text. Tests for the detection ranking + substitution; updated the mode-b reason assertion. Full CI (gofmt/vet/build/test/bench) green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…rror diag Root cause of the persistent mode-b-on-every-request (even with valid model ids): baselineCaps is keyed by exact canonical model (claude-opus-4-8, claude-sonnet-5), but real accounts run other versions (claude-opus-4-7, claude-sonnet-4-5). A miss returned the FULLY-CAPABLE sentinel → reconciliation stripped NOTHING → an opus[1m] request's context-1m beta was forwarded to the down-route sonnet target, which rejects it → 400 → Mode B retried the original. Net: no routing, ~2x latency. Fix: - capsFor now falls back to a conservative per-FAMILY floor (haiku/sonnet/opus) before the fully-capable sentinel. An unrecognized same-family model gets the family's known capabilities (sonnet: no context-1m), so reconciliation strips the incompatible beta instead of forwarding it. Only a truly unknown family stays fully-capable. Aligns with the table's existing "conservative" philosophy. - Mode-B now records the exact upstream rejection in the reason (model, status, error.type, and a bounded error.message) — a silent down-route 400 becomes a one-line diagnosis. Error messages are API-structure text, never prompt text. Tests for the family fallback; full CI green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… context-1m) Live confirmation via the new diagnostic: an unknown sonnet rejected BOTH the context-1m beta and adaptive thinking. The sonnet family floor now supports nothing optional — over-stripping on a down-route is safe, under-stripping 400s. Only opus (top tier, up-routed to) keeps the full feature set. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Live diag caught the next dependent beta: after stripping thinking config + interleaved-thinking + thinking-token-count, a clear_thinking_20251015 beta remained → 400 'requires thinking to be enabled'. Dependent thinking betas are an open-ended family, so the thinking strip now removes ANY beta whose token contains 'thinking' (new removeBetaContaining helper), and detects them the same way. Regression test covers clear_thinking + the two known prefixes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ghest model version Reproduced end-to-end with headless claude through a body-capturing proxy: the persistent clear_thinking 400 was NOT a beta — it was a request BODY field, context_management.edits[].type = clear_thinking_20251015, which requires thinking. Stripping thinking left it dangling → 400. Fix: the thinking strip now also removes context_management edits whose type references thinking (dropping context_management entirely if that empties it), and detects them. Also: model-id detection now prefers the highest version over the config's "model" field, which proved stale — it named opus-4-7 while Claude Code actually sends opus-4-8, so `policy init` had made the smart tier a down-version rewrite (4-8→4-7) that 400'd on an unsupported mid-conv system role. Version-first picks 4-8. Tests for both. Verified live: opus[1m] requests now down-route to sonnet at 200, single round-trip, no mode-b, on 26k-token contexts. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…an't import it app.py does 'import route' (M5), but route.py was missing from the //go:embed list, so 'whittle setup' materialized a sidecar that fails at startup — breaking BOTH router smart mode and compression. Added it + a guard test that every module app.py imports is embedded. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Opus adversarial review found two blocking issues: - context_tokens>120000 in hard-work pinned the whole session to opus (once a few file reads cross 120k, every trivial turn routes to opus). Long context != hard reasoning → removed. - domain:non-technical → fast was a quality inversion: sparse mid-task and substantive-but-casual turns classify 'other' → haiku (weakest model on the hardest/meta turns). Domain doesn't earn its place in a coding workflow → removed the signal and the down-route. Also: added 'migration' (migrate didn't substring-match it), dropped the 'architecture' keyword (redundant with the embedding signal), and enlarged the complexity prototype banks 4→8 each for better recall. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The per-request log line now carries: - "model": the model that ACTUALLY served the response (the tier's model on a routed rewrite; the requested model on no-op/passthrough/mode-b-retry) — distinct from "tier" and "requested". - "in_tokens" / "out_tokens": real usage.input_tokens / output_tokens scanned from the streamed Anthropic response (best-effort, carry-buffered across chunk boundaries, never buffers the stream or reads content — token counts only). Together with "requested", this is enough to compute per-request savings offline: cost(requested, in, out) − cost(model, in, out). Verified live via headless claude: requested=opus-4-8, model=sonnet on a down-route, in/out captured. Tests for the usage scanner incl. a count split across chunks. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…brated default preset
Three first-principles improvements over both our prior design and vSR's, each
validated empirically against the live mmBERT models:
1. Domain signals now threshold PROBABILITY MASS over the category set
(min_mass), using the classifier's full softmax (sidecar returns `probs`).
One scalar subsumes vSR's entropy ladder: mass concentrates only on a
confident in-set classification; split-across-in-set passes with no top-2
special case; an ambiguous distribution fails the threshold and falls to the
safe middle tier (cost-first — deliberately inverting vSR's accuracy-first
"very-uncertain → escalate"). Argmax membership kept as legacy/no-probs
fallback. Measured: kills the ambiguity failures (health 0.51 → no escalate)
while catching confident quant (math 0.98 → escalate).
2. Keywords are now whole-word/phrase (non-alphanumeric boundaries; "c++" still
works). Minimal complete fix for the observed within-word collisions
("migration"↛"immigration", "refactor"↛"refactored") — chosen over vSR's
bm25/ngram/fuzzy which add config surface without fixing the residual
(phrase-in-context) failure class. Presets list derived-form variants.
3. New `default` preset (mixed-use employee), signal composition chosen by
measurement: complexity:hard OR quantitative-mass≥0.7 → opus (the two signals
have complementary failure modes: the theorem-proof misses complexity but is
math 0.98; the misclassified diff-eq misses domain but is complexity-hard);
easy AND NOT high-stakes → haiku (guard blocks "GDPR yes/no" from dropping).
12/12 on the adversarial battery incl. all previous failures. Also fixed the
review-flagged min_band_jump=2 session-anchor bug in coding/heuristic presets.
Docs: schema §2/§7 updated (word-boundary semantics, min_mass with the formal
argument); new docs/ROUTER_CALIBRATION_RESEARCH.md is the point-1 research brief
(labeled tier-sufficiency dataset; notes even vSR has zero coding datasets).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replaces 1,343 lines across five lifecycle-era documents (ROUTER_DESIGN, ROUTER_POLICY_SCHEMA, ROUTER_RECONCILIATION, ROUTER_IMPLEMENTATION_PLAN, ROUTER_CALIBRATION_RESEARCH) with one 346-line docs/ROUTER.md, structured on the ARCHITECTURE.md pattern (matklad): bird's-eye view → request lifecycle → policy reference → signals with the exact math and rationale → reconciliation → failure model → observability → codemap → prior art → honest limitations. Durable content only — the review tables, milestone ACs, GATE experiment logs, and revision histories that made the old docs implementation-lifecycle artifacts are gone (they live in git history). The calibration research brief's substance is folded into §11 as the roadmap. Credits are explicit (§10): vLLM Semantic Router's two mmBERT models, the bank-score blend and contrastive margin math, and the signal-taxonomy idea are borrowed and named; the architecture around them (single-file boolean policy, mass thresholding with the cost-first uncertainty direction, capability reconciliation, the A/B/C fail-open model, cost observability) is presented on its own terms. All code/README references repointed to docs/ROUTER.md sections; no dangling links. CI sequence green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- docs/ROUTER.md §10 now cites vLLM Semantic Router's whitepaper ("Signal-driven
decision routing for mixture-of-modality models",
https://vllm-semantic-router.com/white-paper) alongside the repo link, both at
the credits and at the signal-taxonomy attribution.
- README: the router blurb and the Model routing section were still describing
the pre-M5 design ("intent classifier and few-shot classify"). Rewritten to
reflect what shipped: multi-signal routing (trained 14-subject classifier with
probability-mass thresholding and the uncertainty-falls-to-the-middle
property, contrastive difficulty, embedding similarity, whole-word keywords),
auto-detected model ids in policy init, measurable savings via the
requested/served/token log, and a brief credits line with the whitepaper link.
docs/ROUTER.md linked from both the top blurb and the section.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Live session evidence: "can you dive deep understand and share the architecture?" scores complexity +0.171 (hard → opus) alone, but +0.025 (medium → sonnet) once the 3-turn inspect window joined it with "what is date today?" and an explain-this-file turn. Window averaging systematically suppresses mid-session ML escalation — the first adversarial review predicted exactly this (deferred then, proven now). Fix: split text scopes by signal nature. Keywords keep scanning the inspect window (persistence is a feature — a hard keyword two turns back keeps protecting); domain/embedding/complexity classify LastUserText only (a clean sample), falling back to the window when the last turn has no user text (tool-result-only turns). Regression tests pin both scopes; the user's exact session replayed end-to-end now routes turn 3 to opus. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The old posture (default: main) rewrote the model precisely when we had ZERO evidence about a request (no route matched) — an affirmative action justified by nothing, contradicting whittle's own fail-open principle. New reserved default value "requested": unmatched traffic is a guaranteed no-op passthrough, so every model change (up or down) traces to a rule the author wrote. Deliberately chosen over hardcoding the top tier: Claude Code also sends cheap-model background requests, which a fixed-tier default would silently up-route. - engine: default:"requested" short-circuits to a no-op decision (no session write, no stickiness); IsNoOp treats an empty rewrite target as no-op. - validation: "requested" legal as default, reserved as a tier name. - default preset + docs updated to the new posture (design principle #4 reworded: "No evidence → no rewrite"); tests pin the no-op, the never-up-route-background property, rules-still-fire, and the reserved-name rejection. Trade-off, stated plainly: with default:"requested" the un-flagged middle stays on the user's model (quality-conservative, lower savings); setting a tier default (e.g. "main") remains the aggressive-savings posture and the coding preset keeps it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
"what is date today?" rode the requested model instead of haiku: it scored -0.118 (medium) because the easy prototype bank was ALL editing tasks (fix typo, rephrase, reformat) with no lookup exemplar — a class blind spot, not a threshold problem. Added four factual-lookup exemplars and re-probed the full battery: the lookup class flips decisively (-0.38..-0.50 → haiku), hard controls hold (+0.28..+0.37 → opus), substantive explain/code questions correctly stay medium (→ requested), and the quantitative-domain residual is unchanged (route order: escalate checks first). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…authoring ceiling
'where is mount everest located' scored -0.107 (medium → requested): the lookup
exemplars covered what-questions only. Added where/who/when forms. Measured:
everest flips to easy (-0.150); code where-questions correctly stay medium
('where is the memory leak' +0.068 — the critical guard); hard/writing hold.
But 'who wrote pride and prejudice' (-0.122) and 'when did ww2 end' (-0.140)
still miss DESPITE a near-identical exemplar — object tokens drag cosine under
the bar. Exemplar patching is measurably sub-linear; further coverage belongs to
the calibration loop (mined prototypes + data-tuned threshold), not more
hand-typed phrases. Misses cost savings only (ride the requested model).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Founder call: start conservative and expand with data. Down-routing now requires the domain and complexity signals to AGREE: confidently-casual (other mass ≥ 0.9) ∧ easy → haiku; confidently-casual ∧ medium → sonnet; EVERYTHING else — coding trivia, low-confidence lookups, tone work, all CS traffic — keeps the requested model. Up-routes unchanged (hard OR confident-quantitative → smart). The asymmetry is deliberate: a down-route needs two concordant signals, an up-route needs one — misroute risk is priced by direction. The mass≥0.9 casual gate mathematically subsumes the old high-stakes guard (mass(other) ≥ 0.9 ⟹ stakes-mass ≤ 0.1), so the guard signal is removed and the policy gets simpler. E2E battery 11/11 against measured expectations: date/ capital → haiku; slack-draft/brainstorm → sonnet; everest(0.62)/typo(CS)/ paragraph-flow(0.84)/rephrase-tone(psychology) → untouched; theorem/contract → opus. Preset description + ROUTER.md updated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A user ran `whittle route` without WHITTLE_ROUTER_MODEL_URL and saw everything
fall to default:requested with no indication why — smart-off was silent (only a
BROKEN sidecar tagged the reason; a DISABLED one showed nothing).
- Decision.MLTrace records every ML signal actually computed, with its raw value
("cplx:reasoning=-0.471 dom=other@1.00"), "off" (smart mode disabled) or "err"
(sidecar failure); the per-request log line carries it as "signals". Lazy eval
unchanged — only consulted signals appear, so the field also shows when
heuristics decided first. This is simultaneously the instrumentation the
bank-mining/calibration loop needs (margins per request, free).
- Startup now logs smart mode OFF explicitly, plus a WARNING when the loaded
policy references ML signals that can never fire in that mode.
Verified live both ways: smart-on routes "what day is today?" → casual-easy →
haiku with signals in the line; smart-off shows the warning pair + off markers.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
User-reported with smart mode ON: "what day is today?" fell to default:requested instead of casual-easy→haiku. Captured the actual wire request: Claude Code PREPENDS <system-reminder> text blocks INSIDE the user message (userEmail, currentDate, task nudges), and userTextOf joined all text blocks — so the classifier scored the reminder blob (computer science@0.99, margin -0.106), not the 18-char human question (other@1.00, -0.471). The turn-level dilution bug, recurring at the block level. Fix: text blocks starting with <system-reminder> are framework-injected, not user-authored — excluded from signal text entirely (classifier AND keyword input; reminder content could false-trigger keywords too). A reminder-only turn contributes no user text, like a tool_result-only turn. Regression tests use the exact captured message structure. Verified live with headless claude through the router: the main request now routes route:casual-easy → haiku with signals=-0.471/other@1.00; the internal title-gen background request correctly stays on its requested model. (Diagnosis took minutes because of the signals log field added one commit ago — the observability investment paying for itself immediately.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Live log showed dom=other@0.90 with casual-easy NOT firing — self-contradictory until you realize the gate is >=0.9 and the true mass was just under, hidden by 2-decimal display. The trace now records each domain leaf's EVALUATED mass against its gate at full precision (dom:casual=0.897/0.90), once per signal, and the argmax confidence at 3 decimals. Boundary misses are now unambiguous. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… text Second consecutive real-traffic boundary miss at 0.89x, decomposed by probe: 'aisa's biggest country' scores other=0.899 while 'asia's...' scores 0.983 — the typo is an unknown token leaking 0.087 mass to economics. Real human text carries typos/fragments that systematically shave ~0.05-0.10 versus the clean canonical phrasings the 0.90 gate was eyeballed on. Lowered to 0.85: the conjunction still requires complexity:easy|medium, and the worst measured near-boundary admit (paragraph-flow, 0.84 — still excluded) would land on sonnet, not haiku, so the quality inversion remains impossible. Regression battery: typo'd + clean casual → haiku; writing, code-trivia, legal guards hold. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
Author
Update: live-traffic calibration session (commits 16–21)Routing real Claude Code traffic and interrogating individual log lines surfaced — and fixed — a chain of issues no synthetic battery caught:
Every change regression-gated (batteries against measured expectations) and verified end-to-end with headless |
coding/heuristic presets never went through the adversarial-review + live- calibration loop the default did — shipping uncalibrated alternatives alongside a calibrated default invites worse out-of-the-box outcomes and triples the maintenance surface. Removed; 'default' is now the sole built-in (and the policy init/show default name). Added router/policies/default.md: a small companion explaining what the policy does (the tier table, the two-concordant- signals rule) and how to customize (models, savings lever, banks, gates, the signals log field, escape hatch). All references cleaned (README, ROUTER.md, usage text, tests). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
Everything between "the router exists" (#2) and "an employee can install it and it just works": built-in policies with a
whittle policycommand, the reconciliation fixes found by routing real Claude Code traffic, empirically calibrated ML signals (including one mechanism of our own), and a single consolidated design doc.Out-of-the-box policies (
whittle policy)default(mixed-use),coding,heuristic(no-ML) — withlist/show/init/validatesubcommands.validateruns the real loader; a test guarantees no shipped preset can ever be invalid.policy initauto-detects the model IDs your account actually uses from Claude Code's config (full dated forms — bare IDs 404). The original starter shipped placeholder IDs that made every rewrite fail silently behind the Mode-B retry: correct answers, zero routing, 2× latency.Battle-tested against real traffic (headless-claude reproduction)
Routing live opus[1m] sessions surfaced a chain of silent down-route 400s, each fixed + regression-tested:
clear_thinking_…confirmed live), and thecontext_managementedits that require thinking (a body field, not a beta — found by capturing the reconciled request).route.pywas missing fromgo:embed, crash-looping the whole sidecar on fresh installs (guarded by test now).in_tokens/out_tokensscanned from the streamed response (no buffering, counts only): per-request savings =cost(requested) − cost(model).Calibrated signals (probed against the live mmBERT models)
min_mass): the sidecar returns the classifier's full softmax; a leaf fires iff the probability mass on its category set clears the threshold. One scalar replaces an entropy ladder — and inverts the default cost-first: an ambiguous classification falls to the middle tier, never up to the expensive one. Split-across-set passes with no top-2 special case; argmax remains the no-probs fallback.migrationno longer fires on immigration,refactoron refactored;c++still works. Chosen over bm25/ngram/fuzzy as the minimal complete fix for the observed failure class.defaultpreset's composition is measured, not guessed: probing showedcomplexityanddomainhave complementary failure modes (a theorem proof reads "medium" to prototypes but classifies math 0.98; a misclassified equation still scores complexity-hard) — so escalation ORs them. 12/12 on the adversarial battery, including every previously-failing case. Two opus adversarial reviews folded (incl. amin_band_jumpbug that could anchor whole sessions on one tier).One design doc
Five lifecycle-era docs (1,343 lines of review tables and milestone logs) →
docs/ROUTER.md(346 lines, ARCHITECTURE.md-style): lifecycle → policy reference → signal math with rationale → failure model → codemap → credits → honest limitations & calibration roadmap. Credits are precise: the two mmBERT models, the bank-score/margin math, and the signal taxonomy come from vLLM Semantic Router (whitepaper); the single-file boolean policy, mass thresholding, capability reconciliation, and the A/B/C fail-open model are whittle's own. README rewritten to match.Verify
gofmt/vet/build/test -race/benchgreen. Live-verified end-to-end: real Claude Code traffic routes cleanly across all three tiers (no mode-b), ML signals fire on keyword-less hard prompts, savings visible per-request in the log.🤖 Generated with Claude Code