This document defines how to measure whether the router actually saves cost without losing quality. It is a plan, not a claim of measured savings.
| Hypothesis | Question | Primary metric |
|---|---|---|
| H1 | Does per-turn effort routing reduce cost on routine work without lowering acceptance? | Full cost, acceptance rate |
| H2 | Does the cache/economics gate avoid downgrades that would cost more than they save? | Net cost vs fixed baseline |
| H3 | Does turn-pinning beat per-call model switching on long sessions? | Cache rebuild tokens, total cost |
| H4 | Do upgrade triggers recover hard tasks without turning every turn into an upgrade? | Quality on hard strata, upgrade rate |
| H5 | Does persisted session state survive codex resume and compaction correctly? |
Correct pinned model, stale-state failures |
| H6 | Does the guard layer fail safe when prices, usage or Jev are missing? | Fallback behavior, UNKNOWN handling |
Quality is evaluated before cost. A cheaper run that fails acceptance is not a saving.
| Arm | Model | Effort | Purpose |
|---|---|---|---|
| A. Fixed baseline | strong model | high | Current realistic baseline |
| B. Effort-only | strong model | routed per turn | Isolate the cache-safe lever |
| C. Model + effort | turn-pinned, routed | routed per turn | Full router |
| D. Naive per-call model | routed every call | high | Demonstrate cache rebuild cost |
| E. Fixed cheap | cheap model | high | Lower-bound quality control |
D is a diagnostic arm, not a recommended product mode. Run it only on a small subset because it can be expensive and noisy.
Use paired tasks: the same repository snapshot, prompt, tool environment and acceptance tests for every arm. Keep the task small enough to run repeatedly, but varied enough to cover both simple and hard work.
| ID | Task | Difficulty | Acceptance |
|---|---|---|---|
| T01 | Rename one symbol across two files | trivial | build + targeted tests |
| T02 | Add a docstring to one exported function | trivial | lint + typecheck |
| T03 | Fix a typo in a config value | trivial | config test |
| T04 | Fix one failing unit test with a clear stack | easy | failing test turns green |
| T05 | Add a missing unit test for an existing function | easy | new test passes |
| T06 | Add a small CLI flag and its test | medium | hidden test + existing suite |
| T07 | Extract a helper into a new module | medium | build + all tests |
| T08 | Debug an unknown-cause failure with a misleading stack | hard | hidden regression test |
| T09 | Fix a concurrency race | hard | stress test + review |
| T10 | Tighten an auth/permission check | hard | security test + review |
| T11 | Add a backward-compatible migration | hard | migration + rollback test |
| T12 | Update an API across many files | long | repo-wide test + review |
Add 20–30 real tasks from a public benchmark such as SWE-bench Verified, split into easy/medium/hard and filtered to tasks that fit the local tool environment. Do not mix benchmark tasks and synthetic tasks in one aggregate without showing them separately.
The routing unit is a turn, so the same task can behave differently depending on session shape.
| Shape | Description | What it tests |
|---|---|---|
| S1 One-shot simple | one user turn, 2–5 tool calls | cheap work should route down |
| S2 One-shot hard | one turn, 20–50 tool calls | per-turn effort cannot adapt inside the loop |
| S3 Ten-turn mixed | easy and hard turns alternate | model pin, effort changes, cache reuse |
| S4 Long context | 100k+ input, many turns | downgrade gate and cache rebuild |
| S5 Resume | exit and resume after 5 minutes | persisted pin is reused |
| S6 Long idle | resume after 30+ minutes | cache TTL resets guards |
| S7 Compaction | force context compaction | cache and effort provenance reset |
| S8 Failure/retry | upstream failure then retry | dedupe and upgrade trigger |
| S9 Unsupported effort | model without effort updates | request-level fallback |
| S10 Manual override | user says use ... |
override bypasses gates |
Run S1–S3 for every task where practical. Run S4–S10 as focused scenarios on a smaller task subset.
Quality:
- acceptance test pass/fail;
- hidden tests pass/fail;
- independent review pass/fail;
- retries, rework, and root takeover count.
Cost, per task and per accepted task:
full_cost =
sum(input_uncached × input_price)
+ sum(cached_input × cached_price)
+ sum(cache_write_input × cache_write_price)
+ sum(output × output_price)
+ Jev calls
+ caller-edge charge
+ retries and verification calls
Cache:
- cached token share =
cached_input / input; - cache-write tokens per switch;
- number of model switches per session;
- number of effort updates per session;
- prefix rebuilds avoided by the gate.
Latency:
- time to first output;
- end-to-end latency;
- Jev latency (p50/p95).
Routing behavior:
- model and effort distribution;
- route reasons:
jev,cache-break-even,upgrade-not-triggered,session-lock,switch-budget,switch-cooldown,cache-cold; - fallback and UNKNOWN counts.
Never convert missing usage to zero. A missing bucket makes the task's full
cost UNKNOWN.
- Freeze the repository revision, task prompt, acceptance checks, model catalog, price table, tool versions, policy version and Jev version.
- Create one cache condition per task: cold or warm. Record it before the run.
- Randomize arm order per task; do not always run the baseline first.
- For each arm, start from the same snapshot and clean process state.
- Capture every physical model call, including Jev, retries and verification.
- Record cache read/write tokens from provider usage, not estimates.
- Run acceptance independently; the model's self-report is not evidence.
- Repeat each task 3–5 times. Report per-task results and aggregate paired differences.
- Use a paired bootstrap confidence interval for cost and acceptance deltas. Predefine a quality non-inferiority margin; a default starting point is −5 percentage points absolute, adjusted after pilot data.
- Report all losing strata and
UNKNOWNcosts; do not hide them in an average.
Phase 0 — offline, no live model:
- request-shape tests: turn detection, tool continuation, retry dedupe;
- effort anchor tests: insert, reinsert, invalid anchor, compaction reset;
- gate tests: price break-even, cold cache, upgrade trigger, lock/cooldown;
- session store tests: load, save, TTL, no prompt text.
Phase 1 — pilot:
- 6 synthetic tasks (T01, T04, T06, T08, T11, T12);
- arms A, B, C;
- 3 repetitions;
- cold and warm cache.
Phase 2 — real workload:
- 20 real tasks, arms A/B/C;
- 3 repetitions;
- per-difficulty report.
Phase 3 — session shapes:
- S4–S10 on 3–5 tasks;
- measure cache rebuild, resume correctness and compaction behavior.
A release candidate should meet all of these before claiming savings:
- acceptance non-inferiority: lower CI bound above the frozen margin;
- full cost reduction: lower CI bound above zero, after Jev, cache writes, retries and verification;
- no regression in critical failure or security tasks;
- no stale effort update after compaction;
- no model change inside a tool loop;
- no model switch on an identical retry;
- missing prices/usage produce a documented fallback, not a silent guess.
If quality fails, report cost without claiming savings. If cost is UNKNOWN,
do not claim savings.
These are external, not independent verification of this router. Use them to choose hypotheses, not as our numbers.
| Project | Published result | Caveat |
|---|---|---|
dominicrico/jev-router (Claude Code) |
Small tasks −97%, everyday −65% vs always Opus; hard set: plain Sonnet cheapest | Real Claude Code runs with hidden tests. Long warm sessions: cache guard cut switches 6→1 and halved cache rewrites, but the bill stayed the same; lean routing averaged Opus cost while plain Sonnet was 43% cheaper. |
robertn702/reasoning-router |
Selected SWE-bench tasks: 52% fewer output tokens on a short Flask fix; 5–7% on harder Django/pytest | Selected tasks, not a general savings claim; no quality guarantee from the classifier. |
gakonst/nanocode PR #805 |
Cache-preserving reasoning updates for Astra/Sol | PR states journeys are synthetic; no live provider cache-hit claim. |
| LiteLLM auto-router | 37–69% cheaper across several benchmarks; switch-back cache warm 97.4% / 99.3% | Vendor benchmarks; workload- and TTL-dependent. |
The most useful lesson is the baseline choice:
- Against always Opus/frontier, savings can be very large.
- Against an already-balanced model, savings shrink or disappear.
- On long warm sessions, model switching can fail to pay; plain mid-tier can be cheaper than clever routing.
Every experiment should report both baselines separately. Otherwise the headline number conflates "don't use frontier for everything" with "routing is smart".
Not the current benchmark. The runtime being evaluated is Codex-specific:
- it expects Codex's Responses-shaped requests and auth;
- effort routing uses OpenAI
configuration_update; - Jev supplies the complexity scores;
- the price/cache model assumes the provider's cache accounting.
A DeepSeek-only setup would test a different system:
- DeepSeek is consumed through Chat Completions in most clients;
- its reasoning control and cache semantics differ;
- without Jev, effort routing falls back to
high; - a DeepSeek adapter would need a separate agent harness, and its results would not transfer to Codex + GPT-6.
What can run now without Codex or Jev:
npm test— request shape, turn pin, retry dedupe, gates, effort anchors, compaction reset, session store.- A synthetic economic replay with fixed usage traces and a price table — tests accounting and gate arithmetic, not model quality.
- A mock-upstream HTTP test — tests streaming and request rewriting.
What cannot be concluded without a live Codex/GPT/Jev setup:
- real cache-hit rate;
- real output/reasoning token savings;
- quality non-inferiority;
- latency.
Recommendation: do not spend a full benchmark budget on DeepSeek for this router. Run the offline phases now; run the paired benchmark when Codex + at least two GPT tiers + Jev are available. If DeepSeek support is a product goal, build a separate adapter and evaluate it on its own cache/effort semantics.
The short answer: yes, overpaying a frontier model for simple work is the obvious waste, but the larger cache-safe lever is usually effort, not model tier.
Why:
- Model downgrade changes the rate on every token, but it forces the new model to rebuild the prompt prefix. On a long context, that rebuild can eat several turns of savings.
- Effort downgrade keeps the same model and the same prefix cache. It reduces reasoning and output tokens, which are not cached. It is usually the cleaner win for routine turns.
- Per-call model switching is the worst case: it rebuilds the prefix repeatedly inside one tool loop.
- A simple task still produces output and tool-call tokens. The saving depends on how much of the bill is output/reasoning versus cached input.
- Our current implementation is per-turn effort. A long single turn with many tool calls cannot lower effort mid-loop; per-step effort is the next lever, but it needs full history/provenance support.
A useful rule of thumb:
save by effort first
then switch model only when price break-even clears
never switch model inside a tool loop
Test both, because the answer is workload-dependent.