ADE router: collect per-turn usage data from users, then build the router from it

[arul28]

Goal

Build an "ADE router" that picks the model, effort, account, and worker tier for each task, in a cache-aware and cost-aware way (idea similar to Devin Fusion). First collect real usage data from users. Then build the router from that data.

What is built (lane ade-router-f30a1b5e, PR to follow)

Measurement (observe only; it does not change how providers work):

• Per-turn usage ledger on each machine: ~/.ade/usage/turns-YYYY-MM.jsonl and quota-YYYY-MM.jsonl. One row per finished chat turn for every provider: requested and served model, account kind, token split (input, output, cache read/write, reasoning), context size, requests, provider cost, API list-price cost, plan units, compactions, duration. Kept 3 months.
• Quota burn rates: dollars per 1% of each plan window, learned from quota readings and ledger rows. This is a lower bound, because other clients use the same quota.
• ade usage turns (action usage.getTurnUsageSummary) reads the ledger.
• Cursor dashboard reconcile: the real served model and cost of Cursor turns.
• Provider telemetry gaps closed for Claude, Codex, OpenCode, Cursor, Droid, Pi, Grok, Qwen, Copilot, and Kimi (Kimi checked in code only).

Daily usage research report (to our Cloudflare):

• Each install sends one compact report per local day to the account-directory Worker: POST /usage-research/daily, D1 table usage_research_daily (migration 0010).
• Contents: per provider and model, turns, tokens, costs, context and duration percentiles, hour of day, compactions, served-model mismatches, plan tier, quota burn rates, and the price table in use.
• Never sent: emails, account ids, paths, session ids, prompts. Anonymous install id; accounts are distinguished only by a salted hash.
• Gated on the existing product analytics consent. Kill switch ADE_USAGE_RESEARCH=0.
• One row per install per day (upsert, no duplicates). 32 KB cap per report, fleet daily write cap, 4 GB storage ceiling, 180-day retention. All are Worker vars.
• Merging to main deploys it (deploy-web.yml applies the migration and deploys the Worker).

Provider model-routing fixes found during this work:

• Grok: the model and effort now go through session/set_config_option. The -m flag failed for some models, and a resume restored the old model.
• Copilot: on an Auto-only plan, Copilot overrides the chosen model. ADE now shows the model that answered.
• Qwen: ADE sends only the models Qwen offers, and says when another model answered.
• Kimi: the effort is now sent.
• Cursor: several model/effort/Fast/config-value bugs (switch reverted, cloud re-adoption, headless catalog, config values never sent).
• OpenCode: the effort variant was dropped for most models; Fast plus effort are now combined.
• Pi: switches and effort fixed. ADE no longer writes the user's global Pi defaults.
• Droid: Ultracode effort, cleared effort, and the real served model.

Findings so far (owner's machine)

• Cost is driven by context length x turns: cache reads are 65% of Claude cost. Main-thread context median is about 400k tokens.
• Subagents are 58% of Claude cost. Each starts with a fixed ~53k-token prompt, and re-reading it is 21% of subagent cost.
• Model switches are cheap when rare (<1% of cost). Moving a large context into an expensive model is costly.
• The same subagent workload repriced: GPT-5.6 Luna about 96% cheaper, DeepSeek Flash about 99% cheaper, Opus 5.5 about 48% cheaper. Quality not measured.
• ADE Claude chats start at about 96k tokens, against about 61k in plain Claude Code (MCP tools loaded up front; ENABLE_TOOL_SEARCH).

Next steps (future work)

1. Collect data. Let the daily reports run for 2 to 4 weeks. Then analyze them in D1: cost per provider and model, cache hit ratios, context sizes, served-model mismatches, and burn rates per plan.
2. Lean context PR (first real savings, no router needed). Stop forcing ENABLE_TOOL_SEARCH=auto (measure tool errors and turns before and after). Trim the ~53k subagent start prompt. Compact the coordinator earlier. Measure with the ledger.
3. Tiered workers. The coordinator stays on a frontier model. Narrow worker tasks (search, read, tests, mechanical edits) go to cheap cache-friendly models. Switch only at task boundaries, never mid-thread.
4. Quota-aware account balancing. Use the burn rates to pick the plan with the most headroom. Spill to an API key only when all plans run low.
5. Shadow router first. Log what the router would pick next to what ran. Use free quality signals (tests, reverts, interrupts, "fix it" follow-ups). Turn on live routing only when the shadow log shows a gain.
6. Skip difficulty classifiers and per-turn model switching until real evals exist.

Known limits

• Kimi is not live-verified (no account on the dev machine).
• A local Cursor "auto" turn learns its served model only later, from the dashboard.
• Droid forgets a cleared effort after an app restart until the next choice.
• Burn rates see ADE turns only, so they are a lower bound.

[linear-code]

ADE-159