Repository navigation
Log the real cost of Anthropic (BYOK) calls and Exa research - #179
Conversation
OpenRouter reports cost: 0 for calls billed to our own provider key (every Anthropic model), with the real charge in cost_details.upstream_inference_cost. The calculator now adds it, and a BYOK response without it stays unpriced. Exa searches and page reads now report costDollars.total into the request's research usage, so the parent ai_requests row includes them and raw_usage keeps one entry per Exa call. A migration recomputes the old $0 rows from their saved raw usage. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A sent Exa call that is cancelled, times out, or returns an unreadable body may still be billed. It now reports an incomplete usage entry instead of nothing, so the row's cost is unknown rather than quietly too low. Error statuses are still not billed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Warning Review limit reachedEnable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. Next included review available in 25 minutes. View limit details
Note 🎁 Summarized by CodeRabbit Free
Comment |
# Conflicts: # supabase/functions/_shared/research.test.ts # supabase/functions/_shared/research.ts
|
Deployment failed for project debatelab-staging with the following error: Learn More: https://vercel.com/adam-gaborjanis-projects?upgradeToPro=build-rate-limit |
|
Deployment failed for project debate-navigator with the following error: Learn More: https://vercel.com/adam-gaborjanis-projects?upgradeToPro=build-rate-limit |
Rename the refund migration to 20261022170000: production already has 20261022160000 (#179), so 20261022150000 would be out of order.
What was wrong
Anthropic rows logged $0. Every Claude call (Haiku 5.5, Opus 5.5, Sonnet 5) goes through OpenRouter on our own Anthropic key (BYOK). For those calls OpenRouter's
usage.costis only its own fee, which is 0. The real charge is inusage.cost_details.upstream_inference_cost, and we ignored it. Pricing wasn't missing: we read the wrong field. Example staging row: Haiku 5.5, 1,527 in / 671 out,cost: 0, upstream $0.0004882.Exa wasn't logged at all. Searches and page reads never reached
ai_requests. Exa returns the charge with every response (costDollars.total); we threw it away.What changed
_shared/cost.ts: for a BYOK response the cost is OpenRouter's fee plus the upstream charge. A BYOK response without the upstream figure is logged as unknown (null), not $0. The calculation version goes toopenrouter-usage-v2, so new rows can be told apart from old ones. All OpenRouter paths (main calls, streaming, theai-streamAsk, Jev) share this function._shared/research.ts: every Exa search and page read reports its charge into the request's research usage, including the fallback search and the URL reads Ask does on its own. Exa costs go into the cost of the request's existing row, with oneraw_usage.callsentry per Exa call (endpoint, request id, cost breakdown). There are no new rows, because the app countsai_requestsrows as daily usage. An error status isn't billed and isn't logged. A call that was sent but never settled (cancelled, timed out, unreadable body) makes the row's cost unknown, not lower.20261022120000_backfill_byok_request_costs.sqlrecomputes the old $0 rows from the raw usage each row already stores. It only changes pricedopenrouter-usage-v1rows that contain a BYOK call and have usable numbers. Repaired rows get the versionopenrouter-usage-v1-byok-backfill, so it can't apply twice. It doesn't delete anything or touchraw_usage. If you'd rather not rewrite history, it can be dropped without affecting the rest.What it changes in the numbers
Read-only dry run of the backfill on staging: 124 rows go from $0 to $9.80 in total (Sonnet 5 $7.74, Opus 5.5 $2.03, Haiku 5.5 $0.008, the rest under 2¢).
Local live run (
verify-debatelab --live, real Exa, OpenRouter pinned to Muse Spark): each research request made 8 or 9 Exa calls for about $0.033, next to about $0.0027 for its model calls. Exa was about 90% of what these requests cost and was missing from the logs entirely.Verification
All local, at this branch. GitHub Actions is broken repo-wide, so there is no CI result.
npm run verifyat the first commit: exit 0. docs, lint, typecheck, 305 unit, 30 collaboration and 361 edge tests, build, SW, and 28 browser smokes passed. 2 smokes were flaky (30 s timeouts in the attack page and the announcements page; load average was about 33).npm run test:edge361 passed, eslint clean,deno checkonaiandai-stream.verify-debatelab up --live,drive recipes/live-generationon desktop-chromium: 3 passed. A temporary trigger copied theai_requestsrows before teardown. The generate, attack, brainstorm, factoids and Ask rows all carryopenrouter-usage-v2, and their cost includes every Exa call, with none unpriced. The final commit only changed the unsettled-call path, which this run didn't exercise../.superset/db-gate.sh --db-only: pgTAP 23 files / 441 tests pass. The first run's daily-quota lock-timing check failed under load; the rerun passed everything.main(Run research helper calls on a fixed cheap model #163 helper model, Trim AI prompts and cache Anthropic system prompts #177 prompt budget; the only conflict wasexaGetContentsgainingmaxChars, now its last parameter):npm run test:edge366 passed,deno checkclean onaiandai-stream.cost.test.ts), and research reporting for billed, error-status and unsettled Exa calls (research.test.ts).Not proven locally: a real BYOK response end to end. The live proxy pins to Muse Spark, which isn't BYOK. The unit tests use the exact shape of a staging Haiku row, and the first Anthropic rows on staging after merge are the live check.
Deploying
Merging ships nothing on the backend. For production: apply the migration, then deploy
aiandai-stream(both usecost.ts; onlyaiuses the Exa change).🤖 Generated with Claude Code