Run Claude only on the Anthropic BYOK key, with no fallback provider by Programcode · Pull Request #185 · Programcode/DebateOwl

Programcode · 2026-10-22T13:50:48Z

Every Claude (anthropic/*) call now asks OpenRouter for the anthropic provider only, with fallbacks off. Bedrock, Vertex, Azure, Claude Platform on AWS and every other provider are never tried. Other models keep balanced routing.

Required manual step in OpenRouter

The request body can't stop OpenRouter from serving Anthropic on its own credits when the BYOK key fails; only the key's own setting does that. In OpenRouter → Settings → Integrations → Anthropic key, set Shared capacity fallback to "Never use shared capacity for any model on this provider" (is_byok_only). Without it, only: ["anthropic"] still allows OpenRouter's shared Anthropic capacity when the key is out of credit.

What changed

Routing. supabase/functions/_shared/provider-routing.ts (new) gives anthropic/* the options provider: { only: ["anthropic"], allow_fallbacks: false, require_parameters: true }. Everything else keeps { require_parameters: true } with no sort or order. Every call path uses it:
- ai/provider.ts callOpenRouter: plain and streamed calls, research steps, the truncation and JSON repair retries, and Repair model JSON locally and enforce schemas before paying for a retry #176's strict json_schema calls. Research uses BACKGROUND_MODEL, which is Haiku, so a GPT request's research steps are pinned too.
- ai-stream/index.ts, streaming Ask.
- scripts/prompt-examples.ts.
- Jev (typesafe/jev-1.13, /alpha/decisions) isn't Claude and is unchanged.
Failures. When a Claude call fails (non-2xx, or an SSE error chunk), nothing is retried on another provider. The endpoints answer 503 { code: "provider_unavailable" }. The client already shows that as "Anthropic is temporarily unavailable. Please try again later." in en, es and hu, on plain JSON, live result events and ai-stream SSE. Sparring now shows the translated message as well. The ai_requests row gets error_code = claude_unavailable. Ask no longer resends a provider_unavailable stream failure through the plain endpoint, which would try Claude a second time on a failing key.
Refund, fitted to Weight the daily AI quota by model and cap AI spend at $2 a week (BYOK Claude excluded) #178's weighted quota.
- refundQuotaReservation(userId, rateLimit) gives back exactly RateLimitResult.units. That's the p_units checkRateLimit passed to consume_ai_quota for this request: Opus 5, GPT-6.1 Sol 3, Sonnet 2, DeepSeek V4 Pro 2, Auto Pro 5, others 1.
- The units follow the requested selection, so Auto Pro refunds 5 even when it resolves to Haiku.
- refund_daily_quota(p_user_id, p_units, p_quota_key) lowers today's counter by those units, never below 0. It's SECURITY DEFINER and service-role only.
Weekly $2 cap. A refunded request is logged with cap_cost_usd = 0, so it doesn't count toward the weekly cap.
When a refund happens. Only when nothing was paid for:
- no text written;
- no finished model call: no completed main attempt before a retry, and no finished research step. A research step Claude refused in the same outage doesn't count, so Factoids, Generate, Attack, Brainstorm and Ask with research still refund in an outage;
- no OpenRouter charge against the cap (chargedAgainstCap).
A failure that OpenRouter charged credits for keeps both its quota and its cap_cost_usd, so no real spend is hidden from the cap. Jev routing and source-ranking rows are logged separately and keep their own cap cost.
Guard. A Claude response whose usage shows is_byok: false or cost > 0 logs a claude_not_byok JSON warning with model, source, is_byok, cost and upstream_inference_cost.
Live proxy. e2e/full-stack/fake-openrouter.mjs drops only and allow_fallbacks when the live proxy rewrites the model to Muse Spark, so the rewritten model can still route.

Status

Merged as 17c00f91 and live on production, per Adam: the migration was applied at 17:46 Budapest, and ai v91, ai-stream v28 and voice v11 are deployed. I didn't do the merge, migration or deploy, and I haven't checked production myself.

Deploy order (as applied)

Migrations, in this order:
- 20261022120000_opus_default_to_sonnet.sql (Move saved Opus 5.5 settings to Sonnet 5.5 (data migration, after #164) #180)
- 20261022130000_index_ai_requests_user_created.sql and 20261022130001_weighted_quota_weekly_budget.sql (Weight the daily AI quota by model and cap AI spend at $2 a week (BYOK Claude excluded) #178)
- 20261022160000_backfill_byok_request_costs.sql (Log the real cost of Anthropic (BYOK) calls and Exa research #179)
- this PR's 20261022170000_refund_daily_quota.sql (renamed from 20261022150000 at merge)
Then deploy ai and ai-stream. Both call consume_ai_quota (Weight the daily AI quota by model and cap AI spend at $2 a week (BYOK Claude excluded) #178) and refund_daily_quota (this PR), so the migration has to be applied before they're deployed. If the functions go first, every refund call fails: it's logged and the student still gets the outage message, but the units aren't given back.
Set the OpenRouter key's shared capacity fallback (above).

Notes

cost > 0 also fires for OpenRouter's 5% BYOK fee once past the monthly BYOK allowance. Locally the fake provider reports cost: 0.0001 for every call, so the warning shows in local runs.
A request reserved just before midnight UTC that fails after it lowers the new day's counter, using the same CURRENT_DATE rule as consume_ai_quota.
Jev calls (Auto routing, source ranking) that finish before a Claude failure don't block the refund. Their own rows keep their cap cost, so only the daily unit is given back.
The client's "remaining today" estimate in use-subscription.ts (Fix the 11 bugs from today's QA pass (H1, M1–M5, L1–L5) #184) counts ai_requests rows, including refunded claude_unavailable rows, and ignores weights. The server's count decides, so this is only a display estimate.
Log the real cost of Anthropic (BYOK) calls and Exa research #179 (BYOK and Exa cost logging) landed with its migration renamed to 20261022160000. Its Exa charges go into researchUsage through onExaUsage, and the refund check runs chargedAgainstCap over that merged usage. So a request whose Exa searches cost money is never refunded and keeps its cap cost.

Verification

The full local run below is at 631d05c4 (current main merged in, through #176, #178, #181 and #184). The last commit, 6fa3f2fc, came after it and has its own checks (see Reviews). GitHub Actions is off (billing), so these are local results, not CI.

npm run verify, one step at a time, all exit 0: docs check, lint, typecheck, vitest 403/403, collaboration 30/30, Edge 388/388, build, sw, and Playwright smoke 43/43 on desktop Chromium.
Database gate (./.superset/db-gate.sh --db-only) at 2415463d: 25 files and 470 tests pass, including Weight the daily AI quota by model and cap AI spend at $2 a week (BYOK Claude excluded) #178's ai_quota_weekly_budget_test.sql and this PR's daily_quota_refund_test.sql, plus the three concurrency scripts. The migrations and SQL tests haven't changed since.
- The pgTAP test reserves Opus (5) and Haiku (1) under a limit of 7 with the real consume_ai_quota, so a second Opus is refused. It refunds 5, checks that a new Opus fits again, and checks the floor at 0, earlier days and privileges.
- It also checks that a refunded claude_unavailable row logged with cap_cost_usd 0 leaves consume_ai_quota's weekly check open.
Function-layer tests in provider-errors_test.ts cover:
- plain JSON, ai live streaming and ai-stream SSE, each with an HTTP 402 and an SSE error chunk;
- the outage message and code, and no leaked provider body;
- a pinned route on every attempt;
- the exact refunded units for Haiku 1, Sonnet 2 and Opus 5 through both endpoints, and cap_cost_usd 0 on refunded rows;
- no refund after text, after a completed repair attempt, after a finished research step, after an OpenRouter charge (that row keeps cap_cost_usd 0.25), on success, or on a GPT failure;
- a refund in a full outage where the research step is refused too.
Each guard was mutation-checked: the test fails without it. That covers refunding 1 instead of the reserved units, dropping the cap exclusion in either function, dropping the charged-request check in either function, and counting a refused research step as work.
Two real Haiku calls to OpenRouter on the pinned route (made-up content):
- JSON mode: HTTP 200, served by Anthropic, is_byok: true, cost: 0.
- The exact Repair model JSON locally and enforce schemas before paying for a retry #176 research payload (strict json_schema, reasoning, prompt caching): HTTP 200, served by Anthropic, is_byok: true, cost: 0, valid schema output.
Both used the live-verify key, so they prove BYOK on that OpenRouter account, not necessarily on production's key.
Staging: skipped. Staging runs main's Edge Functions and only gets new ones after a merge, so testing this there would need a staging deploy.
Not verified: Opus and Sonnet on the pinned route, and a real out-of-credit response from OpenRouter. The exact status isn't documented, so any non-2xx or error chunk for Claude is treated as an outage.
Reviews:
- Opus round 1: six findings; five fixed in ffd86044, and the midnight date was accepted.
- Opus round 2 at 2415463d, against Weight the daily AI quota by model and cap AI spend at $2 a week (BYOK Claude excluded) #178: one medium finding, fixed in 631d05c4. In an outage, refused Haiku research steps counted as paid work, so research tools never refunded. Of the four lows, the pinned route with Repair model JSON locally and enforce schemas before paying for a retry #176's schema is now verified by the second real call above. The midnight date, the Jev-before-failure case and the client estimate are listed under Notes.
- The automated security review flagged that zeroing cap_cost_usd could hide a real OpenRouter charge. Fixed in 2415463d: a charged failure is never refunded.
- Codex round 1 stalled on the Codex usage limit.
- Codex round 2 at 631d05c4 (gpt-6.1-sol, xhigh): two medium findings, both confirmed in the code and fixed in 6fa3f2fc.
  - Both stream readers threw on an SSE error frame before reading that frame's usage. A charge reported only on the error frame was dropped, and the failure could then be refunded with cap cost 0. Usage is now kept before the error is thrown. A new test case (error-frame-charged) fails against the old order in each reader.
  - ai/index.ts refunded a Claude refusal that raced a client abort; ai-stream already refused that. The plain handler now checks workSignal.aborted. A new test aborts after the 402 is read and before the refund decision, and fails without the check.
  - Dismissed: an abort during the later usage logging. By then the refund decision rests on a real refusal with no output, so the client leaving doesn't change it.
- Checks at 6fa3f2fc: docs check, lint and typecheck exit 0, and provider-errors_test.ts passes with each new guard mutation-checked. I didn't rerun the full unit, Edge, build and Playwright suites on that commit before it was merged.
PR checks: the Vercel previews hit "Deployment rate limited — retry in 24 hours", and the GitHub Actions jobs are pending (billing). Neither reflects this code.

🤖 Generated with Claude Code

Every anthropic/* call now asks OpenRouter for the anthropic provider only, with fallbacks off, so Bedrock, Vertex and other providers are never tried. Other models keep balanced routing. A refused Claude call returns a 503 outage message the client already translates, logs claude_unavailable, and gives the reserved daily request back through a new refund_daily_quota RPC. A Claude response whose usage shows is_byok false or a cost above zero logs a claude_not_byok warning. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

A stream that fails after the model wrote text, or a repair retry after a completed attempt, already spent tokens on the key, so the request stays counted. The student still sees the outage message. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

vercel · 2026-10-22T13:50:52Z

Deployment failed for project debate-navigator with the following error:

Resource is limited - try again in 24 hours (more than 100, code: "api-deployments-free-per-day").

Learn More: https://vercel.com/adam-gaborjanis-projects?upgradeToPro=build-rate-limit

coderabbitai · 2026-10-22T13:50:59Z

Note

Currently processing new changes in this PR. This may take a few minutes, please wait...

⚙️ Run configuration

Source mode: Auto
Repository: Programcode/DebateLab
Branch: cost/claude-byok-only
File: .coderabbit.yaml
Commit: b25bc99
Layers: Repository YAML (base), Organization UI (inherited)
Review profile: CHILL
Plan: Free
Run ID: 1fe59580-9e48-4b18-8ccd-889fb07e2119

📥 Commits

Reviewing files that changed from the base of the PR and between b493fd6 and b25bc99.

📒 Files selected for processing: 10

docs/architecture.md
supabase/README.md
supabase/functions/_shared/cost.ts
supabase/functions/_shared/entitlements.ts
supabase/functions/ai-stream/index.ts
supabase/functions/ai/index.ts
supabase/functions/ai/provider-errors_test.ts
supabase/functions/ai/provider.ts
supabase/migrations/20261022170000_refund_daily_quota.sql
supabase/tests/daily_quota_refund_test.sql

+----------------------------------+
| Bazinga! You missed a semicolon. |
+----------------------------------+
  \
   \     /)/)
        ( '')
        /  |
      o(___)

ℹ️ Recent review info

⚙️ Run configuration

Source mode: Auto
Repository: Programcode/DebateLab
Branch: cost/claude-byok-only
File: .coderabbit.yaml
Commit: b493fd6
Layers: Repository YAML (base), Organization UI (inherited)
Review profile: CHILL
Plan: Free
Run ID: a2897419-ba13-4373-9c66-86d9f806aa70

📥 Commits

Reviewing files that changed from the base of the PR and between da7473e and b493fd6.

📒 Files selected for processing: 17

docs/architecture.md
e2e/full-stack/fake-openrouter.mjs
scripts/prompt-examples.ts
src/pages/AskPage.tsx
src/pages/SparringPage.tsx
supabase/README.md
supabase/functions/_shared/entitlements.ts
supabase/functions/_shared/provider-routing.test.ts
supabase/functions/_shared/provider-routing.ts
supabase/functions/ai-stream/handler_test.ts
supabase/functions/ai-stream/index.ts
supabase/functions/ai/handler-research.test.ts
supabase/functions/ai/index.ts
supabase/functions/ai/provider-errors_test.ts
supabase/functions/ai/provider.ts
supabase/migrations/20261022150000_refund_daily_quota.sql
supabase/tests/daily_quota_refund_test.sql

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

📝 Walkthrough

Claude requests now use Anthropic-only OpenRouter routing. Provider outages receive a dedicated response, and qualifying failures refund daily quota. The client, tests, and documentation reflect these routing and error-handling changes.

Changes: Claude BYOK request flow

Layer / File(s)	Summary
Provider routing and request behavior `supabase/functions/_shared/provider-routing.ts`, `supabase/functions/ai/provider.ts`, `scripts/prompt-examples.ts`, `e2e/full-stack/fake-openrouter.mjs`, `supabase/functions/_shared/provider-routing.test.ts`, `supabase/functions/ai-stream/handler_test.ts`, `supabase/functions/ai/handler-research.test.ts`	Claude models use Anthropic-only routing with fallbacks disabled. Provider errors and BYOK usage warnings use shared helpers. The prompt example and test proxy use routing-compatible request options. Tests check routing and warning behavior.
Daily quota refund `supabase/migrations/20261022150000_refund_daily_quota.sql`, `supabase/functions/_shared/entitlements.ts`, `supabase/tests/daily_quota_refund_test.sql`	A service-role RPC refunds current-day quota units without reducing the counter below zero. The entitlement helper calls the RPC and logs failures. SQL tests cover permissions, refund behavior, and prior-day counters.
Provider outage responses and refunds `supabase/functions/ai-stream/index.ts`, `supabase/functions/ai/index.ts`, `supabase/functions/ai/provider-errors_test.ts`	The AI handlers classify Claude provider outages separately, return the configured outage response, and record `claude_unavailable`. They refund quota only when the relevant output and usage conditions are met. Tests cover provider failures, successful requests, and refund cases.
Client behavior and routing documentation `src/pages/AskPage.tsx`, `src/pages/SparringPage.tsx`, `docs/architecture.md`, `supabase/README.md`	Ask does not retry through `askChat` after a streamed `provider_unavailable` error. Sparring sends nonblank error messages through the user-facing error formatter. Documentation describes Claude routing, outage responses, refunds, and BYOK usage warnings.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant AskPage
  participant ai-stream
  participant OpenRouter
  participant refundDailyQuota
  participant refund_daily_quota
  AskPage->>ai-stream: Send streaming request
  ai-stream->>OpenRouter: Request with Anthropic-only routing
  OpenRouter-->>ai-stream: Return provider failure
  opt No output was produced
    ai-stream->>refundDailyQuota: Refund reserved quota
    refundDailyQuota->>refund_daily_quota: Call refund RPC
    refund_daily_quota-->>refundDailyQuota: Return updated counter
  end
  ai-stream-->>AskPage: Return provider_unavailable response

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Note

🎁 Summarized by CodeRabbit Free

Your organization is on the Free plan. CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please upgrade your subscription to CodeRabbit Essentials by visiting https://app.coderabbit.ai/settings/billing.

_{Comment @coderabbitai help to get the list of available commands.}

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

…mpt. From the Opus review of #185: - A request whose research steps already ran on Claude keeps its quota. - Ask no longer resends a provider_unavailable stream failure through the plain endpoint, which tried Claude a second time on a failing key. - ai-stream refunds before it sends the error, so an immediate retry sees it. - Sparring shows the translated provider message instead of the raw one. - Tests cover no refund on success, on a non-Claude failure, after a completed repair attempt, and after research. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

vercel · 2026-10-22T14:05:14Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
debatelab-staging	Ready	Preview	Oct 22, 2026 3:03pm UTC

# Conflicts: # docs/architecture.md

vercel · 2026-10-22T14:08:55Z

Deployment failed for project debatelab-staging with the following error:

Resource is limited - try again in 24 hours (more than 100, code: "api-deployments-free-per-day").

Learn More: https://vercel.com/adam-gaborjanis-projects?upgradeToPro=build-rate-limit

# Conflicts: # docs/architecture.md # supabase/functions/ai/provider.ts

# Conflicts: # src/pages/SparringPage.tsx

refund_daily_quota takes p_units (default 1, the reservation every AI request makes today) and never goes below zero. Once the weighted quota in #178 lands, the refund can return what consume_ai_quota reserved for the model. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

#180 added 20261022120000_opus_default_to_sonnet.sql, so the refund migration's identical version failed to apply after the merge from main. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

# Conflicts: # supabase/functions/ai/provider-errors_test.ts

… out of the weekly cap. #178 reserves each request's weight (Opus 5, Sonnet 2, others 1; Auto Pro 5) through consume_ai_quota. A refunded Claude failure now gives back the same units checkRateLimit reserved, taken from its RateLimitResult, and logs cap_cost_usd 0 so it does not count toward the weekly $2 either. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Zeroing cap_cost_usd on such a row would hide real spend from the weekly cap, so a request with a charge against the cap now keeps both its quota and its cap cost. Only free failures are refunded and logged with cap cost 0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Research steps run on Haiku, so in an outage they are refused along with the main call. Such a refusal did no paid work and no longer blocks the refund; a research step that finished or was charged still does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…quest. From Codex review round 2 of #185: - Both stream readers threw on an error frame before reading its usage, so a charge reported only there was dropped, and the failure could then be refunded with cap cost 0. Usage is now kept before the error is thrown. - ai/index.ts refunded a Claude refusal that raced a client abort. ai-stream already refused that; the plain handler now does too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Rename the refund migration to 20261022170000: production already has 20261022160000 (#179), so 20261022150000 would be out of order.

Programcode and others added 2 commits October 22, 2026 13:01

Copilot AI balanced review requested due to automatic review settings October 22, 2026 13:50

Copilot AI reviewed Oct 22, 2026

Merge remote-tracking branch 'origin/main' into cost/claude-byok-only

21ee09b

# Conflicts: # docs/architecture.md

vercel Bot deployed to Preview – debatelab-staging October 22, 2026 14:06

View deployment

Programcode and others added 5 commits October 22, 2026 14:09

Merge remote-tracking branch 'origin/main' into cost/claude-byok-only

91c6fe2

# Conflicts: # docs/architecture.md # supabase/functions/ai/provider.ts

Merge remote-tracking branch 'origin/main' into cost/claude-byok-only

630db01

# Conflicts: # src/pages/SparringPage.tsx

Give the refund migration its own version.

b493fd6

#180 added 20261022120000_opus_default_to_sonnet.sql, so the refund migration's identical version failed to apply after the merge from main. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Merge remote-tracking branch 'origin/main' into cost/claude-byok-only

35f0a2c

# Conflicts: # supabase/functions/ai/provider-errors_test.ts

vercel Bot deployed to Preview – debatelab-staging October 22, 2026 14:39

View deployment

Programcode and others added 4 commits October 22, 2026 14:42

Merge remote-tracking branch 'origin/main' into cost/claude-byok-only

c93eabd

vercel Bot deployed to Preview – debatelab-staging October 22, 2026 15:03

View deployment

Programcode and others added 3 commits October 22, 2026 15:19

Merge main into cost/claude-byok-only

e74563d

Rename the refund migration to 20261022170000: production already has 20261022160000 (#179), so 20261022150000 would be out of order.

Merge remote-tracking branch 'origin/main' into s185

b25bc99

Programcode merged commit 17c00f9 into main Oct 22, 2026
2 of 14 checks passed

Programcode mentioned this pull request Oct 23, 2026

Move default, Auto and test-agent model calls onto the Anthropic BYOK key, with a no-fallback guard #220

Merged

Navigation Menu

Repository navigation

Run Claude only on the Anthropic BYOK key, with no fallback provider - #185

Run Claude only on the Anthropic BYOK key, with no fallback provider#185
Programcode merged 16 commits into
mainfrom
cost/claude-byok-only

Programcode commented Oct 22, 2026 •

edited

Loading

Uh oh!

vercel Bot commented Oct 22, 2026

Uh oh!

coderabbitai Bot commented Oct 22, 2026 •

edited

Loading

Sequence Diagram(s)

Uh oh!

Copilot AI left a comment

Uh oh!

vercel Bot commented Oct 22, 2026 •

edited

Loading

Uh oh!

vercel Bot commented Oct 22, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Programcode commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Required manual step in OpenRouter

What changed

Status

Deploy order (as applied)

Notes

Verification

Uh oh!

vercel Bot commented Oct 22, 2026

Uh oh!

coderabbitai Bot commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Sequence Diagram(s)

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

vercel Bot commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vercel Bot commented Oct 22, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Programcode commented Oct 22, 2026 •

edited

Loading

coderabbitai Bot commented Oct 22, 2026 •

edited

Loading

vercel Bot commented Oct 22, 2026 •

edited

Loading