Cap each AI feature's reply length instead of 32,768 tokens by Programcode · Pull Request #182 · Programcode/DebateOwl

Programcode · 2026-10-22T12:35:17Z

Cost fix 3 of 7. Every AI call used to send max_tokens: 32768, whatever the feature. Each feature now has its own reply cap, kept in one file: supabase/functions/_shared/output-caps.ts.

The caps

Reply budgets in tokens:

Ask (streamed and structured): 1,500
Crossfire question or answer: 1,000
Crossfire debrief: 2,000
Sparring speech: 2,500
Sparring scorecard: 3,000
Generate (any length): 6,000
Attack, Brainstorm, Factoids: 4,000 each
Research sub-calls (search decision, follow-up queries): 400

Four of these differ from the brief, which said roughly 1,000 for Crossfire and Sparring:

A Sparring speech is written for a time limit of up to 7 minutes and may run to 6,000 characters. In Hungarian or Spanish that is well past 1,500 tokens, so 2,500 keeps a full-length speech from being cut off and paid for twice.
The Sparring scorecard has four categories plus moments, areas and a verdict, so it got 3,000. The Crossfire debrief has six lists, each citing turns, and got 2,000.
A bigger cap costs Basic users nothing extra: at low effort Claude's thinking budget stays at its 1,024 floor until max_tokens passes about 5,100.
A single Crossfire question or answer stays at 1,000.
Attack, Brainstorm and Factoids weren't in the brief, so I set 4,000.

Why max_tokens is bigger than the reply cap

OpenRouter counts hidden reasoning against max_tokens. For Claude (the default model) it also sets the thinking budget to max(max_tokens × share, 1024), where the share is 0.2 at low effort, 0.5 at medium and 0.8 at high. A flat max_tokens: 1000 would leave Claude no room to answer.

So max_tokens is the reply cap plus room for reasoning. That room is the larger of two amounts: Claude's full budget, or a fixed floor of 2,048 / 4,096 / 8,192 tokens for low / medium / high. The floor is for OpenAI, Gemini 3 and DeepSeek, which decide on their own how long to think. Examples:

Ask, Basic user (always low effort): 3,548, down from 32,768. Claude's thinking budget drops from 6,553 to 1,024.
Ask, Pro user at medium effort: 5,596. Thinking budget drops from 16,384 to 2,798.
Generate, Pro user at high effort: 30,000. This is close to the old limit, because Claude at high effort keeps 80% for thinking. Capping reasoning separately would mean switching to reasoning.max_tokens, and that changes what the effort setting does.

Cut-off replies

Before this change nothing checked finish_reason: "length". A JSON reply cut off at the limit surfaced as "invalid JSON", then went to a repair retry with the same limit, which couldn't help.

Now:

JSON features. Every ai call retries a cut-off reply once with twice the reply budget. The live draft resets first, and the usage log bills both attempts. A second cut-off fails the request instead of retrying again. Each cut-off logs [AI] Reply hit its <cap> cap (...) with the model and effort, so the numbers can be tuned from logs.
Streamed Ask. In ai-stream the text has already reached the user, so it can't be retried. The server sets replyTruncated: true on the output, and the chat shows "This answer hit its length limit and stops early..." under the answer, translated when shown (en, es, hu). When the model spent the whole cap on reasoning and sent no text, the stream fails instead. The client then falls back to the structured endpoint, which retries with the larger cap.

Verify tooling

The fake OpenRouter gets a /__truncate hook (fake.truncate(feature)) that cuts the next reply off with finish_reason: "length".
It now records each call's maxTokens.
There are two new recipes: gen-cutoff and ask-cutoff.

Verification

GitHub Actions doesn't run repo-wide right now: every job fails in 2 to 3 seconds with "The job was not started because recent account payments have failed or your spending limit needs to be increased." Both Vercel previews are on a 24-hour build rate limit. Everything below ran locally, and a local pass is not a CI pass.

At 78164d2b, npm run verify passed docs check, lint, typecheck, unit, collaboration, edge and build. test:sw timed out once under load, then passed on rerun. The desktop @smoke browser set passed: 26 tests, 4 flaky on pages that make no AI calls.
After merging main (f9159f7c, which has the same tree as the tested rebase), edge (362), unit (338), lint and typecheck all passed.
At dda9bc01 the VPS is short on memory, so only the touched files were rechecked: output-caps.test.ts, ai-stream/, ai/stream.test.ts, ai/handler-research.test.ts and ai/provider-errors_test.ts (26 tests) passed, and eslint passed on the touched files.
The cut-off retry test and the empty cut-off stream test both fail when their fix is removed.
Pending: the verify-debatelab recipes (gen-cutoff, ask-cutoff and the existing Ask, Generate, Sparring and Crossfire ones) on both viewports, plus a live pass to check real reply lengths against the caps. The shared local Supabase stack is busy with other threads.

Reviews

Codex round 1 (e5b4d986) had two findings, both fixed in 78164d2b:

Effort-based models such as Gemini 3 could spend the whole budget on reasoning. They now get a reasoning floor per effort.
The cut-off notice was stored already translated. It's now translated when shown.

Codex round 2 couldn't run: the Codex account hit its usage cap (it resets at 15:05 UTC), and a final round is scheduled after that.

An Opus review of f9159f7c had four findings:

A blank cut-off streamed Ask was logged as a success. Fixed in dda9bc01: it now fails over to the structured endpoint. The test is the cut-off-empty scenario.
The Sparring speech cap was below a full-length speech. Fixed in dda9bc01: the caps are now 2,500 and 3,000.
The cut-off retry doubles Claude's reasoning budget along with the reply. Accepted as is. Under OpenRouter's rule Claude's reply room is (1 − share) × max_tokens, so doubling the reply room means doubling max_tokens. The budget is a ceiling, not spend, and the retry runs only after a cut-off.
Tests computed their expected caps with the same function. Fixed in dda9bc01: output-caps.test.ts pins hard-coded values and checks that every cap leaves its full reply after Claude's budget. It caught a floating-point off-by-one (30,001 instead of 30,000), fixed by using integer percentages.

Out of scope

The default model, the background-step model, quotas, prompt trimming, cost logging and the JSON output format are all handled in other threads.

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… notice on render Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

vercel · 2026-10-22T12:35:18Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
debate-navigator	Ready	Preview	Oct 22, 2026 12:54pm UTC
debatelab-staging	Ready	Preview	Oct 22, 2026 12:54pm UTC

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

coderabbitai · 2026-10-22T12:35:28Z

Warning

Review limit reached

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

Next included review available in 19 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Source mode: Auto
Repository: Programcode/DebateLab
Branch: cost/per-feature-token-caps
File: .coderabbit.yaml
Commit: d231cef
Layers: Repository YAML (base), Organization UI (inherited)
Review profile: CHILL
Plan: Free
Run ID: 7d584a0a-acaf-4a4a-aed9-fc37c66f8a0a

📥 Commits

Reviewing files that changed from the base of the PR and between bf4c34e and d231cef.

📒 Files selected for processing: 31

.claude/skills/verify-debatelab/SKILL.md
.claude/skills/verify-debatelab/features/README.md
.claude/skills/verify-debatelab/features/ai-tools.md
.claude/skills/verify-debatelab/features/case-generation.md
.claude/skills/verify-debatelab/recipes/ai-tools.spec.ts
.claude/skills/verify-debatelab/recipes/case-generation.spec.ts
docs/architecture.md
e2e/full-stack/fake-openrouter.mjs
e2e/full-stack/stack-core.ts
packages/core/src/ai/schemas/ask.ts
src/components/features/ask/AskChat.tsx
src/locales/en.json
src/locales/es.json
src/locales/hu.json
supabase/README.md
supabase/functions/_shared/model-catalog.ts
supabase/functions/_shared/output-caps.test.ts
supabase/functions/_shared/output-caps.ts
supabase/functions/ai-stream/index.ts
supabase/functions/ai-stream/index_test.ts
supabase/functions/ai/features/ask.ts
supabase/functions/ai/features/attack.ts
supabase/functions/ai/features/brainstorm.ts
supabase/functions/ai/features/crossfire.ts
supabase/functions/ai/features/factoids.ts
supabase/functions/ai/features/generate.ts
supabase/functions/ai/features/sparring.ts
supabase/functions/ai/handler-research.test.ts
supabase/functions/ai/index.ts
supabase/functions/ai/provider.ts
supabase/functions/ai/stream.test.ts

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Note

🎁 Summarized by CodeRabbit Free

Your organization is on the Free plan. CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please upgrade your subscription to CodeRabbit Essentials by visiting https://app.coderabbit.ai/settings/billing.

_{Comment @coderabbitai help to get the list of available commands.}

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

vercel · 2026-10-22T12:40:16Z

Deployment failed for project debatelab-staging with the following error:

Resource is limited - try again in 24 hours (more than 100, code: "api-deployments-free-per-day").

Learn More: https://vercel.com/adam-gaborjanis-projects?upgradeToPro=build-rate-limit

vercel · 2026-10-22T12:40:17Z

Deployment failed for project debate-navigator with the following error:

Resource is limited - try again in 24 hours (more than 100, code: "api-deployments-free-per-day").

Learn More: https://vercel.com/adam-gaborjanis-projects?upgradeToPro=build-rate-limit

…the cap math An Ask stream that spent its whole cap on reasoning now fails so the client falls back to the structured endpoint, which retries with a larger cap. The Sparring speech and scorecard caps fit a full-length Hungarian reply. The cap formula uses integer percentages, and a table test pins it from outside. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Keep #182 retry of a reply cut off at its per-feature cap (now with the strict schema) and drop this branch no-retry-on-truncation path, which the per-feature caps superseded.

Programcode · 2026-10-22T16:11:07Z

Post-merge follow-up.

Staging. The staging sync deployed ai, ai-stream and voice from 6260da2 at 14:10 UTC, with a staging-sync success status on the merge commit.

Recipes (fake mode). I ran them at 25c07542, which has the same AI code as the merge. The touched recipe files (ai-tools, case-generation, practice, live-generation) passed all 21 tests on both desktop-chromium and mobile-chromium. That includes the new ask-cutoff and gen-cutoff recipes:

ask-cutoff: a streamed answer cut off at its cap keeps the partial text and shows the translated notice under Uncertainties, with no error toast.
gen-cutoff: the fake provider recorded max_tokens 8,048 on the first attempt and 15,000 on the retry. Those match the formula for a 6,000- and a 12,000-token reply at low effort.

Post-merge Codex review of 6260da20. It had one finding. A stream that sent only whitespace before hitting its cap left an empty answer instead of falling back to the structured Ask. #187 fixes that, with a regression test that fails on main. Its Codex review found nothing.

Programcode and others added 2 commits October 22, 2026 11:16

Cap each AI feature's reply length and retry cut-off replies

e5b4d98

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Keep reasoning room for effort-based models and translate the cut-off…

78164d2

… notice on render Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot AI balanced review requested due to automatic review settings October 22, 2026 12:35

Copilot AI reviewed Oct 22, 2026

vercel Bot deployed to Preview – debatelab-staging October 22, 2026 12:37

View deployment

vercel Bot deployed to Preview – debate-navigator October 22, 2026 12:37

View deployment

Merge main into cost/per-feature-token-caps

f9159f7

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

vercel Bot deployed to Preview – debatelab-staging October 22, 2026 12:54

View deployment

Programcode mentioned this pull request Oct 22, 2026

Fix the 11 bugs from today's QA pass (H1, M1–M5, L1–L5) #184

Merged

Programcode and others added 2 commits October 22, 2026 13:50

Merge main into cost/per-feature-token-caps

25c0754

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Merge remote-tracking branch 'origin/main' into s182

d231cef

Programcode merged commit 6260da2 into main Oct 22, 2026
1 of 14 checks passed

Programcode mentioned this pull request Oct 22, 2026

Run Claude only on the Anthropic BYOK key, with no fallback provider #185

Merged

This was referenced Oct 22, 2026

Fall back to the structured Ask when a failed stream sent only whitespace #187

Merged

Move saved Opus 5.5 settings to Sonnet 5.5 (data migration, after #164) #180

Merged

Repair model JSON locally and enforce schemas before paying for a retry #176

Merged

Programcode deleted the cost/per-feature-token-caps branch October 22, 2026 16:22

Navigation Menu

Repository navigation

Cap each AI feature's reply length instead of 32,768 tokens - #182

Cap each AI feature's reply length instead of 32,768 tokens#182
Programcode merged 6 commits into
mainfrom
cost/per-feature-token-caps

Programcode commented Oct 22, 2026 •

edited

Loading

Uh oh!

vercel Bot commented Oct 22, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

coderabbitai Bot commented Oct 22, 2026 •

edited

Loading

Review limit reached

Uh oh!

vercel Bot commented Oct 22, 2026

Uh oh!

vercel Bot commented Oct 22, 2026

Uh oh!

Uh oh!

Programcode commented Oct 22, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Programcode commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

The caps

Why max_tokens is bigger than the reply cap

Cut-off replies

Verify tooling

Verification

Reviews

Out of scope

Uh oh!

vercel Bot commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review limit reached

Uh oh!

vercel Bot commented Oct 22, 2026

Uh oh!

vercel Bot commented Oct 22, 2026

Uh oh!

Uh oh!

Programcode commented Oct 22, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Programcode commented Oct 22, 2026 •

edited

Loading

vercel Bot commented Oct 22, 2026 •

edited

Loading

coderabbitai Bot commented Oct 22, 2026 •

edited

Loading