Skip to content

Fix case generator issues from case-quality batch 1 - #263

Merged
Programcode merged 4 commits into
mainfrom
fix/case-prompt-batch1
Oct 29, 2026
Merged

Programcode merged 4 commits into
mainfrom
fix/case-prompt-batch1

Conversation

@Programcode

@Programcode Programcode commented Oct 28, 2026 •

Copy link
Copy Markdown
Owner

Fixes the case generator (Generate tool) against the findings from case-quality batch 1 (12 cases, ~/.t3/scratch/case-quality/batch1/). The reference is Adam's own debate prints. In those, R2 brings no new arguments, spends about 2 minutes backing up our arguments, then about 4 minutes on many full "If they say X, we answer Y" blocks.

Fixes, mapped to batch 1

1. Spoken text no longer talks about the research.
Batch 1 had spoken lines like "We do not have the survey numbers in our research" (12, en intro), "a kutatásban nincs olyan adat…" (10, hu R1), "A kapott kutatásból nem tudunk nemzetközi összehasonlító adatot idézni" (09) and "the research shows even governments are unsure" (08). The generate prompt now says the research is for the writer. No spoken field may mention it, what it doesn't cover, missing data or an undated source. With no number, the speaker makes the point without one. A note on missing evidence goes only into sourceQuery with needsSource: true. The evidence rule used to say "describe the evidence still needed", which pushed the gap into a spoken field. It now says to give the best example or reasoning instead. The prompt also overrides the shared research block's "state that evidence is limited" line, which other features still use.

2. Hungarian and Spanish cases look for national sources first.
Batch 1's Hungarian R2 on safe schools (06) cited Brazil, the CDC, YouthTruth and Victoria, and no Hungarian source. deepResearch now accepts optional language and country fields on its existing options object. The signature is unchanged, and calls without them, or with language: "en", send exactly the same requests as before (tested).

  • For hu or es, the first seed search goes to that country's sources: KSH, kormany.hu, Oktatási Hivatal, Hungarian universities and press; or INE, gob.es, BOE, CIS, Spanish universities and press. It comes first so the four-query limit can't drop it. When it finds nothing, it retries without the domain filter. The usual international searches follow as the fallback.
  • Ranking gives national pages (listed domains or any .hu/.es site) +12. Without that bonus, the five pages read in full would still be mostly foreign.
  • The follow-up query prompt asks for at least one query that finds national sources.
  • The Generate handler derives the country from the output language (hu gives HU, es gives ES). The language is already validated server-side.

3. R2: 10-12 full "If they say" blocks, one reinforce length.
Batch 1's R2 cases had 6 blocks of 170-190 words. The prompt now asks for 10-12 blocks covering their main arguments, then the supporting points behind each, then their answers to each of ours. Each block must be complete enough to say word for word. Brief asks for 10 answers of 100-130 words, standard for 12 of 120-150, and comprehensive for 12 of 150-180. That puts comprehensive at about 2,200 words of answers, inside the 12,000-token reply cap even in Hungarian. The contradiction is fixed: the role text and the JSON schema both say reinforce paragraphs are about 100-120 words (two paragraphs is roughly 2 minutes with the opening). control-enforcement.ts now targets 10 (brief) or 12 counters and keeps a reply up to three short (7 or 9 minimum), so a slightly short reply doesn't throw. Extra answers are trimmed. R1 still has no counter minimum in code: batch 1's R1s already had 5-8, the prompt asks for 5-8, and a minimum would break the fake provider's three-counter R1 used by the browser recipes.

4. Word budgets per speech role.
Batch 1's spoken text often ran over the time (the analyzer counted 960 spoken words for case 02, a 6-minute R1). Each role's request now carries a Speech Budget: line at 130 spoken words a minute, and the role text repeats it:

  • Introductor: 4 min, about 520 words, short form only (hook, "we will show", each argument as point, a two or three sentence explanation and one piece of evidence). Its "If they say" list stays, as a short prep list of 3-5 answers of 40-70 words outside the budget.
  • R1: 6 min, about 780 words. Opening, arguments and closing stay within about 600, leaving time for the answers the speaker picks (5-8 answers of 60-100 words).
  • R2: 6 min, about 780 words. Opening, backing up and closing stay within about 300; the rest is for the blocks the other side used.
  • Conclusor: 3 min, about 390 words for the whole speech.
    prompt-builder.test.ts checks that the prompt for every role and length carries its budget, and that a case with no speaker role carries none.

5. Rules that now apply to every speaker, not only R2.
Batch 1's R1 cases had "that cost is real" (01), "we agree" (04), "cross-sectional" (01, 07) and "meta-analysis" (03), and "contentions" turned up in every case. A new "SPOKEN TEXT RULES (every speaker)" section bans conceding, study-method words (say "a study of 10,000 pupils" instead), and debate jargon in sentences ("contentions", "weighing", "warrant", "framework" and similar). It notes these may be JSON keys but never words the speaker says. Sources from the last five years are preferred (it was two).

The case JSON shape is unchanged: no field was added or renamed.

Kept in step

  • e2e/full-stack/fake-openrouter.mjs returns 10 or 12 R2 answers.
  • The verify-debatelab case-generation recipe expects 12 R2 answers on the fake stack and 9-12 live. Its feature doc says twelve.
  • An R2 case saved into the debate context still passes only its first 8 answers to the next speaker (MAX_CASE_COUNTERS), within the 12,000-character case digest. I left that limit alone.

Verification

All local, at head 98e7a984. This is a local pass, not CI.

  • NODE_OPTIONS=--max-old-space-size=3072 npm run verify: exit 0. docs:check, seo:jargon, lint, typecheck clean; vitest 714/714; collaboration 30/30; edge (Deno) 502/502; build ok; test:sw ok; Playwright smoke 47/47.
  • verify-debatelab on the fake provider ($H up, drive recipes/case-generation, $H down): 10/10 on desktop-chromium and 10/10 on mobile-chromium, including the R2 check for 12 answers.
  • New or changed tests: national-first search requests and unchanged requests without a locale (ai/source-quality.test.ts); national ranking bonus; deepResearch in Hungarian searches, asks for and reads Hungarian sources first, and an English call is identical to the old call shape (_shared/research.test.ts); R2 targets 10/12 with the three-short floor (control-enforcement.test.ts, control-matrix.test.ts); speech budget per role and new R2 counts (prompt-builder.test.ts); the new prompt rules (prompt-examples.test.ts).

No live model, staging function or Exa call was made. Everything above is unit and prompt-text tests plus the fake-provider stack, so none of it proves the model now writes better cases.

Before this ships

  • Needs Adam's OK to merge.
  • After merge, the ai Edge Function needs a deploy, staging first and then production. A merge to main only ships the frontend.
  • Batch 2, a rerun of the same 12 cases (about $2 on the live function), needs Adam's spend OK. It's the only way to confirm these prompt changes work on real output.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • New Features
    • R2 debate generation now provides 10 counterarguments for brief cases and 12 for standard or comprehensive cases.
    • Research for Hungarian- and Spanish-language cases prioritizes relevant national sources when available, with international sources as alternatives.
  • Improvements
    • Generated speeches follow clearer role-specific length and speaking-time guidance.
    • Research favors recent sources. Unsupported points can use reasoning or examples instead of spoken references to missing evidence.
    • Notes about gaps in research are kept separate from spoken debate text.

- Spoken text never mentions the research or missing data; gaps go to
  sourceQuery with needsSource.
- Generate research prefers Hungarian or Spanish sources for cases in
  those languages, with international sources as the fallback.
- R2 asks for 10-12 full "If they say" blocks; reinforce paragraphs are
  100-120 words in both the role text and the schema; the R2 counter
  target is 10/12 and a reply up to three short is kept.
- Each role states its word budget at 130 spoken words a minute.
- The no-conceding, no-study-method and no-jargon rules apply to every
  speaker role.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings October 28, 2026 20:17

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

@coderabbitai

coderabbitai Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Recent review info
Run configuration
Commits

Reviewing files that changed from the base of the PR and between fd29987 and a771f86.

Files selected for processing: 8
  • .claude/skills/verify-debatelab/features/case-generation.md
  • docs/architecture.md
  • e2e/full-stack/fake-openrouter.mjs
  • supabase/functions/_shared/output-caps.ts
  • supabase/functions/_shared/prompts/generate.ts
  • supabase/functions/ai/features/generate.ts
  • supabase/functions/ai/r2-case.ts
  • supabase/functions/ai/r2-case_test.ts
Files skipped from review as they are similar to previous changes: 1
  • .claude/skills/verify-debatelab/features/case-generation.md

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

Walkthrough

R2 case generation now requests a plan and writes four parts concurrently. The changes revise R2 answer counts, add role-specific speech guidance, and pass language and country preferences through research for locale-aware source selection and ranking.

Changes: R2 generation

Layer / File(s) Summary
R2 prompts and speech budgets
supabase/functions/_shared/prompts/generate.ts, supabase/functions/_shared/prompts/prompt-examples.test.ts, supabase/functions/ai/prompt-builder.test.ts
Generation prompts add role-specific speech budgets and revise spoken-content and evidence instructions. R2 prompts specify answer counts, lengths, and plan and part-writing instructions.
R2 planning and parallel writing
supabase/functions/ai/r2-case.ts, supabase/functions/ai/r2-case_test.ts, supabase/functions/ai/features/generate.ts, supabase/functions/_shared/output-caps.ts, e2e/full-stack/fake-openrouter.mjs, docs/architecture.md
The R2 writer validates a plan, writes four claim parts concurrently, streams part 1, merges outputs, and tracks usage. The generation entry point routes R2 requests to the writer. Output caps, fake responses, tests, and architecture documentation cover this flow.
R2 answer counts and validation
supabase/functions/_shared/control-enforcement.ts, supabase/functions/_shared/control-enforcement.test.ts, supabase/functions/ai/control-matrix.test.ts, .claude/skills/verify-debatelab/features/case-generation.md, .claude/skills/verify-debatelab/recipes/case-generation.spec.ts
R2 targets increase to 10 answers for brief cases and 12 otherwise. Enforcement tests and case-generation checks cover the updated counts.

Locale-aware research

Layer / File(s) Summary
Locale-specific source selection and ranking
supabase/functions/_shared/source-quality.ts, supabase/functions/ai/source-quality.test.ts
Source-quality helpers resolve Hungarian and Spanish locale preferences. Search requests can add a national-source query, and result ranking adds a bonus for matching country domains. Tests cover supported and unsupported locales.
Locale propagation through research
supabase/functions/ai/features/generate.ts, supabase/functions/_shared/research.ts, supabase/functions/_shared/research.test.ts
Generation passes language and derived country preferences to deep research. Deep research uses the locale for search requests, result ranking, and follow-up-query prompts. Tests check Hungarian source selection and English behavior.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant handleGenerate
  participant deepResearchInner
  participant buildSearchRequests
  participant Exa
  participant rankSearchResults
  handleGenerate->>deepResearchInner: Pass language and country preferences
  deepResearchInner->>buildSearchRequests: Build seed requests with locale
  buildSearchRequests-->>deepResearchInner: Return locale-aware requests
  deepResearchInner->>Exa: Search with seed and follow-up requests
  Exa-->>deepResearchInner: Return search results
  deepResearchInner->>rankSearchResults: Rank results with locale
  rankSearchResults-->>deepResearchInner: Return ranked results
Loading
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Note

🎁 Summarized by CodeRabbit Free

Your organization is on the Free plan. CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please upgrade your subscription to CodeRabbit Essentials by visiting https://app.coderabbit.ai/settings/billing.

Comment @coderabbitai help to get the list of available commands.

Programcode and others added 3 commits October 28, 2026 20:26
…ce-limited rule in the no-research-talk override, give general-role cases R1 counters
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… target

R2 cases came within 6 s of the 130 s request budget (124 s for a Hungarian
case at medium effort). Both Supabase projects are on the Free plan, which
stops a worker at 150 s, so the budget cannot grow. An R2 case is now planned
at low effort (our two arguments and the claims to answer), then written in
four parts at the same time at the student's model and effort. Part 1 writes
the speech and streams to the browser. Each part has its own smaller cap, which
keeps its reasoning budget at 8,000 tokens at medium. Measured: 68 s, 58 s and
53 s, down from 124, 109 and 105. A failed part stops the others and every
call is billed.

R1 speeches ran to 752 and 823 words against a 600-word budget. The Speech
Budget line now gives each argument field a word target derived from the
speaker's time and the number of arguments. Measured: 533 and 531.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Programcode
Programcode merged commit d11ed94 into main Oct 29, 2026
1 of 6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants