Repository navigation
Say why AI requests fail, offer the next step, and measure activation - #264
Merged
Merged
Conversation
ai_request_failed now carries tool, error_code, http_status, stage, streaming and duration_ms, mapped from the client's AI error classes. Each tool's error offers the matching next step: the plan dialog for a limit, Sign in again for a refused session, Retry where sending again can work. Every lazy component and route loader goes through loadChunk, which waits for the stale-chunk reload instead of destructuring a module that loaded as nothing (the /login crash), and reloads once per path when a chunk is missing. New client events: signup_completed, trial_started, paywall_shown, first_ai_success and welcome_completed, through the existing telemetry schema. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Warning Review limit reachedEnable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. Next included review available in 25 minutes. View limit details
Note 🎁 Summarized by CodeRabbit Free
Comment |
A TimeoutError's 408 is the browser's own deadline, so http_status left it reading as a server answer. namedChunk now throws ChunkLoadError for an export that is not there instead of handing React undefined. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ad again, and plainer sign-in copy
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
In the last 14 days, 16 of 56 AI requests failed (29%). That hit 12 of 29 users, and
ai_request_faileddidn't say why. This PR changes only the web client. It touches no Edge Functions, migrations or config.ai_request_failednow says why a request failed, using a fixed set of codes mapped from the client's existing error classes (src/lib/ai/failure-reason.ts). It sends no motion text, prompts, answers, emails or names. The telemetry schema insrc/lib/observability.tsdrops any other value, and a test checks that./loginwith "Your session expired. Sign in again.", and signing in with an email code or a demo code returns to the same page. Google sign-in lands on/app, because the login page only sends the return path through Google when no account was signed in before. The recipe checks the login page, not the return trip. Timeouts, network failures, provider errors, malformed answers and server errors keep Retry. A request that is too large (413) has no Retry. This covers Generate, Attack, Brainstorm, Factoids, Ask, Sparring and Crossfire. Sparring and Crossfire retries already reuse the round'squotaSession/sessionId, so a retry is not charged twice. The PR adds one string, "Sign in again", in EN, HU and ES.React.lazycomponents and every route loader now load through one helper,loadChunkinsrc/lib/lazy-chunk.ts. After a deploy, Vite'svite:preloadErrorlistener reloads the page, and Vite resolves the import toundefined. Callers then destructured nothing, which caused "(destructured parameter) is undefined".loadChunknow waits for that reload. A failed chunk import reloads once per path. ThesessionStorageguard is shared with the listener, so the page can't reload in a loop. A second failure, or one while offline, reaches the route error boundary. Fix rate-limit overrides, AI error reports, and the login crash #190 already guarded the analytics import named in the Sentry stack (import("./index-*.js").then(({Analytics}) => ...)), so that report most likely comes from a tab or service worker still running an older bundle. I haven't confirmed that against the Sentry release. This PR closes the same pattern where it remained (AppLayout,FirstRun, the onboarding tour, the page loaders).signup_completed,trial_started,paywall_shown,first_ai_successandwelcome_completed.Events and properties
ai_request_failedadds these properties:tool: generate, attack, brainstorm, factoids, ask, sparring or crossfireerror_code: limit_reached, model_restricted, auth_expired, timeout, network, aborted, provider_error, invalid_response, server_error or unknown. A 429 rate limit counts aslimit_reached, withhttp_status429.http_status: 100 to 599, when a server answered. A client-side timeout sends none.stage: request, stream (a streamed answer broke) or parse (the answer failed its check)streaming: boolean, now always sentduration_msThe existing
feature,errorName,statusCodeanddurationMsare still sent, because current insights and the Sentry duration metric read them. The overlap is deliberate.New events:
signup_completedwithmethod(email, google or demo). It fires on the first sign-in to an account created in the last 24 hours, once per account per browser. A demo code's account counts the first time the code is redeemed.trial_started. It fires when a new account (same 24-hour rule) first loads on the trial plan, once per account per browser.handle_new_userstarts the trial at sign-up, so it followssignup_completed. Accounts that existed before this ships never send it.paywall_shownwithreason(limit_reached, weekly_budget_reached, trial_ended, payment_failed, subscription_ended or model_restricted) andtool. It fires every time the limit or Pro-model dialog opens.first_ai_successwithtool. It fires on an account's first successful AI answer, once per account per browser.welcome_completed. It fires when the owl welcome closes, once per account per browser.There is no consent layer in the app. These events use the same gates as every other product event:
VITE_POSTHOG_KEYmust be set, and the closed schema must accept the event. The once-per-account guards keep the account id inlocalStorageonly. PostHog runs withpersistence: "memory"and never callsidentify, so it can't link one student's events across page loads. Funnels across sessions would need a decision on identity first. This PR doesn't change that.Verification
NODE_OPTIONS=--max-old-space-size=3072 npm run verifypassed locally atb8f18da1(exit 0). Docs check, SEO jargon check, lint (with knip and the repo rules) and typecheck all passed. Unit tests: 748 in 99 files. Collaboration tests: 30. Edge tests: 498. The production build, the service-worker offline check and the desktop Chromium@smokebrowser set (47 tests) also passed. This is a local pass, not CI. The first run caught three untyped stub functions in the new tests. After that full run, one follow-up commit (180f3da6) drops the 408 a client timeout used to report ashttp_status, and makesnamedChunkthrow a named error for a missing export. I checked that commit with typecheck, lint, and the two test files it touches.New unit tests:
src/lib/ai/failure-reason.test.ts(error mapping, the reason fields onai_request_failed, and the telemetry schema with PostHog mocked),src/test/lib/lazy-chunk.test.tsandsrc/test/lib/activation-events.test.ts(the once-only guards). Writing the mapping test caught a real bug: jsdom'sDOMExceptionisn't anError, so an abort would have been reported asunknown.I added a new verify-debatelab recipe,
recipes/ai-failures(feature filefeatures/ai-failures.md), and ran it on the local stack in fake mode. All 6 tests passed on desktop-chromium and all 6 on mobile-chromium. It forces each failure in the browser and covers Sign in again in Generate, Ask, Sparring and Crossfire, Retry after a network failure in Ask (Retry answers the question once), and the plan dialog on a limit.I also ran the recipes whose Retry flows these changes touch:
case-generation,usage-limitsandpractice. All 44 tests passed across both viewports.Not covered: a real timeout (it needs the 200 s client budget, so only the unit test covers it), staging, production, and whether the events arrive in PostHog. Nothing has been deployed.
Screenshots
Desktop (1280 px) and phone (mobile-chromium) shots of each error state are in
~/.t3/scratch/debateowl-next/ai-failures-shots/on the VPS:*-01-generate-sign-in-again.pngand*-02-login-after-sign-in-again.png*-01-ask-sign-in-again.png*-01-ask-network-retry.pngand*-02-ask-answered-after-retry.png*-01-sparring-sign-in-again.png*-01-crossfire-sign-in-again.png*-01-brainstorm-limit-dialog.pngBefore merging
This needs Adam's OK to merge. Merging to
maindeploys the frontend to production on Vercel, so it also needs his OK to deploy. No Supabase deploy is needed.🤖 Generated with Claude Code