Repository navigation
feat(router): Route turns with a decision model - #2241
Draft
sentry-junior[bot] wants to merge 1 commit into
Draft
sentry-junior[bot] wants to merge 1 commit into
sentry-junior[bot] wants to merge 1 commit into
Conversation
Move the turn router from a structured completeObject call on the fast model to the AI SDK decision API on the configured decision model (AI_DECISION_MODEL, default typesafe-ai/jev). The router asks two choice questions. Profile options come from the configured profile descriptions. Reasoning-level options use the same criteria as before. Confidence is now the lower of the two chosen-option probabilities, so the 0.75 threshold uses model probabilities instead of a self-reported number. Decision models return no text, so the logged reason is now a 'handoff p=0.91, xhigh p=0.88' summary. Fallback, retry, reasoning floor, and profile reasoning overrides are unchanged. Co-Authored-By: David Cramer <david@sentry.io>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Sentry Evalsjunior-behavioral
Base commit junior-integration
Base commit junior-router
Base commit |
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #2238. The turn router now uses the AI SDK decision API on the configured decision model (
AI_DECISION_MODEL, defaulttypesafe-ai/jev), not acompleteObjectcall on the fast model.What changed
choicequestions in one decision call:profile: the options are the configured profiles. Each option's criteria text is that profile's description.reasoning_level: the options arenone…xhigh. Each option uses the same criteria text as the old system prompt.low/medium/highlabel handling is removed. When a provider returns no probabilities, the choice counts as certain.reason(spanlevel_reasonandturn_routed) is nowhandoff p=0.91, xhigh p=0.88style.medium, one retry oninvalid_response(this includes decision refusals), the reasoning floor, and profile reasoning overrides. An unknown profile or level from the model also falls back.botConfig.decisionModelId. The docs now list the turn router underAI_DECISION_MODEL.Tests
tests/unit/services/turn-router.test.tsis rewritten for the decision call. It covers the question shape, confidence from probabilities, missing probabilities, and an unknown-choice fallback.@/chat/pi/clientfakecompleteDecisionand spread the real module, so the model-fake lint count stays at baseline.mockTurnRouterModelnow answers/v4/ai/decision-model.Verification
pnpm lintand typecheck (junior,junior-evals) pass.bot-handlers,agent-continue-slack, and config.evals/router/*) locally because there is no recording token. The CI run on this PR is the Jev comparison against the 12/12 Luna baseline. I did not change the eval cases.via David Cramer.
--
View Junior Session [Sentry]