Repository navigation
feat(slack): Route passive replies with a decision model - #2238
Merged
Merged
Conversation
Upgrade to AI SDK 7 and AI Gateway provider 4, which add experimental_decide and gateway decision models. Pi does not depend on the AI SDK, so the upgrade only touches Junior's direct callers and the gateway URL that tests and evals fake (/v3/ai moved to /v4/ai). The passive subscribed-thread router now asks one choice question (reply, stay_quiet, unsubscribe) through completeDecision instead of a structured LLM completion. The choice probability replaces the self-reported confidence for the existing 0.7 gate. Decision models return no prose, so reasonDetail now records the route and probability. AI_DECISION_MODEL / decisionModelId selects the model. It defaults to typesafe-ai/jev; openai/gpt-6-luna-decisions also works. Route cost uses gateway input rates so passive_reply_routed events keep recording. Co-Authored-By: David Cramer <david@sentry.io>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Sentry Evalsjunior-behavioral
Base commit junior-guardian
Base commit junior-integration
Base commit junior-router
Base commit |
dcramer
marked this pull request as ready for review
October 26, 2026 19:51
dcramer
approved these changes
Oct 26, 2026
Jev routed plain-name messages to a coworker, such as "Alex, ...", and follow-ups in that human exchange as replies. The passive participation evals that stayed quiet on the old router now fail. State the name-address and human-exchange rules in the policy, and give the reply and stay_quiet criteria discriminating definitions instead of restating the route name.
The name-address rule used the same names as the passive evals, and the human-exchange rule listed the eval phrasings. Keep the general rules and drop the examples so the evals measure the policy, not memorized cases. Co-Authored-By: David Cramer <david@sentry.io>
Contributor
|
2 screenshot changes — 2 changed · 0 added · 0 removed Review screenshots in Frameshift
|
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This moves the passive subscribed-thread reply router from a structured LLM completion to the AI SDK decision API, so it can run on a classifier model such as Jev. Passive routing stays behind the
passive-routingexperimental flag.AI SDK 7 upgrade
ai6.0.190 → 7.0.153 and@ai-sdk/gateway3 → 4.0.126. Both are older than the 24h release quarantine, so no exception is needed.chat/pi/client.tsandtools/web/search.tsimportai, and both typecheck as they are./v3/aito/v4/ai. I updated the MSW turn-router fixture, the eval web fixture, and the recording-rule comment to match.chat@4.29.0declares peerai@^6, but the rootaioverride already pins one version. Junior does not importchat/ai, andpnpm peers checkshows no new problems.Router
completeDecisioninchat/pi/client.tswrapsexperimental_decidewithgateway.decisionModel(...). It emits agen_ai.decidespan, maps refusals toinvalid_response, and estimates cost from gateway input rates: Jev $0.042/M, Luna-decisions $0.10/M, from/v1/models. With the cost estimate,passive_reply_routedevents keep recording.choicequestion:reply/stay_quiet/unsubscribe. The policy text is the same Guardian-style floor policy, with outputs renamed to routes. The probability of the chosen route replaces the old self-reportedconfidencefor the 0.7 gate.reasonDetailis now"<route> p=<prob>"instead of a model-written sentence.AI_DECISION_MODEL/decisionModelIdsetting. It defaults totypesafe-ai/jev.openai/gpt-6-luna-decisionsis the OpenAI Decisions alternative.Test bookkeeping
check-test-architecturenow countscompleteDecision:fakes as model fakes, so the baseline counts are unchanged.watched-message-behavior.test.tsdropped under 1,000 lines, so I removed its file-length exception.Checks run
pnpm lintpasses. This includes oxlint, depcruise, file-length, and test architecture.pnpm --filter @sentry/junior typecheckandpnpm --filter @sentry/junior-evals typecheckpass.tests/unit/routing,tests/unit/pi, config tests, andjunior-evalsharness tests pass.watched-message-behaviorpasses 21/21 with--testTimeout=30000. Some agent-reply integration tests in this sandbox time out at the default 5s. These include durable-queue, annotation-cards, event-wake-delay, and bot-image-hydration. They fail the same way on a cleanorigin/mainworktree, and timings are similar on both, so this looks like sandbox slowness, not this change.Not run
evals/conversation/participation.eval.ts). This sandbox has noROACH_TOKENor gateway credentials. The new/v4/aianddecision-modelrequests need fresh recordings from CI. Comparing pass rates with the old Luna path is the main open item.via David Cramer.
--
View Junior Session [Sentry]