Repository navigation
test(evals): Compare passive router on gpt-6-luna-decisions [do not merge] - #2240
Closed
sentry-junior[bot] wants to merge 4 commits into
Closed
sentry-junior[bot] wants to merge 4 commits into
sentry-junior[bot] wants to merge 4 commits into
Conversation
Upgrade to AI SDK 7 and AI Gateway provider 4, which add experimental_decide and gateway decision models. Pi does not depend on the AI SDK, so the upgrade only touches Junior's direct callers and the gateway URL that tests and evals fake (/v3/ai moved to /v4/ai). The passive subscribed-thread router now asks one choice question (reply, stay_quiet, unsubscribe) through completeDecision instead of a structured LLM completion. The choice probability replaces the self-reported confidence for the existing 0.7 gate. Decision models return no prose, so reasonDetail now records the route and probability. AI_DECISION_MODEL / decisionModelId selects the model. It defaults to typesafe-ai/jev; openai/gpt-6-luna-decisions also works. Route cost uses gateway input rates so passive_reply_routed events keep recording. Co-Authored-By: David Cramer <david@sentry.io>
Jev routed plain-name messages to a coworker, such as "Alex, ...", and follow-ups in that human exchange as replies. The passive participation evals that stayed quiet on the old router now fail. State the name-address and human-exchange rules in the policy, and give the reply and stay_quiet criteria discriminating definitions instead of restating the route name.
The name-address rule used the same names as the passive evals, and the human-exchange rule listed the eval phrasings. Keep the general rules and drop the examples so the evals measure the policy, not memorized cases. Co-Authored-By: David Cramer <david@sentry.io>
Experiment only. Switch the default decision model so CI runs the passive evals on OpenAI Decisions for comparison with Jev. Co-Authored-By: David Cramer <david@sentry.io>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Sentry Evalsjunior-behavioral
Base commit junior-guardian
Base commit junior-integration
Base commit junior-router
Base commit |
Contributor
|
2 screenshot changes — 2 changed · 0 added · 0 removed Review screenshots in Frameshift
|
Contributor
Author
|
Comparison done. On the passive evals, Luna-decisions and Jev gave the same results: 7 of 8 passed, and both failed only "only shares topic vocabulary". Closing this experiment, as planned. |
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Experiment only. Do not merge.
This branch is #2238 (at
d5953f8) plus one change: the default decision model isopenai/gpt-6-luna-decisionsinstead oftypesafe-ai/jev. The router code and policy are the same, so the passive evals compare the two decision models directly.It targets
mainbecause the eval workflows run only on pull requests tomain. The diff therefore includes all of #2238.Compare the
evals/conversation/participation.eval.tsresults with the #2238 runs on Jev. Close this PR after the comparison.via David Cramer.
--
View Junior Session [Sentry]