feat(client): grounded judge context and per-judge diagnostics - #84
Draft
apucacao wants to merge 1 commit into
Draft
feat(client): grounded judge context and per-judge diagnostics#84apucacao wants to merge 1 commit into
apucacao wants to merge 1 commit into
Conversation
apucacao
force-pushed
the
alexis/grounded-judge-context
branch
from
September 11, 2026 10:17
1279bfd to
8a96052
Compare
Callers can now hand judges evidence about what actually happened during a request. config() takes a lazy judge_context callback because the value does not exist yet when config() is called: the caller's tools fill it while the primary handler runs. The SDK resolves it exactly once, right after the primary handler succeeds and before output-format parsing, so every request with a callback freezes the same snapshot whether or not a judge is sampled. A resolved context must be acyclic JSON of at most 64 KiB encoded. It is returned unchanged on ProviderResponse.judge_context and reaches a judge only through that judge's message_history variable, between the UNTRUSTED_ACTUATOR_EVIDENCE_BEGIN and UNTRUSTED_ACTUATOR_EVIDENCE_END lines. It never reaches the primary model, the track data, or a span. With no callback the judge prompt is byte-identical to before. Judges are now isolated from each other and from the primary result. Config lookup, provider call, parse and tracking each sit behind their own boundary, bounded by judge_timeout_ms, and a failure produces one JudgeDiagnostic instead of discarding work that already succeeded. A judge that beats the clock and then fails to track keeps its result. A judge that misses the clock has its late completion consumed silently, so it can neither mutate results nor emit the score metric. Diagnostics carry codes only, never exception text, and the same codes are shared with the TypeScript SDK. run_judges and build_judge_tasks now return result objects carrying both the results and the diagnostics, which is a breaking change for direct callers. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> BREAKING CHANGE: run_judges and build_judge_tasks now return a result object (RunJudgesResult with judge_results/judge_diagnostics, and BuildJudgeTasksResult with judge_tasks/judge_diagnostics/judge_context) instead of the bare value. The per-entry shape inside judge_results is unchanged. Both are exported from the package.
apucacao
changed the base branch from
main
to
alexis/judge-output-format-ignored
September 11, 2026 14:09
apucacao
force-pushed
the
alexis/grounded-judge-context
branch
from
September 11, 2026 14:09
8a96052 to
c5635ab
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A judge is a second AI call that grades the first one. Three things were wrong with how it ran. This PR fixes all three.
The judge was grading blind. It saw the user request and the model's answer, but not the data the run actually looked at. So it had to guess. It now receives that evidence as context.
A broken judge could sink a good answer. Judges run after the primary call is finished and billed. If the judge's own config or provider failed, the error propagated out and threw away the response the caller already paid for. Each judge is now isolated: its failure is recorded, not raised.
Failures were invisible. A judge that was disabled, timed out, or errored simply vanished from the results, with no way to tell which. Every skip or failure now leaves a
JudgeDiagnosticbehind.What changed
config(...)gainsjudge_context(a lazy callback, resolved once after the primary call) andjudge_timeout_ms.ProviderResponse, the streamdoneevent, andJudgeTaskgainjudge_context; those plus the graph response gainjudge_diagnostics.message_historyinsideUNTRUSTED_ACTUATOR_EVIDENCE_BEGIN/..._END. It never reaches the primary model and is never recorded in telemetry. A judge's own prompt is a LaunchDarkly AI Config and stays swappable at runtime; the evidence rides in a fixed slot that prompt cannot displace.Breaking change
run_judgesandbuild_judge_tasksnow returnRunJudgesResult { judge_results, judge_diagnostics }andBuildJudgeTasksResult { judge_tasks, judge_diagnostics, judge_context }instead of the bare value. The per-entry shape inside is unchanged. This is a clean break on purpose: a wrapper that returned only results would reintroduce the silent-failure hole point 3 closes.Stacked
Based on #86 (
fix(judges): a judge config's outputFormat must not reach the provider), not onmain, because both changejudges.py. Review #86 first; its diff is small. GitHub retargets this PR tomainautomatically when #86 merges.