Conversation
apucacao
force-pushed
the
alexis/grounded-judge-context
branch
from
September 11, 2026 10:16
1999481 to
8841bc3
Compare
A judge grades a response that already exists. By the time one runs, the provider call is finished and billed. Both judge loops resolved the judge's own AI Config with an unguarded `extractVariation`. That call throws `Variation <key> is not enabled` when the config's served variation is disabled. Four call sites await `runJudges` without a guard, so the error travelled all the way out. The caller paid for a completion and received an exception instead. The trigger is deliberate and harmless. Someone toggles a judge off in LaunchDarkly, and every request that judge is attached to starts to fail. I found this by running the `agent` integration scenario. It died on `Variation launch-darkly-summarization-judge is not enabled`, after the model had already answered. Route both loops through `resolveJudge`. It returns null and logs instead of throwing, so the run skips a judge it cannot resolve. That matches how this file already handles a judge with no compatible handler, and how `inspectConfig` reports a disabled config rather than throwing. A resolution failure on the main config still fails the run. You genuinely cannot run without that config.
apucacao
force-pushed
the
alexis/grounded-judge-context
branch
from
September 11, 2026 14:06
8841bc3 to
63e43ad
Compare
apucacao
changed the base branch from
main
to
alexis/judge-output-format-ignored
September 11, 2026 14:06
apucacao
force-pushed
the
alexis/grounded-judge-context
branch
from
September 11, 2026 18:29
63e43ad to
6ff4d6c
Compare
Callers can now hand config().invoke()/.stream() a lazy judgeContext callback that resolves once, right after the primary handler settles and before output-format parsing — even when no judge ends up sampled, because sampling controls execution, not the freeze boundary. The resolved value must be acyclic JSON under 64 KiB; on success it comes back unchanged on ProviderResponse.judgeContext and is injected into every judge's message_history inside a delimited UNTRUSTED_ACTUATOR_EVIDENCE block, so a judge can ground its verdict in evidence the primary model produced (e.g. via a tool) without that evidence ever reaching the primary model's own prompt, track data, or spans. Every judge now runs inside its own isolation boundary: a disabled judge config, a provider failure, an unparseable verdict, a track() failure, or a per-judge timeout (judgeTimeoutMs, default 30s) each produce one JudgeDiagnostic instead of raising or silently discarding the response the caller already paid for. Duplicate judge keys in a single judgeConfiguration only run once. Judge reasoning is capped at 4 KiB before it reaches judgeResults or a worker payload. The skipJudges path (buildJudgeTasks/runJudge) carries the same resolved context and diagnostics through JudgeTask so a background worker thread reconstructs the identical judge prompt. This also folds in the already-reviewed judge-isolation fix from public PR #17 (cherry-picked as its own commit): a judge's own AI Config resolving to a disabled variation no longer throws out of config().invoke() after the primary call was already billed. Graph nodes do not receive a caller-supplied judgeContext in v1; graph()/resolveGraph() forward judgeDiagnostics from a graph-level judge only. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> BREAKING CHANGE: runJudges and buildJudgeTasks now return an object ({ judgeResults, judgeDiagnostics } and { judgeTasks, judgeDiagnostics }) instead of the bare value. The per-entry shape inside judgeResults is unchanged. runJudges is internal; buildJudgeTasks is exported.
apucacao
force-pushed
the
alexis/grounded-judge-context
branch
from
September 11, 2026 18:29
6ff4d6c to
2e68040
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A judge is a second AI call that grades the first one. Three things were wrong with how it ran. This PR fixes all three.
The judge was grading blind. It saw the user request and the model's answer, but not the data the run actually looked at. So it had to guess. It now receives that evidence as context.
A broken judge could sink a good answer. Judges run after the primary call is finished and billed. If the judge's own config or provider failed, the error propagated out and threw away the response the caller already paid for. Each judge is now isolated: its failure is recorded, not raised.
Failures were invisible. A judge that was disabled, timed out, or errored simply vanished from the results, with no way to tell which. Every skip or failure now leaves a
JudgeDiagnosticbehind.What changed
ConfigArgsgainsjudgeContext(a lazy callback, resolved once after the primary call) andjudgeTimeoutMs.ProviderResponse, the streamdoneevent, andJudgeTaskgainjudgeContext; those plusProviderGraphResponsegainjudgeDiagnostics.message_historyinsideUNTRUSTED_ACTUATOR_EVIDENCE_BEGIN/..._END. It never reaches the primary model and is never recorded in telemetry. A judge's own prompt is a LaunchDarkly AI Config and stays swappable at runtime; the evidence rides in a fixed slot that prompt cannot displace.Breaking change
runJudges(internal) andbuildJudgeTasks(exported) now return{ judgeResults, judgeDiagnostics }and{ judgeTasks, judgeDiagnostics }instead of the bare value. The per-entry shape inside is unchanged. This is a clean break on purpose: a wrapper that returned only results would reintroduce the silent-failure hole point 3 closes.Stacked
Based on #56 (
fix(judges): a judge config's outputFormat must not reach the provider), not onmain, because both changejudges.ts. Review #56 first; its diff is small. GitHub retargets this PR tomainautomatically when #56 merges.