Conversation
A judge is an ordinary AI Config, so it can carry outputFormat (a JSON
Schema). Two code paths read that field with opposite intent: judges.ts
ignores it and parses the reply as { score, reasoning }, but every
provider handler (openai-messages, openai-agents, claude-messages,
claude-agents, langchain-messages/agents) reads config.outputFormat and
hard-constrains the model to the author's schema instead. The model is
told to produce two different shapes, obeys the provider-level schema,
and the judge parser then rejects the verdict, so the judge silently
produces no score and tracks no metric.
This is live: a read-only sweep of 215 AI Config variations across four
LaunchDarkly connections found 30 with mode: judge, and one of those,
some-judge-config in staging project test-ai-config-project, has
outputFormat set to a schema about a completely different task (message,
script, error), with nothing about a score. That judge can never return
a valid verdict.
Strip outputFormat before a judge config reaches a handler or is stored
on a JudgeTask: in runJudges before executeAndTrack, in buildJudgeTasks
before the task is pushed, and defensively in runJudge in case an older
serialized task still carries it. Log once per judge, only when the
field was present, naming the judge key.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A judge is an ordinary AI Config, so it can carry
outputFormat(a JSON Schema). Two code paths read that one field with opposite intent:judges.tsignores it. It appendsFORMATTING_INSTRUCTIONSand parses the verdict as{ score: number 0-1, reasoning: string }. The verdict contract belongs to the SDK.executeAndTrack, which hands the whole judge config to the handler.openai-messagessetstext.format = json_schemafromconfig.outputFormat; claude and langchain handlers read it too.So the model is told to produce two different shapes, obeys the provider-level schema, and the judge parser then rejects the verdict. The judge produces no score and tracks no metric, silently.
This is live. A read-only sweep of 215 AI Config variations across four LaunchDarkly connections found 30 with
mode: judge, and one of those setsoutputFormatto{ message: string, script: string }. Nothing about a score. That judge can never return a valid verdict, and nothing warns its author.Fix
Strip
outputFormatfrom a judge config before it can reach a handler or be stored on aJudgeTask. One helper, applied at the three places a judge config is handed off:runJudges, beforeexecuteAndTrackbuildJudgeTasks, before the config is persisted on the taskrunJudge, defensively, so an older serialized task still behavesWhen the field was actually present, say so once per judge. The helper returns the same reference when the field is absent and never mutates the input, which may be a cached
extractVariationresult.Not in scope, deliberately: the score contract, the parser,
FORMATTING_INSTRUCTIONS, and handler selection are untouched. Making the SDK honor a judge's schema, or substituting the SDK's own{ score, reasoning }schema to get structured output for judges, are both follow-ups.