fix(judges): a judge config's outputFormat must not reach the provider - #86
Draft
apucacao wants to merge 1 commit into
Draft
fix(judges): a judge config's outputFormat must not reach the provider#86apucacao wants to merge 1 commit into
apucacao wants to merge 1 commit into
Conversation
A judge is an ordinary AI Config, so it can carry its own outputFormat (a JSON
Schema). Two readers disagreed about what to do with it: judges.py ignores it
and parses the reply as {score, reasoning}, while every provider handler
(openai/claude/langchain) honors config["outputFormat"] and hard-constrains
generation to the author's schema. The model ends up obeying a shape the judge
parser then rejects, so the judge silently produces no score and tracks no
metric.
This is live: a read-only sweep of 215 AI Config variations across four
LaunchDarkly connections (2026-09-11) found 30 with mode: judge, and one of
those, some-judge-config in staging project test-ai-config-project, has
outputFormat set to a schema with no score field at all. That judge can never
return a valid verdict.
Adds _without_output_format(), a private helper in judges.py that returns a
shallow copy of the config with outputFormat removed (or the input unchanged
when absent, never mutated). Applied in run_judges before execute_and_track,
in build_judge_tasks before the config is stored on a JudgeTask, and
defensively in run_judge for older serialized tasks. Logs a warning once per
judge, naming the judge key, only when outputFormat was actually present.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A judge is an ordinary AI Config, so it can carry
outputFormat(a JSON Schema). Two code paths read that one field with opposite intent:judges.pyignores it. It appends its formatting instructions and parses the verdict as{ score: number 0-1, reasoning: string }. The verdict contract belongs to the SDK.execute_and_track, which hands the whole judge config to the handler.openai-messagessetstext.format = json_schemafromconfig.outputFormat; claude and langchain handlers read it too.So the model is told to produce two different shapes, obeys the provider-level schema, and the judge parser then rejects the verdict. The judge produces no score and tracks no metric, silently.
This is live. A read-only sweep of 215 AI Config variations across four LaunchDarkly connections found 30 with
mode: judge, and one of those setsoutputFormatto{ message: string, script: string }. Nothing about a score. That judge can never return a valid verdict, and nothing warns its author.Fix
Strip
outputFormatfrom a judge config before it can reach a handler or be stored on aJudgeTask. One helper, applied at the three places a judge config is handed off:run_judges, beforeexecute_and_trackbuild_judge_tasks, before the config is persisted on the taskrun_judge, defensively, so an older serialized task still behavesWhen the field was actually present, say so once per judge. The helper returns the same reference when the field is absent and never mutates the input, which may be a cached
extract_variationresult.Not in scope, deliberately: the score contract, the parser, the formatting instructions, and handler selection are untouched. Making the SDK honor a judge's schema, or substituting the SDK's own
{ score, reasoning }schema to get structured output for judges, are both follow-ups.