fix(judges): a judge's own config must not fail the run it grades

[apucacao]

The bug

Warning

This content was written by AI.

A judge grades a response that already exists. By the time the judge runs, the provider call is finished and billed.

Both judge loops resolved the judge's own AI Config with an unguarded extractVariation (judges.ts:62 in runJudges, and judges.ts:188 in buildJudgeTasks). That call throws Variation <key> is not enabled when the judge's served variation is disabled. Four call sites await runJudges without a guard: client.ts:108, client.ts:160, graph.ts:211 and graph.ts:317. The error therefore travelled all the way out.

What the caller saw:

1. The model call succeeds. LaunchDarkly bills it.
2. Someone turns off a judge attached to the config.
3. The whole call fails. The caller gets an exception, not the response they paid for.

The trigger is deliberate and harmless. Someone flips a judge off in the user interface. Every request that judge is attached to then starts to fail.

How I found it

I ran the agent integration scenario against a live environment. It died with Variation launch-darkly-summarization-judge is not enabled, after the model had already answered.

The fix

Both loops now call resolveJudge. It returns null and logs, instead of throwing. The run skips a judge it cannot resolve, and continues.

Two patterns in this codebase already work that way:

• judges.ts already skips a judge that has no compatible handler, rather than failing. The comment says "skip this judge rather than calling the wrong provider's API".
• lifecycle.ts: inspectConfig returns { enabled: false, config: null } for a disabled config, rather than throwing.

A resolution failure on the main config still fails the run. You genuinely cannot run without that config, and client.test.ts has a test that pins the behaviour.

Tests

Three new tests, 823 to 826. I checked each one by reverting judges.ts and confirming that those three, and only those three, fail.

• judges.test.ts: the run skips a disabled judge, returns {}, never calls executeAndTrack, and logs the skip.
• judges.test.ts: with two judges where one is disabled, the healthy judge still runs and produces its score.
• client.test.ts: invoke() leaves the disabled judge out of judgeTasks, and still returns the response.

yarn build, yarn typecheck, yarn test (826 tests), yarn code:check and yarn lint:pkg all pass.

For the reviewer

An earlier change in the pre-migration repo ("fix silent judge errors") made judge error handling louder. This PR keeps the failure visible through console.error. It only stops the failure from ending the run. Please check that this is the balance you want.

Provenance

Ported from launchdarkly-labs/tmp-launchdarkly-ai-sdk#58, which was open against the pre-migration repo and never reached its main, so it is absent from this repo. The commit is unchanged, and every added and removed line here is identical to the original.

---

Note

Overview
Fixes a failure mode where disabling a judge’s LaunchDarkly AI config could throw after the main model call completed, so callers lost the billed response.

Judge resolution now goes through resolveJudge, which catches extractVariation errors, logs Judge '<key>' skipped:, and returns null instead of propagating. runJudges and buildJudgeTasks both skip unresolvable judges and continue, consistent with skipping judges that have no compatible handler.

Main config resolution is unchanged: a disabled primary flag still fails the run.

Tests cover a fully disabled judge, a mix of disabled and working judges, and invoke() with skipJudges: true still returning the LLM response with empty judgeTasks.

^{Reviewed by Cursor Bugbot for commit aba6200. Bugbot is set up for automated code reviews on this repo. Configure here.}