feat: carry judge reasoning in the judge track payload and telemetry

[tracisiebel]

Summary

Judge reasoning is parsed on every sampled evaluation and then thrown away: run_judges puts it in the returned JudgeResult and client.track(...) sends only the score. This carries it off the process, both on the evaluation metric event and on the judge's evaluation span event, behind an explicit opt-in.

• TrackData.judgeReasoning now rides the judge's evaluation metric event, on the inline path and on the deferred run_judge path, so reasoning reaches LaunchDarkly through the evaluation pipeline that already carries the score.
• Off by default. LD_CAPTURE_JUDGE_REASONING=true (1/on/yes) is required before reasoning leaves the process; unset means score only.
• The opt-in is separate from the judge handler's capture_content, so a caller who wants this one field is not forced to ship every request and response.
• Truncated at 4000 characters, so a judge that writes an essay cannot inflate every sampled event.
• Documented with an explicit warning: a judge sees the full text it grades and its reasoning may quote it, and the SDK does no PII/token detection on that text.

 client.track(
     evaluation_metric_key,
     to_ld_context(client, user_context),
-    {**base_track_data, "judgeConfigKey": judge_key},
+    build_judge_track_data(base_track_data, judge_key, reasoning),
     score,
 )

-record_evaluation(numeric_score, reasoning if judge_handler.capture_content else None)
+record_evaluation(numeric_score, judge_explanation(reasoning))

Requested as a must-have: reasoning is visible nowhere in the dashboard because the data never leaves the customer's process, and customers are shipping it to LD Observability by hand to route around the eval pipeline.

Implementation details

Relationship to #43. #43 landed on main and already emits gen_ai.evaluation.result / gen_ai.evaluation.explanation on the judge's own invoke_agent span. This branch originally added a separate launchdarkly.judge span with launchdarkly.judge.* attributes; that is dropped in the merge in favour of #43's semconv shape, so there is one judge telemetry vocabulary rather than two. What remains from this PR is the part #43 does not cover: reasoning on the LD track payload, and the opt-in being a dedicated switch rather than whole-conversation capture.

Why opt-in. An earlier revision of this PR defaulted the capture on, reasoning that a score with no explanation is a number nobody can act on. That is an exfiltration risk: the judge is handed the entire response it grades, and its reasoning is free-form model prose that can quote sensitive spans verbatim. Nothing short of a second inference pass or PII detection can rule that out, so the default is now off and TELEMETRY-CONTRACT.md §4a carries the warning. Keeping the switch independent of capture_content still means opting in costs one field, not the whole conversation.

Scope. No handler package touched, no new dependency. uv.lock churn from running the toolchain was left out of the commits.

Testing. packages/client/tests/test_judges.py covers the track payload, the default-off behaviour (score retained, reasoning absent), empty reasoning, truncation, the capture_content-independent explanation, and the deferred run_judge payload. make lint, make format-check, mypy packages/client/src and the full suite (495 passed, 11 skipped) pass locally.

Not done here. The Monitoring surface that displays reasoning next to the score is a platform-side change; this PR only makes the data available.

Link to Devin session: https://app.devin.ai/sessions/e4c538a50a4141d7af1bcc8c424d0f11
Open in Devin Desktop: https://app.devin.ai/desktop/session/e4c538a50a4141d7af1bcc8c424d0f11?variant=devin
Requested by: @tracisiebel

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[devin-ai-integration]

@cursor review

[tracisiebel]

devin - can you resolve conflicts here plz

[devin-ai-integration]

Conflicts resolved — main now has #43, so I dropped the separate launchdarkly.judge span and kept #43's gen_ai.evaluation.* shape. What's left from this PR: TrackData.judgeReasoning on the metric event (inline + deferred), and gen_ai.evaluation.explanation no longer gated on the judge handler's capture_content — LD_CAPTURE_JUDGE_REASONING=false is the narrow opt-out instead. Full suite passes locally (495 passed, 11 skipped). Description updated.