Skip to content

fix(dspy): record LM history token metrics - #820

Merged
Abhijeet Prasad (AbhiPrasad) merged 3 commits into
mainfrom
abhi/sdk-427-record-dspy-default-engine-token-metrics
Sep 29, 2026
Merged

Abhijeet Prasad (AbhiPrasad) merged 3 commits into
mainfrom
abhi/sdk-427-record-dspy-default-engine-token-metrics

Conversation

@AbhiPrasad

@AbhiPrasad Abhijeet Prasad (AbhiPrasad) commented Sep 28, 2026 •

Copy link
Copy Markdown
Member

Summary

  • Record prompt, completion, and total token metrics from the matching DSPy LM history entry on dspy.lm spans.
  • Correlate entries with each call's inputs and returned output object, including concurrent calls with identical values; omit metrics for failed or unidentifiable calls.
  • Update the callback docs and extend the VCR-backed DSPy test to assert usage from the recorded OpenAI response. DSPy's cache is disabled in the test so it exercises the cassette.

The VCR test uses LiteLLM because DSPy's default lm15 transport uses raw sockets that VCR does not intercept. The cassette verifies token extraction from a real provider response through DSPy's history; it does not exercise lm15's transport itself.

Validation

  • Red: cassette metrics were missing before the tracing fix; stale-history, distinct concurrent-call, and identical concurrent-call regressions failed before correlation was added.
  • Green: mise exec -- nox -s 'test_dspy(latest)' passed (9 tests).
  • Targeted stale-history, concurrent-call, and VCR callback tests passed on both DSPy matrix versions (latest and 2.6.0).
  • Commit hooks passed, including Ruff format/check and codespell.

Fixes SDK-427.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 28, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-28T20:37:28.846046Z 1222b91 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a0b9e712ec

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "Codex (@codex) review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".

Comment thread py/src/braintrust/integrations/dspy/tracing.py Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 584a37acc1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "Codex (@codex) review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".

Comment thread py/src/braintrust/integrations/dspy/tracing.py
@AbhiPrasad
Abhijeet Prasad (AbhiPrasad) merged commit 6da9271 into main Sep 29, 2026
83 checks passed
@AbhiPrasad
Abhijeet Prasad (AbhiPrasad) deleted the abhi/sdk-427-record-dspy-default-engine-token-metrics branch September 29, 2026 14:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants