Skip to content

fix(openai): log output and metrics for incomplete Responses API streams - #829

Closed
Raphael Fakhri (RaphaelFakhri) wants to merge 1 commit into
braintrustdata:mainfrom
RaphaelFakhri:fix/openai-responses-stream-incomplete
Closed

Raphael Fakhri (RaphaelFakhri) wants to merge 1 commit into
braintrustdata:mainfrom
RaphaelFakhri:fix/openai-responses-stream-incomplete

Conversation

@RaphaelFakhri

Copy link
Copy Markdown
Contributor

Summary

Streaming responses.create calls that end with response.incomplete or response.failed now log their final output and token metrics.

ResponseWrapper._postprocess_streaming_results only read the final response object from response.completed events. A stream that stops at max_output_tokens or because of a content filter ends with response.incomplete instead. That event carries the usage, but the span was logged with no token metrics and only the partial output rebuilt from deltas, even though the request was billed. The response.failed event carries the same response object and had the same problem.

This change adds a _TERMINAL_RESPONSE_EVENT_TYPES set in py/src/braintrust/integrations/openai/tracing.py and uses it in both places that check for the final event.

Test plan

  • Added test_responses_stream_terminal_event_provides_output_and_metrics in py/src/braintrust/integrations/openai/test_openai.py, parametrized over response.completed, response.incomplete and response.failed.
  • Before the change, the response.incomplete and response.failed cases fail. After the change, all three pass.
  • ruff check and ruff format --check pass.
cd py && python -m pytest src/braintrust/integrations/openai/test_openai.py -k terminal_event

@AbhiPrasad

Copy link
Copy Markdown
Member

please test this by recording a vcr cassette!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants