Skip to content

test: mocked coverage for multi-turn tool-call formatting in response_api builders - #49

Open
giladsamuels-cognition wants to merge 1 commit into
mainfrom
devin/1790797673-tool-call-builder-tests
Open

giladsamuels-cognition wants to merge 1 commit into
mainfrom
devin/1790797673-tool-call-builder-tests

Conversation

@giladsamuels-cognition

@giladsamuels-cognition giladsamuels-cognition commented Sep 30, 2026 •

Copy link
Copy Markdown

Summary

Until now, the multi-turn tool-call formatting in response_api.py (add_responded_tool_calls and each builder's handle_tool_call) was only tested by @pytest.mark.pricy tests, and CI skips those with -m 'not pricy'. This PR adds non-pricy tests to tests/llm/test_response_api.py. They reuse the existing create_mock_* helpers, and no production code changes.

What the tests check:

  • Full multi-turn loop through each mocked model (OpenAIResponseModel, ClaudeResponseModel, OpenAIChatCompletionModel): call 1 returns a tool call → response_text(...) → add_responded_tool_calls → call 2. The test then asserts the exact input= / messages= payload sent to create on call 2:
    • Responses API: raw output items, then {"type": "function_call_output", "call_id", "output"}
    • Anthropic: {"role": "assistant", "content": raw.content}, then {"role": "user", "content": [{"type": "tool_result", "tool_use_id", "content"}, ...]} (two parallel calls, order preserved)
    • Chat Completions: raw assistant message, then one {"name", "role": "tool", "tool_call_id", "content"} per call (two parallel calls)
  • Responses computer_call branch produces computer_call_output with an input_image output.
  • Guard cases:
    • An image in a function-call or tool response raises AssertionError (all three builders).
    • Text in a computer_call response raises AssertionError.
    • add_responded_tool_calls raises when only some tool calls have a response set.
    • A tool message whose responded_tool_calls is unset raises ValueError.

To confirm the tests catch real regressions, I temporarily broke the source in four places and 7 tests failed:

  • renamed the call_id key
  • changed the tool_use_id value
  • changed the tool_call_id value
  • disabled the all_responses_set assert

Result with the source restored: pytest -m 'not pricy' tests/llm/ gives 44 passed, 3 skipped.

Link to Devin session: https://app.devin.ai/sessions/b30c9ada7dad48f8921cb253abbf1def
Open in Devin Desktop: https://app.devin.ai/desktop/session/b30c9ada7dad48f8921cb253abbf1def?variant=devin
Requested by: @giladsamuels-cognition


Devin Review

Status Commit
⚪ Not started —

Run Devin Review

Devin Review (Staging)


Devin Review

…uilders without API calls

Co-Authored-By: Gilad Samuels <gilad.samuels@cognition.ai>
@devin-ai-integration

Copy link
Copy Markdown

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Devin Review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant