Skip to content

feat(tracing): snapshot effective tools and handoffs per turn - #4629

Closed
sylvesterkaczmarek wants to merge 3 commits into
openai:mainfrom
sylvesterkaczmarek:fix/turn-capability-tracing-4626
Closed

feat(tracing): snapshot effective tools and handoffs per turn#4629
sylvesterkaczmarek wants to merge 3 commits into
openai:mainfrom
sylvesterkaczmarek:fix/turn-capability-tracing-4626

Conversation

@sylvesterkaczmarek

@sylvesterkaczmarek sylvesterkaczmarek commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Summary

This pull request adds per-turn tracing of the effective tools and handoffs visible to each model invocation.

Turn spans snapshot capability names after dynamic enablement, filtering, and tool-name collision resolution, while agent spans retain their existing latest-view and identity behaviour. The snapshots contain names only and work for both streaming and non-streaming runs. Runs with task/turn spans disabled retain the existing compact trace hierarchy.

The review follow-up also ensures that:

  • turn spans record Handoff.tool_name, including tool_name_override
  • namespaced FunctionTool entries use their canonical qualified trace identity
  • existing agent-span handoff and tool identities remain backwards compatible
  • regression coverage exercises both a handoff override and a namespaced function tool, including the exported turn-span payload

Test plan

  • Same-fork staging CI on the final review-fix head passed Ruff formatting/lint, typecheck, Windows mypy, native macOS sandbox, MCP v1 compatibility, prospective release contract, packaged Windows contract, and Python 3.10-3.13 test jobs.
  • The temporary staging PR was closed after the relevant validation above; the remaining Python 3.14 and Windows/packaged matrix legs were still running with no failures reported at that point.
  • Upstream OpenAI Actions is still action_required pending external-contributor approval.

Issue number

Closes #4626

Checks

  • I've added new tests, if relevant
  • I've run .agents/skills/code-change-verification/scripts/run.sh
  • I've confirmed all verification steps pass
  • If using Codex, I've run /review before submitting this PR

@seratch seratch left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the focused contribution. The overall approach in #4629 is the implementation path we want to pursue: capture each turn's resolved capability set on TurnSpanData after dynamic enablement, filtering, and collision resolution.

I also verified the proposed payload with a live Traces ingest probe. The exact custom-span structure, including list[str] values for tools and handoffs, returned HTTP 204 and was retained and rendered correctly in the Traces UI. No tracing-schema redesign is needed.

Before merging, please make these focused identity corrections:

  • Record Handoff.tool_name in TurnSpanData rather than Handoff.agent_name. With the supported tool_name_override, the current value can differ from the identifier offered to the model, and distinct handoff tools targeting the same agent become indistinguishable.
  • Use the existing namespace-aware canonical trace identity for FunctionTool definitions. The current generic helper can reduce namespaced tools to the unqualified tool.name.
  • Split the handoff projections in the helper: continue using agent_name for the released AgentSpanData.handoffs behavior, while using tool_name for the new per-turn capability snapshot.
  • Add focused regression coverage for one handoff with tool_name_override and one namespaced FunctionTool, asserting that the exported turn-span identifiers match the resolved capability identifiers.

The existing streamed/non-streamed dynamic-enablement test already covers the lifecycle placement and does not need broader permutation coverage.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4b516becb0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/agents/run_internal/run_loop.py Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ead01e1b01

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/agents/run_internal/run_loop.py Outdated
@sylvesterkaczmarek
sylvesterkaczmarek force-pushed the fix/turn-capability-tracing-4626 branch from 8624512 to adab4fc Compare August 24, 2026 23:30
@sylvesterkaczmarek

Copy link
Copy Markdown
Contributor Author

Addressed the requested identity corrections: turn spans now use Handoff.tool_name and namespace-aware FunctionTool identities, while legacy agent-span identities remain unchanged. Added regression coverage for tool_name_override and a namespaced FunctionTool. The review threads are resolved; the latest Actions run is waiting for maintainer approval to start.

@sylvesterkaczmarek

sylvesterkaczmarek commented Aug 24, 2026

Copy link
Copy Markdown
Contributor Author

@seratch thanks again, addressed the review feedback! Turn spans now use the model-visible handoff tool name and namespace-aware FunctionTool identity, while agent spans preserve their existing identities.

I also added focused coverage for a tool_name_override handoff and a namespaced FunctionTool, including assertions against the exported turn-span payload.

Same-fork validation is green across the relevant lint, typecheck, platform, and Python 3.10-3.13 test jobs; upstream Actions are still awaiting external-contributor approval.

@seratch

seratch commented Aug 25, 2026

Copy link
Copy Markdown
Member

Thanks for the thoughtful implementation and for working through the review feedback. After validating the behavior with both scripted and live-model traces, I do not think we should add this duplication to the tracing schema.

For each model turn, the existing model-call span already records the exact functions sent to the model, including dynamically enabled tools and handoffs represented as function tools. Adding the same capability list to the turn span would create another source of truth and ongoing identity and maintenance coupling without a demonstrated supported scenario that the model-call trace cannot handle.

I am going to close this PR and #4626. If a concrete tracing workflow shows that the model-call span is insufficient, we can revisit that narrower gap. Thanks again for the careful work here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Trace the effective tools and handoffs for each model turn

2 participants