feat(tracing): snapshot effective tools and handoffs per turn - #4629
feat(tracing): snapshot effective tools and handoffs per turn#4629sylvesterkaczmarek wants to merge 3 commits into
Conversation
There was a problem hiding this comment.
Thanks for the focused contribution. The overall approach in #4629 is the implementation path we want to pursue: capture each turn's resolved capability set on TurnSpanData after dynamic enablement, filtering, and collision resolution.
I also verified the proposed payload with a live Traces ingest probe. The exact custom-span structure, including list[str] values for tools and handoffs, returned HTTP 204 and was retained and rendered correctly in the Traces UI. No tracing-schema redesign is needed.
Before merging, please make these focused identity corrections:
- Record
Handoff.tool_nameinTurnSpanDatarather thanHandoff.agent_name. With the supportedtool_name_override, the current value can differ from the identifier offered to the model, and distinct handoff tools targeting the same agent become indistinguishable. - Use the existing namespace-aware canonical trace identity for
FunctionTooldefinitions. The current generic helper can reduce namespaced tools to the unqualifiedtool.name. - Split the handoff projections in the helper: continue using
agent_namefor the releasedAgentSpanData.handoffsbehavior, while usingtool_namefor the new per-turn capability snapshot. - Add focused regression coverage for one handoff with
tool_name_overrideand one namespacedFunctionTool, asserting that the exported turn-span identifiers match the resolved capability identifiers.
The existing streamed/non-streamed dynamic-enablement test already covers the lifecycle placement and does not need broader permutation coverage.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4b516becb0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ead01e1b01
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
8abab26 to
57eb4fe
Compare
8624512 to
adab4fc
Compare
|
Addressed the requested identity corrections: turn spans now use Handoff.tool_name and namespace-aware FunctionTool identities, while legacy agent-span identities remain unchanged. Added regression coverage for tool_name_override and a namespaced FunctionTool. The review threads are resolved; the latest Actions run is waiting for maintainer approval to start. |
|
@seratch thanks again, addressed the review feedback! Turn spans now use the model-visible handoff tool name and namespace-aware FunctionTool identity, while agent spans preserve their existing identities. I also added focused coverage for a tool_name_override handoff and a namespaced FunctionTool, including assertions against the exported turn-span payload. Same-fork validation is green across the relevant lint, typecheck, platform, and Python 3.10-3.13 test jobs; upstream Actions are still awaiting external-contributor approval. |
|
Thanks for the thoughtful implementation and for working through the review feedback. After validating the behavior with both scripted and live-model traces, I do not think we should add this duplication to the tracing schema. For each model turn, the existing model-call span already records the exact functions sent to the model, including dynamically enabled tools and handoffs represented as function tools. Adding the same capability list to the turn span would create another source of truth and ongoing identity and maintenance coupling without a demonstrated supported scenario that the model-call trace cannot handle. I am going to close this PR and #4626. If a concrete tracing workflow shows that the model-call span is insufficient, we can revisit that narrower gap. Thanks again for the careful work here. |
Summary
This pull request adds per-turn tracing of the effective tools and handoffs visible to each model invocation.
Turn spans snapshot capability names after dynamic enablement, filtering, and tool-name collision resolution, while agent spans retain their existing latest-view and identity behaviour. The snapshots contain names only and work for both streaming and non-streaming runs. Runs with task/turn spans disabled retain the existing compact trace hierarchy.
The review follow-up also ensures that:
Handoff.tool_name, includingtool_name_overrideFunctionToolentries use their canonical qualified trace identityTest plan
action_requiredpending external-contributor approval.Issue number
Closes #4626
Checks
.agents/skills/code-change-verification/scripts/run.sh/reviewbefore submitting this PR