Skip to content

Cancelled function tools vanish without a trace: no FunctionCallOutput, no span error/output attributes, no log #2374

Description

@dtran26

Environment

  • @livekit/agents 1.7.1 (behavior also present on 1.6.x)
  • Telephony pipeline, LiveKit Cloud-hosted agents

Summary

When a running function tool's task is aborted before it resolves — e.g. the activity/session is torn down while the tool executes — the tool disappears without any observable record:

  • no FunctionCallOutput is emitted or committed to the chat context;
  • the tool's telemetry span ends with no lk.function_tool.output and no lk.function_tool.is_error attribute (a successful tool gets both, a throwing tool gets is_error=true plus an exception occurred while executing tool error log);
  • nothing is logged at any level.

So "tool was cancelled" is indistinguishable from "span export glitch" in telemetry, and applications listening for tool completion (e.g. FunctionToolsExecuted-style rewrites, or logic awaiting the tool's chat-context output) silently never fire.

Why it matters — a concrete production case

During a WarmTransferTask, the human agent hung up (CLIENT_INITIATED) while connect_to_caller was executing its merge. The consult session's closeOnDisconnect teardown force-interrupted the in-flight speech and aborted the tool task. Result: the tool span for connect_to_caller (aliased) shows a 14–21s duration and then simply ends with no output/error attributes; worker logs show nothing at all. Combined with #2358 (the room-close listener being detached during the merge), the transfer task never settled and the caller sat on dead hold until an application-level timeout — and the only way we could diagnose it was pulling server-side room events, because the worker-side telemetry contained no evidence that the merge had been attempted and cancelled.

Ask

On tool-task abort, emit an explicit record:

  1. mark the tool span with a status/attribute (e.g. lk.function_tool.cancelled = true), and
  2. log at debug/info (function tool cancelled before resolution, with tool name + call id), and
  3. ideally, emit a synthetic FunctionCallOutput (or a dedicated cancellation event) so application listeners and the chat context aren't left with a dangling FunctionCall that has no output — related to the cooperative-cancellation direction in WarmTransferTask: add cooperative cancellation and a teardown-complete lifecycle #2288 / feat(core): WarmTransferTask cooperative cancellation via abortSignal, bounded cancellation teardown #2292, and to llm_node response telemetry omits function calls and partial output on cancellation #2043 which covered the LLM-side telemetry of cancellation.

Any of these individually would have cut our diagnosis time from days to minutes.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions