Conversation
|
0a979d8 to
473c53d
Compare
473c53d to
70effd6
Compare
70effd6 to
1bb26cb
Compare
1bb26cb to
378dc23
Compare
378dc23 to
434f47e
Compare
434f47e to
0d1517e
Compare
0d1517e to
b2c5e8d
Compare
2268d2d to
f064b33
Compare
0daa94a to
92a97ea
Compare
b6eebad to
289b916
Compare
289b916 to
81f697d
Compare
Port of livekit/agents#7148. The nesting of spans is an emergent property of many call sites; a refactor can move a span under the wrong parent while every existing test passes, because each asserts one edge. agents/src/telemetry/testing/trace_schema.ts writes the rules down once: - SPAN_PARENTS: for every span the JS framework emits, the parents it may have (ROOT for none, ANY for spans that follow their caller). Unknown names are violations, so a new span must be registered. - MAY_OUTLIVE_PARENT: the child/parent edges where the child may end after its parent, each with its reason. Everything else must sit inside its parent, with 2 ms of slack. - checkTrace(): one trace id, every parent present and allowed, bounds, one agent_turn per lk.speech_id, lk.generation_count equal to the speech's own generation events, every eou_wait with an outcome. - fromReadableSpans / fromOtlpJson: an in-memory exporter or an export downloaded from LiveKit Cloud, same rules. As a CLI (`pnpm exec tsx agents/src/telemetry/testing/trace_schema.ts x.json`) it prints the span summary and the violations, tolerating orphans in a partial export. assertTraceWellFormed() now ends the full-session tests: the tool call and plain reply in agent_turn_span, the barge-in and handoff in coverage_spans, the hook and redaction sessions in eou_wait_span, the lifecycle and SIP sessions in session_lifecycle_span. The module is test support: imported by tests only, not exported from the package, and free of framework imports so the CLI runs on the source file. Schema differences from Python, all from what the JS code emits: answering_machine_detection is JS's name for `amd`; JS has no llm_fallback_adapter / tts_fallback_adapter / tts_stream_adapter spans (its adapters emit the plain request spans, which nest under the attempt's *_request_run), no wait_for_video_track, no judge_evaluation; rpc_handler may be a root, since without a session the SDK dispatches on a context carrying no span. Test hardening: the eou_wait full-session helper decided the turn 20 ms after the fake STT final was due, so a loaded host could open a second user turn (the intermittent failure seen in full-suite runs); the endpointing delay now leaves a 170 ms margin. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
81f697d to
f0c186a
Compare
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| const ownGenerations = span.events.filter( | ||
| (event) => | ||
| event.name === 'generation' && | ||
| String(event.attributes['lk.generation_id'] ?? `${speechId}_`).startsWith(`${speechId}_`), |
There was a problem hiding this comment.
🟡 Valid preemptive turns fail checks
After a preemptive handoff, checkTrace excludes the discarded generation. lk.generation_count includes every generation, so valid turns receive a count violation.
Learn more
A preemptive handoff preserves the discarded attempt's generation event on the adopted span. The successor changes lk.speech_id, but withAgentTurn keeps lk.generation_count cumulative across every generation event. Filtering events by the successor's speech ID therefore compares different quantities.
Example: An attempt emits attempt_1, then its successor emits reply_1. The span has lk.speech_id=reply and lk.generation_count=2, but the checker counts only reply_1 and reports 2 but 1.
Recommended fix: Compare lk.generation_count with all generation events on the span. Keep the discarded-event attribution check separate if the schema also needs to validate handoffs.
Was this helpful? React with 👍 or 👎 to provide feedback.
| const traceIds = new Set(spans.map((span) => span.traceId)); | ||
| if (traceIds.size > 1) { | ||
| violations.push(`spans belong to ${traceIds.size} traces, expected one`); | ||
| } |
There was a problem hiding this comment.
🟡 Empty exports pass validation
With no spans, checkTrace accepts zero trace IDs because it rejects only multiple IDs. Empty or unrecognized exports report trace shape OK.
| const traceIds = new Set(spans.map((span) => span.traceId)); | |
| if (traceIds.size > 1) { | |
| violations.push(`spans belong to ${traceIds.size} traces, expected one`); | |
| } | |
| const traceIds = new Set(spans.map((span) => span.traceId)); | |
| if (traceIds.size !== 1) { | |
| violations.push(`spans belong to ${traceIds.size} traces, expected one`); | |
| } |
Was this helpful? React with 👍 or 👎 to provide feedback.
1 participant
Port of livekit/agents#7148. Stacked on #2500.
Description
The nesting of spans is an emergent property of many call sites. A refactor can move a span under the wrong parent while every existing test passes, because each test asserts the one edge it was written for. This layer writes the rules down once and checks every span in every full-session test against them, and applies the same rules to a real run's export.
agents/src/telemetry/testing/trace_schema.ts(test support, not exported from the package):SPAN_PARENTS: for every span name the JS framework emits, the parents it may have (ROOTfor none,ANYfor spans that follow their caller:rpc_call,event_loop_blocked). An unknown name is a violation, so a new span must be registered.MAY_OUTLIVE_PARENT: the child/parent edges where the child may end after its parent, each with its reason (the startup spanssession.start()does not wait for,on_enter, and stalls whose end is one heartbeat late). Everything else must sit inside its parent, with 2 ms of slack.checkTrace(): one trace id, every parent present, parent allowed by the schema, bounds, and the per-turn invariants: oneagent_turnperlk.speech_id,lk.generation_countequal to the number ofgenerationevents, everyeou_waitwith an outcome.fromReadableSpansfor an in-memory exporter,fromOtlpJsonfor an export downloaded from LiveKit Cloud. As a CLI,pnpm exec tsx agents/src/telemetry/testing/trace_schema.ts traces.jsonprints the span summary and the violations, tolerating orphans in a partial export.assertTraceWellFormed(exporter.getFinishedSpans())now ends the full-session tests of the earlier layers:agent_turn_span,coverage_spans,eou_wait_span,session_lifecycle_span.Schema differences from Python
All from what the JS code emits:
answering_machine_detectionis JS's name foramd; JS has nollm_fallback_adapter/tts_fallback_adapter/tts_stream_adapterspans (its adapters emit the plain request spans, which nest under the attempt's*_request_run), nowait_for_video_track, nojudge_evaluation;rpc_handlermay be a root, since without a session the SDK dispatches on a context carrying no span.MAY_OUTLIVE_PARENTis identical to Python's.Testing
telemetry/testing/trace_schema.test.ts(9 tests): schema self-consistency; a sound trace passes; a misparentedllm_request, aneou_detectionoutside its wait, an unknown span, a missing parent, a child outside its parent, a duplicated speech turn, a generation-count mismatch and a second trace id are each reported; the deliberate overruns pass; OTLP and in-memory sources agree; a full fake session is well-formed.eou_waitfull-session helper decided the turn 20 ms after the fake STT final was due, which under load opened a second user turn; the endpointing delay now leaves a 170 ms margin.agentssuite green.🤖 Generated with Claude Code