docs(cookbook): add the voice-agent instrument-and-verify recipe and its checker by abhijaisrivastava15 · Pull Request #844 · future-agi/docs

abhijaisrivastava15 · 2026-09-04T06:27:36Z

The problem

A voice call is not a trace with audio in it. Future AGI finds a call by six conditions at once, and the one people miss is that the call's span must have no parent. Every voice framework opens its own root span the moment a session starts, so an engineer who wraps their call in a span after that produces a trace that is complete, correctly shaped, fully populated, and absent from the Voice tab, from every voice filter and from every voice eval. Nothing errors. No dashboard reads differently. The only way to notice is to go looking for a call that was never there.

The other half is that nothing about a voice call is derived. Duration, turns, talk ratio and the transcript are all read from named attributes on that one span, and the instrumentor writes none of them. A name that is nearly right is a blank column rather than an error.

Why nobody catches it

The existing cookbook cannot cover this. Its checker opens on "at least one LLM span", which a pulled provider call never has, and none of its ten gates looks at whether a span has a parent, because on the text side that never mattered.

What

A voice-agent sibling to instrument-and-verify, same contract section for section, plus the checker it depends on.

src/pages/docs/cookbook/quickstart/instrument-and-verify-voice.mdx (653 lines): six steps, two tracks in Tabs, a real end-to-end run, an eval-binding table, a non-Python section and a troubleshooting table
public/fi_verify_voice.py (305 lines): twelve gates, a conversation-shaped preflight, and attach()
src/lib/navigation.ts: the sibling sidebar entry next to Instrument and Verify

Three things in it are not in any existing page:

The preflight span is conversation-shaped, so a 200 proves the voice path rather than the route, and one call is in the Voice tab before the reader writes a line of agent code
attach() captures at the exporter, after export, not with a span processor. traceai-livekit rewrites span._attributes inside export(), so a processor tee reports on attributes that were never sent: it shows a span kind of None on every LiveKit span
The transcript is written under three keys. fi.conversation.transcript is the only one the call drawer reads on a self-hosted agent, conversation.transcript is what the eval picker resolves, and the flattened conversation.transcript.N.message.* is what the error feed walks. None falls back to another

Two docs defects fixed in the same branch

docs/integrations/traceai/livekit.mdx taught fi_span_kind="agent" in both listings. Its own example therefore produced a call that appears in Traces and in no voice surface at all. Now "conversation", with the reason inline and a closing section
docs/observe/features/voice.mdx listed Vapi and Retell, and was unreachable from the sidebar. It now lists every provider a user can actually pick in Connect a provider, which is Vapi, Retell and Bland.ai. ElevenLabs is commented out of VOICE_CHAT_PROVIDERS on dev and Twilio never reaches the dropdown at all, so neither belongs on a page telling someone to connect one, even though both have server-side parsers

Screenshots

Three, all 3024x1890 off the same run as the gate output below, annotated to the same house spec as the text recipe's three.

The trace list. The row is the conversation span itself, which is the whole reason Step 3 opens it before session.start().

Trace list for voice-cookbook-page-run, seven voice.call traces, the conversation span annotated at the root

The call read as a call. Duration, Latency, User / AI and Silence are blank on purpose and the page says why: those four are computed from per-turn speech timings on the transcript entries, which a text-mode run does not have. Turns, Words and the transcript come from the attributes Step 5 writes.

Voice call detail: transcript with four turns and the Call Analytics strip, Latency annotated as blank without per-turn timing

The Attributes tab filtered to transcript, which is the only place the three transcript keys are visible side by side.

Attributes tab filtered to transcript, the three transcript keys annotated as none of them a fallback

Verification

The page was followed literally, twice, in a clean directory: every listing extracted from the .mdx by its own first-line marker, no retyping, the checker copied from public/ with a matching sha256, then the page's own three commands.

  PASS  V1 ... PASS  V12          12/12, exit 0
  Future AGI sees this as a call
  GREEN LIGHT achieved

Read back through the product's own endpoints, not just the database:

Endpoint	Before Step 5	After Step 5
`list_voice_calls`	call absent	call returned, `talk_ratio 0.727`
`voice_call_detail`	`transcript_available: None`, 0 turns	`transcript_available: True`, `message_count: 4`, all four turns

And the defect the page exists for, same agent, one line moved:

  span opened before session.start()   conversation  ROOT   returned by the Voice list
  span opened after  session.start()   conversation  child  returned by nothing

Eleven of twelve gates pass on the broken run. V3 is the only thing that tells them apart.

The checker's own gates are proved to bind: one real broken run plus 14 mutations of the green capture, one per gate, 14/14 red on the gate they target.

Run against production, by an agent with no context

The runs above were against a local stack. The page was then handed to a coding agent that had never seen it, given the rendered URL and a key file and nothing else, pointed at https://api.futureagi.com with a real account. No FI_BASE_URL override: the SDK and the checker both default there, and that default had never been exercised.

  PASS  V1   preflight ok, delivery ok
  PASS  V2   project_name='voice-cookbook-prod-run' project_type='observe'
  PASS  V3   1 conversation span(s)
  PASS  V4   11 spans, 1 trace(s), 1 root(s), 0 orphan(s)
  PASS  V5   session.id='call_c4bce86b02b5' user.id='acct_10427' on the conversation span
  PASS  V6   call.duration=1.235
  PASS  V7   call.total_turns=2 call.talk_ratio=0.786
  PASS  V8   2 turn(s) flattened; conversation.transcript present; fi.conversation.transcript present
  PASS  V9   provider='livekit'
  PASS  V10  no recording attribute, acknowledged: audio evals cannot bind to this call
  PASS  V11  2 LLM span(s), model on every one, prompt and completion on ['llm_node']
  PASS  V12  no credential in any span attribute

  Future AGI sees this as a call
  GREEN LIGHT achieved

GREEN on the first pass, no gate retried, about three and a half minutes from opening the page. The preflight's controls held against production too: the real keys got 200, a mutated key and a headerless send both got 401, and a trailing slash got 404.

The last commit on this branch is everything that run had to work out for itself: the two __init__.py files no command created, the missing virtualenv against a python every listing assumes, a Step 6 command that could not reproduce Step 6's own printed result, LLM_API_KEY named nowhere outside a listing, an undeclared Groq default behind a variable called OPENAI_BASE_URL, two wrong step cross-references, and a .fi_verify/ path that is relative to wherever you happen to stand.

One finding is not fixed and cannot be: curl https://docs.futureagi.com/fi_verify_voice.py is 404 until this merges and deploys. The sibling https://docs.futureagi.com/fi_verify.py is live and 200 today, so the shape is right and the asset simply is not published yet. The agent worked around it and confirmed the file it got matched the sha256 the page prints.

Docs build: 1175 pages, 0 errors. audit-links: 0 broken nav links, 0 broken content links, orphan pages 34 to 33.

Review round 1

Nine comments from @khushalsonawat, all answered on the branch.

The provider list was the real one. I had read the backend parsers, which do cover ElevenLabs and Twilio; the question a reader is actually asking is which providers they can pick, and that is VOICE_CHAT_PROVIDERS in the frontend. Right source, wrong field.

The Colab notebook is now future-agi/cookbooks#15. Its four listings are extracted from this page's own fenced blocks byte for byte, so the notebook and the page cannot drift.

The row was the text checker's probe, and the cause is in the checker

I first read this as the trace list mangling the span name, and that was wrong. Probe spans named alpha.beta.gamma, one.two.three.four, zz.voice.qq and a fresh futureagi.voice.preflight all render in full. There is no rendering bug.

Both checkers hardcode the same ids:

# public/fi_verify.py:53   and   public/fi_verify_voice.py:80
span = {"traceId": "4bf92f3577b34da6a3ce929d0e0e4736", "spanId": "00f067aa0ba902b7", ...}

default.spans sorts and replaces on (project_id, observation_type, service_name, toStartOfHour(start_time), trace_id, id), so two preflights in one project in one hour collapse to one row. The capture caught the text checker's probe winning that race. Fixed separately, with fresh ids per send; fi_verify.py still carries it.

Docs build after the round: 1175 pages, 0 errors. audit-links: 0 broken nav links, 0 broken content links.

Out of scope

traceai-livekit does not read LiveKit's current attribute names. LiveKit moved conversation content behind a pii segment; the mapper still reads the bare names, so LLM spans arrive with no prompt and no completion. Fixed separately in future-agi/traceAI; until that ships, Step 2 carries an 11-line shim and says when to delete it
The Cost column cannot be filled by a voice cost key. Pricing reads gen_ai.cost.total and llm.cost.total only. Documented, not changed
The TypeScript SDK has no CONVERSATION span kind. The page gives the attribute-level workaround. The SDK is not touched here

…its checker A voice call is selected by six conditions at once, and a span that misses any one of them is absent from the Voice tab while remaining a perfectly healthy trace. The text recipe cannot cover it: its gate set opens on "at least one LLM span", which a pulled provider call never has. New page, same contract as instrument-and-verify.mdx section for section, plus public/fi_verify_voice.py: twelve gates, a conversation-shaped preflight, and a capture that wraps the exporter rather than a span processor, because traceai-livekit rewrites attributes inside export().

Both listings taught fi_span_kind="agent". The Voice tab lists a conversation-typed span with no parent, so the page's own example produced a call that appears in Traces and in no voice surface at all.

…e up the sidebar The page listed Vapi and Retell. The product also pulls ElevenLabs, Bland.ai and Twilio, each with its own parser, and the page itself was unreachable from the sidebar. Adds both sidebar entries: this page under Observability > Guides, and the new voice recipe beside Instrument and Verify.

All three are 3024x1890 full-window dark, taken off the same run as the printed gate output (call ID 1965f5ba, session call_355937e9c818), annotated to the same house spec as the text recipe's three. The list shot carries the point the page is really about: the row is the conversation span itself, so opening it after session.start() costs you the row. The detail shot says out loud why Duration, Latency, User / AI and Silence are blank on a text-mode run. The attributes shot is the Attributes tab filtered to transcript, which is the only place the three transcript keys are visible side by side. The old call-detail.png came off an earlier run and showed a different call ID and a different answer than the transcript printed above it. Replaced. The last line of the printed read-back is now the assistant's answer verbatim, punctuation included, so the block and the screenshot say the same thing.

Run end to end against api.futureagi.com by an agent with no context but the page URL and a key file. It reached GREEN on the first pass, and everything below is something it had to work out for itself on the way. - The Install listing never created `observability/__init__.py` or `observability/futureagi/__init__.py`, which the tree diagram lists but no command produced, so both track listings fail to import until you guess it. - No virtualenv anywhere, while every command says `python`. System python on macOS is 3.9 and a bare `pip install` is refused outright, so the first listing on the page cannot run as written. One venv line fixes both. - Step 6 ran `agent.py` with one ask and the result block below it reports the two-ask default: 4 turns, 16 spans, unreachable from the page's own command. - `LLM_API_KEY` existed only inside `agent.py`. The export block that sets up the run never named it, and the prose only said "one model key". - The example calls Groq by default through a variable named `OPENAI_BASE_URL`, which the page never said out loud. - Two step cross-references were wrong: the conversation span is opened in Step 3, not Step 5, and the checker is downloaded in Install, not Step 1. - `.fi_verify/` is relative to the working directory, so preflight and check run from different places fail with a reason that names neither. Now a troubleshooting row. - Said which track the end-to-end walkthrough is, since it is SDK only.

khushalsonawat

Check comments

khushalsonawat · 2026-09-07T08:42:59Z

+<Prerequisites>
+- Future AGI account → [app.futureagi.com](https://app.futureagi.com)
+- API keys: `FI_API_KEY` and `FI_SECRET_KEY` (see [Get your API keys](/docs/admin-settings))
+- A voice agent you host yourself: LiveKit Agents, Pipecat, or your own STT plus LLM plus TTS loop. If your calls run on Vapi, Retell, ElevenLabs, Bland or Twilio, you write no code at all: see [If your calls come from a managed provider](#if-your-calls-come-from-a-managed-provider)


ElevenLabs is commented out of the Connect-a-provider dropdown (frontend constants.js) and Twilio isn't in it at all, both only have server-side parsers. Selectable today: Vapi, Retell, Bland.ai. Same list at L671

Done, and you are right on both. VOICE_CHAT_PROVIDERS on origin/dev has ElevenLabs commented out, and Twilio appears nowhere in frontend/src outside the generated API contracts. The page now says Vapi, Retell or Bland.ai in the Prerequisites and in the managed-provider section.

khushalsonawat · 2026-09-07T08:42:59Z

+A voice call is not a trace with audio in it. The Voice tab finds a call by six conditions at once, and a span that misses any one of them is invisible there no matter how healthy it looks in Traces. This guide instruments a self-hosted voice agent, then runs `fi_verify_voice.py`, which checks twelve gates against the spans your agent really sent and exits 0 or names the gate that failed. The worked example runs with one model key, no LiveKit account, no phone number and no microphone.
+</TLDR>
+
+| Time | Difficulty | Package |


Every runnable cookbook ships a Colab notebook (ruled 2026-08-18), the sibling has one and this example runs on a single model key. Add the notebook, the badge row and code-repo-url

Done. quickstart/instrument-and-verify-voice.ipynb is up as future-agi/cookbooks#15, and the page carries code-repo-url and the Colab + GitHub badge row in the same slot as the sibling.

The four listings the notebook writes out are extracted from this page's own fenced blocks byte for byte rather than retyped, so the two cannot drift.

khushalsonawat · 2026-09-07T08:42:59Z

+
+The call arrives with sixteen spans: the conversation span you wrote, and fifteen from LiveKit around it. `llm_node` and `llm_request` are the model call, `agent_turn` is one exchange, and the rest are session lifecycle.
+
+<img src="/images/docs/cookbook-instrument-and-verify-voice/call-list.png" alt="The Future AGI trace list for the voice-cookbook-page-run project, showing eight voice.call traces and the checker's futureagi.preflight probe, with input, output, timestamp, status and latency" style={{ borderRadius: '5px' }} />


That UNSET futureagi.preflight row is the text checker's probe, the page says the voice probe is futureagi.voice.preflight. Filter it out of the shot like last time

Re-shot without it, same project, same seven calls.

Worth flagging what I found chasing it, because the row is not what it looks like. That trace is a single span, and in ClickHouse its name is futureagi.voice.preflight:

trace_id: 4bf92f35-77b3-4da6-a3ce-929d0e0e4736 n: 1 names: ['futureagi.voice.preflight']

The only futureagi.preflight on the box that day is in a different project entirely. So it was the voice probe, exactly what Step 1 writes. I then loaded that list in a headless browser and read the rendered text back, and the DOM says futureagi.preflight.

The trace list is dropping the middle segment of the span name when it renders it. Nothing rewrites the name in fi-collector or in the tracer serializers that I could find, so I have the symptom but not the line. Any span named a.b.c is worth a second look.

The row is out of the shot either way: it is the checker's probe, not a call, and the caption says one row per call.

Correction to my reply above, and you were right.

I claimed the trace list drops the middle segment of a span name. It does not. I inserted probe spans named alpha.beta.gamma, one.two.three.four, zz.voice.qq, futureagi.a.b and a fresh futureagi.voice.preflight, and every one of them renders in full. No rendering bug.

The real cause is in the checkers. Both hardcode the same ids:

# public/fi_verify.py:53 and public/fi_verify_voice.py:80 span = {"traceId": "4bf92f3577b34da6a3ce929d0e0e4736", "spanId": "00f067aa0ba902b7", ...}

default.spans sorts on (project_id, observation_type, service_name, toStartOfHour(start_time), trace_id, id) and replaces on it. So two preflights in one project in one hour collapse to a single row, and whichever landed last wins. The row in that capture really was the text checker's probe, exactly as you called it. By the time I queried ClickHouse the voice probe had replaced it, which is what sent me down the wrong path.

Fixed on cookbook/voice-preflight-unique-ids: fresh os.urandom ids per send, verified three sends give three distinct pairs. fi_verify.py still has it and I have not touched that one here.

Sorry for the noise on a thread you had already got right.

khushalsonawat · 2026-09-07T08:42:59Z

+    user       How much baggage can I bring?
+    assistant  You may bring one checked bag up to 23 kg and one carry-on bag up ...
+    user       And is a stroller counted separately?
+    assistant  Yes—strollers are considered a free‑of‑charge personal item and do ...


Em-dash, even if the model wrote it. Trim the printed line before the dash

Done, the line is trimmed before the dash.

It is still in the trace-list screenshot, in the Output column of the top row, because that is real model output in a real capture and I did not want to doctor the image. Say the word and I will re-run the example for an answer without one.

khushalsonawat · 2026-09-07T08:42:59Z

+| `check` says `no preflight receipt` right after `preflight` said it passed | `.fi_verify/` is relative to the working directory, so the two commands ran from different places | Run `preflight`, the agent and `check` from one directory, or set `FI_VERIFY_FILE` to an absolute path for all three |
+| Spans stop arriving as soon as the mapper is enabled | Something wrapped the exporter before `enable_http_attribute_mapping()` replaced it | Call the mapper first, then anything that wraps an exporter, in the Step 2 order |
+
+Instrument the model calls inside the call with [Instrument and Verify](/docs/cookbook/quickstart/instrument-and-verify), and keep one call across a service boundary with [Distributed Tracing](/docs/cookbook/quickstart/distributed-tracing).


One forward link, not two

Done, dropped the Distributed Tracing link and kept Instrument and Verify.

khushalsonawat · 2026-09-07T08:42:59Z

              },
              { title: 'Setup alerts', href: '/docs/observe/guides/setup-alerts' },
              { title: 'Setup evals', href: '/docs/observe/guides/setup-evals' },
+              { title: 'Connect a voice provider', href: '/docs/observe/features/voice' },


Sidebar item and page title should match, the page is titled Voice Observability

Done, the sidebar item is now Voice Observability.

khushalsonawat · 2026-09-07T08:42:59Z

            collapsible: true,
            items: [
              { title: 'Instrument and Verify', href: '/docs/cookbook/quickstart/instrument-and-verify' },
+              { title: 'Instrument and Verify a Voice Agent', href: '/docs/cookbook/quickstart/instrument-and-verify-voice' },


Add the card to platform/index.mdx and bump the hub's Tracing & Debugging count to 9

Done. The card is in cookbook/platform/index.mdx under Tracing & Debugging, and the count on cookbook/index.mdx is 9.

khushalsonawat · 2026-09-07T08:42:59Z

 ---


 ## About


About is banned as a heading, rename it while you're in here

Done, it is now "What voice observability does".

Worth knowing separately: ## About is still the opening heading on alerts, dashboard, evals, quickstart and users in the same folder. Happy to take those in one pass if you want them gone.

khushalsonawat · 2026-09-07T08:42:59Z

+|---|---|
+| Vapi | [dashboard.vapi.ai](https://dashboard.vapi.ai) |
+| Retell | [retellai.com](https://www.retellai.com/) |
+| ElevenLabs | [elevenlabs.io](https://elevenlabs.io/) |


Drop the ElevenLabs and Twilio rows, neither is connectable in the UI. Same in the description and the feature bullet at L27

Done. Both rows are out of the table, and out of the description and the L27 bullet with them.

Providers: only Vapi, Retell and Bland.ai are selectable in Connect a provider. ElevenLabs is commented out of VOICE_CHAT_PROVIDERS and Twilio never reaches the dropdown, so both come out of the page, the feature page and its description. Also: the Colab and GitHub badge row with code-repo-url, the sidebar item matched to the page title, the card added to the platform hub with the count bumped to 9, one forward link instead of two, a heading rename, and the printed transcript trimmed before the dash.

The row read futureagi.preflight, which is not a span name that exists: the span is futureagi.voice.preflight, and the trace list drops the middle segment when it renders the name. Same project, same seven calls, probe trace excluded, so the shot now shows one row per call as its caption says.

khushalsonawat

LGTM

abhijaisrivastava15 added 5 commits September 3, 2026 18:21

docs(integrations): type the LiveKit session span as a conversation

e5d4a58

Both listings taught fi_span_kind="agent". The Voice tab lists a conversation-typed span with no parent, so the page's own example produced a call that appears in Traces and in no voice surface at all.

abhijaisrivastava15 requested a review from khushalsonawat September 4, 2026 11:13

khushalsonawat requested changes Sep 7, 2026

View reviewed changes

abhijaisrivastava15 added 2 commits September 7, 2026 14:59

abhijaisrivastava15 requested a review from khushalsonawat September 7, 2026 09:48

khushalsonawat approved these changes Sep 7, 2026

View reviewed changes

khushalsonawat merged commit 74881c6 into dev Sep 7, 2026
1 check passed

khushalsonawat deleted the cookbook/instrument-and-verify-voice branch September 7, 2026 10:10

This was referenced Sep 7, 2026

fix(checker): give each voice preflight its own trace and span id #847

Merged

Dev to main, 7th Sep, 2026 #848

Merged

docs(livekit): set the session root span kind to conversation so calls reach the voice UI #804

Closed


		The call arrives with sixteen spans: the conversation span you wrote, and fifteen from LiveKit around it. `llm_node` and `llm_request` are the model call, `agent_turn` is one exchange, and the rest are session lifecycle.

		<img src="/images/docs/cookbook-instrument-and-verify-voice/call-list.png" alt="The Future AGI trace list for the voice-cookbook-page-run project, showing eight voice.call traces and the checker's futureagi.preflight probe, with input, output, timestamp, status and latency" style={{ borderRadius: '5px' }} />

Uh oh!

Conversation

abhijaisrivastava15 commented Sep 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

The problem

Why nobody catches it

What

Two docs defects fixed in the same branch

Screenshots

Verification

Run against production, by an agent with no context

Review round 1

The row was the text checker's probe, and the cause is in the checker

Out of scope

Uh oh!

khushalsonawat left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

khushalsonawat left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

abhijaisrivastava15 commented Sep 4, 2026 •

edited

Loading