+## Conversations and Users
+
+Beyond per-call traces, Traceway rolls calls up into conversations and per-user analytics. The AI Traces page has three tabs: **Traces**, **Conversations**, and **Users**.
+
+### How conversations are identified
+
+Each AI call gets a conversation id at ingest, resolved in this order:
+
+1. `gen_ai.conversation.id` span attribute. Set this in your app for reliable multi-turn grouping.
+2. `session.id` span or resource attribute.
+3. The distributed trace id. A single agent run inside one request still groups its calls, even without an explicit id.
+
+Calls ingested before this feature (or with none of the above) have no conversation id and are excluded from conversation analytics.
+
+### Conversations view
+
+Each row is one conversation: turns (LLM call count), user, tool calls, models used, tokens, cost, and first/last seen. Conversations above the 95th percentile in cost or turns are highlighted so outliers stand out.
+
+
+
+Filter by user, model, specific tool, or flagged content. Active filters show as removable pills, and free-text search matches conversation ids, users, models, tool names, and flagged terms, so searching a curse word finds the flagged conversations directly.
+
+
+
+Drill into a conversation to read the full chat timeline with tool calls rendered inline (function name, arguments, and paired results), per-turn cost and tokens, and links to each underlying call.
+
+
+
+### Users view
+
+Per-customer analytics keyed on the `user.id` attribute: conversation count, total calls, median/average/min conversation length, average cost per conversation, total cost, and flagged conversation count. Clicking a user filters the Conversations view to them.
+
+#### What to put in `user.id`
+
+Set `user.id` on every model-call span to a **stable identifier for the end user of your product**, and use the same value across all of that user's sessions and conversations. Good values: your internal account/user id (best), a tenant or organization id for B2B products, or an email if emails are acceptable in your telemetry. Do not use session ids, request ids, or anything random per conversation; that is what `gen_ai.conversation.id` is for, and putting it in `user.id` makes every user look new, which destroys the per-user medians. The value is stored on every AI trace row and is searchable by anyone with project access, so if PII must stay out of telemetry, use an internal id or a hash: the analytics only need stability, not readability.
+
+With OpenRouter's zero-code Broadcast integration, pass the `user` field in your chat completion requests (same guidance for the value) and it arrives as `user.id`. Without it, calls attribute to your OpenRouter account itself and every customer looks like one user.
+
+
+
+### Content flagging
+
+At ingest, the prompt and completion text are scanned against built-in profanity language packs plus custom terms. Packs ship for English, German, Spanish, French, Italian, Portuguese, and Serbian; enable any combination, or none to rely on custom terms only. Custom terms cover anything beyond profanity: competitor names, refund phrases, compliance keywords.
+
+Configure both from the **Flagged terms** button on the Conversations page, or from the project settings sheet (Edit Project, AI tab). Changes take effect for newly ingested calls.
+
+
+
+Matching is whole-word and case-insensitive, so terms never match inside larger words. Matched terms are indexed on the call at ingestion time only; there is no query-time scanning. They show as a Flagged badge, power the flagged-only filter, and are covered by free-text search, so searching a flagged word finds the conversations containing it. Flagging only applies to calls ingested after the configuration changes; historical calls are not re-scanned. Because matching is word-boundary based, packs are only feasible for languages with space-separated words (no CJK).
+
+### Alerting on outliers
+
+Three notification rule types cover AI spend and content:
+
+| Rule type | Fires when |
+|-----------|-----------|
+| AI Trace Cost | A single call exceeds a cost threshold |
+| AI Conversation Cost | A conversation's cumulative cost over the last 24 hours exceeds a threshold |
+| AI Flagged Content | A conversation matches flagged terms (optionally narrowed to specific terms) |
+
## Conversation Storage
Conversation content (prompts and completions) can be large — a single prompt with RAG context might be 50KB+. To keep the database fast for aggregation queries, Traceway stores conversation content in object storage (S3 or local filesystem) and reads it on demand when you view a specific trace.
diff --git a/docs/public/ai-conversation-detail.png b/docs/public/ai-conversation-detail.png
new file mode 100644
index 00000000..1c50f2d1
Binary files /dev/null and b/docs/public/ai-conversation-detail.png differ
diff --git a/docs/public/ai-conversations-filtered.png b/docs/public/ai-conversations-filtered.png
new file mode 100644
index 00000000..b2d0ad92
Binary files /dev/null and b/docs/public/ai-conversations-filtered.png differ
diff --git a/docs/public/ai-conversations.png b/docs/public/ai-conversations.png
new file mode 100644
index 00000000..118fbb19
Binary files /dev/null and b/docs/public/ai-conversations.png differ
diff --git a/docs/public/ai-flagged-terms-settings.png b/docs/public/ai-flagged-terms-settings.png
new file mode 100644
index 00000000..b9025855
Binary files /dev/null and b/docs/public/ai-flagged-terms-settings.png differ
diff --git a/docs/public/ai-traces-detail.png b/docs/public/ai-traces-detail.png
index c5cf6a21..104ce3f0 100644
Binary files a/docs/public/ai-traces-detail.png and b/docs/public/ai-traces-detail.png differ
diff --git a/docs/public/ai-traces.png b/docs/public/ai-traces.png
index f7f0b6da..e5e72f29 100644
Binary files a/docs/public/ai-traces.png and b/docs/public/ai-traces.png differ
diff --git a/docs/public/ai-users.png b/docs/public/ai-users.png
new file mode 100644
index 00000000..b963b938
Binary files /dev/null and b/docs/public/ai-users.png differ
diff --git a/examples/devtesting-embedded/README.md b/examples/devtesting-embedded/README.md
index 7e7c4d63..3d1372c0 100644
--- a/examples/devtesting-embedded/README.md
+++ b/examples/devtesting-embedded/README.md
@@ -10,8 +10,34 @@ go run .
Then:
- App: http://localhost:8080/cdn (no build step) or http://localhost:8080 (requires `cd frontend && npm install && npm run build` first)
+- AI chat: http://localhost:8080/chat (needs `OPENROUTER_API_KEY` in `.env`)
- Dashboard: http://localhost:8082 — login `admin@localhost.com` / `admin`
+## AI chat (conversations, tool calls, sub-agents)
+
+`/chat` is a no-build chat UI backed by OpenRouter, built to exercise the AI conversation analytics end to end. Configuration lives in `.env` (gitignored):
+
+```
+OPENROUTER_API_KEY=sk-or-...
+OPENROUTER_MODEL=anthropic/claude-sonnet-5
+```
+
+How it maps to Traceway:
+
+- Every LLM call emits a `gen_ai.*` span into the `Backend API` project, with real token counts and cost from OpenRouter usage accounting.
+- The chat session id becomes `gen_ai.conversation.id`, so each browser conversation shows up on the Conversations tab. The persona picker sets `user.id` for the Users tab.
+- The main **Support Chat Agent** has fake tools (`get_weather`, `lookup_order`, `get_server_time`) plus two delegation tools that run sub-agents in the same conversation: **Research Sub-Agent** (with `search_knowledge_base`) and **Math Sub-Agent** (with a real `calculate` evaluator). Each agent is a separate trace name on the AI Traces tab.
+- Tool executions run in plain child spans (no `gen_ai.*` attributes) so they appear in the trace waterfall without inflating conversation turn counts; the tool calls themselves are parsed from the completion payloads.
+
+Suggestion buttons in the UI trigger each tool and sub-agent. To test content flagging, swear at the bot or add custom terms in the project settings AI tab. The API is also curl-able:
+
+```bash
+curl -s http://localhost:8080/api/ai/chat -H 'Content-Type: application/json' -d '{
+ "sessionId": "conv-1", "userId": "alice@example.com",
+ "messages": [{"role": "user", "content": "Weather in Paris, and ask the math agent for 12*(3+4)"}]
+}'
+```
+
## What runs
- **Traceway backend**, embedded on port 8082, SQLite storage in `./storage/`.
diff --git a/examples/devtesting-embedded/ai.go b/examples/devtesting-embedded/ai.go
new file mode 100644
index 00000000..dd321c1c
--- /dev/null
+++ b/examples/devtesting-embedded/ai.go
@@ -0,0 +1,590 @@
+package main
+
+import (
+ "bytes"
+ "context"
+ "encoding/json"
+ "fmt"
+ "hash/fnv"
+ "io"
+ "net/http"
+ "os"
+ "strconv"
+ "strings"
+ "time"
+
+ "github.com/gin-gonic/gin"
+ "go.opentelemetry.io/otel/attribute"
+ "go.opentelemetry.io/otel/codes"
+ "go.opentelemetry.io/otel/trace"
+)
+
+// An AI chat agent backed by OpenRouter, instrumented with gen_ai.* span
+// attributes so every LLM call lands in Traceway's AI Traces / Conversations
+// analytics: conversation grouping (gen_ai.conversation.id), per-user stats
+// (user.id), tool calls parsed from the completion payload, and sub-agents as
+// separate trace names inside the same conversation.
+
+const openRouterURL = "https://openrouter.ai/api/v1/chat/completions"
+
+var aiHTTPClient = &http.Client{Timeout: 120 * time.Second}
+
+// --- OpenRouter wire types (OpenAI chat-completions compatible) ---
+
+type orToolCall struct {
+ Id string `json:"id"`
+ Type string `json:"type"`
+ Function struct {
+ Name string `json:"name"`
+ Arguments string `json:"arguments"`
+ } `json:"function"`
+}
+
+type orMessage struct {
+ Role string `json:"role"`
+ Content *string `json:"content,omitempty"`
+ ToolCalls []orToolCall `json:"tool_calls,omitempty"`
+ ToolCallId string `json:"tool_call_id,omitempty"`
+}
+
+func textMsg(role, content string) orMessage {
+ return orMessage{Role: role, Content: &content}
+}
+
+type orTool struct {
+ Type string `json:"type"`
+ Function orToolDef `json:"function"`
+}
+
+type orToolDef struct {
+ Name string `json:"name"`
+ Description string `json:"description"`
+ Parameters json.RawMessage `json:"parameters"`
+}
+
+type orRequest struct {
+ Model string `json:"model"`
+ Messages []orMessage `json:"messages"`
+ Tools []orTool `json:"tools,omitempty"`
+ Usage struct {
+ Include bool `json:"include"`
+ } `json:"usage"`
+}
+
+type orChoice struct {
+ FinishReason string `json:"finish_reason"`
+ Message orMessage `json:"message"`
+}
+
+type orResponse struct {
+ Id string `json:"id"`
+ Model string `json:"model"`
+ Choices []orChoice `json:"choices"`
+ Usage struct {
+ PromptTokens int64 `json:"prompt_tokens"`
+ CompletionTokens int64 `json:"completion_tokens"`
+ TotalTokens int64 `json:"total_tokens"`
+ Cost float64 `json:"cost"`
+ } `json:"usage"`
+ Error *struct {
+ Message string `json:"message"`
+ } `json:"error"`
+}
+
+func mustSchema(v string) json.RawMessage { return json.RawMessage(v) }
+
+// --- Agents ---
+
+type agentDef struct {
+ // TraceName becomes trace.name on the gen_ai spans, so each agent shows up
+ // as its own entry on the AI Traces page while sharing the conversation.
+ TraceName string
+ SystemPrompt string
+ Tools []orTool
+ // MaxTurns bounds the tool-call loop per user message.
+ MaxTurns int
+}
+
+var mainAgent = agentDef{
+ TraceName: "Support Chat Agent",
+ SystemPrompt: "You are a helpful support assistant for the Acme web shop. " +
+ "Use the available tools whenever they can answer the question: weather, order lookups, server time. " +
+ "Delegate research questions (documentation, product knowledge) to the research agent and " +
+ "math/calculation questions to the math agent instead of answering from memory. " +
+ "Keep answers short and conversational.",
+ Tools: []orTool{
+ {Type: "function", Function: orToolDef{
+ Name: "get_weather",
+ Description: "Get the current weather for a city.",
+ Parameters: mustSchema(`{"type":"object","properties":{"city":{"type":"string","description":"City name"}},"required":["city"]}`),
+ }},
+ {Type: "function", Function: orToolDef{
+ Name: "lookup_order",
+ Description: "Look up the status of a customer order by its order id.",
+ Parameters: mustSchema(`{"type":"object","properties":{"order_id":{"type":"string","description":"The order id, e.g. A-1042"}},"required":["order_id"]}`),
+ }},
+ {Type: "function", Function: orToolDef{
+ Name: "get_server_time",
+ Description: "Get the current server date and time.",
+ Parameters: mustSchema(`{"type":"object","properties":{}}`),
+ }},
+ {Type: "function", Function: orToolDef{
+ Name: "ask_research_agent",
+ Description: "Delegate a research or documentation question to the research sub-agent. It has access to the knowledge base.",
+ Parameters: mustSchema(`{"type":"object","properties":{"question":{"type":"string"}},"required":["question"]}`),
+ }},
+ {Type: "function", Function: orToolDef{
+ Name: "ask_math_agent",
+ Description: "Delegate a math or calculation problem to the math sub-agent. It has a calculator.",
+ Parameters: mustSchema(`{"type":"object","properties":{"problem":{"type":"string"}},"required":["problem"]}`),
+ }},
+ },
+ MaxTurns: 6,
+}
+
+var researchAgent = agentDef{
+ TraceName: "Research Sub-Agent",
+ SystemPrompt: "You are a research sub-agent. Answer the question using the search_knowledge_base tool, " +
+ "then reply with a concise summary of what you found. Always search before answering.",
+ Tools: []orTool{
+ {Type: "function", Function: orToolDef{
+ Name: "search_knowledge_base",
+ Description: "Search the internal knowledge base for documentation snippets.",
+ Parameters: mustSchema(`{"type":"object","properties":{"query":{"type":"string"}},"required":["query"]}`),
+ }},
+ },
+ MaxTurns: 4,
+}
+
+var mathAgent = agentDef{
+ TraceName: "Math Sub-Agent",
+ SystemPrompt: "You are a math sub-agent. Use the calculate tool for any arithmetic instead of computing yourself, " +
+ "then reply with just the result and one short sentence.",
+ Tools: []orTool{
+ {Type: "function", Function: orToolDef{
+ Name: "calculate",
+ Description: "Evaluate an arithmetic expression with + - * / and parentheses, e.g. \"12*(3+4)\".",
+ Parameters: mustSchema(`{"type":"object","properties":{"expression":{"type":"string"}},"required":["expression"]}`),
+ }},
+ },
+ MaxTurns: 4,
+}
+
+// --- Chat session context threaded through every LLM/tool call ---
+
+type chatContext struct {
+ svc *otelService
+ model string
+ conversationId string
+ userId string
+ events []toolEvent
+}
+
+type toolEvent struct {
+ Agent string `json:"agent"`
+ Tool string `json:"tool"`
+ Args string `json:"args"`
+ Result string `json:"result"`
+}
+
+// callOpenRouter makes one chat-completions call and emits the gen_ai.* span
+// Traceway promotes to an ai_traces row.
+func (cc *chatContext) callOpenRouter(ctx context.Context, agent agentDef, messages []orMessage) (*orResponse, error) {
+ reqBody := orRequest{Model: cc.model, Messages: messages, Tools: agent.Tools}
+ reqBody.Usage.Include = true
+
+ payload, err := json.Marshal(reqBody)
+ if err != nil {
+ return nil, err
+ }
+
+ spanCtx, span := cc.svc.tr.Start(ctx, "chat "+cc.model, trace.WithSpanKind(trace.SpanKindClient))
+ defer span.End()
+
+ promptJSON, _ := json.Marshal(map[string]any{"messages": messages})
+ span.SetAttributes(
+ attribute.String("trace.name", agent.TraceName),
+ attribute.String("gen_ai.conversation.id", cc.conversationId),
+ attribute.String("user.id", cc.userId),
+ attribute.String("gen_ai.operation.name", "chat"),
+ attribute.String("gen_ai.system", "openrouter"),
+ attribute.String("gen_ai.request.model", cc.model),
+ attribute.String("gen_ai.prompt", string(promptJSON)),
+ )
+
+ httpReq, err := http.NewRequestWithContext(spanCtx, http.MethodPost, openRouterURL, bytes.NewReader(payload))
+ if err != nil {
+ span.SetStatus(codes.Error, err.Error())
+ return nil, err
+ }
+ httpReq.Header.Set("Content-Type", "application/json")
+ httpReq.Header.Set("Authorization", "Bearer "+os.Getenv("OPENROUTER_API_KEY"))
+ httpReq.Header.Set("X-Title", "traceway devtesting-embedded")
+
+ httpResp, err := aiHTTPClient.Do(httpReq)
+ if err != nil {
+ span.SetStatus(codes.Error, err.Error())
+ return nil, err
+ }
+ defer httpResp.Body.Close()
+
+ body, err := io.ReadAll(httpResp.Body)
+ if err != nil {
+ span.SetStatus(codes.Error, err.Error())
+ return nil, err
+ }
+
+ var resp orResponse
+ if err := json.Unmarshal(body, &resp); err != nil {
+ span.SetStatus(codes.Error, err.Error())
+ return nil, fmt.Errorf("openrouter returned non-JSON (%d): %.300s", httpResp.StatusCode, string(body))
+ }
+ if resp.Error != nil {
+ span.SetStatus(codes.Error, resp.Error.Message)
+ return nil, fmt.Errorf("openrouter error: %s", resp.Error.Message)
+ }
+ if len(resp.Choices) == 0 {
+ span.SetStatus(codes.Error, "no choices in response")
+ return nil, fmt.Errorf("openrouter returned no choices (%d): %.300s", httpResp.StatusCode, string(body))
+ }
+
+ completionJSON, _ := json.Marshal(map[string]any{"choices": resp.Choices})
+ span.SetAttributes(
+ attribute.String("gen_ai.response.model", resp.Model),
+ attribute.Int64("gen_ai.usage.input_tokens", resp.Usage.PromptTokens),
+ attribute.Int64("gen_ai.usage.output_tokens", resp.Usage.CompletionTokens),
+ attribute.Int64("gen_ai.usage.total_tokens", resp.Usage.TotalTokens),
+ attribute.String("gen_ai.response.finish_reason", resp.Choices[0].FinishReason),
+ attribute.String("gen_ai.completion", string(completionJSON)),
+ )
+ if resp.Usage.Cost > 0 {
+ span.SetAttributes(attribute.Float64("gen_ai.usage.total_cost", resp.Usage.Cost))
+ }
+
+ return &resp, nil
+}
+
+// runAgent drives one agent's tool-call loop and returns its final text reply.
+func (cc *chatContext) runAgent(ctx context.Context, agent agentDef, userContent string, history []orMessage) (string, error) {
+ messages := []orMessage{textMsg("system", agent.SystemPrompt)}
+ messages = append(messages, history...)
+ messages = append(messages, textMsg("user", userContent))
+
+ for turn := 0; turn < agent.MaxTurns; turn++ {
+ resp, err := cc.callOpenRouter(ctx, agent, messages)
+ if err != nil {
+ return "", err
+ }
+
+ msg := resp.Choices[0].Message
+ if len(msg.ToolCalls) == 0 {
+ if msg.Content != nil {
+ return *msg.Content, nil
+ }
+ return "", nil
+ }
+
+ messages = append(messages, msg)
+ for _, call := range msg.ToolCalls {
+ result := cc.executeTool(ctx, agent, call.Function.Name, call.Function.Arguments)
+ messages = append(messages, orMessage{
+ Role: "tool",
+ Content: &result,
+ ToolCallId: call.Id,
+ })
+ }
+ }
+ return "I hit my tool-call limit before finishing. Please try a simpler question.", nil
+}
+
+// executeTool runs a fake tool (or a sub-agent) inside a plain child span.
+// Tool spans deliberately carry no gen_ai.* attributes: the tool calls are
+// already captured on the LLM rows via the completion payload, and gen_ai
+// attributes here would promote each execution to its own ai_traces row.
+func (cc *chatContext) executeTool(ctx context.Context, agent agentDef, name, args string) string {
+ toolCtx, span := cc.svc.tr.Start(ctx, "tool "+name, trace.WithAttributes(
+ attribute.String("tool.name", name),
+ attribute.String("tool.agent", agent.TraceName),
+ attribute.String("tool.args", truncate(args, 500)),
+ ))
+ defer span.End()
+
+ result := cc.dispatchTool(toolCtx, name, args)
+ span.SetAttributes(attribute.String("tool.result", truncate(result, 500)))
+
+ cc.events = append(cc.events, toolEvent{
+ Agent: agent.TraceName,
+ Tool: name,
+ Args: truncate(args, 300),
+ Result: truncate(result, 300),
+ })
+ return result
+}
+
+func (cc *chatContext) dispatchTool(ctx context.Context, name, args string) string {
+ var params map[string]string
+ if err := json.Unmarshal([]byte(args), ¶ms); err != nil {
+ params = map[string]string{}
+ }
+
+ switch name {
+ case "get_weather":
+ return fakeWeather(params["city"])
+ case "lookup_order":
+ return fakeOrderStatus(params["order_id"])
+ case "get_server_time":
+ return time.Now().Format("Monday, 2 Jan 2006 15:04:05 MST")
+ case "search_knowledge_base":
+ return fakeKnowledgeBase(params["query"])
+ case "calculate":
+ value, err := evalArithmetic(params["expression"])
+ if err != nil {
+ return "error: " + err.Error()
+ }
+ return strconv.FormatFloat(value, 'f', -1, 64)
+ case "ask_research_agent":
+ reply, err := cc.runAgent(ctx, researchAgent, params["question"], nil)
+ if err != nil {
+ return "research agent failed: " + err.Error()
+ }
+ return reply
+ case "ask_math_agent":
+ reply, err := cc.runAgent(ctx, mathAgent, params["problem"], nil)
+ if err != nil {
+ return "math agent failed: " + err.Error()
+ }
+ return reply
+ default:
+ return "unknown tool: " + name
+ }
+}
+
+// --- Fake tool implementations (deterministic per input, no external calls) ---
+
+func pick(seed string, options []string) string {
+ h := fnv.New32a()
+ h.Write([]byte(seed))
+ return options[int(h.Sum32())%len(options)]
+}
+
+func fakeWeather(city string) string {
+ if city == "" {
+ return "error: city is required"
+ }
+ condition := pick(city, []string{"sunny", "partly cloudy", "overcast", "light rain", "thunderstorms", "windy"})
+ h := fnv.New32a()
+ h.Write([]byte(city + "-temp"))
+ temp := 8 + int(h.Sum32())%22
+ return fmt.Sprintf("Weather in %s: %s, %d°C, humidity %d%%.", city, condition, temp, 40+int(h.Sum32())%45)
+}
+
+func fakeOrderStatus(orderId string) string {
+ if orderId == "" {
+ return "error: order_id is required"
+ }
+ status := pick(orderId, []string{
+ "processing in our warehouse",
+ "shipped and in transit with DHL",
+ "out for delivery today",
+ "delivered and signed for",
+ "delayed at customs, new ETA in 2 days",
+ })
+ h := fnv.New32a()
+ h.Write([]byte(orderId + "-eta"))
+ eta := time.Now().AddDate(0, 0, 1+int(h.Sum32())%5).Format("Mon, 2 Jan")
+ return fmt.Sprintf("Order %s is %s. Estimated delivery: %s.", orderId, status, eta)
+}
+
+func fakeKnowledgeBase(query string) string {
+ q := strings.ToLower(query)
+ snippets := []string{}
+ if strings.Contains(q, "traceway") || strings.Contains(q, "observab") || strings.Contains(q, "trace") {
+ snippets = append(snippets, "[doc:overview] Traceway is an error tracking and monitoring platform: endpoints, exceptions, logs, metrics, AI traces, and session replay in one self-hostable binary.")
+ }
+ if strings.Contains(q, "ai") || strings.Contains(q, "conversation") || strings.Contains(q, "llm") {
+ snippets = append(snippets, "[doc:ai-tracing] AI calls with gen_ai.* attributes are grouped into conversations via gen_ai.conversation.id; tool calls are parsed from the completion payload and per-user analytics key on user.id.")
+ }
+ if strings.Contains(q, "return") || strings.Contains(q, "refund") || strings.Contains(q, "policy") {
+ snippets = append(snippets, "[doc:returns] Acme shop policy: returns accepted within 30 days of delivery, refunds are processed to the original payment method within 5 business days.")
+ }
+ if strings.Contains(q, "shipping") || strings.Contains(q, "delivery") {
+ snippets = append(snippets, "[doc:shipping] Standard shipping takes 2-5 business days; express is next-day when ordered before 15:00.")
+ }
+ if len(snippets) == 0 {
+ snippets = append(snippets, "[doc:misc] No exact match found. Closest entry: the Acme handbook says to escalate unknown questions to a human agent.")
+ }
+ return strings.Join(snippets, "\n")
+}
+
+// evalArithmetic is a tiny recursive-descent evaluator for + - * / and
+// parentheses, so the calculate tool actually computes.
+func evalArithmetic(input string) (float64, error) {
+ p := &exprParser{src: strings.ReplaceAll(input, " ", "")}
+ if p.src == "" {
+ return 0, fmt.Errorf("empty expression")
+ }
+ value, err := p.parseExpr()
+ if err != nil {
+ return 0, err
+ }
+ if p.pos != len(p.src) {
+ return 0, fmt.Errorf("unexpected character %q at position %d", p.src[p.pos], p.pos)
+ }
+ return value, nil
+}
+
+type exprParser struct {
+ src string
+ pos int
+}
+
+func (p *exprParser) parseExpr() (float64, error) {
+ left, err := p.parseTerm()
+ if err != nil {
+ return 0, err
+ }
+ for p.pos < len(p.src) && (p.src[p.pos] == '+' || p.src[p.pos] == '-') {
+ op := p.src[p.pos]
+ p.pos++
+ right, err := p.parseTerm()
+ if err != nil {
+ return 0, err
+ }
+ if op == '+' {
+ left += right
+ } else {
+ left -= right
+ }
+ }
+ return left, nil
+}
+
+func (p *exprParser) parseTerm() (float64, error) {
+ left, err := p.parseFactor()
+ if err != nil {
+ return 0, err
+ }
+ for p.pos < len(p.src) && (p.src[p.pos] == '*' || p.src[p.pos] == '/') {
+ op := p.src[p.pos]
+ p.pos++
+ right, err := p.parseFactor()
+ if err != nil {
+ return 0, err
+ }
+ if op == '*' {
+ left *= right
+ } else {
+ if right == 0 {
+ return 0, fmt.Errorf("division by zero")
+ }
+ left /= right
+ }
+ }
+ return left, nil
+}
+
+func (p *exprParser) parseFactor() (float64, error) {
+ if p.pos < len(p.src) && p.src[p.pos] == '-' {
+ p.pos++
+ value, err := p.parseFactor()
+ return -value, err
+ }
+ if p.pos < len(p.src) && p.src[p.pos] == '(' {
+ p.pos++
+ value, err := p.parseExpr()
+ if err != nil {
+ return 0, err
+ }
+ if p.pos >= len(p.src) || p.src[p.pos] != ')' {
+ return 0, fmt.Errorf("missing closing parenthesis")
+ }
+ p.pos++
+ return value, nil
+ }
+ start := p.pos
+ for p.pos < len(p.src) && (p.src[p.pos] >= '0' && p.src[p.pos] <= '9' || p.src[p.pos] == '.') {
+ p.pos++
+ }
+ if start == p.pos {
+ return 0, fmt.Errorf("expected a number at position %d", start)
+ }
+ return strconv.ParseFloat(p.src[start:p.pos], 64)
+}
+
+func truncate(s string, max int) string {
+ if len(s) <= max {
+ return s
+ }
+ return s[:max] + "…"
+}
+
+// --- HTTP handler ---
+
+type chatRequest struct {
+ SessionId string `json:"sessionId"`
+ UserId string `json:"userId"`
+ Messages []struct {
+ Role string `json:"role"`
+ Content string `json:"content"`
+ } `json:"messages"`
+}
+
+func registerAIChatRoutes(router *gin.Engine, svc *otelService) {
+ model := os.Getenv("OPENROUTER_MODEL")
+ if model == "" {
+ model = "anthropic/claude-sonnet-5"
+ }
+
+ router.POST("/api/ai/chat", func(c *gin.Context) {
+ if os.Getenv("OPENROUTER_API_KEY") == "" {
+ c.JSON(http.StatusServiceUnavailable, gin.H{"error": "OPENROUTER_API_KEY is not set; add it to examples/devtesting-embedded/.env"})
+ return
+ }
+
+ var req chatRequest
+ if err := c.ShouldBindJSON(&req); err != nil {
+ c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
+ return
+ }
+ if req.SessionId == "" || len(req.Messages) == 0 {
+ c.JSON(http.StatusBadRequest, gin.H{"error": "sessionId and messages are required"})
+ return
+ }
+ if req.UserId == "" {
+ req.UserId = "anonymous"
+ }
+ last := req.Messages[len(req.Messages)-1]
+ if last.Role != "user" || strings.TrimSpace(last.Content) == "" {
+ c.JSON(http.StatusBadRequest, gin.H{"error": "the last message must be a non-empty user message"})
+ return
+ }
+
+ var history []orMessage
+ for _, m := range req.Messages[:len(req.Messages)-1] {
+ if m.Role == "user" || m.Role == "assistant" {
+ history = append(history, textMsg(m.Role, m.Content))
+ }
+ }
+
+ cc := &chatContext{
+ svc: svc,
+ model: model,
+ conversationId: req.SessionId,
+ userId: req.UserId,
+ }
+
+ reply, err := cc.runAgent(c.Request.Context(), mainAgent, last.Content, history)
+ if err != nil {
+ span := trace.SpanFromContext(c.Request.Context())
+ span.RecordError(err)
+ c.JSON(http.StatusBadGateway, gin.H{"error": err.Error()})
+ return
+ }
+
+ c.JSON(http.StatusOK, gin.H{
+ "reply": reply,
+ "events": cc.events,
+ "model": model,
+ })
+ })
+}
diff --git a/examples/devtesting-embedded/chat.html b/examples/devtesting-embedded/chat.html
new file mode 100644
index 00000000..9cc2ee47
--- /dev/null
+++ b/examples/devtesting-embedded/chat.html
@@ -0,0 +1,238 @@
+
+
+
+
+
++ {getRoleLabel(msg.role)} +
+ {#each getMessageImages(msg.content) as imageUrl} + {#if imageUrl && !imageUrl.includes('REDACTED')} +Matched terms: {terms.join(', ')}
++ Arguments +
+{formattedArguments}
+ {resultText}
+ {/if}
+ user_id or request_id: every distinct value is stored as a separate sample, which bloats storage and slows queries.
+ {:else if activeTab === 'ai'}
+ + Built-in profanity lists to scan conversations with. Deselect all to rely on custom terms only. Matching is per whole word, so these packs only cover languages with space-separated words. +
+
+
+ One term per line. AI conversations containing these terms, in addition to the selected language packs, are marked as flagged and filterable on the Conversations page. Terms match whole words, case-insensitively, and only apply to calls ingested afterward. +
+Conversation
+ + {trace.conversationId} + +- {getRoleLabel(msg.role)} -
- {#each getMessageImages(msg.content) as imageUrl} - {#if imageUrl && !imageUrl.includes('REDACTED')} -Above the 95th percentile turns for this period (P95: {Math.round(thresholds?.p95Turns ?? 0)})
+{conv.toolNames.join(', ')}
+{conv.models.join(', ')}
+Above the 95th percentile cost for this period (P95: {formatCost(thresholds?.p95Cost ?? 0)})
+Started {formatDateTime(conv.firstSeen, { timezone })}
+Turns
+{formatCount(stats.turns)}
+Total cost
+{formatCost(stats.totalCost)}
+Total tokens
+{formatTokens(stats.totalTokens)}
+Tool calls
+{formatCount(stats.toolCallCount)}
+Duration
+{conversationDurationLabel(stats)}
++ Turn {i + 1} · {formatDateTime(turn.recordedAt, { timezone })} +
+ {#if turn.input} +{formatConversationContent(turn.input)}
+ {formatConversationContent(turn.output)}
+ + No conversation payload was captured for this turn. +
+ {/if} +avg {user.avgTurns.toFixed(1)} · min {user.minTurns}
++ Custom flagged terms are managed in the project settings AI tab. +
+- Monitor LLM costs, token usage, latency, and conversations across - every provider. Works with OpenRouter, OpenAI, Anthropic, and any - OpenTelemetry-compatible provider. + Monitor LLM costs, token usage, latency, conversations, and tool + calls across every provider. Group calls into conversations, break + them down per user, and flag the ones that need attention. Works + with OpenRouter, OpenAI, Anthropic, and any OpenTelemetry-compatible + provider.
+ Traceway keys per-customer analytics on the{" "}
+ user.id span attribute (with OpenRouter,
+ just pass the user field in your requests).
+ Set it to a stable identifier for the end user of your
+ product: your internal account id, a tenant id, or an
+ email. The same user must carry the same value across
+ all their conversations.
+
+ Never put session ids or random values there — that is + what the conversation id is for. With a stable id, the + Users view shows conversation count, median conversation + length, and cost per conversation for every customer. + If PII must stay out of telemetry, use an internal id or + a hash; the analytics only need stability. +
+ + ), + }, + { + q: "Can I find conversations containing specific words, like profanity?", + a: "Yes. Traceway scans every prompt and completion at ingestion against built-in profanity lists (seven languages) plus custom terms you configure per project. Matches are stored on the call, so flagged conversations are instantly filterable and searchable, and an alert rule can notify you the moment flagged content appears. Because scanning happens at ingest, there is no expensive query-time content search.", + }, { q: "Can I track costs across multiple models?", a: ( diff --git a/website/components/home-tabs.tsx b/website/components/home-tabs.tsx index 8664a396..3a4659b1 100644 --- a/website/components/home-tabs.tsx +++ b/website/components/home-tabs.tsx @@ -73,7 +73,7 @@ const tabs: Tab[] = [ color: "var(--a3)", heading: "Track every AI call, its cost, and its conversation.", description: - "Monitor LLM costs, token usage, and latency across every provider. See the full prompt and completion for every call, with per-agent and per-model breakdowns.", + "Monitor LLM costs, token usage, and latency across every provider. Group calls into conversations with tool calls rendered inline, break costs down per user, and flag conversations containing terms you care about.", bullets: [ "Per-call cost and token tracking", "Conversation replay with chat view", diff --git a/website/public/images/ai-conversation-detail.png b/website/public/images/ai-conversation-detail.png new file mode 100644 index 00000000..1c50f2d1 Binary files /dev/null and b/website/public/images/ai-conversation-detail.png differ diff --git a/website/public/images/ai-conversations.png b/website/public/images/ai-conversations.png new file mode 100644 index 00000000..118fbb19 Binary files /dev/null and b/website/public/images/ai-conversations.png differ diff --git a/website/public/images/ai-flagged-terms.png b/website/public/images/ai-flagged-terms.png new file mode 100644 index 00000000..b9025855 Binary files /dev/null and b/website/public/images/ai-flagged-terms.png differ