Skip to content

docs(TSP-1334): document new evals table features - #715

Closed
claude[bot] wants to merge 1 commit into
mainfrom
docs/TSP-1334
Closed

docs(TSP-1334): document new evals table features#715
claude[bot] wants to merge 1 commit into
mainfrom
docs/TSP-1334

Conversation

@claude

@claude claude Bot commented Jul 10, 2026

Copy link
Copy Markdown

Summary

  • Added <Note> callout in the Scenario results section documenting column sorting (server-side, cross-page), column resizing, and the column visibility picker. Notes that the Test column is pinned and that visibility preferences are saved per view.
  • Extended the scenario results column table with four new optional cost breakdown columns: Scenario credits, Execution credits, Checks credits, and Checks actions — each noting they are toggled via the column visibility picker.
  • Expanded the Viewing dashboard insights section's runs list from a single bullet into a full column reference table, with a matching <Note> callout for table interaction features.
  • Updated the Cost and billing section to surface that the per-component breakdown is also available inline via the column visibility picker, not only through the click-to-expand modal.
  • Light update to the Runs bullet in the Evals sections overview to mention sorting and the column visibility picker.

Relates to: https://linear.app/relevance/issue/TSP-1334/

Test plan

  • Verify <Note> callouts render correctly in Mintlify preview (no attributes, single paragraph)
  • Confirm new column table rows display as expected
  • Check internal anchor links (#scenario-results, #viewing-dashboard-insights, #cost-and-billing) still resolve
  • Review sentence case on all headings and table column names

🤖 Generated with Claude Code

… column picker, cost breakdown columns)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@claude claude Bot added the docs-drafter Documentation drafted by Claude label Jul 10, 2026
@linear

linear Bot commented Jul 10, 2026

Copy link
Copy Markdown

TSP-1334

@mintlify

mintlify Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated (UTC)
relevanceai 🟢 Ready View Preview Jul 10, 2026, 5:02 AM

💡 Tip: Enable Workflows to automatically generate PRs for you.

@github-actions

Copy link
Copy Markdown
Contributor

🎯 Vibe check

Reviewed: 1 file (1 with issues, 0 clean)

Scores

Dimension Score What's holding it back
🟡 Consistency 7/10 Repeated lowercase "agent" and "tool" in the Cost and billing section — these are Relevance AI product features and should be "Agent" / "Tool". Pattern appears across ~6 locations concentrated in lines 253, 363, 365, 369, 376. Two additional hits at lines 11 and 93.
🟢 Technical clarity 9/10 No issues found. Edge cases are well-covered (truncation behavior, plan-specific columns, Workforce-specific Tool simulation scoping). UI element names are exact.
🟡 Non-technical clarity 8/10 The intro frames Evals as "what it can do" before establishing "why you'd want it." A reader new to AI agent quality assurance has no hook for why testing matters before the feature walkthrough begins.
🟢 Structure 9/10 No issues found. Logical flow from concept → how-to → results → monitoring → publishing → cost → best practices → FAQ. Components are well-chosen for their content type.

Score key: 🟢 9–10, 🟡 6–8, 🔴 1–5. Scores are a single overall judgment about the whole PR.

Overall vibe: Solid, detailed feature doc with specific UI references, good edge-case coverage, and a clear billing breakdown that will save readers a lot of confusion. The only real problem is a capitalization pattern that breaks down in the Cost and billing section — "agent" and "tool" drift to lowercase throughout that section when they should stay "Agent" and "Tool" as Relevance AI product terms.

🔧 Issues (6)
  • build/agents/build-your-agent/evals.mdx:11 — "sub-agents and tools inside a Workforce" → "sub-agents and Tools inside a Workforce" (Tools are a Relevance AI product feature)

  • build/agents/build-your-agent/evals.mdx:93 — "a sub-agent or tool in the Workforce" and "a particular sub-agent used a given tool" — both instances of "tool" → "Tool" (the node and the thing being asserted are Relevance AI Tools)

  • build/agents/build-your-agent/evals.mdx:253 — "each Check plus the agent's tool runs" → "each Check plus the Agent's Tool runs"

  • build/agents/build-your-agent/evals.mdx:363 — "the agent's own run" → "the Agent's own run"

  • build/agents/build-your-agent/evals.mdx:365 — "the agent's own tool runs" → "the Agent's own Tool runs"

  • build/agents/build-your-agent/evals.mdx:369,376 — "the agent's own execution" (line 369), "The agent's own run during the test", "each tool the agent runs", "when the agent ran" (line 376) — all three occurrences of "agent" → "Agent" and "tool" → "Tool"

🧩 Component suggestions (1)
  • build/agents/build-your-agent/evals.mdx:383–402 — The best practices <CardGroup> is borderline per CLAUDE.md ("not appropriate for best practices that read naturally as flowing bullets"). The six cards here do have icons and a short description each, so they carry enough weight to justify the visual treatment — but if these best practices ever grow short (one-liners), consider converting to a bulleted list.
⚠️ Contradictions (1)
  • build/agents/build-your-agent/evals.mdx:11 says "Evals apply to both individual Agents and Workforces, including the sub-agents and tools inside a Workforce." The sibling build-overview.mdx lists the Build page sections (Prompt, Tools, Knowledge, Triggers, Alerts, Memory, Variables, Advanced) with no mention of the Evaluate tab. This isn't a direct factual contradiction, but build-overview.mdx will mislead readers into thinking they've seen the full Agent builder — it should eventually be updated to reference the Evaluate tab. Not a blocker for this PR but worth a follow-up ticket.
🔋 Credit usage
Item Count
Files reviewed 1
Context pages read 2
Total lines processed ~609

Files read: build/agents/build-your-agent/evals.mdx (445 lines), build/agents/build-your-agent/build-overview.mdx (25 lines), build/agents/build-your-agent/memory.mdx (164 lines)

@jordanc-relevanceai

Copy link
Copy Markdown
Collaborator

Superseded by #742, which consolidates the five open PRs editing build/agents/build-your-agent/evals.mdx (#698, #704, #715, #716, #732).

The table-behaviour content is carried over in full — server-side sorting, column resizing, the visibility picker, per-view persistence, the pinned Test and Name columns, and the expanded Monitor runs list. evals-column-sorting is at 100%, so all of that is safe to document.

The four cost-breakdown columns were left out. evals-cost-comparison has exactly one release condition: a person-property match on a single internal email address. No customer can see Scenario credits, Execution credits, Checks credits or Checks actions, so documenting them would send readers looking for picker entries that aren't there. Worth adding back when the flag widens — the wording is preserved in this PR's history.

Two related notes: some of those columns are additionally gated to resourceType === "agent", so they'd never show for Workforces; and I couldn't verify the three-way credits breakdown modal, so that was left out too.

Your "Only appears on usage-based plans" note on Actions checks out and is kept.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs-drafter Documentation drafted by Claude

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant