docs: consolidate evals page updates (replaces #698, #704, #715, #716, #732)

[jordanc-relevanceai]

Summary

Consolidates the five open PRs editing build/agents/build-your-agent/evals.mdx into one change. Branched from main.

• Argument conditions on Tool Usage checks, plus the Node picker for Workforce scoping
• Version dropdown in the run modal, with the Workforce caveat and a link to the MCP equivalent
• Table behaviour on scenario results and the Monitor runs list — sorting, resizing, column visibility, pinned first column
• Alarms and Auto-diagnosis by Inventor
• Workforce eval wording: Evaluate tab on the canvas, per-node Tool simulation with connection labels, and a create-a-workforce.mdx cross-link

Replaces these PRs (now closed)

PR	What it covered	Status here
#698	Workforce evals, node scoping, per-node simulation	Folded in
#704	Tool-usage argument matching	Folded in, minus its flag caveat
#715	Evals table features + cost breakdown columns	Table features folded in; cost columns omitted
#716	Version picker (evals half)	Folded in. Its MCP half went to #741
#732	Alarms + Auto-diagnosis	Folded in, with 2 corrections

Five PRs editing one file in five places would have conflicted on merge even though the sections themselves barely overlapped.

Omitted: the four cost-breakdown columns from #715

#715 documented Scenario credits, Execution credits, Checks credits and Checks actions as columns you toggle via the visibility picker. They are gated on the evals-cost-comparison flag, whose only release condition is a person-property match on one internal email address (danial@relevanceai.com).

No customer can see these columns, so documenting them would send readers hunting for a picker entry that isn't there. Left out entirely rather than caveated. Worth revisiting when the flag widens — the rest of #715's table content is in, since evals-column-sorting is at 100%.

Two smaller notes on #715: some cost columns are additionally gated to resourceType === "agent", so they would never have appeared for Workforces; and I could not verify the three-way credits breakdown modal, so that content is also left out.

Corrections to #732's auto-diagnosis

The required alert channel is on Inventor, not the alarm. #732 said auto-diagnosis "requires at least one alert channel configured on the alarm" and delivers "to the alert channels configured by the alarm creator". In ObservabilityEvalAlertConfig.vue, the check reads inventorSettingsQuery.data.value?.settings.notification_channels — the Inventor settings modal's alerts tab — and the toggle stays hidden until that is set up. The code comment adds: "once enabled, delivery uses the enabler's channels, not the viewer's."

"Inventor is an Enterprise-only feature" softened to the pilot framing. There is no plan or tier gate in the code. Visibility requires three flags — alarm-auto-diagnose-ui (100%), inventor-scheduled-check-ins-fe (100%), and invent-v3, which is cohort-limited to "Relevance AI Employees" and "Invent + Evals Pilot (Mid 2026)". So the real constraint is the Inventor pilot, which is how the Invent page already describes itself.

Stale caveats dropped

• Update evals docs: Add tool-usage argument matching feature #704's "currently behind the evals-toolusage-argument-matcher feature flag" — that flag is at 100% rollout. It also named an internal flag key in customer docs, which we don't do elsewhere on the page.
• Alarms themselves need no caveat: observability-alarms is at 100%.

Verified as accurate and kept

• Four match types — exact, contains, regex, and has-a-value, confirmed in MatcherEditor.vue. Update evals docs: Add tool-usage argument matching feature #704's "Exact match / Contains / Regex / Not empty" naming is closer to the UI than the changelog wording in the recently closed changelog: Tool Argument Matching Check Type for Evals #727, so Update evals docs: Add tool-usage argument matching feature #704's table is used.
• Type constraints documented as a real limit: contains is text-only, exact match is bound to the argument's primitive type or enum values, and unresolvable argument paths are dropped.
• "Only appears on usage-based plans" for Actions columns — the code hides Actions for legacy credit-billed accounts.
• Version picker — eval-version-picker is at 100%, and the Workforce topology-pinning caveat comes straight from the tool schema.

Test plan

• Mintlify preview renders the Argument conditions table inside the Tool Usage accordion
• Alarms section renders between Monitor and Publish
• /integrations/mcp/mcp-server#running-evaluations resolves (depends on docs: consolidate MCP run-only access + cancellation tools, fix invented tool signatures (replaces #730, #733) #741)
• Run modal steps still number 1–4 correctly

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated (UTC)
relevanceai	🟢 Ready	View Preview	Jul 31, 2026, 7:09 AM

💡 Tip: Enable Workflows to automatically generate PRs for you.