💡 Avatar-Narrated Workflow Quality Scorecard: Rubric-Based Output Assessment for Non-Technical Users

[github-actions[bot]]

Summary

After a workflow completes, the avatar doesn't just present the output — it narrates a quality assessment. "Your brainstorming document covers 8 idea categories with strong diversity. The competitor analysis section could benefit from 2-3 more data sources. Want me to strengthen it?" This turns TalkTerm from an output generator into a quality partner that non-technical users can trust to evaluate their work product.

Market Signal

Anthropic Managed Agents introduced "Outcomes" in 2026 — rubric-based grading of agent output quality with structured scoring dimensions. Claude Cowork and Manus both present outputs but leave quality assessment entirely to the user. For non-technical users who cannot independently judge whether AI-generated output is thorough, accurate, or actionable, this is a critical trust gap that no competitor addresses. The broader trend of "AI as copilot, not autopilot" (UX Design Institute 2026 trends) supports giving users quality signals rather than expecting blind trust.

User Signal

The PRD's success criteria state: "A non-technical user completes a full agent workflow and produces a real output artifact." But producing an artifact is necessary, not sufficient — Sarah the PM needs to know her brainstorming document is thorough before sharing it with her team. Without quality feedback, users face two failure modes: over-trust (shipping poor output to their team, damaging credibility) or under-trust (not using TalkTerm for important work, limiting adoption). Both undermine the product's business success criteria.

Technical Opportunity

Claude Agent SDK can execute a follow-up evaluation pass using defined quality rubrics after workflow completion. The evaluation uses a separate prompt with explicit, measurable criteria — not "is this good?" but "count distinct categories, check for evidence citations, verify actionability of each item." Results render in the output panel's comparison table display mode (FR45) while the avatar narrates key findings. This maps cleanly to existing architecture:

• Agent evaluation call via AgentBackend port → structured scoring response
• Scores rendered in ComparisonTable display mode (already specified in FR45/UX-DR13)
• Avatar narrates findings via existing TTS pipeline
• Future upgrade path: Anthropic Managed Agents Outcomes API for standardized rubric grading

Assessment

Dimension	Score	Rationale
Feasibility	high	Implementable via prompt engineering now; one additional API call per workflow
Impact	high	Converts TalkTerm from output generator to quality partner — addresses core trust gap
Urgency	med	Valuable differentiation but not blocking MVP; can ship as a post-MVP enhancement

Adversarial Review

Strongest objection: Using the LLM to evaluate its own output is circular — the same model that generated a mediocre brainstorm will rate it highly. This adds API cost for potentially unreliable assessments. Additionally, Managed Agents Outcomes API is designed for cloud deployments and may never support in-process Electron usage.

Rebuttal: Self-evaluation bias is real but mitigatable with structured rubrics. Research consistently shows that LLMs are better evaluators than generators when given explicit scoring criteria (e.g., "count the number of distinct categories" rather than "rate diversity"). The cost is one additional API call per completed workflow — negligible compared to the multi-turn workflow itself. The proposal doesn't depend on Managed Agents — prompt-based evaluation works now, with a natural upgrade path if/when the Outcomes API supports local deployment.

Suggested Next Step

Define 3-4 quality rubric templates for initial workflow types: brainstorming (category diversity + idea count + actionability per idea), competitive analysis (dimension coverage + evidence density + source recency), architecture review (trade-off completeness + constraint coverage + risk identification). Implement as a post-workflow agent evaluation call that feeds structured scores to the avatar narration and comparison table display mode.