feat(gooddata-eval): check that gen-ai sends the Document wire names

[romrak]

gd-eval's Publisher copilot evaluator now uses the Document names and fails a run when gen-ai still sends a Report name. Do not merge before gen-ai ships the Document wire names (LX-3244): against today's gen-ai every document eval fails, by design.

Jira: LX-3248, epic GDP-3485. Wire names as proposed on LX-3244.

Two commits, review them one by one:

1. refactor — rename only. Evaluator, test kind and score names go from report to document; the old Python names keep working.
2. feat — the wire names flip, plus the new document_wire_names check.

What this implements

Where	Before	After
Test kind	agentic_report_skill	agentic_document_skill
Module	core/agentic/report_skill.py	core/agentic/document_skill.py
Scores	report_drafted, report_ref_matches, …	document_drafted, document_ref_matches, …, plus document_wire_names
Answer part read	type: "report", report, report_ref, saved_report_id, base_report_id	type: "document", document, document_ref, saved_document_id, base_document_id
Tool / skill	draft_report, report_builder	draft_document, document_builder
User context	view.report passed through	view.document; view.report rejected before the first request

What a run against today's gen-ai reports (from the unit test that pins it):

Document skill assertion failed. … gen-ai still sends the Report names, so it predates the Document rename.
Failures: gen-ai sends the Report names: skill 'report_builder', tool 'draft_report', part type 'report',
part key 'report', part key 'report_ref', part key 'base_report_id', part key 'saved_report_id',
document type 'report'; the agent never produced a successful draft_document call.

Decisions

1. The evaluator accepts only the Document names; there is no fallback to the Report ones.
   A fallback would pass a half-switched chain, and the point is to prove every link was renamed.

• src/gooddata_eval/core/agentic/document_skill.py — legacy_wire_names, _LEGACY_* constants
• The conversation stops at an old draft_report instead of replying up to max_iterations times.

2. document_wire_names is a strict check published on every run, and its failure leads the list.
   Without it an old gen-ai reads as "never produced a draft_document call", which hides why. It scans every turn: part type and keys, the embedded document's type, the old tools, and report_builder in either the set_skills arguments or its result.

• It adds one always-present check, so a failing run's quality_score moves (4/6 → 5/7 when it passes). Scores before and after this PR are not comparable anyway, because the names changed.

3. The old Python names keep working; the old test kind does not.
   gdc-nas imports gooddata_eval.core.agentic.report_skill directly, so its names stay. The agentic_report_skill kind is gone: its one dataset needs migrating for the Document wire names anyway, and the Tavern shim calls the function, not the kind.

• core/agentic/report_skill.py re-exports the old names with a DeprecationWarning.
• core/agentic/__init__.py keeps the old aliases.
• An item still on agentic_report_skill is skipped as an unsupported test_kind and listed at the end of the run.

4. Langfuse score names change from report_* to document_*.
   Score history splits at this release: old runs keep report_*, new runs get document_*.

What comes next

• gdc-nas, when bumping the gooddata-eval pin: tests/tavern-e2e/ci/combo_report.py _REPORT_*_FIELDS → document_*, the report_skill_agentic.py import path and the Langfuse dataset kind.
• The Langfuse dataset items: test_kind agentic_report_skill → agentic_document_skill, and view.report → view.document in their user context.

Test plan

• tox py310–py314 for gooddata-eval: all green (1622 passed, 1 skipped on each).
• ruff format --check and ruff check repo-wide: clean. ty check: no new diagnostics in src; the test file keeps its existing test-double diagnostics.
• Each commit passes on its own (1612 and 1622 tests).
• Live run against staging: not possible until LX-3244 ships.

risk: high — a published package that stops passing against current gen-ai.

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features
  • Added evaluation support for agentic document skills, including checks for document content, structure, references, save state, and narrative quality.
  • Added document-skill support to the evaluation runner and public exports.
• Compatibility
  • Preserved existing report-skill imports and names as deprecated compatibility aliases.
• Behavior Changes
  • Document content is recognized in chat responses. Report-skill test kinds are no longer accepted; legacy report wire names fail document-skill checks.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Organization UI
• Review profile: CHILL
• Plan: Essentials
• Run ID: e810e38c-a383-404f-bef2-21b2342218a6

---

---

📥 Commits

Reviewing files that changed from the base of the PR and between b41dd40 and 92fe97e.

---

---

📒 Files selected for processing (2)

• packages/gooddata-eval/src/gooddata_eval/cli/agentic_runner.py
• packages/gooddata-eval/tests/test_agentic_runner.py

---

---

Included review availability: This review used your included allowance. 3 included reviews remain after this review. Your included PR review attempts over the past 7 days set your current allowance at 5 reviews per hour.

---

---

---

📝 Walkthrough

📝 Walkthrough

Walkthrough

The pull request adds document-skill evaluation for document drafts, including validation, scoring, optional narrative judging, multi-run evaluation, and trace scoring. It retains report-skill names as deprecated compatibility aliases and dispatches document-skill inputs through the CLI.

Changes

Document Skill Evaluation

Layer / File(s)	Summary
Document validation and scoring packages/gooddata-eval/src/gooddata_eval/core/agentic/document_skill.py, packages/gooddata-eval/tests/test_agentic_document_skill.py	Adds fixture validation and document response checks for structure, references, saved state, periods, visualizations, narrative slots, and legacy wire names.
Conversation and narrative judging packages/gooddata-eval/src/gooddata_eval/core/agentic/document_skill.py, packages/gooddata-eval/tests/test_agentic_document_skill.py	Adds document conversation handling, clarification diagnostics, optional narrative judging, and per-run results.
Multi-run evaluation and trace scoring packages/gooddata-eval/src/gooddata_eval/core/agentic/document_skill.py, packages/gooddata-eval/tests/test_agentic_document_skill.py	Adds K-run evaluation, scored-run trace submission, gate handling, and result or error reporting.
Public exports and dispatch compatibility packages/gooddata-eval/src/gooddata-eval/core/agentic/...	Exports document-skill APIs and retains report-named aliases. Routes document-skill CLI inputs to the evaluator and recognizes document as a known multipart type. Related tests cover dispatch, aliases, multipart handling, and trace-linker evaluator coverage.

Priority: ⬇️ Low

Estimated code review effort: 4 (Complex) | ~60 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant Runner as run_agentic_document_skill
  participant Chat as ChatClient
  participant Judge as LLMJudge
  Runner->>Chat: Send conversation request
  Chat-->>Runner: Return tool events and document part
  Runner->>Runner: Evaluate document response
  Runner->>Judge: Judge rendered document text when narrative is expected
  Judge-->>Runner: Return narrative verdict

Loading

---

---

Merge Risk: ⚪ Minimal · up to 92fe9

Legacy report-kind items are skipped as intended, and no merge-blocking issue remains after normal checks.

Pre-merge checks | 4 | 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 20.33% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 123 functions across 9 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.
Description Check		Check skipped - CodeRabbit’s high-level summary is enabled.
Title check		The title clearly and concisely describes the primary change: validating that GenAI uses Document wire names instead of legacy Report names.

---

• Fix all pre-merge checks with AI

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Commit to this branch
• Create a new PR

---

---

---

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

_{A rabbit checks each page at dawn
And tests the charts before moving on
It asks the judge to read the prose
Then scores the drafts from nose to toes
Report names stay, as aliases do
The document path is ready too}

_{Comment @coderabbitai help to get the list of available commands.}

[codecov]

Codecov Report

❌ Patch coverage is 99.13978% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.32%. Comparing base (a1ae1c2) to head (92fe97e).
⚠️ Report is 2 commits behind head on master.

Files with missing lines	Patch %	Lines
...l/src/gooddata_eval/core/agentic/document_skill.py	99.09%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1862      +/-   ##
==========================================
+ Coverage   84.19%   84.32%   +0.12%     
==========================================
  Files         333      335       +2     
  Lines       23261    23439     +178     
==========================================
+ Hits        19585    19764     +179     
+ Misses       3676     3675       -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.