feat(gooddata-eval): per-trace latency percentiles and a restyled HTML report

packages/gooddata-eval/README.md

@@ -308,13 +308,69 @@ The other six agentic kinds report `0.0` for the first three. Trace linking itse
 runner rather than by `evaluate_agentic_*`, so a direct library caller sees `langfuse_s: 0.0` even though its
 linking ran. Pass `TAVERN_E2E_SKIP_TRACE_LINK=1` to opt out of linking altogether.
 
+### Latency: single-shot and multi-turn
+
+`avg_latency_s` is one number over a mixed population. The report instead keeps two kinds of item apart, by
+their eval kind, and states latency for each:
+
+- **Single-shot** — kinds that send exactly one chat message per run: every non-agentic kind, plus
+  `agentic_general_question`, `agentic_guardrail` and `agentic_search`.
+- **Multi-turn** — kinds that drive a simulated user or fixture turns: `agentic_visualization` (and
+  `vis_agentic`), `agentic_metric_skill`, `agentic_alert_skill`, `agentic_dashboard_skill`,
+  `agentic_document_skill`, `agentic_kda_skill`, `agentic_anomaly_detection`, `agentic_what_if` and
+  `agentic_conversation`.
+
+```json
+"summary": {
+  "by_mode": {
+    "single_shot": {"total": 12, "passed": 10,
+                    "traces_s": {"n": 24, "mean": 5.39, "trimmed_mean": 5.36, "min": 3.88, "p50": 5.33, "p70": 6.23, "p90": 6.82, "p99": 7.18, "max": 7.22}},
+    "multi_turn":  {"total": 6, "passed": 4,
+                    "traces_s": {"n": 45, "mean": 6.91, "trimmed_mean": 6.84, "min": 2.78, "p50": 6.52, "p70": 8.03, "p90": 10.79, "p99": 13.57, "max": 14.25},
+                    "runs_s":   {"n": 15, "mean": 23.55, "trimmed_mean": 23.31, "min": 12.64, "p50": 21.23, "p70": 28.52, "p90": 33.02, "p99": 36.94, "max": 37.51}}
+  }
+}
+```
+
+`by_mode` is in each run's `summary` and in its `comparison` entry. `total` and `passed` count the evaluated
+items of that mode (skipped ones belong to neither). Every distribution is `null` when nothing was timed.
+Each item also states its `mode`, its `trace_latencies_s` and its `run_latencies_s`.
+
+- **Per trace is the main figure.** gen-ai starts one trace per chat message, so a trace is one chat turn.
+  Its latency is that turn's response time, from sending the message to the end of its stream
+  (`turn_wall_clock_sec`), recorded where the chat client completes the turn. It holds none of the LLM judge,
+  the simulated user or conversation create/delete. `dashboard_summary` is timed the same way around its one
+  request. A turn cut off by its turn or item deadline (or ended by an error event) still counts, with the
+  time it ran for: those are the slowest turns, and the run that scored them is in the per-run figures too. An
+  item that errored outside its evaluator's own failure reporting contributes neither.
+- **Single-shot** runs are exactly one trace, so `traces_s` is also the latency of a run.
+- **Multi-turn** runs span several traces, so they get both views: `traces_s` over every turn of every
+  conversation (comparable with single-shot), and `runs_s`, the wall clock of each whole conversation from just
+  before its first message to the end of its scoring — every turn, simulated-user reply and judge call, but not
+  conversation create/delete. `agentic_conversation` drives its fixture once and contributes one run per item.
+  The two single-shot and multi-turn populations are never mixed.
+- **Percentiles** use linear interpolation between closest ranks (NumPy's default), so `p50` is the median and
+  the report carries no separate median field. On a small sample the high percentiles sit at or near the
+  maximum: P99 over fewer than 100 samples is effectively the slowest one, which is why `n` always travels with it.
+- **Trimmed mean** (`trimmed_mean`) is the average of the middle 80%: the fastest 10% and the slowest 10% of the
+  samples are dropped from each end (`n * 10 // 100` of them, so 25 samples drop two each) before averaging. It is
+  not the median, which is one middle value (`p50`). It is `null` below 10 samples, where there is nothing to
+  trim and the figure would only repeat `mean`.
+- **Older reports** have no `by_mode`. The HTML report says so and nothing is backfilled.
+
+`avg_latency_s` and the other existing fields are unchanged. The console prints each run's pass counts per
+mode and one `Latency, <population> (n=...)` line per distribution beneath its summary line, and Trace
+P50/P90/P99 columns (single-shot) in the multi-model comparison.
+
 ---
 
 ## `gd-eval report`
 
 Turns JSON report(s) into one HTML file you can actually navigate. No server, no
 credentials, no external assets — it opens over `file://`, attaches to a Jira issue and
-survives a Slack thread.
+survives a Slack thread. It follows GoodData's product look (colours, type, status chips). The faces are Inter, with
+Funnel Display for headings, named but never loaded, so a viewer without them gets the system sans and the file stays
+self-contained.
 
 ```bash
 # one run
@@ -333,25 +389,74 @@ gd-eval report results.json -o customer.html --redact
 | `--title TEXT` | Title shown in the report header. |
 | `--redact` | Drop conversation/response ids and raw reasoning, and rename models to `Model A`, `Model B`, … Pass rate, per-item results, questions and latency survive. |
 
-The report is a *view* over the JSON — it computes no numbers of its own. It gives you:
-
-- **Run cards and a comparison table** — pass rate, quality, latency per run.
-- **An item table** with a pass/fail column per run, so a model or run-over-run
-  regression is one glance rather than a hand-assembled spreadsheet.
-- **An expression filter** for cross-cutting questions the fixed filters can't
-  anticipate, e.g. `d.filter_ranking_score === false` or
+The report is a *view* over the JSON. Unfiltered, every figure on it is read from the report, so it cannot
+disagree with the console or the JSON. Narrow the selection with a filter and the figures are recomputed in the
+page from the selected items' own records, by the same rules gd-eval uses. The page reads top to bottom as a
+document, in plain words: a *test case* is one prompt given to the assistant, an *attempt* is one go at it, and the
+two formats are *single-question* tests (one answer, no follow-up) and *conversation* tests (a back-and-forth with a
+simulated user), which are `single_shot` and `multi_turn` in the JSON.
+
+- **A masthead** with the report's title (`--title`, or "Run history", "Run comparison" or "Run results" plus the
+  model and workspace), when it was generated in the reader's own time zone, and a **Run setup** tooltip listing what
+  is the same in every run (model, provider, workspace, dataset, test cases, attempts per test case, pass rule) and
+  what differs. Results from different setups are not directly comparable, so a caution icon on a latency row and the
+  *New* / *Removed* marks in the test case table say where two runs stopped being alike.
+- **Filters in two tiers** — the **Filter the report** bar at the top (category, format, result and run) decides what
+  the whole page covers: the Results cards, the latency tables and the test case table all follow it, so the runs stay
+  comparable. The Test cases section repeats those four and adds the search box and the expression; they narrow the
+  table only, inside whatever the top bar allows, and only offer what the top bar leaves. Category, result and run are
+  attribute filters like the product's: a search box, an All row, the values with an **Only** shortcut and a count,
+  then Cancel and Apply. Changes are staged in the popup and take effect on Apply; closing it any other way discards
+  them. Nothing ticked means all. The run filter chooses which runs are shown; a result matches a test case when any of
+  the runs in play has it (so "Failed" over previous and latest finds every test case that broke in either), and the
+  expression is evaluated once per run in play with the same any-run rule. A filtered page states how many test cases
+  it covers, and each tier has its own Reset.
+- **Results** — three headline cards for the latest run: test cases passed (of those evaluated: skipped ones are not
+  counted, as in the console's percentage), the average quality score and the median time per attempt. With an earlier run each card carries one change line in words, "down from 86% on Sep 28", rather
+  than a signed number. The line is green when that direction is good and red when it is not, with a chevron and
+  screen-reader text (*Better* / *Worse*) so colour is not the only signal. From three runs each card also has a
+  sparkline of every run: hover a point for its value, click it to compare the latest run with that one instead of the
+  previous. A pick that a filter has removed, or that has become the latest run, falls back to the previous run. The
+  pass-rate card links to the test case table to see *which* ones changed; the split by format lives in that table's
+  filters.
+- **Latency sections** — **Single-question tests · response time** first, then **Conversation tests**, which shows
+  both per response (each assistant reply, one gen-ai trace) and per conversation (start to finish, with the
+  simulated user and the judge). Each is a table with one row per run, newest first, giving the fastest, the median
+  (P50), P90 and the slowest. The quickest run in each column is marked green and the slowest red, each with a chevron
+  (down for least time, up for most) so colour is not the only signal, and the latest run is marked in lime. A caution
+  icon flags a run timed on a different number of samples than the one before. Unfiltered they are read from
+  `by_mode`; a filtered view recomputes them in the page by the same rules. P70, P99, the average and the trimmed
+  average stay in the JSON (P99 over a few dozen samples is only the slowest one again). Seconds read with one
+  decimal and no redundant ".0" (`3s`, `4.6s`).
+- **A test case table**, newest first: the latest run's result, a **Since previous run** column, then the earlier runs
+  going back in time (click any result to open the drawer on that run). The change column reads *Regressed* (passed in
+  the previous run, not in the latest), *Fixed*, *Flaky* (flipped between pass and fail more than once), *New*
+  (in no earlier run) or *Removed* (in an earlier run, gone from the latest). A test case skipped in the latest run
+  has no label, since the change would describe earlier runs. With several runs the table opens on the regressions. Two time columns
+  keep the populations apart: **Response time** is the median of the test case's assistant responses, and **Time per
+  attempt** is the mean wall-clock of a whole attempt (the responses plus the judge and, for conversations, the
+  simulated user). With more than two runs shown they follow the latest run only. Hover a value to see every sample
+  behind it; the drawer labels its two latency chips the same way. A report written before per-trace timing has no
+  response figure and shows `avg_latency_s` as the attempt figure.
+- **An expression filter**, folded under *Advanced: filter by a check expression* in the Test cases section, for
+  cross-cutting questions the fixed filters can't anticipate, e.g. `d.filter_ranking_score === false` or
   `d.expected_metric_uris.length > 1 && !d.metrics_correct`. Available variables:
-  `d` (the focused run's `detail`), `it` (its item), `i` (the row, `i.per[label]` for any
-  run), `q` (question), `kind`.
-- **The conversation**, when the item ran the agentic multi-turn path — every turn in
+  `d` (the checks of the run being read, its `detail`), `it` (that run's result), `i` (the row, `i.per[label]` for any
+  run), `q` (question), `kind` (category id).
+- **The conversation**, when the test case ran the agentic multi-turn path — every turn in
   order, with the simulated user marked apart from a real question, so you can see
   whether the agent got there or was handed the answer.
-- **A per-item drawer** — checks as pass/fail chips, expected vs actual side by side,
+- **A per-test-case drawer** — checks as pass/fail chips, expected vs actual side by side,
   full reasoning, conversation/response ids.
 - **A latency timeline** from `detail.latency_breakdown`, in execution order, one bar per
   step. Clicking a step expands its full record, joined by `index`: the paragraph a
   reasoning step was summarised from, or a tool call's arguments and result from
   `detail.tool_calls`.
+- **A print stylesheet** that drops the filters and the drawer, so "Save as PDF" gives a clean page.
+
+A report that merges several files names each run after its file, and a file named for its date (`2026-09-28.json`)
+reads as "Sep 28". The page assumes the runs are one model followed over time; `gd-eval run --html` over several
+models puts different models in one report, which it still lays out as run 1 against run 2.
 
 `--redact` additionally drops `transcript` and `tool_calls`: the exchange shows that the
 simulated user is primed with the expected output, and a tool result carries

packages/gooddata-eval/src/gooddata_eval/cli/agentic_runner.py

@@ -28,6 +28,7 @@
 from gooddata_eval.core.evaluators.visualization import requires_execution_of
 from gooddata_eval.core.models import AgenticEvalOutcome, CreatedVisualization, DatasetItem
 from gooddata_eval.core.runner import EvalReport, ItemReport
+from gooddata_eval.core.timing import TraceRecorder
 
 
 class _LfKw(TypedDict, total=False):
@@ -375,6 +376,18 @@ def _apply_failed_runs(item_report: ItemReport, source: Any) -> None:
         item_report.failed_runs = list(failed_runs)
 
 
+def _apply_run_latencies(item_report: ItemReport, source: Any) -> None:
+    """Copy the per-run wall-clock times off an outcome or a failure, if the kind timed them.
+
+    Read from both like ``_apply_failed_runs``: a failing item reaches the report through the
+    exception. A kind that records none leaves the empty default, so the item adds no sample
+    to the latency distribution rather than a zero.
+    """
+    run_latencies = getattr(source, "run_latencies_s", None)
+    if run_latencies:
+        item_report.run_latencies_s = list(run_latencies)
+
+
 def _apply_timings(item_report: ItemReport, timings: Any) -> None:
     """Copy an outcome's phase breakdown onto the item report, if the kind recorded one.
 
@@ -447,6 +460,10 @@ def _process_item(index: int, item: DatasetItem) -> ItemReport:
         # None, not False, for the kinds _dispatch_agentic passes no gate to: ItemReport.passed
         # then falls back to pass_at_k and gate_passed keeps meaning "a gate ran".
         gated = item.test_kind not in UNGATED_AGENTIC_TEST_KINDS
+        # Started in this worker, where the item runs, so a concurrent item cannot write into it.
+        # Every kind reaches the chat client the same way, which is why the per-trace latencies are
+        # collected here once rather than handed out by each evaluator.
+        traces = TraceRecorder()
         t0 = time.perf_counter()
         try:
             outcome = _dispatch_agentic(
@@ -482,6 +499,7 @@ def _process_item(index: int, item: DatasetItem) -> ItemReport:
             _apply_timings(item_report, getattr(outcome, "timings", None))
             _apply_run_counts(item_report, outcome)
             _apply_failed_runs(item_report, outcome)
+            _apply_run_latencies(item_report, outcome)
         except AssertionError as exc:
             item_report.gate_passed = False if gated else None
             item_report.runs = k
@@ -492,6 +510,7 @@ def _process_item(index: int, item: DatasetItem) -> ItemReport:
             _apply_timings(item_report, getattr(exc, "timings", None))
             _apply_run_counts(item_report, exc)
             _apply_failed_runs(item_report, exc)
+            _apply_run_latencies(item_report, exc)
             # Read off the counts, not off the gate: pass^K fails items where runs did pass,
             # and reporting those as pass_at_k False would contradict the Langfuse score of
             # the same name. Kinds that report no count read as 0, i.e. a clean failure.
@@ -512,6 +531,7 @@ def _process_item(index: int, item: DatasetItem) -> ItemReport:
             _apply_timings(item_report, getattr(exc, "timings", None))
             _apply_run_counts(item_report, exc)
             _apply_failed_runs(item_report, exc)
+            _apply_run_latencies(item_report, exc)
             # Every run ungraded, by definition of this error -- said explicitly rather than
             # left to _apply_run_counts, whose source is detail["unscored_runs"] and which a
             # kind that attaches no diagnostics would leave at 0.
@@ -525,8 +545,15 @@ def _process_item(index: int, item: DatasetItem) -> ItemReport:
             # judge broke should not also report the agent as costing 0s. Kinds that
             # attach no timings to the exception keep their 0.0 defaults.
             _apply_timings(item_report, getattr(exc, "timings", None))
+            # Phase timings stay, but the per-turn response times do not: this item reports no
+            # runs (and so no run latencies), and keeping its turns would put them in the
+            # response population with no attempt beside them. An item that completed 2 of 3
+            # runs and then raised would otherwise show up in the per-response table and not in
+            # the per-conversation one.
+            traces.discard()
         finally:
             item_report.latency_s = time.perf_counter() - t0
+            item_report.trace_latencies_s = traces.stop()
 
         if on_item_done is not None:
             try:

packages/gooddata-eval/src/gooddata_eval/core/agentic/alert_skill.py

@@ -6,6 +6,7 @@
 import json
 import os
 import re
+import time
 from dataclasses import dataclass, field
 from typing import Any
 
@@ -491,6 +492,8 @@ class AlertRunResult:
     # also report operator/threshold/metric/recipients as False.
     exit_reason: LoopExit = LoopExit.BUDGET_EXHAUSTED
     turns_used: int = 0
+    # This run's own wall time, from just before its chat call to the end of its scoring.
+    run_latency_s: float = 0.0
 
 
 @dataclass
@@ -699,6 +702,7 @@ def _run_once(conv_id: str) -> AlertRunResult:
             # that simply runs out of range() is correctly labelled without a trailing else.
             exit_reason = LoopExit.BUDGET_EXHAUSTED
             turns_used = 0
+            run_started = time.monotonic()
             for _iteration in range(max_iterations):
                 turns_used = _iteration + 1
                 try:
@@ -795,6 +799,7 @@ def _run_once(conv_id: str) -> AlertRunResult:
                 reasoning_step_events=all_reasoning_step_events,
                 exit_reason=exit_reason,
                 turns_used=turns_used,
+                run_latency_s=time.monotonic() - run_started,
             )
         finally:
             if alert_id_to_delete:
@@ -1011,6 +1016,7 @@ def _write_scores(ctx: RunTraceContext) -> None:
         exc.runs_passed = runs_passed
         exc.runs_effective = runs_effective
         exc.failed_runs = failed_runs
+        exc.run_latencies_s = [r.run_latency_s for r in summary.run_results]
         raise exc
     return AgenticEvalOutcome(
         runs_passed=runs_passed,
@@ -1022,4 +1028,5 @@ def _write_scores(ctx: RunTraceContext) -> None:
         # Also on the success path: pass@K clears the gate with one passing run, so a
         # 1/3 item reports success while two of its runs failed for reasons worth reading.
         failed_runs=failed_runs,
+        run_latencies_s=[r.run_latency_s for r in summary.run_results],
     )