Repository navigation
Commit 1daa2c4
feat(gooddata-eval): add the agentic anomaly-detection evaluator (#1801)
* feat(gooddata-eval): add the agentic anomaly-detection evaluator
Scores the anomaly-detection skill end to end, with the granularity map aligned to what
the service actually accepts and whole-conversation latency and cost rather than the
goal turn's alone.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(gooddata-eval): let --gate decide an anomaly item, not pass@K alone
`run_agentic_anomaly_detection` already computed `pass_power_k`; the evaluator
never read it, asking `if not summary.pass_at_k` for the verdict, so `--gate
power` on a flaky item reported a pass on the strength of one good run out of K.
Wired the same way as the other multi-run kinds: the evaluator takes `gate`, the
dispatch passes it, `gate_passed(...)` decides, the gate's note goes into the
assertion message -- which matters because the body describes the BEST run, and
under pass^K that can be one that passed -- and pass@K/pass^K/gate_passed reach
Langfuse alongside the run metadata stamp.
The default is unchanged: without --gate this is pass@K exactly as before, pinned
by a test beside the power case. The power test fails against the previous code.
Found by CodeRabbit on #1798. Forecasting is fixed there; what-if, which is
already on master, in #1831.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>1 parent c3c5284 commit 1daa2c4
5 files changed
Lines changed: 1188 additions & 0 deletions
File tree
- packages/gooddata-eval
- src/gooddata_eval
- cli
- core/agentic
- tests
Lines changed: 14 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
12 | 12 | | |
13 | 13 | | |
14 | 14 | | |
| 15 | + | |
15 | 16 | | |
16 | 17 | | |
17 | 18 | | |
| |||
48 | 49 | | |
49 | 50 | | |
50 | 51 | | |
| 52 | + | |
51 | 53 | | |
52 | 54 | | |
53 | 55 | | |
| |||
278 | 280 | | |
279 | 281 | | |
280 | 282 | | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
281 | 295 | | |
282 | 296 | | |
283 | 297 | | |
| |||
0 commit comments