Vis Node statistics with evalstats

[ianarawjo]

What it does

When ChainForge runs locally with pip install chainforge[stats] (evalstats 0.3.2 or newer), a Stats switch in the Vis Node's header shows evalstats results under the plot:

• Executive summary: rank band, mean with 95% CI, and a verdict from evalstats' rank bands (Highest / Tied for highest / Significantly lower) for each group. The wording is direction-neutral, since a higher score isn't always better.
• Pairwise differences: collapsible, with difference, CI and p-value. Pairs evalstats marks significant are bold.
• Methods and notes from evalstats: collapsed. Built from methods(): design, score type, CI methods, simultaneous CIs, the p-value test and correction, the Friedman test, the rank-band rule, resampling, and the evalstats version.
• Error bars on the plot: on true/false accuracy bars, grouped bars, box plots (which also show their means), and numeric bars.

It always uses evalstats' paired design. Items without a score for every group and run are dropped with evalstats.complete_items() and listed, with a prompt to rerun those queries. Below evalstats.MIN_ITEMS (15), no statistics are shown.

Changes

• chainforge/stats.py: builds the evalstats input, then relies on evalstats' public API:

  • complete_items() drops incomplete items;
  • to_dict() supplies rank bands, verdicts, significance and p-values;
  • methods() supplies the Methods list, and result.notes the notes;
  • the typed errors (InsufficientItemsError, MissingCellsError, TooFewGroupsError, AmbiguousLabelsError) become messages.

  No private evalstats helpers are imported.

• flask_app.py: /app/compareEvalStats and /app/checkEvalStatsAvailable, gated on the extra like RAG, plus window.__EVALSTATS_AVAILABLE.

• setup.py / CI / README: the [stats] extra (evalstats>=0.3.2).

• backend/evalStats.ts: pairs results by input (vars, metavars, LLM). It leaves out values that only describe one group, such as an upstream LLM's answer text.

• VisStatsPanel.tsx: the panel.

• VisNode.tsx:

  • numeric bar charts show each group's mean with n on hover, with or without Stats (they used to sum scores, so a per-response metric like response length read as a total);
  • stacked true/false charts become grouped bars;
  • categorical scores become 100%-stacked bars (the share of each group's responses in each category, with count and n on hover), replacing counts titled "Number of 'true' values"; LLM × variable plots use a two-level axis;
  • the graph type menu is disabled for true/false and categorical scores, which are always bars;
  • the graph type menu and Stats switch move to the header, after the AI plot buttons, and are hidden while an AI plot shows; the graph type is saved with the node;
  • the toolbar is compact and no longer sets the node's width;
  • the dark-mode grid is dimmer and the legend matches the axis labels;
  • plots now redraw on resize in development. StrictMode reran the effect cleanup that disconnected the resize observer, without rerunning the ref that created it.

Testing

• Python: 37 tests in tests/test_stats.py, run against evalstats 0.3.2.
  • They cover completeness and the 15-item floor, groups with no results, verdicts and bands, and the methods() lines.
  • They cover model × prompt names containing " / " and names that collide.
  • They check that each typed error becomes a message, and that the paired design is always used.
• Frontend: 11 jest tests for pairing, and the full frontend suite (1011 tests); tsc passes.
• Battle test, statistics: 78 generated datasets went through compare_eval_results, a few also through the Flask route.
  • Data covered: true/false, 0/1, continuous, Likert, unbounded, huge and tiny scales, constant and identical groups, junk scores (strings, None, NaN, inf), 2–10 groups, 15–1000 items, 1–5 runs, model × prompt grids, random and structured missing results, unicode, 500-character, " / ", numeric-looking and empty names, and duplicates.
  • Every result was checked for strict JSON, means inside CIs, non-decreasing bands, verdicts matching bands, pair counts and indices, significant agreeing with the CIs, and each difference equalling the gap between means. None failed.
  • 30,000 rows take about 1 s, and concurrent requests keep their own notes.
• Battle test, pairing: 16 ChainForge-shaped scenarios, including chained flows, per-response ids, tabular metavariables, prompt variants, image variables, missing variables, and 60,000 rows (about 140 ms).
  • This found chained results failing to pair when upstream answers repeat across LLMs, and questions that differed per LLM being merged. Both are fixed.
• In the dev app, with evalstats 0.3.2: HumanEval and sycophancy examples, plus eight edge-case flows. The edge cases were all-true scores, negative numbers, dict-valued scores (both keys), categorical scores, three runs with errors, only 10 items, eight LLMs with long, unicode and " / " names, and LLM × variable with one combination never run. Checked in each:
  • verdicts, bold pairs, the Methods list, notes and messages;
  • error bars aligned on each plot type;
  • the graph type surviving a reload;
  • resizing the node narrower.

Remaining

• The response inspector's plot hasn't been checked in the browser. It has no header, so its graph type menu stays in its toolbar and it has no Stats switch.

🤖 Generated with Claude Code