Skip to content

feat: explain benchmark model choices - #441

Merged
mattrossman merged 19 commits into
mainfrom
mattrossman/ai-1285-explain-why-each-benchmark-experiment-uses-its-model-and
Nov 11, 2026
Merged

mattrossman merged 19 commits into
mainfrom
mattrossman/ai-1285-explain-why-each-benchmark-experiment-uses-its-model-and

Conversation

@mattrossman

@mattrossman mattrossman commented Oct 26, 2026 •

Copy link
Copy Markdown
Collaborator

Adds a "Why these experiments" section to the results page to shed light on why we picked these model / reasoning effort loadouts, since that was questioned here.

Also adds upload dates for the published benchmark experiments to communicate freshness of these explanations (which will typically update at the same time as experiment pins).

Rationales are maintained alongside the pinned Braintrust experiment IDs so they stay in sync with results shown on the site.

Rationales Timestamps dropdown
CleanShot 2026-11-09 at 5 04 15 PM@2x CleanShot 2026-11-09 at 4 29 07 PM@2x

Kimi K3 ranks #30 on OpenCode's usage data, so its text only says it's a deliberate open-weight pick. We may want to consider switching the benchmark a more popular OpenCode model as a follow-up.

Review in the preview build: https://evals-git-mattrossman-ai-1285-explain-why-each-10afd2-supabase.vercel.app/

Closes AI-1285

@vercel

vercel Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
evals Ready Ready Preview Nov 11, 2026 6:39pm UTC

Request Review

@mattrossman
mattrossman changed the base branch from mattrossman/ai-1286-align-the-claude-code-benchmark-presets-with-claude-codes to main November 9, 2026 16:55
@mattrossman
mattrossman marked this pull request as ready for review November 9, 2026 21:47
@mattrossman
mattrossman requested a review from a team as a code owner November 9, 2026 21:47
@mattrossman mattrossman changed the title feat(evals): explain each benchmark experiment's model and effort on the results page feat(evals): explain benchmark model choices Nov 9, 2026
@mattrossman mattrossman changed the title feat(evals): explain benchmark model choices feat: explain benchmark model choices Nov 9, 2026

@raulb raulb left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mattrossman Code LGTM, however I would love to hear @SaxonF's thoughts if he has cycles.

For instance, I'm not sure I love having to click on the arrow to expand the Experiment upload dates, considering there are only a handful of them. Approving to unblock, but I would like to include @SaxonF in the loop.

@mattrossman

Copy link
Copy Markdown
Collaborator Author

@raulb I just changed it to show it flattened w/o the collapsible label, and if we want to make further UI follow ups that's fine!

CleanShot 2026-11-11 at 1 46 21 PM@2x

Note I also explored having this info embedded inside of the "Why these experiments" table but it looked noisy to me, hence the smaller separate table below it. It also isn't relevant just the justifications, but also the results tables above.

@mattrossman
mattrossman merged commit a3c2468 into main Nov 11, 2026
10 checks passed

This branch was successfully deployed

1 active deployment
Preview — 2d6b8846 Deployed Nov 11, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants