Skip to content

Pull requests: OpenRouterTeam/benchmark-harness

Pull request creation is restricted
11 Open 42 Closed
11 Open 42 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Reviews
Filter by reviews
No reviews Review required Approved review Changes requested
Assignee
Filter by who’s assigned
Assigned to nobody Loading
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Pull requests list

feat: persist completed sample outcomes so interrupted runs resume instead of restarting
#48 opened Aug 21, 2026 by jamespsterling • Draft
4 of 5 tasks
@jamespsterling
21
deps: bump the dependencies group with 16 updates
#47 opened Aug 20, 2026 by dependabot Bot Loading…
1
feat(runner): accept caller trace headers for OpenRouter API requests
#45 opened Aug 20, 2026 by jamespsterling Loading…
3 tasks done
@jamespsterling
7
ECO-2484: feat(datasets): cache HF rows pages and task repo checkouts on disk
#44 opened Aug 19, 2026 by abhinav-pola Contributor Loading…
6 tasks done
25
fix(search): resolve server-tool generation ids
#32 opened Aug 14, 2026 by ayush-or Contributor Loading…
fix: recover unclassified Responses output items
#29 opened Aug 13, 2026 by bjoerndanz Contributor Loading…
5 tasks done
@bjoerndanz
3
feat: add agent_dx benchmark for OpenRouter agent-experience evals
#15 opened Aug 7, 2026 by devin-ai-integration Bot Loading…
5 tasks done
7
feat(benchmarks): chess — full games vs Stockfish, first-party original
#4 opened Aug 5, 2026 by LukasParke Loading…
2
feat(benchmarks): EvalManifestV1 — unified eval description schema + first-party manifests
#3 opened Aug 5, 2026 by LukasParke Loading…
2
feat(benchmarks): custom_eval — declarative user evals as run-config data
#2 opened Aug 5, 2026 by LukasParke Loading…
2
feat(harness): durable per-(sample, epoch) result store + fiber-scoped request context
#1 opened Aug 5, 2026 by LukasParke Loading…
3
ProTip! What’s not been updated in a month: updated:<2026-07-25.