Repository navigation
Add TeamoRouter to the RouterArena leaderboard as a submission #171
Description
Activity
Thanks for the interest, @blammectrappora! RouterArena accepts both open-source and commercial routers. To add TeamoRouter, please open a PR (no issue needed) with:
router_inference/config/teamo-router.json— your router config.router_inference/predictions/teamo-router.json— predictions over the full 8,400-query dataset (each base row carrying the selected model, the generated answer, andtoken_usage).router_inference/predictions/teamo-router-robustness.json— the robustness split.- If you use any models not already priced, add them to
model_cost/model_cost.json(with the provider's official per-million-token list price) and register their slugs inuniversal_model_names.py.
Then comment
/evaluateon the PR and the bot will score it. You can use any recently-merged submission as a template — e.g. #181 (AsiaInfo) or #180 (BARouter). Ping me if you hit snags.
Add TeamoRouter to the RouterArena leaderboard as a submission
RouterArena is exactly the kind of neutral evaluation the router ecosystem needs — standard datasets, cost/latency/optimality metrics, and a clear leaderboard. Since the README states that both open-source and commercial routers are welcome, I'd like to propose submitting TeamoRouter as a candidate.
TeamoRouter is a multi-model routing / API gateway that does cost-aware and latency-aware routing across providers, and it serves models like DeepSeek V4 Flash / V4 Pro through OpenAI-compatible endpoints (with prompt caching to control cost). Having it measured on the RouterArena pipeline would give reproducible, comparable numbers that are genuinely useful to developers routing DeepSeek V4 and similar models.
Practical question: what's the submission flow — is there a benchmark harness or config format I should follow, or should this start as a discussion on the evaluation criteria? Happy to run the eval and submit the results if you point me at the harness.