Submit Kilo Auto Efficient production router

[iscekic]

Submission

• Submit Kilo Auto Efficient through the production kilo-auto/efficient gateway endpoint.
• Include 8,400 full predictions with native generated results and 420 robustness predictions.
• Register the eleven concrete model IDs and their public catalog prices, captured on 2026-10-07.
• Add a standalone reproduction driver and the concrete Kilo provider in ModelInference.
• No RouterArena training or tuning was performed for this submission.
• Prompts, temperature, token limits, seeds, and production reasoning settings were not overridden.
• Existing cached model answers were not substituted.

Provider failures and retry policy

• The first pass recorded 8,252 successful generations, 95 explicit provider timeouts, and 53 empty answers.
• Explicit provider errors received at most two additional captured attempts per query.
• Retries recovered 56 timeouts. The final data retains 39 timeouts and all 53 empty answers as failures.
• Empty answers without an explicit provider error did not qualify for retries.
• All 8,252 original successful answers and their token usages remain unchanged, including incorrect answers.
• All 8,400 model selections remain unchanged. Correctness scores were not consulted during retry selection.
• The append-only audit journal preserves 8,547 captured full attempts. Early transport interruptions also required resuming uncaptured work.
• The submitted full JSON has no precomputed accuracy or cost fields, so the official workflow grades independently.

Verification

• Both official prediction validators passed, including the full generated-result check.
• The production gateway, streaming driver, and concrete SDK provider passed real API smoke runs.
• The official cost calculator matched the provider's generation market cost for the first benchmark response.
• Full local grading completed on a dedicated Google Cloud VM without an attached service account.
• Parallel grading hit filelock fork errors. All unchanged answers were then graded sequentially, without new model requests.
• Ruff checks and formatting checks passed for all three changed Python files.

Local metric	Result
Accuracy, including all 92 generation failures	0.7600
Catalog-priced successful generation cost	$27.197310
RouterArena score using that generation cost	0.6980
Robustness score	0.690476

Cost and latency limitations

• The cost above is not all-in router cost or total benchmark spend.
• The stock evaluator excludes classifier overhead, discarded retry attempts, and failed inference with missing usage.
• It also excludes robustness-call costs, cache-price differences, and separate request or search fees.
• The funded account uses BYOK and can report usage.cost=0; evaluation uses public catalog prices, not zero account charges.
• Missing authoritative token usage remains null. No tokens or costs were fabricated.
• Completion duration is not routing-only latency. This submission makes no routing-only latency claim.
• Please review these limitations before approving a cost ranking.

[iscekic]

/evaluate

[github-actions]

Router Evaluation Results

Router: kilo-auto-efficient
Dataset Split: full

RouterArena Metrics

Metric	Value
RouterArena Score	0.6980
Accuracy	76.00%
Total Cost	$27.197310
Avg Cost per Query	$0.003238
Avg Cost per 1K Queries	$3.2378
Number of Queries	8400
Abnormal Entries	92
Robustness Score	0.6905

⚠️ 92 of 8400 queries (1.1%) had no valid generation (inference failed / empty answer) and were scored as incorrect (0). These queries still count toward the denominator, so accuracy and cost reflect the full query set. Please regenerate predictions for these queries and resubmit for a complete evaluation.

---

Evaluation completed by RouterArena automated workflow

[iscekic]

Retried all 92 failed generations once, as explicitly requested. Recovered 43; 49 remain (21 timeouts, 28 empty answers). All 8,308 previously successful rows remain identical. Prompts and router settings are unchanged; no accuracy-based retries occurred. Both submission validators passed. Sequential local grading: accuracy 0.7621 across all 8,400 queries, generation cost $28.909222, RouterArena score 0.6979, robustness 0.6905. Remaining failures count as zero accuracy. Reported cost excludes discarded attempts and classifier overhead, as previously disclosed.

[iscekic]

/evaluate

[iscekic]

Closing this submission at the submitter’s request. A new PR will open only after every generation succeeds. Provider-error retries continue without changing successful results, prompts, or router settings.

[github-actions]

Router Evaluation Results

Router: kilo-auto-efficient
Dataset Split: full

RouterArena Metrics

Metric	Value
RouterArena Score	0.6979
Accuracy	76.21%
Total Cost	$28.909222
Avg Cost per Query	$0.003442
Avg Cost per 1K Queries	$3.4416
Number of Queries	8400
Abnormal Entries	49
Robustness Score	0.6905

⚠️ 49 of 8400 queries (0.6%) had no valid generation (inference failed / empty answer) and were scored as incorrect (0). These queries still count toward the denominator, so accuracy and cost reflect the full query set. Please regenerate predictions for these queries and resubmit for a complete evaluation.

---

Evaluation completed by RouterArena automated workflow