obol-stack-side model ranking: keep, deprecate, or remove? (audit findings)

[bussyjd]

Context

PR #379 (obol model prefer + provider smokes) was closed in favour of doing model control on the calling agent side rather than the LiteLLM side. The premise: each agent (Hermes, OpenClaw, future) has its own ordering and fallback mechanism, so it's redundant for obol-stack to also pick.

This issue documents an audit of whether that premise actually holds today, and what the cleanup path is.

Code under review

obol-stack maintains a model-ranking layer that picks a primary + fallbacks before the agents see the model list:

• Core: internal/model/rank.go::Rank — capability-aware ranker (cloud > local; cloud quality table; local by parameter count). ~221 lines + rank_test.go.
• OpenClaw wrapper: internal/openclaw/openclaw.go::rankModels — wraps Rank, adds openai/ prefix.
• OpenClaw hand-off: internal/openclaw/openclaw.go::patchModelHierarchy — writes agents.defaults.model.{primary,fallbacks} into openclaw-config ConfigMap.
• Hermes wrapper: internal/hermes/hermes.go::rankModels — wraps Rank (no prefix).
• Hermes hand-off: internal/hermes/hermes.go::generateValues bakes primary into the API_SERVER_MODEL_NAME env var via helm values; generateConfig writes model.default = primary into the agent config.

Audit findings

Spawned read-only audits of both runtime repos.

OpenClaw (/Users/bussyjd/Development/R&D/openclaw)

• Treats agents.defaults.model.primary as a hard pin, not a hint — src/agents/model-selection.ts:392-422, src/agents/agent-scope.ts:186-195.
• Does NOT query LiteLLM /v1/models at runtime. Catalog loads once at startup from a bundled registry (src/agents/model-catalog.ts:80+), used only for display/validation.
• Has fallback orchestration on error (src/agents/model-fallback.ts:78+) — walks the explicit fallbacks list in order. No capability ranking or auto-discovery.
• If primary is empty/missing, falls back to hardcoded defaults in src/agents/defaults.ts — silently downgrades every agent.

Hermes (/Users/bussyjd/Development/R&D/hermes-agent, upstream nousresearch/hermes-agent)

• Treats API_SERVER_MODEL_NAME and config model.default as hard pins. Read at cli.py:1797, gateway/platforms/api_server.py:541.
• Does NOT query LiteLLM /v1/models at runtime. No code path discovers models dynamically from the LiteLLM sidecar. Pulls model lists from OpenRouter/Anthropic/provider catalogs only.
• Has fallback chain via run_agent.py:6089::_try_activate_fallback — switches on error using a user-configured fallback_model list. No capability ranking, no auto-discovery.
• If API_SERVER_MODEL_NAME is empty, the API server advertises a placeholder name on /v1/models; the CLI agent itself requires a model and fails to initialize.

Verdict

The premise that "each agent has its own ordering" doesn't fully hold. Both runtimes have reactive fallback (try next on error), not proactive capability ranking. Both treat the obol-stack-supplied primary as a hard pin and would silently downgrade or fail without it.

So Rank is currently load-bearing. Removing it today (without coordinated upstream changes) breaks both runtimes.

Options

A. Keep Rank, document the contract — recommended now

Add a header comment to internal/model/rank.go and to both rankModels wrappers stating: "obol-stack picks a sensible primary because neither Hermes nor OpenClaw auto-discover from the LiteLLM model_list at runtime. Until they do, this layer is load-bearing." No code change. Eliminates "is this dead?" confusion for future readers. ~30 min.

B. Drop the wrappers, inline model.Rank(...) — minor cleanup

Saves ~30 lines of indirection. Doesn't change runtime behavior. Doesn't address the design question. ~1h. Low value if we're going to do A.

C. Full removal — blocked on upstream

Drop Rank, both wrappers, patchModelHierarchy, and the primary env-var/config plumbing. Push only the LiteLLM model_list to agents. Requires both upstream repos to add a startup-time /v1/models query and pick their own primary. Needs design alignment with the OpenClaw and Hermes maintainers before any obol-stack change.

Recommendation

A now. Open follow-up RFCs / coordinated PRs in openclaw (and an upstream issue/PR in nousresearch/hermes-agent) to add /v1/models auto-discovery at startup. Once both runtimes can stand on their own, revisit C.

Closing PR #379 was correct

The closure rationale ("model control on the calling agent side") is the right destination. This issue captures how to get there — the cleanup is gated on upstream changes, not on obol-stack alone.

cc @OisinKyne

[bussyjd]

Update — adding option D (LiteLLM-side smart routing) as the recommended path

The user noted that Rank only delivers value if either (1) the agent itself sees the ranking and picks contextually (e.g. Opus for code, Haiku for subagents), or (2) LiteLLM exposes a smart-routing endpoint that resolves a virtual model name to the best available concrete model. Option (2) is meaningfully better and is already a first-class LiteLLM feature.

Option D — LiteLLM-side smart routing via model alias groups

LiteLLM supports model alias groups: multiple model_list entries can share the same model_name, and LiteLLM picks one per request based on a routing strategy (priority order, latency, cost, simple-shuffle, least-busy, etc.) with explicit fallback on error.

So instead of telling agents "your primary is claude-opus-4-7", obol-stack publishes virtual names like obol-best, obol-cheap, obol-coding, obol-fast. Each name maps to all configured concrete models, ordered by Rank() for that intent. LiteLLM resolves at request time.

# litellm-config (generated by obol-stack from Rank() output)
model_list:
  - model_name: obol-best
    litellm_params: {model: anthropic/claude-opus-4-7, api_key: …}
    model_info: {priority: 1}
  - model_name: obol-best
    litellm_params: {model: anthropic/claude-sonnet-4-6, api_key: …}
    model_info: {priority: 2}
  - model_name: obol-best
    litellm_params: {model: openai/gpt-5, api_key: …}
    model_info: {priority: 3}
  - model_name: obol-best
    litellm_params: {model: ollama/qwen3.5:9b, api_base: …}
    model_info: {priority: 4}
  # …same shape for obol-cheap, obol-coding, obol-fast

Rank() becomes the source of the priority ordering — same algorithm, different output target.

Why this dominates A/B/C

Property	A (document & keep)	B (drop wrappers)	C (full removal, blocked)	D (alias routing)
Smart logic lives in obol-stack	✅	✅	❌ (moves to agents)	✅
Agent code change required	none	none	Hermes + OpenClaw + future	one-line config: point at obol-best
Failover when provider is down	reactive in agent	reactive in agent	reactive in agent	built into LiteLLM
Per-task intent (coding/cheap/fast)	not exposed	not exposed	agent reimplements	free via additional aliases
Aligns with "obol-stack manages via LiteLLM"	partial	partial	yes	yes — and centralised
Adding a new agent runtime	linear cost (re-pin)	linear cost	linear cost (re-implement picker)	zero cost

Concrete obol-stack changes

• Keep internal/model/rank.go::Rank — now used to generate priority ordering inside alias groups.
• Drop internal/openclaw/openclaw.go::patchModelHierarchy (no more pre-pick into OpenClaw config) and the API_SERVER_MODEL_NAME = <concrete primary> baking in internal/hermes/hermes.go::generateValues.
• Drop the rankModels wrappers in internal/openclaw/openclaw.go and internal/hermes/hermes.go (no longer needed at this layer).
• Add alias-group emission to internal/model/model.go::buildModelEntries / ConfigureLiteLLM — emit obol-best (and friends) for every concrete entry, with model_info.priority set from Rank().

Concrete agent-side changes

• OpenClaw: change cluster default in obol-stack ConfigMap write so agents.defaults.model.primary becomes "obol-best" instead of the concrete pick. OpenClaw still hard-pins, but the "pin" is now a virtual name that LiteLLM resolves intelligently. No OpenClaw code change.
• Hermes: change API_SERVER_MODEL_NAME baking so the env var is "obol-best" instead of the concrete pick. No Hermes code change.

This is the smallest ask of both runtimes — they keep treating their input as a hard pin, they just pin to a virtual name. None of the upstream prerequisites option C had.

Follow-up dimension: intent-based routing

Once aliases are in place, the agent can opt into specific intents:

• obol-best — strongest available
• obol-coding — code-tuned ordering (Opus > Sonnet > qwen-coder > deepseek-coder)
• obol-cheap — cheapest first (Haiku > local 3b > 1b)
• obol-fast — local-first for low latency
• obol-tools — tool-use-tuned

Hermes saying model: obol-coding for a code subagent is a far smaller behavioural ask than Hermes evaluating the catalog and picking. The smart logic still lives in obol-stack (Rank), expressed via LiteLLM config, and the agent only needs to pick intent — which is much closer to context it already has.

Updated recommendation

Option D now. It's the path that lets Rank survive AND deletes the pre-pick plumbing AND avoids any upstream change AND gives intent-based routing for free.

Will follow up with a separate implementation issue scoped to: alias-group emission in ConfigureLiteLLM, drop of patchModelHierarchy / primary baking, and the one-line config flips on the OpenClaw + Hermes hand-off paths.