Skip to content

fix(models): sync bundled offline fallback data with models.dev (2026-09-08) - #225

Merged
ltmoerdani merged 1 commit into
mainfrom
chore/models-dev-data-sync
Sep 8, 2026
Merged

ltmoerdani merged 1 commit into
mainfrom
chore/models-dev-data-sync

Conversation

@ltmoerdani

Copy link
Copy Markdown
Owner

Summary

Full audit of the bundled offline fallback tables against the live models.dev registry (api.json, fetched 2026-09-08) found three drift classes. All of these tables only matter when the live models.dev fetch and the gateway /models endpoint are unavailable, but some wrong values were large enough to matter anyway (up to 6x cost under-reporting).

Every drifted value was traced with git log -S before changing it, to separate stale snapshots from deliberate deviations.

1. Context limits (src/models/modelTables.ts)

Model Before After Why
kimi-k2.7-code 256,000 262,144 Verified against models.dev on 2026-06-15 (issue #25); models.dev changed since
minimax-m3 512,000 1,000,000 2026-06-09 snapshot; Zen still lists 512K, Go was raised to 1M
qwen3.6-plus 262,144 1,000,000 Same pattern; Zen still lists 262K
qwen3.7-plus 262,144 1,000,000 Copy-paste from qwen3.6-plus in 1c26fd1 (2026-08-13), never verified

The key signal: models.dev now differentiates Go vs Zen per model, and the bundled Go values exactly matched the Zen ones. That is the same cross-provider contamination pattern fixed in docs/issues/02.

Kept as-is (deliberate, verified against the gateway): deepseek-v4-flash output cap 131,072 (#171) and Go minimax-m2.5 output 65,536.

2. Usage tracker pricing (src/usage/pricing.ts)

The static GO_MODEL_PRICING fallback had drifted badly from the registry:

Model Bundled Actual Impact
mimo-v2.5-pro $1.74 / $3.48 $0.435 / $0.87 4x over-report
deepseek-v4-pro $1.74 / $3.48 $0.66 / $1.98 ~2.6x over-report
mimo-v2-omni $0.14 / $0.28 $0.40 / $2.00 big under-report
deepseek-v4-flash $0.14 / $0.28 $0.22 / $0.66 under-report
minimax-m3 $0.60 / $2.40 $0.30 / $1.20 2x over-report
kimi-k3 missing (fallback $0.5/$2.0) $3.00 / $15.00 6x under-report

Table rewritten from the current Go snapshot: 15 new models added, minimax-m2.1 / minimax-m2 removed (no longer served), hy3-preview renamed to hy3 with the corrected price.

3. Go fallback catalog (src/provider/definitions.ts)

fallbackModels still listed the retired hy3-preview, omitted kimi-k2.7-code and everything added after June, and included deprecated models, contradicting the README's deprecated-filtering promise. Refreshed to the 27-model curated active set.

4. Stale tests

metadata.test.ts, registry.test.ts, thinking.test.ts still referenced hy3-preview (passing only because the routing catch-all treats both ids the same). Updated to hy3. The goUsageTracker and metadata unit-test locks that pinned the old pricing/context values were updated as part of the sync.

Verification

  • npm run lint: all 7 checks pass, 462/462 unit tests
  • models.dev snapshot diffed programmatically before/after (curl + python3)

Caveats

Doc: docs/issues/100-20260908-bundled-model-data-sync.md

…-09-08)

Three drift classes fixed in the offline-only tables:

1. Context limits — kimi-k2.7-code 256K -> 262,144; minimax-m3,
   qwen3.6-plus, qwen3.7-plus -> 1M. models.dev now differentiates
   Go vs Zen per model and the bundled Go values matched the Zen
   ones (issue #2 contamination pattern); qwen3.7-plus was a
   copy-paste, never verified. Deliberate deviations kept:
   deepseek-v4-flash 131,072 output cap (#171), Go minimax-m2.5
   65,536 output.
2. Usage tracker pricing — table rewritten from the current Go
   snapshot: mimo-v2.5-pro / deepseek-v4-pro were 2-4x over-reported,
   mimo-v2-omni / deepseek-v4-flash under-reported, kimi-k3 ($3/$15)
   fell back to a 6x-under-estimating generic price. 15 new models
   added, minimax-m2.1/m2 removed, hy3-preview -> hy3.
3. Go fallbackModels catalog refreshed to the 27-model curated
   active set (still listed hy3-preview + pre-June models).

Stale tests referencing hy3-preview updated; goUsageTracker and
metadata unit-test locks updated to the corrected values.

Doc: docs/issues/100-20260908-bundled-model-data-sync.md
@ltmoerdani
ltmoerdani merged commit efd2583 into main Sep 8, 2026
2 checks passed
@ltmoerdani
ltmoerdani deleted the chore/models-dev-data-sync branch September 23, 2026 14:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant