Skip to content

fix: models.dev last-segment match + official-vendor disambiguation - #1047

Merged
vastsa merged 7 commits into
vastsa:mainfrom
aurorax-neo:fix/models-dev-last-segment-match
Sep 26, 2026
Merged

vastsa merged 7 commits into
vastsa:mainfrom
aurorax-neo:fix/models-dev-last-segment-match

Conversation

@aurorax-neo

@aurorax-neo aurorax-neo commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Fix models.dev enrichment for routed/gateway wire IDs (e.g. route/model-leaf): match by case-insensitive exact last / segment only.
  • When multiple catalog hits share that leaf, prefer a unique official/source provider (anthropic / openai / google* / xai/x-ai) whose provider key agrees with the model’s source vendor (so a gateway listing like google-vertex/xai/grok-* does not count as xAI).
  • If no unique official hit, enrich only when all hits share identical capabilities/thinking; otherwise leave unmatched.
  • UI/identity always keep the full wire ID; unmatched models expose all thinking levels and default to off.
  • Also harden plugin-mcp child teardown (process group + SIGKILL + stdio destroy) so desktop tests no longer hang.

Test plan

  • Shared 1030/1030 + typecheck
  • models.dev targeted 45/45; composer/thinking 42/42
  • Desktop full suite 2861 pass / 0 fail / 1 skip
  • Acceptance against a live OpenAI-compatible gateway catalog: official enrichments for common sonnet/gpt/grok leaves; custom agent-style aliases stay unmatched (no false collapse onto *-latest)

Update upstream tests that assumed release-stamp stripping, deployment
marker fallbacks, or majority/consensus borrowing so they match the
approved vastsa#1047 matcher and official/shared-capabilities disambiguation.
@aurorax-neo
aurorax-neo force-pushed the fix/models-dev-last-segment-match branch from ddbe127 to 9069108 Compare September 26, 2026 15:02
@vastsa

vastsa commented Sep 26, 2026

Copy link
Copy Markdown
Owner

Thanks — I reviewed commit 9069108 and ran the targeted suites (desktop 116/116; shared 22/22). The core last-segment matching and official/shared-capability disambiguation look implemented, and the PR CI is green.

Before merge, I found a few completeness/regression concerns:

  1. initialThinkingLevelForBinding() in packages/shared/src/thinking-levels.ts now returns "off" whenever no default is stored. This helper is also used for normal new-session materialization, not only unmatched models. The current UX spec says a reasoning binding with no stored default falls back to its highest enabled level. Please scope this behavior to unmatched models, or update the product contract and add a user-path regression test.

  2. In thinkingProviderForModel(), an unmatched model with an existing binding whose thinkingLevels is [] still gets an empty supportedThinkingLevels array; the new “all levels selectable, default off” behavior only occurs when there is no binding. Please clarify/fix the empty-binding case or adjust the PR summary.

  3. The model-list row now uses the wire ID, but Composer.tsx still derives the main model chip label from selectedModelInfo.displayName, so the full wire ID is not preserved in every UI surface.

  4. The matching/spec change is not reflected in docs/spec/03-runtime/13-model-catalog-and-selection.md and the related decision/spec docs; they still describe the older suffix/route behavior.

Could you address these before merge?

Keep the existing highest-enabled fallback for catalog-known bindings while applying the PR's off-by-default behavior only to unmatched models. Treat empty unknown-model bindings as generic seeds, and synchronize the model catalog and Composer specifications with the conservative matching contract.
# Conflicts:
#	docs/spec/08-meta/decisions-log.md
@vastsa

vastsa commented Sep 26, 2026

Copy link
Copy Markdown
Owner

Follow-up on my earlier review: I verified that the alias chip is intentional and the exact wire ID remains the model identity and list-row value. The empty thinkingLevels: [] case was a real gap for the PR's unmatched-model contract, and the global default-off change also affected known catalog models. I pushed a minimal landing fix on top of the contributor commits: known catalog matches retain the highest-enabled fallback; unmatched models, including empty generic bindings, start at off while exposing the full thinking ladder; explicit non-empty bindings remain authoritative. Specs and regression tests are synchronized. Relevant candidate checks pass: shared tests 23/23, desktop typecheck, and targeted model/Composer/MCP tests 135/135. The branch also includes the latest origin/main.

Keep the source-contract assertion aligned with the attachment history variable introduced on the latest main line. This removes a stale test failure without changing runtime behavior.
@vastsa

vastsa commented Sep 26, 2026

Copy link
Copy Markdown
Owner

The first post-fix CI run exposed one unrelated latest-main regression: the attachment-history source-contract test still expected the old local name after main changed it to . I synchronized that test in 328eab2 without changing runtime behavior. Local full desktop suite now passes 2971/2971 with the repository loader; the PR branch remains based on the latest origin/main.

@vastsa

vastsa commented Sep 26, 2026

Copy link
Copy Markdown
Owner

Correction to the previous note: the stale attachment-history source-contract test expected the old local name 'shouldInline' after latest main changed it to 'canInline'. I synchronized that test in 328eab2 without changing runtime behavior. Local full desktop suite now passes 2971/2971 with the repository loader, and the PR branch remains based on the latest origin/main.

Use explicit TypeScript extensions for the composer model module graph so the native desktop test runner can load the new model capability regression tests. Runtime behavior is unchanged.
@vastsa

vastsa commented Sep 26, 2026

Copy link
Copy Markdown
Owner

The second CI failure was the native Node test runner unable to resolve extensionless TypeScript imports inside composer model.ts, reached by the new regression test. I added explicit .ts extensions in 5067a90; this is test-loading hygiene only and leaves runtime behavior unchanged. The exact CI command now passes locally: 2971/2971 desktop tests.

@vastsa
vastsa merged commit 0140410 into vastsa:main Sep 26, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants