Skip to content

Seed model offerings in declared priority order - #546

Closed
pratikbuilds wants to merge 7 commits into
corbitsdev:mainfrom
pratikbuilds:cl-541-anthropic-default-model
Closed

pratikbuilds wants to merge 7 commits into
corbitsdev:mainfrom
pratikbuilds:cl-541-anthropic-default-model

Conversation

@pratikbuilds

@pratikbuilds pratikbuilds commented Sep 1, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • assign seeded model offerings distinct provider-major, model-major priorities
  • support ANTHROPIC_MODEL so the configured curated Anthropic model is preferred by both runtime resolution and Settings
  • reconcile existing offering priorities on reseed, including the fast environment-credential plant
  • update onboarding test fakes for the idempotent 409 -> GET/PATCH flow

Configuration

Set ANTHROPIC_API_KEY and optionally set ANTHROPIC_MODEL=claude-opus-5. The model defaults to claude-sonnet-5; setting ANTHROPIC_MODEL without the key fails hub configuration. Boot seeding is authoritative, so restarting or reseeding restores the configured model after a manual catalog reorder.

Verification

  • bun run check (database-gated suites skip locally without DATABASE_URL)
  • 162 focused seeding, configuration, and onboarding tests
  • 26 OAuth route tests
  • Greybeard and Critique reviews

Fixes #541

@TheGreatAxios TheGreatAxios left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review · Request changes

Catalog seeding assigns each declared provider/model pair a deterministic priority.

Findings

  • packages/hub-client/src/seed.ts:1341 — Could the seed reconcile an existing offering's priority rather than skip the 409? Tenants seeded before this change retain equal priorities on rerun, so the intended Sonnet-first order never reaches those catalogs and resolution continues to use the tie-breaker.

@TheGreatAxios TheGreatAxios left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Critic · Comment

Catalog seeding flattens declared provider and model order into offering priorities.

Findings

  • packages/hub-client/src/seed.ts:1341 — Existing offerings are skipped after a 409, so a catalog seeded before this change keeps the former equal priorities even after rerunning the seed command. A re-run regression should establish that upgrade behavior.

@TheGreatAxios TheGreatAxios left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Priority-ordering logic is sound and the offering-conflict-reconciliation addition is a real improvement, but this PR needs a rebase onto the packages/seeding move and currently breaks CI outside the touched package.

Must fix

  • Rebase needed: yes (2 conflicts) — seed code now lives in packages/seeding. packages/hub-client no longer exists on main (it was moved wholesale to packages/seeding/src/seed.ts and packages/seeding/test/seed.test.ts); git merge-tree reports both touched files as modify/delete against main. Neither the offset-priority fix nor the 409-reconciliation logic is present in packages/seeding/src/seed.ts today (still Math.max(0, indexOf(provider)) at seed.ts:1577-1579, and ensureCatalogOffering's 409 branch still just logs "already exists (skipped)"), so this needs to be reapplied there, not merged as-is.
  • CI is red on this PR independent of the rebase: build-test fails on packages/onboarding/test/complete-credential.test.ts → "a second credential save for the same account is idempotent...". That suite's fake catalog-offerings handler returns { status: 409, data: {} } on the second offering POST with no GET/PATCH-offerings route implemented. The new ensureCatalogOffering conflict path now unconditionally lists offerings and PATCHes on a priority mismatch, so it hits a route the fake doesn't provide and throws (or fails to parse {} against ModelOfferingResponse). This is a real regression the PR introduces, not flakiness — worth fixing in packages/onboarding's test fakes as part of this change (or as a directly-following commit) once ported to packages/seeding.

Consider

  • The 409-reconciliation path pages through all catalog offerings every time a conflict is hit, looking for one modelId+providerId match; for a tenant with a large catalog this is an O(n) full scan per conflicting offering during every seed run. Not wrong, but worth a comment noting the assumption (catalogs stay small) if that's intentional.
  • ensureCatalogOffering's new behavior silently promotes every re-run into a potential PATCH when only the freshly-added offerings actually need reordering; on a bench where an operator has since hand-tuned an offering's priority, a re-seed will now overwrite that customization back to the computed value with no distinction between "we planted this" and "operator changed this." Might be fine per the seed contract, but flag it explicitly if seeding is meant to be idempotent-but-not-authoritative over operator edits.

Verified

  • Priority math: offeringPriorityOffset sums prior providers' models.length in CATALOG_SEEDS iteration order (a literal object, so order is stable), then adds modelIndex from seededModels.entries(). This produces a strictly increasing, globally unique priority per (provider, model-declaration-order) pair — no ties possible across providers or within a provider, and the first declared model for a provider (e.g. Anthropic's Sonnet) always gets the lowest priority of that provider's block. Confirmed via git show diff read directly; test "fresh run gives the declared Anthropic default the lowest distinct priority" asserts both uniqueness and the min-priority claim.
  • The 409→list→conditional-PATCH flow parses both the paginated list response and the PATCH response through parseAs/arktype schemas (ModelOfferingResponse, paginatedSchema) rather than casting — consistent with the repo's trust-boundary rule.
  • gh pr checks 546: lint, typecheck, structural, e2e, isolation, db-suites all pass; build-test fails (see above).
  • Scope matches the stated intent (#541): no new env var for model selection, ANTHROPIC_API_KEY stays credential-only.

@pratikbuilds
pratikbuilds force-pushed the cl-541-anthropic-default-model branch from 8711245 to 56f5527 Compare September 3, 2026 17:48
@pratikbuilds

Copy link
Copy Markdown
Contributor Author

Addressed the review feedback in the updated branch:

  • rebased onto current main, where catalog seeding now lives in packages/seeding
  • ported distinct priority assignment and existing-offering reconciliation to that package
  • updated the onboarding Hub fakes for the idempotent credential-save 409 -> GET/PATCH path
  • added ANTHROPIC_MODEL so an ENV-selected curated model (for example, Opus) is made first for both runtime resolution and the Settings catalog; Sonnet remains the default
  • propagated that preference through both system seeding and the fast environment-credential plant
  • documented that boot/reseed configuration is authoritative after a manual priority reorder

Validation completed locally:

  • 162 focused configuration, seeding, and onboarding tests passed
  • 26 OAuth route tests passed
  • typecheck and structural checks passed
  • full local check passed with the repository's documented DB-gated skips when DATABASE_URL is absent
  • Greybeard and Critique re-review found no remaining code findings

The branch head is now 56f552765 and GitHub CI is running.

@TheGreatAxios

Copy link
Copy Markdown
Contributor

Closing: superseded by #599 (same fix on the current cl-7294 branch; this branch is based on the removed hub-client package). Tracked as CL-7294.

TheGreatAxios added a commit that referenced this pull request Sep 4, 2026
…294) (#599)

Fixes CL-7294

Catalog seeding used one priority per provider, so every Anthropic model tied at 0 and runtime `localeCompare` picked `claude-fable-5` over the declared default. Offerings now get distinct priorities from the flattened `CATALOG_SEEDS` order (`claude-sonnet-5` is index 0), and a 409 reconciles via GET then PATCH instead of skipping forever.

Supersedes #546 (stale `hub-client` branch; that package is gone on main).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Anthropic default model is inconsistent between seed configuration and UI

2 participants