Repository navigation
Seed model offerings in declared priority order - #546
pratikbuilds wants to merge 7 commits into
Conversation
TheGreatAxios
left a comment
There was a problem hiding this comment.
Review · Request changes
Catalog seeding assigns each declared provider/model pair a deterministic priority.
Findings
packages/hub-client/src/seed.ts:1341— Could the seed reconcile an existing offering's priority rather than skip the 409? Tenants seeded before this change retain equal priorities on rerun, so the intended Sonnet-first order never reaches those catalogs and resolution continues to use the tie-breaker.
TheGreatAxios
left a comment
There was a problem hiding this comment.
Critic · Comment
Catalog seeding flattens declared provider and model order into offering priorities.
Findings
packages/hub-client/src/seed.ts:1341— Existing offerings are skipped after a 409, so a catalog seeded before this change keeps the former equal priorities even after rerunning the seed command. A re-run regression should establish that upgrade behavior.
TheGreatAxios
left a comment
There was a problem hiding this comment.
Priority-ordering logic is sound and the offering-conflict-reconciliation addition is a real improvement, but this PR needs a rebase onto the packages/seeding move and currently breaks CI outside the touched package.
Must fix
- Rebase needed: yes (2 conflicts) — seed code now lives in
packages/seeding.packages/hub-clientno longer exists onmain(it was moved wholesale topackages/seeding/src/seed.tsandpackages/seeding/test/seed.test.ts);git merge-treereports both touched files as modify/delete againstmain. Neither the offset-priority fix nor the 409-reconciliation logic is present inpackages/seeding/src/seed.tstoday (stillMath.max(0, indexOf(provider))at seed.ts:1577-1579, andensureCatalogOffering's 409 branch still just logs "already exists (skipped)"), so this needs to be reapplied there, not merged as-is. - CI is red on this PR independent of the rebase:
build-testfails onpackages/onboarding/test/complete-credential.test.ts→ "a second credential save for the same account is idempotent...". That suite's fake catalog-offerings handler returns{ status: 409, data: {} }on the second offering POST with no GET/PATCH-offerings route implemented. The newensureCatalogOfferingconflict path now unconditionally lists offerings and PATCHes on a priority mismatch, so it hits a route the fake doesn't provide and throws (or fails to parse{}againstModelOfferingResponse). This is a real regression the PR introduces, not flakiness — worth fixing inpackages/onboarding's test fakes as part of this change (or as a directly-following commit) once ported topackages/seeding.
Consider
- The 409-reconciliation path pages through all catalog offerings every time a conflict is hit, looking for one
modelId+providerIdmatch; for a tenant with a large catalog this is an O(n) full scan per conflicting offering during every seed run. Not wrong, but worth a comment noting the assumption (catalogs stay small) if that's intentional. ensureCatalogOffering's new behavior silently promotes every re-run into a potential PATCH when only the freshly-added offerings actually need reordering; on a bench where an operator has since hand-tuned an offering's priority, a re-seed will now overwrite that customization back to the computed value with no distinction between "we planted this" and "operator changed this." Might be fine per the seed contract, but flag it explicitly if seeding is meant to be idempotent-but-not-authoritative over operator edits.
Verified
- Priority math:
offeringPriorityOffsetsums prior providers'models.lengthinCATALOG_SEEDSiteration order (a literal object, so order is stable), then addsmodelIndexfromseededModels.entries(). This produces a strictly increasing, globally unique priority per (provider, model-declaration-order) pair — no ties possible across providers or within a provider, and the first declared model for a provider (e.g. Anthropic's Sonnet) always gets the lowest priority of that provider's block. Confirmed viagit showdiff read directly; test"fresh run gives the declared Anthropic default the lowest distinct priority"asserts both uniqueness and the min-priority claim. - The 409→list→conditional-PATCH flow parses both the paginated list response and the PATCH response through
parseAs/arktype schemas (ModelOfferingResponse,paginatedSchema) rather than casting — consistent with the repo's trust-boundary rule. gh pr checks 546:lint,typecheck,structural,e2e,isolation,db-suitesall pass;build-testfails (see above).- Scope matches the stated intent (#541): no new env var for model selection,
ANTHROPIC_API_KEYstays credential-only.
ensureCatalogOffering lists existing offerings after a 409 so it can reconcile priority. The onboarding fake Hub only stubbed the POST, so a second credential save threw instead of treating the catalog as already seeded.
8711245 to
56f5527
Compare
|
Addressed the review feedback in the updated branch:
Validation completed locally:
The branch head is now |
|
Closing: superseded by #599 (same fix on the current cl-7294 branch; this branch is based on the removed hub-client package). Tracked as CL-7294. |
…294) (#599) Fixes CL-7294 Catalog seeding used one priority per provider, so every Anthropic model tied at 0 and runtime `localeCompare` picked `claude-fable-5` over the declared default. Offerings now get distinct priorities from the flattened `CATALOG_SEEDS` order (`claude-sonnet-5` is index 0), and a 409 reconciles via GET then PATCH instead of skipping forever. Supersedes #546 (stale `hub-client` branch; that package is gone on main).
Summary
ANTHROPIC_MODELso the configured curated Anthropic model is preferred by both runtime resolution and Settings409 -> GET/PATCHflowConfiguration
Set
ANTHROPIC_API_KEYand optionally setANTHROPIC_MODEL=claude-opus-5. The model defaults toclaude-sonnet-5; settingANTHROPIC_MODELwithout the key fails hub configuration. Boot seeding is authoritative, so restarting or reseeding restores the configured model after a manual catalog reorder.Verification
bun run check(database-gated suites skip locally withoutDATABASE_URL)Fixes #541