ui : add model compatibility estimation - #27957
Open
allozaur wants to merge 2 commits into
Open
Conversation
allozaur
force-pushed
the
allozaur/ui/model-compatibility
branch
2 times, most recently
from
August 31, 2026 10:56
25e0e56 to
e9d6f6d
Compare
allozaur
marked this pull request as ready for review
August 31, 2026 10:56
Contributor
There was a problem hiding this comment.
Pull request overview
Adds client-side utilities in the UI to (1) estimate whether a GGUF quant “fits” into a device’s memory budget and (2) infer tool-calling support by inspecting a model’s chat template. This supports richer model browsing/selection UX without requiring new server flags.
Changes:
- Add a model hardware-compatibility estimation module that tiers GGUF files (
full/limited/none) based on estimated memory use. - Add a chat-template heuristic to detect tool-use capability.
- Export the new utilities from the UI utils barrel, and expose
detectOsfrombrowser-info.ts.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
| tools/ui/src/lib/utils/model-compatibility.ts | New compatibility estimator (memory budget + shard/sidecar grouping + tiering). |
| tools/ui/src/lib/utils/chat-template-tool-detector.ts | New heuristic detector for tool-calling support via template inspection. |
| tools/ui/src/lib/utils/index.ts | Re-export new utility APIs from the utils barrel. |
| tools/ui/src/lib/utils/browser-info.ts | Make detectOs an exported helper (previously file-local). |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+39
to
+60
| const MB = 1024 * 1024; | ||
| /** Hardcoded device RAM (GB) for testing the compatibility UI; 0 disables it. */ | ||
| const TEST_DEVICE_MEMORY_GB = 128; | ||
|
|
||
| /** | ||
| * Resolve the device memory in GB: the user's settings override when set, | ||
| * else the browser's `navigator.deviceMemory` (Chrome/Edge only, capped at 8). | ||
| * Returns 0 when neither is available, which callers treat as "unknown". | ||
| */ | ||
| export function resolveDeviceMemoryGb(configuredGb: number): number { | ||
| // Hardcoded device RAM for testing the compatibility UI; 0 disables the | ||
| // override. TODO: remove once the device-memory source is trusted. | ||
| if (TEST_DEVICE_MEMORY_GB > 0) return TEST_DEVICE_MEMORY_GB; | ||
|
|
||
| if (configuredGb > 0) return configuredGb; | ||
|
|
||
| if (!browser) return 0; | ||
|
|
||
| const nav = navigator as Navigator & { deviceMemory?: number }; | ||
|
|
||
| return typeof nav.deviceMemory === 'number' && nav.deviceMemory > 0 ? nav.deviceMemory : 0; | ||
| } |
Comment on lines
+198
to
+222
| const ctxBytesPer1k = ctxBytesPer1kTokens(nativeCtxTokens); | ||
| const fits = (ctxTokens: number) => | ||
| weightMb + (ctxBytesPer1k * (ctxTokens / 1000)) / MB <= budgetMb; | ||
|
|
||
| if (nativeCtxTokens < MIN_CTX_TOKENS) return 'none'; | ||
|
|
||
| if (fits(nativeCtxTokens)) return 'full'; | ||
|
|
||
| // Find the largest standard tier that still fits within the native window. | ||
| const largestFitting = [...CTX_TIERS] | ||
| .filter((t) => t <= nativeCtxTokens) | ||
| .reverse() | ||
| .find((t) => fits(t)); | ||
|
|
||
| return largestFitting !== undefined ? 'limited' : 'none'; | ||
| } | ||
|
|
||
| /** | ||
| * Approximate KV-cache bytes per 1k tokens. Without a MemProfile probe (which | ||
| * only exists post-launch in llama-macos) we estimate from the native context | ||
| * window; ~0.1 MB per 1k tokens is a conservative mid-range for modern models. | ||
| */ | ||
| function ctxBytesPer1kTokens(_nativeCtxTokens: number): number { | ||
| return 0.1 * MB; | ||
| } |
Comment on lines
+22
to
+28
| export function detectToolUseSupport(t: string): boolean { | ||
| if (!t) return false; | ||
|
|
||
| if (JINJA_TOOLS_VAR.test(t)) return true; | ||
|
|
||
| return TOOL_CALL_TOKENS.some((token) => t.includes(token)); | ||
| } |
Comment on lines
+81
to
+90
| export function computeFileCompatibilityTiers( | ||
| files: HfModelSibling[], | ||
| nativeCtxTokens: number, | ||
| deviceMemoryGb: number | ||
| ): Map<string, CompatibilityTier> { | ||
| const tiers = new Map<string, CompatibilityTier>(); | ||
|
|
||
| // Unknown device memory: leave every file untiered (neutral) rather than | ||
| // guessing a fit we cannot back up. | ||
| if (deviceMemoryGb <= 0) return tiers; |
allozaur
force-pushed
the
allozaur/ui/model-compatibility
branch
from
August 31, 2026 12:49
e9d6f6d to
3259c9c
Compare
allozaur
force-pushed
the
allozaur/ui/model-compatibility
branch
2 times, most recently
from
August 31, 2026 21:46
7e6bc67 to
58677fe
Compare
Port the hardware-compatibility estimator from ggml-org/llama-macos: map every GGUF file in a repo to a full/limited/none tier based on the device memory budget (GPU working set approximated from RAM, less fit slack and an OS floor) and the estimated weight + context memory. Main quants are tiered individually; shards, mmproj and quant-matched draft sidecars inherit their main quant's tier. Sidecar picking mirrors the server's find_best_sibling ranking (deepest directory, exact quant tag, closest bit depth). Also port detectToolUseSupport (infers tool-calling support from a chat template) and the browser get_info fallback helper. Assisted-by: pi
Replace the device-memory tier machinery with a plain file-size estimate: required runtime memory is the model file size with headroom for KV cache and allocator overhead (estimateModelMemoryBytes). Callers present the requirement; there is no device detection and no fit-versus-budget verdict. Drops resolveDeviceMemoryGb, deviceMemoryBudgetMb, computeFileCompatibilityTiers and the CompatibilityTier type, and the barrel keeps only the new estimator. Assisted-by: pi
allozaur
force-pushed
the
allozaur/ui/model-compatibility
branch
from
August 31, 2026 22:50
58677fe to
a07359e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Adds model hardware-compatibility estimation for the web UI:
model-compatibility.tsutil (ported from llama-macos): color-codes each GGUF in a repo asfull(green, fits at native context),limited(yellow, fits at reduced context) ornone(red)--fit-targetlogic: ~75% of RAM for the GPU working set minus 1 GB slack, clamped by a 4 GB OS floorfind_best_siblingnavigator.deviceMemory; unknown memory = no badgesdetectToolUseSupport()util: infers tool-calling support from the chat template (no server flag exists)detectOsfor reuseAdditional information
Requirements