Repository navigation
Commit 5e74277
committed
Add llama.cpp backend (Vulkan/CUDA/CPU) and real Hugging Face search
llama.cpp backend, added alongside Ollama (not a replacement - Ollama
still works exactly as before):
- New provider "llamacpp" backed by node-llama-cpp, with GPU backend
selection (auto/vulkan/cuda/metal/cpu) in Settings, auto-detecting
which backends are actually available on the machine.
- Downloaded GGUF files live in their own configurable directory
(separate from Ollama's OLLAMA_MODELS, since the two use incompatible
on-disk layouts) and show up in the model picker as their own group.
- Agent mode tool-calling isn't implemented for this backend yet -
attempting it throws a clear error rather than silently ignoring
tools. Everything else (streaming, generation params, GPU offload)
works the same as any other provider.
- Known tradeoff: each turn creates a fresh context and replays the
conversation history rather than keeping a session warm across turns,
so long conversations redo more prompt-processing work than an
optimized implementation would. Model weights themselves stay cached
across turns.
- Verified node-llama-cpp's native binary actually loads and correctly
reports Vulkan as an available backend on this machine (not just a
type-check pass).
Hugging Face search: the model search box now queries the real HF Hub
API instead of requiring an exact pasted URL/tag. Typing a query shows
real repos ranked by downloads/likes; expanding one lists its actual
GGUF files with real sizes, downloadable either via Ollama's existing
pull mechanism or directly into the llama.cpp backend's models folder
(with progress reporting).
Not done this round: the broader "make the UI more professional/
scalable" ask - no concrete direction was given for what specifically
needs to change, and a broad visual redesign isn't something to guess
at blindly given how much surface area it would touch.1 parent 2a8d9b7 commit 5e74277
14 files changed
Lines changed: 2279 additions & 119 deletions
File tree
- app
- src
- providers
- frontend/src
- lib
- pages
- types
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
27 | 27 | | |
28 | 28 | | |
29 | 29 | | |
30 | | - | |
| 30 | + | |
| 31 | + | |
31 | 32 | | |
32 | 33 | | |
33 | 34 | | |
| |||
54 | 55 | | |
55 | 56 | | |
56 | 57 | | |
57 | | - | |
| 58 | + | |
58 | 59 | | |
59 | 60 | | |
60 | 61 | | |
| |||
0 commit comments