Skip to content

Commit 5e74277

Browse files
committed
Add llama.cpp backend (Vulkan/CUDA/CPU) and real Hugging Face search
llama.cpp backend, added alongside Ollama (not a replacement - Ollama still works exactly as before): - New provider "llamacpp" backed by node-llama-cpp, with GPU backend selection (auto/vulkan/cuda/metal/cpu) in Settings, auto-detecting which backends are actually available on the machine. - Downloaded GGUF files live in their own configurable directory (separate from Ollama's OLLAMA_MODELS, since the two use incompatible on-disk layouts) and show up in the model picker as their own group. - Agent mode tool-calling isn't implemented for this backend yet - attempting it throws a clear error rather than silently ignoring tools. Everything else (streaming, generation params, GPU offload) works the same as any other provider. - Known tradeoff: each turn creates a fresh context and replays the conversation history rather than keeping a session warm across turns, so long conversations redo more prompt-processing work than an optimized implementation would. Model weights themselves stay cached across turns. - Verified node-llama-cpp's native binary actually loads and correctly reports Vulkan as an available backend on this machine (not just a type-check pass). Hugging Face search: the model search box now queries the real HF Hub API instead of requiring an exact pasted URL/tag. Typing a query shows real repos ranked by downloads/likes; expanding one lists its actual GGUF files with real sizes, downloadable either via Ollama's existing pull mechanism or directly into the llama.cpp backend's models folder (with progress reporting). Not done this round: the broader "make the UI more professional/ scalable" ask - no concrete direction was given for what specifically needs to change, and a broad visual redesign isn't something to guess at blindly given how much surface area it would touch.
1 parent 2a8d9b7 commit 5e74277

14 files changed

Lines changed: 2279 additions & 119 deletions

File tree

‎README.md‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,8 @@ Beyond chat, Modelforge includes an **agentic mode** — the model can read/writ
2727
## Features
2828

2929
**Chat & providers**
30-
- Local Ollama models, OpenAI, and Anthropic in one interface, with token-by-token streaming.
30+
- Local Ollama models, OpenAI, Anthropic, **and llama.cpp** in one interface, with token-by-token streaming.
31+
- **llama.cpp backend** — run GGUF models directly via [node-llama-cpp](https://github.com/withcatai/node-llama-cpp) instead of Ollama, with **Vulkan**, CUDA, or Metal GPU acceleration (auto-detected, selectable in Settings). Useful for Vulkan acceleration or models Ollama doesn't package. This is an *additional* backend, not a replacement — Ollama is still fully supported side by side. Agent mode tool-calling isn't wired up for this backend yet (a clear error explains this if you try); everything else (streaming, GPU offload, all the generation parameters) works the same as any other provider. Model weights load once and stay warm across messages, but each turn currently re-evaluates the full conversation from scratch rather than reusing a cache across turns — correct, but slower on long conversations than it could be.
3132
- Vision support — attach images (or extract frames from a video) for models that can see them. When an image is attached, an **"Analyze as..."** menu fills the composer with a ready-made prompt for common diagram/wireframe tasks — describe the UI, convert it to a Mermaid diagram, generate React + Tailwind code from it, list its components, or review it for usability/accessibility issues.
3233
- Live token usage and estimated cost per message and per session (Ollama is free/local; cloud providers show a running estimate).
3334

@@ -54,7 +55,7 @@ Beyond chat, Modelforge includes an **agentic mode** — the model can read/writ
5455
**Models & hardware**
5556
- Model recommendations based on your actual hardware — RAM and VRAM are detected and summed **across all GPUs**, not just the first one, so multi-GPU machines get accurate suggestions.
5657
- **GPU offload control** — set how many model layers Ollama offloads to GPU (`num_gpu`) per chat, per project, or as a global default; leave it blank to let Ollama decide automatically.
57-
- **Pull models directly from Hugging Face** — paste any GGUF model's Hugging Face URL (or `hf.co/user/repo`) into the model search box alongside the curated catalog and Ollama's own library.
58+
- **Real Hugging Face search** — typing in the model search box queries the actual Hugging Face Hub API (not just "paste an exact URL"), showing real repos ranked by downloads/likes; expand one to see its actual GGUF files with real file sizes, then either pull it via Ollama or download it directly for the llama.cpp backend. Pasting an exact `hf.co/user/repo` tag or a full URL still works too, for Ollama's own pull mechanism.
5859
- Models with reliable tool/function-calling support are flagged with a 🔧 badge, so picking a good Agent mode model doesn't require guesswork.
5960
- **Custom model storage location** — Settings → Ollama Server → "Model storage location" lets you point downloaded models at any folder (e.g. a larger or faster drive) instead of Ollama's default location. If this app started Ollama, it restarts it automatically with the new location; if Ollama is running outside the app, you're told to restart it yourself.
6061

0 commit comments

Comments
 (0)