The installer detects your GPU and provisions the matching llama.cpp backend; you can also do it (or change it) directly:
localm setup-llama # auto-detect
localm setup-llama --backend vulkan # any GPU
localm setup-llama --backend cuda # NVIDIA
localm setup-llama --backend amd-rocm # AMD, self-contained (Windows only)
localm setup-llama --backend hip # AMD, needs a system ROCm/HIP toolkit
localm setup-llama --backend sycl # Intel
localm setup-llama --backend metal # Apple Silicon (macOS only)
localm setup-llama --backend cpu # no GPU
localm setup-llama --from <build dir> # your own llama.cpp buildSee the table below for what each backend needs and when to pick it.
| Backend | Runs on | Notes |
|---|---|---|
vulkan |
any AMD / NVIDIA / Intel GPU | the universal fallback - only the normal display driver, no CUDA/ROCm/oneAPI toolkit; auto-picked for Intel on Linux and for AMD with no ROCm/HIP toolkit detected |
cuda |
NVIDIA | auto-picked on every OS (peak performance); setup fetches the CUDA runtime for you (no Toolkit) on both Windows and Linux, then load-tests and offers vulkan/cpu if it cannot load |
amd-rocm |
AMD RX 6000/7000/9000 (gfx103X/110X/120X), Windows only | self-contained ROCm build matched to your card's family (bundles its runtime, no toolkit needed); auto-picked for RX 6000 (gfx103X), the best-tested family - request it explicitly (--backend amd-rocm) on RX 7000/9000 too, since auto-detect still defaults those to hip/vulkan; on Linux use hip or --from instead |
hip |
AMD (any gfx) | upstream ROCm build; auto-picked when a system ROCm/HIP toolkit is detected present, else vulkan |
sycl |
Intel GPU (Arc/Battlemage, or an 11th-gen+ Core integrated GPU) | self-contained on Windows (bundles the oneAPI runtime) and auto-picked there; needs a system oneAPI install on Linux, so it stays opt-in there |
metal |
Apple Silicon | auto-picked on Apple Silicon (Intel Macs get cpu); experimental, Metal inference unverified - see the note below |
cpu |
no GPU | always works |
Binaries come from official llama.cpp releases (AMD uses a self-contained build); the loader auto-detects the GPU at runtime.
setup fetches the build for the CPU architecture of localm's own Python:
an Intel Mac gets the macOS x64 build and Apple Silicon the arm64 one. On ARM64
Linux and ARM64 Windows, cpu, vulkan and cuda have a build, and cuda is
for Blackwell-class and newer GPUs only (upstream builds ARM64 CUDA on the 13.x
line only), so an older NVIDIA GPU there is set up with vulkan. On ARM64 Linux
the CUDA build and its runtime bundle both come from upstream's release. A backend with no build for the machine's architecture is
refused with the list of backends that have one. Any other architecture (s390x
has cpu only) needs your own build: localm setup-llama --from <build dir>. macOS/Metal is experimental and its inference is unverified - treat it as best-effort.
CI on an Apple Silicon runner checks the install, the Metal auto-pick, localm doctor, a server with no model, and a tiny model loading and generating on the
CPU (0 GPU layers); it does not run Metal inference. If inference fails or hangs
on macOS, run on the CPU with localm config n_gpu_layers 0 (or -g 0 on
localm run / localm serve). --backend cpu does not do this on Apple
Silicon: it installs the same arm64 build as metal.
Before anything is kept installed, setup load-tests the library - and that
load includes localm's own ABI check: does this exact build's struct layout and
enum values still match what localm's native bindings expect (an upstream release
has broken this before without changing the visible API). A build that fails is
never left installed: for a vendor backend (cuda/hip/sycl/amd-rocm) setup
explains why and offers the universal Vulkan build instead; vulkan/cpu are
themselves the universal builds, so a failure there instead names the missing
piece and offers a retry, then - if you decline or it still fails - lets the
rest of setup continue so you can provision a runtime later instead.
On Linux, every cpu/vulkan build upstream publishes links against the
system's OpenMP runtime (libgomp.so.1) but does not ship it, so a bare/minimal
distribution image can be missing it. setup bundles a copy automatically (the
same way manylinux Python wheels do), so this normally never comes up. If a
load still fails naming a different missing .so, that is a genuine system
dependency - e.g. libvulkan1 (or your GPU vendor's Vulkan ICD) for the
vulkan build - install the named package and retry.
Once a backend is provisioned, setup-llama can also change or revert it in
place:
localm setup-llama --tag b10355 # install exactly this llama.cpp release and pin it
localm setup-llama --tag latest # track upstream's newest release (untested by localm)
localm setup-llama --tag default # back to the build localm ships and confirmed
localm setup-llama --rollback # back to the previous build recorded for this backend
localm setup-llama --backend cuda # switch backend on a machine that already has oneBy default you get the newest upstream release localm has confirmed works
(default); an upstream release that turns out broken on your hardware is exactly
what --rollback is for. Both --tag and --rollback stick, including across
localm update. localm doctor shows which build is installed and whether it is
pinned.
The same actions are in the GUI under Settings -> Updates -> Inference
runtime: a Check for runtime update button (compares what is installed
against what setup-llama would install right now, without downloading
anything), an Update runtime button once a change is available (pick a
different backend or an exact build first, or leave both blank to just refresh the
current one), and a Roll back button that appears whenever an earlier build is
on record for the installed backend. This is also how a GUI-only user installs a
runtime for the first time on a machine that has none, or switches backend,
without touching a terminal.
localm uses the native llama.cpp HIP (ROCm) build for AMD GPUs (llama.dll on Windows, libllama.so on Linux). No Python GPU packages are needed for GGUF inference: only the prebuilt native libraries.
--backend amd-rocm detects your card's family from its name and fetches the
matching self-contained build automatically: gfx103X for RX 6000, gfx110X for RX
7000, gfx120X for RX 9000. An unrecognised AMD card falls back to the gfx103X
build. This is Windows only; on Linux use --backend hip (needs a system
ROCm/HIP toolkit) instead.
That build's own CPU backend uses no SIMD instructions, so setup also installs
the AVX2/AVX-512 CPU backend from the same llama.cpp commit (the variant this
CPU supports best) over it. Work that runs on the CPU, such as Mixture-of-Experts
weights kept in system RAM, runs about twice as fast with it. An install made
before this shows a runtime update; localm setup-llama --backend amd-rocm --force applies it.
The layout below is illustrative for the gfx103X build specifically - RX 7000 (gfx110X) and RX 9000 (gfx120X) follow the same shape with their own DLL set, fetched the same way.
Required DLLs (load order matters):
ggml-base.dll
ggml-cpu.dll
ggml-hip.dll ← GPU compute via ROCm HIP
ggml.dll
llama.dll
The DLL directory is auto-detected. To use a custom location:
set LLAMA_CPP_LIB=D:\path\to\llama.dll
localm run mymodel --prompt "hello"Or set it permanently in your shell profile.
If you want to build llama.dll yourself from the llama.cpp source:
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1030 -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j8Output: build\bin\Release\llama.dll and sibling ggml*.dll files.
-DAMDGPU_TARGETS=gfx1030targets RX 6000 (RDNA2). For another AMD GPU set it to your card's gfx arch (e.g.gfx1100for RX 7000 / RDNA3). On Linux,localm setup-llama --backend hipdownloads upstream's prebuilt ROCm build, which loads against the ROCm/HIP runtime installed on the system (it does not bundle one); to use a build you compiled, runlocalm setup-llama --from <build dir>.
Run with a 13B+ model and watch VRAM usage in Task Manager or rocm-smi.
localm prints at startup:
ggml_cuda_init: found 1 ROCm devices (Total VRAM: 16368 MiB):
Device 0: AMD Radeon RX 6900 XT, gfx1030
To limit GPU layers (partial offload):
localm run mymodel --gpu-layers 20 --prompt "hello"The installer recommends cuda for NVIDIA on both Windows and Linux
(peak performance, self-contained - see below). The universal alternative is
localm setup-llama --backend vulkan (runs on the NVIDIA GPU through the
normal driver, no CUDA toolkit) if you would rather skip the runtime download.
On Windows this is a guided, self-contained path: setup checks your
driver, then fetches BOTH the CUDA llama build and the matching cudart
runtime bundle from the same llama.cpp release, so you do not need to
install the CUDA Toolkit (the driver must support CUDA >= 12.4 for most GPUs;
nvidia-smi prints the version it supports). Setup also
detects your GPU's own architecture and picks the CUDA asset line it actually
needs: NVIDIA Blackwell (RTX 50-series and datacenter B100/B200) automatically
gets the newer 13.x line - which needs a newer driver in turn (CUDA >= 13.4) -
since the 12.x build has no kernels for Blackwell; every earlier architecture
stays on the 12.x line. Setup checks your driver against whichever line your
GPU needs and, if it is too old or no NVIDIA is found, uses Vulkan for now and
tells you what to do; after updating a driver, re-run
localm setup-llama --backend cuda --force.
On Linux, --backend cuda is self-contained the same way: upstream
llama.cpp does not publish a Linux CUDA binary itself, so setup fetches one
from an actively-maintained third-party builder, plus the CUDA runtime
libraries (cudart, cuBLAS) as separate small downloads - again, no CUDA
Toolkit install needed, and the same Blackwell-aware asset-line pick applies.
Real-hardware field testing (NVIDIA RTX PRO 4000 Blackwell) confirmed CUDA
works and outperforms Vulkan on Linux too, so it is now the default
recommendation there, the same as Windows.
After provisioning, setup load-tests the library exactly as localm run
will, on every platform. If the CUDA build cannot load on your machine,
setup says why and offers the universal Vulkan build (declining stops setup so
you can fix the cause; a non-interactive run falls back to Vulkan, then CPU,
with a warning), so setup never leaves you with a silently broken runtime.
--sha256 <hex>pins the CUDA build archive only; the cudart runtime bundle (Windows) or the CUDA runtime libraries (Linux) are verified by their own separately published checksums instead.
localm setup-llama --backend sycl is the installer's default on Windows:
it is self-contained (bundles the whole oneAPI/Level-Zero runtime - only the
Intel GPU driver is needed) and often faster than Vulkan; if it cannot load,
setup offers the Vulkan build instead. SYCL works on integrated GPUs too
(11th-gen Core / Tiger Lake and newer, including the built-in Arc iGPU on
Meteor Lake and Lunar Lake), not only discrete Arc/Battlemage cards - though
an iGPU with fewer than 80 execution units will likely be too slow to be
worth it, and needs enough system RAM available as GPU-shared memory to hold
the model.
On Linux, the default stays --backend vulkan (no toolkit) instead: the
Linux SYCL build needs a separate system oneAPI install most users will not
have, so it is opt-in there (--backend sycl) rather than automatic.
Provision the CPU build with localm setup-llama --backend cpu. On Windows the
HIP/CUDA DLL is then absent and the DLL load order becomes:
ggml-base.dll → ggml-cpu.dll → ggml.dll → llama.dll
Set --gpu-layers 0 to disable GPU offloading:
localm run mymodel --gpu-layers 0 --prompt "hello"With more than one GPU, a model too large for one card can be split across
several. This applies to n_gpu_layers/GGUF loading, not to media generation
(ComfyUI drives its own GPU placement separately).
localm config main_gpu_index 0 # which device to load onto (blank = device 0)
localm config gpu_split_indices 0,1 # which devices to split across (one = a GGUF chat model on only that device)
localm config gpu_split_ratios 3,1 # relative weight per device, same order/lengthLeft blank, gpu_split_indices still spreads a model over every detected GPU by
free VRAM; set it explicitly to choose which cards, and gpu_split_ratios to pin
exact relative weights instead of following free VRAM at load time (blank
distributes evenly when free VRAM cannot be measured). On a CUDA or ROCm build the
indices are the ones localm gpus and the GUI list. A computer with an integrated
GPU alongside a discrete one lists the integrated GPU too, and llama.cpp usually does
not use it then: for a GGUF load localm checks llama.cpp's own device list and
renumbers a configured split and main GPU into it, and refuses a split (or single
GPU) that names an integrated GPU llama.cpp leaves out, with a message saying so.
When the runtime keeps the integrated GPU the numbers already match; when localm
cannot match the two lists, llama.cpp's default split is kept and the reason is
logged. On the vulkan or sycl build the indices are llama.cpp's own device order.
In the GUI, the same controls live in Settings' Live Tuning card as a Main
GPU selector and a "Split across GPUs" checkbox row (populated from the
detected device list), with a ratio-weight input beside each checked device for
gpu_split_ratios. Ticking one GPU loads a model on that GPU only; leaving
every box unticked keeps automatic placement.
A Mixture-of-Experts (MoE) model's expert layers can be kept in system RAM
instead of VRAM, so it fits in far less GPU memory. When such a model does not
fit and GPU layers are auto-sized (the default), localm does this by itself: it
keeps the routed experts of as few layers as needed in system RAM and every
layer on the GPU, and only moves whole layers to the CPU when even that does
not fit. On several GPUs it checks that each card can hold its own share; with
a configured split (gpu_split_indices) it leaves the choice to you. To choose
it yourself:
localm config n_cpu_moe 16 # keep 16 layers' worth of experts on system RAM
localm config n_cpu_moe 0 # automatic, only when the model does not fit - the defaultA value above 0, or an explicit n_gpu_layers, is used as given.
Generation speed is then bounded by system RAM bandwidth: each token reads the selected experts of every layer kept in RAM, and the load output prints that amount. Measured on Qwen3-30B-A3B Q4_K_M (18.6 GB) with 12.8 GB free on a 16 GB card: automatic placement kept the experts of 18 of 48 layers in RAM and generated at 26 tokens/s, against 19 tokens/s when whole layers were moved to the CPU instead. A model that activates large experts (for example 2 of 8 experts of several billion parameters each) reads far more per token and is slow from RAM by design. Has no effect on a normal (dense) model, and says so instead of silently doing nothing. The Settings page has the same field ("MoE expert layers on CPU").
A model whose weights do not fit in system RAM can still load when the file is
memory-mapped: only the pages a run touches are resident, and the rest are read
from disk on demand. use_mmap chooses:
localm config use_mmap auto # localm decides - the default
localm config use_mmap on # always memory-map
localm config use_mmap off # always read the whole model into memory firstWith auto and every layer on the GPU, localm reads the weights that stay in
system RAM (the token embeddings, and any MoE experts kept off the GPU) into
memory when they fit available RAM with 2 GiB to spare (inside a container, its
memory limit counts), and memory-maps the model when they do not. The load
output, the GUI load toast and the load response (mmap, mmap_from_disk)
then say it runs from disk-backed memory and that first tokens are slower. A
model split between GPU and CPU, a machine with no GPU, and a load where
available RAM cannot be read use llama.cpp's own default, which memory-maps the
file wherever the device supports it; when the weights such a load keeps in
system RAM (its CPU layers, plus a draft model placed on the CPU) do not fit
available RAM with 2 GiB to spare, the same three places say the CPU layers read
them from disk-backed memory and every token is slower (mmap_cpu_from_disk in
the load response). on and off are used as given; with off, a load whose
weights do not fit available RAM warns before it starts that it may fail or
swap, and the GUI load toast and the load response (mmap_ram_short,
mmap_note) carry the same warning. The
Settings page has the same field ("Memory-map model files").
Loading an image runs the vision projector (mtmd) on the GPU by default,
falling back to CPU only if the GPU attempt genuinely fails (logging why).
Set LOCALM_MTMD_CPU=1 to force the old CPU-only behaviour:
set LOCALM_MTMD_CPU=1
localm run myvisionmodel --image photo.jpg --prompt "Describe this image."The live GPU stats localm shows (VRAM and load percent) work on AMD and Intel
now, not only NVIDIA: NVIDIA reads nvidia-smi; AMD reads the driver's own ADL
sensor for whole-card load (so it is not fooled by ROCm/HIP compute, which does
not show up on the generic Windows GPU-engine counters); everything else on
Windows falls back to those generic counters. This is Windows-first: on Linux
only the NVIDIA reading is currently available. VRAM readings are corrected the
same way, since AMD's own driver query under-reports free VRAM by only counting
what the current process itself has allocated.
GGUF inference needs no PyTorch - this section is only for HF-format models. The
installer auto-detects your GPU and installs the matching torch wheels. To do it
by hand, use the line for your hardware - if you are not sure which NVIDIA line
applies, ask localm's own detector rather than guessing. The commands below use
.venv because that is what the self-contained installers create and use uv
specifically (not plain pip) because --torch-backend is a uv-only feature
that routes only the PyTorch-family packages to that wheel index, leaving every
other dependency (including setuptools) on the normal PyPI index - pip install --index-url ... would replace PyPI outright and can fail to resolve a
plain dependency the vendor index does not carry. If you installed with pip install localm there is no .venv to point at - drop -p .venv and run the
same uv pip install torch torchvision --torch-backend=... in whatever
environment localm is already installed into (uv itself is a separate,
optional tool - pip install uv first if you do not already have it). If setup
put uv inside the clone (the Portable choice), run it as ./.uv/uv (Windows:
.uv\uv.exe) unless it is on your PATH.
.venv/bin/python -m localm.hwdetect torch-args cuda (Windows:
.venv\Scripts\python -m localm.hwdetect torch-args cuda) prints the exact
uv pip install arguments for the card actually in this machine.
# NVIDIA (CUDA), pre-Blackwell (most cards shipped before 2026), any OS:
uv pip install -p .venv torch torchvision --torch-backend=cu126
# NVIDIA (CUDA), Blackwell and newer (RTX 50-series, RTX PRO Blackwell,
# B100/B200) - the cu126 wheels above carry no kernels for these and PyTorch
# would silently run CPU-only:
uv pip install -p .venv torch torchvision --torch-backend=cu130
# Intel (Arc / Xe), any OS - the wheels carry the oneAPI runtime:
uv pip install -p .venv torch torchvision --torch-backend=xpu
# AMD on Linux - upstream ROCm wheels (broad gfx support):
uv pip install -p .venv torch torchvision --torch-backend=rocm6.2
# AMD on Windows, RX 6000 / RDNA2 (gfx103X) - localm's bundled self-contained build.
# `-e ".[gpu]"` is an editable install and needs a source checkout to point at;
# for `pip install localm` use the non-editable equivalent instead:
# pip install --upgrade "localm[gpu]"
uv pip install -p .venv -e ".[gpu]"
# AMD on Windows, RX 7000 / 9000 (RDNA3 / RDNA4) - AMD's Windows ROCm wheels
# (public preview). This repo is a flat wheel listing rather than a pip/uv
# package index, so it needs --find-links with exact pinned versions, never
# --torch-backend/--index-url:
uv pip install -p .venv torch==2.9.1+rocm7.2.1 torchvision==0.24.1+rocm7.2.1 \
--find-links https://repo.radeon.com/rocm/windows/rocm-rel-7.2.1/
# CPU (any machine):
uv pip install -p .venv torch torchvision --torch-backend=cpuThe [gpu] extra is the gfx103X (RX 6000) self-contained build (Windows, Python
3.12) - on other AMD Windows cards use the ROCm preview wheels above, and on
Linux use the upstream ROCm index. AMD ROCm on Windows is a recent public
preview, so expect rough edges there. GPU is selected automatically; override
with --device cuda / --device xpu / --device cpu.
On AMD (ROCm), loading an HF model also picks the fastest rocBLAS kernel for each of
its one-token matrix products (PyTorch TunableOp). The first load of a model on a
card spends a few seconds on this; the results are kept in
<data dir>/cache/rocm-gemm-tuning/ and reused on later loads until torch or ROCm
changes. Setting PYTORCH_TUNABLEOP_HIPBLASLT_ENABLED=1 also considers hipBLASLt
kernels, which some cards (RX 6000) do not have.
If setup or a run fails, localm tells you what broke and why, and (for an unexpected error) offers to save a prefilled, editable bug report you can read before anything is sent. You can also create one any time:
localm bug-report -m "short description of the problem"It collects a hardware/backend snapshot plus an allowlisted, scrubbed subset of
config (context sizing, ports, media URLs, privacy mode) - never the whole
config, and never your API key, environment variables, secrets, or chat/transcript
content. The report is saved under your data dir (bug-reports/); you can edit
it, then send it with localm bug-report --send (no GitHub account needed),
email it to the maintainer, or send the file however you like.