llama : keep explicit host buffer overrides when using mmap

[Inovello]

Overview

Testing revealed that a host buffer type, specifically CUDA_Host was being replaced with CPU buffer type under mmap. Patch makes it so that when the buffer type comes from an explicit -ot rule, it skips the safety check it does (The loader's rule that essentially replaces a host buffer type with a plain CPU one when a model is memory mapped) and keeps the type you asked for which enables CPU resident experts to live in the pinned memory for op offload while the rest of the model stays memory mapped. Prefill of a 26k prompt went from 166 t/s to 330 t/s with a cold page cache, and to 379 t/s once the per-layer embedding rows were cached (Warm), on 2x RTX 3090 with 40 expert layers on the host.

Now as for why the safety check exists and behaves the way it does. Under mmap, the bytes are already stored in the memory through the mapping, so putting a tensor in a host buffer makes a second copy. For the automatic choice that is waste, so the loader downgrades it, and that behaviour is unchanged. The change this PR brings only applies when the user names a host buffer type in an -ot rule. Default loading, --cpu-moe, --n-cpu-moe are not affected so that means the extra copy is the cost the user willingly chooses to pay. If the pinned allocation fails, the buffer falls back to pageable memory. A visible warning can be added if needed.

To test it out, you need to use the -ot rule with the target as a CUDA_Host, for example -ot "ffn_(gate|up|down)_exps\.weight=CUDA_Host", and on a Dual CPU machine (Like Dual Xeons), you run under numactl --interleave=all.

The change lives in the check in create_tensor() in src/llama-model-loader.cpp, which now skips the host buffer downgrade when the new buft_overridden flag is set, and in parse_tensor_buffer_overrides() in common/arg.cpp, which now accepts CUDA_Host as a target.

Additional information

Setup: Qwen3.8-Flash-Next UD-Q6_K_XL (177B MoE, 512 experts, 10 active), 2x RTX 3090 24 GB (PCIe 3.0 x16),
2x Xeon E5-2696 v4, 188 GiB DDR4-2133, Ubuntu 24.04, CUDA 12.0. Context 261,888, -ub 2048 -b 4096, f16 KV,
4 expert layers per GPU, the other 40 layers (85 GiB) in CPU RAM. Server timings from /completion, temperature 0.7,
page cache dropped before each load, --cache-ram 0.

Experts in	Prefill 26k, PLE cold	Prefill 26k, PLE warm	Prefill 131k	Decode short	Decode @131k	Load	RAM
CPU (mmap, pageable)	166 t/s	(not measured, see note)	193 t/s	~16.3 t/s (a)	9.1 t/s	96 s	page cache
CUDA_Host (this PR)	330 t/s	379 t/s	300 t/s	15.9 t/s	11.3 t/s	488 s	89.6 GB pinned

"PLE cold/warm": The model reads the per layer embedding table lazily. The first prompt reads the rows from disk, a second prompt with the same text reads the rows in the page cache. The warm-cache pass for the mmap row was not measured because my box lost power during that run.

Now as the table shows, loading time is about 5x longer because mmap defers reading until pages are touched, while a host buffer is allocated and filled at load time, so all 85 GiB are read from disk and page-locked before the server is ready; that is the 96 s to 488 s difference; it is paid only by users who opt in, in exchange for 2x to 2.3x prefill on a server that runs for days.

(a) The run was cut off at 446 of 512 tokens by a reboot. mmap decode depends on NUMA page placement.

Also worth stating: ik_llama.cpp (fused MoE + pinned) does 353 t/s prefill on the same box, but 10 t/s decode.

Here are some related PR's/Issues I found: #26659, #26110, #25859, #26448, #28136.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: Yes. The code was written by an AI coding agent (Fable) working under my direction on my machine, the benchmarks were run and iterated by the agent on my hardware and I understand the fix proposed.