ggml-cpu : add mirror NUMA strategy (replicate weights on each node)

[matteoscalabrini]

The results below were measured on the development branch, which shares the performance-relevant mirror implementation with this PR.

At steady state, decode throughput matches a fully-converged first-touch --numa distribute setup on this hardware (see the corrected results below). The case for replicas is structural instead: first-touch NUMA and CUDA pinned host memory are mutually exclusive, so a hybrid CPU/GPU setup is forced to choose between DMA-speed prefill and NUMA-local decode - the mirror is the only strategy that provides both (see "Why replicas instead of --numa distribute" below). On top of that, the steady state arrives immediately and unconditionally: no 10-15 minute warmup convergence after every cold start, no page-cache state to manage, no thread-count sensitivity.

What is this?

• Fully replicates large CPU weights on every NUMA node; CPU GEMM paths read from the node-local copy.
• Targets dual-socket systems with large memory pools, especially systems with multiple GPUs split across NUMA nodes.
• Implements GGML_NUMA_STRATEGY_MIRROR - enum value 4, already reserved in ggml/include/ggml-cpu.h but previously unwired.

Why replicas instead of --numa distribute

--numa distribute gets its locality from the kernel's first-touch rule: a page lands on the NUMA node of the thread that first touches it, so when the compute threads fault the weights in, each page lands next to its reader. CUDA pinned host memory breaks that mechanism: when the CPU-resident weights load through the pinned host-buffer path (--no-mmap on a CUDA build places them in a CUDA_Host buffer), every page is faulted by the loader thread at load time - placement is frozen before the compute threads have touched anything, and pinned pages cannot be migrated afterwards. Distribute's locality and pinned memory are mutually exclusive. An operator running hybrid CPU/GPU MoE inference (the DeepSeek/GLM class these machines actually serve) therefore faces a forced choice:

• fast prefill: load pinned (--no-mmap) - DMA-speed expert uploads, but placement frozen wherever the loader ran, or
• NUMA-local decode: load mmap'd - first-touch works, but every expert upload during prefill pays the pageable-copy penalty.

Never both. The mirror removes the choice: locality comes from the replicas rather than from fault order, so the primary copy stays pinnable. Full-rate DMA prefill and node-local decode reads coexist by construction. This is structural, not a benchmark result that better measurement could overturn.

Verified directly on this PR branch - DeepSeek-V4-Flash UD-Q8_K_XL (151 GB), 5x RTX 3090, -ngl 99 -ot exps=CPU, llama-server, one binary, one day; default batching (-b 2048 -ub 512), 9112-token prefill probes and greedy decode probes per arm. Decode is batch-1 and ubatch-independent; the prefill deltas are ub-512-regime numbers - at larger ubatch the upload cost amortizes and the relative pinning gain should compress:

config	prefill t/s	decode t/s
distribute, mmap (fully converged first-touch)	60.0	17.3
distribute, pinned (--no-mmap)	125.1	11.5
mirror, mmap	67.7-69.2	16.4
mirror, pinned (--no-mmap)	128.6	18.2

Pinning buys +86-108% prefill (corroborating a +81% production measurement on a different, 302 GB model). Pinned distribute pays -34% decode, reproduced across two cold-cache runs (11.5 and 11.3 t/s): numastat at server health shows the pinned buffer frozen on the node the loader thread ran on (121/141 GB and 142/142 GB, before any request), and it is byte-identical after generation - nothing migrates, so no warmup ritual can fix pinned placement. Pinned mirror posts the best prefill and the best decode of the entire matrix simultaneously: the self-bound loader places the pinned primary on the home node, the replica covers the other socket (numastat 142.0/141.5 GB), and the replica build costs 9.3 s (memcpy from resident pinned memory). The steady-state decode tie shown in Results is measured in the unpinned regimes; the moment prefill matters enough to pin, distribute forfeits a third of its decode and the mirror forfeits nothing.

Key points

• tensor->data is never modified. The primary copy remains on node 0; KV cache, activations, and all other backends are unaffected.

• Adds one extra weight copy per additional NUMA node. On a dual-socket system, NPS2/NPS4 therefore require 3x/7x additional copies; NPS1 is recommended.

• GPU expert-weight uploads (op offload) read from the replica local to the destination GPU's PCIe NUMA affinity. The NUMA node is obtained from sysfs using cudaDeviceGetPCIBusId, resolved through get_proc_address, and cached per backend. The bytes are identical; only transfer locality changes. GPUs on the home node retain the existing behavior.

  On the development branch, forcing the wrong source node reduced op-offload prefill throughput by ~48% (1375.7 -> 711.0 pp), while remaining bit-exact (160/160). Test hook: GGML_NUMA_MIRROR_SRC_NODE=<n>.

• Remapping occurs only at CPU GEMM read sites (mul_mat, mul_mat_id, llamafile SGEMM x2, repack x2) through ggml_numa_mirror_remap().

• Each thread selects its replica using getcpu() at entry. This remains correct even if a thread migrates, so explicit CPU pinning is not required for correctness.

• Replica discovery is lazy at graph_compute; no allocator hooks are required. Allocation uses anonymous mmap plus raw mbind, with no libnuma dependency.

• No numactl wrapper is needed. Mirror mode sets an MPOL_PREFERRED policy for the home node at backend init, before any weight is allocated, so the primary copy lands on the home node by itself. An explicit policy set by the user (numactl etc., detected via get_mempolicy) is never overridden. PREFERRED rather than BIND, so a model larger than one node degrades gracefully instead of failing allocation.

• The home node is picked automatically: a sysfs PCI scan finds the discrete GPUs (display-class devices from NVIDIA/AMD, which excludes the BMC VGA server boards carry) and takes the node holding the majority of them, since GPU uploads read the primary copy. Falls back to node 0 with no GPUs, logs its choice, and GGML_NUMA_MIRROR_HOME=<n> overrides it. The scan runs before any backend is initialized, which is why it reads sysfs instead of asking the GPU backend.

• If the primary still lands off-home (e.g. a pre-existing conflicting policy), placement self-heals using move_pages() sampling followed by a 1 GiB probe before full migration; unmovable pinned pages produce a warning instead.

• Repack/AMX extra buffer types are mirrored as well.

• Emits a one-shot coverage report and warns when coverage is below 50%.

• Disabled mode, non-Linux platforms, and single-node systems are no-ops. The hot remap path adds one atomic load per call.

Limitations

Currently Linux-only, with a maximum of 16 mirrored buffers.

Related work

This PR builds on the long-running NUMA work discussed in #1437.

The --numa mirror flag name and the mirror concept come from #16000 by @dbsanfte.

The implementation here uses a different mechanism. #16000 turns every tensor->data access into an accessor across 123 files (+4692 lines). This PR leaves tensor->data untouched and confines remapping to the CPU GEMM read sites plus the scheduler's expert-upload source pointer (+516/-10 across 11 files). The sole backend-side addition is a small CUDA helper in ggml-cuda.cu that resolves a GPU's PCIe NUMA node through sysfs; no backend compute or copy path is otherwise modified.

It is also complementary to the sharded-prefetch experiments discussed in #16000. Sharding keeps memory usage at 1x and primarily targets GPU-offload prefill. Mirroring instead targets CPU decode reads: a shard cannot simultaneously provide node-local access to cores on both sockets, whereas a replica can.

Results

Test system: 2x AMD EPYC 7532 (Zen 2, 32 cores/socket), NPS1, 16x64 GB DDR4-2666 (1 DIMM per channel across all 16 channels), 1 TB total RAM, and 5x RTX 3090.

Measured memory read bandwidth:

• Socket-local: 137 GB/s
• Cross-socket (xGMI): 47.7 GB/s
• Remote access penalty: ~2.1x

The main head-to-head table (prefill and decode, distribute vs mirror, mmap and pinned) is in "Why replicas instead of --numa distribute" above.

GLM-5.3-Flash 321B UD-Q4_K_XL

186 GB.

Using the same binary with only --numa mirror toggled and position-matched prompts, decode improved by +46% to +76%, depending on context depth.

The mirror arm ran second and was therefore thermally disadvantaged, making these results conservative. Absolute pairs were not recorded; a fresh llama-bench validation table on the PR branch should supersede these measurements when available.

Coverage sensitivity

Mirror decode gain scales monotonically with the fraction of GEMM weight bytes actually mirrored:

mirrored coverage	decode change
95%	+70.6%
67%	+17.4%
0%	-4.9%

At 0% coverage, mirroring produces a net loss because the node-0 placement penalty remains without any replicas to compensate for it.

This is why the implementation reports coverage and warns below 50%. Mirroring the repack/AMX extra buffers also makes coverage effectively quant-independent: repack on/off was measured within noise (44.08 vs 43.52).

Mechanism microbenchmark

With identical socket and thread count and only weight placement changed:

• Node-local weights: 26.39 t/s
• Node-remote weights: 12.46 t/s
• Ratio: 2.12x

This matches the platform's measured ~2.1x local-to-remote memory read-bandwidth ratio.

Validation on this PR branch (same system)

This section has been corrected twice, both times following methodology critiques by @usrlocalben in the discussion below - the history is preserved there. First: an earlier version of the table wrapped both arms in numactl --membind=0, which prevented first-touch page distribution and handicapped the distribute baseline. Second, and more fundamental: llama-bench structurally cannot converge a first-touch distribute baseline. Each invocation generates only a few hundred tokens while major faults keep trickling for thousands (~36M faults / ~138 GB observed before quiescence), and it runs pp before tg, so at prompt batch sizes the op-offload upload thread faults pages in an arrangement the decode threads never chose - and the page cache freezes that arrangement for every later run.

The llama-bench numbers are therefore kept below as an honest picture of the cold-start / short-run regime (which is what any llama-bench user or fresh server start experiences), and the server-level steady-state comparison is the fair throughput race.

llama-bench tg128, -t 64 (cold-start / short-run regime; no numactl wrapper, drop_caches before every arm, mmap on, distribute warmed across four sequential runs):

model	distribute (warmed x4)	interleave=all	--numa mirror
gemma-4-31B qat Q4_0 (16 GiB, dense, -ngl 0)	3.70	4.93	7.89 +/- 0.14
DeepSeek-V4-Flash UD-Q8_K_XL (151 GiB MoE, -ngl 99 -ot exps=CPU)	11.55	10.29	18.14 +/- 0.16

llama-server steady state (protocol from the discussion: --no-warmup, thousands of warm generation tokens per arm, major faults tracked to quiescence, throughput read from short greedy requests only):

model	distribute steady	mirror steady
gemma-4-31B qat Q4_0 (dense)	7.5-7.6	7.5-7.7
DeepSeek-V4-Flash UD-Q8_K_XL (MoE)	16.9	16.4

The dense mirror figure was reproduced from a cold start with only ~50 warm tokens before measuring: the replica build (17.5 s for 16.1 GiB) is the entire warmup. The distribute arm needs its ~3000-token convergence ritual to post its number.

At steady state neither mode wins on decode throughput on this box. On a bandwidth-rich 2S Genoa-class system (@usrlocalben's independent test in the discussion), fully-warmed distribute reached 34.3 t/s vs 36.4 for the mirror with zero warmup.

(Measurement note: these are cool-DIMM numbers. Sustained multi-minute dense decode on this box drops 30-45% from DIMM heating regardless of NUMA mode - it affects both arms equally and recovers after a few idle minutes. Anyone reproducing with long back-to-back sessions will see it.)

What the mirror buys is the cost of reaching and keeping that steady state:

• Warmup. Distribute pays 10-15 minutes of degraded serving after every cold start while first-touch placement converges (the ~36M major faults above). The mirror front-loads the identical work into the replica build at load time and the first request runs at full speed - an explicit NUMA load mechanism instead of an emergent one.
• Fragility. The first-touch arrangement silently breaks on a stale page cache (measured: 2.88 vs 4.93 on the dense model, -42%), a forgotten drop_caches, or a thread-count change. The mirror has no cache-state or run-ordering dependencies.
• Thread tuning. Mirror decode is flat across t=32/48/64 (18.12 / 18.47 / 18.29) where no-locality configs swing 14.80-18.88 with binding; the thread-count probe stops being needed.
• Pinned primary (GPU prefill). Per "Why replicas instead of --numa distribute" above: a distribute-based setup is locked out of pinned-memory DMA uploads and pays the pageable-copy penalty on every expert upload during prefill; the mirror keeps the primary pinnable (+81% prefill on the production stack; production measurement, not part of the A/B tables above).

The cost is unchanged: 2x RAM for the CPU-resident weights - and, for an operator who performs the warmup ritual correctly and can afford it on every restart, no steady-state MoE decode throughput advantage on this hardware.

Bit-exactness re-verified on this branch: greedy 200-token outputs are byte-identical with the mirror on vs off, with numastat confirming the replica engaged (154.6 GB on node 0 + 153.1 GB on node 1). The upload-sourcing path was exercised with a 4,003-token prompt while forcing the wrong source node (GGML_NUMA_MIRROR_SRC_NODE=1): output stays byte-identical.

The non-default home-node path was validated by forcing GGML_NUMA_MIRROR_HOME=1 on this box (whose GPUs majority-vote node 0): the policy follows the forced node, numastat shows the primary on node 1 with the replica on node 0, greedy output is byte-identical to the home-0 run, and decode matches within noise (7.72 vs 7.88 t/s on the dense CPU-only case). This is the configuration a machine with its GPU on socket 1 now gets automatically.

The registry lifecycle was also validated directly: two sequential model loads in one process with the mirror on peak at 33.9 GB resident (one mirrored model plus KV) - replicas are dropped when their buffer is freed, and freed table slots are reused.

Correctness

Mirror on vs. off:

• 320/320 top-20 logprobs identical
• MTP acceptance identical to 5 decimal places
• 480/480 identical on the instrumented repack path

The replicas contain identical bytes and are consumed by the same kernels; only their NUMA locality differs.

Testing

Further community testing on diverse hardware is needed, particularly on Intel CPUs and AMD/Vulkan systems.

Requirements

• I have read and agree with the contributing guidelines.
• AI usage disclosure: YES - implementation, extraction to a clean branch, and the upload-sourcing port were done with Claude Code under strong architectural supervision. The design, all measurements, and production deployment/verification are mine.

[ggml-gh-bot]

Hi @matteoscalabrini, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

• Multiple backend changes in one PR: When adding support for a new model or feature, focus on CPU support only in the initial PR. Add support for other backends like CUDA in follow-up PRs. If you have a good reason to modify multiple backends in one PR, please explain it.

---

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

[dur-randir]

It's an interesting idea, but for me on a 2S Xeon system it halves pp (900 t/s -> 500 t/s), while not improving tg at all (14 t/s -> 14 t/s) compared to a model loaded & bound to a single socket.

[matteoscalabrini]

It's an interesting idea, but for me on a 2S Xeon system it halves pp (900 t/s -> 500 t/s), while not improving tg at all (14 t/s -> 14 t/s) compared to a model loaded & bound to a single socket.

Can i ask what model did you try and if you did hybrid or cpu only? It works best in cpu only or hybrid with high reliance on cpu.

[dur-randir]

Can i ask what model did you try and if you did hybrid or cpu only? It works best in cpu only or hybrid with high reliance on cpu.

Qwen3.8-Flash-Next-UD-Q3_K_XL-00001-of-00003.gguf, --n-cpu-moe 40, rest offloaded to 5090.

[matteoscalabrini]

Can i ask what model did you try and if you did hybrid or cpu only? It works best in cpu only or hybrid with high reliance on cpu.

Qwen3.8-Flash-Next-UD-Q3_K_XL-00001-of-00003.gguf, --n-cpu-moe 40, rest offloaded to 5090.

A couple of diagnostics would help distinguish the two cases here:

Which commit did you build — before or after 81139af?
Does the startup log show the NUMA-mirror lines at all (NUMA mirror: N nodes / coverage)?
What does numastat -p look like once the model is loaded? In the mirrored case, resident memory should be roughly 2× the CPU-resident model portion and reasonably symmetric across the two NUMA nodes.
Could you retry on current HEAD? Alternatively, for the pp result, try explicitly running with numactl --membind=0 so we can separate the mirror behavior from the placement issue.

For tg, I think your configuration is genuinely inside the regime where binding to one socket wins. The current PR wording around “large memory pools” is probably too vague there, and I can make that crossover more explicit.

[dur-randir]

Which commit did you build — before or after 81139af?

After. I have the following [1] checked out and merged into master, which was on [2].

Does the startup log show the NUMA-mirror lines at all (NUMA mirror: N nodes / coverage)?

Yes.

What does numastat -p look like once the model is loaded? In the mirrored case, resident memory should be roughly 2× the CPU-resident model portion and reasonably symmetric across the two NUMA nodes.

Yes, it correctly takes x2 RAM, split between nodes.

Could you retry on current HEAD?

Only tomorrow. 90% of the time this machine runs Windows VM with 5090, and it's a rather hideous process to switch video back and forth between the host and passthrough :(

Alternatively, for the pp result, try explicitly running with numactl --membind=0 so we can separate the mirror behavior from the placement issue.

I've tried membind=0, -N 0 -l, -N 1 -l (and it fails with the warning you mentioned in docs). After some thinking, my best suspicion is that, since you force initial load on socket == 0, and only later copy to socket == 1, and can't do the reverse (that warning), it may be not prepared that my card is not on socket 0, but on socket 1. You mentioned that you try to use the correct buffer to talk to GPU, but maybe not in all cases? Still, this might explain decreased pp, but doesn't explain same tg. I'll also try tomorrow with --cpu-moe instead, to keep all experts on CPU.

[1] * 81139af 30 Aug 16:44 Okoku - (pr/27986) ggml-cpu : set a home-node memory policy automatically in mirror mode
[2] * f1793c1 30 Aug 16:06 Pascal - (HEAD -> master, origin/master, origin/HEAD) CUDA: use the fast mm_ids_helper path for any n_expert_used (#27978)

[matteoscalabrini]

After some thinking, my best suspicion is that, since you force initial load on socket == 0, and only later copy to socket == 1, and can't do the reverse (that warning), it may be not prepared that my card is not on socket 0, but on socket 1.

I re-traced the code to confirm the consequence: with everything default, primary lands on node 0 and uploads cross UPI. Your workaround — numactl -N 1 -l — actively breaks: the mirror still treats node 0 as home, tries to migrate the primary away from node 1, hits pinned pages, and fires the warning you saw. Under that config replica[0] is the primary sitting physically on node 1, so node-0 threads read remote and the node-1 replica duplicates data already there. The home node needs to be selectable and ideally auto-picked as the GPU's node.

I'm actively working on auto selecting home node using the kernel's PCI device tree in sysfs

[usrlocalben]

It's curious the word drop_caches doesn't appear anywhere in the PR descr, and mmap appears only once. This hints the author may be unaware of this protocol and that the improvements are being compared to an incorrectly initialized --numa distribute. A correctly initialized --numa distribute system for cpu-moe should see relatively little cross-node traffic.

[matteoscalabrini]

It's curious the word drop_caches doesn't appear anywhere in the PR descr, and mmap appears only once. This hints the author may be unaware of this protocol and that the improvements are being compared to an incorrectly initialized --numa distribute. A correctly initialized --numa distribute system for cpu-moe should see relatively little cross-node traffic.

As replied on reddit I report my words for completeness:

Thank you very much for the comment. I am running now tests to verify your suggestions and the effectivity of my code against your suggestion. I'll be back with the results and maybe we can have a look together on them.
I'm very happy to avoid unnecessary code in llama.cpp and therefore treating this seriously.

You are right and I was unaware of this protocol. I am informing myself thoroughly now. and running test to update the data displayed in the pr text.

The error:

• both arms of the published llama-bench A/B ran under numactl --membind=0 ; visible in the table's own methodology line
• that binds every allocation to node 0: the distribute arm had threads spread across both sockets but all weight pages forced onto node 0, so first-touch distribution was impossible and half the reads were remote by construction
• the wrapper was a leftover from before the branch self-bound the mirror's home node
• consequence: the published distribute columns understate a correctly initialized distribute, and the deltas are inflated by that margin

The tests that im now running, (might take a couple of hours):

• echo 3 > /proc/sys/vm/drop_caches before every arm; mmap on (llama-bench default); no numactl wrapper anywhere
• arm A: bare --numa distribute, four sequential invocations without dropping caches between them
• arm B: --interleave=all as the classic reference
• arm C: bare --numa mirror
• arm D: a distribute run on the stale page cache left behind by the mirror arm, no drop
• two models: dense CPU-only (gemma-4-31B, -ngl 0) and hybrid MoE (DeepSeek-V4-Flash, -ngl 99, experts on CPU), -t 64 on 2× EPYC 7532 NPS1

[matteoscalabrini]

It's curious the word drop_caches doesn't appear anywhere in the PR descr, and mmap appears only once. This hints the author may be unaware of this protocol and that the improvements are being compared to an incorrectly initialized --numa distribute. A correctly initialized --numa distribute system for cpu-moe should see relatively little cross-node traffic.

Update after running the tests:

First of all, thanks, your critique was correct. Both baseline arms in the old table were "wrapped" in numactl --membind=0 dirtying the test of --numa distribute. I rerun the full comparison using drop-caches before every arm, mmap enable, no numactl, and pure --numa distribute with warming across 4 runs.

model	distribute (converged)	interleave=all	--numa mirror	mirror vs best baseline
gemma-4-31B qat Q4_0 (16 GiB, dense, -ngl 0, t64)	3.70	4.93	7.89 +/- 0.14	+60%
DeepSeek-V4-Flash UD-Q8_K_XL (151 GiB MoE, -ngl 99 -ot exps=CPU, t64)	11.55	10.29	18.14 +/- 0.16	+57%

Your explaination of the moe behaviour makes sense. The distribute sequence made 9.11 -> 11.40 -> 11.56 -> 11.55. ; warming first-touch placement improves it by about 27%. On that model, converged first-touch distribute also beats interleave (11.55 vs 10.29), which is exactly what you've pointed out.

The stale-cache is real as well. Running distribute against the page cache left by a differentl run gives 2.88 on the dense model, about 42% below interleave.

With those effects accounted for, mirror is still +57% over fully converged distribute on the MoE model and +60% over the best baseline on the dense model.

I also checked the current code path. For these mul_mat_id decode shapes, you end up with exactly nth chunks, so the steal loop exits immediately. Plain mul_mat also takes the static chunk path under ggml_is_numa() which makes the thread-to-row mapping deterministic, so repeated execution can indeed first-touch those mmap-backed pages onto the node whose threads later consume them.

There is still one part I am having difficulties understand:
On the dense model, your protocol is actually placing the mmap'd weights almost perfectly. After warming, /proc//numa_maps for the GGUF mapping shows: N0=2108342 / N1=2108416
So essentially an exact 50-50 split but is still only 3.70 versus 7.89 for mirror, about 2.1x apart.
That seems to rule out simple page placement as the explanation for the gap.
Two things look worth pointing out: 50/50 placement isn't necessarily the same thing as good alignment: a page can be on node 0 or node 1 and still be on the wrong node relative to the thread that becomes its reader. Secondly, there's roughly 21 GB of KV/scratch anonymous memory that lands entirely on one NUMA node under bare distribute, presumably because it's memset at allocation by the main thread, whereas interleave spreads that allocation.

I'm not sure yet how much either of those explains, especially the size of the gap. If you have a read on which one is more likely or if i'm completely missing something (might be this is a bit out of my comfort zone) I'd genuinely be interested.

I've updated the PR body with the corrected table and methodology. The four lifecycle/race/policy issues from the code review are also fixed in 108f651.

[DocShotgun]

The problem with the drop_caches + mmap method for first-touch weights distribution across NUMA nodes is that the PP speed is terrible. The TG speed is nice, but the PP is slower than op offload prompt processing with numactl --interleave=all + --numa distribute with mmap disabled. Really what we need ideally is multi-NUMA tensor parallelism or expert parallelism.

[usrlocalben]

@DocShotgun there is a DMA solution for this here. It will give full pcie5 transfer rates for offloading by overlapping a NUMA gather operation for layer N+1 while layer N is transferred via DMA (CUDA pinned mem).

This gives --distribute decode speed with CUDA pinned mem offload speed for a small cost in addl system ram for a pair of CUDA pinned buffers for each of gate/up/down.

[usrlocalben]

@matteoscalabrini

I reconfigured with NPS1 and ran a comparison.

MXFP4 weights from sokan mxfp4 gguf conversion
Hardware is 1x R6KP + 2x 9B14 w/24x DDR5 4800

llama.cpp master Invocation:

./master/build/bin/llama-server \
--no-repack  \
-lv 4 \
--numa distribute --warmup \
--host 0.0.0.0 --port 4567 \
-t 48 \
-np 1 \
--jinja \
-c $[ 2**17 ] \
-ngl 999 -ot exps=CPU \
-m /model/DeepSeek-V4-Flash-0731/sokann/DeepSeek-V4-Flash-0731.gguf

Llama.cpp warmup is broken, so first 10-20 minutes of "tell me a story about a cat and a doubly linked list in spanish" etc.

4.10.747.830 I slot print_timing: id  0 | task 0 | n_gen =    100, tg =   3.66 t/s, tg_3s =   3.70 t/s
4.13.890.201 I slot print_timing: id  0 | task 0 | n_gen =    125, tg =   4.11 t/s, tg_3s =   7.96 t/s
4.16.899.424 I slot print_timing: id  0 | task 0 | n_gen =    149, tg =   4.46 t/s, tg_3s =   7.98 t/s
4.19.932.240 I slot print_timing: id  0 | task 0 | n_gen =    185, tg =   5.08 t/s, tg_3s =  11.87 t/s
4.22.972.114 I slot print_timing: id  0 | task 0 | n_gen =    227, tg =   5.76 t/s, tg_3s =  13.82 t/s
4.26.142.226 I slot print_timing: id  0 | task 0 | n_gen =    248, tg =   5.82 t/s, tg_3s =   6.62 t/s
4.29.142.866 I slot print_timing: id  0 | task 0 | n_gen =    275, tg =   6.03 t/s, tg_3s =   9.00 t/s
4.32.187.541 I slot print_timing: id  0 | task 0 | n_gen =    314, tg =   6.46 t/s, tg_3s =  12.81 t/s
4.35.254.606 I slot print_timing: id  0 | task 0 | n_gen =    347, tg =   6.71 t/s, tg_3s =  10.76 t/s
4.38.280.853 I slot print_timing: id  0 | task 0 | n_gen =    394, tg =   7.20 t/s, tg_3s =  15.53 t/s
4.41.323.188 I slot print_timing: id  0 | task 0 | n_gen =    435, tg =   7.53 t/s, tg_3s =  13.48 t/s
4.44.337.735 I slot print_timing: id  0 | task 0 | n_gen =    489, tg =   8.05 t/s, tg_3s =  17.91 t/s
4.47.348.686 I slot print_timing: id  0 | task 0 | n_gen =    543, tg =   8.52 t/s, tg_3s =  17.93 t/s
4.50.475.370 I slot print_timing: id  0 | task 0 | n_gen =    590, tg =   8.82 t/s, tg_3s =  15.03 t/s
4.53.480.462 I slot print_timing: id  0 | task 0 | n_gen =    651, tg =   9.32 t/s, tg_3s =  20.30 t/s
4.56.498.814 I slot print_timing: id  0 | task 0 | n_gen =    719, tg =   9.86 t/s, tg_3s =  22.53 t/s
4.59.527.130 I slot print_timing: id  0 | task 0 | n_gen =    782, tg =  10.30 t/s, tg_3s =  20.80 t/s
5.02.557.416 I slot print_timing: id  0 | task 0 | n_gen =    852, tg =  10.79 t/s, tg_3s =  23.10 t/s
5.05.594.493 I slot print_timing: id  0 | task 0 | n_gen =    929, tg =  11.33 t/s, tg_3s =  25.35 t/s
5.08.629.734 I slot print_timing: id  0 | task 0 | n_gen =   1004, tg =  11.81 t/s, tg_3s =  24.71 t/s
5.11.630.212 I slot print_timing: id  0 | task 0 | n_gen =   1079, tg =  12.26 t/s, tg_3s =  25.00 t/s
5.14.667.564 I slot print_timing: id  0 | task 0 | n_gen =   1159, tg =  12.73 t/s, tg_3s =  26.34 t/s
5.17.687.732 I slot print_timing: id  0 | task 0 | n_gen =   1238, tg =  13.16 t/s, tg_3s =  26.16 t/s
5.20.711.790 I slot print_timing: id  0 | task 0 | n_gen =   1313, tg =  13.52 t/s, tg_3s =  24.80 t/s
5.23.733.480 I slot print_timing: id  0 | task 0 | n_gen =   1385, tg =  13.84 t/s, tg_3s =  23.83 t/s
5.26.765.406 I slot print_timing: id  0 | task 0 | n_gen =   1463, tg =  14.19 t/s, tg_3s =  25.73 t/s
5.29.792.661 I slot print_timing: id  0 | task 0 | n_gen =   1548, tg =  14.58 t/s, tg_3s =  28.08 t/s
5.32.802.140 I slot print_timing: id  0 | task 0 | n_gen =   1632, tg =  14.95 t/s, tg_3s =  27.91 t/s
5.35.826.979 I slot print_timing: id  0 | task 0 | n_gen =   1719, tg =  15.32 t/s, tg_3s =  28.76 t/s
<snip>

Idea: Is it possible you read from here the tg= value at this point? It's an easy mistake to make.
It's curiously similar to the measurement difference in your last post.

Eventually there are no more page faults, and the system can be considered to be warmed up.
It takes a long time, e.g.15+ min to finally see zero page faults since llama.cpp's warmup is broken.

Now decode at low context, e.g. 500tok or so. (where cpu-moe is at its highest ratio of attn/ffn duty)

12.14.904.163 I slot launch_slot_: id  0 | task 11536 | processing task, is_child = 0
12.14.904.184 I slot   operator(): id  0 | task 11536 | new prompt, n_ctx_slot = 131072, n_keep = 0, task.n_tokens = 16
12.14.904.189 I slot   operator(): id  0 | task 11536 | checking checkpoint with [11, 11] against 15...
12.14.906.709 I slot   operator(): id  0 | task 11536 | restored context checkpoint (pos_min = 11, pos_max = 11, n_tokens = 12, n_past = 12, size = 12.149 MiB)
12.14.906.721 I slot   operator(): id  0 | task 11536 | cached n_tokens = 12, memory_seq_rm [12, end)
12.14.907.524 I slot init_sampler: id  0 | task 11536 | init sampler, took 0.01 ms, tokens: text = 16, total = 16
12.14.912.579 I slot create_check: id  0 | task 11536 | created context checkpoint 2 of 32 (pos_min = 11, pos_max = 11, n_tokens = 12, size = 12.149 MiB)
12.17.993.174 I slot print_timing: id  0 | task 11536 | n_gen =    103, tg =  33.97 t/s, tg_3s =  34.30 t/s
12.20.996.084 I slot print_timing: id  0 | task 11536 | n_gen =    207, tg =  34.30 t/s, tg_3s =  34.63 t/s
12.24.018.854 I slot print_timing: id  0 | task 11536 | n_gen =    311, tg =  34.34 t/s, tg_3s =  34.41 t/s
12.27.022.350 I slot print_timing: id  0 | task 11536 | n_gen =    414, tg =  34.32 t/s, tg_3s =  34.29 t/s
12.30.031.872 I slot print_timing: id  0 | task 11536 | n_gen =    517, tg =  34.30 t/s, tg_3s =  34.22 t/s

main = ~34.3t/s

Now for the mirror PR #27986

Invocation:

./pr27986-mirror/build/bin/llama-server \
-lv 4 \
--numa mirror --warmup \
--host 0.0.0.0 --port 4567 \
-t 48 \
-np 1 \
--jinja \
-c $[ 2**17 ] \
-ngl 999 -ot exps=CPU \
-m /model/DeepSeek-V4-Flash-0731/sokann/DeepSeek-V4-Flash-0731.gguf

Interesting NUMA startup messages:

<snip>
0.45.842.776 I NUMA mirror: 2 nodes, home node 0, min buffer 1024 MiB
0.45.877.382 I NUMA mirror: primary 0x14c422bf6bc0 138.0 GiB: 100% -> 100% on node 0 (0.0 s)
1.38.193.235 I NUMA mirror: 1 replica(s) of 138.0 GiB built in 52.3 s
1.38.193.260 I NUMA mirror: coverage: 138.0 GiB mirrored, 0.0 GiB not (100% of registered weight buffers)
<snip>

No warmup is needed since this system has an explcict numa load mechanism.

And a request:

1.58.439.880 I slot print_timing: id  0 | task 0 | n_gen =    110, tg =  36.11 t/s, tg_3s =  36.44 t/s
2.01.460.504 I slot print_timing: id  0 | task 0 | n_gen =    221, tg =  36.43 t/s, tg_3s =  36.75 t/s
2.04.463.878 I slot print_timing: id  0 | task 0 | n_gen =    330, tg =  36.38 t/s, tg_3s =  36.29 t/s
2.07.471.049 I slot print_timing: id  0 | task 0 | n_gen =    440, tg =  36.43 t/s, tg_3s =  36.58 t/s
2.10.484.535 I slot print_timing: id  0 | task 0 | n_gen =    549, tg =  36.38 t/s, tg_3s =  36.17 t/s

~36.4t/s

I used 48 threads (out of 192) because I've previously measured this to be optimal for single-token decode on llama.cpp + MXFP4.
It takes a long time to probe for optimal thread count, I don't have resources for that today although the optimal thread count may be different for the mirror.

So the mirror at least hints at (in this simple n=1 test) about +6% decode throughput.
That sounds promising.
I didn't measure prefill, but if the gains transfer to cpu-side small batch throughput then that may be good as well.

If you are repacking the FFN weights, it may explain all of the difference.
In the mmap system, repacking is a challenge:

• They can't be repacked at load time, or at least not trivially, without breaking the first-touch concept, and
• Layer-wise offloading would require CUDA kernels that can handle repacked weights.

[DocShotgun]

@DocShotgun there is a DMA solution for this here. It will give full pcie5 transfer rates for offloading by overlapping a NUMA gather operation for layer N+1 while layer N is transferred via DMA (CUDA pinned mem).

This gives --distribute decode speed with CUDA pinned mem offload speed for a small cost in addl system ram for a pair of CUDA pinned buffers for each of gate/up/down.

Interesting... so in theory implementing this would allow for the PP speed of interleave + no mmap + distribute using CUDA_Host->GPU op offload, with the decode speed of drop_caches + mmap + distribute?

EDIT: So I had DS flash implement the linked hack into my local copy for testing. Doesn't seem to have a speed benefit at least for running DS flash itself, where a majority of the weights were residing on GPU (for DS flash, binding the process to a single socket was still faster in both PP and TG). I'll need to try it using a bigger model, but sheesh stuff is slow to load with mmap...

Also is it normal for the first request after server start to be insanely slow? I thought warmup would already pull all the mmap'd weights into RAM?

[usrlocalben]

@DocShotgun Precisely.

The barrier to 50GB/s PCIe is getting the layer data into something CUDA can DMA w/o the OS.

CUDA has pinned buffers, which ik_llama switched to by default some time ago. I'm not sure what llama.cpp does today. The problem these buffers solve is DMA. (But they make a new problem for NUMA) For the GPU to retrieve from mem directly it needs a buffer that doesn't require Linux kernel negotiation to read. A physical mem mapping that is guaranteed not to be paged/evicted/etc. AFAIK there is no CUDA API to create these w/NUMA awareness.

The idea is to allocate a pair of CUDA pinned buffers to hold A/B copies of one layer of gate/up/down tensors.

When GGML wants to copy layer N to GPU for offload op, grab it from one of the CUDA pinned buffers. While that copies (via DMA, i.e. 50GB/sec pcie5) run e.g. 10 threads of memcpy to get layer N+1 into the second buffer.

When layer #Last is reached, layer #First is preloaded into the buffers so it's ready on the next forward pass.

The xgmi links have more bandwidth than the pcie transfer so the gather memcpy to fill the next layer's buffers during the DMA is no problem.

[DocShotgun]

@DocShotgun Precisely.

The barrier to 50GB/s PCIe is getting the layer data into something CUDA can DMA w/o the OS.

CUDA has pinned buffers, which ik_llama switched to by default some time ago. I'm not sure what llama.cpp does today. The problem these buffers solve is DMA. (But they make a new problem for NUMA) For the GPU to retrieve from mem directly it needs a buffer that doesn't require Linux kernel negotiation to read. A physical mem mapping that is guaranteed not to be paged/evicted/etc. AFAIK there is no CUDA API to create these w/NUMA awareness.

The idea is to allocate a pair of CUDA pinned buffers to hold A/B copies of one layer of gate/up/down tensors.

When GGML wants to copy layer N to GPU for offload op, grab it from one of the CUDA pinned buffers. While that copies (via DMA, i.e. 50GB/sec pcie5) run e.g. 10 threads of memcpy to get layer N+1 into the second buffer.

When layer #Last is reached, layer #First is preloaded into the buffers so it's ready on the next forward pass.

The xgmi links have more bandwidth than the pcie transfer so the gather memcpy to fill the next layer's buffers during the DMA is no problem.

Hmmm... I'm not getting very good results in terms of speed (it's just slower than isolating to a single socket), whether I use DS v4 flash or GLM 5.3 perhaps there's an issue with my invocation?

Dropping caches and then:

GGML_CUDA_GRAPH_OPT=1 \
GGML_OP_OFFLOAD_MIN_BATCH=256 \
GGML_CUDA_NO_PINNED=1 \
GGML_CUDA_EXPS_READAHEAD_THREADS=8 \
~/disable-numa-balancing.sh \
~/llama.cpp/build/bin/llama-server \
-m /mnt/data/models/GGUF/GLM-5.3-Q4_K_M/GLM-5.3-Q4_K_M-00001-of-00011.gguf \
-c 131072 \
-ngl 999 \
-fa on \
-t 120 \
-b 4096 \
-ub 4096 \
--parallel 1 \
--host 0.0.0.0 \
--port 8080 \
-a zai-org/GLM-5.3 \
--jinja \
--offline \
-lm mmap \
-fit off \
-lv 4 \
--numa distribute \
-ot "blk\.([0-9]|1[0-2])\.=CUDA0,exps=CPU" \
--cache-ram 0 \
--temp 1.0 \
--top-k 200 \
--top-p 0.95

[usrlocalben]

@DocShotgun it looks correct at a glance, but the disable-numa-balancing call in the middle of a line-continuation seems odd, I'm not sure if that's what you intended or if it's copy-paste residue.

You should be able to set GGML_CUDA_EXPS_READAHEAD_DEBUG=1 and get some feedback. expect 100% "cache miss" noise during the first prefill batch (learn phase) then "hit" afterwards.

[matteoscalabrini]

Thanks a lot for running the test @usrlocalben.

I actually think your result is a very useful second data point: +6% decode on a VERY high-bandwidth machine, with zero warmup, versus a machine that needs the 15+ minute first-touch procedure before it gets there.

“no warmup is needed since this system has an explicit numa load mechanism”

is probably the description of what this buys on hardware like yours.

I also think the difference between the gain you see and the gain I see is due to the different generations of machines.
One socket in yours has 12 channels of DDR5-4800, so +/- 460 GB/s theoretical bandwidth. Mine whole dual-socket system has 16 channels of DDR4 2666, +/- 340 GB/s aggregate.
And at 48 threads out of 192, your system should still have a lot of memory-bandwidth to give: remote traffic can disappear into that headroom without showing up as a large throughput loss, so there is simply less for the mirror to use/fix/allocate.

My machine is basically the opposite case. Im already bandwidth limited, and measure roughly a 2.1x local vs remote bandwidth ratio. That is pretty much the environment this code grew out of: necessity to squeeze all I can out of my limited hwd.

So to me the 6% on your machine and the +/- 50% on mine aren't contradictory results; they're two points on the same curve: locality matters more as available memory bandwidth gets tighter.
I believe that this is a feature for old dual-socket machines which also happens to describe a surprisingly large portion of the strange hardware people use at home to run models this big.

There is one other part to test I'd be curious to see from you which is totally speculation on my side: you already mentioned that the optimal thread count may move with the mirror, and I think that's quite plausible.

t=48 was tuned on master. There adding threads also means adding more cross-socket traffic: once both sockets have a local copy of the weights, that tradeoff changes, so there is no particular reason the optimum should stay at the same thread count. I see exactly that on my system.

Without locality, keeping 32 cores on one socket beats spreading 64 across both: (18.88 tg/s vs 14.80 tg/s) with the mirror,the 64-thread run wins at 20.81 tg/s.

Let me know what you think might be funny to see such result.

[jukofyork]

Not much to add as @usrlocalben pretty much said what I've found to be optimal:

https://reddit.com/comments/1w2hlm8/comment/p6ug4xx

This seems to work for both my dual E5-2699v4 and dual Xeon Gold 6248 systems.

At one time I got better performance by not using hypercores (either by disabling them in BIOS or via thread pinning). Recently I revisited this and found that I get significantly better PP (30%+) and marginally better TG (~10%) by using all core+hypercores, but I'm not sure if this is quant-type-specific.

[jukofyork]

This might also be of interest:

#16000 (comment)

as I think the default --numa distribute code using modulo n to distribute the threads might actually be a bad idea, and it would be better to have all consecutive thread IDs for one node followed by the next, etc.

If you do this then the ggml_compute_forward_mul_mat_id code actually simplifies to become tensor parallel, eg: for a 2-node system, the first half of each tensor will go to node0 and the second half to node1, etc.

This means there isn't actually any NUMA link cross-traffic at all!

[matteoscalabrini]

The results from my long warmup tests just landed.

You were right about the warmup.

Measured your way (llama-server, thousands of warm tokens, major faults tracked until they go quiet), fully warmed numa distribute reaches 16.9 t/s on this box and mirror reaches 16.4. On MoE decode that is a tie. The +57% in my table was distribute measured mid-warmup -- im sorry for causing confusion but it was quite difficult and time consuming to test it as I did not understand the extent the warm up needed to be until I saw your numbers this morning. The table is getting corrected for the third time.

The reason llama-bench could not see this is worth spelling out. Each invocation generates only a few hundred tokens, and the warmup tail is enormous by comparison: major faults were still trickling in at 8,000 tokens, 36M faults total, roughly 138 GB read from disk. There is a second defect on top of that. llama-bench runs pp before tg, and at batch 64 the op-offload path reads the weights from a single upload thread, so the first pages fault in an arrangement that the 64 decode threads never chose. The page cache then freezes that arrangement for every later run. Short bursts plus the wrong first toucher gives you a baseline that cannot converge inside that harness, no matter how many repetitions you run.

Dense tells the same story, and it gets there faster. One generation pass was enough to fault in the entire 16 GB of weights (4.24M major faults, then a delta of 8 on the following pass), which is what you would expect when every token touches everything. So the correction extends to dense as well: neither mode wins on steady-state throughput, on either model class.

If throughput is equal at steady state, then the case for mirror is not throughput. It is the cost of reaching that steady state and of keeping it.

Distribute pays 10 to 15 minutes of degraded serving after every cold start. That is the story-prompt ritual you described. Mirror front-loads the identical work into the replica build, about a minute on this model class, after which the first request already runs at full speed.

The "first-touch" arrangement breaks silently. A stale page cache costs 42% here, measured. A forgotten drop_caches does it, a change in thread count does it, and nothing in the output tells you it happened. Mirror has no dependency on cache state or on the order in which pages were first touched.

Thread tuning: mirror sits flat at 18.1 to 18.5 t/s across t=32/48/64, while the no-locality configs swing between 14.8 and 18.9. You mentioned in your comment how expensive it is to probe for the optimal thread count. With mirror that probe should not be necessary.

There is one more property that I think matters more than anything above, and it concerns prompt processing on GPU setups. First-touch NUMA and CUDA pinned memory are mutually exclusive. Pinning faults every page on the loader thread at load time, which destroys the first-touch placement you were trying to establish, so a distribute-based system is permanently locked out of DMA-speed uploads and pays the pageable-copy bounce on every expert upload during prefill. Mirror breaks the exclusivity: locality comes from the replicas, so the primary is free to be pinned. You get full-rate DMA prefill and node-local decode reads at the same time, without choosing. On my "production "stack the pinning half alone was worth +81% on prefill, 85 to 153 t/s on a 302 GB model. That is a production measurement, not from the A/B table in this PR. I would point out, respectfully, that this is the same problem your readahead-pool patch builds a manual solution for, with gather threads and double pinned staging buffers. Mirror gets the property by construction instead, and the price for skipping that machinery is the extra RAM. (which im not implying its a small tradeoff)

2x RAM for the CPU-resident weights, and, on a box where the operator performs the ritual correctly and can afford the warmup, no steady-state decode advantage to show for it.

The table correction is going into the PR body. @jukofyork 's independent numbers on the older Xeons would be the most informative next point, since everything above comes from one (mine) machine.

[usrlocalben]

@jukofyork Indeed w/codebook, trellis quants IQ* I see a completely different characterization wrt. threads than e.g. Q4_K. A quant like IQ3_XS can see increased throughput w/threads to the full capacity of the chip, where Q quants will reach their optimum with very few, and more or less proportional to their size/flops ratio.

SMT=on gives bizarre charts with zig-zag shaped long-tail, I leave it off.

ik_llama added PR 2202 which makes experimenting with different thread/tensor alignment very simple. I added 8 counters (counter per numa) and from there one can change the arrangement to expert parallel, row parallel or probably any layout you may want to try.

I thought that EP might be viable in conjunction with speculative, since a batch of N tokens should give ~N*n_routed_exps for better "coverage" but trying this with ik_ and GLM-5.2 didn't work out as I hoped.

If one doesn't mind the binary blob, the fastest system for a number of models (not all) is Lsglang. For the same DSv4-Flash test measured above in this thread, Lsglang gives 50-60 tok/sec decode and with MTP 75-100 or more, and the KV-caches and compute buffers are tiny compared to llama. I find it very impressive. It gives the impression of fundamental differences compared to [ik]llama, especially wrt. MTP results. For GLM-5.2 however I found prefill to be lacking and I preferred ik_llama for that model.

I see @DocShotgun trying GLM-5.3 with the DMA mod so I'll grab that and try it out sometime this week to ensure I'm giving correct advice on running it.

[matteoscalabrini]

@usrlocalben

I gave a go to Lsglang. Its definetely impressive and decode is 50% higher than what I can archive with the best tuned llama.cpp instance I have. Unfortunately prefill is a half of what i'm getting with my current llama setup, thats a shame. I tried tuning it but im quite new to sglang therefore I might have not done the best of jobs.

Worth keeping an eye on. especially for older machines with multiple nodes like mine.

I decided to continue the work on numa mirror privately and update this draft as I'm discovering something worth posting. I still remain of the idea that numa mirror is the best solution for hybrid gpu-cpu inference with cards split on 2 nodes. I'm currently tuning the Big GLM 5.3 in Q4_K_XL and while it barely fits my 500gb per node cap (mirror occupies a total of 870 gb or ram) it shows promising results especially since i have cards split across nodes which can read weights locally without crossing.

Currently getting 10.5tk/s in decode and around 150 in prefill. with 256k context at q8 and a couple of experts offloaded.