llama: batched readahead for lazily read gather tables - #137
llama: batched readahead for lazily read gather tables#137danielhanchen wants to merge 1 commit into
Conversation
TENSOR_READ_LAZY skips the eager pull-in for a large embedding table and marks its range MADV_RANDOM. That also turns off the kernel readahead, so a sparse gather over the table costs one synchronous fault per row. Issue the reads for the rows a batch is about to gather, as one batch of MADV_WILLNEED (PrefetchVirtualMemory on Windows), from set_input. Rows are merged to whole pages first, so 16 gathers become a couple of hints. Wired into the two TENSOR_READ_LAZY users, qwen4exp and gemma4. Hints only, so results are unchanged.
Placing the per-layer embedding table, and the full measurement setCollecting everything measured around this patch, since the placement question comes up Where the PLE table actually lives
The verbose load says so directly, on UD-Q4_K_XL: 51880 MiB is 54.4 GB, the q8_0 PLE table to the byte. So the four knobs, and what each one is worth:
The reverse experiment is the informative one, because it is also the control proving
Moving 36.6 GB and 51.9 GB across the bus changes throughput by under 1% in both directions. The Why the batched readahead is needed
Note that plain Model level A/BOne binary per row group, cold cache, three reps, qwen4exp UD-IQ1_S, 26.8 GiB iq4_nl table
gemma-4-E4B-it Q4_K_M, 1.94 GB q5_K table, the model ggml-org#27794 was measured on
Lazy reading costs 7 to 9x prompt throughput on qwen4exp and 2.7x on gemma4. The patch returns gemma4 also reproduces ggml-org#27794's own finding independently: -12% tg from lazy reading there, against This is the default path, not an opt-in one
A 9.0x prompt regression out of the box. There is also no signal at default verbosity: neither So without the readahead this presents as "the model is slow", not as a flag question. Interaction with ggml-org#27837On current master, Full matrix, cold cache, three reps, one binary, UD-IQ1_S:
Three things worth drawing out. Process memory falls to 1.4 GB, 21x below the same load mode What the page cache actually does over timeThe 0.2% residency above is what a 640 token run touches, not a steady state. Nothing in this patch
Residency grows and is never dropped, but sublinearly: 2.16 GiB in the first 45 s, only +0.71 GiB
The access pattern this is built for
Not one of 4.75M consecutive pairs shared a page, which is structural: the 16 heads sit at offsets The temporal side is what makes it workable anyway: 50% of gathers are served by 1.23 GB of cache, CorrectnessThese are hints and cannot change arithmetic, but that is verified rather than asserted. Greedy Practical guidance, UD-IQ1_S (72.5 GB total: 28.8 GB PLE plus 43.7 GB weights)
Per quant tier, total / PLE / weights in GB, for sizing: What is not claimedWarm cache shows no effect. The first two reps looked like a clean 5% regression with no The "skip prefetch but do not apply MADV_RANDOM" arm was never run at model level. It needs a Partial offload numbers are upper bounds. The offload sweep on UD-IQ1_S with lazy on gives Two harness failures worth recordingThe first microbenchmark run reported this patch as worthless, 14.8 s against 13.3 s with identical Separately, the first byte identity check returned four matching md5s, all of them |
Addendum: ggml-org#27837 measured on its own, without this patchThe matrix in the previous comment was run entirely on the stacked build, so every row of it
ggml-org#27837 alone
Side by side with the stacked buildBoth builds share the same eager rows, which is the control:
RSS is identical between the two builds in both rows, 1.3 to 1.4 GB and 42.9 GB, so the readahead Correction to an earlier estimateI previously described ggml-org#27837 without this patch as landing "at about 200 t/s prompt". That was an The reason is worth stating because it is a point in ggml-org#27837's favour: ggml-org#27837 improves the lazy That does not change the conclusion, and arguably sharpens it. ggml-org#27837 makes the low memory So the two changes are complementary rather than alternatives. ggml-org#27837 decides whether the table |
How to keep the PLE table on disk while everything else stays in RAM and VRAMThe target configuration, stated plainly: every tensor except the per-layer embedding table is The command is: It needs both ggml-org#27837 and this patch. They are not alternatives and neither
For reference, the same build with the table in RAM ( Why each half is requiredggml-org#27837 decides whether the table can stay on disk at all. On master if ((flags & TENSOR_READ_LAZY) && use_mmap && tensor_read_lazy != LLAMA_TENSOR_READ_LAZY_OFF) {so // a lazy context is mapped whatever the load mode, but the memory-fit pass maps nothing
const bool is_lazy_mapped = ctx_key.lazy && !ml.no_alloc;
if ((ml.use_mmap || is_lazy_mapped) && use_mmap_buffer && buffer_from_host_ptr_supported && is_default_buft) {That is the split the configuration depends on: two contexts, two placement policies, one model. This patch decides whether that is worth doing. Lazy reading applies This patch alone changes nothing in this configuration, and that is structural rather than a What this does not require
Master also warns about this from the other side, which is worth quoting because it agrees with the If you want the weights genuinely pinned
// locking a lazy tensor would fault all of it in, which is what lazy avoids
if (lmlocks && !lazy.has(cur)) {So mlock pins everything except the PLE table, by design. I have not benchmarked that combination Recommendation todayggml-org#27837 is still open, so the configuration above is not reachable on master. Until it lands the Correctness is unaffected in every case. The readahead is a hint and cannot change arithmetic; |
Exactly which tensors this affects, and which it does notA question worth answering in the thread, because the flag name suggests something broader than Both are per-layer embedding tables, which is also why this patch has only those two call sites. tok_embd = create_tensor(tn(LLM_TENSOR_TOKEN_EMBD, "weight"), { n_embd, n_vocab }, 0);
output = create_tensor(tn(LLM_TENSOR_OUTPUT, "weight"), { n_embd, n_vocab }, TENSOR_NOT_REQUIRED);Flags The 4 GiB threshold is a filter, not a selectorThis is the part that is easy to read backwards. The size test only ever removes tensors from the if ((flags & TENSOR_READ_LAZY) && use_mmap && tensor_read_lazy != LLAMA_TENSOR_READ_LAZY_OFF) {
constexpr size_t auto_lazy_min_size = 4ull * 1024 * 1024 * 1024;
if (tensor_read_lazy == LLAMA_TENSOR_READ_LAZY_ON || ggml_nbytes(cur) > auto_lazy_min_size) {ggml-org#27837 restructures this into
An unflagged tensor is never lazy at any size. A hypothetical 5 GiB expert tensor stays fully Measured on UD-IQ1_SThe load emits exactly one If the embeddings or the head were eligible there would be three. The resulting placement, with and the three tensors in question, read out of the GGUF:
Both 341.02 MiB tensors are read in full; one sits in the pinned Note that in this particular file the only tensor over 4 GiB is also the only flagged one, so A control for the
|
| arm | lazy read enabled |
override log line | buffer for the table | VmHWM |
|---|---|---|---|---|
lazy on, no -ot |
yes | none | CPU_Mapped 27465.95 MiB |
2.04 GB |
lazy on, -ot ...=CPU |
yes | none | CPU_Mapped 27465.95 MiB |
2.04 GB |
lazy off, -ot ...=CPU |
no | buffer type overridden to CPU |
CPU 27465.95 MiB |
30.16 GB |
The third row is the point. A null result from row 2 on its own is indistinguishable from a
pattern that never matched or a flag that was misspelled. Row 3 passes the identical string and the
override line appears, so the flag, the pattern and the code path are all live. They are simply not
reached when the tensor is lazy.
Three independent signals agree: the debug line, the buffer name (CPU_Mapped is the mmap-backed
buffer_from_host_ptr path, plain CPU is an allocated buffer the data was copied into), and
28.1 GB of peak RSS, which is the table. Rows 1 and 2 differ by 484 KB, which is noise.
…161) * unsloth: repin ggml-org#25731 to the commit that merges onto b10705 edee0e1 stopped applying to the current base tag, which failed the pin preflight. Because the preflight stops at the first conflict, it also meant the two AMD fixes pinned by #160 were never dry-run merged at all. 44eb88e is the head after the upstream conflicts were resolved; the PR is MERGEABLE upstream again as of 2026-08-31T08:06Z. * unsloth: repoint the AMD pins at commits based on an upstream ancestor #157 and #158 were branched from fork master, which is right for a small PR diff and wrong for a pin. Fork master has diverged from upstream (it deletes a dozen upstream workflow files), so merging those commits onto the base tag dragged the whole divergence in and conflicted modify/delete on twelve .github/workflows files. That is what exited the preflight with no message after 'ok #137'. The pins that work are branched from an upstream commit that is an ancestor of both master and the base tag, so they carry only their own delta. Both branches are now rebased onto 11cd988 on that pattern, and verified locally to merge onto b10705 for a net +35/-3 across 4 files, which is the two fixes and nothing else. --------- Co-authored-by: danielhanchen <elliegouldingstuff@gmail.com>
* unsloth: pin ggml-org#27941, #152 and #154 ggml-org#27941 at 02eb201, no longer a draft and MERGEABLE upstream. It fixes four qwen4exp correctness defects and is the only account of the Flash-Next reporter who is on Vulkan only, where the allocator variable is never set. The previous pin PR #148 carried the stale 8161d11 and is closed. #152 at 258345e and #154 at 31e432e both merge onto b10708 on their own. #137 is already pinned at 4e1865e, which is still its head, so no change. #142 and #144 are NOT pinned here: both predate an upstream refactor that folded lazy_mode and model_shared into a lazy struct, so they conflict on src/llama.cpp and src/llama-model-loader.cpp against b10708. b10708 has 'ml.lazy.mode = params.lazy_mode' where those branches still write 'ml.lazy_mode' and 'ml.model_shared'. They need rebasing onto current upstream before they can be pinned; pinning them now would fail the resolve. * unsloth: repin ggml-org#27941 to 6b2b85c The PR moved on at 10:44Z, after 02eb201 was pinned. Verified to merge onto the current base tag: 6 files changed, 480 insertions, 138 deletions. * unsloth: pin #144, rebased, and leave #142 out as contained in it #144 rebased onto b10709 at 6fc8df1. Two of its nine commits were dropped rather than carried: 'qwen4exp: fix QSA correctness defects and harden metadata loading' and the comment tidy on top of it. That work is what ggml-org#27941 supersedes, and keeping both copies is what made the two pins conflict in llama-kv-cells.h and llama-memory-hybrid-idx.cpp. Listed after ggml-org#27941 so the upstream version of that work lands first. #142 is deliberately NOT pinned: its single commit is byte-identical to #144's c7bd6f2 apart from the lazy API spelling, so #144 already contains it and pinning both would apply the same change twice. Verified: b10709 + ggml-org#27941 + #144 merges clean, 24 files, +612/-94. * unsloth: repin ggml-org#27754 to 949f7ef The PR moved on at 10:39Z; 5796547 was pinned by #159 earlier today. Verified to merge onto b10709: 44 files changed, 2673 insertions, 38 deletions. Every entry in the set is now at its PR's current head. --------- Co-authored-by: danielhanchen <elliegouldingstuff@gmail.com>
1 participant
Batched readahead for gather tables read through
TENSOR_READ_LAZY.Based on ggml-org/llama.cpp master
ca3d5a3e1.unslothai/llama.cpp:masterdoes not carryTENSOR_READ_LAZYorqwen4expyet, so this PR targetsbase/upstream-ca3d5a3e1(that upstream commit pushed as a branch) to keep the diff to the six files this change touches.The problem
#27794 added
TENSOR_READ_LAZY, which skips the eager pull-in for a large per-layer embedding table and marks its byte rangeMADV_RANDOM. That saves real memory, butMADV_RANDOMalso switches off the kernel's own readahead, and nothing was put in its place. A sparse gather over the table then takes one synchronous fault per row.Measured on the 26.8 GiB
iq4_nlPLE table with a real 320,000 gather trace, no model and no GPU involved, median of three runs on an evicted page cache:MADV_RANDOMonly, what ships todayMADV_RANDOMplus this changeMADV_RANDOMon its own is 3.7x slower than leaving the mapping alone. This is the same effect ggml-org#27794's own gemma-4 table shows, where theMADV_RANDOMvariant scored worse than skipping the prefetch alone.The change
llama_mmap::prefetch_rows()takes the row indices a batch is about to gather, merges them to whole pages, and issues the reads as one batch:MADV_WILLNEEDon POSIX,PrefetchVirtualMemoryon Windows, no-op elsewhere.llama_model::prefetch_rows()finds the mapping holding the tensor and returns if there is none, so it is inert for anything not read out of a mapping. BothTENSOR_READ_LAZYusers call it fromset_input, which is far enough ahead of the gather for the reads to be in flight.151 insertions, 1 deletion, six files. No
ggmlchanges at all, so no new op and no backend work.No new user-facing flag. The behaviour belongs to
TENSOR_READ_LAZY, which already has--tensor-read-lazy; a second knob for the half that makes the first one usable would be the wrong shape.Results
Cold page cache, three repetitions,
llama-batched-bench -npp 512 -ntg 128 -npl 1, one B200. Four arms per repetition so thatlazy offappears on both builds: those two agree within noise throughout, which is what makes the rest of the table attributable to this change alone.qwen4exp, UD-IQ1_S, 26.8 GiB table
gemma-4-E4B-it Q4_K_M, 1.94 GB table, the model ggml-org#27794 was measured on
Prompt throughput returns to eager parity on both models while keeping the whole memory saving lazy reading buys: 28.4 GB on qwen4exp, 1.9 GB on gemma-4.
Correctness
Readahead cannot change arithmetic, and that is verified rather than assumed. Greedy generation from the base build and from this branch, same prompt, same file,
--tensor-read-lazy on, is byte-identical on both models.What this does not claim
MADV_RANDOMat all would also recover much of the loss, and is a smaller change. The gather benchmark says this is still 7.4x better than that (4,666 against 34,767 ns/gather), but that arm has not been measured at model level.Warm cache: no measurable difference either way
Once the rows a workload touches are resident, the hints buy nothing, and the question is whether they cost anything. Five repetitions, same run repeated with no eviction,
--tensor-read-lazy onon both builds:Means are 1732 against 1692 for prompt and 93.0 against 92.3 for generation, so roughly 2% and 1% in favour of the base build, but the ranges overlap and the ordering flips between repetitions. Five runs cannot resolve a difference this small. The honest statement is that warm behaviour is unchanged, with a possible small overhead from the extra
madvisecalls that would need a much longer run to measure.Worth recording how that number moved: the first two repetitions alone showed a clean 5% loss with no overlap, and three more turned it into noise.
Where the memory actually goes
The peak RSS column above is the whole process, so it is worth separating. Page cache residency after a cold run, measured per file with
mincore:Lazy reading removes essentially all of the table, and that is the whole 28.4 GB of RSS saving. What stays resident is the quantized weights, which are read through the mapping while being copied to the device and are left in page cache afterwards. That is a separate problem from this PR, and
--load-modeis the lever for it.The 0.23% figure is low because this benchmark decodes 640 tokens and touches few rows. A gather trace over 60,000 tokens of wikitext needs about 0.39 GB of page cache to serve half its gathers and 1.91 GB to serve 90%, so a long-running server settles at a few GB resident for the table rather than at zero.