Summary
On integrated GPUs (tested: AMD gfx1151 / Strix Halo APU, ROCm/HIP backend), llama-server can leak one request's conversation content into a later, unrelated request's responses. The KV cache gets corrupted during chunked prefill: K/V rows are scattered to wrong cells, and stale K/V from a previous request remains resident under cells the new request's attention mask marks visible. We hit this in production as verbatim fragments of a private chat appearing in another client's document-QA outputs.
Verified at v0.3.0 (build 10622, commit c1d0e7a), HIP build, -ngl 99 -b 2048 -ub 2048 -fa on -np 1.
Root cause (three links)
-
Zero-copy graph inputs on iGPUs: ggml_backend_cuda_device_supports_buft returns true for pinned-host bufts when integrated is set, so GPU kernels read graph-input tensors (k_idxs, v_idxs, kq_mask, inp_tokens, inp_pos, …) directly from host memory. On discrete GPUs these are split inputs with a protective synchronize+copy inside the sched — which is why this never reproduces on dGPUs.
-
Synchronization gated on pipeline_parallel: in llama_context::process_ubatch, the sync before set_inputs on the graph-reuse path only runs if (cparams.pipeline_parallel) — always false on a single GPU. The graph-rebuild path (ggml_backend_sched_reset + ggml_backend_sched_alloc_graph, which remap the compute buffers) has no synchronization at all.
-
Fire-and-forget prefill: graph_compute ends with ggml_backend_sched_graph_compute_async, and prefill chunks produce no outputs, so llama_decode returns without any sync and the server immediately submits the next chunk. Measured on this hardware: six 2048-token chunks submitted within 60–70 ms while each takes 0.8–1.8 s to execute (~6-deep async pipeline).
Result: ubatch N+1's set_inputs overwrites the very host buffers ubatch N's in-flight kernels are still reading. ggml_set_rows then scatters K/V rows to wrong cells; the intended cells keep stale data from a previous request, and the new request's mask marks them visible → cross-request contamination.
Reproducer / evidence
Model: Qwen3.6-35B-A3B (hybrid GDN) Q6_K_XL, llama-server --ctx-size 262144 -b 2048 -ub 2048 -fa on -np 1 --cache-reuse 1, one client at a time, deterministic sampling (temperature 0, top_k 1) for the probe requests.
Scenario: 2-turn chat about a distinctive topic X → idle → send a ~21k-token synthetic document + 4 questions. The answers quote topic X verbatim (4/4 runs, up to 5 mentions of X's key term per answer).
Instrumentation findings (per-layer FNV-1a hash of individual K-cache rows, logged at every seq_rm):
- A probe cell inside the document's chunk-2 target range holds a non-boot value planted during the chat phase — which never legitimately writes anywhere near that cell — and that value never changes while the document's 10 chunks are prefilled, despite host-side
k_idxs provably containing the correct contiguous indices for those chunks. So the same race has two faces: earlier writes scatter into wrong cells, and later writes fail to land on their intended cells.
- A control cell beyond the reach of both requests keeps its boot-time value throughout.
- Host-zeroing the attention K/V buffers between the two requests eliminates the contamination; zeroing the recurrent (GDN) state does not — the carrier is stale attention K/V, not the recurrent state.
GGML_CUDA_DISABLE_FUSION=1 reduces but does not eliminate it (slower enqueue → shallower pipeline — a timing effect, not a cause).
Fix (verified)
Make the synchronize in process_ubatch unconditional, hoisted above the can_reuse branch so it also covers the rebuild path:
// the previous graph_compute_async may still be running on the GPU. synchronize before
// mutating any graph state: set_inputs overwrites input tensors the in-flight compute
// may still be reading, and sched_reset/alloc_graph remap the compute buffers under it.
// not limited to pipeline parallelism: on integrated GPUs graph inputs are read
// zero-copy from pinned host memory.
ggml_backend_sched_synchronize(sched.get());
if (!graph_reuse_disable && res->can_reuse(gparams)) {
n_reused++;
} else {
...After the fix: 0/12 contaminated outputs across 3 runs of the reproducer (was 4/4 heavy per run), and the K-cell hash trajectory becomes healthy (chat era untouched, document data lands in its cells and stays there through all 4 questions).
Performance cost is zero on this hardware — the async pipeline only queued deeper, the GPU was already saturated:
| metric |
before |
after |
| 21k-token prefill |
1230.8 t/s |
1231.8 t/s |
| 2k-token prefill |
957–970 t/s |
957–966 t/s |
| decode |
48.5 t/s |
48.4 t/s |
Notes
Happy to submit this as a PR.
Summary
On integrated GPUs (tested: AMD gfx1151 / Strix Halo APU, ROCm/HIP backend),
llama-servercan leak one request's conversation content into a later, unrelated request's responses. The KV cache gets corrupted during chunked prefill: K/V rows are scattered to wrong cells, and stale K/V from a previous request remains resident under cells the new request's attention mask marks visible. We hit this in production as verbatim fragments of a private chat appearing in another client's document-QA outputs.Verified at v0.3.0 (build 10622, commit c1d0e7a), HIP build,
-ngl 99 -b 2048 -ub 2048 -fa on -np 1.Root cause (three links)
Zero-copy graph inputs on iGPUs:
ggml_backend_cuda_device_supports_buftreturns true for pinned-host bufts whenintegratedis set, so GPU kernels read graph-input tensors (k_idxs,v_idxs,kq_mask,inp_tokens,inp_pos, …) directly from host memory. On discrete GPUs these are split inputs with a protective synchronize+copy inside the sched — which is why this never reproduces on dGPUs.Synchronization gated on
pipeline_parallel: inllama_context::process_ubatch, the sync beforeset_inputson the graph-reuse path only runsif (cparams.pipeline_parallel)— always false on a single GPU. The graph-rebuild path (ggml_backend_sched_reset+ggml_backend_sched_alloc_graph, which remap the compute buffers) has no synchronization at all.Fire-and-forget prefill:
graph_computeends withggml_backend_sched_graph_compute_async, and prefill chunks produce no outputs, sollama_decodereturns without any sync and the server immediately submits the next chunk. Measured on this hardware: six 2048-token chunks submitted within 60–70 ms while each takes 0.8–1.8 s to execute (~6-deep async pipeline).Result: ubatch N+1's
set_inputsoverwrites the very host buffers ubatch N's in-flight kernels are still reading.ggml_set_rowsthen scatters K/V rows to wrong cells; the intended cells keep stale data from a previous request, and the new request's mask marks them visible → cross-request contamination.Reproducer / evidence
Model: Qwen3.6-35B-A3B (hybrid GDN) Q6_K_XL,
llama-server --ctx-size 262144 -b 2048 -ub 2048 -fa on -np 1 --cache-reuse 1, one client at a time, deterministic sampling (temperature 0, top_k 1) for the probe requests.Scenario: 2-turn chat about a distinctive topic X → idle → send a ~21k-token synthetic document + 4 questions. The answers quote topic X verbatim (4/4 runs, up to 5 mentions of X's key term per answer).
Instrumentation findings (per-layer FNV-1a hash of individual K-cache rows, logged at every
seq_rm):k_idxsprovably containing the correct contiguous indices for those chunks. So the same race has two faces: earlier writes scatter into wrong cells, and later writes fail to land on their intended cells.GGML_CUDA_DISABLE_FUSION=1reduces but does not eliminate it (slower enqueue → shallower pipeline — a timing effect, not a cause).Fix (verified)
Make the synchronize in
process_ubatchunconditional, hoisted above thecan_reusebranch so it also covers the rebuild path:After the fix: 0/12 contaminated outputs across 3 runs of the reproducer (was 4/4 heavy per run), and the K-cell hash trajectory becomes healthy (chat era untouched, document data lands in its cells and stays there through all 4 questions).
Performance cost is zero on this hardware — the async pipeline only queued deeper, the GPU was already saturated:
Notes
Happy to submit this as a PR.