Eval bug: every mrope image decode on hybrid recurrent models (qwen3_5) triggers guaranteed non-consecutive position warnings in llama_memory_recurrent; mechanism fixed by #27941 for hybrid-idx only

[lloyal-research]

Name and Version

Pinned fork at commit d6d0ce821. Before filing we checked the relevant code at current master (9d817213a): the find_slot warning block in src/llama-memory-recurrent.cpp and the memory-class selection for qwen3_5 in src/llama-model.cpp are unchanged, so everything below applies to master as well.

Operating systems

Mac

GGML backends

Metal

Hardware

Apple M2 Pro, 16 GB unified memory

Models

Qwen3.5-4B Q4_K_M + mmproj-F16.gguf (qwen3vl_merger), both from unsloth/Qwen3.5-4B-GGUF

Problem description & steps to reproduce

We embed llama.cpp directly (no llama-server). Multimodal prompts are tokenized with mtmd and image batches get their positions from mtmd_helper_image_get_decoder_pos, the same way mtmd-helper.cpp builds them. The model is hybrid (full attention layers + gated delta net layers), so decodes go through llama_memory_hybrid, which feeds llama_memory_recurrent.

Every single image prefill produces a burst of warnings from llama_memory_recurrent::find_slot:

find_slot: non-consecutive token position 9 after 8 for sequence 0 with 512 new tokens
find_slot: non-consecutive token position 9 after 8 for sequence 0 with 512 new tokens
find_slot: non-consecutive token position 9 after 9 for sequence 0 with 512 new tokens
find_slot: non-consecutive token position 9 after 9 for sequence 0 with 512 new tokens
find_slot: non-consecutive token position 48 after 9 for sequence 0 with 8 new tokens
find_slot: non-consecutive token position 48 after 9 for sequence 0 with 8 new tokens

This is deterministic and, as far as we can tell, structural: mrope gives all rows of an image the same temporal position and the following text resumes at pos + max(nx, ny), while find_slot checks last_pos == cell.pos + n_seq_tokens per sequence. So any mrope image on any hybrid model trips the check on every ubatch of the image plus the first text ubatch after it. The comment above the warning already asks what should happen here; currently cell.pos is overwritten and decoding continues.

Two things made us file this as its own issue rather than only a +1 on #27931:

1. Eval bug: llama-server crashes (stack overflow / access violation) on hybrid recurrent models (qwen3_5_moe) with mmproj vision when alternating text and image turns #27931 reports the same warning signature escalating to crashes, on CUDA, with a qwen3_5_moe model, through llama-server. We see the identical storm on Metal, with a dense 4B, through direct API use. So the warning half at least is not server-specific, not CUDA-specific and not moe-specific.
2. qwen4exp: follow up fixes #27941 appears to have fixed this exact mechanism, but only in llama-memory-hybrid-idx (one of its changes is described as ranking cells instead of using the position because mrope repeats one position across an image). qwen3_5-class models still select plain llama_memory_hybrid + llama_memory_recurrent, so they keep the unfixed path.

Possibly related, reported for pattern matching only: one long session (single image at the start, then ~1300 tokens of text generation) died about 280 s after the warning burst with kIOGPUCommandBufferCallbackErrorOutOfMemory, after which the backend stayed in its error state and every later llama_decode returned -3 until process restart. The box was under heavy memory load from unrelated processes at the time, so we cannot cleanly attribute that crash to this issue. It does resemble the "eventually crashes the same way" long-session mode described in #27931.

We are happy to help debug: we can produce Metal builds with debug symbols, run with MTL_DEBUG_LAYER or GGML asserts enabled, capture lldb backtraces or heap snapshots if we manage to catch a crash, and rerun any candidate patch against our reproduction. The warning burst itself reproduces in seconds on our side.

First Bad Commit

Not a regression as far as we can tell. Present at d6d0ce821 and the code is identical at master 9d817213a.

Relevant log output

find_slot: non-consecutive token position 9 after 8 for sequence 0 with 512 new tokens
find_slot: non-consecutive token position 9 after 9 for sequence 0 with 512 new tokens
find_slot: non-consecutive token position 48 after 9 for sequence 0 with 8 new tokens
...
ggml_metal_synchronize: error: command buffer 1 failed with status 5
error: Insufficient Memory (00000008:kIOGPUCommandBufferCallbackErrorOutOfMemory)
ggml_metal_graph_compute: backend is in error state from a previous command buffer failure - recreate the backend to recover
graph_compute: ggml_backend_sched_graph_compute_async failed with error -1
llama_decode: failed to decode, ret = -3