Qwen4exp correctness fixes

[tarruda]

Overview

Fix issues found by GTP 5.6 Sol by comparing llama.cpp implementation of Qwen4Exp against transformers and vllm.

Additional information

I cannot judge if these are correct or even make sense, so I'm opening as a draft/reference that maintainers can use as they want.

Here's a summary provided by GPT 5.6 Sol. Each item in this list corresponds to one commit:

• Fix sparse-attention block selection. QSA blocks are now formed in each sequence’s logical token order, with only the incomplete tail kept unconditionally visible. Pooled keys also use the first token’s full M-RoPE position. Previously, long-context, unified-cache, multi-sequence, and multimodal inference could select the wrong history blocks, causing silent logit drift, quality loss, and cross-sequence interference.
• Support independent PLE embedding widths. PLE key/value projections are now sized from the concatenated n-gram embedding width rather than assuming it equals the main hidden width. The official model does not satisfy that assumption, so the old implementation could reject a correctly converted GGUF with tensor-shape errors or construct incorrect projections.
• Validate Qwen4Exp metadata during loading. GDN, hyper-connection, QSA, compression, and PLE invariants are checked before graph construction. Malformed or incompatible GGUFs now produce clear load errors instead of later assertions, out-of-bounds fixed-array copies, invalid tensor shapes, or possible crashes. This does not change inference for a valid official model.
• Update the QSA indexer cache during sequence copies. Indexer keys are treated as raw, unrotated data and are copied or updated alongside the attention and recurrent caches. Previously, sequence forks, prompt sharing, cache shifts, or non-unified sequence copies could leave stale indexer keys or incorrectly apply RoPE to them, making child-sequence logits diverge from an equivalent independent decode.
• Enable recurrent-state rollback. Qwen4Exp already had the required rollback implementation, but its architecture capability flag was missing, so n_rs_seq was silently disabled. This primarily affects speculative decoding: rejected draft tokens need the GDN and PLE recurrent state restored exactly. Ordinary non- speculative generation is unaffected.
• Reject tensor-split mode. Qwen4Exp does not currently implement the state and cache semantics required by -sm tensor. Master incorrectly allowed the mode. It now fails cleanly during model loading instead of reaching unsupported device placement, graph failures, or potentially invalid results. Normal layer offloading remains available

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: Yes, 100% done by GPT 5.6 Sol.

[ggerganov]

@tarruda Could you extract just the ggml-metal 2 line changes in a separate PR? If possible with a test in test-backend-ops that reproduces the overflow.

[danielhanchen]

@tarruda @ggerganov I re-checked - some fixes are fine, but some break CUDA - I also did a few more at #27941 if that helps

[tarruda]

@ggerganov done: #27944, also pushed a7fc7e4 to this branch which is something GPT 5.6 Sol found later

[tarruda]

@danielhanchen dfc7932 should address the CUDA issue.

Correct me if I'm wrong, but it seems #27941 is just doing a subset of the things that this PR does but with an alternative QSA implementation, plus the CUDA fix that I just covered with dfc7932

[ggerganov]

I think there is an issue here - with -ub 2048 it tries to allocate 128GB compute buffer. Overall the compute buffer is extremely large with this branch. Haven't looked in details, but building separate graphs for the streams is incorrect. The streams should be batched instead.

[tarruda]

@ggerganov ashamed to say I didn't test this branch in isolation, but only as a subset of the changes of this branch.

A bit more context: that other branch contains a whole lot more than just fixing qwen4exp to match transformers reference. With this PR, I've tried to get the agent to isolate only the fixes to match the transformers implementation without any of the graph/metal optimizations. Clearly it missed important pieces.

Merged master and added a fix on top, and the issue is fixed. As always, feel free to ignore or just pick what you need into separate PRs. I can also do it if it will make reviewing/merging easier.

[danielhanchen]

@tarruda Sorry on the delay - yes some fixes I repurposed and rewrote since it broke CUDA - I also added 2 more as well to #27941

[tarruda]

Assuming #27941 and other recent PRs addressed all issues, so closing