From 82977da47c71d14cde141a696f8fecc213c90f33 Mon Sep 17 00:00:00 2001 From: garnetlyx Date: Tue, 1 Sep 2026 13:43:03 -0700 Subject: [PATCH] glm: keep rope models on the indexed causal attention kernel The dense compact flash-attention prefill added in b0c31af scores the shared latent cache directly and carries no separate RoPE contribution, so it is only exact for models whose DSA cache row has no RoPE tail (GLM-5.3 Flash, n_rot == 0). On GLM-5.2 and full GLM-5.3 (n_rot == 64) the rotated RoPE term from k_rope_cache is silently dropped, and every prompt with at least glm_graph_flash_attention_prefill_min_tokens() (24) tokens gets wrong attention: output turns into fluent but unrelated text, tool calls are never emitted, and generations run to max_tokens. Gate glm_graph_use_dense_compact_attention_prefill() on DS4_N_ROT == 0 so rope models keep the indexed kernel, which adds the RoPE term. Verified on M3 Ultra (Metal), GLM-5.2-UD-Q2_K_RoutedQ2K: git bisect good=4771329 bad=: first bad = b0c31af ./ds4 -m GLM-5.2-UD-Q2_K_RoutedQ2K.gguf --ctx 4096 --tokens 240 --temp 0 -p "Was ist 2+2? Antworte nur mit der Zahl." -> unrelated fluent garbage, differing run to run at temp 0 with this fix: "4", byte-identical across runs; prompt-length ladder (17..273 tokens) answers exactly. Fixes #932 --- ds4.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/ds4.c b/ds4.c index b54075539..f1935213d 100644 --- a/ds4.c +++ b/ds4.c @@ -45275,6 +45275,12 @@ static bool glm_graph_use_flash_attention_prefill(uint32_t n_tokens) { static DS4_MAYBE_UNUSED bool glm_graph_use_dense_compact_attention_prefill( uint32_t n_tokens) { + /* The dense compact path scores the shared latent cache directly and + * carries no separate RoPE contribution, so it is only exact for models + * whose DSA cache row has no RoPE tail (GLM-5.3 Flash, n_rot == 0). + * Models with n_rot == 64 (GLM-5.2, full GLM-5.3) must keep the indexed + * kernel, which adds the rotated RoPE term from k_rope_cache. */ + if (DS4_N_ROT != 0u) return false; if (!glm_graph_use_flash_attention_prefill(n_tokens)) return false; #if !defined(__APPLE__) && !defined(DS4_ROCM_BUILD) && !defined(DS4_NO_GPU) /* The CUDA GEMM setup crosses over the scalar online kernel near 256