glm: keep rope models on the indexed causal attention kernel (fixes #932) - #936
Open
garnetlyx wants to merge 1 commit into
Open
glm: keep rope models on the indexed causal attention kernel (fixes #932)#936garnetlyx wants to merge 1 commit into
garnetlyx wants to merge 1 commit into
Conversation
The dense compact flash-attention prefill added in b0c31af scores the shared latent cache directly and carries no separate RoPE contribution, so it is only exact for models whose DSA cache row has no RoPE tail (GLM-5.3 Flash, n_rot == 0). On GLM-5.2 and full GLM-5.3 (n_rot == 64) the rotated RoPE term from k_rope_cache is silently dropped, and every prompt with at least glm_graph_flash_attention_prefill_min_tokens() (24) tokens gets wrong attention: output turns into fluent but unrelated text, tool calls are never emitted, and generations run to max_tokens. Gate glm_graph_use_dense_compact_attention_prefill() on DS4_N_ROT == 0 so rope models keep the indexed kernel, which adds the RoPE term. Verified on M3 Ultra (Metal), GLM-5.2-UD-Q2_K_RoutedQ2K: git bisect good=4771329 bad=<post-b0c31af>: first bad = b0c31af ./ds4 -m GLM-5.2-UD-Q2_K_RoutedQ2K.gguf --ctx 4096 --tokens 240 --temp 0 -p "Was ist 2+2? Antworte nur mit der Zahl." -> unrelated fluent garbage, differing run to run at temp 0 with this fix: "4", byte-identical across runs; prompt-length ladder (17..273 tokens) answers exactly. Fixes antirez#932
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
b0c31af("Improve GLM 5.3 attention memory and batching") added a densecompact flash-attention prefill path
(
ds4_gpu_glm_attention_dense_compact_lora_causal_tensor) that scores theshared latent cache directly with no separate RoPE contribution. That is only
exact for models whose DSA cache row has no RoPE tail — GLM-5.3 Flash
(
n_rot == 0, "Raw DSA cache row: 512 latent dimensions and no RoPE tail").On GLM-5.2 and full GLM-5.3 (
n_rot == 64), the rotated RoPE term fromk_rope_cacheis silently dropped. Every prompt with at leastglm_graph_flash_attention_prefill_min_tokens()(= 24) tokens gets wrongattention, and generation degenerates into fluent but unrelated text — this is
issue #932.
Evidence
Two independent git bisect runs (mine and GLM 5.2 Q4 on Metal (M3 Ultra) produces fluent but unrelated output after commit b0c31af #932's) land on
b0c31afwith4771329as the last good parent.Threshold matches the gate exactly: 21-token prompts behave, 25-token prompts
produce garbage (
glm_graph_flash_attention_prefill_min_tokens() == 24).Repro on M3 Ultra (Metal),
GLM-5.2-UD-Q2_K_RoutedQ2K.gguf:./ds4 -m gguf/GLM-5.2-UD-Q2_K_RoutedQ2K.gguf --ctx 4096 --tokens 240 \ --temp 0 -p "Was ist 2+2? Antworte nur mit der Zahl." # -> unrelated fluent text (differs run to run at temp 0), never "4"The indexed kernel the old path used documents the invariant in
ds4.c(glm_graph_forward_indexed_tokens): "The compact causal scoreincludes both absorbed MLA and the separate RoPE query/key contribution" —
it passes
layer_k_rope_cache+DS4_N_ROT+ rope freq params. The denseflash path passes only
layer_kv_lora_cache.Fix
Gate
glm_graph_use_dense_compact_attention_prefill()onDS4_N_ROT == 0, sorope models keep the indexed causal kernel. GLM-5.3 Flash (
n_rot == 0)behavior is unchanged; no kernel code is touched.
Verification
M3 Ultra (Metal), GLM-5.2-UD-Q2_K_RoutedQ2K:
4, byte-identical across runs (determinismrestored at temp 0).
returns exactly
LENGTH_AB_OK.DS4_GLM_DISABLE_FLASH_PREFILL=1was the interim workaround; this fix isverified equivalent without it.
Not verified locally: GLM-5.3 Flash regression (no Flash GGUF on this host);
the gate keeps its behavior identical by construction.
Fixes #932