Skip to content

glm: keep rope models on the indexed causal attention kernel (fixes #932) - #936

Open
garnetlyx wants to merge 1 commit into
antirez:mainfrom
garnetlyx:fix/glm-dense-compact-flash-rope
Open

glm: keep rope models on the indexed causal attention kernel (fixes #932)#936
garnetlyx wants to merge 1 commit into
antirez:mainfrom
garnetlyx:fix/glm-dense-compact-flash-rope

Conversation

@garnetlyx

Copy link
Copy Markdown

Summary

b0c31af ("Improve GLM 5.3 attention memory and batching") added a dense
compact flash-attention prefill path
(ds4_gpu_glm_attention_dense_compact_lora_causal_tensor) that scores the
shared latent cache directly with no separate RoPE contribution. That is only
exact for models whose DSA cache row has no RoPE tail — GLM-5.3 Flash
(n_rot == 0, "Raw DSA cache row: 512 latent dimensions and no RoPE tail").

On GLM-5.2 and full GLM-5.3 (n_rot == 64), the rotated RoPE term from
k_rope_cache is silently dropped. Every prompt with at least
glm_graph_flash_attention_prefill_min_tokens() (= 24) tokens gets wrong
attention, and generation degenerates into fluent but unrelated text — this is
issue #932.

Evidence

  • Two independent git bisect runs (mine and GLM 5.2 Q4 on Metal (M3 Ultra) produces fluent but unrelated output after commit b0c31af #932's) land on b0c31af with
    4771329 as the last good parent.

  • Threshold matches the gate exactly: 21-token prompts behave, 25-token prompts
    produce garbage (glm_graph_flash_attention_prefill_min_tokens() == 24).

  • Repro on M3 Ultra (Metal), GLM-5.2-UD-Q2_K_RoutedQ2K.gguf:

    ./ds4 -m gguf/GLM-5.2-UD-Q2_K_RoutedQ2K.gguf --ctx 4096 --tokens 240 \
          --temp 0 -p "Was ist 2+2? Antworte nur mit der Zahl."
    # -> unrelated fluent text (differs run to run at temp 0), never "4"
  • The indexed kernel the old path used documents the invariant in
    ds4.c (glm_graph_forward_indexed_tokens): "The compact causal score
    includes both absorbed MLA and the separate RoPE query/key contribution"

    it passes layer_k_rope_cache + DS4_N_ROT + rope freq params. The dense
    flash path passes only layer_kv_lora_cache.

Fix

Gate glm_graph_use_dense_compact_attention_prefill() on DS4_N_ROT == 0, so
rope models keep the indexed causal kernel. GLM-5.3 Flash (n_rot == 0)
behavior is unchanged; no kernel code is touched.

Verification

M3 Ultra (Metal), GLM-5.2-UD-Q2_K_RoutedQ2K:

  • The repro above now answers 4, byte-identical across runs (determinism
    restored at temp 0).
  • Prompt-length ladder (17/21/25/29/33/81/273 tokens, temp 0): every length
    returns exactly LENGTH_AB_OK.
  • DS4_GLM_DISABLE_FLASH_PREFILL=1 was the interim workaround; this fix is
    verified equivalent without it.

Not verified locally: GLM-5.3 Flash regression (no Flash GGUF on this host);
the gate keeps its behavior identical by construction.

Fixes #932

The dense compact flash-attention prefill added in b0c31af scores the
shared latent cache directly and carries no separate RoPE contribution,
so it is only exact for models whose DSA cache row has no RoPE tail
(GLM-5.3 Flash, n_rot == 0). On GLM-5.2 and full GLM-5.3 (n_rot == 64)
the rotated RoPE term from k_rope_cache is silently dropped, and every
prompt with at least glm_graph_flash_attention_prefill_min_tokens()
(24) tokens gets wrong attention: output turns into fluent but
unrelated text, tool calls are never emitted, and generations run to
max_tokens.

Gate glm_graph_use_dense_compact_attention_prefill() on DS4_N_ROT == 0
so rope models keep the indexed kernel, which adds the RoPE term.

Verified on M3 Ultra (Metal), GLM-5.2-UD-Q2_K_RoutedQ2K:

  git bisect good=4771329 bad=<post-b0c31af>: first bad = b0c31af
  ./ds4 -m GLM-5.2-UD-Q2_K_RoutedQ2K.gguf --ctx 4096 --tokens 240         --temp 0 -p "Was ist 2+2? Antworte nur mit der Zahl."
    -> unrelated fluent garbage, differing run to run at temp 0
  with this fix: "4", byte-identical across runs; prompt-length
  ladder (17..273 tokens) answers exactly.

Fixes antirez#932
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

GLM 5.2 Q4 on Metal (M3 Ultra) produces fluent but unrelated output after commit b0c31af

1 participant