Add GLM-5.3-Flash (glm5next) support with ROCmFP mix qtypes (105/106)

[marcelormendes]

GLM-5.3-Flash Support (GLM5-Next Backend)

Adds a new glm5next backend to lucebox for the GLM-5.3-Flash model architecture.

Architecture

GLM-5.3-Flash (GGUF arch glm5next / glm5-next) features:

• 45 layers: 34 KDA linear attention + 11 NoPE sparse MLA/DSA
• Hierarchical Controller (mHC): 4 streams, 20 Sinkhorn iterations, unweighted mean collapse
• MoE: 288 routed experts (top-8) + 1 shared expert, sigmoid routing, layers 0-2 dense FFN only
• KDA: Key-Disentangled Attention with kimi-linear recurrence (conv1d, f_a/f_b, g_a/g_b gates, A_log decay)
• MLA/DSA: Multi-head Latent Attention with IndexPool sparse selection (kpool=4, index_topk=2048, NoPE)
• Clamped SwiGLU: gate (-inf, 10] → SiLU, up [-10, 10]

Implementation Status

✅ Core Backend

• ModelBackend implementation (Glm5NextBackend)
• Architecture registration in kArchCapabilities with expert offload
• Factory dispatch for glm5next and glm5-next aliases
• CMake build integration

✅ Loader (glm5next_loader.cpp)

• All GGUF hyperparameters read from glm5next.* keys
• Per-layer tensor loading (KDA: ssm_* tensors, MLA: attn_q_a/q_b, attn_wk_b/wv_b, indexer_compressor_*)
• Dense FFN tensors (layers 0-2)
• MoE tensors: moe_gate, moe_exp_probs_b, shared expert, 288 routed experts
• ROCmFP mix qtype (105/106) registration with sidecar codebooks
  • 105 (Q3_1_ROCMFP3_MIX, p4mix): down-experts only, in % 32 == 0
  • 106 (Q2_1_ROCMFP2_MIX, gumix): gate/up/down, in % 128 == 0
  • Codebooks from glm5next.{p4mix,gumix}.sidecar KV keys
  • Missing codebook fails the load (no silent fallback)
• Cleanup: Unregister mix qtypes on backend shutdown

✅ Graph Construction (glm5next_graph.cpp)

mHC (Hierarchical Controller):

• glm5next_hc_pre: RMSNorm, mix projection, pre/post gates, Sinkhorn combine matrix, weighted collapse
• glm5next_hc_sinkhorn: Iterative normalization (20 iterations, eps=1e-10)
• glm5next_hc_mean: Unweighted mean collapse (GLM terminal form, not DS4 learned output HC)
• glm5next_hc_post: Stream reconstruction from sublayer output

✅ MoE FFN (REAL routed expert execution):

• Sigmoid routing: sigmoid(moe_gate @ input + moe_exp_probs_b)
• Top-8 expert selection via ggml_top_k(router_probs, 8)
• Weight normalization
• MoeHybridStorage setup: All 288 experts on GPU (all-hot placement)
• Actual expert execution: Calls build_moe_hybrid_ffn_graph() → ggml_mul_mat_id() for gate/up/down matmuls
  • Expert matmul path: server/src/common/moe_hybrid_ffn_eval.cpp:1658-1665
  • Gate: ggml_mul_mat_id(ctx, desc.ffn_gate_exps, cur_3d, sel) (line 1658)
  • Up: ggml_mul_mat_id(ctx, desc.ffn_up_exps, cur_3d, sel) (line 1660)
  • Down: ggml_mul_mat_id(ctx, desc.ffn_down_exps, gu, sel) (line 1665)
• Clamped SwiGLU activation between gate/up and down
• Shared expert execution + combination with routed output
• Dense FFN for layers 0-2 (no experts)

KDA (Key-Disentangled Attention):

• Conv1d projections: ggml_ssm_conv for Q/K/V with d_conv=4
• Forget gate: g = gate_lower_bound * sigmoid(-ssm_a * (f_b(f_a(x)) + dt_bias))
• Beta mixing with sigmoid(beta)
• Q/K RMSNorm (eps=1e-6)
• Linear attention recurrence (simplified for single-token)
• Output gating: sigmoid(g_b(g_a(x))) * RMSNorm(out) with gate_lower_bound = -5.0
• ⚠️ Limitation: Recurrent state cache management TODO

MLA/DSA (Multi-head Latent Attention + IndexPool):

• Latent Q compression: q_a → RMSNorm → q_a_norm → q_b
• Absorbed KV: wk_b @ input, wv_b @ input
• IndexPool scoring: APE + sigmoid(indexer_gate) * K_compressed, row-wise RMSNorm
• Top-k selection: ggml_top_k(indexer_scores, 2048)
• Always-select-tail: Last kpool=4 tokens (concat logic simplified)
• Standard attention: softmax(Q @ K^T / sqrt(head_dim)) @ V over selected KV
• NoPE (no rotary position encoding)
• ⚠️ Limitation: KV cache management across layers TODO

What Executes vs. TODO

✅ Executes in Forward Pass

1. Backend initialization (Glm5NextBackend::init_hybrid_model):

   • CUDA/HIP backend setup
   • MoeHybridStorage with all-hot placement (all 288 experts GPU-resident)
   • Expert weight slicing and GPU buffer allocation

2. Graph construction (glm5next_build_graph):

   • Embedding lookup
   • 45 layers of:
     • mHC pre (Sinkhorn, weighted collapse)
     • Attention (KDA or MLA based on full_attn_interval)
     • mHC post (stream reconstruction)
     • FFN (dense or real routed MoE)
     • Residual connections
   • Final RMSNorm
   • Output head

3. MoE expert evaluation (layers 3+):

   • Router computes sigmoid probabilities
   • Top-8 expert selection per token
   • ggml_mul_mat_id kernels execute gate/up/down matmuls for selected experts
   • Clamped SwiGLU activation
   • Shared expert runs in parallel
   • Weighted combination of expert outputs

⚠️ Limitations

• KDA/MLA state caches: Single-token approximation. Full prefill/decode with recurrent state and KV cache management is future work.
• Multi-token batching: Current graph is single-token focused. Batched prefill needs attention masking and state init.
• Vision: Text-only. GLM-5.3-Flash vision encoder (mmproj) not ported.

Testing

No weights are downloaded in this PR. Validation requires:

1. A GLM-5.3-Flash GGUF (with or without ROCmFP mix qtypes)
2. Sidecar files (*.p4mix.v1 / *.gumix.s1) if using 105/106 quantization
3. Test that dflash_server accepts the GGUF, constructs the graph, and runs MoE expert execution

Reference

Ported from:

• ggml-org/llama.cpp PR #27752 (src/models/glm5next.cpp, text graph by eauchs)

Reuses from existing lucebox infrastructure:

• ModelBackend, MoeHybridStorage, MoeExpertCompute
• ROCmFP2/FP3 mix qtype registration and kernels
• mHC Sinkhorn helpers (cpu_hc_sinkhorn, deepseek4_hc_cuda.cu)
• MoE hybrid FFN evaluation (build_moe_hybrid_ffn_graph, ggml_mul_mat_id CUDA kernels)

Build

Requires HIP with gfx1100 and gfx1151 targets (existing lucebox build config).

---

Status: Draft - full graph with real routed MoE execution implemented. State cache management and multi-token prefill are future work.