GLM5next Production Readiness: Optional Mix Qtypes + Cache Allocation

[marcelormendes]

This PR implements production-ready GLM-5.3-Flash support in lucebox with real KDA/MLA cache, dual-GPU placement, and full decode loop.

Architecture

GLM-5.3-Flash (45 layers):

• 34 × KDA linear attention (kimi-linear SSM)
• 11 × MLA sparse attention (IndexPool DSA, kpool=4, top-k 2048)
• mHC hierarchical controller (4 streams, Sinkhorn normalization, unweighted mean collapse)
• Dense FFN layers 0-2
• MoE from layer 3: 288 routed experts top-8 + 1 shared expert
• Clamped SwiGLU activation

Implementation Progress

UNIT 1: Real Cache R/W ✅ (commits 4aeb17a, 9fc1ad3)

KDA Recurrent State:

• glm5next_kda_attention reads state_prev from cache.kda_state at layer index
• Computes recurrence: state_new = g * state_prev + (1 - beta) * (k @ v^T)
• Writes state_new back to cache via ggml_cpy into ggml_view_3d
• Fixed dimension: kda_state allocated [head_dim, n_head, n_kda_layers] (was n_embd)

MLA KV Cache:

• glm5next_mla_attention appends new K/V to cache at cur_pos via ggml_cpy
• Reads full cached context [0:cur_pos+n_tokens] for attention
• IndexPool/DSA operates over full cache, not just current token
• APE and top-k selection updated for cached context

Graph:

• Local mla_layer_idx/kda_layer_idx counters (was static globals - concurrent safety)
• cache.cur_pos incremented after each decode step
• Dummy/single-token paths deleted

UNIT 2: Dual-GPU Placement ✅ (commit 2d9062e)

Device 0 (HIP gfx1100 RX 7900 XT, ~20 GiB) - Hot Path:

• tok_embd, output, output_norm
• All attention: KDA (ssm_*), MLA (attn_q/k/v, indexer_*)
• mHC tensors (hc_attn/ffn_*)
• Dense FFN layers 0-2
• Caches (cache.k/v, cache.kda_state)
• MoE router (moe_gate, moe_exp_probs_b)
• Shared expert (small)

Device 1 (HIP gfx1151 Strix Halo, 128 GiB UMA) - Routed Experts:

• All 288 expert stacks: moe_experts_{gate,up,down}
• Via MoeHybridStorage cold_expert_backend=Gpu

Runtime Detection:

• Tries ggml_backend_cuda_init(1) for expert backend
• Success → dual-GPU (hot/cold split)
• Fail → single-GPU fallback (all-hot on device 0)

Placement:

• Dual-GPU: hot_expert_ids empty, all experts in cold (device 1)
• Single-GPU: hot_expert_ids=all, no cold materialization

Reuses existing MoeHybridStorage / build_moe_hybrid_storage with cold_gpu_backend parameter. No DS4 stem/hash/DSpark kernels copied.

UNIT 3: Real Sampling ✅ (commit ab43c0c)

Prefill:

• Processes all prompt tokens in one batched graph
• Reads logits from last token position via ggml_backend_tensor_get
• Updates cache.cur_pos += prompt_len

Decode Loop (mirrors DeepSeek4):

• Samples next token: greedy argmax or sample_logits() with temp/penalties
• Emits token via io.emit()
• Checks for EOS (token 2) and breaks
• Builds graph for single token with sampled next_token
• Computes graph, reads logits, updates cache.cur_pos
• Loops until n_gen or EOS

Sampler Integration:

• Copies req.sampler to sampler_ member
• Seeds sampler_rng_ if do_sample
• Builds history for penalty processing when needed
• Calls sample_logits() from common/sampler.h

Result: n_gen and token stream are now real model output, not constant EOS placeholder.

Known Remaining Work

• MLA always_select_tail is TODO (noted, not blocking)
• Timing metrics (currently zeroed)

Testing

No weight downloads. Target artifact: GLM-5.3-Flash-skiplist.gguf (stock ggml qtypes Q2_K/Q8_0/Q6_K/F32, no mix sidecar). ROCmFP mix qtypes (105/106) remain optional.

Success criteria:

• ✅ Factory dispatches glm5next / glm5-next
• ✅ Init does not abort on stock ggml GGUF (no gumix sidecar)
• ✅ KDA and MLA keep real recurrent/KV state across tokens
• ✅ Dual-GPU placement: hot tensors device 0, routed experts device 1
• ✅ Real multi-token generation with sampling
• ✅ Build targets gfx1100 + gfx1151

Status: Draft - awaiting operator validation on target hardware.