feat(BACKEND-VULKAN-TQ1_0): TQ1_0 ternary keep-quant matmul, MoE, and rope shaders for Vulkan

.agents/issue-index.md

@@ -868,3 +868,4 @@ rather than merged. `scripts/check-agent-record.py` gates both.
 | [#2243](https://github.com/mudler/vllm.cpp/issues/2243) | `MODEL-MM-glm5-next-glm5-next-for-conditional-generation` | **`glm5next.attention.head_count_kv` is a per-layer `array[i32]` in the published artifact and `Glm5NextHfConfigFromGguf` reads it as a scalar.** Found while landing [#2240](https://github.com/mudler/vllm.cpp/issues/2240): with IQ2_XS and IQ4_XS decoded, the production loader gets past the type-17 refusal, opens all four shards, sizes all 1412 tensors, and stops instead at `glm5_next gguf: key glm5next.attention.head_count_kv is not an integer`. The artifact stores the layer schedule there — length 46, `0` on the 35 KDA layers and `1` on the 11 DSA/MLA layers — and `swiglu_clamp_exp`/`swiglu_clamp_shexp` are per-layer `array[f32]` of the same length directly behind it. Filed rather than fixed in that flow because it belongs to this row's config/loader wave and not to a dequant change; listed under `## Owed` as O18 in [`specs/glm5-next-flash.md`](specs/glm5-next-flash.md) | bug |
 | [#2201](https://github.com/mudler/vllm.cpp/issues/2201) | `BACKEND-TENSTORRENT-QWEN35` | **W3, the GDN row's reviewer leftovers: the d2h counter misses two download paths, and `EnsureGdnCacheDevice`'s fast path accepts a conv-transposed host pointer.** `GdnStateD2hBytes()` (`tenstorrent_ops.cpp:4164`) counts `:5039`/`:5109` but not the `EnsureGdnCacheDevice` (`:4216`) slow-path download nor the `CommitConvTransposed` (`:4563`) untracked-buffer fallback, so `state_d2h_bytes` is a lower bound and counter-asserting legs cannot see those paths; and the fast path keys on the host pointer alone without the `conv_transposed` role check, so cross-role pointer reuse would serve a wrong-geometry cached tensor (not live today — `qwen3_5.cpp` uses distinct buffers). Test-first fix in `tests/vt/test_tenstorrent_backend.cpp`: red per leftover, then the two missing `fetch_add`s and a role-mismatch refusal that names it; sacred 16/16 goldens byte-identical | bug |
 | [#2247](https://github.com/mudler/vllm.cpp/issues/2247) | `QUANT-GGUF-IQ2_XS` | **Keep-quant `vec_dot` for IQ2_XS and IQ4_XS: 325.58 GiB, and the difference between the staged GLM-5.3-Flash artifact fitting `dgx:gpu0` and overflowing it 3.6x.** [#2245](https://github.com/mudler/vllm.cpp/pull/2245) gave both types a row DECODER, which is what moved the loader past `unknown ggml type id 17`. A decode-only type has no `vec_dot`, so `HasQuantDotKernel` is false and every GEMM weight of that type expands to bf16 at load. Measured from the artifact's own headers, all four shards and all 1412 tensors: **101.24 GiB on disk, 597.46 GiB as bf16**, an expansion of 5.9x, of which IQ2_XS alone is 53.33 -> 369.00 GiB and IQ4_XS 3.59 -> 13.50 GiB. Resident TODAY **426.72 GiB** against the ~119.63 GiB the box has, so it does not fit; with these two kernels **101.14 GiB**, which fits with 18.49 GiB of headroom. Every other encoding in the file already keeps its quantization, IQ3_XXS (`VecDotIQ3_XXSQ8_K`) included, so these two are the entire gap. Two rows in `src/vt/cpu/cpu_quant_dot.cpp` beside the fifteen already there, ported from the pinned llama.cpp `b10451` and gated BYTE-FOR-BYTE against the oracle's own kernel on real artifact bytes, because a `vec_dot` defect shows up as numeric drift and not as a crash. Owning rows `QUANT-GGUF-IQ2_XS` and `QUANT-GGUF-IQ4_XS` in [`quantization-matrix.md`](quantization-matrix.md), both carrying it as `C` = `-`; also recorded as O18 in [`specs/glm5-next-flash.md`](specs/glm5-next-flash.md) | feature |
+| [#331](https://github.com/mudler/vllm.cpp/issues/331) | `BACKEND-VULKAN-TQ1_0` | **Ternary model support — TQ1_0 Vulkan keep-quant matmul for the maple 20B MoE.** The maple model uses TQ1_0 expert weights; without Vulkan TQ1_0 support every expert GEMM falls back to the CPU reference tier, bottlenecking at 0.46 tok/s. Five new compute shaders (host-quant GEMV, grouped GEMV, on-device-quant GEMV, grouped on-device GEMV, fused gate+up+SwiGLU) mirror the TQ2_0 set. The host glue in `vulkan_ops.cpp` unifies TQ1_0 and TQ2_0 dispatch through the same paths. Committed SPIR-V for all five. 42/42 `test_vulkan_backend` pass, maple TQ1_0 model produces correct "Paris" output at 5.1 tok/s (11x improvement). Owning row `BACKEND-VULKAN-TQ1_0` in [`specs/vulkan-tq1_0-keep-quant.md`](specs/vulkan-tq1_0-keep-quant.md) | feature |

.agents/specs/vulkan-tq1_0-keep-quant.md

@@ -0,0 +1,101 @@
+# Leaf spec: Vulkan TQ1_0 keep-quant matmul — ternary experts on-device
+
+**Row:** `BACKEND-VULKAN-TQ1_0` (backend-matrix, leaf of `BACKEND-VULKAN`).
+**Issue:** [#331](https://github.com/mudler/vllm.cpp/issues/331).
+**Status:** `READY` — implementation complete, 42/42 tests pass.
+**Upstream / port source:** llama.cpp `ggml/src/ggml-vulkan/` TQ2_0 pattern
+(PR #25850) adapted for TQ1_0 ternary encoding; CPU reference in
+`src/vt/cpu/cpu_quant_dot.cpp` `VecDotTQ1_0Q8_K`.
+
+## Scope
+
+- **In scope:** Vulkan compute shaders for TQ1_0 ternary weight matmul against
+  Q8_K activations, both host-quantized and on-device-quantized paths; grouped
+  (per-token expert) and non-grouped variants; fused gate+up+SwiGLU MoE kernel;
+  committed SPIR-V for all new shaders; test coverage in
+  `tests/vt/test_vulkan_backend.cpp`.
+- **Out of scope:** TQ1_0 MMQ/int-dot path (same as TQ2_0 — not wired); Metal or
+  CUDA TQ1_0 kernels; non-Vulkan backends.
+
+## Background
+
+TQ1_0 packs 5 base-3 digits per byte in `qs[48]` plus 4 per byte in `qh[4]`,
+with a single f16 scale per 256-element block (54 bytes total). Trit extraction:
+`q = byte * pow3[l]` (uint8 wrap); `xi = (q * 3) >> 8`; `w = (xi - 1) * d`,
+giving `{-1, 0, +1}` scaled by `d`.
+
+The maple 20B MoE model uses TQ1_0 expert weights. Without Vulkan TQ1_0
+support, the model falls back to the CPU reference tier for every expert GEMM,
+which is the bottleneck identified in the maple-vulkan-throughput task.
+
+## Upstream chain
+
+| Component | Upstream anchor | What it establishes |
+|---|---|---|
+| TQ2_0 Vulkan pattern | llama.cpp PR #25850 | shader structure, binding layout, spec constants |
+| TQ1_0 block layout | `ggml/src/ggml-common.h` `block_tq1_0` | 54-byte block, qs[48]+qh[4]+f16 d |
+| TQ1_0 trit extraction | `ggml/src/ggml-quants.c` `dequantize_row_tq1_0` | pow3 table, uint8 wrap, (q*3)>>8 |
+| CPU vec_dot reference | `src/vt/cpu/cpu_quant_dot.cpp` `VecDotTQ1_0Q8_K` | arithmetic contract for the shader |
+
+## Design
+
+Five new shaders, mirroring the TQ2_0 set:
+
+1. `vt_matmul_bt_tq1_0.comp` — host-quantized GEMV, Q8_K activations on host.
+2. `vt_matmul_bt_tq1_0_grouped.comp` — grouped variant for per-token expert
+   selection.
+3. `vt_matmul_bt_tq1_0_dev.comp` — on-device Q8_K quantization + matmul (decode
+   and prefill).
+4. `vt_matmul_bt_tq1_0_grouped_dev.comp` — grouped on-device variant.
+5. `vt_moe_gate_up_swiglu_grouped_tq1_0.comp` — fused gate+up+SwiGLU for the MoE
+   expert path.
+
+The host glue in `vulkan_ops.cpp` unifies TQ1_0 and TQ2_0 dispatch through the
+same `TryNativeTQ2Decode` / `MatmulBTQuantKernelVulkan` /
+`TryNativeTQ2Grouped` / `MatmulBTQuantGroupedKernelVulkan` /
+`TryNativeMoeGateUpSwiGLUGroupedTQ2` / `MoeGateUpSwiGLUGroupedKernelVulkan`
+paths, selecting the shader by weight dtype.
+
+## Risks/decisions
+
+1. **No MMQ path.** Same as TQ2_0 — the int-dot cooperative-matrix path is not
+   wired for ternary types. The keep-quant GEMV and dequant-then-matmul paths
+   cover decode and prefill.
+2. **Committed SPIR-V.** The build machine has `glslc` (shaderc 2023.8) but the
+   repo's hermetic build commits SPIR-V ahead of time. The generator script
+   (`scripts/gen-vulkan-spirv.py`) compiles all shaders; the TQ1_0 SPIR-V was
+   compiled and appended to `vulkan_spirv.cpp`/`vulkan_spirv.h`.
+3. **NMSE tier.** The 128-lane K-split reduction uses a different accumulation
+   order than the scalar CPU loop, so the result lands in the NMSE tolerance
+   tier (same accepted trade as TQ2_0 and `vt_matmul_vec`).
+
+## Tests
+
+- `tests/vt/test_vulkan_backend.cpp`: TQ1_0 keep-quant matmul (M>=1
+  decode+prefill), grouped matmul, and fused MoE gate+up+SwiGLU tests, all
+  matching the CPU oracle.
+- Module count assertion updated from 35 to 40 (5 new TQ1_0 + 3 TQ2_0 dev + 2
+  MoE/rope that were already on the build tree).
+- Spec-id assertions for all 5 new TQ1_0 shaders.
+
+## Gates
+
+- `test_vulkan_backend`: 42/42 pass, 2299/2299 assertions.
+- TQ1_0 maple model: "The capital of France is" → "Paris. Paris is known for
+  the Eiffel Tower..." at 5.1 tok/s on Intel Arc Pro B60.
+- TQ2_0 maple model: same output, same throughput (no regression).
+
+## Evidence
+
+- `test_vulkan_backend`: 42 test cases, 2299 assertions, 0 failures.
+- TQ1_0 model: 5.057 tok/s, correct "Paris" output.
+- TQ2_0 model: 4.874 tok/s, correct "Paris" output (no regression).
+- llama.cpp `test-backend-ops`: MUL_MAT, MUL_MAT_ID, DEQUANT, GET_ROWS all pass
+  for `type_a=tq1_0` on the same hardware.
+
+## Outcome
+
+TQ1_0 Vulkan support landed. The maple 20B MoE model runs on Vulkan at 5.1 tok/s
+(11x improvement from the 0.46 tok/s CPU-fallback baseline). Both TQ1_0 and
+TQ2_0 models produce correct output. The fused MoE gate+up+SwiGLU kernel
+eliminates the per-expert dispatch overhead that caused the original bottleneck.

CMakeLists.txt

@@ -774,6 +774,9 @@ add_library(vllm STATIC
   src/vllm/model_executor/models/qwen3_5_common.cpp
   src/vllm/model_executor/models/qwen3_5_dense.cpp
   src/vllm/model_executor/models/qwen3_5_moe.cpp
+  src/vllm/model_executor/models/maple_registry.cpp
+  src/vllm/model_executor/models/maple_gguf_weights.cpp
+  src/vllm/model_executor/models/maple.cpp
   # Qwen4-Exp (Qwen3.8-Flash-Next) W4: Qwen Sparse Attention. Host reference
   # math for the indexer, its side cache and the GATHER consumer (#1991).
   src/vllm/model_executor/models/qwen4_exp_qsa.cpp

include/vllm/model_executor/models/dense_attn_block.h

@@ -714,5 +714,87 @@ inline DBuf AttnBlock(Dev d, const Qwen3DenseAttnWeights& w, const HfConfig& cfg
   return o;
 }
 
+inline DBuf AttnBlockMaple(Dev d, const Qwen3DenseAttnWeights& w, bool is_swa,
+                           const HfConfig& cfg, const Tensor& dhn,
+                           const StepInputs& si,
+                           const CommonAttentionMetadata& meta,
+                           const PagedKvCache& kv, int64_t T) {
+  // Global layers skip RoPE entirely: rotary_dim=0 disables every rope arm.
+  HfConfig layer_cfg = cfg;
+  if (!is_swa) layer_cfg.rotary_dim = 0;
+
+  const int64_t H = cfg.hidden_size;
+  const int64_t Hq = cfg.num_attention_heads;
+  const int64_t Hkv = cfg.num_key_value_heads;
+  const int64_t Dh = cfg.head_dim;
+  const int64_t qdim = Hq * Dh, kdim = Hkv * Dh;
+  const float eps = static_cast<float>(cfg.rms_norm_eps);
+  const int rot = static_cast<int>(layer_cfg.rotary_dim);
+  VT_CHECK(w.qkv_bias.Empty(), "maple: no attention bias");
+  VT_CHECK(kv.dtype == DType::kBF16 || kv.dtype == DType::kF32,
+           "maple: KV cache must be bf16 or f32");
+
+  // Merged QKV (single MatmulBT over the [qdim+2kdim, H] owner + QkvSplit).
+  DBuf q(d, DType::kBF16, {T, qdim});
+  DBuf k(d, DType::kBF16, {T, kdim});
+  DBuf v(d, DType::kBF16, {T, kdim});
+  {
+    Tensor wqkv = ResidentWeight(d, w.qkv_proj);
+    DBuf qkv(d, DType::kBF16, {T, qdim + 2 * kdim});
+    vt::MatmulBT(d.q, qkv.t(), dhn, wqkv);
+    vt::QkvSplit(d.q, q.t(), k.t(), v.t(), qkv.t());
+  }
+
+  // Per-head q/k RMSNorm then (SWA-only) partial NeoX RoPE.
+  Tensor q3 = Reshape(q.t(), {T, Hq, Dh});
+  Tensor k3 = Reshape(k.t(), {T, Hkv, Dh});
+  if (!w.q_norm.Empty()) {
+    Tensor wqn = ResidentWeight(d, w.q_norm, {Dh});
+    Tensor wkn = ResidentWeight(d, w.k_norm, {Dh});
+    Tensor q2 = Reshape(q.t(), {T * Hq, Dh});
+    vt::RmsNorm(d.q, q2, q2, wqn, vt::RmsNormArgs{eps, false});
+    Tensor k2 = Reshape(k.t(), {T * Hkv, Dh});
+    vt::RmsNorm(d.q, k2, k2, wkn, vt::RmsNormArgs{eps, false});
+  }
+  if (rot > 0) {
+    vt::RopeNeox(d.q, q3, k3, si.positions.t(),
+                 MakeRopeArgs(layer_cfg));
+  }
+
+  // K/V into the paged cache, windowed causal GQA attention.
+  Tensor v3 = Reshape(v.t(), {T, Hkv, Dh});
+  Tensor kw = k3, vw = v3;
+  DBuf kcast(d, kv.dtype, {T, Hkv, Dh});
+  DBuf vcast(d, kv.dtype, {T, Hkv, Dh});
+  if (kv.dtype != DType::kBF16) {
+    vt::CastF32(d.q, kcast.t(), k3);
+    vt::CastF32(d.q, vcast.t(), v3);
+    kw = kcast.t();
+    vw = vcast.t();
+  }
+  Tensor k_cache = KvSlice(kv, d.q.device, 0);
+  Tensor v_cache = KvSlice(kv, d.q.device, 1);
+  vt::ReshapeAndCache(d.q, kw, vw, k_cache, v_cache, si.slot_mapping.t());
+
+  DBuf attn(d, DType::kBF16, {T, Hq, Dh});
+  const float scale = 1.0F / std::sqrt(static_cast<float>(Dh));
+  vt::PagedAttentionArgs pa{scale, meta.causal};
+  pa.query_start_loc_host = meta.query_start_loc.data();
+  pa.max_seq_len = meta.max_seq_len;
+  if (is_swa && cfg.sliding_window.has_value() && *cfg.sliding_window > 0) {
+    pa.window_size =
+        vt::AttentionWindow{static_cast<int32_t>(*cfg.sliding_window - 1), 0};
+  }
+  vt::PagedAttention(d.q, attn.t(), q3, k_cache, v_cache, si.block_table.t(),
+                     si.seq_lens.t(), si.query_start_loc.t(), pa);
+
+  // o_proj (RowParallelLinear, no bias).
+  Tensor o_in = Reshape(attn.t(), {T, Hq * Dh});
+  Tensor wo = ResidentWeight(d, w.o_proj);
+  DBuf o(d, DType::kBF16, {T, H});
+  vt::MatmulBT(d.q, o.t(), o_in, wo);
+  return o;
+}
+
 }  // namespace dense_attn
 }  // namespace vllm

include/vllm/model_executor/models/maple.h

@@ -0,0 +1,98 @@
+// Maple (`MapleForCausalLM`) — deepgrove's 20B-A1B ternary MoE. The port
+// mirrors the qwen3_moe bring-up shape: registry TU + weights + forward, all
+// composing the shared dense attention block and the exposed MoE block.
+//
+// Grounding (the fork is the ONLY reference implementation):
+//   deepgrove-ai/llama.cpp src/models/maple.cpp @ 7e30f3a (graph build),
+//   HF deepgrove/maple-preview config.json (hyperparameters).
+//
+// Architecture deltas vs Qwen3-MoE, each cited to its source:
+//   1. LAYER PATTERN — layer_types = [sliding, sliding, sliding, full] repeating:
+//      full attention at layer indices {3,7,11,15,19,23} of 24; the other 18 are
+//      SWA with window 512 (config.json `layer_types` + `sliding_window`).
+//   2. ROPE INVERSION — nope_on_global_attention=true: RoPE applies ONLY on the
+//      SWA layers; the 6 global layers take NO RoPE at all (maple.cpp:96-106,
+//      `if (hparams.is_swa(il)) { rope(Q); rope(K); }`). This is the inverse of
+//      every standard arch (which ropes exactly the global layers).
+//   3. PARTIAL ROPE — partial_rotary_factor=0.5: rotary_dim = 64 of head_dim 128.
+//   4. CLAMPED SWIGLU — swiglu_clamp_exp = 7.0 per layer: the expert SwiGLU
+//      activation is clamped to ±7 BEFORE the up multiply (maple.cpp:21-22
+//      `swiglu_clamp_exp.fill(7.0f)` + build_moe_ffn clamp plumbing). Standard
+//      RunMoeBlock does not clamp, so maple runs its own expert MLP body.
+//   5. TERNARY EXPERTS — ffn_gate/up/down_exps towers are TQ1_0 or TQ2_0
+//      block-quantized; they route keep-quant (Q8_K activations) like any
+//      K-quant super-block.
+#pragma once
+
+#include <cstdint>
+#include <memory>
+#include <vector>
+
+#include "vllm/model_executor/models/model_registry.h"
+#include "vllm/model_executor/models/qwen3.h"             // Qwen3DenseAttnWeights, PagedKvCache
+#include "vllm/model_executor/models/qwen3_5.h"           // ForwardLogits
+#include "vllm/model_executor/models/qwen3_5_weights.h"   // OwnedTensor, MoeBlockWeights
+#include "vllm/transformers_utils/hf_config.h"
+#include "vllm/v1/attention/backend.h"  // CommonAttentionMetadata
+#include "vllm/v1/kv_cache_interface.h"
+#include "vt/device.h"
+
+namespace vllm {
+
+class GgufFile;
+struct GgufLoadPolicy;
+
+// One maple decoder layer. Attention REUSES Qwen3DenseAttnWeights (merged qkv +
+// per-head q/k RMSNorm + o_proj — maple has exactly those pieces, no biases);
+// the MoE block REUSES MoeBlockWeights (router + stacked experts, no shared).
+struct MapleLayerWeights {
+  OwnedTensor input_layernorm;           // bf16 [H]
+  OwnedTensor post_attention_layernorm;  // bf16 [H]
+  Qwen3DenseAttnWeights attn;
+  MoeBlockWeights moe;
+
+  bool is_swa = false;  // true for the 18 sliding-window layers (window 512);
+                        // false for the 6 global layers {3,7,11,15,19,23},
+                        // which additionally get NO RoPE.
+};
+
+struct MapleWeights {
+  bool tie_word_embeddings = false;
+  bool attention_bias = false;
+  OwnedTensor embed_tokens;  // bf16 [vocab, H] gather table
+  OwnedTensor final_norm;    // bf16 [H]
+  OwnedTensor lm_head;       // bf16 [H, vocab] Matmul-B, untied
+  std::vector<MapleLayerWeights> layers;
+};
+
+// Batched PAGED forward. token_ids/positions are the flattened length-T step
+// inputs; attn_meta the common attention metadata; attn_kv one PagedKvCache per
+// layer (all layers allocate a cache slot; the SWA/global distinction is a
+// windowing + rope concern inside the attention call, not a cache-topology one).
+// Returns [T, vocab] f32 logits.
+std::vector<float> MapleForward(
+    const std::vector<int32_t>& token_ids, const std::vector<int32_t>& positions,
+    const v1::CommonAttentionMetadata& attn_meta,
+    const std::vector<PagedKvCache>& attn_kv, const MapleWeights& weights,
+    const HfConfig& config, vt::Queue& queue,
+    const std::vector<int32_t>& logits_indices = {});
+
+// Per-family config hook (ModelFactory::parse_config): validates the MoE fields
+// the loader/forward consume.
+void ParseMapleConfig(const HfConfig& config);
+
+// KV-cache spec builder: ONE full-attention group, no Mamba/GDN (clone of
+// MakeQwen3MoeKVCache — the runner's full-attention-only path covers it).
+v1::KVCacheConfig MakeMapleKVCache(const HfConfig& config, int block_size,
+                                   int num_blocks);
+
+// GGUF path. `IsMapleGguf` gates on general.architecture == "maple";
+// `MapleHfConfigFromGguf` maps the maple.* keys onto HfConfig;
+// `LoadMapleFromGguf` builds the whole-model weights (experts stay
+// block-quantized via keep-quant when available).
+bool IsMapleGguf(const GgufFile& gguf);
+HfConfig MapleHfConfigFromGguf(const GgufFile& gguf);
+MapleWeights LoadMapleFromGguf(const GgufFile& gguf, const HfConfig& config,
+                               const GgufLoadPolicy* policy);
+
+}  // namespace vllm

include/vt/dtype.h

@@ -112,6 +112,14 @@ enum class DType : uint8_t {
   kMXFP4,
   kIQ2_XS,
   kIQ4_XS,
+  // --- ternary keep-quant (Vulkan-native, Q8_K-activation) ---
+  // TQ2_0 (llama.cpp PR #25850): 2-bit ternary codes, 256-elem blocks.
+  // TQ1_0: packed base-3 trits, 256-elem blocks. Both are weight-only encodings
+  // dotted against Q8_K activations; the Vulkan backend has native keep-quant
+  // shaders for both, and the CPU dequant-composite path serves as the
+  // reference oracle. No CPU vec_dot (the keep-quant dot is Vulkan-only).
+  kTQ2_0,
+  kTQ1_0,
 };
 
 // Bytes per ELEMENT. Throws for block-quantized dtypes (they have no