CUDA: native TQ1_0 support (MMVQ decode, MMQ prefill)

[bri-prism]

Current behavior

GGML_TYPE_TQ1_0 is not implemented anywhere in the CUDA backend. A grep for
TQ1_0 across ggml/src/ggml-cuda/ returns zero matches, on this fork and on
upstream master alike. The type is CPU-only, with Vulkan and Metal support for its
sibling TQ2_0 having landed since.

The consequence is not an error, it is a silent fallback to the CPU. The type is
rejected by the device capability check, so the scheduler never places a TQ1_0
matmul on the GPU:

• ggml/src/ggml-cuda/ggml-cuda.cu:4985 opens the type switch shared by
  GGML_OP_MUL_MAT and GGML_OP_MUL_MAT_ID. TQ1_0 is absent from the accepted
  list at lines 4986 to 5013 and falls to default: return false at
  ggml-cuda.cu:5015.
• ggml/src/ggml-cuda/ggml-cuda.cu:5023 does the same for GGML_OP_GET_ROWS,
  with TQ1_0 falling to default: return false at ggml-cuda.cu:5055.
• ggml/src/ggml-cuda/mmq.cu:269 gates MMQ on type. TQ1_0 falls to
  default: mmq_supported = false at mmq.cu:298, so even once the capability
  check accepts the type there is no tile loader for the prefill path.
• ggml/src/ggml-cuda/mmvq.cu has no vec_dot entry for the type, and
  ggml/src/ggml-cuda/convert.cu has no dequantize kernel, so neither the
  quantized mat-vec path nor the dequantize-and-cuBLAS path can run.
• ggml/src/ggml-cuda/common.cuh has no ggml_cuda_type_traits<GGML_TYPE_TQ1_0>
  specialization, which every kernel path needs for its block geometry.

The op tests do not catch any of this, because TQ1_0 is commented out of both type
lists it would be exercised from: tests/test-backend-ops.cpp:8347 and
tests/test-backend-ops.cpp:8377, both reading
// GGML_TYPE_TQ1_0, // TODO: implement for all backends. A CUDA run of
test-backend-ops therefore reports a vacuous pass for this type today.

Measured impact

Internal measurement, large mixture-of-experts model quantized to a single low-bit
type throughout, one NVIDIA data-center GPU, llama-bench with r=3. Absolute
throughput in tokens per second:

type	pp512	tg128	CUDA kernel
TQ1_0	36.75	21.72	none, runs on CPU
Q1_0	2722.59	189.23	full MMVQ and MMQ
Q2_0	2252.51	181.72	full MMVQ and MMQ

TQ1_0 prefill is roughly 74x slower and decode roughly 8.7x slower than the low-bit
types that do have kernels, on the same model and the same hardware. The packing
itself was verified correct, by histogramming the per-tensor conversion lines from
llama-quantize, so this is kernel absence and not a per-tensor quantization
fallback.

The gap matters because TQ1_0 is the type the wider ecosystem uses for ternary
weights. Anyone who quantizes to it and then offloads to CUDA gets CPU speed
without being told why.

Upstream state

There is no upstream implementation and no upstream issue tracking one.

ref	what	state
ggml-org#8151	introduced TQ1_0 and TQ2_0, CPU only	merged, 2024-06
ggml-org#11183	ggml-cuda: TQ2_0 kernels (mmvq, MMQ tile load, mma, dequant plus cuBLAS)	open since 2025-01, stalled
ggml-org#25850	Vulkan TQ2_0 support	merged, 2026-07
ggml-org#27765	Vulkan TQ1_0 support (mm, mat-vec, mat-vec-id, dequant, get_rows)	open, approved, active
ggml-org#19743	earlier Vulkan TQ1_0 and TQ2_0 attempt	open, inactive since 2026-02, superseded by ggml-org#27765
ggml-org#22910	SYCL TQ1_0 and TQ2_0 dequant fallback	closed unmerged, 2026-05
ggml-org#23332	CUDA dequantization kernel for a different sparse ternary type	closed unmerged draft, 2026-05

Two things are worth drawing out of that table. First, ggml-org#11183 explicitly scoped
TQ1_0 out ("TQ1_0 is out of scope of this PR, but GPU support for it will also
come eventually") and then stalled, so even TQ2_0 has no CUDA kernels upstream
after more than a year. Second, ggml-org#27765 is the only place anyone has solved the
TQ1_0 mat-vec layout problem on a GPU, and its approach is the one to copy.

Proposed approach

Land it in the same order the accepted upstream pattern uses, smallest piece first,
each piece independently testable:

1. Add ggml_cuda_type_traits<GGML_TYPE_TQ1_0> in common.cuh, a dequantize
   kernel in convert.cu, and the get_rows entry. This alone gets the type onto
   the GPU through the dequantize-and-cuBLAS path and unblocks the op tests.
2. Add a vec_dot for the quantized mat-vec path, laid out to consume a whole
   256-element block per invocation rather than per 32-element chunk. See the
   decode issue for why the chunk contract is the problem.
3. Add an MMQ tile loader so prefill stops going through a full dequantize to F16.
4. Uncomment TQ1_0 in tests/test-backend-ops.cpp at both sites, and add
   MUL_MAT_ID cases at single-token decode shapes with more than one block per
   row. The generic sweep only covers k = 256, which is one block per row, and a
   wrong per-expert base offset is invisible at one block per row.

Steps 1 and 2 exist on this fork's branch feat/tq1_0-cuda (dequantize, vec_dot, traits,
capability entry, test enablement), measured but not landed, because step 2 as written regresses
decode. Details in the decode issue. Related report from a Jetson user: #99.

Prior art to follow rather than reinvent: ggml-org#11183 for the CUDA MMQ and mma tile
loading structure, and ggml-org#27765 for the base-3 decode helpers and the byte-indexed
mat-vec loop.

Acceptance criteria

• test-backend-ops -b CUDA0 -o MUL_MAT passes for tq1_0 on all shapes, with
  TQ1_0 uncommented in all_types and base_types so the run is not vacuous.
• test-backend-ops -b CUDA0 -o MUL_MAT_ID passes for tq1_0, including at least
  one single-token (ne11 = 1) shape with k a multiple of 512 so that more than
  one block per row is exercised, and with fewer experts used than available.
• test-backend-ops -b CUDA0 -o GET_ROWS passes for tq1_0.
• A real ternary model generates coherent output on CUDA with layers offloaded, not
  only op tests. -p 8 -n 4 is enough to catch a MoE dispatch abort that the op
  tests miss.
• llama-bench on the same model and GPU, TQ1_0 against Q2_0 at matching shapes:
  TQ1_0 pp512 and tg128 both within 25 percent of Q2_0. TQ1_0's format rate is
  1.6875 bpw against Q2_0's 2 bpw plus scale overhead, so parity on throughput
  plus the smaller footprint is the win condition. Regressing decode below the CPU
  fallback is an automatic fail, and is the specific failure the current local
  patch hits.
• No change to any other type's numbers. Enabling a type in the shared MUL_MAT
  switch also enables it for MUL_MAT_ID, so confirm the MoE path separately.

[bri-prism]

Children filed: #139 covers the decode mat-vec layout problem, #140 covers the missing MMQ tile loader for prefill. They take independent code paths, so #140 is worth doing even if #139 stays parked.

Also relevant: #99 reports the same CUDA gap from a Jetson user's side, for both ternary types. This issue supersedes the TQ1_0 half of that report and adds the file-level detail; the SM72 arch-list half of #99 is unrelated and still stands on its own.

Existing work to build on rather than restart: this fork's branch feat/tq1_0-cuda already carries the traits, the dequantize kernel, a vec_dot, the capability entry and the test enablement. It applies to the current default branch with two mechanical conflicts, one from an adjacent HIP helper and one from TQ2_0 having since been enabled in the op-test type lists. Both resolve as both-keep. Anyone picking this up should start from that branch, not from scratch, and should read #139 first, because the vec_dot in it is the part that needs restructuring.