Metal: no TQ1_0 kernels, and supports_op claims the type anyway

[bri-prism]

Current behavior

GGML_TYPE_TQ1_0 has no Metal kernels at all. A grep for tq1_0 across
ggml/src/ggml-metal/ returns zero matches, while its sibling TQ2_0 is fully
supported there (dequantize helper, mul_mv, mul_mm, mul_mm_id, mul_mv_id,
get_rows, cpy, set_rows).

Worse than absence, the backend actively claims the type. ggml_metal_device_supports_op
gates these ops by rejecting a single type rather than by accepting a known list:

• ggml/src/ggml-metal/ggml-metal-device.m:1719 for GGML_OP_MUL_MAT and
  GGML_OP_MUL_MAT_ID returns has_simdgroup_reduction && type != GGML_TYPE_NVFP4.
• ggml/src/ggml-metal/ggml-metal-device.m:1786 for GGML_OP_GET_ROWS returns
  type != GGML_TYPE_NVFP4.

So TQ1_0 passes the capability check, the scheduler places the op on Metal, the
pipeline lookup asks for a kernel name that was never compiled, and
ggml_metal_library_compile_pipeline aborts the process. This does not show up
today only because TQ1_0 is commented out of the op-test type lists at
tests/test-backend-ops.cpp:8347 and tests/test-backend-ops.cpp:8377. Enable it
and a Metal op-test run aborts.

This is the third instance of the same pattern found in this backend recently. The
other two were a flash-attention variant that aborts on a missing kernel for an
unsupported KV type, and a low-bit matmul that silently took a generic kernel
instead of its tuned one. A permissive supports_op turns a missing kernel into
either a crash or a silent slowdown, never a clean fallback.

Why support the type here

TQ1_0 is the format the wider ecosystem uses for ternary weights, at 1.6875 bpw
against TQ2_0's 2.0625, so it is the smaller of the two ternary options on disk and
in memory. Apple hardware is a primary target for low-bit inference, and a unified
memory device is exactly where the smaller footprint pays. Right now a ternary
model in this format runs its matmuls on the CPU on Apple silicon, or aborts.

Proposed approach

The type's block layout is block_tq1_0 in ggml/src/ggml-common.h:288-294: 5
trits packed per byte in base 3 for the bulk of the block, plus a qh tail at 4
elements per byte, plus one half-precision scale per 256 elements. That base-3
packing is what makes it different from every byte-aligned type in the backend, and
it is the part to get right first.

Follow the existing TQ2_0 code path file by file, since the plumbing is identical
and only the unpack differs:

1. kernels/dequantize.h: add dequantize_tq1_0, modelled on dequantize_tq2_0
   at line 752. Expand each packed byte into all five of its trits in place rather
   than indexing by element and deriving the byte, so every byte is read once.
   Handle the qh tail separately.
2. kernels/quantize.metal: instantiate kernel_get_rows_tq1_0 and the cpy
   variants, alongside the TQ2_0 instantiations at lines 307, 152 and 163.
3. kernels/mul_mv.metal: add a kernel_mul_mv_tq1_0_f32_impl plus the mul_mv_id
   instantiation, following the TQ2_0 versions at lines 3089 and 3360.
4. kernels/mul_mm.metal: add the four mul_mm and mul_mm_id instantiations,
   following lines 767, 794, 830 and 857.
5. Remove the TQ1_0 rejections from supports_op as each op gains a kernel, and
   narrow those two gates to accept-lists so the next missing kernel produces a
   fallback rather than an abort.

Prior art worth reading before writing the unpack: the Vulkan TQ1_0 work in
ggml-org#27765 (open, approved) is the only existing GPU solution to this
layout. It loads one packed byte per thread, expands all five digits in place, and
packs the powers of three into a single word so they extract with a shift and a
mask. The same structure applies directly to a Metal mat-vec kernel.

Acceptance criteria

• TQ1_0 uncommented in all_types and other_types in tests/test-backend-ops.cpp,
  so the run is not vacuous.
• test-backend-ops -b Metal -o GET_ROWS passes for tq1_0.
• test-backend-ops -b Metal -o MUL_MAT passes for tq1_0 at both single-token and
  batched shapes, so the mat-vec and mat-mul paths are both covered.
• test-backend-ops -b Metal -o MUL_MAT_ID passes for tq1_0, including at least
  one single-token shape with more than one block per row, since a wrong per-expert
  base offset is invisible when a row is a single block.
• A real ternary model in this format generates coherent output on Metal with layers
  offloaded, not only op tests.
• llama-bench on the same model: TQ1_0 decode and prefill within 25 percent of
  TQ2_0 on the same device. Slower than the CPU fallback is an automatic fail.
• The two supports_op gates converted to accept-lists, and a note in the commit
  message pointing at the two other bugs that the permissive form caused.

Interim state

Until kernels exist, the type must be declined rather than claimed. Upstream
already carries that fix as part of ggml-org#27765, and it has been cherry-picked onto this
fork's TQ1_0 branch. Anyone implementing the kernels should remove those rejections
op by op as each kernel lands.

[bri-prism]

Kernels are written and correctness-gated on an Apple silicon device (M5 Pro). Branch: feat/tq1_0-cuda-v7, commit "metal: TQ1_0 kernels (dequantize, get_rows, mat-vec, mat-mul, per-expert)".

What landed on the branch: tq1_0_trit plus dequantize_tq1_0 in kernels/dequantize.h, kernel_get_rows_tq1_0, kernel_mul_mv_tq1_0_f32 and its mul_mv_id instantiation, the four mul_mm and mul_mm_id instantiations, N_R0_TQ1_0 and N_SG_TQ1_0, and the nr0/nsg dispatch at both mat-vec sites. The two supports_op rejections from the interim commit are removed again, since kernels now exist for those ops.

The mat-vec kernel inverts the loop rather than fitting the element-indexed convention: each thread owns packed bytes and expands all five base-3 digits in place, so every packed byte is read exactly once and the strided reads land on the activation vector instead. Per 256-element block that is 32 bytes in the first qs region, 16 in the second, and 4 in the qh tail, spread across a simdgroup's 32 threads.

Correctness: passing

test-backend-ops on the Metal backend, with TQ1_0 uncommented in both type lists so the run is not vacuous. Zero failures and zero "not supported" for the type across GET_ROWS, MUL_MAT and MUL_MAT_ID.

Nine explicit cases were added, because the generic sweep only covers k = 256, which is one block per row, and that hides both block-boundary errors and wrong per-expert base offsets. The new cases include k of 512, 768 and 1024, single-token and batched, with and without a bias, and per-expert shapes with fewer experts used than available. All pass.

A real model in this format also generates coherent output on Metal with all layers offloaded, which the op tests alone would not have caught.

Throughput: below the bar in this issue

llama-bench, small ternary-trained model, all layers offloaded, r=3, two interleaved passes. Tokens per second:

	pp512	tg128
TQ1_0 on Metal	3275 to 3291	142.9 to 143.6
TQ2_0 on Metal	6199 to 6241	235.2 to 240.3
TQ1_0 on CPU (the old fallback)	2313	112.7

So the kernels clear the hard floor in this issue, beating the CPU fallback by 1.42x prefill and 1.27x decode, but TQ1_0 sits at 53 percent of TQ2_0 prefill and 60 percent of its decode. The acceptance criterion asked for within 25 percent. That criterion is not met, and this should not be treated as done.

Part of the gap is intrinsic: extracting a base-3 digit costs a multiply, a shift and a subtract per element, where a 2-bit field costs a shift and a mask. Part is not, and is worth attacking before anyone concludes the format is simply slower on this backend:

• The activation reads in the mat-vec kernel are five separate scalar loads per byte region. The TQ2_0 kernel reads float4 and hoists per-byte coefficients out of the row loop. The same restructuring applies here.
• TQ2_0 expresses its field shifts as float floor operations and folds the zero-point correction into a single hoisted sum. The base-3 equivalent is a small table of digit weights per byte value, which would replace the per-element integer multiply.
• mul_mm goes through the generic dequantize_tq1_0 template, which is element-indexed and therefore pays the five-times-read cost the mat-vec kernel avoids. A block-at-a-time unpack into the shared memory tile would fix prefill specifically, and prefill is the weaker of the two ratios.

Note the measurement model is small, so decode is closer to launch-bound than bandwidth-bound and the GPU-over-CPU ratio understates what a large model would show. The TQ1_0-versus-TQ2_0 ratio is the meaningful number here, not the absolute speedup over CPU.

Remaining before this issue closes: the throughput work above, and converting the two supports_op gates to accept-lists so the next missing kernel produces a clean fallback rather than an abort or a silent generic path.

[bri-prism]

Perf follow-up. The first lever from the comment above is done and measured, and the diagnosis it rests on is worth stating plainly because it reframes the issue.

Where the time was going

TQ1_0 decode was not bandwidth-bound. Against a weights-only bandwidth ceiling it ran at 29 percent, where TQ2_0 reaches 54 percent on the same device. The mat-vec inner loop was doing about seven integer operations plus an integer-to-float conversion per element, and this ISA cannot co-issue integer and floating-point work, so none of that cost overlapped with the activation arithmetic. TQ2_0 is fast partly because it deliberately expresses its field shifts as float floor operations and never leaves the float pipe.

The fix

A packed byte is a base-3 fraction of 256, not an integer to divide. With u = b/256, digit n is g(n+1) - 3g(n) where g(k) = floor(3^k * u). Every such product is exact in fp32, because 3^kb is at most 61965 and dividing by 256 is exact, so the floors are exact and the result matches the integer reference bit for bit over all 256 byte values. No epsilon, no tolerance.

Summing (digit - 1) * activation over a byte's digits then collapses to a form whose coefficients depend only on the activations, so they hoist out of the row loop and are shared across every row in the simdgroup. The inner loop becomes one floor and one fma per element with no integer arithmetic. The qh tail carries four digits rather than five and takes the same treatment truncated.

Measured

Small ternary model, all layers offloaded, r=3, two interleaved passes, tokens per second:

	before	after
decode	143.2	166.1
prefill	3283	3318

Decode is up 16.0 percent, from 29.3 to 34.0 percent of ceiling, and TQ1_0 now reaches 71 percent of TQ2_0 decode rather than 61. Correctness is unchanged: MUL_MAT and MUL_MAT_ID pass with zero failures, including the multi-block-per-row and per-expert cases.

Prefill moved 1.1 percent, which is within noise, and that confirms the diagnosis rather than contradicting it. mul_mm goes through the element-indexed dequantize template and never reaches the mat-vec kernel, so prefill still reads every packed byte five times.

What is left, ranked and quantified

About 1.57x remains before TQ1_0 would match TQ2_0's efficiency on its own smaller footprint, at which point it would beat TQ2_0 outright.

1. Simdgroup load imbalance in the mat-vec, the largest remaining decode lever. A 256-element block is 32 packed bytes in the first qs region, 16 in the second, and 4 in the qh tail. The kernel handles these as three phases, so 32, then 16, then 4 of the 32 threads are busy: 54 percent utilization, and the simdgroup waits for the slowest lane each phase. Flattening all 52 bytes into one list with at most two bytes per thread lifts that to 81 percent. Processing two blocks per iteration would go further.
2. Prefill unpack, the worse of the two ratios and completely untouched. A block-at-a-time unpack into the shared memory tile, rather than the element-indexed template, is what fixes it. TQ1_0 prefill is 53 percent of TQ2_0.
3. Vectorized activation loads. The remaining reads are scalar and strided, five per byte region. TQ2_0 reads float4. This is smaller than the two above but cheap to try.

The supports_op accept-list conversion is still open and unrelated to throughput.

[bri-prism]

One more lever, from a parallel piece of work on this backend rather than from this issue.

A word-parallel formulation was just measured on the 1-bit type on the same device: activations are quantized to int8 once per matmul and stored as eight bit-planes in a separate dispatch, so 32 weights are consumed per AND-plus-popcount instead of being unpacked one at a time. It measured 21 to 31 percent over the previous best kernel at verify batch sizes.

The same idea transfers to a ternary type, and would subsume levers 1 and 3 above rather than sit alongside them. A ternary weight needs two bit-planes rather than one, a nonzero mask and a sign mask, after which a dot product is two AND-popcount pairs per 32 weights with no base-3 arithmetic at all. That is a much larger change than flattening the thread assignment, so the ordering to try is still the load imbalance first, since it is a few lines, then this if the remaining gap justifies it.

Two hard-won details from that work worth carrying over. The bit-plane construction has to be its own dispatch into a scratch buffer, because building planes per threadgroup eats the gain. And the row tiling decides the result: two columns per lane worked while four spilled registers and lost more than the kernel won.

Note also that quantizing activations to int8 changes the numerics, so a word-parallel path here would need a measured divergence gate before being enabled by default, not just op-test parity. The float work in the comment above is bit-exact against the reference and needs no such gate, which is why it went first.