feat(kvcache): sub-byte Q4_0 and Q6_0 KV cache quantization

docs/kv_cache_quantization.md

@@ -0,0 +1,193 @@
+# KV Cache Quantization
+
+> User-facing reference for the `--kv-cache-dtype` family. For the
+> implementation rationale (why mag=8 not 7, why per-block 32 not
+> super-block, why the dual-plane Q6 layout), see the PR bodies
+> (`pr-bodies/PR1_q4_0.md` and `pr-bodies/PR2_q6_0.md`).
+
+## What is `--kv-cache-dtype`?
+
+The KV cache stores the K (key) and V (value) tensors the attention
+layers read from on every decode step. By default it lives in bf16
+(2 bytes per element). Quantizing the cache shrinks the per-element
+storage cost at the cost of some numerical precision on the
+attention computation.
+
+FreeToken supports the following `--kv-cache-dtype` values:
+
+| value | bytes/elem | effective context @ 8 GB | precision vs bf16 |
+|---|---|---|---|
+| `auto` (or unset) | 2.000 (bf16) | ~110K | - |
+| `q8_0` | 1.0625 | ~160K | ~0.6% kernel rel_err |
+| `q6_0` | 0.8125 | ~190K | ~2.4% kernel rel_err |
+| `q4_0` | 0.5625 | ~220K | ~9.4% kernel rel_err |
+| `fp8_e4m3` | 1.0625 | ~160K | ~0.6% kernel rel_err (float path) |
+
+"Effective context" assumes the engine's hybrid MoE backend is
+enabled (the MoE offload cache lives in its own pool sized by
+`--moe-cache-auto`). On 8 GB consumer GPUs the Q4 path is the
+only one that pushes the context window past 200K.
+
+## How to launch
+
+The simplest form:
+
+```bash
+ft serve \
+  --model Qwen/Qwen3.6-35B-A3B \
+  --kv-cache-dtype q4_0 \
+  --kv-reserve-tokens 220000 \
+  --moe-backend hybrid \
+  --moe-cpu-threads 12 \
+  --memory-ratio 0.97 \
+  --moe-cache-auto
+```
+
+`--kv-cache-dtype auto` is the same as not setting it (bf16).
+`--kv-reserve-tokens N` is the size of the K/V pool; pick N to
+match the longest conversation you intend to serve. `--moe-cpu-threads
+12` should be calibrated on the target machine (`ft bench bw` to
+find the best value).
+
+The first request after startup will spend ~3-5 s JIT-compiling the
+quantized store / load kernels. Subsequent requests are at full
+throughput.
+
+## When to pick which dtype
+
+- **Default / when in doubt**: `q8_0` is the upstream PR#103 default
+  and a safe bet; near-bf16 precision, 47% memory savings over bf16.
+- **Need maximum context** (long documents, full-book Q&A): `q4_0`
+  gives 3.5x the bf16 context on the same VRAM, at the cost of
+  ~9% kernel rel_err. Retrieval is unaffected (needle-in-haystack
+  passes at 8K through 220K), but multi-step chain-of-thought
+  degrades measurably: on a six-scheme same-protocol ladder
+  (GSM8K-CoT 8-shot greedy, n=150), q4_0 scored 0.83-0.85 vs
+  0.96-0.97 for q6_0/q8_0/nvfp4 at the same bytes/element and
+  0.95 for the 0.39-byte LM-codebook q3_lm -- the 4-bit amax
+  scale combination is the outlier. Pick `q4_0` when context
+  capacity is the goal and your workload is retrieval-shaped;
+  prefer `nvfp4` (same bytes, no CoT loss) or `q6_0` when
+  reasoning quality matters.
+- **Precision-first sub-byte**: `q6_0` is between Q4 and Q8: ~4x
+  better kernel precision than Q4, 24% more bytes. Use when Q4
+  loses too much on your workload and Q8's context window is
+  too small.
+- **bf16 only**: `auto` (or unset). Required if you see model-output
+  drift on hard reasoning and need the canonical baseline.
+
+The Q4/Q6 paths do **not** require any model quantization: weights
+stay in bf16, only the K/V cache is sub-byte. The Q4/Q6 sub-byte
+path is orthogonal to NVFP4 / FP8 weight quantization -- both
+can be active at the same time.
+
+## How it works (one paragraph)
+
+The K/V pool's last axis is `head_dim`. Quantized schemes pack
+multiple values per byte along that axis:
+
+- **q8_0** / **fp8_e4m3** -- 1 byte per element. One int8 (or fp8)
+  value per slot, plus one fp16 scale per 32 values along head_dim.
+- **q6_0** -- 0.75 byte per element. 32 values are packed into
+  16 bytes (low plane: low 4 bits of each 6-bit value, packed the
+  same as Q4) plus 8 bytes (high plane: top 2 bits of each value,
+  packed four-per-byte at bit positions 0, 2, 4, 6), plus one fp16
+  scale per block.
+- **q4_0** -- 0.5 byte per element. 32 values are packed into
+  16 bytes: byte `j` holds `val[2j]` in the low nibble and
+  `val[2j+1]` in the high nibble, both as unsigned 4-bit. One fp16
+  scale per block.
+
+The attention kernel is told the logical `head_dim` and unpacks
+inside the load. The store kernel packs on the write path. Both
+operations are transparent to the model code.
+
+```
+         logical head_dim (e.g. 128)
+         =======================
+bf16     [v0][v1] ... [v127]            256 bytes per token per layer
+q8_0     [v0][v1] ... [v127]            128 bytes +  8 bytes scale = 136
+q6_0     [v0/lo][v1/lo] ... [v127/lo]   96 bytes +  8 bytes scale = 104
+         [v0/hi, v1/hi, v2/hi, v3/hi] ... (8 bytes, 4 values each)
+q4_0     [v0/lo|v1/hi][v2/lo|v3/hi] ...  64 bytes +  8 bytes scale =  72
+```
+
+## What the Q4/Q6 paths do NOT change
+
+- **Model weights** are still bf16 (or NVFP4 / FP8 if you set the
+  weight quantization separately). Only the K/V cache is sub-byte.
+- **Linear-attention (GatedDeltaNet) layers** are not affected.
+  Hybrid models (e.g. Qwen3.5-35B-A3B's 4 linear + 32 full attention
+  layers) get the full context-length win because the paged pool is
+  what hits the wall, but the linear layers' state pool is untouched.
+- **The MoE offload cache** lives in its own pool sized by
+  `--moe-cache-auto`. Sub-byte KV does not change the MoE cache
+  budget solve.
+- **The OpenAI-compatible API surface** is unchanged. Tokens/s,
+  request formats, response formats, and the streaming protocol are
+  all identical across dtypes; the only knob is the new context
+  budget.
+
+## Hybrid model note
+
+For hybrid models (Qwen3.5 / Qwen3.6 MoE with linear attention
+layers), the K/V pool is sized for the **full-attention** layers
+only. The linear layers' state is held in a separate pool that
+this PR does not touch. Empirically on Qwen3.5-35B-A3B the linear
+layers account for 4 of the 36 layers, so the effective Q4 KV
+context is still the Q4 number from the table; the linear layers'
+state is on top of that, sized separately by the engine.
+
+## Compatibility with the GGUF Q4_0 spec
+
+The byte layout (low-nibble-even, high-nibble-odd, 16 bytes per
+32-value block, 1 fp16 scale) matches the GGUF Q4_0 spec, **except**
+for the `max_magnitude` constant: we use 8 (range `[-8, 7]`) where
+GGUF uses 7 (range `[-7, 7]`). The 8-bound is empirically 5% better
+on K/V-shaped data because the distribution tail biases the per-
+block scale upward, leaving the +7 boundary the more frequent side.
+A Q4_0 cache produced by a tool that uses the GGUF 7-bound will
+round-trip through our dequant with ~5% rel_err; we do not read
+pre-quantized caches from disk, so this only matters if a user
+later writes a converter.
+
+## How to verify it's working
+
+```bash
+# Start the service
+ft serve --model Qwen/Qwen3.6-35B-A3B --kv-cache-dtype q4_0 \
+  --kv-reserve-tokens 220000 --moe-backend hybrid \
+  --moe-cpu-threads 12 --memory-ratio 0.97 --moe-cache-auto
+
+# In another terminal, check the startup log for "Allocating ... tokens
+# for KV cache, K + V = <X> GiB". Q4_0 yields ~1.18 GiB at 160K tokens;
+# Q6_0 yields ~1.24 GiB; q8_0 yields ~1.24 GiB; bf16 yields ~3.20 GiB.
+```
+
+A clean run will also report the per-kernel compile lines on the
+first request; these can be ignored after the first decode.
+
+## How to recover
+
+Reverting to bf16 is one flag change:
+
+```bash
+ft serve ... --kv-cache-dtype auto
+```
+
+There is no data loss across dtype changes -- the K/V cache is
+ephemeral (regenerated on every request) and a session-started
+flag controls the pool allocation at startup. The CLI rejects
+mismatched configurations at startup; if you change `--kv-cache-
+dtype` mid-session, restart the service.
+
+## See also
+
+- `pr-bodies/PR1_q4_0.md` -- the Q4 PR body, with kernel-level
+  numbers, A/B test results, and "why mag=8" rationale
+- `pr-bodies/PR2_q6_0.md` -- the Q6 PR body, with the dual-plane
+  layout, kernel-level numbers, and the "why two PRs" rationale
+- `tests/kvcache/test_subbyte_quant.py` -- spec round-trip tests
+- `tests/kernels/test_attention_subbyte.py` -- kernel parity tests
+- `WALKTHROUGH.md` (in the upload package) -- review-prep doc with
+  the 3 most likely reviewer questions

python/freetoken/checkpoint/ftw.py

@@ -261,7 +261,12 @@ def _map(self, file: str) -> memoryview:
                 if entry is None:
                     fd = os.open(os.path.join(self.dir, file), os.O_RDONLY)
                     try:
-                        m = mmap.mmap(fd, 0, prot=mmap.PROT_READ)
+                        if os.name == "nt":
+                            # Windows mmap has no prot= / PROT_READ; ACCESS_READ is
+                            # the equivalent page-protection mode there.
+                            m = mmap.mmap(fd, 0, access=mmap.ACCESS_READ)
+                        else:
+                            m = mmap.mmap(fd, 0, prot=mmap.PROT_READ)
                     finally:
                         os.close(fd)  # the mapping keeps its own reference to the file
                     try:

python/freetoken/engine/config.py

@@ -80,6 +80,17 @@ class EngineConfig:
     # KV capacity in tokens; resolved into num_page_override by _adjust_config once page_size
     # is final. Mutually exclusive with num_page_override.
     num_token_override: int | None = None
+    # KV element storage (--kv-cache-dtype): "auto" keeps the compute dtype, "q8_0" and
+    # "fp8_e4m3" store 8 bits plus a per-block scale, and the sub-byte "q4_0"/"q6_0"
+    # pack multiple values per byte. Resolved through
+    # freetoken.kvcache.quant.resolve_kv_quant by the pools and the cost model.
+    kv_cache_dtype: str = "auto"
+
+    @cached_property
+    def kv_quant(self):
+        from freetoken.kvcache.quant import resolve_kv_quant
+
+        return resolve_kv_quant(self.kv_cache_dtype)
 
     @cached_property
     def hf_config(self):

python/freetoken/engine/engine.py

@@ -151,6 +151,44 @@ def _resolve_auto_attention_backend(required: frozenset[AttnType]) -> str:
     )
 
 
+def _validate_kv_cache_dtype(config, model_config) -> None:
+    """Gate --kv-cache-dtype against what the quantized path actually implements.
+
+    Quantized KV storage lives in the triton attention kernels and the MHA/hybrid-SWA
+    pools. Every other backend reads the KV slabs through its own kernels (flashinfer's
+    ``kv_data_type``, trtllm's fp8 path) which this has not been wired into, and the
+    MLA/DSA/DSV4/BSA pools have their own slab layouts. Reject those combinations here,
+    at config time, rather than letting a wrong-dtype tensor reach a kernel.
+    """
+    quant = getattr(config, "kv_quant", None)
+    if quant is None or not quant.enabled:
+        return
+
+    from freetoken.kvcache.quant import BLOCK
+
+    backends = [p.strip() for p in config.attention_backend.split(",")]
+    if any(b != "triton" for b in backends):
+        raise ValueError(
+            f"--kv-cache-dtype {quant.name} needs the triton attention backend, but the "
+            f"resolved backend is {config.attention_backend!r}. Pass "
+            "--attention-backend triton, or drop --kv-cache-dtype."
+        )
+
+    specs = [s for s in model_config.kv_cache_group_specs() if s.num_layers > 0]
+    if any(s.mla or s.index_head_dim > 0 for s in specs):
+        raise ValueError(
+            f"--kv-cache-dtype {quant.name} does not support MLA/DSA latent KV pools "
+            "(their slabs alias K and V and carry an index tier); use --kv-cache-dtype auto."
+        )
+    bad = [s for s in specs if s.head_dim % BLOCK]
+    if bad:
+        names = ", ".join(f"{s.name} (head_dim {s.head_dim})" for s in bad)
+        raise ValueError(
+            f"--kv-cache-dtype {quant.name} needs every head_dim to be a multiple of "
+            f"{BLOCK}, the quantization block; this model has {names}."
+        )
+
+
 def _validate_attention_backend_choice(config, override, required: frozenset[AttnType]) -> None:
     """Config-time type x backend capability check for the resolved (or explicit)
     backend string: every comma part must serve every required type and have its
@@ -1027,7 +1065,11 @@ def _ensure_expandable_segments() -> None:
     if os.environ.get("PYTORCH_ALLOC_CONF") or os.environ.get("PYTORCH_CUDA_ALLOC_CONF"):
         return
     try:
-        torch.cuda.memory._set_allocator_settings("expandable_segments:True")
+        # torch 2.9+: _set_allocator_settings -> _C._cuda_setAllocatorSettings
+        try:
+            torch.cuda.memory._set_allocator_settings("expandable_segments:True")
+        except AttributeError:
+            torch._C._cuda_setAllocatorSettings("expandable_segments:True")
     except Exception as exc:  # pragma: no cover - depends on torch build
         logger.info_rank0(f"Could not enable expandable_segments ({exc}); continuing")
         return
@@ -1327,6 +1369,7 @@ def override(attr: str, value: Any):  # this is dangerous, use with caution
         )
         logger.info_rank0(f"Auto-selected attention backend: {config.attention_backend}")
     _validate_attention_backend_choice(config, override, required_attn_types)
+    _validate_kv_cache_dtype(config, model_config)
 
     if config.moe_cache_rate is not None:
         total_experts = config.model_config.num_moe_layers * config.model_config.num_experts

python/freetoken/kernel/csrc/include/freetoken/utils.h

@@ -1,5 +1,12 @@
 #pragma once
 
+// MSVC has no __always_inline (a sys/cdefs.h / GCC predefined macro); its
+// equivalent is __forceinline. The kernel headers use __always_inline
+// throughout, so alias it before anything includes them.
+#if defined(_MSC_VER) && !defined(__always_inline)
+#define __always_inline __forceinline
+#endif
+
 // ref:
 // https://forums.developer.nvidia.com/t/c-20s-source-location-compilation-error-when-using-nvcc-12-1/258026/3
 #ifdef __CUDACC__

python/freetoken/kernel/csrc/include/freetoken/warp.cuh

@@ -1,7 +1,9 @@
 #pragma once
 #include <freetoken/utils.cuh>
 
+#if !defined(_MSC_VER)
 #include <sys/cdefs.h>
+#endif
 
 #include <cstddef>
 

python/freetoken/kernel/csrc/jit/fast_index_copy.cuh

@@ -269,7 +269,7 @@ inline auto get_sync_flag_ptr(
     auto flag_dtype = host::SymbolicDType{};
     host::TensorMatcher({1})
         .with_dtype<int32_t>(flag_dtype)
-        .with_device<kDLCUDA>(device)
+        .template with_device<kDLCUDA>(device)
         .verify(sync_flag);
     return static_cast<int32_t*>(sync_flag.data_ptr());
 }
@@ -344,17 +344,17 @@ struct FastIndexCopyKernel {
 
         TensorMatcher({-1, D})
         .with_dtype(data_dtype)
-        .with_device<kDLCUDA, kDLCUDAHost, kDLCPU>()
+        .template with_device<kDLCUDA, kDLCUDAHost, kDLCPU>()
         .verify(src);
 
         TensorMatcher({-1, D})
         .with_dtype(data_dtype)
-        .with_device<kDLCUDA, kDLCUDAHost, kDLCPU>()
+        .template with_device<kDLCUDA, kDLCUDAHost, kDLCPU>()
         .verify(dst);
 
         TensorMatcher({L})
         .with_dtype<int32_t, int64_t>(indices_dtype)
-        .with_device<kDLCUDA>(device)
+        .template with_device<kDLCUDA>(device)
         .verify(src_indices)
         .verify(dst_indices);
 
@@ -363,7 +363,7 @@ struct FastIndexCopyKernel {
             const auto num_indices_tensor = num_indices.value();
             TensorMatcher({1})
                 .with_dtype<int64_t>(num_indices_dtype)
-                .with_device<kDLCUDA>(device)
+                .template with_device<kDLCUDA>(device)
                 .verify(num_indices_tensor);
 
             num_indices_data_ptr = static_cast<const int64_t*>(num_indices_tensor.data_ptr());
@@ -529,14 +529,14 @@ struct MultiIndexCopyKernel {
         auto indices_dtype = SymbolicDType{};
         auto num_indices_dtype = SymbolicDType{};
 
-        TensorMatcher({B}).with_dtype<int64_t>(ptr_dtype).with_device<kDLCUDA>(device)
+        TensorMatcher({B}).with_dtype<int64_t>(ptr_dtype).template with_device<kDLCUDA>(device)
             .verify(dst_ptrs).verify(src_ptrs).verify(feat_bytes);
-        TensorMatcher({L}).with_dtype<int32_t, int64_t>(indices_dtype).with_device<kDLCUDA>(device)
+        TensorMatcher({L}).with_dtype<int32_t, int64_t>(indices_dtype).template with_device<kDLCUDA>(device)
             .verify(dst_indices).verify(src_indices);
 
         const int64_t* valid_length = nullptr;
         if (num_indices.has_value()) {
-            TensorMatcher({1}).with_dtype<int64_t>(num_indices_dtype).with_device<kDLCUDA>(device)
+            TensorMatcher({1}).with_dtype<int64_t>(num_indices_dtype).template with_device<kDLCUDA>(device)
                 .verify(num_indices.value());
             valid_length = static_cast<const int64_t*>(num_indices.value().data_ptr());
         }

python/freetoken/kernel/csrc/jit/index.cu

@@ -114,15 +114,15 @@ struct IndexKernel {
 
     TensorMatcher({-1, D}) //
         .with_dtype(weights_dtype_)
-        .with_device<kDLCUDA>(device_)
+        .template with_device<kDLCUDA>(device_)
         .verify(weights);
     TensorMatcher({L, D}) //
         .with_dtype(weights_dtype_)
-        .with_device<kDLCUDA>(device_)
+        .template with_device<kDLCUDA>(device_)
         .verify(output);
     TensorMatcher({L}) //
         .with_dtype<int32_t, int64_t>(indices_dtype_)
-        .with_device<kDLCUDA>(device_)
+        .template with_device<kDLCUDA>(device_)
         .verify(indices);
 
     const auto device = device_.unwrap();

python/freetoken/kernel/csrc/jit/store.cu

@@ -84,7 +84,7 @@ struct StoreKernel {
         .verify(v);
     TensorMatcher({L}) //
         .with_device<kDLCUDA>(device_)
-        .with_dtype<int32_t, int64_t>(indices_dtype_)
+        .template with_dtype<int32_t, int64_t>(indices_dtype_)
         .verify(indices);
 
     const auto dtype_size = dtype_bytes(dtype_.unwrap());