From 83b1326e657bd00ca8a0e3479a6c6a9b8a433b79 Mon Sep 17 00:00:00 2001 From: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> Date: Sat, 29 Aug 2026 03:03:10 +0000 Subject: [PATCH] docs: align FP8 KV and placement claims ROCm now implements and gates the FP8 E4M3 KV store and read, while the public usage pages still described it as refused. dots3-note also joined the shared expert-placement path after the README counted five families. Keep the unmeasured CUDA and end-to-end gates explicit, and compact the placement row to its current support and gap summary. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Codex:gpt-5 [Codex] --- README.md | 14 +++++++------- docs/FEATURES.md | 4 ++-- docs/USAGE.md | 5 +++-- 3 files changed, 12 insertions(+), 11 deletions(-) diff --git a/README.md b/README.md index f8c52dfbe..4772e6e3b 100644 --- a/README.md +++ b/README.md @@ -37,16 +37,16 @@ ## News -- **2026-08** **Hybrid CPU/GPU expert placement reaches five architecture families.** Qwen3-MoE, - Qwen3.5/3.6, Nemotron-H, DeepSeek-V2, and Kimi-Linear can run routed experts on the CPU while the - rest of the model stays on the selected accelerator. The end-to-end token and speed gates are - still pending. +- **2026-08** **Hybrid CPU/GPU expert placement reaches six architecture families.** Qwen3-MoE, + Qwen3.5/3.6, Nemotron-H, DeepSeek-V2, Kimi-Linear, and dots3-note can run routed experts on the CPU + while the rest of the model stays on the selected accelerator. The end-to-end token and speed + gates are still pending. - **2026-08** **IndexTTS 2.5 now uses its reference audio.** The clip conditions both the talker and S2Mel stages. Repeated runs with one clip are bit-identical, while different clips change the output. Voice-cloning parity with vLLM-Omni remains unmeasured. -- **2026-08** **FP8 KV cache storage reaches the server.** `--kv-cache-dtype fp8` stores K/V in - 1-byte E4M3 pages. A fixed memory budget holds twice as many blocks, but the throughput trade-off - is not measured yet. +- **2026-08** **FP8 KV cache storage reaches the server and ROCm.** `--kv-cache-dtype fp8` stores K/V + in 1-byte E4M3 pages. The ROCm store and read pass their gfx1100 component gate. CUDA device and + end-to-end memory and throughput gates are still pending. - **2026-08** **The server returns prompt log probabilities.** Both completion endpoints accept `prompt_logprobs`, including `-1` for the full vocabulary, and reject unsupported request shapes with `400`. diff --git a/docs/FEATURES.md b/docs/FEATURES.md index bdbc425f3..5a2124d1a 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -52,7 +52,7 @@ are our reading of their documented behavior, not measurements. | Block-paged KV with refcount and LRU evict | ✅ | ✅ | ✅ | ◐ | | Hybrid KV groups (full attention + GDN/Mamba) | ◐ GDN gate activation resolved from the checkpoint's `output_gate_type` (silu/swish/sigmoid; anything else refused at load, #489) | ✅ | ◐ | ◐ | | Sliding-window and chunked-local attention | ◐ | ✅ | ✅ | ✅ | -| fp8 KV cache | ◐ `--kv-cache-dtype fp8` halves the block, so a fixed `--kv-cache-memory` buys 2x the blocks and the DEFAULT 256-block path halves the pool bytes instead. Costs the bf16-native FA-2/WMMA/vector kernels (net UNMEASURED). 16 archs, MLA, the C ABI are refused before any write; only 1 arch names fp8 back. CUDA UNRUN ([spec](../.agents/specs/fp8-kv-cache.md)) | ✅ | ✅ | ✅ | +| fp8 KV cache | ◐ `--kv-cache-dtype fp8` halves each block. CPU and ROCm store/read component gates pass; CUDA device and end-to-end memory/throughput gates are pending. The path uses the slower attention kernels. Metal, the C ABI, MLA, and 16 architecture-specific attention paths remain unsupported ([spec](../.agents/specs/fp8-kv-cache.md)) | ✅ | ✅ | ✅ | | KV offload to host memory | ✅ | ✅ | ✅ | ☐ | | External KV provider ABI (LMCache) | ☐ | ✅ | ◐ | ☐ | | KV events (block create / evict publish) | ◐ no transport | ✅ | ☐ | ☐ | @@ -62,7 +62,7 @@ are our reading of their documented behavior, not measurements. | Automatic memory sizing (no hand-tuned budget) | ☐ hand-typed block count | ☐ percent, hand-tuned | ☐ | ◐ | | Memory cap with a pre-flight error instead of an OOM | ☐ | ◐ KV pool only | ◐ | ☐ | | Routed-expert weight streaming from disk | ◐ default OFF (`VT_MOE_EXPERT_STREAM=1`), keep-quant/keep-f16 towers (#1378); bounded slot cache; refuses unfittable slices by name. c1-c4 capacity, not throughput. CPU; staging device DECODES, token gate FAILS (#1299) | ☐ blanket `cpu_offload_gb`, not expert-granular | ☐ | ◐ mmap only | -| Hybrid CPU/GPU expert placement (routed-expert compute on the CPU, attention and dense layers on the GPU) | ◐ **Six architecture families.** `RunMoePlaced` routes Qwen3-MoE, Qwen3.5/3.6, Nemotron-H, DeepSeek-V2, Kimi-Linear and dots3-note through one shared placement seam. The `vllm_cpp.placement` object in `--offload-config` maps llama.cpp's `-ot`, `-cmoe`, `-ncmoe`, and `--fit` controls to per-layer decisions. The CPU is the only placement target. Accelerator targets are refused. So is the fp4-resident expert arm, because its device residents are built eagerly at load, so placing it would upload every expert and then compute across the bus, which is a defect a token gate cannot see. Laguna and Gemma4 need different forward interfaces before they can use the seam, and the reasons differ: Laguna runs its expert GEMMs on the DEVICE but presents a per-token host-float FFN boundary, so it has no `[T,H]` block to hand the seam ([#2050](https://github.com/mudler/vllm.cpp/issues/2050)), while Gemma4's expert path is a `void Expert...Accum` accumulate shape rather than a `-> DBuf` one. DeepSeek-V4 runs its experts on the host from host weights, so a placement has nothing to move. dots3-note joined the seam at W5 ([#699](https://github.com/mudler/vllm.cpp/issues/699)): `Dots3NoteMoeBlock` routes through `RunMoePlaced`, so the seam now carries six architecture families, though no dots3-note checkpoint fits any host this project reaches. GLM-5-Next, Kimi-K3 and qwen4_exp have no reachable MoE forward yet, and refuse by name. Unit and round-trip tests pass, but no model has run end to end with placement enabled. The round trip is byte-identical to computing in place, mutation-proven. The token gate is pending, and the speed gate needs a discrete CPU/GPU system ([#149](https://github.com/mudler/vllm.cpp/issues/149), [#2026](https://github.com/mudler/vllm.cpp/issues/2026)) | ☐ CPU MoE kernels exist, but selection requires the whole model to use the CPU platform | ☐ CPU selection also uses module-level platform checks, with no per-layer device override | ✅ `-cmoe` and `-ncmoe` select buffer types from tensor-name patterns, so compute follows weight placement | +| Hybrid CPU/GPU expert placement (routed-expert compute on the CPU, attention and dense layers on the GPU) | ◐ Six families use the shared `RunMoePlaced` path: Qwen3-MoE, Qwen3.5/3.6, Nemotron-H, DeepSeek-V2, Kimi-Linear, and dots3-note. Configure placement with `vllm_cpp.placement` in `--offload-config`; the CPU is the only placement target. FP4-resident experts, Laguna, and Gemma4 remain unsupported. Unit and round-trip tests pass, but the end-to-end token and speed gates are pending ([#149](https://github.com/mudler/vllm.cpp/issues/149), [#2026](https://github.com/mudler/vllm.cpp/issues/2026)) | ☐ CPU MoE kernels exist, but selection requires the whole model to use the CPU platform | ☐ CPU selection also uses module-level platform checks, with no per-layer device override | ✅ `-cmoe` and `-ncmoe` select buffer types from tensor-name patterns, so compute follows weight placement | ## Quantization and weight formats diff --git a/docs/USAGE.md b/docs/USAGE.md index 849ac4b8d..cd5561962 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -248,11 +248,12 @@ anything, rather than writing floats into a half-sized block. Only one of them (Nemotron-H) tells you what you asked for: its refusal names the fp8 KV scheme. Qwen3-VL reaches the store, which names the op that should have been called and says the architecture is not routed for fp8 KV. The other 14 report a dtype rule -instead — 13 say `": KV cache must be bf16 or f32"`, and Gemma-4 dies one +instead: 13 say `": KV cache must be bf16 or f32"`, and Gemma-4 dies one step earlier inside a cast with `"cast_f32: out must be f32"`, which does not even name the architecture. Every one of the 16 refuses before writing, so the half-sized block is never fed floats; what differs is how much the message tells -you. Metal and ROCm refuse it too. See +you. Metal refuses it too. CPU and ROCm store/read component gates pass. The +CUDA device gate is still pending. See [the row spec](../.agents/specs/fp8-kv-cache.md) for the exact list. A refusal arrives AFTER the pool has already been sized at half, which is the