Same tokens as vLLM. Same throughput. 140x less to install.
- Continuous batching, paged KV, 43 registered architectures, CUDA / CPU / Metal / Vulkan. No Python anywhere.
+ Continuous batching, paged KV, 44 registered architectures, CUDA / CPU / Metal / Vulkan. No Python anywhere.
@@ -40,9 +40,14 @@
- **2026-08** **EXL3 checkpoints now generate on CPU and CUDA.** A stock
Llama-3.2-1B-Instruct EXL3 checkpoint loads through the shared dense model path and emits text.
The current CUDA path supports its 3-bit body and 6-bit output head. No speed claim is available.
-- **2026-08** **GGUF gains IQ2_XS and IQ4_XS.** Both formats decode and run directly on their
- compressed blocks on CPU. This lets the 101.25 GiB GLM-5.3-Flash GGUF weight tower load without
- expanding to 426.72 GiB. Its model forward is still incomplete.
+- **2026-08** **GLM-5.3-Flash now generates on CPU from a 101.25 GiB GGUF.** The shipped
+ `UD-Q2_K_XL` artifact emits coherent text while keeping IQ2_XS and IQ4_XS blocks compressed.
+ Both formats also have CUDA keep-quant kernels, but this model's CUDA forward and every speed
+ gate remain pending.
+- **2026-08** **GLM-5.3 joins the model registry.** Its GGUF loader and first-token forward run
+ through the shared expert-streaming path. The real 201.83 GiB artifact has not completed a load,
+ and resumed sparse decoding still needs the indexer side cache, so no real-checkpoint token or
+ speed claim is available.
- **2026-08** **Hybrid CPU/GPU expert placement reaches five architecture families.** Qwen3-MoE,
Qwen3.5/3.6, Nemotron-H, DeepSeek-V2, and Kimi-Linear can run routed experts on the CPU while the
rest of the model stays on the selected accelerator. The end-to-end token and speed gates are
@@ -107,7 +112,7 @@ Where that stands today:
ahead at all six concurrencies but only c1 outside our noise band. Also **1.18x llama.cpp's
prefill** on the same GGUF file (denominator SUPERSEDED, see below), and **ahead of MLX-LM on
prefill** on Apple Silicon. Most other architectures are speed-pending, and say so.
-- **Everything.** 43 registered architectures, 38 tool-parser families, structured output including
+- **Everything.** 44 registered architectures, 38 tool-parser families, structured output including
GBNF, three speculative decoders, image, video, and audio input, music generation, external KV
offload, Prometheus metrics, and the SGLang knobs, all in a library you can `dlopen`. Multimodal
input runs on the single-sequence drivers. No multimodal request is served over HTTP yet
@@ -283,7 +288,7 @@ InternLM2/3, MiniCPM and MiniCPM3, Yi, OPT, plus Qwen3-VL and Qwen3.6-27B vision
and Voxtral (audio).
The full architecture matrix (43 registered architectures grouped by family)
+The full architecture matrix (44 registered architectures grouped by family)
| Architecture | Example checkpoint | GGUF | Correctness | Speed |
|---|---|:---:|---|---|
@@ -297,6 +302,7 @@ and Voxtral (audio).
| DeepSeek-V4-Flash (MLA + MHC + DSA) | DeepSeek-V4-Flash-GGUF (80.7 GB, single GB10) | keep-quant | Coherent (near-tie-robust) | Decode beats ds4 1.144x by default (byte-exact) |
| GLM-4 dense | GLM-4-9B-0414 | - | Token-exact | Speed-pending |
| GLM-4.7-Flash (MLA MoE) | zai-org/GLM-4.7-Flash | - | Token-exact (near-tie-robust) | Speed-pending |
+| GLM-5.3 / GLM-5.3-Flash | unsloth GLM-5.3 GGUFs | keep-quant | Flash emits coherent text; GLM-5.3 synthetic first token only | Speed-pending |
| Laguna-S / Laguna-XS 2.1 (MoE) | poolside/Laguna-S-2.1-NVFP4 | NVFP4 + Q4_K | Near-tie (byte-exact) | vLLM parity+ 1.03x by default |
| Kimi-Linear-48B-A3B (KDA + MLA + MoE) | Kimi-Linear-48B-A3B | - | Near-tie (106/128) | 1.59 tok/s, default off |
| Nemotron-H hybrid (Mamba2 + GQA + MoE) | Nemotron-3.5-Lightning-30B-A3B-NVFP4 | NVFP4 | Host gate strict 96/96; GB10 rerun pending | Speed-pending |
@@ -330,7 +336,7 @@ sampler, no logits); upstream is `vllm-project/vllm-omni`. Five conditioning mod
Compressed-tensors NVFP4A16 (W4A16) dense weights also load and compute natively
(RedHatAI/Qwen3-32B-NVFP4A16). Long-context RoPE (YaRN, Llama-3, LongRoPE, dynamic-NTK) and
sliding-window attention are gated feature-positive. The authoritative per-architecture list, bound
-to the C++ registry (all 43 registered architectures with their tested checkpoint and gate, plus the
+to the C++ registry (all 44 registered architectures with their tested checkpoint and gate, plus the
standalone audio/diffusion lanes and the inventoried-but-blocked archs), is in
[docs/FEATURES.md](docs/FEATURES.md); family-by-family lifecycle detail, including what is
hardware-blocked and why, is linked from [Project status](#project-status).
diff --git a/docs/FEATURES.md b/docs/FEATURES.md
index f8fd2c9f9..97cd8e050 100644
--- a/docs/FEATURES.md
+++ b/docs/FEATURES.md
@@ -25,7 +25,7 @@ are our reading of their documented behavior, not measurements.
| Embeddable behind a C ABI | ✅ | ☐ | ☐ | ✅ |
| Weight formats | Safetensors + GGUF | Safetensors | Safetensors | GGUF |
| Correctness gate | token-exact vs vLLM | reference | own | own |
-| Architectures | 43 registered, 27 gated | 130+ | 100+ | 100+ |
+| Architectures | 44 registered, 27 gated | 130+ | 100+ | 100+ |
| Downloadable server binaries | ✅ v0.0.2: eight indexed archives with checksums, provenance, manifests, and SBOMs. Windows ZIP downloads do not exist; native CPU/Vulkan lanes await hosted runtime, dry-run, prerelease, and authenticated audit gates | ✅ wheels/containers | ✅ wheels/containers | ✅ host-specific binaries |
| Native Windows builds | ◐ CPU/Vulkan: `/MT /W4 /WX`, central `NOMINMAX`, UTF-8, aligned allocation, C++20 `std::numbers` pi, runtime ISA dispatch. Local closure includes the float-domain DeepSeek probe; hosted compile/runtime/release pending | ✅ | ✅ | ✅ |
@@ -102,7 +102,7 @@ are our reading of their documented behavior, not measurements.
The supported set is exactly what the C++ registry registers: every
architecture self-registers via `REGISTER_VLLM_MODEL`, and
`scripts/check-supported-models.py` gates this list against the source so it
-cannot drift. Today that is **43 registered architectures**. Each row names the
+cannot drift. Today that is **44 registered architectures**. Each row names the
checkpoint it was gated against and the verdict; caveats are in
[Project status](../README.md#project-status), agent detail in `.agents/model-matrix.md`. A mergeable
gate/up MLP routes through one shared merged-GEMM method, so a tuned arm added
@@ -150,8 +150,8 @@ speed-pending, which [BENCHMARKS.md](BENCHMARKS.md) tracks.
| `DeepseekV4ForCausalLM` | DeepSeek-V4-Flash GGUF (ds4 q2-imatrix, UD-IQ2); the SAFETENSORS arms now get past the tokenizer ([#1924](https://github.com/mudler/vllm.cpp/issues/1924)) | coherent near-tie vs ds4 oracle (vLLM cannot fit one GB10). Tokenizer ids are exact vs HF `tokenizers` on the checkpoint's own 6.4 MB `tokenizer.json`, and the GGUF arm's `joyai-llm` pre no longer resolves to an APPROXIMATION | decode beats ds4 1.144x, default on, via the `deepseek-v4-gen` CLI; the registered engine publishes DeepSeek-V4's real seven-group / 167-entry cache topology ([#1973](https://github.com/mudler/vllm.cpp/issues/1973)) and the runner now ALLOCATES all 167 of them ([#2068](https://github.com/mudler/vllm.cpp/issues/2068)), handing them to the forward keyed by the name each was published under; the FORWARD then refuses, because no registered forward consumes a cache set keyed that way yet (W5). At the default `--block-size` 32 a run reads the factory's own refusal first, since a compress-ratio-128 page needs 128 or 256. So the engine still cannot serve, one seam further along than it was |
| `Glm4ForCausalLM` | GLM-4-9B-0414 | near-tie 16/16 vs vLLM 0.25.0 | pending |
| `Glm4MoeLiteForCausalLM` | zai-org/GLM-4.7-Flash (31.2B, MLA MoE) | near-tie 8/8 vs vLLM 0.25.0 | pending |
-| `Glm5NextForConditionalGeneration` | GGUF: `unsloth/GLM-5.3-Flash-GGUF` UD-Q2_K_XL @ `d425e572f`, 101.2535 GiB in four shards — **LOADS on `--device cpu`, and the engine's multi-KV guard no longer refuses above the model's forward** (W5b-2c, [#2348](https://github.com/mudler/vllm.cpp/issues/2348); W5c, [#2242](https://github.com/mudler/vllm.cpp/issues/2242); KV-cache spec + MoE W5, [#2223](https://github.com/mudler/vllm.cpp/issues/2223); the forward W5b-2b, [#2337](https://github.com/mudler/vllm.cpp/issues/2337)). MEASURED on `dgx:gpu0` 2026-08-30 ([#2343](https://github.com/mudler/vllm.cpp/issues/2343)): all four shards load and the engine sizes its caches -- `max_model_len` auto-fits from 1048576 to 8192 against 256 blocks of 32 tokens, and `max_num_seqs` drops from 32 to 1 because one 4,390,912-byte GDN state fills a unified page -- in under 26 minutes wall, which is a DURATION and not a throughput number. At THAT change the first step then threw at the `multi_kv` guard at the TOP of `ModelRegistry::Forward`; W5b-2c ([#2348](https://github.com/mudler/vllm.cpp/issues/2348)) is the consuming forward that guard was waiting for, so it no longer fires for this model. **THIS ARTIFACT GENERATES COHERENT TEXT ON `dgx:gpu0`** as of [#2241](https://github.com/mudler/vllm.cpp/issues/2241). MEASURED 2026-08-30 on GB10 in the SHIPPED configuration (no diagnostic env set), `vllm-cli --device cpu --max-tokens 2` at the prompt `The capital of France is`: it emits ` Paris.`, `rc=0`, `prompt_tokens=5 completion_tokens=2`, with **peak RSS 104,792,300 kB = 99.94 GiB** (`VmHWM`, polled) against the 99.47 GiB the broken binary read on the same box. The mechanism and the per-layer bisect come from two instrumented `thor:gpu0` runs the same day, where four tokens read ` Paris. Paris is`, the prefill top-5 is ` Paris` (16.427) ahead of ` one`, ` located`, ` known` and ` a` at a margin of 1.279, and no NaN appears in any of 180 per-layer readings across four steps. The first generation attempt, at W5b-2c on `dgx:gpu0`, emitted token id 0 eight times because the loader repacked all 346 of this file's q8_0 tensors into the `block_q8_0x4` i8mm interleave while the host bridge decoded them as plain blocks -- see the spec's `## Owed` O30 for the bisect. **NO SPEED NUMBER IS CLAIMED and the earlier ones are VOID**: the 73 s/token of the W5b-2c run came from an all-NaN forward whose degenerate expert selection is not this model's work. the GB10 arm is the one measured above | **THE WEIGHT TOWER IS PORTED AND THE FORWARD READS THE ENGINE'S PAGED CACHES.** The config resolves and validates against transformers **v5.16.1**, the only revision of any admissible oracle that implements `glm5_next` — vLLM implements it at NO revision, and [vllm#53906](https://github.com/vllm-project/vllm/pull/53906) is open and therefore inadmissible. All five upstream `validate_architecture` rejections are implemented, and both sources — a `config.json` and a GGUF — descend through ONE parser. The GGUF arm of `load_weights` now returns a real `LoadedModel`: the KDA layer with its three separate depthwise convs, the NoPE MLA with the two SPLIT absorbed halves, the DSA k-pool indexer, the flat mHC pair, the 288 stacked routed experts plus one shared, and the dense MLP on the leading three layers. The name map is gated against the REAL 1412-tensor artifact with no asset, in both directions, and `blk.45` — the multi-token-prediction block — is read, counted and NOT built as a decoder layer. **`ModelRegistry::Forward` DISPATCHES to the model** as of W5b-2b ([#2337](https://github.com/mudler/vllm.cpp/issues/2337)), which is what discharges the six "gated but reached by nothing" debts this row carried, **and the ENGINE path now REACHES that dispatch** as of W5b-2c ([#2348](https://github.com/mudler/vllm.cpp/issues/2348)), which writes the forward the `multi_kv` guard at the top of the same function was waiting for ([#2343](https://github.com/mudler/vllm.cpp/issues/2343), [#2068](https://github.com/mudler/vllm.cpp/issues/2068)): each DSA layer's MLA latent and indexer side cache are found BY NAME on `MultiKvCacheIndex` and read out of the engine's own pages, the 34 KDA states come off `gdn_state` positionally because that channel carries no names, and each step's new rows are written back into those pages rather than kept on the model. The tower stays block-resident exactly as loaded, ONE decoder layer at a time is bridged to host f32 and dropped, and only the 8 of 288 experts a token selects are decoded. That is arithmetic and not preference — a float tower is 426.72 GiB and the 42 sparse layers' expert banks alone are 1,134 GiB, against ~119.63 GiB usable on the largest box this project reaches, while the streamed forward's f32 peak is under 0.75 GiB. The vision tower and the safetensors arm still REFUSE BY NAME, as do a multi-request step (this forward is single-sequence and ragged batching is owed) and a non-CPU queue (every primitive here is a host f32 reference and the device arm is owed). **The KV-cache spec no longer does** (W5, [#2223](https://github.com/mudler/vllm.cpp/issues/2223)): `make_kv_cache` publishes three groups -- an `MLAAttentionSpec` at head 512 for the 11 DSA layers, ONE `MambaSpec` for the 34 KDA layers, and a second `MLAAttentionSpec` at head 257 for the indexer side cache -- and it is REACHED through the production factory hook. W5's 288+1 expert MoE block (`glm5_next_moe`) is now REACHED by the forward, along with W2's KDA arm, W3's DSA indexer, W4's mHC bricks and W5b-1's attention; deleting the production call site in the registry hook reds the focused gate. **Use `--device cpu`, and as of [#2260](https://github.com/mudler/vllm.cpp/issues/2260) the reason is no longer the quantization**: the artifact's 82 IQ2_XS and 3 IQ4_XS tensors now HAVE a CUDA keep-quant kernel, so the expert GEMM no longer drains the stream to the host and the fused MoE seam no longer throws — but this forward is a host f32 reference and refuses a non-CPU queue BY NAME before any GEMM runs, so `--device cuda` is still an error message and the device arm is still owed by this row. **NO end-to-end token gate exists or can exist on this fleet** and that is a measured fact, not a schedule: no oracle registers this architecture at any revision it can also RUN here | none, and no speed claim is admissible from this row until a correctness gate exists |
-| `GlmMoeDsaForCausalLM` | zai-org/GLM-5.3 (753.33B, DSA sparse MLA MoE), HF revision `935644c05e76fc198714f4cca449fd8b970ff6d7` — **REGISTERED AND VALIDATING; IT LOADS NOTHING AND FORWARDS NOTHING** (W2, [#2214](https://github.com/mudler/vllm.cpp/issues/2214)). The architecture resolves, its config parses from a `config.json` and from a `glm-dsa` GGUF header through one validator, and the `glm-dsa` row of the GGUF dispatch table is reached from `LoadedEngine::FromModelDir`. Both `load_weights` arms refuse by name: safetensors permanently (703.74 GiB across 141 shards, no streaming loader, no MoE-expert block-fp8 rung) and GGUF until W7. The forward refuses and names all seven missing primitives — the indexer KV side cache, the expert-streaming seam, sparse prefill, the per-layer indexer schedule reuse, the `IQ4_XS` keep-quant `vec_dot`, the loader and the fp32 router GEMM. The one staged GGUF arm states no `glm-dsa.attention.indexer.types` and is refused rather than resolved off llama.cpp's hardcoded table | **NONE, and none is reachable on this fleet.** vLLM implements this architecture at the pin `5559679229bc961848b121ccdeaa8fa5d79bec98` (`registry.py:117`) and cannot fit it on any device this project reaches, so no wave may promise a token-exact number against it | no run, no number |
+| `Glm5NextForConditionalGeneration` | `unsloth/GLM-5.3-Flash-GGUF` UD-Q2_K_XL @ `d425e572f`, 101.2535 GiB in four shards | **CPU reference forward, coherent output, not oracle-gated.** The real artifact loads in 99.94 GiB peak RSS and emits ` Paris.` for a two-token smoke prompt. The forward reads the engine's paged MLA, KDA, and indexer caches while keeping the expert tower block-resident. It is single-sequence, CPU-only, and GGUF-only. The vision tower, safetensors loader, multi-request steps, and device forward refuse by name. No runnable vLLM revision implements this architecture, so no end-to-end parity or speed claim is available ([#2241](https://github.com/mudler/vllm.cpp/issues/2241), [#2260](https://github.com/mudler/vllm.cpp/issues/2260), [#2348](https://github.com/mudler/vllm.cpp/issues/2348)). | none |
+| `GlmMoeDsaForCausalLM` | `unsloth/GLM-5.3-GGUF` UD-IQ1_S, 201.83 GiB in six shards | **Synthetic first-token forward only.** The GGUF loader and first-token forward run through the shared expert-streaming path and produce finite logits on a complete synthetic model. The published GGUF needs a derived indexer-schedule metadata key, has not completed a real load, and has no generated token. A resumed sparse step still refuses until the indexer side cache is connected. Safetensors remains unsupported because the 703.74 GiB checkpoint cannot fit the project fleet ([#2214](https://github.com/mudler/vllm.cpp/issues/2214), [#2323](https://github.com/mudler/vllm.cpp/issues/2323)). | no real-checkpoint run or speed number |
| `LagunaForCausalLM` | poolside/Laguna-S-2.1-NVFP4, GGUF-Q4_K, Laguna-XS | byte-exact near-tie (distributional vs vLLM) | vLLM parity+ 1.03x, default on, via the `laguna-gen` CLI; the registered engine forward VT_CHECKs non-bf16 (`ARCH-ONE-SURFACE` fold) |
| `KimiLinearForCausalLM` | Kimi-Linear-48B-A3B (KDA + NoPE-MLA + MoE) | **Folded onto the shared paged runner (ROW 7 §21, #122): engine==CLI 128/128 byte-identical; vs golden 122/128 (the intrinsic near-tie profile); FA2 paged MLA default-ON; SACRED post-fold green** | Served via `vllm_engine_load` + `vllm_complete_tokens` (ABI v13); server 19.0 tok/s wall vs vLLM ~21 (~0.90×), speed residual open |
| `KimiK3ForConditionalGeneration` | Kimi-K3 (2.8T MoE) | scaffold: registry+config+enumeration gated, forward refuses | HW-infeasible (~1.56 TB); no run |
@@ -218,9 +218,9 @@ Enumerated in `.agents/model-matrix.md`, not registered, no runnable GB10 gate:
| `MiniMaxM2ForCausalLM` | MiniMax-M2 | ~230B, ~428 GiB bf16, ~4x over the unified pool |
| `Dots3NoteMTPModel` | dots3-note nextn head (the target arch `Dots3NoteForCausalLM` IS registered; see the supported table above) | W10 owns it and it is deliberately NOT registered: a speculator that cannot propose makes the engine accept a speculative config it then dies on mid-run. The checkpoint ships exactly one nextn layer, and since W5c ([#2176](https://github.com/mudler/vllm.cpp/issues/2176)) its 19 tensors are a NAMED W10 deferral in the language tower's accounting rather than a refusal, which is what vLLM's own loader does with them. Blocked behind the target row: no oracle runs here, 298.67 GB fp8 against a 122 GiB ceiling, so NO number is claimable on any axis ([spec](../.agents/specs/dots3-note.md), #699) |
-27 of the 39 registered text-generation architectures carry a passing
+27 of the 40 registered text-generation architectures carry a passing
correctness gate today; the rest are honestly marked scaffold or blocked above.
-(The 43 registered total also covers 3 Parakeet ASR entry points and the
+(The 44 registered total also covers 3 Parakeet ASR entry points and the
`LlamaModel` embedding arch, which are not text generation.)
vLLM registers 130+ text architectures, so this is a curated, gated subset, not
a breadth claim. The first EMBEDDING architecture is registered and live