dots3-note W6a: the DENSE vision tower, on a SERVED image request

CMakeLists.txt

@@ -808,6 +808,7 @@ add_library(vllm STATIC
   src/vllm/model_executor/models/dots3_note.cpp
   src/vllm/model_executor/models/dots3_note_attn.cpp
   src/vllm/model_executor/models/dots3_note_device.cpp
+  src/vllm/model_executor/models/dots3_note_vision.cpp
   src/vllm/model_executor/models/dots3_note_registry.cpp
   src/vllm/model_executor/models/qwen4_exp_ple.cpp
   src/vllm/model_executor/models/qwen4_exp.cpp
@@ -1080,6 +1081,7 @@ add_library(vllm STATIC
   # Per-modality input-count limits and the refusal they carry
   # (ENG-MM-INPUT-PIPELINE L1, #607).
   src/vllm/multimodal/processing/context.cpp
+  src/vllm/multimodal/dots3_note_processor.cpp
   src/vllm/multimodal/qwen3vl_processor.cpp
   src/vllm/multimodal/audio_processor.cpp
   # Parakeet / FastConformer audio encoder + CTC head + its log-mel front end
@@ -1243,6 +1245,7 @@ add_library(vllm STATIC
   # registry, and Qwen3-VL's registration into it. The registration TU is
   # SELF-REGISTERING and is reached through vllm's INTERFACE --whole-archive.
   src/vllm/entrypoints/openai/mm_chat_registry.cpp
+  src/vllm/entrypoints/openai/mm_chat_dots3note.cpp
   src/vllm/entrypoints/openai/mm_chat_qwen3vl.cpp
   src/vllm/entrypoints/openai/serving_utils.cpp
   src/vllm/entrypoints/openai/serving_completion.cpp

docs/FEATURES.md

@@ -155,7 +155,7 @@ speed-pending, which [BENCHMARKS.md](BENCHMARKS.md) tracks.
 | `LagunaForCausalLM` | poolside/Laguna-S-2.1-NVFP4, GGUF-Q4_K, Laguna-XS | byte-exact near-tie (distributional vs vLLM) | vLLM parity+ 1.03x, default on, via the `laguna-gen` CLI; the registered engine forward VT_CHECKs non-bf16 (`ARCH-ONE-SURFACE` fold) |
 | `KimiLinearForCausalLM` | Kimi-Linear-48B-A3B (KDA + NoPE-MLA + MoE) | **Folded onto the shared paged runner (ROW 7 §21, #122): engine==CLI 128/128 byte-identical; vs golden 122/128 (the intrinsic near-tie profile); FA2 paged MLA default-ON; SACRED post-fold green** | Served via `vllm_engine_load` + `vllm_complete_tokens` (ABI v13); server 19.0 tok/s wall vs vLLM ~21 (~0.90×), speed residual open |
 | `KimiK3ForConditionalGeneration` | Kimi-K3 (2.8T MoE) | scaffold: registry+config+enumeration gated, forward refuses | HW-infeasible (~1.56 TB); no run |
-| `Dots3NoteForCausalLM` | `dots-studio/dots3-note-prev` @`1e1e7b0c` (280B-A16B multimodal MoE, 576,886,825,984 bytes bf16; the `-fp8` sibling `dots-studio/dots3-note-prev-fp8` @`7c14222e` is 298,673,280,504 bytes = 298.67 GB decimal / 278.16 GiB binary). Headers only — no tensor byte downloaded | W1+W2 scaffold: registry + config gated off the REAL released `config.json`, with one assertion per §4 config trap (ungrouped 1/1 router, GPT-J indexer RoPE, one nextn layer, the two LoRA rescales, the sliding theta); name map accounted **38006/38006** over the WHOLE released index — at W2 that split read 35381 language / 2195 vision / 430 audio, and W5c re-bucketed it (see below) — with the two tower files carried as named W6/W7 deferrals rather than dropped; W4a+W4b-2 put BOTH attention geometries on the DECODE PATH, reached through `ModelRegistry::Forward`: the 13 full-attention layers with the two LoRA rescales, `k_rope_only_layernorm` and the headwise gate, and the 33 sliding-window layers over a PADDED 1088-wide MLA cache row that each layer narrows to its own logical width on read; `vt::MlaDecodeAttention` and `vt::MlaPrefillAttention` grew an optional window whose absent state is bit-identical to no window; W4b-3c put the DSA lightning indexer's SELECTION on the same path, so a SINGLE-SHOT prefill longer than `index_topk` is now served sparsely instead of refused — `vt::MlaDecodeAttention` grew an optional selected-slot arm whose absent state is bit-identical to no selection and whose FULL selection reproduces the dense answer byte for byte, beside a new `vt::DsaIndexerLogits` / `vt::DsaTopkSelect` pair on CPU and CUDA. A STEP in which any request has CACHED CONTEXT and any request is past `index_topk` still REFUSES BY NAME — the sparse route is a property of the step, not of one request — because the indexer's own key cache is a second attention group owned by `KV-DSV4-MULTICACHE` (#1925). W5 put the 45 MoE layers on the same path through `Dots3NoteMoeBlock` over the shared `RunMoePlaced` seam — the UNGROUPED (n_group=1 / topk_group=1) noaux_tc router at 256/8 with the F32 `e_score_correction_bias` feeding the SELECTION and the unbiased scores feeding the routing weights, plus one shared expert at `moe_intermediate_size * n_shared_experts` = 1536 and NOT `intermediate_size` = 13824 — with no `vt` op changed. W5c removed the last refusal, and that one was a DEFECT rather than a gap: the nextn branch was STRICTER THAN UPSTREAM, which drops `model.layers.46.*` and `model.mtp.*` from the main model instead of refusing (`utils.py:542` -> `deepseek_v2.py:1618-1620`; `models/dots3_note/nvidia/model.py:624`), so those 19 tensors are now a NAMED W10 deferral with their own accounting bucket ([#2176](https://github.com/mudler/vllm.cpp/issues/2176)) and the split reads 35362 language / 19 nextn / 2195 vision / 430 audio. **So `Dots3NoteDeviceRefusal` is EMPTY for the released `config.json`, and that is representable rather than runnable**: the MoE alone is 545.82 GB of a 576.89 GB checkpoint (94.62%), which no host this project reaches can hold. What still refuses BY NAME is the blockwise-FP8 sibling (W9 — `weight_block_size [128, 128]` with a `weight_scale_inv` per projection, which this port's bf16 loaders cannot read), GGUF k-quants (W9), and both towers (W6/W7). `supports_multimodal` went TRUE -> FALSE at W5 because the released config became loadable while the vision tower, the audio tower and the multimodal front end are still W6/W7/W8; W8 flips it back | **No oracle, on any host we own** (298.67 GB fp8 against a 122 GiB ceiling), so NO number is claimable on any axis and the e2e gate is an open gap by construction ([spec](../.agents/specs/dots3-note.md) §6.4, #699) |
+| `Dots3NoteForCausalLM` | `dots-studio/dots3-note-prev` @`1e1e7b0c` (280B-A16B multimodal MoE, 576,886,825,984 bytes bf16; the `-fp8` sibling `dots-studio/dots3-note-prev-fp8` @`7c14222e` is 298,673,280,504 bytes = 298.67 GB decimal / 278.16 GiB binary). Headers only — no tensor byte downloaded | W1+W2 scaffold: registry + config gated off the REAL released `config.json`, with one assertion per §4 config trap (ungrouped 1/1 router, GPT-J indexer RoPE, one nextn layer, the two LoRA rescales, the sliding theta); name map accounted **38006/38006** over the WHOLE released index — at W2 that split read 35381 language / 2195 vision / 430 audio, and W5c re-bucketed it (see below) — with the two tower files carried as named W6/W7 deferrals rather than dropped; W4a+W4b-2 put BOTH attention geometries on the DECODE PATH, reached through `ModelRegistry::Forward`: the 13 full-attention layers with the two LoRA rescales, `k_rope_only_layernorm` and the headwise gate, and the 33 sliding-window layers over a PADDED 1088-wide MLA cache row that each layer narrows to its own logical width on read; `vt::MlaDecodeAttention` and `vt::MlaPrefillAttention` grew an optional window whose absent state is bit-identical to no window; W4b-3c put the DSA lightning indexer's SELECTION on the same path, so a SINGLE-SHOT prefill longer than `index_topk` is now served sparsely instead of refused — `vt::MlaDecodeAttention` grew an optional selected-slot arm whose absent state is bit-identical to no selection and whose FULL selection reproduces the dense answer byte for byte, beside a new `vt::DsaIndexerLogits` / `vt::DsaTopkSelect` pair on CPU and CUDA. A STEP in which any request has CACHED CONTEXT and any request is past `index_topk` still REFUSES BY NAME — the sparse route is a property of the step, not of one request — because the indexer's own key cache is a second attention group owned by `KV-DSV4-MULTICACHE` (#1925). W5 put the 45 MoE layers on the same path through `Dots3NoteMoeBlock` over the shared `RunMoePlaced` seam — the UNGROUPED (n_group=1 / topk_group=1) noaux_tc router at 256/8 with the F32 `e_score_correction_bias` feeding the SELECTION and the unbiased scores feeding the routing weights, plus one shared expert at `moe_intermediate_size * n_shared_experts` = 1536 and NOT `intermediate_size` = 13824 — with no `vt` op changed. W5c removed the last refusal, and that one was a DEFECT rather than a gap: the nextn branch was STRICTER THAN UPSTREAM, which drops `model.layers.46.*` and `model.mtp.*` from the main model instead of refusing (`utils.py:542` -> `deepseek_v2.py:1618-1620`; `models/dots3_note/nvidia/model.py:624`), so those 19 tensors are now a NAMED W10 deferral with their own accounting bucket ([#2176](https://github.com/mudler/vllm.cpp/issues/2176)) and the split reads 35362 language / 19 nextn / 2195 vision / 430 audio. **So `Dots3NoteDeviceRefusal` is EMPTY for the released `config.json`, and that is representable rather than runnable**: the MoE alone is 545.82 GB of a 576.89 GB checkpoint (94.62%), which no host this project reaches can hold. What still refuses BY NAME is the blockwise-FP8 sibling (W9 — `weight_block_size [128, 128]` with a `weight_scale_inv` per projection, which this port's bf16 loaders cannot read), GGUF k-quants (W9), and both towers (W6/W7). **W6a ([#2512](https://github.com/mudler/vllm.cpp/issues/2512)) puts the DENSE half of the VISION TOWER on a SERVED request**: an `image_url` chat request travels `ApiServer::handle_chat_completions` -> the architecture-dispatched multimodal chat seam (a `REGISTER_VLLM_MM_CHAT` translation unit of its own, #2481) -> `GPUModelRunner::execute_mm_encoder` (#2398) -> `ModelRegistry::EmbedMm` -> `ModelRegistry::Forward`, and `kDots3NoteFactory` now carries `encode_mm` and `embed_mm`. The arm is `patch_embed` -> blocks 0-24 (fused `attn.qkv` with no bias, per-head `q_norm`/`k_norm` applied BEFORE a 2-D vision RoPE, bidirectional attention, and the three-tensor SwiGLU routed through `layers::MlpGateUpMethodBase`) -> `post_trunk_norm` -> the `patch_merger` adapter folding 4x1536 to 5120, plus a dots3 image processor with its own per-channel normalization, its own `resized_size` and its own marker. **The RELEASED checkpoint still refuses**: 17 of its 42 vision blocks are pyramid MoE (608 routed experts, 1960 of the 2195 `vision_encoder.*` tensors), so the tower refuses at block 25 naming W6b, the chat seam installs REFUSING, and an image request is HTTP 400 while text keeps serving. That is W3's polarity applied to the second tower, not a new exception. Ported from vLLM `nvidia/vision.py` (677 lines) / `nvidia/vision_attention.py` (477 lines) / `common/processor.py` read at `9035151d6`, which is BEYOND our pin `5559679229`; every anchor names that SHA because `vision_attention.py` is already 494 lines at vLLM `main` `7a100bb61`. The gate is a CONSISTENCY gate (spec §6.4 option B): an independent in-test double-precision reference sharing no helper with the implementation, plus a served-request reachability suite whose load-bearing case is two DIFFERENT images compared on logprobs. `supports_multimodal` went TRUE -> FALSE at W5 and FALSE -> TRUE at W6a, and the trail is the honest record of what this port could back at each point; W7 (audio) and W8 (the video/audio front end) are still owed | **No oracle, on any host we own** (298.67 GB fp8 against a 122 GiB ceiling), so NO number is claimable on any axis and the e2e gate is an open gap by construction ([spec](../.agents/specs/dots3-note.md) §6.4, #699) |
 | `NemotronHForCausalLM` | Nemotron-3.5-Lightning-30B-A3B-NVFP4 (`nvidia` @`29f2d174`) | config+enumeration+KV-shape gated; hybrid forward COMPUTES; loader materializes 18487/18487 as SHIPPED; A3 e2e gate 96/96 `STRICT PASS` on GB10 at `0ea5d249f` (#1221); NO run against current `main` | **PAGED (#810 A2-P): K/V go to the runner's pages; conv+SSM rows carry at the metadata's state indices.** G-SAFE: `num_reqs <= 1`. Device `lm_head` (A2-Q2b), UNMEASURED. Owed: FP8 mamba (A2-Q1), MTP, GGUF |
 | `MuseGlimmerForCausalLM` | real tensors, **bf16 depth 4/52 only**: 5 prefill argmax positions match a torch transcription of vllm#51655 and HF. GGUF full depth generates coherently (#347, #359) but is **NOT token-exact** | text forward + loader vs an fp32 reference, per-mechanism property tests, scaffold 11/11, GGUF gate 17/17. An ABSENT config key now takes the architecture's constant (#412): GGUF post-norms ran at 1e-5, not 1e-8 | no vLLM denominator (pin cannot load it); SECONDARY llama.cpp, same GGUF, GB10 CPU: prefill tie **0.997x**, decode 0.232x, RSS 1.92x (#333) |
 | `MuseGlimmerForConditionalGeneration` | vision: **no reference run of any kind**; enumeration gated vs the released 30B index (1436/1436). Image/video need bf16 safetensors: `mmproj-kquant.gguf` is refused by name | perception encoder loaded and wired, so an image or video prompt runs; `perception_emb_norm` now armed by default (#405). Reachability plus placeholder scatter only, no image or video correctness | not measurable; anchored to open vllm#51655 |