diff --git a/.agents/model-matrix.md b/.agents/model-matrix.md index c339fee66..d360a63df 100644 --- a/.agents/model-matrix.md +++ b/.agents/model-matrix.md @@ -121,7 +121,7 @@ Engaged architectures (the 55 non-`INVENTORIED` rows): | 🚧 | `KimiLinearForCausalLM` | Kimi-Linear-48B-A3B | **PAGED-INCREMENTAL DECODE LANDS the 5Γ— speed win (2026-08-07, Β§19, `row/KIMI-PAGED-INCREMENTAL` #113):** the Β§18 real lever (e) BUILT β€” `KimiDecodeCache` + `ForwardPrefillIncremental`/`ForwardDecodeStepIncremental` (`kimi_linear_device.cpp`): prefill-once (KDA recurrent+conv state carried via `vt::KdaGatedDeltaRule` state in/out + `vt::CausalConv1dFwd` tap-carry; NoPE-MLA latent-KV cached) + recurrent decode-step, MIRRORING vLLM `kimi_gdn_linear_attn._forward` (prefill=`chunk_kda_with_fused_gate` output_final_state / decode=`fused_recurrent_kda` initial_state, `vllm-src` `a4e3cb4`; divergences: host state vs paged slot cache, materialized-MHA MLA vs paged-FA2 β€” named residuals). CPU byte-exact state-carry gate `test_kimi_linear_forward` **15/15Β·875** (NEW case l: carried decode == fresh full-recompute byte-identical + greedy-identical). Full 48.9B GB10 (single-load/config, flock, drop_caches, min-avail 18-21 GiB, no reboot, Β§12 golden md5 `bfa5bdbf`): recompute 122/128 @ 4.23 tok/s (reproduces #111); incremental+recurrence 120/128 @ 16.63; **incremental+chunk-prefill 122/128 @ 18.87/19.03 tok/s (2 runs) β€” token-IDENTICAL to recompute (Gate A PASS, p7 `got` byte-exact) = 4.5Γ— over recompute, 0.90Γ— of vLLM ~21 (the 5Γ— decode gap 0.20Γ—β†’0.90Γ—)**. Gate B STRICT NOT reached (122/128): chunk-prefill in the RIGHT vehicle reproduces recompute EXACTLY, does NOT close p7 β€” REFUTES the #111 "p7 in the right vehicle β†’ STRICT" hypothesis; p7 intrinsic (Β§13/Β§14 f32-vs-bf16 near-tie at a comma). Decode decomposition (nsys, ours, 99 steps, same-tool): **~90% is the SAME cuBLAS `internal::gemvx::kernel` vLLM calls (batch-1 GEMV-parity)**, KdaScanKernel 2.3%, MoE glue 2.3%, CastBf16 3%; chunk kernels 20 inst = prefill only (prefill=chunk/decode=recurrent IN VIVO). Killing O(nΒ²) ALONE reaches parity-class; no lever load-bearing beyond it β€” residual = ~15% host-orchestration idle + 3% CastBf16 (a bf16 residual stream = the ONE lever ALSO closing the p7-STRICT near-tie) + paged-FA2 MLA decode. vLLM-live-nsys@0.82 NOT run (box-safety: 95-98 GiB reservation + nsys below the 15 GiB LIFE-CRITICAL floor). `--incremental` opt-in; `VT_KIMI_DEVICE_KDA`/`_CHUNK` STAY OFF (122/128 β‰  STRICT). Row STAYS 🚧. **chunk_kda PREFILL PHASE-2 MEASURED β€” op CORRECT, chunk-EVERY-STEP REGRESSES 122β†’102 (2026-08-07, Β§18, `row/KIMI-CHUNK-KDA-P2` #111):** the `chunk_kda` prefill family regenerated + vendored for ALL 6 arches (reproducible β€” only new `kda_*`+MANIFEST; GDN cubins byte-identical; drift GREEN) + wired through the new op `vt::KdaChunkPrefill` (the 6-cubin `_chunk_kda_fwd_with_cumulative_g`; `cuda_gdn.cu.o` -Werror clean; RED-first unit `test_ops_kda_chunk_prefill` **2/2Β·4** on GB10 β€” chunk-vs-recurrence mean_abs **4.68e-5**, wrong-gate **72Γ—**; GDN untouched 66/66Β·4242). Full 48.9B GB10 gate (flock, min-avail 21 GiB, no reboot): control device-KDA reproduces **122/128, 4.24 tok/s** EXACTLY; **+chunk-prefill (`VT_KIMI_DEVICE_KDA_CHUNK`) REGRESSES to 102/128, 4.08 tok/s** (p3 16β†’3, p6 16β†’11). Root cause: the island's O(nΒ²) recompute applies chunk EVERY decode step over the growing sequence β€” NOT vLLM's prefill=chunk/decode=recurrent split β€” so it coin-flips near-ties the recurrence-every-step (control) doesn't (the recurrence matches vLLM's DECODE; chunk only matches its PREFILL). vLLM speed arm (Β§12 recipe, util 0.82, triton MoE, eager, single-seq; min-avail 15 GiB, no reboot): **~21 tok/s median** 16-token aggregate (25.3 cold-discarded; TTFT not isolable in 0.25.0) vs ours **4.24** (recurrence) / **4.08** (chunk) STEADY decode β†’ **ours/vLLM β‰ˆ 0.20** (vLLM ~5Γ— faster on decode β€” the O(nΒ²)-recompute vs paged-incremental distance, = the coupled STRICT+speed lever). `VT_KIMI_DEVICE_KDA_CHUNK` STAYS OFF (a regression isn't a flip); device-KDA (122, OFF) still best. The op + regen are the validated prefill half of the named real lever (e) paged-incremental decode (chunk-prefill ONCE + recurrent-decode over PERSISTENT state β€” kills the O(nΒ²); the STRICT + speed lever, coupled). Row STAYS 🚧. **DEVICE-KDA GB10 122/128 + 4.24 tok/s (Β§15, #104); device NoPE-MLA lever MEASURED-NEGATIVE (2026-08-07, Β§16, `row/KIMI-STRICT-CLOSE` #107):** the per-channel-decay device recurrence `vt::KdaGatedDeltaRule` moves 106β†’**122/128** (p0-p6 16/16; sole p7 pos-6 comma near-tie) AND **1.35β†’4.24 tok/s (3.1Γ—)** β€” vLLM's ACTUAL f32-on-bf16 arithmetic, beats Β§14's host-precision 120. The Β§15 residual (d) was attempted in device-COMPUTE form: `VT_KIMI_DEVICE_MLA` routes the 7 NoPE-MLA layers' softmax core through `vt::Attention` (pad-V: value zero-padded qk_nope+qk_rope=192 vs v=128, `out[:,:,:v]` byte-exact). CPU RED-first gate GREEN (`test_kimi_linear_forward` **14/14Β·825**, pad-V==f64 ref rtol 3e-3; perturbation fails 108). Full 48.9B GB10 gate (single-load, flock, min-avail 21 GiB, no reboot): control device-KDA reproduces **122/128, 4.24 tok/s** EXACTLY; **+device-MLA REGRESSES to 109/128 AND 3.89 tok/s** β€” `vt::Attention`'s f32 online-softmax is the right math but a DIFFERENT reduction order than vLLM's FA2, so it coin-flips near-ties (breaks p3 16β†’3 into Β§14's `163586Γ—` repeat) and the per-(t,h) build slows the O(nΒ²) recompute. `VT_KIMI_DEVICE_MLA` STAYS OFF, kept as a documented-MEASURED-NEGATIVE A/B knob (Β§14 `ISLAND_F32ACC` precedent). MLA dims VERIFIED from the real config (nah=32, qk_nope=128, qk_rope=64, v=128, kv_lora=512, q_lora=None; 7 full-attn/20 KDA). Both device knobs default OFF (122 β‰  STRICT, K=3-deterministic golden). STRICT residual, sharpened: needs vLLM's ACTUAL kernels β€” (c) chunk_kda prefill family (Triton-AOT regen for sm_121a) + (d) paged FA2 `mla::ForwardMlaAttentionBlock` (NOT the vt::Attention approximation) + (e) paged-incremental decode (needs a decode/paged-attn op, query_lenβ‰ key_len; kills the O(nΒ²)) β€” each a substantial multi-kernel brick (Β§16). Row STAYS 🚧. **FULL-MODEL GB10 e2e RUNS β€” NEAR-TIE 106/128 (2026-08-06, `row/MODEL-KIMI-LINEAR-BF16`):** the bf16-resident path CLEARS the f32-loader block β€” the full 48.9B model now runs e2e on one GB10. dgx CUDA build (`-Werror` clean, 14 GDN AOT symbols nm-linked, `test_kimi_linear_forward` 13/13Β·656 in the CUDA binary); `kimi-linear-gen --gpu` greedy-decodes the Β§12 8-prompt battery x16 vs `greedy_ids.npy`. MEMORY: load 117.6s, host RSS PEAK **1.7 GiB** (stage-then-ReleaseHost), device peak 98.5 GiB, min-avail **21.6 GiB** (above the 15 GiB floor, matches the ~25 GiB pool-math headroom), NO OOM/reboot. TOKEN gate **NEAR-TIE 106/128 (82.8%)** β€” prompts 0,1,3,4,5,6 are 16/16 token-exact, p2/p7 diverge at punctuation/word near-ties; 96 consecutive exact tokens across 6 prompts prove the WIRING (a wiring bug can't). Root cause (honest): the f32 residual stream + host-f64 islands are MORE precise than vLLM's bf16 device kernels, so they flip the argmax where vLLM's deterministic bf16 top-1 has a small margin. STRICT path = the named W7-speed residuals (device GDN/MLA islands -> bf16 stream matching vLLM's rounding). 1.59 tok/s (recompute+island rate). `VT_KIMI_DEVICE_COMPUTE` STAYS OFF (parity-enablers: near-tie != token-exact). Row STAYS 🚧. **bf16-RESIDENT loader/forward IMPLEMENTED + CPU-gated (2026-08-06, `row/MODEL-KIMI-LINEAR-BF16`):** the Β§13 design is coded β€” `LoadKimiLinearResidentBf16Weights`/`StageKimiResidentBf16`/`BuildKimiResidentFromHost` (`kimi_linear_weights.cpp`; `LoadBf16Direct` -> `OwnedTensor`, per-tensor stage-to-`d_dev` + `ReleaseHost`, tiny vectors host f32), `KimiLinearResidentWeights` (`kimi_linear.h`), bf16 device forward `DeviceForwardBodyBf16` + `Gemm Bf16` cast-act at ~20 GEMM sites with the two host-fallback islands EXTRACTED+shared (`kimi_linear_device.cpp`), `ForwardDevice` resident-path dispatch (`kimi_linear.cpp`), and the `kimi-linear-gen` e2e harness. CPU **13/13Β·656** (12/12Β·614 f32 path UNTOUCHED + NEW tiny-config bf16-vs-f32 gate). PENDING: dgx CUDA build + full-model GB10 e2e vs the STRICT golden. Row STAYS 🚧. **bf16-RESIDENT brick POOL-MATH+DESIGN (2026-08-06, `row/MODEL-KIMI-LINEAR-BF16`):** pool math CLOSES (91.5 GiB bf16 device-resident + ~2.4 GiB act/norms/ctx β‰ˆ 94 GiB, ~25 GiB headroom); design grounded Β§13 (Laguna `GemmBf16` cast-act + `OwnedTensor::d_dev`, `LoadBf16Direct`, f32 `MaterializeHost` kept for the unit gate). Impl (loader/forward rewrite + gate + e2e) pending. Row STAYS 🚧. **Β§8 GOLDEN CAPTURED β€” STRICT (2026-08-06, `row/MODEL-KIMI-LINEAR-E2E`):** the Β§8 SACRED oracle golden is captured on GB10 (0.25.0-stage, util 0.82, moe=triton, min 15 GiB avail, NO reboot), **8/8 prompts DETERMINISTIC over K=3 β†’ STRICT gate**, committed at `tests/parity/goldens/kimi_linear_greedy/`. Full our-engine e2e BLOCKED on OUR f32 loader (materializes ~183 GiB > 119 pool), the bf16-residency residual; row STAYS 🚧. **W7 GPU-VERIFY (2026-08-06, branch `row/MODEL-KIMI-LINEAR-GPU`):** the device compute runs **12/12Β·614 GREEN on GB10 sm_121a CUDA build**, BOTH arms (`VT_KIMI_DEVICE_COMPUTE=1` + host-ref); prod stack (CUTLASS-NVFP4 GEMM + FA2 ENABLED + Triton-AOT GDN, 14 cubins nm-verified); f32 device==W2 ref, no divergence, no DeepSeek-class trap. Oracle gateability re-confirmed (0.25.0-stage registers `KimiLinearForCausalLM`). e2e Β§8 SACRED golden STILL disk-blocked (91.5 GiB checkpoint absent, dgx root 100% full, 34G free). Row STAYS 🚧. **W7 DBuf-resident device COMPUTE landed, CPU-gated** (`CLAIM-KIMI-LINEAR-W7`): the real device compute (`ForwardDeviceCompute`, `kimi_linear_device.cpp`) composes the whole 27-layer KDA/NoPE-MLA + 256-expert-MoE hybrid over pooled f32 `DBuf`s through the SHARED `vt::` ops (embed/`FusedChain` add+RMSNorm/`MatmulBT` projections/`CausalConv1dFwd` convs/`L2Norm`/`RmsNormGated`/`MoeRouterTopK` sigmoid-`noaux_tc`/`MoeSiluMul`/`MoeCombine`/lm_head), returning DEVICE-RESIDENT logits; 2 documented HOST-FALLBACK islands (the KDA per-k-channel gated-delta recurrence + its exp/softplus decay gate β€” `vt::GdnDecode` carries only a per-HEAD scalar decay; the NoPE-MLA softmax core β€” the paged `mla::ForwardMlaAttentionBlock` device path is born-on-runner) are the W7-speed residuals. CPU-gated vs the W2 host reference (the CPU backend runs the SAME `vt::` dispatch): `test_kimi_linear_forward` **12/12Β·614** (per-op KDA/NoPE-MLA/MoE/dense device==ref within f32-accumulation tolerance; the whole `ForwardDeviceCompute` == ref logits + greedy-token-identical + device-resident). Runner opt-in via `VT_KIMI_DEVICE_COMPUTE=1` (default OFF keeps the CPU-verified W6 host-ref compose). GPU numerics (bf16 activations, GDN Triton-AOT cubins, paged het-KV, grouped-MoE slabs) + the e2e SACRED golden stay a NAMED pending (box down) β€” row STAYS 🚧. ON TOP OF **W6 DEVICE forward SEAM** (`CLAIM-KIMI-LINEAR-W6`): the born-on-the-runner `ForwardDevice` (the DEFAULT `gather_logits` runner path) no longer refuses β€” it composes the `[rows,vocab]` logits via the CPU reference and hands them back DEVICE-RESIDENT (a pooled `DBuf`, wrapped like deepseek_v2 `WrapDeviceLogits`; `on_device()==true` on CPU+CUDA) so the on-GPU sampler consumes them with NO host download. Kimi-Linear now ROUTES device-resident (`check-runner-routing-consistency` reclassifies it, refuse-skipped stubs 2β†’1, NO allowlist; `check-fusion-consistency` green); `test_kimi_linear_forward` **7/7Β·300** (adds the `ForwardDevice`==host-ref device-resident gate). The DBuf-resident device COMPUTE (KDA via the GDN family, NoPE-MLA via `mla::ForwardMlaAttentionBlock`, DeepSeek-V2 grouped-MoE over the paged het-KV; full plan in `kimi_linear.cpp`) is the GPU-verify-pending W7 residual. ON TOP OF **W2-W6 CPU REFERENCE forward** (`CLAIM-KIMI-LINEAR-W2`): the real host `KimiLinearModel::Forward` composes the whole 27-layer hybrid from the landed primitives (KDA layer via `vllm::kimi_kda` refs + the gated-delta recurrence; NoPE-MLA materialized-MHA ref; sigmoid `noaux_tc` MoE + shared expert; dense SwiGLU); loader now materializes host float weights; `test_kimi_linear_forward` 6/6Β·246 (per-op gates + finite whole forward + greedy decode). ON TOP OF **W1 scaffolding** (registry + `ParseKimiLinearParams` 20 KDA + 7 NoPE-MLA + index-verified name-map + het-KV spec). e2e-gateable (FITS one GB10, 0.77Γ— pool). RESIDUAL = the DEVICE born-on-runner forward (KDA kernel/absorbed-MLA/grouped-MoE slabs) + the W0/W7 e2e SACRED golden. **ROW 7 RUNNER FOLD (2026-08-07, Β§21, #122): Kimi decode now runs THROUGH `ModelRegistry::Forward` on the runner's paged state (B1 config synthesis, B2 KDA-paged via `KdaChunkPrefill`/`KdaGatedDeltaRule` over `gdn_state`, B3 paged NoPE-MLA with `mla::ForwardMlaAttentionBlock` default-ON, B4 `vllm_complete_tokens` ABI v13 + thin-client example). GB10: engine==CLI 128/128 byte-identical, vs golden 122/128 (the intrinsic near-tie profile), SACRED post-fold green (35B 315/315, 27B 235/235); server stream 19.0 tok/s wall vs vLLM ~21 (~0.90Γ—) = the open speed residual.** Row stays `ACTIVE` on the speed thread | `MODEL-TEXT-kimi-linear-kimi-linear-for-causal-lm` | | πŸ“‹ | `MuseGlimmerForConditionalGeneration` | Muse Glimmer 30B (Meta, agentic multimodal; iRoPE + gated attn + perception encoder + DFlash) | **W0-W7 ON `row/MODEL-MUSE-GLIMMER` (PR #279, NOT merged): 52-layer text forward, 50-layer perception encoder, mm wiring, ATEM reasoning + tool parsers β€” the forward no longer refuses.** The evidence behind that is NARROW and says so: text agrees with a torch transcription of #51655 and with HF's own `muse_glimmer` on REAL 30B tensors at **reduced depth 4/52 only** (5 prefill argmax positions, no generated tokens); full depth never ran, multi-step decode is untested, the perception encoder has NO reference check, nothing has run through the server, and the ATEM parsers' channel scoping does not work at server defaults (the seam has no `adjust_request` dispatch site and `skip_special_tokens` defaults true, so the framing is stripped first). The lifecycle token stays `SPIKE`/πŸ“‹ deliberately: advancing it owes `docs/STATUS.md` and `.agents/NOW.md` rows, and both are byte-exactly at their shrink-only ratchet/budget, so the advance belongs to the landing commit that can pay for them. Released 2026-08-08, **BEYOND-PIN**: no Muse code at `555967922` nor on vLLM `main`; the only upstream impl is the OPEN PR [#51655](https://github.com/vllm-project/vllm/pull/51655) head `075d645af` (approved, 3/20 CI red), ported from on explicit developer direction as a tracked exception (porting-inventory Β§9 deviation 16). **No oracle β‡’ no speed axis is claimable**; correctness gates against the HF reference. Text tower fully traced (Gemma-style sandwich norms w/ baked +1 offset + split eps, Llama-4-style iRoPE NoPE/full vs RoPE/sliding, weightless pre-RoPE QK-norm, ~3.87 query pre-scale w/ dual config schema, Qwen3.5-style attn output gate reading the LAYER INPUT); 50-layer perception encoder (linear patchify, bilinear pos-emb interp, width-first 2D RoPE, block-windowed attn, pixel-shuffle merge) reuses the Qwen3-VL vision seam; DFlash is recognition + `is_neox_style` threading on the existing speculator, not a new drafter | `MODEL-MM-muse-glimmer-muse-glimmer-for-conditional-generation` | | πŸ“‹ | `KimiK3ForConditionalGeneration` | Kimi K3 (2.8T MoE + MoonViT-V2, DERIVE-AND-SHIP) | **W2/W5 CPU scaffolding landed** (registry stub + nested text/vision/quant config descent + text-backbone structural name-map + REFUSE-by-name forward + MXFP4-refuse loader; clean CPU build, scaffold gate 6/6). text backbone IS `KimiLinearForCausalLM` (KDA+MLA+MoE hybrid, HEAVY reuse); **does NOT fit GB10 (~1.56 TB MXFP4, ~12Γ—)** and NOT in the pinned oracle β‡’ no on-box golden β€” DERIVED, proxy-gated on Kimi-Linear-48B; forward + MXFP4 + KDA delta + MoonViT-V2 not implemented (NOT-YET-BUILDABLE) | `MODEL-MM-kimi-k3-kimi-k3-for-conditional-generation` | -| πŸ“‹ | `Dots3NoteForCausalLM` | dots3-note (280B-A16B multimodal MoE: DeepSeek-V3.2 text tower + sliding-window MLA, MoE ViT, `dots` audio encoder) | **W2 LANDED (the whole weight map): the complete released shard index read for real β€” 38006/38006 accounted, 35381 language / 2195 vision / 430 audio, every bucket asserted BY NUMBER, zero unaccounted, and the two tower files carried as NAMED W6/W7 deferral records rather than integer counters.** Headers only: 4770592 bytes over 133 safetensors headers, no tensor byte. It settled three things the W1 slice could not β€” the backbone has exactly FOUR distinct layer shapes so the 1/2 repeat holds, the full/sliding split derived from the shipped indexer tensors matches `config.layer_types` exactly, and the checkpoint carries 62 F32 tensors in TWO families (the 45 language `e_score_correction_bias` W1 predicted plus 17 vision `router_bias`, spec R5's shape). And it FOUND one: the index declares `indexer_rope_layout: "leading"` / `converted_from: "tail"`, which no upstream code reads ([#1846](https://github.com/mudler/vllm.cpp/issues/1846)); W3 owes the slice. **W1 LANDED (config + registry): the arch RESOLVES, parses the REAL released `config.json`, and accounts 1614/1614 tensors on a committed slice of the released shard index; load, GGUF and the forward each REFUSE BY NAME.** All six Β§4 config traps are gated RED-first (evidence table in spec Β§4.1), which matters more here than on any other row because spec Β§6.4 records that NO oracle runs on hardware we own, so no token gate exists downstream. W1 also RESOLVED Β§1.4 from the checkpoint: exactly ONE nextn layer (`model.layers.46.*`, 18 tensors) with the SLIDING geometry and a DENSE MLP, and no `shared_head.head.weight` (`has_own_lm_head=False`); and CORRECTED spec Β§4 item 6 ([#1804](https://github.com/mudler/vllm.cpp/issues/1804)) β€” BOTH MLA ropes are GPT-J (`deepseek_v2.py:1093-1098` for the full class), the geometries differ on the THETA (5e4 vs 8e7), not the layout. Memory format recorded: `mlp.gate.e_score_correction_bias` ships F32 in an otherwise BF16 tower. Row stays `SPIKE` deliberately β€” making an arch resolve is not porting a model, and the Β§8.1 heading restructure `ACTIVE` requires belongs to the brick where the forward stops refusing. W0 scope, unchanged below. Upstream subclasses DeepSeek (`Dots3NoteModel(DeepseekV32Model)`, `Dots3NoteMoE(DeepseekV2MoE)`), so our gated MLA + DSA indexer + `noaux_tc` router + Qwen3-VL vision + Voxtral audio carry most of it; net-new is windowed MLA over 33 of 46 layers with a second latent geometry (576 vs 1088 rows), the padded/heterogeneous MLA KV spec, the headwise attention gate, the MoE ViT and the `dots` audio stem. **BEYOND-PIN** (vLLM `main` only, `#51255`, still being patched) and **oracle-memory-infeasible**: ~576 GB bf16 / ~290 GB fp8 against a 119-122 GiB ceiling on every host we own, so no performance axis is claimable and no token gate can bind until Β§6.4 of the spec is answered | `MODEL-MM-dots3-note-dots3-note-for-causal-lm` | +| πŸ“‹ | `Dots3NoteForCausalLM` | dots3-note (280B-A16B multimodal MoE: DeepSeek-V3.2 text tower + sliding-window MLA, MoE ViT, `dots` audio encoder) | **W6a LANDED (2026-09-01, [#2512](https://github.com/mudler/vllm.cpp/issues/2512)): the DENSE half of the VISION TOWER is SERVED end to end.** An `image_url` chat request travels `ApiServer::handle_chat_completions` -> the architecture-dispatched multimodal chat seam (`REGISTER_VLLM_MM_CHAT`, #2481) -> `GPUModelRunner::execute_mm_encoder` (#2398) -> `ModelRegistry::EmbedMm` -> `ModelRegistry::Forward`, and `kDots3NoteFactory` now carries `encode_mm`/`embed_mm` with `supports_multimodal` TRUE again. The arm is `patch_embed` -> blocks 0-24 (fused `attn.qkv`, per-head `q_norm`/`k_norm` BEFORE a 2-D vision RoPE, bidirectional attention, three-tensor SwiGLU through `layers::MlpGateUpMethodBase`) -> `post_trunk_norm` -> the `patch_merger` adapter folding 4x1536 to 5120, plus a dots3 image processor whose placeholder marker is the checkpoint's own image start/pad/end token triple rather than Qwen3-VL's. **The RELEASED checkpoint still REFUSES**, by name, at its first MoE ViT block: 17 of 42 vision blocks are MoE and 1960 of the 2195 `vision_encoder.*` tensors are W6b's β€” W3's polarity applied to the second tower, not a new exception. Ported from vLLM `vision.py` (677 lines) / `vision_attention.py` (477 lines) / `common/processor.py` read at `9035151d6`, which is BEYOND our pin `5559679229`; every anchor names that SHA because `vision_attention.py` is already 494 lines at vLLM `main` `7a100bb61`. Gate is a CONSISTENCY gate (spec Β§6.4 option B): an independent in-test double-precision reference sharing no helper with the implementation, RED-first, plus a served-request reachability suite whose load-bearing case is two DIFFERENT images compared on LOGPROBS. No performance number is claimable on any axis. Spec Β§7's vision FP8 formula and Β§8's R5 MOVED from W6 to W9 in the same change, because the W2 census below shows the bf16 checkpoint ships no scale tensor at all. **W5 LANDED: `Dots3NoteDeviceRefusal` returns "" for the RELEASED config** β€” the 45 MoE layers decode through `Dots3NoteMoeBlock` over the shared `RunMoePlaced` seam, and W5c ([#2176](https://github.com/mudler/vllm.cpp/issues/2176)) removed the nextn refusal, which was STRICTER THAN UPSTREAM. Representable is not runnable: the MoE alone is 545.82 GB of a 576.89 GB checkpoint. **W4b-3c LANDED: the DSA lightning indexer's SELECTION is on the decode path** β€” a step whose requests are all single-shot prefills is served SPARSELY; a step in which any request resumes is refused by name, because the indexer's own key cache is `KV-DSV4-MULTICACHE` ([#1925](https://github.com/mudler/vllm.cpp/issues/1925)). **W4b-2 LANDED: the SLIDING arm decodes over a PADDED 1088-wide MLA cache row** narrowed per layer, its two windowed CUDA ops compiled and run on `orin:gpu0` (sm_87) under an `rc` lease. **W4b-1 LANDED** the sliding host maths, **W4a LANDED** the full-attention layer on the decode path with dots3's two LoRA rescales, its `k_rope_only_layernorm` and its headwise gate, and **W3 LANDED** the first host reference on this row. **W2 LANDED (the whole weight map): the complete released shard index read for real β€” 38006/38006 accounted, 35381 language / 2195 vision / 430 audio, every bucket asserted BY NUMBER, zero unaccounted, and the two tower files carried as NAMED W6/W7 deferral records rather than integer counters.** Headers only: 4770592 bytes over 133 safetensors headers, no tensor byte. It settled three things the W1 slice could not β€” the backbone has exactly FOUR distinct layer shapes so the 1/2 repeat holds, the full/sliding split derived from the shipped indexer tensors matches `config.layer_types` exactly, and the checkpoint carries 62 F32 tensors in TWO families (the 45 language `e_score_correction_bias` W1 predicted plus 17 vision `router_bias`, spec R5's shape). And it FOUND one: the index declares `indexer_rope_layout: "leading"` / `converted_from: "tail"`, which no upstream code reads ([#1846](https://github.com/mudler/vllm.cpp/issues/1846)); W3 owes the slice. **W1 LANDED (config + registry): the arch RESOLVES, parses the REAL released `config.json`, and accounts 1614/1614 tensors on a committed slice of the released shard index; load, GGUF and the forward each REFUSE BY NAME.** All six Β§4 config traps are gated RED-first (evidence table in spec Β§4.1), which matters more here than on any other row because spec Β§6.4 records that NO oracle runs on hardware we own, so no token gate exists downstream. W1 also RESOLVED Β§1.4 from the checkpoint: exactly ONE nextn layer (`model.layers.46.*`, 18 tensors) with the SLIDING geometry and a DENSE MLP, and no `shared_head.head.weight` (`has_own_lm_head=False`); and CORRECTED spec Β§4 item 6 ([#1804](https://github.com/mudler/vllm.cpp/issues/1804)) β€” BOTH MLA ropes are GPT-J (`deepseek_v2.py:1093-1098` for the full class), the geometries differ on the THETA (5e4 vs 8e7), not the layout. Memory format recorded: `mlp.gate.e_score_correction_bias` ships F32 in an otherwise BF16 tower. Row stays `SPIKE` deliberately β€” making an arch resolve is not porting a model, and the Β§8.1 heading restructure `ACTIVE` requires belongs to the brick where the forward stops refusing. W0 scope, unchanged below. Upstream subclasses DeepSeek (`Dots3NoteModel(DeepseekV32Model)`, `Dots3NoteMoE(DeepseekV2MoE)`), so our gated MLA + DSA indexer + `noaux_tc` router + Qwen3-VL vision + Voxtral audio carry most of it; net-new is windowed MLA over 33 of 46 layers with a second latent geometry (576 vs 1088 rows), the padded/heterogeneous MLA KV spec, the headwise attention gate, the MoE ViT and the `dots` audio stem. **BEYOND-PIN** (vLLM `main` only, `#51255`, still being patched) and **oracle-memory-infeasible**: ~576 GB bf16 / ~290 GB fp8 against a 119-122 GiB ceiling on every host we own, so no performance axis is claimable and no token gate can bind until Β§6.4 of the spec is answered | `MODEL-MM-dots3-note-dots3-note-for-causal-lm` | | 🚧 | `MiniMaxH3DiTModel` | MiniMax-H3 (33.1B omni-modal video+audio DiT, DERIVE-AND-SHIP) | **W1/W2 landed**: packed layout (fl2va + ref2va, fp64 position grid BIT-EXACT), latent packing, euler-ancestral eta0 scheduler, and the full DiT forward all parity-gated against the UPSTREAM vLLM-Omni modules executed at reduced dimensions (**max abs diff 1.6e-7**, 10/10 cases / 2539 assertions). NOT autoregressive (no KV cache, no sampler, no logits) and **e2e HW-BLOCKED** (~354 GB checkpoint, ~133 GB/rank on 4x B300 vs 119 GiB unified); bf16 production stream + request planning + the ComfyUI-GGUF arm also landed (535 REAL tensors resolve onto our contract, geometry from shapes alone). **HW verdict CORRECTED: quantized arms FIT (~41 GB in 119 GiB)**, so e2e + speed are reachable; encoder/VAEs/audio VAE DONE (4.2e-9 vs the checkpoint's remote code); NVFP4 layout GATED as identical to ours (speed path is loader wiring); BOTH VAE DECODERS done (audio 4.2e-9, video ViT3D 8.9e-8); video tiling + 3D-CNN encoder (conditioning only) pending; encoder TEXT tower done (1.2e-7); **serving `/v1/videos` DONE and the DEVICE-RESIDENT forward (W2b, f32) LANDED + GPU-VERIFIED on Thor sm_110 at video 1.49e-7 / audio 8.94e-8**; bf16 stream + fusion folds + the FP4 path (needs sm_121a) + a real-checkpoint run pending. **2026-08-05: the AUDIO-VAE ENCODER is ported** (DAC analysis stack + `pre_block` AttnProjection + `mean_proj`, gated stage by stage vs the checkpoint's own remote code at 2.98e-8 / 1.64e-7 / 1.86e-8) with its own checkpoint loader gated on the real 1087-tensor manifest β€” so **ref2va AUDIO and VIDEO+AUDIO references are now WIRED** (audio rows move by 0.51 / 0.71; a different waveform still moves them by 7.1e-4). Both VAEs are now complete in both directions. **bf16 13-SHARD RELEASE INDEXES 2026-08-07 (`row/H3-BF16-SHARDED-DIT`)**: `MiniMaxH3ShardedCheckpoint` resolves the ORIGINAL 66.3 GB release through its own `model.safetensors.index.json` (a tensor named in the index but missing from its shard throws BY NAME), `EnumerateMiniMaxH3ShardedTensors` feeds the shared shapes-only geometry parser, and `LoadMiniMaxH3DitFromShards` is the host-f32 reference loader. Gated CPU-only at 72/72/54497 (post-rebase): every tensor resolves to the shard the index named AND to the bytes written there, the derived geometry equals the single-file path field for field, and a SPARSE 13-shard release with the REAL 535 tensors at REAL shapes (66.3 GB declared, 144 KB on disk) derives the SHIPPED geometry (50/5376/56/128/14336/24/32/1x2x2/5120). **STREAMS 2026-08-07 (`row/H3-BF16-SHARDED-STREAM`)**: `StreamMiniMaxH3ShardedToDeviceBf16` uploads it one tensor at a time β€” a BF16 tensor bound for a bf16 device slot goes straight from the mmap with ZERO host buffer, so peak host is bounded by ONE tensor (observed `host_peak=8192`, `direct=37 converted=9`); bit-exact vs the non-streamed `StageMiniMaxH3DitWeights` reference over all 46 views with identical logits, rope.inv_freq host-resident, 73/73/55203. Spec Β§8.14. **bf16 TEXT ENCODER + THE CONDITIONING NUMBER 2026-08-07 (`row/H3-ENC-BF16-COND-DIFF`)**: the 14-shard 63 GB bf16 Qwen3-VL-32B encoder streams to device too (`StreamMiniMaxH3EncoderShardsToDevice`, q/k/v and gate/up fused ON DEVICE), `--encoder-only` runs the tower alone (peak ~96 -> ~49 GiB by not loading the DiT first), and the widening is gated BIT-IDENTICAL vs an f32-staged tower so the A/B cannot be confounded. MEASURED on Thor over 233 tokens: Q4_K_M vs bf16 conditioning is cos 0.99745 mean / 0.909 min, rel RMS 6.85% excluding the attention sink, median rotation 3.5 deg β€” same energy as a ONE-WORD prompt edit but DIFFUSE (232/233 tokens rotate vs 172/233). Whether the RENDER changes is NOT established. 75/75/55609. Spec Β§8.15. This UNBLOCKS the quantization-quality question; no bf16-vs-quant render or speed number is claimed. Spec Β§8.13. **W-FP4a LANDED (CPU) 2026-08-06 (`row/H3-FP4-SPEED`)**: the device DiT forward now routes the NVFP4 projections through the shared Marlin W4A16 dispatcher (fp4 kept packed; no new quant code), fp4-vs-bf16 wiring gate GREEN (62/62Β·30039). **W-FP4a GB10 leg LANDED 2026-08-06 (`row/H3-FP4-GPU-E2E`, PR #64):** on sm_121a the Marlin W4A16 path RAN for all 11 projections (`dense_gemms==11` default β€” VT_MARLIN_DENSE is default-ON β†’ vLLM's own DENSE Marlin GEMM, not the grouped route; `marlin_gemms==11` under VT_MARLIN_DENSE=0; `fallback_gemms==0`), fp4-vs-bf16 BYTE-EXACT (max\|diff\|=0), and the fp4 arm is a MEMORY win not a diffusion-forward speed win (per-forward bf16/fp4 3.47Γ— @seq64 β†’ 0.79–0.83Γ— @seq4224–7040; ~16 vs ~66 GB device). Real-checkpoint fp4-resident t2va e2e RUNS (real 18.75 GB NVFP4 DiT + VAEs + GGUF Qwen3-VL-32B encoder β†’ valid mp4/wav; DiT s/step 5.45/20.0/209 s @512/768/REF-209f) but frames are a non-scene patch-grid at 12/20/50 steps β†’ OPEN render bug (device VAE/denoise). vLLM-Omni has no quantized H3 arm (BF16-only) so any comparison is HW/loader-forced-indirect β€” spec Β§8. **ONE-SURFACE ROW 2 LANDED 2026-08-08 (`row/H3-VIDEO-ABI`)**: the whole assembly pipeline is library-owned (`vllm::multimodal::MiniMaxH3VideoEngine`, `minimax_h3_video.cpp`) behind the ABI v12 `vllm_video_*` entry points; `/v1/videos` routes through the SAME seam; `minimax_h3_gen`+`minimax_h3_mux` are thin `vllm.h` clients, frames+WAV byte-identical to the pre-fold binary on the committed fold fixture (`test_minimax_h3_video_fold` 3-arm gate + the v12 `test_capi` section); GB10 real-video re-verify via the v12 ABI = named residual. **ROW 2 DEVICE-SEAM FOLLOW-UP (#135; replaces #134):** ABI 0/1 maps once to `vt::DeviceType`; shared code dispatches through `GetBackend(device_type)`, restoring DSR 34β†’32 without a baseline/allowlist change; CPU compile/fold test pending in CI due shared-disk pressure **PRUNED CHECKPOINTS LOAD AND RUN 2026-08-10 (`row/H3-PRUNED`, #241, spec Β§8.21):** the community `pruned` variants (`unsloth/MiniMax-H3-GGUF` Q2_K..Q8_0, `lilcheaty/MiniMax-H3-NVFP4` `*_pruned_nvfp4`) are not lossily pruned but ComfyUI's AdaLN timestep-CURVE refactor (`comfy/ldm/minimax/model.py:419-432,610-615`): an `adaln_t_table` [1025, 8] lerp replaces the sinusoidal+MLP time embedder, no SiLU before the AdaLN linear, and its in_features drop 2688 -> 8, collapsing `adaln_proj` from 13.04B of 33.12B to 0.04B and the DiT to 20.11B β€” so a pruned Q8_0 (21.4 GB) costs what our unpruned Q4_K_M (19.9 GB) costs. Gated on the REAL 532-tensor pruned GGUF manifest (532 = 535 - 4 `time_embedder.*` + 1 table, header-only), the clamped-lerp golden, and a CONSTRUCTED-curve test proving the pruned forward is numerically identical to the unpruned one; 79/79/57299. | `MODEL-DIFFUSION-minimax-h3-mini-max-h3-dit` | | 🚧 | `MiniMaxMusic3ForConditionalGeneration` | MiniMax-Music3 (text-to-music: ~8.6B Qwen3 global LLM + 0.646B 8-codebook RVQ depth decoder + 2.4B fp32 flow-matching DiT + 0.054B DAC Flow-VAE) | **W0 DONE + W1 LANDED (#672, [spec](specs/minimax-music3.md)).** Diffusers oracle gateable with committed per-stage goldens (#708); modular loader in the tree, 1413/1413 assertions against the real checkpoint (#714). Geometry MEASURED from every component `config.json` plus safetensors headers by range request. First row to exercise the fallback-oracle rule (#647): absent from vLLM AND vLLM-Omni, so its primary oracle is the OPEN diffusers PR #14456 head `c6da9936` with SGLang-Omni `748a0b43` as the e2e cross-check. **W2/W3 LANDED**: the AR half computes and is gated. **W4/W5 LANDED**: the ACOUSTIC half β€” flow-matching DiT, `FlowMatchEulerDiscreteScheduler` with `invert_sigmas`, the CFG mix, the denoise loop's window bookkeeping and the DAC Flow-VAE vocoder over the shared `vocoder1d` primitives β€” gated at reduced dimensions (27 cases / 265 assertions, no checkpoint) AND at full scale against the committed capture (scheduler BIT-EXACT 22 016/22 016; waveform 88 064 samples, 0 outside a tolerance calibrated against torch reproducing its own goldens). **W6 LANDED**: registered as a `SpeechRegistry` family (`minimax-music3`, detection INSPECTS `modular_model_index.json`), `SpeechGenParams` extended ADDITIVELY with `lyrics`/`description` plus three generation controls (IndexTTS-2.5 byte-identical, 0 lines changed), the new `vllm_speech_*` C-ABI surface at v20, and `POST /v1/audio/speech` on the example server as a thin ABI client. The denoise + decode composition reproduces the capture's `waveform.npy` β€” 88 064 values, 0 outside tolerance, max abs-diff 4.52e-06 β€” and the WAV payload is BIT-EXACT against the quantization of that golden. **W7 LANDED: ONE ARM IMPLEMENTED AND VALUE-GATED, the rest refused.** The RVQ depth decoder loads from GGUF Q4_K (`audio-cpp/MiniMax-Music3-GGUF`@`c36aaeed`, `rvq_depth_decoder_q4_k.gguf`, 405 752 480 B, sha256 `4c5d41b2...c70cbdd0` β€” pinned because an unpinned quantized checkpoint is not reproducible): 47 tensors as 36 Q4_K + 9 BF16 + 2 F16, dequantized to bf16 through the SHARED `gguf_dequant.h` seam, lineage keyed on `audiocpp.model_spec.family` because `general.architecture` reads `audiocpp`/`mm3`/`qwen3`/`wan` for this ONE model and `wan` collides with real Wan video GGUFs. Gated 6 cases / 319 assertions against the W3 golden's 716 800 values at a bound DERIVED rather than chosen: worst per-tensor relative L2 0.0742 (bound 0.10), full-scale mean abs-diff 0.0324 / max abs-diff 0.3125 / 2.84% bit-identical. **THE GATE-DESIGN FINDING, which generalizes past this row: an upper-bound-only tolerance CANNOT distinguish a real quantized arm from a silent dequant fallback, because the fallback is CLOSER to the golden** β€” bf16 weights through the identical forward give mean abs-diff 0.00182, 17.8x nearer than the genuine Q4_K arm, so every plausible upper bound PASSES the failure. What catches it is a LOWER bound (5e-3, sitting between the two), gated by a positive control that actually runs the fallback and asserts the gate rejects it. Two further INDEPENDENT legs prove the quantized path was taken rather than inferring it: the RESIDENT ggml type of every tensor (loader bookkeeping β€” localizes a fault to a name, but a loader that lied about its tallies would pass it, which is why mutation QM2 exists) and the Q4_K LATTICE (a property of the DATA needing no loader cooperation β€” 0 of 524 288 32-element windows exceed 16 distinct values, against a bf16 control from the same file at 127 of 128). Second finding: the 2 F16 islands do NOT round-trip exactly (rel 3.23e-08 / 2.36e-08) because the quantizer re-encoded them BF16->F16 and F16's exponent range is NARROWER, flushing weights below ~6e-08 to zero β€” so the gate splits the islands, BF16 bit-exact and F16 bounded; asserting "unquantized means exact" for all 11 REDS a correct reader. 18 mutations across both layers all fire, one of them (QM4, the lineage guard) only after a COVERAGE HOLE it exposed was closed with synthetic GGUF fixtures. The 4 OTHER components' GGUF arms, the `mm3` and ComfyUI lineages, and every non-GGUF format are REFUSED BY NAME and owed. Quantized MiniMax-Music3 checkpoints DO exist β€” a HuggingFace survey on 2026-08-14 found 14 community repositories in 5 formats (GGUF of all five components, int8/w4a8, MLX, plus a proprietary Cortiq arm), recorded query-by-query with result counts in spec Β§9.1. `minimax_music3_quant.{h,cpp}` is a SEPARATE translation unit per porting-a-model.md that diagnoses 8 formats at the THREE places a quantized checkpoint announces itself (tree `.gguf`, manifest sidecars plus the dtype-only fp8/int8 cases, and `quantization_config`/MLX `quantization`) and refuses each BY NAME with the evidence, the missing piece, the supported arm, the phase and the issue. 29 cases / 125 assertions without a checkpoint plus 6 cases / 319 assertions against the pinned artifact, RED first at 8/8 probe checks, 18 of 18 mutations fire; every tolerance is DERIVED from a printed measurement rather than chosen, and no speed number is claimed. GGUF headers MEASURED by range request (56 MiB, no weight bytes): "the GGUF arm" is THREE MUTUALLY INCOMPATIBLE LINEAGES and `general.architecture` cannot separate them (`audiocpp`/`mm3`/`qwen3`/`wan`, and `wan` collides with real Wan video GGUFs), the ComfyUI lineage ships the DiT + condition encoder ONLY so it can never generate audio, and `comfy.gguf.orig_shape.*` is absent from all ten files. The 8.6B **W2's remainder LANDED (this PR): the 8.6B `Qwen3ForCausalLM` forward runs.** `Qwen3DenseModel::ForwardEmbeds` gives the landed dense path the `inputs_embeds` entry its multimodal siblings already had (`qwen3_vl.h:145,159`, `gemma4.h:210-218`, `muse_glimmer.h:369-380`), additively and proved BIT-IDENTICAL to the token-id `Forward` in logits AND in the paged KV it writes. Teacher-forced on the capture's own codes, 25 steps reproduce `frame_hiddens[:, :4096]` - 102 400 values, 9337 bit-identical, mean abs-diff 1.763e-02 - INSIDE a matched control (upstream's own model under `sdpa_kernel(MATH)`: 12 036 bit-identical, mean abs-diff 1.475e-02), with the per-step error FLAT rather than compounding. The oracle's OWN sampled semantic codes rank 2.48 on average under our guided logits where chance is 8191.5, and the gate carries its own NEGATIVE CONTROL: a one-step-shifted alignment reads mean abs-diff 0.803, 46x outside. **The e2e case at `POST /v1/audio/speech` is OBSERVED TO PASS (2026-08-15, #852)**: 200 `audio/wav`, 12332 bytes, 2 AR frames -> 6 latent frames -> 3072 samples/channel of 44100 Hz stereo, 6144 int16 samples all non-zero, 0 clipped, 2818 of 3072 positions differing between L and R; 21 assertions in 7:54, and 5 cases / 535 assertions for the whole file in 31:14. It had never been seen because the request body posted `audio_duration_s`, the FIELD name, where `ParseSpeechRequest` reads `audio_duration` - so the key was silently dropped and the family's 60 s default substituted, making every run a ~750x job (1500 AR frames not 2, 8 denoise windows not 1, 5167 vocoder latents not 6) that could never have passed its own payload assertion. The near-miss key is now REFUSED (#925). **The earlier diagnosis was wrong and the way it was wrong generalizes**: open LM shard fds do NOT locate the program counter, because `LoadBf16Direct` BORROWS the mapping and a borrowed `OwnedTensor` keeps it alive for the whole request, while the depth decoder's tensors are COPIED so its fd closes on return. A symbol-resolved profile puts the LM weight load at 180 s (1 thread, state `D`, ~92 MB/s off the NAS) in BOTH binaries, and the cost in `LinearNoBias` 42-57% / `Threadpool::Barrier` 25% / `vocoder1d::ConvTranspose1d` 88.5%; the suspected 200000 x 4096 `lm_head` transpose is 1.388 s. The GGUF k-quant ARM for the remaining components is still owed; spec Β§5's token-exact gate is WITHDRAWN because upstream's AR stage has no greedy path **PARITY SWEEP + MUSIC-ONLY SERVER LANDED (#672, #953)**: upstream's whole user-facing surface was enumerated field by field (SGLang-Omni `748a0b43` `models/minimax_music3/`, diffusers `c6da9936` `modular_pipelines/minimax_music3/`) and recorded with `file:line` in spec Β§10.1. **`--model` is now OPTIONAL when `--speech-model` is given**, so `vllm-server --speech-model ` serves `/v1/audio/speech` and registers NOTHING else β€” upstream's own shape (`sgl-omni serve --model MiniMaxAI/MiniMax-Music3`, `config.py:29-63`: three stages, no chat LLM), and the third instance of the task-conditional dispatch already in `server_main.cpp` for pooling and transcription-only checkpoints (vLLM `api_server.py:255-265`). ADDITIVE and proved: the ONLY case whose verdict changes is BOTH flags absent, which was an error and remains one with a message naming both options; the route table is gated in BOTH directions over a REAL SOCKET. New `examples/minimax_music3_gen` β€” the music family was the one generative family in this tree with NO example β€” a THIN C-ABI client (`vllm_speech_engine_load` + `vllm_synthesize`, `include/vllm.h` only). **Five keys upstream refuses BY NAME were SILENT here** and are now refused (#953): `temperature`/`top_p`/`top_k`/`repetition_penalty` (`request_builders.py:14-19,109-114`; the AR stage has ONE sampler, a fixed top-50 draw) and `max_new_tokens` (`request_builders.py:56-68`; upstream's length spelling in 25 Hz FRAMES, not seconds) β€” the #925 class exactly. **The e2e gate no longer reports a SKIP WEARING A PASS**: it read 5 cases / 5 passed with `assertions: 0` when the checkpoint was absent, the same shape that fooled this project on `test_qwen3_paged_engine`. Split into a CHECKPOINT-FREE half that runs unconditionally in CI (request contract, both ceilings, the speech-only route table over a real socket with a stub synthesizer) and the env-gated checkpoint half, whose HTTP case now runs over a REAL SOCKET against the music-only server shape rather than calling `handle_audio_speech`; a coverage-report case prints which arms ran every run. No-env arm 9 cases / 37 assertions where it was 5 / 0. **The WEIGHTS are documented** per `porting-a-model.md` Β§2.1: `docs/USAGE.md` carries component-by-component tables for the diffusers arm at `MiniMaxAI/MiniMax-Music3`@`fbdf52fbaaca799592917417eb05f1899f1255ec` (**28.5 GB resident, 28 517 617 303 B MEASURED**, out of a 57.4 GB repository, and why they differ), the native `.pth` arm we refuse and SGLang-Omni serves, the one implemented GGUF Q4_K artifact with its sha256, and the 14 THIRD-PARTY quantized repositories in 5 formats each marked refused. The revision is VERIFIED not copied: `condition_encoder/diffusion_pytorch_model.safetensors` on disk hashes to `83179c5e…a202c2a4d`, that revision's own LFS record. OWED and named rather than discovered: the non-`wav` response formats (no encoder vendored; note upstream DOWNMIXES TO MONO for every non-wav format, `client/audio.py:328-334`), request batching and `/v1/audio/speech/batch` (SGLang runs continuous batching at 16 with TWO engine rows per request for the CFG twin, `engine_builder.py:74-77`), and the 32 kHz delivery resample. Streaming is a PERMANENT refusal, not a debt: neither upstream arm has it (`supports_streaming_vocoder=False`). One place we are AHEAD of both arms: `guidance_scale` is a real per-request control here, where diffusers freezes it at 1.7 into the guider component (`denoise.py:180`) and SGLang exposes it only as a serve-time knob. **DEVICE ARM LANDED (#672, this PR) β€” PARTIAL, and the split is the point.** `SpeechModelParams` grew `device` (0 = CPU, 1 = the accelerator this build resolves; the `VideoModelParams::device` polarity, NOT `vllm_model_params.device`'s 0=auto, because every Music3 gate was taken on the CPU arm and `auto` would move every zero-filled caller off it), resolved through a NEW SHARED `multimodal::SpeechEngineDeviceType` rather than a third copy of a mapping the tree already carried twice (`minimax_h3_video.cpp:255`, `ltx2_video.cpp:706`). The engine's `vt::Queue` was a COMPILE-TIME CONSTANT (`minimax_music3_speech.cpp:492`, whose own comment said "a device arm is a queue, not a fork"); it is now that queue, built once in the constructor BEFORE any weight is read (the unified-memory recipe). `Music3LmSession` allocates its 36-layer paged KV on the queue's device, because `dense_attn::KvSlice` labels the pointer with `d.q.device` and a host `std::vector` handed to a CUDA forward is a host pointer wearing a device tensor's label. C ABI **v21**: `vllm_speech_model_params.device` + `vllm_speech_engine_device`, which reports what was GRANTED not what was requested; `--speech-device` on the server, `--device` on `minimax-music3-gen`. **WHAT MOVES is the 8.6B `Qwen3ForCausalLM` half and nothing else**: it already rides the shared `Qwen3DenseModel::ForwardEmbeds` five text registrations use, so no model was forked. **WHAT DOES NOT, named rather than implied**: the 0.646B RVQ depth decoder and the 2.4B fp32 DiT are host `std::vector` scalar loops under `-ffp-contract=off` (kept reproducible against torch by W2-W5) and are OWED; the DAC Flow-VAE vocoder needs `ConvTranspose1d`, and **`vt` HAS NO SUCH OP** β€” `vt::Conv2d` and `vt::DepthwiseConv1d` exist but are registered for the **CPU only** and neither expresses a transposed convolution β€” so it is recorded as owed rather than hand-rolled outside the seam. The CPU arm is BIT-IDENTICAL: it takes the same host code with the same `std::vector` KV, and every Music3 gate is unchanged. | `MODEL-MUSIC-minimax-music3-mini-max-music3-for-conditional-generation` | | πŸ“‹ | `LTX2VideoTransformer3DModel` | LTX-2.5 (21.00B joint video+audio flow-matching DiT, Lightricks) | **L0 spec committed 2026-08-11 (#435, [spec](specs/ltx-2-5.md)).** Geometry MEASURED from the FP8 checkpoint's own safetensors header by HTTP range request (6124 tensors, 881,048-byte header, no payload downloaded): **21.00B** params β€” 48 blocks @ 386.7M = 18.560B, audio embeddings connector 2.016B, global 0.427B. The filename says `22b` and the Diffusers card says ~19B; the MEASURED count is what this row uses. Video stream 4096 (32 heads x 128), audio stream 2048 (32 heads x 64), in/out channels 128 both. Per block SIX attentions β€” `attn1` (video self), `attn2` (video<->text, cross 4096), `audio_attn1`, `audio_attn2` (cross 2048), and the two CROSS-MODAL `audio_to_video_attn` / `video_to_audio_attn` β€” which is the structural break from MiniMax-H3: H3 packs every modality into ONE sequence with per-row token tags, LTX runs TWO streams coupled by explicit cross-attention. **Per-head gated attention on every attention** (`to_gate_logits` = `Linear(query_dim, heads, bias=True)`, `attention.py:513-514`, applied AFTER the attention output at `:577`) β€” H3 has no analogue and getting it wrong renders plausibly-wrong rather than erroring. FFN is gelu-approximate 4096->16384->4096 with **NO bias** while `audio_ff` 2048->8192->2048 **HAS** bias, which independently confirms `ff_bias=false` / `audio_ff_bias=true` from `model_configurator.py:78-80` β€” checkpoint and source agreeing, not either alone. **RETRACTED 2026-08-12, was billed as a FREE WIN.** The spec claimed 2.5 sets `use_prompt_adaln_single=false` so the cross-attention K/V are timestep-free and cacheable. The shipped checkpoint DISPROVES it: it carries 12 `prompt_adaln_single`/`audio_prompt_adaln_single` tensors including a `timestep_embedder.linear_1 [4096, 256]` (256 = the sinusoidal timestep width), and `model.py:223-227` builds that module ONLY when the flag is TRUE. `transformer.py:441` was quoted as proof of no timestep term, but `:442-443` add one whenever `prompt_timestep` is not None, and the comment above them says exactly that. NO SHIPPED DEFECT: `ltx2_dit.cpp:672` refuses the cache by name when the flag is on, so the feature is correct-and-inapplicable rather than silently wrong, and stays gated bit-identical and prompt-bound for any checkpoint that does set it false. **ORACLE:** vLLM-Omni does NOT support 2.5 β€” `resolve_ltx_pipeline_recipe` keys on 2 / 2.3 only and RAISES otherwise (`ltx2_recipes.py:162-166`), with upstream [vllm-omni#6066](https://github.com/vllm-project/vllm-omni/issues/6066) filed 2026-08-11 β€” but its `DiffusersAdapterPipeline` is generic (`DiffusionPipeline.from_pretrained`, `pipeline_diffusers_adapter.py:116`), so vLLM-Omni CAN execute 2.5 via `--load-format diffusers`. Binding oracle = that adapter; immediate cross-check = Lightricks `ltx-pipelines`. **HW: FITS ONE GB10** at ~29 GB (NVFP4 DiT 18.72 + NVFP4 Gemma-4 TE 7.40 + VAEs 1.83 + upscaler 1.00) vs H3's ~41 GB. **OWED UP FRONT, not to be discovered later:** the speed axis lands `PENDING` because the adapter is a black box (`supports_step_execution=False`, `supports_request_batch=False`) and therefore NOT vLLM's production configuration, which AGENTS.md requires as the denominator; DiffVAE (`NADiffusionDecoder`, neighborhood attention) is REFUSED BY NAME until its own row rather than silently downgraded to the Conv VAE; and no render-quality claim is made from structural e2e. | `MODEL-DIFFUSION-ltx-2-5-ltx2-video-transformer-3d-model` | diff --git a/.agents/specs/dots3-note.md b/.agents/specs/dots3-note.md index a75bfa9f5..bbeabdb3e 100644 --- a/.agents/specs/dots3-note.md +++ b/.agents/specs/dots3-note.md @@ -3967,6 +3967,357 @@ widening it. Stop if the nearest mechanism cannot be pushed above the residue by fixture design. Stop if W5c needs more than the classifier-deferral shape. Stop and return `NEEDS_DECISION` if the seam needs a new `vt` op. +### 4.11 W6a puts the DENSE vision tower on a SERVED request, and refuses the pyramid by name + +W6a is the first brick on this row whose output a client can ask for. Every +brick before it ended at a `ctest` binary: W3 wrote host maths, W4a/W4b put the +two attention geometries on the decode path, W5 put the MoE layers there. None +of them could be reached from `ApiServer::handle_chat_completions`, because +until #2398 and #2481 landed there was no engine path by which a vision tower +could be fed from a production entry point at all. Both are on `main` now, so +the remaining work was this model's own half: a tower, a processor, a chat +registration and the two `ModelFactory` hooks. + +**The oracle, and the SHA every anchor here was read at.** vLLM is the oracle +and no secondary oracle is admissible, because vLLM implements this tower. It is +BEYOND our parity pin: `5559679229bc961848b121ccdeaa8fa5d79bec98` has no +`dots3_note` directory at all. The sources were read in the local clone +`~/_git/vllm` at **`9035151d6`**, the merge of +[vllm#51255](https://github.com/vllm-project/vllm/pull/51255) that added them: + +| Upstream | Lines at `9035151d6` | What W6a ported from it | +|---|---|---| +| `vllm/models/dots3_note/nvidia/vision.py` | 677 | `DotsMoEVitConfig:27`, `RMSNorm:107`, `DotsSwiGLUFFN:126`, `DotsPatchEmbed:302`, `MoEVisionBlock:334`, `PatchMergerAdapter:441`, `DotsMoEVitModel:492` (`get_pos_ids_by_grid:566`, `rot_pos_emb:604`, `_build_single_temporal_cu_seqlens_from_grid:625`, `forward:634`) | +| `vllm/models/dots3_note/nvidia/vision_attention.py` | 477 | `rotate_half:33`, `apply_rotary_pos_emb_vision:39`, `VisionRotaryEmbedding:52`, `_RMSNorm:97`, `_VisionAttentionBase:134` (`_qkv_with_rope:149`), `VisionAttentionV2:207`, `apply_vision_attention_residual:436` | +| `vllm/models/dots3_note/common/processor.py` | 811 | `IMAGE_START/PAD/END:41-43`, `Dots3NoteImageProcessor:63` (`resized_size:97`, `preprocess:147`) | +| `vllm/models/dots3_note/nvidia/multimodal.py` | 304 | `hf_to_vllm_mapper:54-62` (`vision_encoder.` -> `visual.`), `get_placeholder_str:65`, `_process_image_input:144` | +| `vllm/models/dots3_note/nvidia/vision_moe.py` | 149 | NOTHING. It is the W6b/W9 arm and is refused by name. | + +**Every anchor above names the SHA it was read at, because upstream has already +moved.** `vision_attention.py` is 477 lines at `9035151d6` and 494 lines at vLLM +`main` `7a100bb61`. An anchor read in the wrong tree is a recorded failure mode +on this project, and a `file:line` with no revision beside it is one. + +#### 4.11.1 The geometry, and one word in #2512 that the fixture corrects + +Every number below was read from the COMMITTED fixture +(`tests/vllm/models/fixtures/dots3_note_prev/config.json` and +`index_full.json`), not from the issue text: + +| `vision_config` key | Released value | Consequence | +|---|---|---| +| `embed_dim` | 1536 | tower width; `head_dim` = 1536/24 = 64 | +| `num_attention_heads` | 24 | | +| `num_hidden_layers` | 42 | 25 dense + 17 MoE | +| `intermediate_size` | 4224 | the DENSE SwiGLU width | +| `moe_intermediate_size` | 2112 | W6b's, unread here | +| `patch_size` | 14 | patch row is 3*1*14*14 = 588 wide | +| `temporal_patch_size` | 1 | `DotsPatchEmbed.forward` takes `[:, :, 0]` of a one-deep temporal axis | +| `spatial_merge_size` | 2 | | +| `rms_norm_eps` | 1e-05 | | +| `use_bias` | false | `attn.qkv`, `attn.proj` and every `mlp.fc*` carry NO bias | +| `use_qk_norm` | true | `q_norm`/`k_norm` [64], per head, BEFORE rope | +| `is_causal` | false | the attention is bidirectional | +| `post_norm` | true | `post_trunk_norm` exists | +| `pre_pixel_shuffle` | true | the PREPROCESSOR emits 2x2-grouped patch rows and RoPE regroups to match | +| `adapter_type` | `"patch_merger"` | `PatchMergerAdapter`, NOT `PixelShuffleAdapter` | +| `adapter_in_dim` / `adapter_out_dim` | 1536 / 5120 | 4x1536 = 6144 folded to the text tower's 5120 | +| `pyramid_num_routed` | `[-1 x 25, 4, 8, ..., 60, 64, 64]` | `is_moe` is `pyramid_num_routed[i] > 0` (vision.py:346-350), so -1 is DENSE | + +**One word in #2512 needs correcting, and the fixture is what corrects it.** The +issue's scope prose says the dense arm is "`post_trunk_norm` -> pixel shuffle -> +`adapter`". The released `vision_config` sets `adapter_type: "patch_merger"`, +and `PatchMergerAdapter` is upstream's own name for the arm that **skips the +pixel-shuffle permutation** and instead views every 4 consecutive 2x2-grouped +tokens as one row (`vision.py:441-449`, its docstring). The 2x2 regrouping has +not disappeared; `pre_pixel_shuffle: true` moved it into the PREPROCESSOR +(`processor.py:185-197`, the nine-way reshape and the `(0,3,6,4,7,2,1,5,8)` +transpose) and into the RoPE position builder +(`get_pos_ids_by_grid:566-575`, `rope_merge_size = spatial_merge_size`). + +This is not a disagreement about geometry, and it did not need escalating. +#2512's own tensor inventory says `adapter.{ln_q, mlp.0, mlp.2}`, which is +`PatchMergerAdapter`'s state dict and nothing else β€” `PixelShuffleAdapter` +spells its parameters `proj.0` / `proj.1` / `proj.3` (`vision.py:397-406`). The +inventory is right and the prose word is loose. W6a implements +`patch_merger`, and `pixel_shuffle_mlp` is REFUSED BY NAME rather than +silently mapped onto it, because the two produce different token orders from +the same pixels and neither shape-checks against the other. + +#### 4.11.2 The 235 dense tensors, counted + +Of the 2195 `vision_encoder.*` tensors in the released index, W6a's arm claims +**235** and refuses **1960**: + +``` +dense blocks 0..24, 9 each 225 norm_1, norm_2, attn.{qkv,proj,q_norm,k_norm}, + mlp.{fc1,fc2,fc3} (all BF16) +patch_embed 3 proj.weight [1536,3,14,14], proj.bias [1536], + norm.weight [1536] +post_trunk_norm 1 +adapter 6 ln_q.{weight,bias} [1536], + mlp.0.{weight,bias} [6144,6144]/[6144], + mlp.2.{weight,bias} [5120,6144]/[5120] + ---- + 235 +MoE blocks 25..41 (W6b) 1960 17 x {norm_1, norm_2, attn x4, gate_weight, + router_bias} = 136, plus 608 experts x 3 = 1824 + ---- + 2195 +``` + +The 608 is the sum of `pyramid_num_routed[25..41]`, and it is what makes the +released checkpoint still refuse. + +#### 4.11.3 What W6a refuses, BY NAME + +The released `dots-studio/dots3-note-prev` has 17 MoE ViT blocks, so a load of +it REFUSES at the vision tower and names W6b. **That is correct and it is this +row's established pattern rather than a new exception.** W3 refused the language +tower's MoE layers by name for four bricks before W5 lifted it; the vision +tower is at W3's stage, not at W5's. Refusing is what stops the port from +serving a tower whose pyramid it silently skipped, on a row that Β§6.4 records as +having no oracle to catch it. + +| Refused | Named brick | Where | +|---|---|---| +| any block with `pyramid_num_routed[i] > 0` | **W6b** | `Dots3NoteVisionRefusal` | +| `quantization_config.weight_block_size` on the vision tower | **W9** | `Dots3NoteVisionRefusal` | +| `video` modality | **W7** | `EncodeMmDots3Note`, and the chat seam's `allowed_limits` | +| `audio` modality | **W8** | same | +| `adapter_type == "pixel_shuffle_mlp"` | W6b | `ParseDots3NoteVisionParams` | +| `post_norm == false` | W6b | `ParseDots3NoteVisionParams` | +| `use_bias == true` | W6b | `ParseDots3NoteVisionParams` | +| `temporal_patch_size != 1` | W7 (video) | `ParseDots3NoteVisionParams` | +| `adapter_out_dim != config.hidden_size` | none β€” unservable | `Dots3NoteVisionRefusal` | +| `adapter_merge_size != spatial_merge_size` | none β€” unservable | `Dots3NoteVisionRefusal` | + +**The last two rows name no brick, and that is the point of them.** They are not +capabilities owed to a later brick; they are configs no dots3-note tower can be +served under at all. They are here because the fresh review of #2523 measured +the refusal predicate to be a strict SUBSET of the `VT_CHECK`s +`EncodeMmDots3NoteForCausalLM` makes on a served request. Three of those checks +β€” the adapter width against the text width +(`dots3_note_registry.cpp`), the emitted row count against the placeholder span, +and `L % merge_unit` inside the tower β€” were reachable from an all-dense config +the seam ACCEPTED. Reaching any of them throws inside the engine's busy loop, +which sets `AsyncLLM::errored_` permanently (`async_llm.cpp:584-601`), so the +server starts, text works, the first image request 500s, and every later +request is dead for the life of the process. That is the exact cascade the +factory-side refusal was introduced to remove, still reachable through a +narrower door. **A refusal and its route predicate must be the SAME predicate**, +which is this row's second recurrence of the finding: the W4b-3c review made it +about sparse routing, recorded in the first `## Owed` entry. + +A tautology went with them. `adapter_merge_size**2 * adapter_in_dim != +merged_dim()` read as a cross-key check and was `x != x` β€” `merged_dim()` is +that product, reordered (`dots3_note_vision.h`) β€” so it could never fire, and it +was the only refusal in the table that named no brick because it stood for no +condition. The `adapter_merge_size` row above is the real cross-key check that +belongs in its place. + +A refused tower leaves the 2195 `vision_encoder.*` tensors in the accounting's +existing `vision` bucket as a NAMED deferral, exactly as before, so every W2 +count assertion is byte-for-byte unchanged. What moved is the deferral's +`brick` field: `W6` -> `W6b`, because W6a is landed and W6b is what is owed. + +#### 4.11.4 The gate is a CONSISTENCY gate, and says so + +Β§6.4's option B stands. The checkpoint is 298.67 GB fp8 / 576.89 GB bf16 against +119-122 GiB hosts, so vLLM cannot be run on it here and no denominator exists. +Correctness for the tower is therefore argued by an **independent in-test +double-precision reference** written from `vision.py` / `vision_attention.py` at +`9035151d6`, sharing NO helper with the implementation, with RED-first mutation +proof. + +**That establishes two implementations agree. It does not establish that either +matches vLLM.** No performance number is claimable on any axis while B holds, +and none is claimed. The reference and the implementation differ deliberately at +every step that has a choice: the reference is a scalar `double` loop with its +own softmax, its own rope and its own norms; the implementation is +`vt::MatmulBT` / `vt::RmsNorm` / `vt::RopeFromCache` / `vt::AttentionDenseFlash` +over bf16 device buffers through the shared seams. + +**One formula difference is deliberate and is recorded rather than hidden.** +Upstream's vision `RMSNorm.forward` (`vision.py:112-114`) casts the normalized +value back to the activation dtype BEFORE multiplying by the weight; `vt::RmsNorm` +keeps f32 through the weight multiply and rounds once on the store (its own +header says so). Using the shared op is the seam rule. The reference does NOT +copy the cast: at infinite precision the two are the same function, so a double +reference is the algebra BOTH implement, and the gate's tolerance covers our +bf16 storage and upstream's intermediate cast together. Copying the cast into +the reference would make the reference agree with a rounding choice instead of +with the maths. Β§4.11.6 records the measured deviation, the bound, and the +mutation that proves the bound is not a mute switch. + +**The discrete-selection rule does not bind W6a and that is a fact about the +arm, not an omission.** The dense blocks have no top-k anywhere: routing is +W6b's. When W6b lands it owes a SET-equality assertion on the router's top-k +plus the printed minimum decision margin, because a tolerance alone cannot see a +bimodal selection flip. + +**The memory format is asserted against upstream explicitly** (porting.md), and +this row has already been bitten on that axis: W2's F1 fixture row proves a +re-typed `router_bias` fires. Every dense vision tensor is BF16 on disk and BF16 +in the resident tower, and the gate asserts the tower's stored dtype rather than +only its values β€” a token gate cannot see a dtype that is too wide. + +#### 4.11.5 Reachability β€” the production entry point, and what it costs to fake + +The production entry point is `ApiServer::handle_chat_completions` on the +server's default configuration. The smallest failing test enters THROUGH it, over +a synthetic in-memory checkpoint at tiny geometry with a generated tokenizer +fixture whose added tokens are `<|img|>` / `<|imgpad|>` / `<|endofimg|>`. A unit +test that constructs the tower by hand proves the class works, never that +anything reaches it. + +The load-bearing case is **two DIFFERENT images, one prompt, compared on +LOGPROBS**. It is the only one that survives a tower replaced by a correctly +SHAPED constant: every text-only assertion β€” status 200, `prompt_tokens`, +`completion_tokens` β€” passes under that mutation, and the logprobs of the first +generated token do not. + +#### 4.11.6 Evidence, measured 2026-09-01 + +Host: the developer's x86-64 Linux box, CPU queue, `-DVLLM_CPP_SERVER=ON +-DVLLM_CPP_BUILD_TESTS=ON -DVLLM_CPP_CUDA=OFF -DCMAKE_BUILD_TYPE=Release`. No +GPU lease was taken and no number below is a performance number. + +| Suite | Result | +|---|---| +| `test_dots3_note_vision` (new) | 8 cases, **4996 assertions**, 0 failed | +| `test_openai_api_server_dots3_mm_forward` (new) | 9 cases, **68 assertions**, 0 failed | +| `test_dots3_note_scaffold` | 26 cases, **110835 assertions**, 0 failed | +| `test_dots3_note_attn` | 51 cases, **6888 assertions**, 0 failed | +| `test_openai_api_server_mm_forward` (Qwen3-VL, untouched) | 9 cases, **73 assertions**, 0 failed | +| `test_model_registry` (repaired here) | 24 cases, **993 assertions**, 0 failed | +| **the FULL gate** β€” `ninja` all 1381 targets then `ctest -j 2` | **702/702, 0 failed**, 7 skipped for absent checkpoints, `NINJA_RC=0`, `CTEST_RC=0`, 158.06 s of `ctest` (rerun 2026-09-02 for the fresh-review repair) | +| `scripts/agent-preflight.sh` | rc 0 | +| `check-commit-style.py` / `check-commit-trailers.py` over `$(git merge-base origin/main HEAD)..HEAD` | rc 0 / rc 0 | + +**The full gate found one thing reading did not.** `test_model_registry`'s +`registry_model_property` partitions every registration into hybrid, +multimodal-non-hybrid and text-only and asserts `supports_multimodal` per branch; +W5 had moved `Dots3NoteForCausalLM` into the text-only branch, and W6a's flag flip +made it red. It is repaired here, and its comment records the true -> false -> +true round trip rather than erasing it. + +**The consistency measurement.** The tower against the independent +double-precision reference: `max |diff| 0.0533141 over a scale of 6.31441 => +relative 8.44e-3`. The gate's bound is **0.02**, a 2.4x margin β€” wide enough that +a different libm or a different GEMM reduction order does not red it, tight +enough that M5 below (relative 5.9e-2) exceeds it by 2.95x. A bound at the round +0.05 a first draft carried would have cleared M5 by only 1.18x, which is one +compiler from a mute switch. + +**WHAT THE 0.02 BOUND CANNOT SEE.** On a row with no oracle the honest statement +of what the gate does NOT detect is part of its evidence, not a caveat outside +it. Both numbers below were measured by the fresh review of #2523, on this +tree's own gate: + +- **A uniform scale error passes until about 1.5%.** Multiplying the tower's + output by 1.01 reads relative **0.0152** against the 0.02 bound, and the gate + stays GREEN. The detection floor for a systematic MULTIPLICATIVE error is + therefore ~1.5%: a missing or doubled scalar smaller than that is invisible + here, and no assertion in the suite bounds the output's SCALE independently of + its shape. +- **A named formula choice is below the gate's resolution.** Replacing the + exact-erf GELU with the tanh approximation leaves the measurement + BYTE-IDENTICAL to the baseline β€” the same printed digits, max |diff| + **0.0533141** β€” because the bf16 store of `fc1` absorbs the whole difference. + The gate cannot tell the two formulas apart at this geometry, so "we use + upstream's GELU" is a claim the CODE and the upstream anchor carry, never one + this measurement supports. + +Neither weakens the two claims above: the 8.44e-3 agreement and M5's 5.9e-2 red +both stand. What they bound is the CLASS of defect the gate detects β€” a change +to the ORDER or the STRUCTURE of the arithmetic, which is what M5 is β€” and not a +small uniform rescale, and not a rounding-equivalent formula swap. + +**Mutations.** Each was applied to the tree, REBUILT, and its test binary's +sha256 compared against the green baseline β€” a mutation that never reached the +binary reads as a passing test. Each file was then restored and `cmp` reported +byte-for-byte identity, and the rebuilt binaries hashed back to the EXACT green +baselines (`c6e83b90...` served, `b03f59e3...` tower). + +**A CHANGED SHA IS NECESSARY AND NOT SUFFICIENT, and the argument below +originally overstated it.** The sha proves the build was not STALE, which is the +trap it was chosen for. It does not prove the mutation reached the code under +test: M4 changed the TOWER gate's binary sha purely by relinking a translation +unit that gate does not exercise, while that binary's behaviour was unchanged β€” +8/8, 4981 assertions, as the paragraph after the table records. The evidence +that a mutation was DETECTED is the CASE COUNT in the Result column, and nothing +else in this table can carry that weight. + +| # | Mutation | Binary sha256 (served gate) | Result | +|---|---|---|---| +| β€” | green baseline | `c6e83b90359f1402…` | 7/7, 55 assertions | +| M1 | the `Dots3NoteVisionForward` call inside `encode_mm` DELETED | `936574cab156b69b…` | **RED** β€” 3 of 7 cases, 43 assertions reached | +| M2 | the tower REPLACED by a correctly-SHAPED constant | `5664578c254c8fa4…` | **RED β€” and only ONE case: "two DIFFERENT images give two different forwards".** 6 of 7 pass, including status 200, `prompt_tokens` and `completion_tokens`. This is the measurement behind Β§4.11.5: without the logprob case this mutation is invisible | +| M3 | the `.mm` read in `Dots3NoteModel::ForwardDevice` DELETED (`have_mm_embeds = false`) | `7285bd3e44221bcb…` | **RED** β€” the same single case, for the same reason: the vision rows never reach the residual stream | +| M4 | the production `MaterializeDots3NoteVision` call site in the LOADER deleted | `80901ec80645b598…` | **RED** β€” 3 of 7 cases | +| M5 | the per-head `q_norm`/`k_norm` moved from BEFORE the rope to AFTER it | tower gate `b39c3e36a0abbfad…` | **RED** β€” relative 5.9e-2 against the 2.0e-2 bound, a 7x jump from the green 8.4e-3 | + +**THE REFUSAL REPAIR IS RED-FIRST, and its RED is the CASCADE rather than a +missing message.** The two served cases were written and built BEFORE the +refusal was widened, on binary `05848d1f4226e416…` (tower gate +`2d2dc86de602c004…`). Both failed at the install assertion β€” `kInstalled` where +`kRefusing` is required β€” so to show what that install then costs, the two +`REQUIRE`s were downgraded to `CHECK` in a scratch build (`34ff9a39094e0e18…`) +and the cases ran to the end. Verbatim, from that run: + +```text +engine-fatal: EngineCore busy loop threw: vt: Dots3NoteForCausalLM encoder: the + vision adapter emits 24-wide rows but the text tower is 16 wide + (`adapter_out_dim`, vision.py:461 @ 9035151d6) at dots3_note_registry.cpp:202 +async-llm: output handler saw engine death: EngineCore encountered an issue. + CHECK( r.status == 400 ) is NOT correct! values: CHECK( 500 == 400 ) + ...the TEXT request sent AFTERWARDS on the same server: + {"error":{"code":500,"message":"EngineCore encountered an issue. ... + [request submitted to a stopped AsyncLLM]"}} + CHECK( t.status == 200 ) is NOT correct! values: CHECK( 500 == 200 ) +``` + +The merge-size case reaches the OTHER assert on the same path +(`dots3_note_registry.cpp:221`, "the tower produced 16 embedding rows for a +placeholder span of 4 tokens") and ends in the same +`request submitted to a stopped AsyncLLM`. The tower gate's own RED was 11 +failed assertions over 4996 in 1 of 8 cases. The scratch file was restored and +`cmp` reported byte-for-byte identity before the fix was applied. + +| # | State | Served-gate sha256 | Tower-gate sha256 | Result | +|---|---|---|---|---| +| β€” | new cases, refusal NOT widened | `05848d1f4226e416…` | `2d2dc86de602c004…` | **RED** β€” 2 of 9 cases; 11 of 4996 tower assertions | +| β€” | the same, `REQUIRE` -> `CHECK` so the cases run on | `34ff9a39094e0e18…` | β€” | **RED**, and the 500 + `stopped AsyncLLM` above is why | +| β€” | refusal widened (this repair) | `c8d9573ef3d605a7…` | `8f2b81436dcdac81…` | **GREEN** β€” 9/9, 68 assertions; 8/8, 4996 assertions | + +**M4 leaves the TOWER GATE GREEN, and that is the point of having two files.** +`test_dots3_note_vision` materializes the tower itself, so deleting the +production call site does not move it: 8/8, 4981 assertions, on a tree where +nothing in the loader builds a vision tower at all. Only the served-request gate +sees it. That is AGENTS.md's "a unit test that constructs the type by hand proves +that the class works, never that anything reaches it", demonstrated rather than +argued. + +**M5's first attempt did not compile** (`-Werror=unused-but-set-variable` on the +two tensor views the move orphaned) and the stale binary printed the GREEN +result. A build failure reading as a passing test is a recorded trap in this +tree; the row above is the SECOND attempt, whose `RC=0` and changed sha256 are +what make it evidence. + +**One behaviour changed under measurement, and it is recorded rather than +smoothed over.** The first version refused a MoE tower only inside +`EncodeMmDots3NoteForCausalLM`. That throw happens in the engine's busy loop: it +stopped `AsyncLLM`, and every LATER request β€” TEXT ones included β€” came back 500. +The served-request gate caught it. The refusal now also runs in the chat +FACTORY, which turns it into a REFUSING seam: HTTP 400 naming the architecture +and the block, with the text path still answering afterwards. The encoder check +stays as defence in depth, on the same polarity Qwen3-VL's carries ("reaching +this point is a defect"). The gate asserts BOTH halves: 400 on the image, 200 on +a text request sent after it. + +--- + ## 5. Gates **Correctness first, and the gate form is chosen by measurement, not in advance** @@ -4447,14 +4798,38 @@ dispatchable in order, under the constraints that answer imposes. `Dots3NoteDeviceRefusal(released_params)` EMPTY. What that does NOT mean is recorded in Β§4.10 and in the row header: the MoE is 94.62% of a 576.89 GB checkpoint and nothing here can hold it. -- **W6 β€” vision tower.** Dense ViT half first, then the pyramid MoE and the - FP32-scale FP8 formula. Reuses `qwen3_vl_vision` structure. +- **W6a β€” the DENSE vision tower, SERVED. LANDED** (evidence Β§4.11, + [#2512](https://github.com/mudler/vllm.cpp/issues/2512)). `patch_embed` -> + blocks 0-24 (`attn.qkv` + per-head `q_norm`/`k_norm` + 2-D vision RoPE + + bidirectional attention, then the three-tensor SwiGLU) -> `post_trunk_norm` -> + the `patch_merger` adapter, reached from + `ApiServer::handle_chat_completions` through the model's `encode_mm` / + `embed_mm` hooks and its own `REGISTER_VLLM_MM_CHAT` translation unit. + Structurally reuses `qwen3_vl_vision`'s outline; shares no code with it, + because the two towers agree on almost nothing below that outline (RMSNorm vs + LayerNorm, no bias, qk-norm, a three-tensor SwiGLU, a patch-merger adapter, no + DeepStack, no position-embedding table, no M-RoPE). +- **W6b β€” the pyramid MoE ViT.** Blocks 25-41: `mlp.gate_weight` + + `mlp.router_bias`, sigmoid scoring, `capacity_factor`-derived top-k, and the + `moe_intermediate_size` experts. It OWES a SET-equality assertion on the top-k + plus the printed minimum decision margin; a tolerance alone cannot see a + bimodal selection flip. The RELEASED checkpoint has 17 such blocks, so its + vision tower still refuses BY NAME until this lands, which is W3's polarity + applied to the second tower rather than a new exception. - **W7 β€” audio tower.** The `dots` stem deltas over our Whisper encoder. - **W8 β€” MM front end + ABI.** Processor, video sampling, placeholder expansion, `<|audio_comp_*|>`, `include/vllm.h` surface, the example server as a thin client. - **W9 β€” quantized arms.** Blockwise FP8 and the owed GGUF k-quant arm + - converter. + converter. **The vision MoE's FP32-scale FP8 formula belongs HERE, not to W6** + (moved 2026-09-01 with W6a, [#2512](https://github.com/mudler/vllm.cpp/issues/2512)). + The evidence that moved it is the row's own W2 census: the released bf16 + `dots-studio/dots3-note-prev` carries 37944 BF16 + 62 F32 tensors and NO scale + tensors at all (Β§4.4), so there is no FP8 formula anywhere in the arm W6 + loads; `MoESwiGLUFFNFP8.process_weights_after_loading` (`vision.py:226-268` @ + `9035151d6`) CASTS bf16 experts to block-FP8 at load, which is a quantized + path this port does not take, and the `-fp8` sibling that ships those scales + is already refused BY NAME as W9. **R5 moved with it** (Β§8). - **W10 β€” MTP.** `Dots3NoteMTPModel` over the existing speculator seam. - **W11 β€” gates.** Whatever Β§6.4 permits: full SACRED if A, the recorded-gap form if B. @@ -4476,7 +4851,15 @@ dispatchable in order, under the constraints that answer imposes. - **R5 β€” the vision MoE's FP32 activation scales** (Β§2.4) are the exact shape of a too-wide/too-narrow dtype defect that a token gate cannot see. Check the memory format against upstream explicitly, per - [porting.md](../porting.md). + [porting.md](../porting.md). **FILED AGAINST W9, not W6** (moved 2026-09-01 + with W6a, [#2512](https://github.com/mudler/vllm.cpp/issues/2512)). It was + filed against W6 when this section was written, and the row's own W2 census + (Β§4.4) contradicts that: the released bf16 checkpoint ships 37944 BF16 + 62 + F32 tensors and no scale tensor at all, so W6's arm has no activation scale to + get wrong, while the `-fp8` sibling that does ship them is refused BY NAME as + W9. The RISK is unchanged and is not waived; only its owner moved. The + memory-format obligation it names still binds every brick, and W6a discharged + its own share of it in Β§4.11.4. - **R6 β€” no llama.cpp comparison** for the GGUF arm, so the quantized floor has no external reference. Record it as a gap rather than substituting a different model's number. @@ -4497,6 +4880,35 @@ change as the lifecycle move, not afterwards. Carried openly under option B (Β§6.4), not waived: +- **The image processor REFUSES instead of resizing, so no non-conformant image + is servable.** `Dots3NoteImageProcessor::ProcessImage` + (`src/vllm/multimodal/dots3_note_processor.cpp`) computes the resized size and + then throws when it differs from the input size, rather than performing the + `Image.Resampling.BICUBIC` resample upstream always performs + (`common/processor.py:174` @ `9035151d6`). `Dots3NoteResizedSize` itself is + ported and correct, including the `min_pixels`/`max_pixels` rebalance + (`processor.py:97`); it is the resample AFTER it that is missing. **This is a + capability gap and not only a bookkeeping one.** `factor` is + `patch_size * merge_size`, which on the released `dots-studio/dots3-note-prev` + is 28, so an image is servable only when BOTH dimensions are already multiples + of 28 and the pixel count already sits inside the bounds β€” which almost no + real photograph or screenshot does. W6a never reaches the throw because its + fixture image is conformant by construction, and once W6b lifts the MoE ViT + refusal this becomes what a user hits instead. Refusing remains the right + INTERIM behaviour: patchifying at the wrong grid changes the placeholder count + and serves a well-shaped wrong prompt, which Β§6.4 records as having no oracle + to catch it. Closing it needs the resample matched to PIL's kernel + (`a = -0.5`, its support radius, its per-axis two-pass order and its clamping), + a gate that measures the resampler against a reference on a NON-conformant + image rather than only checking that the placeholder count comes out right, + and the refusal deleted in the same change so message and behaviour cannot + drift. **The record it claimed did not exist.** The code comment and the + runtime message both said "Recorded under `## Owed` in + `.agents/specs/dots3-note.md`" while this section named it nowhere and no + issue tracked it; the fresh review of #2523 found that, and this entry and + the issue below are the repair. Owner: this row, W8 (the MM front end brick + that owns the processor, Β§7). Issue + [#2537](https://github.com/mudler/vllm.cpp/issues/2537). - **PER-REQUEST sparse routing for a MIXED step, and the refusal that stands in for it.** The W4b-3c review found the route predicate and the refusal predicate to be different predicates with a reachable gap between them, and @@ -5180,8 +5592,34 @@ released config becoming loadable made a claim this port cannot honour: the 2195 vision and 430 audio tensors are named W6/W7 deferrals and the multimodal front end (W8) does not exist. W8 flips it back. -**Next dispatchable: W6 β€” the vision tower**, or W9 for the quantized arms if -the fp8 sibling is wanted before the towers. `## Owed` is unchanged except that -`vt::QuantFp8Group`'s missing `use_ue8m0` rounding is now recorded against W9 -with the reason, because upstream's blockwise-fp8 MoE routes through DeepGEMM -with e8m0 scales. +**W6a β€” LANDED, and this row can now be asked for something over HTTP.** +([#2512](https://github.com/mudler/vllm.cpp/issues/2512), evidence Β§4.11.) The +DENSE half of the vision tower β€” `patch_embed`, blocks 0-24, `post_trunk_norm` +and the `patch_merger` adapter β€” runs on a served `image_url` chat request +through `ApiServer::handle_chat_completions` -> the architecture-dispatched chat +seam -> `GPUModelRunner::execute_mm_encoder` -> `ModelRegistry::EmbedMm` -> +`ModelRegistry::Forward`. `supports_multimodal` went FALSE -> TRUE, which is the +second half of the true -> false -> true trail W5 predicted, and this time the +flag is backed: `kDots3NoteFactory` sets `encode_mm` and `embed_mm`, and +`Dots3NoteForCausalLM` has its own `REGISTER_VLLM_MM_CHAT` translation unit. + +**Say the other half in the same breath, again.** The RELEASED checkpoint STILL +REFUSES, at its first MoE ViT block. 17 of its 42 vision blocks are MoE, so 1960 +of the 2195 `vision_encoder.*` tensors are W6b's and the tower refuses BY NAME +before it loads one of them. That is W3's polarity applied to the second tower, +not a new exception. What W6a changed is that a config whose vision blocks are +all DENSE is now served end to end rather than refused, and that the seam every +future arm plugs into exists and is gated. + +**The gate is a CONSISTENCY gate and nothing more** (Β§6.4 option B, Β§4.11.4): +an independent in-test double-precision reference written from `vision.py` and +`vision_attention.py` at `9035151d6`, sharing no helper with the implementation. +It establishes that two implementations agree. It does not establish that either +matches vLLM, and no performance number is claimed on any axis. + +**Next dispatchable: W6b β€” the pyramid MoE ViT**, which is what the released +checkpoint's vision tower is waiting on, or W7 for the audio tower, or W9 for +the quantized arms. `## Owed` is unchanged except that `vt::QuantFp8Group`'s +missing `use_ue8m0` rounding is now recorded against W9 with the reason, because +upstream's blockwise-fp8 MoE routes through DeepGEMM with e8m0 scales, and that +R5 and the vision FP8 formula moved from W6 to W9 (Β§7, Β§8). diff --git a/CMakeLists.txt b/CMakeLists.txt index c5b849779..49f495784 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -808,6 +808,7 @@ add_library(vllm STATIC src/vllm/model_executor/models/dots3_note.cpp src/vllm/model_executor/models/dots3_note_attn.cpp src/vllm/model_executor/models/dots3_note_device.cpp + src/vllm/model_executor/models/dots3_note_vision.cpp src/vllm/model_executor/models/dots3_note_registry.cpp src/vllm/model_executor/models/qwen4_exp_ple.cpp src/vllm/model_executor/models/qwen4_exp.cpp @@ -1080,6 +1081,7 @@ add_library(vllm STATIC # Per-modality input-count limits and the refusal they carry # (ENG-MM-INPUT-PIPELINE L1, #607). src/vllm/multimodal/processing/context.cpp + src/vllm/multimodal/dots3_note_processor.cpp src/vllm/multimodal/qwen3vl_processor.cpp src/vllm/multimodal/audio_processor.cpp # Parakeet / FastConformer audio encoder + CTC head + its log-mel front end @@ -1243,6 +1245,7 @@ add_library(vllm STATIC # registry, and Qwen3-VL's registration into it. The registration TU is # SELF-REGISTERING and is reached through vllm's INTERFACE --whole-archive. src/vllm/entrypoints/openai/mm_chat_registry.cpp + src/vllm/entrypoints/openai/mm_chat_dots3note.cpp src/vllm/entrypoints/openai/mm_chat_qwen3vl.cpp src/vllm/entrypoints/openai/serving_utils.cpp src/vllm/entrypoints/openai/serving_completion.cpp diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 7ef7009e4..6db42f8c8 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -155,7 +155,7 @@ speed-pending, which [BENCHMARKS.md](BENCHMARKS.md) tracks. | `LagunaForCausalLM` | poolside/Laguna-S-2.1-NVFP4, GGUF-Q4_K, Laguna-XS | byte-exact near-tie (distributional vs vLLM) | vLLM parity+ 1.03x, default on, via the `laguna-gen` CLI; the registered engine forward VT_CHECKs non-bf16 (`ARCH-ONE-SURFACE` fold) | | `KimiLinearForCausalLM` | Kimi-Linear-48B-A3B (KDA + NoPE-MLA + MoE) | **Folded onto the shared paged runner (ROW 7 Β§21, #122): engine==CLI 128/128 byte-identical; vs golden 122/128 (the intrinsic near-tie profile); FA2 paged MLA default-ON; SACRED post-fold green** | Served via `vllm_engine_load` + `vllm_complete_tokens` (ABI v13); server 19.0 tok/s wall vs vLLM ~21 (~0.90Γ—), speed residual open | | `KimiK3ForConditionalGeneration` | Kimi-K3 (2.8T MoE) | scaffold: registry+config+enumeration gated, forward refuses | HW-infeasible (~1.56 TB); no run | -| `Dots3NoteForCausalLM` | `dots-studio/dots3-note-prev` @`1e1e7b0c` (280B-A16B multimodal MoE, 576,886,825,984 bytes bf16; the `-fp8` sibling `dots-studio/dots3-note-prev-fp8` @`7c14222e` is 298,673,280,504 bytes = 298.67 GB decimal / 278.16 GiB binary). Headers only β€” no tensor byte downloaded | W1+W2 scaffold: registry + config gated off the REAL released `config.json`, with one assertion per Β§4 config trap (ungrouped 1/1 router, GPT-J indexer RoPE, one nextn layer, the two LoRA rescales, the sliding theta); name map accounted **38006/38006** over the WHOLE released index β€” at W2 that split read 35381 language / 2195 vision / 430 audio, and W5c re-bucketed it (see below) β€” with the two tower files carried as named W6/W7 deferrals rather than dropped; W4a+W4b-2 put BOTH attention geometries on the DECODE PATH, reached through `ModelRegistry::Forward`: the 13 full-attention layers with the two LoRA rescales, `k_rope_only_layernorm` and the headwise gate, and the 33 sliding-window layers over a PADDED 1088-wide MLA cache row that each layer narrows to its own logical width on read; `vt::MlaDecodeAttention` and `vt::MlaPrefillAttention` grew an optional window whose absent state is bit-identical to no window; W4b-3c put the DSA lightning indexer's SELECTION on the same path, so a SINGLE-SHOT prefill longer than `index_topk` is now served sparsely instead of refused β€” `vt::MlaDecodeAttention` grew an optional selected-slot arm whose absent state is bit-identical to no selection and whose FULL selection reproduces the dense answer byte for byte, beside a new `vt::DsaIndexerLogits` / `vt::DsaTopkSelect` pair on CPU and CUDA. A STEP in which any request has CACHED CONTEXT and any request is past `index_topk` still REFUSES BY NAME β€” the sparse route is a property of the step, not of one request β€” because the indexer's own key cache is a second attention group owned by `KV-DSV4-MULTICACHE` (#1925). W5 put the 45 MoE layers on the same path through `Dots3NoteMoeBlock` over the shared `RunMoePlaced` seam β€” the UNGROUPED (n_group=1 / topk_group=1) noaux_tc router at 256/8 with the F32 `e_score_correction_bias` feeding the SELECTION and the unbiased scores feeding the routing weights, plus one shared expert at `moe_intermediate_size * n_shared_experts` = 1536 and NOT `intermediate_size` = 13824 β€” with no `vt` op changed. W5c removed the last refusal, and that one was a DEFECT rather than a gap: the nextn branch was STRICTER THAN UPSTREAM, which drops `model.layers.46.*` and `model.mtp.*` from the main model instead of refusing (`utils.py:542` -> `deepseek_v2.py:1618-1620`; `models/dots3_note/nvidia/model.py:624`), so those 19 tensors are now a NAMED W10 deferral with their own accounting bucket ([#2176](https://github.com/mudler/vllm.cpp/issues/2176)) and the split reads 35362 language / 19 nextn / 2195 vision / 430 audio. **So `Dots3NoteDeviceRefusal` is EMPTY for the released `config.json`, and that is representable rather than runnable**: the MoE alone is 545.82 GB of a 576.89 GB checkpoint (94.62%), which no host this project reaches can hold. What still refuses BY NAME is the blockwise-FP8 sibling (W9 β€” `weight_block_size [128, 128]` with a `weight_scale_inv` per projection, which this port's bf16 loaders cannot read), GGUF k-quants (W9), and both towers (W6/W7). `supports_multimodal` went TRUE -> FALSE at W5 because the released config became loadable while the vision tower, the audio tower and the multimodal front end are still W6/W7/W8; W8 flips it back | **No oracle, on any host we own** (298.67 GB fp8 against a 122 GiB ceiling), so NO number is claimable on any axis and the e2e gate is an open gap by construction ([spec](../.agents/specs/dots3-note.md) Β§6.4, #699) | +| `Dots3NoteForCausalLM` | `dots-studio/dots3-note-prev` @`1e1e7b0c` (280B-A16B multimodal MoE, 576,886,825,984 bytes bf16; the `-fp8` sibling `dots-studio/dots3-note-prev-fp8` @`7c14222e` is 298,673,280,504 bytes = 298.67 GB decimal / 278.16 GiB binary). Headers only β€” no tensor byte downloaded | W1+W2 scaffold: registry + config gated off the REAL released `config.json`, with one assertion per Β§4 config trap (ungrouped 1/1 router, GPT-J indexer RoPE, one nextn layer, the two LoRA rescales, the sliding theta); name map accounted **38006/38006** over the WHOLE released index β€” at W2 that split read 35381 language / 2195 vision / 430 audio, and W5c re-bucketed it (see below) β€” with the two tower files carried as named W6/W7 deferrals rather than dropped; W4a+W4b-2 put BOTH attention geometries on the DECODE PATH, reached through `ModelRegistry::Forward`: the 13 full-attention layers with the two LoRA rescales, `k_rope_only_layernorm` and the headwise gate, and the 33 sliding-window layers over a PADDED 1088-wide MLA cache row that each layer narrows to its own logical width on read; `vt::MlaDecodeAttention` and `vt::MlaPrefillAttention` grew an optional window whose absent state is bit-identical to no window; W4b-3c put the DSA lightning indexer's SELECTION on the same path, so a SINGLE-SHOT prefill longer than `index_topk` is now served sparsely instead of refused β€” `vt::MlaDecodeAttention` grew an optional selected-slot arm whose absent state is bit-identical to no selection and whose FULL selection reproduces the dense answer byte for byte, beside a new `vt::DsaIndexerLogits` / `vt::DsaTopkSelect` pair on CPU and CUDA. A STEP in which any request has CACHED CONTEXT and any request is past `index_topk` still REFUSES BY NAME β€” the sparse route is a property of the step, not of one request β€” because the indexer's own key cache is a second attention group owned by `KV-DSV4-MULTICACHE` (#1925). W5 put the 45 MoE layers on the same path through `Dots3NoteMoeBlock` over the shared `RunMoePlaced` seam β€” the UNGROUPED (n_group=1 / topk_group=1) noaux_tc router at 256/8 with the F32 `e_score_correction_bias` feeding the SELECTION and the unbiased scores feeding the routing weights, plus one shared expert at `moe_intermediate_size * n_shared_experts` = 1536 and NOT `intermediate_size` = 13824 β€” with no `vt` op changed. W5c removed the last refusal, and that one was a DEFECT rather than a gap: the nextn branch was STRICTER THAN UPSTREAM, which drops `model.layers.46.*` and `model.mtp.*` from the main model instead of refusing (`utils.py:542` -> `deepseek_v2.py:1618-1620`; `models/dots3_note/nvidia/model.py:624`), so those 19 tensors are now a NAMED W10 deferral with their own accounting bucket ([#2176](https://github.com/mudler/vllm.cpp/issues/2176)) and the split reads 35362 language / 19 nextn / 2195 vision / 430 audio. **So `Dots3NoteDeviceRefusal` is EMPTY for the released `config.json`, and that is representable rather than runnable**: the MoE alone is 545.82 GB of a 576.89 GB checkpoint (94.62%), which no host this project reaches can hold. What still refuses BY NAME is the blockwise-FP8 sibling (W9 β€” `weight_block_size [128, 128]` with a `weight_scale_inv` per projection, which this port's bf16 loaders cannot read), GGUF k-quants (W9), and both towers (W6/W7). **W6a ([#2512](https://github.com/mudler/vllm.cpp/issues/2512)) puts the DENSE half of the VISION TOWER on a SERVED request**: an `image_url` chat request travels `ApiServer::handle_chat_completions` -> the architecture-dispatched multimodal chat seam (a `REGISTER_VLLM_MM_CHAT` translation unit of its own, #2481) -> `GPUModelRunner::execute_mm_encoder` (#2398) -> `ModelRegistry::EmbedMm` -> `ModelRegistry::Forward`, and `kDots3NoteFactory` now carries `encode_mm` and `embed_mm`. The arm is `patch_embed` -> blocks 0-24 (fused `attn.qkv` with no bias, per-head `q_norm`/`k_norm` applied BEFORE a 2-D vision RoPE, bidirectional attention, and the three-tensor SwiGLU routed through `layers::MlpGateUpMethodBase`) -> `post_trunk_norm` -> the `patch_merger` adapter folding 4x1536 to 5120, plus a dots3 image processor with its own per-channel normalization, its own `resized_size` and its own marker. **The RELEASED checkpoint still refuses**: 17 of its 42 vision blocks are pyramid MoE (608 routed experts, 1960 of the 2195 `vision_encoder.*` tensors), so the tower refuses at block 25 naming W6b, the chat seam installs REFUSING, and an image request is HTTP 400 while text keeps serving. That is W3's polarity applied to the second tower, not a new exception. Ported from vLLM `nvidia/vision.py` (677 lines) / `nvidia/vision_attention.py` (477 lines) / `common/processor.py` read at `9035151d6`, which is BEYOND our pin `5559679229`; every anchor names that SHA because `vision_attention.py` is already 494 lines at vLLM `main` `7a100bb61`. The gate is a CONSISTENCY gate (spec Β§6.4 option B): an independent in-test double-precision reference sharing no helper with the implementation, plus a served-request reachability suite whose load-bearing case is two DIFFERENT images compared on logprobs. `supports_multimodal` went TRUE -> FALSE at W5 and FALSE -> TRUE at W6a, and the trail is the honest record of what this port could back at each point; W7 (audio) and W8 (the video/audio front end) are still owed | **No oracle, on any host we own** (298.67 GB fp8 against a 122 GiB ceiling), so NO number is claimable on any axis and the e2e gate is an open gap by construction ([spec](../.agents/specs/dots3-note.md) Β§6.4, #699) | | `NemotronHForCausalLM` | Nemotron-3.5-Lightning-30B-A3B-NVFP4 (`nvidia` @`29f2d174`) | config+enumeration+KV-shape gated; hybrid forward COMPUTES; loader materializes 18487/18487 as SHIPPED; A3 e2e gate 96/96 `STRICT PASS` on GB10 at `0ea5d249f` (#1221); NO run against current `main` | **PAGED (#810 A2-P): K/V go to the runner's pages; conv+SSM rows carry at the metadata's state indices.** G-SAFE: `num_reqs <= 1`. Device `lm_head` (A2-Q2b), UNMEASURED. Owed: FP8 mamba (A2-Q1), MTP, GGUF | | `MuseGlimmerForCausalLM` | real tensors, **bf16 depth 4/52 only**: 5 prefill argmax positions match a torch transcription of vllm#51655 and HF. GGUF full depth generates coherently (#347, #359) but is **NOT token-exact** | text forward + loader vs an fp32 reference, per-mechanism property tests, scaffold 11/11, GGUF gate 17/17. An ABSENT config key now takes the architecture's constant (#412): GGUF post-norms ran at 1e-5, not 1e-8 | no vLLM denominator (pin cannot load it); SECONDARY llama.cpp, same GGUF, GB10 CPU: prefill tie **0.997x**, decode 0.232x, RSS 1.92x (#333) | | `MuseGlimmerForConditionalGeneration` | vision: **no reference run of any kind**; enumeration gated vs the released 30B index (1436/1436). Image/video need bf16 safetensors: `mmproj-kquant.gguf` is refused by name | perception encoder loaded and wired, so an image or video prompt runs; `perception_emb_norm` now armed by default (#405). Reachability plus placeholder scatter only, no image or video correctness | not measurable; anchored to open vllm#51655 | diff --git a/docs/USAGE.md b/docs/USAGE.md index 5f98194f6..5aa091e08 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -647,7 +647,7 @@ repository in this project's history. | GLM-5.3 config | `config.json` | 29,464 bytes | `zai-org/GLM-5.3` @ `935644c05e76fc198714f4cca449fd8b970ff6d7` | Committed verbatim in-tree as `tests/vllm/models/glm_moe_dsa_config_glm53.inc`, so the config layer is gated against what the checkpoint says rather than against what a port's author believed it says | It is the ONLY authoritative source of the 78-entry `indexer_types` list β€” 21 `full`, at layers {0,1,2} and every fourth from 6 to 74 β€” which three independent derivations agree on bit for bit (the list itself, vLLM's rule at `deepseek_v2.py:1097-1101`, and llama.cpp's `GLM_5_2_DEFAULT_INDEXER_TYPES`) | The GGUF above does not carry this list, which is why it cannot be fed as published | | Qwen3.5-0.8B (Tenstorrent P150 arm) | `model.safetensors-00001-of-00001.safetensors` | 1,746,942,600 bytes | `Qwen/Qwen3.5-0.8B` @ `2fc06364715b967f1860aea9cf38778875588b17`, authorized 2026-08-23 | `04b1c301231dd422b8860db31311ab2721511346a32cb1e079c4c4e5f1fe4696` (non-quantized; hashed anyway from the local bytes the gates and the eager profile consumed) | bf16 on the Tenstorrent P150: the sacred greedy pair, both ambient legs, and the #1715/#2107 profile legs all ran from this snapshot | **Arms refused by name:** GGUF k-quant arms on TT β€” no TT kernels exist for them, refused at load; Qwen3.8-27B on TT β€” no arm fits the P150 (bf16 53.8 GB), refused at load | | dots3-note bf16 language tower | `model-000{01..131}-of-00131.safetensors` | 561,371,869,568 bytes total (522.82 GiB), of which the MoE is 545,823,175,680 | `dots-studio/dots3-note-prev` @ `1e1e7b0cd37a3a48a6c8d7fa55d5f9d14377006b` | Owed: **no tensor byte has been fetched**, so no local hash exists to state, and an unauthenticated tree hash is not a pin here | The bf16 text tower this port loads: 46 backbone layers, both MLA geometries, and since W5 the 45 MoE layers β€” the ungrouped noaux_tc router at 256/8 plus one shared expert at `moe_intermediate_size * n_shared_experts` = 1536. Everything except `mlp.gate.e_score_correction_bias` is BF16; that one is F32, on both sides | **Nothing has ever loaded these bytes.** The tower alone is 522.82 GiB against a 122 GiB ceiling on the largest host this project reaches (spec Β§6.2), so the arm is representable and unfeedable, and the e2e gate is an OPEN GAP by construction. GGUF k-quants are refused by name (W9). The 19-tensor nextn tail is a NAMED W10 deferral rather than a refusal since #2176 | -| dots3-note vision tower | `model-vision.safetensors` | 13,742,557,056 bytes | `dots-studio/dots3-note-prev` @ `1e1e7b0cd37a3a48a6c8d7fa55d5f9d14377006b` | Owed, as above | none | **Refused as a NAMED W6 deferral**: 2195 `vision_encoder.*` tensors, accounted for by number and loaded by nothing. The MoE ViT and its pyramid schedule are W6 | +| dots3-note vision tower | `model-vision.safetensors` | 13,742,557,056 bytes | `dots-studio/dots3-note-prev` @ `1e1e7b0cd37a3a48a6c8d7fa55d5f9d14377006b` | Owed, as above | none | **W6a ports the DENSE half; this checkpoint's tower still REFUSES BY NAME.** Of its 2195 `vision_encoder.*` tensors, 235 are the dense arm W6a loads β€” `patch_embed`, blocks 0-24 and the `patch_merger` adapter β€” and 1960 are the PYRAMID MoE half (17 blocks, 608 routed experts) that belongs to W6b. Because 17 of the 42 blocks are routed, a load of THIS checkpoint refuses at block 25 naming W6b, the tensors stay a named deferral, and a served image request is answered HTTP 400 at the entrypoint rather than taking the engine down. A checkpoint whose vision blocks are all dense is served end to end | | dots3-note audio tower | `model-audio.safetensors` | 1,772,399,360 bytes | `dots-studio/dots3-note-prev` @ `1e1e7b0cd37a3a48a6c8d7fa55d5f9d14377006b` | Owed, as above | none | **Refused as a NAMED W7 deferral**: 430 `audio_encoder.*` tensors. The `dots` Whisper-variant stem is W7 | | dots3-note blockwise-FP8 sibling | `model-000{01..131}-of-00131.safetensors` plus the two tower files | 298,673,280,504 bytes total (278.16 GiB) across 133 safetensors, read 2026-08-28 | `dots-studio/dots3-note-prev-fp8` @ `7c14222e22423d6df6848eb0d1c5c3a88a00311a` | Owed: only `config.json` and `model.safetensors.index.json` were read | none | **Refused BY NAME at the forward, naming W9.** Its `quantization_config` is `{"quant_method": "fp8", "fmt": "e4m3", "activation_scheme": "dynamic", "weight_block_size": [128, 128]}` and its index (73,029 entries) ships a `weight_scale_inv` beside every projection β€” at the routed experts' `[1536, 5120]` that scale is `[12, 40]`. This port's bf16 loaders read a per-tensor or per-output-ROW `_scale` and nothing else, so without the named refusal the load would fail with a bare "tensor not found". It does not fit either: 278.16 GiB against the same 122 GiB ceiling | | Qwen3.8-Flash-Next GGUF | `Qwen3.8-Flash-Next-UD-IQ1_S-0000{1..3}-of-00003.gguf` | 72,546,461,344 bytes total (67.564 GiB) across three shards (10,946,624 + 49,990,818,368 + 22,544,696,352); 1224 tensors | `unsloth/Qwen3.8-Flash-Next-GGUF` @ `8bdc666649440e9bdc97e16f3f75782c98478ff5`, path `UD-IQ1_S` | `88a1420825a9304063e882ada29d438263617f51ac8923d438d927496693bafd` (shard 1); `3a62e35bbf9add4733bd1438ebd3a67649d5edd6cb0e72bb78e33c913992b2b6` (shard 2); `0e25ceaeb89b8a80aa973c6c0c7448943682f7408c2855b2ebd016b7643a861a` (shard 3). Shard 1's digest was recomputed TWICE for this row -- on the development box and again INSIDE the `thor` lease against the bytes the server actually opened. Shards 2 and 3 carry the digests recorded in [the ladder-arm evidence file](bench-evidence/qwen4exp-llamacpp-ladder-arm-20260829.md), which recomputed all three on the staged copy on 29 August 2026; **this wave did not re-derive those two**, because the hash was killed mid-run for reading the same CIFS share as the load being measured | **LOADS on `--device cpu`, and the server LISTENS -- it produces NO TOKEN.** Measured on `thor:gpu0` 2026-08-30 (`rc` job `0f188dd1`, [evidence](bench-evidence/qwen4exp-released-checkpoint-serve-20260830.md)): all three shards load through `LoadedEngine::FromModelDir`, the engine sizes all three published cache groups, the tokenizer and the 9993-character chat template come out of the GGUF's own metadata, and `examples/server` answers on `/health`. **Load wall time 4446 s (74.1 min); peak RSS `VmHWM` 69.206 GiB against a 67.564 GiB artifact.** Residency is keep-quant: anonymous memory moved 4 -> 11 GiB across a load whose n-gram table alone would have added 95.368 GiB there, so all nine encodings in the file (F32, Q8_0, Q4_K, Q5_K, Q6_K, IQ2_XXS, IQ1_S, IQ4_NL, BF16) keep their blocks. `POST /v1/completions` then returns **500** and zero tokens | **THE FORWARD REFUSED THIS ARTIFACT BY NAME ON THAT RUN, AND W5p REMOVED THE REFUSAL**: `vt: qwen4_exp_gated_residual: input_mix_weight_down must be float (f32/bf16 for outputs)`. The file stores all **194** hyper-connection mix weights (`blk.N.hc_{attn,ffn}_{down,up}.weight` and `output_hc_{down,up}.weight`) as **Q8_0**; our loader correctly keeps them quantized (`qwen4_exp_weights.cpp` -> `LoadMatmul`), and `vt::Qwen4ExpGatedResidual` accepted only float, while every arm of the synthetic fixture wrote those same names as ggml type 0 (F32) -- so every prior wave gated the float case only and none could see this. Since W5p the three PROJECTION operands (`mix_down`, `mix_up`, `block_inject`) accept a block-quantized `[N,K]` weight and route through `vt::MatmulBT`/`kMatmulBTQuant`, mirroring llama.cpp, which merged this architecture on 2026-08-27 (`6c84c7d5d`, first tag `b10660`) and declares all six of them `GGML_OP_MUL_MAT`; the ELEMENTWISE `hc_*_norm` gamma is still refused by name, which is llama.cpp's own split. `FixtureOpts::hc_mix_q8_0` is the fixture arm that was missing. **W5q RE-RAN THIS ARTIFACT ON 2026-08-31** ([evidence](bench-evidence/qwen4exp-released-checkpoint-serve-20260831.md)): staged to worker-local disk it loads in **61 s** rather than 4446 s, `VmHWM` 73.935 GiB, the prefill and eight decode steps complete with nothing thrown, and `POST /v1/completions` returns **200** with 8 tokens. **Every token was id 0 (`!`) and two different prompts returned a byte-identical answer.** **W5s RE-RAN IT ON 2026-08-31 ON `origin/main` `52f7ccbfc`, WHICH CARRIES W5r, AND THE TOKENS ARE REAL** ([evidence](bench-evidence/qwen4exp-released-checkpoint-tokens-20260831.md)): `"The capital of France is"` -> `" Paris. Given this fact, what is"` and `"Water boils at"` -> `" 100Β°C at sea level"`, eight distinct token ids none of them 0, loaded in 60 s from the staged copy at `VmHWM` 73.93 GiB with system `used` flat at 11 GiB. **The cause of W5q's degeneracy was the dropped repack marker W5r fixed**: on this aarch64 i8mm box `kMatmulBTQuant` had been reading `block_q8_0x4` buffers as flat `q8_0`, putting a NaN in layer 0 that collapsed to an all-zero logit row, and `argmax` over a row with no maximum returns index 0. `VT_CPU_QUANT_REPACK=0` now gives byte-identical output to the default. **WHAT RUNS IS EXACTLY THIS AND NO MORE: `--device cpu`, ONE SEQUENCE AT A TIME, the UD-IQ1_S arm.** It is **NOT a token gate** β€” no oracle decoded these prompts, and there is no speed number. ISSUE OWED (this account is suspended for GitHub **API** writes -- `gh issue create` returns `HTTP 403: Sorry. Your account was suspended`, while `git push` over SSH succeeds, which is how this row reached `main`); scoped under `## Owed` in [the spec](../.agents/specs/qwen4-exp-flash-next.md). **Also refused or absent:** the other six published quants (UD-IQ1_M, UD-Q2_K_XL, UD-IQ3_XXS, UD-Q3_K_XL, UD-IQ4_XS, UD-Q4_K_XL) are staged but **none has been run**; every safetensors artifact (~360 GB bf16, ~180 GB FP8, ~128 GB NVFP4) exceeds the 122.80 GiB of the largest box in this fleet; any non-CPU device refuses by name ahead of tensor I/O, because `DeviceQuantGatherSupported` is true for `kCPU` alone and the n-gram table would expand from 26.822 GiB to 95.368 GiB ([#2083](https://github.com/mudler/vllm.cpp/issues/2083)), and although W6-CUDA and W6-CUDA-B between them gave ALL SIX `qwen4_exp` ops plus `vt::RmsNormGroup` a CUDA arm, the block-decoding n-gram gather still has none (`EmbeddingKernelCuda` refuses a block-quantized table by name) and `ModelRegistry::Forward` is all-or-nothing, so no `qwen4_exp` step reaches a CUDA queue; `num_reqs > 1` is refused by name; MTP is absent (**zero** `nextn`/`mtp` tensors of 1224 against 31 in the safetensors repo, [#1993](https://github.com/mudler/vllm.cpp/issues/1993)); and the file is TEXT-ONLY (no `v.blk.*`), so the multimodal arm has no artifact | diff --git a/include/vllm/multimodal/dots3_note_processor.h b/include/vllm/multimodal/dots3_note_processor.h new file mode 100644 index 000000000..f6155e008 --- /dev/null +++ b/include/vllm/multimodal/dots3_note_processor.h @@ -0,0 +1,120 @@ +// dots3-note IMAGE processor (W6a, #2512). +// +// Ported from `vllm/models/dots3_note/common/processor.py` read in `~/_git/vllm` +// at **`9035151d6`** β€” the merge of vllm#51255. `dots3_note` does not exist at +// our parity pin `5559679229bc961848b121ccdeaa8fa5d79bec98`, so every anchor +// here names that SHA; upstream has already moved under this row once. +// +// IMAGE_START / IMAGE_PAD / IMAGE_END :41-43 +// Dots3NoteImageProcessor.__init__ :63-79 +// .factor :83-84 +// ._round_by_factor / _ceil / _floor :86-96 +// .resized_size :97-146 +// .preprocess :147-218 +// +// WHY THIS IS NOT `Qwen3VLImageProcessor` WITH DIFFERENT NUMBERS. Three things +// differ and each is silent when wrong: +// +// 1. `resized_size` is NOT `smart_resize`. Upstream dots3 rounds each side +// INDEPENDENTLY to a multiple of `factor` and only then applies the pixel +// budget (`processed.py:139-146`), where `smart_resize` +// (transformers `image_processing_qwen2_vl.py:62`) does the same rounding +// but with different guards and a different min-pixel branch. The two +// agree on many images and disagree on some, and a disagreement moves the +// GRID, which moves the placeholder count, which changes the prompt. +// 2. `image_mean` / `image_std` are PER-CHANNEL lists, not the single scalar +// Qwen3-VL's 0.5/0.5 collapses to. +// 3. The patch row order is selected by `pre_pixel_shuffle`. TRUE is the +// 2x2-grouped order (which happens to be byte-identical to Qwen3-VL's +// merge-grouped patchify); FALSE is plain row-major. The released +// checkpoint sets TRUE, and the tower's RoPE position builder reads the +// SAME flag β€” so a processor and a tower that disagree on it produce a +// well-shaped, wrong answer. +// +// The placeholder EXPANSION is shared rather than re-written: +// `multimodal::ExpandImagePlaceholders` already takes the image token id, the +// merge size and the grids, and dots3's rule is upstream's same +// `grid.prod(-1) // merge_size**2` (`multimodal.py:151-155` @ `9035151d6`). +#ifndef VLLM_MULTIMODAL_DOTS3_NOTE_PROCESSOR_H_ +#define VLLM_MULTIMODAL_DOTS3_NOTE_PROCESSOR_H_ + +#include +#include +#include +#include + +#include "vllm/multimodal/inputs.h" + +namespace vllm::multimodal { + +// The subset of `preprocessor_config.json` + `config.json` the image path +// needs. Defaults are upstream's own where upstream has one; the three token +// ids have none upstream (they come from `added_tokens.json`, read at +// `multimodal.py:82-90` @ `9035151d6`) and are therefore REQUIRED by the +// loader below rather than defaulted to a number this port invented. +struct Dots3NoteProcessorConfig { + int patch_size = 14; + int temporal_patch_size = 1; + int merge_size = 2; // == vision_config.spatial_merge_size + bool pre_pixel_shuffle = true; + // Per channel, in the checkpoint's own order (R, G, B). + std::array image_mean{0.5, 0.5, 0.5}; + std::array image_std{0.5, 0.5, 0.5}; + double rescale_factor = 1.0 / 255.0; + int64_t min_pixels = 3136; + int64_t max_pixels = 12845056; + + // `<|img|>` / `<|imgpad|>` / `<|endofimg|>` in the checkpoint's tokenizer. + int32_t image_token_id = -1; + int32_t image_start_token_id = -1; + int32_t image_end_token_id = -1; + + std::string model_id = "dots-studio/dots3-note-prev"; // for the mm-hash +}; + +// Load from the two HF json documents. THROWS BY NAME when the three image +// token ids cannot be resolved: a processor that guessed them would inject a +// marker the tokenizer maps to something else, and the request would be served +// as text with the image dropped. +Dots3NoteProcessorConfig LoadDots3NoteProcessorConfig( + const std::string& preprocessor_config_json_path, + const std::string& config_json_path, const std::string& model_id); + +// `Dots3NoteImageProcessor.resized_size` (`common/processor.py:97-146` @ +// `9035151d6`) β€” the height/width both divisible by `factor` whose product lies +// in `[min_pixels, max_pixels]`. Throws upstream's two refusals: a side under +// `factor / 4`, and an aspect ratio over 200. +std::array Dots3NoteResizedSize(int64_t height, int64_t width, + int64_t factor, int64_t min_pixels, + int64_t max_pixels); + +class Dots3NoteImageProcessor { + public: + explicit Dots3NoteImageProcessor(Dots3NoteProcessorConfig cfg) + : cfg_(std::move(cfg)) {} + + const Dots3NoteProcessorConfig& config() const { return cfg_; } + + int64_t factor() const { + return static_cast(cfg_.patch_size) * cfg_.merge_size; + } + + // Preprocess ONE RGB image (HWC uint8, height*width*3) into + // `pixel_values [num_patches, channel*temporal*patch*patch]` + + // `image_grid_thw`. A genuine bicubic RESIZE is a NAMED residual, exactly as + // it is on the Qwen3-VL processor beside this one: an image whose dimensions + // `Dots3NoteResizedSize` would change is REFUSED with both sizes in the + // message rather than silently patchified at the wrong grid. + ImageKwargs ProcessImage(const uint8_t* rgb, int64_t height, + int64_t width) const; + + std::string HashImage(const uint8_t* rgb, int64_t height, + int64_t width) const; + + private: + Dots3NoteProcessorConfig cfg_; +}; + +} // namespace vllm::multimodal + +#endif // VLLM_MULTIMODAL_DOTS3_NOTE_PROCESSOR_H_ diff --git a/src/vllm/entrypoints/openai/mm_chat_dots3note.cpp b/src/vllm/entrypoints/openai/mm_chat_dots3note.cpp new file mode 100644 index 000000000..631d122c8 --- /dev/null +++ b/src/vllm/entrypoints/openai/mm_chat_dots3note.cpp @@ -0,0 +1,272 @@ +// dots3-note's multimodal CHAT seam, registered on its own architecture +// (W6a, #2512). +// +// This is the SECOND architecture to reach the registry #2481 built, and it is +// what that row was for: adding a multimodal model is a NEW translation unit +// plus one `REGISTER_VLLM_MM_CHAT` line, with ZERO edits to any shared table. +// Before #2481 the server decided whether it could serve images by asking +// whether `/preprocessor_config.json` existed and then built +// Qwen3-VL's processor unconditionally β€” so this model would have been served +// Qwen3-VL's patch geometry, its merge size and its token ids, against its own +// `vision_config`, and answered 200. +// +// Ported from vLLM read in `~/_git/vllm` at **`9035151d6`**: +// `Dots3NoteForCausalLM.get_placeholder_str` (`nvidia/multimodal.py:65-72`) +// image -> f"{IMAGE_START}{IMAGE_PAD}{IMAGE_END}" +// `IMAGE_START` / `IMAGE_PAD` / `IMAGE_END` (`common/processor.py:41-43`) +// "<|img|>" / "<|imgpad|>" / "<|endofimg|>" +// `_process_image_input` (`nvidia/multimodal.py:144-155`) +// the placeholder run is `grid.prod(-1) // merge_size**2` +// `MULTIMODAL_REGISTRY.register_processor` (`nvidia/multimodal.py:44-48`) +// the registration sits ON THE MODEL and edits no shared table β€” the +// mechanism this file mirrors. +// +// WHY THE MARKER IS BUILT HERE AND NOT TAKEN FROM `chat_mm.h`. That header's +// `ImagePlaceholderString()` returns Qwen3-VL's +// "<|vision_start|><|image_pad|><|vision_end|>" (`qwen3_vl.py:1716`), and +// `BuildMarkerInjectedContent` dispatches through it. Those are one +// architecture's markers, which is exactly the coupling #2475 removed from the +// install path; reaching for them here would put it back one layer down. What +// IS shared is everything that is not per-architecture: `ValidateChatMmLimits` +// (the per-item limit walk), `DecodeImageUrlPart` (the data-URI decode), +// `BaseProcessingInfo` (the `--limit-mm-per-prompt` fold) and +// `multimodal::ExpandImagePlaceholders` (the expansion rule, which is upstream's +// same `prod(grid) // merge**2` for both models). +#include +#include +#include +#include +#include +#include +#include +#include + +#include "vllm/entrypoints/openai/chat_mm.h" +#include "vllm/entrypoints/openai/mm_chat_registry.h" +#include "vllm/multimodal/dots3_note_processor.h" +#include "vllm/multimodal/processing/context.h" +#include "vllm/model_executor/models/dots3_note_vision.h" // the tower refusal +#include "vllm/multimodal/qwen3vl_processor.h" // ExpandImagePlaceholders +#include "vllm/tokenizer/tokenizer.h" +#include "vllm/transformers_utils/hf_config.h" + +namespace vllm::entrypoints::openai { + +namespace { + +using vllm::Dots3NoteVisionRefusalFor; +using vllm::LoadHfConfig; + +namespace fs = std::filesystem; + +// TU-local, matching `mm_chat_qwen3vl.cpp:38-56`. A checkpoint path arrives as +// UTF-8 and `fs::path` is `wchar_t`-based on Windows. +fs::path NativeUtf8Path(const std::string& value) { +#if defined(_WIN32) + const std::u8string utf8(reinterpret_cast(value.data()), + value.size()); + return fs::path(utf8); +#else + return fs::path(value); +#endif +} + +std::string PathUtf8(const fs::path& path) { +#if defined(_WIN32) + const std::u8string utf8 = path.u8string(); + return std::string(reinterpret_cast(utf8.data()), utf8.size()); +#else + return path.string(); +#endif +} + +// `get_placeholder_str` (`nvidia/multimodal.py:65-68` @ `9035151d6`), the image +// branch. The SINGLE `<|imgpad|>` in the middle is what the tokenizer maps to +// ONE `image_token_id`, which `ExpandImagePlaceholders` then expands to N. +std::string Dots3NoteImageMarker() { + return "<|img|><|imgpad|><|endofimg|>"; +} + +// `BuildMarkerInjectedContent`'s dots3 twin: walk the parts IN ORDER, append +// each text part's text and each image part's marker at its position. A +// bare-string message is returned unchanged, so the text path is byte-identical. +std::string BuildDots3NoteMarkerContent(const ChatMessage& message) { + if (!message.content_parts.has_value()) { + return message.content.has_value() ? *message.content : std::string(); + } + std::string out; + for (const ChatContentPart& part : *message.content_parts) { + if (part.type == "text") { + out += part.text; + } else if (part.type == "image_url") { + out += Dots3NoteImageMarker(); + } + // Every other part type contributes nothing here and is refused earlier by + // ValidateChatMmLimits, whose supported-limit map below declares only + // "image". + } + return out; +} + +// This seam's OWN ceiling β€” the other operand of the `min()` fold +// (`context.py:392-405`). Upstream's `Dots3NoteProcessingInfo` handles N images +// and video and audio; THIS seam locates exactly one image part and has no +// video or audio arm at all, so the honest ceiling is `{"image": 1}` and every +// other modality is ABSENT, which `context.py:414-415` reads as limit 0. A user +// limit can only LOWER it, so `--limit-mm-per-prompt image=99` still refuses the +// second image. Video is W7's and audio is W8's; when they land they raise +// these numbers here and nothing else changes. +std::map> Dots3NoteChatSupportedMmLimits() { + return {{"image", std::optional(1)}}; +} + +// `RouteImageRgb`'s dots3 twin (`chat_mm.cpp:162-187`): preprocess, expand the +// single placeholder id to `prod(grid)/merge^2` copies, and build the +// `mm_features` handle the engine's multimodal generate overload carries onto +// `Request.mm_features`. +multimodal::MultiModalInputs RouteDots3NoteImageRgb( + const multimodal::Dots3NoteImageProcessor& proc, const uint8_t* rgb, + int64_t height, int64_t width, const std::vector& prompt_ids) { + const multimodal::Dots3NoteProcessorConfig& cfg = proc.config(); + multimodal::ImageKwargs kw = proc.ProcessImage(rgb, height, width); + const std::array grid = kw.image_grid_thw; + + std::vector> grids{grid}; + std::vector> placeholders; + std::vector expanded = multimodal::ExpandImagePlaceholders( + prompt_ids, cfg.image_token_id, cfg.merge_size, grids, &placeholders); + + multimodal::MultiModalInputs out; + out.prompt_token_ids = std::move(expanded); + if (!placeholders.empty()) { + multimodal::MultiModalFeatureSpec spec; + spec.modality = "image"; + spec.offset = placeholders[0][0]; + spec.length = placeholders[0][1]; + spec.mm_hash = proc.HashImage(rgb, height, width); + spec.data = std::make_shared(std::move(kw)); + out.mm_features.push_back(std::move(spec)); + } + return out; +} + +MultiModalChatFn MakeDots3NoteImageChatFn( + std::shared_ptr proc, + const vllm::tok::Tokenizer& tokenizer, ChatPromptRenderFn prompt_fn, + ImageCodecFn codec, + std::shared_ptr info) { + return [proc, info, &tokenizer, prompt_fn = std::move(prompt_fn), + codec = std::move(codec)](const std::vector& messages) + -> std::optional { + // STEP 0: the per-item limit check, BEFORE anything is decoded or dropped + // (`chat_utils.py:662` validates as it tracks, for the same reason). + ValidateChatMmLimits(*info, messages); + + const ChatContentPart* image_part = nullptr; + for (const ChatMessage& m : messages) { + if (!m.content_parts.has_value()) continue; + for (const ChatContentPart& part : *m.content_parts) { + if (part.type == "image_url") { + image_part = ∂ + break; + } + } + if (image_part != nullptr) break; + } + if (image_part == nullptr) return std::nullopt; // the text path, untouched + + // 1. Inject dots3-note's OWN marker at each image part's position and + // render the templated prompt. + std::vector rendered = messages; + for (ChatMessage& m : rendered) { + if (m.content_parts.has_value()) { + m.content = BuildDots3NoteMarkerContent(m); + m.content_parts.reset(); + } + } + const std::string prompt = + prompt_fn(rendered, /*add_generation_prompt=*/true, {}, + nlohmann::ordered_json::object()); + + // 2. Tokenize WITH special tokens: the single `<|imgpad|>` becomes ONE + // `image_token_id` (added tokens match leftmost-longest). + const std::vector prompt_ids = + tokenizer.EncodeWithSpecialTokens(prompt); + + // 3. Decode + route: expand that id to N and build the mm_features. + const DecodedMedia media = DecodeImageUrlPart(*image_part); + const DecodedImageRgb img = codec(media); + return RouteDots3NoteImageRgb(*proc, img.rgb.data(), img.height, img.width, + prompt_ids); + }; +} + +MultiModalChatSeam MakeDots3NoteChatSeam(const MultiModalChatContext& ctx) { + if (ctx.tokenizer == nullptr || ctx.mm_config == nullptr || !ctx.prompt_fn || + !ctx.codec) { + // Refuse by name rather than dereference. The install's catch turns this + // into a REFUSING seam, which is an HTTP 400 naming the architecture β€” never + // a silent text answer. + throw std::runtime_error( + "dots3-note multimodal chat seam: the install context is incomplete " + "(tokenizer, multimodal config, chat-prompt renderer and image codec " + "are all required)"); + } + + const std::string preprocessor_config_path = + PathUtf8(NativeUtf8Path(ctx.model_dir) / "preprocessor_config.json"); + if (!fs::exists(NativeUtf8Path(preprocessor_config_path))) { + throw std::runtime_error( + "dots3-note multimodal chat seam: '" + preprocessor_config_path + + "' is missing; the image processor's patch/merge geometry, its " + "per-channel normalization and its pixel bounds are read from it"); + } + + // THE TOWER'S OWN REFUSAL, ASKED HERE AND NOT IN THE ENGINE. The DENSE arm is + // W6a's; a checkpoint whose vision blocks are routed belongs to W6b, and the + // RELEASED `dots-studio/dots3-note-prev` is one β€” 17 of its 42 blocks are. + // + // Asking at INSTALL is not a preference. `EncodeMmDots3NoteForCausalLM` + // refuses too, but it runs inside the engine's busy loop: throwing there stops + // `AsyncLLM` and turns every LATER request, TEXT ONES INCLUDED, into a 500. + // That was measured on this row's served-request gate before this check + // existed. Refusing here installs a REFUSING seam instead, which is upstream's + // own shape for "this server does not accept images for this model": HTTP 400 + // naming the architecture and the reason, with the text path untouched. The + // encoder's check stays as defence in depth, on the same polarity Qwen3-VL's + // carries ("reaching this point is a defect"). + { + const std::string why = + Dots3NoteVisionRefusalFor(LoadHfConfig(ctx.config_path)); + if (!why.empty()) { + throw std::runtime_error( + "dots3-note multimodal chat seam: this checkpoint's vision tower is " + "not ported β€” " + why + + ". See .agents/specs/dots3-note.md Β§4.11 and issue #2512."); + } + } + + auto proc = std::make_shared( + multimodal::LoadDots3NoteProcessorConfig( + preprocessor_config_path, ctx.config_path, ctx.served_model_name)); + // The engine's limits (`--limit-mm-per-prompt`, `--language-model-only`) + // folded by min() against this seam's own ceiling. The MultiModalConfig is + // held BY REFERENCE (`context.h:105`); the engine owns it and outlives the + // seam. + auto info = std::make_shared( + *ctx.mm_config, Dots3NoteChatSupportedMmLimits()); + + MultiModalChatSeam seam; + seam.allowed_limits = info->AllowedMmLimits(); + seam.detail = "dots3-note processor from " + preprocessor_config_path + + " (DENSE vision arm; the pyramid MoE ViT is W6b)"; + seam.chat_fn = MakeDots3NoteImageChatFn(proc, *ctx.tokenizer, ctx.prompt_fn, + ctx.codec, info); + return seam; +} + +} // namespace + +REGISTER_VLLM_MM_CHAT(dots3_note, "Dots3NoteForCausalLM", &MakeDots3NoteChatSeam) + +} // namespace vllm::entrypoints::openai diff --git a/src/vllm/model_executor/models/dots3_note.cpp b/src/vllm/model_executor/models/dots3_note.cpp index 650c52c9f..b200d026d 100644 --- a/src/vllm/model_executor/models/dots3_note.cpp +++ b/src/vllm/model_executor/models/dots3_note.cpp @@ -585,8 +585,16 @@ const std::vector& Dots3NoteDeferredTowers() { // revision 1e1e7b0cd37a3a48a6c8d7fa55d5f9d14377006b: each tower ships whole // in one standalone file rather than across the 131 numbered language shards. static const std::vector kTowers{ - {"vision_encoder.", "model-vision.safetensors", "W6", - "the MoE ViT vision tower (nvidia/vision.py, nvidia/vision_moe.py)"}, + // W6a LANDED the DENSE half of this tower (#2512), so what is DEFERRED + // here is the PYRAMID half and the brick moved with it: `W6` -> `W6b`. + // The prefix did not move, and neither did the count. This row fires only + // when `Dots3NoteVisionRefusal` refused the tower β€” which for the + // RELEASED checkpoint it does, at block 25 β€” and in that case all 2195 + // `vision_encoder.*` names land here exactly as they did before, because + // a tower that is not loaded is not partly loaded. + {"vision_encoder.", "model-vision.safetensors", "W6b", + "the PYRAMID MoE half of the ViT vision tower (nvidia/vision.py's " + "MoESwiGLUFFN, nvidia/vision_moe.py); W6a ships the DENSE blocks"}, {"audio_encoder.", "model-audio.safetensors", "W7", "the `dots` Whisper-variant audio tower (nvidia/audio_encoder.py)"}, }; @@ -743,6 +751,25 @@ Dots3NoteWeights LoadDots3NoteWeights(const std::vector& shards w.device = MaterializeDots3NoteDevice(shards, w.params); w.materialized = w.device.present; } + + // W6a (#2512): the VISION tower, on the same polarity and for the same + // reason. `Dots3NoteVisionRefusal` is empty only for a tower whose every + // block is DENSE and whose adapter is `patch_merger`; for the RELEASED + // `dots-studio/dots3-note-prev` it is NOT, because 17 of its 42 vision blocks + // are pyramid MoE, so nothing below runs and its 2195 `vision_encoder.*` + // tensors stay in the accounting's `vision` deferral bucket exactly as they + // did before this brick. Every W2 count assertion is therefore unchanged. + // + // THE MESSAGE IS KEPT, not just the boolean. The encoder hook reports it + // verbatim, so an operator who sends an image to a checkpoint whose tower is + // owed is told WHICH block is routed and WHICH brick owes it, rather than + // that a tower is missing. + w.vision_params = ParseDots3NoteVisionParams(config); + w.vision_refusal = Dots3NoteVisionRefusal(w.vision_params, w.params.quant_method, + w.params.weight_block_size); + if (w.vision_refusal.empty()) { + w.vision = MaterializeDots3NoteVision(shards, w.vision_params); + } return w; } diff --git a/src/vllm/model_executor/models/dots3_note.h b/src/vllm/model_executor/models/dots3_note.h index 7b75fed05..1a469cd6e 100644 --- a/src/vllm/model_executor/models/dots3_note.h +++ b/src/vllm/model_executor/models/dots3_note.h @@ -45,6 +45,7 @@ #include #include +#include "vllm/model_executor/models/dots3_note_vision.h" // W6a: the DENSE ViT arm #include "vllm/model_executor/models/mla_attention.h" // MlaBlockDims / the seam #include "vllm/model_executor/models/model_registry.h" #include "vllm/model_executor/models/qwen3_5.h" // PagedKvCache, ForwardLogits @@ -490,6 +491,22 @@ struct Dots3NoteWeights { Dots3NoteAccounting accounting; bool materialized = false; Dots3NoteDeviceWeights device; + + // W6a (#2512) β€” the DENSE vision tower. `vision.present` is TRUE only when + // the config HAS a `vision_config` AND `Dots3NoteVisionRefusal` accepted it, + // which for the RELEASED `dots-studio/dots3-note-prev` it does NOT: 17 of its + // 42 vision blocks are pyramid MoE and belong to W6b. `vision_refusal` keeps + // that message so the encoder hook can report WHY rather than only that the + // tower is absent β€” a refusal a caller can only see as a missing pointer is + // one nobody can act on. + // + // The two are kept BESIDE the language tower rather than folded into + // `device`, because they have different lifetimes: a config can have a + // runnable language tower and a refused vision tower (that is the released + // checkpoint) and the reverse is a defect, not a state. + Dots3NoteVisionParams vision_params; + Dots3NoteVisionWeights vision; + std::string vision_refusal; }; // Why the DEVICE forward cannot run `params`, or "" when it can. The message @@ -594,12 +611,27 @@ Dots3NoteWeights LoadDots3NoteWeights(const std::vector& shards // refuses the shape now, so this cannot come back unseen. class Dots3NoteModel { public: + // `mm` is `ModelForwardInput::mm` (W6a, #2512). When it carries + // `inputs_embeds` the forward SKIPS the embedding lookup and starts from + // those rows, which is upstream's `inputs_embeds is not None` arm + // (`deepseek_v2.py`'s `DeepseekV2Model.forward`, mirrored for dots3 at + // `nvidia/model.py`) and the ONLY way the vision rows the runner merged can + // reach the residual stream. NULL for every text step, so the text path is + // byte-identical. + // + // dots3-note is NOT an M-RoPE model β€” upstream's `Dots3NoteForCausalLM` is + // `SupportsMultiModal, SupportsPP` and not `SupportsMRoPE` + // (`nvidia/multimodal.py:49` @ `9035151d6`) β€” so `mm->positions3` is never + // read and `kDots3NoteFactory` leaves `mrope_prompt_positions` null. The + // runner then hands the ordinary 1-D `positions`, which is what this forward + // has always consumed. static ForwardLogits ForwardDevice( const std::vector& token_ids, const std::vector& positions, const v1::CommonAttentionMetadata& attn_meta, const std::vector& attn_kv, const Dots3NoteWeights& weights, - vt::Queue& queue, const std::vector& logits_indices); + vt::Queue& queue, const std::vector& logits_indices, + const MultiModalForwardInput* mm = nullptr); }; // The MLA KV topology. Both classes share ONE physical row of diff --git a/src/vllm/model_executor/models/dots3_note_device.cpp b/src/vllm/model_executor/models/dots3_note_device.cpp index c4d68c0ee..21ff56e7a 100644 --- a/src/vllm/model_executor/models/dots3_note_device.cpp +++ b/src/vllm/model_executor/models/dots3_note_device.cpp @@ -1079,7 +1079,8 @@ ForwardLogits Dots3NoteModel::ForwardDevice( const std::vector& token_ids, const std::vector& positions, const v1::CommonAttentionMetadata& attn_meta, const std::vector& attn_kv, const Dots3NoteWeights& weights, - vt::Queue& queue, const std::vector& logits_indices) { + vt::Queue& queue, const std::vector& logits_indices, + const MultiModalForwardInput* mm) { const Dots3NoteParams& p = weights.params; // The scope boundary, stated as a refusal rather than as a comment. Every // branch names ONE unrepresentable feature and the brick that owes it; the @@ -1192,8 +1193,35 @@ ForwardLogits Dots3NoteModel::ForwardDevice( "dots3-note forward: one KV cache per backbone layer is required"); // ── embed ───────────────────────────────────────────────────────────────── + // + // W6a (#2512): `inputs_embeds is not None` is upstream's own arm, and it is + // the ONLY way the vision rows reach the residual stream. The runner ran the + // encoder, gathered its rows and asked the model's `embed_mm` hook to scatter + // them over the placeholder positions; what arrives here is the RESULT of + // that scatter, so the embedding-table lookup must be SKIPPED rather than run + // and overwritten. Running it anyway would be invisible on a text step and + // would silently discard every vision row on an image step. DBuf hidden_buf(d, DType::kBF16, {T, H}); - { + const bool have_mm_embeds = + mm != nullptr && mm->inputs_embeds.data != nullptr; + if (have_mm_embeds) { + const Tensor& emb = mm->inputs_embeds; + VT_CHECK(emb.rank == 2 && emb.shape[0] == T && emb.shape[1] == H, + "dots3-note forward: ModelForwardInput.mm carries inputs_embeds of " + "rank " + std::to_string(emb.rank) + " shaped [" + + std::to_string(emb.shape[0]) + ", " + + std::to_string(emb.shape[1]) + "], expected [" + + std::to_string(T) + ", " + std::to_string(H) + + "]. A merged embedding stream that does not cover the step's " + "tokens would splice vision features onto the wrong rows."); + VT_CHECK(emb.dtype == DType::kBF16, + "dots3-note forward: mm.inputs_embeds is not BF16. The model dtype " + "IS bf16 here and a wider stream moves twice the bytes through " + "every layer while producing tokens a gate cannot tell apart " + "(porting.md's memory-format rule)."); + d.b.Copy(d.q, hidden_buf.ptr(), emb.data, + static_cast(T * H) * vt::SizeOf(DType::kBF16)); + } else { DBuf ids(d, DType::kI32, {T}, token_ids.data()); Tensor tab = ResidentWeight(d, dw.embed_tokens, {vocab, H}); Tensor h = hidden_buf.t(); diff --git a/src/vllm/model_executor/models/dots3_note_registry.cpp b/src/vllm/model_executor/models/dots3_note_registry.cpp index b0b6783a9..e4cbeebc7 100644 --- a/src/vllm/model_executor/models/dots3_note_registry.cpp +++ b/src/vllm/model_executor/models/dots3_note_registry.cpp @@ -31,11 +31,17 @@ #include #include +#include #include +#include +#include "vllm/model_executor/models/dense_attn_block.h" // MakeTensor #include "vllm/model_executor/models/dots3_note.h" +#include "vllm/model_executor/models/dots3_note_vision.h" #include "vllm/model_executor/models/qwen3_5.h" // ForwardLogits carrier +#include "vllm/multimodal/inputs.h" // MultiModalFeatureSpec #include "vllm/v1/kv_cache_interface.h" +#include "vt/dtype.h" namespace vllm { namespace { @@ -44,35 +50,46 @@ namespace { // tree's sense (both attention classes are attention over a paged MLA cache; // the sliding half is a window on the same cache, not a recurrent state). // -// ─── `supports_multimodal` IS FALSE, AND IT WAS TRUE UNTIL W5 ──────────────── +// ─── `supports_multimodal` IS TRUE AGAIN, AND THE TRAIL IS THE RECORD ──────── // W1 set it TRUE because upstream registers this architecture in // `_MULTIMODAL_MODELS` and `multimodal.py`::Dots3NoteForCausalLM -// .get_placeholder_str (:80-88, the three branches at :82-87) handles image, -// video AND audio. That is a true statement about UPSTREAM and it was harmless -// while the released config was refused at its first MoE layer: nothing could -// load, so nothing could read the flag and act on it. +// .get_placeholder_str handles image, video AND audio. That is a true statement +// about UPSTREAM, and it was harmless while the released config was refused at +// its first MoE layer: nothing could load, so nothing could read the flag. // // W5 and W5c made the released config loadable, and at that moment the flag -// became a claim about THIS port that this port cannot honour. There is no -// vision tower (W6), no audio tower (W7) and no multimodal front end at all -// (W8) β€” `EnumerateDots3NoteTensors` does not claim one tensor of either tower -// and `Dots3NoteDeferredTowers()` records all 2625 of them as deferrals. A -// registry entry is a support claim, not a code-coverage claim, and the same -// argument `deepseek_v2_registry.cpp` makes about V3. +// became a claim about THIS port that this port could not honour β€” no vision +// tower, no audio tower, no multimodal front end β€” so W5 set it FALSE. // -// MEASURED before flipping it: `supports_multimodal` has NO production reader -// anywhere in `src/`, `include/`, `examples/` or `scripts/` β€” every occurrence -// outside `include/vllm/model_executor/models/model_registry.h` is either a -// registration writing it or a test reading it. So the flip changes no -// behaviour today; what it changes is what the record SAYS while W6/W7/W8 are -// owed. W8 flips it back, and the true -> false -> true trail is the honest -// version of that history. +// W6a ([#2512](https://github.com/mudler/vllm.cpp/issues/2512)) BACKS IT. The +// DENSE vision tower is on a SERVED request: `encode_mm` and `embed_mm` below +// are non-null, which is what makes `ModelRegistry::SupportsMmInputs` true and +// what turns on the runner's whole multimodal arm; and +// `mm_chat_dots3note.cpp` registers this architecture's own chat seam through +// `REGISTER_VLLM_MM_CHAT`, which is what makes the SERVER able to build image +// features for it. A registry entry is a support claim, and this one now has +// both halves behind it. +// +// SAY THE OTHER HALF IN THE SAME BREATH. The RELEASED +// `dots-studio/dots3-note-prev` still REFUSES an image, by name, because 17 of +// its 42 vision blocks are pyramid MoE (W6b) and 1960 of its 2195 +// `vision_encoder.*` tensors belong to that brick. The flag says this +// ARCHITECTURE has a served multimodal path, which is now true; it has never +// meant that every checkpoint of it loads. VIDEO (W7) and AUDIO (W8) are +// refused by name in `EncodeMmDots3NoteForCausalLM`, and the chat seam declares +// a ceiling of exactly one IMAGE so a video part is refused at the entrypoint +// with upstream's own message. +// +// MEASURED, and still true: `supports_multimodal` has no production reader +// outside `model_registry.h` β€” every other occurrence is a registration writing +// it or a test reading it. What gates the capability is the two hooks, not the +// flag, which is why the scaffold gate asserts all three together. inline constexpr ModelInfo kDots3NoteInfo{ .is_text_generation_model = true, .is_pooling_model = false, .is_hybrid = false, .has_inner_state = false, - .supports_multimodal = false, + .supports_multimodal = true, .score_type = "bi-encoder", }; @@ -131,10 +148,209 @@ ForwardLogits ForwardDots3NoteForCausalLM(LoadedModel& model, // model from the ForwardDevice impl its hook delegates to, and a model with // no recognizable producer lands in the silently-exempt NONE bucket. This // shape reports REFUSE, and it is the signature W3 fills in. - return Dots3NoteModel::ForwardDevice(input.token_ids, input.positions, - input.attn_meta, input.attn_kv, - d3.weights(), input.queue, - input.logits_indices); + // `input.mm` (W6a, #2512). The runner sets it ONLY on a step in which some + // request carries multimodal features, and `Dots3NoteModel::ForwardDevice` + // reads `inputs_embeds` off it in place of the embedding lookup. A text step + // passes nullptr and is byte-identical to every step this row has ever run. + return Dots3NoteModel::ForwardDevice( + input.token_ids, input.positions, input.attn_meta, input.attn_kv, + d3.weights(), input.queue, input.logits_indices, + input.mm.has_value() ? &input.mm.value() : nullptr); +} + +// ─── W6a: the two ModelFactory multimodal hooks (#2512) ────────────────────── +// +// The split between them is upstream's own and is the runner's contract +// (`ENG-MM-INPUT-PIPELINE` P2, #2379). `encode_mm` is +// `_process_image_input` (`nvidia/multimodal.py:144-155` @ `9035151d6`): run +// the tower on ONE item and return its rows. `embed_mm` is +// `get_input_embeddings` + the masked scatter: the token lookup for the whole +// step, with the gathered encoder rows written over the placeholder positions. +// The runner owns the encoder CACHE and the gather between them; it never sees +// a tower. + +MmEncoderOutput EncodeMmDots3NoteForCausalLM( + LoadedModel& model, const HfConfig& config, vt::Queue& queue, + const multimodal::MultiModalFeatureSpec& item) { + auto& d3 = ModelAs(model, "Dots3NoteForCausalLM"); + const Dots3NoteWeights& w = d3.weights(); + + // THE REFUSAL CARRIES ITS REASON. `vision_refusal` is the message + // `Dots3NoteVisionRefusal` produced at load; for the RELEASED checkpoint it + // names the first pyramid MoE block and the brick that owes it. Reporting + // only "no tower" would tell an operator nothing they could act on. + VT_CHECK(w.vision.present, + "Dots3NoteForCausalLM encoder: this load carries no vision tower β€” " + + (w.vision_refusal.empty() + ? std::string("the loader did not materialize one") + : w.vision_refusal) + + ". See .agents/specs/dots3-note.md Β§4.11 and issue #2512."); + VT_CHECK(item.modality == "image", + "Dots3NoteForCausalLM encoder: modality '" + item.modality + + "' is not ported. IMAGE is (W6a); VIDEO needs the multi-frame " + "cu_seqlens builder and the video sampler and is W7; AUDIO has " + "no tower at all and is W8. Refused by name rather than served " + "from the image path. See issue #2512."); + VT_CHECK(item.data != nullptr && !item.data->empty(), + "Dots3NoteForCausalLM encoder: the multimodal item carries no " + "processed image features (MultiModalFeatureSpec::data)."); + + const Dots3NoteVisionParams& v = w.vision_params; + const int64_t width = v.adapter_out_dim; + // The tower lands in the TEXT hidden space, and a checkpoint whose adapter + // does not is one whose rows cannot be scattered into the prompt at all. + // + // DEFENCE IN DEPTH, and no longer the FIRST line. `Dots3NoteVisionRefusal` + // now makes this same comparison, and the one on the merge sizes below it, at + // INSTALL β€” because reaching either of them here throws inside the engine's + // busy loop, which stops `AsyncLLM` for the life of the process. Keep the two + // predicates identical: a check added here and not there re-opens that + // cascade through a narrower door (fresh review of #2523). + VT_CHECK(width == config.hidden_size, + "Dots3NoteForCausalLM encoder: the vision adapter emits " + + std::to_string(width) + "-wide rows but the text tower is " + + std::to_string(config.hidden_size) + + " wide (`adapter_out_dim`, vision.py:461 @ 9035151d6)"); + + vt::Backend& backend = vt::GetBackend(queue.device.type); + // THE TOWER. This call is the point of the brick: before it, nothing on this + // row ran anything on a served request. + const std::vector tower = Dots3NoteVisionForward( + item.data->pixel_values_bf16, item.data->image_grid_thw, w.vision, v, + backend); + const int64_t rows = + width > 0 ? static_cast(tower.size()) / width : 0; + VT_CHECK(rows > 0 && rows * width == static_cast(tower.size()), + "Dots3NoteForCausalLM encoder: the tower produced " + + std::to_string(tower.size()) + + " floats, which is not a whole number of " + + std::to_string(width) + "-wide rows"); + VT_CHECK(rows == static_cast(item.length), + "Dots3NoteForCausalLM encoder: the tower produced " + + std::to_string(rows) + " embedding rows for a placeholder span " + "of " + std::to_string(item.length) + + " tokens. The processor's placeholder expansion " + "(`prod(grid) // merge**2`) and the tower's adapter merge " + "disagree, and a masked scatter would then splice the wrong rows " + "into the prompt."); + + // Stored in the MODEL dtype. bf16 is not a compression here: the whole dense + // vision arm is bf16 on disk and every GEMM above ran in it, so a f32 store + // would double the encoder cache for values that carry no extra information + // (porting.md's memory-format rule, pointed the other way). + std::vector bits(tower.size()); + for (size_t i = 0; i < tower.size(); ++i) bits[i] = vt::F32ToBF16(tower[i]); + const size_t bytes = bits.size() * vt::SizeOf(vt::DType::kBF16); + void* p = backend.Alloc(bytes); + std::shared_ptr storage(p, [&backend](void* q) { backend.Free(q); }); + backend.Copy(queue, p, bits.data(), bytes); + backend.Synchronize(queue); + MmEncoderOutput out; + out.storage = std::move(storage); + out.embeds = dense_attn::MakeTensor(p, vt::DType::kBF16, queue.device, + {rows, width}); + return out; +} + +MmForwardBuffers EmbedMmDots3NoteForCausalLM(LoadedModel& model, + const HfConfig& config, + vt::Queue& queue, + const MmEmbedInputs& inputs) { + auto& d3 = ModelAs(model, "Dots3NoteForCausalLM"); + const Dots3NoteWeights& w = d3.weights(); + VT_CHECK(w.materialized && w.device.present, + "Dots3NoteForCausalLM embed: the language tower was not " + "materialized, so there is no embedding table to look tokens up in."); + VT_CHECK(inputs.token_ids != nullptr && inputs.is_mm_embed != nullptr && + inputs.mm_embeds != nullptr, + "Dots3NoteForCausalLM embed: the runner passed a null MmEmbedInputs " + "channel"); + const std::vector& token_ids = *inputs.token_ids; + const int64_t T = static_cast(token_ids.size()); + const int64_t H = config.hidden_size; + VT_CHECK(T > 0, "Dots3NoteForCausalLM embed: empty step"); + VT_CHECK(static_cast(inputs.is_mm_embed->size()) == T, + "Dots3NoteForCausalLM embed: is_mm_embed has " + + std::to_string(inputs.is_mm_embed->size()) + " entries for " + + std::to_string(T) + " tokens"); + + vt::Backend& backend = vt::GetBackend(queue.device.type); + dense_attn::Dev d{backend, queue}; + + // `embed_input_ids`, the token half: the plain lookup, identical to the one + // the text path runs. + std::vector merged(static_cast(T * H)); + { + dense_attn::DBuf ids(d, vt::DType::kI32, {T}, token_ids.data()); + dense_attn::DBuf emb(d, vt::DType::kBF16, {T, H}); + vt::Tensor table = dense_attn::ResidentWeight( + d, w.device.embed_tokens, {config.vocab_size, H}); + vt::Embedding(d.q, emb.t(), table, ids.t()); + emb.Download(d, merged.data()); + } + + // The gathered encoder rows, concatenated in mask order. + int64_t n_rows = 0; + for (const vt::Tensor& slice : *inputs.mm_embeds) { + VT_CHECK(slice.rank == 2 && slice.shape[1] == H, + "Dots3NoteForCausalLM embed: a gathered encoder slice is " + + std::to_string(slice.shape[1]) + " wide, expected " + + std::to_string(H)); + VT_CHECK(slice.dtype == vt::DType::kBF16, + "Dots3NoteForCausalLM embed: a gathered encoder slice is not " + "BF16, so the scatter below would write the wrong bytes."); + n_rows += slice.shape[0]; + } + int64_t n_masked = 0; + for (int64_t t = 0; t < T; ++t) + if ((*inputs.is_mm_embed)[static_cast(t)] != 0) ++n_masked; + VT_CHECK(n_rows == n_masked, + "Dots3NoteForCausalLM embed: " + std::to_string(n_rows) + + " gathered encoder rows for " + std::to_string(n_masked) + + " masked placeholder positions. A masked scatter that does not " + "balance splices vision features onto text rows."); + + // `merge_multimodal_embeddings`, the masked scatter. It is a pure COPY of + // bf16 rows over bf16 rows: no arithmetic, so nothing here rounds and the + // encoder's own output reaches the residual stream bit-for-bit. Qwen3-VL's + // hook beside this one goes through f32 because its DeepStack split needs to; + // this model has no DeepStack and pays nothing. + if (n_rows > 0) { + std::vector gathered(static_cast(n_rows * H)); + size_t offset = 0; + for (const vt::Tensor& slice : *inputs.mm_embeds) { + const size_t n = static_cast(slice.shape[0] * H); + backend.Copy(queue, gathered.data() + offset, slice.data, + n * vt::SizeOf(vt::DType::kBF16)); + offset += n; + } + backend.Synchronize(queue); + int64_t r = 0; + for (int64_t t = 0; t < T; ++t) { + if ((*inputs.is_mm_embed)[static_cast(t)] == 0) continue; + std::copy(gathered.begin() + static_cast(r * H), + gathered.begin() + static_cast((r + 1) * H), + merged.begin() + static_cast(t * H)); + ++r; + } + } + + MmForwardBuffers out; + const size_t bytes = merged.size() * vt::SizeOf(vt::DType::kBF16); + void* p = backend.Alloc(bytes); + out.storage.emplace_back(p, [&backend](void* q) { backend.Free(q); }); + backend.Copy(queue, p, merged.data(), bytes); + out.mm.inputs_embeds = + dense_attn::MakeTensor(p, vt::DType::kBF16, queue.device, {T, H}); + // NO `positions3`, deliberately. dots3-note is not an M-RoPE model + // (`nvidia/multimodal.py:49` @ `9035151d6`: `SupportsMultiModal, SupportsPP` + // and NOT `SupportsMRoPE`), so `mrope_prompt_positions` is null on the + // factory, the runner leaves `MmEmbedInputs::mrope_positions` empty, and the + // forward reads the ordinary 1-D `ModelForwardInput::positions`. Filling a + // 3-D field the forward never reads would be a claim about this model that + // upstream does not make. + backend.Synchronize(queue); + return out; } const ModelFactory kDots3NoteFactory{ @@ -143,6 +359,13 @@ const ModelFactory kDots3NoteFactory{ .prepare = &PrepareDots3NoteForCausalLM, .forward = &ForwardDots3NoteForCausalLM, .make_kv_cache = &MakeDots3NoteKVCache, + // W6a (#2512). Setting these two is what makes + // `ModelRegistry::SupportsMmInputs` true for this architecture, and the + // runner's whole multimodal arm hangs on that predicate β€” which is DERIVED + // from these pointers and never stored. `mrope_prompt_positions` stays + // NULL: upstream does not declare `SupportsMRoPE` for this model. + .encode_mm = &EncodeMmDots3NoteForCausalLM, + .embed_mm = &EmbedMmDots3NoteForCausalLM, .is_dense_model = false, }; diff --git a/src/vllm/model_executor/models/dots3_note_vision.cpp b/src/vllm/model_executor/models/dots3_note_vision.cpp new file mode 100644 index 000000000..e82783d66 --- /dev/null +++ b/src/vllm/model_executor/models/dots3_note_vision.cpp @@ -0,0 +1,762 @@ +// dots3-note VISION tower β€” the DENSE arm (W6a, #2512). See the header for the +// complete port map and for the ONE deliberate RMSNorm rounding difference. +// +// Every anchor in this file was read in `~/_git/vllm` at `9035151d6`. +#include "vllm/model_executor/models/dots3_note_vision.h" + +#include +#include +#include +#include +#include +#include +#include +#include + +#include "vllm/model_executor/layers/linear.h" // UnquantizedMlpGateUpMethod seam +#include "vllm/model_executor/model_loader/safetensors_reader.h" +#include "vllm/model_executor/models/dense_attn_block.h" +#include "vllm/model_executor/models/dense_weight_loaders.h" +#include "vllm/model_executor/models/dots3_note.h" +#include "vllm/transformers_utils/hf_config.h" +#include "vt/backend.h" +#include "vt/dtype.h" +#include "vt/ops.h" + +namespace vllm { + +using vt::DType; +using vt::Tensor; +using namespace dense_attn; // Dev / DBuf / MakeTensor / ResidentWeight + +namespace { + +using dense_loaders::LoadBf16Direct; +using dense_loaders::LoadMergedBf16RawNK; + +// ── config resolution ─────────────────────────────────────────────────────── +// +// The reader mirrors `dots3_note.cpp`'s: a MISSING key that upstream's +// `DotsMoEVitConfig.__init__` defaults gets that default; a key that is present +// and of the wrong TYPE refuses by name, because a silently-ignored value is +// the shape of every Β§4 trap on this row. +int64_t ReadIntOr(const nlohmann::json& j, const char* key, int64_t fallback) { + const auto it = j.find(key); + if (it == j.end() || it->is_null()) return fallback; + VT_CHECK(it->is_number_integer() || it->is_number_unsigned(), + std::string("dots3-note vision_config: '") + key + + "' must be an integer (DotsMoEVitConfig, vision.py:27-105 @ " + "9035151d6), got " + it->dump()); + return it->get(); +} + +double ReadNumOr(const nlohmann::json& j, const char* key, double fallback) { + const auto it = j.find(key); + if (it == j.end() || it->is_null()) return fallback; + VT_CHECK(it->is_number(), + std::string("dots3-note vision_config: '") + key + + "' must be a number, got " + it->dump()); + return it->get(); +} + +bool ReadBoolOr(const nlohmann::json& j, const char* key, bool fallback) { + const auto it = j.find(key); + if (it == j.end() || it->is_null()) return fallback; + VT_CHECK(it->is_boolean(), + std::string("dots3-note vision_config: '") + key + + "' must be a boolean, got " + it->dump()); + return it->get(); +} + +std::string ReadStrOr(const nlohmann::json& j, const char* key, + const std::string& fallback) { + const auto it = j.find(key); + if (it == j.end() || it->is_null()) return fallback; + VT_CHECK(it->is_string(), + std::string("dots3-note vision_config: '") + key + + "' must be a string, got " + it->dump()); + return it->get(); +} + +std::string ShapeOf(const std::vector& s) { + std::string out = "["; + for (size_t i = 0; i < s.size(); ++i) { + if (i != 0) out += ", "; + out += std::to_string(s[i]); + } + return out + "]"; +} + +void RequireVisionShape(const OwnedTensor& t, const std::string& name, + const std::vector& want) { + std::vector got(t.shape, t.shape + t.rank); + VT_CHECK(got == want, + "dots3-note vision tower: '" + name + "' ships " + ShapeOf(got) + + ", the config implies " + ShapeOf(want) + + ". Refusing rather than reading a differently-shaped weight. " + "See .agents/specs/dots3-note.md Β§4.11 and issue #2512."); + // THE MEMORY FORMAT, asserted rather than assumed (porting.md). Every dense + // vision tensor is BF16 in the released index; a widened one still passes + // every shape check and every token gate while moving twice the bytes, and + // this row's W2 F1 fixture row already proves a re-typed tensor fires. + VT_CHECK(t.dtype == DType::kBF16, + "dots3-note vision tower: '" + name + + "' is not BF16 after load. The released checkpoint carries the " + "whole dense vision arm in BF16; a wider store is a memory-format " + "defect a token gate cannot see (porting.md)."); +} + +} // namespace + +Dots3NoteVisionParams ParseDots3NoteVisionParams(const HfConfig& config) { + Dots3NoteVisionParams v; + const auto it = config.raw.find("vision_config"); + if (it == config.raw.end() || it->is_null()) return v; // present == false + VT_CHECK(it->is_object(), + "dots3-note: `vision_config` must be an object, got " + it->dump()); + const nlohmann::json& j = *it; + v.present = true; + + v.embed_dim = ReadIntOr(j, "embed_dim", 1536); + // From the LANGUAGE config, not from `j`. See the field's comment: this is + // the width `EncodeMmDots3NoteForCausalLM` compares `adapter_out_dim` + // against, and reading `vision_config`'s own copy of it instead would leave + // the refusal answering a different question from the route. + v.text_hidden_size = config.hidden_size; + v.intermediate_size = ReadIntOr(j, "intermediate_size", 4224); + v.moe_intermediate_size = ReadIntOr(j, "moe_intermediate_size", 2112); + v.num_hidden_layers = ReadIntOr(j, "num_hidden_layers", 42); + v.num_attention_heads = ReadIntOr(j, "num_attention_heads", 24); + v.num_channels = ReadIntOr(j, "num_channels", 3); + v.patch_size = ReadIntOr(j, "patch_size", 14); + v.spatial_merge_size = ReadIntOr(j, "spatial_merge_size", 2); + v.temporal_patch_size = ReadIntOr(j, "temporal_patch_size", 1); + v.rms_norm_eps = ReadNumOr(j, "rms_norm_eps", 1e-5); + v.use_bias = ReadBoolOr(j, "use_bias", false); + v.use_qk_norm = ReadBoolOr(j, "use_qk_norm", true); + v.is_causal = ReadBoolOr(j, "is_causal", false); + v.post_norm = ReadBoolOr(j, "post_norm", true); + v.pre_pixel_shuffle = ReadBoolOr(j, "pre_pixel_shuffle", false); + v.capacity_factor = ReadNumOr(j, "capacity_factor", 2.0); + v.router_scoring_func = ReadStrOr(j, "router_scoring_func", "sigmoid"); + v.router_scale = ReadNumOr(j, "router_scale", 1.0); + v.adapter_type = ReadStrOr(j, "adapter_type", "pixel_shuffle_mlp"); + v.adapter_in_dim = ReadIntOr(j, "adapter_in_dim", 1536); + v.adapter_out_dim = ReadIntOr(j, "adapter_out_dim", 2048); + v.adapter_merge_size = ReadIntOr(j, "adapter_merge_size", 2); + + const auto pyr = j.find("pyramid_num_routed"); + if (pyr != j.end() && !pyr->is_null()) { + VT_CHECK(pyr->is_array(), + "dots3-note vision_config: `pyramid_num_routed` must be a list " + "(vision.py:90 @ 9035151d6), got " + pyr->dump()); + for (const nlohmann::json& e : *pyr) { + VT_CHECK(e.is_number_integer(), + "dots3-note vision_config: `pyramid_num_routed` entries must be " + "integers, got " + e.dump()); + v.pyramid_num_routed.push_back(e.get()); + } + } + + // Upstream's own validation, mirrored: `adapter_type` is checked in the + // constructor and raises there (`vision.py:98-102`). Anything else is a + // config this port cannot represent AT ALL, so it refuses at PARSE rather + // than at load β€” an unknown adapter is not owed to a later brick. + VT_CHECK(v.adapter_type == "pixel_shuffle_mlp" || + v.adapter_type == "patch_merger", + "dots3-note vision_config: adapter_type must be 'pixel_shuffle_mlp' " + "or 'patch_merger' (vision.py:98-102 @ 9035151d6), got '" + + v.adapter_type + "'"); + + // Geometry that cannot be true of any dots3-note tower, checked where the key + // name is still in hand. + VT_CHECK(v.num_attention_heads > 0 && v.embed_dim > 0 && + v.embed_dim % v.num_attention_heads == 0, + "dots3-note vision_config: embed_dim " + + std::to_string(v.embed_dim) + + " is not a whole multiple of num_attention_heads " + + std::to_string(v.num_attention_heads)); + VT_CHECK(v.head_dim() % 2 == 0, + "dots3-note vision_config: head_dim " + + std::to_string(v.head_dim()) + + " is odd; the 2-D vision RoPE splits it into a height half and a " + "width half (vision.py:503-504 @ 9035151d6)"); + VT_CHECK(v.spatial_merge_size > 0 && v.adapter_merge_size > 0, + "dots3-note vision_config: spatial_merge_size and adapter_merge_size " + "must be positive"); + VT_CHECK(static_cast(v.pyramid_num_routed.size()) == 0 || + static_cast(v.pyramid_num_routed.size()) >= + v.num_hidden_layers, + "dots3-note vision_config: `pyramid_num_routed` has " + + std::to_string(v.pyramid_num_routed.size()) + " entries for " + + std::to_string(v.num_hidden_layers) + + " blocks. Upstream indexes it by layer number " + "(vision.py:346-350 @ 9035151d6), so a short list would make the " + "tail silently dense."); + return v; +} + +std::string Dots3NoteVisionRefusal( + const Dots3NoteVisionParams& v, const std::string& quant_method, + const std::vector& weight_block_size) { + if (!v.present) { + return "the checkpoint's `config.json` carries no `vision_config`, so this " + "load has no vision tower to build (multimodal.py:113-118 @ " + "9035151d6)"; + } + // ORDER IS BRICK ORDER, and the message names ONE thing: a reader is told + // what to build, not that something is missing. + if (!weight_block_size.empty() || quant_method == "fp8") { + return "the checkpoint is BLOCKWISE-QUANTIZED (`quantization_config" + ".weight_block_size`), and the vision tower's FP32-scale FP8 arm " + "(`MoESwiGLUFFNFP8`, vision.py:222-297 @ 9035151d6, and " + "nvidia/vision_moe.py's `note_vision_fused_moe_fp8`) is W9"; + } + const int64_t moe = v.num_moe_blocks(); + if (moe > 0) { + int64_t first = -1; + for (int64_t i = 0; i < v.num_hidden_layers; ++i) { + if (v.is_moe_block(i)) { first = i; break; } + } + return "vision block " + std::to_string(first) + " is a PYRAMID MoE block (" + + std::to_string(v.pyramid_num_routed[static_cast(first)]) + + " routed experts), and " + std::to_string(moe) + " of the tower's " + + std::to_string(v.num_hidden_layers) + + " blocks are. The MoE ViT β€” `mlp.gate_weight` + `mlp.router_bias`, " + "sigmoid scoring and the capacity-factor top-k (`MoESwiGLUFFN`, " + "vision.py:139-219 @ 9035151d6) β€” is W6b. W6a ships the DENSE arm " + "only, and refusing is what stops this port from serving a tower " + "whose pyramid it silently skipped on a row that has no oracle " + "(spec Β§6.4)"; + } + if (v.adapter_type != "patch_merger") { + return "`adapter_type` is '" + v.adapter_type + + "'. W6a implements `patch_merger` (`PatchMergerAdapter`, " + "vision.py:441-472 @ 9035151d6), which is what the released " + "`dots-studio/dots3-note-prev` selects. `pixel_shuffle_mlp` is a " + "DIFFERENT token order from the same pixels and a DIFFERENT state " + "dict (`proj.0`/`proj.1`/`proj.3` against `ln_q`/`mlp.0`/`mlp.2`), " + "so it is refused rather than mapped onto this one. W6b"; + } + if (!v.post_norm) { + return "`post_norm` is false, so upstream builds no `post_trunk_norm` " + "(vision.py:513-514 @ 9035151d6). Every published dots3-note tower " + "sets it true and W6a's arm assumes it. W6b"; + } + if (v.use_bias) { + return "`use_bias` is true, so `attn.qkv`, `attn.proj` and every " + "`mlp.fc*` would carry a bias the released checkpoint does not ship " + "(vision.py:143-144 of vision_attention.py @ 9035151d6). W6b"; + } + if (!v.use_qk_norm) { + return "`use_qk_norm` is false, so attention would run with no per-head " + "`q_norm`/`k_norm` (vision_attention.py:145-147 @ 9035151d6) while " + "the checkpoint ships both. W6b"; + } + if (v.is_causal) { + return "`is_causal` is true. The dots3-note ViT attends bidirectionally " + "(vision_attention.py:118 @ 9035151d6) and W6a's attention call " + "passes `causal=false`. W6b"; + } + if (v.temporal_patch_size != 1) { + return "`temporal_patch_size` is " + std::to_string(v.temporal_patch_size) + + ", which is the VIDEO arm: `DotsPatchEmbed.forward` takes " + "`[:, :, 0]` of the temporal axis (vision.py:317-325 @ 9035151d6) " + "and the multi-frame `cu_seqlens` builder is a different one " + "(vision.py:613-624). Video is W7"; + } + if (v.adapter_in_dim != v.embed_dim) { + return "`adapter_in_dim` " + std::to_string(v.adapter_in_dim) + + " is not the tower's `embed_dim` " + std::to_string(v.embed_dim) + + ", so `ln_q` would normalize a width the trunk does not produce " + "(vision.py:466 @ 9035151d6). W6b"; + } + // ── THE TWO THE ENCODER ASSERTS ON ───────────────────────────────────────── + // + // These name NO brick, because nothing is owed: they are configs no + // dots3-note tower can be served under. They are here because a refusal + // predicate that is a strict SUBSET of the request-time asserts is not a + // refusal. `EncodeMmDots3NoteForCausalLM` makes both comparisons again inside + // the ENGINE's busy loop, where a throw sets `AsyncLLM::errored_` + // permanently (`async_llm.cpp:584-601`) β€” the server then starts, serves + // text, 500s the first image, and answers every LATER request, text ones + // included, with "request submitted to a stopped AsyncLLM". Asking here turns + // the same answer into a REFUSING seam: HTTP 400, text path untouched. + // + // The refusal and the route must be the SAME predicate. This is the row's + // second recurrence of that finding (the first is the sparse-routing entry + // under `## Owed`), and it is what retired the tautology that used to sit + // where the second check now is: `adapter_merge_size**2 * adapter_in_dim != + // merged_dim()` compared `merged_dim()` against its own definition. + if (v.adapter_out_dim != v.text_hidden_size) { + return "`adapter_out_dim` " + std::to_string(v.adapter_out_dim) + + " is not the TEXT tower's `hidden_size` " + + std::to_string(v.text_hidden_size) + + ", so `adapter.mlp.2` emits rows that cannot be scattered into the " + "prompt at all (vision.py:461 @ 9035151d6 against " + "`config.hidden_size`). This is the comparison " + "`EncodeMmDots3NoteForCausalLM` makes on a served request"; + } + if (v.adapter_merge_size != v.spatial_merge_size) { + return "`adapter_merge_size` " + std::to_string(v.adapter_merge_size) + + " is not `spatial_merge_size` " + + std::to_string(v.spatial_merge_size) + + ". The PROMPT side expands one image marker into " + "`prod(grid) // spatial_merge_size**2` placeholders " + "(multimodal.py:151-155 @ 9035151d6, and this port's " + "`Dots3NoteProcessorConfig::merge_size`, which is read from that " + "key) while the ADAPTER folds `adapter_merge_size**2` trunk tokens " + "into each emitted row (vision.py:441-449). Upstream keeps the two " + "as independent keys with independent defaults, so a checkpoint can " + "carry them disagreeing; serving it would either leave the trunk " + "length not grouping into whole merger rows, or emit a row count " + "the placeholder span cannot hold"; + } + return ""; +} + +std::string Dots3NoteVisionRefusalFor(const HfConfig& config) { + // The `quantization_config` block is read HERE rather than through + // `ParseDots3NoteParams`, because this overload must answer for a checkpoint + // whose LANGUAGE config the caller has not validated β€” the chat seam runs at + // server start and holds only a path. + std::string quant_method; + std::vector weight_block_size; + const auto qc = config.raw.find("quantization_config"); + if (qc != config.raw.end() && qc->is_object()) { + const auto qm = qc->find("quant_method"); + if (qm != qc->end() && qm->is_string()) quant_method = qm->get(); + const auto wb = qc->find("weight_block_size"); + if (wb != qc->end() && wb->is_array()) { + for (const nlohmann::json& e : *wb) + if (e.is_number_integer()) weight_block_size.push_back(e.get()); + } + } + return Dots3NoteVisionRefusal(ParseDots3NoteVisionParams(config), quant_method, + weight_block_size); +} + +std::vector EnumerateDots3NoteVisionTensors( + const Dots3NoteVisionParams& v) { + std::vector out; + if (!v.present) return out; + const std::string p = "vision_encoder."; + out.push_back({p + "patch_embed.proj.weight", "vision.patch_embed"}); + out.push_back({p + "patch_embed.proj.bias", "vision.patch_embed"}); + out.push_back({p + "patch_embed.norm.weight", "vision.patch_embed"}); + for (int64_t b = 0; b < v.num_hidden_layers; ++b) { + if (v.is_moe_block(b)) continue; // W6b's + const std::string pre = p + "blocks." + std::to_string(b) + "."; + out.push_back({pre + "norm_1.weight", "vision.block.norm_1"}); + out.push_back({pre + "norm_2.weight", "vision.block.norm_2"}); + out.push_back({pre + "attn.qkv.weight", "vision.block.attn.qkv"}); + out.push_back({pre + "attn.proj.weight", "vision.block.attn.proj"}); + out.push_back({pre + "attn.q_norm.weight", "vision.block.attn.q_norm"}); + out.push_back({pre + "attn.k_norm.weight", "vision.block.attn.k_norm"}); + out.push_back({pre + "mlp.fc1.weight", "vision.block.mlp.gate"}); + out.push_back({pre + "mlp.fc2.weight", "vision.block.mlp.down"}); + out.push_back({pre + "mlp.fc3.weight", "vision.block.mlp.up"}); + } + if (v.post_norm) { + out.push_back({p + "post_trunk_norm.weight", "vision.post_trunk_norm"}); + } + out.push_back({p + "adapter.ln_q.weight", "vision.adapter.ln_q"}); + out.push_back({p + "adapter.ln_q.bias", "vision.adapter.ln_q"}); + out.push_back({p + "adapter.mlp.0.weight", "vision.adapter.mlp.0"}); + out.push_back({p + "adapter.mlp.0.bias", "vision.adapter.mlp.0"}); + out.push_back({p + "adapter.mlp.2.weight", "vision.adapter.mlp.2"}); + out.push_back({p + "adapter.mlp.2.bias", "vision.adapter.mlp.2"}); + return out; +} + +Dots3NoteVisionWeights MaterializeDots3NoteVision( + const std::vector& shards, + const Dots3NoteVisionParams& v) { + std::unordered_map where; + for (const SafetensorsFile& f : shards) { + for (const std::string& n : f.Names()) where.emplace(n, &f); + } + const TensorResolver get = + [&where](const std::string& name) -> const StTensor& { + auto it = where.find(name); + VT_CHECK(it != where.end(), + "dots3-note vision tower: tensor not found: " + name); + return it->second->Get(name); + }; + + const int64_t E = v.embed_dim, I = v.intermediate_size, D = v.head_dim(); + const std::string p = "vision_encoder."; + Dots3NoteVisionWeights w; + + // `DotsPatchEmbed.proj` is an `nn.Conv2d(C, E, kernel=stride=patch)` + // (vision.py:308-314), so its weight ships [E, C, p, p]. The forward takes + // `[:, :, 0]` of a temporal axis of size 1 and then applies a kernel that + // covers exactly one patch with no overlap, which is a Linear over the + // flattened patch row β€” so it is read as [E, C*tp*p*p] here. The shape + // OVERRIDE is what makes that reinterpretation explicit rather than implied, + // and the check below is against the ON-DISK rank-4 shape. + { + const StTensor& t = get(p + "patch_embed.proj.weight"); + const std::vector want{E, v.num_channels, v.patch_size, + v.patch_size}; + VT_CHECK(t.shape == want, + "dots3-note vision tower: '" + p + + "patch_embed.proj.weight' ships " + ShapeOf(t.shape) + + ", the config implies " + ShapeOf(want)); + } + w.patch_proj_w = LoadBf16Direct(get, p + "patch_embed.proj.weight", + {E, v.patch_row()}); + RequireVisionShape(w.patch_proj_w, p + "patch_embed.proj.weight", + {E, v.patch_row()}); + w.patch_proj_b = LoadBf16Direct(get, p + "patch_embed.proj.bias"); + RequireVisionShape(w.patch_proj_b, p + "patch_embed.proj.bias", {E}); + w.patch_norm = LoadBf16Direct(get, p + "patch_embed.norm.weight"); + RequireVisionShape(w.patch_norm, p + "patch_embed.norm.weight", {E}); + + for (int64_t b = 0; b < v.num_hidden_layers; ++b) { + VT_CHECK(!v.is_moe_block(b), + "dots3-note vision tower: block " + std::to_string(b) + + " is a pyramid MoE block and W6a does not load it. " + "MaterializeDots3NoteVision was reached with a config " + "Dots3NoteVisionRefusal should have refused; that is a caller " + "defect, not a checkpoint one."); + const std::string pre = p + "blocks." + std::to_string(b) + "."; + Dots3NoteVisionBlockWeights bw; + bw.norm_1 = LoadBf16Direct(get, pre + "norm_1.weight"); + RequireVisionShape(bw.norm_1, pre + "norm_1.weight", {E}); + bw.norm_2 = LoadBf16Direct(get, pre + "norm_2.weight"); + RequireVisionShape(bw.norm_2, pre + "norm_2.weight", {E}); + bw.qkv = LoadBf16Direct(get, pre + "attn.qkv.weight"); + RequireVisionShape(bw.qkv, pre + "attn.qkv.weight", {3 * E, E}); + bw.proj = LoadBf16Direct(get, pre + "attn.proj.weight"); + RequireVisionShape(bw.proj, pre + "attn.proj.weight", {E, E}); + bw.q_norm = LoadBf16Direct(get, pre + "attn.q_norm.weight"); + RequireVisionShape(bw.q_norm, pre + "attn.q_norm.weight", {D}); + bw.k_norm = LoadBf16Direct(get, pre + "attn.k_norm.weight"); + RequireVisionShape(bw.k_norm, pre + "attn.k_norm.weight", {D}); + // fc1 = the SwiGLU gate, fc3 = the up projection + // (`fc2(F.silu(fc1(x)) * fc3(x))`, vision.py:132-133). The merge is what + // routes this pair through `layers::MlpGateUpMethodBase` instead of a + // hand-written parallel path. + bw.gate_up = LoadMergedBf16RawNK( + get, {pre + "mlp.fc1.weight", pre + "mlp.fc3.weight"}); + RequireVisionShape(bw.gate_up, pre + "mlp.{fc1,fc3}.weight", {2 * I, E}); + bw.down = LoadBf16Direct(get, pre + "mlp.fc2.weight"); + RequireVisionShape(bw.down, pre + "mlp.fc2.weight", {E, I}); + w.blocks.push_back(std::move(bw)); + } + + if (v.post_norm) { + w.post_trunk_norm = LoadBf16Direct(get, p + "post_trunk_norm.weight"); + RequireVisionShape(w.post_trunk_norm, p + "post_trunk_norm.weight", {E}); + } + const int64_t M = v.merged_dim(), O = v.adapter_out_dim; + w.adapter_ln_w = LoadBf16Direct(get, p + "adapter.ln_q.weight"); + RequireVisionShape(w.adapter_ln_w, p + "adapter.ln_q.weight", + {v.adapter_in_dim}); + w.adapter_ln_b = LoadBf16Direct(get, p + "adapter.ln_q.bias"); + RequireVisionShape(w.adapter_ln_b, p + "adapter.ln_q.bias", + {v.adapter_in_dim}); + w.adapter_mlp0_w = LoadBf16Direct(get, p + "adapter.mlp.0.weight"); + RequireVisionShape(w.adapter_mlp0_w, p + "adapter.mlp.0.weight", {M, M}); + w.adapter_mlp0_b = LoadBf16Direct(get, p + "adapter.mlp.0.bias"); + RequireVisionShape(w.adapter_mlp0_b, p + "adapter.mlp.0.bias", {M}); + w.adapter_mlp2_w = LoadBf16Direct(get, p + "adapter.mlp.2.weight"); + RequireVisionShape(w.adapter_mlp2_w, p + "adapter.mlp.2.weight", {O, M}); + w.adapter_mlp2_b = LoadBf16Direct(get, p + "adapter.mlp.2.bias"); + RequireVisionShape(w.adapter_mlp2_b, p + "adapter.mlp.2.bias", {O}); + w.present = true; + return w; +} + +std::vector> Dots3NoteVisionPosIds( + const std::array& grid_thw, const Dots3NoteVisionParams& v) { + // `get_pos_ids_by_grid` (vision.py:566-603 @ 9035151d6). When + // `pre_pixel_shuffle` is set the positions follow the qwen `merge_size` + // GROUPED layout, because the preprocessor already emitted the patch rows in + // that order; otherwise they are flat row-major regardless of + // `spatial_merge_size` (upstream's own comment at :567-570). + const int64_t rope_merge = + v.pre_pixel_shuffle ? (v.spatial_merge_size > 1 ? v.spatial_merge_size : 2) + : 1; + const int64_t t = grid_thw[0], h = grid_thw[1], wgrid = grid_thw[2]; + VT_CHECK(h % rope_merge == 0 && wgrid % rope_merge == 0, + "dots3-note vision tower: grid " + std::to_string(h) + "x" + + std::to_string(wgrid) + + " is not divisible by the RoPE merge size " + + std::to_string(rope_merge) + + ", so the grouped position reshape (vision.py:576-590 @ " + "9035151d6) has no answer"); + std::vector> one; + one.reserve(static_cast(h * wgrid)); + // `reshape(h/m, m, w/m, m).permute(0, 2, 1, 3).flatten()` over an [h, w] + // array whose value is the ROW index (h_pos) or the COLUMN index (w_pos). + for (int64_t bh = 0; bh < h / rope_merge; ++bh) { + for (int64_t bw = 0; bw < wgrid / rope_merge; ++bw) { + for (int64_t sh = 0; sh < rope_merge; ++sh) { + for (int64_t sw = 0; sw < rope_merge; ++sw) { + one.push_back({bh * rope_merge + sh, bw * rope_merge + sw}); + } + } + } + } + std::vector> out; + out.reserve(one.size() * static_cast(t)); + for (int64_t f = 0; f < t; ++f) + out.insert(out.end(), one.begin(), one.end()); + return out; +} + +std::vector Dots3NoteVisionRopeCache( + const std::array& grid_thw, const Dots3NoteVisionParams& v) { + // `VisionRotaryEmbedding(head_dim // 2)` (vision.py:503-504) builds + // `inv_freq = 1 / theta ** (arange(0, dim, 2) / dim)` with `dim = head_dim/2` + // (vision_attention.py:67), i.e. head_dim/4 frequencies per spatial axis. + // `rot_pos_emb` gathers [L, 2, nf] and flattens to [L, 2*nf] = [L, head_dim/2] + // (vision.py:604-611). `apply_rotary_pos_emb_vision` then repeats that to + // head_dim as [f | f] and applies NeoX rotate_half + // (vision_attention.py:39-49) β€” which is exactly the [cos(hd/2) | sin(hd/2)] + // cache `vt::RopeFromCache` consumes at `rotary_dim == head_dim`. + const int64_t hd = v.head_dim(); + const int64_t dim = hd / 2; + const int64_t nf = dim / 2; // frequencies per axis + VT_CHECK(nf * 2 == dim, + "dots3-note vision tower: head_dim/2 is odd, so the rope frequency " + "table has no whole per-axis half"); + std::vector inv_freq(static_cast(nf)); + for (int64_t i = 0; i < nf; ++i) { + inv_freq[static_cast(i)] = + 1.0 / std::pow(10000.0, static_cast(2 * i) / + static_cast(dim)); + } + const std::vector> pos = + Dots3NoteVisionPosIds(grid_thw, v); + const int64_t L = static_cast(pos.size()); + std::vector cache(static_cast(L * hd)); + for (int64_t r = 0; r < L; ++r) { + for (int64_t axis = 0; axis < 2; ++axis) { + const double p = static_cast(pos[static_cast(r)][ + static_cast(axis)]); + for (int64_t i = 0; i < nf; ++i) { + const double ang = p * inv_freq[static_cast(i)]; + const int64_t c = axis * nf + i; + cache[static_cast(r * hd + c)] = + static_cast(std::cos(ang)); + cache[static_cast(r * hd + dim + c)] = + static_cast(std::sin(ang)); + } + } + } + return cache; +} + +namespace { + +// out[M,N] = x[M,K] @ W[N,K]^T (+ bias[N]). bf16 throughout, exactly the +// projection shape the rest of this tree spells. +void LinearBias(Dev d, DBuf& out, const Tensor& x, const Tensor& w, + const Tensor* bias) { + vt::MatmulBT(d.q, out.t(), x, w); + if (bias != nullptr) vt::Add(d.q, out.t(), out.t(), *bias); +} + +std::vector DownloadF32(Dev d, DBuf& buf, int64_t n) { + std::vector bits(static_cast(n)); + buf.Download(d, bits.data()); + std::vector out(static_cast(n)); + for (int64_t i = 0; i < n; ++i) + out[static_cast(i)] = vt::BF16ToF32(bits[static_cast(i)]); + return out; +} + +} // namespace + +std::vector Dots3NoteVisionForward( + const std::vector& pixel_values_bf16, + const std::array& grid_thw, const Dots3NoteVisionWeights& w, + const Dots3NoteVisionParams& v, vt::Backend& backend, + Dots3NoteVisionCapture* cap) { + VT_CHECK(w.present, + "dots3-note vision tower: the weights were never materialized. The " + "loader only materializes a tower Dots3NoteVisionRefusal accepted."); + VT_CHECK(grid_thw[0] == 1, + "dots3-note vision tower: grid_t is " + std::to_string(grid_thw[0]) + + ". W6a serves single-frame IMAGE items; the multi-frame " + "`cu_seqlens` builder (vision.py:613-624 @ 9035151d6) is the " + "VIDEO arm and belongs to W7."); + const int64_t E = v.embed_dim, I = v.intermediate_size; + const int64_t nh = v.num_attention_heads, hd = v.head_dim(); + const int64_t L = grid_thw[0] * grid_thw[1] * grid_thw[2]; + const int64_t P = v.patch_row(); + VT_CHECK(L > 0, "dots3-note vision tower: an empty grid"); + VT_CHECK(static_cast(pixel_values_bf16.size()) == L * P, + "dots3-note vision tower: the processor produced " + + std::to_string(pixel_values_bf16.size()) + + " patch values for a grid implying " + std::to_string(L) + " x " + + std::to_string(P) + " = " + std::to_string(L * P)); + + vt::Queue q = backend.CreateQueue(); + Dev d{backend, q}; + const vt::RmsNormArgs rms{static_cast(v.rms_norm_eps), /*gemma=*/false}; + + // ── patch_embed (vision.py:302-332) ──────────────────────────────────────── + DBuf hidden(d, DType::kBF16, {L, E}); + { + DBuf px(d, DType::kBF16, {L, P}, pixel_values_bf16.data()); + DBuf proj(d, DType::kBF16, {L, E}); + Tensor pw = ResidentWeight(d, w.patch_proj_w, {E, P}); + Tensor pb = ResidentWeight(d, w.patch_proj_b, {E}); + LinearBias(d, proj, px.t(), pw, &pb); + Tensor nw = ResidentWeight(d, w.patch_norm, {E}); + vt::RmsNorm(d.q, hidden.t(), proj.t(), nw, rms); + } + if (cap != nullptr) cap->patch_embed_out = DownloadF32(d, hidden, L * E); + + // ── the 2-D vision rope cache, host f32 -> bf16 device (vision.py:604-611) ── + const std::vector cache_f = Dots3NoteVisionRopeCache(grid_thw, v); + if (cap != nullptr) cap->rope_cache = cache_f; + std::vector cache_bits(cache_f.size()); + for (size_t i = 0; i < cache_f.size(); ++i) + cache_bits[i] = vt::F32ToBF16(cache_f[i]); + DBuf cache(d, DType::kBF16, {L, hd}, cache_bits.data()); + std::vector pos_idx(static_cast(L)); + for (int64_t i = 0; i < L; ++i) pos_idx[static_cast(i)] = + static_cast(i); + DBuf posb(d, DType::kI32, {L}, pos_idx.data()); + vt::RopeArgs ra; + ra.rotary_dim = static_cast(hd); + ra.is_neox_style = true; + + // Softmax scale: `1 / sqrt(head_dim)` (vision_attention.py:199, :233). + const float scale = + 1.0f / std::sqrt(static_cast(hd)); + const vt::AttentionArgs aargs{scale, /*causal=*/false}; + + DBuf n1(d, DType::kBF16, {L, E}); + DBuf qkv(d, DType::kBF16, {L, 3 * E}); + DBuf qb(d, DType::kBF16, {L, E}); + DBuf kb(d, DType::kBF16, {L, E}); + DBuf vb(d, DType::kBF16, {L, E}); + DBuf ao(d, DType::kBF16, {L, nh, hd}); + DBuf attn(d, DType::kBF16, {L, E}); + DBuf n2(d, DType::kBF16, {L, E}); + DBuf mlp_out(d, DType::kBF16, {L, E}); + + for (size_t b = 0; b < w.blocks.size(); ++b) { + const Dots3NoteVisionBlockWeights& bw = w.blocks[b]; + // `apply_vision_attention_residual` (vision_attention.py:436-456): + // `hidden + attn(norm_1(hidden))`. PRE-norm, and the residual is the + // UN-normalized stream. + vt::RmsNorm(d.q, n1.t(), hidden.t(), ResidentWeight(d, bw.norm_1, {E}), rms); + { + Tensor wq = ResidentWeight(d, bw.qkv, {3 * E, E}); + vt::MatmulBT(d.q, qkv.t(), n1.t(), wq); + vt::QkvSplit(d.q, qb.t(), kb.t(), vb.t(), qkv.t()); + } + // `_qkv_with_rope` (vision_attention.py:149-166): the per-head q/k RMSNorm + // runs BEFORE the rope. Swapping the two is silent β€” same shapes, same + // magnitudes, different numbers β€” and this row has no oracle downstream to + // catch it, which is why the order has its own gate case. + { + Tensor qn = qb.t(); + qn.rank = 2; qn.shape[0] = L * nh; qn.shape[1] = hd; + qn.stride[0] = hd; qn.stride[1] = 1; + Tensor kn = kb.t(); + kn.rank = 2; kn.shape[0] = L * nh; kn.shape[1] = hd; + kn.stride[0] = hd; kn.stride[1] = 1; + vt::RmsNorm(d.q, qn, qn, ResidentWeight(d, bw.q_norm, {hd}), rms); + vt::RmsNorm(d.q, kn, kn, ResidentWeight(d, bw.k_norm, {hd}), rms); + } + Tensor q3 = qb.t(); + q3.rank = 3; q3.shape[0] = L; q3.shape[1] = nh; q3.shape[2] = hd; + q3.stride[0] = nh * hd; q3.stride[1] = hd; q3.stride[2] = 1; + Tensor k3 = kb.t(); + k3.rank = 3; k3.shape[0] = L; k3.shape[1] = nh; k3.shape[2] = hd; + k3.stride[0] = nh * hd; k3.stride[1] = hd; k3.stride[2] = 1; + Tensor v3 = vb.t(); + v3.rank = 3; v3.shape[0] = L; v3.shape[1] = nh; v3.shape[2] = hd; + v3.stride[0] = nh * hd; v3.stride[1] = hd; v3.stride[2] = 1; + vt::RopeFromCache(d.q, q3, &k3, posb.t(), cache.t(), ra); + // ONE window: `grid_t == 1` is asserted above, so `cu_seqlens` is + // `[0, h*w]` under either of upstream's two builders and the whole item is + // a single bidirectional block. + vt::AttentionDenseFlash(d.q, ao.t(), q3, k3, v3, aargs); + { + Tensor ao2 = ao.t(); + ao2.rank = 2; ao2.shape[0] = L; ao2.shape[1] = E; + ao2.stride[0] = E; ao2.stride[1] = 1; + Tensor wp = ResidentWeight(d, bw.proj, {E, E}); + LinearBias(d, attn, ao2, wp, nullptr); + } + vt::Add(d.q, hidden.t(), hidden.t(), attn.t()); + + // `hidden + mlp(norm_2(hidden))` (vision.py:369). + vt::RmsNorm(d.q, n2.t(), hidden.t(), ResidentWeight(d, bw.norm_2, {E}), rms); + { + // THE SHARED SEAM. `fc2(silu(fc1(x)) * fc3(x))` is a mergeable gate/up + // pair, so it rides `layers::MlpGateUpMethodBase` rather than two + // hand-written GEMMs (AGENTS.md, "Shared seams"). + DBuf act = layers::UnquantizedMlpGateUpMethod(&bw.gate_up, I).Apply(d, n2.t()); + Tensor wd = ResidentWeight(d, bw.down, {E, I}); + LinearBias(d, mlp_out, act.t(), wd, nullptr); + } + vt::Add(d.q, hidden.t(), hidden.t(), mlp_out.t()); + if (cap != nullptr && b == 0) cap->block0_out = DownloadF32(d, hidden, L * E); + } + + // ── post_trunk_norm (vision.py:513-514, 676-677) ─────────────────────────── + DBuf trunk(d, DType::kBF16, {L, E}); + if (v.post_norm) { + vt::RmsNorm(d.q, trunk.t(), hidden.t(), + ResidentWeight(d, w.post_trunk_norm, {E}), rms); + } else { + backend.Copy(q, trunk.ptr(), hidden.ptr(), hidden.bytes()); + } + if (cap != nullptr) cap->trunk_out = DownloadF32(d, trunk, L * E); + + // ── the patch_merger adapter (vision.py:441-490) ─────────────────────────── + // + // `ln_q` normalizes over the PER-TOKEN dim with a HARD-CODED eps of 1e-6 + // (vision.py:466) β€” NOT `rms_norm_eps`, and it is a LayerNorm with a bias, + // not an RMSNorm. Then `reshape(-1, merged_dim)` views every 4 consecutive + // 2x2-grouped tokens as one row: no permutation, because `pre_pixel_shuffle` + // put the tokens in that order already. + const int64_t merge_unit = v.adapter_merge_size * v.adapter_merge_size; + VT_CHECK(L % merge_unit == 0, + "dots3-note vision tower: " + std::to_string(L) + + " trunk tokens do not group into whole " + + std::to_string(merge_unit) + + "-token merger rows. The processor's grid and the adapter's " + "merge size disagree."); + const int64_t Nm = L / merge_unit; + const int64_t M = v.merged_dim(), O = v.adapter_out_dim; + + DBuf lnq(d, DType::kBF16, {L, E}); + { + Tensor lw = ResidentWeight(d, w.adapter_ln_w, {v.adapter_in_dim}); + Tensor lb = ResidentWeight(d, w.adapter_ln_b, {v.adapter_in_dim}); + vt::LayerNorm(d.q, lnq.t(), trunk.t(), &lw, &lb, vt::LayerNormArgs{1e-6f}); + } + DBuf fc1(d, DType::kBF16, {Nm, M}); + { + Tensor xv = lnq.t(); // [L, E] contiguous IS [Nm, M] + xv.rank = 2; xv.shape[0] = Nm; xv.shape[1] = M; + xv.stride[0] = M; xv.stride[1] = 1; + Tensor w0 = ResidentWeight(d, w.adapter_mlp0_w, {M, M}); + Tensor b0 = ResidentWeight(d, w.adapter_mlp0_b, {M}); + LinearBias(d, fc1, xv, w0, &b0); + } + // `nn.GELU()` with no `approximate` argument is the EXACT erf gelu + // (vision.py:469). The tanh approximation is a different function and a + // silent one at this magnitude. + vt::GeluErf(d.q, fc1.t(), fc1.t()); + DBuf out(d, DType::kBF16, {Nm, O}); + { + Tensor w2 = ResidentWeight(d, w.adapter_mlp2_w, {O, M}); + Tensor b2 = ResidentWeight(d, w.adapter_mlp2_b, {O}); + LinearBias(d, out, fc1.t(), w2, &b2); + } + return DownloadF32(d, out, Nm * O); +} + +} // namespace vllm diff --git a/src/vllm/model_executor/models/dots3_note_vision.h b/src/vllm/model_executor/models/dots3_note_vision.h new file mode 100644 index 000000000..c60d4b83b --- /dev/null +++ b/src/vllm/model_executor/models/dots3_note_vision.h @@ -0,0 +1,283 @@ +// dots3-note VISION tower β€” the DENSE arm (W6a, #2512). +// +// Ported from vLLM read in the local clone `~/_git/vllm` at +// **`9035151d6`**, the merge of vllm#51255 that added the architecture. That +// SHA is written beside every anchor in this file on purpose: `dots3_note` does +// not exist at our parity pin `5559679229bc961848b121ccdeaa8fa5d79bec98`, and +// upstream has ALREADY moved β€” `nvidia/vision_attention.py` is 477 lines at +// `9035151d6` and 494 lines at vLLM `main` `7a100bb61`. An anchor with no +// revision beside it is a line number read in the wrong tree. +// +// vllm/models/dots3_note/nvidia/vision.py @ 9035151d6 (677 lines) +// DotsMoEVitConfig :27 -> Dots3NoteVisionParams +// RMSNorm :107 -> vt::RmsNorm (one deliberate +// rounding difference, see below) +// DotsSwiGLUFFN :126 -> layers::UnquantizedMlpGateUpMethod +// DotsPatchEmbed :302 -> the patch GEMM + RMSNorm +// MoEVisionBlock :334 -> Dots3NoteVisionBlockWeights +// PatchMergerAdapter :441 -> the adapter +// DotsMoEVitModel :492 -> Dots3NoteVisionForward +// get_pos_ids_by_grid :566 -> Dots3NoteVisionPosIds +// rot_pos_emb :604 -> Dots3NoteVisionRopeCache +// forward :634 +// vllm/models/dots3_note/nvidia/vision_attention.py @ 9035151d6 (477 lines) +// rotate_half :33 +// apply_rotary_pos_emb_vision :39 -> vt::RopeFromCache (NeoX, rotary_dim +// == head_dim, [L, head_dim] cache) +// VisionRotaryEmbedding :52 -> Dots3NoteVisionRopeCache +// _RMSNorm :97 -> the per-head q_norm/k_norm +// _VisionAttentionBase :134 -> the qkv/proj pair +// _qkv_with_rope :149 -> qk-norm BEFORE rope; the order is +// load-bearing and silent when wrong +// VisionAttentionV2 :207 -> vt::AttentionDenseFlash, causal=false +// apply_vision_attention_residual :436 +// +// WHAT THIS FILE IS NOT. `nvidia/vision_moe.py` @ `9035151d6` (149 lines) and +// `MoESwiGLUFFN` / `MoESwiGLUFFNFP8` (`vision.py:139`, `:222`) are the PYRAMID +// arm. Nothing here reads them, and `Dots3NoteVisionRefusal` names W6b for any +// block the config marks routed. The RELEASED `dots-studio/dots3-note-prev` has +// 17 such blocks out of 42, so its vision tower still refuses BY NAME β€” which +// is exactly what W3 did to the language tower for four bricks before W5 lifted +// it, not a new exception. See `.agents/specs/dots3-note.md` Β§4.11. +// +// WHY IT SHARES NO CODE WITH `qwen3_vl_vision`. The two towers agree on the +// block OUTLINE and on almost nothing below it: RMSNorm vs LayerNorm, no bias +// anywhere vs bias everywhere, a per-head qk-norm applied BEFORE rope, a +// three-tensor SwiGLU vs a two-tensor GELU MLP, a patch-merger adapter vs a +// pixel-shuffle merger, no DeepStack, no interpolated position-embedding table. +// Extending `Qwen3VLVisionConfig` to carry all of that would be a parallel path +// wearing one struct's name. What IS shared is every seam underneath: the +// `vt::` ops, `dense_attn`'s device glue, `dense_loaders`' weight readers and +// `layers::MlpGateUpMethodBase` for the mergeable projections. +// +// THE ONE DELIBERATE FORMULA DIFFERENCE, named rather than left to be found. +// Upstream's `RMSNorm.forward` (`vision.py:112-114`) is +// `self._norm(x.float()).type_as(x) * self.weight` β€” it rounds BACK to the +// activation dtype before multiplying by the weight. `vt::RmsNorm` keeps f32 +// through the weight multiply and rounds once on the store (`vt/ops.h`'s own +// note on it). Using the shared op is the seam rule; the difference is one bf16 +// rounding step, it is what the gate's tolerance carries, and the in-test +// double-precision reference mirrors UPSTREAM rather than this file so that the +// difference is measured rather than defined away. +#ifndef VLLM_MODEL_EXECUTOR_MODELS_DOTS3_NOTE_VISION_H_ +#define VLLM_MODEL_EXECUTOR_MODELS_DOTS3_NOTE_VISION_H_ + +#include +#include +#include +#include + +#include "vllm/model_executor/models/qwen3_5_weights.h" // OwnedTensor +#include "vt/backend.h" + +namespace vllm { + +class SafetensorsFile; +struct HfConfig; +struct Dots3NoteTensor; + +// `DotsMoEVitConfig.__init__` (`vision.py:27-105` @ `9035151d6`), reduced to +// the fields this arm reads plus the ones it REFUSES on. Every default here is +// upstream's own default, so a config that omits a key gets what upstream's +// constructor would have given it β€” never a value chosen locally. +struct Dots3NoteVisionParams { + // False when `config.json` carries no `vision_config` at all. Upstream builds + // no `DotsMoEVitModel` in that case (`multimodal.py:113-118` @ `9035151d6`, + // the `vision_config_dict is not None` guard), and neither does this port. + bool present = false; + + int64_t embed_dim = 1536; + // The TEXT tower's `hidden_size`, copied from the LANGUAGE config at parse. + // It is NOT a `vision_config` key, and it replaces one that was: upstream's + // `vision_config.hidden_size` is a SECOND copy of the same number, nothing + // read it, and a refusal that read it would still not be the predicate the + // encoder applies. `EncodeMmDots3NoteForCausalLM` compares `adapter_out_dim` + // against `config.hidden_size`, so that is the value that has to be here for + // `Dots3NoteVisionRefusal` to answer the same question the route asks. + int64_t text_hidden_size = 0; + int64_t intermediate_size = 4224; // the DENSE SwiGLU width + int64_t moe_intermediate_size = 2112; // W6b's; read only to report it + int64_t num_hidden_layers = 42; + int64_t num_attention_heads = 24; + int64_t num_channels = 3; + int64_t patch_size = 14; + int64_t spatial_merge_size = 2; + int64_t temporal_patch_size = 1; + double rms_norm_eps = 1e-5; + bool use_bias = false; + bool use_qk_norm = true; + bool is_causal = false; + bool post_norm = true; + // `pre_pixel_shuffle` (`vision.py:60-63`): when TRUE the PREPROCESSOR already + // emits patch rows in 2x2-grouped order (`common/processor.py:185-197`), so + // the RoPE positions are regrouped to match (`get_pos_ids_by_grid:566-575`). + // It is NOT the same switch as `adapter_type`, and conflating the two is the + // reading #2512's prose invites β€” see `.agents/specs/dots3-note.md` Β§4.11.1. + // + // FALSE is `DotsMoEVitConfig`'s own default (`vision.py:64`), which is why it + // is the default here; the RELEASED checkpoint sets it TRUE in its + // `vision_config` and the parser reads that. The struct default matters + // because this field selects between two INCOMPATIBLE token orders, so a + // default-constructed params must not silently claim the regrouped one. + bool pre_pixel_shuffle = false; + + // The per-block routed-expert counts (`vision.py:90`). `is_moe` is + // `pyramid_num_routed[i] > 0` (`vision.py:346-350`), so the released + // checkpoint's leading 25 entries of `-1` are DENSE and the trailing 17 + // (4, 8, ... 60, 64, 64) are W6b's. + std::vector pyramid_num_routed; + double capacity_factor = 2.0; // W6b's top-k; read only to report it + std::string router_scoring_func = "sigmoid"; + double router_scale = 1.0; + + // The adapter (`vision.py:441-472`). `patch_merger` is the arm the released + // checkpoint selects; `pixel_shuffle_mlp` is a DIFFERENT token order from the + // same pixels and is refused by name. + std::string adapter_type = "pixel_shuffle_mlp"; + int64_t adapter_in_dim = 1536; + int64_t adapter_out_dim = 2048; + int64_t adapter_merge_size = 2; + + int64_t head_dim() const { return embed_dim / num_attention_heads; } + // `merged_dim = in_dim * merge_size**2` (`vision.py:462`). + int64_t merged_dim() const { + return adapter_in_dim * adapter_merge_size * adapter_merge_size; + } + // One patch row as the processor ships it: + // `channel * temporal_patch_size * patch_size * patch_size` + // (`common/processor.py:216-218` @ `9035151d6`). + int64_t patch_row() const { + return num_channels * temporal_patch_size * patch_size * patch_size; + } + bool is_moe_block(int64_t layer) const { + return layer >= 0 && + layer < static_cast(pyramid_num_routed.size()) && + pyramid_num_routed[static_cast(layer)] > 0; + } + // How many DENSE blocks this config has, counted over ALL blocks rather than + // as a leading run: a config that interleaved them would report the truth + // instead of the length of its first run. + int64_t num_dense_blocks() const { + int64_t n = 0; + for (int64_t i = 0; i < num_hidden_layers; ++i) + if (!is_moe_block(i)) ++n; + return n; + } + int64_t num_moe_blocks() const { return num_hidden_layers - num_dense_blocks(); } +}; + +// Resolve + validate `config.json`'s `vision_config`. Returns `present=false` +// when the key is absent. Throws (VT_CHECK) naming the key on a value this arm +// cannot represent AND that no later brick owns β€” a shape that IS owed to a +// later brick is reported by `Dots3NoteVisionRefusal` instead, because a +// checkpoint whose tower is owed must still LOAD its language half. +Dots3NoteVisionParams ParseDots3NoteVisionParams(const HfConfig& config); + +// Why the vision tower cannot be materialized, or "" when it can. Names ONE +// thing β€” the first unrepresentable feature in brick order β€” and the brick that +// owes it. A non-empty answer leaves the 2195 `vision_encoder.*` tensors in the +// accounting's existing `vision` deferral bucket, exactly as before W6a, so +// every W2 count assertion is unchanged. +std::string Dots3NoteVisionRefusal(const Dots3NoteVisionParams& v, + const std::string& quant_method, + const std::vector& weight_block_size); + +// The same answer from a CONFIG alone, for a caller that holds a checkpoint +// directory and no loaded model. The multimodal CHAT seam is that caller, and +// it is the reason this overload exists rather than a convenience. +// +// A refusal raised from `encode_mm` is FATAL: it is thrown inside the engine's +// busy loop, which stops `AsyncLLM` and turns every later request β€” including +// TEXT ones β€” into a 500. Measured on this row's served-request gate before +// this function existed. The entrypoint is where a "this server cannot serve +// images for this checkpoint" answer belongs, and `InstallMultiModalChatSeam` +// already has the shape for it: a factory that throws installs a REFUSING seam, +// which answers an image request with HTTP 400 naming the architecture and the +// reason while the text path keeps working. The `encode_mm` check stays as +// defence in depth, on the same polarity as Qwen3-VL's ("reaching this point is +// a defect"). +std::string Dots3NoteVisionRefusalFor(const HfConfig& config); + +// One DENSE block's weights, by the names the checkpoint ships +// (`vision_encoder.blocks.{B}.*`). Every tensor is BF16 on disk and BF16 here: +// the released index carries 37944 BF16 + 62 F32 and every F32 of those is a +// `router_bias` or an `e_score_correction_bias`, neither of which this arm +// reads. Widening any of these would be invisible to a token gate and is what +// `porting.md`'s memory-format rule is about. +struct Dots3NoteVisionBlockWeights { + OwnedTensor norm_1; // [E] + OwnedTensor norm_2; // [E] + OwnedTensor qkv; // [3E, E] (NO bias: use_bias == false) + OwnedTensor proj; // [E, E] + OwnedTensor q_norm; // [head_dim] + OwnedTensor k_norm; // [head_dim] + // fc1 (the SwiGLU gate) and fc3 (the up projection) MERGED into one [2I, E] + // raw-NK operand, so the pair rides `layers::MlpGateUpMethodBase` rather than + // a hand-written parallel path (AGENTS.md, "Shared seams"). The order is + // gate-then-up because `vt::SiluAndMul` reads `silu(x[:, :I]) * x[:, I:]` and + // upstream is `fc2(F.silu(fc1(x)) * fc3(x))` (`vision.py:132-133`). + OwnedTensor gate_up; // [2I, E] = concat(fc1, fc3) + OwnedTensor down; // fc2 [E, I] +}; + +struct Dots3NoteVisionWeights { + bool present = false; + OwnedTensor patch_proj_w; // [E, C*tp*p*p] (on disk [E, C, p, p]) + OwnedTensor patch_proj_b; // [E] + OwnedTensor patch_norm; // [E] + std::vector blocks; // the DENSE blocks, in order + OwnedTensor post_trunk_norm; // [E] + OwnedTensor adapter_ln_w; // [in_dim] + OwnedTensor adapter_ln_b; // [in_dim] + OwnedTensor adapter_mlp0_w; // [merged_dim, merged_dim] + OwnedTensor adapter_mlp0_b; // [merged_dim] + OwnedTensor adapter_mlp2_w; // [out_dim, merged_dim] + OwnedTensor adapter_mlp2_b; // [out_dim] +}; + +// Every `vision_encoder.*` name the DENSE arm claims, with its named consumer. +// Over the released checkpoint this is exactly 235 of the 2195 vision tensors; +// the other 1960 belong to W6b and stay deferred. +std::vector EnumerateDots3NoteVisionTensors( + const Dots3NoteVisionParams& v); + +// Read the DENSE tower out of `shards`. REFUSES BY NAME on the first tensor +// whose shape disagrees with the config. Only called when +// `Dots3NoteVisionRefusal` is empty. +Dots3NoteVisionWeights MaterializeDots3NoteVision( + const std::vector& shards, const Dots3NoteVisionParams& v); + +// `get_pos_ids_by_grid` (`vision.py:566-603` @ `9035151d6`) for ONE grid. +// Returns L = t*h*w pairs {h_pos, w_pos} in the tower's token order. +std::vector> Dots3NoteVisionPosIds( + const std::array& grid_thw, const Dots3NoteVisionParams& v); + +// `VisionRotaryEmbedding` + `rot_pos_emb` (`vision_attention.py:52-89`, +// `vision.py:604-611`) folded into the [L, head_dim] = [cos(hd/2) | sin(hd/2)] +// cache `vt::RopeFromCache` consumes. f32 host precompute, deterministic. +std::vector Dots3NoteVisionRopeCache( + const std::array& grid_thw, const Dots3NoteVisionParams& v); + +// Optional intermediate capture, for the unit gate only. Production passes +// nullptr and pays nothing. +struct Dots3NoteVisionCapture { + std::vector rope_cache; // [L, head_dim] + std::vector patch_embed_out; // [L, E] + std::vector block0_out; // [L, E] + std::vector trunk_out; // [L, E], after post_trunk_norm +}; + +// THE TOWER. `pixel_values_bf16` is [L, patch_row()] raw bf16 bits as the +// processor ships them; `grid_thw` is {t, h, w}. Returns +// [L / merge^2, adapter_out_dim] host f32 β€” the rows the placeholder span is +// expanded to (`multimodal.py:151-155` @ `9035151d6`: +// `grid.prod(-1) // merge_size**2`). +std::vector Dots3NoteVisionForward( + const std::vector& pixel_values_bf16, + const std::array& grid_thw, const Dots3NoteVisionWeights& w, + const Dots3NoteVisionParams& v, vt::Backend& backend, + Dots3NoteVisionCapture* capture = nullptr); + +} // namespace vllm + +#endif // VLLM_MODEL_EXECUTOR_MODELS_DOTS3_NOTE_VISION_H_ diff --git a/src/vllm/multimodal/dots3_note_processor.cpp b/src/vllm/multimodal/dots3_note_processor.cpp new file mode 100644 index 000000000..2f06a29a1 --- /dev/null +++ b/src/vllm/multimodal/dots3_note_processor.cpp @@ -0,0 +1,325 @@ +// dots3-note IMAGE processor (W6a, #2512). Ported from +// `vllm/models/dots3_note/common/processor.py` read in `~/_git/vllm` at +// `9035151d6`. See the header for the full provenance and for the three ways +// this differs from the Qwen3-VL processor beside it. +#include "vllm/multimodal/dots3_note_processor.h" + +#include +#include +#include +#include +#include + +#include + +#include "vllm/multimodal/hasher.h" +#include "vt/dtype.h" + +namespace vllm::multimodal { + +namespace { + +nlohmann::json LoadJson(const std::string& path) { + std::ifstream f(path); + if (!f) throw std::runtime_error("cannot open json: " + path); + nlohmann::json j; + f >> j; + return j; +} + +// `round(v / factor) * factor` with Python's round-half-to-EVEN +// (`processor.py:86-88` @ `9035151d6`). `std::round` is half-away-from-zero and +// would disagree at exactly .5, which is one grid row. +int64_t RoundByFactor(int64_t v, int64_t factor) { + return static_cast(std::nearbyint(static_cast(v) / + static_cast(factor))) * + factor; +} +int64_t CeilByFactor(double v, int64_t factor) { + return static_cast( + std::ceil(v / static_cast(factor))) * factor; +} +int64_t FloorByFactor(double v, int64_t factor) { + return static_cast( + std::floor(v / static_cast(factor))) * factor; +} + +// Resolve one of the three image token ids from `added_tokens.json` (upstream's +// own source, `multimodal.py:82-90` @ `9035151d6`) or, failing that, from the +// `config.json` key a converted checkpoint may carry. Returns -1 when neither +// answers; the caller refuses BY NAME rather than defaulting. +int32_t ResolveTokenId(const nlohmann::json& added, const nlohmann::json& cfg, + const char* marker, const char* config_key) { + if (added.is_object()) { + const auto it = added.find(marker); + if (it != added.end() && it->is_number_integer()) + return it->get(); + } + if (cfg.is_object()) { + const auto it = cfg.find(config_key); + if (it != cfg.end() && it->is_number_integer()) + return it->get(); + } + return -1; +} + +} // namespace + +Dots3NoteProcessorConfig LoadDots3NoteProcessorConfig( + const std::string& preprocessor_config_json_path, + const std::string& config_json_path, const std::string& model_id) { + Dots3NoteProcessorConfig cfg; + cfg.model_id = model_id; + + const nlohmann::json pp = LoadJson(preprocessor_config_json_path); + cfg.patch_size = pp.value("patch_size", cfg.patch_size); + cfg.temporal_patch_size = + pp.value("temporal_patch_size", cfg.temporal_patch_size); + cfg.merge_size = pp.value("merge_size", cfg.merge_size); + cfg.pre_pixel_shuffle = pp.value("pre_pixel_shuffle", cfg.pre_pixel_shuffle); + cfg.min_pixels = pp.value("min_pixels", cfg.min_pixels); + cfg.max_pixels = pp.value("max_pixels", cfg.max_pixels); + // The `size` shorthand the HF image-processor family also writes. + if (pp.contains("size") && pp["size"].is_object()) { + const auto& sz = pp["size"]; + cfg.min_pixels = sz.value("shortest_edge", cfg.min_pixels); + cfg.max_pixels = sz.value("longest_edge", cfg.max_pixels); + } + // PER CHANNEL. Reading only `[0]` β€” which is what the Qwen3-VL loader beside + // this one can afford, because its mean and std are 0.5 on all three β€” would + // silently normalize green and blue with red's statistics. + const auto read3 = [](const nlohmann::json& j, const char* key, + std::array* out) { + if (!j.contains(key)) return; + const auto& a = j[key]; + if (!a.is_array()) { + throw std::runtime_error(std::string("dots3-note processor: '") + key + + "' must be a 3-entry list, got " + a.dump()); + } + if (a.size() == 1) { + (*out) = {a[0].get(), a[0].get(), a[0].get()}; + return; + } + if (a.size() != 3) { + throw std::runtime_error( + std::string("dots3-note processor: '") + key + "' has " + + std::to_string(a.size()) + + " entries; the RGB pipeline needs 1 or 3 (processor.py:76-77 @ " + "9035151d6)"); + } + for (int i = 0; i < 3; ++i) (*out)[static_cast(i)] = a[i].get(); + }; + read3(pp, "image_mean", &cfg.image_mean); + read3(pp, "image_std", &cfg.image_std); + cfg.rescale_factor = pp.value("rescale_factor", cfg.rescale_factor); + + const nlohmann::json cj = LoadJson(config_json_path); + // `vision_config` is the AUTHORITY on the patch/merge geometry, exactly as it + // is for the tower: a `preprocessor_config.json` that disagrees with the + // model it belongs to would move the grid. + if (cj.contains("vision_config") && cj["vision_config"].is_object()) { + const auto& vc = cj["vision_config"]; + cfg.merge_size = vc.value("spatial_merge_size", cfg.merge_size); + cfg.patch_size = vc.value("patch_size", cfg.patch_size); + cfg.temporal_patch_size = + vc.value("temporal_patch_size", cfg.temporal_patch_size); + cfg.pre_pixel_shuffle = vc.value("pre_pixel_shuffle", cfg.pre_pixel_shuffle); + } + + // The three ids. Upstream reads `added_tokens.json` and RAISES when + // `<|imgpad|>` is absent (`multimodal.py:86-90` @ `9035151d6`); this mirrors + // that, and extends it to the two markers around it, because injecting a + // start/end marker the tokenizer does not know breaks the prompt just as + // quietly. + nlohmann::json added = nlohmann::json::object(); + { + const std::string dir = + config_json_path.substr(0, config_json_path.find_last_of("/\\") + 1); + std::ifstream f(dir + "added_tokens.json"); + if (f) f >> added; + } + cfg.image_token_id = + ResolveTokenId(added, cj, "<|imgpad|>", "image_token_id"); + cfg.image_start_token_id = + ResolveTokenId(added, cj, "<|img|>", "image_start_token_id"); + cfg.image_end_token_id = + ResolveTokenId(added, cj, "<|endofimg|>", "image_end_token_id"); + const auto require = [](int32_t id, const char* marker, const char* key) { + if (id >= 0) return; + throw std::runtime_error( + std::string("dots3-note processor: the image token '") + marker + + "' has no id. Upstream reads it from `added_tokens.json` and raises " + "when it is missing (multimodal.py:86-90 @ 9035151d6); this port also " + "accepts `config.json`'s `" + key + + "`. Refusing rather than guessing an id: a marker the tokenizer does " + "not know is injected as ordinary text and the image is dropped."); + }; + require(cfg.image_token_id, "<|imgpad|>", "image_token_id"); + require(cfg.image_start_token_id, "<|img|>", "image_start_token_id"); + require(cfg.image_end_token_id, "<|endofimg|>", "image_end_token_id"); + return cfg; +} + +std::array Dots3NoteResizedSize(int64_t height, int64_t width, + int64_t factor, int64_t min_pixels, + int64_t max_pixels) { + // `processor.py:131-146` @ `9035151d6`, in upstream's own order. + if (std::min(height, width) < factor / 4) { + throw std::runtime_error( + "dots3-note processor: image height and width must be at least " + + std::to_string(factor / 4) + ", got " + std::to_string(height) + "x" + + std::to_string(width)); + } + if (std::min(height, width) <= 0 || + static_cast(std::max(height, width)) / + static_cast(std::min(height, width)) > + 200.0) { + throw std::runtime_error( + "dots3-note processor: image aspect ratio must be smaller than 200"); + } + int64_t rh = std::max(factor, RoundByFactor(height, factor)); + int64_t rw = std::max(factor, RoundByFactor(width, factor)); + const double hw = static_cast(height) * static_cast(width); + if (rh * rw > max_pixels) { + const double beta = std::sqrt(hw / static_cast(max_pixels)); + rh = std::max(factor, FloorByFactor(static_cast(height) / beta, factor)); + rw = std::max(factor, FloorByFactor(static_cast(width) / beta, factor)); + } else if (rh * rw < min_pixels) { + const double beta = std::sqrt(static_cast(min_pixels) / hw); + rh = CeilByFactor(static_cast(height) * beta, factor); + rw = CeilByFactor(static_cast(width) * beta, factor); + if (rh * rw > max_pixels) { + const double b2 = std::sqrt(static_cast(rh) * + static_cast(rw) / + static_cast(max_pixels)); + rh = std::max(factor, FloorByFactor(static_cast(rh) / b2, factor)); + rw = std::max(factor, FloorByFactor(static_cast(rw) / b2, factor)); + } + } + return {rh, rw}; +} + +std::string Dots3NoteImageProcessor::HashImage(const uint8_t* rgb, + int64_t height, + int64_t width) const { + return MultiModalHasher::HashImageRGB(cfg_.model_id, rgb, height, width); +} + +ImageKwargs Dots3NoteImageProcessor::ProcessImage(const uint8_t* rgb, + int64_t height, + int64_t width) const { + const int64_t patch = cfg_.patch_size; + const int64_t merge = cfg_.merge_size; + const int64_t tp = cfg_.temporal_patch_size; + const int64_t f = factor(); + + const std::array rs = + Dots3NoteResizedSize(height, width, f, cfg_.min_pixels, cfg_.max_pixels); + const int64_t rh = rs[0], rw = rs[1]; + if (rh != height || rw != width) { + // A genuine bicubic resize (`Image.Resampling.BICUBIC`, + // `processor.py:174`). NAMED, exactly as the Qwen3-VL processor beside this + // one names it: patchifying at the wrong grid would change the placeholder + // count and serve a well-shaped wrong prompt. + // + // THIS IS A CAPABILITY GAP, NOT A CORNER. `factor` is + // `patch_size * merge_size`, which on the released checkpoint is 28, and + // upstream ALWAYS resizes β€” so once W6b lifts the MoE ViT refusal, almost + // no real image clears this. It is owed by W8 (the MM front end brick that + // owns the processor), recorded under `## Owed` in + // `.agents/specs/dots3-note.md`, and tracked by issue #2537. Both this + // comment and the message below claimed that record before it existed; the + // fresh review of #2523 found the claim false and this is the repair. + throw std::runtime_error( + "Dots3NoteImageProcessor: image requires resize (" + + std::to_string(width) + "x" + std::to_string(height) + " -> " + + std::to_string(rw) + "x" + std::to_string(rh) + + "); the bicubic resize path is not ported (W6a uses conformant " + "images). Owed by W8 and tracked by issue #2537; see " + ".agents/specs/dots3-note.md `## Owed`."); + } + + const int64_t grid_h = rh / patch; + const int64_t grid_w = rw / patch; + const int64_t grid_t = 1; // ONE image; video grids are W7's + if (cfg_.pre_pixel_shuffle && (grid_h % merge != 0 || grid_w % merge != 0)) { + throw std::runtime_error( + "Dots3NoteImageProcessor: the " + std::to_string(grid_h) + "x" + + std::to_string(grid_w) + + " patch grid does not group into whole " + std::to_string(merge) + + "x" + std::to_string(merge) + + " blocks, which `pre_pixel_shuffle` requires (processor.py:185-197 @ " + "9035151d6)"); + } + const int64_t num_patches = grid_t * grid_h * grid_w; + const int64_t feat = 3 * tp * patch * patch; + + // Fused rescale + normalize, PER CHANNEL: + // (raw * rescale - mean) / std == (raw - mean/rescale) / (std/rescale) + // `processor.py:166-167`. + double shift[3], scale[3]; + for (int c = 0; c < 3; ++c) { + shift[c] = cfg_.image_mean[static_cast(c)] / cfg_.rescale_factor; + scale[c] = cfg_.image_std[static_cast(c)] / cfg_.rescale_factor; + } + + ImageKwargs out; + out.num_patches = num_patches; + out.patch_feature_dim = feat; + out.image_grid_thw = {grid_t, grid_h, grid_w}; + out.pixel_values_f32.resize(static_cast(num_patches * feat)); + out.pixel_values_bf16.resize(static_cast(num_patches * feat)); + + const int64_t rowstride = width * 3; // HWC uint8 source stride + // The column index is the same under both row orders: + // k = ((c * tp + t) * patch + ph) * patch + pw + // because both transposes end `..., C, tp, ph, pw` (`processor.py:196`, + // `:207`). Only the ROW index differs. + const auto emit = [&](int64_t r, int64_t src_h0, int64_t src_w0) { + for (int64_t c = 0; c < 3; ++c) { + for (int64_t t = 0; t < tp; ++t) { + for (int64_t ph = 0; ph < patch; ++ph) { + const int64_t H = src_h0 + ph; + for (int64_t pw = 0; pw < patch; ++pw) { + const int64_t W = src_w0 + pw; + const uint8_t raw = rgb[H * rowstride + W * 3 + c]; + const float v = static_cast( + (static_cast(raw) - shift[c]) / scale[c]); + const int64_t k = ((c * tp + t) * patch + ph) * patch + pw; + const size_t idx = static_cast(r * feat + k); + out.pixel_values_f32[idx] = v; + out.pixel_values_bf16[idx] = vt::F32ToBF16(v); + } + } + } + } + }; + + if (cfg_.pre_pixel_shuffle) { + // `reshape(t, tp, C, Gh, m, p, Gw, m, p).transpose(0,3,6,4,7,2,1,5,8)` + // (`processor.py:185-197`): row index + // r = ((gh * Gw + gw) * m + mh) * m + mw + const int64_t Gh = grid_h / merge, Gw = grid_w / merge; + for (int64_t gh = 0; gh < Gh; ++gh) { + for (int64_t gw = 0; gw < Gw; ++gw) { + for (int64_t mh = 0; mh < merge; ++mh) { + for (int64_t mw = 0; mw < merge; ++mw) { + const int64_t r = ((gh * Gw + gw) * merge + mh) * merge + mw; + emit(r, (gh * merge + mh) * patch, (gw * merge + mw) * patch); + } + } + } + } + } else { + // `reshape(t, tp, C, gh, p, gw, p).transpose(0,3,5,2,1,4,6)` + // (`processor.py:199-208`): plain row-major, r = gh * grid_w + gw. + for (int64_t gh = 0; gh < grid_h; ++gh) { + for (int64_t gw = 0; gw < grid_w; ++gw) { + emit(gh * grid_w + gw, gh * patch, gw * patch); + } + } + } + return out; +} + +} // namespace vllm::multimodal diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 5b3896bac..160051d1f 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -1239,6 +1239,24 @@ target_compile_definitions(test_dots3_note_attn PRIVATE # Same model-private-header arrangement as the scaffold gate above. target_include_directories(test_dots3_note_attn PRIVATE ${CMAKE_SOURCE_DIR}/src) +# dots3-note W6a (#2512, spec Β§4.11): the DENSE VISION TOWER against an +# INDEPENDENT double-precision reference written from vLLM `nvidia/vision.py` + +# `nvidia/vision_attention.py` at `9035151d6`, plus the config traps, the +# refuse-by-name table, the 235-tensor enumeration and the image processor. +# +# A CONSISTENCY gate and not a correctness one, and the file says so in its own +# header: spec Β§6.4 option B records that NO oracle for this model runs on any +# hardware this project owns, so nothing here is compared against vLLM and no +# performance number is claimable. CPU-only, no checkpoint download, no GPU. +vllm_cpp_add_test(test_dots3_note_vision vllm/models/test_dots3_note_vision.cpp) +target_compile_definitions(test_dots3_note_vision PRIVATE + DOTS3_NOTE_CKPT_FIXTURE_DIR="${CMAKE_CURRENT_SOURCE_DIR}/vllm/models/fixtures/dots3_note_prev") +# Same model-private-header arrangement as the two gates above: dots3_note.h and +# dots3_note_vision.h live under src/, because W1 ships nothing on the public ABI +# and include/vllm/ is a USER_USAGE_PREFIXES surface (#515). +target_include_directories(test_dots3_note_vision PRIVATE + ${CMAKE_SOURCE_DIR}/src ${CMAKE_CURRENT_SOURCE_DIR}/vllm/models) + # Nemotron-H (NemotronHForCausalLM) W3 STRUCTURAL gate (#517, # .agents/specs/nemotron-h-model.md Β§4 W3): registry-resolve + config descent off # the REAL released config.json (52-entry layers_block_type schedule, the legacy @@ -2180,6 +2198,20 @@ if(VLLM_CPP_SERVER) # fixture (one whose added tokens are the vision markers). vllm_cpp_add_test(test_openai_api_server_mm_forward vllm/entrypoints/openai/test_api_server_mm_forward.cpp) + # dots3-note W6a (#2512): the SERVED image request for the SECOND multimodal + # architecture, end to end through the same production chain. Its own target + # for the same reason the Qwen3-VL one has one: a different model (a registry + # LoadedModel carrying a dots3-note DENSE vision tower, loaded from a + # synthetic on-disk checkpoint) and a different tokenizer fixture (one whose + # added tokens are dots3-note's own image markers). + vllm_cpp_add_test(test_openai_api_server_dots3_mm_forward + vllm/entrypoints/openai/test_api_server_dots3_mm_forward.cpp) + target_compile_definitions(test_openai_api_server_dots3_mm_forward PRIVATE + DOTS3_NOTE_CKPT_FIXTURE_DIR="${CMAKE_CURRENT_SOURCE_DIR}/vllm/models/fixtures/dots3_note_prev") + # The tiny-checkpoint fixture header lives beside the model gates, and + # dots3_note.h is a MODEL-PRIVATE header under src/ (#515). + target_include_directories(test_openai_api_server_dots3_mm_forward PRIVATE + ${CMAKE_SOURCE_DIR}/src ${CMAKE_CURRENT_SOURCE_DIR}/vllm/models) # #1982: WHEN the first chat SSE frame reaches the client. Its own target so # the gated-runner fixture cannot perturb the keepalive suite's timings, and # inside this guard because it drives the production ApiServer dispatch, diff --git a/tests/vllm/entrypoints/openai/test_api_server_dots3_mm_forward.cpp b/tests/vllm/entrypoints/openai/test_api_server_dots3_mm_forward.cpp new file mode 100644 index 000000000..72f3eedca --- /dev/null +++ b/tests/vllm/entrypoints/openai/test_api_server_dots3_mm_forward.cpp @@ -0,0 +1,689 @@ +// dots3-note W6a (#2512) β€” THE SERVED IMAGE REQUEST, end to end. +// +// One OpenAI `image_url` chat request travels the whole production chain on a +// CPU queue over the real serving stack: +// +// ApiServer::handle_chat_completions +// -> OpenAIServingChat (the ARCHITECTURE-dispatched multimodal chat seam: +// `<|img|><|imgpad|><|endofimg|>` marker injection, tokenize, +// Dots3NoteImageProcessor, placeholder EXPANSION, mm_features) +// -> AsyncLLM::generate(MultiModalInputs) -> EngineCore +// -> Scheduler::schedule (encoder admission + budget) +// -> Executor -> GPUModelRunner::execute_model +// (the encoder step runs the DENSE VISION TOWER, the gather slices its +// rows, `EmbedMmDots3NoteForCausalLM` scatters them, `.mm` is set) +// -> ModelRegistry::Forward -> ForwardDots3NoteForCausalLM +// -> Dots3NoteModel::ForwardDevice, which reads `mm->inputs_embeds` +// +// WHY THIS FILE EXISTS RATHER THAN A UNIT TEST (AGENTS.md, "Nothing lands +// dead"). `test_dots3_note_vision.cpp` proves the tower computes the right +// numbers. It would pass just as well on a tree where nothing calls the tower: +// it constructs the weights and calls the forward itself. This file cannot. It +// enters through the HTTP dispatch on the server's default configuration and +// asserts that the pixels reached the model. +// +// THE WEIGHTS ARE SYNTHETIC AND THE TOKENS ARE NOT CHECKED, and on this row +// they never can be: `.agents/specs/dots3-note.md` Β§6.4 records option B β€” the +// checkpoint is 298.67 GB fp8 / 576.89 GB bf16 against 119-122 GiB hosts, so +// vLLM cannot be run on this model here and NO token-exact denominator exists. +// What this file gates is REACHABILITY and the shape of what flows: which stage +// ran, on how many rows, and that a request whose tower is owed is refused BY +// NAME rather than answered from the text path. +// +// THE LOAD-BEARING CASE IS THE TWO-IMAGE LOGPROB ONE. Status 200, +// `prompt_tokens` and `completion_tokens` all pass on a tree where the tower is +// replaced by a correctly SHAPED constant. The logprobs of the first generated +// token do not. +#include "vllm/entrypoints/openai/api_server.h" + +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +#include "dots3_note_tiny_fixture.h" +#include "vllm/config/multimodal.h" +#include "vllm/config/scheduler.h" +#include "vllm/entrypoints/openai/chat_mm.h" +#include "vllm/entrypoints/openai/mm_chat_registry.h" +#include "vllm/entrypoints/openai/serving_chat.h" +#include "vllm/entrypoints/openai/serving_completion.h" +#include "vllm/entrypoints/openai/serving_models.h" +#include "vllm/model_executor/model_loader/safetensors_reader.h" +#include "vllm/model_executor/models/dots3_note.h" +#include "vllm/model_executor/models/model_registry.h" +#include "vllm/sampling_params.h" +#include "vllm/tokenizer/tokenizer.h" +#include "vllm/transformers_utils/hf_config.h" +#include "vllm/v1/core/kv_cache_utils.h" +#include "vllm/v1/core/sched/scheduler.h" +#include "vllm/v1/engine/async_llm.h" +#include "vllm/v1/engine/input_processor.h" +#include "vllm/v1/engine/output_processor.h" +#include "vllm/v1/executor/executor.h" +#include "vllm/v1/kv_cache_interface.h" +#include "vllm/v1/worker/gpu/runner.h" +#include "vt/dtype.h" + +namespace oai = vllm::entrypoints::openai; +using nlohmann::json; +using dots3_tiny::TinyCheckpoint; +using dots3_tiny::TinySpec; +using oai::ApiServer; +using oai::ChatMessage; +using oai::OpenAIServingChat; +using oai::OpenAIServingCompletion; +using oai::OpenAIServingModels; +using vllm::HfConfig; +using vllm::SchedulerConfig; +using vllm::tok::Tokenizer; +using vllm::v1::AsyncLLM; +using vllm::v1::Executor; +using vllm::v1::get_request_block_hasher; +using vllm::v1::GPUModelRunner; +using vllm::v1::init_none_hash; +using vllm::v1::InputProcessor; +using vllm::v1::KVCacheConfig; +using vllm::v1::OutputProcessor; +using vllm::v1::Scheduler; +using vllm::v1::sha256_cbor; + +namespace { + +std::string FixtureDir() { return DOTS3_NOTE_CKPT_FIXTURE_DIR; } + +constexpr int kBlockSize = 16, kNumBlocks = 64, kMaxModelLen = 128; +// The architecture the model registry resolves this fixture to, and the one +// dots3-note's chat factory is registered under. +constexpr const char* kDots3Arch = "Dots3NoteForCausalLM"; +// An architecture NOTHING registers a chat seam for; the premise of the +// refusal case is asserted rather than assumed. +constexpr const char* kUnregisteredMmArch = "Dots3NoteNotRegisteredForCausalLM"; + +vt::Queue Q() { return vt::Queue{vt::Device{vt::DeviceType::kCPU, 0}, nullptr}; } + +// A BPE fixture whose ADDED tokens are dots3-note's three image markers, so the +// string the chat seam injects tokenizes to exactly [14, 15, 16]. +Tokenizer BuildFixture() { + static int counter = 0; + const std::string path = + (std::filesystem::temp_directory_path() / + ("vllm_dots3mm_tok_" + std::to_string(counter++) + ".json")).string(); + json doc; + doc["version"] = "1.0"; + doc["added_tokens"] = json::array( + {{{"id", dots3_tiny::kImgStartId}, {"content", "<|img|>"}, {"special", true}}, + {{"id", dots3_tiny::kImgPadId}, {"content", "<|imgpad|>"}, {"special", true}}, + {{"id", dots3_tiny::kImgEndId}, {"content", "<|endofimg|>"}, {"special", true}}}); + doc["normalizer"] = nullptr; + doc["pre_tokenizer"] = { + {"type", "Sequence"}, + {"pretokenizers", + json::array( + {{{"type", "Split"}, + {"pattern", + {{"Regex", + R"((?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\r\n\p{L}\p{N}]?[\p{L}\p{M}]+|\p{N}| ?[^\s\p{L}\p{M}\p{N}]+[\r\n]*|\s*[\r\n]+|\s+(?!\S)|\s+)"}}}, + {"behavior", "Isolated"}, + {"invert", false}}, + {{"type", "ByteLevel"}, + {"add_prefix_space", false}, + {"trim_offsets", false}, + {"use_regex", false}}})}}; + json vocab = {{"h", 0}, {"e", 1}, {"l", 2}, {"o", 3}, {"w", 4}, + {"r", 5}, {"d", 6}, {"Δ ", 7}, {"1", 8}, {"2", 9}, + {"ll", 10}, {"he", 11}, {"llo", 12}, {"hello", 13}}; + doc["model"] = {{"type", "BPE"}, + {"ignore_merges", false}, + {"vocab", vocab}, + {"merges", json::array({json::array({"l", "l"}), + json::array({"h", "e"}), + json::array({"ll", "o"}), + json::array({"he", "llo"})})}}; + std::ofstream(path, std::ios::binary) << doc.dump(); + Tokenizer tok = Tokenizer::FromHfJson(path); + std::remove(path.c_str()); + return tok; +} + +const Tokenizer& Fixture() { + static const Tokenizer tok = BuildFixture(); + return tok; +} + +// The chat prompt seam: concatenate the rendered contents. The mm seam has +// already replaced each image part with the marker string, so this is where the +// markers enter the prompt. +std::string ConcatChatPrompt( + const std::vector& messages, bool, + const std::vector&, + const nlohmann::ordered_json&) { + std::string p; + for (const ChatMessage& m : messages) + if (m.content.has_value()) p += *m.content; + return p; +} + +// The raw-RGB passthrough codec, the same shape `server_main.cpp` installs. +oai::ImageCodecFn RawRgbCodec() { + return [](const oai::DecodedMedia& media) -> oai::DecodedImageRgb { + REQUIRE(media.media_type == "image/x-raw-rgb"); + const std::size_t px = media.bytes.size() / 3; + const auto side = static_cast( + std::llround(std::sqrt(static_cast(px)))); + REQUIRE(static_cast(side * side * 3) == media.bytes.size()); + oai::DecodedImageRgb out; + out.rgb = media.bytes; + out.height = side; + out.width = side; + return out; + }; +} + +std::string EncodeBase64(const std::vector& raw) { + static const char* kAlpha = + "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"; + std::string out; + size_t i = 0; + for (; i + 2 < raw.size(); i += 3) { + const uint32_t v = (uint32_t(raw[i]) << 16) | (uint32_t(raw[i + 1]) << 8) | + uint32_t(raw[i + 2]); + out += kAlpha[(v >> 18) & 63]; + out += kAlpha[(v >> 12) & 63]; + out += kAlpha[(v >> 6) & 63]; + out += kAlpha[v & 63]; + } + if (i + 1 == raw.size()) { + const uint32_t v = uint32_t(raw[i]) << 16; + out += kAlpha[(v >> 18) & 63]; + out += kAlpha[(v >> 12) & 63]; + out += "=="; + } else if (i + 2 == raw.size()) { + const uint32_t v = (uint32_t(raw[i]) << 16) | (uint32_t(raw[i + 1]) << 8); + out += kAlpha[(v >> 18) & 63]; + out += kAlpha[(v >> 12) & 63]; + out += kAlpha[(v >> 6) & 63]; + out += '='; + } + return out; +} + +std::string ImageDataUri(int variant) { + return "data:image/x-raw-rgb;base64," + + EncodeBase64(dots3_tiny::FixtureImage(variant)); +} + +json ChatBody(int max_tokens, int variant, bool logprobs) { + json body = { + {"model", "test-model"}, + {"messages", + json::array({{{"role", "user"}, + {"content", + json::array({{{"type", "image_url"}, + {"image_url", {{"url", ImageDataUri(variant)}}}}, + {{"type", "text"}, {"text", "hello"}}})}}})}, + {"max_completion_tokens", max_tokens}, + {"temperature", 0.0}}; + if (logprobs) { + body["logprobs"] = true; + body["top_logprobs"] = 3; + } + return body; +} + +// The whole production serving stack over the tiny dots3-note, on a CPU queue. +struct MmServerHarness { + MmServerHarness(const HfConfig& c, vllm::LoadedModel& model, + const Tokenizer& tok) + : scheduler(MakeSchedulerConfig(), MakeKv(c), kBlockSize, + /*enable_caching=*/true), + runner(c, model, MakeKv(c), Q(), /*max_num_reqs=*/1, kMaxModelLen, + kMaxModelLen * 4), + executor(runner), + input_processor(tok, c), + output_processor(&tok), + async_engine(input_processor, scheduler, executor, output_processor, + Hasher()), + models("test-model"), + completion(async_engine, "test-model"), + chat(async_engine, "test-model", &ConcatChatPrompt, "hermes"), + server(completion, chat, models, "9.9.9") {} + + // The SAME topology `MakeDots3NoteKVCache` publishes for this architecture: + // ONE MLA group at the padded physical latent row. + static KVCacheConfig MakeKv(const HfConfig& c) { + return vllm::MakeDots3NoteKVCache(c, kBlockSize, kNumBlocks); + } + static SchedulerConfig MakeSchedulerConfig() { + SchedulerConfig cfg; + cfg.max_num_seqs = 1; + cfg.max_num_batched_tokens = kMaxModelLen * 4; + // Chunked prefill is ON because it is the PRODUCTION value, not because + // this file measures it: the budget is 512 against a seven-token prompt, so + // nothing ever chunks here and the straddling-item clamp is never reached. + cfg.enable_chunked_prefill = true; + cfg.max_model_len = kMaxModelLen; + cfg.watermark = 0.0; + cfg.max_num_encoder_input_tokens = kMaxModelLen * 4; + cfg.encoder_cache_size = kMaxModelLen * 4; + return cfg; + } + static vllm::v1::BlockHasher Hasher() { + static bool init = false; + if (!init) { + init_none_hash(sha256_cbor); + init = true; + } + return get_request_block_hasher(kBlockSize, sha256_cbor); + } + + // THE PRODUCTION INSTALL, not a copy of it. `server_main.cpp` builds the same + // context and calls the same function; what it adds on top is reading the + // architecture and the multimodal declaration off `LoadedEngine`. + oai::MultiModalChatInstall install(std::string_view architecture, + bool is_multimodal_model, + const TinyCheckpoint& ckpt, + std::ostream& log) { + oai::MultiModalChatContext ctx; + ctx.architecture = architecture; + ctx.model_dir = ckpt.dir(); + ctx.config_path = ckpt.config_path(); + ctx.served_model_name = "tiny-dots3-note"; + ctx.tokenizer = &Fixture(); + ctx.prompt_fn = &ConcatChatPrompt; + ctx.codec = RawRgbCodec(); + ctx.mm_config = &mm_cfg; + return oai::InstallMultiModalChatSeam(chat, is_multimodal_model, ctx, log); + } + + // Declared FIRST so it outlives `chat`: the seam's `BaseProcessingInfo` holds + // it by reference, exactly as the engine's own config is held in production. + vllm::MultiModalConfig mm_cfg; + Scheduler scheduler; + GPUModelRunner runner; + Executor executor; + InputProcessor input_processor; + OutputProcessor output_processor; + AsyncLLM async_engine; + OpenAIServingModels models; + OpenAIServingCompletion completion; + OpenAIServingChat chat; + ApiServer server; +}; + +// Everything a served case needs: the tiny checkpoint on disk, the config, and +// the model the REAL registry loader returned from it. +struct Served { + TinySpec spec; + TinyCheckpoint ckpt; + HfConfig config; + std::unique_ptr model; + + explicit Served(TinySpec s = TinySpec{}) + : spec(s), ckpt(FixtureDir(), s), config(vllm::LoadHfConfig(ckpt.config_path())) { + const std::vector arch{kDots3Arch}; + const vllm::ModelRegistration& reg = vllm::ModelRegistry::Resolve(arch); + std::vector shards; + shards.push_back(vllm::SafetensorsFile::Open(ckpt.weights_path())); + const vllm::ModelSource source = vllm::ModelSource::FromSafetensors(shards); + model = reg.factory->load_weights(reg, config, source); + vt::Queue q = Q(); + vllm::ModelRegistry::Prepare(*model, config, q); + } +}; + +} // namespace + +// --------------------------------------------------------------------------- +// 1. THE TWO REGISTRATIONS. The model registration says this model can EMBED +// image features; the chat registration says the server can BUILD them from +// an `image_url` part. Both are keyed on the architecture, and both are +// reached through the static library's `--whole-archive` INTERFACE β€” so a +// link that dropped either translation unit reads as an ABSENT registration +// rather than as a subtly wrong one. +// --------------------------------------------------------------------------- +TEST_CASE("dots3-note W6a: both halves of the multimodal seam are registered on the architecture") { + const std::vector arch{kDots3Arch}; + const vllm::ModelRegistration& reg = vllm::ModelRegistry::Resolve(arch); + REQUIRE(reg.factory != nullptr); + CHECK(reg.info.supports_multimodal); + CHECK(reg.factory->encode_mm != nullptr); + CHECK(reg.factory->embed_mm != nullptr); + // NOT M-RoPE: upstream's Dots3NoteForCausalLM is `SupportsMultiModal, + // SupportsPP` and not `SupportsMRoPE` (nvidia/multimodal.py:49 @ 9035151d6). + CHECK(reg.factory->mrope_prompt_positions == nullptr); + + const oai::MultiModalChatRegistration* seam = + oai::MultiModalChatRegistry::Find(kDots3Arch); + REQUIRE(seam != nullptr); + CHECK(seam->architecture == kDots3Arch); + CHECK(seam->make_seam != nullptr); + const std::vector archs = + oai::MultiModalChatRegistry::SupportedArchs(); + CHECK(std::find(archs.begin(), archs.end(), std::string_view(kDots3Arch)) != + archs.end()); + // TWO architectures now hold seams, which is the whole point of #2481: the + // registry is keyed, not a single entry with a filename check in front of it. + CHECK(std::find(archs.begin(), archs.end(), + std::string_view("Qwen3VLForConditionalGeneration")) != + archs.end()); + // The premise of the refusal case below, asserted rather than assumed. + CHECK(oai::MultiModalChatRegistry::Find(kUnregisteredMmArch) == nullptr); + + // And the predicate the runner reads is DERIVED from the two hooks, never + // stored. + const Served s; + CHECK(vllm::ModelRegistry::SupportsMmInputs(*s.model)); + CHECK_FALSE(vllm::ModelRegistry::UsesMrope(*s.model)); +} + +// --------------------------------------------------------------------------- +// 2. THE SERVED REQUEST. One image chat request, through HTTP dispatch. +// --------------------------------------------------------------------------- +TEST_CASE("dots3-note W6a: a served image chat request reaches the model forward") { + Served s; + MmServerHarness h(s.config, *s.model, Fixture()); + std::ostringstream log; + REQUIRE(h.install(kDots3Arch, /*is_multimodal_model=*/true, s.ckpt, log) == + oai::MultiModalChatInstall::kInstalled); + INFO("install log: ", log.str()); + // The install announced WHICH processor it built and from where, and it says + // out loud that the pyramid half is owed. + CHECK(log.str().find("dots3-note processor") != std::string::npos); + + const ApiServer::DispatchResult r = + h.server.handle_chat_completions(ChatBody(/*max_tokens=*/3, 0, false).dump()); + // A 500 here is the interesting failure and it is what this brick is about: + // before W6a the factory had no `encode_mm`, so `SupportsMmInputs` was false + // and the runner's whole multimodal arm was never entered. + INFO("body: ", r.body); + REQUIRE(r.status == 200); + const json j = json::parse(r.body); + CHECK(j.at("object") == "chat.completion"); + CHECK(j.at("usage").at("completion_tokens") == 3); + // The prompt the engine actually ran is the EXPANDED one: `<|img|>` + FOUR + // image tokens + `<|endofimg|>` + "hello". A seam that dropped the expansion + // would report 3 prompt tokens and still answer 200. + CHECK(j.at("usage").at("prompt_tokens") == 3 + dots3_tiny::kExpectedImageTokens); +} + +// --------------------------------------------------------------------------- +// 3. TWO DIFFERENT IMAGES GIVE TWO DIFFERENT FORWARDS. +// +// Case 2 proves the request arrives. It does NOT prove the pixels do: an +// encoder hook returning a correctly SHAPED constant satisfies every +// assertion there, and so does a scatter that wrote the token embedding it +// was about to overwrite. This case is the one that separates them, and it +// is why deleting the `Dots3NoteVisionForward` call is a mutation with +// something to detect rather than only a shape check to trip. +// +// Compared on LOGPROBS rather than on the sampled text: on a random tiny +// checkpoint the argmax over 17 vocabulary entries is saturated and does not +// have to move, while the float logprobs move for ANY change in the hidden +// state. Different `mm_hash` on each leg, so the encoder cache cannot answer +// the second from the first. +// --------------------------------------------------------------------------- +TEST_CASE("dots3-note W6a: two DIFFERENT images give two different forwards") { + Served s; + MmServerHarness h(s.config, *s.model, Fixture()); + std::ostringstream log; + REQUIRE(h.install(kDots3Arch, true, s.ckpt, log) == + oai::MultiModalChatInstall::kInstalled); + + const ApiServer::DispatchResult a = + h.server.handle_chat_completions(ChatBody(4, /*variant=*/0, true).dump()); + const ApiServer::DispatchResult b = + h.server.handle_chat_completions(ChatBody(4, /*variant=*/1, true).dump()); + INFO("a: ", a.body); + INFO("b: ", b.body); + REQUIRE(a.status == 200); + REQUIRE(b.status == 200); + + const json ja = json::parse(a.body); + const json jb = json::parse(b.body); + const json& la = ja.at("choices").at(0).at("logprobs").at("content").at(0); + const json& lb = jb.at("choices").at(0).at("logprobs").at("content").at(0); + MESSAGE("image A logprob0 ", la.dump()); + MESSAGE("image B logprob0 ", lb.dump()); + // Same prompt, same weights, greedy sampling: the ONLY difference between the + // two forwards is the pixels. If the logprobs match, the pixels did not reach + // the forward. + CHECK(la != lb); +} + +// --------------------------------------------------------------------------- +// 4. THE SAME IMAGE TWICE hits the encoder cache by mm_hash and still answers +// with the same shape β€” the tower runs once, both legs are served. +// --------------------------------------------------------------------------- +TEST_CASE("dots3-note W6a: the same image twice is served twice, and the second hits the encoder cache") { + Served s; + MmServerHarness h(s.config, *s.model, Fixture()); + std::ostringstream log; + REQUIRE(h.install(kDots3Arch, true, s.ckpt, log) == + oai::MultiModalChatInstall::kInstalled); + for (int i = 0; i < 2; ++i) { + const ApiServer::DispatchResult r = + h.server.handle_chat_completions(ChatBody(2, 0, false).dump()); + INFO("attempt ", i, " body: ", r.body); + REQUIRE(r.status == 200); + const json j = json::parse(r.body); + CHECK(j.at("usage").at("completion_tokens") == 2); + CHECK(j.at("usage").at("prompt_tokens") == + 3 + dots3_tiny::kExpectedImageTokens); + } +} + +// --------------------------------------------------------------------------- +// 5. A CHECKPOINT WHOSE VISION TOWER IS OWED REFUSES BY NAME, and the message +// names the block and the brick. +// +// THIS IS THE RELEASED CHECKPOINT'S CASE, at tiny scale: 17 of its 42 vision +// blocks are pyramid MoE, so its tower refuses at block 25. A port that +// served such a request would be serving a tower whose pyramid it silently +// skipped, on a row Β§6.4 records as having no oracle to catch it. +// +// The request still reaches the FORWARD path β€” the seam builds the features +// and the runner asks for the encoder β€” so this is the refusal firing where +// it can name what is missing, not the request being turned away earlier. +// --------------------------------------------------------------------------- +TEST_CASE("dots3-note W6a: a checkpoint with a PYRAMID vision block refuses the image BY NAME") { + TinySpec spec; + spec.v_pyramid = {-1, 4}; // block 1 is routed: W6b's + Served s(spec); + MmServerHarness h(s.config, *s.model, Fixture()); + std::ostringstream log; + // The chat FACTORY asks the tower's own refusal and throws, so the install + // reports kRefusing rather than wiring a seam that would take the engine down + // on the first image. That ordering is measured, not chosen: before the + // factory asked, the refusal fired inside the engine's busy loop, stopped + // `AsyncLLM`, and every LATER request β€” text ones included β€” came back 500. + REQUIRE(h.install(kDots3Arch, true, s.ckpt, log) == + oai::MultiModalChatInstall::kRefusing); + INFO("install log: ", log.str()); + CHECK(log.str().find("W6b") != std::string::npos); + + const ApiServer::DispatchResult r = + h.server.handle_chat_completions(ChatBody(2, 0, false).dump()); + INFO("body: ", r.body); + // A 200 here is the defect: it would mean the tower was skipped and the + // placeholder rows kept whatever the embedding table gave them, which is a + // fluent wrong answer. A 500 is the OTHER defect, and it is the one this + // ordering removes: an engine that died takes the text path with it. + CHECK(r.status == 400); + CHECK(r.body.find("W6b") != std::string::npos); + CHECK(r.body.find("MoE") != std::string::npos); + // ...and the TEXT path over the SAME server, AFTER the refused image request, + // still answers. This is the assertion the engine-fatal shape could not pass. + const json text = {{"model", "test-model"}, + {"messages", json::array({{{"role", "user"}, + {"content", "hello"}}})}, + {"max_completion_tokens", 2}, + {"temperature", 0.0}}; + const ApiServer::DispatchResult t = + h.server.handle_chat_completions(text.dump()); + INFO("text body: ", t.body); + CHECK(t.status == 200); + CHECK(t.body.find("W6b") == std::string::npos); +} + +// --------------------------------------------------------------------------- +// 6. THE WRONG ARCHITECTURE GETS NOTHING (#2475), through the same dispatch. +// This model directory carries a `preprocessor_config.json`, so an install +// that keyed on the FILE β€” which is what the server did before #2481 β€” would +// build a processor here and answer 200. Only a lookup that reads the +// architecture refuses. +// --------------------------------------------------------------------------- +TEST_CASE("dots3-note W6a: an architecture with no registered seam REFUSES the image by name") { + Served s; + MmServerHarness h(s.config, *s.model, Fixture()); + std::ostringstream log; + REQUIRE(h.install(kUnregisteredMmArch, true, s.ckpt, log) == + oai::MultiModalChatInstall::kRefusing); + INFO("install log: ", log.str()); + CHECK(log.str().find(kUnregisteredMmArch) != std::string::npos); + + const ApiServer::DispatchResult r = + h.server.handle_chat_completions(ChatBody(2, 0, false).dump()); + INFO("body: ", r.body); + CHECK(r.status == 400); + const json j = json::parse(r.body); + const std::string message = j.at("error").at("message").get(); + CHECK(message.find(kUnregisteredMmArch) != std::string::npos); + CHECK(message.find("multimodal input is not available") != std::string::npos); +} + +// --------------------------------------------------------------------------- +// 7. THE SEAM'S CEILING IS ONE IMAGE, and a second one is REFUSED with +// upstream's own message rather than silently dropped (#686). This seam +// locates a single image part, so `{"image": 1}` is its implemented arm +// stated as a number β€” and VIDEO and AUDIO are ABSENT from the map, which +// `context.py:414-415` reads as unsupported. +// --------------------------------------------------------------------------- +TEST_CASE("dots3-note W6a: the chat seam declares ONE image, and refuses a second") { + Served s; + MmServerHarness h(s.config, *s.model, Fixture()); + std::ostringstream log; + REQUIRE(h.install(kDots3Arch, true, s.ckpt, log) == + oai::MultiModalChatInstall::kInstalled); + + const json two = { + {"model", "test-model"}, + {"messages", + json::array({{{"role", "user"}, + {"content", + json::array({{{"type", "image_url"}, + {"image_url", {{"url", ImageDataUri(0)}}}}, + {{"type", "image_url"}, + {"image_url", {{"url", ImageDataUri(1)}}}}, + {{"type", "text"}, {"text", "hello"}}})}}})}, + {"max_completion_tokens", 2}, + {"temperature", 0.0}}; + const ApiServer::DispatchResult r = + h.server.handle_chat_completions(two.dump()); + INFO("body: ", r.body); + // 200 here would be the #686 defect: the second image dropped without a word. + CHECK(r.status == 400); + CHECK(r.body.find("image") != std::string::npos); +} + +// --------------------------------------------------------------------------- +// 8 and 9. THE REFUSAL PREDICATE AND THE ROUTE PREDICATE ARE THE SAME +// PREDICATE, or the engine-fatal cascade case 5 removed is still reachable +// (fresh review of #2523). +// +// Case 5 proves the shape works for ONE condition β€” a pyramid block. These +// two prove it holds for the conditions the ENCODER asserts on. Before this +// repair `Dots3NoteVisionRefusal` was a strict SUBSET of +// `EncodeMmDots3NoteForCausalLM`'s `VT_CHECK`s: an all-dense checkpoint whose +// `adapter_out_dim` is not the text width, or whose `adapter_merge_size` is +// not `spatial_merge_size`, INSTALLED cleanly, served text, and then threw +// inside the engine's busy loop on the first image β€” after which +// `AsyncLLM::errored_` is set for the life of the process +// (`async_llm.cpp:584-601`) and every later request, text included, is dead. +// +// Each case therefore asserts BOTH halves, exactly as case 5 does: HTTP 400 +// on the image, and HTTP 200 on a TEXT request sent afterwards on the SAME +// server. The second assertion is the one the pre-repair tree cannot pass. +// --------------------------------------------------------------------------- +TEST_CASE("dots3-note W6a: an adapter that does not land in the TEXT hidden space refuses at INSTALL") { + TinySpec spec; + // Every block is dense and every other key is conformant, so this checkpoint + // clears every refusal case 5 exercises. What it gets wrong is the ONE thing + // the encoder compares `adapter_out_dim` against: the text tower's width. + spec.v_adapter_out_override = spec.hidden + 8; + Served s(spec); + MmServerHarness h(s.config, *s.model, Fixture()); + std::ostringstream log; + REQUIRE(h.install(kDots3Arch, true, s.ckpt, log) == + oai::MultiModalChatInstall::kRefusing); + INFO("install log: ", log.str()); + CHECK(log.str().find("adapter_out_dim") != std::string::npos); + + const ApiServer::DispatchResult r = + h.server.handle_chat_completions(ChatBody(2, 0, false).dump()); + INFO("body: ", r.body); + CHECK(r.status == 400); + CHECK(r.body.find("adapter_out_dim") != std::string::npos); + + // THE ASSERTION THE PRE-REPAIR TREE FAILS. A 500 here β€” "request submitted to + // a stopped AsyncLLM" β€” is the cascade: the image request threw inside the + // busy loop and took the text path down with it. + const json text = {{"model", "test-model"}, + {"messages", json::array({{{"role", "user"}, + {"content", "hello"}}})}, + {"max_completion_tokens", 2}, + {"temperature", 0.0}}; + const ApiServer::DispatchResult t = + h.server.handle_chat_completions(text.dump()); + INFO("text body: ", t.body); + CHECK(t.status == 200); + CHECK(t.body.find("stopped") == std::string::npos); +} + +TEST_CASE("dots3-note W6a: an adapter merge that is not the PROMPT's merge refuses at INSTALL") { + TinySpec spec; + // `spatial_merge_size` stays 2, so the prompt side expands the placeholder to + // prod(grid)/4 = FOUR tokens; the adapter folds `adapter_merge_size**2` = ONE + // trunk token per row and emits SIXTEEN. Upstream keeps the two as + // independent keys with independent defaults, so this is a config a + // checkpoint can carry β€” not a shape the parse can rule out. + spec.v_adapter_merge_override = 1; + Served s(spec); + MmServerHarness h(s.config, *s.model, Fixture()); + std::ostringstream log; + REQUIRE(h.install(kDots3Arch, true, s.ckpt, log) == + oai::MultiModalChatInstall::kRefusing); + INFO("install log: ", log.str()); + CHECK(log.str().find("adapter_merge_size") != std::string::npos); + + const ApiServer::DispatchResult r = + h.server.handle_chat_completions(ChatBody(2, 0, false).dump()); + INFO("body: ", r.body); + CHECK(r.status == 400); + CHECK(r.body.find("adapter_merge_size") != std::string::npos); + CHECK(r.body.find("spatial_merge_size") != std::string::npos); + + const json text = {{"model", "test-model"}, + {"messages", json::array({{{"role", "user"}, + {"content", "hello"}}})}, + {"max_completion_tokens", 2}, + {"temperature", 0.0}}; + const ApiServer::DispatchResult t = + h.server.handle_chat_completions(text.dump()); + INFO("text body: ", t.body); + CHECK(t.status == 200); + CHECK(t.body.find("stopped") == std::string::npos); +} diff --git a/tests/vllm/models/dots3_note_tiny_fixture.h b/tests/vllm/models/dots3_note_tiny_fixture.h new file mode 100644 index 000000000..9956a7856 --- /dev/null +++ b/tests/vllm/models/dots3_note_tiny_fixture.h @@ -0,0 +1,492 @@ +// A TINY, COMPLETE dots3-note checkpoint on disk β€” language tower + DENSE +// vision tower β€” shared by the W6a tower gate and the W6a served-request gate +// (#2512). +// +// WHY A SHARED HEADER AND NOT A COPY. The two gates must agree on the geometry +// byte-for-byte: the tower gate measures the arithmetic at a grid, and the +// server gate asserts the PLACEHOLDER COUNT that same grid implies. Two +// hand-typed copies of "8x8 image, patch 2, merge 2 -> 4 placeholder tokens" are +// one edit away from disagreeing, and the disagreement would show up as a +// passing tower gate beside a server gate measuring a different model. +// +// WHY IT IS NEW RATHER THAN EXTRACTED FROM `test_dots3_note_attn.cpp`. That +// file's checkpoint builder is 5867 lines deep in one anonymous namespace and +// carries W3's double-precision attention reference with it. Lifting it would be +// a refactor of four bricks' evidence in a brick that is adding a tower. This +// header builds only what a LOAD needs, which is much less. +// +// The language geometry mirrors `test_dots3_note_attn.cpp`'s device bench and +// for the same measured reason (its review finding F1): `q_lora` 3 and +// `kv_lora` 2 over `hidden` 16 give the two Β§4-trap-5 rescales sqrt(16/3) and +// sqrt(16/2), which are DIFFERENT from each other and both far from 1, so a +// dropped or swapped rescale cannot hide. +#ifndef VLLM_TESTS_DOTS3_NOTE_TINY_FIXTURE_H_ +#define VLLM_TESTS_DOTS3_NOTE_TINY_FIXTURE_H_ + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +#include "vt/dtype.h" + +namespace dots3_tiny { + +// ── the geometry ──────────────────────────────────────────────────────────── +// +// Every dimension is the smallest that still exercises the branch it stands +// for: +// * vision `head_dim` 8, so the 2-D rope has TWO frequencies per spatial axis +// (`head_dim/2 = 4` split into a height half and a width half, +// vision.py:503-504 @ 9035151d6). At `head_dim` 4 there would be one +// frequency per axis and a swapped height/width axis could not show. +// * TWO dense vision blocks, so a block that read the previous block's +// weights would move the answer. +// * a 4x4 patch grid over a 2x2 merge, so the adapter folds FOUR rows into +// one and produces FOUR merger rows. One row would make a mis-sized scatter +// invisible. +// * `intermediate_size` 6 against `embed_dim` 16, so the SwiGLU is not square +// and a transposed gate/up merge refuses on shape instead of computing. +struct TinySpec { + // text tower + int64_t hidden = 16; + int64_t heads = 2; + int64_t qk_nope = 4; + int64_t qk_rope = 4; + int64_t v_head = 8; + int64_t q_lora = 3; + int64_t kv_lora = 2; + int64_t layers = 2; + int64_t vocab = 17; + int64_t inter = 10; + int64_t max_pos = 64; + int64_t index_topk = 32; + int64_t index_n_heads = 2; + int64_t index_head_dim = 6; + double rope_theta = 137.0; + double rms_eps = 1e-3; + + // vision tower + bool with_vision = true; + int64_t v_embed = 16; + int64_t v_heads = 2; + int64_t v_layers = 2; + int64_t v_inter = 6; + int64_t v_patch = 2; + int64_t v_merge = 2; + int64_t v_channels = 3; + int64_t v_temporal = 1; + double v_rms_eps = 1e-3; + // `pyramid_num_routed` written into the config. EMPTY means "no MoE at all"; + // a case that wants the W6b refusal sets one entry positive. + std::vector v_pyramid{-1, -1}; + std::string v_adapter_type = "patch_merger"; + bool v_pre_pixel_shuffle = true; + bool v_post_norm = true; + // THE TWO ADAPTER KEYS A CONFORMANT CHECKPOINT TIES TO SOMETHING ELSE, and + // the only reason either is settable. `adapter_out_dim` is the TEXT hidden + // size because the encoder's rows are scattered into the prompt; and + // `adapter_merge_size` is `spatial_merge_size` because the PROMPT side + // expands placeholders by that key while the tower folds by this one. A case + // that wants the install-time refusal for a checkpoint whose keys disagree + // sets one of these positive. Zero means "follow the conformant value". + int64_t v_adapter_out_override = 0; + int64_t v_adapter_merge_override = 0; + + int64_t v_head_dim() const { return v_embed / v_heads; } + int64_t v_patch_row() const { + return v_channels * v_temporal * v_patch * v_patch; + } + int64_t v_adapter_merge() const { + return v_adapter_merge_override > 0 ? v_adapter_merge_override : v_merge; + } + int64_t v_merged_dim() const { + return v_embed * v_adapter_merge() * v_adapter_merge(); + } + // The adapter lands in the TEXT hidden space; anything else cannot be + // scattered into the prompt. + int64_t v_adapter_out() const { + return v_adapter_out_override > 0 ? v_adapter_out_override : hidden; + } + int64_t qk_head_dim() const { return qk_nope + qk_rope; } +}; + +// The three vision marker ids. They are the tokenizer fixture's ADDED tokens in +// the server gate, and `config.json`'s `image_token_id` / `image_start_token_id` +// / `image_end_token_id` here, so the marker string the chat seam injects +// tokenizes to exactly one `<|imgpad|>` id the expansion can expand. +inline constexpr int32_t kImgStartId = 14; +inline constexpr int32_t kImgPadId = 15; +inline constexpr int32_t kImgEndId = 16; + +// An 8x8 RGB image over a 2-pixel patch and a 2x2 merge: `factor` is 4, the +// image needs no resize, the grid is (1, 4, 4) = 16 patches, and +// 16 / (2*2) = FOUR placeholder tokens. +inline constexpr int64_t kImageSide = 8; +inline constexpr int64_t kExpectedImageTokens = 4; + +// ── deterministic values ──────────────────────────────────────────────────── +inline uint64_t Mix(uint64_t x) { + x += 0x9E3779B97F4A7C15ULL; + x = (x ^ (x >> 30)) * 0xBF58476D1CE4E5B9ULL; + x = (x ^ (x >> 27)) * 0x94D049BB133111EBULL; + return x ^ (x >> 31); +} + +// Values already ROUNDED to the bf16 the checkpoint stores, so a comparison +// against a double reference measures the FORWARD rather than the weights' +// storage width. +inline std::vector Values(int64_t n, uint64_t seed, double amp, + double bias = 0.0) { + std::vector v(static_cast(n)); + for (int64_t i = 0; i < n; ++i) { + const double u = + static_cast(Mix(seed + static_cast(i)) >> 40) / + static_cast(1 << 24); + const float f = static_cast((u * 2.0 - 1.0) * amp + bias); + v[static_cast(i)] = + static_cast(vt::BF16ToF32(vt::F32ToBF16(f))); + } + return v; +} + +// ── safetensors ───────────────────────────────────────────────────────────── +struct StOut { + std::string name; + std::vector shape; + std::vector values; // already rounded to `dtype` + std::string dtype = "BF16"; +}; + +inline void WriteSafetensors(const std::vector& entries, + const std::string& path) { + nlohmann::json header = nlohmann::json::object(); + size_t off = 0; + for (const StOut& e : entries) { + size_t n = 1; + for (int64_t s : e.shape) n *= static_cast(s); + const size_t w = e.dtype == "F32" ? 4u : 2u; + header[e.name] = {{"dtype", e.dtype}, + {"shape", e.shape}, + {"data_offsets", {off, off + n * w}}}; + off += n * w; + } + const std::string hs = header.dump(); + std::ofstream out(path, std::ios::binary); + const uint64_t hlen = hs.size(); + out.write(reinterpret_cast(&hlen), 8); + out.write(hs.data(), static_cast(hs.size())); + for (const StOut& e : entries) { + for (double v : e.values) { + if (e.dtype == "F32") { + const float f = static_cast(v); + out.write(reinterpret_cast(&f), 4); + } else { + const uint16_t b = vt::F32ToBF16(static_cast(v)); + out.write(reinterpret_cast(&b), 2); + } + } + } +} + +// ── the config document ───────────────────────────────────────────────────── +// +// Built on the COMMITTED released `config.json`, with the geometry overridden. +// That is deliberate: all 36 keys `ParseDots3NoteParams` requires, and the whole +// W1 validation, still apply to this fixture, so a tiny config cannot pass +// through a hole the released one would have hit. +inline nlohmann::json TinyConfigDoc(const std::string& fixture_dir, + const TinySpec& s) { + nlohmann::json d; + { + std::ifstream in(fixture_dir + "/config.json"); + in >> d; + } + d["hidden_size"] = s.hidden; + d["num_hidden_layers"] = s.layers; + nlohmann::json lt = nlohmann::json::array(); + for (int64_t i = 0; i < s.layers; ++i) lt.push_back("full_attention"); + d["layer_types"] = lt; + d["num_attention_heads"] = s.heads; + d["num_key_value_heads"] = s.heads; + d["qk_nope_head_dim"] = s.qk_nope; + d["qk_rope_head_dim"] = s.qk_rope; + d["v_head_dim"] = s.v_head; + d["q_lora_rank"] = s.q_lora; + d["kv_lora_rank"] = s.kv_lora; + d["rope_theta"] = s.rope_theta; + d["rms_norm_eps"] = s.rms_eps; + d["max_position_embeddings"] = s.max_pos; + d["index_n_heads"] = s.index_n_heads; + d["index_head_dim"] = s.index_head_dim; + d["index_topk"] = s.index_topk; + d["indexer_rope_interleave"] = true; + // The SWA geometry is required by the parse even with zero sliding layers; + // `swa_kv_lora_rank == kv_lora_rank` keeps the PHYSICAL latent row equal to + // the logical one. + d["swa_num_attention_heads"] = 1; + d["swa_num_key_value_heads"] = 1; + d["swa_q_lora_rank"] = s.q_lora; + d["swa_kv_lora_rank"] = s.kv_lora; + d["swa_qk_nope_head_dim"] = s.qk_nope; + d["swa_qk_rope_head_dim"] = s.qk_rope; + d["swa_v_head_dim"] = s.v_head; + d["vocab_size"] = s.vocab; + d["intermediate_size"] = s.inter; + d["moe_intermediate_size"] = 6; + d["n_routed_experts"] = 4; + d["num_experts_per_tok"] = 2; + d["first_k_dense_replace"] = s.layers; // every layer DENSE + d["num_nextn_predict_layers"] = 0; // no MTP tail (W10 owns it) + d["tie_word_embeddings"] = false; + // The three image marker ids, so the processor can resolve them from + // `config.json` the way a converted checkpoint carries them. + d["image_token_id"] = kImgPadId; + d["image_start_token_id"] = kImgStartId; + d["image_end_token_id"] = kImgEndId; + + if (s.with_vision) { + nlohmann::json v = nlohmann::json::object(); + v["embed_dim"] = s.v_embed; + v["hidden_size"] = s.hidden; + v["intermediate_size"] = s.v_inter; + v["moe_intermediate_size"] = 4; + v["num_hidden_layers"] = s.v_layers; + v["num_attention_heads"] = s.v_heads; + v["num_channels"] = s.v_channels; + v["patch_size"] = s.v_patch; + v["spatial_merge_size"] = s.v_merge; + v["temporal_patch_size"] = s.v_temporal; + v["rms_norm_eps"] = s.v_rms_eps; + v["use_bias"] = false; + v["use_qk_norm"] = true; + v["is_causal"] = false; + v["post_norm"] = s.v_post_norm; + v["pre_pixel_shuffle"] = s.v_pre_pixel_shuffle; + v["pyramid_num_routed"] = s.v_pyramid; + v["capacity_factor"] = 2; + v["router_scoring_func"] = "sigmoid"; + v["router_scale"] = 1.0; + v["adapter_type"] = s.v_adapter_type; + v["adapter_in_dim"] = s.v_embed; + v["adapter_out_dim"] = s.v_adapter_out(); + v["adapter_merge_size"] = s.v_adapter_merge(); + d["vision_config"] = v; + } else { + d.erase("vision_config"); + } + // The AUDIO tower is W7's and this fixture ships none, so the key that would + // make the loader expect one is removed. + d.erase("audio_config"); + return d; +} + +// The `preprocessor_config.json` the chat seam's factory reads. Its geometry +// reproduces the `vision_config`'s, and `min_pixels`/`max_pixels` are chosen so +// `Dots3NoteResizedSize` is the IDENTITY on the fixture image β€” the bicubic +// resize path is a named residual, so a fixture that needed it would refuse. +inline nlohmann::json TinyPreprocessorDoc(const TinySpec& s) { + return nlohmann::json{ + {"patch_size", s.v_patch}, + {"temporal_patch_size", s.v_temporal}, + {"merge_size", s.v_merge}, + {"pre_pixel_shuffle", s.v_pre_pixel_shuffle}, + {"image_mean", {0.5, 0.45, 0.4}}, + {"image_std", {0.25, 0.3, 0.35}}, + {"min_pixels", 16}, + {"max_pixels", 1 << 20}}; +} + +inline nlohmann::json TinyAddedTokensDoc() { + return nlohmann::json{{"<|img|>", kImgStartId}, + {"<|imgpad|>", kImgPadId}, + {"<|endofimg|>", kImgEndId}}; +} + +// ── the tensors ───────────────────────────────────────────────────────────── +// +// The LANGUAGE half is exactly what `EnumerateDots3NoteTensors` claims for a +// config whose every layer is full-attention with a dense MLP, and the VISION +// half is exactly what `EnumerateDots3NoteVisionTensors` claims for an all-dense +// tower. A name this list gets wrong refuses the load by name rather than being +// skipped, which is what makes the fixture self-checking. +inline std::vector TinyEntries(const TinySpec& s, uint64_t seed = 7) { + const int64_t H = s.hidden, N = s.heads, QK = s.qk_head_dim(); + std::vector e; + uint64_t k = seed; + const auto next = [&k]() { return (k += 0x9E37ULL) * 1000003ULL; }; + // A norm weight sits around 1.0; a projection sits around 0. + const auto norm = [&](int64_t n) { return Values(n, next(), 0.25, 1.0); }; + const auto proj = [&](int64_t n) { return Values(n, next(), 0.5); }; + + e.push_back({"model.embed_tokens.weight", {s.vocab, H}, proj(s.vocab * H)}); + e.push_back({"model.norm.weight", {H}, norm(H)}); + e.push_back({"lm_head.weight", {s.vocab, H}, proj(s.vocab * H)}); + for (int64_t l = 0; l < s.layers; ++l) { + const std::string p = "model.layers." + std::to_string(l) + "."; + const std::string sa = p + "self_attn."; + e.push_back({p + "input_layernorm.weight", {H}, norm(H)}); + e.push_back({p + "post_attention_layernorm.weight", {H}, norm(H)}); + e.push_back({sa + "q_a_proj.weight", {s.q_lora, H}, proj(s.q_lora * H)}); + e.push_back({sa + "q_a_layernorm.weight", {s.q_lora}, norm(s.q_lora)}); + e.push_back({sa + "q_b_proj.weight", {N * QK, s.q_lora}, + proj(N * QK * s.q_lora)}); + e.push_back({sa + "kv_a_proj_with_mqa.weight", + {s.kv_lora + s.qk_rope, H}, proj((s.kv_lora + s.qk_rope) * H)}); + e.push_back({sa + "kv_a_layernorm.weight", {s.kv_lora}, norm(s.kv_lora)}); + e.push_back({sa + "kv_b_proj.weight", + {N * (s.qk_nope + s.v_head), s.kv_lora}, + proj(N * (s.qk_nope + s.v_head) * s.kv_lora)}); + e.push_back({sa + "o_proj.weight", {H, N * s.v_head}, proj(H * N * s.v_head)}); + e.push_back({sa + "g_proj.weight", {N, H}, proj(N * H)}); + e.push_back({sa + "k_rope_only_layernorm.weight", {s.qk_rope}, + norm(s.qk_rope)}); + e.push_back({sa + "indexer.wq_b.weight", + {s.index_n_heads * s.index_head_dim, s.q_lora}, + proj(s.index_n_heads * s.index_head_dim * s.q_lora)}); + e.push_back({sa + "indexer.wk.weight", {s.index_head_dim, H}, + proj(s.index_head_dim * H)}); + e.push_back({sa + "indexer.k_norm.weight", {s.index_head_dim}, + norm(s.index_head_dim)}); + e.push_back({sa + "indexer.k_norm.bias", {s.index_head_dim}, + Values(s.index_head_dim, next(), 0.1)}); + e.push_back({sa + "indexer.weights_proj.weight", {s.index_n_heads, H}, + proj(s.index_n_heads * H)}); + e.push_back({p + "mlp.gate_proj.weight", {s.inter, H}, proj(s.inter * H)}); + e.push_back({p + "mlp.up_proj.weight", {s.inter, H}, proj(s.inter * H)}); + e.push_back({p + "mlp.down_proj.weight", {H, s.inter}, proj(H * s.inter)}); + } + + if (!s.with_vision) return e; + + const int64_t E = s.v_embed, VI = s.v_inter, D = s.v_head_dim(); + const std::string vp = "vision_encoder."; + e.push_back({vp + "patch_embed.proj.weight", + {E, s.v_channels, s.v_patch, s.v_patch}, + proj(E * s.v_patch_row())}); + e.push_back({vp + "patch_embed.proj.bias", {E}, Values(E, next(), 0.2)}); + e.push_back({vp + "patch_embed.norm.weight", {E}, norm(E)}); + for (int64_t b = 0; b < s.v_layers; ++b) { + const std::string pre = vp + "blocks." + std::to_string(b) + "."; + const bool moe = b < static_cast(s.v_pyramid.size()) && + s.v_pyramid[static_cast(b)] > 0; + e.push_back({pre + "norm_1.weight", {E}, norm(E)}); + e.push_back({pre + "norm_2.weight", {E}, norm(E)}); + e.push_back({pre + "attn.qkv.weight", {3 * E, E}, proj(3 * E * E)}); + e.push_back({pre + "attn.proj.weight", {E, E}, proj(E * E)}); + e.push_back({pre + "attn.q_norm.weight", {D}, norm(D)}); + e.push_back({pre + "attn.k_norm.weight", {D}, norm(D)}); + if (moe) { + // Present so a REFUSAL case can load a checkpoint that really does carry + // a pyramid block, rather than one that merely says so in its config. + const int64_t ne = s.v_pyramid[static_cast(b)]; + e.push_back({pre + "mlp.gate_weight", {ne, E}, proj(ne * E)}); + e.push_back({pre + "mlp.router_bias", {ne}, Values(ne, next(), 0.1), + "F32"}); + for (int64_t x = 0; x < ne; ++x) { + const std::string ep = pre + "mlp.experts." + std::to_string(x) + "."; + e.push_back({ep + "fc1.weight", {4, E}, proj(4 * E)}); + e.push_back({ep + "fc2.weight", {E, 4}, proj(E * 4)}); + e.push_back({ep + "fc3.weight", {4, E}, proj(4 * E)}); + } + } else { + e.push_back({pre + "mlp.fc1.weight", {VI, E}, proj(VI * E)}); + e.push_back({pre + "mlp.fc2.weight", {E, VI}, proj(E * VI)}); + e.push_back({pre + "mlp.fc3.weight", {VI, E}, proj(VI * E)}); + } + } + if (s.v_post_norm) { + e.push_back({vp + "post_trunk_norm.weight", {E}, norm(E)}); + } + const int64_t M = s.v_merged_dim(), O = s.v_adapter_out(); + e.push_back({vp + "adapter.ln_q.weight", {E}, norm(E)}); + e.push_back({vp + "adapter.ln_q.bias", {E}, Values(E, next(), 0.1)}); + e.push_back({vp + "adapter.mlp.0.weight", {M, M}, proj(M * M)}); + e.push_back({vp + "adapter.mlp.0.bias", {M}, Values(M, next(), 0.1)}); + e.push_back({vp + "adapter.mlp.2.weight", {O, M}, proj(O * M)}); + e.push_back({vp + "adapter.mlp.2.bias", {O}, Values(O, next(), 0.1)}); + return e; +} + +// ── the checkpoint DIRECTORY ──────────────────────────────────────────────── +// +// A complete model directory: `config.json`, `model.safetensors`, +// `preprocessor_config.json` and `added_tokens.json`. The last two are what the +// PRODUCTION chat factory reads, so the server gate loads its processor off +// disk on the production path rather than being handed one pre-built. +class TinyCheckpoint { + public: + TinyCheckpoint(const std::string& fixture_dir, const TinySpec& spec, + uint64_t seed = 7) + : entries_(TinyEntries(spec, seed)) { + static int counter = 0; + static const unsigned salt = std::random_device{}(); + dir_ = std::filesystem::temp_directory_path() / + ("dots3_note_tiny_" + std::to_string(salt) + "_" + + std::to_string(counter++)); + std::filesystem::create_directories(dir_); + std::ofstream(dir_ / "config.json", std::ios::binary) + << TinyConfigDoc(fixture_dir, spec).dump(); + std::ofstream(dir_ / "preprocessor_config.json", std::ios::binary) + << TinyPreprocessorDoc(spec).dump(); + std::ofstream(dir_ / "added_tokens.json", std::ios::binary) + << TinyAddedTokensDoc().dump(); + WriteSafetensors(entries_, (dir_ / "model.safetensors").string()); + } + ~TinyCheckpoint() { + std::error_code ec; + std::filesystem::remove_all(dir_, ec); + } + TinyCheckpoint(const TinyCheckpoint&) = delete; + TinyCheckpoint& operator=(const TinyCheckpoint&) = delete; + + std::string dir() const { return dir_.string(); } + std::string config_path() const { return (dir_ / "config.json").string(); } + std::string weights_path() const { + return (dir_ / "model.safetensors").string(); + } + const std::vector& entries() const { return entries_; } + // The bf16-rounded values of one tensor, by name. Used by the tower gate to + // drive its DOUBLE reference from the SAME bytes the loader read, so the + // comparison measures the forward and not a second copy of the weights. + const std::vector& value_of(const std::string& name) const { + for (const StOut& e : entries_) + if (e.name == name) return e.values; + // AN UNKNOWN NAME IS A TEST DEFECT AND SAYS SO. Returning an empty vector + // made a typo or a renamed tensor read as a tensor of zero elements, which + // `ref::Tower` then INDEXES β€” undefined behaviour whose most likely shape + // is a reference driven from garbage that the gate compares against + // itself. An instrument must be able to report its own failure. + throw std::runtime_error( + "dots3_tiny::TinyCheckpoint::value_of: this fixture checkpoint has no " + "tensor named '" + name + "'"); + } + + private: + std::filesystem::path dir_; + std::vector entries_; +}; + +// The fixture image: HWC uint8, `kImageSide` square. `variant` picks a +// genuinely DIFFERENT image rather than a shifted one β€” a high-frequency +// sawtooth against a smooth vertical ramp β€” so the two disagree in every patch. +inline std::vector FixtureImage(int variant) { + std::vector rgb(static_cast(kImageSide * kImageSide * 3)); + for (size_t i = 0; i < rgb.size(); ++i) { + rgb[i] = variant == 0 + ? static_cast((i * 37 + 11) & 0xFF) + : static_cast(((i / (kImageSide * 3)) * 29) & 0xFF); + } + return rgb; +} + +} // namespace dots3_tiny + +#endif // VLLM_TESTS_DOTS3_NOTE_TINY_FIXTURE_H_ diff --git a/tests/vllm/models/test_dots3_note_scaffold.cpp b/tests/vllm/models/test_dots3_note_scaffold.cpp index 3df42b772..a138d97dc 100644 --- a/tests/vllm/models/test_dots3_note_scaffold.cpp +++ b/tests/vllm/models/test_dots3_note_scaffold.cpp @@ -218,17 +218,29 @@ TEST_CASE("dots3-note: the architecture resolves through the model registry") { CHECK(reg.factory->forward != nullptr); CHECK(reg.factory->make_kv_cache != nullptr); CHECK(reg.info.is_text_generation_model); - // `supports_multimodal` is FALSE, and it was TRUE until W5 (#699). Upstream - // does put this architecture in `_MULTIMODAL_MODELS` with image, video AND - // audio (registry.py:381, multimodal.py:82-87 β€” the three branches of - // `get_placeholder_str` at :80-88), which is why W1 set it β€” but - // that is a statement about UPSTREAM, and it only became misleading about - // THIS port once W5 and W5c made the released config loadable. There is no - // vision tower (W6), no audio tower (W7) and no multimodal front end (W8): - // `EnumerateDots3NoteTensors` claims not one tensor of either tower and - // `Dots3NoteDeferredTowers()` records all 2625 as deferrals. W8 flips it - // back, and the true -> false -> true trail is the honest record. - CHECK_FALSE(reg.info.supports_multimodal); + // `supports_multimodal` is TRUE AGAIN, and the true -> false -> true trail is + // the honest record of what this port could back at each point (#699, #2512). + // W1 set it TRUE because upstream registers this architecture in + // `_MULTIMODAL_MODELS` with image, video AND audio (registry.py:381, + // multimodal.py:82-87) β€” a statement about UPSTREAM. W5 set it FALSE, because + // making the released config loadable turned that into a claim about THIS + // port that this port could not honour: there was no vision tower, no audio + // tower and no multimodal front end at all. + // + // W6a backs it. `kDots3NoteFactory` now carries `encode_mm` and `embed_mm`, + // `Dots3NoteForCausalLM` has its own `REGISTER_VLLM_MM_CHAT` translation + // unit, and a served `image_url` chat request reaches the model forward. The + // two assertions below are what makes this a CAPABILITY claim rather than a + // flag: `SupportsMmInputs` is DERIVED from the two hooks, never stored. + CHECK(reg.info.supports_multimodal); + CHECK(reg.factory->encode_mm != nullptr); + CHECK(reg.factory->embed_mm != nullptr); + // NOT an M-RoPE model: upstream's `Dots3NoteForCausalLM` is + // `SupportsMultiModal, SupportsPP` and NOT `SupportsMRoPE` + // (nvidia/multimodal.py:49 @ 9035151d6), so the runner hands it the ordinary + // 1-D positions. A non-null hook here would make the runner compute 3-D + // positions this forward never reads. + CHECK(reg.factory->mrope_prompt_positions == nullptr); // Both attention classes page the same MLA cache; the sliding half is a // window on it, not a recurrent state. CHECK_FALSE(reg.info.is_hybrid); @@ -1124,7 +1136,11 @@ TEST_CASE("dots3-note W2: the two tower files are NAMED W6/W7 deferrals, all 262 REQUIRE(towers.size() == 2); CHECK(std::string(towers[0].prefix) == "vision_encoder."); CHECK(std::string(towers[0].file) == "model-vision.safetensors"); - CHECK(std::string(towers[0].brick) == "W6"); + // W6a (#2512) landed the DENSE half of this tower, so the deferral that + // remains is the PYRAMID half and the brick moved with it. The COUNT did not + // move and neither did the prefix: a tower whose refusal fires is not partly + // loaded, so all 2195 `vision_encoder.*` names still land in this bucket. + CHECK(std::string(towers[0].brick) == "W6b"); CHECK(std::string(towers[1].prefix) == "audio_encoder."); CHECK(std::string(towers[1].file) == "model-audio.safetensors"); CHECK(std::string(towers[1].brick) == "W7"); @@ -1658,7 +1674,7 @@ TEST_CASE("dots3-note: the unported arms REFUSE BY NAME") { // W2: the message distinguishes UNKNOWN from DEFERRED by printing the // deferral table, so a reader is not left guessing which of the two the // loader thinks it hit. - CHECK_MESSAGE(refusal.find("vision_encoder.* (W6)") != std::string::npos, + CHECK_MESSAGE(refusal.find("vision_encoder.* (W6b)") != std::string::npos, "the refusal does not name the vision deferral: " << refusal); CHECK_MESSAGE(refusal.find("audio_encoder.* (W7)") != std::string::npos, "the refusal does not name the audio deferral: " << refusal); diff --git a/tests/vllm/models/test_dots3_note_vision.cpp b/tests/vllm/models/test_dots3_note_vision.cpp new file mode 100644 index 000000000..f6f8f389b --- /dev/null +++ b/tests/vllm/models/test_dots3_note_vision.cpp @@ -0,0 +1,907 @@ +// dots3-note W6a β€” the DENSE vision tower, against an INDEPENDENT +// double-precision reference (#2512, `.agents/specs/dots3-note.md` Β§4.11). +// +// WHAT THIS GATE IS, IN ITS OWN WORDS. Β§6.4 of the spec records option B, +// decided 2026-08-15: the checkpoint is 298.67 GB fp8 / 576.89 GB bf16 against +// 119-122 GiB hosts, so vLLM cannot be run on it on any hardware this project +// owns and NO DENOMINATOR EXISTS. This is therefore a CONSISTENCY gate. It +// establishes that two implementations of the same formula agree. It does NOT +// establish that either matches vLLM, and no performance number is claimable on +// any axis while B holds. Nothing below claims otherwise. +// +// The reference is written from `vllm/models/dots3_note/nvidia/vision.py` and +// `nvidia/vision_attention.py` read in `~/_git/vllm` at **`9035151d6`** β€” the +// merge of vllm#51255 β€” and shares NO helper with the implementation. It is a +// scalar `double` loop with its own GEMM, its own softmax, its own rope and its +// own norms; the implementation is `vt::MatmulBT` / `vt::RmsNorm` / +// `vt::RopeFromCache` / `vt::AttentionDenseFlash` over bf16 device buffers +// through the shared seams. Every anchor names that SHA because upstream has +// already moved under this row: `vision_attention.py` is 477 lines at +// `9035151d6` and 494 at vLLM `main` `7a100bb61`. +// +// THE ONE FORMULA DIFFERENCE, and why the reference does not copy it. Upstream's +// `RMSNorm.forward` (`vision.py:112-114`) casts the normalized value back to the +// ACTIVATION dtype before multiplying by the weight; `vt::RmsNorm` keeps f32 +// through that multiply. At infinite precision the two are the same function, +// so the reference β€” which is double throughout β€” is the algebra BOTH implement +// and the tolerance below covers our bf16 storage AND upstream's intermediate +// cast together. Copying the cast into the reference would make the reference +// agree with a rounding choice instead of with the maths. +#include "vllm/model_executor/models/dots3_note_vision.h" + +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +#include "dots3_note_tiny_fixture.h" +#include "vllm/model_executor/model_loader/safetensors_reader.h" +#include "vllm/model_executor/models/dots3_note.h" +#include "vllm/model_executor/models/model_registry.h" +#include "vllm/multimodal/dots3_note_processor.h" +#include "vllm/transformers_utils/hf_config.h" +#include "vt/backend.h" +#include "vt/dtype.h" + +namespace { + +using dots3_tiny::TinyCheckpoint; +using dots3_tiny::TinySpec; +using vllm::Dots3NoteVisionForward; +using vllm::Dots3NoteVisionParams; +using vllm::Dots3NoteVisionPosIds; +using vllm::Dots3NoteVisionRefusal; +using vllm::Dots3NoteVisionWeights; +using vllm::HfConfig; +using vllm::LoadHfConfig; +using vllm::ParseDots3NoteVisionParams; + +std::string FixtureDir() { return DOTS3_NOTE_CKPT_FIXTURE_DIR; } + +nlohmann::json ReleasedConfigDoc() { + std::ifstream in(FixtureDir() + "/config.json"); + REQUIRE_MESSAGE(in.good(), "cannot open " << FixtureDir() << "/config.json"); + nlohmann::json j; + in >> j; + return j; +} + +// A throwaway `config.json` holding an arbitrary document, so a case can drive +// the REAL `LoadHfConfig` -> `ParseDots3NoteVisionParams` path rather than +// building an `HfConfig` by hand. +class TempConfig { + public: + explicit TempConfig(const nlohmann::json& doc) { + static int counter = 0; + static const unsigned salt = std::random_device{}(); + dir_ = std::filesystem::temp_directory_path() / + ("dots3_vision_cfg_" + std::to_string(salt) + "_" + + std::to_string(counter++)); + std::filesystem::create_directories(dir_); + std::ofstream(dir_ / "config.json", std::ios::binary) << doc.dump(); + } + ~TempConfig() { + std::error_code ec; + std::filesystem::remove_all(dir_, ec); + } + std::string path() const { return (dir_ / "config.json").string(); } + + private: + std::filesystem::path dir_; +}; + +Dots3NoteVisionParams ParseDoc(const nlohmann::json& doc) { + const TempConfig cfg(doc); + return ParseDots3NoteVisionParams(LoadHfConfig(cfg.path())); +} + +// ═══════════════════════════════════════════════════════════════════════════ +// THE INDEPENDENT REFERENCE. Every line below is written from the upstream +// source at `9035151d6`; it calls nothing the implementation calls. +// ═══════════════════════════════════════════════════════════════════════════ +namespace ref { + +// out[M,N] = x[M,K] @ w[N,K]^T (+ bias). Plain triple loop, double accumulator. +std::vector Linear(const std::vector& x, + const std::vector& w, + const std::vector* bias, int64_t M, + int64_t K, int64_t N) { + std::vector out(static_cast(M * N), 0.0); + for (int64_t m = 0; m < M; ++m) { + for (int64_t n = 0; n < N; ++n) { + double acc = bias != nullptr ? (*bias)[static_cast(n)] : 0.0; + for (int64_t k = 0; k < K; ++k) { + acc += x[static_cast(m * K + k)] * + w[static_cast(n * K + k)]; + } + out[static_cast(m * N + n)] = acc; + } + } + return out; +} + +// `RMSNorm.forward` (vision.py:107-124) and `_RMSNorm` (vision_attention.py:97-110), +// which are the same function: `x * rsqrt(mean(x^2) + eps) * weight`. The +// intermediate `.type_as(x)` is a no-op in double (see the file header). +std::vector Rms(const std::vector& x, + const std::vector& w, int64_t rows, int64_t dim, + double eps) { + std::vector out(x.size()); + for (int64_t r = 0; r < rows; ++r) { + double ss = 0.0; + for (int64_t c = 0; c < dim; ++c) { + const double v = x[static_cast(r * dim + c)]; + ss += v * v; + } + const double inv = 1.0 / std::sqrt(ss / static_cast(dim) + eps); + for (int64_t c = 0; c < dim; ++c) { + out[static_cast(r * dim + c)] = + x[static_cast(r * dim + c)] * inv * w[static_cast(c)]; + } + } + return out; +} + +// `nn.LayerNorm` with weight and bias, the adapter's `ln_q` (vision.py:466). +std::vector LayerNorm(const std::vector& x, + const std::vector& w, + const std::vector& b, int64_t rows, + int64_t dim, double eps) { + std::vector out(x.size()); + for (int64_t r = 0; r < rows; ++r) { + double mean = 0.0; + for (int64_t c = 0; c < dim; ++c) mean += x[static_cast(r * dim + c)]; + mean /= static_cast(dim); + double var = 0.0; + for (int64_t c = 0; c < dim; ++c) { + const double d = x[static_cast(r * dim + c)] - mean; + var += d * d; + } + var /= static_cast(dim); + const double inv = 1.0 / std::sqrt(var + eps); + for (int64_t c = 0; c < dim; ++c) { + out[static_cast(r * dim + c)] = + (x[static_cast(r * dim + c)] - mean) * inv * + w[static_cast(c)] + + b[static_cast(c)]; + } + } + return out; +} + +// `get_pos_ids_by_grid` (vision.py:566-603), written straight from the reshape / +// permute / flatten upstream spells, rather than from the loop the +// implementation collapsed it into. +std::vector> PosIds(int64_t t, int64_t h, int64_t w, + int64_t rope_merge) { + // hpos[i][j] = i ; wpos[i][j] = j, both [h, w] + std::vector hp(static_cast(h * w)); + std::vector wp(static_cast(h * w)); + for (int64_t i = 0; i < h; ++i) { + for (int64_t j = 0; j < w; ++j) { + hp[static_cast(i * w + j)] = i; + wp[static_cast(i * w + j)] = j; + } + } + // reshape(h/m, m, w/m, m) then permute(0, 2, 1, 3) then flatten. + const int64_t m = rope_merge; + const auto regroup = [&](const std::vector& src) { + std::vector out; + out.reserve(src.size()); + for (int64_t a = 0; a < h / m; ++a) { + for (int64_t c = 0; c < w / m; ++c) { + for (int64_t b = 0; b < m; ++b) { + for (int64_t dd = 0; dd < m; ++dd) { + // index into the [h/m, m, w/m, m] view: (a, b, c, dd) + const int64_t row = a * m + b; + const int64_t col = c * m + dd; + out.push_back(src[static_cast(row * w + col)]); + } + } + } + } + return out; + }; + const std::vector hf = regroup(hp); + const std::vector wf = regroup(wp); + std::vector> out; + for (int64_t f = 0; f < t; ++f) { + for (size_t i = 0; i < hf.size(); ++i) out.push_back({hf[i], wf[i]}); + } + return out; +} + +// `apply_rotary_pos_emb_vision` (vision_attention.py:33-49) applied to ONE +// [L, nh, hd] tensor, with the frequency table built as +// `VisionRotaryEmbedding(hd // 2)` (vision_attention.py:60-89). +void ApplyRope(std::vector* x, + const std::vector>& pos, int64_t L, + int64_t nh, int64_t hd) { + const int64_t dim = hd / 2; // the table's own `dim` + const int64_t nf = dim / 2; // frequencies per spatial axis + for (int64_t l = 0; l < L; ++l) { + // freqs[l] is [2, nf] flattened to [dim]; cos/sin are then REPEATED to hd + // as [f | f] (`.repeat(1, 1, 2)` at vision_attention.py:46-47). + std::vector c(static_cast(hd)), s(static_cast(hd)); + for (int64_t axis = 0; axis < 2; ++axis) { + for (int64_t i = 0; i < nf; ++i) { + const double invf = + 1.0 / std::pow(10000.0, static_cast(2 * i) / + static_cast(dim)); + const double ang = + static_cast(pos[static_cast(l)][ + static_cast(axis)]) * invf; + const int64_t k = axis * nf + i; + c[static_cast(k)] = std::cos(ang); + c[static_cast(dim + k)] = std::cos(ang); + s[static_cast(k)] = std::sin(ang); + s[static_cast(dim + k)] = std::sin(ang); + } + } + for (int64_t h = 0; h < nh; ++h) { + const size_t base = static_cast((l * nh + h) * hd); + std::vector in(x->begin() + static_cast(base), + x->begin() + static_cast(base + hd)); + for (int64_t d = 0; d < hd; ++d) { + // rotate_half: (-x2, x1) over the two halves (vision_attention.py:33-36) + const double rot = d < dim ? -in[static_cast(d + dim)] + : in[static_cast(d - dim)]; + (*x)[base + static_cast(d)] = + in[static_cast(d)] * c[static_cast(d)] + + rot * s[static_cast(d)]; + } + } + } +} + +// `VisionAttentionV2.forward` (vision_attention.py:210-239) for ONE window: +// bidirectional, scaled by 1/sqrt(head_dim), softmax in f32 (here, double). +std::vector Attention(const std::vector& q, + const std::vector& k, + const std::vector& v, int64_t L, + int64_t nh, int64_t hd) { + std::vector out(static_cast(L * nh * hd), 0.0); + const double scale = 1.0 / std::sqrt(static_cast(hd)); + for (int64_t h = 0; h < nh; ++h) { + for (int64_t i = 0; i < L; ++i) { + std::vector sc(static_cast(L)); + double mx = -1e300; + for (int64_t j = 0; j < L; ++j) { + double acc = 0.0; + for (int64_t d = 0; d < hd; ++d) { + acc += q[static_cast((i * nh + h) * hd + d)] * + k[static_cast((j * nh + h) * hd + d)]; + } + sc[static_cast(j)] = acc * scale; + mx = std::max(mx, sc[static_cast(j)]); + } + double sum = 0.0; + for (int64_t j = 0; j < L; ++j) { + sc[static_cast(j)] = std::exp(sc[static_cast(j)] - mx); + sum += sc[static_cast(j)]; + } + for (int64_t j = 0; j < L; ++j) { + const double p = sc[static_cast(j)] / sum; + for (int64_t d = 0; d < hd; ++d) { + out[static_cast((i * nh + h) * hd + d)] += + p * v[static_cast((j * nh + h) * hd + d)]; + } + } + } + } + return out; +} + +double Silu(double x) { return x / (1.0 + std::exp(-x)); } +// `nn.GELU()` with no `approximate=` is the EXACT erf gelu. +double GeluErf(double x) { + return 0.5 * x * (1.0 + std::erf(x / std::sqrt(2.0))); +} + +// `DotsMoEVitModel.forward` (vision.py:634-677), the all-DENSE path. +std::vector Tower(const TinySpec& s, const TinyCheckpoint& ck, + const std::vector& pixels, int64_t t, + int64_t gh, int64_t gw) { + const int64_t E = s.v_embed, nh = s.v_heads, hd = s.v_head_dim(); + const int64_t L = t * gh * gw, P = s.v_patch_row(), VI = s.v_inter; + const double eps = s.v_rms_eps; + const std::string vp = "vision_encoder."; + + // patch_embed: the Conv2d over one non-overlapping patch IS a Linear over the + // flattened patch row, then RMSNorm (vision.py:317-331). + std::vector hidden = + Linear(pixels, ck.value_of(vp + "patch_embed.proj.weight"), + &ck.value_of(vp + "patch_embed.proj.bias"), L, P, E); + hidden = Rms(hidden, ck.value_of(vp + "patch_embed.norm.weight"), L, E, eps); + + const int64_t rope_merge = + s.v_pre_pixel_shuffle ? (s.v_merge > 1 ? s.v_merge : 2) : 1; + const std::vector> pos = PosIds(t, gh, gw, rope_merge); + + for (int64_t b = 0; b < s.v_layers; ++b) { + const std::string pre = vp + "blocks." + std::to_string(b) + "."; + // `apply_vision_attention_residual`: hidden + attn(norm_1(hidden)) + const std::vector n1 = + Rms(hidden, ck.value_of(pre + "norm_1.weight"), L, E, eps); + const std::vector qkv = + Linear(n1, ck.value_of(pre + "attn.qkv.weight"), nullptr, L, E, 3 * E); + // `.reshape(L, 3, nh, -1).permute(1, 0, 2, 3).unbind(0)` + std::vector qh(static_cast(L * E)); + std::vector kh(static_cast(L * E)); + std::vector vh(static_cast(L * E)); + for (int64_t l = 0; l < L; ++l) { + for (int64_t c = 0; c < E; ++c) { + qh[static_cast(l * E + c)] = + qkv[static_cast(l * 3 * E + c)]; + kh[static_cast(l * E + c)] = + qkv[static_cast(l * 3 * E + E + c)]; + vh[static_cast(l * E + c)] = + qkv[static_cast(l * 3 * E + 2 * E + c)]; + } + } + // Q/K NORM FIRST, ROPE SECOND (vision_attention.py:161-165). The order is + // silent when swapped: same shapes, same magnitudes, different numbers. + qh = Rms(qh, ck.value_of(pre + "attn.q_norm.weight"), L * nh, hd, eps); + kh = Rms(kh, ck.value_of(pre + "attn.k_norm.weight"), L * nh, hd, eps); + ApplyRope(&qh, pos, L, nh, hd); + ApplyRope(&kh, pos, L, nh, hd); + const std::vector ao = Attention(qh, kh, vh, L, nh, hd); + const std::vector proj = + Linear(ao, ck.value_of(pre + "attn.proj.weight"), nullptr, L, E, E); + for (size_t i = 0; i < hidden.size(); ++i) hidden[i] += proj[i]; + + // hidden + mlp(norm_2(hidden)), `fc2(silu(fc1(x)) * fc3(x))` + const std::vector n2 = + Rms(hidden, ck.value_of(pre + "norm_2.weight"), L, E, eps); + const std::vector g = + Linear(n2, ck.value_of(pre + "mlp.fc1.weight"), nullptr, L, E, VI); + const std::vector u = + Linear(n2, ck.value_of(pre + "mlp.fc3.weight"), nullptr, L, E, VI); + std::vector act(g.size()); + for (size_t i = 0; i < g.size(); ++i) act[i] = Silu(g[i]) * u[i]; + const std::vector down = + Linear(act, ck.value_of(pre + "mlp.fc2.weight"), nullptr, L, VI, E); + for (size_t i = 0; i < hidden.size(); ++i) hidden[i] += down[i]; + } + + if (s.v_post_norm) { + hidden = Rms(hidden, ck.value_of(vp + "post_trunk_norm.weight"), L, E, eps); + } + + // `PatchMergerAdapter.forward` (vision.py:474-484): ln_q over the per-token + // dim at a HARD-CODED eps of 1e-6, then `reshape(-1, merged_dim)`, then the + // two-layer MLP with an exact-erf GELU between. + const std::vector lnq = + LayerNorm(hidden, ck.value_of(vp + "adapter.ln_q.weight"), + ck.value_of(vp + "adapter.ln_q.bias"), L, E, 1e-6); + const int64_t M = s.v_merged_dim(), O = s.v_adapter_out(); + const int64_t Nm = L * E / M; + std::vector f1 = + Linear(lnq, ck.value_of(vp + "adapter.mlp.0.weight"), + &ck.value_of(vp + "adapter.mlp.0.bias"), Nm, M, M); + for (double& x : f1) x = GeluErf(x); + return Linear(f1, ck.value_of(vp + "adapter.mlp.2.weight"), + &ck.value_of(vp + "adapter.mlp.2.bias"), Nm, M, O); +} + +} // namespace ref + +// The whole loaded model, built once per case through the REAL registry over the +// REAL loader β€” never a hand-built weights struct. +struct Bench { + TinySpec spec; + TinyCheckpoint ckpt; + HfConfig config; + std::unique_ptr model; + + explicit Bench(TinySpec s = TinySpec{}) + : spec(s), + ckpt(FixtureDir(), s), + config(LoadHfConfig(ckpt.config_path())) { + const std::vector arch{"Dots3NoteForCausalLM"}; + const vllm::ModelRegistration& reg = vllm::ModelRegistry::Resolve(arch); + std::vector shards; + shards.push_back(vllm::SafetensorsFile::Open(ckpt.weights_path())); + const vllm::ModelSource source = vllm::ModelSource::FromSafetensors(shards); + model = reg.factory->load_weights(reg, config, source); + } +}; + +// The bf16 patch rows the tower consumes, widened to double, so the reference +// and the implementation start from the SAME bytes and the comparison measures +// the forward rather than the processor. +std::vector WidenBf16(const std::vector& bits) { + std::vector out(bits.size()); + for (size_t i = 0; i < bits.size(); ++i) + out[i] = static_cast(vt::BF16ToF32(bits[i])); + return out; +} + +double MaxAbs(const std::vector& v) { + double m = 0.0; + for (double x : v) m = std::max(m, std::abs(x)); + return m; +} + +} // namespace + +// --------------------------------------------------------------------------- +// 1. THE RELEASED `vision_config` resolves to the geometry the checkpoint's own +// shard index carries. Every number here was read from the COMMITTED +// fixture, not from the issue text. +// --------------------------------------------------------------------------- +TEST_CASE("dots3-note W6a: the RELEASED vision_config resolves to the measured geometry") { + const Dots3NoteVisionParams v = ParseDoc(ReleasedConfigDoc()); + REQUIRE(v.present); + CHECK(v.embed_dim == 1536); + CHECK(v.num_attention_heads == 24); + CHECK(v.head_dim() == 64); + CHECK(v.num_hidden_layers == 42); + CHECK(v.intermediate_size == 4224); + CHECK(v.moe_intermediate_size == 2112); + CHECK(v.patch_size == 14); + CHECK(v.temporal_patch_size == 1); + CHECK(v.spatial_merge_size == 2); + CHECK(v.rms_norm_eps == doctest::Approx(1e-5)); + CHECK_FALSE(v.use_bias); + CHECK(v.use_qk_norm); + CHECK_FALSE(v.is_causal); + CHECK(v.post_norm); + + // THE TWO FLAGS #2512's PROSE CONFLATES, asserted apart. `adapter_type` is + // `patch_merger`, which is upstream's name for the arm that SKIPS the + // pixel-shuffle permutation (vision.py:441-449 @ 9035151d6); the 2x2 + // regrouping did not disappear, `pre_pixel_shuffle` moved it into the + // PREPROCESSOR and into the RoPE. The issue's own tensor inventory β€” + // `adapter.{ln_q, mlp.0, mlp.2}` β€” is `PatchMergerAdapter`'s state dict and + // agrees with this; `PixelShuffleAdapter` spells its parameters + // `proj.0`/`proj.1`/`proj.3` (vision.py:397-406). See spec Β§4.11.1. + CHECK(v.adapter_type == "patch_merger"); + CHECK(v.pre_pixel_shuffle); + CHECK(v.adapter_in_dim == 1536); + CHECK(v.adapter_out_dim == 5120); + CHECK(v.adapter_merge_size == 2); + CHECK(v.merged_dim() == 6144); // 4 x 1536 + + // 25 dense + 17 MoE, counted from `pyramid_num_routed` rather than assumed: + // `is_moe` is `> 0` (vision.py:346-350), so the leading -1s are DENSE. + REQUIRE(v.pyramid_num_routed.size() == 42u); + CHECK(v.num_dense_blocks() == 25); + CHECK(v.num_moe_blocks() == 17); + CHECK(v.is_moe_block(24) == false); + CHECK(v.is_moe_block(25) == true); + CHECK(v.pyramid_num_routed[25] == 4); + CHECK(v.pyramid_num_routed[41] == 64); + int64_t experts = 0; + for (int64_t i = 25; i < 42; ++i) experts += v.pyramid_num_routed[static_cast(i)]; + // 1960 of the 2195 vision tensors: 17 x 8 block tensors + 608 x 3 experts. + CHECK(experts == 608); + CHECK(17 * 8 + experts * 3 == 1960); +} + +// --------------------------------------------------------------------------- +// 2. THE RELEASED CHECKPOINT STILL REFUSES, by name, and names W6b. +// This is the row's established pattern β€” W3 refused the LANGUAGE tower's +// MoE for four bricks before W5 lifted it β€” not a new exception. +// --------------------------------------------------------------------------- +TEST_CASE("dots3-note W6a: the RELEASED vision tower REFUSES BY NAME, and names W6b") { + const Dots3NoteVisionParams v = ParseDoc(ReleasedConfigDoc()); + const std::string why = Dots3NoteVisionRefusal(v, "", {}); + INFO("refusal: ", why); + REQUIRE_FALSE(why.empty()); + CHECK(why.find("W6b") != std::string::npos); + CHECK(why.find("25") != std::string::npos); // the FIRST routed block + CHECK(why.find("17") != std::string::npos); // how many there are + CHECK(why.find("MoE") != std::string::npos); + // The message names what to build, not that something is missing. + CHECK(why.find("gate_weight") != std::string::npos); + CHECK(why.find("router_bias") != std::string::npos); +} + +TEST_CASE("dots3-note W6a: every unported vision shape refuses BY NAME with its brick") { + const nlohmann::json released = ReleasedConfigDoc(); + + SUBCASE("an all-DENSE tower is ACCEPTED β€” the premise of every case below") { + nlohmann::json d = released; + for (auto& e : d["vision_config"]["pyramid_num_routed"]) e = -1; + CHECK(Dots3NoteVisionRefusal(ParseDoc(d), "", {}).empty()); + } + SUBCASE("the BLOCKWISE-FP8 arm is W9, and it outranks the MoE refusal") { + nlohmann::json d = released; + for (auto& e : d["vision_config"]["pyramid_num_routed"]) e = -1; + const std::string why = + Dots3NoteVisionRefusal(ParseDoc(d), "fp8", {128, 128}); + INFO(why); + CHECK(why.find("W9") != std::string::npos); + CHECK(why.find("weight_block_size") != std::string::npos); + } + SUBCASE("`pixel_shuffle_mlp` is a DIFFERENT token order and is refused") { + nlohmann::json d = released; + for (auto& e : d["vision_config"]["pyramid_num_routed"]) e = -1; + d["vision_config"]["adapter_type"] = "pixel_shuffle_mlp"; + const std::string why = Dots3NoteVisionRefusal(ParseDoc(d), "", {}); + INFO(why); + CHECK(why.find("pixel_shuffle_mlp") != std::string::npos); + CHECK(why.find("patch_merger") != std::string::npos); + } + SUBCASE("an UNKNOWN adapter refuses at PARSE, not at load") { + nlohmann::json d = released; + d["vision_config"]["adapter_type"] = "something_else"; + CHECK_THROWS_AS((void)ParseDoc(d), std::runtime_error); + } + SUBCASE("`temporal_patch_size != 1` is the VIDEO arm and names W7") { + nlohmann::json d = released; + for (auto& e : d["vision_config"]["pyramid_num_routed"]) e = -1; + d["vision_config"]["temporal_patch_size"] = 2; + const std::string why = Dots3NoteVisionRefusal(ParseDoc(d), "", {}); + INFO(why); + CHECK(why.find("W7") != std::string::npos); + } + SUBCASE("`use_bias`, `use_qk_norm`, `is_causal` and `post_norm` each refuse") { + for (const char* key : {"use_bias", "use_qk_norm", "is_causal", "post_norm"}) { + nlohmann::json d = released; + for (auto& e : d["vision_config"]["pyramid_num_routed"]) e = -1; + d["vision_config"][key] = !d["vision_config"][key].get(); + const std::string why = Dots3NoteVisionRefusal(ParseDoc(d), "", {}); + INFO("key ", key, " -> ", why); + CHECK_MESSAGE(!why.empty(), "flipping " << key << " was accepted"); + CHECK(why.find(key) != std::string::npos); + } + } + // THE THREE CONDITIONS THE ENCODER ASSERTS ON, asked HERE too. Before the + // fresh review of #2523 the refusal was a strict SUBSET of + // `EncodeMmDots3NoteForCausalLM`'s `VT_CHECK`s, and the gap was reachable + // from an all-dense config the seam ACCEPTED β€” after which the throw lands in + // the engine's busy loop and stops `AsyncLLM` for the life of the process. + // The refusal predicate and the route predicate must be the SAME predicate. + SUBCASE("`adapter_out_dim` that is not the TEXT hidden_size refuses") { + nlohmann::json d = released; + for (auto& e : d["vision_config"]["pyramid_num_routed"]) e = -1; + d["vision_config"]["adapter_out_dim"] = d["hidden_size"].get() + 8; + const std::string why = Dots3NoteVisionRefusal(ParseDoc(d), "", {}); + INFO(why); + CHECK_FALSE(why.empty()); + CHECK(why.find("adapter_out_dim") != std::string::npos); + CHECK(why.find("hidden_size") != std::string::npos); + } + SUBCASE("the TEXT hidden_size is what it compares against, not vision_config's") { + // The two documents' copies of the number are made to DISAGREE. The + // encoder's assert reads `config.hidden_size`, so a refusal that read + // `vision_config.hidden_size` instead would accept the left case and refuse + // the right one β€” both of them backwards. + nlohmann::json d = released; + for (auto& e : d["vision_config"]["pyramid_num_routed"]) e = -1; + d["vision_config"]["hidden_size"] = d["hidden_size"].get() + 8; + CHECK(Dots3NoteVisionRefusal(ParseDoc(d), "", {}).empty()); + + nlohmann::json e2 = released; + for (auto& e : e2["vision_config"]["pyramid_num_routed"]) e = -1; + e2["hidden_size"] = e2["hidden_size"].get() + 8; + const std::string why = Dots3NoteVisionRefusal(ParseDoc(e2), "", {}); + INFO(why); + CHECK_FALSE(why.empty()); + CHECK(why.find("adapter_out_dim") != std::string::npos); + } + SUBCASE("`adapter_merge_size` that is not `spatial_merge_size` refuses") { + // BOTH directions, because they reach DIFFERENT asserts on a served + // request: 1 makes the tower emit more rows than the placeholder span holds + // (`rows == item.length`), and 3 makes the trunk length not group into + // whole merger rows at all (`L % merge_unit == 0`). One refusal covers + // both, because both are the same disagreement between two keys. + for (int64_t m : {int64_t{1}, int64_t{3}}) { + nlohmann::json d = released; + for (auto& e : d["vision_config"]["pyramid_num_routed"]) e = -1; + d["vision_config"]["adapter_merge_size"] = m; + const std::string why = Dots3NoteVisionRefusal(ParseDoc(d), "", {}); + INFO("adapter_merge_size ", m, " -> ", why); + CHECK_FALSE(why.empty()); + CHECK(why.find("adapter_merge_size") != std::string::npos); + CHECK(why.find("spatial_merge_size") != std::string::npos); + } + } + SUBCASE("a checkpoint with NO vision_config refuses, naming the absence") { + nlohmann::json d = released; + d.erase("vision_config"); + const Dots3NoteVisionParams v = ParseDoc(d); + CHECK_FALSE(v.present); + const std::string why = Dots3NoteVisionRefusal(v, "", {}); + CHECK(why.find("vision_config") != std::string::npos); + } +} + +// --------------------------------------------------------------------------- +// 3. THE 235 DENSE TENSORS, counted against the COMMITTED shard index. +// "Nothing was left over" is also true of a map that claimed the MoE blocks +// as dense, so the counts are asserted BY NUMBER and cross-checked against +// the released index's own names. +// --------------------------------------------------------------------------- +TEST_CASE("dots3-note W6a: the DENSE arm claims 235 of the released tower's 2195 tensors") { + const Dots3NoteVisionParams v = ParseDoc(ReleasedConfigDoc()); + const std::vector claimed = + vllm::EnumerateDots3NoteVisionTensors(v); + CHECK(claimed.size() == 235u); + + // 25 blocks x 9, plus 3 patch_embed, 1 post_trunk_norm, 6 adapter. + CHECK(25 * 9 + 3 + 1 + 6 == 235); + + std::set names; + for (const vllm::Dots3NoteTensor& t : claimed) { + CHECK_MESSAGE(!t.consumer.empty(), t.name << " has no named consumer"); + CHECK_MESSAGE(names.insert(t.name).second, t.name << " is claimed twice"); + CHECK(t.name.rfind("vision_encoder.", 0) == 0); + } + + // NOT ONE MoE block's tensor is claimed. A map that walked all 42 blocks + // would claim 42 x 9 = 378 names, of which 153 do not exist on disk β€” and the + // load would then refuse for the wrong reason. + for (int64_t b = 25; b < 42; ++b) { + const std::string pre = "vision_encoder.blocks." + std::to_string(b) + "."; + for (const std::string& n : names) { + CHECK_MESSAGE(n.rfind(pre, 0) != 0, + "the DENSE arm claims " << n << ", which is a W6b block"); + } + } + // ...and every DENSE block IS claimed, so a leading-run bug that stopped + // early would show. + for (int64_t b = 0; b < 25; ++b) { + const std::string pre = "vision_encoder.blocks." + std::to_string(b) + "."; + CHECK_MESSAGE(names.count(pre + "attn.qkv.weight") == 1, + "block " << b << " is dense and unclaimed"); + CHECK_MESSAGE(names.count(pre + "mlp.fc3.weight") == 1, + "block " << b << " is dense and its fc3 is unclaimed"); + } + CHECK(names.count("vision_encoder.adapter.ln_q.bias") == 1); + CHECK(names.count("vision_encoder.adapter.mlp.2.weight") == 1); + CHECK(names.count("vision_encoder.post_trunk_norm.weight") == 1); + CHECK(names.count("vision_encoder.patch_embed.proj.bias") == 1); +} + +// --------------------------------------------------------------------------- +// 4. THE POSITION GRID. `pre_pixel_shuffle` selects between two DIFFERENT token +// orders, and the flag is read by the processor AND by the tower β€” so a case +// that only checked one order could not see the two disagreeing. +// --------------------------------------------------------------------------- +TEST_CASE("dots3-note W6a: pre_pixel_shuffle regroups the RoPE positions, and NOT setting it does not") { + TinySpec s; + const std::array grid{1, 4, 4}; + + s.v_pre_pixel_shuffle = true; + Dots3NoteVisionParams grouped = ParseDoc(dots3_tiny::TinyConfigDoc(FixtureDir(), s)); + s.v_pre_pixel_shuffle = false; + Dots3NoteVisionParams flat = ParseDoc(dots3_tiny::TinyConfigDoc(FixtureDir(), s)); + + const auto ours_grouped = Dots3NoteVisionPosIds(grid, grouped); + const auto ours_flat = Dots3NoteVisionPosIds(grid, flat); + const auto ref_grouped = ref::PosIds(1, 4, 4, /*rope_merge=*/2); + const auto ref_flat = ref::PosIds(1, 4, 4, /*rope_merge=*/1); + + REQUIRE(ours_grouped.size() == 16u); + CHECK(ours_grouped == ref_grouped); + CHECK(ours_flat == ref_flat); + // THE PREMISE, asserted rather than assumed: the two orders really differ, so + // agreeing with the wrong one is a detectable defect rather than a tie. + CHECK(ours_grouped != ours_flat); + // Spot the grouped layout by hand: token 1 is the 2x2 block's top-RIGHT, so + // it is row 0 column 1, where the flat order would put row 0 column 1 too but + // token 2 apart β€” flat token 2 is (0, 2), grouped token 2 is (1, 0). + CHECK(ours_grouped[2] == std::array{1, 0}); + CHECK(ours_flat[2] == std::array{0, 2}); +} + +// --------------------------------------------------------------------------- +// 5. THE TOWER, against the independent double-precision reference. +// THE CONSISTENCY GATE. It says two implementations agree; it does not say +// either matches vLLM, because vLLM cannot be run on this model here +// (spec Β§6.4 option B). +// --------------------------------------------------------------------------- +TEST_CASE("dots3-note W6a: the DENSE tower agrees with an INDEPENDENT double reference") { + Bench bench; + // The processor is the production one, so the patch rows the tower sees are + // the rows a served request would produce. + vllm::multimodal::Dots3NoteProcessorConfig pcfg = + vllm::multimodal::LoadDots3NoteProcessorConfig( + bench.ckpt.dir() + "/preprocessor_config.json", + bench.ckpt.config_path(), "tiny-dots3"); + const vllm::multimodal::Dots3NoteImageProcessor proc(pcfg); + const std::vector rgb = dots3_tiny::FixtureImage(0); + const vllm::multimodal::ImageKwargs kw = + proc.ProcessImage(rgb.data(), dots3_tiny::kImageSide, + dots3_tiny::kImageSide); + REQUIRE(kw.image_grid_thw[0] == 1); + REQUIRE(kw.image_grid_thw[1] == 4); + REQUIRE(kw.image_grid_thw[2] == 4); + REQUIRE(kw.num_patches == 16); + + const Dots3NoteVisionParams v = ParseDots3NoteVisionParams(bench.config); + REQUIRE(Dots3NoteVisionRefusal(v, "", {}).empty()); + Dots3NoteVisionWeights vw = vllm::MaterializeDots3NoteVision( + [&] { + std::vector shards; + shards.push_back( + vllm::SafetensorsFile::Open(bench.ckpt.weights_path())); + return shards; + }(), + v); + REQUIRE(vw.present); + + // THE MEMORY FORMAT (porting.md). A widened store passes every shape check + // and every token gate while moving twice the bytes, and this row's W2 F1 + // fixture row already proves a re-typed tensor fires. + CHECK(vw.patch_proj_w.dtype == vt::DType::kBF16); + CHECK(vw.adapter_mlp2_w.dtype == vt::DType::kBF16); + for (const auto& blk : vw.blocks) { + CHECK(blk.qkv.dtype == vt::DType::kBF16); + CHECK(blk.gate_up.dtype == vt::DType::kBF16); + } + // The merge is real: [2I, E], gate then up. + REQUIRE(vw.blocks.size() == static_cast(bench.spec.v_layers)); + CHECK(vw.blocks[0].gate_up.shape[0] == 2 * bench.spec.v_inter); + CHECK(vw.blocks[0].gate_up.shape[1] == bench.spec.v_embed); + + vt::Backend& backend = vt::GetBackend(vt::DeviceType::kCPU); + const std::vector ours = Dots3NoteVisionForward( + kw.pixel_values_bf16, kw.image_grid_thw, vw, v, backend); + + const std::vector want = + ref::Tower(bench.spec, bench.ckpt, WidenBf16(kw.pixel_values_bf16), 1, 4, 4); + REQUIRE(ours.size() == want.size()); + // FOUR merger rows (16 patches / 2x2), each in the TEXT hidden space. + CHECK(ours.size() == + static_cast(dots3_tiny::kExpectedImageTokens * bench.spec.hidden)); + + double max_abs = 0.0; + for (size_t i = 0; i < ours.size(); ++i) + max_abs = std::max(max_abs, std::abs(static_cast(ours[i]) - want[i])); + const double scale = MaxAbs(want); + const double rel = scale > 0.0 ? max_abs / scale : max_abs; + MESSAGE("tower vs double reference: max |diff| ", max_abs, " over a scale of ", + scale, " => relative ", rel); + // THE BOUND, and where it comes from. The implementation stores every + // activation and every weight in bf16 (8 mantissa bits, ~3.9e-3 relative) and + // runs two blocks plus a 64-wide adapter GEMM over it; the reference is + // double throughout and also absorbs upstream's intermediate `.type_as(x)` + // cast, which is a no-op at infinite precision. The bound is set at a + // MEASURED multiple of the observed deviation rather than at a round number, + // and the mutation evidence in the spec is what proves it is still tight + // enough to see a defect β€” a bound above the real error is a mute switch. + // MEASURED 2026-09-01 on this fixture: max |diff| 0.0533 over a scale of + // 6.314, i.e. 8.44e-3 relative. The bound is 0.02 β€” a 2.4x margin over the + // observation, wide enough that a different libm or a different GEMM + // reduction order does not red it, and tight enough that the mutations + // recorded in spec Β§4.11 all exceed it. + CHECK(rel < 0.02); + // ...and the two are not trivially equal, which would mean one of them is + // reading the other's answer. + CHECK(scale > 1e-3); +} + +// --------------------------------------------------------------------------- +// 6. THE PROCESSOR. `resized_size` is NOT `smart_resize`, the normalization is +// PER CHANNEL, and the placeholder count follows from the grid. +// --------------------------------------------------------------------------- +TEST_CASE("dots3-note W6a: the image processor mirrors upstream's own resize and normalization") { + using vllm::multimodal::Dots3NoteResizedSize; + // `factor = patch * merge` = 28 on the released geometry. + SUBCASE("a conformant size is the identity") { + const auto rs = Dots3NoteResizedSize(56, 84, 28, 16, 1 << 22); + CHECK(rs[0] == 56); + CHECK(rs[1] == 84); + } + SUBCASE("each side rounds INDEPENDENTLY to a multiple of factor") { + const auto rs = Dots3NoteResizedSize(57, 83, 28, 16, 1 << 22); + CHECK(rs[0] == 56); + CHECK(rs[1] == 84); + } + SUBCASE("a side under factor/4 refuses, naming the bound") { + CHECK_THROWS_AS((void)Dots3NoteResizedSize(6, 84, 28, 16, 1 << 22), + std::runtime_error); + } + SUBCASE("an aspect ratio over 200 refuses") { + CHECK_THROWS_AS((void)Dots3NoteResizedSize(28, 28 * 300, 28, 16, 1 << 30), + std::runtime_error); + } + SUBCASE("the max-pixel budget shrinks BOTH sides by the same beta") { + const auto rs = Dots3NoteResizedSize(280, 280, 28, 16, 28 * 28 * 4); + CHECK(rs[0] % 28 == 0); + CHECK(rs[1] % 28 == 0); + CHECK(rs[0] * rs[1] <= 28 * 28 * 4); + } + + // PER-CHANNEL normalization, which is the second thing that separates this + // processor from Qwen3-VL's. The fixture's mean/std differ per channel, so a + // processor that read only `image_mean[0]` would put red's statistics on + // green and blue. + TinySpec s; + TinyCheckpoint ck(FixtureDir(), s); + const vllm::multimodal::Dots3NoteProcessorConfig cfg = + vllm::multimodal::LoadDots3NoteProcessorConfig( + ck.dir() + "/preprocessor_config.json", ck.config_path(), "tiny"); + CHECK(cfg.image_mean[0] == doctest::Approx(0.5)); + CHECK(cfg.image_mean[1] == doctest::Approx(0.45)); + CHECK(cfg.image_mean[2] == doctest::Approx(0.4)); + CHECK(cfg.image_std[0] == doctest::Approx(0.25)); + CHECK(cfg.image_std[2] == doctest::Approx(0.35)); + // The three marker ids resolved from `added_tokens.json`. + CHECK(cfg.image_token_id == dots3_tiny::kImgPadId); + CHECK(cfg.image_start_token_id == dots3_tiny::kImgStartId); + CHECK(cfg.image_end_token_id == dots3_tiny::kImgEndId); + + const vllm::multimodal::Dots3NoteImageProcessor proc(cfg); + const std::vector rgb = dots3_tiny::FixtureImage(0); + const vllm::multimodal::ImageKwargs kw = proc.ProcessImage( + rgb.data(), dots3_tiny::kImageSide, dots3_tiny::kImageSide); + CHECK(kw.num_patches == 16); + CHECK(kw.patch_feature_dim == s.v_patch_row()); + // The placeholder run: `prod(grid) // merge**2` (multimodal.py:151-155). + CHECK(kw.num_patches / (s.v_merge * s.v_merge) == + dots3_tiny::kExpectedImageTokens); + + // One patch row, computed by hand against the pre_pixel_shuffle layout: + // row 0 is the 2x2 block (0,0)'s sub-patch (0,0), i.e. source pixels + // [0..1] x [0..1], and its channel-c element (ph, pw) is + // (raw - mean_c/rescale) / (std_c/rescale). + const int64_t patch = s.v_patch; + for (int64_t c = 0; c < 3; ++c) { + for (int64_t ph = 0; ph < patch; ++ph) { + for (int64_t pw = 0; pw < patch; ++pw) { + const uint8_t raw = + rgb[static_cast(ph * dots3_tiny::kImageSide * 3 + pw * 3 + c)]; + const double shift = cfg.image_mean[static_cast(c)] / + cfg.rescale_factor; + const double sc = + cfg.image_std[static_cast(c)] / cfg.rescale_factor; + const double want = (static_cast(raw) - shift) / sc; + const int64_t k = (c * patch + ph) * patch + pw; + CHECK(kw.pixel_values_f32[static_cast(k)] == + doctest::Approx(want).epsilon(1e-6)); + } + } + } + + // A non-conformant image is REFUSED by name rather than patchified at the + // wrong grid: the bicubic resize is a named residual, not a silent path. + std::vector odd(static_cast(9 * 9 * 3), 128); + CHECK_THROWS_AS((void)proc.ProcessImage(odd.data(), 9, 9), std::runtime_error); +} + +// --------------------------------------------------------------------------- +// 7. THE LOADER refuses a MoE tower over a checkpoint that really ships one, +// and leaves the language tower loadable beside it. A checkpoint whose +// vision arm is owed must still load its text half β€” that is what makes the +// refusal a deferral rather than a load failure. +// --------------------------------------------------------------------------- +TEST_CASE("dots3-note W6a: a PYRAMID vision block defers the tower and leaves the text tower loaded") { + TinySpec s; + s.v_pyramid = {-1, 4}; // block 1 is routed + Bench bench(s); + REQUIRE(bench.model != nullptr); + + const Dots3NoteVisionParams v = ParseDots3NoteVisionParams(bench.config); + const std::string why = Dots3NoteVisionRefusal(v, "", {}); + INFO(why); + CHECK_FALSE(why.empty()); + CHECK(why.find("W6b") != std::string::npos); + + // The LANGUAGE tower still materialized: the load did not fail, and the + // registration still resolves. A refusal that took the whole checkpoint down + // would make the released model unloadable, which it is not. + const std::vector arch{"Dots3NoteForCausalLM"}; + const vllm::ModelRegistration& reg = vllm::ModelRegistry::Resolve(arch); + CHECK(reg.factory->encode_mm != nullptr); + CHECK(reg.factory->embed_mm != nullptr); + CHECK(vllm::ModelRegistry::SupportsMmInputs(*bench.model)); + // ...and NOT an M-RoPE model. + CHECK_FALSE(vllm::ModelRegistry::UsesMrope(*bench.model)); +} diff --git a/tests/vllm/models/test_model_registry.cpp b/tests/vllm/models/test_model_registry.cpp index 271dc6904..257d50014 100644 --- a/tests/vllm/models/test_model_registry.cpp +++ b/tests/vllm/models/test_model_registry.cpp @@ -308,6 +308,7 @@ TEST_CASE("registry_model_property: Qwen registrations match pinned _ModelInfo") CHECK(registration.info.is_hybrid); CHECK_FALSE(registration.info.supports_multimodal); } else if (registration.architecture == "Qwen3VLForConditionalGeneration" || + registration.architecture == "Dots3NoteForCausalLM" || registration.architecture == "Gemma4ForConditionalGeneration" || registration.architecture == "Gemma4UnifiedForConditionalGeneration" || @@ -322,14 +323,24 @@ TEST_CASE("registry_model_property: Qwen registrations match pinned _ModelInfo") // with the hybrids; its vision tower is scaffolded, not yet forwarding. // The non-hybrid multimodal registrations. // - // `Dots3NoteForCausalLM` USED TO BE IN THIS LIST and moved out at W5 - // (#699). Upstream does register it in `_MULTIMODAL_MODELS` with image, - // video AND audio towers (multimodal.py:82-87, inside - // `get_placeholder_str` at :80-88), which is why W1 put it - // here β€” but this port has none of the three (W6/W7 for the towers, W8 - // for the front end), and the flag only became misleading once W5/W5c - // made the released config loadable. It now sits in the text-only branch - // below, and W8 moves it back. + // `Dots3NoteForCausalLM` LEFT THIS LIST AT W5 AND CAME BACK AT W6a, and + // the round trip is the honest record rather than churn (#699, #2512). + // W1 put it here because upstream registers it in `_MULTIMODAL_MODELS` + // with image, video AND audio towers (multimodal.py:82-87, inside + // `get_placeholder_str` at :80-88) β€” a statement about UPSTREAM. W5/W5c + // made the released config loadable, at which point the flag became a + // claim about THIS port that this port could not honour, so it moved to + // the text-only branch below. + // + // W6a moved it back and BACKED it: `kDots3NoteFactory` carries `encode_mm` + // and `embed_mm`, and `mm_chat_dots3note.cpp` registers this + // architecture's own chat seam, so a served `image_url` request reaches + // the model forward. It is NOT hybrid for the same reason the others here + // are not: both its attention classes page one MLA cache, and the sliding + // half is a window on it rather than a recurrent state. What is still + // refused BY NAME is the pyramid MoE ViT (W6b), which is why the RELEASED + // checkpoint's tower does not load β€” a flag about the ARCHITECTURE, not a + // promise about every checkpoint of it. CHECK_FALSE(registration.info.is_hybrid); CHECK(registration.info.supports_multimodal); } else {