Add DeepSeek-V4-Flash-Vision-Exp support with sm80 (Ampere) fixes - #58
Open
kaka86mm wants to merge 10 commits into
Open
Add DeepSeek-V4-Flash-Vision-Exp support with sm80 (Ampere) fixes#58kaka86mm wants to merge 10 commits into
kaka86mm wants to merge 10 commits into
Conversation
DeepSeek-V4-Flash-Vision-Exp keeps the V4-Flash language backbone unchanged (the config diff against 0731 removes nothing and adds only the ten vision_* keys), but its checkpoint carries weights the text model has no home for: - a ViT, an aligner and four sentinel embeddings, and - a per-layer vision expert bias `ffn.gate.bias_vl`, plus a plain `ffn.gate.bias` on the first num_hash_layers layers, which the text-only checkpoint omits because hash routing ignores it. Drop the vision tensors and allocate the hash-layer gate bias so the backbone loads and serves text from a vision checkpoint. The skip mapper has to run before hf_to_vllm_mapper: its ".ffn.gate.bias" substring rule also matches ".ffn.gate.bias_vl" and would rename it into a parameter that does not exist. Verified on 4x A6000 (TP4, fp8_ds_mla, 8k ctx): weights load, KV cache and CUDA graphs build, and greedy text generation is coherent (127*43 -> 5461, capital of France -> Paris). Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
A faithful port of the reference vision tower: a 32-block bidirectional ViT over one image's 14x14 patches with 2D RoPE, followed by an aligner that folds each 3x3 square of ViT tokens into one language-model token. The tower is ~0.3B, so it is replicated per rank rather than sharded. The rotary tables are built on CPU and then copied to the device. This looks gratuitous but is load-bearing: building them on device costs one fp32 ULP in cos/sin, and bf16 rounding compounds that across the 32 residual blocks into ~9e-2 of drift in the tower output (against a ViT output std of ~1e-1). With CPU-built tables the port is bit-exact. Checked against the reference implementation on inference/examples/images/carrots.jpeg with the checkpoint's own weights: a 42x61 patch grid encodes to [2562, 1024] and aligns to [294, 4096], both bitwise identical to the reference. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
Owner
|
From what I would tell, vllm-project/vllm#54566 is not stable enough to backport so that's why I put it in a standalone branch at this moment. |
the PR is merged in the meanwhile. Is it possible to adopt into your project? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds full multimodal support for DeepSeek-V4-Flash-Vision-Exp on top of the
dsv4-vision-expbranch, including three sm80-specific fixes that make it actually work on Ampere GPUs (CMP 170HX / A100).What's included
From upstream vllm PR #54566 (cherry-picked):
vl_model.py,vl_stub.py)mm_preprocess.py,common/vision.py)bias_vl(CUDA kernel + router changes)fix breakable cg— 2-line config fix for CUDA-graph text corruptionenable mtp— speculative decoding plumbing for VLsm80 / Ampere fixes (new, tested on 4× CMP 170HX):
bias_vlrouting requiresinput_idson all PP ranks, but non-first ranks only receive hidden states. Relaysinput_idsthroughIntermediateTensorsbroadcast (3 hunks innvidia/model.py).embed_tokensonly builds on the first rank. Now builds it on the last rank when speculative decoding is active (+1 GB VRAM).fp8_sm80software encode/decode) instead of crashing withfp8e4nv not supported.Tested on
4× CMP 170HX (unlocked to 64GB each, GA100 sm80, PCIe Gen2 x4, PP4):
get_weather({"city": "Beijing"})with correct finish_reasonNotes for reviewers
5ab628dd1) is 2 lines inconfig/vllm.py+models/config.py— without it, text queries return content from unrelated requests under CUDA graphsbias_vlrouting crashes withvision MoE routing requires input_idson PP ranks 1-3Not included (deliberately)