Repository navigation
Conversation
MXFP4 checkpoints cannot use the M2N DeepGEMM executor, which reads BF16 or block-FP8 FusedMoE weights. Add a "marlin" expert backend that reads the parameters vLLM's Mxfp4MoEMethod leaves after the Marlin repack and runs fused_marlin_moe on the DeepEP-received rows (top_k=1 per row, the expert derived on the device from the prefix sum, graph safe). - mxfp4_experts.py: Mxfp4MarlinExpertWeights, the executor and the expected tensor specs for the model-info handshake. - The routed FFN pins --moe-backend marlin (auto -> marlin) when the checkpoint's experts are MXFP4 and keeps triton otherwise. - The handshake accepts mxfp4_marlin_w4a16 experts with the BF16 payload and checks that every FFN rank agrees on the weight format. - has_mxfp4_experts also recognizes compressed-tensors W4A8 MXFP4 experts (for example RedHatAI/GLM-5.3-MXFP4xFP8), resolved before load with the same scheme lookup and predicate vLLM uses to pick its MoE method. Ported from the DeepSeek-V4.1-Flash work (connector-generic parts only). Tests: backend pin from a real compressed-tensors GLM-5.3 config (MXFP4 experts pin marlin, FP8 experts keep triton), handshake and model-info checks, and tests/gpu Marlin parity against a dense MXFP4 reference (eager and CUDA graph), 6/6 on H100. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
vLLM 0.32 serves GlmMoeDsaForCausalLM on CUDA with vllm.models.deepseek_v32.nvidia (DSA backbone, fused embed/RMSNorm, MLP output reduced in the next layer's fused all-reduce + RMSNorm, breakable CUDA graphs), while AFD routed it to the DeepSeek V2-derived wrapper around the generic deepseek_v2 module that vLLM now uses only on XPU/CPU. Add a CUDA wrapper over the native DSA model that keeps its forward and loader and makes only construction and checkpoint loading role-aware: - Attention: DSA attention with the indexer top-k buffer, the dense MLPs, and per MoE layer the existing routed-experts shell (gate, native router, shared experts, dispatch/combine). - FFN: the routed experts only (native DeepseekV2MoE without shared experts, external routing). - One-pass checkpoint filter in front of the native loader. - Fail closed outside the first contract: DeepEP M2N only, Attention TP 1 (the native layer would all-reduce complete connector rows TP times), no PP/SP, no DBO, no MTP. GlmMoeDsaForCausalLM resolves to the new wrapper on CUDA and keeps the DeepSeek V2-derived wrapper on NPU; the M2N connector admits it. The idea of admitting GLM-5.3 on M2N follows an earlier teammate branch. Tests: construction on GLM-5.3's 78-layer schedule per role, the fail-closed table, role filtering of real GLM-5.3 FP8 and MXFP4xFP8 tensor names, registration and the M2N admission check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
- recipe/gpu/DeepEPM2NAFDConnector/glm_5_3: MXFP4xFP8 4A4F eager and graph on one 8-GPU node (Marlin experts, BF16 payload) and a block-128 FP8 4A4F,4A4F two-pod graph recipe (DeepGEMM, FP8 payload). - recipe/gpu/baseline/glm_5_3/native_dp8.sh: the native vLLM DP8 + EP control with the same checkpoint, KV cache dtype and Marlin MoE kernel. - tests/e2e/models/glm_5_3: native baseline and M2N 4A4F eager/graph GSM8K cases; Deployment.connector_extra_config accepts string values (expert_backend). - Connector guide, support matrix and model-integration design document list GLM-5.x, the marlin expert backend and the MXFP4 backend pin. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
GLM-5.3 MXFP4xFP8 on one 8 x H200 node, vLLM 0.32.0, GSM8K-200 (flexible/strict): native DP8 + EP eager with Marlin 0.925/0.925, AFD DeepEP M2N 4A4F eager 0.930/0.930, graph (FULL_DECODE_ONLY, breakable CUDA graphs as native) 0.935/0.935. The stock vllm-openai v0.32.0 image's nvidia-nccl-cu13 2.29.7 lacks the GIN request API that afdEP-comm v1.1.0 compiles against; the recipe README asks for the 2.30.7 wheel. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
github-actions
Bot
added
the
documentation
…ch top-k MTP: the native DeepseekV32MTP draft runs on the Attention ranks with its own (EP-local) experts; the FFN serves only the target's routed experts. The wrapper accepts method "mtp" on the Attention role only. DBO: GLM-5.x runs the lightning indexer on one layer in four and later layers reuse its top-k from one model-level buffer and per-indexer SparseMLAIndexGroup workspaces. The routed-experts connector yields between decoder layers, so the other micro-batch overwrote that state (GSM8K-200 0.685 with garbled output). gpu_deepseek_v32_dbo gives each micro-batch its own buffer and group copies (fresh side stream/events), resolving every holder (model, DSA attention, indexer, indexer op, sparse MLA impl incl. its private buffer) by dbo_current_ubatch_id(), and fails at startup if any object still holds the shared state. It also leases the micro-batch compute SM budget to the DSA indexer planner (as the V4 DBO path does, vLLM #60732); without it the AFD world deadlocked. Remaining limits, fail closed: - micro-batched prefill still corrupts a fraction of requests (0.90 with garbled output vs 0.93): DBO requires dbo_prefill_token_threshold above max_num_batched_tokens; - with CUDA graphs a padding-only micro-batch trips vLLM 0.32's sparse FP8 MLA metadata builder (assert decode_query_len > 0): DBO requires eager; - vLLM ModelRunnerV2 rejects DBO with speculative decoding: no MTP + DBO. Tests: per-micro-batch resolution and isolation over vLLM's SharedTopkIndicesBuffer mixin and real index groups, missed-holder detection, the DBO install on construction and the updated deployment table. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
- 4a4f_graph_mtp_dp4tp1.sh: MTP (3 speculative tokens by default) on the Attention role only. - 4a4f_dbo_dp4tp1.sh: eager DBO, decode steps only. - Connector guide, support matrix, model-integration design and recipe README describe the MTP/DBO contract and its fail-closed limits. GLM-5.3 MXFP4xFP8 on one 8 x H200 node, vLLM 0.32.0, GSM8K-200: MTP native graph 0.93 vs M2N 4A4F graph 0.92 (mean acceptance length 3.4-3.5 vs 3.4); DBO eager decode-only 0.93 (concurrency 32) and 0.91 (64) vs 0.93 without DBO. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
…s report - config_dbo_spec_decode: for AFD configs, drop the MRV2 "DBO with speculative decoding" rejection and raise the DBO thresholds to one request per microbatch (#61464). - gpu_mrv2_dbo_spec_decode (worker): request-boundary split of uniform multi-token (verification) batches from the DP-agreed uniform count, and a draft-prefill pre-step that rebuilds full-batch metadata on the draft builders after a microbatched target forward (eager). - GLM-5.x accepts MTP together with eager decode-only DBO. - docs/reports/glm-5.3-afd-followups.md: open work (prefill DBO, graph DBO root cause, MTP+DBO GPU validation) and suggested upstream changes. Unit tests for the split, the gate and the pre-step pass (15); MTP+DBO is not GPU-validated yet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
…P+DBO eager vLLM 0.32's sync_cudagraph_and_dp_padding falls back from a microbatched FULL graph only when 2 * min(tokens) < padded. With an even padded count a rank holding exactly half the tokens replays a graph whose second micro-batch is padding only, and the sparse FP8 MLA builder asserts decode_query_len > 0. A marked copy in gpu_mrv2_ubatching uses the exact midpoint-split condition, and AFD's DP1 dispatch uses the same check. GLM-5.3 MXFP4xFP8 4A4F GSM8K-200 on 8 x H200: graph DBO 0.93, eager MTP + DBO 0.94 (MTP alone 0.92). The eager-only DBO rule is lifted; MTP with graph DBO hit an illegal memory access (draft-prefill graphs still capture on the target's builders) and is rejected. Also: 4a4f_dbo_dp4tp1.sh takes MODE (graph default), new 4a4f_mtp_dbo_dp4tp1.sh, docs and the follow-ups report (prefill-DBO request-aligned split diagnostic removes the garbled answers), and the ubatch-prepare signature test fake gains speculative_config. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
…arison Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
At vLLM 0.32 the FULL draft-prefill graphs are captured on the target's attention builders and replay the full-batch metadata the target forward last wrote there. A microbatched target forward writes only per-microbatch metadata, so the replay read stale sequence lengths and block tables (illegal memory access with GLM-5.3 MTP + graph DBO). After a microbatched step the speculator now dispatches only draft prefill eager, on the metadata the #61464 backport rebuilds on the draft builders; the target and draft decode keep their graphs. The adapter accepts MTP with graph DBO. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
…1464) Complete the #61464 backport for CUDA graphs: under DBO the FULL draft-prefill graphs are captured on the draft's own attention builders (scoped swap of target_attn_groups around TargetDependentARSpeculator .capture), and each draft-prefill replay first refreshes those builders with this batch's full-batch metadata (scoped wrap of the prefill manager's run_fullgraph within propose). This replaces the eager draft prefill fallback, which confirmed the cause. GLM-5.3 MXFP4xFP8 4A4F graph MTP + DBO GSM8K-200 on 8 x H200: 0.95, no errors, acceptance 3.30-3.69 (eager-draft-prefill fallback 0.92, eager MTP + DBO 0.94, MTP alone 0.92). Docs and the follow-ups report updated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
Map each open item and upstream suggestion to the vLLM issues and PRs found on main and the v0.33.0 release branch, with their status and coverage. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.