diff --git a/MODELS.md b/MODELS.md index 060c053b87..017c524986 100644 --- a/MODELS.md +++ b/MODELS.md @@ -52,7 +52,7 @@ Speculative-decoding A/B retirements apply to each pair below. The spec-decode a Rationale: `dsv4` carries the largest single-turn footprint in the repository. 45 active config keys use the 8k1k scenario, 32 in `configs/nvidia-master.yaml` and 13 in `configs/amd-master.yaml`, spanning H200, B200, B300, GB200, GB300, MI300X, MI325X, and MI355X across vLLM, SGLang, TensorRT-LLM, ATOM, Dynamo, and llm-d. That is a large share of every full sweep. AgentX trace replay is the scenario AI labs and the ML community ask about, and DeepSeek-V4-Pro's 19 agentic config keys are the part of `dsv4` that feeds the published North Star Pareto frontier. Retiring the fixed-sequence-length arm frees cluster hours for AgentX and for new frontier models such as Qwen3.8-Flash-Next without reducing what we publish for this model. Single-turn 8k1k stays active for the other models that still list it. -**Status: not yet enacted.** All 45 8k1k config keys still run. On enactment they are removed from the active master configs and archived under [`configs/deprecated/`](configs/deprecated/), with their benchmark scripts moved to the sibling `deprecated/` directories, matching how [#2493](https://github.com/SemiAnalysisAI/InferenceX/pull/2493) and [#2527](https://github.com/SemiAnalysisAI/InferenceX/pull/2527) were carried out. The SPEED-Bench acceptance-length scripts for `dsv4` are intentionally kept. Speedbench is driven by `speedbench-al.yml`, not the master configs. +**Enacted on 2026-09-09** in [#2921](https://github.com/SemiAnalysisAI/InferenceX/pull/2921): 46 `dsv4` 8k1k config keys were removed from the active master configs and archived under [`configs/deprecated/`](configs/deprecated/) as `nvidia-dsv4-8k1k-master.yaml` (33) and `amd-dsv4-8k1k-master.yaml` (13), and their 28 benchmark scripts were moved to the sibling `deprecated/` directories (25 under `benchmarks/single_node/fixed_seq_len/`, 3 under `benchmarks/multi_node/`), matching how [#2493](https://github.com/SemiAnalysisAI/InferenceX/pull/2493) and [#2527](https://github.com/SemiAnalysisAI/InferenceX/pull/2527) were carried out. The count is 46 rather than the 45 quoted above because `dsv4-fp4-b200-dynamo-sglang` landed after this notice was written. The 19 agentic-coding keys are untouched: `dsv4` continues to run and publish with agentic coding as its only scenario. The SPEED-Bench acceptance-length scripts for `dsv4` are intentionally kept. Speedbench is driven by `speedbench-al.yml`, not the master configs. The srt-slurm and llm-d recipe YAMLs referenced by the archived multi-node keys stay in place as inert reference data, as #2493 and #2527 left theirs. ## Scenarios @@ -120,7 +120,7 @@ The table also records both the agreed plan-of-record (PoR) draft-model mapping | Model | Primary native/upstream engines | Agreed draft model(s) (PoR) | Proposed draft model(s) pending partner alignment | Additional engines | |---|---|---|---|---| -| DeepSeek-V4-Pro 1.6T (`dsv4`) | native/upstream vLLM engine and native/upstream SGLang engine | native MTP (Single-turn 8k1k); native DSpark heads `deepseek-ai/DeepSeek-V4-Pro-0813` (Agentic coding only, under the same synthetic-acceptance methodology as Kimi-K3's DSpark PoR) | None | Additional non-vLLM/SGLang engines under the ordering guideline and exceptions above | +| DeepSeek-V4-Pro 1.6T (`dsv4`) | native/upstream vLLM engine and native/upstream SGLang engine | native MTP (Agentic coding MTP arms; formerly also Single-turn 8k1k, retired 2026-09-09); native DSpark heads `deepseek-ai/DeepSeek-V4-Pro-0813` (Agentic coding only, under the same synthetic-acceptance methodology as Kimi-K3's DSpark PoR) | None | Additional non-vLLM/SGLang engines under the ordering guideline and exceptions above | | Kimi-K3 (`kimik3`) | native/upstream vLLM engine | `Inferact/Kimi-K3-DSpark` | None | Additional non-vLLM/SGLang engines under the ordering guideline and exceptions above | | MiniMax-M3 (`minimaxm3`) | native/upstream vLLM engine | `Inferact/MiniMax-M3-EAGLE3` and/or `Inferact/MiniMax-M3-EAGLE3-GQA` | None | Additional non-vLLM/SGLang engines under the ordering guideline and exceptions above | | GLM-5.2 (`glm5.2`) | native/upstream SGLang engine | native MTP | None | Additional non-vLLM/SGLang engines under the ordering guideline and exceptions above | @@ -154,7 +154,7 @@ Other offloading tiers, including NVMe KV cache offloading, are outside the init | Kimi-K3 | `kimik3` | 2026-07-27 ([#2391](https://github.com/SemiAnalysisAI/InferenceX/pull/2391)) | Agentic coding (DSpark only) | Agentic coding non-DSpark arm (deprecated from day 0) | | GLM-5.2 | `glm5.2` | 2026-07-18 ([#2268](https://github.com/SemiAnalysisAI/InferenceX/pull/2268)) | Agentic coding (the non-MTP arm still runs while the MTP-only transition remains pending, as explained in the Deprecation Notice) | | | MiniMax-M3 | `minimaxm3` | 2026-06-12 ([#1724](https://github.com/SemiAnalysisAI/InferenceX/pull/1724)) | Agentic coding | Single-turn 1k1k, Single-turn 8k1k (removed 2026-08-04, [#2493](https://github.com/SemiAnalysisAI/InferenceX/pull/2493)) | -| DeepSeek-V4-Pro | `dsv4` | 2026-04-24 ([#1130](https://github.com/SemiAnalysisAI/InferenceX/pull/1130)) | Single-turn 8k1k (last day 2026-09-08, see [Deprecation Notice](#deprecation-notice)), Agentic coding (the non-MTP arm still runs while the MTP-only transition remains pending, as explained in the Deprecation Notice) | Single-turn 1k1k | +| DeepSeek-V4-Pro | `dsv4` | 2026-04-24 ([#1130](https://github.com/SemiAnalysisAI/InferenceX/pull/1130)) | Agentic coding (the non-MTP arm still runs while the MTP-only transition remains pending, as explained in the Deprecation Notice) | Single-turn 1k1k, Single-turn 8k1k (removed 2026-09-09, [#2921](https://github.com/SemiAnalysisAI/InferenceX/pull/2921)) | | GLM-5 / GLM-5.1 | `glm5`, `glm5.1` | 2026-03-06 ([#762](https://github.com/SemiAnalysisAI/InferenceX/pull/762)), with GLM-5.1 added 2026-04-21 ([#1098](https://github.com/SemiAnalysisAI/InferenceX/pull/1098)) | None (retired 2026-07-18, [#2276](https://github.com/SemiAnalysisAI/InferenceX/pull/2276)) | Single-turn 1k1k, Single-turn 1k8k (GLM-5 only), Single-turn 8k1k | | MiniMax-M2.5/2.7 | `minimaxm2.5` | 2026-02-18 ([#755](https://github.com/SemiAnalysisAI/InferenceX/pull/755)) | None (retired 2026-06-20, [#1874](https://github.com/SemiAnalysisAI/InferenceX/pull/1874)) | Single-turn 1k1k, Single-turn 1k8k, Single-turn 8k1k | | Kimi-K2.5/2.6/2.7-Code | `kimik2.5` | 2026-02-17 ([#734](https://github.com/SemiAnalysisAI/InferenceX/pull/734)) | None (fully retired 2026-08-07, [#2527](https://github.com/SemiAnalysisAI/InferenceX/pull/2527)) | Single-turn 1k1k, Single-turn 1k8k, Agentic coding (removed 2026-08-04, [#2493](https://github.com/SemiAnalysisAI/InferenceX/pull/2493)), Single-turn 8k1k (removed 2026-08-07, [#2527](https://github.com/SemiAnalysisAI/InferenceX/pull/2527)) | diff --git a/MODELS_zh.md b/MODELS_zh.md index 5b674b4607..e2c5178771 100644 --- a/MODELS_zh.md +++ b/MODELS_zh.md @@ -52,7 +52,7 @@ InferenceX-e2e 运行在数量固定且有限的 GPU 资源池上，并由一支 原因：`dsv4` 是本仓库中单轮场景占用最大的模型。当前有 45 个启用的配置项使用 8k1k 场景（`configs/nvidia-master.yaml` 32 个，`configs/amd-master.yaml` 13 个），覆盖 H200、B200、B300、GB200、GB300、MI300X、MI325X 与 MI355X，涉及 vLLM、SGLang、TensorRT-LLM、ATOM、Dynamo 与 llm-d，在每一轮完整 sweep 中占比可观。AgentX 轨迹回放才是 AI 实验室与 ML 社区真正关注的场景，而 DeepSeek-V4-Pro 的 19 个智能体编码配置项正是 `dsv4` 中支撑已发布北极星（North Star）帕累托前沿的部分。下线固定序列长度分支可为 AgentX 以及 Qwen3.8-Flash-Next 等新前沿模型腾出集群机时，同时不减少该模型对外发布的内容。对于仍列有该场景的其他模型，单轮 8k1k 保持启用。 -**状态：尚未执行。** 全部 45 个 8k1k 配置项仍在运行。执行时将从启用的主配置中移除并归档至 [`configs/deprecated/`](configs/deprecated/)，对应基准测试脚本移入同级 `deprecated/` 目录，与 [#2493](https://github.com/SemiAnalysisAI/InferenceX/pull/2493) 和 [#2527](https://github.com/SemiAnalysisAI/InferenceX/pull/2527) 的做法一致。`dsv4` 的 SPEED-Bench 接受长度脚本予以保留。Speedbench 由 `speedbench-al.yml` 驱动，不经过主配置。 +**已于 2026-09-09 执行**（[#2921](https://github.com/SemiAnalysisAI/InferenceX/pull/2921)）：46 个 `dsv4` 8k1k 配置项已从启用的主配置中移除并归档至 [`configs/deprecated/`](configs/deprecated/)，即 `nvidia-dsv4-8k1k-master.yaml`（33 个）与 `amd-dsv4-8k1k-master.yaml`（13 个）；对应的 28 个基准测试脚本移入同级 `deprecated/` 目录（`benchmarks/single_node/fixed_seq_len/` 下 25 个，`benchmarks/multi_node/` 下 3 个），与 [#2493](https://github.com/SemiAnalysisAI/InferenceX/pull/2493) 和 [#2527](https://github.com/SemiAnalysisAI/InferenceX/pull/2527) 的做法一致。数量为 46 而非上文所述的 45，是因为 `dsv4-fp4-b200-dynamo-sglang` 在本公告发布后才合入。19 个智能体编码配置项未做改动：`dsv4` 以智能体编码为唯一场景继续运行与发布。`dsv4` 的 SPEED-Bench 接受长度脚本予以保留。Speedbench 由 `speedbench-al.yml` 驱动，不经过主配置。已归档多节点配置项所引用的 srt-slurm 与 llm-d 配方 YAML 作为惰性参考数据原地保留，与 #2493 和 #2527 的处理一致。 ## 场景 @@ -120,7 +120,7 @@ InferenceX 支持 SGLang 和 vLLM 双方的维护者，并响应 AI 实验室和 | 模型 | 首选原生/上游引擎 | 已达成一致的草稿模型（PoR） | 待合作伙伴对齐的草稿模型提案 | 其他引擎 | |---|---|---|---|---| -| DeepSeek-V4-Pro 1.6T（`dsv4`） | 原生/上游 vLLM 引擎和原生/上游 SGLang 引擎 | 原生 MTP（单轮 8k1k）；`deepseek-ai/DeepSeek-V4-Pro-0813`（仅用于智能体编码，遵循与 Kimi-K3 DSpark PoR 相同的合成接受方法） | 无 | 按照上述提交顺序指南及例外处理的其他非 vLLM/SGLang 引擎 | +| DeepSeek-V4-Pro 1.6T（`dsv4`） | 原生/上游 vLLM 引擎和原生/上游 SGLang 引擎 | 原生 MTP（智能体编码 MTP 分支；此前亦用于单轮 8k1k，已于 2026-09-09 退役）；`deepseek-ai/DeepSeek-V4-Pro-0813`（仅用于智能体编码，遵循与 Kimi-K3 DSpark PoR 相同的合成接受方法） | 无 | 按照上述提交顺序指南及例外处理的其他非 vLLM/SGLang 引擎 | | Kimi-K3（`kimik3`） | 原生/上游 vLLM 引擎 | `Inferact/Kimi-K3-DSpark` | 无 | 按照上述提交顺序指南及例外处理的其他非 vLLM/SGLang 引擎 | | MiniMax-M3（`minimaxm3`） | 原生/上游 vLLM 引擎 | `Inferact/MiniMax-M3-EAGLE3` 和/或 `Inferact/MiniMax-M3-EAGLE3-GQA` | 无 | 按照上述提交顺序指南及例外处理的其他非 vLLM/SGLang 引擎 | | GLM-5.2（`glm5.2`） | 原生/上游 SGLang 引擎 | 原生 MTP | 无 | 按照上述提交顺序指南及例外处理的其他非 vLLM/SGLang 引擎 | @@ -154,7 +154,7 @@ InferenceX 支持 SGLang 和 vLLM 双方的维护者，并响应 AI 实验室和 | Kimi-K3 | `kimik3` | 2026-07-27 ([#2391](https://github.com/SemiAnalysisAI/InferenceX/pull/2391)) | 智能体编码（仅 DSpark） | 智能体编码非 DSpark 分支（自第 0 天起弃用） | | GLM-5.2 | `glm5.2` | 2026-07-18（[#2268](https://github.com/SemiAnalysisAI/InferenceX/pull/2268)） | 智能体编码（非 MTP 分支仍在运行，「仅 MTP」转换仍待执行，见弃用公告） | | | MiniMax-M3 | `minimaxm3` | 2026-06-12（[#1724](https://github.com/SemiAnalysisAI/InferenceX/pull/1724)） | 智能体编码 | 单轮 1k1k、单轮 8k1k（2026-08-04 移除，[#2493](https://github.com/SemiAnalysisAI/InferenceX/pull/2493)） | -| DeepSeek-V4-Pro | `dsv4` | 2026-04-24（[#1130](https://github.com/SemiAnalysisAI/InferenceX/pull/1130)） | 单轮 8k1k、智能体编码（非 MTP 分支仍在运行，「仅 MTP」转换仍待执行，见弃用公告） | 单轮 1k1k | +| DeepSeek-V4-Pro | `dsv4` | 2026-04-24（[#1130](https://github.com/SemiAnalysisAI/InferenceX/pull/1130)） | 智能体编码（非 MTP 分支仍在运行，「仅 MTP」转换仍待执行，见弃用公告） | 单轮 1k1k、单轮 8k1k（已于 2026-09-09 移除，[#2921](https://github.com/SemiAnalysisAI/InferenceX/pull/2921)） | | GLM-5 / GLM-5.1 | `glm5`、`glm5.1` | 2026-03-06（[#762](https://github.com/SemiAnalysisAI/InferenceX/pull/762)），GLM-5.1 于 2026-04-21 加入（[#1098](https://github.com/SemiAnalysisAI/InferenceX/pull/1098)） | 无（2026-07-18 退役，[#2276](https://github.com/SemiAnalysisAI/InferenceX/pull/2276)） | 单轮 1k1k、单轮 1k8k（仅 GLM-5）、单轮 8k1k | | MiniMax-M2.5/2.7 | `minimaxm2.5` | 2026-02-18（[#755](https://github.com/SemiAnalysisAI/InferenceX/pull/755)） | 无（2026-06-20 退役，[#1874](https://github.com/SemiAnalysisAI/InferenceX/pull/1874)） | 单轮 1k1k、单轮 1k8k、单轮 8k1k | | Kimi-K2.5/2.6/2.7-Code | `kimik2.5` | 2026-02-17（[#734](https://github.com/SemiAnalysisAI/InferenceX/pull/734)） | 无（2026-08-07 完全退役，[#2527](https://github.com/SemiAnalysisAI/InferenceX/pull/2527)） | 单轮 1k1k、单轮 1k8k、智能体编码（2026-08-04 移除，[#2493](https://github.com/SemiAnalysisAI/InferenceX/pull/2493)）、单轮 8k1k（2026-08-07 移除，[#2527](https://github.com/SemiAnalysisAI/InferenceX/pull/2527)） | diff --git a/benchmarks/multi_node/dsv4_fp4_gb200_llmd-vllm-disagg.sh b/benchmarks/multi_node/deprecated/dsv4_fp4_gb200_llmd-vllm-disagg.sh similarity index 100% rename from benchmarks/multi_node/dsv4_fp4_gb200_llmd-vllm-disagg.sh rename to benchmarks/multi_node/deprecated/dsv4_fp4_gb200_llmd-vllm-disagg.sh diff --git a/benchmarks/multi_node/dsv4_fp4_mi355x_atom-disagg.sh b/benchmarks/multi_node/deprecated/dsv4_fp4_mi355x_atom-disagg.sh similarity index 100% rename from benchmarks/multi_node/dsv4_fp4_mi355x_atom-disagg.sh rename to benchmarks/multi_node/deprecated/dsv4_fp4_mi355x_atom-disagg.sh diff --git a/benchmarks/multi_node/dsv4_fp4_mi355x_sglang-disagg.sh b/benchmarks/multi_node/deprecated/dsv4_fp4_mi355x_sglang-disagg.sh similarity index 100% rename from benchmarks/multi_node/dsv4_fp4_mi355x_sglang-disagg.sh rename to benchmarks/multi_node/deprecated/dsv4_fp4_mi355x_sglang-disagg.sh diff --git a/benchmarks/single_node/agentic/dsv4_fp4_mi355x_vllm_mtp.sh b/benchmarks/single_node/agentic/dsv4_fp4_mi355x_vllm_mtp.sh index daa0704e56..32751d5ce9 100644 --- a/benchmarks/single_node/agentic/dsv4_fp4_mi355x_vllm_mtp.sh +++ b/benchmarks/single_node/agentic/dsv4_fp4_mi355x_vllm_mtp.sh @@ -16,7 +16,7 @@ set -x # # Serving flags follow the validated MI355X recipe from # https://recipes.vllm.ai/deepseek-ai/DeepSeek-V4-Pro?hardware=mi355x -# https://github.com/SemiAnalysisAI/InferenceX/blob/main/benchmarks/single_node/fixed_seq_len/dsv4_fp4_mi355x_vllm.sh +# https://github.com/SemiAnalysisAI/InferenceX/blob/main/benchmarks/single_node/fixed_seq_len/deprecated/dsv4_fp4_mi355x_vllm.sh # Image is configured in amd-master.yaml. # # Required env vars: diff --git a/benchmarks/single_node/fixed_seq_len/dsv4_fp4_b200.sh b/benchmarks/single_node/fixed_seq_len/deprecated/dsv4_fp4_b200.sh similarity index 100% rename from benchmarks/single_node/fixed_seq_len/dsv4_fp4_b200.sh rename to benchmarks/single_node/fixed_seq_len/deprecated/dsv4_fp4_b200.sh diff --git a/benchmarks/single_node/fixed_seq_len/dsv4_fp4_b200_trt.sh b/benchmarks/single_node/fixed_seq_len/deprecated/dsv4_fp4_b200_trt.sh similarity index 100% rename from benchmarks/single_node/fixed_seq_len/dsv4_fp4_b200_trt.sh rename to benchmarks/single_node/fixed_seq_len/deprecated/dsv4_fp4_b200_trt.sh diff --git a/benchmarks/single_node/fixed_seq_len/dsv4_fp4_b200_trt_mtp.sh b/benchmarks/single_node/fixed_seq_len/deprecated/dsv4_fp4_b200_trt_mtp.sh similarity index 100% rename from benchmarks/single_node/fixed_seq_len/dsv4_fp4_b200_trt_mtp.sh rename to benchmarks/single_node/fixed_seq_len/deprecated/dsv4_fp4_b200_trt_mtp.sh diff --git a/benchmarks/single_node/fixed_seq_len/dsv4_fp4_b200_vllm.sh b/benchmarks/single_node/fixed_seq_len/deprecated/dsv4_fp4_b200_vllm.sh similarity index 100% rename from benchmarks/single_node/fixed_seq_len/dsv4_fp4_b200_vllm.sh rename to benchmarks/single_node/fixed_seq_len/deprecated/dsv4_fp4_b200_vllm.sh diff --git a/benchmarks/single_node/fixed_seq_len/dsv4_fp4_b200_vllm_mtp.sh b/benchmarks/single_node/fixed_seq_len/deprecated/dsv4_fp4_b200_vllm_mtp.sh similarity index 100% rename from benchmarks/single_node/fixed_seq_len/dsv4_fp4_b200_vllm_mtp.sh rename to benchmarks/single_node/fixed_seq_len/deprecated/dsv4_fp4_b200_vllm_mtp.sh diff --git a/benchmarks/single_node/fixed_seq_len/dsv4_fp4_b300_sglang.sh b/benchmarks/single_node/fixed_seq_len/deprecated/dsv4_fp4_b300_sglang.sh similarity index 100% rename from benchmarks/single_node/fixed_seq_len/dsv4_fp4_b300_sglang.sh rename to benchmarks/single_node/fixed_seq_len/deprecated/dsv4_fp4_b300_sglang.sh diff --git a/benchmarks/single_node/fixed_seq_len/dsv4_fp4_b300_sglang_mtp.sh b/benchmarks/single_node/fixed_seq_len/deprecated/dsv4_fp4_b300_sglang_mtp.sh similarity index 100% rename from benchmarks/single_node/fixed_seq_len/dsv4_fp4_b300_sglang_mtp.sh rename to benchmarks/single_node/fixed_seq_len/deprecated/dsv4_fp4_b300_sglang_mtp.sh diff --git a/benchmarks/single_node/fixed_seq_len/dsv4_fp4_b300_trt.sh b/benchmarks/single_node/fixed_seq_len/deprecated/dsv4_fp4_b300_trt.sh similarity index 100% rename from benchmarks/single_node/fixed_seq_len/dsv4_fp4_b300_trt.sh rename to benchmarks/single_node/fixed_seq_len/deprecated/dsv4_fp4_b300_trt.sh diff --git a/benchmarks/single_node/fixed_seq_len/dsv4_fp4_b300_trt_mtp.sh b/benchmarks/single_node/fixed_seq_len/deprecated/dsv4_fp4_b300_trt_mtp.sh similarity index 100% rename from benchmarks/single_node/fixed_seq_len/dsv4_fp4_b300_trt_mtp.sh rename to benchmarks/single_node/fixed_seq_len/deprecated/dsv4_fp4_b300_trt_mtp.sh diff --git a/benchmarks/single_node/fixed_seq_len/dsv4_fp4_b300_vllm.sh b/benchmarks/single_node/fixed_seq_len/deprecated/dsv4_fp4_b300_vllm.sh similarity index 100% rename from benchmarks/single_node/fixed_seq_len/dsv4_fp4_b300_vllm.sh rename to benchmarks/single_node/fixed_seq_len/deprecated/dsv4_fp4_b300_vllm.sh diff --git a/benchmarks/single_node/fixed_seq_len/dsv4_fp4_b300_vllm_mtp.sh b/benchmarks/single_node/fixed_seq_len/deprecated/dsv4_fp4_b300_vllm_mtp.sh similarity index 100% rename from benchmarks/single_node/fixed_seq_len/dsv4_fp4_b300_vllm_mtp.sh rename to benchmarks/single_node/fixed_seq_len/deprecated/dsv4_fp4_b300_vllm_mtp.sh diff --git a/benchmarks/single_node/fixed_seq_len/dsv4_fp4_mi355x_atom.sh b/benchmarks/single_node/fixed_seq_len/deprecated/dsv4_fp4_mi355x_atom.sh similarity index 100% rename from benchmarks/single_node/fixed_seq_len/dsv4_fp4_mi355x_atom.sh rename to benchmarks/single_node/fixed_seq_len/deprecated/dsv4_fp4_mi355x_atom.sh diff --git a/benchmarks/single_node/fixed_seq_len/dsv4_fp4_mi355x_atom_mtp.sh b/benchmarks/single_node/fixed_seq_len/deprecated/dsv4_fp4_mi355x_atom_mtp.sh similarity index 100% rename from benchmarks/single_node/fixed_seq_len/dsv4_fp4_mi355x_atom_mtp.sh rename to benchmarks/single_node/fixed_seq_len/deprecated/dsv4_fp4_mi355x_atom_mtp.sh diff --git a/benchmarks/single_node/fixed_seq_len/dsv4_fp4_mi355x_sglang.sh b/benchmarks/single_node/fixed_seq_len/deprecated/dsv4_fp4_mi355x_sglang.sh similarity index 100% rename from benchmarks/single_node/fixed_seq_len/dsv4_fp4_mi355x_sglang.sh rename to benchmarks/single_node/fixed_seq_len/deprecated/dsv4_fp4_mi355x_sglang.sh diff --git a/benchmarks/single_node/fixed_seq_len/dsv4_fp4_mi355x_sglang_mtp.sh b/benchmarks/single_node/fixed_seq_len/deprecated/dsv4_fp4_mi355x_sglang_mtp.sh similarity index 100% rename from benchmarks/single_node/fixed_seq_len/dsv4_fp4_mi355x_sglang_mtp.sh rename to benchmarks/single_node/fixed_seq_len/deprecated/dsv4_fp4_mi355x_sglang_mtp.sh diff --git a/benchmarks/single_node/fixed_seq_len/dsv4_fp4_mi355x_vllm.sh b/benchmarks/single_node/fixed_seq_len/deprecated/dsv4_fp4_mi355x_vllm.sh similarity index 100% rename from benchmarks/single_node/fixed_seq_len/dsv4_fp4_mi355x_vllm.sh rename to benchmarks/single_node/fixed_seq_len/deprecated/dsv4_fp4_mi355x_vllm.sh diff --git a/benchmarks/single_node/fixed_seq_len/dsv4_fp4_mi355x_vllm_mtp.sh b/benchmarks/single_node/fixed_seq_len/deprecated/dsv4_fp4_mi355x_vllm_mtp.sh similarity index 100% rename from benchmarks/single_node/fixed_seq_len/dsv4_fp4_mi355x_vllm_mtp.sh rename to benchmarks/single_node/fixed_seq_len/deprecated/dsv4_fp4_mi355x_vllm_mtp.sh diff --git a/benchmarks/single_node/fixed_seq_len/dsv4_fp8_h200.sh b/benchmarks/single_node/fixed_seq_len/deprecated/dsv4_fp8_h200.sh similarity index 100% rename from benchmarks/single_node/fixed_seq_len/dsv4_fp8_h200.sh rename to benchmarks/single_node/fixed_seq_len/deprecated/dsv4_fp8_h200.sh diff --git a/benchmarks/single_node/fixed_seq_len/dsv4_fp8_h200_mtp.sh b/benchmarks/single_node/fixed_seq_len/deprecated/dsv4_fp8_h200_mtp.sh similarity index 100% rename from benchmarks/single_node/fixed_seq_len/dsv4_fp8_h200_mtp.sh rename to benchmarks/single_node/fixed_seq_len/deprecated/dsv4_fp8_h200_mtp.sh diff --git a/benchmarks/single_node/fixed_seq_len/dsv4_fp8_h200_sglang.sh b/benchmarks/single_node/fixed_seq_len/deprecated/dsv4_fp8_h200_sglang.sh similarity index 100% rename from benchmarks/single_node/fixed_seq_len/dsv4_fp8_h200_sglang.sh rename to benchmarks/single_node/fixed_seq_len/deprecated/dsv4_fp8_h200_sglang.sh diff --git a/benchmarks/single_node/fixed_seq_len/dsv4_fp8_h200_sglang_mtp.sh b/benchmarks/single_node/fixed_seq_len/deprecated/dsv4_fp8_h200_sglang_mtp.sh similarity index 100% rename from benchmarks/single_node/fixed_seq_len/dsv4_fp8_h200_sglang_mtp.sh rename to benchmarks/single_node/fixed_seq_len/deprecated/dsv4_fp8_h200_sglang_mtp.sh diff --git a/benchmarks/single_node/fixed_seq_len/dsv4_fp8_mi300x.sh b/benchmarks/single_node/fixed_seq_len/deprecated/dsv4_fp8_mi300x.sh similarity index 100% rename from benchmarks/single_node/fixed_seq_len/dsv4_fp8_mi300x.sh rename to benchmarks/single_node/fixed_seq_len/deprecated/dsv4_fp8_mi300x.sh diff --git a/benchmarks/single_node/fixed_seq_len/dsv4_fp8_mi300x_mtp.sh b/benchmarks/single_node/fixed_seq_len/deprecated/dsv4_fp8_mi300x_mtp.sh similarity index 100% rename from benchmarks/single_node/fixed_seq_len/dsv4_fp8_mi300x_mtp.sh rename to benchmarks/single_node/fixed_seq_len/deprecated/dsv4_fp8_mi300x_mtp.sh diff --git a/benchmarks/single_node/fixed_seq_len/dsv4_fp8_mi325x.sh b/benchmarks/single_node/fixed_seq_len/deprecated/dsv4_fp8_mi325x.sh similarity index 100% rename from benchmarks/single_node/fixed_seq_len/dsv4_fp8_mi325x.sh rename to benchmarks/single_node/fixed_seq_len/deprecated/dsv4_fp8_mi325x.sh diff --git a/benchmarks/single_node/fixed_seq_len/dsv4_fp8_mi325x_mtp.sh b/benchmarks/single_node/fixed_seq_len/deprecated/dsv4_fp8_mi325x_mtp.sh similarity index 100% rename from benchmarks/single_node/fixed_seq_len/dsv4_fp8_mi325x_mtp.sh rename to benchmarks/single_node/fixed_seq_len/deprecated/dsv4_fp8_mi325x_mtp.sh diff --git a/configs/amd-master.yaml b/configs/amd-master.yaml index 18007db0dd..7ed680d85f 100644 --- a/configs/amd-master.yaml +++ b/configs/amd-master.yaml @@ -92,53 +92,6 @@ dsr1-fp8-mi325x-sglang: search-space: - { tp: 8, conc-start: 4, conc-end: 64 } -# DeepSeek-V4-Pro FP8 single-node on MI300X (gfx942) via vLLM. -# EXTRAPOLATED bring-up. sglang was abandoned: on gfx942 (no native FP4) the -# dsv4 sglang backend's nvfp4 MoE / TileLang-MLA kernels have no gfx942 build -# (gfx950/MI355X only). vLLM runs the checkpoint in FP8 via --quantization -# deepseek_v4_fp8 (dequant FP4 MoE -> FP8), the same path the H200 dsv4 vLLM -# recipe uses. Config mirrors the same-model dsv4-fp4-mi355x-vllm (TP8, conc -# 4-512); the FP4->FP8 dequant (~1.05TB) fits 8x192GB only at TP8. Launch -# script dsv4_fp8_mi300x.sh carries the deepseek_v4 + gfx942 AITER flags. -dsv4-fp8-mi300x-vllm: - image: vllm/vllm-openai-rocm:nightly-09663abde0f50944a8d5ea30120666024b503faa - model: deepseek-ai/DeepSeek-V4-Pro - model-prefix: dsv4 - runner: mi300x - precision: fp8 - framework: vllm - multinode: false - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - # 8k1k KV fits only ~20x concurrency (9472-token requests), so conc512 is - # ~25x oversubscribed -> request timeouts. conc256 is proven green; cap here. - - { tp: 8, conc-start: 4, conc-end: 256 } - -# MTP variant of dsv4-fp8-mi300x-vllm. Mirrors the base recipe and adds -# DeepSeek-V4 built-in MTP via --speculative-config (num_speculative_tokens=2), -# routing to dsv4_fp8_mi300x_mtp.sh; benchmark uses --dsv4 chat-template -# encoding (required for meaningful MTP acceptance). -dsv4-fp8-mi300x-vllm-mtp: - image: vllm/vllm-openai-rocm:nightly-09663abde0f50944a8d5ea30120666024b503faa - model: deepseek-ai/DeepSeek-V4-Pro - model-prefix: dsv4 - runner: mi300x - precision: fp8 - framework: vllm - multinode: false - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - # 8k1k MTP KV is tighter than normal (draft model): conc256 failed here - # (19.2% req failures) and conc512 was 55.8%; conc128 passed cleanly. - # Cap 8k1k MTP at 128 (normal holds 256, 1k1k holds 512). - - { tp: 8, conc-start: 4, conc-end: 128, spec-decoding: mtp } - dsr1-fp8-mi355x-sglang: image: lmsysorg/sglang:v0.5.12-rocm700-mi35x model: deepseek-ai/DeepSeek-R1-0528 @@ -986,305 +939,6 @@ dsr1-fp4-mi355x-sglang-disagg-8k1k-mtp: - "DECODE_NODES=1" - "DECODE_MTP_SIZE=1" - -dsv4-fp4-mi355x-sglang-disagg: - image: lmsysorg/sglang-rocm:v0.5.14-rocm720-mi35x-20260701 - model: deepseek-ai/DeepSeek-V4-Pro - model-prefix: dsv4 - runner: cluster:mi355x-amds - precision: fp4 - framework: sglang-disagg - router: { name: sglang-router, version: "0.3.2" } - kv-p2p-transfer: mori - multinode: true - disagg: true - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - # non-MTP configurations - # 1P1D pure TP8 (mori KV transfer) - - spec-decoding: "none" - conc-list: [ 1, 2, 4, 8, 16, 32, 64, 128 ] - prefill: - num-worker: 1 - tp: 8 - ep: 1 - dp-attn: false - additional-settings: - - "PREFILL_NODES=1" - decode: - num-worker: 1 - tp: 8 - ep: 1 - dp-attn: false - additional-settings: - - "DECODE_NODES=1" - - "DECODE_MTP_SIZE=0" - # 1P1D DEP8 (mori KV transfer + mori MoE a2a, dp-attention) - - spec-decoding: "none" - conc-list: [ 256 ] - prefill: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - additional-settings: - - "PREFILL_NODES=1" - decode: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - additional-settings: - - "DECODE_NODES=1" - - "DECODE_MTP_SIZE=0" - - - spec-decoding: "none" - conc-list: [ 512 ] - prefill: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - additional-settings: - - "PREFILL_NODES=1" - decode: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - additional-settings: - - "DECODE_NODES=1" - - "DECODE_MTP_SIZE=0" - -# MTP variant of dsv4-fp4-mi355x-sglang-disagg (base run 29821778206). Mirrors -# the base disagg search space and topologies exactly (same 20260701 image, -# mori KV transfer + MoE a2a, sglang-router --pd-disaggregation) and enables -# DeepSeek-V4 built-in MTP on the decode side via spec-decoding: mtp — the same -# enablement path as dsr1-fp8-mi355x-sglang-disagg-mtp. DECODE_MTP_SIZE=1 (draft -# depth 1) on every arm. Depth 2 was tried on the pure-TP8 (bottom-of-curve) arm, -# but this 20260701 build's 2-step EAGLE verification is lossy: greedy gsm8k fell -# to ~0.85 (below the 0.91 dsv4 floor), while depth 1 is lossless (~0.96, matching -# the spec-none base). So all arms use depth 1. The dsv4 ROCm MTP -# CUDA-graph fix (sgl-project/sglang#26383, merged 2026-05-27) is already -# present in the pinned 20260701 nightly, so no image bump is needed. -dsv4-fp4-mi355x-sglang-disagg-mtp: - image: lmsysorg/sglang-rocm:v0.5.15-rocm720-mi35x-20260713 - model: deepseek-ai/DeepSeek-V4-Pro - model-prefix: dsv4 - runner: cluster:mi355x-amds - precision: fp4 - framework: sglang-disagg - router: { name: sglang-router, version: "0.3.2" } - kv-p2p-transfer: mori - multinode: true - disagg: true - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - # MTP configurations - # 1P1D pure TP8 (mori KV transfer) - - spec-decoding: "mtp" - conc-list: [ 1, 2, 4, 8, 16, 32, 64, 128 ] - prefill: - num-worker: 1 - tp: 8 - ep: 1 - dp-attn: false - additional-settings: - - "PREFILL_NODES=1" - decode: - num-worker: 1 - tp: 8 - ep: 1 - dp-attn: false - additional-settings: - - "DECODE_NODES=1" - - "DECODE_MTP_SIZE=2" - # 1P1D DEP8 (mori KV transfer + mori MoE a2a, dp-attention) - - spec-decoding: "mtp" - conc-list: [ 256 ] - prefill: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - additional-settings: - - "PREFILL_NODES=1" - decode: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - additional-settings: - - "DECODE_NODES=1" - - "DECODE_MTP_SIZE=1" - - - spec-decoding: "mtp" - conc-list: [ 512 ] - prefill: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - additional-settings: - - "PREFILL_NODES=1" - decode: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - additional-settings: - - "DECODE_NODES=1" - - "DECODE_MTP_SIZE=1" - - -dsv4-fp4-mi355x-sglang: - image: lmsysorg/sglang-rocm:v0.5.14-rocm720-mi35x-20260706 - model: deepseek-ai/DeepSeek-V4-Pro - model-prefix: dsv4 - runner: mi355x - precision: fp4 - framework: sglang - multinode: false - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - - { tp: 8, dp-attn: true, conc-start: 64, conc-end: 2048 } - - { tp: 4, dp-attn: true, conc-start: 16, conc-end: 128 } - - { tp: 4, dp-attn: false, conc-start: 1, conc-end: 32 } - -# MTP variant of dsv4-fp4-mi355x-sglang. Mirrors the base search space and adds -# spec-decoding: mtp, which routes to dsv4_fp4_mi355x_sglang_mtp.sh (EAGLE -# speculative decoding), per sgl-project/sglang#26383 ([AMD][DSV4] DSV4 MTP -# graph + sparse triton attn optimizations, merged to main 2026-05-27). That PR -# fixes the ROCm HIP-radix MTP CUDA-graph bug (the false-EOS symptom in sgl -# #20404) and validates GSM8K 0.950 with MTP on. -dsv4-fp4-mi355x-sglang-mtp: - image: lmsysorg/sglang-rocm:v0.5.19-rocm720-mi35x-20260907 - model: deepseek-ai/DeepSeek-V4-Pro - model-prefix: dsv4 - runner: mi355x - precision: fp4 - framework: sglang - multinode: false - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - - { tp: 8, dp-attn: true, conc-start: 64, conc-end: 2048, spec-decoding: mtp } - - { tp: 8, dp-attn: false, conc-list: [1, 2, 4, 16, 32], spec-decoding: mtp } - -# DSv4 on MI355X via vLLM, using the official vllm/vllm-openai-rocm -# nightly image. DSv4 base ROCm support (vllm-project/vllm#40871) merged -# on 2026-05-05, so any nightly built after that includes the -# DeepseekV4ForCausalLM model class. -# -# IMPORTANT: pin to a digest-suffixed nightly tag rather than the -# floating `:nightly`. launch_mi355x-amds.sh caches enroot squashfs -# files keyed on the image string and short-circuits re-import if the -# file already exists, so the floating tag silently keeps a stale build -# even after Docker Hub updates `:nightly`. -# -# DeepSeek-V4-Pro is FP4+FP8 mixed (FP4 MoE expert weights, FP8 for the -# rest); InferenceX classifies this as fp4 — same as the sister sglang -# and atom DSv4 mi355x entries below. The serving recipe enables AITER, -# shared-expert fusion, and INT4 quick-reduce explicitly; the pinned nightly -# then auto-selects its ROCm DeepSeek-V4 mHC, sparse-indexer, attention, and -# compressor-GEMM optimizations. Shared-expert fusion (FHMoE) needs -# vllm-project/vllm#53161, first shipped in this nightly; on earlier builds -# it self-disabled because this checkpoint path was not eligible. -dsv4-fp4-mi355x-vllm: - image: vllm/vllm-openai-rocm:nightly-d9105ea8001e0a6d77a96327d17515bb5791fb36 - model: deepseek-ai/DeepSeek-V4-Pro - model-prefix: dsv4 - runner: mi355x - precision: fp4 - framework: vllm - multinode: false - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - - { tp: 8, conc-start: 4, conc-end: 512 } - -# MTP variant of dsv4-fp4-mi355x-vllm. Mirrors the base recipe's search space -# and adds spec-decoding: mtp, which routes to dsv4_fp4_mi355x_vllm_mtp.sh -# (--speculative-config '{"method":"mtp","num_speculative_tokens":2}'), per -# vllm-project/vllm#43385 (ROCm DeepSeek-V4 MTP, merged 2026-05-24, included in -# v0.22.0). Full conc 4-512 range maps the complete crossover curve: MTP wins -# at low batch (PR perf data: +75% @ conc1, +38% @ conc8) and falls behind STP -# above ~conc32 (-37% @ conc32). Image reuses the base entry's pinned ROCm -# nightly, which includes the MTP path and subsequent DSV4 kernel updates. -dsv4-fp4-mi355x-vllm-mtp: - image: vllm/vllm-openai-rocm:nightly-d9105ea8001e0a6d77a96327d17515bb5791fb36 - model: deepseek-ai/DeepSeek-V4-Pro - model-prefix: dsv4 - runner: mi355x - precision: fp4 - framework: vllm - multinode: false - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - - { tp: 8, conc-start: 4, conc-end: 512, spec-decoding: mtp } - -dsv4-fp4-mi355x-atom: - image: rocm/atom-dev:nightly_202607231538 - model: deepseek-ai/DeepSeek-V4-Pro - model-prefix: dsv4 - runner: mi355x - precision: fp4 - framework: atom - multinode: false - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - # conc<64: tp4 dp-off (best at low conc); conc>=64: dp-on + TBO. - # tp8 dp-off low-conc kept as redundancy to recheck the crossover on the new image. - # dp-on cells always set max_req=conc (recipe .sh) to avoid OOM from the large - # default max_num_seqs (dp-attn keeps a full KV pool per rank); TBO stays conc>=64. - - { tp: 4, ep: 1, dp-attn: false, conc-start: 1, conc-end: 64 } - - { tp: 4, ep: 1, dp-attn: true, conc-start: 16, conc-end: 128 } - - { tp: 8, ep: 1, dp-attn: false, conc-start: 1, conc-end: 64 } - - { tp: 8, ep: 1, dp-attn: true, conc-start: 64, conc-end: 2048 } - -dsv4-fp4-mi355x-atom-mtp: - image: rocm/atom-dev:nightly_202607231538 - model: deepseek-ai/DeepSeek-V4-Pro - model-prefix: dsv4 - runner: mi355x - precision: fp4 - framework: atom - multinode: false - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - # Aligns ATOM official DSv4 MTP (models.json): MTP3 dp-off at low-mid conc, - # DPA MTP3 (dp-attn) at mid-high conc. dp-on starts at c32 (dp-attn helps from c32). - # prefill-only TBO on dp-attn cells at conc>=256 (measured crossover: TBO hurts - # c64/c128 -10~14%, helps c256+ +8~14%; MTP removes low-conc latency so TBO there is - # pure overhead). MTP-safe — only decode-TBO (--enable-tbo all) drops spec metadata. - - { tp: 8, ep: 1, dp-attn: false, conc-start: 4, conc-end: 256, spec-decoding: mtp } - - { tp: 8, ep: 1, dp-attn: true, conc-start: 32, conc-end: 1024, spec-decoding: mtp } - # tp4 removed: tp4+MTP+dp-on OOMs — 806G/4=201G/GPU weight leaves KV budget - # NEGATIVE (-10.98GB) even at gpu-mem-util 0.90 (run 30277221626). #2327 tp4 - # works only because it's non-MTP (no draft KV layer). MTP stays tp8-only. - dsr1-fp8-mi325x-sglang-mtp: image: lmsysorg/sglang:v0.5.12-rocm700-mi30x model: deepseek-ai/DeepSeek-R1-0528 @@ -1481,58 +1135,6 @@ dsr1-fp4-mi355x-sglang-disagg-mtp: - "DECODE_NODES=1" - "DECODE_MTP_SIZE=1" -# MiniMax-M3 MXFP8 MI355X recipe: -# https://github.com/vllm-project/recipes/commit/2a3728ed9892debfd767a72a58ebc90b33f186e5 -# MXFP8 runs from TP=4 on gfx950; block size 128 is mandatory for MSA. -dsv4-fp4-mi355x-atom-disagg: - image: rocm/atom-dev:nightly_202606101403 - model: deepseek-ai/DeepSeek-V4-Pro - model-prefix: dsv4 - runner: mi355x - precision: fp4 - framework: atom-disagg - router: { name: atomesh, version: "087b82d9c1f630e79149ba37e6213257ec9a76f8" } - kv-p2p-transfer: mooncake - multinode: true - disagg: true - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - # 2P1D DPA+TP8 - - conc-list: [ 256, 512, 768, 1024, 2048 ] - prefill: - num-worker: 2 - tp: 8 - ep: 1 - dp-attn: true - additional-settings: - - "PREFILL_NODES=2" - decode: - num-worker: 1 - tp: 8 - ep: 1 - dp-attn: true - additional-settings: - - "DECODE_NODES=1" - # 1P1D TP8 - - conc-list: [ 4, 8, 16, 32, 64, 128 ] - prefill: - num-worker: 1 - tp: 8 - ep: 1 - dp-attn: false - additional-settings: - - "PREFILL_NODES=1" - decode: - num-worker: 1 - tp: 8 - ep: 1 - dp-attn: false - additional-settings: - - "DECODE_NODES=1" - # 1P1D TP8 minimaxm3-fp8-mi300x-vllm-agentic-mtp: image: vllm/vllm-openai-rocm:nightly-d9105ea8001e0a6d77a96327d17515bb5791fb36 model: MiniMaxAI/MiniMax-M3-MXFP8 @@ -1634,56 +1236,6 @@ dsv4-fp4-mi355x-sglang-disagg-agentic-hicache-mtp: - "DECODE_NODES=1" - "DECODE_MTP_SIZE=3" -# DeepSeek-V4-Pro FP8 single-node on MI325X (gfx942) via vLLM. -# EXTRAPOLATED bring-up. Same rationale as dsv4-fp8-mi300x-vllm: sglang has no -# gfx942 build of the dsv4 nvfp4 MoE / TileLang-MLA kernels, so vLLM runs the -# checkpoint in FP8 via --quantization deepseek_v4_fp8 (dequant FP4 MoE -> FP8), -# the H200 dsv4 vLLM path. Config mirrors the same-model dsv4-fp4-mi355x-vllm -# (TP8, conc 4-512); 8x256GB (2TB) has ample headroom for the ~1.05TB FP8 -# footprint. Launch script dsv4_fp8_mi325x.sh carries the deepseek_v4 + gfx942 -# AITER flags. -dsv4-fp8-mi325x-vllm: - image: vllm/vllm-openai-rocm:nightly-09663abde0f50944a8d5ea30120666024b503faa - model: deepseek-ai/DeepSeek-V4-Pro - model-prefix: dsv4 - # mi325x (mi325x-amds pool) runners are down; use the mi325x-tw cluster. - runner: cluster:mi325x-tw - precision: fp8 - framework: vllm - multinode: false - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - # 8k1k KV fits only ~20x concurrency (9472-token requests), so conc512 is - # ~25x oversubscribed -> request timeouts. conc256 is proven green; cap here. - - { tp: 8, conc-start: 4, conc-end: 256 } - -# MTP variant of dsv4-fp8-mi325x-vllm. Mirrors the base recipe and adds -# DeepSeek-V4 built-in MTP via --speculative-config (num_speculative_tokens=2), -# routing to dsv4_fp8_mi325x_mtp.sh; benchmark uses --dsv4 chat-template -# encoding (required for meaningful MTP acceptance). -dsv4-fp8-mi325x-vllm-mtp: - image: vllm/vllm-openai-rocm:nightly-09663abde0f50944a8d5ea30120666024b503faa - model: deepseek-ai/DeepSeek-V4-Pro - model-prefix: dsv4 - # mi325x (mi325x-amds pool) runners are down; use the mi325x-tw cluster. - runner: cluster:mi325x-tw - precision: fp8 - framework: vllm - multinode: false - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - # 8k1k MTP KV is tighter than normal (draft model): conc256 is borderline - # (MI300X 19.2% req failures, flaked pass->fail across runs) and conc512 - # is 55.8%. conc128 passed cleanly on the memory-tightest SKU (MI300X); - # cap 8k1k MTP at 128 (normal holds 256, 1k1k holds 512). - - { tp: 8, conc-start: 4, conc-end: 128, spec-decoding: mtp } - minimaxm3-fp4-mi355x-vllm-agentic-mtp: image: vllm/vllm-openai-rocm:nightly-8a728663c1c3eeace834a95f5654fa653cc1998c model: amd/MiniMax-M3-MXFP4 @@ -1740,8 +1292,6 @@ glm5.2-fp4-mi355x-atom-agentic-mtp: - { tp: 4, kv-offloading: dram, kv-offload-backend: { name: lmcache, version: "0.4.5" }, conc-list: [2, 4, 8, 10], spec-decoding: mtp } - { tp: 8, kv-offloading: none, conc-list: [1, 2, 4], spec-decoding: mtp } - - dsv4-fp4-mi355x-sglang-agentic-mtp: image: lmsysorg/sglang-rocm:v0.5.18-rocm720-mi35x-20260902 model: deepseek-ai/DeepSeek-V4-Pro diff --git a/configs/deprecated/amd-dsv4-8k1k-master.yaml b/configs/deprecated/amd-dsv4-8k1k-master.yaml new file mode 100644 index 0000000000..16f0077c19 --- /dev/null +++ b/configs/deprecated/amd-dsv4-8k1k-master.yaml @@ -0,0 +1,452 @@ +# Deprecated DeepSeek-V4-Pro 1.6T Single-turn 8k1k entries archived from +# amd-master.yaml. Deprecated after Tuesday, September 8, 2026 (see the +# Deprecation Notice in MODELS.md). Agentic coding is unaffected and stays +# active for dsv4, including its MTP and DSpark arms; the model is not retired. +# Removed from the active master config so sweep generation no longer selects them. + +# DeepSeek-V4-Pro FP8 single-node on MI300X (gfx942) via vLLM. +# EXTRAPOLATED bring-up. sglang was abandoned: on gfx942 (no native FP4) the +# dsv4 sglang backend's nvfp4 MoE / TileLang-MLA kernels have no gfx942 build +# (gfx950/MI355X only). vLLM runs the checkpoint in FP8 via --quantization +# deepseek_v4_fp8 (dequant FP4 MoE -> FP8), the same path the H200 dsv4 vLLM +# recipe uses. Config mirrors the same-model dsv4-fp4-mi355x-vllm (TP8, conc +# 4-512); the FP4->FP8 dequant (~1.05TB) fits 8x192GB only at TP8. Launch +# script dsv4_fp8_mi300x.sh carries the deepseek_v4 + gfx942 AITER flags. +dsv4-fp8-mi300x-vllm: + image: vllm/vllm-openai-rocm:nightly-09663abde0f50944a8d5ea30120666024b503faa + model: deepseek-ai/DeepSeek-V4-Pro + model-prefix: dsv4 + runner: mi300x + precision: fp8 + framework: vllm + multinode: false + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + # 8k1k KV fits only ~20x concurrency (9472-token requests), so conc512 is + # ~25x oversubscribed -> request timeouts. conc256 is proven green; cap here. + - { tp: 8, conc-start: 4, conc-end: 256 } + +# MTP variant of dsv4-fp8-mi300x-vllm. Mirrors the base recipe and adds +# DeepSeek-V4 built-in MTP via --speculative-config (num_speculative_tokens=2), +# routing to dsv4_fp8_mi300x_mtp.sh; benchmark uses --dsv4 chat-template +# encoding (required for meaningful MTP acceptance). +dsv4-fp8-mi300x-vllm-mtp: + image: vllm/vllm-openai-rocm:nightly-09663abde0f50944a8d5ea30120666024b503faa + model: deepseek-ai/DeepSeek-V4-Pro + model-prefix: dsv4 + runner: mi300x + precision: fp8 + framework: vllm + multinode: false + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + # 8k1k MTP KV is tighter than normal (draft model): conc256 failed here + # (19.2% req failures) and conc512 was 55.8%; conc128 passed cleanly. + # Cap 8k1k MTP at 128 (normal holds 256, 1k1k holds 512). + - { tp: 8, conc-start: 4, conc-end: 128, spec-decoding: mtp } + +dsv4-fp4-mi355x-sglang-disagg: + image: lmsysorg/sglang-rocm:v0.5.14-rocm720-mi35x-20260701 + model: deepseek-ai/DeepSeek-V4-Pro + model-prefix: dsv4 + runner: cluster:mi355x-amds + precision: fp4 + framework: sglang-disagg + router: { name: sglang-router, version: "0.3.2" } + kv-p2p-transfer: mori + multinode: true + disagg: true + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + # non-MTP configurations + # 1P1D pure TP8 (mori KV transfer) + - spec-decoding: "none" + conc-list: [ 1, 2, 4, 8, 16, 32, 64, 128 ] + prefill: + num-worker: 1 + tp: 8 + ep: 1 + dp-attn: false + additional-settings: + - "PREFILL_NODES=1" + decode: + num-worker: 1 + tp: 8 + ep: 1 + dp-attn: false + additional-settings: + - "DECODE_NODES=1" + - "DECODE_MTP_SIZE=0" + # 1P1D DEP8 (mori KV transfer + mori MoE a2a, dp-attention) + - spec-decoding: "none" + conc-list: [ 256 ] + prefill: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + additional-settings: + - "PREFILL_NODES=1" + decode: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + additional-settings: + - "DECODE_NODES=1" + - "DECODE_MTP_SIZE=0" + + - spec-decoding: "none" + conc-list: [ 512 ] + prefill: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + additional-settings: + - "PREFILL_NODES=1" + decode: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + additional-settings: + - "DECODE_NODES=1" + - "DECODE_MTP_SIZE=0" + +# MTP variant of dsv4-fp4-mi355x-sglang-disagg (base run 29821778206). Mirrors +# the base disagg search space and topologies exactly (same 20260701 image, +# mori KV transfer + MoE a2a, sglang-router --pd-disaggregation) and enables +# DeepSeek-V4 built-in MTP on the decode side via spec-decoding: mtp — the same +# enablement path as dsr1-fp8-mi355x-sglang-disagg-mtp. DECODE_MTP_SIZE=1 (draft +# depth 1) on every arm. Depth 2 was tried on the pure-TP8 (bottom-of-curve) arm, +# but this 20260701 build's 2-step EAGLE verification is lossy: greedy gsm8k fell +# to ~0.85 (below the 0.91 dsv4 floor), while depth 1 is lossless (~0.96, matching +# the spec-none base). So all arms use depth 1. The dsv4 ROCm MTP +# CUDA-graph fix (sgl-project/sglang#26383, merged 2026-05-27) is already +# present in the pinned 20260701 nightly, so no image bump is needed. +dsv4-fp4-mi355x-sglang-disagg-mtp: + image: lmsysorg/sglang-rocm:v0.5.15-rocm720-mi35x-20260713 + model: deepseek-ai/DeepSeek-V4-Pro + model-prefix: dsv4 + runner: cluster:mi355x-amds + precision: fp4 + framework: sglang-disagg + router: { name: sglang-router, version: "0.3.2" } + kv-p2p-transfer: mori + multinode: true + disagg: true + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + # MTP configurations + # 1P1D pure TP8 (mori KV transfer) + - spec-decoding: "mtp" + conc-list: [ 1, 2, 4, 8, 16, 32, 64, 128 ] + prefill: + num-worker: 1 + tp: 8 + ep: 1 + dp-attn: false + additional-settings: + - "PREFILL_NODES=1" + decode: + num-worker: 1 + tp: 8 + ep: 1 + dp-attn: false + additional-settings: + - "DECODE_NODES=1" + - "DECODE_MTP_SIZE=2" + # 1P1D DEP8 (mori KV transfer + mori MoE a2a, dp-attention) + - spec-decoding: "mtp" + conc-list: [ 256 ] + prefill: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + additional-settings: + - "PREFILL_NODES=1" + decode: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + additional-settings: + - "DECODE_NODES=1" + - "DECODE_MTP_SIZE=1" + + - spec-decoding: "mtp" + conc-list: [ 512 ] + prefill: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + additional-settings: + - "PREFILL_NODES=1" + decode: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + additional-settings: + - "DECODE_NODES=1" + - "DECODE_MTP_SIZE=1" + +dsv4-fp4-mi355x-sglang: + image: lmsysorg/sglang-rocm:v0.5.14-rocm720-mi35x-20260706 + model: deepseek-ai/DeepSeek-V4-Pro + model-prefix: dsv4 + runner: mi355x + precision: fp4 + framework: sglang + multinode: false + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - { tp: 8, dp-attn: true, conc-start: 64, conc-end: 2048 } + - { tp: 4, dp-attn: true, conc-start: 16, conc-end: 128 } + - { tp: 4, dp-attn: false, conc-start: 1, conc-end: 32 } + +# MTP variant of dsv4-fp4-mi355x-sglang. Mirrors the base search space and adds +# spec-decoding: mtp, which routes to dsv4_fp4_mi355x_sglang_mtp.sh (EAGLE +# speculative decoding), per sgl-project/sglang#26383 ([AMD][DSV4] DSV4 MTP +# graph + sparse triton attn optimizations, merged to main 2026-05-27). That PR +# fixes the ROCm HIP-radix MTP CUDA-graph bug (the false-EOS symptom in sgl +# #20404) and validates GSM8K 0.950 with MTP on. +dsv4-fp4-mi355x-sglang-mtp: + image: lmsysorg/sglang-rocm:v0.5.19-rocm720-mi35x-20260907 + model: deepseek-ai/DeepSeek-V4-Pro + model-prefix: dsv4 + runner: mi355x + precision: fp4 + framework: sglang + multinode: false + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - { tp: 8, dp-attn: true, conc-start: 64, conc-end: 2048, spec-decoding: mtp } + - { tp: 8, dp-attn: false, conc-list: [1, 2, 4, 16, 32], spec-decoding: mtp } + +# DSv4 on MI355X via vLLM, using the official vllm/vllm-openai-rocm +# nightly image. DSv4 base ROCm support (vllm-project/vllm#40871) merged +# on 2026-05-05, so any nightly built after that includes the +# DeepseekV4ForCausalLM model class. +# +# IMPORTANT: pin to a digest-suffixed nightly tag rather than the +# floating `:nightly`. launch_mi355x-amds.sh caches enroot squashfs +# files keyed on the image string and short-circuits re-import if the +# file already exists, so the floating tag silently keeps a stale build +# even after Docker Hub updates `:nightly`. +# +# DeepSeek-V4-Pro is FP4+FP8 mixed (FP4 MoE expert weights, FP8 for the +# rest); InferenceX classifies this as fp4 — same as the sister sglang +# and atom DSv4 mi355x entries below. The serving recipe enables AITER, +# shared-expert fusion, and INT4 quick-reduce explicitly; the pinned nightly +# then auto-selects its ROCm DeepSeek-V4 mHC, sparse-indexer, attention, and +# compressor-GEMM optimizations. Shared-expert fusion (FHMoE) needs +# vllm-project/vllm#53161, first shipped in this nightly; on earlier builds +# it self-disabled because this checkpoint path was not eligible. +dsv4-fp4-mi355x-vllm: + image: vllm/vllm-openai-rocm:nightly-d9105ea8001e0a6d77a96327d17515bb5791fb36 + model: deepseek-ai/DeepSeek-V4-Pro + model-prefix: dsv4 + runner: mi355x + precision: fp4 + framework: vllm + multinode: false + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - { tp: 8, conc-start: 4, conc-end: 512 } + +# MTP variant of dsv4-fp4-mi355x-vllm. Mirrors the base recipe's search space +# and adds spec-decoding: mtp, which routes to dsv4_fp4_mi355x_vllm_mtp.sh +# (--speculative-config '{"method":"mtp","num_speculative_tokens":2}'), per +# vllm-project/vllm#43385 (ROCm DeepSeek-V4 MTP, merged 2026-05-24, included in +# v0.22.0). Full conc 4-512 range maps the complete crossover curve: MTP wins +# at low batch (PR perf data: +75% @ conc1, +38% @ conc8) and falls behind STP +# above ~conc32 (-37% @ conc32). Image reuses the base entry's pinned ROCm +# nightly, which includes the MTP path and subsequent DSV4 kernel updates. +dsv4-fp4-mi355x-vllm-mtp: + image: vllm/vllm-openai-rocm:nightly-d9105ea8001e0a6d77a96327d17515bb5791fb36 + model: deepseek-ai/DeepSeek-V4-Pro + model-prefix: dsv4 + runner: mi355x + precision: fp4 + framework: vllm + multinode: false + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - { tp: 8, conc-start: 4, conc-end: 512, spec-decoding: mtp } + +dsv4-fp4-mi355x-atom: + image: rocm/atom-dev:nightly_202607231538 + model: deepseek-ai/DeepSeek-V4-Pro + model-prefix: dsv4 + runner: mi355x + precision: fp4 + framework: atom + multinode: false + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + # conc<64: tp4 dp-off (best at low conc); conc>=64: dp-on + TBO. + # tp8 dp-off low-conc kept as redundancy to recheck the crossover on the new image. + # dp-on cells always set max_req=conc (recipe .sh) to avoid OOM from the large + # default max_num_seqs (dp-attn keeps a full KV pool per rank); TBO stays conc>=64. + - { tp: 4, ep: 1, dp-attn: false, conc-start: 1, conc-end: 64 } + - { tp: 4, ep: 1, dp-attn: true, conc-start: 16, conc-end: 128 } + - { tp: 8, ep: 1, dp-attn: false, conc-start: 1, conc-end: 64 } + - { tp: 8, ep: 1, dp-attn: true, conc-start: 64, conc-end: 2048 } + +dsv4-fp4-mi355x-atom-mtp: + image: rocm/atom-dev:nightly_202607231538 + model: deepseek-ai/DeepSeek-V4-Pro + model-prefix: dsv4 + runner: mi355x + precision: fp4 + framework: atom + multinode: false + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + # Aligns ATOM official DSv4 MTP (models.json): MTP3 dp-off at low-mid conc, + # DPA MTP3 (dp-attn) at mid-high conc. dp-on starts at c32 (dp-attn helps from c32). + # prefill-only TBO on dp-attn cells at conc>=256 (measured crossover: TBO hurts + # c64/c128 -10~14%, helps c256+ +8~14%; MTP removes low-conc latency so TBO there is + # pure overhead). MTP-safe — only decode-TBO (--enable-tbo all) drops spec metadata. + - { tp: 8, ep: 1, dp-attn: false, conc-start: 4, conc-end: 256, spec-decoding: mtp } + - { tp: 8, ep: 1, dp-attn: true, conc-start: 32, conc-end: 1024, spec-decoding: mtp } + # tp4 removed: tp4+MTP+dp-on OOMs — 806G/4=201G/GPU weight leaves KV budget + # NEGATIVE (-10.98GB) even at gpu-mem-util 0.90 (run 30277221626). #2327 tp4 + # works only because it's non-MTP (no draft KV layer). MTP stays tp8-only. + +# MiniMax-M3 MXFP8 MI355X recipe: +# https://github.com/vllm-project/recipes/commit/2a3728ed9892debfd767a72a58ebc90b33f186e5 +# MXFP8 runs from TP=4 on gfx950; block size 128 is mandatory for MSA. +dsv4-fp4-mi355x-atom-disagg: + image: rocm/atom-dev:nightly_202606101403 + model: deepseek-ai/DeepSeek-V4-Pro + model-prefix: dsv4 + runner: mi355x + precision: fp4 + framework: atom-disagg + router: { name: atomesh, version: "087b82d9c1f630e79149ba37e6213257ec9a76f8" } + kv-p2p-transfer: mooncake + multinode: true + disagg: true + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + # 2P1D DPA+TP8 + - conc-list: [ 256, 512, 768, 1024, 2048 ] + prefill: + num-worker: 2 + tp: 8 + ep: 1 + dp-attn: true + additional-settings: + - "PREFILL_NODES=2" + decode: + num-worker: 1 + tp: 8 + ep: 1 + dp-attn: true + additional-settings: + - "DECODE_NODES=1" + # 1P1D TP8 + - conc-list: [ 4, 8, 16, 32, 64, 128 ] + prefill: + num-worker: 1 + tp: 8 + ep: 1 + dp-attn: false + additional-settings: + - "PREFILL_NODES=1" + decode: + num-worker: 1 + tp: 8 + ep: 1 + dp-attn: false + additional-settings: + - "DECODE_NODES=1" + # 1P1D TP8 + +# DeepSeek-V4-Pro FP8 single-node on MI325X (gfx942) via vLLM. +# EXTRAPOLATED bring-up. Same rationale as dsv4-fp8-mi300x-vllm: sglang has no +# gfx942 build of the dsv4 nvfp4 MoE / TileLang-MLA kernels, so vLLM runs the +# checkpoint in FP8 via --quantization deepseek_v4_fp8 (dequant FP4 MoE -> FP8), +# the H200 dsv4 vLLM path. Config mirrors the same-model dsv4-fp4-mi355x-vllm +# (TP8, conc 4-512); 8x256GB (2TB) has ample headroom for the ~1.05TB FP8 +# footprint. Launch script dsv4_fp8_mi325x.sh carries the deepseek_v4 + gfx942 +# AITER flags. +dsv4-fp8-mi325x-vllm: + image: vllm/vllm-openai-rocm:nightly-09663abde0f50944a8d5ea30120666024b503faa + model: deepseek-ai/DeepSeek-V4-Pro + model-prefix: dsv4 + # mi325x (mi325x-amds pool) runners are down; use the mi325x-tw cluster. + runner: cluster:mi325x-tw + precision: fp8 + framework: vllm + multinode: false + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + # 8k1k KV fits only ~20x concurrency (9472-token requests), so conc512 is + # ~25x oversubscribed -> request timeouts. conc256 is proven green; cap here. + - { tp: 8, conc-start: 4, conc-end: 256 } + +# MTP variant of dsv4-fp8-mi325x-vllm. Mirrors the base recipe and adds +# DeepSeek-V4 built-in MTP via --speculative-config (num_speculative_tokens=2), +# routing to dsv4_fp8_mi325x_mtp.sh; benchmark uses --dsv4 chat-template +# encoding (required for meaningful MTP acceptance). +dsv4-fp8-mi325x-vllm-mtp: + image: vllm/vllm-openai-rocm:nightly-09663abde0f50944a8d5ea30120666024b503faa + model: deepseek-ai/DeepSeek-V4-Pro + model-prefix: dsv4 + # mi325x (mi325x-amds pool) runners are down; use the mi325x-tw cluster. + runner: cluster:mi325x-tw + precision: fp8 + framework: vllm + multinode: false + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + # 8k1k MTP KV is tighter than normal (draft model): conc256 is borderline + # (MI300X 19.2% req failures, flaked pass->fail across runs) and conc512 + # is 55.8%. conc128 passed cleanly on the memory-tightest SKU (MI300X); + # cap 8k1k MTP at 128 (normal holds 256, 1k1k holds 512). + - { tp: 8, conc-start: 4, conc-end: 128, spec-decoding: mtp } diff --git a/configs/deprecated/nvidia-dsv4-8k1k-master.yaml b/configs/deprecated/nvidia-dsv4-8k1k-master.yaml new file mode 100644 index 0000000000..a6cb64aa59 --- /dev/null +++ b/configs/deprecated/nvidia-dsv4-8k1k-master.yaml @@ -0,0 +1,2416 @@ +# Deprecated DeepSeek-V4-Pro 1.6T Single-turn 8k1k entries archived from +# nvidia-master.yaml. Deprecated after Tuesday, September 8, 2026 (see the +# Deprecation Notice in MODELS.md). Agentic coding is unaffected and stays +# active for dsv4, including its MTP and DSpark arms; the model is not retired. +# Removed from the active master config so sweep generation no longer selects them. + +dsv4-fp4-b200-sglang: + image: lmsysorg/sglang:nightly-dev-cu13-20260628-da802ddc + model: deepseek-ai/DeepSeek-V4-Pro + model-prefix: dsv4 + runner: cluster:b200-nscale + precision: fp4 + framework: sglang + multinode: false + # Two recipes from https://docs.sglang.io/cookbook/autoregressive/DeepSeek/DeepSeek-V4 + # are selected inside benchmarks/single_node/dsv4_fp4_b200.sh by DP_ATTENTION: + # low-latency (DP_ATTENTION=false): TP-only, flashinfer_mxfp4 + # DP-attention (DP_ATTENTION=true): DP-attn + DeepEP + mega_moe opts + # The DP-attention recipe covers both "balanced" (conc 64-128) and + # "max-throughput" (conc 256+) CONC ranges with identical flags; + # only --max-running-requests scales with CONC. + # ep is implicit in sglang: --moe-a2a-backend deepep forces ep_size=tp_size, + # while low-latency leaves ep_size at the default of 1. + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + # low-latency (DP_ATTENTION=false) + - { tp: 8, ep: 1, conc-start: 1, conc-end: 32 } + # DP-attention (DP_ATTENTION=true) — balanced CONC range + - { tp: 8, ep: 8, dp-attn: true, conc-start: 64, conc-end: 128 } + # DP-attention (DP_ATTENTION=true) — max-throughput CONC range + - { tp: 8, ep: 8, dp-attn: true, conc-start: 256, conc-end: 1024 } + +dsv4-fp4-b200-vllm: + image: vllm/vllm-openai:nightly-7c5dc571cbd1064ecc8a9b1045637ff647aa22cb + model: deepseek-ai/DeepSeek-V4-Pro + model-prefix: dsv4 + runner: cluster:b200-nscale + precision: fp4 + framework: vllm + multinode: false + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - { tp: 8, conc-start: 1, conc-end: 256 } + - { tp: 8, ep: 8, conc-start: 1, conc-end: 256 } + - { tp: 8, ep: 8, dp-attn: true, conc-start: 64, conc-end: 1024 } + +dsv4-fp4-b200-trt: + image: ghcr.io#semianalysisai/trtllm-deepseek-v4:feat-deepseek_v4-c185066 + model: deepseek-ai/DeepSeek-V4-Pro + model-prefix: dsv4 + runner: cluster:b200-nscale + precision: fp4 + framework: trt + multinode: false + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - { tp: 8, conc-start: 1, conc-end: 32 } + - { tp: 8, ep: 8, dp-attn: true, conc-start: 32, conc-end: 512 } + +dsv4-fp4-b200-trt-mtp: + image: ghcr.io#semianalysisai/trtllm-deepseek-v4:feat-deepseek_v4-c185066 + model: deepseek-ai/DeepSeek-V4-Pro + model-prefix: dsv4 + runner: cluster:b200-nscale + precision: fp4 + framework: trt + multinode: false + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - { tp: 8, conc-start: 1, conc-end: 32, spec-decoding: mtp } + - { tp: 8, ep: 8, dp-attn: true, conc-start: 32, conc-end: 256, spec-decoding: mtp } + +# MTP variant of dsv4-fp4-b200-vllm. Mirrors the base search space and adds +# --speculative-config '{"method":"mtp","num_speculative_tokens":2}'. +dsv4-fp4-b200-vllm-mtp: + image: vllm/vllm-openai:v0.25.0 + model: deepseek-ai/DeepSeek-V4-Pro + model-prefix: dsv4 + runner: cluster:b200-nscale + precision: fp4 + framework: vllm + multinode: false + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - { tp: 8, conc-start: 1, conc-end: 64, spec-decoding: mtp } + - { tp: 8, ep: 8, conc-start: 128, conc-end: 256, spec-decoding: mtp } + - { tp: 8, ep: 8, dp-attn: true, conc-start: 64, conc-end: 256, spec-decoding: mtp } + +# DeepSeek-V4-Pro on B300 with sglang (non-MTP). +# Uses nightly image with megamoe backend for high-concurrency profiles. +dsv4-fp4-b300-sglang: + image: lmsysorg/sglang:nightly-dev-cu13-20260624-b2c8f7a2 + model: deepseek-ai/DeepSeek-V4-Pro + model-prefix: dsv4 + runner: b300 + precision: fp4 + framework: sglang + multinode: false + # Recipes are selected inside benchmarks/single_node/dsv4_fp4_b300_sglang.sh + # by CONC: + # CONC 1|32: TP-only, flashinfer_mxfp4 + # CONC 512: DP-attn, flashinfer_mxfp4 + # CONC 2048-8192: DP-attn, megamoe + # ep is implicit in sglang: --moe-a2a-backend megamoe forces ep_size=tp_size, + # while low-latency leaves ep_size at the default of 1. + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - { tp: 8, ep: 1, conc-start: 1, conc-end: 1 } + - { tp: 4, ep: 1, conc-start: 32, conc-end: 32 } + - { tp: 4, ep: 1, dp-attn: true, conc-start: 512, conc-end: 512 } + - { tp: 8, ep: 8, dp-attn: true, conc-start: 2048, conc-end: 2048 } + - { tp: 8, ep: 8, dp-attn: true, conc-start: 4096, conc-end: 4096 } + +dsv4-fp4-b300-sglang-mtp: + image: lmsysorg/sglang:nightly-dev-cu13-20260610-f332e526 + model: deepseek-ai/DeepSeek-V4-Pro + model-prefix: dsv4 + runner: b300 + precision: fp4 + framework: sglang + multinode: false + # Three CONC bands: + # A: TP=8 ep=1 -- conc 1-8 EAGLE (3,1,4) TP-only fallback + # B: TP=4 ep=1 -- conc 4-32 EAGLE (3,1,4) TP-only mid batch + # C: TP=4 ep=1 dp-attn -- conc 16-256 EAGLE (1,1,2) DP-attn flashinfer + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - { tp: 8, ep: 1, conc-start: 1, conc-end: 8, spec-decoding: mtp } + - { tp: 4, ep: 1, conc-start: 4, conc-end: 32, spec-decoding: mtp } + +# DeepSeek-V4-Pro H200 recipe from https://vllm.ai/blog/deepseek-v4 +# Uses the cu129 image. H200 has no FP4 path, so the FP4 indexer cache +# flag is omitted. Max-model-len is pinned at 800k per the recipe. +dsv4-fp8-h200-vllm: + image: vllm/vllm-openai:v0.25.1 + model: deepseek-ai/DeepSeek-V4-Pro + model-prefix: dsv4 + runner: h200 + precision: fp8 + framework: vllm + multinode: false + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - { tp: 8, ep: 1, dp-attn: false, conc-start: 1, conc-end: 256 } + - { tp: 8, ep: 8, dp-attn: true, conc-start: 1, conc-end: 256 } + +# MTP variant of dsv4-fp8-h200-vllm. Mirrors the non-MTP image and adds +# --speculative-config '{"method":"mtp","num_speculative_tokens":2}'. +dsv4-fp8-h200-vllm-mtp: + image: vllm/vllm-openai:v0.25.1 + model: deepseek-ai/DeepSeek-V4-Pro + model-prefix: dsv4 + runner: h200 + precision: fp8 + framework: vllm + multinode: false + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - { tp: 8, ep: 1, dp-attn: false, conc-start: 1, conc-end: 256, spec-decoding: mtp } + - { tp: 8, ep: 8, dp-attn: true, conc-start: 1, conc-end: 256, spec-decoding: mtp } + +# DeepSeek-V4-Pro H200 single-node with SGLang (Marlin FP8, TP-only). +# Pinned to the h200-dgxc-slurm runner pool because the deepseek-v4-hopper +# image needs the /ix mount layout that only launch_h200-dgxc-slurm.sh sets up. + +# MTP variant of dsv4-fp8-h200-sglang. Mirrors the non-MTP recipe (same image, +# runner pool, search space) and adds EAGLE speculative decoding via +# --speculative-algorithm EAGLE with the (3,1,4) chain matching dsv4-fp4-b300-sglang-mtp. +dsv4-fp8-h200-sglang: + image: lmsysorg/sglang:deepseek-v4-hopper@sha256:1bf5d508ab110cc0fe1659a5f21d1be02a7f0d7ca8f58cea7e7f4e11f6ae208f + model: deepseek-ai/DeepSeek-V4-Pro + model-prefix: dsv4 + runner: h200-dgxc + precision: fp8 + framework: sglang + multinode: false + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - { tp: 8, ep: 1, conc-start: 1, conc-end: 1 } + - { tp: 8, ep: 1, conc-start: 4, conc-end: 64 } + +# MTP variant of dsv4-fp8-h200-sglang. Mirrors the non-MTP recipe (same image, +# runner pool, search space) and adds EAGLE speculative decoding via +# --speculative-algorithm EAGLE with the (3,1,4) chain matching dsv4-fp4-b300-sglang-mtp. +dsv4-fp8-h200-sglang-mtp: + image: lmsysorg/sglang:deepseek-v4-hopper@sha256:1bf5d508ab110cc0fe1659a5f21d1be02a7f0d7ca8f58cea7e7f4e11f6ae208f + model: deepseek-ai/DeepSeek-V4-Pro + model-prefix: dsv4 + runner: h200-dgxc + precision: fp8 + framework: sglang + multinode: false + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - { tp: 8, ep: 1, conc-start: 1, conc-end: 1, spec-decoding: mtp } + - { tp: 8, ep: 1, conc-start: 4, conc-end: 64, spec-decoding: mtp } + + # DeepSeek-V4-Pro B300 single-node aggregate recipe from the submitted B300 + # pareto sweep. The single-node schema has no explicit data-parallel-size + # field, so dp-attn=true is used as the existing vLLM script switch for DP4 + # layouts on 4 allocated GPUs. + +dsv4-fp4-b300-vllm: + image: vllm/vllm-openai:v0.25.0 + model: deepseek-ai/DeepSeek-V4-Pro + model-prefix: dsv4 + runner: cluster:b300-nv + precision: fp4 + framework: vllm + multinode: false + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - { tp: 4, conc-start: 1, conc-end: 64 } + - { tp: 8, conc-start: 1, conc-end: 4 } + - { tp: 4, ep: 4, dp-attn: true, conc-start: 256, conc-end: 512 } + - { tp: 8, ep: 8, dp-attn: true, conc-start: 2048, conc-end: 2048 } + +dsv4-fp4-b300-trt: + image: ghcr.io#semianalysisai/trtllm-deepseek-v4:feat-deepseek_v4-c185066 + model: deepseek-ai/DeepSeek-V4-Pro + model-prefix: dsv4 + runner: b300 + precision: fp4 + framework: trt + multinode: false + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - { tp: 4, conc-start: 1, conc-end: 32 } + - { tp: 4, ep: 4, dp-attn: true, conc-start: 64, conc-end: 64 } + - { tp: 8, ep: 8, dp-attn: true, conc-start: 256, conc-end: 1024 } + +dsv4-fp4-b300-trt-mtp: + image: ghcr.io#semianalysisai/trtllm-deepseek-v4:feat-deepseek_v4-c185066 + model: deepseek-ai/DeepSeek-V4-Pro + model-prefix: dsv4 + runner: b300 + precision: fp4 + framework: trt + multinode: false + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - { tp: 4, conc-start: 1, conc-end: 32, spec-decoding: mtp } + - { tp: 4, ep: 4, dp-attn: true, conc-start: 64, conc-end: 64, spec-decoding: mtp } + - { tp: 8, ep: 8, dp-attn: true, conc-start: 256, conc-end: 1024, spec-decoding: mtp } + +dsv4-fp4-b300-vllm-mtp: + image: vllm/vllm-openai:v0.25.0 + model: deepseek-ai/DeepSeek-V4-Pro + model-prefix: dsv4 + runner: cluster:b300-nv + precision: fp4 + framework: vllm + multinode: false + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - { tp: 4, conc-start: 1, conc-end: 64, spec-decoding: mtp } + - { tp: 8, conc-start: 1, conc-end: 8, spec-decoding: mtp } + - { tp: 4, ep: 4, conc-start: 64, conc-end: 256, spec-decoding: mtp } + - { tp: 4, ep: 4, dp-attn: true, conc-start: 256, conc-end: 512, spec-decoding: mtp } + +dsv4-fp4-b200-dynamo-vllm: + image: vllm/vllm-openai:v0.23.0 + model: deepseek-ai/DeepSeek-V4-Pro + model-prefix: dsv4 + runner: cluster:b200-nscale + precision: fp4 + framework: dynamo-vllm + router: { name: dynamo-router, version: "1.2.0.dev20260426" } + kv-p2p-transfer: nixl + multinode: true + disagg: true + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + # B200 adaptation of the DSV4 GB200 vLLM disagg recipes. Each worker + # maps to one full 8-GPU B200 node. + - conc-list: [1] + prefill: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/deepseek-v4/8k1k/disagg-b200-low-latency-c1.yaml" + decode: + num-worker: 1 + tp: 8 + ep: 1 + dp-attn: false + - conc-list: [32, 128] + prefill: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/deepseek-v4/8k1k/disagg-b200-low-latency-c32-c128.yaml" + decode: + num-worker: 1 + tp: 8 + ep: 1 + dp-attn: false + - conc-list: [64] + prefill: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/deepseek-v4/8k1k/disagg-b200-low-latency-c64.yaml" + decode: + num-worker: 1 + tp: 8 + ep: 1 + dp-attn: false + - conc-list: [256] + prefill: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/deepseek-v4/8k1k/disagg-b200-low-middle-c256.yaml" + decode: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + - conc-list: [512] + prefill: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/deepseek-v4/8k1k/disagg-b200-low-middle-c512.yaml" + decode: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + +dsv4-fp4-gb200-dynamo-vllm: + image: vllm/vllm-openai:v0.25.1 + model: deepseek-ai/DeepSeek-V4-Pro + model-prefix: dsv4 + runner: gb200 + precision: fp4 + framework: dynamo-vllm + router: { name: dynamo-router, version: "1.2.0.dev20260426" } + kv-p2p-transfer: nixl + multinode: true + disagg: true + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + # Validated 8k/1k points mirrored from NVIDIA/srt-slurm + # aflowers/vllm-gb200-v0.20.0 history. conc-list values match each + # recipe's benchmark.concurrencies. + + # Low latency: 1 prefill (DEP=8) + 1 decode (TP=8). 5 nodes total with + # a dedicated NATS/etcd infra node. + - conc-list: [1] + prefill: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/deepseek-v4/8k1k/disagg-gb200-low-latency.yaml" + decode: + num-worker: 1 + tp: 8 + ep: 1 + dp-attn: false + + # Low-middle curve: 1 prefill (DEP=8) + 4 decode (TP=8). 11 nodes total + # with a dedicated NATS/etcd infra node. + - conc-list: [256, 512] + prefill: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/deepseek-v4/8k1k/disagg-gb200-low-middle-curve.yaml" + decode: + num-worker: 4 + tp: 8 + ep: 1 + dp-attn: false + + # MegaMOE mid curve: 1 prefill (DEP=8) + 1 decode (DEP=8). 5 nodes + # total with a dedicated NATS/etcd infra node. + - conc-list: [256, 512, 1024] + prefill: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/deepseek-v4/8k1k/disagg-gb200-mid-curve-megamoe.yaml" + decode: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + + # MegaMOE high throughput: 2 prefill (DEP=8 each) + 1 decode (DEP=8). + # 7 nodes total with a dedicated NATS/etcd infra node. + - conc-list: [4096] + prefill: + num-worker: 2 + tp: 8 + ep: 8 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/deepseek-v4/8k1k/disagg-gb200-high-tpt-megamoe.yaml" + decode: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + + # MegaMOE max throughput: 3 prefill (DEP=8 each) + 1 decode (DEP=8). + # 9 nodes total with a dedicated NATS/etcd infra node. + - conc-list: [4096] + prefill: + num-worker: 3 + tp: 8 + ep: 8 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/deepseek-v4/8k1k/disagg-gb200-max-tpt-megamoe.yaml" + decode: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + +# TODO: change image to official llmd image. +# Build source: benchmarks/llm-d/Dockerfile. +dsv4-fp4-gb200-llmd-vllm: + image: quay.io/rh-ee-imarkov/llm-d-nokube-vllm:vllm0.26@sha256:a9095d4c835935c4070be2040de0a5ef3b44098f603092ab66d743b0e731b7b4 + model: deepseek-ai/DeepSeek-V4-Pro + model-prefix: dsv4 + runner: gb200 + precision: fp4 + framework: llmd-vllm + router: { name: llm-d-router, version: "0.9.0" } + kv-p2p-transfer: nixl + multinode: true + disagg: true + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + # Low latency: 1 prefill DEP8 + 1 decode TP8. + - spec-decoding: "none" + conc-list: [1] + prefill: + num-worker: 1 + tp: 1 + ep: 8 + dp-attn: true + additional-settings: + - "PREFILL_NODES=2" + - "GPUS_PER_NODE=4" + - "CONFIG_FILE=dsv4-fp4-gb200-low-latency.yaml" + decode: + num-worker: 1 + tp: 8 + ep: 1 + dp-attn: false + additional-settings: + - "DECODE_NODES=2" + - "GPUS_PER_NODE=4" + + # Mid curve: 1 prefill DEP8 + 1 decode DEP8. + - spec-decoding: "none" + conc-list: [256, 512, 1024] + prefill: + num-worker: 1 + tp: 1 + ep: 8 + dp-attn: true + additional-settings: + - "PREFILL_NODES=2" + - "GPUS_PER_NODE=4" + - "CONFIG_FILE=dsv4-fp4-gb200-mid-curve-megamoe.yaml" + decode: + num-worker: 1 + tp: 1 + ep: 8 + dp-attn: true + additional-settings: + - "DECODE_NODES=2" + - "GPUS_PER_NODE=4" + + # Max throughput: 3 prefill DEP8 + 1 decode DEP8. + - spec-decoding: "none" + conc-list: [4096] + prefill: + num-worker: 3 + tp: 1 + ep: 8 + dp-attn: true + additional-settings: + - "PREFILL_NODES=6" + - "PREFILL_WORKERS=3" + - "GPUS_PER_NODE=4" + - "CONFIG_FILE=dsv4-fp4-gb200-mid-curve-megamoe.yaml" + decode: + num-worker: 1 + tp: 1 + ep: 8 + dp-attn: true + additional-settings: + - "DECODE_NODES=2" + - "GPUS_PER_NODE=4" + +# MTP2 variant of dsv4-fp4-gb200-dynamo-vllm. Uses the vLLM 0.20.1 image +# and hand-picked 8k/1k Pareto points mirrored from NVIDIA/srt-slurm. +dsv4-fp4-gb200-dynamo-vllm-mtp2: + image: vllm/vllm-openai:v0.20.1-ubuntu2404 + model: deepseek-ai/DeepSeek-V4-Pro + model-prefix: dsv4 + runner: gb200 + precision: fp4 + framework: dynamo-vllm + router: { name: dynamo-router, version: "1.2.0.dev20260426" } + kv-p2p-transfer: nixl + multinode: true + disagg: true + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + # Aggregate low latency: TP=8, max-num-seqs=4. + - conc-list: [1] + spec-decoding: mtp + prefill: + num-worker: 1 + tp: 8 + ep: 1 + dp-attn: false + additional-settings: + - "SYNTHETIC_ACCEPTANCE=true" + - "SYNTHETIC_ACCEPTANCE_LENGTH=2.27" + - "CONFIG_FILE=recipes/vllm/deepseek-v4/8k1k/agg-gb200-low-latency-mtp2.yaml" + decode: + num-worker: 0 + tp: 8 + ep: 1 + dp-attn: false + + # Low-latency bridge: 1 prefill (DEP=8) + 4 decode (TP=8), no offload. + - conc-list: [16, 32, 64] + spec-decoding: mtp + prefill: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + additional-settings: + - "SYNTHETIC_ACCEPTANCE=true" + - "SYNTHETIC_ACCEPTANCE_LENGTH=2.27" + - "CONFIG_FILE=recipes/vllm/deepseek-v4/8k1k/disagg-gb200-low-latency-mtp2.yaml" + decode: + num-worker: 4 + tp: 8 + ep: 1 + dp-attn: false + + # MegaMOE mid curve: 1 prefill (DEP=8) + 1 decode (DEP=8). + # 5 nodes total with a dedicated NATS/etcd infra node. + - conc-list: [128, 256, 512, 1024] + spec-decoding: mtp + prefill: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + additional-settings: + - "SYNTHETIC_ACCEPTANCE=true" + - "SYNTHETIC_ACCEPTANCE_LENGTH=2.27" + - "CONFIG_FILE=recipes/vllm/deepseek-v4/8k1k/disagg-gb200-mid-curve-megamoe-mtp2.yaml" + decode: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + + # MegaMOE high throughput: 2 prefill (DEP=8 each) + 1 decode (DEP=8). + # 7 nodes total with a dedicated NATS/etcd infra node. + - conc-list: [1024] + spec-decoding: mtp + prefill: + num-worker: 2 + tp: 8 + ep: 8 + dp-attn: true + additional-settings: + - "SYNTHETIC_ACCEPTANCE=true" + - "SYNTHETIC_ACCEPTANCE_LENGTH=2.27" + - "CONFIG_FILE=recipes/vllm/deepseek-v4/8k1k/disagg-gb200-high-tpt-megamoe-mtp2.yaml" + decode: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + +# Baseline (no synthetic acceptance) variant of dsv4-fp4-gb200-dynamo-vllm-mtp2 +# for before/after Pareto comparison. Identical topology and recipes, only the +# SYNTHETIC_ACCEPTANCE envs are removed. +dsv4-fp4-gb200-dynamo-vllm-mtp2-nosynthetic: + image: vllm/vllm-openai:v0.20.1-ubuntu2404 + model: deepseek-ai/DeepSeek-V4-Pro + model-prefix: dsv4 + runner: gb200 + precision: fp4 + framework: dynamo-vllm + router: { name: dynamo-router, version: "1.2.0.dev20260426" } + multinode: true + disagg: true + kv-p2p-transfer: nixl + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + # Aggregate low latency: TP=8, max-num-seqs=4. + - conc-list: [1] + spec-decoding: mtp + prefill: + num-worker: 1 + tp: 8 + ep: 1 + dp-attn: false + additional-settings: + - "CONFIG_FILE=recipes/vllm/deepseek-v4/8k1k/agg-gb200-low-latency-mtp2.yaml" + decode: + num-worker: 0 + tp: 8 + ep: 1 + dp-attn: false + + # Low-latency bridge: 1 prefill (DEP=8) + 4 decode (TP=8), no offload. + - conc-list: [16, 32, 64] + spec-decoding: mtp + prefill: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/deepseek-v4/8k1k/disagg-gb200-low-latency-mtp2.yaml" + decode: + num-worker: 4 + tp: 8 + ep: 1 + dp-attn: false + + # MegaMOE mid curve: 1 prefill (DEP=8) + 1 decode (DEP=8). + # 5 nodes total with a dedicated NATS/etcd infra node. + - conc-list: [128, 256, 512, 1024] + spec-decoding: mtp + prefill: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/deepseek-v4/8k1k/disagg-gb200-mid-curve-megamoe-mtp2.yaml" + decode: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + + # MegaMOE high throughput: 2 prefill (DEP=8 each) + 1 decode (DEP=8). + # 7 nodes total with a dedicated NATS/etcd infra node. + - conc-list: [1024] + spec-decoding: mtp + prefill: + num-worker: 2 + tp: 8 + ep: 8 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/deepseek-v4/8k1k/disagg-gb200-high-tpt-megamoe-mtp2.yaml" + decode: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + +# DSV4 B200 disaggregated Dynamo SGLang STP configuration. +dsv4-fp4-b200-dynamo-sglang: + image: lmsysorg/sglang:nightly-dev-cu13-20260821-f825d729 + model: deepseek-ai/DeepSeek-V4-Pro + model-prefix: dsv4 + runner: cluster:b200-nscale + precision: fp4 + framework: dynamo-sglang + router: { name: dynamo-router, version: "86f84b9486905c2bfaa4b756c9fa39f04ecf5ebd" } + kv-p2p-transfer: nixl + multinode: true + disagg: true + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + # 1p1d-tp8-tp8 (pure TP). 2 nodes (1+1). + - conc-list: [1] + prefill: + num-worker: 1 + tp: 8 + ep: 1 + dp-attn: false + additional-settings: + - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-b200-1p1d-tp8-tp8.yaml" + decode: + num-worker: 1 + tp: 8 + ep: 1 + dp-attn: false + # 1p1d-dep8-tp8: DEP8 prefill + TP8 decode. 2 nodes (1+1). + - conc-list: [32, 64] + prefill: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-b200-1p1d-dep8-tp8.yaml" + decode: + num-worker: 1 + tp: 8 + ep: 1 + dp-attn: false + # 1p1d-dep8-dep8: DEP8 prefill + DEP8 decode. 2 nodes (1+1). + - conc-list: [256, 512] + prefill: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-b200-1p1d-dep8-dep8.yaml" + decode: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + # 1p4d-dep8-tp8: DEP8 prefill + 4x TP8 decode. 5 nodes (1+4). + - conc-list: [64] + prefill: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-b200-1p4d-dep8-tp8.yaml" + decode: + num-worker: 4 + tp: 8 + ep: 1 + dp-attn: false + # 1p2d-dep8-dep8: DEP8 prefill + 2x DEP8 decode. 3 nodes (1+2). + - conc-list: [256] + prefill: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-b200-1p2d-dep8-dep8.yaml" + decode: + num-worker: 2 + tp: 8 + ep: 8 + dp-attn: true + # 1p4d-dep8-dep8: DEP8 prefill + 4x DEP8 decode. 5 nodes (1+4). + - conc-list: [256] + prefill: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-b200-1p4d-dep8-dep8.yaml" + decode: + num-worker: 4 + tp: 8 + ep: 8 + dp-attn: true + # 3p2d-dep8-dep8: 3x DEP8 prefill + 2x DEP8 decode. 5 nodes (3+2). + - conc-list: [2048] + prefill: + num-worker: 3 + tp: 8 + ep: 8 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-b200-3p2d-dep8-dep8.yaml" + decode: + num-worker: 2 + tp: 8 + ep: 8 + dp-attn: true + # 4p2d-dep8-dep8: 4x DEP8 prefill + 2x DEP8 decode. 6 nodes (4+2). + - conc-list: [4096] + prefill: + num-worker: 4 + tp: 8 + ep: 8 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-b200-4p2d-dep8-dep8.yaml" + decode: + num-worker: 2 + tp: 8 + ep: 8 + dp-attn: true + # 5p2d-dep8-dep8: 5x DEP8 prefill + 2x DEP8 decode. 7 nodes (5+2). + - conc-list: [6144] + prefill: + num-worker: 5 + tp: 8 + ep: 8 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-b200-5p2d-dep8-dep8.yaml" + decode: + num-worker: 2 + tp: 8 + ep: 8 + dp-attn: true + # 6p2d-dep8-dep8: 6x DEP8 prefill + 2x DEP8 decode. 8 nodes (6+2). + - conc-list: [8192] + prefill: + num-worker: 6 + tp: 8 + ep: 8 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-b200-6p2d-dep8-dep8.yaml" + decode: + num-worker: 2 + tp: 8 + ep: 8 + dp-attn: true + +dsv4-fp4-gb200-dynamo-sglang: + image: lmsysorg/sglang:nightly-dev-cu13-20260528-0abe6a85 + model: deepseek-ai/DeepSeek-V4-Pro + model-prefix: dsv4 + runner: gb200 + precision: fp4 + framework: dynamo-sglang + router: { name: dynamo-router, version: "92f5b3b8d7dd5ab9179d4b1034bd2c1c0803693e" } + kv-p2p-transfer: mooncake + multinode: true + disagg: true + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + # Low latency: 1p1d-tp8-tp8. 4 nodes. + - conc-list: [1] + prefill: + num-worker: 1 + tp: 8 + ep: 1 + dp-attn: false + additional-settings: + - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-gb200-1p1d-tp8-tp8-4-c1.yaml" + decode: + num-worker: 1 + tp: 8 + ep: 1 + dp-attn: false + # 1p4d-dep8-tp8. 10 nodes. + - conc-list: [64] + prefill: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-gb200-1p4d-dep8-tp8-10-c64.yaml" + decode: + num-worker: 4 + tp: 8 + ep: 1 + dp-attn: false + # WideEP TP=16 decode: 1p2d-dep8-dep16. 10 nodes. + - conc-list: [256] + prefill: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-gb200-1p2d-dep8-dep16-10-c256.yaml" + decode: + num-worker: 2 + tp: 16 + ep: 16 + dp-attn: true + # WideEP TP=16 decode: 1p1d-dep8-dep16. 6 nodes. + - conc-list: [512] + prefill: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-gb200-1p1d-dep8-dep16-6-c512.yaml" + decode: + num-worker: 1 + tp: 16 + ep: 16 + dp-attn: true + # WideEP TP=16 decode: 2p1d-dep8-dep16. 8 nodes. + - conc-list: [1536] + prefill: + num-worker: 2 + tp: 8 + ep: 8 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-gb200-2p1d-dep8-dep16-8-c1536.yaml" + decode: + num-worker: 1 + tp: 16 + ep: 16 + dp-attn: true + # WideEP TP=16 decode: 4p1d-dep8-dep16. 12 nodes. + - conc-list: [4096] + prefill: + num-worker: 4 + tp: 8 + ep: 8 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-gb200-4p1d-dep8-dep16-12-c4096.yaml" + decode: + num-worker: 1 + tp: 16 + ep: 16 + dp-attn: true + # WideEP TP=16 decode: 5p1d-dep8-dep16. 14 nodes. + - conc-list: [8192] + prefill: + num-worker: 5 + tp: 8 + ep: 8 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-gb200-5p1d-dep8-dep16-14-c8192.yaml" + decode: + num-worker: 1 + tp: 16 + ep: 16 + dp-attn: true + # WideEP TP=12 decode: 6p1d-dep8-dep12. 15 nodes. + - conc-list: [8192] + prefill: + num-worker: 6 + tp: 8 + ep: 8 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-gb200-6p1d-dep8-dep12-15-c8192.yaml" + decode: + num-worker: 1 + tp: 12 + ep: 12 + dp-attn: true + +# MTP variant of dsv4-fp4-gb200-dynamo-sglang. +dsv4-fp4-gb200-dynamo-sglang-mtp: + image: lmsysorg/sglang:nightly-dev-cu13-20260528-0abe6a85 + model: deepseek-ai/DeepSeek-V4-Pro + model-prefix: dsv4 + runner: gb200 + precision: fp4 + framework: dynamo-sglang + router: { name: dynamo-router, version: "92f5b3b8d7dd5ab9179d4b1034bd2c1c0803693e" } + kv-p2p-transfer: mooncake + multinode: true + disagg: true + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + # Low-latency baseline: 1p1d-tp8-tp8. 4 nodes. + - spec-decoding: "mtp" + conc-list: [1] + prefill: + num-worker: 1 + tp: 8 + ep: 1 + dp-attn: false + additional-settings: + - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-gb200-low-latency-1p1d-tp8-tp8-mtp.yaml" + decode: + num-worker: 1 + tp: 8 + ep: 1 + dp-attn: false + # Low-latency 1p6d-dep8-tp8: 1P (DEP=8) + 6 TP=8 decode workers. 14 nodes. + # Recipe runs concurrencies=32x64x128; matrix tracks the max. + - spec-decoding: "mtp" + conc-list: [128] + prefill: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-gb200-low-latency-1p6d-dep8-tp8-mtp.yaml" + decode: + num-worker: 6 + tp: 8 + ep: 1 + dp-attn: false + # Mid curve 1p1d-dep8-dep16. 6 nodes. + - spec-decoding: "mtp" + conc-list: [1024] + prefill: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-gb200-mid-curve-1p1d-dep8-dep16-mtp.yaml" + decode: + num-worker: 1 + tp: 16 + ep: 16 + dp-attn: true + # Mid curve 2p1d-dep8-dep16. 8 nodes. + - spec-decoding: "mtp" + conc-list: [2048] + prefill: + num-worker: 2 + tp: 8 + ep: 8 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-gb200-mid-curve-2p1d-dep8-dep16-mtp.yaml" + decode: + num-worker: 1 + tp: 16 + ep: 16 + dp-attn: true + # Mid curve 3p1d-dep8-dep16. 10 nodes. + - spec-decoding: "mtp" + conc-list: [3072] + prefill: + num-worker: 3 + tp: 8 + ep: 8 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-gb200-mid-curve-3p1d-dep8-dep16-mtp.yaml" + decode: + num-worker: 1 + tp: 16 + ep: 16 + dp-attn: true + # Mid curve 4p1d-dep8-dep16. 12 nodes. + - spec-decoding: "mtp" + conc-list: [6144] + prefill: + num-worker: 4 + tp: 8 + ep: 8 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-gb200-mid-curve-4p1d-dep8-dep16-mtp.yaml" + decode: + num-worker: 1 + tp: 16 + ep: 16 + dp-attn: true + # Mid curve 5p1d-dep8-dep16. 14 nodes. + - spec-decoding: "mtp" + conc-list: [8192] + prefill: + num-worker: 5 + tp: 8 + ep: 8 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-gb200-mid-curve-5p1d-dep8-dep16-mtp.yaml" + decode: + num-worker: 1 + tp: 16 + ep: 16 + dp-attn: true + # Mid curve 6p1d-dep8-dep16. 16 nodes. + - spec-decoding: "mtp" + conc-list: [16384] + prefill: + num-worker: 6 + tp: 8 + ep: 8 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-gb200-mid-curve-6p1d-dep8-dep16-mtp.yaml" + decode: + num-worker: 1 + tp: 16 + ep: 16 + dp-attn: true + + # ---------- 1k1k high-throughput (wide-EP decode, MTP) ---------- + +dsv4-fp4-b300-dynamo-vllm: + image: vllm/vllm-openai:v0.23.0 + model: deepseek-ai/DeepSeek-V4-Pro + model-prefix: dsv4 + runner: b300 + precision: fp4 + framework: dynamo-vllm + router: { name: dynamo-router, version: "1.2.0.dev20260426" } + kv-p2p-transfer: nixl + multinode: true + disagg: true + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + # B300 adaptation of the DSV4 B200/GB200 vLLM disagg recipes. Each + # prefill/decode worker maps to one full 8-GPU B300 node. + - conc-list: [1, 32, 64, 128] + prefill: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/deepseek-v4/8k1k/disagg-b300-low-latency.yaml" + decode: + num-worker: 1 + tp: 8 + ep: 1 + dp-attn: false + - conc-list: [256, 1024] + prefill: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/deepseek-v4/8k1k/disagg-b300-mid-curve-megamoe.yaml" + decode: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + - conc-list: [4096] + prefill: + num-worker: 2 + tp: 8 + ep: 8 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/deepseek-v4/8k1k/disagg-b300-high-tpt-megamoe.yaml" + decode: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + +# DSV4 B300 disaggregated dynamo-sglang STP configuration. +# Each search-space entry maps one-to-one to a checked-in srt-slurm recipe under +# recipes/sglang/deepseek-v4/8k1k/. Runs on the b300 pool via +# runners/launch_b300-nv.sh. Prefill workers are TP4/DEP4 (two workers per +# 8-GPU B300 node); the decode worker is a single DEP8 node. +dsv4-fp4-b300-dynamo-sglang: + image: lmsysorg/sglang:nightly-dev-cu13-20260721-8905cbd4 + model: deepseek-ai/DeepSeek-V4-Pro + model-prefix: dsv4 + runner: b300 + precision: fp4 + framework: dynamo-sglang + kv-p2p-transfer: nixl + multinode: true + disagg: true + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + # 1p1d-tp4-tp4 (pure TP, no dp-attn). 2 nodes (1P + 1D). + - conc-list: [1, 8, 16, 32] + prefill: + num-worker: 1 + tp: 4 + ep: 1 + dp-attn: false + additional-settings: + - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-b300-1p1d-tp4-tp4.yaml" + decode: + num-worker: 1 + tp: 4 + ep: 1 + dp-attn: false + # 1p1d-dep4-dep8: 1x DEP4 prefill + DEP8 decode. 2 nodes (1P + 1D). + - conc-list: [256] + prefill: + num-worker: 1 + tp: 4 + ep: 4 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-b300-1p1d-dep4-dep8.yaml" + decode: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + # 2p1d-dep4-dep8: 2x DEP4 prefill + DEP8 decode. 2 nodes (1P + 1D). + - conc-list: [768] + prefill: + num-worker: 2 + tp: 4 + ep: 4 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-b300-2p1d-dep4-dep8.yaml" + decode: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + # 4p1d-dep4-dep8: 4x DEP4 prefill + DEP8 decode. 3 nodes (2P + 1D). + - conc-list: [2048] + prefill: + num-worker: 4 + tp: 4 + ep: 4 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-b300-4p1d-dep4-dep8.yaml" + decode: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + # 6p1d-dep4-dep8: 6x DEP4 prefill + DEP8 decode. 4 nodes (3P + 1D). + - conc-list: [3072] + prefill: + num-worker: 6 + tp: 4 + ep: 4 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-b300-6p1d-dep4-dep8.yaml" + decode: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + +dsv4-fp4-gb300-dynamo-vllm: + image: vllm/vllm-openai:dsv4-megamoe-mxfp4-arm64-cu130-4ba0a72 + model: deepseek-ai/DeepSeek-V4-Pro + model-prefix: dsv4 + runner: gb300-nv + precision: fp4 + framework: dynamo-vllm + router: { name: dynamo-router, version: "1.2.0.dev20260426" } + kv-p2p-transfer: nixl + multinode: true + disagg: true + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - conc-list: [192] + prefill: + num-worker: 1 + tp: 4 + ep: 4 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/deepseek-v4/8k1k/disagg-gb300-1p6d-dep4-tp4.yaml" + decode: + num-worker: 6 + tp: 4 + ep: 1 + dp-attn: false + - conc-list: [18] + prefill: + num-worker: 1 + tp: 4 + ep: 4 + dp-attn: false + additional-settings: + - "CONFIG_FILE=recipes/vllm/deepseek-v4/8k1k/disagg-gb300-1p9d-tep4-tp4.yaml" + decode: + num-worker: 9 + tp: 4 + ep: 1 + dp-attn: false + - conc-list: [4096] + prefill: + num-worker: 4 + tp: 4 + ep: 4 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/deepseek-v4/8k1k/disagg-gb300-4p1d-dep4-dep8-24-c4096.yaml" + decode: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + - conc-list: [4096] + prefill: + num-worker: 5 + tp: 4 + ep: 4 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/deepseek-v4/8k1k/disagg-gb300-5p1d-dep4-dep8-28-c4096.yaml" + decode: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + - conc-list: [4096] + prefill: + num-worker: 6 + tp: 4 + ep: 4 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/deepseek-v4/8k1k/disagg-gb300-6p1d-dep4-dep8-32-c4096.yaml" + decode: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + - conc-list: [3072] + prefill: + num-worker: 7 + tp: 4 + ep: 4 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/deepseek-v4/8k1k/disagg-gb300-7p2d-dep4-dep16.yaml" + decode: + num-worker: 2 + tp: 16 + ep: 16 + dp-attn: true + +dsv4-fp4-b200-dynamo-sglang-mtp: + image: lmsysorg/sglang:nightly-dev-cu13-20260710-cfc66e05 + model: deepseek-ai/DeepSeek-V4-Pro + model-prefix: dsv4 + runner: cluster:b200-nscale + precision: fp4 + framework: dynamo-sglang + router: { name: dynamo-router, version: "92f5b3b8d7dd5ab9179d4b1034bd2c1c0803693e" } + kv-p2p-transfer: nixl + multinode: true + disagg: true + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - spec-decoding: "mtp" + conc-list: [1] + prefill: + num-worker: 1 + tp: 8 + ep: 1 + dp-attn: false + additional-settings: + - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-b200-low-latency-1p1d-tp8-tp8-mtp.yaml" + decode: + num-worker: 1 + tp: 8 + ep: 1 + dp-attn: false + - spec-decoding: "mtp" + conc-list: [32, 64, 128] + prefill: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-b200-low-latency-1p6d-dep8-tp8-mtp.yaml" + decode: + num-worker: 6 + tp: 8 + ep: 1 + dp-attn: false + - spec-decoding: "mtp" + conc-list: [256, 1024] + prefill: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-b200-mid-curve-1p1d-dep8-dep8-mtp.yaml" + decode: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + - spec-decoding: "mtp" + conc-list: [256] + prefill: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-b200-mid-curve-1p2d-dep8-dep8-mtp.yaml" + decode: + num-worker: 2 + tp: 8 + ep: 8 + dp-attn: true + - spec-decoding: "mtp" + conc-list: [6144] + prefill: + num-worker: 5 + tp: 8 + ep: 8 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-b200-mid-curve-5p3d-dep8-dep8-mtp.yaml" + decode: + num-worker: 3 + tp: 8 + ep: 8 + dp-attn: true + +dsv4-fp4-b200-dynamo-vllm-mtp: + image: vllm/vllm-openai:vllm-x86_64-cu13-0.25.1-7a33ba9 + model: deepseek-ai/DeepSeek-V4-Pro + model-prefix: dsv4 + runner: cluster:b200-nscale + precision: fp4 + framework: dynamo-vllm + multinode: true + disagg: true + kv-p2p-transfer: nixl + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + # B200 MTP disagg recipes. Each worker maps to one full 8-GPU B200 node + # (prefill dep8; decode tp8, except the low-latency tp8 variant). + # SYNTHETIC_ACCEPTANCE is a throughput-only control (injected into the + # recipe by runners/inject_synthetic_acceptance.py); EVAL_ONLY runs keep + # the checked-in real-MTP recipe so verification is measured normally. + - spec-decoding: "mtp" + conc-list: [1, 8, 16] + prefill: + num-worker: 1 + tp: 1 + ep: 8 + dp-attn: true + additional-settings: + - "SYNTHETIC_ACCEPTANCE=true" + - "SYNTHETIC_ACCEPTANCE_LENGTH=2.49" + - "CONFIG_FILE=recipes/vllm/deepseek-v4/8k1k/disagg-b200-1p1d-dep8-tp8-mtp.yaml" + decode: + num-worker: 1 + tp: 8 + ep: 1 + dp-attn: false + - spec-decoding: "mtp" + conc-list: [128, 256] + prefill: + num-worker: 1 + tp: 1 + ep: 8 + dp-attn: true + additional-settings: + - "SYNTHETIC_ACCEPTANCE=true" + - "SYNTHETIC_ACCEPTANCE_LENGTH=2.49" + - "CONFIG_FILE=recipes/vllm/deepseek-v4/8k1k/disagg-b200-1p2d-dep8-dep8-mtp.yaml" + decode: + num-worker: 2 + tp: 1 + ep: 8 + dp-attn: true + - spec-decoding: "mtp" + conc-list: [512] + prefill: + num-worker: 1 + tp: 1 + ep: 8 + dp-attn: true + additional-settings: + - "SYNTHETIC_ACCEPTANCE=true" + - "SYNTHETIC_ACCEPTANCE_LENGTH=2.49" + - "CONFIG_FILE=recipes/vllm/deepseek-v4/8k1k/disagg-b200-1p1d-dep8-dep8-mtp.yaml" + decode: + num-worker: 1 + tp: 1 + ep: 8 + dp-attn: true + - spec-decoding: "mtp" + conc-list: [1024] + prefill: + num-worker: 2 + tp: 1 + ep: 8 + dp-attn: true + additional-settings: + - "SYNTHETIC_ACCEPTANCE=true" + - "SYNTHETIC_ACCEPTANCE_LENGTH=2.49" + - "CONFIG_FILE=recipes/vllm/deepseek-v4/8k1k/disagg-b200-2p1d-dep8-dep8-mtp.yaml" + decode: + num-worker: 1 + tp: 1 + ep: 8 + dp-attn: true + - spec-decoding: "mtp" + conc-list: [4096] + prefill: + num-worker: 3 + tp: 1 + ep: 8 + dp-attn: true + additional-settings: + - "SYNTHETIC_ACCEPTANCE=true" + - "SYNTHETIC_ACCEPTANCE_LENGTH=1.79" + - "CONFIG_FILE=recipes/vllm/deepseek-v4/8k1k/disagg-b200-3p1d-dep8-dep8-mtp.yaml" + decode: + num-worker: 1 + tp: 1 + ep: 8 + dp-attn: true + +dsv4-fp4-gb300-dynamo-trt: + image: nvcr.io#nvidia/ai-dynamo/tensorrtllm-runtime:1.3.0-deepseek-v4-dev.1 + model: deepseek-ai/DeepSeek-V4-Pro + model-prefix: dsv4 + runner: gb300-nv + precision: fp4 + framework: dynamo-trt + router: { name: dynamo-router, version: "v1.3.0-deepseek-v4-dev.1" } + kv-p2p-transfer: nixl + multinode: true + disagg: true + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - conc-list: [4] + prefill: + num-worker: 1 + tp: 4 + ep: 4 + dp-attn: true + additional-settings: + # https://github.com/NVIDIA/srt-slurm/blob/sa-submission-q2-2026/recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/STP/ctx1dep4_gen4tep8_batch1_eplb0_mtp0.yaml + - "CONFIG_FILE=recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/STP/ctx1dep4_gen4tep8_batch1_eplb0_mtp0.yaml" + decode: + num-worker: 4 + tp: 8 + ep: 8 + dp-attn: false + - conc-list: [5] + prefill: + num-worker: 1 + tp: 4 + ep: 4 + dp-attn: true + additional-settings: + # https://github.com/NVIDIA/srt-slurm/blob/sa-submission-q2-2026/recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/STP/ctx1dep4_gen5tep4_batch1_eplb0_mtp0.yaml + - "CONFIG_FILE=recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/STP/ctx1dep4_gen5tep4_batch1_eplb0_mtp0.yaml" + decode: + num-worker: 5 + tp: 4 + ep: 4 + dp-attn: false + - conc-list: [15] + prefill: + num-worker: 1 + tp: 4 + ep: 4 + dp-attn: true + additional-settings: + # https://github.com/NVIDIA/srt-slurm/blob/sa-submission-q2-2026/recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/STP/ctx1dep4_gen5tep4_batch2_eplb0_mtp0.yaml + - "CONFIG_FILE=recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/STP/ctx1dep4_gen5tep4_batch2_eplb0_mtp0.yaml" + decode: + num-worker: 5 + tp: 4 + ep: 4 + dp-attn: false + - conc-list: [25] + prefill: + num-worker: 1 + tp: 4 + ep: 4 + dp-attn: true + additional-settings: + # https://github.com/NVIDIA/srt-slurm/blob/sa-submission-q2-2026/recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/STP/ctx1dep4_gen5tep4_batch4_eplb0_mtp0.yaml + - "CONFIG_FILE=recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/STP/ctx1dep4_gen5tep4_batch4_eplb0_mtp0.yaml" + decode: + num-worker: 5 + tp: 4 + ep: 4 + dp-attn: false + - conc-list: [55] + prefill: + num-worker: 1 + tp: 4 + ep: 4 + dp-attn: true + additional-settings: + # https://github.com/NVIDIA/srt-slurm/blob/sa-submission-q2-2026/recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/STP/ctx1dep4_gen5tep4_batch8_eplb0_mtp0.yaml + - "CONFIG_FILE=recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/STP/ctx1dep4_gen5tep4_batch8_eplb0_mtp0.yaml" + decode: + num-worker: 5 + tp: 4 + ep: 4 + dp-attn: false + - conc-list: [154] + prefill: + num-worker: 1 + tp: 4 + ep: 4 + dp-attn: true + additional-settings: + # https://github.com/NVIDIA/srt-slurm/blob/sa-submission-q2-2026/recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/STP/ctx1dep4_gen1dep32_batch4_eplb384_mtp0.yaml + - "CONFIG_FILE=recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/STP/ctx1dep4_gen1dep32_batch4_eplb384_mtp0.yaml" + decode: + num-worker: 1 + tp: 32 + ep: 32 + dp-attn: true + - conc-list: [308] + prefill: + num-worker: 2 + tp: 4 + ep: 4 + dp-attn: true + additional-settings: + # https://github.com/NVIDIA/srt-slurm/blob/sa-submission-q2-2026/recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/STP/ctx2dep4_gen1dep32_batch8_eplb384_mtp0.yaml + - "CONFIG_FILE=recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/STP/ctx2dep4_gen1dep32_batch8_eplb384_mtp0.yaml" + decode: + num-worker: 1 + tp: 32 + ep: 32 + dp-attn: true + - conc-list: [615] + prefill: + num-worker: 4 + tp: 4 + ep: 4 + dp-attn: true + additional-settings: + # https://github.com/NVIDIA/srt-slurm/blob/sa-submission-q2-2026/recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/STP/ctx4dep4_gen1dep32_batch16_eplb384_mtp0.yaml + - "CONFIG_FILE=recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/STP/ctx4dep4_gen1dep32_batch16_eplb384_mtp0.yaml" + decode: + num-worker: 1 + tp: 32 + ep: 32 + dp-attn: true + - conc-list: [1127] + prefill: + num-worker: 6 + tp: 4 + ep: 4 + dp-attn: true + additional-settings: + # https://github.com/NVIDIA/srt-slurm/blob/sa-submission-q2-2026/recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/STP/ctx6dep4_gen1dep32_batch32_eplb384_mtp0.yaml + - "CONFIG_FILE=recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/STP/ctx6dep4_gen1dep32_batch32_eplb384_mtp0.yaml" + decode: + num-worker: 1 + tp: 32 + ep: 32 + dp-attn: true + - conc-list: [1229] + prefill: + num-worker: 5 + tp: 4 + ep: 4 + dp-attn: true + additional-settings: + # https://github.com/NVIDIA/srt-slurm/blob/sa-submission-q2-2026/recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/STP/ctx5dep4_gen1dep16_batch64_eplb384_mtp0.yaml + - "CONFIG_FILE=recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/STP/ctx5dep4_gen1dep16_batch64_eplb384_mtp0.yaml" + decode: + num-worker: 1 + tp: 16 + ep: 16 + dp-attn: true + - conc-list: [2253] + prefill: + num-worker: 6 + tp: 4 + ep: 4 + dp-attn: true + additional-settings: + # https://github.com/NVIDIA/srt-slurm/blob/sa-submission-q2-2026/recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/STP/ctx6dep4_gen1dep8_batch256_eplb384_mtp0.yaml + - "CONFIG_FILE=recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/STP/ctx6dep4_gen1dep8_batch256_eplb384_mtp0.yaml" + decode: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + - conc-list: [2253] + prefill: + num-worker: 9 + tp: 4 + ep: 4 + dp-attn: true + additional-settings: + # https://github.com/NVIDIA/srt-slurm/blob/sa-submission-q2-2026/recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/STP/ctx9dep4_gen1dep16_batch128_eplb384_mtp0.yaml + - "CONFIG_FILE=recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/STP/ctx9dep4_gen1dep16_batch128_eplb384_mtp0.yaml" + decode: + num-worker: 1 + tp: 16 + ep: 16 + dp-attn: true + - conc-list: [4301] + prefill: + num-worker: 10 + tp: 4 + ep: 4 + dp-attn: true + additional-settings: + # https://github.com/NVIDIA/srt-slurm/blob/sa-submission-q2-2026/recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/STP/ctx10dep4_gen1dep8_batch512_eplb384_mtp0.yaml + - "CONFIG_FILE=recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/STP/ctx10dep4_gen1dep8_batch512_eplb384_mtp0.yaml" + decode: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + +dsv4-fp4-gb300-dynamo-trt-mtp: + image: nvcr.io#nvidia/ai-dynamo/tensorrtllm-runtime:1.3.0-deepseek-v4-dev.1 + model: deepseek-ai/DeepSeek-V4-Pro + model-prefix: dsv4 + runner: gb300-nv + precision: fp4 + framework: dynamo-trt + router: { name: dynamo-router, version: "v1.3.0-deepseek-v4-dev.1" } + kv-p2p-transfer: nixl + multinode: true + disagg: true + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - spec-decoding: "mtp" + conc-list: [8] + prefill: + num-worker: 1 + tp: 4 + ep: 4 + dp-attn: true + additional-settings: + # https://github.com/NVIDIA/srt-slurm/blob/sa-submission-q2-2026/recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/MTP/ctx1dep4_gen4tep8_batch1_eplb0_mtp3.yaml + - "CONFIG_FILE=recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/MTP/ctx1dep4_gen4tep8_batch1_eplb0_mtp3.yaml" + decode: + num-worker: 4 + tp: 8 + ep: 8 + dp-attn: false + - spec-decoding: "mtp" + conc-list: [10] + prefill: + num-worker: 1 + tp: 4 + ep: 4 + dp-attn: true + additional-settings: + # https://github.com/NVIDIA/srt-slurm/blob/sa-submission-q2-2026/recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/MTP/ctx1dep4_gen5tep4_batch1_eplb0_mtp3.yaml + - "CONFIG_FILE=recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/MTP/ctx1dep4_gen5tep4_batch1_eplb0_mtp3.yaml" + decode: + num-worker: 5 + tp: 4 + ep: 4 + dp-attn: false + - spec-decoding: "mtp" + conc-list: [15] + prefill: + num-worker: 1 + tp: 4 + ep: 4 + dp-attn: true + additional-settings: + # https://github.com/NVIDIA/srt-slurm/blob/sa-submission-q2-2026/recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/MTP/ctx1dep4_gen5tep4_batch2_eplb0_mtp3.yaml + - "CONFIG_FILE=recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/MTP/ctx1dep4_gen5tep4_batch2_eplb0_mtp3.yaml" + decode: + num-worker: 5 + tp: 4 + ep: 4 + dp-attn: false + - spec-decoding: "mtp" + conc-list: [30] + prefill: + num-worker: 1 + tp: 4 + ep: 4 + dp-attn: true + additional-settings: + # https://github.com/NVIDIA/srt-slurm/blob/sa-submission-q2-2026/recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/MTP/ctx1dep4_gen5tep4_batch4_eplb0_mtp3.yaml + - "CONFIG_FILE=recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/MTP/ctx1dep4_gen5tep4_batch4_eplb0_mtp3.yaml" + decode: + num-worker: 5 + tp: 4 + ep: 4 + dp-attn: false + - spec-decoding: "mtp" + conc-list: [84] + prefill: + num-worker: 2 + tp: 4 + ep: 4 + dp-attn: true + additional-settings: + # https://github.com/NVIDIA/srt-slurm/blob/sa-submission-q2-2026/recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/MTP/ctx2dep4_gen1dep32_batch2_eplb384_mtp3.yaml + - "CONFIG_FILE=recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/MTP/ctx2dep4_gen1dep32_batch2_eplb384_mtp3.yaml" + decode: + num-worker: 1 + tp: 32 + ep: 32 + dp-attn: true + - spec-decoding: "mtp" + conc-list: [180] + prefill: + num-worker: 3 + tp: 4 + ep: 4 + dp-attn: true + additional-settings: + # https://github.com/NVIDIA/srt-slurm/blob/sa-submission-q2-2026/recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/MTP/ctx3dep4_gen1dep32_batch4_eplb384_mtp3.yaml + - "CONFIG_FILE=recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/MTP/ctx3dep4_gen1dep32_batch4_eplb384_mtp3.yaml" + decode: + num-worker: 1 + tp: 32 + ep: 32 + dp-attn: true + - spec-decoding: "mtp" + conc-list: [333] + prefill: + num-worker: 4 + tp: 4 + ep: 4 + dp-attn: true + additional-settings: + # https://github.com/NVIDIA/srt-slurm/blob/sa-submission-q2-2026/recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/MTP/ctx4dep4_gen1dep32_batch8_eplb384_mtp3.yaml + - "CONFIG_FILE=recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/MTP/ctx4dep4_gen1dep32_batch8_eplb384_mtp3.yaml" + decode: + num-worker: 1 + tp: 32 + ep: 32 + dp-attn: true + - spec-decoding: "mtp" + conc-list: [615] + prefill: + num-worker: 8 + tp: 4 + ep: 4 + dp-attn: true + additional-settings: + # https://github.com/NVIDIA/srt-slurm/blob/sa-submission-q2-2026/recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/MTP/ctx8dep4_gen1dep32_batch16_eplb384_mtp3.yaml + - "CONFIG_FILE=recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/MTP/ctx8dep4_gen1dep32_batch16_eplb384_mtp3.yaml" + decode: + num-worker: 1 + tp: 32 + ep: 32 + dp-attn: true + - spec-decoding: "mtp" + conc-list: [666] + prefill: + num-worker: 6 + tp: 4 + ep: 4 + dp-attn: true + additional-settings: + # https://github.com/NVIDIA/srt-slurm/blob/sa-submission-q2-2026/recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/MTP/ctx6dep4_gen1dep16_batch32_eplb384_mtp3.yaml + - "CONFIG_FILE=recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/MTP/ctx6dep4_gen1dep16_batch32_eplb384_mtp3.yaml" + decode: + num-worker: 1 + tp: 16 + ep: 16 + dp-attn: true + - spec-decoding: "mtp" + conc-list: [1229] + prefill: + num-worker: 7 + tp: 4 + ep: 4 + dp-attn: true + additional-settings: + # https://github.com/NVIDIA/srt-slurm/blob/sa-submission-q2-2026/recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/MTP/ctx7dep4_gen1dep8_batch128_eplb384_mtp3.yaml + - "CONFIG_FILE=recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/MTP/ctx7dep4_gen1dep8_batch128_eplb384_mtp3.yaml" + decode: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + - spec-decoding: "mtp" + conc-list: [1229] + prefill: + num-worker: 10 + tp: 4 + ep: 4 + dp-attn: true + additional-settings: + # https://github.com/NVIDIA/srt-slurm/blob/sa-submission-q2-2026/recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/MTP/ctx10dep4_gen1dep16_batch64_eplb384_mtp3.yaml + - "CONFIG_FILE=recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/MTP/ctx10dep4_gen1dep16_batch64_eplb384_mtp3.yaml" + decode: + num-worker: 1 + tp: 16 + ep: 16 + dp-attn: true + - spec-decoding: "mtp" + conc-list: [2253] + prefill: + num-worker: 9 + tp: 4 + ep: 4 + dp-attn: true + additional-settings: + # https://github.com/NVIDIA/srt-slurm/blob/sa-submission-q2-2026/recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/MTP/ctx9dep4_gen1dep8_batch256_eplb384_mtp1.yaml + - "CONFIG_FILE=recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/MTP/ctx9dep4_gen1dep8_batch256_eplb384_mtp1.yaml" + decode: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + - spec-decoding: "mtp" + conc-list: [4301] + prefill: + num-worker: 12 + tp: 4 + ep: 4 + dp-attn: true + additional-settings: + # https://github.com/NVIDIA/srt-slurm/blob/sa-submission-q2-2026/recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/MTP/ctx12dep4_gen1dep8_batch512_eplb384_mtp1.yaml + - "CONFIG_FILE=recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/MTP/ctx12dep4_gen1dep8_batch512_eplb384_mtp1.yaml" + decode: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + +dsv4-fp4-gb300-dynamo-sglang: + image: lmsysorg/sglang:v0.5.14-cu130@sha256:5027e95bf6ec536856b1b52a91d1f35ff5c564ab83e8a94758a169ff09bb8df3 + model: deepseek-ai/DeepSeek-V4-Pro + model-prefix: dsv4 + runner: gb300 + precision: fp4 + framework: dynamo-sglang + router: { name: dynamo-router, version: "81d0555ee23519cea80a42b4fe824e30368b7300" } + kv-p2p-transfer: mooncake + multinode: true + disagg: true + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + # WideEP TP=16 decode: 1p1d-dep4-dep16. 5 nodes (4P + 16D = 20 GPUs). + - conc-list: [1024] + prefill: + num-worker: 1 + tp: 4 + ep: 4 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-gb300-1p1d-dep4-dep16-5-c1024.yaml" + decode: + num-worker: 1 + tp: 16 + ep: 16 + dp-attn: true + # Low concurrency: 1p1d-tp4-tp4. 2 nodes. + - conc-list: [1] + prefill: + num-worker: 1 + tp: 4 + ep: 1 + dp-attn: false + additional-settings: + - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-gb300-1p1d-tp4-tp4-2-c1.yaml" + decode: + num-worker: 1 + tp: 4 + ep: 1 + dp-attn: false + # --- Weiliang wide-EP sweep (srt-slurm PR#173), 18 nodes total --- + # EP=12: 15P+3D, conc=12000. + - conc-list: [12000] + prefill: + num-worker: 15 + tp: 4 + ep: 4 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-gb300-15p1d-dep4-dep12-18-c12000.yaml" + decode: + num-worker: 1 + tp: 12 + ep: 12 + dp-attn: true + # EP=16: 14P+4D, conc=8192. + - conc-list: [8192] + prefill: + num-worker: 14 + tp: 4 + ep: 4 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-gb300-14p1d-dep4-dep16-18-c8192.yaml" + decode: + num-worker: 1 + tp: 16 + ep: 16 + dp-attn: true + # EP=24: 12P+6D, conc=3000. + - conc-list: [3000] + prefill: + num-worker: 12 + tp: 4 + ep: 4 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-gb300-12p1d-dep4-dep24-18-c3000.yaml" + decode: + num-worker: 1 + tp: 24 + ep: 24 + dp-attn: true + # EP=32: 10P+8D, conc=2500. + - conc-list: [2500] + prefill: + num-worker: 10 + tp: 4 + ep: 4 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-gb300-10p1d-dep4-dep32-18-c2500.yaml" + decode: + num-worker: 1 + tp: 32 + ep: 32 + dp-attn: true + # EP=40: 8P+10D, conc=2048. + - conc-list: [2048] + prefill: + num-worker: 8 + tp: 4 + ep: 4 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-gb300-8p1d-dep4-dep40-18-c2048.yaml" + decode: + num-worker: 1 + tp: 40 + ep: 40 + dp-attn: true + +# MTP variant of dsv4-fp4-gb300-dynamo-sglang. +dsv4-fp4-gb300-dynamo-sglang-mtp: + image: lmsysorg/sglang:nightly-dev-20260527-14f81a67 + model: deepseek-ai/DeepSeek-V4-Pro + model-prefix: dsv4 + runner: gb300 + precision: fp4 + framework: dynamo-sglang + router: { name: dynamo-router, version: "81d0555ee23519cea80a42b4fe824e30368b7300" } + kv-p2p-transfer: mooncake + multinode: true + disagg: true + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + # Low-latency baseline: 1p1d-tp4-tp4. 2 nodes. + - spec-decoding: "mtp" + conc-list: [1] + prefill: + num-worker: 1 + tp: 4 + ep: 1 + dp-attn: false + additional-settings: + - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-low-latency-1p1d-tp4-tp4-mtp.yaml" + decode: + num-worker: 1 + tp: 4 + ep: 1 + dp-attn: false + # Low-latency 1p6d-dep4-tp4: 1P (DEP=4) + 6 TP=4 decode workers. 7 nodes. + # Recipe runs concurrencies=8x32x64; matrix tracks the max. + - spec-decoding: "mtp" + conc-list: [64] + prefill: + num-worker: 1 + tp: 4 + ep: 4 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-low-latency-1p6d-dep4-tp4-mtp.yaml" + decode: + num-worker: 6 + tp: 4 + ep: 1 + dp-attn: false + # Mid curve 1p1d-dep4-dep8. 3 nodes. + - spec-decoding: "mtp" + conc-list: [256] + prefill: + num-worker: 1 + tp: 4 + ep: 4 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-mid-curve-1p1d-dep4-dep8-mtp.yaml" + decode: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + # Mid curve 1p1d-dep4-dep16. 5 nodes. + - spec-decoding: "mtp" + conc-list: [256] + prefill: + num-worker: 1 + tp: 4 + ep: 4 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-mid-curve-1p1d-dep4-dep16-mtp.yaml" + decode: + num-worker: 1 + tp: 16 + ep: 16 + dp-attn: true + # Mid curve 2p1d-dep4-dep8. 4 nodes. + - spec-decoding: "mtp" + conc-list: [512] + prefill: + num-worker: 2 + tp: 4 + ep: 4 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-mid-curve-2p1d-dep4-dep8-mtp.yaml" + decode: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + # Mid curve 4p1d-dep4-dep8. 6 nodes. + - spec-decoding: "mtp" + conc-list: [1024] + prefill: + num-worker: 4 + tp: 4 + ep: 4 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-mid-curve-4p1d-dep4-dep8-mtp.yaml" + decode: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + # High concurrency 6p1d-dep4-dep8. 8 nodes. + - spec-decoding: "mtp" + conc-list: [4096] + prefill: + num-worker: 6 + tp: 4 + ep: 4 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-high-conc-6p1d-dep4-dep8-mtp.yaml" + decode: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + # High concurrency 8p1d-dep4-dep8. 10 nodes. + - spec-decoding: "mtp" + conc-list: [8192] + prefill: + num-worker: 8 + tp: 4 + ep: 4 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-high-conc-8p1d-dep4-dep8-mtp.yaml" + decode: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + +# DSV4 B300 disaggregated dynamo-sglang MTP configuration. +# Each search-space entry maps one-to-one to a checked-in srt-slurm recipe under +# recipes/sglang/deepseek-v4/8k1k/. Runs on the b300 pool via +# runners/launch_b300-nv.sh. Prefill workers are TP4/DEP4 (two workers per +# 8-GPU B300 node); decode is DEP8 (one full node) except the TP4 topologies. +# EAGLE draft depth is tapered down as concurrency grows +# (3 steps / 4 draft tokens -> 2/3 at 6p1d -> 1/2 at 8p1d). +dsv4-fp4-b300-dynamo-sglang-mtp: + image: lmsysorg/sglang:nightly-dev-cu13-20260721-8905cbd4 + model: deepseek-ai/DeepSeek-V4-Pro + model-prefix: dsv4 + runner: b300 + precision: fp4 + framework: dynamo-sglang + kv-p2p-transfer: nixl + multinode: true + disagg: true + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + # 1p1d-tp4-tp4 (pure TP, no dp-attn). 2 nodes (1P + 1D). + - spec-decoding: "mtp" + conc-list: [1] + prefill: + num-worker: 1 + tp: 4 + ep: 1 + dp-attn: false + additional-settings: + - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-b300-1p1d-tp4-tp4-mtp.yaml" + decode: + num-worker: 1 + tp: 4 + ep: 1 + dp-attn: false + # Scale-out: 1x DEP4 prefill + 6x TP4 decode. 4 nodes (1P + 3D). + - spec-decoding: "mtp" + conc-list: [8, 32, 64] + prefill: + num-worker: 1 + tp: 4 + ep: 4 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-b300-1p6d-dep4-tp4-mtp.yaml" + decode: + num-worker: 6 + tp: 4 + ep: 1 + dp-attn: false + # 1p1d-dep4-dep8: 1x DEP4 prefill + DEP8 decode. 2 nodes (1P + 1D). + - spec-decoding: "mtp" + conc-list: [256] + prefill: + num-worker: 1 + tp: 4 + ep: 4 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-b300-1p1d-dep4-dep8-mtp.yaml" + decode: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + # 1p2d-dep4-dep8: 1x DEP4 prefill + 2x DEP8 decode. 3 nodes (1P + 2D). + - spec-decoding: "mtp" + conc-list: [256] + prefill: + num-worker: 1 + tp: 4 + ep: 4 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-b300-1p2d-dep4-dep8-mtp.yaml" + decode: + num-worker: 2 + tp: 8 + ep: 8 + dp-attn: true + # 2p1d-dep4-dep8: 2x DEP4 prefill + DEP8 decode. 2 nodes (1P + 1D). + - spec-decoding: "mtp" + conc-list: [512] + prefill: + num-worker: 2 + tp: 4 + ep: 4 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-b300-2p1d-dep4-dep8-mtp.yaml" + decode: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + # 4p1d-dep4-dep8: 4x DEP4 prefill + DEP8 decode. 3 nodes (2P + 1D). + - spec-decoding: "mtp" + conc-list: [1024] + prefill: + num-worker: 4 + tp: 4 + ep: 4 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-b300-4p1d-dep4-dep8-mtp.yaml" + decode: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + # 6p1d-dep4-dep8: 6x DEP4 prefill + DEP8 decode. 4 nodes (3P + 1D). + - spec-decoding: "mtp" + conc-list: [2048] + prefill: + num-worker: 6 + tp: 4 + ep: 4 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-b300-6p1d-dep4-dep8-mtp.yaml" + decode: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + # 8p1d-dep4-dep8. 5 nodes (4P + 1D). + - spec-decoding: "mtp" + conc-list: [3072] + prefill: + num-worker: 8 + tp: 4 + ep: 4 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-b300-8p1d-dep4-dep8-mtp.yaml" + decode: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true diff --git a/configs/nvidia-master.yaml b/configs/nvidia-master.yaml index 0c44ca1af5..54a115f90d 100644 --- a/configs/nvidia-master.yaml +++ b/configs/nvidia-master.yaml @@ -899,35 +899,6 @@ dsr1-fp4-b200-sglang-mtp: - { tp: 4, ep: 1, conc-start: 1, conc-end: 32, spec-decoding: mtp } - { tp: 4, ep: 4, dp-attn: true, conc-start: 64, conc-end: 256, spec-decoding: mtp } -dsv4-fp4-b200-sglang: - image: lmsysorg/sglang:nightly-dev-cu13-20260628-da802ddc - model: deepseek-ai/DeepSeek-V4-Pro - model-prefix: dsv4 - runner: cluster:b200-nscale - precision: fp4 - framework: sglang - multinode: false - # Two recipes from https://docs.sglang.io/cookbook/autoregressive/DeepSeek/DeepSeek-V4 - # are selected inside benchmarks/single_node/dsv4_fp4_b200.sh by DP_ATTENTION: - # low-latency (DP_ATTENTION=false): TP-only, flashinfer_mxfp4 - # DP-attention (DP_ATTENTION=true): DP-attn + DeepEP + mega_moe opts - # The DP-attention recipe covers both "balanced" (conc 64-128) and - # "max-throughput" (conc 256+) CONC ranges with identical flags; - # only --max-running-requests scales with CONC. - # ep is implicit in sglang: --moe-a2a-backend deepep forces ep_size=tp_size, - # while low-latency leaves ep_size at the default of 1. - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - # low-latency (DP_ATTENTION=false) - - { tp: 8, ep: 1, conc-start: 1, conc-end: 32 } - # DP-attention (DP_ATTENTION=true) — balanced CONC range - - { tp: 8, ep: 8, dp-attn: true, conc-start: 64, conc-end: 128 } - # DP-attention (DP_ATTENTION=true) — max-throughput CONC range - - { tp: 8, ep: 8, dp-attn: true, conc-start: 256, conc-end: 1024 } - dsv4-fp4-b200-sglang-agentic-hicache-mtp: image: lmsysorg/sglang:nightly-dev-20260827-20621aa1 model: deepseek-ai/DeepSeek-V4-Pro-0813 @@ -944,23 +915,6 @@ dsv4-fp4-b200-sglang-agentic-hicache-mtp: - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: hicache }, spec-decoding: draft_model, conc-list: [8, 10, 16] } - { tp: 8, ep: 8, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: hicache }, spec-decoding: draft_model, conc-list: [64, 96, 128, 160], router: { name: sglang-router, version: "0.3.2" } } -dsv4-fp4-b200-vllm: - image: vllm/vllm-openai:nightly-7c5dc571cbd1064ecc8a9b1045637ff647aa22cb - model: deepseek-ai/DeepSeek-V4-Pro - model-prefix: dsv4 - runner: cluster:b200-nscale - precision: fp4 - framework: vllm - multinode: false - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - - { tp: 8, conc-start: 1, conc-end: 256 } - - { tp: 8, ep: 8, conc-start: 1, conc-end: 256 } - - { tp: 8, ep: 8, dp-attn: true, conc-start: 64, conc-end: 1024 } - dsv4-fp4-b200-vllm-agentic-mtp: image: vllm/vllm-openai:nightly-dev-x86_64-cu13.0.1-426e59f model: deepseek-ai/DeepSeek-V4-Pro @@ -978,57 +932,6 @@ dsv4-fp4-b200-vllm-agentic-mtp: # DEP8 SimpleCPU + MTP - { tp: 8, ep: 8, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: vllm-simple }, spec-decoding: mtp, conc-list: [32, 64, 96, 128, 160, 196], router: { name: vllm-router, version: "0.1.14" } } -dsv4-fp4-b200-trt: - image: ghcr.io#semianalysisai/trtllm-deepseek-v4:feat-deepseek_v4-c185066 - model: deepseek-ai/DeepSeek-V4-Pro - model-prefix: dsv4 - runner: cluster:b200-nscale - precision: fp4 - framework: trt - multinode: false - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - - { tp: 8, conc-start: 1, conc-end: 32 } - - { tp: 8, ep: 8, dp-attn: true, conc-start: 32, conc-end: 512 } - -dsv4-fp4-b200-trt-mtp: - image: ghcr.io#semianalysisai/trtllm-deepseek-v4:feat-deepseek_v4-c185066 - model: deepseek-ai/DeepSeek-V4-Pro - model-prefix: dsv4 - runner: cluster:b200-nscale - precision: fp4 - framework: trt - multinode: false - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - - { tp: 8, conc-start: 1, conc-end: 32, spec-decoding: mtp } - - { tp: 8, ep: 8, dp-attn: true, conc-start: 32, conc-end: 256, spec-decoding: mtp } - -# MTP variant of dsv4-fp4-b200-vllm. Mirrors the base search space and adds -# --speculative-config '{"method":"mtp","num_speculative_tokens":2}'. -dsv4-fp4-b200-vllm-mtp: - image: vllm/vllm-openai:v0.25.0 - model: deepseek-ai/DeepSeek-V4-Pro - model-prefix: dsv4 - runner: cluster:b200-nscale - precision: fp4 - framework: vllm - multinode: false - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - - { tp: 8, conc-start: 1, conc-end: 64, spec-decoding: mtp } - - { tp: 8, ep: 8, conc-start: 128, conc-end: 256, spec-decoding: mtp } - - { tp: 8, ep: 8, dp-attn: true, conc-start: 64, conc-end: 256, spec-decoding: mtp } - # NOTE: At the time of submission, https://cookbook.sglang.io/autoregressive/DeepSeek/DeepSeek-R1 # does not have a B300-specific recipe, so this config reuses the existing DSR1 FP4 # B200 SGLang recipe as-is until B300-specific tuning is available. @@ -1126,34 +1029,6 @@ dsr1-fp8-b300-sglang: - { tp: 8, ep: 1, conc-start: 1, conc-end: 4 } - { tp: 4, ep: 1, conc-start: 1, conc-end: 32 } -# DeepSeek-V4-Pro on B300 with sglang (non-MTP). -# Uses nightly image with megamoe backend for high-concurrency profiles. -dsv4-fp4-b300-sglang: - image: lmsysorg/sglang:nightly-dev-cu13-20260624-b2c8f7a2 - model: deepseek-ai/DeepSeek-V4-Pro - model-prefix: dsv4 - runner: b300 - precision: fp4 - framework: sglang - multinode: false - # Recipes are selected inside benchmarks/single_node/dsv4_fp4_b300_sglang.sh - # by CONC: - # CONC 1|32: TP-only, flashinfer_mxfp4 - # CONC 512: DP-attn, flashinfer_mxfp4 - # CONC 2048-8192: DP-attn, megamoe - # ep is implicit in sglang: --moe-a2a-backend megamoe forces ep_size=tp_size, - # while low-latency leaves ep_size at the default of 1. - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - - { tp: 8, ep: 1, conc-start: 1, conc-end: 1 } - - { tp: 4, ep: 1, conc-start: 32, conc-end: 32 } - - { tp: 4, ep: 1, dp-attn: true, conc-start: 512, conc-end: 512 } - - { tp: 8, ep: 8, dp-attn: true, conc-start: 2048, conc-end: 2048 } - - { tp: 8, ep: 8, dp-attn: true, conc-start: 4096, conc-end: 4096 } - dsv4-fp4-b300-sglang-agentic-hicache-mtp: image: lmsysorg/sglang:nightly-dev-cu13-20260827-20621aa1 model: deepseek-ai/DeepSeek-V4-Pro @@ -1179,26 +1054,6 @@ dsv4-fp4-b300-sglang-agentic-hicache-mtp: # + swa-full-tokens-ratio 0.075 # + prefill-decode-interval 20 # Both paths share EAGLE (3,1,4) and max-running-requests 2*CONC. -dsv4-fp4-b300-sglang-mtp: - image: lmsysorg/sglang:nightly-dev-cu13-20260610-f332e526 - model: deepseek-ai/DeepSeek-V4-Pro - model-prefix: dsv4 - runner: b300 - precision: fp4 - framework: sglang - multinode: false - # Three CONC bands: - # A: TP=8 ep=1 -- conc 1-8 EAGLE (3,1,4) TP-only fallback - # B: TP=4 ep=1 -- conc 4-32 EAGLE (3,1,4) TP-only mid batch - # C: TP=4 ep=1 dp-attn -- conc 16-256 EAGLE (1,1,2) DP-attn flashinfer - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - - { tp: 8, ep: 1, conc-start: 1, conc-end: 8, spec-decoding: mtp } - - { tp: 4, ep: 1, conc-start: 4, conc-end: 32, spec-decoding: mtp } - qwen3.5-fp8-b200-sglang: image: lmsysorg/sglang:v0.5.19-cu130 model: Qwen/Qwen3.5-397B-A17B-FP8 @@ -1570,47 +1425,6 @@ dsr1-fp8-h200-sglang-mtp: search-space: - { tp: 8, ep: 1, conc-start: 4, conc-end: 64, spec-decoding: mtp } -# DeepSeek-V4-Pro H200 recipe from https://vllm.ai/blog/deepseek-v4 -# Uses the cu129 image. H200 has no FP4 path, so the FP4 indexer cache -# flag is omitted. Max-model-len is pinned at 800k per the recipe. -dsv4-fp8-h200-vllm: - image: vllm/vllm-openai:v0.25.1 - model: deepseek-ai/DeepSeek-V4-Pro - model-prefix: dsv4 - runner: h200 - precision: fp8 - framework: vllm - multinode: false - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - - { tp: 8, ep: 1, dp-attn: false, conc-start: 1, conc-end: 256 } - - { tp: 8, ep: 8, dp-attn: true, conc-start: 1, conc-end: 256 } - -# MTP variant of dsv4-fp8-h200-vllm. Mirrors the non-MTP image and adds -# --speculative-config '{"method":"mtp","num_speculative_tokens":2}'. -dsv4-fp8-h200-vllm-mtp: - image: vllm/vllm-openai:v0.25.1 - model: deepseek-ai/DeepSeek-V4-Pro - model-prefix: dsv4 - runner: h200 - precision: fp8 - framework: vllm - multinode: false - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - - { tp: 8, ep: 1, dp-attn: false, conc-start: 1, conc-end: 256, spec-decoding: mtp } - - { tp: 8, ep: 8, dp-attn: true, conc-start: 1, conc-end: 256, spec-decoding: mtp } - -# DeepSeek-V4-Pro H200 single-node with SGLang (Marlin FP8, TP-only). -# Pinned to the h200-dgxc-slurm runner pool because the deepseek-v4-hopper -# image needs the /ix mount layout that only launch_h200-dgxc-slurm.sh sets up. - # DeepSeek-V4-Pro AgentX on one aggregated TP8 H200 worker. Keep the serving # topology fixed and sweep only concurrency to produce the Pareto curve. dsv4-fp8-h200-dynamo-sglang-agentic-agg: @@ -1639,66 +1453,6 @@ dsv4-fp8-h200-dynamo-sglang-agentic-agg: dp-attn: false additional-settings: - "CONFIG_FILE=recipes/sglang/deepseek-v4/agentic/agg-h200-tp8-mtp-kvoffload.yaml" -# MTP variant of dsv4-fp8-h200-sglang. Mirrors the non-MTP recipe (same image, -# runner pool, search space) and adds EAGLE speculative decoding via -# --speculative-algorithm EAGLE with the (3,1,4) chain matching dsv4-fp4-b300-sglang-mtp. -dsv4-fp8-h200-sglang: - image: lmsysorg/sglang:deepseek-v4-hopper@sha256:1bf5d508ab110cc0fe1659a5f21d1be02a7f0d7ca8f58cea7e7f4e11f6ae208f - model: deepseek-ai/DeepSeek-V4-Pro - model-prefix: dsv4 - runner: h200-dgxc - precision: fp8 - framework: sglang - multinode: false - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - - { tp: 8, ep: 1, conc-start: 1, conc-end: 1 } - - { tp: 8, ep: 1, conc-start: 4, conc-end: 64 } - -# MTP variant of dsv4-fp8-h200-sglang. Mirrors the non-MTP recipe (same image, -# runner pool, search space) and adds EAGLE speculative decoding via -# --speculative-algorithm EAGLE with the (3,1,4) chain matching dsv4-fp4-b300-sglang-mtp. -dsv4-fp8-h200-sglang-mtp: - image: lmsysorg/sglang:deepseek-v4-hopper@sha256:1bf5d508ab110cc0fe1659a5f21d1be02a7f0d7ca8f58cea7e7f4e11f6ae208f - model: deepseek-ai/DeepSeek-V4-Pro - model-prefix: dsv4 - runner: h200-dgxc - precision: fp8 - framework: sglang - multinode: false - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - - { tp: 8, ep: 1, conc-start: 1, conc-end: 1, spec-decoding: mtp } - - { tp: 8, ep: 1, conc-start: 4, conc-end: 64, spec-decoding: mtp } - - # DeepSeek-V4-Pro B300 single-node aggregate recipe from the submitted B300 - # pareto sweep. The single-node schema has no explicit data-parallel-size - # field, so dp-attn=true is used as the existing vLLM script switch for DP4 - # layouts on 4 allocated GPUs. -dsv4-fp4-b300-vllm: - image: vllm/vllm-openai:v0.25.0 - model: deepseek-ai/DeepSeek-V4-Pro - model-prefix: dsv4 - runner: cluster:b300-nv - precision: fp4 - framework: vllm - multinode: false - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - - { tp: 4, conc-start: 1, conc-end: 64 } - - { tp: 8, conc-start: 1, conc-end: 4 } - - { tp: 4, ep: 4, dp-attn: true, conc-start: 256, conc-end: 512 } - - { tp: 8, ep: 8, dp-attn: true, conc-start: 2048, conc-end: 2048 } - dsv4-fp4-b300-vllm-agentic-mtp: image: vllm/vllm-openai:nightly-dev-x86_64-cu13.0.1-426e59f model: deepseek-ai/DeepSeek-V4-Pro @@ -1722,58 +1476,6 @@ dsv4-fp4-b300-vllm-agentic-mtp: # DEP8 SimpleCPU + MTP (num_speculative_tokens=3) - { tp: 8, ep: 8, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: vllm-simple }, spec-decoding: mtp, conc-list: [128, 256, 384, 512, 576], router: { name: vllm-router, version: "0.1.14" } } -dsv4-fp4-b300-trt: - image: ghcr.io#semianalysisai/trtllm-deepseek-v4:feat-deepseek_v4-c185066 - model: deepseek-ai/DeepSeek-V4-Pro - model-prefix: dsv4 - runner: b300 - precision: fp4 - framework: trt - multinode: false - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - - { tp: 4, conc-start: 1, conc-end: 32 } - - { tp: 4, ep: 4, dp-attn: true, conc-start: 64, conc-end: 64 } - - { tp: 8, ep: 8, dp-attn: true, conc-start: 256, conc-end: 1024 } - -dsv4-fp4-b300-trt-mtp: - image: ghcr.io#semianalysisai/trtllm-deepseek-v4:feat-deepseek_v4-c185066 - model: deepseek-ai/DeepSeek-V4-Pro - model-prefix: dsv4 - runner: b300 - precision: fp4 - framework: trt - multinode: false - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - - { tp: 4, conc-start: 1, conc-end: 32, spec-decoding: mtp } - - { tp: 4, ep: 4, dp-attn: true, conc-start: 64, conc-end: 64, spec-decoding: mtp } - - { tp: 8, ep: 8, dp-attn: true, conc-start: 256, conc-end: 1024, spec-decoding: mtp } - -dsv4-fp4-b300-vllm-mtp: - image: vllm/vllm-openai:v0.25.0 - model: deepseek-ai/DeepSeek-V4-Pro - model-prefix: dsv4 - runner: cluster:b300-nv - precision: fp4 - framework: vllm - multinode: false - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - - { tp: 4, conc-start: 1, conc-end: 64, spec-decoding: mtp } - - { tp: 8, conc-start: 1, conc-end: 8, spec-decoding: mtp } - - { tp: 4, ep: 4, conc-start: 64, conc-end: 256, spec-decoding: mtp } - - { tp: 4, ep: 4, dp-attn: true, conc-start: 256, conc-end: 512, spec-decoding: mtp } - qwen3.5-fp8-h200-sglang: image: lmsysorg/sglang:v0.5.14-cu130 model: Qwen/Qwen3.5-397B-A17B-FP8 @@ -3942,2202 +3644,62 @@ dsr1-fp8-b200-dynamo-sglang-mtp: dp-attn: true - spec-decoding: "mtp" conc-list: [160, 288] - prefill: - num-worker: 1 - tp: 8 - ep: 1 - dp-attn: true - additional-settings: - # https://github.com/NVIDIA/srt-slurm/blob/sa-submission-q2-2026/recipes/b200-fp8/8k1k_mtp_maxtpt_1.yaml - - "CONFIG_FILE=recipes/b200-fp8/8k1k_mtp_maxtpt_1.yaml" - router: { name: dynamo-router, version: "0.9.1" } - decode: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - - spec-decoding: "mtp" - conc-list: [512] - prefill: - num-worker: 2 - tp: 8 - ep: 1 - dp-attn: true - additional-settings: - # https://github.com/NVIDIA/srt-slurm/blob/sa-submission-q2-2026/recipes/b200-fp8/8k1k_mtp_maxtpt_2.yaml - - "CONFIG_FILE=recipes/b200-fp8/8k1k_mtp_maxtpt_2.yaml" - router: { name: dynamo-router, version: "0.9.1" } - decode: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - - spec-decoding: "mtp" - conc-list: [1024] - prefill: - num-worker: 3 - tp: 8 - ep: 1 - dp-attn: true - additional-settings: - # https://github.com/NVIDIA/srt-slurm/blob/sa-submission-q2-2026/recipes/b200-fp8/8k1k_mtp_maxtpt_3.yaml - - "CONFIG_FILE=recipes/b200-fp8/8k1k_mtp_maxtpt_3.yaml" - router: { name: dynamo-router, version: "0.9.1" } - decode: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - -dsr1-fp4-b200-dynamo-sglang-mtp: - image: "lmsysorg/sglang:v0.5.12.post1" - model: deepseek-r1-fp4 - model-prefix: dsr1 - runner: cluster:b200-nscale - precision: fp4 - framework: dynamo-sglang - router: { name: dynamo-router, version: "5b4bc1dd70965017a737c71b19db5a0aeaa88727" } - kv-p2p-transfer: nixl - multinode: true - disagg: true - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - # 1p5d low-latency (decode-heavy). - - spec-decoding: "mtp" - conc-list: [4, 8, 16, 32] - prefill: - num-worker: 1 - tp: 4 - ep: 1 - dp-attn: false - additional-settings: - - "CONFIG_FILE=recipes/sglang/dsr1/b200-fp4/8k1k/disagg/mtp/8k1k_mtp_lowlat_0.yaml" - decode: - num-worker: 5 - tp: 8 - ep: 1 - dp-attn: false - # 1p3d low-latency. - - spec-decoding: "mtp" - conc-list: [32, 64] - prefill: - num-worker: 1 - tp: 4 - ep: 1 - dp-attn: false - additional-settings: - - "CONFIG_FILE=recipes/sglang/dsr1/b200-fp4/8k1k/disagg/mtp/8k1k_mtp_lowlat_1.yaml" - decode: - num-worker: 3 - tp: 8 - ep: 1 - dp-attn: false - # 1p1d low-latency. - - spec-decoding: "mtp" - conc-list: [32] - prefill: - num-worker: 1 - tp: 4 - ep: 1 - dp-attn: false - additional-settings: - - "CONFIG_FILE=recipes/sglang/dsr1/b200-fp4/8k1k/disagg/mtp/8k1k_mtp_lowlat_2.yaml" - decode: - num-worker: 1 - tp: 8 - ep: 1 - dp-attn: false - # MTP2 high-throughput (DEP4 prefill / DEP8 decode), one Pareto point each. - # 1p1d throughput. - - spec-decoding: "mtp" - conc-list: [512] - prefill: - num-worker: 1 - tp: 4 - ep: 4 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/sglang/dsr1/b200-fp4/8k1k/disagg/mtp/8k1k_mtp2_throughput_1p1d.yaml" - decode: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - # 2p1d throughput. - - spec-decoding: "mtp" - conc-list: [768] - prefill: - num-worker: 2 - tp: 4 - ep: 4 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/sglang/dsr1/b200-fp4/8k1k/disagg/mtp/8k1k_mtp2_throughput_2p1d.yaml" - decode: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - # 3p1d throughput. - - spec-decoding: "mtp" - conc-list: [1024] - prefill: - num-worker: 3 - tp: 4 - ep: 4 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/sglang/dsr1/b200-fp4/8k1k/disagg/mtp/8k1k_mtp2_throughput_3p1d.yaml" - decode: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - # 4p1d throughput. - - spec-decoding: "mtp" - conc-list: [512] - prefill: - num-worker: 4 - tp: 4 - ep: 4 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/sglang/dsr1/b200-fp4/8k1k/disagg/mtp/8k1k_mtp2_throughput_4p1d.yaml" - decode: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - # 5p1d throughput. - - spec-decoding: "mtp" - conc-list: [2048] - prefill: - num-worker: 5 - tp: 4 - ep: 4 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/sglang/dsr1/b200-fp4/8k1k/disagg/mtp/8k1k_mtp2_throughput_5p1d.yaml" - decode: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - # 4p1d high-throughput (dedicated c2048 tune). - - spec-decoding: "mtp" - conc-list: [2048] - prefill: - num-worker: 4 - tp: 4 - ep: 4 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/sglang/dsr1/b200-fp4/8k1k/disagg/mtp/8k1k_mtp_4p1d_c2048.yaml" - decode: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - -kimik2.6-fp4-b200-dynamo-vllm: - image: vllm/vllm-openai:v0.25.1 - model: nvidia/Kimi-K2.6-NVFP4 - model-prefix: kimik2.6 - runner: cluster:b200-nscale - precision: fp4 - framework: dynamo-vllm - router: { name: dynamo-router, version: "1.3.0.dev20260721" } - kv-p2p-transfer: nixl - multinode: true - disagg: true - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - - conc-list: [1] - prefill: - num-worker: 1 - tp: 1 - ep: 8 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/kimi-k2.6/b200-fp4/8k1k/disagg-b200-1p1d-dep8-tp8-c1.yaml" - decode: - num-worker: 1 - tp: 8 - ep: 1 - dp-attn: false - - conc-list: [32] - prefill: - num-worker: 1 - tp: 1 - ep: 4 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/kimi-k2.6/b200-fp4/8k1k/disagg-b200-1p8d-dep4-tp4-c32.yaml" - decode: - num-worker: 8 - tp: 4 - ep: 1 - dp-attn: false - - conc-list: [128] - prefill: - num-worker: 1 - tp: 1 - ep: 4 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/kimi-k2.6/b200-fp4/8k1k/disagg-b200-1p8d-dep4-tp4-c128.yaml" - decode: - num-worker: 8 - tp: 4 - ep: 1 - dp-attn: false - - conc-list: [512] - prefill: - num-worker: 1 - tp: 1 - ep: 4 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/kimi-k2.6/b200-fp4/8k1k/disagg-b200-1p4d-dep4-tp4-c512.yaml" - decode: - num-worker: 4 - tp: 4 - ep: 1 - dp-attn: false - - conc-list: [2048] - prefill: - num-worker: 1 - tp: 1 - ep: 8 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/kimi-k2.6/b200-fp4/8k1k/disagg-b200-1p1d-dep8-dep8-c2048.yaml" - decode: - num-worker: 1 - tp: 1 - ep: 8 - dp-attn: true - - conc-list: [8192] - prefill: - num-worker: 2 - tp: 1 - ep: 8 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/kimi-k2.6/b200-fp4/8k1k/disagg-b200-2p1d-dep8-dep8-c8192.yaml" - decode: - num-worker: 1 - tp: 1 - ep: 8 - dp-attn: true - -dsv4-fp4-b200-dynamo-vllm: - image: vllm/vllm-openai:v0.23.0 - model: deepseek-ai/DeepSeek-V4-Pro - model-prefix: dsv4 - runner: cluster:b200-nscale - precision: fp4 - framework: dynamo-vllm - router: { name: dynamo-router, version: "1.2.0.dev20260426" } - kv-p2p-transfer: nixl - multinode: true - disagg: true - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - # B200 adaptation of the DSV4 GB200 vLLM disagg recipes. Each worker - # maps to one full 8-GPU B200 node. - - conc-list: [1] - prefill: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/deepseek-v4/8k1k/disagg-b200-low-latency-c1.yaml" - decode: - num-worker: 1 - tp: 8 - ep: 1 - dp-attn: false - - conc-list: [32, 128] - prefill: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/deepseek-v4/8k1k/disagg-b200-low-latency-c32-c128.yaml" - decode: - num-worker: 1 - tp: 8 - ep: 1 - dp-attn: false - - conc-list: [64] - prefill: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/deepseek-v4/8k1k/disagg-b200-low-latency-c64.yaml" - decode: - num-worker: 1 - tp: 8 - ep: 1 - dp-attn: false - - conc-list: [256] - prefill: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/deepseek-v4/8k1k/disagg-b200-low-middle-c256.yaml" - decode: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - - conc-list: [512] - prefill: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/deepseek-v4/8k1k/disagg-b200-low-middle-c512.yaml" - decode: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - -dsv4-fp4-gb200-dynamo-vllm: - image: vllm/vllm-openai:v0.25.1 - model: deepseek-ai/DeepSeek-V4-Pro - model-prefix: dsv4 - runner: gb200 - precision: fp4 - framework: dynamo-vllm - router: { name: dynamo-router, version: "1.2.0.dev20260426" } - kv-p2p-transfer: nixl - multinode: true - disagg: true - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - # Validated 8k/1k points mirrored from NVIDIA/srt-slurm - # aflowers/vllm-gb200-v0.20.0 history. conc-list values match each - # recipe's benchmark.concurrencies. - - # Low latency: 1 prefill (DEP=8) + 1 decode (TP=8). 5 nodes total with - # a dedicated NATS/etcd infra node. - - conc-list: [1] - prefill: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/deepseek-v4/8k1k/disagg-gb200-low-latency.yaml" - decode: - num-worker: 1 - tp: 8 - ep: 1 - dp-attn: false - - # Low-middle curve: 1 prefill (DEP=8) + 4 decode (TP=8). 11 nodes total - # with a dedicated NATS/etcd infra node. - - conc-list: [256, 512] - prefill: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/deepseek-v4/8k1k/disagg-gb200-low-middle-curve.yaml" - decode: - num-worker: 4 - tp: 8 - ep: 1 - dp-attn: false - - # MegaMOE mid curve: 1 prefill (DEP=8) + 1 decode (DEP=8). 5 nodes - # total with a dedicated NATS/etcd infra node. - - conc-list: [256, 512, 1024] - prefill: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/deepseek-v4/8k1k/disagg-gb200-mid-curve-megamoe.yaml" - decode: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - - # MegaMOE high throughput: 2 prefill (DEP=8 each) + 1 decode (DEP=8). - # 7 nodes total with a dedicated NATS/etcd infra node. - - conc-list: [4096] - prefill: - num-worker: 2 - tp: 8 - ep: 8 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/deepseek-v4/8k1k/disagg-gb200-high-tpt-megamoe.yaml" - decode: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - - # MegaMOE max throughput: 3 prefill (DEP=8 each) + 1 decode (DEP=8). - # 9 nodes total with a dedicated NATS/etcd infra node. - - conc-list: [4096] - prefill: - num-worker: 3 - tp: 8 - ep: 8 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/deepseek-v4/8k1k/disagg-gb200-max-tpt-megamoe.yaml" - decode: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - -# TODO: change image to official llmd image. -# Build source: benchmarks/llm-d/Dockerfile. -dsv4-fp4-gb200-llmd-vllm: - image: quay.io/rh-ee-imarkov/llm-d-nokube-vllm:vllm0.26@sha256:a9095d4c835935c4070be2040de0a5ef3b44098f603092ab66d743b0e731b7b4 - model: deepseek-ai/DeepSeek-V4-Pro - model-prefix: dsv4 - runner: gb200 - precision: fp4 - framework: llmd-vllm - router: { name: llm-d-router, version: "0.9.0" } - kv-p2p-transfer: nixl - multinode: true - disagg: true - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - # Low latency: 1 prefill DEP8 + 1 decode TP8. - - spec-decoding: "none" - conc-list: [1] - prefill: - num-worker: 1 - tp: 1 - ep: 8 - dp-attn: true - additional-settings: - - "PREFILL_NODES=2" - - "GPUS_PER_NODE=4" - - "CONFIG_FILE=dsv4-fp4-gb200-low-latency.yaml" - decode: - num-worker: 1 - tp: 8 - ep: 1 - dp-attn: false - additional-settings: - - "DECODE_NODES=2" - - "GPUS_PER_NODE=4" - - # Mid curve: 1 prefill DEP8 + 1 decode DEP8. - - spec-decoding: "none" - conc-list: [256, 512, 1024] - prefill: - num-worker: 1 - tp: 1 - ep: 8 - dp-attn: true - additional-settings: - - "PREFILL_NODES=2" - - "GPUS_PER_NODE=4" - - "CONFIG_FILE=dsv4-fp4-gb200-mid-curve-megamoe.yaml" - decode: - num-worker: 1 - tp: 1 - ep: 8 - dp-attn: true - additional-settings: - - "DECODE_NODES=2" - - "GPUS_PER_NODE=4" - - # Max throughput: 3 prefill DEP8 + 1 decode DEP8. - - spec-decoding: "none" - conc-list: [4096] - prefill: - num-worker: 3 - tp: 1 - ep: 8 - dp-attn: true - additional-settings: - - "PREFILL_NODES=6" - - "PREFILL_WORKERS=3" - - "GPUS_PER_NODE=4" - - "CONFIG_FILE=dsv4-fp4-gb200-mid-curve-megamoe.yaml" - decode: - num-worker: 1 - tp: 1 - ep: 8 - dp-attn: true - additional-settings: - - "DECODE_NODES=2" - - "GPUS_PER_NODE=4" - -# MTP2 variant of dsv4-fp4-gb200-dynamo-vllm. Uses the vLLM 0.20.1 image -# and hand-picked 8k/1k Pareto points mirrored from NVIDIA/srt-slurm. -dsv4-fp4-gb200-dynamo-vllm-mtp2: - image: vllm/vllm-openai:v0.20.1-ubuntu2404 - model: deepseek-ai/DeepSeek-V4-Pro - model-prefix: dsv4 - runner: gb200 - precision: fp4 - framework: dynamo-vllm - router: { name: dynamo-router, version: "1.2.0.dev20260426" } - kv-p2p-transfer: nixl - multinode: true - disagg: true - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - # Aggregate low latency: TP=8, max-num-seqs=4. - - conc-list: [1] - spec-decoding: mtp - prefill: - num-worker: 1 - tp: 8 - ep: 1 - dp-attn: false - additional-settings: - - "SYNTHETIC_ACCEPTANCE=true" - - "SYNTHETIC_ACCEPTANCE_LENGTH=2.27" - - "CONFIG_FILE=recipes/vllm/deepseek-v4/8k1k/agg-gb200-low-latency-mtp2.yaml" - decode: - num-worker: 0 - tp: 8 - ep: 1 - dp-attn: false - - # Low-latency bridge: 1 prefill (DEP=8) + 4 decode (TP=8), no offload. - - conc-list: [16, 32, 64] - spec-decoding: mtp - prefill: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - additional-settings: - - "SYNTHETIC_ACCEPTANCE=true" - - "SYNTHETIC_ACCEPTANCE_LENGTH=2.27" - - "CONFIG_FILE=recipes/vllm/deepseek-v4/8k1k/disagg-gb200-low-latency-mtp2.yaml" - decode: - num-worker: 4 - tp: 8 - ep: 1 - dp-attn: false - - # MegaMOE mid curve: 1 prefill (DEP=8) + 1 decode (DEP=8). - # 5 nodes total with a dedicated NATS/etcd infra node. - - conc-list: [128, 256, 512, 1024] - spec-decoding: mtp - prefill: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - additional-settings: - - "SYNTHETIC_ACCEPTANCE=true" - - "SYNTHETIC_ACCEPTANCE_LENGTH=2.27" - - "CONFIG_FILE=recipes/vllm/deepseek-v4/8k1k/disagg-gb200-mid-curve-megamoe-mtp2.yaml" - decode: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - - # MegaMOE high throughput: 2 prefill (DEP=8 each) + 1 decode (DEP=8). - # 7 nodes total with a dedicated NATS/etcd infra node. - - conc-list: [1024] - spec-decoding: mtp - prefill: - num-worker: 2 - tp: 8 - ep: 8 - dp-attn: true - additional-settings: - - "SYNTHETIC_ACCEPTANCE=true" - - "SYNTHETIC_ACCEPTANCE_LENGTH=2.27" - - "CONFIG_FILE=recipes/vllm/deepseek-v4/8k1k/disagg-gb200-high-tpt-megamoe-mtp2.yaml" - decode: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - -# Baseline (no synthetic acceptance) variant of dsv4-fp4-gb200-dynamo-vllm-mtp2 -# for before/after Pareto comparison. Identical topology and recipes, only the -# SYNTHETIC_ACCEPTANCE envs are removed. -dsv4-fp4-gb200-dynamo-vllm-mtp2-nosynthetic: - image: vllm/vllm-openai:v0.20.1-ubuntu2404 - model: deepseek-ai/DeepSeek-V4-Pro - model-prefix: dsv4 - runner: gb200 - precision: fp4 - framework: dynamo-vllm - router: { name: dynamo-router, version: "1.2.0.dev20260426" } - multinode: true - disagg: true - kv-p2p-transfer: nixl - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - # Aggregate low latency: TP=8, max-num-seqs=4. - - conc-list: [1] - spec-decoding: mtp - prefill: - num-worker: 1 - tp: 8 - ep: 1 - dp-attn: false - additional-settings: - - "CONFIG_FILE=recipes/vllm/deepseek-v4/8k1k/agg-gb200-low-latency-mtp2.yaml" - decode: - num-worker: 0 - tp: 8 - ep: 1 - dp-attn: false - - # Low-latency bridge: 1 prefill (DEP=8) + 4 decode (TP=8), no offload. - - conc-list: [16, 32, 64] - spec-decoding: mtp - prefill: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/deepseek-v4/8k1k/disagg-gb200-low-latency-mtp2.yaml" - decode: - num-worker: 4 - tp: 8 - ep: 1 - dp-attn: false - - # MegaMOE mid curve: 1 prefill (DEP=8) + 1 decode (DEP=8). - # 5 nodes total with a dedicated NATS/etcd infra node. - - conc-list: [128, 256, 512, 1024] - spec-decoding: mtp - prefill: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/deepseek-v4/8k1k/disagg-gb200-mid-curve-megamoe-mtp2.yaml" - decode: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - - # MegaMOE high throughput: 2 prefill (DEP=8 each) + 1 decode (DEP=8). - # 7 nodes total with a dedicated NATS/etcd infra node. - - conc-list: [1024] - spec-decoding: mtp - prefill: - num-worker: 2 - tp: 8 - ep: 8 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/deepseek-v4/8k1k/disagg-gb200-high-tpt-megamoe-mtp2.yaml" - decode: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - -# DSV4 B200 disaggregated Dynamo SGLang STP configuration. -dsv4-fp4-b200-dynamo-sglang: - image: lmsysorg/sglang:nightly-dev-cu13-20260821-f825d729 - model: deepseek-ai/DeepSeek-V4-Pro - model-prefix: dsv4 - runner: cluster:b200-nscale - precision: fp4 - framework: dynamo-sglang - router: { name: dynamo-router, version: "86f84b9486905c2bfaa4b756c9fa39f04ecf5ebd" } - kv-p2p-transfer: nixl - multinode: true - disagg: true - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - # 1p1d-tp8-tp8 (pure TP). 2 nodes (1+1). - - conc-list: [1] - prefill: - num-worker: 1 - tp: 8 - ep: 1 - dp-attn: false - additional-settings: - - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-b200-1p1d-tp8-tp8.yaml" - decode: - num-worker: 1 - tp: 8 - ep: 1 - dp-attn: false - # 1p1d-dep8-tp8: DEP8 prefill + TP8 decode. 2 nodes (1+1). - - conc-list: [32, 64] - prefill: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-b200-1p1d-dep8-tp8.yaml" - decode: - num-worker: 1 - tp: 8 - ep: 1 - dp-attn: false - # 1p1d-dep8-dep8: DEP8 prefill + DEP8 decode. 2 nodes (1+1). - - conc-list: [256, 512] - prefill: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-b200-1p1d-dep8-dep8.yaml" - decode: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - # 1p4d-dep8-tp8: DEP8 prefill + 4x TP8 decode. 5 nodes (1+4). - - conc-list: [64] - prefill: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-b200-1p4d-dep8-tp8.yaml" - decode: - num-worker: 4 - tp: 8 - ep: 1 - dp-attn: false - # 1p2d-dep8-dep8: DEP8 prefill + 2x DEP8 decode. 3 nodes (1+2). - - conc-list: [256] - prefill: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-b200-1p2d-dep8-dep8.yaml" - decode: - num-worker: 2 - tp: 8 - ep: 8 - dp-attn: true - # 1p4d-dep8-dep8: DEP8 prefill + 4x DEP8 decode. 5 nodes (1+4). - - conc-list: [256] - prefill: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-b200-1p4d-dep8-dep8.yaml" - decode: - num-worker: 4 - tp: 8 - ep: 8 - dp-attn: true - # 3p2d-dep8-dep8: 3x DEP8 prefill + 2x DEP8 decode. 5 nodes (3+2). - - conc-list: [2048] - prefill: - num-worker: 3 - tp: 8 - ep: 8 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-b200-3p2d-dep8-dep8.yaml" - decode: - num-worker: 2 - tp: 8 - ep: 8 - dp-attn: true - # 4p2d-dep8-dep8: 4x DEP8 prefill + 2x DEP8 decode. 6 nodes (4+2). - - conc-list: [4096] - prefill: - num-worker: 4 - tp: 8 - ep: 8 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-b200-4p2d-dep8-dep8.yaml" - decode: - num-worker: 2 - tp: 8 - ep: 8 - dp-attn: true - # 5p2d-dep8-dep8: 5x DEP8 prefill + 2x DEP8 decode. 7 nodes (5+2). - - conc-list: [6144] - prefill: - num-worker: 5 - tp: 8 - ep: 8 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-b200-5p2d-dep8-dep8.yaml" - decode: - num-worker: 2 - tp: 8 - ep: 8 - dp-attn: true - # 6p2d-dep8-dep8: 6x DEP8 prefill + 2x DEP8 decode. 8 nodes (6+2). - - conc-list: [8192] - prefill: - num-worker: 6 - tp: 8 - ep: 8 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-b200-6p2d-dep8-dep8.yaml" - decode: - num-worker: 2 - tp: 8 - ep: 8 - dp-attn: true - -dsv4-fp4-gb200-dynamo-sglang: - image: lmsysorg/sglang:nightly-dev-cu13-20260528-0abe6a85 - model: deepseek-ai/DeepSeek-V4-Pro - model-prefix: dsv4 - runner: gb200 - precision: fp4 - framework: dynamo-sglang - router: { name: dynamo-router, version: "92f5b3b8d7dd5ab9179d4b1034bd2c1c0803693e" } - kv-p2p-transfer: mooncake - multinode: true - disagg: true - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - # Low latency: 1p1d-tp8-tp8. 4 nodes. - - conc-list: [1] - prefill: - num-worker: 1 - tp: 8 - ep: 1 - dp-attn: false - additional-settings: - - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-gb200-1p1d-tp8-tp8-4-c1.yaml" - decode: - num-worker: 1 - tp: 8 - ep: 1 - dp-attn: false - # 1p4d-dep8-tp8. 10 nodes. - - conc-list: [64] - prefill: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-gb200-1p4d-dep8-tp8-10-c64.yaml" - decode: - num-worker: 4 - tp: 8 - ep: 1 - dp-attn: false - # WideEP TP=16 decode: 1p2d-dep8-dep16. 10 nodes. - - conc-list: [256] - prefill: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-gb200-1p2d-dep8-dep16-10-c256.yaml" - decode: - num-worker: 2 - tp: 16 - ep: 16 - dp-attn: true - # WideEP TP=16 decode: 1p1d-dep8-dep16. 6 nodes. - - conc-list: [512] - prefill: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-gb200-1p1d-dep8-dep16-6-c512.yaml" - decode: - num-worker: 1 - tp: 16 - ep: 16 - dp-attn: true - # WideEP TP=16 decode: 2p1d-dep8-dep16. 8 nodes. - - conc-list: [1536] - prefill: - num-worker: 2 - tp: 8 - ep: 8 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-gb200-2p1d-dep8-dep16-8-c1536.yaml" - decode: - num-worker: 1 - tp: 16 - ep: 16 - dp-attn: true - # WideEP TP=16 decode: 4p1d-dep8-dep16. 12 nodes. - - conc-list: [4096] - prefill: - num-worker: 4 - tp: 8 - ep: 8 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-gb200-4p1d-dep8-dep16-12-c4096.yaml" - decode: - num-worker: 1 - tp: 16 - ep: 16 - dp-attn: true - # WideEP TP=16 decode: 5p1d-dep8-dep16. 14 nodes. - - conc-list: [8192] - prefill: - num-worker: 5 - tp: 8 - ep: 8 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-gb200-5p1d-dep8-dep16-14-c8192.yaml" - decode: - num-worker: 1 - tp: 16 - ep: 16 - dp-attn: true - # WideEP TP=12 decode: 6p1d-dep8-dep12. 15 nodes. - - conc-list: [8192] - prefill: - num-worker: 6 - tp: 8 - ep: 8 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-gb200-6p1d-dep8-dep12-15-c8192.yaml" - decode: - num-worker: 1 - tp: 12 - ep: 12 - dp-attn: true - -qwen3.5-fp8-gb200-dynamo-sglang: - image: lmsysorg/sglang:nightly-dev-cu13-20260608-303757cc - model: Qwen/Qwen3.5-397B-A17B-FP8 - model-prefix: qwen3.5 - runner: gb200 - precision: fp8 - framework: dynamo-sglang - router: { name: dynamo-router, version: "46520ca59afe992fb5ef61b3197b2316f8df9b2b" } - kv-p2p-transfer: mooncake - multinode: true - disagg: true - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - # 1P1D STP: TP4 prefill + TP4 decode (pure tensor parallel). 2 nodes (1+1). - - spec-decoding: "none" - conc-list: [1, 2, 4, 8, 16, 32, 64, 128] - prefill: - num-worker: 1 - tp: 4 - ep: 1 - dp-attn: false - additional-settings: - - "CONFIG_FILE=recipes/sglang/qwen3.5/gb200-fp8/8k1k/1p1d-tp4-tp4.yaml" - decode: - num-worker: 1 - tp: 4 - ep: 1 - dp-attn: false - # 4P1D wide-EP: 4 prefill DEP4 + decode DEP16. 8 nodes (4+4). - - spec-decoding: "none" - conc-list: [1024] - prefill: - num-worker: 4 - tp: 4 - ep: 4 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/sglang/qwen3.5/gb200-fp8/8k1k/4p1d-dep4-dep16.yaml" - decode: - num-worker: 1 - tp: 16 - ep: 16 - dp-attn: true - # 8P1D wide-EP: 8 prefill DEP4 + decode DEP16. 12 nodes (8+4). - - spec-decoding: "none" - conc-list: [2048, 4096] - prefill: - num-worker: 8 - tp: 4 - ep: 4 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/sglang/qwen3.5/gb200-fp8/8k1k/8p1d-dep4-dep16.yaml" - decode: - num-worker: 1 - tp: 16 - ep: 16 - dp-attn: true - -# MTP variant of dsv4-fp4-gb200-dynamo-sglang. -dsv4-fp4-gb200-dynamo-sglang-mtp: - image: lmsysorg/sglang:nightly-dev-cu13-20260528-0abe6a85 - model: deepseek-ai/DeepSeek-V4-Pro - model-prefix: dsv4 - runner: gb200 - precision: fp4 - framework: dynamo-sglang - router: { name: dynamo-router, version: "92f5b3b8d7dd5ab9179d4b1034bd2c1c0803693e" } - kv-p2p-transfer: mooncake - multinode: true - disagg: true - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - # Low-latency baseline: 1p1d-tp8-tp8. 4 nodes. - - spec-decoding: "mtp" - conc-list: [1] - prefill: - num-worker: 1 - tp: 8 - ep: 1 - dp-attn: false - additional-settings: - - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-gb200-low-latency-1p1d-tp8-tp8-mtp.yaml" - decode: - num-worker: 1 - tp: 8 - ep: 1 - dp-attn: false - # Low-latency 1p6d-dep8-tp8: 1P (DEP=8) + 6 TP=8 decode workers. 14 nodes. - # Recipe runs concurrencies=32x64x128; matrix tracks the max. - - spec-decoding: "mtp" - conc-list: [128] - prefill: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-gb200-low-latency-1p6d-dep8-tp8-mtp.yaml" - decode: - num-worker: 6 - tp: 8 - ep: 1 - dp-attn: false - # Mid curve 1p1d-dep8-dep16. 6 nodes. - - spec-decoding: "mtp" - conc-list: [1024] - prefill: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-gb200-mid-curve-1p1d-dep8-dep16-mtp.yaml" - decode: - num-worker: 1 - tp: 16 - ep: 16 - dp-attn: true - # Mid curve 2p1d-dep8-dep16. 8 nodes. - - spec-decoding: "mtp" - conc-list: [2048] - prefill: - num-worker: 2 - tp: 8 - ep: 8 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-gb200-mid-curve-2p1d-dep8-dep16-mtp.yaml" - decode: - num-worker: 1 - tp: 16 - ep: 16 - dp-attn: true - # Mid curve 3p1d-dep8-dep16. 10 nodes. - - spec-decoding: "mtp" - conc-list: [3072] - prefill: - num-worker: 3 - tp: 8 - ep: 8 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-gb200-mid-curve-3p1d-dep8-dep16-mtp.yaml" - decode: - num-worker: 1 - tp: 16 - ep: 16 - dp-attn: true - # Mid curve 4p1d-dep8-dep16. 12 nodes. - - spec-decoding: "mtp" - conc-list: [6144] - prefill: - num-worker: 4 - tp: 8 - ep: 8 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-gb200-mid-curve-4p1d-dep8-dep16-mtp.yaml" - decode: - num-worker: 1 - tp: 16 - ep: 16 - dp-attn: true - # Mid curve 5p1d-dep8-dep16. 14 nodes. - - spec-decoding: "mtp" - conc-list: [8192] - prefill: - num-worker: 5 - tp: 8 - ep: 8 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-gb200-mid-curve-5p1d-dep8-dep16-mtp.yaml" - decode: - num-worker: 1 - tp: 16 - ep: 16 - dp-attn: true - # Mid curve 6p1d-dep8-dep16. 16 nodes. - - spec-decoding: "mtp" - conc-list: [16384] - prefill: - num-worker: 6 - tp: 8 - ep: 8 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-gb200-mid-curve-6p1d-dep8-dep16-mtp.yaml" - decode: - num-worker: 1 - tp: 16 - ep: 16 - dp-attn: true - - # ---------- 1k1k high-throughput (wide-EP decode, MTP) ---------- -dsv4-fp4-b300-dynamo-vllm: - image: vllm/vllm-openai:v0.23.0 - model: deepseek-ai/DeepSeek-V4-Pro - model-prefix: dsv4 - runner: b300 - precision: fp4 - framework: dynamo-vllm - router: { name: dynamo-router, version: "1.2.0.dev20260426" } - kv-p2p-transfer: nixl - multinode: true - disagg: true - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - # B300 adaptation of the DSV4 B200/GB200 vLLM disagg recipes. Each - # prefill/decode worker maps to one full 8-GPU B300 node. - - conc-list: [1, 32, 64, 128] - prefill: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/deepseek-v4/8k1k/disagg-b300-low-latency.yaml" - decode: - num-worker: 1 - tp: 8 - ep: 1 - dp-attn: false - - conc-list: [256, 1024] - prefill: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/deepseek-v4/8k1k/disagg-b300-mid-curve-megamoe.yaml" - decode: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - - conc-list: [4096] - prefill: - num-worker: 2 - tp: 8 - ep: 8 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/deepseek-v4/8k1k/disagg-b300-high-tpt-megamoe.yaml" - decode: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - -# DSV4 B300 disaggregated dynamo-sglang STP configuration. -# Each search-space entry maps one-to-one to a checked-in srt-slurm recipe under -# recipes/sglang/deepseek-v4/8k1k/. Runs on the b300 pool via -# runners/launch_b300-nv.sh. Prefill workers are TP4/DEP4 (two workers per -# 8-GPU B300 node); the decode worker is a single DEP8 node. -dsv4-fp4-b300-dynamo-sglang: - image: lmsysorg/sglang:nightly-dev-cu13-20260721-8905cbd4 - model: deepseek-ai/DeepSeek-V4-Pro - model-prefix: dsv4 - runner: b300 - precision: fp4 - framework: dynamo-sglang - kv-p2p-transfer: nixl - multinode: true - disagg: true - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - # 1p1d-tp4-tp4 (pure TP, no dp-attn). 2 nodes (1P + 1D). - - conc-list: [1, 8, 16, 32] - prefill: - num-worker: 1 - tp: 4 - ep: 1 - dp-attn: false - additional-settings: - - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-b300-1p1d-tp4-tp4.yaml" - decode: - num-worker: 1 - tp: 4 - ep: 1 - dp-attn: false - # 1p1d-dep4-dep8: 1x DEP4 prefill + DEP8 decode. 2 nodes (1P + 1D). - - conc-list: [256] - prefill: - num-worker: 1 - tp: 4 - ep: 4 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-b300-1p1d-dep4-dep8.yaml" - decode: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - # 2p1d-dep4-dep8: 2x DEP4 prefill + DEP8 decode. 2 nodes (1P + 1D). - - conc-list: [768] - prefill: - num-worker: 2 - tp: 4 - ep: 4 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-b300-2p1d-dep4-dep8.yaml" - decode: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - # 4p1d-dep4-dep8: 4x DEP4 prefill + DEP8 decode. 3 nodes (2P + 1D). - - conc-list: [2048] - prefill: - num-worker: 4 - tp: 4 - ep: 4 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-b300-4p1d-dep4-dep8.yaml" - decode: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - # 6p1d-dep4-dep8: 6x DEP4 prefill + DEP8 decode. 4 nodes (3P + 1D). - - conc-list: [3072] - prefill: - num-worker: 6 - tp: 4 - ep: 4 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-b300-6p1d-dep4-dep8.yaml" - decode: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - -dsv4-fp4-gb300-dynamo-vllm: - image: vllm/vllm-openai:dsv4-megamoe-mxfp4-arm64-cu130-4ba0a72 - model: deepseek-ai/DeepSeek-V4-Pro - model-prefix: dsv4 - runner: gb300-nv - precision: fp4 - framework: dynamo-vllm - router: { name: dynamo-router, version: "1.2.0.dev20260426" } - kv-p2p-transfer: nixl - multinode: true - disagg: true - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - - conc-list: [192] - prefill: - num-worker: 1 - tp: 4 - ep: 4 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/deepseek-v4/8k1k/disagg-gb300-1p6d-dep4-tp4.yaml" - decode: - num-worker: 6 - tp: 4 - ep: 1 - dp-attn: false - - conc-list: [18] - prefill: - num-worker: 1 - tp: 4 - ep: 4 - dp-attn: false - additional-settings: - - "CONFIG_FILE=recipes/vllm/deepseek-v4/8k1k/disagg-gb300-1p9d-tep4-tp4.yaml" - decode: - num-worker: 9 - tp: 4 - ep: 1 - dp-attn: false - - conc-list: [4096] - prefill: - num-worker: 4 - tp: 4 - ep: 4 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/deepseek-v4/8k1k/disagg-gb300-4p1d-dep4-dep8-24-c4096.yaml" - decode: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - - conc-list: [4096] - prefill: - num-worker: 5 - tp: 4 - ep: 4 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/deepseek-v4/8k1k/disagg-gb300-5p1d-dep4-dep8-28-c4096.yaml" - decode: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - - conc-list: [4096] - prefill: - num-worker: 6 - tp: 4 - ep: 4 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/deepseek-v4/8k1k/disagg-gb300-6p1d-dep4-dep8-32-c4096.yaml" - decode: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - - conc-list: [3072] - prefill: - num-worker: 7 - tp: 4 - ep: 4 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/deepseek-v4/8k1k/disagg-gb300-7p2d-dep4-dep16.yaml" - decode: - num-worker: 2 - tp: 16 - ep: 16 - dp-attn: true - -dsv4-fp4-b200-dynamo-sglang-mtp: - image: lmsysorg/sglang:nightly-dev-cu13-20260710-cfc66e05 - model: deepseek-ai/DeepSeek-V4-Pro - model-prefix: dsv4 - runner: cluster:b200-nscale - precision: fp4 - framework: dynamo-sglang - router: { name: dynamo-router, version: "92f5b3b8d7dd5ab9179d4b1034bd2c1c0803693e" } - kv-p2p-transfer: nixl - multinode: true - disagg: true - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - - spec-decoding: "mtp" - conc-list: [1] - prefill: - num-worker: 1 - tp: 8 - ep: 1 - dp-attn: false - additional-settings: - - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-b200-low-latency-1p1d-tp8-tp8-mtp.yaml" - decode: - num-worker: 1 - tp: 8 - ep: 1 - dp-attn: false - - spec-decoding: "mtp" - conc-list: [32, 64, 128] - prefill: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-b200-low-latency-1p6d-dep8-tp8-mtp.yaml" - decode: - num-worker: 6 - tp: 8 - ep: 1 - dp-attn: false - - spec-decoding: "mtp" - conc-list: [256, 1024] - prefill: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-b200-mid-curve-1p1d-dep8-dep8-mtp.yaml" - decode: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - - spec-decoding: "mtp" - conc-list: [256] - prefill: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-b200-mid-curve-1p2d-dep8-dep8-mtp.yaml" - decode: - num-worker: 2 - tp: 8 - ep: 8 - dp-attn: true - - spec-decoding: "mtp" - conc-list: [6144] - prefill: - num-worker: 5 - tp: 8 - ep: 8 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-b200-mid-curve-5p3d-dep8-dep8-mtp.yaml" - decode: - num-worker: 3 - tp: 8 - ep: 8 - dp-attn: true - -dsv4-fp4-b200-dynamo-vllm-mtp: - image: vllm/vllm-openai:vllm-x86_64-cu13-0.25.1-7a33ba9 - model: deepseek-ai/DeepSeek-V4-Pro - model-prefix: dsv4 - runner: cluster:b200-nscale - precision: fp4 - framework: dynamo-vllm - multinode: true - disagg: true - kv-p2p-transfer: nixl - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - # B200 MTP disagg recipes. Each worker maps to one full 8-GPU B200 node - # (prefill dep8; decode tp8, except the low-latency tp8 variant). - # SYNTHETIC_ACCEPTANCE is a throughput-only control (injected into the - # recipe by runners/inject_synthetic_acceptance.py); EVAL_ONLY runs keep - # the checked-in real-MTP recipe so verification is measured normally. - - spec-decoding: "mtp" - conc-list: [1, 8, 16] - prefill: - num-worker: 1 - tp: 1 - ep: 8 - dp-attn: true - additional-settings: - - "SYNTHETIC_ACCEPTANCE=true" - - "SYNTHETIC_ACCEPTANCE_LENGTH=2.49" - - "CONFIG_FILE=recipes/vllm/deepseek-v4/8k1k/disagg-b200-1p1d-dep8-tp8-mtp.yaml" - decode: - num-worker: 1 - tp: 8 - ep: 1 - dp-attn: false - - spec-decoding: "mtp" - conc-list: [128, 256] - prefill: - num-worker: 1 - tp: 1 - ep: 8 - dp-attn: true - additional-settings: - - "SYNTHETIC_ACCEPTANCE=true" - - "SYNTHETIC_ACCEPTANCE_LENGTH=2.49" - - "CONFIG_FILE=recipes/vllm/deepseek-v4/8k1k/disagg-b200-1p2d-dep8-dep8-mtp.yaml" - decode: - num-worker: 2 - tp: 1 - ep: 8 - dp-attn: true - - spec-decoding: "mtp" - conc-list: [512] - prefill: - num-worker: 1 - tp: 1 - ep: 8 - dp-attn: true - additional-settings: - - "SYNTHETIC_ACCEPTANCE=true" - - "SYNTHETIC_ACCEPTANCE_LENGTH=2.49" - - "CONFIG_FILE=recipes/vllm/deepseek-v4/8k1k/disagg-b200-1p1d-dep8-dep8-mtp.yaml" - decode: - num-worker: 1 - tp: 1 - ep: 8 - dp-attn: true - - spec-decoding: "mtp" - conc-list: [1024] - prefill: - num-worker: 2 - tp: 1 - ep: 8 - dp-attn: true - additional-settings: - - "SYNTHETIC_ACCEPTANCE=true" - - "SYNTHETIC_ACCEPTANCE_LENGTH=2.49" - - "CONFIG_FILE=recipes/vllm/deepseek-v4/8k1k/disagg-b200-2p1d-dep8-dep8-mtp.yaml" - decode: - num-worker: 1 - tp: 1 - ep: 8 - dp-attn: true - - spec-decoding: "mtp" - conc-list: [4096] - prefill: - num-worker: 3 - tp: 1 - ep: 8 - dp-attn: true - additional-settings: - - "SYNTHETIC_ACCEPTANCE=true" - - "SYNTHETIC_ACCEPTANCE_LENGTH=1.79" - - "CONFIG_FILE=recipes/vllm/deepseek-v4/8k1k/disagg-b200-3p1d-dep8-dep8-mtp.yaml" - decode: - num-worker: 1 - tp: 1 - ep: 8 - dp-attn: true - -dsv4-fp4-gb300-dynamo-trt: - image: nvcr.io#nvidia/ai-dynamo/tensorrtllm-runtime:1.3.0-deepseek-v4-dev.1 - model: deepseek-ai/DeepSeek-V4-Pro - model-prefix: dsv4 - runner: gb300-nv - precision: fp4 - framework: dynamo-trt - router: { name: dynamo-router, version: "v1.3.0-deepseek-v4-dev.1" } - kv-p2p-transfer: nixl - multinode: true - disagg: true - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - - conc-list: [4] - prefill: - num-worker: 1 - tp: 4 - ep: 4 - dp-attn: true - additional-settings: - # https://github.com/NVIDIA/srt-slurm/blob/sa-submission-q2-2026/recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/STP/ctx1dep4_gen4tep8_batch1_eplb0_mtp0.yaml - - "CONFIG_FILE=recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/STP/ctx1dep4_gen4tep8_batch1_eplb0_mtp0.yaml" - decode: - num-worker: 4 - tp: 8 - ep: 8 - dp-attn: false - - conc-list: [5] - prefill: - num-worker: 1 - tp: 4 - ep: 4 - dp-attn: true - additional-settings: - # https://github.com/NVIDIA/srt-slurm/blob/sa-submission-q2-2026/recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/STP/ctx1dep4_gen5tep4_batch1_eplb0_mtp0.yaml - - "CONFIG_FILE=recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/STP/ctx1dep4_gen5tep4_batch1_eplb0_mtp0.yaml" - decode: - num-worker: 5 - tp: 4 - ep: 4 - dp-attn: false - - conc-list: [15] - prefill: - num-worker: 1 - tp: 4 - ep: 4 - dp-attn: true - additional-settings: - # https://github.com/NVIDIA/srt-slurm/blob/sa-submission-q2-2026/recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/STP/ctx1dep4_gen5tep4_batch2_eplb0_mtp0.yaml - - "CONFIG_FILE=recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/STP/ctx1dep4_gen5tep4_batch2_eplb0_mtp0.yaml" - decode: - num-worker: 5 - tp: 4 - ep: 4 - dp-attn: false - - conc-list: [25] - prefill: - num-worker: 1 - tp: 4 - ep: 4 - dp-attn: true - additional-settings: - # https://github.com/NVIDIA/srt-slurm/blob/sa-submission-q2-2026/recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/STP/ctx1dep4_gen5tep4_batch4_eplb0_mtp0.yaml - - "CONFIG_FILE=recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/STP/ctx1dep4_gen5tep4_batch4_eplb0_mtp0.yaml" - decode: - num-worker: 5 - tp: 4 - ep: 4 - dp-attn: false - - conc-list: [55] - prefill: - num-worker: 1 - tp: 4 - ep: 4 - dp-attn: true - additional-settings: - # https://github.com/NVIDIA/srt-slurm/blob/sa-submission-q2-2026/recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/STP/ctx1dep4_gen5tep4_batch8_eplb0_mtp0.yaml - - "CONFIG_FILE=recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/STP/ctx1dep4_gen5tep4_batch8_eplb0_mtp0.yaml" - decode: - num-worker: 5 - tp: 4 - ep: 4 - dp-attn: false - - conc-list: [154] - prefill: - num-worker: 1 - tp: 4 - ep: 4 - dp-attn: true - additional-settings: - # https://github.com/NVIDIA/srt-slurm/blob/sa-submission-q2-2026/recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/STP/ctx1dep4_gen1dep32_batch4_eplb384_mtp0.yaml - - "CONFIG_FILE=recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/STP/ctx1dep4_gen1dep32_batch4_eplb384_mtp0.yaml" - decode: - num-worker: 1 - tp: 32 - ep: 32 - dp-attn: true - - conc-list: [308] - prefill: - num-worker: 2 - tp: 4 - ep: 4 - dp-attn: true - additional-settings: - # https://github.com/NVIDIA/srt-slurm/blob/sa-submission-q2-2026/recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/STP/ctx2dep4_gen1dep32_batch8_eplb384_mtp0.yaml - - "CONFIG_FILE=recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/STP/ctx2dep4_gen1dep32_batch8_eplb384_mtp0.yaml" - decode: - num-worker: 1 - tp: 32 - ep: 32 - dp-attn: true - - conc-list: [615] - prefill: - num-worker: 4 - tp: 4 - ep: 4 - dp-attn: true - additional-settings: - # https://github.com/NVIDIA/srt-slurm/blob/sa-submission-q2-2026/recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/STP/ctx4dep4_gen1dep32_batch16_eplb384_mtp0.yaml - - "CONFIG_FILE=recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/STP/ctx4dep4_gen1dep32_batch16_eplb384_mtp0.yaml" - decode: - num-worker: 1 - tp: 32 - ep: 32 - dp-attn: true - - conc-list: [1127] - prefill: - num-worker: 6 - tp: 4 - ep: 4 - dp-attn: true - additional-settings: - # https://github.com/NVIDIA/srt-slurm/blob/sa-submission-q2-2026/recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/STP/ctx6dep4_gen1dep32_batch32_eplb384_mtp0.yaml - - "CONFIG_FILE=recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/STP/ctx6dep4_gen1dep32_batch32_eplb384_mtp0.yaml" - decode: - num-worker: 1 - tp: 32 - ep: 32 - dp-attn: true - - conc-list: [1229] - prefill: - num-worker: 5 - tp: 4 - ep: 4 - dp-attn: true - additional-settings: - # https://github.com/NVIDIA/srt-slurm/blob/sa-submission-q2-2026/recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/STP/ctx5dep4_gen1dep16_batch64_eplb384_mtp0.yaml - - "CONFIG_FILE=recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/STP/ctx5dep4_gen1dep16_batch64_eplb384_mtp0.yaml" - decode: - num-worker: 1 - tp: 16 - ep: 16 - dp-attn: true - - conc-list: [2253] - prefill: - num-worker: 6 - tp: 4 - ep: 4 - dp-attn: true - additional-settings: - # https://github.com/NVIDIA/srt-slurm/blob/sa-submission-q2-2026/recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/STP/ctx6dep4_gen1dep8_batch256_eplb384_mtp0.yaml - - "CONFIG_FILE=recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/STP/ctx6dep4_gen1dep8_batch256_eplb384_mtp0.yaml" - decode: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - - conc-list: [2253] - prefill: - num-worker: 9 - tp: 4 - ep: 4 - dp-attn: true - additional-settings: - # https://github.com/NVIDIA/srt-slurm/blob/sa-submission-q2-2026/recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/STP/ctx9dep4_gen1dep16_batch128_eplb384_mtp0.yaml - - "CONFIG_FILE=recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/STP/ctx9dep4_gen1dep16_batch128_eplb384_mtp0.yaml" - decode: - num-worker: 1 - tp: 16 - ep: 16 - dp-attn: true - - conc-list: [4301] - prefill: - num-worker: 10 - tp: 4 - ep: 4 - dp-attn: true - additional-settings: - # https://github.com/NVIDIA/srt-slurm/blob/sa-submission-q2-2026/recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/STP/ctx10dep4_gen1dep8_batch512_eplb384_mtp0.yaml - - "CONFIG_FILE=recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/STP/ctx10dep4_gen1dep8_batch512_eplb384_mtp0.yaml" - decode: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - -dsv4-fp4-gb300-dynamo-trt-mtp: - image: nvcr.io#nvidia/ai-dynamo/tensorrtllm-runtime:1.3.0-deepseek-v4-dev.1 - model: deepseek-ai/DeepSeek-V4-Pro - model-prefix: dsv4 - runner: gb300-nv - precision: fp4 - framework: dynamo-trt - router: { name: dynamo-router, version: "v1.3.0-deepseek-v4-dev.1" } - kv-p2p-transfer: nixl - multinode: true - disagg: true - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - - spec-decoding: "mtp" - conc-list: [8] - prefill: - num-worker: 1 - tp: 4 - ep: 4 - dp-attn: true - additional-settings: - # https://github.com/NVIDIA/srt-slurm/blob/sa-submission-q2-2026/recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/MTP/ctx1dep4_gen4tep8_batch1_eplb0_mtp3.yaml - - "CONFIG_FILE=recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/MTP/ctx1dep4_gen4tep8_batch1_eplb0_mtp3.yaml" - decode: - num-worker: 4 - tp: 8 - ep: 8 - dp-attn: false - - spec-decoding: "mtp" - conc-list: [10] - prefill: - num-worker: 1 - tp: 4 - ep: 4 - dp-attn: true - additional-settings: - # https://github.com/NVIDIA/srt-slurm/blob/sa-submission-q2-2026/recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/MTP/ctx1dep4_gen5tep4_batch1_eplb0_mtp3.yaml - - "CONFIG_FILE=recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/MTP/ctx1dep4_gen5tep4_batch1_eplb0_mtp3.yaml" - decode: - num-worker: 5 - tp: 4 - ep: 4 - dp-attn: false - - spec-decoding: "mtp" - conc-list: [15] - prefill: - num-worker: 1 - tp: 4 - ep: 4 - dp-attn: true - additional-settings: - # https://github.com/NVIDIA/srt-slurm/blob/sa-submission-q2-2026/recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/MTP/ctx1dep4_gen5tep4_batch2_eplb0_mtp3.yaml - - "CONFIG_FILE=recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/MTP/ctx1dep4_gen5tep4_batch2_eplb0_mtp3.yaml" - decode: - num-worker: 5 - tp: 4 - ep: 4 - dp-attn: false - - spec-decoding: "mtp" - conc-list: [30] - prefill: - num-worker: 1 - tp: 4 - ep: 4 - dp-attn: true - additional-settings: - # https://github.com/NVIDIA/srt-slurm/blob/sa-submission-q2-2026/recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/MTP/ctx1dep4_gen5tep4_batch4_eplb0_mtp3.yaml - - "CONFIG_FILE=recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/MTP/ctx1dep4_gen5tep4_batch4_eplb0_mtp3.yaml" - decode: - num-worker: 5 - tp: 4 - ep: 4 - dp-attn: false - - spec-decoding: "mtp" - conc-list: [84] - prefill: - num-worker: 2 - tp: 4 - ep: 4 - dp-attn: true - additional-settings: - # https://github.com/NVIDIA/srt-slurm/blob/sa-submission-q2-2026/recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/MTP/ctx2dep4_gen1dep32_batch2_eplb384_mtp3.yaml - - "CONFIG_FILE=recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/MTP/ctx2dep4_gen1dep32_batch2_eplb384_mtp3.yaml" - decode: - num-worker: 1 - tp: 32 - ep: 32 - dp-attn: true - - spec-decoding: "mtp" - conc-list: [180] - prefill: - num-worker: 3 - tp: 4 - ep: 4 - dp-attn: true - additional-settings: - # https://github.com/NVIDIA/srt-slurm/blob/sa-submission-q2-2026/recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/MTP/ctx3dep4_gen1dep32_batch4_eplb384_mtp3.yaml - - "CONFIG_FILE=recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/MTP/ctx3dep4_gen1dep32_batch4_eplb384_mtp3.yaml" - decode: - num-worker: 1 - tp: 32 - ep: 32 - dp-attn: true - - spec-decoding: "mtp" - conc-list: [333] - prefill: - num-worker: 4 - tp: 4 - ep: 4 - dp-attn: true - additional-settings: - # https://github.com/NVIDIA/srt-slurm/blob/sa-submission-q2-2026/recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/MTP/ctx4dep4_gen1dep32_batch8_eplb384_mtp3.yaml - - "CONFIG_FILE=recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/MTP/ctx4dep4_gen1dep32_batch8_eplb384_mtp3.yaml" - decode: - num-worker: 1 - tp: 32 - ep: 32 - dp-attn: true - - spec-decoding: "mtp" - conc-list: [615] - prefill: - num-worker: 8 - tp: 4 - ep: 4 - dp-attn: true - additional-settings: - # https://github.com/NVIDIA/srt-slurm/blob/sa-submission-q2-2026/recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/MTP/ctx8dep4_gen1dep32_batch16_eplb384_mtp3.yaml - - "CONFIG_FILE=recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/MTP/ctx8dep4_gen1dep32_batch16_eplb384_mtp3.yaml" - decode: - num-worker: 1 - tp: 32 - ep: 32 - dp-attn: true - - spec-decoding: "mtp" - conc-list: [666] - prefill: - num-worker: 6 - tp: 4 - ep: 4 - dp-attn: true - additional-settings: - # https://github.com/NVIDIA/srt-slurm/blob/sa-submission-q2-2026/recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/MTP/ctx6dep4_gen1dep16_batch32_eplb384_mtp3.yaml - - "CONFIG_FILE=recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/MTP/ctx6dep4_gen1dep16_batch32_eplb384_mtp3.yaml" - decode: - num-worker: 1 - tp: 16 - ep: 16 - dp-attn: true - - spec-decoding: "mtp" - conc-list: [1229] - prefill: - num-worker: 7 - tp: 4 - ep: 4 - dp-attn: true - additional-settings: - # https://github.com/NVIDIA/srt-slurm/blob/sa-submission-q2-2026/recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/MTP/ctx7dep4_gen1dep8_batch128_eplb384_mtp3.yaml - - "CONFIG_FILE=recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/MTP/ctx7dep4_gen1dep8_batch128_eplb384_mtp3.yaml" - decode: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - - spec-decoding: "mtp" - conc-list: [1229] - prefill: - num-worker: 10 - tp: 4 - ep: 4 - dp-attn: true - additional-settings: - # https://github.com/NVIDIA/srt-slurm/blob/sa-submission-q2-2026/recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/MTP/ctx10dep4_gen1dep16_batch64_eplb384_mtp3.yaml - - "CONFIG_FILE=recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/MTP/ctx10dep4_gen1dep16_batch64_eplb384_mtp3.yaml" - decode: - num-worker: 1 - tp: 16 - ep: 16 - dp-attn: true - - spec-decoding: "mtp" - conc-list: [2253] - prefill: - num-worker: 9 - tp: 4 - ep: 4 - dp-attn: true - additional-settings: - # https://github.com/NVIDIA/srt-slurm/blob/sa-submission-q2-2026/recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/MTP/ctx9dep4_gen1dep8_batch256_eplb384_mtp1.yaml - - "CONFIG_FILE=recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/MTP/ctx9dep4_gen1dep8_batch256_eplb384_mtp1.yaml" - decode: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - - spec-decoding: "mtp" - conc-list: [4301] - prefill: - num-worker: 12 - tp: 4 - ep: 4 - dp-attn: true - additional-settings: - # https://github.com/NVIDIA/srt-slurm/blob/sa-submission-q2-2026/recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/MTP/ctx12dep4_gen1dep8_batch512_eplb384_mtp1.yaml - - "CONFIG_FILE=recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/MTP/ctx12dep4_gen1dep8_batch512_eplb384_mtp1.yaml" - decode: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - -dsv4-fp4-gb300-dynamo-sglang: - image: lmsysorg/sglang:v0.5.14-cu130@sha256:5027e95bf6ec536856b1b52a91d1f35ff5c564ab83e8a94758a169ff09bb8df3 - model: deepseek-ai/DeepSeek-V4-Pro - model-prefix: dsv4 - runner: gb300 - precision: fp4 - framework: dynamo-sglang - router: { name: dynamo-router, version: "81d0555ee23519cea80a42b4fe824e30368b7300" } - kv-p2p-transfer: mooncake - multinode: true - disagg: true - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - # WideEP TP=16 decode: 1p1d-dep4-dep16. 5 nodes (4P + 16D = 20 GPUs). - - conc-list: [1024] - prefill: - num-worker: 1 - tp: 4 - ep: 4 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-gb300-1p1d-dep4-dep16-5-c1024.yaml" - decode: - num-worker: 1 - tp: 16 - ep: 16 - dp-attn: true - # Low concurrency: 1p1d-tp4-tp4. 2 nodes. - - conc-list: [1] - prefill: - num-worker: 1 - tp: 4 - ep: 1 - dp-attn: false - additional-settings: - - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-gb300-1p1d-tp4-tp4-2-c1.yaml" - decode: - num-worker: 1 - tp: 4 - ep: 1 - dp-attn: false - # --- Weiliang wide-EP sweep (srt-slurm PR#173), 18 nodes total --- - # EP=12: 15P+3D, conc=12000. - - conc-list: [12000] - prefill: - num-worker: 15 - tp: 4 - ep: 4 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-gb300-15p1d-dep4-dep12-18-c12000.yaml" - decode: - num-worker: 1 - tp: 12 - ep: 12 - dp-attn: true - # EP=16: 14P+4D, conc=8192. - - conc-list: [8192] - prefill: - num-worker: 14 - tp: 4 - ep: 4 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-gb300-14p1d-dep4-dep16-18-c8192.yaml" - decode: - num-worker: 1 - tp: 16 - ep: 16 - dp-attn: true - # EP=24: 12P+6D, conc=3000. - - conc-list: [3000] - prefill: - num-worker: 12 - tp: 4 - ep: 4 + prefill: + num-worker: 1 + tp: 8 + ep: 1 dp-attn: true additional-settings: - - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-gb300-12p1d-dep4-dep24-18-c3000.yaml" + # https://github.com/NVIDIA/srt-slurm/blob/sa-submission-q2-2026/recipes/b200-fp8/8k1k_mtp_maxtpt_1.yaml + - "CONFIG_FILE=recipes/b200-fp8/8k1k_mtp_maxtpt_1.yaml" + router: { name: dynamo-router, version: "0.9.1" } decode: num-worker: 1 - tp: 24 - ep: 24 + tp: 8 + ep: 8 dp-attn: true - # EP=32: 10P+8D, conc=2500. - - conc-list: [2500] + - spec-decoding: "mtp" + conc-list: [512] prefill: - num-worker: 10 - tp: 4 - ep: 4 + num-worker: 2 + tp: 8 + ep: 1 dp-attn: true additional-settings: - - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-gb300-10p1d-dep4-dep32-18-c2500.yaml" + # https://github.com/NVIDIA/srt-slurm/blob/sa-submission-q2-2026/recipes/b200-fp8/8k1k_mtp_maxtpt_2.yaml + - "CONFIG_FILE=recipes/b200-fp8/8k1k_mtp_maxtpt_2.yaml" + router: { name: dynamo-router, version: "0.9.1" } decode: num-worker: 1 - tp: 32 - ep: 32 + tp: 8 + ep: 8 dp-attn: true - # EP=40: 8P+10D, conc=2048. - - conc-list: [2048] + - spec-decoding: "mtp" + conc-list: [1024] prefill: - num-worker: 8 - tp: 4 - ep: 4 + num-worker: 3 + tp: 8 + ep: 1 dp-attn: true additional-settings: - - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-gb300-8p1d-dep4-dep40-18-c2048.yaml" + # https://github.com/NVIDIA/srt-slurm/blob/sa-submission-q2-2026/recipes/b200-fp8/8k1k_mtp_maxtpt_3.yaml + - "CONFIG_FILE=recipes/b200-fp8/8k1k_mtp_maxtpt_3.yaml" + router: { name: dynamo-router, version: "0.9.1" } decode: num-worker: 1 - tp: 40 - ep: 40 + tp: 8 + ep: 8 dp-attn: true -# MTP variant of dsv4-fp4-gb300-dynamo-sglang. -dsv4-fp4-gb300-dynamo-sglang-mtp: - image: lmsysorg/sglang:nightly-dev-20260527-14f81a67 - model: deepseek-ai/DeepSeek-V4-Pro - model-prefix: dsv4 - runner: gb300 +dsr1-fp4-b200-dynamo-sglang-mtp: + image: "lmsysorg/sglang:v0.5.12.post1" + model: deepseek-r1-fp4 + model-prefix: dsr1 + runner: cluster:b200-nscale precision: fp4 framework: dynamo-sglang - router: { name: dynamo-router, version: "81d0555ee23519cea80a42b4fe824e30368b7300" } - kv-p2p-transfer: mooncake + router: { name: dynamo-router, version: "5b4bc1dd70965017a737c71b19db5a0aeaa88727" } + kv-p2p-transfer: nixl multinode: true disagg: true scenarios: @@ -6145,142 +3707,151 @@ dsv4-fp4-gb300-dynamo-sglang-mtp: - isl: 8192 osl: 1024 search-space: - # Low-latency baseline: 1p1d-tp4-tp4. 2 nodes. + # 1p5d low-latency (decode-heavy). - spec-decoding: "mtp" - conc-list: [1] + conc-list: [4, 8, 16, 32] prefill: num-worker: 1 tp: 4 ep: 1 dp-attn: false additional-settings: - - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-low-latency-1p1d-tp4-tp4-mtp.yaml" + - "CONFIG_FILE=recipes/sglang/dsr1/b200-fp4/8k1k/disagg/mtp/8k1k_mtp_lowlat_0.yaml" decode: + num-worker: 5 + tp: 8 + ep: 1 + dp-attn: false + # 1p3d low-latency. + - spec-decoding: "mtp" + conc-list: [32, 64] + prefill: num-worker: 1 tp: 4 ep: 1 dp-attn: false - # Low-latency 1p6d-dep4-tp4: 1P (DEP=4) + 6 TP=4 decode workers. 7 nodes. - # Recipe runs concurrencies=8x32x64; matrix tracks the max. + additional-settings: + - "CONFIG_FILE=recipes/sglang/dsr1/b200-fp4/8k1k/disagg/mtp/8k1k_mtp_lowlat_1.yaml" + decode: + num-worker: 3 + tp: 8 + ep: 1 + dp-attn: false + # 1p1d low-latency. - spec-decoding: "mtp" - conc-list: [64] + conc-list: [32] prefill: num-worker: 1 tp: 4 - ep: 4 - dp-attn: true + ep: 1 + dp-attn: false additional-settings: - - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-low-latency-1p6d-dep4-tp4-mtp.yaml" + - "CONFIG_FILE=recipes/sglang/dsr1/b200-fp4/8k1k/disagg/mtp/8k1k_mtp_lowlat_2.yaml" decode: - num-worker: 6 - tp: 4 + num-worker: 1 + tp: 8 ep: 1 dp-attn: false - # Mid curve 1p1d-dep4-dep8. 3 nodes. + # MTP2 high-throughput (DEP4 prefill / DEP8 decode), one Pareto point each. + # 1p1d throughput. - spec-decoding: "mtp" - conc-list: [256] + conc-list: [512] prefill: num-worker: 1 tp: 4 ep: 4 dp-attn: true additional-settings: - - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-mid-curve-1p1d-dep4-dep8-mtp.yaml" + - "CONFIG_FILE=recipes/sglang/dsr1/b200-fp4/8k1k/disagg/mtp/8k1k_mtp2_throughput_1p1d.yaml" decode: num-worker: 1 tp: 8 ep: 8 dp-attn: true - # Mid curve 1p1d-dep4-dep16. 5 nodes. + # 2p1d throughput. - spec-decoding: "mtp" - conc-list: [256] + conc-list: [768] prefill: - num-worker: 1 + num-worker: 2 tp: 4 ep: 4 dp-attn: true additional-settings: - - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-mid-curve-1p1d-dep4-dep16-mtp.yaml" + - "CONFIG_FILE=recipes/sglang/dsr1/b200-fp4/8k1k/disagg/mtp/8k1k_mtp2_throughput_2p1d.yaml" decode: num-worker: 1 - tp: 16 - ep: 16 + tp: 8 + ep: 8 dp-attn: true - # Mid curve 2p1d-dep4-dep8. 4 nodes. + # 3p1d throughput. - spec-decoding: "mtp" - conc-list: [512] + conc-list: [1024] prefill: - num-worker: 2 + num-worker: 3 tp: 4 ep: 4 dp-attn: true additional-settings: - - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-mid-curve-2p1d-dep4-dep8-mtp.yaml" + - "CONFIG_FILE=recipes/sglang/dsr1/b200-fp4/8k1k/disagg/mtp/8k1k_mtp2_throughput_3p1d.yaml" decode: num-worker: 1 tp: 8 ep: 8 dp-attn: true - # Mid curve 4p1d-dep4-dep8. 6 nodes. + # 4p1d throughput. - spec-decoding: "mtp" - conc-list: [1024] + conc-list: [512] prefill: num-worker: 4 tp: 4 ep: 4 dp-attn: true additional-settings: - - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-mid-curve-4p1d-dep4-dep8-mtp.yaml" + - "CONFIG_FILE=recipes/sglang/dsr1/b200-fp4/8k1k/disagg/mtp/8k1k_mtp2_throughput_4p1d.yaml" decode: num-worker: 1 tp: 8 ep: 8 dp-attn: true - # High concurrency 6p1d-dep4-dep8. 8 nodes. + # 5p1d throughput. - spec-decoding: "mtp" - conc-list: [4096] + conc-list: [2048] prefill: - num-worker: 6 + num-worker: 5 tp: 4 ep: 4 dp-attn: true additional-settings: - - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-high-conc-6p1d-dep4-dep8-mtp.yaml" + - "CONFIG_FILE=recipes/sglang/dsr1/b200-fp4/8k1k/disagg/mtp/8k1k_mtp2_throughput_5p1d.yaml" decode: num-worker: 1 tp: 8 ep: 8 dp-attn: true - # High concurrency 8p1d-dep4-dep8. 10 nodes. + # 4p1d high-throughput (dedicated c2048 tune). - spec-decoding: "mtp" - conc-list: [8192] + conc-list: [2048] prefill: - num-worker: 8 + num-worker: 4 tp: 4 ep: 4 dp-attn: true additional-settings: - - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-high-conc-8p1d-dep4-dep8-mtp.yaml" + - "CONFIG_FILE=recipes/sglang/dsr1/b200-fp4/8k1k/disagg/mtp/8k1k_mtp_4p1d_c2048.yaml" decode: num-worker: 1 tp: 8 ep: 8 dp-attn: true -# DSV4 B300 disaggregated dynamo-sglang MTP configuration. -# Each search-space entry maps one-to-one to a checked-in srt-slurm recipe under -# recipes/sglang/deepseek-v4/8k1k/. Runs on the b300 pool via -# runners/launch_b300-nv.sh. Prefill workers are TP4/DEP4 (two workers per -# 8-GPU B300 node); decode is DEP8 (one full node) except the TP4 topologies. -# EAGLE draft depth is tapered down as concurrency grows -# (3 steps / 4 draft tokens -> 2/3 at 6p1d -> 1/2 at 8p1d). -dsv4-fp4-b300-dynamo-sglang-mtp: - image: lmsysorg/sglang:nightly-dev-cu13-20260721-8905cbd4 - model: deepseek-ai/DeepSeek-V4-Pro - model-prefix: dsv4 - runner: b300 +kimik2.6-fp4-b200-dynamo-vllm: + image: vllm/vllm-openai:v0.25.1 + model: nvidia/Kimi-K2.6-NVFP4 + model-prefix: kimik2.6 + runner: cluster:b200-nscale precision: fp4 - framework: dynamo-sglang + framework: dynamo-vllm + router: { name: dynamo-router, version: "1.3.0.dev20260721" } kv-p2p-transfer: nixl multinode: true disagg: true @@ -6289,125 +3860,145 @@ dsv4-fp4-b300-dynamo-sglang-mtp: - isl: 8192 osl: 1024 search-space: - # 1p1d-tp4-tp4 (pure TP, no dp-attn). 2 nodes (1P + 1D). - - spec-decoding: "mtp" - conc-list: [1] + - conc-list: [1] prefill: num-worker: 1 - tp: 4 - ep: 1 - dp-attn: false + tp: 1 + ep: 8 + dp-attn: true additional-settings: - - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-b300-1p1d-tp4-tp4-mtp.yaml" + - "CONFIG_FILE=recipes/vllm/kimi-k2.6/b200-fp4/8k1k/disagg-b200-1p1d-dep8-tp8-c1.yaml" decode: num-worker: 1 - tp: 4 + tp: 8 ep: 1 dp-attn: false - # Scale-out: 1x DEP4 prefill + 6x TP4 decode. 4 nodes (1P + 3D). - - spec-decoding: "mtp" - conc-list: [8, 32, 64] + - conc-list: [32] prefill: num-worker: 1 - tp: 4 + tp: 1 ep: 4 dp-attn: true additional-settings: - - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-b300-1p6d-dep4-tp4-mtp.yaml" + - "CONFIG_FILE=recipes/vllm/kimi-k2.6/b200-fp4/8k1k/disagg-b200-1p8d-dep4-tp4-c32.yaml" decode: - num-worker: 6 + num-worker: 8 tp: 4 ep: 1 dp-attn: false - # 1p1d-dep4-dep8: 1x DEP4 prefill + DEP8 decode. 2 nodes (1P + 1D). - - spec-decoding: "mtp" - conc-list: [256] + - conc-list: [128] prefill: num-worker: 1 - tp: 4 + tp: 1 ep: 4 dp-attn: true additional-settings: - - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-b300-1p1d-dep4-dep8-mtp.yaml" + - "CONFIG_FILE=recipes/vllm/kimi-k2.6/b200-fp4/8k1k/disagg-b200-1p8d-dep4-tp4-c128.yaml" decode: + num-worker: 8 + tp: 4 + ep: 1 + dp-attn: false + - conc-list: [512] + prefill: num-worker: 1 - tp: 8 - ep: 8 + tp: 1 + ep: 4 dp-attn: true - # 1p2d-dep4-dep8: 1x DEP4 prefill + 2x DEP8 decode. 3 nodes (1P + 2D). - - spec-decoding: "mtp" - conc-list: [256] + additional-settings: + - "CONFIG_FILE=recipes/vllm/kimi-k2.6/b200-fp4/8k1k/disagg-b200-1p4d-dep4-tp4-c512.yaml" + decode: + num-worker: 4 + tp: 4 + ep: 1 + dp-attn: false + - conc-list: [2048] prefill: num-worker: 1 - tp: 4 - ep: 4 + tp: 1 + ep: 8 dp-attn: true additional-settings: - - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-b300-1p2d-dep4-dep8-mtp.yaml" + - "CONFIG_FILE=recipes/vllm/kimi-k2.6/b200-fp4/8k1k/disagg-b200-1p1d-dep8-dep8-c2048.yaml" decode: - num-worker: 2 - tp: 8 + num-worker: 1 + tp: 1 ep: 8 dp-attn: true - # 2p1d-dep4-dep8: 2x DEP4 prefill + DEP8 decode. 2 nodes (1P + 1D). - - spec-decoding: "mtp" - conc-list: [512] + - conc-list: [8192] prefill: num-worker: 2 - tp: 4 - ep: 4 + tp: 1 + ep: 8 dp-attn: true additional-settings: - - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-b300-2p1d-dep4-dep8-mtp.yaml" + - "CONFIG_FILE=recipes/vllm/kimi-k2.6/b200-fp4/8k1k/disagg-b200-2p1d-dep8-dep8-c8192.yaml" decode: num-worker: 1 - tp: 8 + tp: 1 ep: 8 dp-attn: true - # 4p1d-dep4-dep8: 4x DEP4 prefill + DEP8 decode. 3 nodes (2P + 1D). - - spec-decoding: "mtp" - conc-list: [1024] + +qwen3.5-fp8-gb200-dynamo-sglang: + image: lmsysorg/sglang:nightly-dev-cu13-20260608-303757cc + model: Qwen/Qwen3.5-397B-A17B-FP8 + model-prefix: qwen3.5 + runner: gb200 + precision: fp8 + framework: dynamo-sglang + router: { name: dynamo-router, version: "46520ca59afe992fb5ef61b3197b2316f8df9b2b" } + kv-p2p-transfer: mooncake + multinode: true + disagg: true + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + # 1P1D STP: TP4 prefill + TP4 decode (pure tensor parallel). 2 nodes (1+1). + - spec-decoding: "none" + conc-list: [1, 2, 4, 8, 16, 32, 64, 128] prefill: - num-worker: 4 + num-worker: 1 tp: 4 - ep: 4 - dp-attn: true + ep: 1 + dp-attn: false additional-settings: - - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-b300-4p1d-dep4-dep8-mtp.yaml" + - "CONFIG_FILE=recipes/sglang/qwen3.5/gb200-fp8/8k1k/1p1d-tp4-tp4.yaml" decode: num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - # 6p1d-dep4-dep8: 6x DEP4 prefill + DEP8 decode. 4 nodes (3P + 1D). - - spec-decoding: "mtp" - conc-list: [2048] + tp: 4 + ep: 1 + dp-attn: false + # 4P1D wide-EP: 4 prefill DEP4 + decode DEP16. 8 nodes (4+4). + - spec-decoding: "none" + conc-list: [1024] prefill: - num-worker: 6 + num-worker: 4 tp: 4 ep: 4 dp-attn: true additional-settings: - - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-b300-6p1d-dep4-dep8-mtp.yaml" + - "CONFIG_FILE=recipes/sglang/qwen3.5/gb200-fp8/8k1k/4p1d-dep4-dep16.yaml" decode: num-worker: 1 - tp: 8 - ep: 8 + tp: 16 + ep: 16 dp-attn: true - # 8p1d-dep4-dep8. 5 nodes (4P + 1D). - - spec-decoding: "mtp" - conc-list: [3072] + # 8P1D wide-EP: 8 prefill DEP4 + decode DEP16. 12 nodes (8+4). + - spec-decoding: "none" + conc-list: [2048, 4096] prefill: num-worker: 8 tp: 4 ep: 4 dp-attn: true additional-settings: - - "CONFIG_FILE=recipes/sglang/deepseek-v4/8k1k/disagg-b300-8p1d-dep4-dep8-mtp.yaml" + - "CONFIG_FILE=recipes/sglang/qwen3.5/gb200-fp8/8k1k/8p1d-dep4-dep16.yaml" decode: num-worker: 1 - tp: 8 - ep: 8 + tp: 16 + ep: 16 dp-attn: true qwen3.5-fp8-h100-sglang: @@ -7352,7 +4943,6 @@ qwen3.5-fp4-b300-sglang-agentic-mtp: - { tp: 2, ep: 2, spec-decoding: mtp, kv-offloading: none, conc-list: [1, 4, 8, 12, 16, 20, 24, 28, 32] } - { tp: 2, ep: 2, spec-decoding: mtp, kv-offloading: dram, kv-offload-backend: { name: hicache }, conc-list: [36, 44, 52] } - # Qwen3.8-Flash-Next NVFP4 AgentX on B300 via SGLang with native NEXTN MTP. # Day-zero recipe; mirrors the B200 arm. TP1 per the cookbook's verified # single-node command: 126 GiB of NVFP4 weights fit on one B300. @@ -7522,7 +5112,6 @@ qwen3.5-fp8-h200-sglang-agentic-mtp: - { tp: 8, ep: 8, spec-decoding: mtp, kv-offloading: none, conc-list: [4, 8, 12, 16] } - { tp: 8, ep: 8, spec-decoding: mtp, kv-offloading: dram, kv-offload-backend: { name: hicache }, conc-list: [4, 8, 12, 16] } - # Qwen3.8-Flash-Next FP8 AgentX on H200 via SGLang with native NEXTN MTP. # Day-zero recipe. H200 is Hopper, so this arm is FP8: NVFP4 is greyed out for # this part in the SGLang cookbook. TP4 with EP4 per the cookbook's verified @@ -7557,7 +5146,6 @@ qwen3.5-fp8-h200-sglang-agentic-hicache-mtp: search-space: - { tp: 8, ep: 1, spec-decoding: mtp, kv-offloading: dram, kv-offload-backend: { name: hicache }, conc-list: [2, 4, 8, 10, 12, 16, 20, 24] } - qwen3.5-fp8-h100-sglang-agentic-mtp: image: lmsysorg/sglang:v0.5.16-cu130 model: Qwen/Qwen3.5-397B-A17B-FP8 @@ -7573,8 +5161,6 @@ qwen3.5-fp8-h100-sglang-agentic-mtp: - { tp: 8, ep: 8, spec-decoding: mtp, kv-offloading: none, conc-list: [1, 4, 8, 12, 16] } - { tp: 8, ep: 8, spec-decoding: mtp, kv-offloading: dram, kv-offload-backend: { name: hicache }, conc-list: [4, 8, 12, 16] } - - # Qwen3.8-Flash-Next FP8 AgentX on H100 via SGLang with native NEXTN MTP. # Day-zero recipe. H100 is Hopper, so FP8: NVFP4 needs SM100 tensor cores. The # SGLang cookbook does not list H100, so this mirrors the H200 arm adjusted for @@ -9869,7 +7455,6 @@ kimik3-fp4-gb300-dynamo-vllm-agentic-mooncake-dcp8-agg: additional-settings: - "CONFIG_FILE=recipes/vllm/kimi-k3/agentic/agg-gb300-dcp8-nospec-mooncake-agentic.yaml" - # Kimi-K3 MXFP4 B200 aggregated vLLM (TP8 x PP2, 2 nodes / 16 GPUs), agentic # coding. The native MXFP4 checkpoint (2.8T total params, ~1.4TB weights) does # not fit one 8xB200 node, so TP8 shards attention/dense and PP2 splits layers. diff --git a/utils/test_gb300_power_official_contract.py b/utils/test_gb300_power_official_contract.py index 32f5e284d7..6097ba2643 100644 --- a/utils/test_gb300_power_official_contract.py +++ b/utils/test_gb300_power_official_contract.py @@ -2,15 +2,12 @@ import os import subprocess -from collections.abc import Iterator from pathlib import Path import pytest -import yaml REPO_ROOT = Path(__file__).resolve().parents[1] LAUNCHER_PATH = REPO_ROOT / "runners/launch_gb300-nv.sh" -MASTER_CONFIG_PATH = REPO_ROOT / "configs/nvidia-master.yaml" # Controlled routing inputs, deliberately independent of the deployed pins. FORK_URL = "https://example.test/power-producer.git" PRODUCER_PIN = "a" * 40 @@ -123,26 +120,6 @@ def _run_dsv4_route( marker = repo_dir / "recipes/sglang/deepseek-v4/8k1k/overlay-marker.txt" return route_log.read_text().splitlines(), workspace, repo_dir, marker - - -def _config_file_values(value: object) -> Iterator[str]: - """Yield every CONFIG_FILE value reachable below a config search space.""" - if isinstance(value, dict): - for child in value.values(): - yield from _config_file_values(child) - elif isinstance(value, list): - for child in value: - yield from _config_file_values(child) - elif isinstance(value, str) and value.startswith("CONFIG_FILE="): - yield value.removeprefix("CONFIG_FILE=").split(":", 1)[0] - - -def _workspace_recipe_path(config_file: str) -> Path: - assert config_file.startswith("recipes/") - relative = config_file.removeprefix("recipes/") - return REPO_ROOT / "benchmarks/multi_node/srt-slurm-recipes" / relative - - @pytest.mark.parametrize( ("launcher_name", "indent", "cache_directory"), [ @@ -234,24 +211,3 @@ def test_dsv4_power_route_rejects_unexpected_checkout_before_publishing_stamp(tm _run_dsv4_route(tmp_path, uses_dcgm_power=True, reported_head="b" * 40) assert not (tmp_path / "workspace/power-producer-sha.txt").exists() - - -def test_gb300_dsv4_recipe_images_match_their_master_configs(): - master = yaml.safe_load(MASTER_CONFIG_PATH.read_text()) - configs = { - key: config - for key, config in master.items() - if isinstance(config, dict) - and config.get("runner") == "gb300" - and config.get("framework") == "dynamo-sglang" - and config.get("model-prefix") == "dsv4" - } - assert configs - for key, config in configs.items(): - config_files = set(_config_file_values(config["scenarios"])) - assert config_files, key - for config_file in config_files: - recipe_path = _workspace_recipe_path(config_file) - assert recipe_path.is_file(), (key, config_file) - recipe_image = yaml.safe_load(recipe_path.read_text())["model"]["container"] - assert recipe_image == config["image"], (key, config_file)