Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions MODELS.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ Speculative-decoding A/B retirements apply to each pair below. The spec-decode a

Rationale: `dsv4` carries the largest single-turn footprint in the repository. 45 active config keys use the 8k1k scenario, 32 in `configs/nvidia-master.yaml` and 13 in `configs/amd-master.yaml`, spanning H200, B200, B300, GB200, GB300, MI300X, MI325X, and MI355X across vLLM, SGLang, TensorRT-LLM, ATOM, Dynamo, and llm-d. That is a large share of every full sweep. AgentX trace replay is the scenario AI labs and the ML community ask about, and DeepSeek-V4-Pro's 19 agentic config keys are the part of `dsv4` that feeds the published North Star Pareto frontier. Retiring the fixed-sequence-length arm frees cluster hours for AgentX and for new frontier models such as Qwen3.8-Flash-Next without reducing what we publish for this model. Single-turn 8k1k stays active for the other models that still list it.

**Status: not yet enacted.** All 45 8k1k config keys still run. On enactment they are removed from the active master configs and archived under [`configs/deprecated/`](configs/deprecated/), with their benchmark scripts moved to the sibling `deprecated/` directories, matching how [#2493](https://github.com/SemiAnalysisAI/InferenceX/pull/2493) and [#2527](https://github.com/SemiAnalysisAI/InferenceX/pull/2527) were carried out. The SPEED-Bench acceptance-length scripts for `dsv4` are intentionally kept. Speedbench is driven by `speedbench-al.yml`, not the master configs.
**Enacted on 2026-09-09** in [#2921](https://github.com/SemiAnalysisAI/InferenceX/pull/2921): 46 `dsv4` 8k1k config keys were removed from the active master configs and archived under [`configs/deprecated/`](configs/deprecated/) as `nvidia-dsv4-8k1k-master.yaml` (33) and `amd-dsv4-8k1k-master.yaml` (13), and their 28 benchmark scripts were moved to the sibling `deprecated/` directories (25 under `benchmarks/single_node/fixed_seq_len/`, 3 under `benchmarks/multi_node/`), matching how [#2493](https://github.com/SemiAnalysisAI/InferenceX/pull/2493) and [#2527](https://github.com/SemiAnalysisAI/InferenceX/pull/2527) were carried out. The count is 46 rather than the 45 quoted above because `dsv4-fp4-b200-dynamo-sglang` landed after this notice was written. The 19 agentic-coding keys are untouched: `dsv4` continues to run and publish with agentic coding as its only scenario. The SPEED-Bench acceptance-length scripts for `dsv4` are intentionally kept. Speedbench is driven by `speedbench-al.yml`, not the master configs. The srt-slurm and llm-d recipe YAMLs referenced by the archived multi-node keys stay in place as inert reference data, as #2493 and #2527 left theirs.

## Scenarios

Expand Down Expand Up @@ -120,7 +120,7 @@ The table also records both the agreed plan-of-record (PoR) draft-model mapping

| Model | Primary native/upstream engines | Agreed draft model(s) (PoR) | Proposed draft model(s) pending partner alignment | Additional engines |
|---|---|---|---|---|
| DeepSeek-V4-Pro 1.6T (`dsv4`) | native/upstream vLLM engine and native/upstream SGLang engine | native MTP (Single-turn 8k1k); native DSpark heads `deepseek-ai/DeepSeek-V4-Pro-0813` (Agentic coding only, under the same synthetic-acceptance methodology as Kimi-K3's DSpark PoR) | None | Additional non-vLLM/SGLang engines under the ordering guideline and exceptions above |
| DeepSeek-V4-Pro 1.6T (`dsv4`) | native/upstream vLLM engine and native/upstream SGLang engine | native MTP (Agentic coding MTP arms; formerly also Single-turn 8k1k, retired 2026-09-09); native DSpark heads `deepseek-ai/DeepSeek-V4-Pro-0813` (Agentic coding only, under the same synthetic-acceptance methodology as Kimi-K3's DSpark PoR) | None | Additional non-vLLM/SGLang engines under the ordering guideline and exceptions above |
| Kimi-K3 (`kimik3`) | native/upstream vLLM engine | `Inferact/Kimi-K3-DSpark` | None | Additional non-vLLM/SGLang engines under the ordering guideline and exceptions above |
| MiniMax-M3 (`minimaxm3`) | native/upstream vLLM engine | `Inferact/MiniMax-M3-EAGLE3` and/or `Inferact/MiniMax-M3-EAGLE3-GQA` | None | Additional non-vLLM/SGLang engines under the ordering guideline and exceptions above |
| GLM-5.2 (`glm5.2`) | native/upstream SGLang engine | native MTP | None | Additional non-vLLM/SGLang engines under the ordering guideline and exceptions above |
Expand Down Expand Up @@ -154,7 +154,7 @@ Other offloading tiers, including NVMe KV cache offloading, are outside the init
| Kimi-K3 | `kimik3` | 2026-07-27 ([#2391](https://github.com/SemiAnalysisAI/InferenceX/pull/2391)) | Agentic coding (DSpark only) | Agentic coding non-DSpark arm (deprecated from day 0) |
| GLM-5.2 | `glm5.2` | 2026-07-18 ([#2268](https://github.com/SemiAnalysisAI/InferenceX/pull/2268)) | Agentic coding (the non-MTP arm still runs while the MTP-only transition remains pending, as explained in the Deprecation Notice) | |
| MiniMax-M3 | `minimaxm3` | 2026-06-12 ([#1724](https://github.com/SemiAnalysisAI/InferenceX/pull/1724)) | Agentic coding | Single-turn 1k1k, Single-turn 8k1k (removed 2026-08-04, [#2493](https://github.com/SemiAnalysisAI/InferenceX/pull/2493)) |
| DeepSeek-V4-Pro | `dsv4` | 2026-04-24 ([#1130](https://github.com/SemiAnalysisAI/InferenceX/pull/1130)) | Single-turn 8k1k (last day 2026-09-08, see [Deprecation Notice](#deprecation-notice)), Agentic coding (the non-MTP arm still runs while the MTP-only transition remains pending, as explained in the Deprecation Notice) | Single-turn 1k1k |
| DeepSeek-V4-Pro | `dsv4` | 2026-04-24 ([#1130](https://github.com/SemiAnalysisAI/InferenceX/pull/1130)) | Agentic coding (the non-MTP arm still runs while the MTP-only transition remains pending, as explained in the Deprecation Notice) | Single-turn 1k1k, Single-turn 8k1k (removed 2026-09-09, [#2921](https://github.com/SemiAnalysisAI/InferenceX/pull/2921)) |
| GLM-5 / GLM-5.1 | `glm5`, `glm5.1` | 2026-03-06 ([#762](https://github.com/SemiAnalysisAI/InferenceX/pull/762)), with GLM-5.1 added 2026-04-21 ([#1098](https://github.com/SemiAnalysisAI/InferenceX/pull/1098)) | None (retired 2026-07-18, [#2276](https://github.com/SemiAnalysisAI/InferenceX/pull/2276)) | Single-turn 1k1k, Single-turn 1k8k (GLM-5 only), Single-turn 8k1k |
| MiniMax-M2.5/2.7 | `minimaxm2.5` | 2026-02-18 ([#755](https://github.com/SemiAnalysisAI/InferenceX/pull/755)) | None (retired 2026-06-20, [#1874](https://github.com/SemiAnalysisAI/InferenceX/pull/1874)) | Single-turn 1k1k, Single-turn 1k8k, Single-turn 8k1k |
| Kimi-K2.5/2.6/2.7-Code | `kimik2.5` | 2026-02-17 ([#734](https://github.com/SemiAnalysisAI/InferenceX/pull/734)) | None (fully retired 2026-08-07, [#2527](https://github.com/SemiAnalysisAI/InferenceX/pull/2527)) | Single-turn 1k1k, Single-turn 1k8k, Agentic coding (removed 2026-08-04, [#2493](https://github.com/SemiAnalysisAI/InferenceX/pull/2493)), Single-turn 8k1k (removed 2026-08-07, [#2527](https://github.com/SemiAnalysisAI/InferenceX/pull/2527)) |
Expand Down
6 changes: 3 additions & 3 deletions MODELS_zh.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ InferenceX-e2e 运行在数量固定且有限的 GPU 资源池上,并由一支

原因:`dsv4` 是本仓库中单轮场景占用最大的模型。当前有 45 个启用的配置项使用 8k1k 场景(`configs/nvidia-master.yaml` 32 个,`configs/amd-master.yaml` 13 个),覆盖 H200、B200、B300、GB200、GB300、MI300X、MI325X 与 MI355X,涉及 vLLM、SGLang、TensorRT-LLM、ATOM、Dynamo 与 llm-d,在每一轮完整 sweep 中占比可观。AgentX 轨迹回放才是 AI 实验室与 ML 社区真正关注的场景,而 DeepSeek-V4-Pro 的 19 个智能体编码配置项正是 `dsv4` 中支撑已发布北极星(North Star)帕累托前沿的部分。下线固定序列长度分支可为 AgentX 以及 Qwen3.8-Flash-Next 等新前沿模型腾出集群机时,同时不减少该模型对外发布的内容。对于仍列有该场景的其他模型,单轮 8k1k 保持启用。

**状态:尚未执行。** 全部 45 个 8k1k 配置项仍在运行。执行时将从启用的主配置中移除并归档至 [`configs/deprecated/`](configs/deprecated/),对应基准测试脚本移入同级 `deprecated/` 目录与 [#2493](https://github.com/SemiAnalysisAI/InferenceX/pull/2493) 和 [#2527](https://github.com/SemiAnalysisAI/InferenceX/pull/2527) 的做法一致。`dsv4` 的 SPEED-Bench 接受长度脚本予以保留。Speedbench 由 `speedbench-al.yml` 驱动,不经过主配置。
**已于 2026-09-09 执行**([#2921](https://github.com/SemiAnalysisAI/InferenceX/pull/2921)):46 个 `dsv4` 8k1k 配置项已从启用的主配置中移除并归档至 [`configs/deprecated/`](configs/deprecated/),即 `nvidia-dsv4-8k1k-master.yaml`(33 个)与 `amd-dsv4-8k1k-master.yaml`(13 个);对应的 28 个基准测试脚本移入同级 `deprecated/` 目录(`benchmarks/single_node/fixed_seq_len/` 下 25 个,`benchmarks/multi_node/` 下 3 个),与 [#2493](https://github.com/SemiAnalysisAI/InferenceX/pull/2493) 和 [#2527](https://github.com/SemiAnalysisAI/InferenceX/pull/2527) 的做法一致。数量为 46 而非上文所述的 45,是因为 `dsv4-fp4-b200-dynamo-sglang` 在本公告发布后才合入。19 个智能体编码配置项未做改动:`dsv4` 以智能体编码为唯一场景继续运行与发布。`dsv4` 的 SPEED-Bench 接受长度脚本予以保留。Speedbench 由 `speedbench-al.yml` 驱动,不经过主配置。已归档多节点配置项所引用的 srt-slurm 与 llm-d 配方 YAML 作为惰性参考数据原地保留,与 #2493 和 #2527 的处理一致

## 场景

Expand Down Expand Up @@ -120,7 +120,7 @@ InferenceX 支持 SGLang 和 vLLM 双方的维护者,并响应 AI 实验室和

| 模型 | 首选原生/上游引擎 | 已达成一致的草稿模型(PoR) | 待合作伙伴对齐的草稿模型提案 | 其他引擎 |
|---|---|---|---|---|
| DeepSeek-V4-Pro 1.6T(`dsv4`) | 原生/上游 vLLM 引擎和原生/上游 SGLang 引擎 | 原生 MTP(单轮 8k1k);`deepseek-ai/DeepSeek-V4-Pro-0813`(仅用于智能体编码,遵循与 Kimi-K3 DSpark PoR 相同的合成接受方法) | 无 | 按照上述提交顺序指南及例外处理的其他非 vLLM/SGLang 引擎 |
| DeepSeek-V4-Pro 1.6T(`dsv4`) | 原生/上游 vLLM 引擎和原生/上游 SGLang 引擎 | 原生 MTP(智能体编码 MTP 分支;此前亦用于单轮 8k1k,已于 2026-09-09 退役);`deepseek-ai/DeepSeek-V4-Pro-0813`(仅用于智能体编码,遵循与 Kimi-K3 DSpark PoR 相同的合成接受方法) | 无 | 按照上述提交顺序指南及例外处理的其他非 vLLM/SGLang 引擎 |
| Kimi-K3(`kimik3`) | 原生/上游 vLLM 引擎 | `Inferact/Kimi-K3-DSpark` | 无 | 按照上述提交顺序指南及例外处理的其他非 vLLM/SGLang 引擎 |
| MiniMax-M3(`minimaxm3`) | 原生/上游 vLLM 引擎 | `Inferact/MiniMax-M3-EAGLE3` 和/或 `Inferact/MiniMax-M3-EAGLE3-GQA` | 无 | 按照上述提交顺序指南及例外处理的其他非 vLLM/SGLang 引擎 |
| GLM-5.2(`glm5.2`) | 原生/上游 SGLang 引擎 | 原生 MTP | 无 | 按照上述提交顺序指南及例外处理的其他非 vLLM/SGLang 引擎 |
Expand Down Expand Up @@ -154,7 +154,7 @@ InferenceX 支持 SGLang 和 vLLM 双方的维护者,并响应 AI 实验室和
| Kimi-K3 | `kimik3` | 2026-07-27 ([#2391](https://github.com/SemiAnalysisAI/InferenceX/pull/2391)) | 智能体编码(仅 DSpark) | 智能体编码非 DSpark 分支(自第 0 天起弃用) |
| GLM-5.2 | `glm5.2` | 2026-07-18([#2268](https://github.com/SemiAnalysisAI/InferenceX/pull/2268)) | 智能体编码(非 MTP 分支仍在运行,「仅 MTP」转换仍待执行,见弃用公告) | |
| MiniMax-M3 | `minimaxm3` | 2026-06-12([#1724](https://github.com/SemiAnalysisAI/InferenceX/pull/1724)) | 智能体编码 | 单轮 1k1k、单轮 8k1k(2026-08-04 移除,[#2493](https://github.com/SemiAnalysisAI/InferenceX/pull/2493)) |
| DeepSeek-V4-Pro | `dsv4` | 2026-04-24([#1130](https://github.com/SemiAnalysisAI/InferenceX/pull/1130)) | 单轮 8k1k、智能体编码(非 MTP 分支仍在运行,「仅 MTP」转换仍待执行,见弃用公告) | 单轮 1k1k |
| DeepSeek-V4-Pro | `dsv4` | 2026-04-24([#1130](https://github.com/SemiAnalysisAI/InferenceX/pull/1130)) | 智能体编码(非 MTP 分支仍在运行,「仅 MTP」转换仍待执行,见弃用公告) | 单轮 1k1k、单轮 8k1k(已于 2026-09-09 移除,[#2921](https://github.com/SemiAnalysisAI/InferenceX/pull/2921)) |
| GLM-5 / GLM-5.1 | `glm5`、`glm5.1` | 2026-03-06([#762](https://github.com/SemiAnalysisAI/InferenceX/pull/762)),GLM-5.1 于 2026-04-21 加入([#1098](https://github.com/SemiAnalysisAI/InferenceX/pull/1098)) | 无(2026-07-18 退役,[#2276](https://github.com/SemiAnalysisAI/InferenceX/pull/2276)) | 单轮 1k1k、单轮 1k8k(仅 GLM-5)、单轮 8k1k |
| MiniMax-M2.5/2.7 | `minimaxm2.5` | 2026-02-18([#755](https://github.com/SemiAnalysisAI/InferenceX/pull/755)) | 无(2026-06-20 退役,[#1874](https://github.com/SemiAnalysisAI/InferenceX/pull/1874)) | 单轮 1k1k、单轮 1k8k、单轮 8k1k |
| Kimi-K2.5/2.6/2.7-Code | `kimik2.5` | 2026-02-17([#734](https://github.com/SemiAnalysisAI/InferenceX/pull/734)) | 无(2026-08-07 完全退役,[#2527](https://github.com/SemiAnalysisAI/InferenceX/pull/2527)) | 单轮 1k1k、单轮 1k8k、智能体编码(2026-08-04 移除,[#2493](https://github.com/SemiAnalysisAI/InferenceX/pull/2493))、单轮 8k1k(2026-08-07 移除,[#2527](https://github.com/SemiAnalysisAI/InferenceX/pull/2527)) |
Expand Down
2 changes: 1 addition & 1 deletion benchmarks/single_node/agentic/dsv4_fp4_mi355x_vllm_mtp.sh
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ set -x
#
# Serving flags follow the validated MI355X recipe from
# https://recipes.vllm.ai/deepseek-ai/DeepSeek-V4-Pro?hardware=mi355x
# https://github.com/SemiAnalysisAI/InferenceX/blob/main/benchmarks/single_node/fixed_seq_len/dsv4_fp4_mi355x_vllm.sh
# https://github.com/SemiAnalysisAI/InferenceX/blob/main/benchmarks/single_node/fixed_seq_len/deprecated/dsv4_fp4_mi355x_vllm.sh
# Image is configured in amd-master.yaml.
#
# Required env vars:
Expand Down
Loading