From 578b4a6f839d76173d6afc0f79dbd5d76f26d561 Mon Sep 17 00:00:00 2001 From: Duyi-Wang Date: Tue, 8 Sep 2026 05:14:51 +0000 Subject: [PATCH 1/5] perf(dsv4-mi355x): tuned DP+MoE-TP configuration, upstream v0.5.19 image Moves dsv4-fp4-mi355x-sglang-agentic-mtp onto the stock upstream lmsysorg v0.5.19-rocm720-mi35x-20260907 image and applies the configuration a conc 64-256 sweep on MI355X converged on. Measured on that image, DP+MoE-TP: 171,732 / 246,399 / 351,252 tok/s at conc 64 / 128 / 256. v0.5.19 is what makes the image change possible: it is the first public lmsysorg tag that accepts --enable-deepseek-v4-fp4-indexer on ROCm. On v0.5.18 argument resolution raises unless is_sm100 or is_sm120, both False on a ROCm torch build, which is why this key previously needed a vendor branch image to carry the indexer. Re-running the identical configuration on a vendor image reproduces these numbers to within 2% at every concurrency, so nothing here depends on out-of-tree patches. Recipe changes, each swept with everything else held fixed: * chunked-prefill base 16384 -> 8192 at tp 8. Won conc 64 and 256 outright, lost conc 128 by 3.6%. It also avoids a cliff: 16384 with two-batch overlap at conc 256 spent 5h54m in warmup and then served 16,929 tok/s against 345,412, a 20x drop that neither knob reproduces on its own. * --enable-two-batch-overlap dropped on the DP path. Cost 15.3% throughput at conc 256 and won only at conc 128. * mem-fraction-static 0.92 on the DP path (pure TP keeps 0.86). The ladder at conc 256 measured 237,643 / 274,619 / 322,961 tok/s for 0.80 / 0.90 / 0.92 as prefix-cache chip hit went 75.75% -> 94.62%. Scoped to DP because tp 4 cannot start above ~0.89. * shared-experts fusion stays enabled under DP attention; the disable moves to the EP branch, where its +10.0% was actually measured and where it also drops the mori dispatch topk from 7 to 6. That mechanism does not exist without expert parallelism. * swa-full-tokens-ratio back to the global 0.10 on the DP path. * max-running-requests loses its 256 ceiling. It is a whole-engine budget that sglang already divides by the DP degree, so the cap silently gave conc 256 the same 32-requests-per-rank scheduler width as conc 128. The B200 and B300 siblings have no cap. * --prefill-delayer-token-usage-low-watermark 0.7 on the DP path. Upstream leaves this off entirely -- the option is Optional[float] with no fallback, so --enable-prefill-delayer alone disables the check. Worth -3.1% / +6.7% / +3.6% throughput and TTFT p50 -59% / -78% / -71% at conc 64 / 128 / 256, for a 6-18% TPOT cost. * MTP draft length by concurrency: 3 below conc 256, 1 at and above it. A depth 0/1/2/3 sweep found the optimum reverses between 128 and 256 (338,280 vs 287,516 tok/s at conc 256). num-draft-tokens is computed as num-steps + 1 rather than relying on sglang's silent correction. Search space trimmed to 7 points: tp4 conc 1/4, tp8 pure TP conc 16/32/48, tp8 DP+MoE-TP conc 128/256. Pure TP flattens out around conc 32-48 (163,203 tok/s at 48) and the DP arm carries the curve above it. --- .../agentic/dsv4_fp4_mi355x_sglang_mtp.sh | 51 ++++++++++--------- configs/amd-master.yaml | 8 +-- 2 files changed, 30 insertions(+), 29 deletions(-) diff --git a/benchmarks/single_node/agentic/dsv4_fp4_mi355x_sglang_mtp.sh b/benchmarks/single_node/agentic/dsv4_fp4_mi355x_sglang_mtp.sh index ccceb2b1d1..245f4374ab 100644 --- a/benchmarks/single_node/agentic/dsv4_fp4_mi355x_sglang_mtp.sh +++ b/benchmarks/single_node/agentic/dsv4_fp4_mi355x_sglang_mtp.sh @@ -121,11 +121,9 @@ if agentic_kv_offload_enabled; then fi # ---- Parallelism ------------------------------------------------------------ -# NOTE: the DP-attention path below is currently DORMANT (no dp-attn arms in -# amd-master.yaml for this key). It is kept so a future arm can enable it -# without rebuilding the router plumbing: sglang-router fronts the DP ranks -# with consistent hashing on the AIPerf correlation id, keeping multi-turn -# sessions on the DP rank that holds their radix/hicache prefix. +# The DP-attention path below is live: sglang-router fronts the DP ranks with +# consistent hashing on the AIPerf correlation id, keeping multi-turn sessions +# on the DP rank that holds their radix/hicache prefix. USE_SGLANG_ROUTER=false SGLANG_BACKEND_PORT="$PORT" # Small prefill chunks interleave long-context agentic prefills across @@ -133,14 +131,10 @@ SGLANG_BACKEND_PORT="$PORT" # (the conc>=16 queue-saturation / decode-stall failure mode). 8192 = 32*256, # a page-size multiple well under the dsv4 compressor kernel's uint16 token # cap; same value the multi-node DeepSeek-V4-Pro-AgentX no_dp profile uses. -if [ "$TP" -eq 8 ]; then - CHUNKED_PREFILL_SIZE=16384 -elif [ "$TP" -eq 4 ]; then - CHUNKED_PREFILL_SIZE=8192 -else - echo "Error: unsupported TP '$TP' (expected: 4 or 8)" >&2 - exit 1 -fi +case "$TP" in + 4|8) CHUNKED_PREFILL_SIZE=8192 ;; + *) echo "Error: unsupported TP '$TP' (expected: 4 or 8)" >&2; exit 1 ;; +esac MEM_FRACTION_STATIC="${MEM_FRACTION_STATIC:-0.86}" PARALLEL_ARGS=(--tensor-parallel-size "$TP") SHARED_EXPERTS_ARGS=(--enforce-shared-experts-fusion) @@ -158,8 +152,7 @@ if [ "$DP_ATTENTION" = "true" ]; then export SGLANG_DP_USE_GATHERV=1 export SGLANG_DP_USE_REDUCE_SCATTER=1 export GPU_MAX_HW_QUEUES="${GPU_MAX_HW_QUEUES_DP:-5}" - SHARED_EXPERTS_ARGS=(--disable-shared-experts-fusion) - SWA_FULL_TOKENS_RATIO="${SWA_FULL_TOKENS_RATIO_DP:-0.15}" + MEM_FRACTION_STATIC="${MEM_FRACTION_STATIC_DP:-0.92}" # Chunked prefill is a whole-engine budget, so widen it by the DP degree. CHUNKED_PREFILL_SIZE=$((CHUNKED_PREFILL_SIZE * TP)) @@ -167,23 +160,23 @@ if [ "$DP_ATTENTION" = "true" ]; then --dp "$TP" --enable-dp-attention --enable-prefill-delayer - --enable-two-batch-overlap --enable-dp-attention-local-control-broadcast --tokenizer-worker-num "$TP" --stream-interval 20 --prefill-decode-interval 10 + --prefill-delayer-token-usage-low-watermark "${DP_PREFILL_DELAYER_LOW_WATERMARK:-0.7}" ) fi if [ "$EP_SIZE" -gt 1 ]; then PARALLEL_ARGS+=(--ep-size "$EP_SIZE") + SHARED_EXPERTS_ARGS=(--disable-shared-experts-fusion) fi # AgentX concurrency counts live session trees, not individual requests. # Subagent fan-out can push instantaneous request concurrency above CONC, so # leave 2x headroom rather than clipping those bursts at the scheduler. MAX_RUNNING_REQUESTS=$((2 * CONC)) -[ "$MAX_RUNNING_REQUESTS" -gt 256 ] && MAX_RUNNING_REQUESTS=256 CUDA_GRAPH_MAX_BS=$MAX_RUNNING_REQUESTS [ "$CUDA_GRAPH_MAX_BS" -gt 128 ] && CUDA_GRAPH_MAX_BS=128 @@ -196,24 +189,32 @@ fi # ---- Speculative decoding --------------------------------------------------- # DeepSeek-V4 ships a built-in MTP head, loaded through the EAGLE spec path # with eagle-topk 1 (a single MTP chain); NOT NEXTN, whose V3/R1 loader -# crashes on the V4 architecture. Depth 3 matches the vLLM agentic sibling -# (dsv4-fp4-mi355x-vllm-agentic-mtp) and the fixed-seq-len SGLang MTP recipe. +# crashes on the V4 architecture. +if [ "$CONC" -ge 256 ]; then + DSV4_SPEC_NUM_STEPS=1 + DSV4_GOLDEN_AL=1.79 +else + DSV4_SPEC_NUM_STEPS=3 + DSV4_GOLDEN_AL=2.49 +fi + SPEC_ARGS=( --speculative-algorithm EAGLE - --speculative-num-steps 3 + --speculative-num-steps "$DSV4_SPEC_NUM_STEPS" --speculative-eagle-topk 1 - --speculative-num-draft-tokens 4 + --speculative-num-draft-tokens $((DSV4_SPEC_NUM_STEPS + 1)) ) # Throughput runs pin acceptance to the committed golden AL for this model, -# thinking mode, and draft length (golden_al_distribution/dsv4_mtp.yaml: -# thinking_on, 3 -> 2.49). Eval-only runs keep real target verification so -# accuracy stays meaningful. +# thinking mode, and draft length (golden_al_distribution/dsv4_mtp.yaml, +# thinking_on column: 3 -> 2.49, 1 -> 1.79). Eval-only runs keep real target +# verification so accuracy stays meaningful. if [ "${EVAL_ONLY:-false}" != "true" ]; then - export SGLANG_SIMULATE_ACC_LEN=2.49 + export SGLANG_SIMULATE_ACC_LEN="$DSV4_GOLDEN_AL" export SGLANG_SIMULATE_ACC_METHOD=match-expected export SGLANG_SIMULATE_ACC_TOKEN_MODE=real-draft-token fi +echo "MTP draft length: num_steps=$DSV4_SPEC_NUM_STEPS (conc $CONC), golden AL=$DSV4_GOLDEN_AL" # ---- Launch ----------------------------------------------------------------- # No --chat-template: the AgentX traces are tool-heavy, and diff --git a/configs/amd-master.yaml b/configs/amd-master.yaml index 2510dc3b8d..ed665fa835 100644 --- a/configs/amd-master.yaml +++ b/configs/amd-master.yaml @@ -1742,7 +1742,7 @@ glm5.2-fp4-mi355x-atom-agentic-mtp: dsv4-fp4-mi355x-sglang-agentic-mtp: - image: lmsysorg/sglang-rocm:v0.5.18-rocm720-mi35x-20260902 + image: lmsysorg/sglang-rocm:v0.5.19-rocm720-mi35x-20260907 model: deepseek-ai/DeepSeek-V4-Pro model-prefix: dsv4 runner: cluster:mi355x-amds @@ -1753,7 +1753,7 @@ dsv4-fp4-mi355x-sglang-agentic-mtp: agentic-coding: - dram-utilization: 0.80 search-space: - - { tp: 4, ep: 1, dp-attn: false, kv-offloading: none, conc-list: [1, 4, 8], spec-decoding: mtp } - - { tp: 8, ep: 1, dp-attn: false, kv-offloading: dram, kv-offload-backend: { name: hicache }, conc-list: [32, 48], spec-decoding: mtp } + - { tp: 4, ep: 1, dp-attn: false, kv-offloading: none, conc-list: [1, 4], spec-decoding: mtp } - { tp: 8, ep: 1, dp-attn: false, kv-offloading: none, conc-list: [16], spec-decoding: mtp } - - { tp: 8, ep: 1, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: hicache }, conc-list: [64, 96, 128, 160], spec-decoding: mtp } + - { tp: 8, ep: 1, dp-attn: false, kv-offloading: dram, kv-offload-backend: { name: hicache }, conc-list: [32, 48], spec-decoding: mtp } + - { tp: 8, ep: 1, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: hicache }, conc-list: [128, 256], spec-decoding: mtp } From 4a642d0ee07f2513702b7dce14b215dd866094cc Mon Sep 17 00:00:00 2001 From: thomawan Date: Tue, 8 Sep 2026 14:19:15 +0800 Subject: [PATCH 2/5] Revert tp8-only to use 16k prefill chunk size --- .../agentic/dsv4_fp4_mi355x_sglang_mtp.sh | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/benchmarks/single_node/agentic/dsv4_fp4_mi355x_sglang_mtp.sh b/benchmarks/single_node/agentic/dsv4_fp4_mi355x_sglang_mtp.sh index 245f4374ab..b701fd0819 100644 --- a/benchmarks/single_node/agentic/dsv4_fp4_mi355x_sglang_mtp.sh +++ b/benchmarks/single_node/agentic/dsv4_fp4_mi355x_sglang_mtp.sh @@ -126,15 +126,20 @@ fi # on the DP rank that holds their radix/hicache prefix. USE_SGLANG_ROUTER=false SGLANG_BACKEND_PORT="$PORT" -# Small prefill chunks interleave long-context agentic prefills across -# requests instead of letting one ~100K-token prefill monopolize the engine -# (the conc>=16 queue-saturation / decode-stall failure mode). 8192 = 32*256, -# a page-size multiple well under the dsv4 compressor kernel's uint16 token -# cap; same value the multi-node DeepSeek-V4-Pro-AgentX no_dp profile uses. +# Small prefill chunks interleave long-context agentic prefills. The flag is +# engine-wide and DP divides it by dp_size (=TP), so DP uses 8192*TP to keep +# 8192 per rank. TP-only: 8192 at TP4, 16384 at TP8. case "$TP" in - 4|8) CHUNKED_PREFILL_SIZE=8192 ;; + 4|8) ;; *) echo "Error: unsupported TP '$TP' (expected: 4 or 8)" >&2; exit 1 ;; esac +if [ "$DP_ATTENTION" = "true" ]; then + CHUNKED_PREFILL_SIZE=$((8192 * TP)) +elif [ "$TP" -eq 8 ]; then + CHUNKED_PREFILL_SIZE=16384 +else + CHUNKED_PREFILL_SIZE=8192 +fi MEM_FRACTION_STATIC="${MEM_FRACTION_STATIC:-0.86}" PARALLEL_ARGS=(--tensor-parallel-size "$TP") SHARED_EXPERTS_ARGS=(--enforce-shared-experts-fusion) @@ -154,8 +159,6 @@ if [ "$DP_ATTENTION" = "true" ]; then export GPU_MAX_HW_QUEUES="${GPU_MAX_HW_QUEUES_DP:-5}" MEM_FRACTION_STATIC="${MEM_FRACTION_STATIC_DP:-0.92}" - # Chunked prefill is a whole-engine budget, so widen it by the DP degree. - CHUNKED_PREFILL_SIZE=$((CHUNKED_PREFILL_SIZE * TP)) PARALLEL_ARGS+=( --dp "$TP" --enable-dp-attention From 187004fd9b470ef63f85fc14369ca546c716bd33 Mon Sep 17 00:00:00 2001 From: thomawan Date: Tue, 8 Sep 2026 14:21:13 +0800 Subject: [PATCH 3/5] Turn to tp8 on low conc --- configs/amd-master.yaml | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/configs/amd-master.yaml b/configs/amd-master.yaml index ed665fa835..7c8d55796f 100644 --- a/configs/amd-master.yaml +++ b/configs/amd-master.yaml @@ -1753,7 +1753,6 @@ dsv4-fp4-mi355x-sglang-agentic-mtp: agentic-coding: - dram-utilization: 0.80 search-space: - - { tp: 4, ep: 1, dp-attn: false, kv-offloading: none, conc-list: [1, 4], spec-decoding: mtp } - - { tp: 8, ep: 1, dp-attn: false, kv-offloading: none, conc-list: [16], spec-decoding: mtp } + - { tp: 8, ep: 1, dp-attn: false, kv-offloading: none, conc-list: [1, 4, 16], spec-decoding: mtp } - { tp: 8, ep: 1, dp-attn: false, kv-offloading: dram, kv-offload-backend: { name: hicache }, conc-list: [32, 48], spec-decoding: mtp } - { tp: 8, ep: 1, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: hicache }, conc-list: [128, 256], spec-decoding: mtp } From 0668fd0e59df98c7e5cee554eb2d36667efc1291 Mon Sep 17 00:00:00 2001 From: thomawan Date: Tue, 8 Sep 2026 14:26:32 +0800 Subject: [PATCH 4/5] Update perf changelog --- perf-changelog.yaml | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/perf-changelog.yaml b/perf-changelog.yaml index e0ef448195..f792ce89fe 100644 --- a/perf-changelog.yaml +++ b/perf-changelog.yaml @@ -6947,3 +6947,16 @@ - "Update vLLM ROCm image from vllm/vllm-openai-rocm:v0.27.1 (v0.27.1 release) to vllm/vllm-openai-rocm:nightly-d9105ea8001e0a6d77a96327d17515bb5791fb36 (2026-09-07 upstream ROCm nightly, digest sha256:74d4a95f3ae672ecddf9acb7296917def82d9eca51687fa2862ae72b03ff1907, tag commit vllm-project/vllm@d9105ea8; Docker Hub last pushed 2026-09-07T05:26:48Z). benchmarks/single_node/agentic/minimaxm3_fp8_mi300x_mtp.sh is unchanged: TRITON_ATTN attention, fp8 KV, block-size 128, EAGLE3 speculative decoding with the Inferact MiniMax-M3 EAGLE3-GQA draft and the committed golden synthetic acceptance length, minimax_m3 tool-call and reasoning parsers; the search space is unchanged. The upstream commit-pinned ROCm nightly is the same vllm commit the B200 MiniMax-M3 AgentX recipe moved to in #2860. Note that vllm-openai-rocm commit-nightly tags have expired from Docker Hub within days in the past; node squash caches keep merged configs running, but a re-pin to a durable tag may be needed later." - "Add --compilation-config cudagraph_mode=FULL_DECODE_ONLY to the serve command. The upstream nightly does not torch-compile MiniMaxM3SparseForConditionalGeneration, so with VLLM_USE_BREAKABLE_CUDAGRAPH=0 the default FULL_AND_PIECEWISE mode aborts at engine init with piecewise CUDA graphs unavailable (first sweep, run 34174124043, eval cell); full decode-only graphs are what the MI355X MiniMax-M3 sibling runs on its nightly (#2825)." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2873 + +- config-keys: + - dsv4-fp4-mi355x-sglang-agentic-mtp + scenario-type: + - agentic-coding + description: + - "Bump image to lmsysorg/sglang-rocm:v0.5.19-rocm720-mi35x-20260907." + - "Keep TP-only chunked-prefill at 8192 for TP4 and 16384 for TP8. The DP-attention path uses an 8192 per-rank budget (engine-wide 8192*TP, 65536 at TP8) instead of scaling the TP8 16384 base." + - "DP-attention serving: drop --enable-two-batch-overlap; set mem-fraction-static to 0.92 (pure TP stays 0.86); keep --enforce-shared-experts-fusion and move --disable-shared-experts-fusion to the EP branch only; keep swa-full-tokens-ratio at 0.10; add --prefill-delayer-token-usage-low-watermark 0.7." + - "Remove the 256 cap on max-running-requests." + - "MTP draft length by concurrency: speculative-num-steps 3 (golden AL 2.49) below conc 256, and 1 (golden AL 1.79) at and above it." + - "Trim the search space to TP8 no-offload conc [1, 4, 16], TP8 hicache conc [32, 48], and TP8 DP-attention hicache conc [128, 256]." + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2885 From a62e35f99fc7e87f726e6dc805e1a8419d357293 Mon Sep 17 00:00:00 2001 From: Thomas Wang <1am9trash@gmail.com> Date: Wed, 9 Sep 2026 23:57:31 +0800 Subject: [PATCH 5/5] Fix format --- perf-changelog.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/perf-changelog.yaml b/perf-changelog.yaml index 1c04558d1e..f0c124730d 100644 --- a/perf-changelog.yaml +++ b/perf-changelog.yaml @@ -7009,4 +7009,4 @@ - "Remove the 256 cap on max-running-requests." - "MTP draft length by concurrency: speculative-num-steps 3 (golden AL 2.49) below conc 256, and 1 (golden AL 1.79) at and above it." - "Trim the search space to TP8 no-offload conc [1, 4, 16], TP8 hicache conc [32, 48], and TP8 DP-attention hicache conc [128, 256]." - pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2885 \ No newline at end of file + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2885