Skip to content

perf(sglang): use DSpark6 for B300 DSV4 AgentX / B300 DSV4 AgentX 使用 DSpark6 #2918

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
nvpohanh wants to merge 4 commits into
base: main
Choose a base branch
from
+66 −28
Open

perf(sglang): use DSpark6 for B300 DSV4 AgentX / B300 DSV4 AgentX 使用 DSpark6 #2918

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
2f17ca0
perf(sglang): use DSpark6 for B300 DSV4 AgentX
nvpohanh Sep 9, 2026
b98a821
chore: link perf changelog to #2918
nvpohanh Sep 9, 2026
ef33a73
Merge remote-tracking branch 'origin/main' into dsv4-fp4-b300-sglang-…
nvpohanh Sep 10, 2026
3f52b08
Merge remote-tracking branch 'origin/main' into dsv4-fp4-b300-sglang-…
nvpohanh Sep 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 12 additions & 16 deletions benchmarks/single_node/agentic/dsv4_fp4_b300_sglang_mtp.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
set -eo pipefail
set -x

# Agentic trace replay for DeepSeek-V4-Pro FP4 on B300 with native EAGLE MTP.
# Agentic trace replay for DeepSeek-V4-Pro-0813 FP4 on B300 with DSpark K=6.
# Throughput uses the committed golden synthetic AL; eval retains real target
# verification.
#
Expand Down Expand Up @@ -122,12 +122,14 @@ if [ "$DP_ATTENTION" = "true" ]; then
--enable-prefill-delayer
--prefill-decode-interval 20
--enable-dp-attention
--enable-dp-lm-head
--enable-dp-attention-local-control-broadcast
--incremental-streaming-output
--stream-interval 20
--dist-init-addr "127.0.0.1:$((PORT + 2000))"
--ep-size "$EP_SIZE"
--moe-a2a-backend megamoe
--enable-w4a4-mxfp4-megamoe
--enable-deepseek-v4-fp4-indexer
--disable-flashinfer-autotune
)
Expand All @@ -147,17 +149,16 @@ if [ "$DP_ATTENTION" = "true" ]; then
MEM_FRACTION_STATIC=0.86
elif [ "$CONC" -ge 384 ]; then
MEM_FRACTION_STATIC=0.88
elif [ "$CONC" -ge 256 ]; then
MEM_FRACTION_STATIC=0.9
elif [ "$CONC" -ge 32 ]; then
MEM_FRACTION_STATIC=0.90
fi
else
# DEP4 is squeezed from both sides: weights occupy ~90% of each GPU when
# the model is sharded over half the node, so the engine refuses to start
# below ~0.902 (no KV left), while megamoe still needs its ~7 GB
# workspace above the static budget. 0.95 leaves only ~11 GB free -- the
# same margin that OOM'd a rank at DEP8 conc 256 -- so use 0.93, which
# gives the ~16 GB that DEP8 conc 128 runs with at the same per-rank load
# (max-running-requests/dp = 32 in both cases).
# leaves approximately 16 GB for the MegaMoE workspace.
MEM_FRACTION_STATIC=0.93
fi
# --chunked-prefill-size is a GLOBAL budget: server_args.py divides it by
Expand Down Expand Up @@ -194,7 +195,7 @@ CUDA_GRAPH_MAX_BS=$((CONC * 4))
CUDA_GRAPH_ARGS=(--cuda-graph-max-bs "$CUDA_GRAPH_MAX_BS")
SWA_FULL_TOKENS_RATIO=0.1
if [ "$DP_ATTENTION" = "true" ]; then
# Decode graphs must cover the padded MTP batch across all DP ranks, which
# Decode graphs must cover the padded speculative batch across all DP ranks, which
# exceeds CONC; capping at 64 would fall back to eager decode.
CUDA_GRAPH_ARGS=(--cuda-graph-max-bs-decode 544)
SWA_FULL_TOKENS_RATIO=0.075
Expand All @@ -219,18 +220,12 @@ export SGLANG_OPT_USE_JIT_INDEXER_METADATA=1
export SGLANG_OPT_USE_TOPK_V2=1
export SGLANG_OPT_USE_CUSTOM_ALL_REDUCE_V2=1
if [ "$DP_ATTENTION" = "true" ]; then
# MegaMoE's FP4/MXF4 activation path is opt-in -- both flags default False,
# so --moe-a2a-backend megamoe alone runs a different kernel than the one
# measured. DG_USE_FP4_ACTS / DG_USE_MXF4_KIND are forwarded to DeepGEMM
# automatically from these two.
export SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_FP4_ACTS=1
export SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_MXF4_KIND=1
# Must cover the per-rank prefill budget (8192) or startup raises; the
# extra 128 is headroom over the exact-fit boundary.
export SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=8320
fi
if [ "${EVAL_ONLY}" != "true" ]; then
export SGLANG_SIMULATE_ACC_LEN=2.49
export SGLANG_SIMULATE_ACC_LEN=3.77
export SGLANG_SIMULATE_ACC_METHOD=match-expected
export SGLANG_SIMULATE_ACC_TOKEN_MODE=real-draft-token
fi
Expand Down Expand Up @@ -261,10 +256,11 @@ SGLANG_CMD=(
--reasoning-parser deepseek-v4
--chat-template "$SCRIPT_DIR/../chat_templates/deepseek_v4_thinking.jinja"
--watchdog-timeout 1800
--speculative-algorithm EAGLE
--speculative-num-steps 3
--speculative-algorithm DSPARK
--speculative-dspark-block-size 6
--speculative-num-steps 1
--speculative-eagle-topk 1
--speculative-num-draft-tokens 4
--speculative-num-draft-tokens 7
"${MODEL_ARGS[@]}"
"${METRICS_ARGS[@]}"
"${CACHE_ARGS[@]}"
Expand Down
15 changes: 8 additions & 7 deletions configs/nvidia-master.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1155,21 +1155,21 @@ dsv4-fp4-b300-sglang:
- { tp: 8, ep: 8, dp-attn: true, conc-start: 4096, conc-end: 4096 }

dsv4-fp4-b300-sglang-agentic-hicache-mtp:
image: lmsysorg/sglang:nightly-dev-cu13-20260827-20621aa1
model: deepseek-ai/DeepSeek-V4-Pro
image: lmsysorg/sglang:nightly-dev-20260901-07c8f729
model: deepseek-ai/DeepSeek-V4-Pro-0813
model-prefix: dsv4
runner: cluster:b300-nv
runner: cluster:b300-dsxe
precision: fp4
framework: sglang
multinode: false
scenarios:
agentic-coding:
- dram-utilization: 0.95
search-space:
- { tp: 8, kv-offloading: none, spec-decoding: mtp, conc-list: [1, 4, 8, 16, 32] }
- { tp: 8, ep: 8, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: hicache }, spec-decoding: mtp, conc-list: [32, 256, 384, 512, 576], router: { name: sglang-router, version: "0.3.2" } }
- { tp: 8, kv-offloading: none, spec-decoding: draft_model, conc-list: [1, 4, 8, 16, 32] }
- { tp: 8, ep: 8, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: hicache }, spec-decoding: draft_model, conc-list: [32, 64, 128, 256, 384, 512, 576], router: { name: sglang-router, version: "0.3.2" } }
Comment on lines +1169 to +1170

@claude claude Bot Sep 9, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 (optional) Changing spec-decoding from mtp to draft_model for this recipe silently drops the CI scheduling priority boost: utils/ci_priority.py's calculate_priority looks up adjustments.spec-decoding[entry['spec-decoding']] against configs/ci-priority.yaml, which only defines mtp/eagle/eagle3 (each +0.75), not draft_model, so these two search-space rows now score +0 instead of +0.75 and queue behind other jobs with no functional reason. Pre-existing gap (3 other draft_model entries already hit it before this PR) but this diff doubles the affected population by converting 2 more rows from mtp to draft_model. Fix: add a draft_model key (matching mtp's weight, or the intended weight for non-MTP speculative decoding) to adjustments.spec-decoding in configs/ci-priority.yaml.

Extended reasoning...

utils/ci_priority.py:161-164 computes score += adjustments.get('spec-decoding', {}).get(str(entry.get('spec-decoding', '')), 0). configs/ci-priority.yaml:19-22 only has mtp/eagle/eagle3 mapped to 0.75; there is no 'draft_model' key anywhere in that file (grep confirms 0 hits, even after this PR). Before this diff, dsv4-fp4-b300-sglang-agentic-hicache-mtp's two search-space entries had spec-decoding: mtp and got +0.75 priority in run-sweep.yml's CI queue ordering; after this diff they read spec-decoding: draft_model and get +0, so the resulting benchmark jobs sort lower in the priority queue purely from this label swap, unrelated to the DSpark change's actual urgency. Verified: base commit already had 3 draft_model entries in nvidia-master.yaml (lines showing they predate this PR) that hit the same gap, so the mechanism is pre-existing, but this PR converts 2 more rows onto the ungapped path, widening exposure.

Verification: nit. The mechanism is real and reachable. configs/ci-priority.yaml only maps spec-decoding mtp/eagle/eagle3 to 0.75 (no draft_model key anywhere). utils/ci_priority.py:161-163 uses an exact-match lookup with default 0: score += _decimal(adjustments.get("spec-decoding", {}).get(str(entry.get("spec-decoding", "")), 0)) — and unlike framework-prefix there is no prefix fallback for… | nit.…


# DeepSeek-V4-Pro on B300 with EAGLE/MTP speculative decoding. Recipe is
# DeepSeek-V4-Pro-0813 on B300 with DSpark speculative decoding. Recipe is
# selected inside benchmarks/single_node/dsv4_fp4_b300_sglang_mtp.sh by
# DP_ATTENTION:
# dp-attn: false -> TP-only + flashinfer_mxfp4 + chunked-prefill 8192
Expand All @@ -1178,7 +1178,8 @@ dsv4-fp4-b300-sglang-agentic-hicache-mtp:
# + chunked-prefill 65536 + mem-fraction 0.90
# + swa-full-tokens-ratio 0.075
# + prefill-decode-interval 20
# Both paths share EAGLE (3,1,4) and max-running-requests 2*CONC.
# Both paths share DSpark K=6 (1 step, 7 draft tokens) and
# max-running-requests 2*CONC.
dsv4-fp4-b300-sglang-mtp:
image: lmsysorg/sglang:nightly-dev-cu13-20260610-f332e526
model: deepseek-ai/DeepSeek-V4-Pro
Expand Down
11 changes: 11 additions & 0 deletions perf-changelog.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7286,3 +7286,14 @@
description:
- "Use thinking-on golden synthetic AL 3.51 for five-token DSpark throughput; disable adaptive verification and retain real verification for evals"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2974

- config-keys:
- dsv4-fp4-b300-sglang-agentic-hicache-mtp
scenario-type:
- agentic-coding
description:
- "Replace the B300 DeepSeek-V4-Pro AgentX native EAGLE/MTP serving path with DSpark block size 6 on the DeepSeek-V4-Pro-0813 checkpoint."
- "Use one speculative step, seven draft tokens, and thinking-on golden synthetic acceptance length 3.77 for throughput; eval retains real DSpark verification."
- "Use SGLang's current W4A4 MegaMoE and DP LM-head flags for DP attention, with additional MegaMoE workspace headroom at concurrency 32, 64, and 128."
- "Route the workload through the B300 DSXE launcher, read weights from /data/models, and mount a writable Hugging Face/Xet cache."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2918
20 changes: 15 additions & 5 deletions runners/launch_b300-dsxe.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ SQUASH_DIR="/data/home/sa-gha-runner/squash"
# it is read-only from the job's point of view. Anything not in STAGED_MODELS is
# downloaded into WRITABLE_MODELS_DIR (shared Lustre) by the single-node scripts.
MODEL_ROOT="/scratch/models"
SHARED_MODEL_ROOT="/data/models"
WRITABLE_MODELS_DIR="/data/home/sa-gha-runner/models"

# Official power (dcgm-power) runs use a separate, pinned producer; CI derives
Expand Down Expand Up @@ -394,15 +395,23 @@ done
find . -name '.nfs*' -delete 2>/dev/null || true

else
# HF_HUB_CACHE is set to help with dataset download inside the container
# for eval jobs.
export HF_HUB_CACHE="$HOME/.cache/huggingface"
# AgentX trace datasets need a writable persistent cache. Keep the host and
# container paths separate so the cache remains valid with
# --no-container-mount-home.
HF_CACHE_HOST_DIR="${B300_HF_CACHE_HOST_DIR:-$HOME/.cache/huggingface}"
HF_CACHE_CONTAINER_DIR="${B300_HF_CACHE_CONTAINER_DIR:-/hf_hub_cache}"
mkdir -p "$HF_CACHE_HOST_DIR/hub" "$HF_CACHE_HOST_DIR/xet"
export HF_HOME="$HF_CACHE_CONTAINER_DIR"
export HF_HUB_CACHE="$HF_CACHE_CONTAINER_DIR/hub"
export HF_XET_CACHE="$HF_CACHE_CONTAINER_DIR/xet"

# MODEL stays the HF id for the client; MODEL_PATH is where the server reads
# weights. Only the root holding MODEL_PATH is mounted -- mounting both roots
# makes pyxis fail whenever the unused one is absent on the node.
MODEL_BASENAME="${MODEL##*/}"
if [[ " ${STAGED_MODELS[*]} " == *" ${MODEL_BASENAME} "* ]]; then
if [[ "$MODEL_BASENAME" == "DeepSeek-V4-Pro-0813" ]]; then
MODEL_MOUNT_DIR="$SHARED_MODEL_ROOT"
elif [[ " ${STAGED_MODELS[*]} " == *" ${MODEL_BASENAME} "* ]]; then
MODEL_MOUNT_DIR="$MODEL_ROOT"
else
MODEL_MOUNT_DIR="$WRITABLE_MODELS_DIR"
Expand All @@ -411,7 +420,7 @@ else
export MODEL_PATH="${MODEL_MOUNT_DIR}/${MODEL_BASENAME}"

SQUASH_FILE="$SQUASH_DIR/$(echo "$IMAGE" | sed 's/[\/:@#]/_/g').sqsh"
SPEC_SUFFIX=$([[ "$SPEC_DECODING" == "mtp" ]] && printf '_mtp' || printf '')
SPEC_SUFFIX=$([[ "$SPEC_DECODING" == "mtp" || "$SPEC_DECODING" == "draft_model" ]] && printf '_mtp' || printf '')
# Prefer a framework-tagged script (e.g. dsv4_fp4_b300_sglang.sh); fall back to
# the untagged historical name for scripts that haven't been retagged yet.
BENCH_BASE="benchmarks/single_node/${SCENARIO_SUBDIR}${EXP_NAME%%_*}_${PRECISION}_b300"
Expand Down Expand Up @@ -468,6 +477,7 @@ else
CONTAINER_MOUNTS=(
"$GITHUB_WORKSPACE:$CONTAINER_MOUNT_DIR"
"$MODEL_MOUNT_DIR:$MODEL_MOUNT_DIR"
"$HF_CACHE_HOST_DIR:$HF_CACHE_CONTAINER_DIR"
)
CONTAINER_MOUNTS_ARG=$(IFS=,; printf '%s' "${CONTAINER_MOUNTS[*]}")

Expand Down
20 changes: 20 additions & 0 deletions runners/test_slurm_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -169,6 +169,26 @@ def test_copy_agentic_results_fails_when_aggregate_is_missing(
assert "no run_conc*.json results found" in result.stderr


def test_b300_dsxe_draft_model_uses_public_paths_and_writable_hf_cache() -> None:
launcher = (REPO_ROOT / "runners/launch_b300-dsxe.sh").read_text()

assert 'SLURM_ACCOUNT="benchmark"' in launcher
assert 'SQUASH_DIR="/data/home/sa-gha-runner/squash"' in launcher
assert 'SHARED_MODEL_ROOT="/data/models"' in launcher
assert 'WRITABLE_MODELS_DIR="/data/home/sa-gha-runner/models"' in launcher
assert '[[ "$MODEL_BASENAME" == "DeepSeek-V4-Pro-0813" ]]' in launcher
assert 'MODEL_MOUNT_DIR="$SHARED_MODEL_ROOT"' in launcher
assert "nv-gha-runner" not in launcher
assert (
'[[ "$SPEC_DECODING" == "mtp" || "$SPEC_DECODING" == "draft_model" ]]'
in launcher
)
assert 'export HF_HOME="$HF_CACHE_CONTAINER_DIR"' in launcher
assert 'export HF_HUB_CACHE="$HF_CACHE_CONTAINER_DIR/hub"' in launcher
assert 'export HF_XET_CACHE="$HF_CACHE_CONTAINER_DIR/xet"' in launcher
assert '"$HF_CACHE_HOST_DIR:$HF_CACHE_CONTAINER_DIR"' in launcher
Comment on lines +172 to +189

@claude claude Bot Sep 9, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 (optional) New test freezes literal source-code strings from launch_b300-dsxe.sh (e.g. SLURM_ACCOUNT="benchmark", SQUASH_DIR path, exact HF_HOME/HF_HUB_CACHE export lines) as raw substring assertions, directly violating AGENTS.md Test quality: "Do not freeze current recipe counts, hardware/framework inventories, image tags, pins, enum values, or source-code strings in assertions." Any harmless refactor of the launcher (renaming a var, reformatting a line, changing quoting) breaks this test even though behavior is unchanged. Fix: replace with a behavioral test that sources/executes the relevant launcher logic (as the file's other tests do via run_bash) and asserts on the resulting env vars/mount args, not on literal script text.

Extended reasoning...

The test does launcher = Path(...).read_text() then assert 'SLURM_ACCOUNT="benchmark"' in launcher and similar literal checks for MODEL_MOUNT_DIR, HF_HOME/HF_HUB_CACHE/HF_XET_CACHE export statements, and the container-mounts string. Every other test in this file (e.g. test_copy_fixed_sequence_results_, test_patch_srt_eval_dispatch_) instead runs the shell function/patch and asserts on observable output, per the file's own established pattern and AGENTS.md's explicit prohibition (line 23) on freezing source-code strings. A future contributor who reformats or renames these variables for clarity gets a spurious CI failure unrelated to any behavior change.

Verification: nit. The new test_b300_dsxe_draft_model_uses_public_paths_and_writable_hf_cache (runners/test_slurm_utils.py:151-168) does launcher = (REPO_ROOT / "runners/launch_b300-dsxe.sh").read_text() and then a series of raw substring assertions against the launcher's source text: 'SLURM_ACCOUNT="benchmark"', 'SQUASH_DIR="/data/home/sa-gha-runner/squash"',… | nit (test-quality convention…



def test_patch_srt_eval_dispatch_forwards_selection_and_is_idempotent(
tmp_path: Path,
) -> None:
Expand Down
Loading