Conversation
…DEP8) Ports the agentX (llm-manifesto/agentx-mvp) DeepSeek-V4-Pro 1P-EP8/1D-EP8 K8s deployment to InferenceX's SLURM-based llmd-vllm framework on B200 (8 GPUs/node, so each DEP8 role fits on one node vs GB200's 4 GPUs/node needing two). New master-config key dsv4-fp4-b200-llmd-vllm-agentx and recipe carry the agentX EPP scheduling config (disagg-profile-handler's `deciders:` shape, prefix-cache-affinity-filter, token-load-scorer, etc., ported verbatim from a real agentx-mvp results snapshot) and per-role vLLM tuning, dropping the Kubernetes/manifesto-specific Mooncake KV chain in favor of server.sh's existing plain NixlConnector. Wires a self-contained FRAMEWORK==llmd-vllm branch into runners/launch_b200-dgxc.sh (ahead of the srtctl/srt-slurm path used by the other B200 multinode frameworks, which llm-d bypasses entirely), mirroring launch_gb200-nv.sh's pyxis/enroot + slurm_utils.sh helpers pattern. Pins a B200-only EPP/pd-sidecar v0.10.0 binary override (LLMD_BIN_DIR) for the newer EPP plugin shape, isolated from GB200's shared v0.9.0 binaries/path. Co-Authored-By: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Cursor <cursoragent@cursor.com>
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
- dsv4-fp4-b200-llmd-vllm-agentx: drop c256 from the 1P-DEP8/1D-DEP8 conc-list per request. - extract-binaries.sh: support BINARIES_ENV_FILE to source an alternate pins file, so a cluster-scoped router-version bump doesn't require editing the shared binaries.env (whose EPP_FROM_IMAGE/ ROUTING_SIDECAR_IMAGE are plain assignments and can't be overridden via pre-exported env vars). - Add binaries-b200-v0.10.0.env: B200-only v0.10.0 EPP/pd-sidecar pins, linux/amd64 platform (B200 is x86_64, unlike GB200's arm64 default), dedicated LLMD_BIN_DIR. - launch_b200-dgxc.sh: fix the extraction command documented in the LLMD_BIN_DIR comment, which previously relied on the non-overridable env vars above and would have silently extracted v0.9.0 binaries. Co-authored-by: Cursor <cursoragent@cursor.com>
Sibling of the P/D disagg benchmark: one engine per topology does both
prefill and decode (1 node / 8 GPUs each), no P/D KV handoff.
- server.sh/job.slurm: add an aggregated mode (DECODE_NODES=0) - no
decode role, no pd-sidecar, no --kv_transfer_config (there is nothing
to hand KV off to), and the sole engine's rank 0 takes over the
EPP/Envoy/bench coordinator duties normally owned by the decode
leader. Also fixes the underlying DECODE_NODES=0 bugs: job.slurm's
out-of-bounds IPS[] read for DECODE_LEADER_IP, and server.sh emitting
a bogus empty-address decode-0 endpoint in endpoints.yaml.
- New recipes dsv4-fp4-b200-agentx-agg-{tp8,dep8}.yaml: per-role vLLM
tuning ported from agentX's ix-agg-base/agg-tp8 and
ix-agg-dep-base/agg-dep8, plus a minimal single-schedulingProfile EPP
config (no disagg-profile-handler needed - llm-d auto-selects
single-profile-handler for one profile). Needs no EPP router bump
since every plugin used is already in the default v0.9.0 binaries.
- New dsv4_fp4_b200_llmd-vllm-agg.sh wrapper + launch_b200-dgxc.sh now
branches its llmd-vllm wrapper-script choice on $DISAGG instead of
hardcoding the disagg wrapper.
- New dsv4-fp4-b200-llmd-vllm-agentx-agg master-config key (disagg:
false, prefill num-worker=1 / decode num-worker=0 per the
process_result.py / dynamo-vllm aggregated convention). conc-lists
mirror the agentX dspark historical runs (results_ilmarkov-2-ix-agg-
tp8-dspark-v3-long, results_ilmarkov-ix-agg-dep8-dspark-1308-long).
- README.md + perf-changelog.yaml updated.
Co-authored-by: Cursor <cursoragent@cursor.com>
|
Claude finished @ilmarkov's task in 13m 58s —— View job Review of PR #2719 — DeepSeek-V4-Pro llm-d B200 (AgentX)
SummaryFound 3 blocking and 2 warning issues (posted as inline comments):
Doc-consistency nits (no inline comments): many comments reference Positive notes: server.sh's aggregated-mode gating ( |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| if [[ "$registry" == "registry-1.docker.io" && "$repository" != */* ]]; then | ||
| repository="library/$repository" | ||
| fi | ||
| printf 'docker://%s#%s:%s\n' "$registry" "$repository" "$digest" |
There was a problem hiding this comment.
🟡 WARNING: The digest branch keeps the tag inside repository, producing a malformed enroot URI for repo:tag@sha256:... images.
Why it matters: For an image like the sibling GB200 llm-d pin quay.io/rh-ee-imarkov/llm-d-nokube-vllm:vllm0.26@sha256:a909..., image_without_digest still contains :vllm0.26, so this prints docker://quay.io#rh-ee-imarkov/llm-d-nokube-vllm:vllm0.26:sha256:a909... — exactly the tag@digest ambiguity this function exists to avoid (per its own header comment). It's latent today only because the new B200 image is tag-only; the moment this image gets digest-pinned like its GB200 counterpart, the import breaks.
Fix:
| if [[ "$registry" == "registry-1.docker.io" && "$repository" != */* ]]; then | |
| repository="library/$repository" | |
| fi | |
| printf 'docker://%s#%s:%s\n' "$registry" "$repository" "$digest" | |
| if [[ "$registry" == "registry-1.docker.io" && "$repository" != */* ]]; then | |
| repository="library/$repository" | |
| fi | |
| # tag@digest form: drop the tag; enroot wants registry#repo:digest. | |
| repository="${repository%%:*}" | |
| printf 'docker://%s#%s:%s\n' "$registry" "$repository" "$digest" |
| # benchmarks/llm-d/extract-binaries.sh | ||
| # before this recipe can actually run; job.slurm's pyxis mount loop is | ||
| # a no-op (falls back to the image's baked-in v0.9.0) until then. | ||
| export LLMD_BIN_DIR="${LLMD_BIN_DIR:-/home/sa-shared/llm-d-bins-v0.10.0}" |
There was a problem hiding this comment.
🟡 WARNING: The v0.10.0 LLMD_BIN_DIR override is applied to every B200 llmd-vllm run, including the aggregated configs that declare router: { version: "0.9.0" }.
Why it matters: This export happens before the DISAGG branch, and job.slurm's mount loop mounts whatever executables it finds there over the image's baked-in binaries. Once /home/sa-shared/llm-d-bins-v0.10.0 is populated (a prerequisite for the disagg recipe to work at all), the dsv4-fp4-b200-llmd-vllm-agentx-agg runs will silently execute the v0.10.0 EPP/pd-sidecar while their master-config metadata, the recipe comments, and the perf-changelog all state they use the default v0.9.0 binaries — the published router version won't match what actually ran.
Fix: Either gate the override on the disagg path (move the export LLMD_BIN_DIR=... inside the DISAGG == true branch, added after the wrapper-selection if), or update the agg key's router: version/comments to 0.10.0 so the metadata matches reality.
There was a problem hiding this comment.
Beyond the inline findings, I checked the ruled-out changelog entry more closely: the Mooncake entry's config-keys (dsv4-fp4-b200-llmd-vllm-agentx) does match its "disagg DEP8 arm" description, since that key names the disagg config (not the -agg aggregated one), so there's no additional config-keys mismatch there.
Extended reasoning...
This run's bug hunt already surfaced five confirmed, non-trivial issues (env-var export bug breaking SLURM submission, append-only violation and invalid PLACEHOLDER pr-links in the byte-sensitive perf-changelog.yaml, a misnamed gb200 prefix on B200-only recipe files, and a malformed enroot URI for tag@ digest images), which on their own are sufficient to require human attention and block approval. I independently re-verified the one ruled-out candidate (the Mooncake entry's config-keys pointing at the disagg key dsv4-fp4-b200-llmd-vllm-agentx versus its "disagg DEP8 arm" description) by reading the surrounding perf-changelog.yaml entries directly, and confirmed the key correctly refers to the disagg config (the aggregated variant is the separate -agg key), so there is no mismatch there. Given the confirmed findings already speak for themselves as inline comments, this defer note is limited to recording that one additional, distinct concern was checked and ruled out rather than restating the existing findings.
Findings marked 🟡 are optional suggestions and need no follow-up push.
| if [[ "$FRAMEWORK" == "llmd-vllm" ]]; then | ||
| # Self-contained: bypasses the srtctl/srt-slurm path entirely (the | ||
| # "Validate framework" guard and the dsv4-only-dynamo-vllm guard | ||
| # further below are srtctl-specific and are never reached from here). | ||
| # MODEL_PATH is already resolved above (the dsv4/fp4 branch); llm-d | ||
| # additionally needs MODEL_NAME (served-model-name), which this file | ||
| # doesn't otherwise set outside the srtctl block - reuse $MODEL (the | ||
| # master-config `model:` field), matching this file's own | ||
| # `export SERVED_MODEL_NAME=$MODEL` convention below. | ||
| if [[ ! ( "$MODEL_PREFIX" == "dsv4" && "$PRECISION" == "fp4" ) ]]; then | ||
| echo "Unsupported MODEL_PREFIX/PRECISION for llmd-vllm on B200: $MODEL_PREFIX/$PRECISION" >&2 | ||
| exit 1 | ||
| fi | ||
| export MODEL_NAME="$MODEL" |
There was a problem hiding this comment.
🔴 The new llmd-vllm block never exports SLURM_PARTITION/SLURM_ACCOUNT before handing off to bash "$BENCH_SCRIPT", which forks submit.sh (a separate process) whose check_env SLURM_ACCOUNT/check_env SLURM_PARTITION require them in the environment; every B200 llmd-vllm job submission fails immediately with "Error: SLURM_ACCOUNT not set". Fix: add export SLURM_PARTITION SLURM_ACCOUNT in the llmd-vllm block, mirroring the sibling tilert branch a few lines below (line 249) which does exactly this before its own subprocess handoff.
Extended reasoning...
SLURM_PARTITION/SLURM_ACCOUNT are set as plain (non-exported) shell vars at the top of the file (lines 5-6) and no workflow sets them as job-level env, so they exist only in this script's local scope. The llmd-vllm branch runs JOB_ID=$(bash "$BENCH_SCRIPT") which execs dsv4_fp4_b200_llmd-vllm-agg.sh/disagg.sh, which runs bash ./submit.sh ... - a brand-new process that only inherits exported vars. submit.sh's check_env SLURM_ACCOUNT / check_env SLURM_PARTITION (both required first) then exit 1 before sbatch ever runs. Contrast with the tilert branch immediately below, which does export SLURM_PARTITION SLURM_ACCOUNT right before its own exec bash "$TILERT_DISAGG" handoff - the same pattern the new block omits.
Verification: normal — the new llmd-vllm block forks a subprocess without exporting the SLURM credentials that subprocess requires. runners/launch_b200-nscale-compat.sh:5-6 set SLURM_PARTITION="${SLURM_PARTITION:-batch_1}" / SLURM_ACCOUNT="${SLURM_ACCOUNT:-benchmark}" as plain (non-exported) vars. The new llmd-vllm branch runs JOB_ID=$(bash "$BENCH_SCRIPT") which execs… | normal — new code…
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33643687916 |
Conflicts resolved: - perf-changelog.yaml: keep both our PR #2719 dsv4-gb200 entry and all upstream new entries (Kimi-K3 GB300, MiniMax-M3 MI355X, etc.) - runners/launch_gb200-nv.sh: take upstream version which adds AGENTX_POWER_RC validation gate and fixes EVAL_ONLY default to :-false Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…light The previous mock always wrote METRICS_BODY to --output and never printed to stdout, so agentic.sh's envoy preflight check got an empty HTTP status instead of "404" and exited before any assertion could run. Now when --write-out is in args (preflight call), print "404" to stdout. Worker scrape calls (no --write-out) write METRICS_BODY to --output and record the URL in METRICS_REQUESTS as before. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Commit 9411ed9 updated agentic.sh to read endpoint['name'], endpoint['labels']['llm-d.ai/role'], and DECODE_NODES (argv[4]) but did not update the test fixture. The endpoint dicts only had address/port so agentic.sh crashed with KeyError: 'name' before reaching the curl calls. Add name and labels to each endpoint, and DECODE_NODES=0 to the env. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 36c6f4d. Configure here.
| decode: | ||
| num-worker: 1 | ||
| tp: 8 | ||
| ep: 8 |
There was a problem hiding this comment.
DEP8 topologies labeled as TP8
Medium Severity
DEP8 search-space entries set tp: 8 with ep: 8, but the recipes they launch use tp: 1 and expert parallel. Existing GB200 DEP8 keys in this file already use tp: 1 / ep: 8. Job names, result filenames, and ingested topology metadata will claim TP8 while vLLM actually runs TP1/DP8/EP.
Additional Locations (2)
Reviewed by Cursor Bugbot for commit 36c6f4d. Configure here.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34861485243 |
|
/stage-results 34861485243 |
|
@ilmarkov staged run 34861485243: https://inferencemax-app-git-staging-semianalysisai.vercel.app/inference?i_dates=2026-09-14~r34861485243 This run remains available across future |
Disaggregated p1d1 (server.sh): - Remove SimpleCPUOffloadConnector from decode role. In EAGER mode it wrote every decoded KV block to mooncake, polluting the prefix cache with non-reusable decode blocks and evicting prefill prefix entries. Prefill retains SimpleCPUOffload (~38 GB CPU DRAM staging). Aggregated DEP8 mooncake (server.sh): - Add SimpleCPUOffloadConnector (~40 GB) to the IS_AGGREGATED+mooncake path, matching the local DEP8 v1 reference configuration. EPP agentic recipes (lruCapacityPerServer, peakPrefillThroughput, queueThresholdTokens): - p1d1: lruCapacityPerServer 45272→43949 (GPU 10,746 + CPU 33,203 per prefill DP rank, cross-validated against v13 kv_cache_config) - DEP8 no-mooncake: lru 8967→5858 (GPU only, 1,499,703 tokens / 256), peakPrefillThroughput 4783→20000, queueThreshold 2295620→1499703 - DEP8 mooncake: lru 8967→40810 (GPU 5,858 + CPU 34,952 per DP rank), peakPrefillThroughput 4783→20000, queueThreshold 3518020→1499703 All queueThresholdTokens are now calibrated per DP rank (one endpoint = one DP rank), and lruCapacityPerServer reflects actual GPU+CPU KV pool sizes reported by vLLM at the measured gpu_memory_utilization. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Bugbot is paused — on-demand spend limit reachedBugbot uses usage-based billing for this team and has hit its on-demand spend limit. A team admin can raise the spend limit in the Cursor dashboard, or wait for the next billing cycle to continue. |
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=35239304468 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=35239304468 |
Replace hybrid LB and two-endpoint discovery with multi-port external LB, per-DP-rank EPP registration, multi-port pd-sidecar, supervisor health on 8100, Nixl-only decode KV, and rank-aware AgentX metrics scraping. 将 DEP8 AgentX 从 hybrid LB 与双端点发现改为多端口外部 LB、按 DP rank 注册 EPP、多端口 pd-sidecar、8100 supervisor 健康检查、decode 仅 Nixl KV， 以及按 rank 抓取 AgentX 指标。 Co-authored-by: Cursor <cursoragent@cursor.com>
Resolve llm-d external-LB conflicts, keep branch perf-changelog entries after upstream updates, and relocate llmd agentic adapter tests. 将 upstream/main 合并入 imarkov/dsv4-pro-agentic：解决 llm-d 外部 LB 冲突，在保留上游 perf-changelog 更新后将本分支条目追加到末尾，并迁移 llmd agentic 适配器测试。 Co-authored-by: Cursor <cursoragent@cursor.com>
Re-remove dsv4-fp4-gb200-llmd-vllm from active master after the upstream merge reintroduced it, and restore the thin AgentX disagg launcher so P/D llmd-vllm jobs can submit again. 重新移除 upstream 合并误带回的 dsv4-fp4-gb200-llmd-vllm 8k1k 主配置项， 并恢复 AgentX P/D llmd-vllm 所需的精简 disagg launcher。 Co-authored-by: Cursor <cursoragent@cursor.com>
Merge latest main and resolve perf-changelog conflicts after restoring the GB200 llmd-vllm disagg wrapper. 合并最新 main，并在恢复 GB200 llmd-vllm disagg wrapper 后解决 perf-changelog 冲突。 Co-authored-by: Cursor <cursoragent@cursor.com>
Raise vLLM logging on llmd-vllm runtime defaults and GB200 DSpark AgentX recipes so external-LB DEP8 worker deaths leave actionable traces. 将 llmd-vllm 运行时默认与 GB200 DSpark AgentX 配方的 vLLM 日志级别 提升至 DEBUG，以便外部 LB DEP8 worker 异常退出时留下可诊断日志。 Co-authored-by: Cursor <cursoragent@cursor.com>
7456ad7 to
4742446
Compare
|
/reuse-sweep-run 35239304468 |
|
/stage-results 35239304468 |
|
@cquil11 staged run 35239304468: https://inferencemax-app-git-staging-semianalysisai.vercel.app/inference?i_dates=2026-09-17~r35239304468 This run remains available across future |
2 participants
Changes
cluster:gb200-nv: aggregated TP8/DEP8 on 2 nodes (8 GPUs), or 1P-DEP8/1D-DEP8 on 4 nodes (16 GPUs).Validation
Generated GB200 matrices have 2-node aggregate and 4-node disaggregated allocations. Live sweep validation is in progress; no successful GPU result is claimed yet.
Note
Medium Risk
Touches multi-node serving (Mooncake master, KV connectors, SLURM env contract) and benchmark invariants (golden AL, offload metadata); mostly benchmark infra but misconfiguration could fail jobs or skew metrics.
Overview
Adds DeepSeek-V4-Pro-0813 AgentX on GB200 via llm-d (
cluster:gb200-nv): master configs for aggregated TP8/DEP8 (2 nodes) and 1P-DEP8/1D-DEP8 disagg (4 nodes), plus GB200 DSpark recipes (prefix/token-load EPP, optional Mooncake DRAM offload).Runtime/orchestration changes:
DECODE_NODES=0aggregated mode (rank 0 runs EPP/Envoy/bench, no pd-sidecar).recipe.pyenforces AgentX offload metadata, injects DSpark golden AL for throughput, and renders Mooncake store config.server.shwires MultiConnector (NIXL ± Mooncake), longer KV lease (1800s) for Mooncake P/D, and routes AgentX throughagentic.sh.agentic.shscrapes per-node vLLM/metricsfrom discovery; Envoy now 404s/metricson the frontend to avoid duplicated counters. Thin GB200 wrappers,submit.shhardware→pyxis/docker mapping, andlaunch_gb200-nv.sh(0813 checkpoint, agg vs disagg script, AgentX log copy). AgentX uses image-bundled router v0.10.0 (no legacy binary mounts on pyxis).Reviewed by Cursor Bugbot for commit 36c6f4d. Bugbot is set up for automated code reviews on this repo. Configure here.