perf(amd): switch DSV4 ATOM AgentX to native RCCL DEP / 将 DSV4 ATOM AgentX 切换到原生 RCCL DEP - #2912
perf(amd): switch DSV4 ATOM AgentX to native RCCL DEP / 将 DSV4 ATOM AgentX 切换到原生 RCCL DEP#2912yhl-amd wants to merge 7 commits into
Conversation
将 DeepSeek-V4-Pro MI355X ATOM AgentX 的 c48 及以上测试切换到本地验证过的原生 RCCL DEP 配置，并保持低并发 TP 测试不变。 同步固定的 post-merge ATOM 镜像、EP8 元数据、关闭 TBO/EPLB、真实 MTP 接受率以及本地验证过的路由和 AIPerf 参数。
将 DeepSeek-V4-Pro RCCL DEP 性能变更记录中的占位链接替换为实际的 InferenceX PR 链接。
精简 DeepSeek-V4-Pro AgentX RCCL DEP 配置，移除与 CLI 或公共默认值重复的环境变量，并补齐本地验证使用的 3600 秒 warmup grace。
Restore the request-equivalent weight, prefill delayer, and decode interval requested for the AgentX run. Remove the newly introduced terminal MTP overrides while keeping the rest of the cleanup unchanged.
Keep the 3600-second agentic warmup allowance, but rely on the server keep-alive setting and AIPerf default benchmark grace period.
Remove the DEP-only 32K override so both TP and DEP retain the original 8192-token state checkpoint interval.
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 5c91ae8. Configure here.
| # affinity is required: otherwise consecutive turns can land on another DPA | ||
| # rank and lose access to the prefix KV produced by the previous turn. | ||
| DEP_ARGS=() | ||
| STATE_CHECKPOINT_INTERVAL_TOKENS=8192 |
There was a problem hiding this comment.
Checkpoint interval misses validated 32K
Medium Severity
STATE_CHECKPOINT_INTERVAL_TOKENS stays at 8192, but the validated native RCCL DEP c48 run and this PR’s changelog both set the state checkpoint interval to 32768 tokens. The DEP band will checkpoint four times more often than the measured recipe, changing prefix-cache behavior and adding extra publisher forwards.
Reviewed by Cursor Bugbot for commit 5c91ae8. Configure here.
| # x-correlation-id) so the DPA router pins each conversation to one rank. | ||
| export AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID=true | ||
| export AIPERF_HTTP_X_SESSION_ID_FROM_CORRELATION_ID=true | ||
| DP_ATTN_ARGS=(--enable-dp-attention --enable-tbo) |
There was a problem hiding this comment.
Request equivalence not set to zero
Medium Severity
ATOM_DP_LB_REQ_EQUIV remains 512 even though the validated DEP recipe and changelog require zero request equivalence. With the new least_tokens policy, load is prompt_tokens + (ATOM_DP_LB_REQ_EQUIV * in-flight requests), so 512 reweights routing away from the measured pure token balance.
Reviewed by Cursor Bugbot for commit 5c91ae8. Configure here.
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.
Beyond the inline finding, I also checked the dropped AIPERF_HTTP_X_SESSION_ID_FROM_CORRELATION_ID export in the DEP branch — the surrounding comment now only describes the dynamo-specific session-id header for DPA routing, so removing the generic fallback appears to be an intentional simplification rather than a regression.
Extended reasoning...
One confirmed finding (perf-changelog.yaml claiming zero request equivalence and a 32768-token checkpoint interval that contradict the script's ATOM_DP_LB_REQ_EQUIV=512 and hardcoded STATE_CHECKPOINT_INTERVAL_TOKENS=8192) is already queued for posting as an inline comment, so it is not restated here. I independently reviewed the script diff, confirming this discrepancy is real (STATE_CHECKPOINT_INTERVAL_TOKENS=8192 is set unconditionally at line ~57 with no path to 32768, and ATOM_DP_LB_REQ_EQUIV=512 is set explicitly for the DP_ATTENTION branch). Given a verified finding exists, approval is not appropriate. I additionally examined the other ruled-out candidate — the removal of AIPERF_HTTP_X_SESSION_ID_FROM_CORRELATION_ID alongside AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID — and concluded it looks intentional: the updated comment describes only the dynamo-specific session header path for the new DEP transport, with no remaining reference to a generic fallback, consistent with dropping the redundant export. This is worth surfacing briefly as it was a real candidate concern that a reviewer might otherwise re-raise, but does not need extensive discussion. No new top-level architectural or security concerns were found beyond what's already captured in the inline finding.
| - "Move the concurrency 48/64/96/128/256 ATOM AgentX band from DPA with TBO and the default MoE transport to native RCCL DEP (TP8/DPA8/EP8), while preserving the TP-only concurrency 1/2/4/8/16 band." | ||
| - "Match the validated DEP run: disable EPLB and TBO, use least-tokens routing with session affinity and zero request equivalence, set the state checkpoint interval to 32768 tokens, and use real MTP acceptance on DEP." | ||
| - "Carry over the validated AgentX transport, terminal-MTP, and AIPerf timeout/failure settings for stable multi-turn replay." | ||
| pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2886 |
There was a problem hiding this comment.
🟡 (optional) The changelog entry's third bullet claims "zero request equivalence" and a "32768" token checkpoint interval, but the script actually sets ATOM_DP_LB_REQ_EQUIV=512 and STATE_CHECKPOINT_INTERVAL_TOKENS=8192 unconditionally — readers/auditors relying on the changelog to explain observed throughput will be misled about the real DEP config. Fix: correct the bullet to state ATOM_DP_LB_REQ_EQUIV=512 and an 8192-token checkpoint interval, matching benchmarks/single_node/agentic/dsv4_fp4_mi355x_atom_mtp.sh lines 65 and 57.
Extended reasoning...
perf-changelog.yaml lines 6958-6959 (the new dsv4-fp4-mi355x-atom-agentic-mtp entry) state 'zero request equivalence' and 'set the state checkpoint interval to 32768 tokens'. The script sets export ATOM_DP_LB_REQ_EQUIV=512 (line 65) and STATE_CHECKPOINT_INTERVAL_TOKENS=8192 (line 57) with no code path that changes either value to 0 or 32768. Since perf-changelog.yaml is append-only and treated as the authoritative record of what changed and why (per AGENTS.md invariant #1), this factual error will mislead future engineers debugging DEP throughput/latency differences from the documented values.
Verification: nit. The changelog claim and the script genuinely disagree. perf-changelog.yaml line 6958 (new dsv4-fp4-mi355x-atom-agentic-mtp entry) states: "...use least-tokens routing with session affinity and zero request equivalence, set the state checkpoint interval to 32768 tokens...". The script benchmarks/single_node/agentic/dsv4_fp4_mi355x_atom_mtp.sh sets STATE_CHECKPOINT_INTERVAL_TOKENS=8192…
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34311990815 |
4 similar comments
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34311990815 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34311990815 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34311990815 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34311990815 |
2 participants
Summary / 概要
Switch the DeepSeek-V4-Pro MI355X ATOM AgentX concurrency 48/64/96/128/256 band introduced by perf(amd): add DP-attention band to DSV4 ATOM MI355X AgentX MTP #2778 from DPA+TBO with the default MoE transport to native RCCL DEP: TP8, DPA8, EP8, RCCL all-to-all, TBO off, and EPLB off.
Keep the concurrency 1/2/4/8/16 TP band unchanged at TP8/EP1 and retain its golden MTP acceptance length.
Pin
rocm/atom-dev:nightly_202609071454(sha256:f252ec9a30e3d17b91f15887ea32baa1a0c5503b775f5adf42ff7cee8cfff769), built by the successful ROCm/ATOM release run at merge commit5a9c2068containing native RCCL DEP.Match the validated local c48 run: least-tokens scheduling, session affinity,
ATOM_DP_LB_REQ_EQUIV=0, 32K state checkpoints, terminal-MTP optimizations, real MTP acceptance, and the validated AIPerf timeout/failure settings.将 perf(amd): add DP-attention band to DSV4 ATOM MI355X AgentX MTP #2778 引入的 DeepSeek-V4-Pro MI355X ATOM AgentX c48/64/96/128/256 档位，从默认 MoE 通信的 DPA+TBO 切换为原生 RCCL DEP：TP8、DPA8、EP8、RCCL all-to-all，并关闭 TBO 和 EPLB。
保持 c1/2/4/8/16 档位为原来的 TP8/EP1，同时保留该档位的 MTP golden acceptance length，不改变低并发测试语义。
固定使用
rocm/atom-dev:nightly_202609071454（sha256:f252ec9a30e3d17b91f15887ea32baa1a0c5503b775f5adf42ff7cee8cfff769）。该镜像由成功的 ROCm/ATOM 发布任务基于包含原生 RCCL DEP 的合并提交5a9c2068构建。对齐本地已验证的 c48 参数：least-tokens 调度、session affinity、
ATOM_DP_LB_REQ_EQUIV=0、32K state checkpoint、terminal-MTP 优化、真实 MTP 接受率，以及已验证的 AIPerf 超时和失败阈值。Local validation / 本地验证
Validated source run:
atom-native-rccl-dpa8-ep8-eplboff-agentx-c48-ci3600-realmtp-fp4idx-r4-20260902本地 c48 实跑完成 3,920 个请求且请求错误数为 0；输出吞吐为 1,066.71 token/s，prompt cache-read rate 为 96.55%，ITL 中位数为 19.12 ms。
Test plan / 测试计划
bash -n benchmarks/single_node/agentic/dsv4_fp4_mi355x_atom_mtp.shParse
configs/amd-master.yaml,configs/runners.yaml, andperf-changelog.yamlwith PyYAMLgit diff --checkGenerate the exact
dsv4-fp4-mi355x-atom-agentic-mtpmatrix withgenerate_sweep_configs.pyVerify TP points are c1/2/4/8/16 with EP1 and DEP points are c48/64/96/128/256 with EP8+DPA
Bash、YAML 和补丁格式检查通过
仓库矩阵生成器成功展开全部十个测试点，并确认低并发 TP 档和高并发 DEP 档的边界正确
References / 参考
Note
Medium Risk
Changes official benchmark topology, MoE/DPA transport, and MTP acceptance semantics for high-concurrency AgentX runs; low-concurrency TP behavior is preserved but cluster image and server launch paths change.
Overview
Switches the DeepSeek-V4-Pro MI355X ATOM AgentX high-concurrency sweep (c48–256) from DPA + TBO with default MoE transport to native RCCL DEP (TP8 / DPA8 / EP8), while leaving the low-concurrency TP8/EP1 band unchanged.
dsv4_fp4_mi355x_atom_mtp.shnow drives expert parallel fromEP_SIZE, validatesEP_SIZE == TPwhenDP_ATTENTIONis on, and passes DEP server flags (--all2all-backend rccl,--dp-load-balance least_tokens,--moe-backend standard) instead of--enable-tboand the old TBO/NUMA env knobs. Golden synthetic MTP acceptance (AL 2.49) applies only to the TP throughput band; DEP and eval use real MTP acceptance. Client/session pinning and AgentX warmup grace are adjusted for multi-turn replay under DEP.configs/amd-master.yamlbumps the ATOM image torocm/atom-dev:nightly_202609071454and setsep: 8on thedp-attn: truesearch-space row.perf-changelog.yamlrecords the image pin and topology change.Reviewed by Cursor Bugbot for commit f5a688c. Bugbot is set up for automated code reviews on this repo. Configure here.
This PR supersedes #2886. The commit history is unchanged; the head branch has been moved from the personal fork to SemiAnalysisAI/InferenceX as requested by the maintainers.