Conversation
|
Thanks for the contribution!
中文感谢你的贡献！
|
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Beyond the inline findings, I also checked two other items flagged by the repo conventions: the new recipe YAMLs' model.container/image fields all match the lmsysorg/sglang:nightly-dev-20260916-c9a8fba9 pin in configs/nvidia-master.yaml, and the disagg scenarios' prefill/decode num-worker counts (1P1D and 2P1D) follow the same pattern as existing disagg entries elsewhere in the file, with the node-count label in the workflow computed dynamically rather than touched by this diff - no additional issues found in either area.
Extended reasoning...
Two confirmed findings (hard-coded SYNTHETIC_ACCEPTANCE_LENGTH violating the AGENTS.md golden_al_distribution auto-selection rule, and a non-canonical astral-sh/setup-uv pin) are already queued as inline comments, so a human review is warranted and I should not approve. Beyond those, I independently verified two of the repo-specific conventions called out for this diff: (1) all six new recipe YAMLs' model.container/image values match the lmsysorg/sglang:nightly-dev-20260916-c9a8fba9 image pinned in the two new configs/nvidia-master.yaml entries, so there is no image-mismatch issue; and (2) the disagg scenarios' node/worker allocation (1P1D at c64/c128, 2P1D at c256) follows the exact same num-worker-based pattern used by other pre-existing disagg entries in nvidia-master.yaml (no num-nodes field is expected or present for disagg search-space entries, matching precedent), and the nodes:N scheduling label in benchmark-multinode-tmpl.yml is derived dynamically from inputs.node-count rather than being touched by this diff, so no node-count mismatch was introduced. These checks turned up nothing beyond the already-reported findings.
Findings marked 🟡 are optional suggestions and need no follow-up push.
| - "SYNTHETIC_ACCEPTANCE=true" | ||
| - "SYNTHETIC_ACCEPTANCE_LENGTH=3.77" |
There was a problem hiding this comment.
🔴 Six new scenarios hard-code SYNTHETIC_ACCEPTANCE_LENGTH=3.77, which AGENTS.md's "SRT Slurm synthetic acceptance" section explicitly forbids: InferenceX must auto-select the measured value from golden_al_distribution/ instead. This makes all six new DSV4 AgentX throughput benchmarks report numbers from a fixed simulated acceptance length rather than the model's actual measured one, producing misleading perf results merged into perf-changelog.yaml. Fix: remove the manual SYNTHETIC_ACCEPTANCE_LENGTH (and SYNTHETIC_ACCEPTANCE) additional-settings and let apply_srt_recipe's automatic golden_al_distribution selection apply, as required for every speculative AgentX scenario. …
Extended reasoning...
…Same pattern at configs/nvidia-master.yaml:8180,8194,8208,8237,8255,8273 (agg c1/c4/c8 and disagg c64/c128/c256 prefill worker additional-settings).
AGENTS.md lines 52-55 state: do not hard-code synthetic acceptance lengths in SRT recipes, master configs, or launchers; do not add manual SYNTHETIC_ACCEPTANCE_LENGTH; InferenceX automatically selects the measured value from golden_al_distribution/ for speculative AgentX throughput runs via apply_srt_recipe / infx/srt_slurm/synthetic_acceptance.py. The new dsv4-fp4-b200-dynamo-sglang-agentic-agg and -disagg entries in configs/nvidia-master.yaml set additional-settings including "SYNTHETIC_ACCEPTANCE=true" and "SYNTHETIC_ACCEPTANCE_LENGTH=3.77" for all 3 agg concurrency points and all 3 disagg prefill workers (6 sites total, lines 8180, 8194, 8208, 8237, 8255, 8273). Because these are passed straight through as SRT overrides in the recipe/launch path (mtp spec-decoding scenarios), the throughput runs simulate acceptance at a fixed 3.77 tokens instead of using the measured golden distribution, so every reported decode…
Verification: normal. AGENTS.md line 54 (under "## SRT Slurm synthetic acceptance", lines 52-58) states unambiguously: "Do not hard-code synthetic acceptance lengths in SRT recipes, master configs, or launchers. ... Do not add manual SYNTHETIC_ACCEPTANCE_LENGTH ... InferenceX automatically selects the measured value from golden_al_distribution/". The diff to configs/nvidia-master.yaml adds two new…
| # pinned action so the B200 launcher does not depend on that endpoint. | ||
| - name: Install uv for NScale B200 | ||
| if: ${{ inputs.runner == 'cluster:b200-nscale' }} | ||
| uses: astral-sh/setup-uv@c771a70e6277c0a99b617c7a806ffedaca235ff9 # v9.0.0 |
There was a problem hiding this comment.
🟡 (optional) Operators running cluster:b200-nscale multi-node jobs get a uv install pinned to a different, non-canonical astral-sh/setup-uv commit than every other call site in the repo. Line 310 pins @ c771a70e6277c0a99b617c7a806ffedaca235ff9 # v9.0.0, while every other workflow (ci.yml, claude.yml, run-sweep.yml, collect-results.yml, etc.) and docs/klaud.md's pin table use @ 20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1. Fix: use the repo's canonical setup-uv pin here too; if this SHA is stale or invalid, every b200-nscale multi-node job fails at this step before ever reaching the launcher's curl fallback.
Extended reasoning...
benchmark-multinode-tmpl.yml:310 is the only astral-sh/setup-uv call site in the repo not using 20cfd1bf945f4377ade1205e4dbc17946fc9a30d (v10.0.1), which docs/klaud.md:125 documents as the current canonical pin and which ci.yml:47, claude.yml:39/316, run-sweep.yml (4 sites), collect-results.yml:36, collect-evals.yml:35, stage-results.yml:35, profile.yml:78, zizmor.yml:38, klaud-plan.yml:46, klaud-candidate.yml:47, codeowner-signoff-verify.yml:56, operatorx-sweep.yml:56, e2e-tests.yml (2 sites), speedbench-al.yml:131 and collectivex-sweep.yml:72 all match exactly. This new step has no continue-on-error and runs (gated on inputs.runner == 'cluster:b200-nscale') right before the 'Launch multi-node job script' step at line 312. If c771a70e6277c0a99b617c7a806ffedaca235ff9 does not resolve to a real astral-sh/setup-uv commit/tag, GitHub Actions fails to resolve the action reference and the whole job fails immediately, before runners/launch_b200-nscale-slurm.sh's own 'if ! command -v uv; then curl ...; fi' fallback (lines 111-113) is ever reached. I could not confirm SHA validity from this…
Verification: nit — the factual claim holds but nothing functionally breaks. .github/workflows/benchmark-multinode-tmpl.yml:310 (added by this diff) pins astral-sh/setup-uv@ c771a70e6277c0a99b617c7a806ffedaca235ff9 # v9.0.0, and a repo-wide grep confirms this is the ONLY setup-uv call site not on the canonical 20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1: ci.yml:47, claude.yml:39/316,… | nit — pin…
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=35345548475 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=35345548475 |
1 participant
Description
Add DeepSeek-V4-Pro-0813 FP4 B200 Dynamo+SGLang AgentX configurations:
Validation:
AI model disclosure
Related Issue
None.
Type of Change
Checklist
perf-changelog.yamland have not edited historical entriesOWNER/MEMBER/COLLABORATOR) has commented/use <run_id>(or the legacy/reuse-sweep-run) on this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the sweep label will no longer automatically kick off new sweeps. Remove and re-add the label to force one.中文
说明
新增 DeepSeek-V4-Pro-0813 FP4 B200 Dynamo+SGLang AgentX 配置：聚合 TP8 HiCache 并发 1、4、8；解耦 DEP8/DEP8 的 1P1D 并发 64、128，以及 2P1D 并发 256。使用模型内置 DSpark 自草稿器与 SGLang nightly-dev-20260916-c9a8fba9，并为 HiCache 分配完整 NScale B200 节点 CPU 和 DRAM。1P1D decode 使用 0.01 的 SWA/full 比例，2P1D c256 使用 0.005，以增加 full-attention KV 容量。
AI 模型使用说明
验证
变更日志处理器生成六个性能测试项和三个 AgentX eval 项；已验证配方名称、注册路径、GPU 资源计算、DSpark 一致性以及 CPU/DRAM 指令；300 项相关测试全部通过。