[Klaud Cold] glm5.2-fp8-b300-sglang-agentic-mtp: day-zero GLM-5.2 FP8 SGLang AgentX MTP on B300 / B300 上 GLM-5.2 FP8 SGLang AgentX MTP 首发配方 - #2884
Conversation
… SGLang AgentX MTP on B300 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
d0976d3 to
0feaf99
Compare
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
2 similar comments
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
I reviewed this PR and didn't find any bugs. Because it's a new 376-line benchmark launcher plus a configs/nvidia-master.yaml entry (a CODEOWNER-covered file) with non-trivial memory/HiCache sizing and locking logic, a human look would still be worthwhile.
What was reviewed: the flock-serialized/completeness-checked hf download staging path, the HiCache/--mem-fraction-static memory budget arithmetic against the stated B300 HBM/DRAM figures, the EAGLE/MTP spec-decode arg wiring and golden-AL env pinning, and the perf-changelog.yaml append (confirmed it only adds a new tail entry, no prior bytes touched). Also checked the fixed 270 GB/rank --hicache-size not scaling with TOTAL_CPU_DRAM_GB — this mirrors the already-shipped NVFP4 B300 sibling's design (per its own comments/#2651) rather than being an oversight.
Extended reasoning...
Overview
This PR adds one new file (a 376-line bash launcher for a GLM-5.2 FP8 agentic-coding SGLang benchmark on B300), one new config entry in configs/nvidia-master.yaml, and one new append-only entry in perf-changelog.yaml. It is explicitly described (and appears in the code) as a mechanical port combining an existing FP8 B200 sibling script with B300-specific deltas from an existing NVFP4 B300 sibling (fixed --mem-fraction-static 0.85 and a fixed 270 GB/rank HiCache target pool). No test files, docs, or other infra code are touched.
Security risks
No injection, auth, or data-exposure surface is introduced. The script does shell out to hf download, flock, and python3 for a completeness check of downloaded shards, but all inputs are internal (env vars set by the harness/launcher) rather than user-controlled, and the same patterns already exist in the sibling scripts this is ported from.
Level of scrutiny
configs/nvidia-master.yaml is a CODEOWNER-covered path (per .github/CODEOWNERS: @ ankur-singh @ kedarpotdar-nv @ InferenceX/core), which per the approval guidelines means this should not be auto-approved regardless of how mechanical the change looks. The launcher script itself has some real logic (flock-based download serialization, checkpoint completeness verification, HiCache/memory-fraction sizing arithmetic, golden-AL simulated-acceptance env wiring) that is worth a domain-expert's eyes even though the bug-hunting pass found nothing and the memory/HiCache math in the comments is internally consistent with the stated hardware numbers.
Other factors
The bug-hunting system reported zero findings, and one candidate issue (fixed 270 GB/rank HiCache size not scaling with TOTAL_CPU_DRAM_GB) was investigated and ruled out as intentional, mirroring the already-merged NVFP4 B300 sibling's design rather than being a regression introduced by this PR. The perf-changelog.yaml addition correctly appends at the tail without altering prior bytes, consistent with the append-only convention noted for that file. Given the CODEOWNER ownership and the presence of non-trivial (if apparently correct) logic, I'm deferring rather than approving, even though no concrete bug was found.
This review covers commit d0976d3, which is no longer the latest commit on this pull request; later commits are not covered by it.
…GLE DSA fp8 read-door None guard) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34177606213 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34179028866 |
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34179028866 |
1 participant
Summary
Day-zero GLM-5.2 FP8 AgentX (agentic-coding) recipe on B300 with SGLang native EAGLE/MTP:
glm5.2-fp8-b300-sglang-agentic-mtp, the FP8 precision sibling ofglm5.2-fp4-b300-sglang-agentic-mtpand the B300 counterpart of the FP8 B200 recipe in #2863.zai-org/GLM-5.2-FP8(~756 GB, 141 shards,quant_method=fp8, 128x128 weight blocks). Not in the DSXESTAGED_MODELSlist, so the launcher places it under its writable models dir and the script's completeness-checked, flock-serializedhf downloadstages it once on the day-zero run.benchmarks/single_node/agentic/glm5.2_fp8_b300_sglang_mtp.sh, a port of the FP8 B200 script with the two B300 deltas the NVFP4 B300 sibling carries:--mem-fraction-static 0.85and a fixed 270 GB/rank HiCache target pool at every concurrency (perf: HiCache tuning for glm5.2-fp4-b300-sglang-agentic-mtp / 性能：glm5.2-fp4-b300-sglang-agentic-mtp HiCache 调优 #2651). Serve flags otherwise identical: TP8 low-latency arm, fp8_e4m3 KV, cutedsl bf16 GEMM, chunked-prefill 8192,--quantization fp8, glm47/glm45 parsers, EAGLE num-steps 3 / topk 1 / 4 draft tokens, golden AL 2.99 (measured on this FP8 checkpoint).lmsysorg/sglang:nightly-dev-cu13-20260907-30705c00(verified on Docker Hub, pushed 2026-09-07T01:43:42Z, digestsha256:19b8fa1223cc339c1eae7a5b703f1a8c2543b5b119155bf3d7efaef18f77f007), same as [Klaud Cold] glm5.2-fp8-b200-sglang-agentic-mtp: day-zero GLM-5.2 FP8 SGLang AgentX MTP on B200 / B200 上 GLM-5.2 FP8 SGLang AgentX MTP 首发配方 #2863 and the B300 SGLang bumps [Klaud Cold] Update glm5.2-fp4-b300-sglang-agentic-mtp SGLang image to nightly-dev-cu13-20260908-20ca564b and move to cluster:b300-dsxe #2880-[Klaud Cold] Update qwen3.5-fp4-b300-sglang-agentic-mtp SGLang image to nightly-dev-cu13-20260907-30705c00 and move to cluster:b300-dsxe #2882.cluster:b300-dsxefrom the start (cluster:b300-nvwas retired in feat(runners): add the B300 DSXE cluster and retire the B300 NV launcher / 添加 B300 DSXE 集群并下线 B300 NV 启动脚本 #2826).Validation:
bash -n, both YAMLs parse,generate_sweep_configs.py test-configwith the runner config yields the expected 5 cells oncluster:b300-dsxe, and the DSXE launcher's script-name construction resolves to the new file.Recipes touched:
glm5.2-fp8-b300-sglang-agentic-mtpTest plan
🤖 Generated with Claude Code
Note
Low Risk
Adds benchmark scripts and YAML recipe entries only; no changes to application runtime, auth, or data paths beyond new cluster sweep configuration.
Overview
Introduces the first GLM-5.2 FP8 AgentX (
agentic-coding) recipe on B300: config keyglm5.2-fp8-b300-sglang-agentic-mtponcluster:b300-dsxe, matching the NVFP4 B300 sibling’s TP8 + HiCache search space (conc 1–16).Adds
benchmarks/single_node/agentic/glm5.2_fp8_b300_sglang_mtp.sh, ported from the FP8 B200 script with B300 tuning (--mem-fraction-static 0.85, fixed 270 GB/rank HiCache),--quantization fp8, EAGLE/MTP, golden AL 2.99, and flock-serializedhf downloadwith a complete-checkpoint guard for the unstaged ~756 GB weights.Documents the recipe in
perf-changelog.yamland pins SGLang tonightly-dev-cu13-20260908-20ca564b(not the 2026-09-07 tag) to pick up sglang#38318, avoiding intermittent DSA fp8 KV / EAGLEAttributeErrorcrashes on long MTP runs.Reviewed by Cursor Bugbot for commit 9d7ce52. Bugbot is set up for automated code reviews on this repo. Configure here.
Update: re-pinned to the 2026-09-08 nightly
The 2026-09-07 cu13 nightly (
nightly-dev-cu13-20260907-30705c00) carries an unguardedkv_index_translator.translate_dcp_read_idscall on the DSA fp8 KV read path; the EAGLE draft backend never binds a translator, so GLM-5.2 MTP runs crash intermittently withAttributeError: 'NoneType' object has no attribute 'translate_dcp_read_ids'(seen on the MI355X FP8 sibling, run 34173459478, after 74 minutes of clean serving). sgl-project/sglang#38318 (merged 2026-09-07T20:03Z) adds the None guard. Re-pinned tolmsysorg/sglang:nightly-dev-cu13-20260908-20ca564b(pushed 2026-09-08T01:40:59Z, digestsha256:9a352a35c973a2357372e85f3bcb5388b6b3c46c1329165987260f3b089647dc), which is six commits past that fix. Recipe otherwise unchanged.