Skip to content

Optimize GLM-5.2-MXFP4 SGLang Agentic Performance on MI355X #2900

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
jiejingzhangamd wants to merge 7 commits into
base: main
Choose a base branch
from
+14 −5
Open

Optimize GLM-5.2-MXFP4 SGLang Agentic Performance on MI355X #2900

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
3050458
update config for glm5.2-mxfp4 on mi355x gpu
lxgsbqylbk Sep 8, 2026
5361b95
update pr link
lxgsbqylbk Sep 8, 2026
09c2e8b
Merge remote-tracking branch 'origin/main' into dev/limou/glm5.2-sgla…
jiejingzhangamd Sep 8, 2026
c8c7ca7
Update changelog link for replacement PR
jiejingzhangamd Sep 8, 2026
4937165
Update changelog link for upstream PR
jiejingzhangamd Sep 8, 2026
52835e9
Merge remote-tracking branch 'origin/main' into dev/jiejing/glm5.2-sg…
jiejingzhangamd Sep 8, 2026
3ac32bd
Merge branch 'main' into dev/jiejing/glm5.2-sglang-opt-0908
chunfangamd Sep 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions benchmarks/single_node/agentic/glm5.2_fp4_mi355x_sglang_mtp.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,9 +91,9 @@ if agentic_kv_offload_enabled; then
# env-var override for maximum throughput on nodes with >4 TB DRAM.
HICACHE_RATIO="${HICACHE_RATIO:-1.5}"
fi
# write_through_selective skips DRAM writes for non-reusable KV blocks,
# reducing host-bus traffic without affecting the cache hit rate.
HICACHE_WRITE_POLICY="${HICACHE_WRITE_POLICY:-write_through_selective}"
# Keep write_through as the validated baseline. Override
# HICACHE_WRITE_POLICY explicitly for selective-write experiments.
HICACHE_WRITE_POLICY="${HICACHE_WRITE_POLICY:-write_through}"
HICACHE_IO_BACKEND="${HICACHE_IO_BACKEND:-direct}"
HICACHE_MEM_LAYOUT="${HICACHE_MEM_LAYOUT:-page_first_direct}"
case "$KV_OFFLOAD_BACKEND" in
Expand Down
2 changes: 1 addition & 1 deletion configs/amd-master.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1708,7 +1708,7 @@ minimaxm3-fp4-mi355x-vllm-agentic-mtp:
# c6 and c8 removed after sweep validation: dominated by TP4/EP4 arm.
# SA selects the Pareto-optimal arm per concurrency point.
glm5.2-fp4-mi355x-sglang-agentic-mtp:
image: lmsysorg/sglang-rocm:v0.5.16-rocm720-mi35x-20260728
image: lmsysorg/sglang-rocm:v0.5.19-rocm720-mi35x-20260907
model: amd/GLM-5.2-MXFP4
model-prefix: glm5.2
runner: cluster:mi355x-amds
Expand Down
11 changes: 10 additions & 1 deletion perf-changelog.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6948,6 +6948,16 @@
- "Add --compilation-config cudagraph_mode=FULL_DECODE_ONLY to the serve command. The upstream nightly does not torch-compile MiniMaxM3SparseForConditionalGeneration, so with VLLM_USE_BREAKABLE_CUDAGRAPH=0 the default FULL_AND_PIECEWISE mode aborts at engine init with piecewise CUDA graphs unavailable (first sweep, run 34174124043, eval cell); full decode-only graphs are what the MI355X MiniMax-M3 sibling runs on its nightly (#2825)."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2873

- config-keys:
- glm5.2-fp4-mi355x-sglang-agentic-mtp
scenario-type:
- agentic-coding
description:
- "Update the SGLang ROCm image from lmsysorg/sglang-rocm:v0.5.16-rocm720-mi35x-20260728 to lmsysorg/sglang-rocm:v0.5.19-rocm720-mi35x-20260907."
- "Pick up our recent SGLang main-branch optimizations for GLM-5.2-MXFP4 serving."
- "Restore HiCache write_through as the default write policy to optimize GLM-5.2-MXFP4 output interactivity and per-GPU throughput in the MI355X AgentX configuration."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2900

@cursor cursor Bot Sep 8, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changelog entry not appended at tail

Medium Severity

The new glm5.2-fp4-mi355x-sglang-agentic-mtp changelog block was inserted before later main entries and a historical separator line was dropped. perf-changelog.yaml is append-only: new entries belong at the physical end, and existing bytes including separator whitespace stay unchanged. A YAML parse does not catch this; validate_raw_change and the reuse merge helper require a byte-identical historical prefix.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 52835e9. Configure here.


- config-keys:
- qwen3.5-fp8-b200-sglang
scenario-type:
Expand All @@ -6957,7 +6967,6 @@
- "Tune the 8k/1k serving recipe with --mamba-full-memory-ratio 0.37 and --linear-attn-prefill-backend flashinfer; raise max-prefill-tokens and chunked-prefill-size from 16384 to 32768 and mem-fraction-static from 0.8 to 0.86."
- "Expand the TP8 and TP4/EP1 sweep coverage through concurrency 640."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2866

@claude claude Bot Sep 8, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 (optional) The diff deletes the file's pre-existing trailing whitespace-only line (old file ended "...pull/2866\n \n", new ends "...pull/2866\n"), violating AGENTS.md's rule that perf-changelog.yaml is append-only/byte-sensitive: preserve all existing bytes and separator whitespace, append only at the tail. Fix: re-add the removed trailing line so no existing bytes are altered; only append the new entry above it, never edit or truncate the tail.

Extended reasoning...

Base file (at 4caeb31) ends with the last pr-link line followed by a line containing two spaces and a newline (confirmed via od -c). HEAD's perf-changelog.yaml drops that trailing line entirely (git diff shows a bare deletion with no matching addition at file end). AGENTS.md line 31 states the file is append-only and byte-sensitive and existing bytes/separator whitespace must be preserved, appends only at the tail — this diff instead mutates the tail by removing a byte sequence, which any tooling relying on byte-stable diffs/checksums against this file would flag or choke on, and sets precedent that trailing separator content can be silently dropped in future PRs.

Verification: normal (violates a documented non-negotiable invariant this diff introduces). Base perf-changelog.yaml at 4caeb31 ends with ...pull/2866\n \n — od -c on the base tail shows 2866 \n \n (a trailing line of two spaces plus newline after the final pr-link). HEAD's file ends ...pull/2866\n — od -c on the current tail shows 2866 \n with the whitespace-only line gone. The diff's last…



- config-keys:
- dsv4-fp4-b200-sglang-agentic-hicache-mtp
Expand Down
Loading