GLM-5.3 series: engine image, example kits, and parked e2e cases

deploy/docker/Dockerfile.sglang

@@ -3,10 +3,24 @@
 # the command.
 #
 # ⚠️ The base tag is PINNED and must stay pinned: the GLM-5.2 DSA patches near
-# the end of this file are context diffs applied at --fuzz=0 against this exact
+# the end of this file are context diffs applied at --fuzz=0 against a specific
 # sglang release. Bumping the base fails the build there rather than silently
 # mis-applying. Build with APPLY_SGLANG_DSA_PATCHES=0 if you need a newer base.
-ARG SGLANG_BASE_IMAGE=lmsysorg/sglang:v0.5.17-rocm720-mi35x
+#
+# Moved v0.5.17 -> v0.5.18 for the GLM-5.3 series. GLM-5.3 and GLM-5.3-MXFP4 are
+# `glm_moe_dsa` / `GlmMoeDsaForCausalLM`, field-for-field identical to GLM-5.2
+# except `transformers_version`, so the released engine already serves them
+# through glm4_moe.py and this image needs no new model source. What the bump
+# cost was one hunk: on v0.5.18 the DSA set still applies except for
+# draft_cuda_graph_dp_vote's min()-reduce in dp_attn.py, which upstream renamed
+# out from under it -- carried by an anchor port, see
+# scripts/apply_sglang_dsa_patches.sh.
+#
+# GLM-5.3-**Flash** is NOT served by this image: it is `glm5_next`, which no
+# published sglang release carries at all. That needs the build-time source
+# overlay in Dockerfile.sglang.glm53, and deliberately stays out of this file so
+# the default build path for Kimi-K3 and GLM-5.2 never fetches unreleased source.
+ARG SGLANG_BASE_IMAGE=lmsysorg/sglang:v0.5.18-rocm720-mi35x
 FROM ${SGLANG_BASE_IMAGE}
 
 WORKDIR /opt/infera
@@ -113,10 +127,19 @@ RUN if [ "${APPLY_SGLANG_DSA_PATCHES}" = "1" ]; then \
 # prefill.py / mooncake/conn.py, which no DSA diff does. A script here exits non-zero
 # when its anchors drifted, i.e. when the fix did not go in — an engine image that
 # silently corrupts long prompts is worse than a failed build.
+# Two of these anchors moved between v0.5.17 and v0.5.18 — the typing import in
+# mooncake/conn.py grew `Set`, and serving_responses.py's `background=` argument
+# gained `and not request.stream`. Neither fix landed upstream, so the patches
+# are still needed; only their anchors are stale. reanchor_...glm53.sh retargets
+# them into a COPY, leaving the originals untouched for the v0.5.17-based
+# gfx942 image. It fails loudly if an old literal is gone, which would mean
+# upstream moved again (or took the fix) and this stage should be re-derived.
 COPY deploy/docker/patches/sglang_disagg/ /tmp/sglang-disagg-patches/
+COPY deploy/docker/scripts/reanchor_sglang_disagg_glm53.sh /tmp/reanchor.sh
 RUN set -eu; \
-    for f in /tmp/sglang-disagg-patches/*.py; do echo "[sglang-patch] $f"; python "$f"; done; \
-    rm -rf /tmp/sglang-disagg-patches
+    bash /tmp/reanchor.sh /tmp/sglang-disagg-patches /tmp/sglang-disagg-v0518; \
+    for f in /tmp/sglang-disagg-v0518/*.py; do echo "[sglang-patch] $f"; python "$f"; done; \
+    rm -rf /tmp/sglang-disagg-patches /tmp/sglang-disagg-v0518 /tmp/reanchor.sh
 
 # ---- sglang Responses API patches (not PD-specific) ------------------------
 # REQUIRED FOR ANY Codex-driven workload against a model with a custom chat

deploy/docker/patch.upstream.status.md

@@ -137,6 +137,66 @@ so expect upstream to close this differently than we did.
 > V4 stack runs on this branch, so that gate would be untested — the script's SCOPE
 > section records it for whoever gets there.
 
+## sglang carried — `patches/sglang_carried/` (**baked by nothing**)
+
+Held, reviewable, and deliberately not built. No Dockerfile copies or runs this
+directory, which is why these scripts are not in `sglang_rocm/` beside their
+siblings: every other patch directory is consumed by an unconditional
+`for f in .../*.py; do python "$f"; done`, so a file placed there is wired into
+every image that copies it, with no per-file opt-out.
+`patches/sglang_carried/README.md` carries the rule and the exit criteria.
+
+| patch | fixes | upstream issue | upstream PR | ours? | PR state |
+|---|---|---|---|---|---|
+| `sglang_carried/patch_glm_moe_gate_bias_fp32.py` | `MoEGate.__init__` allocates the MoE `e_score_correction_bias` as **bf16** whenever a quant_config is present and `_use_aiter`, and `biased_grouped_topk_gpu` casts it down again at the aiter call. GLM's bias is a narrow band at a large offset, so bf16 cannot hold it: measured on the checkpoints themselves, GLM-5.3 / -MXFP4 collapse **238 distinct fp32 values to 8** and GLM-5.3-Flash-MXFP4 **282 to 11**, which reorders `noaux_tc` top-k routing | none found | [sglang#37133](https://github.com/sgl-project/sglang/pull/37133) `[GLM-5.2] Keep GlmMoeDsa MoE e_score_correction_bias in fp32` (`xiaobochen-amd`) — same defect, **narrower gate**: see below | no (`xiaobochen-amd`) | OPEN |
+
+**Status: carried, not applied.** Verified as text and as predicate logic only —
+applies to copies of `v0.5.18`, `c821c425` and the live source extracted from
+`infera/engine-sglang:v0518-glm53` (result byte-identical to the patched
+`v0.5.18` tree), idempotent on re-run, and both drift cases exit 1 having written
+neither file. **It has never been executed on a GPU, and no accuracy or
+throughput delta has been measured.** Do not read the row above as validation.
+
+Not wired because the defect is a routing perturbation, not a crash: the server
+starts, answers, and reports plausible numbers either way. Rebuilding the engine
+image while an alignment campaign is in flight would make our arm the only one
+running the fixed router and destroy the like-for-like property those ratios rest
+on. The exit criterion is a hardware run plus a measurement of the thing it
+claims to change.
+
+> **Why the gate is not upstream's.** #37133 gates on
+> `any("GlmMoeDsa" in arch for arch in config.architectures)`. That predicate is
+> False for the whole Flash family and would ship a fix that does not fix it:
+> `Glm5NextForConditionalGeneration.__init__` does
+> `self.config = config.text_config` and passes the **text** config down, and
+> GLM-5.3-Flash-MXFP4's `text_config` has `architectures: None` — there is no arch
+> string on the object `MoEGate` receives, so widening the string list cannot
+> help. Flash imports `DeepseekV2MoE` (hence the same `MoEGate`) and has the same
+> collapse.
+>
+> This script's `_moe_gate_bias_wants_fp32()` is a union of three tests:
+> `moe_router_dtype == "float32"` (the model author's own declaration — carried by
+> all four GLM-5.3 configs and GLM-5.2-FP8, and **sglang has zero references to
+> the field**, which is the actual root cause); `model_type` (`glm_moe_dsa` /
+> `glm5_next*`, needed because GLM-5.2-MXFP4 and GLM-5.1-FP8 predate that field,
+> and it survives the NextN rewrite in `configs/model_config.py:623`); and
+> upstream's arch test, kept so this stays a superset of #37133 and becomes a
+> no-op once it lands. Checked against all 38 checkpoints on this host: 8/8 GLM
+> keep fp32, 30/30 non-GLM byte-identical.
+>
+> **Both edits or neither, and that is not stylistic.** aiter's launcher
+> (`csrc/kernels/topk_softmax_kernels_group.cu:1156`) dispatches on
+> `gating_output.dtype()` and then `reinterpret_cast`s the bias pointer to that
+> same `scalar_t` **without checking the bias tensor's own dtype**. An fp32 bias
+> under a bf16 gating tensor is neither an error nor a cast — it reads fp32 bytes
+> as bf16. So applying the `deepseek_v2.py` half alone is strictly worse than the
+> defect, and the script plans both files before writing either.
+>
+> **Drop this patch** when a base sglang keeps the GLM bias fp32 on both sides;
+> the replacement text is then already present and the script reports "already
+> present" and no-ops. If only #37133's form lands, keep it — the Flash half would
+> go with it.
+
 ## Mooncake C++ — `patches/mooncake_cpp/`
 
 SGLang now builds Mooncake `faae8dd4` directly and carries no private Mooncake

deploy/docker/patches/sglang_carried/README.md

@@ -0,0 +1,53 @@
+# `patches/sglang_carried/` — patches we hold, and deliberately do not build
+
+**No Dockerfile copies or runs anything in this directory.** That is the whole
+point of it, and it is the reason these scripts do not live beside their
+siblings in `patches/sglang_rocm/`.
+
+Every other patch directory is consumed the same way:
+
+```dockerfile
+COPY deploy/docker/patches/sglang_rocm/ /tmp/sglang-rocm-patches/
+RUN set -eu; \
+    for f in /tmp/sglang-rocm-patches/*.py; do echo "[sglang-patch] $f"; python "$f"; done; \
+```
+
+The glob is `*.py` and the loop is unconditional, so **dropping a file into one
+of those directories wires it into every image that copies the directory** —
+`Dockerfile.sglang` and `Dockerfile.sglang.gfx942` for `sglang_rocm/`. There is
+no per-file opt-out. A patch that is ready to read but not ready to build
+therefore cannot be stored there without changing two images.
+
+This directory is the opt-out. Nothing globs it, so a script here is carried,
+reviewable, and inert.
+
+## When a patch belongs here
+
+When all three hold:
+
+- the defect is established well enough to write the fix down,
+- the fix has **not** been executed on hardware, or its effect has not been
+  measured, and
+- wiring it would perturb something in flight — a benchmark campaign, an
+  alignment comparison, a packup whose numbers are already reported.
+
+The last one is the common case, and it is why "carried" is a status and not a
+euphemism for "unfinished". A silently changed image invalidates every number
+measured before and after it; an unwired script costs nothing and loses nothing.
+
+## Moving one out
+
+Move the file into the directory of the image that should carry it
+(`sglang_rocm/`, `sglang_dsa/`, …) and update its row in
+`../../patch.upstream.status.md` from *carried, not applied* to the image list
+it is now baked by. The glob picks it up from there — no Dockerfile edit is
+needed, which is exactly why the move must be deliberate.
+
+Before moving one out, it needs what it did not have when it was parked: a run
+on hardware, and a measurement of the thing it claims to change.
+
+## Contents
+
+| script | what it fixes | why it is not wired |
+|---|---|---|
+| `patch_glm_moe_gate_bias_fp32.py` | GLM MoE `e_score_correction_bias` is allocated bf16 under aiter+quant and downcast again at the aiter router boundary; GLM's bias band collapses from 238 distinct fp32 values to 8 in bf16 (Flash: 282 → 11), reordering `noaux_tc` top-k routing | verified as text and as predicate logic only — **never executed on a GPU**, and no accuracy or throughput delta measured. Wiring it would rebuild the engine image mid-campaign and confound an alignment comparison whose validity rests on both arms running the same code |