From aac6c4558b12813d36f20cebad62ecc19fd0c62e Mon Sep 17 00:00:00 2001 From: functionstackx <47992694+functionstackx@users.noreply.github.com> Date: Mon, 7 Sep 2026 20:55:50 -0400 Subject: [PATCH 1/5] [Klaud Cold] Update qwen3.8next-fp8-h200-sglang-agentic-mtp SGLang image to dev-cu13-qwen38-next-local (2026-09-07, digest-pinned) Co-Authored-By: Claude Fable 5.1 --- configs/nvidia-master.yaml | 2 +- perf-changelog.yaml | 8 ++++++++ 2 files changed, 9 insertions(+), 1 deletion(-) diff --git a/configs/nvidia-master.yaml b/configs/nvidia-master.yaml index d9b13b7d2c..9b0c7c1079 100644 --- a/configs/nvidia-master.yaml +++ b/configs/nvidia-master.yaml @@ -7409,7 +7409,7 @@ qwen3.5-fp8-h200-sglang-agentic-mtp: # low-latency single-node command, which shards the 512-expert MoE with expert # parallelism rather than sharding attention eight ways. qwen3.8next-fp8-h200-sglang-agentic-mtp: - image: lmsysorg/sglang:qwen38flashnext + image: lmsysorg/sglang:dev-cu13-qwen38-next-local@sha256:9d2a843c706c74bc259c0d9abf360551eb2734e1e7d255ab012a6965f10480b6 model: Qwen/Qwen3.8-Flash-Next-FP8 model-prefix: qwen3.8next runner: cluster:h200-dgxc diff --git a/perf-changelog.yaml b/perf-changelog.yaml index e0ef448195..5648c07656 100644 --- a/perf-changelog.yaml +++ b/perf-changelog.yaml @@ -6947,3 +6947,11 @@ - "Update vLLM ROCm image from vllm/vllm-openai-rocm:v0.27.1 (v0.27.1 release) to vllm/vllm-openai-rocm:nightly-d9105ea8001e0a6d77a96327d17515bb5791fb36 (2026-09-07 upstream ROCm nightly, digest sha256:74d4a95f3ae672ecddf9acb7296917def82d9eca51687fa2862ae72b03ff1907, tag commit vllm-project/vllm@d9105ea8; Docker Hub last pushed 2026-09-07T05:26:48Z). benchmarks/single_node/agentic/minimaxm3_fp8_mi300x_mtp.sh is unchanged: TRITON_ATTN attention, fp8 KV, block-size 128, EAGLE3 speculative decoding with the Inferact MiniMax-M3 EAGLE3-GQA draft and the committed golden synthetic acceptance length, minimax_m3 tool-call and reasoning parsers; the search space is unchanged. The upstream commit-pinned ROCm nightly is the same vllm commit the B200 MiniMax-M3 AgentX recipe moved to in #2860. Note that vllm-openai-rocm commit-nightly tags have expired from Docker Hub within days in the past; node squash caches keep merged configs running, but a re-pin to a durable tag may be needed later." - "Add --compilation-config cudagraph_mode=FULL_DECODE_ONLY to the serve command. The upstream nightly does not torch-compile MiniMaxM3SparseForConditionalGeneration, so with VLLM_USE_BREAKABLE_CUDAGRAPH=0 the default FULL_AND_PIECEWISE mode aborts at engine init with piecewise CUDA graphs unavailable (first sweep, run 34174124043, eval cell); full decode-only graphs are what the MI355X MiniMax-M3 sibling runs on its nightly (#2825)." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2873 + +- config-keys: + - qwen3.8next-fp8-h200-sglang-agentic-mtp + scenario-type: + - agentic-coding + description: + - "Move the image from the mutable model-branch tag lmsysorg/sglang:qwen38flashnext (last re-pushed 2026-09-03T20:54:46Z, digest sha256:5ae5816783d58e2e56e84d2e863f5441425056f500b7fbd7448c4aae017a2521) to the newest Qwen3.8-Flash-Next model-branch build, lmsysorg/sglang:dev-cu13-qwen38-next-local@sha256:9d2a843c706c74bc259c0d9abf360551eb2734e1e7d255ab012a6965f10480b6 (Docker Hub last pushed 2026-09-07T11:12:42Z), pinned by digest so every node runs the same build. Qwen3.8-Flash-Next (architecture Qwen4ExpForConditionalGeneration) is not in SGLang main, v0.5.19, or the 2026-09-07 nightly (nightly-dev-cu13-20260907-30705c00), so no upstream nightly can serve it; the model-branch dev images are the only lineage. benchmarks/single_node/agentic/qwen3.8next_fp8_h200_sglang_mtp.sh is unchanged: TP4/EP4, mem-fraction 0.85, flashinfer attention (sm_90), native NEXTN MTP at three speculative tokens, float32 Mamba SSM state, golden thinking_on acceptance length 2.32; the conc [1, 4, 8, 12, 16] grid is unchanged. The sweep is the validation that this branch build still carries the fixes the recipe relies on." + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2877 From c64b680dabddac61f422eb56a92ae5d84e84f8b1 Mon Sep 17 00:00:00 2001 From: functionstackx <47992694+functionstackx@users.noreply.github.com> Date: Tue, 8 Sep 2026 00:00:27 -0400 Subject: [PATCH 2/5] Resolve the model from the local HF cache when the shared-cache lock is not writable Co-Authored-By: Claude Fable 5.1 --- .../agentic/qwen3.8next_fp8_h200_sglang_mtp.sh | 13 ++++++++++++- perf-changelog.yaml | 1 + 2 files changed, 13 insertions(+), 1 deletion(-) diff --git a/benchmarks/single_node/agentic/qwen3.8next_fp8_h200_sglang_mtp.sh b/benchmarks/single_node/agentic/qwen3.8next_fp8_h200_sglang_mtp.sh index 5199d539cc..4e1d884d52 100755 --- a/benchmarks/single_node/agentic/qwen3.8next_fp8_h200_sglang_mtp.sh +++ b/benchmarks/single_node/agentic/qwen3.8next_fp8_h200_sglang_mtp.sh @@ -50,7 +50,18 @@ if [[ -n "${MODEL_PATH:-}" ]]; then hf download "$MODEL" --local-dir "$MODEL_PATH" fi else - hf download "$MODEL" + # The shared /mnt/hf_hub_cache on cluster:h200-dgxc holds lock files under + # .locks// owned by whichever runner account first downloaded a blob; + # filelock creates them 0644, so a job on a different runner account gets + # EACCES opening the lock even when every file is already cached (run + # 34174941536: "Fetching 144 files: 94%" then PermissionError on + # .locks/models--Qwen--Qwen3.8-Flash-Next-FP8/.lock). Only the + # online revalidation takes locks, so on failure resolve the snapshot from + # the local cache alone; a genuinely missing file still fails loudly there. + if ! hf download "$MODEL"; then + echo "hf download failed online (shared-cache lock permissions?); resolving $MODEL from the local HF cache only" + HF_HUB_OFFLINE=1 hf download "$MODEL" + fi export MODEL_PATH="$MODEL" fi nvidia-smi diff --git a/perf-changelog.yaml b/perf-changelog.yaml index 5648c07656..63912b45da 100644 --- a/perf-changelog.yaml +++ b/perf-changelog.yaml @@ -6954,4 +6954,5 @@ - agentic-coding description: - "Move the image from the mutable model-branch tag lmsysorg/sglang:qwen38flashnext (last re-pushed 2026-09-03T20:54:46Z, digest sha256:5ae5816783d58e2e56e84d2e863f5441425056f500b7fbd7448c4aae017a2521) to the newest Qwen3.8-Flash-Next model-branch build, lmsysorg/sglang:dev-cu13-qwen38-next-local@sha256:9d2a843c706c74bc259c0d9abf360551eb2734e1e7d255ab012a6965f10480b6 (Docker Hub last pushed 2026-09-07T11:12:42Z), pinned by digest so every node runs the same build. Qwen3.8-Flash-Next (architecture Qwen4ExpForConditionalGeneration) is not in SGLang main, v0.5.19, or the 2026-09-07 nightly (nightly-dev-cu13-20260907-30705c00), so no upstream nightly can serve it; the model-branch dev images are the only lineage. benchmarks/single_node/agentic/qwen3.8next_fp8_h200_sglang_mtp.sh is unchanged: TP4/EP4, mem-fraction 0.85, flashinfer attention (sm_90), native NEXTN MTP at three speculative tokens, float32 Mamba SSM state, golden thinking_on acceptance length 2.32; the conc [1, 4, 8, 12, 16] grid is unchanged. The sweep is the validation that this branch build still carries the fixes the recipe relies on." + - "Fall back to HF_HUB_OFFLINE=1 hf download when the online download fails: the shared /mnt/hf_hub_cache on cluster:h200-dgxc carries .locks// lock files owned by whichever runner account first fetched a blob (filelock creates them 0644), so a job on another runner account hits PermissionError acquiring the lock even with every file already cached (run 34174941536, four cells, Fetching 144 files: 94% then EACCES). Only online revalidation takes locks; the offline resolve serves the cached snapshot and still fails loudly on a genuinely missing file. Serve flags unchanged." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2877 From 677c3ee4fc034f325f5977c4eb88abbbabd7df50 Mon Sep 17 00:00:00 2001 From: functionstackx <47992694+functionstackx@users.noreply.github.com> Date: Tue, 8 Sep 2026 00:11:24 -0400 Subject: [PATCH 3/5] Retrigger CI: no workflow run registered for 387f2f77a From ec419012e8af17bc57f088e0483753ebaa4d4b04 Mon Sep 17 00:00:00 2001 From: functionstackx <47992694+functionstackx@users.noreply.github.com> Date: Tue, 8 Sep 2026 01:00:01 -0400 Subject: [PATCH 4/5] Serve a complete cached snapshot when the shared-cache lock blocks hf download Co-Authored-By: Claude Fable 5.1 --- .../qwen3.8next_fp8_h200_sglang_mtp.sh | 45 +++++++++++++++---- perf-changelog.yaml | 1 + 2 files changed, 37 insertions(+), 9 deletions(-) diff --git a/benchmarks/single_node/agentic/qwen3.8next_fp8_h200_sglang_mtp.sh b/benchmarks/single_node/agentic/qwen3.8next_fp8_h200_sglang_mtp.sh index 4e1d884d52..9b1f5470ea 100755 --- a/benchmarks/single_node/agentic/qwen3.8next_fp8_h200_sglang_mtp.sh +++ b/benchmarks/single_node/agentic/qwen3.8next_fp8_h200_sglang_mtp.sh @@ -53,16 +53,43 @@ else # The shared /mnt/hf_hub_cache on cluster:h200-dgxc holds lock files under # .locks// owned by whichever runner account first downloaded a blob; # filelock creates them 0644, so a job on a different runner account gets - # EACCES opening the lock even when every file is already cached (run - # 34174941536: "Fetching 144 files: 94%" then PermissionError on - # .locks/models--Qwen--Qwen3.8-Flash-Next-FP8/.lock). Only the - # online revalidation takes locks, so on failure resolve the snapshot from - # the local cache alone; a genuinely missing file still fails loudly there. - if ! hf download "$MODEL"; then - echo "hf download failed online (shared-cache lock permissions?); resolving $MODEL from the local HF cache only" - HF_HUB_OFFLINE=1 hf download "$MODEL" + # EACCES opening the lock. Qwen/Qwen3.8-Flash-Next-FP8 changed upstream on + # 2026-08-31 (a README and config revision) after this recipe merged, so + # every `hf download` now wants one new blob and dies on that lock (runs + # 34174941536 and 34188408693: "Fetching 144 files: 99%" then + # PermissionError; HF_HUB_OFFLINE cannot help because the new revision is + # not fully cached). When the online download fails, serve the newest + # cached snapshot that is complete -- tokenizer, shard index, and every + # shard the index names -- which is the same set of weights this recipe was + # validated on. A cache with no complete snapshot still fails loudly. + if hf download "$MODEL"; then + export MODEL_PATH="$MODEL" + else + echo "hf download failed online (shared-cache lock permissions?); looking for a complete cached snapshot of $MODEL" + HF_SNAPSHOT_ROOT="${HF_HUB_CACHE:-$HOME/.cache/huggingface/hub}/models--${MODEL//\//--}/snapshots" + SNAPSHOT="" + for d in $(ls -1dt "$HF_SNAPSHOT_ROOT"/*/ 2>/dev/null); do + if [[ -f "$d/config.json" && -f "$d/tokenizer_config.json" && -f "$d/model.safetensors.index.json" ]] \ + && CKPT_DIR="$d" python3 - <<'PYEOF' +import json, os, sys +d = os.environ["CKPT_DIR"] +with open(os.path.join(d, "model.safetensors.index.json")) as fh: + shards = sorted(set(json.load(fh)["weight_map"].values())) +missing = [s for s in shards if not os.path.isfile(os.path.join(d, s))] +sys.exit(1 if missing else 0) +PYEOF + then + SNAPSHOT="${d%/}" + break + fi + done + if [[ -z "$SNAPSHOT" ]]; then + echo "Error: no complete cached snapshot of $MODEL under $HF_SNAPSHOT_ROOT and the online download failed." >&2 + exit 1 + fi + echo "Serving cached snapshot $SNAPSHOT" + export MODEL_PATH="$SNAPSHOT" fi - export MODEL_PATH="$MODEL" fi nvidia-smi diff --git a/perf-changelog.yaml b/perf-changelog.yaml index 63912b45da..550e1e48fc 100644 --- a/perf-changelog.yaml +++ b/perf-changelog.yaml @@ -6955,4 +6955,5 @@ description: - "Move the image from the mutable model-branch tag lmsysorg/sglang:qwen38flashnext (last re-pushed 2026-09-03T20:54:46Z, digest sha256:5ae5816783d58e2e56e84d2e863f5441425056f500b7fbd7448c4aae017a2521) to the newest Qwen3.8-Flash-Next model-branch build, lmsysorg/sglang:dev-cu13-qwen38-next-local@sha256:9d2a843c706c74bc259c0d9abf360551eb2734e1e7d255ab012a6965f10480b6 (Docker Hub last pushed 2026-09-07T11:12:42Z), pinned by digest so every node runs the same build. Qwen3.8-Flash-Next (architecture Qwen4ExpForConditionalGeneration) is not in SGLang main, v0.5.19, or the 2026-09-07 nightly (nightly-dev-cu13-20260907-30705c00), so no upstream nightly can serve it; the model-branch dev images are the only lineage. benchmarks/single_node/agentic/qwen3.8next_fp8_h200_sglang_mtp.sh is unchanged: TP4/EP4, mem-fraction 0.85, flashinfer attention (sm_90), native NEXTN MTP at three speculative tokens, float32 Mamba SSM state, golden thinking_on acceptance length 2.32; the conc [1, 4, 8, 12, 16] grid is unchanged. The sweep is the validation that this branch build still carries the fixes the recipe relies on." - "Fall back to HF_HUB_OFFLINE=1 hf download when the online download fails: the shared /mnt/hf_hub_cache on cluster:h200-dgxc carries .locks// lock files owned by whichever runner account first fetched a blob (filelock creates them 0644), so a job on another runner account hits PermissionError acquiring the lock even with every file already cached (run 34174941536, four cells, Fetching 144 files: 94% then EACCES). Only online revalidation takes locks; the offline resolve serves the cached snapshot and still fails loudly on a genuinely missing file. Serve flags unchanged." + - "Replace the HF_HUB_OFFLINE fallback with a complete-cached-snapshot fallback: Qwen/Qwen3.8-Flash-Next-FP8 changed upstream on 2026-08-31 (README and config revision) after this recipe merged, so every hf download now needs one new blob and dies on the foreign-owned .locks entry (run 34188408693: Fetching 144 files: 99% then PermissionError, and offline resolution then fails because the new revision is not fully cached). When the online download fails the script now serves the newest cached snapshot that has the tokenizer, the shard index, and every shard the index names, i.e. the weights this recipe was validated on; a cache with no complete snapshot still fails loudly. Serve flags unchanged." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2877 From 1a6698ff022b221e553d6f39cacce459fd57046b Mon Sep 17 00:00:00 2001 From: functionstackx <47992694+functionstackx@users.noreply.github.com> Date: Tue, 8 Sep 2026 01:00:32 -0400 Subject: [PATCH 5/5] Load the tokenizer from the same resolved path as the weights Co-Authored-By: Claude Fable 5.1 --- .../single_node/agentic/qwen3.8next_fp8_h200_sglang_mtp.sh | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/benchmarks/single_node/agentic/qwen3.8next_fp8_h200_sglang_mtp.sh b/benchmarks/single_node/agentic/qwen3.8next_fp8_h200_sglang_mtp.sh index 9b1f5470ea..a7a4fc7856 100755 --- a/benchmarks/single_node/agentic/qwen3.8next_fp8_h200_sglang_mtp.sh +++ b/benchmarks/single_node/agentic/qwen3.8next_fp8_h200_sglang_mtp.sh @@ -235,7 +235,10 @@ SGLANG_CMD=( --stream-interval 50 --scheduler-recv-interval "$SCHEDULER_RECV_INTERVAL" --tokenizer-worker-num 6 - --tokenizer-path "$MODEL" + # Same path as the weights: when the cached-snapshot fallback is in effect + # the HF id would send the tokenizer load back through the shared-cache + # lock this run just failed on. Identical to "$MODEL" otherwise. + --tokenizer-path "$MODEL_PATH" --enable-metrics "${CACHE_ARGS[@]}" )