[Klaud Cold] Update qwen3.8next-fp8-h200-sglang-agentic-mtp SGLang image to dev-cu13-qwen38-next-local (2026-09-07, digest-pinned)

benchmarks/single_node/agentic/qwen3.8next_fp8_h200_sglang_mtp.sh

@@ -50,8 +50,46 @@ if [[ -n "${MODEL_PATH:-}" ]]; then
         hf download "$MODEL" --local-dir "$MODEL_PATH"
     fi
 else
-    hf download "$MODEL"
-    export MODEL_PATH="$MODEL"
+    # The shared /mnt/hf_hub_cache on cluster:h200-dgxc holds lock files under
+    # .locks/<repo>/ owned by whichever runner account first downloaded a blob;
+    # filelock creates them 0644, so a job on a different runner account gets
+    # EACCES opening the lock. Qwen/Qwen3.8-Flash-Next-FP8 changed upstream on
+    # 2026-08-31 (a README and config revision) after this recipe merged, so
+    # every `hf download` now wants one new blob and dies on that lock (runs
+    # 34174941536 and 34188408693: "Fetching 144 files: 99%" then
+    # PermissionError; HF_HUB_OFFLINE cannot help because the new revision is
+    # not fully cached). When the online download fails, serve the newest
+    # cached snapshot that is complete -- tokenizer, shard index, and every
+    # shard the index names -- which is the same set of weights this recipe was
+    # validated on. A cache with no complete snapshot still fails loudly.
+    if hf download "$MODEL"; then
+        export MODEL_PATH="$MODEL"
+    else
+        echo "hf download failed online (shared-cache lock permissions?); looking for a complete cached snapshot of $MODEL"
+        HF_SNAPSHOT_ROOT="${HF_HUB_CACHE:-$HOME/.cache/huggingface/hub}/models--${MODEL//\//--}/snapshots"
+        SNAPSHOT=""
+        for d in $(ls -1dt "$HF_SNAPSHOT_ROOT"/*/ 2>/dev/null); do
+            if [[ -f "$d/config.json" && -f "$d/tokenizer_config.json" && -f "$d/model.safetensors.index.json" ]] \
+               && CKPT_DIR="$d" python3 - <<'PYEOF'
+import json, os, sys
+d = os.environ["CKPT_DIR"]
+with open(os.path.join(d, "model.safetensors.index.json")) as fh:
+    shards = sorted(set(json.load(fh)["weight_map"].values()))
+missing = [s for s in shards if not os.path.isfile(os.path.join(d, s))]
+sys.exit(1 if missing else 0)
+PYEOF
+            then
+                SNAPSHOT="${d%/}"
+                break
+            fi
+        done
+        if [[ -z "$SNAPSHOT" ]]; then
+            echo "Error: no complete cached snapshot of $MODEL under $HF_SNAPSHOT_ROOT and the online download failed." >&2
+            exit 1
+        fi
+        echo "Serving cached snapshot $SNAPSHOT"
+        export MODEL_PATH="$SNAPSHOT"
+    fi
 fi
 nvidia-smi
 
@@ -197,7 +235,10 @@ SGLANG_CMD=(
     --stream-interval 50
     --scheduler-recv-interval "$SCHEDULER_RECV_INTERVAL"
     --tokenizer-worker-num 6
-    --tokenizer-path "$MODEL"
+    # Same path as the weights: when the cached-snapshot fallback is in effect
+    # the HF id would send the tokenizer load back through the shared-cache
+    # lock this run just failed on. Identical to "$MODEL" otherwise.
+    --tokenizer-path "$MODEL_PATH"
     --enable-metrics
     "${CACHE_ARGS[@]}"
 )

configs/nvidia-master.yaml

@@ -7409,7 +7409,7 @@ qwen3.5-fp8-h200-sglang-agentic-mtp:
 # low-latency single-node command, which shards the 512-expert MoE with expert
 # parallelism rather than sharding attention eight ways.
 qwen3.8next-fp8-h200-sglang-agentic-mtp:
-  image: lmsysorg/sglang:qwen38flashnext
+  image: lmsysorg/sglang:dev-cu13-qwen38-next-local@sha256:9d2a843c706c74bc259c0d9abf360551eb2734e1e7d255ab012a6965f10480b6
   model: Qwen/Qwen3.8-Flash-Next-FP8
   model-prefix: qwen3.8next
   runner: cluster:h200-dgxc

perf-changelog.yaml

@@ -6947,3 +6947,13 @@
     - "Update vLLM ROCm image from vllm/vllm-openai-rocm:v0.27.1 (v0.27.1 release) to vllm/vllm-openai-rocm:nightly-d9105ea8001e0a6d77a96327d17515bb5791fb36 (2026-09-07 upstream ROCm nightly, digest sha256:74d4a95f3ae672ecddf9acb7296917def82d9eca51687fa2862ae72b03ff1907, tag commit vllm-project/vllm@d9105ea8; Docker Hub last pushed 2026-09-07T05:26:48Z). benchmarks/single_node/agentic/minimaxm3_fp8_mi300x_mtp.sh is unchanged: TRITON_ATTN attention, fp8 KV, block-size 128, EAGLE3 speculative decoding with the Inferact MiniMax-M3 EAGLE3-GQA draft and the committed golden synthetic acceptance length, minimax_m3 tool-call and reasoning parsers; the search space is unchanged. The upstream commit-pinned ROCm nightly is the same vllm commit the B200 MiniMax-M3 AgentX recipe moved to in #2860. Note that vllm-openai-rocm commit-nightly tags have expired from Docker Hub within days in the past; node squash caches keep merged configs running, but a re-pin to a durable tag may be needed later."
     - "Add --compilation-config cudagraph_mode=FULL_DECODE_ONLY to the serve command. The upstream nightly does not torch-compile MiniMaxM3SparseForConditionalGeneration, so with VLLM_USE_BREAKABLE_CUDAGRAPH=0 the default FULL_AND_PIECEWISE mode aborts at engine init with piecewise CUDA graphs unavailable (first sweep, run 34174124043, eval cell); full decode-only graphs are what the MI355X MiniMax-M3 sibling runs on its nightly (#2825)."
   pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2873
+
+- config-keys:
+    - qwen3.8next-fp8-h200-sglang-agentic-mtp
+  scenario-type:
+    - agentic-coding
+  description:
+    - "Move the image from the mutable model-branch tag lmsysorg/sglang:qwen38flashnext (last re-pushed 2026-09-03T20:54:46Z, digest sha256:5ae5816783d58e2e56e84d2e863f5441425056f500b7fbd7448c4aae017a2521) to the newest Qwen3.8-Flash-Next model-branch build, lmsysorg/sglang:dev-cu13-qwen38-next-local@sha256:9d2a843c706c74bc259c0d9abf360551eb2734e1e7d255ab012a6965f10480b6 (Docker Hub last pushed 2026-09-07T11:12:42Z), pinned by digest so every node runs the same build. Qwen3.8-Flash-Next (architecture Qwen4ExpForConditionalGeneration) is not in SGLang main, v0.5.19, or the 2026-09-07 nightly (nightly-dev-cu13-20260907-30705c00), so no upstream nightly can serve it; the model-branch dev images are the only lineage. benchmarks/single_node/agentic/qwen3.8next_fp8_h200_sglang_mtp.sh is unchanged: TP4/EP4, mem-fraction 0.85, flashinfer attention (sm_90), native NEXTN MTP at three speculative tokens, float32 Mamba SSM state, golden thinking_on acceptance length 2.32; the conc [1, 4, 8, 12, 16] grid is unchanged. The sweep is the validation that this branch build still carries the fixes the recipe relies on."
+    - "Fall back to HF_HUB_OFFLINE=1 hf download when the online download fails: the shared /mnt/hf_hub_cache on cluster:h200-dgxc carries .locks/<repo>/ lock files owned by whichever runner account first fetched a blob (filelock creates them 0644), so a job on another runner account hits PermissionError acquiring the lock even with every file already cached (run 34174941536, four cells, Fetching 144 files: 94% then EACCES). Only online revalidation takes locks; the offline resolve serves the cached snapshot and still fails loudly on a genuinely missing file. Serve flags unchanged."
+    - "Replace the HF_HUB_OFFLINE fallback with a complete-cached-snapshot fallback: Qwen/Qwen3.8-Flash-Next-FP8 changed upstream on 2026-08-31 (README and config revision) after this recipe merged, so every hf download now needs one new blob and dies on the foreign-owned .locks entry (run 34188408693: Fetching 144 files: 99% then PermissionError, and offline resolution then fails because the new revision is not fully cached). When the online download fails the script now serves the newest cached snapshot that has the tokenizer, the shard index, and every shard the index names, i.e. the weights this recipe was validated on; a cache with no complete snapshot still fails loudly. Serve flags unchanged."
+  pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2877