[Klaud Cold] Update qwen3.8next-fp8-h200-sglang-agentic-mtp SGLang image to dev-cu13-qwen38-next-local (2026-09-07, digest-pinned)

[functionstackx]

Summary

Move the H200 Qwen3.8-Flash-Next FP8 AgentX MTP recipe from the mutable model-branch tag lmsysorg/sglang:qwen38flashnext to the newest Qwen3.8-Flash-Next model-branch build, lmsysorg/sglang:dev-cu13-qwen38-next-local@sha256:9d2a843c706c74bc259c0d9abf360551eb2734e1e7d255ab012a6965f10480b6, pinned by digest.

• Why not an upstream nightly: Qwen3.8-Flash-Next's architecture (Qwen4ExpForConditionalGeneration) is not in SGLang main, v0.5.19, or today's nightly nightly-dev-cu13-20260907-30705c00. It exists only on SGLang model branches shipped as dev images, so the model-branch lineage is the only one that can serve this recipe.
• Old tag: qwen38flashnext was last re-pushed 2026-09-03T20:54:46Z (digest sha256:5ae58167...), after the recipe merged on 2026-08-27, so nodes with an older squash cache may have been running a different build than nodes that pulled later. The digest pin removes that ambiguity.
• New image: dev-cu13-qwen38-next-local, Docker Hub last pushed 2026-09-07T11:12:42Z, digest sha256:9d2a843c706c74bc259c0d9abf360551eb2734e1e7d255ab012a6965f10480b6. Provenance is a model branch, not upstream main; this sweep is the validation that it still carries the fixes the recipe relies on (NEXTN MTP verify with float32 SSM state on sm_90).
• benchmarks/single_node/agentic/qwen3.8next_fp8_h200_sglang_mtp.sh is unchanged: TP4/EP4, mem-fraction 0.85, flashinfer attention, NEXTN MTP at 3 speculative tokens, golden AL 2.32, conc [1, 4, 8, 12, 16].

Recipes touched: qwen3.8next-fp8-h200-sglang-agentic-mtp

Test plan

• full-sweep-enabled sweep passes on cluster:h200-dgxc.

🤖 Generated with Claude Code

---

Note

Medium Risk
Benchmark reliability and reproducibility change (image pin plus model-path resolution); serve flags and search space are unchanged, but wrong snapshot selection could serve stale weights if validation logic were wrong.

Overview
Pins the qwen3.8next-fp8-h200-sglang-agentic-mtp recipe to digest lmsysorg/sglang:dev-cu13-qwen38-next-local@sha256:9d2a843c… instead of the mutable qwen38flashnext tag, and documents the change in perf-changelog.yaml.

H200 benchmark script: when MODEL_PATH is not preset, a failed online hf download (e.g. PermissionError on shared /mnt/hf_hub_cache lock files after an upstream repo revision) now selects the newest locally cached snapshot that has tokenizer, index, and all weight shards, sets MODEL_PATH to that directory, and still exits with an error if none qualify. --tokenizer-path is switched from the HF repo id to $MODEL_PATH so tokenizer load does not re-hit the same cache locks during snapshot fallback.

^{Reviewed by Cursor Bugbot for commit 1a6698f. Bugbot is set up for automated code reviews on this repo. Configure here.}

Update: shared-cache lock fallback

First sweep (run 34174941536) failed all four H200 cells before serving: hf download Qwen/Qwen3.8-Flash-Next-FP8 had 135/144 files cached and then hit PermissionError acquiring /mnt/hf_hub_cache/.locks/models--Qwen--Qwen3.8-Flash-Next-FP8/<etag>.lock. The lock files are owned by whichever runner account first fetched the blob (filelock creates them 0644), so a job on another runner account cannot open them; this is a cluster-cache permission issue, not the image (all three images run as root, and the launcher remaps root). The script now falls back to HF_HUB_OFFLINE=1 hf download when the online download fails, which resolves the cached snapshot without taking locks and still fails loudly on a genuinely missing file. Serve flags unchanged.

Update 2: complete-snapshot fallback

The offline fallback did not help (run 34188408693): the cache holds 143/144 files and the missing blob is the new README from the model repo's 2026-08-31 revision, the same blob whose lock cannot be created, so HF_HUB_OFFLINE had nothing to resolve. The script now serves the newest cached snapshot that is complete (tokenizer, shard index, and every shard present) when the online download fails, which is the weight set this recipe was validated on. A cache with no complete snapshot still fails loudly. The underlying issue is cluster-side: /mnt/hf_hub_cache/.locks/models--Qwen--Qwen3.8-Flash-Next-FP8/ is owned by another runner account and should be made group-writable.

[github-actions]

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs

---

感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have。

如需进行 PR 验证，请为此 PR 添加 full-sweep-fail-fast 标签（强烈推荐）— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled。

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

[claude]

Beyond the inline pr-link finding, I also checked whether the two sibling recipes (qwen3.8next-fp4-b300 and qwen3.8next-fp8-h100-sglang-agentic-mtp) needed the same image bump — the diff only touches the h200 recipe intentionally per the PR description, so that's not a gap here.

Extended reasoning...

A confirmed finding (invalid pr-link: PRLINK_PLACEHOLDER value in perf-changelog.yaml, which will fail utils/validate_perf_changelog.py's placeholder check) is being posted inline, so this run cannot approve. Beyond that, I independently verified via git diff that the nvidia-master.yaml change is exactly the single-line image bump described (line 7412), with no other fields touched. I also considered whether the two sibling qwen3.8next recipes (fp4-b300 at line 7240, fp8-h100 at line 7464) should have received the same digest pin, but the PR description explicitly scopes this change to only the h200 recipe, and there's no indication those siblings use the same mutable tag being replaced here, so that candidate is ruled out rather than a real gap.

[claude]

🔴 The new entry's pr-link value PRLINK_PLACEHOLDER is not a recognized placeholder: utils/validate_perf_changelog.py's validate_added_pr_link only accepts the canonical PR URL or XXX/.../pull/XXX, so this will fail the changelog validation gate on CI, blocking merge. Fix: use XXX (or https://github.com/SemiAnalysisAI/InferenceX/pull/XXX) as documented/enforced, then replace with the real PR URL once opened.

Extended reasoning...

utils/validate_perf_changelog.py:24-27 defines PR_LINK_PLACEHOLDERS = {"XXX", "https://github.com/SemiAnalysisAI/InferenceX/pull/XXX"}; validate_added_pr_link (lines 144-160) raises ChangelogValidationError if link is not in that set and not the exact expected .../pull/<pr_number> URL. The appended entry's pr-link: PRLINK_PLACEHOLDER (perf-changelog.yaml, new last line) matches neither, so any CI job or pre-merge tooling invoking this validator (e.g. the changelog gate tests directory utils/changelog_gate_tests) will reject the PR until the placeholder is corrected.

Verification: normal. The appended entry's pr-link: PRLINK_PLACEHOLDER (perf-changelog.yaml new final line) is not a recognized value. utils/validate_perf_changelog.py:24-27 defines PR_LINK_PLACEHOLDERS = {"XXX", "https://github.com/SemiAnalysisAI/InferenceX/pull/XXX"}; validate_added_pr_link (lines 144-160) raises ChangelogValidationError when the link is neither in that set nor the exact…

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34174941536
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=34174941536

[cursor]

Stale Bugbot comment from a previous run.

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34188408693
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=34188408693

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 1a6698f. Configure here.}

[cursor]

Snapshot check skips tokenizer files

Medium Severity

The cached-snapshot fallback treats tokenizer_config.json as proof the tokenizer is present, then points --tokenizer-path at that directory. A snapshot can pass this check and still lack the files AutoTokenizer needs, so serve fails after the download fallback already ran.

Additional Locations (1)

• benchmarks/single_node/agentic/qwen3.8next_fp8_h200_sglang_mtp.sh#L240-L241

 

^{Reviewed by Cursor Bugbot for commit 1a6698f. Configure here.}

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34198065264
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=34198065264