Conversation
|
Thanks for the contribution!
中文感谢你的贡献！
|
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
I reviewed this PR and didn't find any bugs. Because it adds a new entry to configs/nvidia-master.yaml, which is a CODEOWNER-protected file (owned by ankur-singh, kedarpotdar-nv, SemiAnalysisAI/core per .github/CODEOWNERS), a human sign-off from one of those owners is still needed as an additional step.
What was reviewed:
- The new recipe yaml and its matching
configs/nvidia-master.yamlentry —model.containerandimageboth pinlmsysorg/sglang:nightly-dev-cu13-20260914-4358a161, and the dynamo router revision matches in both places. - Golden acceptance-length lookup:
golden_al_distribution/qwen3.5_mtp.yamlhas a3(num-steps) entry for both thinking modes, so the recipe's NEXTN/3-step MTP config resolves to a real measured curve rather than failing or silently defaulting. perf-changelog.yamldiff — confirmed it only appends 9 lines at the tail; no prior bytes were touched (only diff-header---lines matched^-).- The
CONFIG_FILE=recipes/qwen3.5/sglang/...path used in the new master-configadditional-settingsentry matches the established path convention used by sibling qwen3.5/dsr1 entries in the same file.
Extended reasoning...
Overview
This PR adds a single new multi-node SGLang recipe (benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/b200-fp8/agentx/disagg-1p1d-p-tp4-d-tp4-mtp.yaml), a matching entry in configs/nvidia-master.yaml, and an append-only perf-changelog.yaml entry. It is a config-only addition (no code/script changes) describing a disaggregated 1P/1D (TP4/TP4) B200 FP8 AgentX benchmark for Qwen3.5-397B-A17B-FP8 with NEXTN MTP speculative decoding via Dynamo+SGLang.
Security risks
None. There is no code execution path introduced here — these are declarative slurm/recipe YAML configs consumed by existing, unmodified tooling (srtctl, apply_srt_recipe, synthetic_acceptance.py). No secrets, no auth/permission logic, no user input handling.
Level of scrutiny
This is a mechanical, additive change following an extremely well-established pattern (dozens of near-identical sibling entries exist for other model/hardware/precision combinations in the same files). I verified the cross-file consistency requirements called out in AGENTS.md: container/image pinning matches, the perf-changelog append is byte-preserving, and the golden AL curve the recipe implicitly depends on (keyed by model-prefix: qwen3.5 + method mtp + num_speculative_tokens: 3) actually exists and has values for the configured MTP depth, so apply_srt_recipe/synthetic_acceptance.py won't silently omit or invent an acceptance length. Nothing found that would need a code-level fix. However, configs/nvidia-master.yaml is one of only two files in the whole repo with narrower CODEOWNERS than the default @ SemiAnalysisAI/core catch-all, which per the reviewing guidelines means an automated approval is not appropriate regardless of how mechanical the change looks — a named owner should still sign off (GitHub will already enforce this via required reviews, but it's worth surfacing explicitly).
Other factors
The PR description is candid that this is an unvalidated "barebones draft for steering" — GPU qualification, full sweep, and accuracy evals are explicitly still pending, and no GPU benchmark has been dispatched (blocked on a DNS issue accessing the B200 cluster). That doesn't create a correctness bug in the diff itself, but it reinforces that this is exactly the kind of change where a domain-expert human (who can judge whether the topology/flags are sane for this new model/precision/engine combination before it gets GPU time) adds real value beyond what static review can catch.
…8-dynamo-agentx # Conflicts: # perf-changelog.yaml
cquil11
changed the title
cquil11
added
all-evals
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=35404276619 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=35404276619 |
cquil11
changed the title
|
/use 35404276619 |
1 participant
Adds Qwen3.5-397B-A17B FP8 disaggregated AgentX recipes for B200 using Dynamo + SGLang through pinned srt-slurm. The matrix covers concurrency 8, 16, 24, 32, 48, and 64 with one TP4 prefill worker and one TP4 decode worker colocated on eight GPUs. C8–C32 use 104 GB of prefill HiCache per rank. C48 and C64 use 215 GB per rank with first-use write-through backup and a decode static-memory fraction of 0.88. Both use Mooncake intra-node NVLink transfer. Paired B300 PR: #3268.
The serving image is
lmsysorg/sglang:nightly-dev-cu13-20260918-20518d85, Dynamo is pinned to805a77f053d807b0d8def5d27f674a6df0ed839e, and srt-slurm to984180e5b8755aef85e9995048b5a16cb5336bce. Native NEXTN MTP settings are explicit; InferenceX selects measured synthetic acceptance automatically for throughput and removes simulation for real-verification evals.Only new disaggregated points are proposed for publication. Existing aggregate curve scopes remain unchanged: publishing a sparse replacement aggregate sweep would discard other winning points under the dashboard's latest-run selection. Same-image aggregate and 2P1D experiments are comparison evidence.
Performance qualification
All six canonical 3,600-second profiles have completed measurement. Throughput counts logical input plus output tokens, including cached input, divided by all eight active prefill/decode GPUs; each job also allocates eight GPUs. Interactivity is reciprocal p90 full-response ITL.
Every profile has valid submission metadata and passing duration coverage. C24's two drain cancellations are retained in its accounting; it has no client request errors. All six independent native power audits and actual strict CI power checks pass, covering all eight GPUs. All six raw/CI artifact reconciliations pass. Each paired real-verification GSM8K evaluation used all 1,319 examples with five-shot prompting and no sample limit; strict accuracy ranges from 97.4223% to 97.9530%, above the 94% threshold. The official sweep completed successfully on the preserved benchmark commit, including a verified six-point combined eval artifact. C8's dependency-startup stall required a same-head failed-only retry through the dashboard after the other eleven GPU jobs and their artifacts were safe; its retry passed at 97.5739% accuracy.
The published database was refreshed read-only at 2026-09-19 02:42 UTC: all 33 B200 FP8 AgentX measurements were unchanged. All six full-hour PD measurements are mutually nondominated and extend that discrete published frontier; no interpolation or rounded cutoff is used for eligibility. Adding them while retaining the existing aggregate scopes would expand the primary frontier from 14 to 16 points:
Same-image aggregate controls also completed canonical full-hour profiles with valid power, real accuracy, and actual CI artifacts: TP4 C4 reached 6,779.70/GPU at 309.423 interactivity; TP4 C32 HiCache reached 38,171.39 at 73.904; TP4 C64 HiCache reached 48,347.74 at 51.648 (p90 TTFT 16.489s). These controls use four active GPUs within an eight-GPU allocation; comparing their active-GPU-normalized throughput favors aggregate and still does not dominate any selected PD point. They remain comparison evidence because a sparse aggregate publication would replace the existing aggregate scope.
Short aggregate and 2P1D experiments informed the selection. Tested 2P1D points did not improve the normalized frontier and are excluded; the 2P1D C32 result (14,511.73/GPU at 228.969) is dominated by the full-hour 1P1D C24 result above. Short-run measurements do not qualify publication. All six selected points now pass the full-duration, request-validity, power, paired accuracy, actual CI and published-frontier checks on the preserved benchmark commit.
Implementation and validation
The existing B200 pool launcher supports compute-node-local checkpoint paths while retaining worker loading checks. It now verifies terminal Slurm allocation status and stages diagnostic/eval artifacts before returning failure; an aborted Slurm job cannot appear successful merely by disappearing from the queue. Required native SRT telemetry captures all eight GPU roles and the profiling window. Each fixed-concurrency recipe explicitly supplies the custom measurement type, exact concurrency and result root; the native producer supplies the window directory. The launcher joins the finalized power package through the existing strict AgentX adapter before archiving; missing or invalid power fails the job while preserving diagnostic artifacts. Eval-only jobs skip the power gate. The shared log-stream helper retains its existing behavior for other callers. NVIDIA dependencies use the official vendor wheel index with integrity verification retained.
Exact config generation, native pinned SRT recipe validation, shell syntax, strict whole-PR changelog/all-evals planning, and focused launcher/synthetic-acceptance/cluster-configuration tests pass. Eight launcher integration cases exercise success, failure, signal termination, cancellation, delayed/missing accounting, required-power failure with retained artifacts, and eval-only handling against the real launcher. Real running-window publication succeeds with each of the six actual recipe environments against the native directory/mount fixture. Exactly one PR changelog entry covers the complete final scope; published
mainchangelog bytes are preserved.unsafe_overridesubmission invalidity and is not throughput qualification; official runs use 3,600 seconds.The expanded official
run-sweepsucceeded in attempt 2 on0b75e138be098de0122959460c765860f79b95fdwithfull-sweep-enabledandall-evals: six canonical 3,600-second throughput profiles and six full real-verification evals. The final eval collector also succeeded; its neweval_results_allartifact contains exactly C8, C16, C24, C32, C48 and C64. All twelve GPU jobs are successful, with exact-head provenance and archived artifacts verified.The current head
c8480f36c6bac6fcf7af9d1c30651116bf5ca012merges main to resolve its append-only changelog conflict and retains the tested0b75e138commit in this PR. Main’s historical changelog bytes and this PR’s single entry are preserved. The B200 recipes, launcher and shared benchmark code are unchanged; the complete six-throughput/six-eval matrix is identical. Changelog validation and 139 focused tests pass. The repository accepted reuse of the successful sweep; synchronization validation passed and skipped all duplicate GPU jobs. CI passed lint and 1,798 tests (one skipped).AI model disclosure
Prepared with Codex, including delegated source and performance analysis. The runtime does not expose a verifiable exact model/version identifier.