Add B200 TP2 DSv4.1 Flash CPU offload / 新增 B200 TP2 DSv4.1 Flash CPU 卸载 - #3216
functionstackx wants to merge 16 commits into
Conversation
新增 B200 TP2 DeepSeek-V4.1-Flash Engram CPU 卸载的 8K/1K 和 AgentX 扫描，保留 TP4 基线并增加 tokenizer 支持。
|
Thanks for the contribution!
中文感谢你的贡献！
|
将 B200 TP2 扫描 changelog 关联到 PR 3216。
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Beyond the inline finding on the missing stop_gpu_monitor call, I also checked the CUDAGRAPH_CAPTURE_SIZE doubling loop for a potential infinite-loop if DSV41_MIN_CUDAGRAPH_CAPTURE_SIZE is "0" — that pattern (CAPTURE_SIZE=$((CAPTURE_SIZE * 2))) is copied from existing scripts (e.g. the deprecated minimaxm3 recipes) rather than introduced fresh here, so it's a pre-existing gap and not a new regression.
Extended reasoning...
The inline CONFIRMED finding (missing stop_gpu_monitor pairing, leaving the background nvidia-smi sampler running) is the substantive issue in this PR and already flags it for human attention via inline comment. I independently verified this by checking that every other active script in benchmarks/single_node/fixed_seq_len/ pairs start_gpu_monitor with stop_gpu_monitor, while the new dsv41flash_fp4_b200_vllm_mtp.sh only calls start_gpu_monitor. I also independently verified the ruled-out candidate about the CAPTURE_SIZE doubling loop: grepping for the same CAPTURE_SIZE * 2 pattern shows it already exists in several deprecated scripts, confirming it's a duplicated pre-existing gap rather than a new bug introduced by this PR, so it doesn't need separate emphasis. Given a CONFIRMED bug is already flagged inline, this should not be approved, and per the narrow exception in the guidelines I'm adding a short defer note recording what else was checked and ruled out, without restating the inline finding itself.
| SPEC_CONFIG='{"method":"dspark","num_speculative_tokens":5,"draft_sample_method":"probabilistic","rejection_sample_method":"synthetic","synthetic_acceptance_length":3.51,"enable_adaptive_verification":false}' | ||
| fi | ||
|
|
||
| start_gpu_monitor |
There was a problem hiding this comment.
🔴 New B200 TP2 fixed-seq-len recipe calls start_gpu_monitor at line 43 but never calls stop_gpu_monitor, unlike every other script in benchmarks/single_node/fixed_seq_len/ which pairs the two. The background nvidia-smi sampler (GPU_MONITOR_PID) is left running when the script exits normally, so it is never killed via the normal path, its CSV never gets the final one-shot sample or the truncated-tail repair that stop_gpu_monitor performs. Fix: add a stop_gpu_monitor call after the benchmark/eval step (matching run_benchmark_serving/run_eval calls above) so the monitor process is reaped and gpu_metrics.csv is finalized for every conc-list run in this new recipe.
Extended reasoning...
benchmark_lib.sh's start_gpu_monitor (line 321) forks 'nvidia-smi ... -l $interval > $output &' and only stop_gpu_monitor (line 369) kills GPU_MONITOR_PID, appends the final boundary sample needed for window interpolation, and repairs a truncated trailing CSV row. The new script at benchmarks/single_node/fixed_seq_len/dsv41flash_fp4_b200_vllm_mtp.sh calls start_gpu_monitor (line 43) then goes straight to wait_for_server_ready/run_benchmark_serving/run_eval and exits; no stop_gpu_monitor. Backgrounded child processes in bash are not killed when the parent script exits normally (no huponexit), so the nvidia-smi sampler keeps running and appending to gpu_metrics.csv indefinitely across every concurrency step in the [1,2,4,8,16,32,64,128] sweep for this recipe, and the CSV never receives the final sample or truncation repair, corrupting downstream power/energy analysis for this recipe's runs.
Verification: normal. The new recipe benchmarks/single_node/fixed_seq_len/dsv41flash_fp4_b200_vllm_mtp.sh calls start_gpu_monitor at line 43 and then goes straight to wait_for_server_ready (line 62), run_eval (line 65) or run_benchmark_serving (lines 67-74), and exits — with no stop_gpu_monitor anywhere. Grep count over benchmarks/single_node/fixed_seq_len confirms it is the ONLY script (active or deprecated)…
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=35387151990 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=35387151990 |
Append-only resolution of perf-changelog.yaml. 合并 origin/main；perf-changelog.yaml 以仅追加方式解决冲突。 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
TP2 c32-c128 died in memory profiling with -10.8 GiB of KV budget (run 35180394796). Cap batched tokens at 4096, bound max-num-seqs and stop graph capture at 512 tokens for TP2 only; TP4/TP8 keep the defaults. 仅对 TP2 设置批处理 token、max-num-seqs 与 graph 捕获上限。 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…repo root The workflow reads $RESULT_FILENAME.json from the repository root, which the dsv41flash launchers mount at /ix; run 35314631817 wrote it under RESULT_DIR. 固定序列结果写入容器内仓库根目录。 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…r repo root 为固定序列结果目录断言容器内仓库根目录。 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
c64 and c128 ended vLLM memory profiling with negative KV budget next to 145 GiB of weights per GPU (run 35316389982); the test expectation follows. 固定序列脚本中 TP2 的 graph 捕获上限设为 256，并同步测试预期。 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The hardcoded --max-model-len 1048576 made the 8k1k TP2 arm pay the 1M indexer buffer and full-context KV reservation (run 35316389982: 0.97 GiB of KV at c32, negative at c64/c128). Use MAX_MODEL_LEN, EVAL_MAX_MODEL_LEN for evals, and relax the TP2 capture cap to 512. 固定序列脚本改为服务矩阵提供的上下文长度。 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ontext Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…tp2-cpu-offload # Conflicts: # perf-changelog.yaml
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…tp2-cpu-offload # Conflicts: # perf-changelog.yaml
…pace Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
# Conflicts: # perf-changelog.yaml
The c128 eval ran ~125 GSM8K requests at once above the 256-token capture, and the eager DSpark draft lm_head GEMM died with cudaErrorIllegalAddress (run 35399984613). Pin lm-eval concurrency to 32 on TP2. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…P2 eval entry check-changelog in run 35403589805 rejected the misspelled key. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
1 participant
Description
Add B200 TP2 with
--engram-config '{"cpu_offload":true}'for fixed 8K/1K and AgentX, at concurrency 1, 2, 4, 8, 16, 32, 64, 128. Preserve the TP4 AgentX baseline and existing pinned vLLM image.Keep DSpark5, golden AL 3.51 for throughput, real block rejection with adaptive verification for evals, and GPU-resident KV. Add
deepseek_v41tokenizer support to the fixed-sequence client.Validation
22 focused local tests passed, along with Bash syntax, Ruff, matrix generation, and append-only changelog validation. GPU memory fit, correctness, and performance remain unverified.
Use
full-sweep-enabledas requested so matrix jobs continue after failures. Do not merge automatically. The corresponding upstream recipe update and normal review remain merge prerequisites.中文
新增 B200 TP2 Engram CPU 卸载的固定 8K/1K 和 AgentX 配方，并发为 1、2、4、8、16、32、64、128。保留 TP4 AgentX 基线和现有固定 vLLM 镜像。吞吐使用 DSpark5 和黄金 AL 3.51；eval 使用真实块拒绝采样和自适应验证，KV 保留在 GPU 上。固定序列客户端新增
deepseek_v41tokenizer 支持。22 项本地测试以及 Bash 语法、Ruff、矩阵生成和 changelog 追加验证通过。GPU 显存容量、正确性和性能仍待验证。按要求使用
full-sweep-enabled，失败后其他矩阵任务继续运行。不自动合并；合并前仍需对应的上游配方更新和正常审核。