Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion benchmarks/multi_node/srt-slurm-recipes/RECIPES.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,12 @@

InferenceX owns the recipes in this directory. Every NVIDIA srt-slurm launcher uses `setup_srt_slurm()` in [`runners/slurm_utils.sh`](../../../runners/slurm_utils.sh), makes a job-local Git clone of the pinned submodule, and copies this entire tree into `recipes/`. The shared helper records the actual revision in `srt-slurm-sha.txt`; power lanes copy that revision into `power-producer-sha.txt` for result validation.

The shared version is the Git submodule pointer at [`utils/srt-slurm`](../../../utils/srt-slurm), currently [v2.2.1](https://github.com/NVIDIA/srt-slurm/releases/tag/v2.2.1) (`984180e5b8755aef85e9995048b5a16cb5336bce`). Update that submodule pointer when upgrading, then run the recipe and integration checks. Do not add model-specific checkout branches to launchers.
The shared version is the Git submodule pointer at [`utils/srt-slurm`](../../../utils/srt-slurm), currently [v2.11.0](https://github.com/NVIDIA/srt-slurm/releases/tag/v2.11.0) (`8939ac513c2646a301dfef46971ff8bb48a9d6ff`). Update that submodule pointer when upgrading, then run the recipe and integration checks. Do not add model-specific checkout branches to launchers.

InferenceX requires srt-slurm 2.0 or newer and `schema: 2` recipes. Legacy recipe layouts are unsupported; migrate them before adding them to this tree.

The MiniMax-M3 Mooncake setup installs its published wheel with `--no-deps` and `--require-hashes`. It clears `UV_OVERRIDE` for that command only so unrelated image-wide package overrides do not enter the wheel's hash check. Other package installers retain the image's overrides.

## Directory and filename convention

Store every recipe at `<model-prefix>/<engine>/<gpu>-<precision>/<workload>/<recipe>.yaml`:
Expand Down
4 changes: 3 additions & 1 deletion benchmarks/multi_node/srt-slurm-recipes/RECIPES_zh.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,12 @@

InferenceX 负责维护本目录中的配置。所有 NVIDIA srt-slurm 启动器均调用 [`runners/slurm_utils.sh`](../../../runners/slurm_utils.sh) 中的 `setup_srt_slurm()`,为作业创建固定版本子模块的本地 Git 克隆,并将整个目录复制到 `recipes/`。共享函数将实际提交记录到 `srt-slurm-sha.txt`;功耗测试路径还会将其复制到 `power-producer-sha.txt`,供结果校验使用。

统一版本由 [`utils/srt-slurm`](../../../utils/srt-slurm) 的 Git 子模块指针指定,目前为 [v2.2.1](https://github.com/NVIDIA/srt-slurm/releases/tag/v2.2.1)(`984180e5b8755aef85e9995048b5a16cb5336bce`)。升级时更新该子模块指针,然后运行配置和集成检查。不要在启动器中新增按模型选择检出版本的分支。
统一版本由 [`utils/srt-slurm`](../../../utils/srt-slurm) 的 Git 子模块指针指定,目前为 [v2.11.0](https://github.com/NVIDIA/srt-slurm/releases/tag/v2.11.0)(`8939ac513c2646a301dfef46971ff8bb48a9d6ff`)。升级时更新该子模块指针,然后运行配置和集成检查。不要在启动器中新增按模型选择检出版本的分支。

InferenceX 要求 srt-slurm 2.0 或更新版本,且配置必须声明 `schema: 2`。不支持旧版配置结构;加入本目录前必须先完成迁移。

MiniMax-M3 的 Mooncake 安装脚本使用 `--no-deps` 和 `--require-hashes` 安装已发布的 wheel。仅对该命令清除 `UV_OVERRIDE`,避免镜像中无关的软件包版本覆盖项进入 wheel 的哈希校验。其他安装命令仍使用镜像的版本覆盖配置。

## 目录和文件命名规范

所有配置统一存放在 `<model-prefix>/<engine>/<gpu>-<precision>/<workload>/<recipe>.yaml`:
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
#!/usr/bin/env bash
set -eo pipefail
# Download, verify, and install the published wheel inside each backend container.
# Image-wide overrides are unrelated to this dependency-free wheel install.
env -u UV_OVERRIDE uv pip install --system --no-deps --reinstall --require-hashes \
'https://test-files.pythonhosted.org/packages/f3/00/fb2847f5564864be132d0f70384dde5688689d9f91b2c81d2609632f36e4/mooncake_transfer_engine_cuda13-0.3.14.dev20260910-cp312-cp312-manylinux_2_28_aarch64.whl#sha256=c55fcc42cf189fcdecdcdf9d776ef84b348a9970a9bd3aad2fad64cb2d9e6f82'

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 (optional) All four disaggregated GB300 MiniMax-M3 recipes fail their setup step once this wheel is pruned, since it is fetched from test-files.pythonhosted.org, TestPyPI's sandbox file host, not the production files.pythonhosted.org used elsewhere in this repo (e.g. benchmarks/benchmark_lib.sh:1457 for bfcl_eval). TestPyPI explicitly does not guarantee permanence of uploaded files, unlike PyPI proper. Fix: host the Mooncake wheel on a durable, project-controlled or production PyPI-backed location (or vendor it) so disagg jobs do not depend on a testing sandbox that can delete the file at any time.

Extended reasoning...

setup-disagg.sh runs uv pip install --require-hashes 'https://test-files.pythonhosted.org/packages/.../mooncake_transfer_engine_cuda13-0.3.14.dev20260910-...whl#sha256=...' inside every prefill/decode container for the four disagg recipes (dep4-tep4-c64, dep4-tp2-c90/c100/c110). test-files.pythonhosted.org backs test.pypi.org, which is documented as a sandbox with no durability guarantee for uploaded files. If the upload is removed or expires, the download 404s, set -eo pipefail aborts the script, and every disagg job using this setup script fails at startup with no fallback. Compare to benchmarks/benchmark_lib.sh:1457 which pins bfcl_eval from the production files.pythonhosted.org host for the same durability reason.

Verification: nit. Verified: benchmarks/multi_node/srt-slurm-recipes/configs/minimaxm3-mooncake/setup-disagg.sh:5 fetches the Mooncake wheel from https://test-files.pythonhosted.org/packages/.../mooncake_transfer_engine_cuda13-0.3.14.dev20260910-...whl#sha256=.... test-files.pythonhosted.org is TestPyPI's sandbox file host, not the production files.pythonhosted.org used everywhere else in the repo (e.g.…

Original file line number Diff line number Diff line change
@@ -0,0 +1,136 @@
schema: 2
name: minimax-m3-gb300-agg-tp2-c20
dynamo:
install: true
source:
wheel: 1.5.0.dev20260914
request_plane: tcp
services:
- name: etcd
type: etcd
placement:
node: infra
- name: nats
type: nats
placement:
node: infra
options:
max_payload_mb: 32
frontend:
type: dynamo
enable_multiple_frontends: false
args:
router-mode: least-loaded
router-session-affinity-ttl-secs: 14400
env:
DYN_TOKENIZER: fastokens
DYN_TOKENIZER_CACHE_BYTES: '8589934592'
DYN_TCP_CONNECT_TIMEOUT: '120'
model:
path: nvidia/MiniMax-M3-NVFP4
container: vllm/vllm-openai:nightly-0bfc7a15d095fe83ecc82b50561a93c177fece2d@sha256:ac75d722e093a1d6b1515fdb0d9cff692225e47e8bcbaf614a1a220d302fa116
precision: fp4
identity:
model:
repo: nvidia/MiniMax-M3-NVFP4
container:
image: vllm/vllm-openai:nightly-0bfc7a15d095fe83ecc82b50561a93c177fece2d@sha256:ac75d722e093a1d6b1515fdb0d9cff692225e47e8bcbaf614a1a220d302fa116
frameworks:
dynamo: 1.5.0.dev20260914
resources:
gpu_type: gb300
gpus_per_node: 4
engine:
type: vllm
connector: null
dp_launch_mode: per_gpu
roles:
agg:
nodes: 1
workers: 1
gpus: 2
env:
VLLM_ENGINE_READY_TIMEOUT_S: '3600'
DYN_TCP_CONNECT_TIMEOUT: '120'
VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS: '1800'
VLLM_FLOAT32_MATMUL_PRECISION: high
VLLM_FLASHINFER_ALLREDUCE_BACKEND: trtllm
VLLM_USE_NCCL_SYMM_MEM: '0'
VLLM_ALLREDUCE_USE_SYMM_MEM: '0'
VLLM_USE_SIMPLE_KV_OFFLOAD: '1'
VLLM_LOG_STATS_INTERVAL: '1'
NCCL_CUMEM_ENABLE: '1'
NCCL_MNNVL_ENABLE: '1'
NCCL_NVLS_ENABLE: '1'
HF_HUB_CACHE: /hf_hub_cache
HUGGINGFACE_HUB_CACHE: /hf_hub_cache
args:
served-model-name: nvidia/MiniMax-M3-NVFP4
tensor-parallel-size: 2
pipeline-parallel-size: 1
trust-remote-code: true
enable-prefix-caching: true
kv-cache-dtype: fp8
block-size: 128
gpu-memory-utilization: 0.9
max-model-len: 1048576
language-model-only: true
no-enable-flashinfer-autotune: true
attention-config: '{"backend":"FLASHINFER","use_trtllm_attention":true,"indexer_kv_dtype":"fp8"}'
speculative-config: '{"method":"eagle3","model":"Inferact/MiniMax-M3-EAGLE3-GQA","num_speculative_tokens":3,"attention_backend":"FLASHINFER"}'
kv-transfer-config: '{"kv_connector":"SimpleCPUOffloadConnector","kv_role":"kv_both","kv_connector_extra_config":{"cpu_bytes_to_use":429496729600,"cpu_bytes_to_use_per_rank":214748364800,"lazy_offload":true}}'
max-cudagraph-capture-size: 512
max-num-batched-tokens: 16384
stream-interval: 20
reasoning-parser: minimax_m3
dyn-tool-call-parser: minimax_m3
dyn-reasoning-parser: minimax_m3
dyn-default-thinking-mode: enabled
benchmark:
type: custom
use_chat_template: true
command: bash /infmax-workspace/benchmarks/multi_node/agentic_srt.sh
env:
AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID: 'true'
AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING: '0'
AIPERF_DYNAMO_SESSION_TIMEOUT_SECONDS: '14400'
ENABLE_AGENTX_POWER: '0'
AIPERF_EXTRA_INPUTS: thinking:true
AIPERF_REQUIRED_SERVER_METRIC_PREFIX: 'vllm:'
AIPERF_DATASET_MMAP_CACHE_DIR: /aiperf_mmap_cache
AIPERF_SERVER_METRICS_COLLECTION_INTERVAL: '1.0'
AIPERF_WARMUP_REQUESTS_PER_LANE: '10'
AIPERF_TRACE_IDLE_GAP_CAP_SECONDS: '300'
AIPERF_SERVICE_PROFILE_CONFIGURE_TIMEOUT: '1800'
AIPERF_DATASET_CONFIGURATION_TIMEOUT: '1800'
AGENTIC_WARMUP_GRACE_PERIOD: '1800'
AIPERF_DRAIN_TIMEOUT_SECONDS: '1800'
AIPERF_TIMING_CANCEL_DRAIN_TIMEOUT: '300'
INFMAX_CONTAINER_WORKSPACE: /infmax-workspace
RESULT_DIR: /logs/agentic
PORT: '8000'
TP: '2'
EP_SIZE: '1'
IS_MULTINODE: 'false'
DISAGG: 'false'
HF_HUB_CACHE: /hf_hub_cache
WEKA_LOADER_OVERRIDE: semianalysis_cc_traces_weka_062126
observability:
tachometer:
enabled: false
telemetry:
enabled: false
health_check:
max_attempts: 2160
interval_seconds: 10
sbatch_directives:
cpus-per-task: '144'
mem: '0'
srun_options:
container-remap-root: ''
cpu-bind: cores
slurm:
time_limit: '8:00:00'
environment:
PYTHONHASHSEED: '0'
WITH_NVIDIA_PEERMEM: '0'
Original file line number Diff line number Diff line change
@@ -0,0 +1,136 @@
schema: 2
name: minimax-m3-gb300-agg-tp2-c24
dynamo:
install: true
source:
wheel: 1.5.0.dev20260914
request_plane: tcp
services:
- name: etcd
type: etcd
placement:
node: infra
- name: nats
type: nats
placement:
node: infra
options:
max_payload_mb: 32
frontend:
type: dynamo
enable_multiple_frontends: false
args:
router-mode: least-loaded
router-session-affinity-ttl-secs: 14400
env:
DYN_TOKENIZER: fastokens
DYN_TOKENIZER_CACHE_BYTES: '8589934592'
DYN_TCP_CONNECT_TIMEOUT: '120'
model:
path: nvidia/MiniMax-M3-NVFP4
container: vllm/vllm-openai:nightly-0bfc7a15d095fe83ecc82b50561a93c177fece2d@sha256:ac75d722e093a1d6b1515fdb0d9cff692225e47e8bcbaf614a1a220d302fa116
precision: fp4
identity:
model:
repo: nvidia/MiniMax-M3-NVFP4
container:
image: vllm/vllm-openai:nightly-0bfc7a15d095fe83ecc82b50561a93c177fece2d@sha256:ac75d722e093a1d6b1515fdb0d9cff692225e47e8bcbaf614a1a220d302fa116
frameworks:
dynamo: 1.5.0.dev20260914
resources:
gpu_type: gb300
gpus_per_node: 4
engine:
type: vllm
connector: null
dp_launch_mode: per_gpu
roles:
agg:
nodes: 1
workers: 1
gpus: 2
env:
VLLM_ENGINE_READY_TIMEOUT_S: '3600'
DYN_TCP_CONNECT_TIMEOUT: '120'
VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS: '1800'
VLLM_FLOAT32_MATMUL_PRECISION: high
VLLM_FLASHINFER_ALLREDUCE_BACKEND: trtllm
VLLM_USE_NCCL_SYMM_MEM: '0'
VLLM_ALLREDUCE_USE_SYMM_MEM: '0'
VLLM_USE_SIMPLE_KV_OFFLOAD: '1'
VLLM_LOG_STATS_INTERVAL: '1'
NCCL_CUMEM_ENABLE: '1'
NCCL_MNNVL_ENABLE: '1'
NCCL_NVLS_ENABLE: '1'
HF_HUB_CACHE: /hf_hub_cache
HUGGINGFACE_HUB_CACHE: /hf_hub_cache
args:
served-model-name: nvidia/MiniMax-M3-NVFP4
tensor-parallel-size: 2
pipeline-parallel-size: 1
trust-remote-code: true
enable-prefix-caching: true
kv-cache-dtype: fp8
block-size: 128
gpu-memory-utilization: 0.9
max-model-len: 1048576
language-model-only: true
no-enable-flashinfer-autotune: true
attention-config: '{"backend":"FLASHINFER","use_trtllm_attention":true,"indexer_kv_dtype":"fp8"}'
speculative-config: '{"method":"eagle3","model":"Inferact/MiniMax-M3-EAGLE3-GQA","num_speculative_tokens":3,"attention_backend":"FLASHINFER"}'
kv-transfer-config: '{"kv_connector":"SimpleCPUOffloadConnector","kv_role":"kv_both","kv_connector_extra_config":{"cpu_bytes_to_use":429496729600,"cpu_bytes_to_use_per_rank":214748364800,"lazy_offload":true}}'
max-cudagraph-capture-size: 512
max-num-batched-tokens: 16384
stream-interval: 20
reasoning-parser: minimax_m3
dyn-tool-call-parser: minimax_m3
dyn-reasoning-parser: minimax_m3
dyn-default-thinking-mode: enabled
benchmark:
type: custom
use_chat_template: true
command: bash /infmax-workspace/benchmarks/multi_node/agentic_srt.sh
env:
AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID: 'true'
AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING: '0'
AIPERF_DYNAMO_SESSION_TIMEOUT_SECONDS: '14400'
ENABLE_AGENTX_POWER: '0'
AIPERF_EXTRA_INPUTS: thinking:true
AIPERF_REQUIRED_SERVER_METRIC_PREFIX: 'vllm:'
AIPERF_DATASET_MMAP_CACHE_DIR: /aiperf_mmap_cache
AIPERF_SERVER_METRICS_COLLECTION_INTERVAL: '1.0'
AIPERF_WARMUP_REQUESTS_PER_LANE: '10'
AIPERF_TRACE_IDLE_GAP_CAP_SECONDS: '300'
AIPERF_SERVICE_PROFILE_CONFIGURE_TIMEOUT: '1800'
AIPERF_DATASET_CONFIGURATION_TIMEOUT: '1800'
AGENTIC_WARMUP_GRACE_PERIOD: '1800'
AIPERF_DRAIN_TIMEOUT_SECONDS: '1800'
AIPERF_TIMING_CANCEL_DRAIN_TIMEOUT: '300'
INFMAX_CONTAINER_WORKSPACE: /infmax-workspace
RESULT_DIR: /logs/agentic
PORT: '8000'
TP: '2'
EP_SIZE: '1'
IS_MULTINODE: 'false'
DISAGG: 'false'
HF_HUB_CACHE: /hf_hub_cache
WEKA_LOADER_OVERRIDE: semianalysis_cc_traces_weka_062126
observability:
tachometer:
enabled: false
telemetry:
enabled: false
health_check:
max_attempts: 2160
interval_seconds: 10
sbatch_directives:
cpus-per-task: '144'
mem: '0'
srun_options:
container-remap-root: ''
cpu-bind: cores
slurm:
time_limit: '8:00:00'
environment:
PYTHONHASHSEED: '0'
WITH_NVIDIA_PEERMEM: '0'
Loading
Loading