From 064157572782382df4a960ae88836341d2032c65 Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Sun, 9 Aug 2026 19:01:47 -0500 Subject: [PATCH 01/67] docs(amd): define srt-slurm bring-up contract Document the MI300X aggregate and disaggregated validation plan, cluster assumptions, safety boundaries, and current srt-slurm development pin. --- .../srt-slurm-recipes/AMD_BRINGUP.md | 74 +++++++++++++++++++ 1 file changed, 74 insertions(+) create mode 100644 benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md diff --git a/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md b/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md new file mode 100644 index 0000000000..a5ffa4e470 --- /dev/null +++ b/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md @@ -0,0 +1,74 @@ +# AMD srt-slurm bring-up + +This document tracks the work-in-progress integration of +[`SemiAnalysisAI/srt-slurm`](https://github.com/SemiAnalysisAI/srt-slurm) with +InferenceX AMD Slurm clusters. The project is a functional orchestration +bring-up, not a performance-tuning exercise. + +Current development pin: + +- repository: `SemiAnalysisAI/srt-slurm` +- branch: `agent/amd-multinode-runtime` +- commit: `3be6482cb879f3de0315bf28bc9d3c4904d23972` + +## Scope + +1. Prove a single-node aggregate vLLM deployment on MI300X. +2. Prove a multi-node Dynamo-vLLM prefill/decode deployment on MI300X. +3. Exercise both paths with fixed input/output sequence lengths and lightweight + models before introducing production-size models. +4. Validate the same paths through the upstream InferenceX GitHub Actions + runner infrastructure. +5. Port a representative existing MI355X disaggregated configuration after the + MI300X runtime contract is stable. + +## MI300X cluster contract under validation + +- eight AMD GPUs per healthy compute node; +- Slurm GPU allocation through `--gres=gpu:` rather than the current + srt-slurm `--gpus-per-node` default; +- no site-specific `--segment` directive; +- ROCm device access through `/dev/kfd` and `/dev/dri`; +- Pyxis/Enroot writable, remap-root, and mount-home behavior matching the + established MI300X launcher; +- the shared Hugging Face cache and runner workspace remain user-owned; +- the routable inter-node network interface is selected from live cluster + evidence rather than copied from an NVIDIA recipe. + +The launcher will continue to exclude compute nodes already documented as +unsuitable. It must not resume down nodes, cancel or preempt existing jobs, or +alter unrelated shared software. + +## Acceptance criteria + +### Aggregate + +- one srt-slurm allocation starts one aggregate vLLM service; +- every requested GPU is visible to ROCm and vLLM exactly once; +- the OpenAI-compatible health/model endpoint becomes ready; +- a fixed-sequence request completes successfully; +- srt-slurm tears down all owned processes and exits successfully. + +### Disaggregated + +- one allocation places distinct prefill and decode roles across multiple + MI300X nodes; +- NATS, etcd, Dynamo, and vLLM workers become healthy without bespoke + per-recipe orchestration; +- role endpoints use routable node addresses and unique ports; +- KV transfer completes across AMD nodes and a fixed-sequence request succeeds; +- teardown removes only processes owned by the allocation. + +### Regression safety + +- NVIDIA remains the default accelerator runtime in srt-slurm; +- existing NVIDIA recipes and device binding tests remain green; +- AMD-specific mounts, Slurm directives, and environment variables live in a + reusable cluster profile rather than duplicated recipe shell fragments. + +## Current status + +The srt-slurm branch now contains the first accelerator-aware runtime slice: +cluster configuration accepts `accelerator_vendor: amd`, partial-GPU workers +use Linux ROCm's `ROCR_VISIBLE_DEVICES`, and legacy NVIDIA/CUDA behavior remains +the default. MI300X runtime-profile and recipe implementation follows next. From 3a3e72387474a543c4b946e22f3ebbe1750ac7fb Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Sun, 9 Aug 2026 19:14:04 -0500 Subject: [PATCH 02/67] feat(amd): add MI300X srt-slurm aggregate scaffold Add the reusable MI300X cluster runtime profile and a minimal stable-vLLM fixed-sequence aggregate recipe for functional bring-up. --- .../srt-slurm-recipes/AMD_BRINGUP.md | 6 +- .../cluster-configs/mi300x-amds.yaml | 28 ++++++++ .../vllm/qwen3-0.6b/mi300x/agg-fixed-seq.yaml | 69 +++++++++++++++++++ 3 files changed, 101 insertions(+), 2 deletions(-) create mode 100644 benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi300x-amds.yaml create mode 100644 benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/agg-fixed-seq.yaml diff --git a/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md b/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md index a5ffa4e470..d762f0a92a 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md +++ b/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md @@ -9,7 +9,7 @@ Current development pin: - repository: `SemiAnalysisAI/srt-slurm` - branch: `agent/amd-multinode-runtime` -- commit: `3be6482cb879f3de0315bf28bc9d3c4904d23972` +- commit: `1edbcfc011a48a7637f9d4407b4ce4e2a656e062` ## Scope @@ -71,4 +71,6 @@ alter unrelated shared software. The srt-slurm branch now contains the first accelerator-aware runtime slice: cluster configuration accepts `accelerator_vendor: amd`, partial-GPU workers use Linux ROCm's `ROCR_VISIBLE_DEVICES`, and legacy NVIDIA/CUDA behavior remains -the default. MI300X runtime-profile and recipe implementation follows next. +the default. It also supports `gpu_sbatch_directive: gres` without changing the +legacy NVIDIA `--gpus-per-node` default. The initial MI300X cluster profile and +small-model aggregate recipe are checked in alongside this document. diff --git a/benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi300x-amds.yaml b/benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi300x-amds.yaml new file mode 100644 index 0000000000..267b8ed716 --- /dev/null +++ b/benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi300x-amds.yaml @@ -0,0 +1,28 @@ +# srt-slurm cluster profile for the MI300X AMD Slurm cluster. +# Runtime-specific source/output paths are supplied by the launcher. + +cluster: mi300x-amds +default_partition: compute +default_time_limit: "01:00:00" + +gpus_per_node: 8 +accelerator_vendor: amd +network_interface: ens51f1np1 + +# This cluster allocates accelerators through GRES and does not implement the +# NVIDIA cluster's topology-segment directive. +gpu_sbatch_directive: gres +use_segment_sbatch_directive: false +use_exclusive_sbatch_directive: false + +default_sbatch_directives: + cpus-per-task: "32" + mem: "128G" + +default_mounts: + /dev/kfd: /dev/kfd + /dev/dri: /dev/dri + /raid/hf-hub-cache: /hf_hub_cache + +default_bash_preamble: "ulimit -n 1048576" +nginx_raise_ulimit: false diff --git a/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/agg-fixed-seq.yaml b/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/agg-fixed-seq.yaml new file mode 100644 index 0000000000..14199bce93 --- /dev/null +++ b/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/agg-fixed-seq.yaml @@ -0,0 +1,69 @@ +# Minimal MI300X aggregate validation. This recipe deliberately uses one GPU, +# a small public model, and a short fixed-sequence workload. It validates +# srt-slurm orchestration rather than performance. + +name: "mi300x-vllm-qwen3-0.6b-agg-fixed-seq" + +model: + path: "hf:Qwen/Qwen3-0.6B" + container: "vllm/vllm-openai-rocm:v0.26.0" + precision: "fp16" + +identity: + model: + repo: "Qwen/Qwen3-0.6B" + container: + image: "vllm/vllm-openai-rocm:v0.26.0" + frameworks: + vllm: "0.26.0" + +slurm: + time_limit: "00:30:00" + +resources: + gpu_type: "mi300x" + gpus_per_node: 1 + agg_nodes: 1 + agg_workers: 1 + gpus_per_agg: 1 + +frontend: + type: vllm + enable_multiple_frontends: false + +backend: + type: vllm + connector: null + aggregated_environment: + HF_HUB_CACHE: "/hf_hub_cache" + HUGGINGFACE_HUB_CACHE: "/hf_hub_cache" + TRANSFORMERS_CACHE: "/hf_hub_cache" + PYTHONUNBUFFERED: "1" + vllm_config: + aggregated: + served-model-name: "Qwen/Qwen3-0.6B" + tensor-parallel-size: 1 + gpu-memory-utilization: 0.50 + max-model-len: 2048 + max-num-seqs: 8 + disable-log-requests: true + +srun_options: + container-mount-home: "" + container-writable: "" + container-remap-root: "" + mem: "0" + +health_check: + max_attempts: 180 + interval_seconds: 5 + +benchmark: + type: sa-bench + isl: 128 + osl: 32 + concurrencies: [1, 4] + req_rate: inf + random_range_ratio: 0.0 + num_warmup_mult: 1 + num_prompts_mult: 2 From fc07a3e20239d6ea04ef70d1095e1440103e6b1f Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Sun, 9 Aug 2026 19:36:18 -0500 Subject: [PATCH 03/67] fix(amd): align MI300X recipe with vLLM 0.26 --- .../srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/agg-fixed-seq.yaml | 1 - 1 file changed, 1 deletion(-) diff --git a/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/agg-fixed-seq.yaml b/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/agg-fixed-seq.yaml index 14199bce93..fccf40fcac 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/agg-fixed-seq.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/agg-fixed-seq.yaml @@ -46,7 +46,6 @@ backend: gpu-memory-utilization: 0.50 max-model-len: 2048 max-num-seqs: 8 - disable-log-requests: true srun_options: container-mount-home: "" From b95b4b8f66c31f6f42a4c7d30d6131ee49c74ad3 Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Sun, 9 Aug 2026 19:48:27 -0500 Subject: [PATCH 04/67] fix(amd): honor MI300X Pyxis home policy --- .../srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/agg-fixed-seq.yaml | 1 - 1 file changed, 1 deletion(-) diff --git a/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/agg-fixed-seq.yaml b/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/agg-fixed-seq.yaml index fccf40fcac..17454b3fb0 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/agg-fixed-seq.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/agg-fixed-seq.yaml @@ -48,7 +48,6 @@ backend: max-num-seqs: 8 srun_options: - container-mount-home: "" container-writable: "" container-remap-root: "" mem: "0" From aa681ff96725642e6a1708a7362e321f6728fc03 Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Sun, 9 Aug 2026 19:57:39 -0500 Subject: [PATCH 05/67] fix(amd): align MI300X worker launch policy --- benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md | 2 +- .../srt-slurm-recipes/cluster-configs/mi300x-amds.yaml | 1 - .../srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/agg-fixed-seq.yaml | 1 + 3 files changed, 2 insertions(+), 2 deletions(-) diff --git a/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md b/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md index d762f0a92a..5e4bfd6ff9 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md +++ b/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md @@ -9,7 +9,7 @@ Current development pin: - repository: `SemiAnalysisAI/srt-slurm` - branch: `agent/amd-multinode-runtime` -- commit: `1edbcfc011a48a7637f9d4407b4ce4e2a656e062` +- commit: `8dd98d5d550b4cc4714b800aed1ae6d5b902822c` ## Scope diff --git a/benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi300x-amds.yaml b/benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi300x-amds.yaml index 267b8ed716..acca7c2771 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi300x-amds.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi300x-amds.yaml @@ -24,5 +24,4 @@ default_mounts: /dev/dri: /dev/dri /raid/hf-hub-cache: /hf_hub_cache -default_bash_preamble: "ulimit -n 1048576" nginx_raise_ulimit: false diff --git a/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/agg-fixed-seq.yaml b/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/agg-fixed-seq.yaml index 17454b3fb0..422f5fb585 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/agg-fixed-seq.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/agg-fixed-seq.yaml @@ -34,6 +34,7 @@ frontend: backend: type: vllm connector: null + set_visible_devices: true aggregated_environment: HF_HUB_CACHE: "/hf_hub_cache" HUGGINGFACE_HUB_CACHE: "/hf_hub_cache" From 1b256b937eae3780ec3451fdeaf08b4fe90f3787 Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Sun, 9 Aug 2026 20:32:03 -0500 Subject: [PATCH 06/67] feat(amd): use InferenceX fixed-sequence benchmark --- .../srt-slurm-recipes/AMD_BRINGUP.md | 4 ++ .../vllm/qwen3-0.6b/mi300x/agg-fixed-seq.yaml | 39 +++++++++++++---- utils/test_mi300x_srt_slurm_contract.py | 42 +++++++++++++++++++ 3 files changed, 76 insertions(+), 9 deletions(-) create mode 100644 utils/test_mi300x_srt_slurm_contract.py diff --git a/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md b/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md index 5e4bfd6ff9..03a3db4a59 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md +++ b/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md @@ -32,6 +32,10 @@ Current development pin: - Pyxis/Enroot writable, remap-root, and mount-home behavior matching the established MI300X launcher; - the shared Hugging Face cache and runner workspace remain user-owned; +- fixed-sequence validation runs InferenceX's existing + `utils/bench_serving/benchmark_serving.py` through srt-slurm's `custom` + benchmark hook rather than maintaining a second benchmark copy in + srt-slurm; - the routable inter-node network interface is selected from live cluster evidence rather than copied from an NVIDIA recipe. diff --git a/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/agg-fixed-seq.yaml b/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/agg-fixed-seq.yaml index 422f5fb585..a9fbaa2cb0 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/agg-fixed-seq.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/agg-fixed-seq.yaml @@ -15,7 +15,7 @@ identity: container: image: "vllm/vllm-openai-rocm:v0.26.0" frameworks: - vllm: "0.26.0" + vllm: "0.26.0+rocm723" slurm: time_limit: "00:30:00" @@ -58,11 +58,32 @@ health_check: interval_seconds: 5 benchmark: - type: sa-bench - isl: 128 - osl: 32 - concurrencies: [1, 4] - req_rate: inf - random_range_ratio: 0.0 - num_warmup_mult: 1 - num_prompts_mult: 2 + type: custom + command: >- + set -euo pipefail; + mkdir -p /logs/fixed-seq; + for concurrency in 1 4; do + python3 /infmax-workspace/utils/bench_serving/benchmark_serving.py + --backend openai-chat + --base-url "http://${SRT_FRONTEND_HOST}:${SRT_FRONTEND_PORT}" + --endpoint /v1/chat/completions + --model Qwen/Qwen3-0.6B + --tokenizer Qwen/Qwen3-0.6B + --dataset-name random + --random-input-len 128 + --random-output-len 32 + --random-range-ratio 1.0 + --random-num-workers 1 + --num-warmups "${concurrency}" + --num-prompts "$((concurrency * 4))" + --max-concurrency "${concurrency}" + --request-rate inf + --ignore-eos + --disable-tqdm + --save-result + --result-dir /logs/fixed-seq + --result-filename "qwen3-0.6b-isl128-osl32-c${concurrency}.json"; + done + env: + HF_HUB_CACHE: /hf_hub_cache + HUGGINGFACE_HUB_CACHE: /hf_hub_cache diff --git a/utils/test_mi300x_srt_slurm_contract.py b/utils/test_mi300x_srt_slurm_contract.py new file mode 100644 index 0000000000..3e0f98c68c --- /dev/null +++ b/utils/test_mi300x_srt_slurm_contract.py @@ -0,0 +1,42 @@ +"""High-signal contract checks for the MI300X srt-slurm bring-up lane.""" + +from pathlib import Path + +import yaml + + +REPO_ROOT = Path(__file__).resolve().parents[1] +RECIPE_PATH = ( + REPO_ROOT + / "benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/agg-fixed-seq.yaml" +) +CLUSTER_PATH = ( + REPO_ROOT + / "benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi300x-amds.yaml" +) + + +def test_mi300x_cluster_uses_the_rocm_slurm_contract(): + cluster = yaml.safe_load(CLUSTER_PATH.read_text()) + + assert cluster["accelerator_vendor"] == "amd" + assert cluster["gpu_sbatch_directive"] == "gres" + assert cluster["use_segment_sbatch_directive"] is False + assert cluster["default_mounts"]["/dev/kfd"] == "/dev/kfd" + assert cluster["default_mounts"]["/dev/dri"] == "/dev/dri" + + +def test_fixed_sequence_recipe_uses_inferencex_custom_benchmark(): + recipe = yaml.safe_load(RECIPE_PATH.read_text()) + benchmark = recipe["benchmark"] + command = benchmark["command"] + + assert benchmark["type"] == "custom" + assert "/infmax-workspace/utils/bench_serving/benchmark_serving.py" in command + assert "--backend openai-chat" in command + assert "--endpoint /v1/chat/completions" in command + assert "--random-input-len 128" in command + assert "--random-output-len 32" in command + assert "--random-range-ratio 1.0" in command + assert "best-of" not in command + assert "sa-bench" not in command From 16c5efc858f197b6cdefe2521be1515b7f147582 Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Sun, 9 Aug 2026 20:35:14 -0500 Subject: [PATCH 07/67] fix(amd): exclude unsuitable MI300X nodes --- .../srt-slurm-recipes/cluster-configs/mi300x-amds.yaml | 1 + utils/test_mi300x_srt_slurm_contract.py | 3 +++ 2 files changed, 4 insertions(+) diff --git a/benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi300x-amds.yaml b/benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi300x-amds.yaml index acca7c2771..ab613c1025 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi300x-amds.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi300x-amds.yaml @@ -18,6 +18,7 @@ use_exclusive_sbatch_directive: false default_sbatch_directives: cpus-per-task: "32" mem: "128G" + exclude: "chi-mi300x-049,chi-mi300x-121" default_mounts: /dev/kfd: /dev/kfd diff --git a/utils/test_mi300x_srt_slurm_contract.py b/utils/test_mi300x_srt_slurm_contract.py index 3e0f98c68c..8b987e70cb 100644 --- a/utils/test_mi300x_srt_slurm_contract.py +++ b/utils/test_mi300x_srt_slurm_contract.py @@ -22,6 +22,9 @@ def test_mi300x_cluster_uses_the_rocm_slurm_contract(): assert cluster["accelerator_vendor"] == "amd" assert cluster["gpu_sbatch_directive"] == "gres" assert cluster["use_segment_sbatch_directive"] is False + assert cluster["default_sbatch_directives"]["exclude"] == ( + "chi-mi300x-049,chi-mi300x-121" + ) assert cluster["default_mounts"]["/dev/kfd"] == "/dev/kfd" assert cluster["default_mounts"]["/dev/dri"] == "/dev/dri" From b1aa29a5a88c2d9140a9d98cd7750466647b5d61 Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Sun, 9 Aug 2026 20:44:42 -0500 Subject: [PATCH 08/67] feat(amd): add MI300X disaggregated validation target --- .../srt-slurm-recipes/AMD_BRINGUP.md | 4 +- .../vllm/qwen3-0.6b/mi300x/agg-fixed-seq.yaml | 2 + .../mi300x/disagg-1p1d-fixed-seq.yaml | 115 ++++++++++++++++++ 3 files changed, 120 insertions(+), 1 deletion(-) create mode 100644 benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml diff --git a/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md b/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md index 03a3db4a59..b563170718 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md +++ b/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md @@ -77,4 +77,6 @@ cluster configuration accepts `accelerator_vendor: amd`, partial-GPU workers use Linux ROCm's `ROCR_VISIBLE_DEVICES`, and legacy NVIDIA/CUDA behavior remains the default. It also supports `gpu_sbatch_directive: gres` without changing the legacy NVIDIA `--gpus-per-node` default. The initial MI300X cluster profile and -small-model aggregate recipe are checked in alongside this document. +small-model aggregate recipe are checked in alongside this document. A two-node +1-prefill/1-decode recipe uses stable Dynamo 1.2.1 and vLLM's NIXL connector as +the first disaggregated validation target. diff --git a/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/agg-fixed-seq.yaml b/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/agg-fixed-seq.yaml index a9fbaa2cb0..815e5746cb 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/agg-fixed-seq.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/agg-fixed-seq.yaml @@ -36,6 +36,7 @@ backend: connector: null set_visible_devices: true aggregated_environment: + HF_HOME: "/hf_hub_cache" HF_HUB_CACHE: "/hf_hub_cache" HUGGINGFACE_HUB_CACHE: "/hf_hub_cache" TRANSFORMERS_CACHE: "/hf_hub_cache" @@ -85,5 +86,6 @@ benchmark: --result-filename "qwen3-0.6b-isl128-osl32-c${concurrency}.json"; done env: + HF_HOME: /hf_hub_cache HF_HUB_CACHE: /hf_hub_cache HUGGINGFACE_HUB_CACHE: /hf_hub_cache diff --git a/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml b/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml new file mode 100644 index 0000000000..2f8a1279c5 --- /dev/null +++ b/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml @@ -0,0 +1,115 @@ +# Minimal two-node MI300X prefill/decode validation. This recipe exercises +# Dynamo discovery and NIXL KV transfer without attempting performance tuning. + +name: "mi300x-vllm-qwen3-0.6b-disagg-1p1d-fixed-seq" + +model: + path: "hf:Qwen/Qwen3-0.6B" + container: "vllm/vllm-openai-rocm:v0.26.0" + precision: "fp16" + +identity: + model: + repo: "Qwen/Qwen3-0.6B" + container: + image: "vllm/vllm-openai-rocm:v0.26.0" + frameworks: + vllm: "0.26.0+rocm723" + dynamo: "1.2.1" + +dynamo: + version: "1.2.1" + install: true + request_plane: tcp + +slurm: + time_limit: "00:45:00" + +resources: + gpu_type: "mi300x" + gpus_per_node: 1 + prefill_nodes: 1 + decode_nodes: 1 + prefill_workers: 1 + decode_workers: 1 + gpus_per_prefill: 1 + gpus_per_decode: 1 + +frontend: + type: dynamo + enable_multiple_frontends: false + +backend: + type: vllm + connector: nixl + set_visible_devices: true + prefill_environment: + HF_HOME: "/hf_hub_cache" + HF_HUB_CACHE: "/hf_hub_cache" + HUGGINGFACE_HUB_CACHE: "/hf_hub_cache" + TRANSFORMERS_CACHE: "/hf_hub_cache" + PYTHONUNBUFFERED: "1" + NIXL_LOG_LEVEL: INFO + decode_environment: + HF_HOME: "/hf_hub_cache" + HF_HUB_CACHE: "/hf_hub_cache" + HUGGINGFACE_HUB_CACHE: "/hf_hub_cache" + TRANSFORMERS_CACHE: "/hf_hub_cache" + PYTHONUNBUFFERED: "1" + NIXL_LOG_LEVEL: INFO + vllm_config: + prefill: + served-model-name: "Qwen/Qwen3-0.6B" + tensor-parallel-size: 1 + gpu-memory-utilization: 0.50 + max-model-len: 2048 + max-num-seqs: 8 + enforce-eager: true + decode: + served-model-name: "Qwen/Qwen3-0.6B" + tensor-parallel-size: 1 + gpu-memory-utilization: 0.50 + max-model-len: 2048 + max-num-seqs: 8 + enforce-eager: true + +srun_options: + container-writable: "" + container-remap-root: "" + mem: "0" + +health_check: + max_attempts: 240 + interval_seconds: 5 + +benchmark: + type: custom + command: >- + set -euo pipefail; + mkdir -p /logs/fixed-seq; + for concurrency in 1 4; do + python3 /infmax-workspace/utils/bench_serving/benchmark_serving.py + --backend openai-chat + --base-url "http://${SRT_FRONTEND_HOST}:${SRT_FRONTEND_PORT}" + --endpoint /v1/chat/completions + --model Qwen/Qwen3-0.6B + --tokenizer Qwen/Qwen3-0.6B + --dataset-name random + --random-input-len 128 + --random-output-len 32 + --random-range-ratio 1.0 + --random-num-workers 1 + --num-warmups "${concurrency}" + --num-prompts "$((concurrency * 4))" + --max-concurrency "${concurrency}" + --request-rate inf + --ignore-eos + --disable-tqdm + --save-result + --result-dir /logs/fixed-seq + --result-filename "qwen3-0.6b-disagg-isl128-osl32-c${concurrency}.json"; + done + env: + HF_HOME: /hf_hub_cache + HF_HUB_CACHE: /hf_hub_cache + HUGGINGFACE_HUB_CACHE: /hf_hub_cache From 181459d071e9711b842999123af9d062fbcf61fc Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Sun, 9 Aug 2026 20:48:15 -0500 Subject: [PATCH 09/67] fix(amd): preserve custom benchmark arguments --- .../vllm/qwen3-0.6b/mi300x/agg-fixed-seq.yaml | 44 +++++++++---------- .../mi300x/disagg-1p1d-fixed-seq.yaml | 44 +++++++++---------- utils/test_mi300x_srt_slurm_contract.py | 37 ++++++++++++++++ 3 files changed, 81 insertions(+), 44 deletions(-) diff --git a/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/agg-fixed-seq.yaml b/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/agg-fixed-seq.yaml index 815e5746cb..7b5f2a9e40 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/agg-fixed-seq.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/agg-fixed-seq.yaml @@ -60,29 +60,29 @@ health_check: benchmark: type: custom - command: >- - set -euo pipefail; - mkdir -p /logs/fixed-seq; + command: | + set -euo pipefail + mkdir -p /logs/fixed-seq for concurrency in 1 4; do - python3 /infmax-workspace/utils/bench_serving/benchmark_serving.py - --backend openai-chat - --base-url "http://${SRT_FRONTEND_HOST}:${SRT_FRONTEND_PORT}" - --endpoint /v1/chat/completions - --model Qwen/Qwen3-0.6B - --tokenizer Qwen/Qwen3-0.6B - --dataset-name random - --random-input-len 128 - --random-output-len 32 - --random-range-ratio 1.0 - --random-num-workers 1 - --num-warmups "${concurrency}" - --num-prompts "$((concurrency * 4))" - --max-concurrency "${concurrency}" - --request-rate inf - --ignore-eos - --disable-tqdm - --save-result - --result-dir /logs/fixed-seq + python3 /infmax-workspace/utils/bench_serving/benchmark_serving.py \ + --backend openai-chat \ + --base-url "http://${SRT_FRONTEND_HOST}:${SRT_FRONTEND_PORT}" \ + --endpoint /v1/chat/completions \ + --model Qwen/Qwen3-0.6B \ + --tokenizer Qwen/Qwen3-0.6B \ + --dataset-name random \ + --random-input-len 128 \ + --random-output-len 32 \ + --random-range-ratio 1.0 \ + --random-num-workers 1 \ + --num-warmups "${concurrency}" \ + --num-prompts "$((concurrency * 4))" \ + --max-concurrency "${concurrency}" \ + --request-rate inf \ + --ignore-eos \ + --disable-tqdm \ + --save-result \ + --result-dir /logs/fixed-seq \ --result-filename "qwen3-0.6b-isl128-osl32-c${concurrency}.json"; done env: diff --git a/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml b/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml index 2f8a1279c5..c330efa3ca 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml @@ -84,29 +84,29 @@ health_check: benchmark: type: custom - command: >- - set -euo pipefail; - mkdir -p /logs/fixed-seq; + command: | + set -euo pipefail + mkdir -p /logs/fixed-seq for concurrency in 1 4; do - python3 /infmax-workspace/utils/bench_serving/benchmark_serving.py - --backend openai-chat - --base-url "http://${SRT_FRONTEND_HOST}:${SRT_FRONTEND_PORT}" - --endpoint /v1/chat/completions - --model Qwen/Qwen3-0.6B - --tokenizer Qwen/Qwen3-0.6B - --dataset-name random - --random-input-len 128 - --random-output-len 32 - --random-range-ratio 1.0 - --random-num-workers 1 - --num-warmups "${concurrency}" - --num-prompts "$((concurrency * 4))" - --max-concurrency "${concurrency}" - --request-rate inf - --ignore-eos - --disable-tqdm - --save-result - --result-dir /logs/fixed-seq + python3 /infmax-workspace/utils/bench_serving/benchmark_serving.py \ + --backend openai-chat \ + --base-url "http://${SRT_FRONTEND_HOST}:${SRT_FRONTEND_PORT}" \ + --endpoint /v1/chat/completions \ + --model Qwen/Qwen3-0.6B \ + --tokenizer Qwen/Qwen3-0.6B \ + --dataset-name random \ + --random-input-len 128 \ + --random-output-len 32 \ + --random-range-ratio 1.0 \ + --random-num-workers 1 \ + --num-warmups "${concurrency}" \ + --num-prompts "$((concurrency * 4))" \ + --max-concurrency "${concurrency}" \ + --request-rate inf \ + --ignore-eos \ + --disable-tqdm \ + --save-result \ + --result-dir /logs/fixed-seq \ --result-filename "qwen3-0.6b-disagg-isl128-osl32-c${concurrency}.json"; done env: diff --git a/utils/test_mi300x_srt_slurm_contract.py b/utils/test_mi300x_srt_slurm_contract.py index 8b987e70cb..f519dc80bc 100644 --- a/utils/test_mi300x_srt_slurm_contract.py +++ b/utils/test_mi300x_srt_slurm_contract.py @@ -1,5 +1,7 @@ """High-signal contract checks for the MI300X srt-slurm bring-up lane.""" +import os +import subprocess from pathlib import Path import yaml @@ -10,6 +12,7 @@ REPO_ROOT / "benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/agg-fixed-seq.yaml" ) +DISAGG_RECIPE_PATH = RECIPE_PATH.with_name("disagg-1p1d-fixed-seq.yaml") CLUSTER_PATH = ( REPO_ROOT / "benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi300x-amds.yaml" @@ -43,3 +46,37 @@ def test_fixed_sequence_recipe_uses_inferencex_custom_benchmark(): assert "--random-range-ratio 1.0" in command assert "best-of" not in command assert "sa-bench" not in command + + +def test_fixed_sequence_commands_keep_all_arguments_attached(tmp_path): + fake_bin = tmp_path / "bin" + fake_bin.mkdir() + fake_python = fake_bin / "python3" + fake_python.write_text( + "#!/bin/bash\n" + 'printf "%s\\n" "$@" >> "$FAKE_ARGS_LOG"\n' + 'printf "%s\\n" --CALL-END-- >> "$FAKE_ARGS_LOG"\n' + ) + fake_python.chmod(0o755) + + for recipe_path in (RECIPE_PATH, DISAGG_RECIPE_PATH): + command = yaml.safe_load(recipe_path.read_text())["benchmark"]["command"] + result_dir = tmp_path / recipe_path.stem + command = command.replace("/logs/fixed-seq", str(result_dir)) + args_log = tmp_path / f"{recipe_path.stem}.args" + env = { + **os.environ, + "PATH": f"{fake_bin}:{os.environ['PATH']}", + "FAKE_ARGS_LOG": str(args_log), + "SRT_FRONTEND_HOST": "127.0.0.1", + "SRT_FRONTEND_PORT": "8000", + } + + subprocess.run(["bash", "-n"], input=command, text=True, check=True) + subprocess.run(["bash", "-c", command], env=env, check=True) + + calls = args_log.read_text().split("--CALL-END--\n") + calls = [[arg for arg in call.splitlines() if arg] for call in calls if call] + assert len(calls) == 2 + assert all("--model" in call and "Qwen/Qwen3-0.6B" in call for call in calls) + assert [call[call.index("--num-prompts") + 1] for call in calls] == ["4", "16"] From 9b728aba072bc7e50c143da6193020e41bf81855 Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Sun, 9 Aug 2026 20:55:44 -0500 Subject: [PATCH 10/67] Stage MI300X runtime config on compute nodes --- .../srt-slurm-recipes/cluster-configs/mi300x-amds.yaml | 5 +++++ utils/test_mi300x_srt_slurm_contract.py | 1 + 2 files changed, 6 insertions(+) diff --git a/benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi300x-amds.yaml b/benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi300x-amds.yaml index ab613c1025..b6025703da 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi300x-amds.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi300x-amds.yaml @@ -15,6 +15,11 @@ gpu_sbatch_directive: gres use_segment_sbatch_directive: false use_exclusive_sbatch_directive: false +# The login node and compute nodes do not share this output path. Carry the +# exact resolved recipe in the batch script and materialize it on the allocated +# head node instead of depending on a submitter-side copy. +runtime_config_transport: embedded + default_sbatch_directives: cpus-per-task: "32" mem: "128G" diff --git a/utils/test_mi300x_srt_slurm_contract.py b/utils/test_mi300x_srt_slurm_contract.py index f519dc80bc..095d9b0010 100644 --- a/utils/test_mi300x_srt_slurm_contract.py +++ b/utils/test_mi300x_srt_slurm_contract.py @@ -25,6 +25,7 @@ def test_mi300x_cluster_uses_the_rocm_slurm_contract(): assert cluster["accelerator_vendor"] == "amd" assert cluster["gpu_sbatch_directive"] == "gres" assert cluster["use_segment_sbatch_directive"] is False + assert cluster["runtime_config_transport"] == "embedded" assert cluster["default_sbatch_directives"]["exclude"] == ( "chi-mi300x-049,chi-mi300x-121" ) From 542ddda60f5c0b2d10c1c36316fdaf7b057f9b14 Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Sun, 9 Aug 2026 21:10:03 -0500 Subject: [PATCH 11/67] Reuse stable ROCm container artifact --- .../srt-slurm-recipes/cluster-configs/mi300x-amds.yaml | 6 ++++++ .../vllm/qwen3-0.6b/mi300x/agg-fixed-seq.yaml | 2 +- .../vllm/qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml | 2 +- utils/test_mi300x_srt_slurm_contract.py | 9 +++++++++ 4 files changed, 17 insertions(+), 2 deletions(-) diff --git a/benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi300x-amds.yaml b/benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi300x-amds.yaml index b6025703da..5c67ef30d2 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi300x-amds.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi300x-amds.yaml @@ -30,4 +30,10 @@ default_mounts: /dev/dri: /dev/dri /raid/hf-hub-cache: /hf_hub_cache +# Pyxis otherwise re-imports the large ROCm image for every srun step. The +# validation launcher materializes this immutable squashfs on each healthy +# node before submitting a sweep. +containers: + vllm-rocm-v0.26.0: /raid/hf-hub-cache/inferencex/srt-slurm/containers/vllm-openai-rocm-v0.26.0.sqsh + nginx_raise_ulimit: false diff --git a/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/agg-fixed-seq.yaml b/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/agg-fixed-seq.yaml index 7b5f2a9e40..9627bdd49e 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/agg-fixed-seq.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/agg-fixed-seq.yaml @@ -6,7 +6,7 @@ name: "mi300x-vllm-qwen3-0.6b-agg-fixed-seq" model: path: "hf:Qwen/Qwen3-0.6B" - container: "vllm/vllm-openai-rocm:v0.26.0" + container: "vllm-rocm-v0.26.0" precision: "fp16" identity: diff --git a/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml b/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml index c330efa3ca..a98a51c0d6 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml @@ -5,7 +5,7 @@ name: "mi300x-vllm-qwen3-0.6b-disagg-1p1d-fixed-seq" model: path: "hf:Qwen/Qwen3-0.6B" - container: "vllm/vllm-openai-rocm:v0.26.0" + container: "vllm-rocm-v0.26.0" precision: "fp16" identity: diff --git a/utils/test_mi300x_srt_slurm_contract.py b/utils/test_mi300x_srt_slurm_contract.py index 095d9b0010..9b29f1f69d 100644 --- a/utils/test_mi300x_srt_slurm_contract.py +++ b/utils/test_mi300x_srt_slurm_contract.py @@ -31,6 +31,15 @@ def test_mi300x_cluster_uses_the_rocm_slurm_contract(): ) assert cluster["default_mounts"]["/dev/kfd"] == "/dev/kfd" assert cluster["default_mounts"]["/dev/dri"] == "/dev/dri" + image_path = cluster["containers"]["vllm-rocm-v0.26.0"] + assert image_path.endswith("/vllm-openai-rocm-v0.26.0.sqsh") + + for recipe_path in (RECIPE_PATH, DISAGG_RECIPE_PATH): + recipe = yaml.safe_load(recipe_path.read_text()) + assert recipe["model"]["container"] == "vllm-rocm-v0.26.0" + assert recipe["identity"]["container"]["image"] == ( + "vllm/vllm-openai-rocm:v0.26.0" + ) def test_fixed_sequence_recipe_uses_inferencex_custom_benchmark(): From 2df850878be1fcac2e2bcb3cf920751bb1a3c2c0 Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Sun, 9 Aug 2026 21:51:20 -0500 Subject: [PATCH 12/67] fix(amd): align disagg runtime dependencies --- benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md | 4 ++-- .../vllm/qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md b/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md index b563170718..f6ee3dbceb 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md +++ b/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md @@ -9,7 +9,7 @@ Current development pin: - repository: `SemiAnalysisAI/srt-slurm` - branch: `agent/amd-multinode-runtime` -- commit: `8dd98d5d550b4cc4714b800aed1ae6d5b902822c` +- commit: `d66b3aea91e3ab9a3d88f22d82fdb20c34abb745` ## Scope @@ -78,5 +78,5 @@ use Linux ROCm's `ROCR_VISIBLE_DEVICES`, and legacy NVIDIA/CUDA behavior remains the default. It also supports `gpu_sbatch_directive: gres` without changing the legacy NVIDIA `--gpus-per-node` default. The initial MI300X cluster profile and small-model aggregate recipe are checked in alongside this document. A two-node -1-prefill/1-decode recipe uses stable Dynamo 1.2.1 and vLLM's NIXL connector as +1-prefill/1-decode recipe uses stable Dynamo 1.3.1 and vLLM's NIXL connector as the first disaggregated validation target. diff --git a/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml b/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml index a98a51c0d6..b2e9c72895 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml @@ -15,10 +15,10 @@ identity: image: "vllm/vllm-openai-rocm:v0.26.0" frameworks: vllm: "0.26.0+rocm723" - dynamo: "1.2.1" + dynamo: "1.3.1" dynamo: - version: "1.2.1" + version: "1.3.1" install: true request_plane: tcp From b0376b34c13b274ac00e94aca301f6aa9cc10446 Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Sun, 9 Aug 2026 21:58:50 -0500 Subject: [PATCH 13/67] docs(amd): pin runtime source override head --- benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md b/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md index f6ee3dbceb..f745bca4e0 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md +++ b/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md @@ -9,7 +9,7 @@ Current development pin: - repository: `SemiAnalysisAI/srt-slurm` - branch: `agent/amd-multinode-runtime` -- commit: `d66b3aea91e3ab9a3d88f22d82fdb20c34abb745` +- commit: `c459a45bdeb0efd713bc63d5e4bd357c5ec9c510` ## Scope From 9afc9a1c5c640e08b075c0bc51c696c33211a1c3 Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Sun, 9 Aug 2026 22:24:35 -0500 Subject: [PATCH 14/67] fix: route MI300X orchestration over private fabric --- benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md | 7 +++++-- .../srt-slurm-recipes/cluster-configs/mi300x-amds.yaml | 2 +- utils/test_mi300x_srt_slurm_contract.py | 1 + 3 files changed, 7 insertions(+), 3 deletions(-) diff --git a/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md b/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md index f745bca4e0..c71ce20664 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md +++ b/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md @@ -9,7 +9,7 @@ Current development pin: - repository: `SemiAnalysisAI/srt-slurm` - branch: `agent/amd-multinode-runtime` -- commit: `c459a45bdeb0efd713bc63d5e4bd357c5ec9c510` +- commit: `9fdb303e3564ede93c192b28a62397bbaeaad09d` ## Scope @@ -79,4 +79,7 @@ the default. It also supports `gpu_sbatch_directive: gres` without changing the legacy NVIDIA `--gpus-per-node` default. The initial MI300X cluster profile and small-model aggregate recipe are checked in alongside this document. A two-node 1-prefill/1-decode recipe uses stable Dynamo 1.3.1 and vLLM's NIXL connector as -the first disaggregated validation target. +the first disaggregated validation target. Stable Dynamo releases install into +a writable job-local overlay instead of the immutable container root, and the +control-plane endpoints use the cluster-selected private interface +(`ens61f1np1`) rather than Slurm hostnames or the public default route. diff --git a/benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi300x-amds.yaml b/benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi300x-amds.yaml index 5c67ef30d2..170fd23fd9 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi300x-amds.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi300x-amds.yaml @@ -7,7 +7,7 @@ default_time_limit: "01:00:00" gpus_per_node: 8 accelerator_vendor: amd -network_interface: ens51f1np1 +network_interface: ens61f1np1 # This cluster allocates accelerators through GRES and does not implement the # NVIDIA cluster's topology-segment directive. diff --git a/utils/test_mi300x_srt_slurm_contract.py b/utils/test_mi300x_srt_slurm_contract.py index 9b29f1f69d..fe5369644b 100644 --- a/utils/test_mi300x_srt_slurm_contract.py +++ b/utils/test_mi300x_srt_slurm_contract.py @@ -23,6 +23,7 @@ def test_mi300x_cluster_uses_the_rocm_slurm_contract(): cluster = yaml.safe_load(CLUSTER_PATH.read_text()) assert cluster["accelerator_vendor"] == "amd" + assert cluster["network_interface"] == "ens61f1np1" assert cluster["gpu_sbatch_directive"] == "gres" assert cluster["use_segment_sbatch_directive"] is False assert cluster["runtime_config_transport"] == "embedded" From 8e139bdd8e8399e2f0a4c051f78d1c57d3bbfbb5 Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Sun, 9 Aug 2026 22:42:36 -0500 Subject: [PATCH 15/67] fix: discover MI300X private fabric addresses --- benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md | 8 +++++--- .../srt-slurm-recipes/cluster-configs/mi300x-amds.yaml | 4 +++- utils/test_mi300x_srt_slurm_contract.py | 2 +- 3 files changed, 9 insertions(+), 5 deletions(-) diff --git a/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md b/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md index c71ce20664..aa7b2097c0 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md +++ b/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md @@ -9,7 +9,7 @@ Current development pin: - repository: `SemiAnalysisAI/srt-slurm` - branch: `agent/amd-multinode-runtime` -- commit: `9fdb303e3564ede93c192b28a62397bbaeaad09d` +- commit: `90ffbda0cb0887af8e6060475d051a8e35e6d546` ## Scope @@ -81,5 +81,7 @@ small-model aggregate recipe are checked in alongside this document. A two-node 1-prefill/1-decode recipe uses stable Dynamo 1.3.1 and vLLM's NIXL connector as the first disaggregated validation target. Stable Dynamo releases install into a writable job-local overlay instead of the immutable container root, and the -control-plane endpoints use the cluster-selected private interface -(`ens61f1np1`) rather than Slurm hostnames or the public default route. +control-plane endpoints use automatic RFC1918-preferring discovery because the +private NIC name varies across MI300X node generations. Slurm validation 11718 +reached vLLM KV-cache initialization on both nodes; its next isolated blocker +is ROCm KV-memory registration in the NIXL/UCX data plane. diff --git a/benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi300x-amds.yaml b/benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi300x-amds.yaml index 170fd23fd9..84968c1528 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi300x-amds.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi300x-amds.yaml @@ -7,7 +7,9 @@ default_time_limit: "01:00:00" gpus_per_node: 8 accelerator_vendor: amd -network_interface: ens61f1np1 +# Private NIC names vary by node generation (ens51f1np1 vs ens61f1np1), so +# use srt-slurm's RFC1918-preferring automatic resolver. +network_interface: null # This cluster allocates accelerators through GRES and does not implement the # NVIDIA cluster's topology-segment directive. diff --git a/utils/test_mi300x_srt_slurm_contract.py b/utils/test_mi300x_srt_slurm_contract.py index fe5369644b..a87e642cf6 100644 --- a/utils/test_mi300x_srt_slurm_contract.py +++ b/utils/test_mi300x_srt_slurm_contract.py @@ -23,7 +23,7 @@ def test_mi300x_cluster_uses_the_rocm_slurm_contract(): cluster = yaml.safe_load(CLUSTER_PATH.read_text()) assert cluster["accelerator_vendor"] == "amd" - assert cluster["network_interface"] == "ens61f1np1" + assert cluster["network_interface"] is None assert cluster["gpu_sbatch_directive"] == "gres" assert cluster["use_segment_sbatch_directive"] is False assert cluster["runtime_config_transport"] == "embedded" From 6f25810fb2fde7f13b0c77ad9cdb12170e025dd5 Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Sun, 9 Aug 2026 23:01:50 -0500 Subject: [PATCH 16/67] feat: validate MI300X MoRI-IO routing --- .../srt-slurm-recipes/AMD_BRINGUP.md | 27 ++++++++++--------- .../cluster-configs/mi300x-amds.yaml | 1 + .../mi300x/disagg-1p1d-fixed-seq.yaml | 22 +++++++-------- utils/test_mi300x_srt_slurm_contract.py | 23 ++++++++++++++++ 4 files changed, 50 insertions(+), 23 deletions(-) diff --git a/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md b/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md index aa7b2097c0..1cae260eaa 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md +++ b/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md @@ -9,12 +9,13 @@ Current development pin: - repository: `SemiAnalysisAI/srt-slurm` - branch: `agent/amd-multinode-runtime` -- commit: `90ffbda0cb0887af8e6060475d051a8e35e6d546` +- commit: `7f53c35015e12ef3581f474018b5349ce454e977` ## Scope 1. Prove a single-node aggregate vLLM deployment on MI300X. -2. Prove a multi-node Dynamo-vLLM prefill/decode deployment on MI300X. +2. Prove a multi-node vLLM Router prefill/decode deployment on MI300X using + AMD's supported MoRI-IO KV connector. 3. Exercise both paths with fixed input/output sequence lengths and lightweight models before introducing production-size models. 4. Validate the same paths through the upstream InferenceX GitHub Actions @@ -57,8 +58,8 @@ alter unrelated shared software. - one allocation places distinct prefill and decode roles across multiple MI300X nodes; -- NATS, etcd, Dynamo, and vLLM workers become healthy without bespoke - per-recipe orchestration; +- vLLM Router and direct vLLM workers become healthy without Dynamo, NATS, + etcd, or bespoke per-recipe orchestration; - role endpoints use routable node addresses and unique ports; - KV transfer completes across AMD nodes and a fixed-sequence request succeeds; - teardown removes only processes owned by the allocation. @@ -77,11 +78,13 @@ cluster configuration accepts `accelerator_vendor: amd`, partial-GPU workers use Linux ROCm's `ROCR_VISIBLE_DEVICES`, and legacy NVIDIA/CUDA behavior remains the default. It also supports `gpu_sbatch_directive: gres` without changing the legacy NVIDIA `--gpus-per-node` default. The initial MI300X cluster profile and -small-model aggregate recipe are checked in alongside this document. A two-node -1-prefill/1-decode recipe uses stable Dynamo 1.3.1 and vLLM's NIXL connector as -the first disaggregated validation target. Stable Dynamo releases install into -a writable job-local overlay instead of the immutable container root, and the -control-plane endpoints use automatic RFC1918-preferring discovery because the -private NIC name varies across MI300X node generations. Slurm validation 11718 -reached vLLM KV-cache initialization on both nodes; its next isolated blocker -is ROCm KV-memory registration in the NIXL/UCX data plane. +small-model aggregate recipe are checked in alongside this document. The first +aggregate path uses a direct private `vllm serve` endpoint. The two-node +1-prefill/1-decode path uses the official vLLM Router and vLLM's ROCm-only +`MoRIIOConnector`: srt-slurm owns the router discovery port and generates +role-aware worker registration config from the realized Slurm topology. The +control-plane endpoints use automatic RFC1918-preferring discovery because +private NIC names vary across MI300X node generations. The earlier +Dynamo/NIXL experiment reached KV-cache initialization but failed ROCm memory +registration; that NVIDIA-oriented data plane is now explicitly out of scope +rather than patched into the AMD implementation. diff --git a/benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi300x-amds.yaml b/benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi300x-amds.yaml index 84968c1528..1f532ac161 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi300x-amds.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi300x-amds.yaml @@ -37,5 +37,6 @@ default_mounts: # node before submitting a sweep. containers: vllm-rocm-v0.26.0: /raid/hf-hub-cache/inferencex/srt-slurm/containers/vllm-openai-rocm-v0.26.0.sqsh + vllm-router-20260716: /raid/hf-hub-cache/inferencex/srt-slurm/containers/vllm-router-nightly-20260716-1fbcde7.sqsh nginx_raise_ulimit: false diff --git a/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml b/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml index b2e9c72895..7709cdedc6 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml @@ -1,5 +1,5 @@ # Minimal two-node MI300X prefill/decode validation. This recipe exercises -# Dynamo discovery and NIXL KV transfer without attempting performance tuning. +# vLLM Router discovery and AMD MoRI-IO KV transfer without performance tuning. name: "mi300x-vllm-qwen3-0.6b-disagg-1p1d-fixed-seq" @@ -15,12 +15,7 @@ identity: image: "vllm/vllm-openai-rocm:v0.26.0" frameworks: vllm: "0.26.0+rocm723" - dynamo: "1.3.1" - -dynamo: - version: "1.3.1" - install: true - request_plane: tcp + vllm-router: "nightly-20260716-1fbcde7" slurm: time_limit: "00:45:00" @@ -36,12 +31,17 @@ resources: gpus_per_decode: 1 frontend: - type: dynamo + type: vllm-router enable_multiple_frontends: false + container_image: "vllm-router-20260716" + args: + policy: consistent_hash + prefill-policy: consistent_hash + decode-policy: consistent_hash backend: type: vllm - connector: nixl + connector: moriio set_visible_devices: true prefill_environment: HF_HOME: "/hf_hub_cache" @@ -49,14 +49,14 @@ backend: HUGGINGFACE_HUB_CACHE: "/hf_hub_cache" TRANSFORMERS_CACHE: "/hf_hub_cache" PYTHONUNBUFFERED: "1" - NIXL_LOG_LEVEL: INFO + MORI_IO_SQ_BACKOFF_TIMEOUT_US: "50000" decode_environment: HF_HOME: "/hf_hub_cache" HF_HUB_CACHE: "/hf_hub_cache" HUGGINGFACE_HUB_CACHE: "/hf_hub_cache" TRANSFORMERS_CACHE: "/hf_hub_cache" PYTHONUNBUFFERED: "1" - NIXL_LOG_LEVEL: INFO + MORI_IO_SQ_BACKOFF_TIMEOUT_US: "50000" vllm_config: prefill: served-model-name: "Qwen/Qwen3-0.6B" diff --git a/utils/test_mi300x_srt_slurm_contract.py b/utils/test_mi300x_srt_slurm_contract.py index a87e642cf6..0573b11b74 100644 --- a/utils/test_mi300x_srt_slurm_contract.py +++ b/utils/test_mi300x_srt_slurm_contract.py @@ -34,6 +34,8 @@ def test_mi300x_cluster_uses_the_rocm_slurm_contract(): assert cluster["default_mounts"]["/dev/dri"] == "/dev/dri" image_path = cluster["containers"]["vllm-rocm-v0.26.0"] assert image_path.endswith("/vllm-openai-rocm-v0.26.0.sqsh") + router_path = cluster["containers"]["vllm-router-20260716"] + assert router_path.endswith("/vllm-router-nightly-20260716-1fbcde7.sqsh") for recipe_path in (RECIPE_PATH, DISAGG_RECIPE_PATH): recipe = yaml.safe_load(recipe_path.read_text()) @@ -43,6 +45,27 @@ def test_mi300x_cluster_uses_the_rocm_slurm_contract(): ) +def test_disaggregated_recipe_uses_native_router_and_moriio(): + recipe = yaml.safe_load(DISAGG_RECIPE_PATH.read_text()) + + assert recipe["frontend"] == { + "type": "vllm-router", + "enable_multiple_frontends": False, + "container_image": "vllm-router-20260716", + "args": { + "policy": "consistent_hash", + "prefill-policy": "consistent_hash", + "decode-policy": "consistent_hash", + }, + } + assert recipe["backend"]["connector"] == "moriio" + assert "dynamo" not in recipe + serialized = DISAGG_RECIPE_PATH.read_text().lower() + assert "nixl" not in serialized + assert "nats" not in serialized + assert "etcd" not in serialized + + def test_fixed_sequence_recipe_uses_inferencex_custom_benchmark(): recipe = yaml.safe_load(RECIPE_PATH.read_text()) benchmark = recipe["benchmark"] From 5b45946d5ff4d959827fb63bba6a026ab093b108 Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Sun, 9 Aug 2026 23:08:04 -0500 Subject: [PATCH 17/67] fix: pin published vllm router image --- .../srt-slurm-recipes/cluster-configs/mi300x-amds.yaml | 2 +- .../vllm/qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml | 4 ++-- utils/test_mi300x_srt_slurm_contract.py | 6 +++--- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi300x-amds.yaml b/benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi300x-amds.yaml index 1f532ac161..21d84329aa 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi300x-amds.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi300x-amds.yaml @@ -37,6 +37,6 @@ default_mounts: # node before submitting a sweep. containers: vllm-rocm-v0.26.0: /raid/hf-hub-cache/inferencex/srt-slurm/containers/vllm-openai-rocm-v0.26.0.sqsh - vllm-router-20260716: /raid/hf-hub-cache/inferencex/srt-slurm/containers/vllm-router-nightly-20260716-1fbcde7.sqsh + vllm-router-20260809: /raid/hf-hub-cache/inferencex/srt-slurm/containers/vllm-router-nightly-20260809-d2ba586.sqsh nginx_raise_ulimit: false diff --git a/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml b/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml index 7709cdedc6..babee1bf3b 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml @@ -15,7 +15,7 @@ identity: image: "vllm/vllm-openai-rocm:v0.26.0" frameworks: vllm: "0.26.0+rocm723" - vllm-router: "nightly-20260716-1fbcde7" + vllm-router: "nightly-20260809-d2ba586" slurm: time_limit: "00:45:00" @@ -33,7 +33,7 @@ resources: frontend: type: vllm-router enable_multiple_frontends: false - container_image: "vllm-router-20260716" + container_image: "vllm-router-20260809" args: policy: consistent_hash prefill-policy: consistent_hash diff --git a/utils/test_mi300x_srt_slurm_contract.py b/utils/test_mi300x_srt_slurm_contract.py index 0573b11b74..232d17695b 100644 --- a/utils/test_mi300x_srt_slurm_contract.py +++ b/utils/test_mi300x_srt_slurm_contract.py @@ -34,8 +34,8 @@ def test_mi300x_cluster_uses_the_rocm_slurm_contract(): assert cluster["default_mounts"]["/dev/dri"] == "/dev/dri" image_path = cluster["containers"]["vllm-rocm-v0.26.0"] assert image_path.endswith("/vllm-openai-rocm-v0.26.0.sqsh") - router_path = cluster["containers"]["vllm-router-20260716"] - assert router_path.endswith("/vllm-router-nightly-20260716-1fbcde7.sqsh") + router_path = cluster["containers"]["vllm-router-20260809"] + assert router_path.endswith("/vllm-router-nightly-20260809-d2ba586.sqsh") for recipe_path in (RECIPE_PATH, DISAGG_RECIPE_PATH): recipe = yaml.safe_load(recipe_path.read_text()) @@ -51,7 +51,7 @@ def test_disaggregated_recipe_uses_native_router_and_moriio(): assert recipe["frontend"] == { "type": "vllm-router", "enable_multiple_frontends": False, - "container_image": "vllm-router-20260716", + "container_image": "vllm-router-20260809", "args": { "policy": "consistent_hash", "prefill-policy": "consistent_hash", From 568327b24f5f7a425a5b2dbc1ec32684640de500 Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Sun, 9 Aug 2026 23:18:09 -0500 Subject: [PATCH 18/67] docs: update AMD runtime pin --- benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md b/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md index 1cae260eaa..2ad779abf7 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md +++ b/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md @@ -9,7 +9,7 @@ Current development pin: - repository: `SemiAnalysisAI/srt-slurm` - branch: `agent/amd-multinode-runtime` -- commit: `7f53c35015e12ef3581f474018b5349ce454e977` +- commit: `3abe13529792e61562c265cd63446f94ea81c090` ## Scope From 1a6c28e4c47a700f3dbcd78edf65b72f5a0a559f Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Sun, 9 Aug 2026 23:32:45 -0500 Subject: [PATCH 19/67] fix(amd): use contiguous AITER cache for MoRI --- benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md | 7 +++++++ .../vllm/qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml | 4 ++++ utils/test_mi300x_srt_slurm_contract.py | 7 +++++++ 3 files changed, 18 insertions(+) diff --git a/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md b/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md index 2ad779abf7..8a75259514 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md +++ b/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md @@ -88,3 +88,10 @@ private NIC names vary across MI300X node generations. The earlier Dynamo/NIXL experiment reached KV-cache initialization but failed ROCm memory registration; that NVIDIA-oriented data plane is now explicitly out of scope rather than patched into the AMD implementation. + +The aggregate recipe has completed end to end on MI300X with both fixed-length +concurrency points. The disaggregated recipe pins ROCm's supported AITER Flash +Attention backend for both roles. This is required by the released MoRI-IO +connector's registered-memory contract: AITER exposes a contiguous logical KV +cache tensor, whereas the default Triton NHD view is strided and cannot be +registered by `mori.io` without copying or patching vLLM. diff --git a/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml b/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml index babee1bf3b..aa48f8edf8 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml @@ -49,6 +49,7 @@ backend: HUGGINGFACE_HUB_CACHE: "/hf_hub_cache" TRANSFORMERS_CACHE: "/hf_hub_cache" PYTHONUNBUFFERED: "1" + VLLM_ROCM_USE_AITER: "1" MORI_IO_SQ_BACKOFF_TIMEOUT_US: "50000" decode_environment: HF_HOME: "/hf_hub_cache" @@ -56,6 +57,7 @@ backend: HUGGINGFACE_HUB_CACHE: "/hf_hub_cache" TRANSFORMERS_CACHE: "/hf_hub_cache" PYTHONUNBUFFERED: "1" + VLLM_ROCM_USE_AITER: "1" MORI_IO_SQ_BACKOFF_TIMEOUT_US: "50000" vllm_config: prefill: @@ -65,6 +67,7 @@ backend: max-model-len: 2048 max-num-seqs: 8 enforce-eager: true + attention-backend: "ROCM_AITER_FA" decode: served-model-name: "Qwen/Qwen3-0.6B" tensor-parallel-size: 1 @@ -72,6 +75,7 @@ backend: max-model-len: 2048 max-num-seqs: 8 enforce-eager: true + attention-backend: "ROCM_AITER_FA" srun_options: container-writable: "" diff --git a/utils/test_mi300x_srt_slurm_contract.py b/utils/test_mi300x_srt_slurm_contract.py index 232d17695b..3e4d9d1c9b 100644 --- a/utils/test_mi300x_srt_slurm_contract.py +++ b/utils/test_mi300x_srt_slurm_contract.py @@ -59,6 +59,13 @@ def test_disaggregated_recipe_uses_native_router_and_moriio(): }, } assert recipe["backend"]["connector"] == "moriio" + for role in ("prefill", "decode"): + assert recipe["backend"][f"{role}_environment"][ + "VLLM_ROCM_USE_AITER" + ] == "1" + assert recipe["backend"]["vllm_config"][role][ + "attention-backend" + ] == "ROCM_AITER_FA" assert "dynamo" not in recipe serialized = DISAGG_RECIPE_PATH.read_text().lower() assert "nixl" not in serialized From 5f0e320a80e4ce7374279531889fc42d6b9c5efe Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Sun, 9 Aug 2026 23:46:50 -0500 Subject: [PATCH 20/67] docs(amd): pin dynamic readiness runtime --- benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md b/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md index 8a75259514..b3ced2c431 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md +++ b/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md @@ -9,7 +9,7 @@ Current development pin: - repository: `SemiAnalysisAI/srt-slurm` - branch: `agent/amd-multinode-runtime` -- commit: `3abe13529792e61562c265cd63446f94ea81c090` +- commit: `d1e42f85a74e88adc7806883cd6a7295f76723a9` ## Scope From 8a59c1ace2e9a438946513d363dd6dc52041cedb Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Sun, 9 Aug 2026 23:53:17 -0500 Subject: [PATCH 21/67] docs(amd): pin CI-compatible runtime --- benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md b/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md index b3ced2c431..6a00243b2c 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md +++ b/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md @@ -9,7 +9,7 @@ Current development pin: - repository: `SemiAnalysisAI/srt-slurm` - branch: `agent/amd-multinode-runtime` -- commit: `d1e42f85a74e88adc7806883cd6a7295f76723a9` +- commit: `bd176ad03fdd8636fdefaaf1a882954cc3c9308d` ## Scope From 7476a9221ab419d51c599792a1f7f5bb97006f8a Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Sun, 9 Aug 2026 23:54:45 -0500 Subject: [PATCH 22/67] docs(amd): pin formatted runtime head --- benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md b/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md index 6a00243b2c..a6a99077e2 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md +++ b/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md @@ -9,7 +9,7 @@ Current development pin: - repository: `SemiAnalysisAI/srt-slurm` - branch: `agent/amd-multinode-runtime` -- commit: `bd176ad03fdd8636fdefaaf1a882954cc3c9308d` +- commit: `cf2d60b821a2f22eafb811c193fccbe31438c43d` ## Scope From 738471eb9633875dd53669d30d01e3dafb1e0e08 Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Mon, 10 Aug 2026 00:00:00 -0500 Subject: [PATCH 23/67] docs: update AMD srt-slurm pin --- benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md b/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md index a6a99077e2..ab57396bab 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md +++ b/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md @@ -9,7 +9,7 @@ Current development pin: - repository: `SemiAnalysisAI/srt-slurm` - branch: `agent/amd-multinode-runtime` -- commit: `cf2d60b821a2f22eafb811c193fccbe31438c43d` +- commit: `411ec5971bac368725f59b0fda419353a6c603aa` ## Scope From 64652fec7d0ee03d7d46e6b5b261941d40a9f444 Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Mon, 10 Aug 2026 00:25:10 -0500 Subject: [PATCH 24/67] ci(amd): validate srt-slurm disaggregation --- .../cluster-configs/mi300x-amds.yaml | 1 + .../vllm/qwen3-0.6b/mi300x/agg-fixed-seq.yaml | 6 +- .../mi300x/disagg-1p1d-fixed-seq.yaml | 6 +- configs/amd-master.yaml | 32 ++++ runners/launch_mi300x-amds-srt.sh | 153 ++++++++++++++++++ runners/launch_mi300x-amds.sh | 4 + utils/test_mi300x_srt_slurm_contract.py | 30 +++- 7 files changed, 227 insertions(+), 5 deletions(-) create mode 100644 runners/launch_mi300x-amds-srt.sh diff --git a/benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi300x-amds.yaml b/benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi300x-amds.yaml index 21d84329aa..ac3952a8fb 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi300x-amds.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi300x-amds.yaml @@ -31,6 +31,7 @@ default_mounts: /dev/kfd: /dev/kfd /dev/dri: /dev/dri /raid/hf-hub-cache: /hf_hub_cache + # The launcher appends per-run /infmax-workspace and /results mounts here. # Pyxis otherwise re-imports the large ROCm image for every srun step. The # validation launcher materializes this immutable squashfs on each healthy diff --git a/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/agg-fixed-seq.yaml b/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/agg-fixed-seq.yaml index 9627bdd49e..2f201560be 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/agg-fixed-seq.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/agg-fixed-seq.yaml @@ -62,7 +62,9 @@ benchmark: type: custom command: | set -euo pipefail - mkdir -p /logs/fixed-seq + result_root="/results/${SLURM_JOB_ID}" + mkdir -p "${result_root}/fixed-seq" + trap 'tar -C /logs -czf "'"${result_root}"'/runtime-logs.tar.gz" . 2>/dev/null || true' EXIT for concurrency in 1 4; do python3 /infmax-workspace/utils/bench_serving/benchmark_serving.py \ --backend openai-chat \ @@ -82,7 +84,7 @@ benchmark: --ignore-eos \ --disable-tqdm \ --save-result \ - --result-dir /logs/fixed-seq \ + --result-dir "${result_root}/fixed-seq" \ --result-filename "qwen3-0.6b-isl128-osl32-c${concurrency}.json"; done env: diff --git a/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml b/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml index aa48f8edf8..239e2b6cc4 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml @@ -90,7 +90,9 @@ benchmark: type: custom command: | set -euo pipefail - mkdir -p /logs/fixed-seq + result_root="/results/${SLURM_JOB_ID}" + mkdir -p "${result_root}/fixed-seq" + trap 'tar -C /logs -czf "'"${result_root}"'/runtime-logs.tar.gz" . 2>/dev/null || true' EXIT for concurrency in 1 4; do python3 /infmax-workspace/utils/bench_serving/benchmark_serving.py \ --backend openai-chat \ @@ -110,7 +112,7 @@ benchmark: --ignore-eos \ --disable-tqdm \ --save-result \ - --result-dir /logs/fixed-seq \ + --result-dir "${result_root}/fixed-seq" \ --result-filename "qwen3-0.6b-disagg-isl128-osl32-c${concurrency}.json"; done env: diff --git a/configs/amd-master.yaml b/configs/amd-master.yaml index c6580af43d..5cc0eac408 100644 --- a/configs/amd-master.yaml +++ b/configs/amd-master.yaml @@ -1497,6 +1497,38 @@ minimaxm3-fp8-mi325x-vllm-agentic: - { tp: 8, ep: 8, kv-offloading: dram, kv-offload-backend: { name: mooncake, version: "0.3.11.post1" }, conc-list: [10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 32] } - { tp: 8, ep: 8, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: mooncake, version: "0.3.11.post1" }, conc-list: [24, 32, 36, 40, 44, 48, 52, 56, 60, 64, 72, 80, 96], router: { name: vllm-router, version: "0.1.14" } } +# Small-model orchestration validation for SemiAnalysisAI/srt-slurm on AMD. +# This is intentionally a correctness lane, not a performance submission. +qwen3-0.6b-fp16-mi300x-vllm-srt-disagg: + image: vllm/vllm-openai-rocm:v0.26.0 + model: Qwen/Qwen3-0.6B + model-prefix: qwen3-0.6b + runner: mi300x-disagg + precision: fp16 + framework: vllm + multinode: true + disagg: true + router: { name: vllm-router, version: "nightly-20260809-d2ba586" } + kv-p2p-transfer: moriio + scenarios: + fixed-seq-len: + - isl: 128 + osl: 32 + search-space: + - conc-list: [1, 4] + prefill: + num-worker: 1 + tp: 1 + ep: 1 + dp-attn: false + additional-settings: + - "CONFIG_FILE=recipes/vllm/qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml" + decode: + num-worker: 1 + tp: 1 + ep: 1 + dp-attn: false + minimaxm3-fp4-mi355x-vllm-agentic: image: vllm/vllm-openai-rocm:nightly-dcfebf93f4eccf30f71872283331eee757915daf model: amd/MiniMax-M3-MXFP4 diff --git a/runners/launch_mi300x-amds-srt.sh b/runners/launch_mi300x-amds-srt.sh new file mode 100644 index 0000000000..7e53cb9b4c --- /dev/null +++ b/runners/launch_mi300x-amds-srt.sh @@ -0,0 +1,153 @@ +#!/usr/bin/env bash +set -euo pipefail + +# MI300X srt-slurm validation path. The existing launcher remains the default; +# matrix rows opt in by exporting CONFIG_FILE through additional-settings. +SRT_SLURM_REPOSITORY="https://github.com/SemiAnalysisAI/srt-slurm.git" +SRT_SLURM_COMMIT="411ec5971bac368725f59b0fda419353a6c603aa" +SLURM_PARTITION="compute" +EXCLUDED_NODES="chi-mi300x-049,chi-mi300x-121" +REMOTE_BASE="/raid/hf-hub-cache/inferencex/srt-slurm" + +: "${CONFIG_FILE:?CONFIG_FILE must name an srt-slurm recipe}" +: "${GITHUB_WORKSPACE:?GITHUB_WORKSPACE must be set by Actions}" +: "${RESULT_FILENAME:?RESULT_FILENAME must be set by the benchmark workflow}" + +CONFIG_PATH="${CONFIG_FILE%%:*}" +LOCAL_RECIPE="${GITHUB_WORKSPACE}/benchmarks/multi_node/srt-slurm-recipes/${CONFIG_PATH#recipes/}" +CLUSTER_PROFILE="${GITHUB_WORKSPACE}/benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi300x-amds.yaml" +[[ -f "$LOCAL_RECIPE" ]] || { echo "Missing recipe: $LOCAL_RECIPE" >&2; exit 1; } +[[ -f "$CLUSTER_PROFILE" ]] || { echo "Missing cluster profile: $CLUSTER_PROFILE" >&2; exit 1; } + +RUN_KEY="${GITHUB_RUN_ID:-manual}-${GITHUB_RUN_ATTEMPT:-0}-${RUNNER_NAME:-runner}" +REMOTE_RUNTIME="${REMOTE_BASE}/runtime/inferencex-${RUN_KEY}" +REMOTE_RESULTS="${REMOTE_BASE}/results" +WORK_DIR="${GITHUB_WORKSPACE}/.srt-slurm-${RUN_KEY}" +SRT_REPO_DIR="${WORK_DIR}/srt-slurm" +mkdir -p "$WORK_DIR" + +# The login and compute nodes do not share a filesystem. Stage only the +# unchanged InferenceX benchmark client onto every eligible node. The batch job +# exits normally; it does not cancel or preempt any allocation. +RUNTIME_ARCHIVE="${WORK_DIR}/inferencex-benchmark.tar.gz" +tar -C "$GITHUB_WORKSPACE" -czf "$RUNTIME_ARCHIVE" utils/bench_serving +RUNTIME_PAYLOAD=$(base64 -w0 "$RUNTIME_ARCHIVE") +STAGE_SCRIPT="${WORK_DIR}/stage-runtime.sbatch" +cat > "$STAGE_SCRIPT" < "\$source_archive" +sbcast --force "\$source_archive" "\$node_archive" +srun --ntasks-per-node=1 bash -c ' + set -euo pipefail + runtime="${REMOTE_RUNTIME}" + mkdir -p "\$runtime" "${REMOTE_RESULTS}" + tar -xzf "/tmp/inferencex-benchmark-\${SLURM_JOB_ID}.tar.gz" -C "\$runtime" + printf "%s\\n" "${GITHUB_SHA:-unknown}" > "\$runtime/.inferencex-source-head" +' +EOF +STAGE_JOB_ID=$(sbatch --wait --parsable "$STAGE_SCRIPT") +echo "Staged InferenceX benchmark client with Slurm job ${STAGE_JOB_ID}" + +git clone "$SRT_SLURM_REPOSITORY" "$SRT_REPO_DIR" +git -C "$SRT_REPO_DIR" checkout "$SRT_SLURM_COMMIT" +ACTUAL_SRT_COMMIT=$(git -C "$SRT_REPO_DIR" rev-parse HEAD) +[[ "$ACTUAL_SRT_COMMIT" == "$SRT_SLURM_COMMIT" ]] || { + echo "srt-slurm checkout mismatch: $ACTUAL_SRT_COMMIT" >&2 + exit 1 +} + +mkdir -p "${SRT_REPO_DIR}/$(dirname "$CONFIG_PATH")" +cp "$LOCAL_RECIPE" "${SRT_REPO_DIR}/${CONFIG_PATH}" +cp "$CLUSTER_PROFILE" "${WORK_DIR}/srtslurm.yaml" +python3 - "${WORK_DIR}/srtslurm.yaml" "$REMOTE_RUNTIME" "$REMOTE_RESULTS" <<'PY' +import sys +from pathlib import Path + +path = Path(sys.argv[1]) +runtime, results = sys.argv[2:] +needle = " /raid/hf-hub-cache: /hf_hub_cache\n" +text = path.read_text() +if text.count(needle) != 1: + raise SystemExit("expected exactly one Hugging Face cache mount") +path.write_text( + text.replace( + needle, + needle + f" {runtime}: /infmax-workspace\n {results}: /results\n", + ) +) +PY + +export PATH="$HOME/.local/bin:$PATH" +cd "$SRT_REPO_DIR" +uv venv --python 3.12 +uv pip install -e . +source .venv/bin/activate +export SRTSLURM_CONFIG="${WORK_DIR}/srtslurm.yaml" +export INFMAX_WORKSPACE="$REMOTE_RUNTIME" + +echo "Submitting ${CONFIG_PATH} with srt-slurm ${SRT_SLURM_COMMIT}" +SRTCTL_OUTPUT=$(srtctl apply -f "$CONFIG_FILE" \ + --tags "mi300x,inferencex,github-actions,${RUN_KEY}" 2>&1) +echo "$SRTCTL_OUTPUT" +JOB_ID=$(grep -oE 'Job [0-9]+' <<< "$SRTCTL_OUTPUT" | awk '{print $2}' | tail -1) +[[ -n "$JOB_ID" ]] || { echo "Unable to parse srt-slurm job ID" >&2; exit 1; } +echo "SRT_SLURM_JOB_ID=$JOB_ID" + +while squeue --noheader --jobs "$JOB_ID" | grep -q .; do + squeue --noheader --jobs "$JOB_ID" --format='srt-slurm %i %T %M %R' + sleep 15 +done + +read -r JOB_STATE JOB_EXIT JOB_NODE < <( + sacct -X --noheader --parsable2 --jobs "$JOB_ID" \ + --format=State,ExitCode,NodeList | head -1 | tr '|' ' ' +) +echo "srt-slurm job ${JOB_ID}: state=${JOB_STATE} exit=${JOB_EXIT} node=${JOB_NODE}" + +# Results live on the allocation's node-local RAID. Retrieve the small result +# bundle with a separate completed Slurm job on the batch node. +RETRIEVE_DIR="${WORK_DIR}/retrieved" +mkdir -p "$RETRIEVE_DIR" +RESULT_PAYLOAD=$(srun --partition="$SLURM_PARTITION" --nodes=1 --ntasks=1 \ + --cpus-per-task=1 --time=00:05:00 --nodelist="$JOB_NODE" \ + bash -c "tar -C '${REMOTE_RESULTS}/${JOB_ID}' -czf - . | base64 -w0") +printf '%s' "$RESULT_PAYLOAD" | base64 -d | tar -xzf - -C "$RETRIEVE_DIR" + +mkdir -p "$GITHUB_WORKSPACE/LOGS" +if [[ -f "$RETRIEVE_DIR/runtime-logs.tar.gz" ]]; then + cp "$RETRIEVE_DIR/runtime-logs.tar.gz" "$GITHUB_WORKSPACE/multinode_server_logs.tar.gz" +fi +cp -R "$RETRIEVE_DIR/." "$GITHUB_WORKSPACE/LOGS/" + +PREFILL_GPUS=$((PREFILL_NUM_WORKERS * PREFILL_TP)) +DECODE_GPUS=$((DECODE_NUM_WORKERS * DECODE_TP)) +TOTAL_GPUS=$((PREFILL_GPUS + DECODE_GPUS)) +shopt -s nullglob +RESULTS=("$RETRIEVE_DIR"/fixed-seq/*.json) +shopt -u nullglob +[[ ${#RESULTS[@]} -gt 0 ]] || { echo "No fixed-sequence results retrieved" >&2; exit 1; } +for result in "${RESULTS[@]}"; do + concurrency=$(basename "$result" | sed -n 's/.*-c\([0-9][0-9]*\)\.json/\1/p') + [[ -n "$concurrency" ]] || { echo "Cannot parse concurrency from $result" >&2; exit 1; } + output="${GITHUB_WORKSPACE}/${RESULT_FILENAME}_srt-${JOB_ID}_conc${concurrency}_gpus_${TOTAL_GPUS}_ctx_${PREFILL_GPUS}_gen_${DECODE_GPUS}.json" + cp "$result" "$output" + echo "Collected $output" +done + +if [[ "$JOB_STATE" != COMPLETED || "$JOB_EXIT" != 0:0 ]]; then + echo "srt-slurm validation failed: ${JOB_STATE} (${JOB_EXIT})" >&2 + exit 1 +fi + +printf '%s\n' "$SRT_SLURM_COMMIT" > "$GITHUB_WORKSPACE/srt-slurm-producer-sha.txt" +echo "MI300X srt-slurm validation completed successfully" diff --git a/runners/launch_mi300x-amds.sh b/runners/launch_mi300x-amds.sh index fdd03889a0..a69e22edbe 100644 --- a/runners/launch_mi300x-amds.sh +++ b/runners/launch_mi300x-amds.sh @@ -1,6 +1,10 @@ #!/usr/bin/env bash set -eo pipefail +if [[ -n "${CONFIG_FILE:-}" ]]; then + exec bash "$(dirname "${BASH_SOURCE[0]}")/launch_mi300x-amds-srt.sh" +fi + export HF_HUB_CACHE_MOUNT="/raid/hf-hub-cache/" export PORT=8888 diff --git a/utils/test_mi300x_srt_slurm_contract.py b/utils/test_mi300x_srt_slurm_contract.py index 3e4d9d1c9b..d0c1777153 100644 --- a/utils/test_mi300x_srt_slurm_contract.py +++ b/utils/test_mi300x_srt_slurm_contract.py @@ -17,6 +17,8 @@ REPO_ROOT / "benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi300x-amds.yaml" ) +MASTER_CONFIG_PATH = REPO_ROOT / "configs/amd-master.yaml" +SRT_LAUNCHER_PATH = REPO_ROOT / "runners/launch_mi300x-amds-srt.sh" def test_mi300x_cluster_uses_the_rocm_slurm_contract(): @@ -73,6 +75,27 @@ def test_disaggregated_recipe_uses_native_router_and_moriio(): assert "etcd" not in serialized +def test_official_matrix_routes_disagg_through_the_pinned_srt_launcher(): + config = yaml.safe_load(MASTER_CONFIG_PATH.read_text())[ + "qwen3-0.6b-fp16-mi300x-vllm-srt-disagg" + ] + search = config["scenarios"]["fixed-seq-len"][0]["search-space"][0] + launcher = SRT_LAUNCHER_PATH.read_text() + + assert config["runner"] == "mi300x-disagg" + assert config["router"] == { + "name": "vllm-router", + "version": "nightly-20260809-d2ba586", + } + assert config["kv-p2p-transfer"] == "moriio" + assert search["prefill"]["additional-settings"] == [ + "CONFIG_FILE=recipes/vllm/qwen3-0.6b/mi300x/" + "disagg-1p1d-fixed-seq.yaml" + ] + assert "411ec5971bac368725f59b0fda419353a6c603aa" in launcher + assert "scancel" not in launcher + + def test_fixed_sequence_recipe_uses_inferencex_custom_benchmark(): recipe = yaml.safe_load(RECIPE_PATH.read_text()) benchmark = recipe["benchmark"] @@ -80,6 +103,7 @@ def test_fixed_sequence_recipe_uses_inferencex_custom_benchmark(): assert benchmark["type"] == "custom" assert "/infmax-workspace/utils/bench_serving/benchmark_serving.py" in command + assert 'result_root="/results/${SLURM_JOB_ID}"' in command assert "--backend openai-chat" in command assert "--endpoint /v1/chat/completions" in command assert "--random-input-len 128" in command @@ -103,7 +127,10 @@ def test_fixed_sequence_commands_keep_all_arguments_attached(tmp_path): for recipe_path in (RECIPE_PATH, DISAGG_RECIPE_PATH): command = yaml.safe_load(recipe_path.read_text())["benchmark"]["command"] result_dir = tmp_path / recipe_path.stem - command = command.replace("/logs/fixed-seq", str(result_dir)) + command = command.replace( + 'result_root="/results/${SLURM_JOB_ID}"', + f'result_root="{result_dir}"', + ) args_log = tmp_path / f"{recipe_path.stem}.args" env = { **os.environ, @@ -111,6 +138,7 @@ def test_fixed_sequence_commands_keep_all_arguments_attached(tmp_path): "FAKE_ARGS_LOG": str(args_log), "SRT_FRONTEND_HOST": "127.0.0.1", "SRT_FRONTEND_PORT": "8000", + "SLURM_JOB_ID": "123", } subprocess.run(["bash", "-n"], input=command, text=True, check=True) From d548a4f2ca4615a15ecfc8ddac13451578e73b85 Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Mon, 10 Aug 2026 00:34:17 -0500 Subject: [PATCH 25/67] ci(amd): bootstrap srt-slurm compute runtime --- benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md | 2 +- runners/launch_mi300x-amds-srt.sh | 10 +++++++++- utils/test_mi300x_srt_slurm_contract.py | 3 ++- 3 files changed, 12 insertions(+), 3 deletions(-) diff --git a/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md b/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md index ab57396bab..6b4aed6c2f 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md +++ b/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md @@ -9,7 +9,7 @@ Current development pin: - repository: `SemiAnalysisAI/srt-slurm` - branch: `agent/amd-multinode-runtime` -- commit: `411ec5971bac368725f59b0fda419353a6c603aa` +- commit: `105824810a67d52b58761077ad3b94d4a05eb3ac` ## Scope diff --git a/runners/launch_mi300x-amds-srt.sh b/runners/launch_mi300x-amds-srt.sh index 7e53cb9b4c..f92a178d1a 100644 --- a/runners/launch_mi300x-amds-srt.sh +++ b/runners/launch_mi300x-amds-srt.sh @@ -4,7 +4,7 @@ set -euo pipefail # MI300X srt-slurm validation path. The existing launcher remains the default; # matrix rows opt in by exporting CONFIG_FILE through additional-settings. SRT_SLURM_REPOSITORY="https://github.com/SemiAnalysisAI/srt-slurm.git" -SRT_SLURM_COMMIT="411ec5971bac368725f59b0fda419353a6c603aa" +SRT_SLURM_COMMIT="105824810a67d52b58761077ad3b94d4a05eb3ac" SLURM_PARTITION="compute" EXCLUDED_NODES="chi-mi300x-049,chi-mi300x-121" REMOTE_BASE="/raid/hf-hub-cache/inferencex/srt-slurm" @@ -91,14 +91,22 @@ export PATH="$HOME/.local/bin:$PATH" cd "$SRT_REPO_DIR" uv venv --python 3.12 uv pip install -e . +make setup-compute ARCH=x86_64 source .venv/bin/activate export SRTSLURM_CONFIG="${WORK_DIR}/srtslurm.yaml" export INFMAX_WORKSPACE="$REMOTE_RUNTIME" echo "Submitting ${CONFIG_PATH} with srt-slurm ${SRT_SLURM_COMMIT}" +set +e SRTCTL_OUTPUT=$(srtctl apply -f "$CONFIG_FILE" \ --tags "mi300x,inferencex,github-actions,${RUN_KEY}" 2>&1) +SRTCTL_RC=$? +set -e echo "$SRTCTL_OUTPUT" +if [[ $SRTCTL_RC -ne 0 ]]; then + echo "srtctl apply failed with exit code ${SRTCTL_RC}" >&2 + exit "$SRTCTL_RC" +fi JOB_ID=$(grep -oE 'Job [0-9]+' <<< "$SRTCTL_OUTPUT" | awk '{print $2}' | tail -1) [[ -n "$JOB_ID" ]] || { echo "Unable to parse srt-slurm job ID" >&2; exit 1; } echo "SRT_SLURM_JOB_ID=$JOB_ID" diff --git a/utils/test_mi300x_srt_slurm_contract.py b/utils/test_mi300x_srt_slurm_contract.py index d0c1777153..4672fe3f9f 100644 --- a/utils/test_mi300x_srt_slurm_contract.py +++ b/utils/test_mi300x_srt_slurm_contract.py @@ -92,7 +92,8 @@ def test_official_matrix_routes_disagg_through_the_pinned_srt_launcher(): "CONFIG_FILE=recipes/vllm/qwen3-0.6b/mi300x/" "disagg-1p1d-fixed-seq.yaml" ] - assert "411ec5971bac368725f59b0fda419353a6c603aa" in launcher + assert "105824810a67d52b58761077ad3b94d4a05eb3ac" in launcher + assert "make setup-compute ARCH=x86_64" in launcher assert "scancel" not in launcher From 8a1f8ccb465ddf4f2a6bd176689a18355aadaad2 Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Mon, 10 Aug 2026 00:38:50 -0500 Subject: [PATCH 26/67] ci(amd): validate node-local runtime images --- benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md | 6 ++++++ runners/launch_mi300x-amds-srt.sh | 3 +++ utils/test_mi300x_srt_slurm_contract.py | 3 +++ 3 files changed, 12 insertions(+) diff --git a/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md b/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md index 6b4aed6c2f..13aad4ce2c 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md +++ b/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md @@ -11,6 +11,12 @@ Current development pin: - branch: `agent/amd-multinode-runtime` - commit: `105824810a67d52b58761077ad3b94d4a05eb3ac` +The MI300X launcher uses srt-slurm's supported `--no-preflight` submission mode +because the immutable squashfs files live on compute-node-local RAID rather +than the login node. Before submission, the launcher stages the benchmark +runtime across the same eligible node pool; missing container files still fail +loudly when Pyxis starts the allocation. + ## Scope 1. Prove a single-node aggregate vLLM deployment on MI300X. diff --git a/runners/launch_mi300x-amds-srt.sh b/runners/launch_mi300x-amds-srt.sh index f92a178d1a..8ecf1f1166 100644 --- a/runners/launch_mi300x-amds-srt.sh +++ b/runners/launch_mi300x-amds-srt.sh @@ -51,6 +51,8 @@ srun --ntasks-per-node=1 bash -c ' set -euo pipefail runtime="${REMOTE_RUNTIME}" mkdir -p "\$runtime" "${REMOTE_RESULTS}" + test -r /raid/hf-hub-cache/inferencex/srt-slurm/containers/vllm-openai-rocm-v0.26.0.sqsh + test -r /raid/hf-hub-cache/inferencex/srt-slurm/containers/vllm-router-nightly-20260809-d2ba586.sqsh tar -xzf "/tmp/inferencex-benchmark-\${SLURM_JOB_ID}.tar.gz" -C "\$runtime" printf "%s\\n" "${GITHUB_SHA:-unknown}" > "\$runtime/.inferencex-source-head" ' @@ -99,6 +101,7 @@ export INFMAX_WORKSPACE="$REMOTE_RUNTIME" echo "Submitting ${CONFIG_PATH} with srt-slurm ${SRT_SLURM_COMMIT}" set +e SRTCTL_OUTPUT=$(srtctl apply -f "$CONFIG_FILE" \ + --no-preflight \ --tags "mi300x,inferencex,github-actions,${RUN_KEY}" 2>&1) SRTCTL_RC=$? set -e diff --git a/utils/test_mi300x_srt_slurm_contract.py b/utils/test_mi300x_srt_slurm_contract.py index 4672fe3f9f..4c3874c141 100644 --- a/utils/test_mi300x_srt_slurm_contract.py +++ b/utils/test_mi300x_srt_slurm_contract.py @@ -94,6 +94,9 @@ def test_official_matrix_routes_disagg_through_the_pinned_srt_launcher(): ] assert "105824810a67d52b58761077ad3b94d4a05eb3ac" in launcher assert "make setup-compute ARCH=x86_64" in launcher + assert "--no-preflight" in launcher + assert "test -r /raid/hf-hub-cache/inferencex/srt-slurm/containers/vllm-openai-rocm-v0.26.0.sqsh" in launcher + assert "test -r /raid/hf-hub-cache/inferencex/srt-slurm/containers/vllm-router-nightly-20260809-d2ba586.sqsh" in launcher assert "scancel" not in launcher From 03c480df96577ea47c68991a13c62387a635b851 Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Mon, 10 Aug 2026 00:43:52 -0500 Subject: [PATCH 27/67] ci(amd): stage srt-slurm on compute nodes --- benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md | 6 ++++++ runners/launch_mi300x-amds-srt.sh | 10 ++++++++++ utils/test_mi300x_srt_slurm_contract.py | 4 ++++ 3 files changed, 20 insertions(+) diff --git a/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md b/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md index 13aad4ce2c..d8226fdbcb 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md +++ b/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md @@ -17,6 +17,12 @@ than the login node. Before submission, the launcher stages the benchmark runtime across the same eligible node pool; missing container files still fail loudly when Pyxis starts the allocation. +The MI300X login and compute nodes also do not share the Actions checkout. The +staging allocation checks out the exact pinned srt-slurm commit on every +eligible compute node, installs its compute-only runtime, and injects that +node-local path through srt-slurm's `SRTCTL_RUNTIME_SOURCE_DIR` transport +override. The submitter continues to validate against its local pinned checkout. + ## Scope 1. Prove a single-node aggregate vLLM deployment on MI300X. diff --git a/runners/launch_mi300x-amds-srt.sh b/runners/launch_mi300x-amds-srt.sh index 8ecf1f1166..cc9a644ff6 100644 --- a/runners/launch_mi300x-amds-srt.sh +++ b/runners/launch_mi300x-amds-srt.sh @@ -21,6 +21,7 @@ CLUSTER_PROFILE="${GITHUB_WORKSPACE}/benchmarks/multi_node/srt-slurm-recipes/clu RUN_KEY="${GITHUB_RUN_ID:-manual}-${GITHUB_RUN_ATTEMPT:-0}-${RUNNER_NAME:-runner}" REMOTE_RUNTIME="${REMOTE_BASE}/runtime/inferencex-${RUN_KEY}" +REMOTE_SRT_RUNTIME="${REMOTE_BASE}/runtime/srt-slurm-${SRT_SLURM_COMMIT}" REMOTE_RESULTS="${REMOTE_BASE}/results" WORK_DIR="${GITHUB_WORKSPACE}/.srt-slurm-${RUN_KEY}" SRT_REPO_DIR="${WORK_DIR}/srt-slurm" @@ -50,9 +51,17 @@ sbcast --force "\$source_archive" "\$node_archive" srun --ntasks-per-node=1 bash -c ' set -euo pipefail runtime="${REMOTE_RUNTIME}" + srt_runtime="${REMOTE_SRT_RUNTIME}" mkdir -p "\$runtime" "${REMOTE_RESULTS}" test -r /raid/hf-hub-cache/inferencex/srt-slurm/containers/vllm-openai-rocm-v0.26.0.sqsh test -r /raid/hf-hub-cache/inferencex/srt-slurm/containers/vllm-router-nightly-20260809-d2ba586.sqsh + if [[ ! -d "\$srt_runtime/.git" ]]; then + git clone --quiet "${SRT_SLURM_REPOSITORY}" "\$srt_runtime" + fi + git -C "\$srt_runtime" fetch --quiet origin "${SRT_SLURM_COMMIT}" + git -C "\$srt_runtime" checkout --quiet --detach "${SRT_SLURM_COMMIT}" + test "\$(git -C "\$srt_runtime" rev-parse HEAD)" = "${SRT_SLURM_COMMIT}" + make -C "\$srt_runtime" --no-print-directory setup-compute ARCH=x86_64 tar -xzf "/tmp/inferencex-benchmark-\${SLURM_JOB_ID}.tar.gz" -C "\$runtime" printf "%s\\n" "${GITHUB_SHA:-unknown}" > "\$runtime/.inferencex-source-head" ' @@ -96,6 +105,7 @@ uv pip install -e . make setup-compute ARCH=x86_64 source .venv/bin/activate export SRTSLURM_CONFIG="${WORK_DIR}/srtslurm.yaml" +export SRTCTL_RUNTIME_SOURCE_DIR="$REMOTE_SRT_RUNTIME" export INFMAX_WORKSPACE="$REMOTE_RUNTIME" echo "Submitting ${CONFIG_PATH} with srt-slurm ${SRT_SLURM_COMMIT}" diff --git a/utils/test_mi300x_srt_slurm_contract.py b/utils/test_mi300x_srt_slurm_contract.py index 4c3874c141..67709e2ccb 100644 --- a/utils/test_mi300x_srt_slurm_contract.py +++ b/utils/test_mi300x_srt_slurm_contract.py @@ -97,6 +97,10 @@ def test_official_matrix_routes_disagg_through_the_pinned_srt_launcher(): assert "--no-preflight" in launcher assert "test -r /raid/hf-hub-cache/inferencex/srt-slurm/containers/vllm-openai-rocm-v0.26.0.sqsh" in launcher assert "test -r /raid/hf-hub-cache/inferencex/srt-slurm/containers/vllm-router-nightly-20260809-d2ba586.sqsh" in launcher + assert 'REMOTE_SRT_RUNTIME="${REMOTE_BASE}/runtime/srt-slurm-${SRT_SLURM_COMMIT}"' in launcher + assert 'git -C "\\$srt_runtime" checkout --quiet --detach "${SRT_SLURM_COMMIT}"' in launcher + assert 'make -C "\\$srt_runtime" --no-print-directory setup-compute ARCH=x86_64' in launcher + assert 'export SRTCTL_RUNTIME_SOURCE_DIR="$REMOTE_SRT_RUNTIME"' in launcher assert "scancel" not in launcher From ba5e7c77dacee155b2bb96e686f59871b5d04c96 Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Mon, 10 Aug 2026 00:57:44 -0500 Subject: [PATCH 28/67] ci(amd): validate aggregate srt-slurm serving --- configs/amd-master.yaml | 17 +++++++++++ runners/launch_mi300x-amds-srt.sh | 36 ++++++++++++++++++----- utils/test_mi300x_srt_slurm_contract.py | 39 +++++++++++++++++++++++++ 3 files changed, 84 insertions(+), 8 deletions(-) diff --git a/configs/amd-master.yaml b/configs/amd-master.yaml index 5cc0eac408..e586041406 100644 --- a/configs/amd-master.yaml +++ b/configs/amd-master.yaml @@ -1499,6 +1499,23 @@ minimaxm3-fp8-mi325x-vllm-agentic: # Small-model orchestration validation for SemiAnalysisAI/srt-slurm on AMD. # This is intentionally a correctness lane, not a performance submission. +qwen3-0.6b-fp16-mi300x-vllm-srt-agg: + image: vllm/vllm-openai-rocm:v0.26.0 + model: Qwen/Qwen3-0.6B + model-prefix: qwen3-0.6b + runner: mi300x-disagg + precision: fp16 + framework: vllm + multinode: false + scenarios: + fixed-seq-len: + - isl: 128 + osl: 32 + search-space: + # The srt-slurm custom benchmark exercises both c1 and c4 in one + # allocation; keep one matrix point so the server is launched once. + - { tp: 1, conc-list: [1] } + qwen3-0.6b-fp16-mi300x-vllm-srt-disagg: image: vllm/vllm-openai-rocm:v0.26.0 model: Qwen/Qwen3-0.6B diff --git a/runners/launch_mi300x-amds-srt.sh b/runners/launch_mi300x-amds-srt.sh index cc9a644ff6..6b3f3090a1 100644 --- a/runners/launch_mi300x-amds-srt.sh +++ b/runners/launch_mi300x-amds-srt.sh @@ -9,10 +9,17 @@ SLURM_PARTITION="compute" EXCLUDED_NODES="chi-mi300x-049,chi-mi300x-121" REMOTE_BASE="/raid/hf-hub-cache/inferencex/srt-slurm" -: "${CONFIG_FILE:?CONFIG_FILE must name an srt-slurm recipe}" : "${GITHUB_WORKSPACE:?GITHUB_WORKSPACE must be set by Actions}" : "${RESULT_FILENAME:?RESULT_FILENAME must be set by the benchmark workflow}" +# The existing MI300X orchestration-capable runner pool defaults to the +# aggregate recipe. Multinode rows continue to pass their recipe explicitly +# through the existing worker additional-settings contract. +if [[ -z "${CONFIG_FILE:-}" && "${RUNNER_TYPE:-}" == "mi300x-disagg" ]]; then + CONFIG_FILE="recipes/vllm/qwen3-0.6b/mi300x/agg-fixed-seq.yaml" +fi +: "${CONFIG_FILE:?CONFIG_FILE must name an srt-slurm recipe}" + CONFIG_PATH="${CONFIG_FILE%%:*}" LOCAL_RECIPE="${GITHUB_WORKSPACE}/benchmarks/multi_node/srt-slurm-recipes/${CONFIG_PATH#recipes/}" CLUSTER_PROFILE="${GITHUB_WORKSPACE}/benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi300x-amds.yaml" @@ -129,18 +136,23 @@ while squeue --noheader --jobs "$JOB_ID" | grep -q .; do sleep 15 done -read -r JOB_STATE JOB_EXIT JOB_NODE < <( +read -r JOB_STATE JOB_EXIT JOB_NODELIST < <( sacct -X --noheader --parsable2 --jobs "$JOB_ID" \ --format=State,ExitCode,NodeList | head -1 | tr '|' ' ' ) -echo "srt-slurm job ${JOB_ID}: state=${JOB_STATE} exit=${JOB_EXIT} node=${JOB_NODE}" +JOB_BATCH_HOST=$(scontrol show job "$JOB_ID" -dd | sed -n 's/.*BatchHost=\([^ ]*\).*/\1/p' | head -1) +[[ -n "$JOB_BATCH_HOST" ]] || { + echo "Unable to resolve BatchHost for srt-slurm job ${JOB_ID}" >&2 + exit 1 +} +echo "srt-slurm job ${JOB_ID}: state=${JOB_STATE} exit=${JOB_EXIT} nodes=${JOB_NODELIST} batch_host=${JOB_BATCH_HOST}" # Results live on the allocation's node-local RAID. Retrieve the small result # bundle with a separate completed Slurm job on the batch node. RETRIEVE_DIR="${WORK_DIR}/retrieved" mkdir -p "$RETRIEVE_DIR" RESULT_PAYLOAD=$(srun --partition="$SLURM_PARTITION" --nodes=1 --ntasks=1 \ - --cpus-per-task=1 --time=00:05:00 --nodelist="$JOB_NODE" \ + --cpus-per-task=1 --time=00:05:00 --nodelist="$JOB_BATCH_HOST" \ bash -c "tar -C '${REMOTE_RESULTS}/${JOB_ID}' -czf - . | base64 -w0") printf '%s' "$RESULT_PAYLOAD" | base64 -d | tar -xzf - -C "$RETRIEVE_DIR" @@ -150,9 +162,13 @@ if [[ -f "$RETRIEVE_DIR/runtime-logs.tar.gz" ]]; then fi cp -R "$RETRIEVE_DIR/." "$GITHUB_WORKSPACE/LOGS/" -PREFILL_GPUS=$((PREFILL_NUM_WORKERS * PREFILL_TP)) -DECODE_GPUS=$((DECODE_NUM_WORKERS * DECODE_TP)) -TOTAL_GPUS=$((PREFILL_GPUS + DECODE_GPUS)) +if [[ "${DISAGG:-false}" == "true" ]]; then + PREFILL_GPUS=$((PREFILL_NUM_WORKERS * PREFILL_TP)) + DECODE_GPUS=$((DECODE_NUM_WORKERS * DECODE_TP)) + TOTAL_GPUS=$((PREFILL_GPUS + DECODE_GPUS)) +else + TOTAL_GPUS=$((TP * ${PP_SIZE:-1} * ${PCP_SIZE:-1})) +fi shopt -s nullglob RESULTS=("$RETRIEVE_DIR"/fixed-seq/*.json) shopt -u nullglob @@ -160,7 +176,11 @@ shopt -u nullglob for result in "${RESULTS[@]}"; do concurrency=$(basename "$result" | sed -n 's/.*-c\([0-9][0-9]*\)\.json/\1/p') [[ -n "$concurrency" ]] || { echo "Cannot parse concurrency from $result" >&2; exit 1; } - output="${GITHUB_WORKSPACE}/${RESULT_FILENAME}_srt-${JOB_ID}_conc${concurrency}_gpus_${TOTAL_GPUS}_ctx_${PREFILL_GPUS}_gen_${DECODE_GPUS}.json" + if [[ "${DISAGG:-false}" == "true" ]]; then + output="${GITHUB_WORKSPACE}/${RESULT_FILENAME}_srt-${JOB_ID}_conc${concurrency}_gpus_${TOTAL_GPUS}_ctx_${PREFILL_GPUS}_gen_${DECODE_GPUS}.json" + else + output="${GITHUB_WORKSPACE}/${RESULT_FILENAME}_srt-${JOB_ID}_conc${concurrency}_gpus_${TOTAL_GPUS}.json" + fi cp "$result" "$output" echo "Collected $output" done diff --git a/utils/test_mi300x_srt_slurm_contract.py b/utils/test_mi300x_srt_slurm_contract.py index 67709e2ccb..ef3f02a269 100644 --- a/utils/test_mi300x_srt_slurm_contract.py +++ b/utils/test_mi300x_srt_slurm_contract.py @@ -104,6 +104,45 @@ def test_official_matrix_routes_disagg_through_the_pinned_srt_launcher(): assert "scancel" not in launcher +def test_official_matrix_routes_aggregate_through_the_pinned_srt_launcher(): + config = yaml.safe_load(MASTER_CONFIG_PATH.read_text())[ + "qwen3-0.6b-fp16-mi300x-vllm-srt-agg" + ] + search = config["scenarios"]["fixed-seq-len"][0]["search-space"] + launcher = SRT_LAUNCHER_PATH.read_text() + + assert config["runner"] == "mi300x-disagg" + assert config["multinode"] is False + assert search == [{"tp": 1, "conc-list": [1]}] + assert ( + 'CONFIG_FILE="recipes/vllm/qwen3-0.6b/mi300x/agg-fixed-seq.yaml"' + in launcher + ) + assert 'JOB_BATCH_HOST=$(scontrol show job "$JOB_ID" -dd' in launcher + assert '--nodelist="$JOB_BATCH_HOST"' in launcher + + +def test_aggregate_recipe_uses_direct_vllm_without_dynamo_or_a_router(): + recipe = yaml.safe_load(RECIPE_PATH.read_text()) + + assert recipe["resources"] == { + "gpu_type": "mi300x", + "gpus_per_node": 1, + "agg_nodes": 1, + "agg_workers": 1, + "gpus_per_agg": 1, + } + assert recipe["frontend"] == { + "type": "vllm", + "enable_multiple_frontends": False, + } + assert recipe["backend"]["connector"] is None + serialized = RECIPE_PATH.read_text().lower() + assert "dynamo" not in serialized + assert "nixl" not in serialized + assert "moriio" not in serialized + + def test_fixed_sequence_recipe_uses_inferencex_custom_benchmark(): recipe = yaml.safe_load(RECIPE_PATH.read_text()) benchmark = recipe["benchmark"] From d7956415ec54bf5756cf990cadfcf43e151d2ddb Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Mon, 10 Aug 2026 01:02:51 -0500 Subject: [PATCH 29/67] ci(amd): route aggregate through srt-slurm --- configs/amd-master.yaml | 19 +++++++++++++++++-- runners/launch_mi300x-amds-srt.sh | 5 ++++- utils/test_mi300x_srt_slurm_contract.py | 22 ++++++++++++++++++++-- 3 files changed, 41 insertions(+), 5 deletions(-) diff --git a/configs/amd-master.yaml b/configs/amd-master.yaml index e586041406..8c980f4652 100644 --- a/configs/amd-master.yaml +++ b/configs/amd-master.yaml @@ -1506,7 +1506,12 @@ qwen3-0.6b-fp16-mi300x-vllm-srt-agg: runner: mi300x-disagg precision: fp16 framework: vllm - multinode: false + # Route through the multinode workflow even though the recipe itself owns a + # single aggregate node. That workflow is the srt-slurm orchestration entry + # point; disagg remains false and the decode worker count stays zero so the + # aggregate GPU is counted exactly once. + multinode: true + disagg: false scenarios: fixed-seq-len: - isl: 128 @@ -1514,7 +1519,17 @@ qwen3-0.6b-fp16-mi300x-vllm-srt-agg: search-space: # The srt-slurm custom benchmark exercises both c1 and c4 in one # allocation; keep one matrix point so the server is launched once. - - { tp: 1, conc-list: [1] } + - conc-list: [1] + prefill: + num-worker: 1 + tp: 1 + ep: 1 + dp-attn: false + decode: + num-worker: 0 + tp: 1 + ep: 1 + dp-attn: false qwen3-0.6b-fp16-mi300x-vllm-srt-disagg: image: vllm/vllm-openai-rocm:v0.26.0 diff --git a/runners/launch_mi300x-amds-srt.sh b/runners/launch_mi300x-amds-srt.sh index 6b3f3090a1..8ee6d9dc06 100644 --- a/runners/launch_mi300x-amds-srt.sh +++ b/runners/launch_mi300x-amds-srt.sh @@ -167,7 +167,10 @@ if [[ "${DISAGG:-false}" == "true" ]]; then DECODE_GPUS=$((DECODE_NUM_WORKERS * DECODE_TP)) TOTAL_GPUS=$((PREFILL_GPUS + DECODE_GPUS)) else - TOTAL_GPUS=$((TP * ${PP_SIZE:-1} * ${PCP_SIZE:-1})) + # Aggregate srt-slurm rows intentionally use the multinode workflow so + # this launcher owns orchestration. The aggregate worker is represented + # by the prefill-shaped matrix fields; decode workers are zero. + TOTAL_GPUS=$((PREFILL_NUM_WORKERS * PREFILL_TP * ${PREFILL_PP_SIZE:-1} * ${PREFILL_PCP_SIZE:-1})) fi shopt -s nullglob RESULTS=("$RETRIEVE_DIR"/fixed-seq/*.json) diff --git a/utils/test_mi300x_srt_slurm_contract.py b/utils/test_mi300x_srt_slurm_contract.py index ef3f02a269..44d899009a 100644 --- a/utils/test_mi300x_srt_slurm_contract.py +++ b/utils/test_mi300x_srt_slurm_contract.py @@ -112,14 +112,32 @@ def test_official_matrix_routes_aggregate_through_the_pinned_srt_launcher(): launcher = SRT_LAUNCHER_PATH.read_text() assert config["runner"] == "mi300x-disagg" - assert config["multinode"] is False - assert search == [{"tp": 1, "conc-list": [1]}] + assert config["multinode"] is True + assert config["disagg"] is False + assert search == [ + { + "conc-list": [1], + "prefill": { + "num-worker": 1, + "tp": 1, + "ep": 1, + "dp-attn": False, + }, + "decode": { + "num-worker": 0, + "tp": 1, + "ep": 1, + "dp-attn": False, + }, + } + ] assert ( 'CONFIG_FILE="recipes/vllm/qwen3-0.6b/mi300x/agg-fixed-seq.yaml"' in launcher ) assert 'JOB_BATCH_HOST=$(scontrol show job "$JOB_ID" -dd' in launcher assert '--nodelist="$JOB_BATCH_HOST"' in launcher + assert "TOTAL_GPUS=$((PREFILL_NUM_WORKERS * PREFILL_TP" in launcher def test_aggregate_recipe_uses_direct_vllm_without_dynamo_or_a_router(): From 524b1a21c9f6c46bc7c760f62a814e9e6ba3d103 Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Mon, 10 Aug 2026 01:07:01 -0500 Subject: [PATCH 30/67] fix(amd): select aggregate srt recipe explicitly --- configs/amd-master.yaml | 2 ++ runners/launch_mi300x-amds-srt.sh | 6 ------ utils/test_mi300x_srt_slurm_contract.py | 9 +++++---- 3 files changed, 7 insertions(+), 10 deletions(-) diff --git a/configs/amd-master.yaml b/configs/amd-master.yaml index 8c980f4652..2e01ca28c1 100644 --- a/configs/amd-master.yaml +++ b/configs/amd-master.yaml @@ -1525,6 +1525,8 @@ qwen3-0.6b-fp16-mi300x-vllm-srt-agg: tp: 1 ep: 1 dp-attn: false + additional-settings: + - "CONFIG_FILE=recipes/vllm/qwen3-0.6b/mi300x/agg-fixed-seq.yaml" decode: num-worker: 0 tp: 1 diff --git a/runners/launch_mi300x-amds-srt.sh b/runners/launch_mi300x-amds-srt.sh index 8ee6d9dc06..f56bb0b62a 100644 --- a/runners/launch_mi300x-amds-srt.sh +++ b/runners/launch_mi300x-amds-srt.sh @@ -12,12 +12,6 @@ REMOTE_BASE="/raid/hf-hub-cache/inferencex/srt-slurm" : "${GITHUB_WORKSPACE:?GITHUB_WORKSPACE must be set by Actions}" : "${RESULT_FILENAME:?RESULT_FILENAME must be set by the benchmark workflow}" -# The existing MI300X orchestration-capable runner pool defaults to the -# aggregate recipe. Multinode rows continue to pass their recipe explicitly -# through the existing worker additional-settings contract. -if [[ -z "${CONFIG_FILE:-}" && "${RUNNER_TYPE:-}" == "mi300x-disagg" ]]; then - CONFIG_FILE="recipes/vllm/qwen3-0.6b/mi300x/agg-fixed-seq.yaml" -fi : "${CONFIG_FILE:?CONFIG_FILE must name an srt-slurm recipe}" CONFIG_PATH="${CONFIG_FILE%%:*}" diff --git a/utils/test_mi300x_srt_slurm_contract.py b/utils/test_mi300x_srt_slurm_contract.py index 44d899009a..25c42ca2cd 100644 --- a/utils/test_mi300x_srt_slurm_contract.py +++ b/utils/test_mi300x_srt_slurm_contract.py @@ -122,6 +122,10 @@ def test_official_matrix_routes_aggregate_through_the_pinned_srt_launcher(): "tp": 1, "ep": 1, "dp-attn": False, + "additional-settings": [ + "CONFIG_FILE=recipes/vllm/qwen3-0.6b/mi300x/" + "agg-fixed-seq.yaml" + ], }, "decode": { "num-worker": 0, @@ -131,10 +135,7 @@ def test_official_matrix_routes_aggregate_through_the_pinned_srt_launcher(): }, } ] - assert ( - 'CONFIG_FILE="recipes/vllm/qwen3-0.6b/mi300x/agg-fixed-seq.yaml"' - in launcher - ) + assert ': "${CONFIG_FILE:?CONFIG_FILE must name an srt-slurm recipe}"' in launcher assert 'JOB_BATCH_HOST=$(scontrol show job "$JOB_ID" -dd' in launcher assert '--nodelist="$JOB_BATCH_HOST"' in launcher assert "TOTAL_GPUS=$((PREFILL_NUM_WORKERS * PREFILL_TP" in launcher From 71e908ae80176059e7c29576c30ad277cf49fe2f Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Mon, 10 Aug 2026 01:30:47 -0500 Subject: [PATCH 31/67] ci(amd): add MI355X srt-slurm validation --- .../cluster-configs/mi355x-amds.yaml | 32 ++++ .../qwen3-0.6b/mi355x/agg-fixed-seq.yaml | 94 ++++++++++ .../mi355x/disagg-1p1d-fixed-seq.yaml | 107 +++++++++++ configs/amd-master.yaml | 62 +++++++ runners/launch_mi355x-amds-srt.sh | 173 ++++++++++++++++++ runners/launch_mi355x-amds.sh | 4 + utils/test_mi355x_srt_slurm_contract.py | 146 +++++++++++++++ 7 files changed, 618 insertions(+) create mode 100644 benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi355x-amds.yaml create mode 100644 benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3-0.6b/mi355x/agg-fixed-seq.yaml create mode 100644 benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3-0.6b/mi355x/disagg-1p1d-fixed-seq.yaml create mode 100755 runners/launch_mi355x-amds-srt.sh create mode 100644 utils/test_mi355x_srt_slurm_contract.py diff --git a/benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi355x-amds.yaml b/benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi355x-amds.yaml new file mode 100644 index 0000000000..51154f85e0 --- /dev/null +++ b/benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi355x-amds.yaml @@ -0,0 +1,32 @@ +# srt-slurm cluster profile for the MI355X AMD Slurm cluster. The login and +# compute nodes share /it-share, so source, output, image, and result paths do +# not need node-local transport. + +cluster: mi355x-amds +default_partition: compute +default_time_limit: "01:00:00" +output_dir: /it-share/inferencex/srt-slurm/outputs + +gpus_per_node: 8 +accelerator_vendor: amd +network_interface: eno0 + +gpu_sbatch_directive: gres +use_segment_sbatch_directive: false +use_exclusive_sbatch_directive: false +runtime_config_transport: shared-filesystem + +default_sbatch_directives: + cpus-per-task: "32" + mem: "128G" + +default_mounts: + /dev/kfd: /dev/kfd + /dev/dri: /dev/dri + /it-share/hf-hub-cache: /hf_hub_cache + # The launcher appends the exact shared workspace and result roots. + +containers: + sglang-rocm-v0.5.16-mi35x: /it-share/inferencex/srt-slurm/containers/sglang-rocm-v0.5.16-mi35x-20260728.sqsh + +nginx_raise_ulimit: false diff --git a/benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3-0.6b/mi355x/agg-fixed-seq.yaml b/benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3-0.6b/mi355x/agg-fixed-seq.yaml new file mode 100644 index 0000000000..938dae7434 --- /dev/null +++ b/benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3-0.6b/mi355x/agg-fixed-seq.yaml @@ -0,0 +1,94 @@ +# Minimal MI355X aggregate validation. It exercises native SGLang Router and +# one aggregate SGLang worker; this is orchestration validation, not tuning. + +name: "mi355x-sglang-qwen3-0.6b-agg-fixed-seq" + +model: + path: "hf:Qwen/Qwen3-0.6B" + container: "sglang-rocm-v0.5.16-mi35x" + precision: "fp16" + +identity: + model: + repo: "Qwen/Qwen3-0.6B" + container: + image: "lmsysorg/sglang-rocm:v0.5.16-rocm720-mi35x-20260728" + frameworks: + sglang: "0.5.16.dev20260728+g32c30c0f96" + sglang-router: "0.3.2" + +slurm: + time_limit: "00:30:00" + +resources: + gpu_type: "mi355x" + gpus_per_node: 1 + agg_nodes: 1 + agg_workers: 1 + gpus_per_agg: 1 + +frontend: + type: sgl-router + enable_multiple_frontends: false + args: + policy: round_robin + +backend: + type: sglang + aggregated_environment: + HF_HOME: "/hf_hub_cache" + HF_HUB_CACHE: "/hf_hub_cache" + HUGGINGFACE_HUB_CACHE: "/hf_hub_cache" + PYTHONUNBUFFERED: "1" + sglang_config: + aggregated: + served-model-name: "Qwen/Qwen3-0.6B" + tensor-parallel-size: 1 + mem-fraction-static: 0.50 + context-length: 2048 + max-running-requests: 8 + disable-cuda-graph: true + enable-metrics: true + +srun_options: + container-writable: "" + container-remap-root: "" + mem: "0" + +health_check: + max_attempts: 180 + interval_seconds: 5 + +benchmark: + type: custom + command: | + set -euo pipefail + result_root="/results/${SLURM_JOB_ID}" + mkdir -p "${result_root}/fixed-seq" + trap 'tar -C /logs -czf "'"${result_root}"'/runtime-logs.tar.gz" . 2>/dev/null || true' EXIT + for concurrency in 1 4; do + python3 /infmax-workspace/utils/bench_serving/benchmark_serving.py \ + --backend openai-chat \ + --base-url "http://${SRT_FRONTEND_HOST}:${SRT_FRONTEND_PORT}" \ + --endpoint /v1/chat/completions \ + --model Qwen/Qwen3-0.6B \ + --tokenizer Qwen/Qwen3-0.6B \ + --dataset-name random \ + --random-input-len 128 \ + --random-output-len 32 \ + --random-range-ratio 1.0 \ + --random-num-workers 1 \ + --num-warmups "${concurrency}" \ + --num-prompts "$((concurrency * 4))" \ + --max-concurrency "${concurrency}" \ + --request-rate inf \ + --ignore-eos \ + --disable-tqdm \ + --save-result \ + --result-dir "${result_root}/fixed-seq" \ + --result-filename "qwen3-0.6b-agg-isl128-osl32-c${concurrency}.json"; + done + env: + HF_HOME: /hf_hub_cache + HF_HUB_CACHE: /hf_hub_cache + HUGGINGFACE_HUB_CACHE: /hf_hub_cache diff --git a/benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3-0.6b/mi355x/disagg-1p1d-fixed-seq.yaml b/benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3-0.6b/mi355x/disagg-1p1d-fixed-seq.yaml new file mode 100644 index 0000000000..3b2686033f --- /dev/null +++ b/benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3-0.6b/mi355x/disagg-1p1d-fixed-seq.yaml @@ -0,0 +1,107 @@ +# Minimal two-node MI355X P/D validation. SGLang Router provides the request +# plane and the bundled AMD MoRI transport moves KV between prefill and decode. + +name: "mi355x-sglang-qwen3-0.6b-disagg-1p1d-fixed-seq" + +model: + path: "hf:Qwen/Qwen3-0.6B" + container: "sglang-rocm-v0.5.16-mi35x" + precision: "fp16" + +identity: + model: + repo: "Qwen/Qwen3-0.6B" + container: + image: "lmsysorg/sglang-rocm:v0.5.16-rocm720-mi35x-20260728" + frameworks: + sglang: "0.5.16.dev20260728+g32c30c0f96" + sglang-router: "0.3.2" + amd-mori: "0.5.16.dev20260728+g32c30c0f96" + +slurm: + time_limit: "00:45:00" + +resources: + gpu_type: "mi355x" + gpus_per_node: 1 + prefill_nodes: 1 + decode_nodes: 1 + prefill_workers: 1 + decode_workers: 1 + gpus_per_prefill: 1 + gpus_per_decode: 1 + +frontend: + type: sgl-router + enable_multiple_frontends: false + args: + policy: round_robin + prefill-policy: round_robin + decode-policy: round_robin + +backend: + type: sglang + prefill_environment: &worker_environment + HF_HOME: "/hf_hub_cache" + HF_HUB_CACHE: "/hf_hub_cache" + HUGGINGFACE_HUB_CACHE: "/hf_hub_cache" + PYTHONUNBUFFERED: "1" + IBDEVICES: "rdma0,rdma1,rdma2,rdma3,rdma4,rdma5,rdma6,rdma7" + MORI_RDMA_TC: "104" + SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT: "1800" + SGLANG_DISAGGREGATION_WAITING_TIMEOUT: "1800" + decode_environment: *worker_environment + sglang_config: + prefill: &worker_config + served-model-name: "Qwen/Qwen3-0.6B" + tensor-parallel-size: 1 + disaggregation-transfer-backend: mori + mem-fraction-static: 0.50 + context-length: 2048 + max-running-requests: 8 + disable-cuda-graph: true + enable-metrics: true + decode: *worker_config + +srun_options: + container-writable: "" + container-remap-root: "" + mem: "0" + +health_check: + max_attempts: 240 + interval_seconds: 5 + +benchmark: + type: custom + command: | + set -euo pipefail + result_root="/results/${SLURM_JOB_ID}" + mkdir -p "${result_root}/fixed-seq" + trap 'tar -C /logs -czf "'"${result_root}"'/runtime-logs.tar.gz" . 2>/dev/null || true' EXIT + for concurrency in 1 4; do + python3 /infmax-workspace/utils/bench_serving/benchmark_serving.py \ + --backend openai-chat \ + --base-url "http://${SRT_FRONTEND_HOST}:${SRT_FRONTEND_PORT}" \ + --endpoint /v1/chat/completions \ + --model Qwen/Qwen3-0.6B \ + --tokenizer Qwen/Qwen3-0.6B \ + --dataset-name random \ + --random-input-len 128 \ + --random-output-len 32 \ + --random-range-ratio 1.0 \ + --random-num-workers 1 \ + --num-warmups "${concurrency}" \ + --num-prompts "$((concurrency * 4))" \ + --max-concurrency "${concurrency}" \ + --request-rate inf \ + --ignore-eos \ + --disable-tqdm \ + --save-result \ + --result-dir "${result_root}/fixed-seq" \ + --result-filename "qwen3-0.6b-disagg-isl128-osl32-c${concurrency}.json"; + done + env: + HF_HOME: /hf_hub_cache + HF_HUB_CACHE: /hf_hub_cache + HUGGINGFACE_HUB_CACHE: /hf_hub_cache diff --git a/configs/amd-master.yaml b/configs/amd-master.yaml index 2e01ca28c1..ee6f9d1c6d 100644 --- a/configs/amd-master.yaml +++ b/configs/amd-master.yaml @@ -1563,6 +1563,68 @@ qwen3-0.6b-fp16-mi300x-vllm-srt-disagg: ep: 1 dp-attn: false +# Small-model correctness lanes for native SGLang Router orchestration on +# MI355X. Both use the unchanged InferenceX fixed-sequence benchmark; the P/D +# lane uses AMD MoRI and neither lane starts Dynamo, NATS, etcd, or NIXL. +qwen3-0.6b-fp16-mi355x-sglang-srt-agg: + image: lmsysorg/sglang-rocm:v0.5.16-rocm720-mi35x-20260728 + model: Qwen/Qwen3-0.6B + model-prefix: qwen3-0.6b + runner: cluster:mi355x-amds + precision: fp16 + framework: sglang + multinode: true + disagg: false + router: { name: sglang-router, version: "0.3.2" } + scenarios: + fixed-seq-len: + - isl: 128 + osl: 32 + search-space: + - conc-list: [1] + prefill: + num-worker: 1 + tp: 1 + ep: 1 + dp-attn: false + additional-settings: + - "CONFIG_FILE=recipes/sglang/qwen3-0.6b/mi355x/agg-fixed-seq.yaml" + decode: + num-worker: 0 + tp: 1 + ep: 1 + dp-attn: false + +qwen3-0.6b-fp16-mi355x-sglang-srt-disagg: + image: lmsysorg/sglang-rocm:v0.5.16-rocm720-mi35x-20260728 + model: Qwen/Qwen3-0.6B + model-prefix: qwen3-0.6b + runner: cluster:mi355x-amds + precision: fp16 + framework: sglang-disagg + multinode: true + disagg: true + router: { name: sglang-router, version: "0.3.2" } + kv-p2p-transfer: mori + scenarios: + fixed-seq-len: + - isl: 128 + osl: 32 + search-space: + - conc-list: [1] + prefill: + num-worker: 1 + tp: 1 + ep: 1 + dp-attn: false + additional-settings: + - "CONFIG_FILE=recipes/sglang/qwen3-0.6b/mi355x/disagg-1p1d-fixed-seq.yaml" + decode: + num-worker: 1 + tp: 1 + ep: 1 + dp-attn: false + minimaxm3-fp4-mi355x-vllm-agentic: image: vllm/vllm-openai-rocm:nightly-dcfebf93f4eccf30f71872283331eee757915daf model: amd/MiniMax-M3-MXFP4 diff --git a/runners/launch_mi355x-amds-srt.sh b/runners/launch_mi355x-amds-srt.sh new file mode 100755 index 0000000000..f51945a47e --- /dev/null +++ b/runners/launch_mi355x-amds-srt.sh @@ -0,0 +1,173 @@ +#!/usr/bin/env bash +set -euo pipefail + +# MI355X validation path for the AMD-capable srt-slurm branch. Matrix rows opt +# in explicitly with CONFIG_FILE; all existing MI355X launch behavior remains +# unchanged for every other row. +SRT_SLURM_REPOSITORY="https://github.com/SemiAnalysisAI/srt-slurm.git" +SRT_SLURM_COMMIT="105824810a67d52b58761077ad3b94d4a05eb3ac" +SLURM_PARTITION="compute" +SGLANG_IMAGE="lmsysorg/sglang-rocm:v0.5.16-rocm720-mi35x-20260728" +SHARED_BASE="/it-share/inferencex/srt-slurm" +SHARED_IMAGE="${SHARED_BASE}/containers/sglang-rocm-v0.5.16-mi35x-20260728.sqsh" +SHARED_HF_CACHE="/it-share/hf-hub-cache" +SHARED_RESULTS="${SHARED_BASE}/results" + +: "${GITHUB_WORKSPACE:?GITHUB_WORKSPACE must be set by Actions}" +: "${RESULT_FILENAME:?RESULT_FILENAME must be set by the benchmark workflow}" +: "${CONFIG_FILE:?CONFIG_FILE must name an srt-slurm recipe}" + +CONFIG_PATH="${CONFIG_FILE%%:*}" +LOCAL_RECIPE="${GITHUB_WORKSPACE}/benchmarks/multi_node/srt-slurm-recipes/${CONFIG_PATH#recipes/}" +CLUSTER_PROFILE="${GITHUB_WORKSPACE}/benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi355x-amds.yaml" +[[ -f "$LOCAL_RECIPE" ]] || { echo "Missing recipe: $LOCAL_RECIPE" >&2; exit 1; } +[[ -f "$CLUSTER_PROFILE" ]] || { echo "Missing cluster profile: $CLUSTER_PROFILE" >&2; exit 1; } + +RUN_KEY="${GITHUB_RUN_ID:-manual}-${GITHUB_RUN_ATTEMPT:-0}-${RUNNER_NAME:-runner}" +WORK_DIR="${GITHUB_WORKSPACE}/.srt-slurm-${RUN_KEY}" +SRT_REPO_DIR="${WORK_DIR}/srt-slurm" +mkdir -p "$WORK_DIR" "$SHARED_RESULTS" + +# Materialize one immutable, shared squashfs and the small public validation +# model. This job exits normally and never cancels or preempts another job. +# The shared lock makes concurrent aggregate/disaggregated validations safe. +STAGE_SCRIPT="${WORK_DIR}/stage-mi355x-runtime.sbatch" +cat > "$STAGE_SCRIPT" <"${SHARED_IMAGE}.lock" +flock -w 2400 9 +if ! unsquashfs -s "$SHARED_IMAGE" >/dev/null 2>&1; then + tmp="${SHARED_IMAGE}.tmp.\${SLURM_JOB_ID}" + rm -f "\$tmp" + local_image="/var/lib/squash/lmsysorg_sglang-rocm_v0.5.16-rocm720-mi35x-20260728.sqsh" + if unsquashfs -s "\$local_image" >/dev/null 2>&1; then + cp --sparse=always "\$local_image" "\$tmp" + else + enroot import -o "\$tmp" "docker://${SGLANG_IMAGE}" + fi + unsquashfs -s "\$tmp" >/dev/null + mv "\$tmp" "$SHARED_IMAGE" +fi +flock -u 9 +srun --nodes=1 --ntasks=1 \ + --container-image="$SHARED_IMAGE" \ + --container-mounts="$SHARED_HF_CACHE:/hf_hub_cache" \ + --container-writable --container-remap-root --no-container-entrypoint \ + --export=ALL,HF_HOME=/hf_hub_cache,HF_HUB_CACHE=/hf_hub_cache,HUGGINGFACE_HUB_CACHE=/hf_hub_cache \ + python3 -c 'from huggingface_hub import snapshot_download; snapshot_download("Qwen/Qwen3-0.6B")' +EOF +STAGE_JOB_ID=$(sbatch --wait --parsable "$STAGE_SCRIPT") +STAGE_JOB_ID="${STAGE_JOB_ID%%;*}" +echo "MI355X runtime prerequisites verified with Slurm job ${STAGE_JOB_ID}" + +git clone "$SRT_SLURM_REPOSITORY" "$SRT_REPO_DIR" +git -C "$SRT_REPO_DIR" checkout "$SRT_SLURM_COMMIT" +ACTUAL_SRT_COMMIT=$(git -C "$SRT_REPO_DIR" rev-parse HEAD) +[[ "$ACTUAL_SRT_COMMIT" == "$SRT_SLURM_COMMIT" ]] || { + echo "srt-slurm checkout mismatch: $ACTUAL_SRT_COMMIT" >&2 + exit 1 +} + +mkdir -p "${SRT_REPO_DIR}/$(dirname "$CONFIG_PATH")" +cp "$LOCAL_RECIPE" "${SRT_REPO_DIR}/${CONFIG_PATH}" +cp "$CLUSTER_PROFILE" "${WORK_DIR}/srtslurm.yaml" +python3 - "${WORK_DIR}/srtslurm.yaml" "$GITHUB_WORKSPACE" "$SHARED_RESULTS" <<'PY' +import sys +from pathlib import Path + +path = Path(sys.argv[1]) +workspace, results = sys.argv[2:] +needle = " /it-share/hf-hub-cache: /hf_hub_cache\n" +text = path.read_text() +if text.count(needle) != 1: + raise SystemExit("expected exactly one Hugging Face cache mount") +path.write_text( + text.replace( + needle, + needle + f" {workspace}: /infmax-workspace\n {results}: /results\n", + ) +) +PY + +export PATH="$HOME/.local/bin:$PATH" +cd "$SRT_REPO_DIR" +uv venv --python 3.12 +uv pip install -e . +make setup-compute ARCH=x86_64 +source .venv/bin/activate +export SRTSLURM_CONFIG="${WORK_DIR}/srtslurm.yaml" +export SRTCTL_RUNTIME_SOURCE_DIR="$SRT_REPO_DIR" + +echo "Submitting ${CONFIG_PATH} with srt-slurm ${SRT_SLURM_COMMIT}" +set +e +SRTCTL_OUTPUT=$(srtctl apply -f "$CONFIG_FILE" \ + --tags "mi355x,inferencex,github-actions,${RUN_KEY}" 2>&1) +SRTCTL_RC=$? +set -e +echo "$SRTCTL_OUTPUT" +if [[ $SRTCTL_RC -ne 0 ]]; then + echo "srtctl apply failed with exit code ${SRTCTL_RC}" >&2 + exit "$SRTCTL_RC" +fi +JOB_ID=$(grep -oE 'Job [0-9]+' <<< "$SRTCTL_OUTPUT" | awk '{print $2}' | tail -1) +[[ -n "$JOB_ID" ]] || { echo "Unable to parse srt-slurm job ID" >&2; exit 1; } +echo "SRT_SLURM_JOB_ID=$JOB_ID" + +while squeue --noheader --jobs "$JOB_ID" | grep -q .; do + squeue --noheader --jobs "$JOB_ID" --format='srt-slurm %i %T %M %R' + sleep 15 +done + +read -r JOB_STATE JOB_EXIT JOB_NODELIST < <( + sacct -X --noheader --parsable2 --jobs "$JOB_ID" \ + --format=State,ExitCode,NodeList | head -1 | tr '|' ' ' +) +echo "srt-slurm job ${JOB_ID}: state=${JOB_STATE} exit=${JOB_EXIT} nodes=${JOB_NODELIST}" + +RESULT_DIR="${SHARED_RESULTS}/${JOB_ID}" +mkdir -p "$GITHUB_WORKSPACE/LOGS" +if [[ -f "$RESULT_DIR/runtime-logs.tar.gz" ]]; then + cp "$RESULT_DIR/runtime-logs.tar.gz" "$GITHUB_WORKSPACE/multinode_server_logs.tar.gz" +fi +cp -R "$RESULT_DIR/." "$GITHUB_WORKSPACE/LOGS/" + +if [[ "${DISAGG:-false}" == "true" ]]; then + PREFILL_GPUS=$((PREFILL_NUM_WORKERS * PREFILL_TP)) + DECODE_GPUS=$((DECODE_NUM_WORKERS * DECODE_TP)) + TOTAL_GPUS=$((PREFILL_GPUS + DECODE_GPUS)) +else + TOTAL_GPUS=$((PREFILL_NUM_WORKERS * PREFILL_TP * ${PREFILL_PP_SIZE:-1} * ${PREFILL_PCP_SIZE:-1})) +fi + +shopt -s nullglob +RESULTS=("$RESULT_DIR"/fixed-seq/*.json) +shopt -u nullglob +[[ ${#RESULTS[@]} -gt 0 ]] || { echo "No fixed-sequence results found in $RESULT_DIR" >&2; exit 1; } +for result in "${RESULTS[@]}"; do + concurrency=$(basename "$result" | sed -n 's/.*-c\([0-9][0-9]*\)\.json/\1/p') + [[ -n "$concurrency" ]] || { echo "Cannot parse concurrency from $result" >&2; exit 1; } + if [[ "${DISAGG:-false}" == "true" ]]; then + output="${GITHUB_WORKSPACE}/${RESULT_FILENAME}_srt-${JOB_ID}_conc${concurrency}_gpus_${TOTAL_GPUS}_ctx_${PREFILL_GPUS}_gen_${DECODE_GPUS}.json" + else + output="${GITHUB_WORKSPACE}/${RESULT_FILENAME}_srt-${JOB_ID}_conc${concurrency}_gpus_${TOTAL_GPUS}.json" + fi + cp "$result" "$output" + echo "Collected $output" +done + +if [[ "$JOB_STATE" != COMPLETED || "$JOB_EXIT" != 0:0 ]]; then + echo "srt-slurm validation failed: ${JOB_STATE} (${JOB_EXIT})" >&2 + exit 1 +fi + +printf '%s\n' "$SRT_SLURM_COMMIT" > "$GITHUB_WORKSPACE/srt-slurm-producer-sha.txt" +echo "MI355X srt-slurm validation completed successfully" diff --git a/runners/launch_mi355x-amds.sh b/runners/launch_mi355x-amds.sh index 75a67c1c58..0c892ba46e 100644 --- a/runners/launch_mi355x-amds.sh +++ b/runners/launch_mi355x-amds.sh @@ -1,5 +1,9 @@ #!/usr/bin/env bash +if [[ -n "${CONFIG_FILE:-}" ]]; then + exec bash "$(dirname "${BASH_SOURCE[0]}")/launch_mi355x-amds-srt.sh" +fi + scancel_sync() { local jobid=$1 local timeout=${2:-600} diff --git a/utils/test_mi355x_srt_slurm_contract.py b/utils/test_mi355x_srt_slurm_contract.py new file mode 100644 index 0000000000..b2ca5b2c16 --- /dev/null +++ b/utils/test_mi355x_srt_slurm_contract.py @@ -0,0 +1,146 @@ +"""High-signal contract checks for the MI355X srt-slurm bring-up lanes.""" + +import os +import subprocess +from pathlib import Path + +import yaml + + +REPO_ROOT = Path(__file__).resolve().parents[1] +RECIPE_ROOT = ( + REPO_ROOT + / "benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3-0.6b/mi355x" +) +AGG_RECIPE = RECIPE_ROOT / "agg-fixed-seq.yaml" +DISAGG_RECIPE = RECIPE_ROOT / "disagg-1p1d-fixed-seq.yaml" +CLUSTER_PROFILE = ( + REPO_ROOT + / "benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi355x-amds.yaml" +) +MASTER_CONFIG = REPO_ROOT / "configs/amd-master.yaml" +LAUNCHER = REPO_ROOT / "runners/launch_mi355x-amds-srt.sh" +LEGACY_LAUNCHER = REPO_ROOT / "runners/launch_mi355x-amds.sh" + + +def test_cluster_profile_matches_the_mi355x_rocm_slurm_contract(): + cluster = yaml.safe_load(CLUSTER_PROFILE.read_text()) + + assert cluster["accelerator_vendor"] == "amd" + assert cluster["network_interface"] == "eno0" + assert cluster["gpu_sbatch_directive"] == "gres" + assert cluster["use_segment_sbatch_directive"] is False + assert cluster["use_exclusive_sbatch_directive"] is False + assert cluster["runtime_config_transport"] == "shared-filesystem" + assert cluster["default_mounts"]["/dev/kfd"] == "/dev/kfd" + assert cluster["default_mounts"]["/dev/dri"] == "/dev/dri" + assert cluster["containers"]["sglang-rocm-v0.5.16-mi35x"].endswith( + "/sglang-rocm-v0.5.16-mi35x-20260728.sqsh" + ) + + +def test_recipes_use_native_sglang_router_and_only_disagg_uses_mori(): + agg = yaml.safe_load(AGG_RECIPE.read_text()) + disagg = yaml.safe_load(DISAGG_RECIPE.read_text()) + + assert agg["resources"] == { + "gpu_type": "mi355x", + "gpus_per_node": 1, + "agg_nodes": 1, + "agg_workers": 1, + "gpus_per_agg": 1, + } + assert agg["frontend"]["type"] == "sgl-router" + assert agg["backend"]["type"] == "sglang" + assert disagg["frontend"]["type"] == "sgl-router" + assert disagg["backend"]["type"] == "sglang" + for role in ("prefill", "decode"): + assert disagg["backend"]["sglang_config"][role][ + "disaggregation-transfer-backend" + ] == "mori" + assert "rdma0" in disagg["backend"][f"{role}_environment"]["IBDEVICES"] + + assert "mori" not in AGG_RECIPE.read_text().lower() + for recipe in (AGG_RECIPE, DISAGG_RECIPE): + text = recipe.read_text().lower() + assert "dynamo" not in text + assert "nixl" not in text + assert "nats" not in text + assert "etcd" not in text + + +def test_matrix_rows_explicitly_select_the_srt_recipes(): + master = yaml.safe_load(MASTER_CONFIG.read_text()) + expected = { + "qwen3-0.6b-fp16-mi355x-sglang-srt-agg": ( + False, + "recipes/sglang/qwen3-0.6b/mi355x/agg-fixed-seq.yaml", + ), + "qwen3-0.6b-fp16-mi355x-sglang-srt-disagg": ( + True, + "recipes/sglang/qwen3-0.6b/mi355x/disagg-1p1d-fixed-seq.yaml", + ), + } + for name, (is_disagg, recipe) in expected.items(): + config = master[name] + search = config["scenarios"]["fixed-seq-len"][0]["search-space"][0] + assert config["runner"] == "cluster:mi355x-amds" + assert config["multinode"] is True + assert config["disagg"] is is_disagg + assert search["prefill"]["additional-settings"] == [ + f"CONFIG_FILE={recipe}" + ] + + +def test_launcher_pins_runtime_and_preserves_legacy_default(): + launcher = LAUNCHER.read_text() + legacy = LEGACY_LAUNCHER.read_text() + + assert "105824810a67d52b58761077ad3b94d4a05eb3ac" in launcher + assert "v0.5.16-rocm720-mi35x-20260728" in launcher + assert "make setup-compute ARCH=x86_64" in launcher + assert "SRTCTL_RUNTIME_SOURCE_DIR" in launcher + assert "scancel" not in launcher + assert 'if [[ -n "${CONFIG_FILE:-}" ]]; then' in legacy + assert "launch_mi355x-amds-srt.sh" in legacy + + +def test_fixed_sequence_commands_execute_with_attached_arguments(tmp_path): + fake_bin = tmp_path / "bin" + fake_bin.mkdir() + fake_python = fake_bin / "python3" + fake_python.write_text( + "#!/bin/bash\n" + 'printf "%s\\n" "$@" >> "$FAKE_ARGS_LOG"\n' + 'printf "%s\\n" --CALL-END-- >> "$FAKE_ARGS_LOG"\n' + ) + fake_python.chmod(0o755) + + for recipe_path in (AGG_RECIPE, DISAGG_RECIPE): + command = yaml.safe_load(recipe_path.read_text())["benchmark"]["command"] + result_dir = tmp_path / recipe_path.stem + command = command.replace( + 'result_root="/results/${SLURM_JOB_ID}"', + f'result_root="{result_dir}"', + ) + args_log = tmp_path / f"{recipe_path.stem}.args" + env = { + **os.environ, + "PATH": f"{fake_bin}:{os.environ['PATH']}", + "FAKE_ARGS_LOG": str(args_log), + "SRT_FRONTEND_HOST": "127.0.0.1", + "SRT_FRONTEND_PORT": "8000", + "SLURM_JOB_ID": "123", + } + + subprocess.run(["bash", "-n"], input=command, text=True, check=True) + subprocess.run(["bash", "-c", command], env=env, check=True) + calls = args_log.read_text().split("--CALL-END--\n") + calls = [[arg for arg in call.splitlines() if arg] for call in calls if call] + + assert len(calls) == 2 + assert [call[call.index("--num-prompts") + 1] for call in calls] == [ + "4", + "16", + ] + assert all("--model" in call and "Qwen/Qwen3-0.6B" in call for call in calls) From 33f2cfcd2b3874276c6bad3067e6d2afbf2e8891 Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Mon, 10 Aug 2026 01:41:52 -0500 Subject: [PATCH 32/67] fix(amd): use writable MI355X shared root --- .../srt-slurm-recipes/cluster-configs/mi355x-amds.yaml | 4 ++-- runners/launch_mi355x-amds-srt.sh | 2 +- utils/test_mi355x_srt_slurm_contract.py | 6 ++++++ 3 files changed, 9 insertions(+), 3 deletions(-) diff --git a/benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi355x-amds.yaml b/benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi355x-amds.yaml index 51154f85e0..d4ca309b1d 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi355x-amds.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi355x-amds.yaml @@ -5,7 +5,7 @@ cluster: mi355x-amds default_partition: compute default_time_limit: "01:00:00" -output_dir: /it-share/inferencex/srt-slurm/outputs +output_dir: /it-share/gharunners2/srt-slurm/outputs gpus_per_node: 8 accelerator_vendor: amd @@ -27,6 +27,6 @@ default_mounts: # The launcher appends the exact shared workspace and result roots. containers: - sglang-rocm-v0.5.16-mi35x: /it-share/inferencex/srt-slurm/containers/sglang-rocm-v0.5.16-mi35x-20260728.sqsh + sglang-rocm-v0.5.16-mi35x: /it-share/gharunners2/srt-slurm/containers/sglang-rocm-v0.5.16-mi35x-20260728.sqsh nginx_raise_ulimit: false diff --git a/runners/launch_mi355x-amds-srt.sh b/runners/launch_mi355x-amds-srt.sh index f51945a47e..9950fd3f61 100755 --- a/runners/launch_mi355x-amds-srt.sh +++ b/runners/launch_mi355x-amds-srt.sh @@ -8,7 +8,7 @@ SRT_SLURM_REPOSITORY="https://github.com/SemiAnalysisAI/srt-slurm.git" SRT_SLURM_COMMIT="105824810a67d52b58761077ad3b94d4a05eb3ac" SLURM_PARTITION="compute" SGLANG_IMAGE="lmsysorg/sglang-rocm:v0.5.16-rocm720-mi35x-20260728" -SHARED_BASE="/it-share/inferencex/srt-slurm" +SHARED_BASE="/it-share/gharunners2/srt-slurm" SHARED_IMAGE="${SHARED_BASE}/containers/sglang-rocm-v0.5.16-mi35x-20260728.sqsh" SHARED_HF_CACHE="/it-share/hf-hub-cache" SHARED_RESULTS="${SHARED_BASE}/results" diff --git a/utils/test_mi355x_srt_slurm_contract.py b/utils/test_mi355x_srt_slurm_contract.py index b2ca5b2c16..edbadc2e28 100644 --- a/utils/test_mi355x_srt_slurm_contract.py +++ b/utils/test_mi355x_srt_slurm_contract.py @@ -37,6 +37,10 @@ def test_cluster_profile_matches_the_mi355x_rocm_slurm_contract(): assert cluster["containers"]["sglang-rocm-v0.5.16-mi35x"].endswith( "/sglang-rocm-v0.5.16-mi35x-20260728.sqsh" ) + assert cluster["output_dir"].startswith("/it-share/gharunners2/srt-slurm/") + assert cluster["containers"]["sglang-rocm-v0.5.16-mi35x"].startswith( + "/it-share/gharunners2/srt-slurm/" + ) def test_recipes_use_native_sglang_router_and_only_disagg_uses_mori(): @@ -100,6 +104,8 @@ def test_launcher_pins_runtime_and_preserves_legacy_default(): assert "v0.5.16-rocm720-mi35x-20260728" in launcher assert "make setup-compute ARCH=x86_64" in launcher assert "SRTCTL_RUNTIME_SOURCE_DIR" in launcher + assert 'SHARED_BASE="/it-share/gharunners2/srt-slurm"' in launcher + assert "/it-share/inferencex" not in launcher assert "scancel" not in launcher assert 'if [[ -n "${CONFIG_FILE:-}" ]]; then' in legacy assert "launch_mi355x-amds-srt.sh" in legacy From 9a7d23378c694edf30af2b74a6c902a1d7b0f177 Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Mon, 10 Aug 2026 02:28:09 -0500 Subject: [PATCH 33/67] add production MI355X srt-slurm validation --- .../mi355x/disagg-1p1d-tp8-fixed-seq.yaml | 135 ++++++++++++++++++ configs/amd-master.yaml | 33 +++++ perf-changelog.yaml | 14 ++ runners/launch_mi355x-amds-srt.sh | 7 +- utils/test_mi355x_srt_slurm_contract.py | 39 ++++- 5 files changed, 223 insertions(+), 5 deletions(-) create mode 100644 benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3.5/mi355x/disagg-1p1d-tp8-fixed-seq.yaml diff --git a/benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3.5/mi355x/disagg-1p1d-tp8-fixed-seq.yaml b/benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3.5/mi355x/disagg-1p1d-tp8-fixed-seq.yaml new file mode 100644 index 0000000000..c27d4f5e81 --- /dev/null +++ b/benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3.5/mi355x/disagg-1p1d-tp8-fixed-seq.yaml @@ -0,0 +1,135 @@ +# Production-scale MI355X port of the existing Qwen3.5 FP8 1P1D recipe. +# Each role owns one full 8-GPU node; the native SGLang Router provides the +# request plane and AMD MoRI moves KV directly between the P/D workers. + +name: "mi355x-sglang-qwen3.5-fp8-disagg-1p1d-tp8-fixed-seq" + +model: + path: "hf:Qwen/Qwen3.5-397B-A17B-FP8" + container: "sglang-rocm-v0.5.16-mi35x" + precision: "fp8" + +identity: + model: + repo: "Qwen/Qwen3.5-397B-A17B-FP8" + container: + image: "lmsysorg/sglang-rocm:v0.5.16-rocm720-mi35x-20260728" + frameworks: + sglang: "0.5.16.dev20260728+g32c30c0f96" + sglang-router: "0.3.2" + amd-mori: "0.5.16.dev20260728+g32c30c0f96" + +slurm: + time_limit: "02:00:00" + +resources: + gpu_type: "mi355x" + gpus_per_node: 8 + prefill_nodes: 1 + decode_nodes: 1 + prefill_workers: 1 + decode_workers: 1 + gpus_per_prefill: 8 + gpus_per_decode: 8 + +frontend: + type: sgl-router + enable_multiple_frontends: false + args: + policy: round_robin + prefill-policy: round_robin + decode-policy: round_robin + +backend: + type: sglang + prefill_environment: &common_environment + HF_HOME: "/hf_hub_cache" + HF_HUB_CACHE: "/hf_hub_cache" + HUGGINGFACE_HUB_CACHE: "/hf_hub_cache" + PYTHONUNBUFFERED: "1" + IBDEVICES: "rdma0,rdma1,rdma2,rdma3,rdma4,rdma5,rdma6,rdma7" + GLOO_SOCKET_IFNAME: "eno0" + NCCL_SOCKET_IFNAME: "eno0" + MORI_RDMA_TC: "104" + MORI_IO_TC: "104" + MORI_IO_SQ_BACKOFF_TIMEOUT_US: "50000" + MORI_IO_QP_MAX_SEND_WR: "16384" + MORI_IO_QP_MAX_CQE: "32768" + MORI_IO_QP_MAX_SGE: "2" + MORI_SHMEM_MODE: "ISOLATION" + SGLANG_USE_AITER: "1" + AITER_LOG_LEVEL: "ERROR" + SGLANG_MORI_DISPATCH_DTYPE: "auto" + SGLANG_MORI_QP_PER_TRANSFER: "4" + SGLANG_MORI_NUM_WORKERS: "4" + SGLANG_DISAGGREGATION_NUM_PRE_ALLOCATE_REQS: "32" + SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT: "3600" + SGLANG_DISAGGREGATION_WAITING_TIMEOUT: "3600" + SGLANG_HEALTH_CHECK_TIMEOUT: "600" + decode_environment: *common_environment + sglang_config: + prefill: &common_config + served-model-name: "Qwen/Qwen3.5-397B-A17B-FP8" + tensor-parallel-size: 8 + disaggregation-transfer-backend: mori + disaggregation-ib-device: "rdma0,rdma1,rdma2,rdma3,rdma4,rdma5,rdma6,rdma7" + kv-cache-dtype: "fp8_e4m3" + attention-backend: aiter + moe-dense-tp-size: 1 + load-balance-method: round_robin + watchdog-timeout: 3600 + decode-log-interval: 1000 + log-level: warning + mem-fraction-static: 0.80 + max-running-requests: 128 + chunked-prefill-size: 262144 + context-length: 16384 + disable-radix-cache: true + cuda-graph-bs: [1, 2, 3, 4, 5, 6, 7, 8] + enable-metrics: true + decode: + <<: *common_config + mem-fraction-static: 0.85 + prefill-round-robin-balance: true + +srun_options: + container-writable: "" + container-remap-root: "" + mem: "0" + +health_check: + max_attempts: 720 + interval_seconds: 5 + +benchmark: + type: custom + command: | + set -euo pipefail + result_root="/results/${SLURM_JOB_ID}" + mkdir -p "${result_root}/fixed-seq" + trap 'tar -C /logs -czf "'"${result_root}"'/runtime-logs.tar.gz" . 2>/dev/null || true' EXIT + concurrency=8 + python3 /infmax-workspace/utils/bench_serving/benchmark_serving.py \ + --backend openai-chat \ + --base-url "http://${SRT_FRONTEND_HOST}:${SRT_FRONTEND_PORT}" \ + --endpoint /v1/chat/completions \ + --model Qwen/Qwen3.5-397B-A17B-FP8 \ + --tokenizer Qwen/Qwen3.5-397B-A17B-FP8 \ + --dataset-name random \ + --random-input-len 8192 \ + --random-output-len 1024 \ + --random-range-ratio 1.0 \ + --random-num-workers 1 \ + --num-warmups "${concurrency}" \ + --num-prompts "$((concurrency * 2))" \ + --max-concurrency "${concurrency}" \ + --request-rate inf \ + --ignore-eos \ + --disable-tqdm \ + --save-result \ + --result-dir "${result_root}/fixed-seq" \ + --result-filename "qwen3.5-fp8-disagg-isl8192-osl1024-c${concurrency}.json" + env: + HF_HOME: /hf_hub_cache + HF_HUB_CACHE: /hf_hub_cache + HUGGINGFACE_HUB_CACHE: /hf_hub_cache diff --git a/configs/amd-master.yaml b/configs/amd-master.yaml index ee6f9d1c6d..a6d492d394 100644 --- a/configs/amd-master.yaml +++ b/configs/amd-master.yaml @@ -1625,6 +1625,39 @@ qwen3-0.6b-fp16-mi355x-sglang-srt-disagg: ep: 1 dp-attn: false +# Production-scale migration of the existing Qwen3.5 FP8 MI355X P/D lane to +# srt-slurm. One full TP8 node serves prefill, one TP8 node serves decode, and +# the request/KV planes remain native SGLang Router + AMD MoRI. +qwen3.5-fp8-mi355x-sglang-srt-disagg: + image: lmsysorg/sglang-rocm:v0.5.16-rocm720-mi35x-20260728 + model: Qwen/Qwen3.5-397B-A17B-FP8 + model-prefix: qwen3.5 + runner: cluster:mi355x-amds + precision: fp8 + framework: sglang-disagg + multinode: true + disagg: true + router: { name: sglang-router, version: "0.3.2" } + kv-p2p-transfer: mori + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - conc-list: [8] + prefill: + num-worker: 1 + tp: 8 + ep: 1 + dp-attn: false + additional-settings: + - "CONFIG_FILE=recipes/sglang/qwen3.5/mi355x/disagg-1p1d-tp8-fixed-seq.yaml" + decode: + num-worker: 1 + tp: 8 + ep: 1 + dp-attn: false + minimaxm3-fp4-mi355x-vllm-agentic: image: vllm/vllm-openai-rocm:nightly-dcfebf93f4eccf30f71872283331eee757915daf model: amd/MiniMax-M3-MXFP4 diff --git a/perf-changelog.yaml b/perf-changelog.yaml index 471981f60c..eef886bfe6 100644 --- a/perf-changelog.yaml +++ b/perf-changelog.yaml @@ -5709,3 +5709,17 @@ - "Enable SGLang metrics on every aggregate, prefill, and decode engine." - "Use supported header-based Dynamo session routing with the in-repo AIPerf build." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2520 + +- config-keys: + - qwen3-0.6b-fp16-mi300x-vllm-srt-agg + - qwen3-0.6b-fp16-mi300x-vllm-srt-disagg + - qwen3-0.6b-fp16-mi355x-sglang-srt-agg + - qwen3-0.6b-fp16-mi355x-sglang-srt-disagg + - qwen3.5-fp8-mi355x-sglang-srt-disagg + description: + - "Add native AMD orchestration to srt-slurm and validate aggregate plus disaggregated serving on MI300X and MI355X without Dynamo, NIXL, NATS, or etcd." + - "Use the native vLLM Router with MoRIIO for MI300X disaggregation and the native SGLang Router with AMD MoRI for MI355X disaggregation; aggregate paths use the corresponding direct backend." + - "Run the repository's unchanged utils/bench_serving/benchmark_serving.py through srt-slurm's custom benchmark contract for fixed-sequence validation." + - "Port the production Qwen3.5-397B-A17B-FP8 MI355X topology as two full TP8 nodes, one prefill and one decode, using lmsysorg/sglang-rocm:v0.5.16-rocm720-mi35x-20260728, SGLang Router 0.3.2, MoRI, AITer attention, FP8 KV cache, and an 8192-input/1024-output concurrency-8 workload." + - "Pin the InferenceX launcher to the exact srt-slurm AMD runtime commit and validate model, container, backend, router, and MoRI package identity before serving." + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2542 diff --git a/runners/launch_mi355x-amds-srt.sh b/runners/launch_mi355x-amds-srt.sh index 9950fd3f61..4ac41eb4ae 100755 --- a/runners/launch_mi355x-amds-srt.sh +++ b/runners/launch_mi355x-amds-srt.sh @@ -5,7 +5,7 @@ set -euo pipefail # in explicitly with CONFIG_FILE; all existing MI355X launch behavior remains # unchanged for every other row. SRT_SLURM_REPOSITORY="https://github.com/SemiAnalysisAI/srt-slurm.git" -SRT_SLURM_COMMIT="105824810a67d52b58761077ad3b94d4a05eb3ac" +SRT_SLURM_COMMIT="25e1e4e71dc8e7383b1857041decff1a9ae0339e" SLURM_PARTITION="compute" SGLANG_IMAGE="lmsysorg/sglang-rocm:v0.5.16-rocm720-mi35x-20260728" SHARED_BASE="/it-share/gharunners2/srt-slurm" @@ -16,6 +16,7 @@ SHARED_RESULTS="${SHARED_BASE}/results" : "${GITHUB_WORKSPACE:?GITHUB_WORKSPACE must be set by Actions}" : "${RESULT_FILENAME:?RESULT_FILENAME must be set by the benchmark workflow}" : "${CONFIG_FILE:?CONFIG_FILE must name an srt-slurm recipe}" +: "${MODEL_PATH:?MODEL_PATH must identify the Hugging Face model}" CONFIG_PATH="${CONFIG_FILE%%:*}" LOCAL_RECIPE="${GITHUB_WORKSPACE}/benchmarks/multi_node/srt-slurm-recipes/${CONFIG_PATH#recipes/}" @@ -62,8 +63,8 @@ srun --nodes=1 --ntasks=1 \ --container-image="$SHARED_IMAGE" \ --container-mounts="$SHARED_HF_CACHE:/hf_hub_cache" \ --container-writable --container-remap-root --no-container-entrypoint \ - --export=ALL,HF_HOME=/hf_hub_cache,HF_HUB_CACHE=/hf_hub_cache,HUGGINGFACE_HUB_CACHE=/hf_hub_cache \ - python3 -c 'from huggingface_hub import snapshot_download; snapshot_download("Qwen/Qwen3-0.6B")' + --export=ALL,HF_HOME=/hf_hub_cache,HF_HUB_CACHE=/hf_hub_cache,HUGGINGFACE_HUB_CACHE=/hf_hub_cache,MODEL_REPO=${MODEL_PATH} \ + python3 -c 'import os; from huggingface_hub import snapshot_download; snapshot_download(os.environ["MODEL_REPO"])' EOF STAGE_JOB_ID=$(sbatch --wait --parsable "$STAGE_SCRIPT") STAGE_JOB_ID="${STAGE_JOB_ID%%;*}" diff --git a/utils/test_mi355x_srt_slurm_contract.py b/utils/test_mi355x_srt_slurm_contract.py index edbadc2e28..2b492e54b1 100644 --- a/utils/test_mi355x_srt_slurm_contract.py +++ b/utils/test_mi355x_srt_slurm_contract.py @@ -14,6 +14,10 @@ ) AGG_RECIPE = RECIPE_ROOT / "agg-fixed-seq.yaml" DISAGG_RECIPE = RECIPE_ROOT / "disagg-1p1d-fixed-seq.yaml" +PRODUCTION_DISAGG_RECIPE = ( + REPO_ROOT + / "benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3.5/mi355x/disagg-1p1d-tp8-fixed-seq.yaml" +) CLUSTER_PROFILE = ( REPO_ROOT / "benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi355x-amds.yaml" @@ -65,7 +69,7 @@ def test_recipes_use_native_sglang_router_and_only_disagg_uses_mori(): assert "rdma0" in disagg["backend"][f"{role}_environment"]["IBDEVICES"] assert "mori" not in AGG_RECIPE.read_text().lower() - for recipe in (AGG_RECIPE, DISAGG_RECIPE): + for recipe in (AGG_RECIPE, DISAGG_RECIPE, PRODUCTION_DISAGG_RECIPE): text = recipe.read_text().lower() assert "dynamo" not in text assert "nixl" not in text @@ -84,6 +88,10 @@ def test_matrix_rows_explicitly_select_the_srt_recipes(): True, "recipes/sglang/qwen3-0.6b/mi355x/disagg-1p1d-fixed-seq.yaml", ), + "qwen3.5-fp8-mi355x-sglang-srt-disagg": ( + True, + "recipes/sglang/qwen3.5/mi355x/disagg-1p1d-tp8-fixed-seq.yaml", + ), } for name, (is_disagg, recipe) in expected.items(): config = master[name] @@ -100,13 +108,14 @@ def test_launcher_pins_runtime_and_preserves_legacy_default(): launcher = LAUNCHER.read_text() legacy = LEGACY_LAUNCHER.read_text() - assert "105824810a67d52b58761077ad3b94d4a05eb3ac" in launcher + assert "25e1e4e71dc8e7383b1857041decff1a9ae0339e" in launcher assert "v0.5.16-rocm720-mi35x-20260728" in launcher assert "make setup-compute ARCH=x86_64" in launcher assert "SRTCTL_RUNTIME_SOURCE_DIR" in launcher assert 'SHARED_BASE="/it-share/gharunners2/srt-slurm"' in launcher assert "/it-share/inferencex" not in launcher assert "scancel" not in launcher + assert 'snapshot_download(os.environ["MODEL_REPO"])' in launcher assert 'if [[ -n "${CONFIG_FILE:-}" ]]; then' in legacy assert "launch_mi355x-amds-srt.sh" in legacy @@ -150,3 +159,29 @@ def test_fixed_sequence_commands_execute_with_attached_arguments(tmp_path): "16", ] assert all("--model" in call and "Qwen/Qwen3-0.6B" in call for call in calls) + + +def test_production_disagg_recipe_uses_two_full_nodes_and_the_existing_workload(): + recipe = yaml.safe_load(PRODUCTION_DISAGG_RECIPE.read_text()) + + assert recipe["resources"] == { + "gpu_type": "mi355x", + "gpus_per_node": 8, + "prefill_nodes": 1, + "decode_nodes": 1, + "prefill_workers": 1, + "decode_workers": 1, + "gpus_per_prefill": 8, + "gpus_per_decode": 8, + } + assert recipe["frontend"]["type"] == "sgl-router" + for role in ("prefill", "decode"): + config = recipe["backend"]["sglang_config"][role] + assert config["tensor-parallel-size"] == 8 + assert config["disaggregation-transfer-backend"] == "mori" + assert config["attention-backend"] == "aiter" + command = recipe["benchmark"]["command"] + assert "utils/bench_serving/benchmark_serving.py" in command + assert "--random-input-len 8192" in command + assert "--random-output-len 1024" in command + assert "--max-concurrency \"${concurrency}\"" in command From cc0b9f78d7a13f13ae608a340c9bd0ef1388ffbd Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Mon, 10 Aug 2026 02:37:12 -0500 Subject: [PATCH 34/67] fix MI355X model staging input --- runners/launch_mi355x-amds-srt.sh | 4 ++-- utils/test_mi355x_srt_slurm_contract.py | 2 ++ 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/runners/launch_mi355x-amds-srt.sh b/runners/launch_mi355x-amds-srt.sh index 4ac41eb4ae..80a80c2a72 100755 --- a/runners/launch_mi355x-amds-srt.sh +++ b/runners/launch_mi355x-amds-srt.sh @@ -16,7 +16,7 @@ SHARED_RESULTS="${SHARED_BASE}/results" : "${GITHUB_WORKSPACE:?GITHUB_WORKSPACE must be set by Actions}" : "${RESULT_FILENAME:?RESULT_FILENAME must be set by the benchmark workflow}" : "${CONFIG_FILE:?CONFIG_FILE must name an srt-slurm recipe}" -: "${MODEL_PATH:?MODEL_PATH must identify the Hugging Face model}" +: "${MODEL:?MODEL must identify the Hugging Face model}" CONFIG_PATH="${CONFIG_FILE%%:*}" LOCAL_RECIPE="${GITHUB_WORKSPACE}/benchmarks/multi_node/srt-slurm-recipes/${CONFIG_PATH#recipes/}" @@ -63,7 +63,7 @@ srun --nodes=1 --ntasks=1 \ --container-image="$SHARED_IMAGE" \ --container-mounts="$SHARED_HF_CACHE:/hf_hub_cache" \ --container-writable --container-remap-root --no-container-entrypoint \ - --export=ALL,HF_HOME=/hf_hub_cache,HF_HUB_CACHE=/hf_hub_cache,HUGGINGFACE_HUB_CACHE=/hf_hub_cache,MODEL_REPO=${MODEL_PATH} \ + --export=ALL,HF_HOME=/hf_hub_cache,HF_HUB_CACHE=/hf_hub_cache,HUGGINGFACE_HUB_CACHE=/hf_hub_cache,MODEL_REPO=${MODEL} \ python3 -c 'import os; from huggingface_hub import snapshot_download; snapshot_download(os.environ["MODEL_REPO"])' EOF STAGE_JOB_ID=$(sbatch --wait --parsable "$STAGE_SCRIPT") diff --git a/utils/test_mi355x_srt_slurm_contract.py b/utils/test_mi355x_srt_slurm_contract.py index 2b492e54b1..cc67fb95fe 100644 --- a/utils/test_mi355x_srt_slurm_contract.py +++ b/utils/test_mi355x_srt_slurm_contract.py @@ -115,6 +115,8 @@ def test_launcher_pins_runtime_and_preserves_legacy_default(): assert 'SHARED_BASE="/it-share/gharunners2/srt-slurm"' in launcher assert "/it-share/inferencex" not in launcher assert "scancel" not in launcher + assert ': "${MODEL:?MODEL must identify the Hugging Face model}"' in launcher + assert "MODEL_REPO=${MODEL}" in launcher assert 'snapshot_download(os.environ["MODEL_REPO"])' in launcher assert 'if [[ -n "${CONFIG_FILE:-}" ]]; then' in legacy assert "launch_mi355x-amds-srt.sh" in legacy From 1d60fb1d8f4c5f9ab9d5acd452e60f34eab4a54a Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Mon, 10 Aug 2026 02:44:40 -0500 Subject: [PATCH 35/67] bump AMD runtime cache fix --- runners/launch_mi355x-amds-srt.sh | 2 +- utils/test_mi355x_srt_slurm_contract.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/runners/launch_mi355x-amds-srt.sh b/runners/launch_mi355x-amds-srt.sh index 80a80c2a72..d9f132c3d6 100755 --- a/runners/launch_mi355x-amds-srt.sh +++ b/runners/launch_mi355x-amds-srt.sh @@ -5,7 +5,7 @@ set -euo pipefail # in explicitly with CONFIG_FILE; all existing MI355X launch behavior remains # unchanged for every other row. SRT_SLURM_REPOSITORY="https://github.com/SemiAnalysisAI/srt-slurm.git" -SRT_SLURM_COMMIT="25e1e4e71dc8e7383b1857041decff1a9ae0339e" +SRT_SLURM_COMMIT="42b91455f9db13cd2cdb8822baaa834d05c40909" SLURM_PARTITION="compute" SGLANG_IMAGE="lmsysorg/sglang-rocm:v0.5.16-rocm720-mi35x-20260728" SHARED_BASE="/it-share/gharunners2/srt-slurm" diff --git a/utils/test_mi355x_srt_slurm_contract.py b/utils/test_mi355x_srt_slurm_contract.py index cc67fb95fe..e811a2260d 100644 --- a/utils/test_mi355x_srt_slurm_contract.py +++ b/utils/test_mi355x_srt_slurm_contract.py @@ -108,7 +108,7 @@ def test_launcher_pins_runtime_and_preserves_legacy_default(): launcher = LAUNCHER.read_text() legacy = LEGACY_LAUNCHER.read_text() - assert "25e1e4e71dc8e7383b1857041decff1a9ae0339e" in launcher + assert "42b91455f9db13cd2cdb8822baaa834d05c40909" in launcher assert "v0.5.16-rocm720-mi35x-20260728" in launcher assert "make setup-compute ARCH=x86_64" in launcher assert "SRTCTL_RUNTIME_SOURCE_DIR" in launcher From 87933a9384e5990925f259de2dadffaf9b067427 Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Mon, 10 Aug 2026 02:55:21 -0500 Subject: [PATCH 36/67] fix(amd): select Qwen3.5 text loader --- .../sglang/qwen3.5/mi355x/disagg-1p1d-tp8-fixed-seq.yaml | 5 +++++ utils/test_mi355x_srt_slurm_contract.py | 3 +++ 2 files changed, 8 insertions(+) diff --git a/benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3.5/mi355x/disagg-1p1d-tp8-fixed-seq.yaml b/benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3.5/mi355x/disagg-1p1d-tp8-fixed-seq.yaml index c27d4f5e81..f4b081f4bc 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3.5/mi355x/disagg-1p1d-tp8-fixed-seq.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3.5/mi355x/disagg-1p1d-tp8-fixed-seq.yaml @@ -70,6 +70,11 @@ backend: sglang_config: prefill: &common_config served-model-name: "Qwen/Qwen3.5-397B-A17B-FP8" + # The published FP8 checkpoint is text-only (model_type=qwen3_5_moe_text, + # no vision_config), but its top-level architecture still names the + # multimodal conditional-generation class. Select SGLang's matching + # built-in text loader so it does not initialize a Qwen-VL processor. + json-model-override-args: '{"architectures":["Qwen3_5MoeForCausalLM"]}' tensor-parallel-size: 8 disaggregation-transfer-backend: mori disaggregation-ib-device: "rdma0,rdma1,rdma2,rdma3,rdma4,rdma5,rdma6,rdma7" diff --git a/utils/test_mi355x_srt_slurm_contract.py b/utils/test_mi355x_srt_slurm_contract.py index e811a2260d..836e6e013a 100644 --- a/utils/test_mi355x_srt_slurm_contract.py +++ b/utils/test_mi355x_srt_slurm_contract.py @@ -182,6 +182,9 @@ def test_production_disagg_recipe_uses_two_full_nodes_and_the_existing_workload( assert config["tensor-parallel-size"] == 8 assert config["disaggregation-transfer-backend"] == "mori" assert config["attention-backend"] == "aiter" + assert config["json-model-override-args"] == ( + '{"architectures":["Qwen3_5MoeForCausalLM"]}' + ) command = recipe["benchmark"]["command"] assert "utils/bench_serving/benchmark_serving.py" in command assert "--random-input-len 8192" in command From a55bed72c0bcf3728b3bdd6d034dca7d5cae5a39 Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Mon, 10 Aug 2026 03:03:33 -0500 Subject: [PATCH 37/67] fix(amd): allocate full MI355X CPU topology --- .../sglang/qwen3.5/mi355x/disagg-1p1d-tp8-fixed-seq.yaml | 7 +++++++ utils/test_mi355x_srt_slurm_contract.py | 4 ++++ 2 files changed, 11 insertions(+) diff --git a/benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3.5/mi355x/disagg-1p1d-tp8-fixed-seq.yaml b/benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3.5/mi355x/disagg-1p1d-tp8-fixed-seq.yaml index f4b081f4bc..4a47caf6cf 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3.5/mi355x/disagg-1p1d-tp8-fixed-seq.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3.5/mi355x/disagg-1p1d-tp8-fixed-seq.yaml @@ -22,6 +22,13 @@ identity: slurm: time_limit: "02:00:00" +# This topology owns every GPU on each MI355X node. Give SGLang the complete +# 128-core cpuset as well: its ROCm image enables per-GPU CPU affinity, which +# maps the eight TP ranks across the full dual-socket CPU topology. +sbatch_directives: + cpus-per-task: "128" + mem: "0" + resources: gpu_type: "mi355x" gpus_per_node: 8 diff --git a/utils/test_mi355x_srt_slurm_contract.py b/utils/test_mi355x_srt_slurm_contract.py index 836e6e013a..3648210c77 100644 --- a/utils/test_mi355x_srt_slurm_contract.py +++ b/utils/test_mi355x_srt_slurm_contract.py @@ -176,6 +176,10 @@ def test_production_disagg_recipe_uses_two_full_nodes_and_the_existing_workload( "gpus_per_prefill": 8, "gpus_per_decode": 8, } + assert recipe["sbatch_directives"] == { + "cpus-per-task": "128", + "mem": "0", + } assert recipe["frontend"]["type"] == "sgl-router" for role in ("prefill", "decode"): config = recipe["backend"]["sglang_config"][role] From d3ed345084932b7a91130c70d8cd99141bfb2a7a Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Mon, 10 Aug 2026 03:15:15 -0500 Subject: [PATCH 38/67] fix(amd): align Qwen3.5 Hugging Face cache root --- .../mi355x/disagg-1p1d-tp8-fixed-seq.yaml | 16 +++++++--------- utils/test_mi355x_srt_slurm_contract.py | 9 ++++++--- 2 files changed, 13 insertions(+), 12 deletions(-) diff --git a/benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3.5/mi355x/disagg-1p1d-tp8-fixed-seq.yaml b/benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3.5/mi355x/disagg-1p1d-tp8-fixed-seq.yaml index 4a47caf6cf..1941a52be6 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3.5/mi355x/disagg-1p1d-tp8-fixed-seq.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3.5/mi355x/disagg-1p1d-tp8-fixed-seq.yaml @@ -51,8 +51,11 @@ backend: type: sglang prefill_environment: &common_environment HF_HOME: "/hf_hub_cache" - HF_HUB_CACHE: "/hf_hub_cache" - HUGGINGFACE_HUB_CACHE: "/hf_hub_cache" + # Hugging Face stores hub snapshots under $HF_HOME/hub. Keep the explicit + # cache variables on that same path so srt-slurm's prefetch and every + # backend process resolve the identical, current snapshot. + HF_HUB_CACHE: "/hf_hub_cache/hub" + HUGGINGFACE_HUB_CACHE: "/hf_hub_cache/hub" PYTHONUNBUFFERED: "1" IBDEVICES: "rdma0,rdma1,rdma2,rdma3,rdma4,rdma5,rdma6,rdma7" GLOO_SOCKET_IFNAME: "eno0" @@ -77,11 +80,6 @@ backend: sglang_config: prefill: &common_config served-model-name: "Qwen/Qwen3.5-397B-A17B-FP8" - # The published FP8 checkpoint is text-only (model_type=qwen3_5_moe_text, - # no vision_config), but its top-level architecture still names the - # multimodal conditional-generation class. Select SGLang's matching - # built-in text loader so it does not initialize a Qwen-VL processor. - json-model-override-args: '{"architectures":["Qwen3_5MoeForCausalLM"]}' tensor-parallel-size: 8 disaggregation-transfer-backend: mori disaggregation-ib-device: "rdma0,rdma1,rdma2,rdma3,rdma4,rdma5,rdma6,rdma7" @@ -143,5 +141,5 @@ benchmark: --result-filename "qwen3.5-fp8-disagg-isl8192-osl1024-c${concurrency}.json" env: HF_HOME: /hf_hub_cache - HF_HUB_CACHE: /hf_hub_cache - HUGGINGFACE_HUB_CACHE: /hf_hub_cache + HF_HUB_CACHE: /hf_hub_cache/hub + HUGGINGFACE_HUB_CACHE: /hf_hub_cache/hub diff --git a/utils/test_mi355x_srt_slurm_contract.py b/utils/test_mi355x_srt_slurm_contract.py index 3648210c77..d826dcf312 100644 --- a/utils/test_mi355x_srt_slurm_contract.py +++ b/utils/test_mi355x_srt_slurm_contract.py @@ -181,14 +181,17 @@ def test_production_disagg_recipe_uses_two_full_nodes_and_the_existing_workload( "mem": "0", } assert recipe["frontend"]["type"] == "sgl-router" + for role in ("prefill", "decode"): + environment = recipe["backend"][f"{role}_environment"] + assert environment["HF_HOME"] == "/hf_hub_cache" + assert environment["HF_HUB_CACHE"] == "/hf_hub_cache/hub" + assert environment["HUGGINGFACE_HUB_CACHE"] == "/hf_hub_cache/hub" for role in ("prefill", "decode"): config = recipe["backend"]["sglang_config"][role] assert config["tensor-parallel-size"] == 8 assert config["disaggregation-transfer-backend"] == "mori" assert config["attention-backend"] == "aiter" - assert config["json-model-override-args"] == ( - '{"architectures":["Qwen3_5MoeForCausalLM"]}' - ) + assert "json-model-override-args" not in config command = recipe["benchmark"]["command"] assert "utils/bench_serving/benchmark_serving.py" in command assert "--random-input-len 8192" in command From 3b3acff331116dfbf9d3df5d170ad88e5509956c Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Mon, 10 Aug 2026 04:03:44 -0500 Subject: [PATCH 39/67] fix(amd): use large-MR-capable MI355X runtime --- .../cluster-configs/mi355x-amds.yaml | 2 +- .../qwen3-0.6b/mi355x/agg-fixed-seq.yaml | 6 ++-- .../mi355x/disagg-1p1d-fixed-seq.yaml | 8 ++--- .../mi355x/disagg-1p1d-tp8-fixed-seq.yaml | 8 ++--- configs/amd-master.yaml | 6 ++-- perf-changelog.yaml | 4 ++- runners/launch_mi355x-amds-srt.sh | 8 ++--- utils/test_mi355x_srt_slurm_contract.py | 29 ++++++++++++++----- 8 files changed, 44 insertions(+), 27 deletions(-) diff --git a/benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi355x-amds.yaml b/benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi355x-amds.yaml index d4ca309b1d..078ef196c9 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi355x-amds.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi355x-amds.yaml @@ -27,6 +27,6 @@ default_mounts: # The launcher appends the exact shared workspace and result roots. containers: - sglang-rocm-v0.5.16-mi35x: /it-share/gharunners2/srt-slurm/containers/sglang-rocm-v0.5.16-mi35x-20260728.sqsh + sglang-rocm-v0.5.17-mi35x: /it-share/gharunners2/srt-slurm/containers/sglang-rocm-v0.5.17-mi35x-20260809.sqsh nginx_raise_ulimit: false diff --git a/benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3-0.6b/mi355x/agg-fixed-seq.yaml b/benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3-0.6b/mi355x/agg-fixed-seq.yaml index 938dae7434..3d3ebbe1f4 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3-0.6b/mi355x/agg-fixed-seq.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3-0.6b/mi355x/agg-fixed-seq.yaml @@ -5,16 +5,16 @@ name: "mi355x-sglang-qwen3-0.6b-agg-fixed-seq" model: path: "hf:Qwen/Qwen3-0.6B" - container: "sglang-rocm-v0.5.16-mi35x" + container: "sglang-rocm-v0.5.17-mi35x" precision: "fp16" identity: model: repo: "Qwen/Qwen3-0.6B" container: - image: "lmsysorg/sglang-rocm:v0.5.16-rocm720-mi35x-20260728" + image: "lmsysorg/sglang-rocm:v0.5.17-rocm720-mi35x-20260809" frameworks: - sglang: "0.5.16.dev20260728+g32c30c0f96" + sglang: "0.5.17.dev20260809+g7120f3ee13" sglang-router: "0.3.2" slurm: diff --git a/benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3-0.6b/mi355x/disagg-1p1d-fixed-seq.yaml b/benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3-0.6b/mi355x/disagg-1p1d-fixed-seq.yaml index 3b2686033f..cf3eca6fd1 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3-0.6b/mi355x/disagg-1p1d-fixed-seq.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3-0.6b/mi355x/disagg-1p1d-fixed-seq.yaml @@ -5,18 +5,18 @@ name: "mi355x-sglang-qwen3-0.6b-disagg-1p1d-fixed-seq" model: path: "hf:Qwen/Qwen3-0.6B" - container: "sglang-rocm-v0.5.16-mi35x" + container: "sglang-rocm-v0.5.17-mi35x" precision: "fp16" identity: model: repo: "Qwen/Qwen3-0.6B" container: - image: "lmsysorg/sglang-rocm:v0.5.16-rocm720-mi35x-20260728" + image: "lmsysorg/sglang-rocm:v0.5.17-rocm720-mi35x-20260809" frameworks: - sglang: "0.5.16.dev20260728+g32c30c0f96" + sglang: "0.5.17.dev20260809+g7120f3ee13" sglang-router: "0.3.2" - amd-mori: "0.5.16.dev20260728+g32c30c0f96" + amd-mori: "0.5.17.dev20260809+g7120f3ee13" slurm: time_limit: "00:45:00" diff --git a/benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3.5/mi355x/disagg-1p1d-tp8-fixed-seq.yaml b/benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3.5/mi355x/disagg-1p1d-tp8-fixed-seq.yaml index 1941a52be6..a7f422fa35 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3.5/mi355x/disagg-1p1d-tp8-fixed-seq.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3.5/mi355x/disagg-1p1d-tp8-fixed-seq.yaml @@ -6,18 +6,18 @@ name: "mi355x-sglang-qwen3.5-fp8-disagg-1p1d-tp8-fixed-seq" model: path: "hf:Qwen/Qwen3.5-397B-A17B-FP8" - container: "sglang-rocm-v0.5.16-mi35x" + container: "sglang-rocm-v0.5.17-mi35x" precision: "fp8" identity: model: repo: "Qwen/Qwen3.5-397B-A17B-FP8" container: - image: "lmsysorg/sglang-rocm:v0.5.16-rocm720-mi35x-20260728" + image: "lmsysorg/sglang-rocm:v0.5.17-rocm720-mi35x-20260809" frameworks: - sglang: "0.5.16.dev20260728+g32c30c0f96" + sglang: "0.5.17.dev20260809+g7120f3ee13" sglang-router: "0.3.2" - amd-mori: "0.5.16.dev20260728+g32c30c0f96" + amd-mori: "0.5.17.dev20260809+g7120f3ee13" slurm: time_limit: "02:00:00" diff --git a/configs/amd-master.yaml b/configs/amd-master.yaml index a6d492d394..97842f3443 100644 --- a/configs/amd-master.yaml +++ b/configs/amd-master.yaml @@ -1567,7 +1567,7 @@ qwen3-0.6b-fp16-mi300x-vllm-srt-disagg: # MI355X. Both use the unchanged InferenceX fixed-sequence benchmark; the P/D # lane uses AMD MoRI and neither lane starts Dynamo, NATS, etcd, or NIXL. qwen3-0.6b-fp16-mi355x-sglang-srt-agg: - image: lmsysorg/sglang-rocm:v0.5.16-rocm720-mi35x-20260728 + image: lmsysorg/sglang-rocm:v0.5.17-rocm720-mi35x-20260809 model: Qwen/Qwen3-0.6B model-prefix: qwen3-0.6b runner: cluster:mi355x-amds @@ -1596,7 +1596,7 @@ qwen3-0.6b-fp16-mi355x-sglang-srt-agg: dp-attn: false qwen3-0.6b-fp16-mi355x-sglang-srt-disagg: - image: lmsysorg/sglang-rocm:v0.5.16-rocm720-mi35x-20260728 + image: lmsysorg/sglang-rocm:v0.5.17-rocm720-mi35x-20260809 model: Qwen/Qwen3-0.6B model-prefix: qwen3-0.6b runner: cluster:mi355x-amds @@ -1629,7 +1629,7 @@ qwen3-0.6b-fp16-mi355x-sglang-srt-disagg: # srt-slurm. One full TP8 node serves prefill, one TP8 node serves decode, and # the request/KV planes remain native SGLang Router + AMD MoRI. qwen3.5-fp8-mi355x-sglang-srt-disagg: - image: lmsysorg/sglang-rocm:v0.5.16-rocm720-mi35x-20260728 + image: lmsysorg/sglang-rocm:v0.5.17-rocm720-mi35x-20260809 model: Qwen/Qwen3.5-397B-A17B-FP8 model-prefix: qwen3.5 runner: cluster:mi355x-amds diff --git a/perf-changelog.yaml b/perf-changelog.yaml index eef886bfe6..ac396591ac 100644 --- a/perf-changelog.yaml +++ b/perf-changelog.yaml @@ -5720,6 +5720,8 @@ - "Add native AMD orchestration to srt-slurm and validate aggregate plus disaggregated serving on MI300X and MI355X without Dynamo, NIXL, NATS, or etcd." - "Use the native vLLM Router with MoRIIO for MI300X disaggregation and the native SGLang Router with AMD MoRI for MI355X disaggregation; aggregate paths use the corresponding direct backend." - "Run the repository's unchanged utils/bench_serving/benchmark_serving.py through srt-slurm's custom benchmark contract for fixed-sequence validation." - - "Port the production Qwen3.5-397B-A17B-FP8 MI355X topology as two full TP8 nodes, one prefill and one decode, using lmsysorg/sglang-rocm:v0.5.16-rocm720-mi35x-20260728, SGLang Router 0.3.2, MoRI, AITer attention, FP8 KV cache, and an 8192-input/1024-output concurrency-8 workload." + - "Port the production Qwen3.5-397B-A17B-FP8 MI355X topology as two full TP8 nodes, one prefill and one decode, using lmsysorg/sglang-rocm:v0.5.17-rocm720-mi35x-20260809, SGLang Router, MoRI, AITer attention, FP8 KV cache, and an 8192-input/1024-output concurrency-8 workload." + - "Use the current steady ROCm image because its MoRI IO path contains the merged automatic dma-buf fallback for large GPU memory registrations; the prior 20260728 image used plain ibv_reg_mr and failed reproducibly above 3.75 GiB." + - "Propagate runtime Slurm options to native static routers so the router inherits the same writable/remapped container contract as workers and benchmarks." - "Pin the InferenceX launcher to the exact srt-slurm AMD runtime commit and validate model, container, backend, router, and MoRI package identity before serving." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2542 diff --git a/runners/launch_mi355x-amds-srt.sh b/runners/launch_mi355x-amds-srt.sh index d9f132c3d6..84867d1597 100755 --- a/runners/launch_mi355x-amds-srt.sh +++ b/runners/launch_mi355x-amds-srt.sh @@ -5,11 +5,11 @@ set -euo pipefail # in explicitly with CONFIG_FILE; all existing MI355X launch behavior remains # unchanged for every other row. SRT_SLURM_REPOSITORY="https://github.com/SemiAnalysisAI/srt-slurm.git" -SRT_SLURM_COMMIT="42b91455f9db13cd2cdb8822baaa834d05c40909" +SRT_SLURM_COMMIT="7beb0a1b86bee281ac36affb6f2c9cbb23a3a616" SLURM_PARTITION="compute" -SGLANG_IMAGE="lmsysorg/sglang-rocm:v0.5.16-rocm720-mi35x-20260728" +SGLANG_IMAGE="lmsysorg/sglang-rocm:v0.5.17-rocm720-mi35x-20260809" SHARED_BASE="/it-share/gharunners2/srt-slurm" -SHARED_IMAGE="${SHARED_BASE}/containers/sglang-rocm-v0.5.16-mi35x-20260728.sqsh" +SHARED_IMAGE="${SHARED_BASE}/containers/sglang-rocm-v0.5.17-mi35x-20260809.sqsh" SHARED_HF_CACHE="/it-share/hf-hub-cache" SHARED_RESULTS="${SHARED_BASE}/results" @@ -49,7 +49,7 @@ flock -w 2400 9 if ! unsquashfs -s "$SHARED_IMAGE" >/dev/null 2>&1; then tmp="${SHARED_IMAGE}.tmp.\${SLURM_JOB_ID}" rm -f "\$tmp" - local_image="/var/lib/squash/lmsysorg_sglang-rocm_v0.5.16-rocm720-mi35x-20260728.sqsh" + local_image="/var/lib/squash/lmsysorg_sglang-rocm_v0.5.17-rocm720-mi35x-20260809.sqsh" if unsquashfs -s "\$local_image" >/dev/null 2>&1; then cp --sparse=always "\$local_image" "\$tmp" else diff --git a/utils/test_mi355x_srt_slurm_contract.py b/utils/test_mi355x_srt_slurm_contract.py index d826dcf312..43be84e75e 100644 --- a/utils/test_mi355x_srt_slurm_contract.py +++ b/utils/test_mi355x_srt_slurm_contract.py @@ -38,11 +38,11 @@ def test_cluster_profile_matches_the_mi355x_rocm_slurm_contract(): assert cluster["runtime_config_transport"] == "shared-filesystem" assert cluster["default_mounts"]["/dev/kfd"] == "/dev/kfd" assert cluster["default_mounts"]["/dev/dri"] == "/dev/dri" - assert cluster["containers"]["sglang-rocm-v0.5.16-mi35x"].endswith( - "/sglang-rocm-v0.5.16-mi35x-20260728.sqsh" + assert cluster["containers"]["sglang-rocm-v0.5.17-mi35x"].endswith( + "/sglang-rocm-v0.5.17-mi35x-20260809.sqsh" ) assert cluster["output_dir"].startswith("/it-share/gharunners2/srt-slurm/") - assert cluster["containers"]["sglang-rocm-v0.5.16-mi35x"].startswith( + assert cluster["containers"]["sglang-rocm-v0.5.17-mi35x"].startswith( "/it-share/gharunners2/srt-slurm/" ) @@ -50,6 +50,7 @@ def test_cluster_profile_matches_the_mi355x_rocm_slurm_contract(): def test_recipes_use_native_sglang_router_and_only_disagg_uses_mori(): agg = yaml.safe_load(AGG_RECIPE.read_text()) disagg = yaml.safe_load(DISAGG_RECIPE.read_text()) + production_disagg = yaml.safe_load(PRODUCTION_DISAGG_RECIPE.read_text()) assert agg["resources"] == { "gpu_type": "mi355x", @@ -69,12 +70,23 @@ def test_recipes_use_native_sglang_router_and_only_disagg_uses_mori(): assert "rdma0" in disagg["backend"][f"{role}_environment"]["IBDEVICES"] assert "mori" not in AGG_RECIPE.read_text().lower() - for recipe in (AGG_RECIPE, DISAGG_RECIPE, PRODUCTION_DISAGG_RECIPE): - text = recipe.read_text().lower() + expected_image = "lmsysorg/sglang-rocm:v0.5.17-rocm720-mi35x-20260809" + expected_version = "0.5.17.dev20260809+g7120f3ee13" + for recipe_path, recipe in ( + (AGG_RECIPE, agg), + (DISAGG_RECIPE, disagg), + (PRODUCTION_DISAGG_RECIPE, production_disagg), + ): + text = recipe_path.read_text().lower() assert "dynamo" not in text assert "nixl" not in text assert "nats" not in text assert "etcd" not in text + assert recipe["model"]["container"] == "sglang-rocm-v0.5.17-mi35x" + assert recipe["identity"]["container"]["image"] == expected_image + assert recipe["identity"]["frameworks"]["sglang"] == expected_version + for recipe in (disagg, production_disagg): + assert recipe["identity"]["frameworks"]["amd-mori"] == expected_version def test_matrix_rows_explicitly_select_the_srt_recipes(): @@ -97,6 +109,9 @@ def test_matrix_rows_explicitly_select_the_srt_recipes(): config = master[name] search = config["scenarios"]["fixed-seq-len"][0]["search-space"][0] assert config["runner"] == "cluster:mi355x-amds" + assert config["image"] == ( + "lmsysorg/sglang-rocm:v0.5.17-rocm720-mi35x-20260809" + ) assert config["multinode"] is True assert config["disagg"] is is_disagg assert search["prefill"]["additional-settings"] == [ @@ -108,8 +123,8 @@ def test_launcher_pins_runtime_and_preserves_legacy_default(): launcher = LAUNCHER.read_text() legacy = LEGACY_LAUNCHER.read_text() - assert "42b91455f9db13cd2cdb8822baaa834d05c40909" in launcher - assert "v0.5.16-rocm720-mi35x-20260728" in launcher + assert "7beb0a1b86bee281ac36affb6f2c9cbb23a3a616" in launcher + assert "v0.5.17-rocm720-mi35x-20260809" in launcher assert "make setup-compute ARCH=x86_64" in launcher assert "SRTCTL_RUNTIME_SOURCE_DIR" in launcher assert 'SHARED_BASE="/it-share/gharunners2/srt-slurm"' in launcher From c5ce0a4a1f939f094e6cd2a3124669b2529cd85f Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Mon, 10 Aug 2026 04:46:36 -0500 Subject: [PATCH 40/67] fix(amd): make runtime staging reproducible --- .../srt-slurm-recipes/AMD_BRINGUP.md | 10 +++-- .../qwen3-0.6b/mi355x/agg-fixed-seq.yaml | 8 ++-- .../mi355x/disagg-1p1d-fixed-seq.yaml | 8 ++-- perf-changelog.yaml | 2 + runners/launch_mi300x-amds-srt.sh | 39 +++++++++++++++---- runners/launch_mi355x-amds-srt.sh | 4 +- utils/test_mi300x_srt_slurm_contract.py | 14 +++++-- utils/test_mi355x_srt_slurm_contract.py | 20 +++++++++- 8 files changed, 80 insertions(+), 25 deletions(-) diff --git a/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md b/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md index d8226fdbcb..f8548d3686 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md +++ b/benchmarks/multi_node/srt-slurm-recipes/AMD_BRINGUP.md @@ -5,17 +5,19 @@ This document tracks the work-in-progress integration of InferenceX AMD Slurm clusters. The project is a functional orchestration bring-up, not a performance-tuning exercise. -Current development pin: +Current development pin for both AMD launchers: - repository: `SemiAnalysisAI/srt-slurm` - branch: `agent/amd-multinode-runtime` -- commit: `105824810a67d52b58761077ad3b94d4a05eb3ac` +- commit: `315e4b06a7e0806194a646ea21832e750e896a46` The MI300X launcher uses srt-slurm's supported `--no-preflight` submission mode because the immutable squashfs files live on compute-node-local RAID rather than the login node. Before submission, the launcher stages the benchmark -runtime across the same eligible node pool; missing container files still fail -loudly when Pyxis starts the allocation. +runtime across the same eligible node pool. Missing engine and router images +are imported atomically under per-image locks from the pinned public +`vllm/vllm-openai-rocm:v0.26.0` and +`vllm/vllm-router:nightly-20260809-d2ba586` images. The MI300X login and compute nodes also do not share the Actions checkout. The staging allocation checks out the exact pinned srt-slurm commit on every diff --git a/benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3-0.6b/mi355x/agg-fixed-seq.yaml b/benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3-0.6b/mi355x/agg-fixed-seq.yaml index 3d3ebbe1f4..fb508e7865 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3-0.6b/mi355x/agg-fixed-seq.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3-0.6b/mi355x/agg-fixed-seq.yaml @@ -37,8 +37,8 @@ backend: type: sglang aggregated_environment: HF_HOME: "/hf_hub_cache" - HF_HUB_CACHE: "/hf_hub_cache" - HUGGINGFACE_HUB_CACHE: "/hf_hub_cache" + HF_HUB_CACHE: "/hf_hub_cache/hub" + HUGGINGFACE_HUB_CACHE: "/hf_hub_cache/hub" PYTHONUNBUFFERED: "1" sglang_config: aggregated: @@ -90,5 +90,5 @@ benchmark: done env: HF_HOME: /hf_hub_cache - HF_HUB_CACHE: /hf_hub_cache - HUGGINGFACE_HUB_CACHE: /hf_hub_cache + HF_HUB_CACHE: /hf_hub_cache/hub + HUGGINGFACE_HUB_CACHE: /hf_hub_cache/hub diff --git a/benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3-0.6b/mi355x/disagg-1p1d-fixed-seq.yaml b/benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3-0.6b/mi355x/disagg-1p1d-fixed-seq.yaml index cf3eca6fd1..6282e5dbea 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3-0.6b/mi355x/disagg-1p1d-fixed-seq.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3-0.6b/mi355x/disagg-1p1d-fixed-seq.yaml @@ -43,8 +43,8 @@ backend: type: sglang prefill_environment: &worker_environment HF_HOME: "/hf_hub_cache" - HF_HUB_CACHE: "/hf_hub_cache" - HUGGINGFACE_HUB_CACHE: "/hf_hub_cache" + HF_HUB_CACHE: "/hf_hub_cache/hub" + HUGGINGFACE_HUB_CACHE: "/hf_hub_cache/hub" PYTHONUNBUFFERED: "1" IBDEVICES: "rdma0,rdma1,rdma2,rdma3,rdma4,rdma5,rdma6,rdma7" MORI_RDMA_TC: "104" @@ -103,5 +103,5 @@ benchmark: done env: HF_HOME: /hf_hub_cache - HF_HUB_CACHE: /hf_hub_cache - HUGGINGFACE_HUB_CACHE: /hf_hub_cache + HF_HUB_CACHE: /hf_hub_cache/hub + HUGGINGFACE_HUB_CACHE: /hf_hub_cache/hub diff --git a/perf-changelog.yaml b/perf-changelog.yaml index ac396591ac..2ef42148b6 100644 --- a/perf-changelog.yaml +++ b/perf-changelog.yaml @@ -5724,4 +5724,6 @@ - "Use the current steady ROCm image because its MoRI IO path contains the merged automatic dma-buf fallback for large GPU memory registrations; the prior 20260728 image used plain ibv_reg_mr and failed reproducibly above 3.75 GiB." - "Propagate runtime Slurm options to native static routers so the router inherits the same writable/remapped container contract as workers and benchmarks." - "Pin the InferenceX launcher to the exact srt-slurm AMD runtime commit and validate model, container, backend, router, and MoRI package identity before serving." + - "Stage every MI355X model into the standard $HF_HOME/hub layout used by all aggregate, prefill, decode, and benchmark processes, avoiding a second flat cache tree on cold clusters." + - "Recreate missing MI300X engine and router squashfs files from their pinned public vLLM images under per-node locks, validate each temporary squashfs, and publish it atomically before submission." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2542 diff --git a/runners/launch_mi300x-amds-srt.sh b/runners/launch_mi300x-amds-srt.sh index f56bb0b62a..ba06c6c651 100644 --- a/runners/launch_mi300x-amds-srt.sh +++ b/runners/launch_mi300x-amds-srt.sh @@ -4,10 +4,14 @@ set -euo pipefail # MI300X srt-slurm validation path. The existing launcher remains the default; # matrix rows opt in by exporting CONFIG_FILE through additional-settings. SRT_SLURM_REPOSITORY="https://github.com/SemiAnalysisAI/srt-slurm.git" -SRT_SLURM_COMMIT="105824810a67d52b58761077ad3b94d4a05eb3ac" +SRT_SLURM_COMMIT="315e4b06a7e0806194a646ea21832e750e896a46" SLURM_PARTITION="compute" EXCLUDED_NODES="chi-mi300x-049,chi-mi300x-121" REMOTE_BASE="/raid/hf-hub-cache/inferencex/srt-slurm" +VLLM_IMAGE="vllm/vllm-openai-rocm:v0.26.0" +VLLM_ROUTER_IMAGE="vllm/vllm-router:nightly-20260809-d2ba586" +VLLM_SQSH="${REMOTE_BASE}/containers/vllm-openai-rocm-v0.26.0.sqsh" +VLLM_ROUTER_SQSH="${REMOTE_BASE}/containers/vllm-router-nightly-20260809-d2ba586.sqsh" : "${GITHUB_WORKSPACE:?GITHUB_WORKSPACE must be set by Actions}" : "${RESULT_FILENAME:?RESULT_FILENAME must be set by the benchmark workflow}" @@ -29,8 +33,9 @@ SRT_REPO_DIR="${WORK_DIR}/srt-slurm" mkdir -p "$WORK_DIR" # The login and compute nodes do not share a filesystem. Stage only the -# unchanged InferenceX benchmark client onto every eligible node. The batch job -# exits normally; it does not cancel or preempt any allocation. +# unchanged InferenceX benchmark client and immutable public container images +# onto every eligible node. The batch job exits normally; it does not cancel or +# preempt any allocation. RUNTIME_ARCHIVE="${WORK_DIR}/inferencex-benchmark.tar.gz" tar -C "$GITHUB_WORKSPACE" -czf "$RUNTIME_ARCHIVE" utils/bench_serving RUNTIME_PAYLOAD=$(base64 -w0 "$RUNTIME_ARCHIVE") @@ -41,7 +46,7 @@ cat > "$STAGE_SCRIPT" </dev/null 2>&1; then + return + fi + exec {lock_fd}>"\${target}.lock" + flock -w 2400 "\$lock_fd" + if ! unsquashfs -s "\$target" >/dev/null 2>&1; then + tmp="\${target}.tmp.\${SLURM_JOB_ID}" + rm -f "\$tmp" + enroot import -o "\$tmp" "docker://\${image}" + unsquashfs -s "\$tmp" >/dev/null + mv "\$tmp" "\$target" + fi + flock -u "\$lock_fd" + exec {lock_fd}>&- + } + ensure_container_image "${VLLM_SQSH}" "${VLLM_IMAGE}" + ensure_container_image "${VLLM_ROUTER_SQSH}" "${VLLM_ROUTER_IMAGE}" if [[ ! -d "\$srt_runtime/.git" ]]; then git clone --quiet "${SRT_SLURM_REPOSITORY}" "\$srt_runtime" fi diff --git a/runners/launch_mi355x-amds-srt.sh b/runners/launch_mi355x-amds-srt.sh index 84867d1597..8873cf7a98 100755 --- a/runners/launch_mi355x-amds-srt.sh +++ b/runners/launch_mi355x-amds-srt.sh @@ -5,7 +5,7 @@ set -euo pipefail # in explicitly with CONFIG_FILE; all existing MI355X launch behavior remains # unchanged for every other row. SRT_SLURM_REPOSITORY="https://github.com/SemiAnalysisAI/srt-slurm.git" -SRT_SLURM_COMMIT="7beb0a1b86bee281ac36affb6f2c9cbb23a3a616" +SRT_SLURM_COMMIT="315e4b06a7e0806194a646ea21832e750e896a46" SLURM_PARTITION="compute" SGLANG_IMAGE="lmsysorg/sglang-rocm:v0.5.17-rocm720-mi35x-20260809" SHARED_BASE="/it-share/gharunners2/srt-slurm" @@ -63,7 +63,7 @@ srun --nodes=1 --ntasks=1 \ --container-image="$SHARED_IMAGE" \ --container-mounts="$SHARED_HF_CACHE:/hf_hub_cache" \ --container-writable --container-remap-root --no-container-entrypoint \ - --export=ALL,HF_HOME=/hf_hub_cache,HF_HUB_CACHE=/hf_hub_cache,HUGGINGFACE_HUB_CACHE=/hf_hub_cache,MODEL_REPO=${MODEL} \ + --export=ALL,HF_HOME=/hf_hub_cache,MODEL_REPO=${MODEL} \ python3 -c 'import os; from huggingface_hub import snapshot_download; snapshot_download(os.environ["MODEL_REPO"])' EOF STAGE_JOB_ID=$(sbatch --wait --parsable "$STAGE_SCRIPT") diff --git a/utils/test_mi300x_srt_slurm_contract.py b/utils/test_mi300x_srt_slurm_contract.py index 25c42ca2cd..d558d7afb8 100644 --- a/utils/test_mi300x_srt_slurm_contract.py +++ b/utils/test_mi300x_srt_slurm_contract.py @@ -92,11 +92,19 @@ def test_official_matrix_routes_disagg_through_the_pinned_srt_launcher(): "CONFIG_FILE=recipes/vllm/qwen3-0.6b/mi300x/" "disagg-1p1d-fixed-seq.yaml" ] - assert "105824810a67d52b58761077ad3b94d4a05eb3ac" in launcher + assert "315e4b06a7e0806194a646ea21832e750e896a46" in launcher assert "make setup-compute ARCH=x86_64" in launcher assert "--no-preflight" in launcher - assert "test -r /raid/hf-hub-cache/inferencex/srt-slurm/containers/vllm-openai-rocm-v0.26.0.sqsh" in launcher - assert "test -r /raid/hf-hub-cache/inferencex/srt-slurm/containers/vllm-router-nightly-20260809-d2ba586.sqsh" in launcher + assert 'VLLM_IMAGE="vllm/vllm-openai-rocm:v0.26.0"' in launcher + assert ( + 'VLLM_ROUTER_IMAGE="vllm/vllm-router:nightly-20260809-d2ba586"' + in launcher + ) + assert 'enroot import -o "\\$tmp" "docker://\\${image}"' in launcher + assert 'exec {lock_fd}>"\\${target}.lock"' in launcher + assert 'flock -w 2400 "\\$lock_fd"' in launcher + assert 'unsquashfs -s "\\$tmp"' in launcher + assert 'mv "\\$tmp" "\\$target"' in launcher assert 'REMOTE_SRT_RUNTIME="${REMOTE_BASE}/runtime/srt-slurm-${SRT_SLURM_COMMIT}"' in launcher assert 'git -C "\\$srt_runtime" checkout --quiet --detach "${SRT_SLURM_COMMIT}"' in launcher assert 'make -C "\\$srt_runtime" --no-print-directory setup-compute ARCH=x86_64' in launcher diff --git a/utils/test_mi355x_srt_slurm_contract.py b/utils/test_mi355x_srt_slurm_contract.py index 43be84e75e..33845e48dc 100644 --- a/utils/test_mi355x_srt_slurm_contract.py +++ b/utils/test_mi355x_srt_slurm_contract.py @@ -123,7 +123,7 @@ def test_launcher_pins_runtime_and_preserves_legacy_default(): launcher = LAUNCHER.read_text() legacy = LEGACY_LAUNCHER.read_text() - assert "7beb0a1b86bee281ac36affb6f2c9cbb23a3a616" in launcher + assert "315e4b06a7e0806194a646ea21832e750e896a46" in launcher assert "v0.5.17-rocm720-mi35x-20260809" in launcher assert "make setup-compute ARCH=x86_64" in launcher assert "SRTCTL_RUNTIME_SOURCE_DIR" in launcher @@ -132,11 +132,29 @@ def test_launcher_pins_runtime_and_preserves_legacy_default(): assert "scancel" not in launcher assert ': "${MODEL:?MODEL must identify the Hugging Face model}"' in launcher assert "MODEL_REPO=${MODEL}" in launcher + assert "HF_HOME=/hf_hub_cache,MODEL_REPO=${MODEL}" in launcher + assert "HF_HUB_CACHE=/hf_hub_cache,MODEL_REPO" not in launcher assert 'snapshot_download(os.environ["MODEL_REPO"])' in launcher assert 'if [[ -n "${CONFIG_FILE:-}" ]]; then' in legacy assert "launch_mi355x-amds-srt.sh" in legacy +def test_all_mi355x_roles_use_the_staged_hugging_face_hub_cache(): + for recipe_path in (AGG_RECIPE, DISAGG_RECIPE, PRODUCTION_DISAGG_RECIPE): + recipe = yaml.safe_load(recipe_path.read_text()) + environments = [recipe["benchmark"]["env"]] + backend = recipe["backend"] + environments.extend( + value + for key, value in backend.items() + if key.endswith("_environment") + ) + for environment in environments: + assert environment["HF_HOME"] == "/hf_hub_cache" + assert environment["HF_HUB_CACHE"] == "/hf_hub_cache/hub" + assert environment["HUGGINGFACE_HUB_CACHE"] == "/hf_hub_cache/hub" + + def test_fixed_sequence_commands_execute_with_attached_arguments(tmp_path): fake_bin = tmp_path / "bin" fake_bin.mkdir() From e22bf36a306acc18545ea1475efa0493d063f278 Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Mon, 10 Aug 2026 05:14:08 -0500 Subject: [PATCH 41/67] fix(amd): pin staged hub cache path --- runners/launch_mi355x-amds-srt.sh | 2 +- utils/test_mi355x_srt_slurm_contract.py | 7 +++++-- 2 files changed, 6 insertions(+), 3 deletions(-) diff --git a/runners/launch_mi355x-amds-srt.sh b/runners/launch_mi355x-amds-srt.sh index 8873cf7a98..4e919af0ed 100755 --- a/runners/launch_mi355x-amds-srt.sh +++ b/runners/launch_mi355x-amds-srt.sh @@ -63,7 +63,7 @@ srun --nodes=1 --ntasks=1 \ --container-image="$SHARED_IMAGE" \ --container-mounts="$SHARED_HF_CACHE:/hf_hub_cache" \ --container-writable --container-remap-root --no-container-entrypoint \ - --export=ALL,HF_HOME=/hf_hub_cache,MODEL_REPO=${MODEL} \ + --export=ALL,HF_HOME=/hf_hub_cache,HF_HUB_CACHE=/hf_hub_cache/hub,HUGGINGFACE_HUB_CACHE=/hf_hub_cache/hub,MODEL_REPO=${MODEL} \ python3 -c 'import os; from huggingface_hub import snapshot_download; snapshot_download(os.environ["MODEL_REPO"])' EOF STAGE_JOB_ID=$(sbatch --wait --parsable "$STAGE_SCRIPT") diff --git a/utils/test_mi355x_srt_slurm_contract.py b/utils/test_mi355x_srt_slurm_contract.py index 33845e48dc..93ee27ca8f 100644 --- a/utils/test_mi355x_srt_slurm_contract.py +++ b/utils/test_mi355x_srt_slurm_contract.py @@ -132,8 +132,11 @@ def test_launcher_pins_runtime_and_preserves_legacy_default(): assert "scancel" not in launcher assert ': "${MODEL:?MODEL must identify the Hugging Face model}"' in launcher assert "MODEL_REPO=${MODEL}" in launcher - assert "HF_HOME=/hf_hub_cache,MODEL_REPO=${MODEL}" in launcher - assert "HF_HUB_CACHE=/hf_hub_cache,MODEL_REPO" not in launcher + assert ( + "HF_HOME=/hf_hub_cache,HF_HUB_CACHE=/hf_hub_cache/hub," + "HUGGINGFACE_HUB_CACHE=/hf_hub_cache/hub,MODEL_REPO=${MODEL}" + in launcher + ) assert 'snapshot_download(os.environ["MODEL_REPO"])' in launcher assert 'if [[ -n "${CONFIG_FILE:-}" ]]; then' in legacy assert "launch_mi355x-amds-srt.sh" in legacy From 147b2ec6196fa9d4eb6b8455badb3d57bf53f44a Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Mon, 10 Aug 2026 20:18:51 -0500 Subject: [PATCH 42/67] chore(amd): pin rebased srt-slurm runtime --- runners/launch_mi300x-amds-srt.sh | 2 +- runners/launch_mi355x-amds-srt.sh | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/runners/launch_mi300x-amds-srt.sh b/runners/launch_mi300x-amds-srt.sh index ba06c6c651..fb27e8a2b2 100644 --- a/runners/launch_mi300x-amds-srt.sh +++ b/runners/launch_mi300x-amds-srt.sh @@ -4,7 +4,7 @@ set -euo pipefail # MI300X srt-slurm validation path. The existing launcher remains the default; # matrix rows opt in by exporting CONFIG_FILE through additional-settings. SRT_SLURM_REPOSITORY="https://github.com/SemiAnalysisAI/srt-slurm.git" -SRT_SLURM_COMMIT="315e4b06a7e0806194a646ea21832e750e896a46" +SRT_SLURM_COMMIT="f94026bd08612ed9f7677fdf0a42f8d8391fd736" SLURM_PARTITION="compute" EXCLUDED_NODES="chi-mi300x-049,chi-mi300x-121" REMOTE_BASE="/raid/hf-hub-cache/inferencex/srt-slurm" diff --git a/runners/launch_mi355x-amds-srt.sh b/runners/launch_mi355x-amds-srt.sh index 4e919af0ed..d353b121d9 100755 --- a/runners/launch_mi355x-amds-srt.sh +++ b/runners/launch_mi355x-amds-srt.sh @@ -5,7 +5,7 @@ set -euo pipefail # in explicitly with CONFIG_FILE; all existing MI355X launch behavior remains # unchanged for every other row. SRT_SLURM_REPOSITORY="https://github.com/SemiAnalysisAI/srt-slurm.git" -SRT_SLURM_COMMIT="315e4b06a7e0806194a646ea21832e750e896a46" +SRT_SLURM_COMMIT="f94026bd08612ed9f7677fdf0a42f8d8391fd736" SLURM_PARTITION="compute" SGLANG_IMAGE="lmsysorg/sglang-rocm:v0.5.17-rocm720-mi35x-20260809" SHARED_BASE="/it-share/gharunners2/srt-slurm" From dd498164544dd774698a0c4d9cbbbd6555eeb9f5 Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Mon, 10 Aug 2026 20:26:32 -0500 Subject: [PATCH 43/67] fix(amd): pin native router infra correction --- runners/launch_mi300x-amds-srt.sh | 2 +- runners/launch_mi355x-amds-srt.sh | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/runners/launch_mi300x-amds-srt.sh b/runners/launch_mi300x-amds-srt.sh index fb27e8a2b2..bfe38b488f 100644 --- a/runners/launch_mi300x-amds-srt.sh +++ b/runners/launch_mi300x-amds-srt.sh @@ -4,7 +4,7 @@ set -euo pipefail # MI300X srt-slurm validation path. The existing launcher remains the default; # matrix rows opt in by exporting CONFIG_FILE through additional-settings. SRT_SLURM_REPOSITORY="https://github.com/SemiAnalysisAI/srt-slurm.git" -SRT_SLURM_COMMIT="f94026bd08612ed9f7677fdf0a42f8d8391fd736" +SRT_SLURM_COMMIT="dd0109d4043141072ad37c043f1100332008b77f" SLURM_PARTITION="compute" EXCLUDED_NODES="chi-mi300x-049,chi-mi300x-121" REMOTE_BASE="/raid/hf-hub-cache/inferencex/srt-slurm" diff --git a/runners/launch_mi355x-amds-srt.sh b/runners/launch_mi355x-amds-srt.sh index d353b121d9..26cbba4568 100755 --- a/runners/launch_mi355x-amds-srt.sh +++ b/runners/launch_mi355x-amds-srt.sh @@ -5,7 +5,7 @@ set -euo pipefail # in explicitly with CONFIG_FILE; all existing MI355X launch behavior remains # unchanged for every other row. SRT_SLURM_REPOSITORY="https://github.com/SemiAnalysisAI/srt-slurm.git" -SRT_SLURM_COMMIT="f94026bd08612ed9f7677fdf0a42f8d8391fd736" +SRT_SLURM_COMMIT="dd0109d4043141072ad37c043f1100332008b77f" SLURM_PARTITION="compute" SGLANG_IMAGE="lmsysorg/sglang-rocm:v0.5.17-rocm720-mi35x-20260809" SHARED_BASE="/it-share/gharunners2/srt-slurm" From 7b3ee00c6c6fc35842aac6163bae0a4ced621ac9 Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Mon, 10 Aug 2026 20:35:09 -0500 Subject: [PATCH 44/67] fix(amd): select the native SGLang Router frontend --- .../sglang/qwen3-0.6b/mi355x/agg-fixed-seq.yaml | 2 +- .../sglang/qwen3-0.6b/mi355x/disagg-1p1d-fixed-seq.yaml | 2 +- .../sglang/qwen3.5/mi355x/disagg-1p1d-tp8-fixed-seq.yaml | 2 +- runners/launch_mi355x-amds-srt.sh | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3-0.6b/mi355x/agg-fixed-seq.yaml b/benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3-0.6b/mi355x/agg-fixed-seq.yaml index fb508e7865..beb4777979 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3-0.6b/mi355x/agg-fixed-seq.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3-0.6b/mi355x/agg-fixed-seq.yaml @@ -28,7 +28,7 @@ resources: gpus_per_agg: 1 frontend: - type: sgl-router + type: sglang enable_multiple_frontends: false args: policy: round_robin diff --git a/benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3-0.6b/mi355x/disagg-1p1d-fixed-seq.yaml b/benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3-0.6b/mi355x/disagg-1p1d-fixed-seq.yaml index 6282e5dbea..b630a0ef65 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3-0.6b/mi355x/disagg-1p1d-fixed-seq.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3-0.6b/mi355x/disagg-1p1d-fixed-seq.yaml @@ -32,7 +32,7 @@ resources: gpus_per_decode: 1 frontend: - type: sgl-router + type: sglang enable_multiple_frontends: false args: policy: round_robin diff --git a/benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3.5/mi355x/disagg-1p1d-tp8-fixed-seq.yaml b/benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3.5/mi355x/disagg-1p1d-tp8-fixed-seq.yaml index a7f422fa35..7af3198531 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3.5/mi355x/disagg-1p1d-tp8-fixed-seq.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3.5/mi355x/disagg-1p1d-tp8-fixed-seq.yaml @@ -40,7 +40,7 @@ resources: gpus_per_decode: 8 frontend: - type: sgl-router + type: sglang enable_multiple_frontends: false args: policy: round_robin diff --git a/runners/launch_mi355x-amds-srt.sh b/runners/launch_mi355x-amds-srt.sh index 26cbba4568..a5e94c6a02 100755 --- a/runners/launch_mi355x-amds-srt.sh +++ b/runners/launch_mi355x-amds-srt.sh @@ -5,7 +5,7 @@ set -euo pipefail # in explicitly with CONFIG_FILE; all existing MI355X launch behavior remains # unchanged for every other row. SRT_SLURM_REPOSITORY="https://github.com/SemiAnalysisAI/srt-slurm.git" -SRT_SLURM_COMMIT="dd0109d4043141072ad37c043f1100332008b77f" +SRT_SLURM_COMMIT="31e72da43ed21fe941c039be51b2cad1a3cf428a" SLURM_PARTITION="compute" SGLANG_IMAGE="lmsysorg/sglang-rocm:v0.5.17-rocm720-mi35x-20260809" SHARED_BASE="/it-share/gharunners2/srt-slurm" From 9403f0b1969bcaf43e0c178b08e0d6a9f67dabee Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Mon, 10 Aug 2026 20:46:07 -0500 Subject: [PATCH 45/67] test(amd): enforce the native SGLang frontend contract --- utils/test_mi355x_srt_slurm_contract.py | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/utils/test_mi355x_srt_slurm_contract.py b/utils/test_mi355x_srt_slurm_contract.py index 93ee27ca8f..569fdd4f22 100644 --- a/utils/test_mi355x_srt_slurm_contract.py +++ b/utils/test_mi355x_srt_slurm_contract.py @@ -59,9 +59,9 @@ def test_recipes_use_native_sglang_router_and_only_disagg_uses_mori(): "agg_workers": 1, "gpus_per_agg": 1, } - assert agg["frontend"]["type"] == "sgl-router" + assert agg["frontend"]["type"] == "sglang" assert agg["backend"]["type"] == "sglang" - assert disagg["frontend"]["type"] == "sgl-router" + assert disagg["frontend"]["type"] == "sglang" assert disagg["backend"]["type"] == "sglang" for role in ("prefill", "decode"): assert disagg["backend"]["sglang_config"][role][ @@ -123,7 +123,7 @@ def test_launcher_pins_runtime_and_preserves_legacy_default(): launcher = LAUNCHER.read_text() legacy = LEGACY_LAUNCHER.read_text() - assert "315e4b06a7e0806194a646ea21832e750e896a46" in launcher + assert "31e72da43ed21fe941c039be51b2cad1a3cf428a" in launcher assert "v0.5.17-rocm720-mi35x-20260809" in launcher assert "make setup-compute ARCH=x86_64" in launcher assert "SRTCTL_RUNTIME_SOURCE_DIR" in launcher @@ -216,7 +216,7 @@ def test_production_disagg_recipe_uses_two_full_nodes_and_the_existing_workload( "cpus-per-task": "128", "mem": "0", } - assert recipe["frontend"]["type"] == "sgl-router" + assert recipe["frontend"]["type"] == "sglang" for role in ("prefill", "decode"): environment = recipe["backend"][f"{role}_environment"] assert environment["HF_HOME"] == "/hf_hub_cache" From 72ee996f7ad3ef365b863af818f3838e44d740cd Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Mon, 10 Aug 2026 20:50:48 -0500 Subject: [PATCH 46/67] test: drop redundant AMD integration contract file --- utils/test_mi355x_srt_slurm_contract.py | 235 ------------------------ 1 file changed, 235 deletions(-) delete mode 100644 utils/test_mi355x_srt_slurm_contract.py diff --git a/utils/test_mi355x_srt_slurm_contract.py b/utils/test_mi355x_srt_slurm_contract.py deleted file mode 100644 index 569fdd4f22..0000000000 --- a/utils/test_mi355x_srt_slurm_contract.py +++ /dev/null @@ -1,235 +0,0 @@ -"""High-signal contract checks for the MI355X srt-slurm bring-up lanes.""" - -import os -import subprocess -from pathlib import Path - -import yaml - - -REPO_ROOT = Path(__file__).resolve().parents[1] -RECIPE_ROOT = ( - REPO_ROOT - / "benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3-0.6b/mi355x" -) -AGG_RECIPE = RECIPE_ROOT / "agg-fixed-seq.yaml" -DISAGG_RECIPE = RECIPE_ROOT / "disagg-1p1d-fixed-seq.yaml" -PRODUCTION_DISAGG_RECIPE = ( - REPO_ROOT - / "benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3.5/mi355x/disagg-1p1d-tp8-fixed-seq.yaml" -) -CLUSTER_PROFILE = ( - REPO_ROOT - / "benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi355x-amds.yaml" -) -MASTER_CONFIG = REPO_ROOT / "configs/amd-master.yaml" -LAUNCHER = REPO_ROOT / "runners/launch_mi355x-amds-srt.sh" -LEGACY_LAUNCHER = REPO_ROOT / "runners/launch_mi355x-amds.sh" - - -def test_cluster_profile_matches_the_mi355x_rocm_slurm_contract(): - cluster = yaml.safe_load(CLUSTER_PROFILE.read_text()) - - assert cluster["accelerator_vendor"] == "amd" - assert cluster["network_interface"] == "eno0" - assert cluster["gpu_sbatch_directive"] == "gres" - assert cluster["use_segment_sbatch_directive"] is False - assert cluster["use_exclusive_sbatch_directive"] is False - assert cluster["runtime_config_transport"] == "shared-filesystem" - assert cluster["default_mounts"]["/dev/kfd"] == "/dev/kfd" - assert cluster["default_mounts"]["/dev/dri"] == "/dev/dri" - assert cluster["containers"]["sglang-rocm-v0.5.17-mi35x"].endswith( - "/sglang-rocm-v0.5.17-mi35x-20260809.sqsh" - ) - assert cluster["output_dir"].startswith("/it-share/gharunners2/srt-slurm/") - assert cluster["containers"]["sglang-rocm-v0.5.17-mi35x"].startswith( - "/it-share/gharunners2/srt-slurm/" - ) - - -def test_recipes_use_native_sglang_router_and_only_disagg_uses_mori(): - agg = yaml.safe_load(AGG_RECIPE.read_text()) - disagg = yaml.safe_load(DISAGG_RECIPE.read_text()) - production_disagg = yaml.safe_load(PRODUCTION_DISAGG_RECIPE.read_text()) - - assert agg["resources"] == { - "gpu_type": "mi355x", - "gpus_per_node": 1, - "agg_nodes": 1, - "agg_workers": 1, - "gpus_per_agg": 1, - } - assert agg["frontend"]["type"] == "sglang" - assert agg["backend"]["type"] == "sglang" - assert disagg["frontend"]["type"] == "sglang" - assert disagg["backend"]["type"] == "sglang" - for role in ("prefill", "decode"): - assert disagg["backend"]["sglang_config"][role][ - "disaggregation-transfer-backend" - ] == "mori" - assert "rdma0" in disagg["backend"][f"{role}_environment"]["IBDEVICES"] - - assert "mori" not in AGG_RECIPE.read_text().lower() - expected_image = "lmsysorg/sglang-rocm:v0.5.17-rocm720-mi35x-20260809" - expected_version = "0.5.17.dev20260809+g7120f3ee13" - for recipe_path, recipe in ( - (AGG_RECIPE, agg), - (DISAGG_RECIPE, disagg), - (PRODUCTION_DISAGG_RECIPE, production_disagg), - ): - text = recipe_path.read_text().lower() - assert "dynamo" not in text - assert "nixl" not in text - assert "nats" not in text - assert "etcd" not in text - assert recipe["model"]["container"] == "sglang-rocm-v0.5.17-mi35x" - assert recipe["identity"]["container"]["image"] == expected_image - assert recipe["identity"]["frameworks"]["sglang"] == expected_version - for recipe in (disagg, production_disagg): - assert recipe["identity"]["frameworks"]["amd-mori"] == expected_version - - -def test_matrix_rows_explicitly_select_the_srt_recipes(): - master = yaml.safe_load(MASTER_CONFIG.read_text()) - expected = { - "qwen3-0.6b-fp16-mi355x-sglang-srt-agg": ( - False, - "recipes/sglang/qwen3-0.6b/mi355x/agg-fixed-seq.yaml", - ), - "qwen3-0.6b-fp16-mi355x-sglang-srt-disagg": ( - True, - "recipes/sglang/qwen3-0.6b/mi355x/disagg-1p1d-fixed-seq.yaml", - ), - "qwen3.5-fp8-mi355x-sglang-srt-disagg": ( - True, - "recipes/sglang/qwen3.5/mi355x/disagg-1p1d-tp8-fixed-seq.yaml", - ), - } - for name, (is_disagg, recipe) in expected.items(): - config = master[name] - search = config["scenarios"]["fixed-seq-len"][0]["search-space"][0] - assert config["runner"] == "cluster:mi355x-amds" - assert config["image"] == ( - "lmsysorg/sglang-rocm:v0.5.17-rocm720-mi35x-20260809" - ) - assert config["multinode"] is True - assert config["disagg"] is is_disagg - assert search["prefill"]["additional-settings"] == [ - f"CONFIG_FILE={recipe}" - ] - - -def test_launcher_pins_runtime_and_preserves_legacy_default(): - launcher = LAUNCHER.read_text() - legacy = LEGACY_LAUNCHER.read_text() - - assert "31e72da43ed21fe941c039be51b2cad1a3cf428a" in launcher - assert "v0.5.17-rocm720-mi35x-20260809" in launcher - assert "make setup-compute ARCH=x86_64" in launcher - assert "SRTCTL_RUNTIME_SOURCE_DIR" in launcher - assert 'SHARED_BASE="/it-share/gharunners2/srt-slurm"' in launcher - assert "/it-share/inferencex" not in launcher - assert "scancel" not in launcher - assert ': "${MODEL:?MODEL must identify the Hugging Face model}"' in launcher - assert "MODEL_REPO=${MODEL}" in launcher - assert ( - "HF_HOME=/hf_hub_cache,HF_HUB_CACHE=/hf_hub_cache/hub," - "HUGGINGFACE_HUB_CACHE=/hf_hub_cache/hub,MODEL_REPO=${MODEL}" - in launcher - ) - assert 'snapshot_download(os.environ["MODEL_REPO"])' in launcher - assert 'if [[ -n "${CONFIG_FILE:-}" ]]; then' in legacy - assert "launch_mi355x-amds-srt.sh" in legacy - - -def test_all_mi355x_roles_use_the_staged_hugging_face_hub_cache(): - for recipe_path in (AGG_RECIPE, DISAGG_RECIPE, PRODUCTION_DISAGG_RECIPE): - recipe = yaml.safe_load(recipe_path.read_text()) - environments = [recipe["benchmark"]["env"]] - backend = recipe["backend"] - environments.extend( - value - for key, value in backend.items() - if key.endswith("_environment") - ) - for environment in environments: - assert environment["HF_HOME"] == "/hf_hub_cache" - assert environment["HF_HUB_CACHE"] == "/hf_hub_cache/hub" - assert environment["HUGGINGFACE_HUB_CACHE"] == "/hf_hub_cache/hub" - - -def test_fixed_sequence_commands_execute_with_attached_arguments(tmp_path): - fake_bin = tmp_path / "bin" - fake_bin.mkdir() - fake_python = fake_bin / "python3" - fake_python.write_text( - "#!/bin/bash\n" - 'printf "%s\\n" "$@" >> "$FAKE_ARGS_LOG"\n' - 'printf "%s\\n" --CALL-END-- >> "$FAKE_ARGS_LOG"\n' - ) - fake_python.chmod(0o755) - - for recipe_path in (AGG_RECIPE, DISAGG_RECIPE): - command = yaml.safe_load(recipe_path.read_text())["benchmark"]["command"] - result_dir = tmp_path / recipe_path.stem - command = command.replace( - 'result_root="/results/${SLURM_JOB_ID}"', - f'result_root="{result_dir}"', - ) - args_log = tmp_path / f"{recipe_path.stem}.args" - env = { - **os.environ, - "PATH": f"{fake_bin}:{os.environ['PATH']}", - "FAKE_ARGS_LOG": str(args_log), - "SRT_FRONTEND_HOST": "127.0.0.1", - "SRT_FRONTEND_PORT": "8000", - "SLURM_JOB_ID": "123", - } - - subprocess.run(["bash", "-n"], input=command, text=True, check=True) - subprocess.run(["bash", "-c", command], env=env, check=True) - calls = args_log.read_text().split("--CALL-END--\n") - calls = [[arg for arg in call.splitlines() if arg] for call in calls if call] - - assert len(calls) == 2 - assert [call[call.index("--num-prompts") + 1] for call in calls] == [ - "4", - "16", - ] - assert all("--model" in call and "Qwen/Qwen3-0.6B" in call for call in calls) - - -def test_production_disagg_recipe_uses_two_full_nodes_and_the_existing_workload(): - recipe = yaml.safe_load(PRODUCTION_DISAGG_RECIPE.read_text()) - - assert recipe["resources"] == { - "gpu_type": "mi355x", - "gpus_per_node": 8, - "prefill_nodes": 1, - "decode_nodes": 1, - "prefill_workers": 1, - "decode_workers": 1, - "gpus_per_prefill": 8, - "gpus_per_decode": 8, - } - assert recipe["sbatch_directives"] == { - "cpus-per-task": "128", - "mem": "0", - } - assert recipe["frontend"]["type"] == "sglang" - for role in ("prefill", "decode"): - environment = recipe["backend"][f"{role}_environment"] - assert environment["HF_HOME"] == "/hf_hub_cache" - assert environment["HF_HUB_CACHE"] == "/hf_hub_cache/hub" - assert environment["HUGGINGFACE_HUB_CACHE"] == "/hf_hub_cache/hub" - for role in ("prefill", "decode"): - config = recipe["backend"]["sglang_config"][role] - assert config["tensor-parallel-size"] == 8 - assert config["disaggregation-transfer-backend"] == "mori" - assert config["attention-backend"] == "aiter" - assert "json-model-override-args" not in config - command = recipe["benchmark"]["command"] - assert "utils/bench_serving/benchmark_serving.py" in command - assert "--random-input-len 8192" in command - assert "--random-output-len 1024" in command - assert "--max-concurrency \"${concurrency}\"" in command From 3105070b87ba3b0f406f0d5f953bd2d51b626c36 Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Tue, 11 Aug 2026 10:18:40 -0500 Subject: [PATCH 47/67] feat: add ATOM and Infera validation lanes --- .../qwen3-0.6b/mi300x/agg-2w-fixed-seq.yaml | 96 ++++++++++++++++ .../mi300x/disagg-1p1d-fixed-seq.yaml | 104 ++++++++++++++++++ .../cluster-configs/mi300x-amds.yaml | 1 + .../cluster-configs/mi355x-amds.yaml | 1 + configs/amd-master.yaml | 63 +++++++++++ perf-changelog.yaml | 9 ++ runners/launch_mi300x-amds-srt.sh | 29 ++++- runners/launch_mi355x-amds-srt.sh | 28 ++++- utils/test_mi300x_srt_slurm_contract.py | 45 +++++++- 9 files changed, 363 insertions(+), 13 deletions(-) create mode 100644 benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi300x/agg-2w-fixed-seq.yaml create mode 100644 benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml diff --git a/benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi300x/agg-2w-fixed-seq.yaml b/benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi300x/agg-2w-fixed-seq.yaml new file mode 100644 index 0000000000..9158a722a0 --- /dev/null +++ b/benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi300x/agg-2w-fixed-seq.yaml @@ -0,0 +1,96 @@ +# Small two-worker aggregate correctness lane for the srt-slurm ATOM backend +# and Infera's dynamic KV-aware router. This validates orchestration, not tuning. + +name: "mi300x-atom-qwen3-0.6b-agg-2w-fixed-seq" + +model: + path: "hf:Qwen/Qwen3-0.6B" + container: "infera-atom-v0.1.1" + precision: "fp16" + +identity: + model: + repo: "Qwen/Qwen3-0.6B" + container: + image: "rocm/infera:atom-v0.1.1" + frameworks: + atom: "0.1.1" + infera: "0.1.1" + +slurm: + time_limit: "00:45:00" + +resources: + gpu_type: "mi300x" + gpus_per_node: 8 + agg_nodes: 1 + agg_workers: 2 + gpus_per_agg: 1 + +frontend: + type: infera + enable_multiple_frontends: false + args: + router-policy: kv-aware + +backend: + type: atom + enable_kv_events: true + aggregated_environment: + HF_HOME: "/hf_hub_cache" + HF_HUB_CACHE: "/hf_hub_cache" + HUGGINGFACE_HUB_CACHE: "/hf_hub_cache" + PYTHONUNBUFFERED: "1" + OMP_NUM_THREADS: "1" + atom_config: + aggregated: + kv_cache_dtype: fp8 + gpu-memory-utilization: 0.50 + max-model-len: 2048 + max-num-seqs: 8 + block-size: 16 + enforce-eager: true + +srun_options: + container-writable: "" + container-remap-root: "" + mem: "0" + +health_check: + max_attempts: 240 + interval_seconds: 5 + +benchmark: + type: custom + command: | + set -euo pipefail + result_root="/results/${SLURM_JOB_ID}" + mkdir -p "${result_root}/fixed-seq" + trap 'tar -C /logs -czf "'"${result_root}"'/runtime-logs.tar.gz" . 2>/dev/null || true' EXIT + for concurrency in 1 4; do + python3 /infmax-workspace/utils/bench_serving/benchmark_serving.py \ + --backend openai \ + --base-url "http://${SRT_FRONTEND_HOST}:${SRT_FRONTEND_PORT}" \ + --endpoint /v1/completions \ + --model Qwen/Qwen3-0.6B \ + --tokenizer Qwen/Qwen3-0.6B \ + --dataset-name random \ + --random-input-len 128 \ + --random-output-len 32 \ + --random-prefix-len 96 \ + --random-range-ratio 1.0 \ + --random-num-workers 1 \ + --num-warmups "${concurrency}" \ + --num-prompts "$((concurrency * 4))" \ + --max-concurrency "${concurrency}" \ + --request-rate inf \ + --ignore-eos \ + --disable-tqdm \ + --save-result \ + --result-dir "${result_root}/fixed-seq" \ + --result-filename "qwen3-0.6b-atom-agg-isl128-osl32-c${concurrency}.json"; + done + env: + HF_HOME: /hf_hub_cache + HF_HUB_CACHE: /hf_hub_cache + HUGGINGFACE_HUB_CACHE: /hf_hub_cache diff --git a/benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml b/benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml new file mode 100644 index 0000000000..45e9864d0b --- /dev/null +++ b/benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml @@ -0,0 +1,104 @@ +# Small two-node ATOM P/D correctness lane. Infera discovers both workers, +# routes the completions API, and ATOM transfers KV through Mooncake RDMA. + +name: "mi300x-atom-qwen3-0.6b-disagg-1p1d-fixed-seq" + +model: + path: "hf:Qwen/Qwen3-0.6B" + container: "infera-atom-v0.1.1" + precision: "fp16" + +identity: + model: + repo: "Qwen/Qwen3-0.6B" + container: + image: "rocm/infera:atom-v0.1.1" + frameworks: + atom: "0.1.1" + infera: "0.1.1" + mooncake: "bundled" + +slurm: + time_limit: "00:45:00" + +resources: + gpu_type: "mi300x" + gpus_per_node: 8 + prefill_nodes: 1 + decode_nodes: 1 + prefill_workers: 1 + decode_workers: 1 + gpus_per_prefill: 1 + gpus_per_decode: 1 + +frontend: + type: infera + enable_multiple_frontends: false + args: + router-policy: kv-aware + +backend: + type: atom + connector: mooncake + enable_kv_events: true + prefill_environment: &worker_environment + HF_HOME: "/hf_hub_cache" + HF_HUB_CACHE: "/hf_hub_cache" + HUGGINGFACE_HUB_CACHE: "/hf_hub_cache" + PYTHONUNBUFFERED: "1" + OMP_NUM_THREADS: "1" + decode_environment: *worker_environment + atom_config: + prefill: &worker_config + kv_cache_dtype: fp8 + gpu-memory-utilization: 0.50 + max-model-len: 2048 + max-num-seqs: 8 + block-size: 16 + enforce-eager: true + no-enable_prefix_caching: true + decode: *worker_config + +srun_options: + container-writable: "" + container-remap-root: "" + mem: "0" + +health_check: + max_attempts: 240 + interval_seconds: 5 + +benchmark: + type: custom + command: | + set -euo pipefail + result_root="/results/${SLURM_JOB_ID}" + mkdir -p "${result_root}/fixed-seq" + trap 'tar -C /logs -czf "'"${result_root}"'/runtime-logs.tar.gz" . 2>/dev/null || true' EXIT + for concurrency in 1 4; do + python3 /infmax-workspace/utils/bench_serving/benchmark_serving.py \ + --backend openai \ + --base-url "http://${SRT_FRONTEND_HOST}:${SRT_FRONTEND_PORT}" \ + --endpoint /v1/completions \ + --model Qwen/Qwen3-0.6B \ + --tokenizer Qwen/Qwen3-0.6B \ + --dataset-name random \ + --random-input-len 128 \ + --random-output-len 32 \ + --random-prefix-len 96 \ + --random-range-ratio 1.0 \ + --random-num-workers 1 \ + --num-warmups "${concurrency}" \ + --num-prompts "$((concurrency * 4))" \ + --max-concurrency "${concurrency}" \ + --request-rate inf \ + --ignore-eos \ + --disable-tqdm \ + --save-result \ + --result-dir "${result_root}/fixed-seq" \ + --result-filename "qwen3-0.6b-atom-disagg-isl128-osl32-c${concurrency}.json"; + done + env: + HF_HOME: /hf_hub_cache + HF_HUB_CACHE: /hf_hub_cache + HUGGINGFACE_HUB_CACHE: /hf_hub_cache diff --git a/benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi300x-amds.yaml b/benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi300x-amds.yaml index ac3952a8fb..d661afc42a 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi300x-amds.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi300x-amds.yaml @@ -39,5 +39,6 @@ default_mounts: containers: vllm-rocm-v0.26.0: /raid/hf-hub-cache/inferencex/srt-slurm/containers/vllm-openai-rocm-v0.26.0.sqsh vllm-router-20260809: /raid/hf-hub-cache/inferencex/srt-slurm/containers/vllm-router-nightly-20260809-d2ba586.sqsh + infera-atom-v0.1.1: /raid/hf-hub-cache/inferencex/srt-slurm/containers/infera-atom-v0.1.1.sqsh nginx_raise_ulimit: false diff --git a/benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi355x-amds.yaml b/benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi355x-amds.yaml index 078ef196c9..3b4cd881c1 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi355x-amds.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi355x-amds.yaml @@ -28,5 +28,6 @@ default_mounts: containers: sglang-rocm-v0.5.17-mi35x: /it-share/gharunners2/srt-slurm/containers/sglang-rocm-v0.5.17-mi35x-20260809.sqsh + infera-atom-v0.1.1: /it-share/gharunners2/srt-slurm/containers/infera-atom-v0.1.1.sqsh nginx_raise_ulimit: false diff --git a/configs/amd-master.yaml b/configs/amd-master.yaml index 39b9e2d5d5..9ffc1b4cae 100644 --- a/configs/amd-master.yaml +++ b/configs/amd-master.yaml @@ -1563,6 +1563,69 @@ qwen3-0.6b-fp16-mi300x-vllm-srt-disagg: ep: 1 dp-attn: false +# Small-model correctness lanes for the native ATOM backend and the current +# open-source Infera router (the successor to ATOMesh). The aggregate lane +# runs two independent ATOM workers so discovery and routing are exercised; +# the P/D lane validates Mooncake transfer across two MI300X nodes. +qwen3-0.6b-fp16-mi300x-atom-infera-srt-agg: + image: rocm/infera:atom-v0.1.1 + model: Qwen/Qwen3-0.6B + model-prefix: qwen3-0.6b + runner: mi300x-disagg + precision: fp16 + framework: atom + router: { name: infera, version: "0.1.1" } + multinode: true + disagg: false + scenarios: + fixed-seq-len: + - isl: 128 + osl: 32 + search-space: + - conc-list: [1] + prefill: + num-worker: 2 + tp: 1 + ep: 1 + dp-attn: false + additional-settings: + - "CONFIG_FILE=recipes/atom/qwen3-0.6b/mi300x/agg-2w-fixed-seq.yaml" + decode: + num-worker: 0 + tp: 1 + ep: 1 + dp-attn: false + +qwen3-0.6b-fp16-mi300x-atom-infera-srt-disagg: + image: rocm/infera:atom-v0.1.1 + model: Qwen/Qwen3-0.6B + model-prefix: qwen3-0.6b + runner: mi300x-disagg + precision: fp16 + framework: atom-disagg + router: { name: infera, version: "0.1.1" } + kv-p2p-transfer: mooncake + multinode: true + disagg: true + scenarios: + fixed-seq-len: + - isl: 128 + osl: 32 + search-space: + - conc-list: [1] + prefill: + num-worker: 1 + tp: 1 + ep: 1 + dp-attn: false + additional-settings: + - "CONFIG_FILE=recipes/atom/qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml" + decode: + num-worker: 1 + tp: 1 + ep: 1 + dp-attn: false + # Small-model correctness lanes for native SGLang Router orchestration on # MI355X. Both use the unchanged InferenceX fixed-sequence benchmark; the P/D # lane uses AMD MoRI and neither lane starts Dynamo, NATS, etcd, or NIXL. diff --git a/perf-changelog.yaml b/perf-changelog.yaml index efd8c51ddd..3d737d5e3a 100644 --- a/perf-changelog.yaml +++ b/perf-changelog.yaml @@ -5788,3 +5788,12 @@ - "models.yaml: add mtp_flags (--speculative-algorithm EAGLE --speculative-eagle-topk 1) to DeepSeek-V4-Pro-AgentX; num-steps/num-draft-tokens are derived from DECODE_MTP_SIZE in server_sglang.sh. DeepSeek-V4's MTP head is a native NextN/EAGLE draft shipped with the model, so no --speculative-draft-model-path is needed." - "Two search-space arms: TP8/EP1 no-DP at conc-list [2,4,8,16,32], and TP8/EP8/DPA (ep=8, dp-attn=true) at conc-list [64,96,128], both on image lmsysorg/sglang-rocm:v0.5.15.post1-rocm720-mi35x-20260719." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2309 + +- config-keys: + - qwen3-0.6b-fp16-mi300x-atom-infera-srt-agg + - qwen3-0.6b-fp16-mi300x-atom-infera-srt-disagg + description: + - "Add srt-slurm correctness lanes for the ROCm ATOM engine and the open-source Infera successor to ATOMesh." + - "Validate two aggregate ATOM workers behind Infera KV-aware routing and a two-node 1P/1D ATOM deployment using Mooncake RDMA." + - "Use rocm/infera:atom-v0.1.1 with Qwen3-0.6B fixed-sequence requests on MI300X and pin the combined AMD, vLLM Router, ATOM, and Infera srt-slurm branch." + pr-link: TBD diff --git a/runners/launch_mi300x-amds-srt.sh b/runners/launch_mi300x-amds-srt.sh index bfe38b488f..f9913e8af5 100644 --- a/runners/launch_mi300x-amds-srt.sh +++ b/runners/launch_mi300x-amds-srt.sh @@ -4,20 +4,41 @@ set -euo pipefail # MI300X srt-slurm validation path. The existing launcher remains the default; # matrix rows opt in by exporting CONFIG_FILE through additional-settings. SRT_SLURM_REPOSITORY="https://github.com/SemiAnalysisAI/srt-slurm.git" -SRT_SLURM_COMMIT="dd0109d4043141072ad37c043f1100332008b77f" +SRT_SLURM_COMMIT="d93b48165ff60c6441feb5dd04504337f0bd7bc5" SLURM_PARTITION="compute" EXCLUDED_NODES="chi-mi300x-049,chi-mi300x-121" REMOTE_BASE="/raid/hf-hub-cache/inferencex/srt-slurm" VLLM_IMAGE="vllm/vllm-openai-rocm:v0.26.0" VLLM_ROUTER_IMAGE="vllm/vllm-router:nightly-20260809-d2ba586" +ATOM_IMAGE="rocm/infera:atom-v0.1.1" VLLM_SQSH="${REMOTE_BASE}/containers/vllm-openai-rocm-v0.26.0.sqsh" VLLM_ROUTER_SQSH="${REMOTE_BASE}/containers/vllm-router-nightly-20260809-d2ba586.sqsh" +ATOM_SQSH="${REMOTE_BASE}/containers/infera-atom-v0.1.1.sqsh" : "${GITHUB_WORKSPACE:?GITHUB_WORKSPACE must be set by Actions}" : "${RESULT_FILENAME:?RESULT_FILENAME must be set by the benchmark workflow}" : "${CONFIG_FILE:?CONFIG_FILE must name an srt-slurm recipe}" +case "${IMAGE:?IMAGE must identify the recipe container}" in + "$VLLM_IMAGE") + ENGINE_IMAGE="$VLLM_IMAGE" + ENGINE_SQSH="$VLLM_SQSH" + AUX_IMAGE="$VLLM_ROUTER_IMAGE" + AUX_SQSH="$VLLM_ROUTER_SQSH" + ;; + "$ATOM_IMAGE") + ENGINE_IMAGE="$ATOM_IMAGE" + ENGINE_SQSH="$ATOM_SQSH" + AUX_IMAGE="" + AUX_SQSH="" + ;; + *) + echo "Unsupported MI300X srt-slurm image: $IMAGE" >&2 + exit 1 + ;; +esac + CONFIG_PATH="${CONFIG_FILE%%:*}" LOCAL_RECIPE="${GITHUB_WORKSPACE}/benchmarks/multi_node/srt-slurm-recipes/${CONFIG_PATH#recipes/}" CLUSTER_PROFILE="${GITHUB_WORKSPACE}/benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi300x-amds.yaml" @@ -79,8 +100,10 @@ srun --ntasks-per-node=1 bash -c ' flock -u "\$lock_fd" exec {lock_fd}>&- } - ensure_container_image "${VLLM_SQSH}" "${VLLM_IMAGE}" - ensure_container_image "${VLLM_ROUTER_SQSH}" "${VLLM_ROUTER_IMAGE}" + ensure_container_image "${ENGINE_SQSH}" "${ENGINE_IMAGE}" + if [[ -n "${AUX_IMAGE}" ]]; then + ensure_container_image "${AUX_SQSH}" "${AUX_IMAGE}" + fi if [[ ! -d "\$srt_runtime/.git" ]]; then git clone --quiet "${SRT_SLURM_REPOSITORY}" "\$srt_runtime" fi diff --git a/runners/launch_mi355x-amds-srt.sh b/runners/launch_mi355x-amds-srt.sh index a5e94c6a02..6d12d6d65d 100755 --- a/runners/launch_mi355x-amds-srt.sh +++ b/runners/launch_mi355x-amds-srt.sh @@ -5,11 +5,11 @@ set -euo pipefail # in explicitly with CONFIG_FILE; all existing MI355X launch behavior remains # unchanged for every other row. SRT_SLURM_REPOSITORY="https://github.com/SemiAnalysisAI/srt-slurm.git" -SRT_SLURM_COMMIT="31e72da43ed21fe941c039be51b2cad1a3cf428a" +SRT_SLURM_COMMIT="d93b48165ff60c6441feb5dd04504337f0bd7bc5" SLURM_PARTITION="compute" SGLANG_IMAGE="lmsysorg/sglang-rocm:v0.5.17-rocm720-mi35x-20260809" +ATOM_IMAGE="rocm/infera:atom-v0.1.1" SHARED_BASE="/it-share/gharunners2/srt-slurm" -SHARED_IMAGE="${SHARED_BASE}/containers/sglang-rocm-v0.5.17-mi35x-20260809.sqsh" SHARED_HF_CACHE="/it-share/hf-hub-cache" SHARED_RESULTS="${SHARED_BASE}/results" @@ -18,6 +18,23 @@ SHARED_RESULTS="${SHARED_BASE}/results" : "${CONFIG_FILE:?CONFIG_FILE must name an srt-slurm recipe}" : "${MODEL:?MODEL must identify the Hugging Face model}" +case "${IMAGE:?IMAGE must identify the recipe container}" in + "$SGLANG_IMAGE") + RUNTIME_IMAGE="$SGLANG_IMAGE" + SHARED_IMAGE="${SHARED_BASE}/containers/sglang-rocm-v0.5.17-mi35x-20260809.sqsh" + LOCAL_FALLBACK_IMAGE="/var/lib/squash/lmsysorg_sglang-rocm_v0.5.17-rocm720-mi35x-20260809.sqsh" + ;; + "$ATOM_IMAGE") + RUNTIME_IMAGE="$ATOM_IMAGE" + SHARED_IMAGE="${SHARED_BASE}/containers/infera-atom-v0.1.1.sqsh" + LOCAL_FALLBACK_IMAGE="" + ;; + *) + echo "Unsupported MI355X srt-slurm image: $IMAGE" >&2 + exit 1 + ;; +esac + CONFIG_PATH="${CONFIG_FILE%%:*}" LOCAL_RECIPE="${GITHUB_WORKSPACE}/benchmarks/multi_node/srt-slurm-recipes/${CONFIG_PATH#recipes/}" CLUSTER_PROFILE="${GITHUB_WORKSPACE}/benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi355x-amds.yaml" @@ -49,11 +66,10 @@ flock -w 2400 9 if ! unsquashfs -s "$SHARED_IMAGE" >/dev/null 2>&1; then tmp="${SHARED_IMAGE}.tmp.\${SLURM_JOB_ID}" rm -f "\$tmp" - local_image="/var/lib/squash/lmsysorg_sglang-rocm_v0.5.17-rocm720-mi35x-20260809.sqsh" - if unsquashfs -s "\$local_image" >/dev/null 2>&1; then - cp --sparse=always "\$local_image" "\$tmp" + if [[ -n "${LOCAL_FALLBACK_IMAGE}" ]] && unsquashfs -s "${LOCAL_FALLBACK_IMAGE}" >/dev/null 2>&1; then + cp --sparse=always "${LOCAL_FALLBACK_IMAGE}" "\$tmp" else - enroot import -o "\$tmp" "docker://${SGLANG_IMAGE}" + enroot import -o "\$tmp" "docker://${RUNTIME_IMAGE}" fi unsquashfs -s "\$tmp" >/dev/null mv "\$tmp" "$SHARED_IMAGE" diff --git a/utils/test_mi300x_srt_slurm_contract.py b/utils/test_mi300x_srt_slurm_contract.py index d558d7afb8..9da9396fa5 100644 --- a/utils/test_mi300x_srt_slurm_contract.py +++ b/utils/test_mi300x_srt_slurm_contract.py @@ -13,6 +13,11 @@ / "benchmarks/multi_node/srt-slurm-recipes/vllm/qwen3-0.6b/mi300x/agg-fixed-seq.yaml" ) DISAGG_RECIPE_PATH = RECIPE_PATH.with_name("disagg-1p1d-fixed-seq.yaml") +ATOM_RECIPE_PATH = ( + REPO_ROOT + / "benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi300x/agg-2w-fixed-seq.yaml" +) +ATOM_DISAGG_RECIPE_PATH = ATOM_RECIPE_PATH.with_name("disagg-1p1d-fixed-seq.yaml") CLUSTER_PATH = ( REPO_ROOT / "benchmarks/multi_node/srt-slurm-recipes/cluster-configs/mi300x-amds.yaml" @@ -92,7 +97,7 @@ def test_official_matrix_routes_disagg_through_the_pinned_srt_launcher(): "CONFIG_FILE=recipes/vllm/qwen3-0.6b/mi300x/" "disagg-1p1d-fixed-seq.yaml" ] - assert "315e4b06a7e0806194a646ea21832e750e896a46" in launcher + assert "d93b48165ff60c6441feb5dd04504337f0bd7bc5" in launcher assert "make setup-compute ARCH=x86_64" in launcher assert "--no-preflight" in launcher assert 'VLLM_IMAGE="vllm/vllm-openai-rocm:v0.26.0"' in launcher @@ -187,6 +192,32 @@ def test_fixed_sequence_recipe_uses_inferencex_custom_benchmark(): assert "sa-bench" not in command +def test_atom_recipes_use_infera_and_keep_worker_metrics_honest(): + cluster = yaml.safe_load(CLUSTER_PATH.read_text()) + aggregate = yaml.safe_load(ATOM_RECIPE_PATH.read_text()) + disaggregate = yaml.safe_load(ATOM_DISAGG_RECIPE_PATH.read_text()) + launcher = SRT_LAUNCHER_PATH.read_text() + + assert cluster["containers"]["infera-atom-v0.1.1"].endswith( + "/infera-atom-v0.1.1.sqsh" + ) + assert aggregate["resources"]["agg_workers"] == 2 + for recipe in (aggregate, disaggregate): + assert recipe["model"]["container"] == "infera-atom-v0.1.1" + assert recipe["identity"]["container"]["image"] == ( + "rocm/infera:atom-v0.1.1" + ) + assert recipe["frontend"]["type"] == "infera" + assert recipe["frontend"]["args"]["router-policy"] == "kv-aware" + assert recipe["backend"]["type"] == "atom" + assert recipe["backend"]["enable_kv_events"] is True + command = recipe["benchmark"]["command"] + assert "--backend openai" in command + assert "--endpoint /v1/completions" in command + assert disaggregate["backend"]["connector"] == "mooncake" + assert 'ATOM_IMAGE="rocm/infera:atom-v0.1.1"' in launcher + + def test_fixed_sequence_commands_keep_all_arguments_attached(tmp_path): fake_bin = tmp_path / "bin" fake_bin.mkdir() @@ -198,14 +229,20 @@ def test_fixed_sequence_commands_keep_all_arguments_attached(tmp_path): ) fake_python.chmod(0o755) - for recipe_path in (RECIPE_PATH, DISAGG_RECIPE_PATH): + recipe_paths = ( + RECIPE_PATH, + DISAGG_RECIPE_PATH, + ATOM_RECIPE_PATH, + ATOM_DISAGG_RECIPE_PATH, + ) + for index, recipe_path in enumerate(recipe_paths): command = yaml.safe_load(recipe_path.read_text())["benchmark"]["command"] - result_dir = tmp_path / recipe_path.stem + result_dir = tmp_path / f"recipe-{index}" command = command.replace( 'result_root="/results/${SLURM_JOB_ID}"', f'result_root="{result_dir}"', ) - args_log = tmp_path / f"{recipe_path.stem}.args" + args_log = tmp_path / f"recipe-{index}.args" env = { **os.environ, "PATH": f"{fake_bin}:{os.environ['PATH']}", From 226a87504c39cbee655ead599225de9735dc9061 Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Tue, 11 Aug 2026 10:19:27 -0500 Subject: [PATCH 48/67] docs: link ATOM validation pull request --- perf-changelog.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/perf-changelog.yaml b/perf-changelog.yaml index 3d737d5e3a..cdb3b30b85 100644 --- a/perf-changelog.yaml +++ b/perf-changelog.yaml @@ -5796,4 +5796,4 @@ - "Add srt-slurm correctness lanes for the ROCm ATOM engine and the open-source Infera successor to ATOMesh." - "Validate two aggregate ATOM workers behind Infera KV-aware routing and a two-node 1P/1D ATOM deployment using Mooncake RDMA." - "Use rocm/infera:atom-v0.1.1 with Qwen3-0.6B fixed-sequence requests on MI300X and pin the combined AMD, vLLM Router, ATOM, and Infera srt-slurm branch." - pr-link: TBD + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2558 From dd597d5eb1ca709d0ae3674f2ba585fc7dfe06b5 Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Tue, 11 Aug 2026 10:33:48 -0500 Subject: [PATCH 49/67] fix: use writable Enroot staging runtime --- runners/launch_mi300x-amds-srt.sh | 4 +++- runners/launch_mi355x-amds-srt.sh | 4 +++- utils/test_mi300x_srt_slurm_contract.py | 1 + 3 files changed, 7 insertions(+), 2 deletions(-) diff --git a/runners/launch_mi300x-amds-srt.sh b/runners/launch_mi300x-amds-srt.sh index f9913e8af5..fa8442da54 100644 --- a/runners/launch_mi300x-amds-srt.sh +++ b/runners/launch_mi300x-amds-srt.sh @@ -79,7 +79,9 @@ srun --ntasks-per-node=1 bash -c ' set -euo pipefail runtime="${REMOTE_RUNTIME}" srt_runtime="${REMOTE_SRT_RUNTIME}" - mkdir -p "\$runtime" "${REMOTE_RESULTS}" "${REMOTE_BASE}/containers" + export ENROOT_RUNTIME_PATH="\${TMPDIR:-/tmp}/enroot-runtime-\${UID}" + mkdir -p "\$ENROOT_RUNTIME_PATH" "\$runtime" "${REMOTE_RESULTS}" "${REMOTE_BASE}/containers" + chmod 700 "\$ENROOT_RUNTIME_PATH" ensure_container_image() { local target="\$1" local image="\$2" diff --git a/runners/launch_mi355x-amds-srt.sh b/runners/launch_mi355x-amds-srt.sh index 6d12d6d65d..bf46fdf2fb 100755 --- a/runners/launch_mi355x-amds-srt.sh +++ b/runners/launch_mi355x-amds-srt.sh @@ -60,7 +60,9 @@ cat > "$STAGE_SCRIPT" <"${SHARED_IMAGE}.lock" flock -w 2400 9 if ! unsquashfs -s "$SHARED_IMAGE" >/dev/null 2>&1; then diff --git a/utils/test_mi300x_srt_slurm_contract.py b/utils/test_mi300x_srt_slurm_contract.py index 9da9396fa5..fa8b167230 100644 --- a/utils/test_mi300x_srt_slurm_contract.py +++ b/utils/test_mi300x_srt_slurm_contract.py @@ -100,6 +100,7 @@ def test_official_matrix_routes_disagg_through_the_pinned_srt_launcher(): assert "d93b48165ff60c6441feb5dd04504337f0bd7bc5" in launcher assert "make setup-compute ARCH=x86_64" in launcher assert "--no-preflight" in launcher + assert 'ENROOT_RUNTIME_PATH="\\${TMPDIR:-/tmp}/enroot-runtime-\\${UID}"' in launcher assert 'VLLM_IMAGE="vllm/vllm-openai-rocm:v0.26.0"' in launcher assert ( 'VLLM_ROUTER_IMAGE="vllm/vllm-router:nightly-20260809-d2ba586"' From 98ac36d4658d19343943402a36d6c64bb4828f1e Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Tue, 11 Aug 2026 10:44:51 -0500 Subject: [PATCH 50/67] fix: retry transient container imports --- runners/launch_mi300x-amds-srt.sh | 14 ++++++++++++-- runners/launch_mi355x-amds-srt.sh | 13 ++++++++++++- utils/test_mi300x_srt_slurm_contract.py | 1 + 3 files changed, 25 insertions(+), 3 deletions(-) diff --git a/runners/launch_mi300x-amds-srt.sh b/runners/launch_mi300x-amds-srt.sh index fa8442da54..016cd7c4c1 100644 --- a/runners/launch_mi300x-amds-srt.sh +++ b/runners/launch_mi300x-amds-srt.sh @@ -94,8 +94,18 @@ srun --ntasks-per-node=1 bash -c ' flock -w 2400 "\$lock_fd" if ! unsquashfs -s "\$target" >/dev/null 2>&1; then tmp="\${target}.tmp.\${SLURM_JOB_ID}" - rm -f "\$tmp" - enroot import -o "\$tmp" "docker://\${image}" + for attempt in 1 2 3; do + rm -f "\$tmp" + if enroot import -o "\$tmp" "docker://\${image}"; then + break + fi + if [[ "\$attempt" -eq 3 ]]; then + echo "Failed to import \${image} after \${attempt} attempts" >&2 + return 1 + fi + echo "Retrying \${image} import after attempt \${attempt}" >&2 + sleep "\$((attempt * 10))" + done unsquashfs -s "\$tmp" >/dev/null mv "\$tmp" "\$target" fi diff --git a/runners/launch_mi355x-amds-srt.sh b/runners/launch_mi355x-amds-srt.sh index bf46fdf2fb..97ce2be163 100755 --- a/runners/launch_mi355x-amds-srt.sh +++ b/runners/launch_mi355x-amds-srt.sh @@ -71,7 +71,18 @@ if ! unsquashfs -s "$SHARED_IMAGE" >/dev/null 2>&1; then if [[ -n "${LOCAL_FALLBACK_IMAGE}" ]] && unsquashfs -s "${LOCAL_FALLBACK_IMAGE}" >/dev/null 2>&1; then cp --sparse=always "${LOCAL_FALLBACK_IMAGE}" "\$tmp" else - enroot import -o "\$tmp" "docker://${RUNTIME_IMAGE}" + for attempt in 1 2 3; do + rm -f "\$tmp" + if enroot import -o "\$tmp" "docker://${RUNTIME_IMAGE}"; then + break + fi + if [[ "\$attempt" -eq 3 ]]; then + echo "Failed to import ${RUNTIME_IMAGE} after \${attempt} attempts" >&2 + exit 1 + fi + echo "Retrying ${RUNTIME_IMAGE} import after attempt \${attempt}" >&2 + sleep "\$((attempt * 10))" + done fi unsquashfs -s "\$tmp" >/dev/null mv "\$tmp" "$SHARED_IMAGE" diff --git a/utils/test_mi300x_srt_slurm_contract.py b/utils/test_mi300x_srt_slurm_contract.py index fa8b167230..357ee3e5de 100644 --- a/utils/test_mi300x_srt_slurm_contract.py +++ b/utils/test_mi300x_srt_slurm_contract.py @@ -101,6 +101,7 @@ def test_official_matrix_routes_disagg_through_the_pinned_srt_launcher(): assert "make setup-compute ARCH=x86_64" in launcher assert "--no-preflight" in launcher assert 'ENROOT_RUNTIME_PATH="\\${TMPDIR:-/tmp}/enroot-runtime-\\${UID}"' in launcher + assert "for attempt in 1 2 3" in launcher assert 'VLLM_IMAGE="vllm/vllm-openai-rocm:v0.26.0"' in launcher assert ( 'VLLM_ROUTER_IMAGE="vllm/vllm-router:nightly-20260809-d2ba586"' From c16663b0c530c843784957a9fffa72a8800aa28d Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Tue, 11 Aug 2026 10:53:18 -0500 Subject: [PATCH 51/67] fix: install orchestration binaries on MI300X --- runners/launch_mi300x-amds-srt.sh | 2 +- utils/test_mi300x_srt_slurm_contract.py | 3 ++- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/runners/launch_mi300x-amds-srt.sh b/runners/launch_mi300x-amds-srt.sh index 016cd7c4c1..cbe33de6d8 100644 --- a/runners/launch_mi300x-amds-srt.sh +++ b/runners/launch_mi300x-amds-srt.sh @@ -163,7 +163,7 @@ export PATH="$HOME/.local/bin:$PATH" cd "$SRT_REPO_DIR" uv venv --python 3.12 uv pip install -e . -make setup-compute ARCH=x86_64 +make setup ARCH=x86_64 source .venv/bin/activate export SRTSLURM_CONFIG="${WORK_DIR}/srtslurm.yaml" export SRTCTL_RUNTIME_SOURCE_DIR="$REMOTE_SRT_RUNTIME" diff --git a/utils/test_mi300x_srt_slurm_contract.py b/utils/test_mi300x_srt_slurm_contract.py index 357ee3e5de..4e1bb5aff6 100644 --- a/utils/test_mi300x_srt_slurm_contract.py +++ b/utils/test_mi300x_srt_slurm_contract.py @@ -98,7 +98,8 @@ def test_official_matrix_routes_disagg_through_the_pinned_srt_launcher(): "disagg-1p1d-fixed-seq.yaml" ] assert "d93b48165ff60c6441feb5dd04504337f0bd7bc5" in launcher - assert "make setup-compute ARCH=x86_64" in launcher + assert "setup-compute ARCH=x86_64" in launcher + assert "\nmake setup ARCH=x86_64\n" in launcher assert "--no-preflight" in launcher assert 'ENROOT_RUNTIME_PATH="\\${TMPDIR:-/tmp}/enroot-runtime-\\${UID}"' in launcher assert "for attempt in 1 2 3" in launcher From 73d5e8e190ba91cea15b40d88d36347c2c4d969b Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Tue, 11 Aug 2026 10:57:31 -0500 Subject: [PATCH 52/67] fix: stage srt-slurm infrastructure binaries --- runners/launch_mi300x-amds-srt.sh | 2 +- utils/test_mi300x_srt_slurm_contract.py | 5 ++--- 2 files changed, 3 insertions(+), 4 deletions(-) diff --git a/runners/launch_mi300x-amds-srt.sh b/runners/launch_mi300x-amds-srt.sh index cbe33de6d8..51c3f74a3f 100644 --- a/runners/launch_mi300x-amds-srt.sh +++ b/runners/launch_mi300x-amds-srt.sh @@ -122,7 +122,7 @@ srun --ntasks-per-node=1 bash -c ' git -C "\$srt_runtime" fetch --quiet origin "${SRT_SLURM_COMMIT}" git -C "\$srt_runtime" checkout --quiet --detach "${SRT_SLURM_COMMIT}" test "\$(git -C "\$srt_runtime" rev-parse HEAD)" = "${SRT_SLURM_COMMIT}" - make -C "\$srt_runtime" --no-print-directory setup-compute ARCH=x86_64 + make -C "\$srt_runtime" --no-print-directory setup ARCH=x86_64 tar -xzf "/tmp/inferencex-benchmark-\${SLURM_JOB_ID}.tar.gz" -C "\$runtime" printf "%s\\n" "${GITHUB_SHA:-unknown}" > "\$runtime/.inferencex-source-head" ' diff --git a/utils/test_mi300x_srt_slurm_contract.py b/utils/test_mi300x_srt_slurm_contract.py index 4e1bb5aff6..94512086b6 100644 --- a/utils/test_mi300x_srt_slurm_contract.py +++ b/utils/test_mi300x_srt_slurm_contract.py @@ -98,8 +98,7 @@ def test_official_matrix_routes_disagg_through_the_pinned_srt_launcher(): "disagg-1p1d-fixed-seq.yaml" ] assert "d93b48165ff60c6441feb5dd04504337f0bd7bc5" in launcher - assert "setup-compute ARCH=x86_64" in launcher - assert "\nmake setup ARCH=x86_64\n" in launcher + assert launcher.count("setup ARCH=x86_64") == 2 assert "--no-preflight" in launcher assert 'ENROOT_RUNTIME_PATH="\\${TMPDIR:-/tmp}/enroot-runtime-\\${UID}"' in launcher assert "for attempt in 1 2 3" in launcher @@ -115,7 +114,7 @@ def test_official_matrix_routes_disagg_through_the_pinned_srt_launcher(): assert 'mv "\\$tmp" "\\$target"' in launcher assert 'REMOTE_SRT_RUNTIME="${REMOTE_BASE}/runtime/srt-slurm-${SRT_SLURM_COMMIT}"' in launcher assert 'git -C "\\$srt_runtime" checkout --quiet --detach "${SRT_SLURM_COMMIT}"' in launcher - assert 'make -C "\\$srt_runtime" --no-print-directory setup-compute ARCH=x86_64' in launcher + assert 'make -C "\\$srt_runtime" --no-print-directory setup ARCH=x86_64' in launcher assert 'export SRTCTL_RUNTIME_SOURCE_DIR="$REMOTE_SRT_RUNTIME"' in launcher assert "scancel" not in launcher From e27c76fc2f57b333f73cec06723040362f26ce31 Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Tue, 11 Aug 2026 11:10:43 -0500 Subject: [PATCH 53/67] fix(srt): pin Infera worker-registry readiness --- runners/launch_mi300x-amds-srt.sh | 2 +- runners/launch_mi355x-amds-srt.sh | 4 ++-- utils/test_mi300x_srt_slurm_contract.py | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/runners/launch_mi300x-amds-srt.sh b/runners/launch_mi300x-amds-srt.sh index 51c3f74a3f..39a2928fd7 100644 --- a/runners/launch_mi300x-amds-srt.sh +++ b/runners/launch_mi300x-amds-srt.sh @@ -4,7 +4,7 @@ set -euo pipefail # MI300X srt-slurm validation path. The existing launcher remains the default; # matrix rows opt in by exporting CONFIG_FILE through additional-settings. SRT_SLURM_REPOSITORY="https://github.com/SemiAnalysisAI/srt-slurm.git" -SRT_SLURM_COMMIT="d93b48165ff60c6441feb5dd04504337f0bd7bc5" +SRT_SLURM_COMMIT="297da661ad058bb1ea4bad06be528ce4a0bbe9e2" SLURM_PARTITION="compute" EXCLUDED_NODES="chi-mi300x-049,chi-mi300x-121" REMOTE_BASE="/raid/hf-hub-cache/inferencex/srt-slurm" diff --git a/runners/launch_mi355x-amds-srt.sh b/runners/launch_mi355x-amds-srt.sh index 97ce2be163..941b983f34 100755 --- a/runners/launch_mi355x-amds-srt.sh +++ b/runners/launch_mi355x-amds-srt.sh @@ -5,7 +5,7 @@ set -euo pipefail # in explicitly with CONFIG_FILE; all existing MI355X launch behavior remains # unchanged for every other row. SRT_SLURM_REPOSITORY="https://github.com/SemiAnalysisAI/srt-slurm.git" -SRT_SLURM_COMMIT="d93b48165ff60c6441feb5dd04504337f0bd7bc5" +SRT_SLURM_COMMIT="297da661ad058bb1ea4bad06be528ce4a0bbe9e2" SLURM_PARTITION="compute" SGLANG_IMAGE="lmsysorg/sglang-rocm:v0.5.17-rocm720-mi35x-20260809" ATOM_IMAGE="rocm/infera:atom-v0.1.1" @@ -132,7 +132,7 @@ export PATH="$HOME/.local/bin:$PATH" cd "$SRT_REPO_DIR" uv venv --python 3.12 uv pip install -e . -make setup-compute ARCH=x86_64 +make setup ARCH=x86_64 source .venv/bin/activate export SRTSLURM_CONFIG="${WORK_DIR}/srtslurm.yaml" export SRTCTL_RUNTIME_SOURCE_DIR="$SRT_REPO_DIR" diff --git a/utils/test_mi300x_srt_slurm_contract.py b/utils/test_mi300x_srt_slurm_contract.py index 94512086b6..e075491063 100644 --- a/utils/test_mi300x_srt_slurm_contract.py +++ b/utils/test_mi300x_srt_slurm_contract.py @@ -97,7 +97,7 @@ def test_official_matrix_routes_disagg_through_the_pinned_srt_launcher(): "CONFIG_FILE=recipes/vllm/qwen3-0.6b/mi300x/" "disagg-1p1d-fixed-seq.yaml" ] - assert "d93b48165ff60c6441feb5dd04504337f0bd7bc5" in launcher + assert "297da661ad058bb1ea4bad06be528ce4a0bbe9e2" in launcher assert launcher.count("setup ARCH=x86_64") == 2 assert "--no-preflight" in launcher assert 'ENROOT_RUNTIME_PATH="\\${TMPDIR:-/tmp}/enroot-runtime-\\${UID}"' in launcher From 5021e0e60e37b27abdc4b995079443cc4f4eec6e Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Tue, 11 Aug 2026 11:28:11 -0500 Subject: [PATCH 54/67] fix(atom): validate upstream Infera KV event decoder --- .../qwen3-0.6b/mi300x/agg-2w-fixed-seq.yaml | 3 +++ .../mi300x/disagg-1p1d-fixed-seq.yaml | 3 +++ perf-changelog.yaml | 1 + runners/launch_mi300x-amds-srt.sh | 19 ++++++++++++++++--- runners/launch_mi355x-amds-srt.sh | 18 +++++++++++++++--- utils/test_mi300x_srt_slurm_contract.py | 11 +++++++++++ 6 files changed, 49 insertions(+), 6 deletions(-) diff --git a/benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi300x/agg-2w-fixed-seq.yaml b/benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi300x/agg-2w-fixed-seq.yaml index 9158a722a0..e3f368aa22 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi300x/agg-2w-fixed-seq.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi300x/agg-2w-fixed-seq.yaml @@ -30,6 +30,8 @@ resources: frontend: type: infera enable_multiple_frontends: false + env: + PYTHONPATH: "/infera-source" args: router-policy: kv-aware @@ -40,6 +42,7 @@ backend: HF_HOME: "/hf_hub_cache" HF_HUB_CACHE: "/hf_hub_cache" HUGGINGFACE_HUB_CACHE: "/hf_hub_cache" + PYTHONPATH: "/infera-source" PYTHONUNBUFFERED: "1" OMP_NUM_THREADS: "1" atom_config: diff --git a/benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml b/benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml index 45e9864d0b..6a187a6512 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml @@ -34,6 +34,8 @@ resources: frontend: type: infera enable_multiple_frontends: false + env: + PYTHONPATH: "/infera-source" args: router-policy: kv-aware @@ -45,6 +47,7 @@ backend: HF_HOME: "/hf_hub_cache" HF_HUB_CACHE: "/hf_hub_cache" HUGGINGFACE_HUB_CACHE: "/hf_hub_cache" + PYTHONPATH: "/infera-source" PYTHONUNBUFFERED: "1" OMP_NUM_THREADS: "1" decode_environment: *worker_environment diff --git a/perf-changelog.yaml b/perf-changelog.yaml index cdb3b30b85..1876e23e8b 100644 --- a/perf-changelog.yaml +++ b/perf-changelog.yaml @@ -5796,4 +5796,5 @@ - "Add srt-slurm correctness lanes for the ROCm ATOM engine and the open-source Infera successor to ATOMesh." - "Validate two aggregate ATOM workers behind Infera KV-aware routing and a two-node 1P/1D ATOM deployment using Mooncake RDMA." - "Use rocm/infera:atom-v0.1.1 with Qwen3-0.6B fixed-sequence requests on MI300X and pin the combined AMD, vLLM Router, ATOM, and Infera srt-slurm branch." + - "Overlay the exact upstream Infera KV-event decoder fix from AMD-AGI/Infera#114 so ATOM's tagged-map events populate the router cache view during pre-release hardware validation." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2558 diff --git a/runners/launch_mi300x-amds-srt.sh b/runners/launch_mi300x-amds-srt.sh index 39a2928fd7..db9d2503e8 100644 --- a/runners/launch_mi300x-amds-srt.sh +++ b/runners/launch_mi300x-amds-srt.sh @@ -5,6 +5,8 @@ set -euo pipefail # matrix rows opt in by exporting CONFIG_FILE through additional-settings. SRT_SLURM_REPOSITORY="https://github.com/SemiAnalysisAI/srt-slurm.git" SRT_SLURM_COMMIT="297da661ad058bb1ea4bad06be528ce4a0bbe9e2" +INFERA_REPOSITORY="https://github.com/cquil11/Infera.git" +INFERA_COMMIT="8ed8f1728c745d4e91ba9eaa09ed81159aa57e41" SLURM_PARTITION="compute" EXCLUDED_NODES="chi-mi300x-049,chi-mi300x-121" REMOTE_BASE="/raid/hf-hub-cache/inferencex/srt-slurm" @@ -48,6 +50,7 @@ CLUSTER_PROFILE="${GITHUB_WORKSPACE}/benchmarks/multi_node/srt-slurm-recipes/clu RUN_KEY="${GITHUB_RUN_ID:-manual}-${GITHUB_RUN_ATTEMPT:-0}-${RUNNER_NAME:-runner}" REMOTE_RUNTIME="${REMOTE_BASE}/runtime/inferencex-${RUN_KEY}" REMOTE_SRT_RUNTIME="${REMOTE_BASE}/runtime/srt-slurm-${SRT_SLURM_COMMIT}" +REMOTE_INFERA_RUNTIME="${REMOTE_BASE}/runtime/infera-${INFERA_COMMIT}" REMOTE_RESULTS="${REMOTE_BASE}/results" WORK_DIR="${GITHUB_WORKSPACE}/.srt-slurm-${RUN_KEY}" SRT_REPO_DIR="${WORK_DIR}/srt-slurm" @@ -79,6 +82,7 @@ srun --ntasks-per-node=1 bash -c ' set -euo pipefail runtime="${REMOTE_RUNTIME}" srt_runtime="${REMOTE_SRT_RUNTIME}" + infera_runtime="${REMOTE_INFERA_RUNTIME}" export ENROOT_RUNTIME_PATH="\${TMPDIR:-/tmp}/enroot-runtime-\${UID}" mkdir -p "\$ENROOT_RUNTIME_PATH" "\$runtime" "${REMOTE_RESULTS}" "${REMOTE_BASE}/containers" chmod 700 "\$ENROOT_RUNTIME_PATH" @@ -123,6 +127,12 @@ srun --ntasks-per-node=1 bash -c ' git -C "\$srt_runtime" checkout --quiet --detach "${SRT_SLURM_COMMIT}" test "\$(git -C "\$srt_runtime" rev-parse HEAD)" = "${SRT_SLURM_COMMIT}" make -C "\$srt_runtime" --no-print-directory setup ARCH=x86_64 + if [[ ! -d "\$infera_runtime/.git" ]]; then + git clone --quiet "${INFERA_REPOSITORY}" "\$infera_runtime" + fi + git -C "\$infera_runtime" fetch --quiet origin "${INFERA_COMMIT}" + git -C "\$infera_runtime" checkout --quiet --detach "${INFERA_COMMIT}" + test "\$(git -C "\$infera_runtime" rev-parse HEAD)" = "${INFERA_COMMIT}" tar -xzf "/tmp/inferencex-benchmark-\${SLURM_JOB_ID}.tar.gz" -C "\$runtime" printf "%s\\n" "${GITHUB_SHA:-unknown}" > "\$runtime/.inferencex-source-head" ' @@ -141,12 +151,12 @@ ACTUAL_SRT_COMMIT=$(git -C "$SRT_REPO_DIR" rev-parse HEAD) mkdir -p "${SRT_REPO_DIR}/$(dirname "$CONFIG_PATH")" cp "$LOCAL_RECIPE" "${SRT_REPO_DIR}/${CONFIG_PATH}" cp "$CLUSTER_PROFILE" "${WORK_DIR}/srtslurm.yaml" -python3 - "${WORK_DIR}/srtslurm.yaml" "$REMOTE_RUNTIME" "$REMOTE_RESULTS" <<'PY' +python3 - "${WORK_DIR}/srtslurm.yaml" "$REMOTE_RUNTIME" "$REMOTE_RESULTS" "$REMOTE_INFERA_RUNTIME" <<'PY' import sys from pathlib import Path path = Path(sys.argv[1]) -runtime, results = sys.argv[2:] +runtime, results, infera_runtime = sys.argv[2:] needle = " /raid/hf-hub-cache: /hf_hub_cache\n" text = path.read_text() if text.count(needle) != 1: @@ -154,7 +164,10 @@ if text.count(needle) != 1: path.write_text( text.replace( needle, - needle + f" {runtime}: /infmax-workspace\n {results}: /results\n", + needle + + f" {runtime}: /infmax-workspace\n" + + f" {results}: /results\n" + + f" {infera_runtime}: /infera-source\n", ) ) PY diff --git a/runners/launch_mi355x-amds-srt.sh b/runners/launch_mi355x-amds-srt.sh index 941b983f34..fdf66745a4 100755 --- a/runners/launch_mi355x-amds-srt.sh +++ b/runners/launch_mi355x-amds-srt.sh @@ -6,12 +6,15 @@ set -euo pipefail # unchanged for every other row. SRT_SLURM_REPOSITORY="https://github.com/SemiAnalysisAI/srt-slurm.git" SRT_SLURM_COMMIT="297da661ad058bb1ea4bad06be528ce4a0bbe9e2" +INFERA_REPOSITORY="https://github.com/cquil11/Infera.git" +INFERA_COMMIT="8ed8f1728c745d4e91ba9eaa09ed81159aa57e41" SLURM_PARTITION="compute" SGLANG_IMAGE="lmsysorg/sglang-rocm:v0.5.17-rocm720-mi35x-20260809" ATOM_IMAGE="rocm/infera:atom-v0.1.1" SHARED_BASE="/it-share/gharunners2/srt-slurm" SHARED_HF_CACHE="/it-share/hf-hub-cache" SHARED_RESULTS="${SHARED_BASE}/results" +SHARED_INFERA_RUNTIME="${SHARED_BASE}/runtime/infera-${INFERA_COMMIT}" : "${GITHUB_WORKSPACE:?GITHUB_WORKSPACE must be set by Actions}" : "${RESULT_FILENAME:?RESULT_FILENAME must be set by the benchmark workflow}" @@ -63,6 +66,12 @@ set -euo pipefail export ENROOT_RUNTIME_PATH="\${TMPDIR:-/tmp}/enroot-runtime-\${UID}" mkdir -p "\$ENROOT_RUNTIME_PATH" "$(dirname "$SHARED_IMAGE")" "$SHARED_HF_CACHE" chmod 700 "\$ENROOT_RUNTIME_PATH" +if [[ ! -d "${SHARED_INFERA_RUNTIME}/.git" ]]; then + git clone --quiet "${INFERA_REPOSITORY}" "${SHARED_INFERA_RUNTIME}" +fi +git -C "${SHARED_INFERA_RUNTIME}" fetch --quiet origin "${INFERA_COMMIT}" +git -C "${SHARED_INFERA_RUNTIME}" checkout --quiet --detach "${INFERA_COMMIT}" +test "\$(git -C "${SHARED_INFERA_RUNTIME}" rev-parse HEAD)" = "${INFERA_COMMIT}" exec 9>"${SHARED_IMAGE}.lock" flock -w 2400 9 if ! unsquashfs -s "$SHARED_IMAGE" >/dev/null 2>&1; then @@ -110,12 +119,12 @@ ACTUAL_SRT_COMMIT=$(git -C "$SRT_REPO_DIR" rev-parse HEAD) mkdir -p "${SRT_REPO_DIR}/$(dirname "$CONFIG_PATH")" cp "$LOCAL_RECIPE" "${SRT_REPO_DIR}/${CONFIG_PATH}" cp "$CLUSTER_PROFILE" "${WORK_DIR}/srtslurm.yaml" -python3 - "${WORK_DIR}/srtslurm.yaml" "$GITHUB_WORKSPACE" "$SHARED_RESULTS" <<'PY' +python3 - "${WORK_DIR}/srtslurm.yaml" "$GITHUB_WORKSPACE" "$SHARED_RESULTS" "$SHARED_INFERA_RUNTIME" <<'PY' import sys from pathlib import Path path = Path(sys.argv[1]) -workspace, results = sys.argv[2:] +workspace, results, infera_runtime = sys.argv[2:] needle = " /it-share/hf-hub-cache: /hf_hub_cache\n" text = path.read_text() if text.count(needle) != 1: @@ -123,7 +132,10 @@ if text.count(needle) != 1: path.write_text( text.replace( needle, - needle + f" {workspace}: /infmax-workspace\n {results}: /results\n", + needle + + f" {workspace}: /infmax-workspace\n" + + f" {results}: /results\n" + + f" {infera_runtime}: /infera-source\n", ) ) PY diff --git a/utils/test_mi300x_srt_slurm_contract.py b/utils/test_mi300x_srt_slurm_contract.py index e075491063..483a37c599 100644 --- a/utils/test_mi300x_srt_slurm_contract.py +++ b/utils/test_mi300x_srt_slurm_contract.py @@ -218,6 +218,17 @@ def test_atom_recipes_use_infera_and_keep_worker_metrics_honest(): assert "--endpoint /v1/completions" in command assert disaggregate["backend"]["connector"] == "mooncake" assert 'ATOM_IMAGE="rocm/infera:atom-v0.1.1"' in launcher + assert 'INFERA_COMMIT="8ed8f1728c745d4e91ba9eaa09ed81159aa57e41"' in launcher + assert 'REMOTE_INFERA_RUNTIME="${REMOTE_BASE}/runtime/infera-${INFERA_COMMIT}"' in launcher + for recipe in (aggregate, disaggregate): + assert recipe["frontend"]["env"]["PYTHONPATH"] == "/infera-source" + role_environments = [ + value + for key, value in recipe["backend"].items() + if key.endswith("_environment") + ] + assert role_environments + assert all(env["PYTHONPATH"] == "/infera-source" for env in role_environments) def test_fixed_sequence_commands_keep_all_arguments_attached(tmp_path): From f307b55e936c60739d42030d6bba1b4f46bf51a2 Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Tue, 11 Aug 2026 11:47:03 -0500 Subject: [PATCH 55/67] fix(mi300x): stage every eligible srt node Signed-off-by: Cam Quilici --- runners/launch_mi300x-amds-srt.sh | 7 ++++--- utils/test_mi300x_srt_slurm_contract.py | 1 + 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/runners/launch_mi300x-amds-srt.sh b/runners/launch_mi300x-amds-srt.sh index db9d2503e8..43aacf3d57 100644 --- a/runners/launch_mi300x-amds-srt.sh +++ b/runners/launch_mi300x-amds-srt.sh @@ -58,8 +58,9 @@ mkdir -p "$WORK_DIR" # The login and compute nodes do not share a filesystem. Stage only the # unchanged InferenceX benchmark client and immutable public container images -# onto every eligible node. The batch job exits normally; it does not cancel or -# preempt any allocation. +# onto every eligible node. The cluster has eight nodes and this validation +# excludes two, so the staging allocation must cover all six remaining nodes. +# The batch job exits normally; it does not cancel or preempt any allocation. RUNTIME_ARCHIVE="${WORK_DIR}/inferencex-benchmark.tar.gz" tar -C "$GITHUB_WORKSPACE" -czf "$RUNTIME_ARCHIVE" utils/bench_serving RUNTIME_PAYLOAD=$(base64 -w0 "$RUNTIME_ARCHIVE") @@ -67,7 +68,7 @@ STAGE_SCRIPT="${WORK_DIR}/stage-runtime.sbatch" cat > "$STAGE_SCRIPT" < Date: Tue, 11 Aug 2026 11:49:00 -0500 Subject: [PATCH 56/67] fix(mi300x): cover the complete eligible node set Signed-off-by: Cam Quilici --- runners/launch_mi300x-amds-srt.sh | 6 +++--- utils/test_mi300x_srt_slurm_contract.py | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/runners/launch_mi300x-amds-srt.sh b/runners/launch_mi300x-amds-srt.sh index 43aacf3d57..06476d86e2 100644 --- a/runners/launch_mi300x-amds-srt.sh +++ b/runners/launch_mi300x-amds-srt.sh @@ -58,8 +58,8 @@ mkdir -p "$WORK_DIR" # The login and compute nodes do not share a filesystem. Stage only the # unchanged InferenceX benchmark client and immutable public container images -# onto every eligible node. The cluster has eight nodes and this validation -# excludes two, so the staging allocation must cover all six remaining nodes. +# onto every eligible node. The cluster has nine nodes and this validation +# excludes two, so the staging allocation must cover all seven remaining nodes. # The batch job exits normally; it does not cancel or preempt any allocation. RUNTIME_ARCHIVE="${WORK_DIR}/inferencex-benchmark.tar.gz" tar -C "$GITHUB_WORKSPACE" -czf "$RUNTIME_ARCHIVE" utils/bench_serving @@ -68,7 +68,7 @@ STAGE_SCRIPT="${WORK_DIR}/stage-runtime.sbatch" cat > "$STAGE_SCRIPT" < Date: Tue, 11 Aug 2026 11:57:33 -0500 Subject: [PATCH 57/67] fix(mi300x): recover incomplete runtime checkouts Signed-off-by: Cam Quilici --- runners/launch_mi300x-amds-srt.sh | 35 ++++++++++++++++--------- utils/test_mi300x_srt_slurm_contract.py | 9 ++++++- 2 files changed, 31 insertions(+), 13 deletions(-) diff --git a/runners/launch_mi300x-amds-srt.sh b/runners/launch_mi300x-amds-srt.sh index 06476d86e2..fe2ebd1de0 100644 --- a/runners/launch_mi300x-amds-srt.sh +++ b/runners/launch_mi300x-amds-srt.sh @@ -117,23 +117,34 @@ srun --ntasks-per-node=1 bash -c ' flock -u "\$lock_fd" exec {lock_fd}>&- } + ensure_git_checkout() { + local target="\$1" + local repository="\$2" + local commit="\$3" + local temporary="\${target}.tmp.\${SLURM_JOB_ID}.\${BASHPID}" + local quarantine="\${target}.incomplete.\${SLURM_JOB_ID}.\${BASHPID}" + if [[ ! -d "\$target/.git" ]]; then + if [[ -e "\$target" ]]; then + mv "\$target" "\$quarantine" + fi + git clone --quiet "\$repository" "\$temporary" + git -C "\$temporary" fetch --quiet origin "\$commit" + git -C "\$temporary" checkout --quiet --detach "\$commit" + test "\$(git -C "\$temporary" rev-parse HEAD)" = "\$commit" + mv "\$temporary" "\$target" + else + git -C "\$target" fetch --quiet origin "\$commit" + git -C "\$target" checkout --quiet --detach "\$commit" + test "\$(git -C "\$target" rev-parse HEAD)" = "\$commit" + fi + } ensure_container_image "${ENGINE_SQSH}" "${ENGINE_IMAGE}" if [[ -n "${AUX_IMAGE}" ]]; then ensure_container_image "${AUX_SQSH}" "${AUX_IMAGE}" fi - if [[ ! -d "\$srt_runtime/.git" ]]; then - git clone --quiet "${SRT_SLURM_REPOSITORY}" "\$srt_runtime" - fi - git -C "\$srt_runtime" fetch --quiet origin "${SRT_SLURM_COMMIT}" - git -C "\$srt_runtime" checkout --quiet --detach "${SRT_SLURM_COMMIT}" - test "\$(git -C "\$srt_runtime" rev-parse HEAD)" = "${SRT_SLURM_COMMIT}" + ensure_git_checkout "\$srt_runtime" "${SRT_SLURM_REPOSITORY}" "${SRT_SLURM_COMMIT}" make -C "\$srt_runtime" --no-print-directory setup ARCH=x86_64 - if [[ ! -d "\$infera_runtime/.git" ]]; then - git clone --quiet "${INFERA_REPOSITORY}" "\$infera_runtime" - fi - git -C "\$infera_runtime" fetch --quiet origin "${INFERA_COMMIT}" - git -C "\$infera_runtime" checkout --quiet --detach "${INFERA_COMMIT}" - test "\$(git -C "\$infera_runtime" rev-parse HEAD)" = "${INFERA_COMMIT}" + ensure_git_checkout "\$infera_runtime" "${INFERA_REPOSITORY}" "${INFERA_COMMIT}" tar -xzf "/tmp/inferencex-benchmark-\${SLURM_JOB_ID}.tar.gz" -C "\$runtime" printf "%s\\n" "${GITHUB_SHA:-unknown}" > "\$runtime/.inferencex-source-head" ' diff --git a/utils/test_mi300x_srt_slurm_contract.py b/utils/test_mi300x_srt_slurm_contract.py index d64cb4d012..55d4ec2889 100644 --- a/utils/test_mi300x_srt_slurm_contract.py +++ b/utils/test_mi300x_srt_slurm_contract.py @@ -113,10 +113,17 @@ def test_official_matrix_routes_disagg_through_the_pinned_srt_launcher(): assert 'unsquashfs -s "\\$tmp"' in launcher assert 'mv "\\$tmp" "\\$target"' in launcher assert 'REMOTE_SRT_RUNTIME="${REMOTE_BASE}/runtime/srt-slurm-${SRT_SLURM_COMMIT}"' in launcher - assert 'git -C "\\$srt_runtime" checkout --quiet --detach "${SRT_SLURM_COMMIT}"' in launcher + assert ( + 'ensure_git_checkout "\\$srt_runtime" "${SRT_SLURM_REPOSITORY}" ' + '"${SRT_SLURM_COMMIT}"' + in launcher + ) assert 'make -C "\\$srt_runtime" --no-print-directory setup ARCH=x86_64' in launcher assert 'export SRTCTL_RUNTIME_SOURCE_DIR="$REMOTE_SRT_RUNTIME"' in launcher assert '#SBATCH --nodes=7' in launcher + assert 'ensure_git_checkout()' in launcher + assert 'mv "\\$target" "\\$quarantine"' in launcher + assert 'mv "\\$temporary" "\\$target"' in launcher assert "scancel" not in launcher From bb97fd7a4a7822492983c8d3ed99000a7d7c4a9d Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Tue, 11 Aug 2026 12:03:20 -0500 Subject: [PATCH 58/67] test(mi355x): add ATOM and Infera validation lanes Signed-off-by: Cam Quilici --- .../qwen3-0.6b/mi355x/agg-2w-fixed-seq.yaml | 99 ++++++++++++++++ .../mi355x/disagg-1p1d-fixed-seq.yaml | 107 ++++++++++++++++++ configs/amd-master.yaml | 62 ++++++++++ perf-changelog.yaml | 4 +- utils/test_mi355x_atom_srt_contract.py | 78 +++++++++++++ 5 files changed, 349 insertions(+), 1 deletion(-) create mode 100644 benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi355x/agg-2w-fixed-seq.yaml create mode 100644 benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi355x/disagg-1p1d-fixed-seq.yaml create mode 100644 utils/test_mi355x_atom_srt_contract.py diff --git a/benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi355x/agg-2w-fixed-seq.yaml b/benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi355x/agg-2w-fixed-seq.yaml new file mode 100644 index 0000000000..340b2071bf --- /dev/null +++ b/benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi355x/agg-2w-fixed-seq.yaml @@ -0,0 +1,99 @@ +# Small two-worker aggregate correctness lane for ATOM and Infera on MI355X. +# It validates orchestration and KV-aware routing rather than performance. + +name: "mi355x-atom-qwen3-0.6b-agg-2w-fixed-seq" + +model: + path: "hf:Qwen/Qwen3-0.6B" + container: "infera-atom-v0.1.1" + precision: "fp16" + +identity: + model: + repo: "Qwen/Qwen3-0.6B" + container: + image: "rocm/infera:atom-v0.1.1" + frameworks: + atom: "0.1.1" + infera: "0.1.1" + +slurm: + time_limit: "00:30:00" + +resources: + gpu_type: "mi355x" + gpus_per_node: 8 + agg_nodes: 1 + agg_workers: 2 + gpus_per_agg: 1 + +frontend: + type: infera + enable_multiple_frontends: false + env: + PYTHONPATH: "/infera-source" + args: + router-policy: kv-aware + +backend: + type: atom + enable_kv_events: true + aggregated_environment: + HF_HOME: "/hf_hub_cache" + HF_HUB_CACHE: "/hf_hub_cache/hub" + HUGGINGFACE_HUB_CACHE: "/hf_hub_cache/hub" + PYTHONPATH: "/infera-source" + PYTHONUNBUFFERED: "1" + OMP_NUM_THREADS: "1" + atom_config: + aggregated: + kv_cache_dtype: fp8 + gpu-memory-utilization: 0.50 + max-model-len: 2048 + max-num-seqs: 8 + block-size: 16 + enforce-eager: true + +srun_options: + container-writable: "" + container-remap-root: "" + mem: "0" + +health_check: + max_attempts: 180 + interval_seconds: 5 + +benchmark: + type: custom + command: | + set -euo pipefail + result_root="/results/${SLURM_JOB_ID}" + mkdir -p "${result_root}/fixed-seq" + trap 'tar -C /logs -czf "'"${result_root}"'/runtime-logs.tar.gz" . 2>/dev/null || true' EXIT + for concurrency in 1 4; do + python3 /infmax-workspace/utils/bench_serving/benchmark_serving.py \ + --backend openai \ + --base-url "http://${SRT_FRONTEND_HOST}:${SRT_FRONTEND_PORT}" \ + --endpoint /v1/completions \ + --model Qwen/Qwen3-0.6B \ + --tokenizer Qwen/Qwen3-0.6B \ + --dataset-name random \ + --random-input-len 128 \ + --random-output-len 32 \ + --random-prefix-len 96 \ + --random-range-ratio 1.0 \ + --random-num-workers 1 \ + --num-warmups "${concurrency}" \ + --num-prompts "$((concurrency * 4))" \ + --max-concurrency "${concurrency}" \ + --request-rate inf \ + --ignore-eos \ + --disable-tqdm \ + --save-result \ + --result-dir "${result_root}/fixed-seq" \ + --result-filename "qwen3-0.6b-atom-mi355x-agg-isl128-osl32-c${concurrency}.json"; + done + env: + HF_HOME: /hf_hub_cache + HF_HUB_CACHE: /hf_hub_cache/hub + HUGGINGFACE_HUB_CACHE: /hf_hub_cache/hub diff --git a/benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi355x/disagg-1p1d-fixed-seq.yaml b/benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi355x/disagg-1p1d-fixed-seq.yaml new file mode 100644 index 0000000000..d2f2f6c4c2 --- /dev/null +++ b/benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi355x/disagg-1p1d-fixed-seq.yaml @@ -0,0 +1,107 @@ +# Small two-node ATOM P/D correctness lane on MI355X. Infera discovers the +# role-specific workers and ATOM transfers KV through Mooncake. + +name: "mi355x-atom-qwen3-0.6b-disagg-1p1d-fixed-seq" + +model: + path: "hf:Qwen/Qwen3-0.6B" + container: "infera-atom-v0.1.1" + precision: "fp16" + +identity: + model: + repo: "Qwen/Qwen3-0.6B" + container: + image: "rocm/infera:atom-v0.1.1" + frameworks: + atom: "0.1.1" + infera: "0.1.1" + mooncake: "bundled" + +slurm: + time_limit: "00:45:00" + +resources: + gpu_type: "mi355x" + gpus_per_node: 8 + prefill_nodes: 1 + decode_nodes: 1 + prefill_workers: 1 + decode_workers: 1 + gpus_per_prefill: 1 + gpus_per_decode: 1 + +frontend: + type: infera + enable_multiple_frontends: false + env: + PYTHONPATH: "/infera-source" + args: + router-policy: kv-aware + +backend: + type: atom + connector: mooncake + enable_kv_events: true + prefill_environment: &worker_environment + HF_HOME: "/hf_hub_cache" + HF_HUB_CACHE: "/hf_hub_cache/hub" + HUGGINGFACE_HUB_CACHE: "/hf_hub_cache/hub" + PYTHONPATH: "/infera-source" + PYTHONUNBUFFERED: "1" + OMP_NUM_THREADS: "1" + decode_environment: *worker_environment + atom_config: + prefill: &worker_config + kv_cache_dtype: fp8 + gpu-memory-utilization: 0.50 + max-model-len: 2048 + max-num-seqs: 8 + block-size: 16 + enforce-eager: true + no-enable_prefix_caching: true + decode: *worker_config + +srun_options: + container-writable: "" + container-remap-root: "" + mem: "0" + +health_check: + max_attempts: 240 + interval_seconds: 5 + +benchmark: + type: custom + command: | + set -euo pipefail + result_root="/results/${SLURM_JOB_ID}" + mkdir -p "${result_root}/fixed-seq" + trap 'tar -C /logs -czf "'"${result_root}"'/runtime-logs.tar.gz" . 2>/dev/null || true' EXIT + for concurrency in 1 4; do + python3 /infmax-workspace/utils/bench_serving/benchmark_serving.py \ + --backend openai \ + --base-url "http://${SRT_FRONTEND_HOST}:${SRT_FRONTEND_PORT}" \ + --endpoint /v1/completions \ + --model Qwen/Qwen3-0.6B \ + --tokenizer Qwen/Qwen3-0.6B \ + --dataset-name random \ + --random-input-len 128 \ + --random-output-len 32 \ + --random-prefix-len 96 \ + --random-range-ratio 1.0 \ + --random-num-workers 1 \ + --num-warmups "${concurrency}" \ + --num-prompts "$((concurrency * 4))" \ + --max-concurrency "${concurrency}" \ + --request-rate inf \ + --ignore-eos \ + --disable-tqdm \ + --save-result \ + --result-dir "${result_root}/fixed-seq" \ + --result-filename "qwen3-0.6b-atom-mi355x-disagg-isl128-osl32-c${concurrency}.json"; + done + env: + HF_HOME: /hf_hub_cache + HF_HUB_CACHE: /hf_hub_cache/hub + HUGGINGFACE_HUB_CACHE: /hf_hub_cache/hub diff --git a/configs/amd-master.yaml b/configs/amd-master.yaml index 9ffc1b4cae..15ec613e4c 100644 --- a/configs/amd-master.yaml +++ b/configs/amd-master.yaml @@ -1626,6 +1626,68 @@ qwen3-0.6b-fp16-mi300x-atom-infera-srt-disagg: ep: 1 dp-attn: false +# Small-model correctness lanes for ATOM and Infera orchestration on MI355X. +# Aggregate uses two workers so KV-event-aware routing is exercised; P/D uses +# one prefill and one decode worker with Mooncake transfer. +qwen3-0.6b-fp16-mi355x-atom-infera-srt-agg: + image: rocm/infera:atom-v0.1.1 + model: Qwen/Qwen3-0.6B + model-prefix: qwen3-0.6b + runner: cluster:mi355x-amds + precision: fp16 + framework: atom + multinode: true + disagg: false + router: { name: infera, version: "0.1.1" } + scenarios: + fixed-seq-len: + - isl: 128 + osl: 32 + search-space: + - conc-list: [1] + prefill: + num-worker: 2 + tp: 1 + ep: 1 + dp-attn: false + additional-settings: + - "CONFIG_FILE=recipes/atom/qwen3-0.6b/mi355x/agg-2w-fixed-seq.yaml" + decode: + num-worker: 0 + tp: 1 + ep: 1 + dp-attn: false + +qwen3-0.6b-fp16-mi355x-atom-infera-srt-disagg: + image: rocm/infera:atom-v0.1.1 + model: Qwen/Qwen3-0.6B + model-prefix: qwen3-0.6b + runner: cluster:mi355x-amds + precision: fp16 + framework: atom-disagg + multinode: true + disagg: true + router: { name: infera, version: "0.1.1" } + kv-p2p-transfer: mooncake + scenarios: + fixed-seq-len: + - isl: 128 + osl: 32 + search-space: + - conc-list: [1] + prefill: + num-worker: 1 + tp: 1 + ep: 1 + dp-attn: false + additional-settings: + - "CONFIG_FILE=recipes/atom/qwen3-0.6b/mi355x/disagg-1p1d-fixed-seq.yaml" + decode: + num-worker: 1 + tp: 1 + ep: 1 + dp-attn: false + # Small-model correctness lanes for native SGLang Router orchestration on # MI355X. Both use the unchanged InferenceX fixed-sequence benchmark; the P/D # lane uses AMD MoRI and neither lane starts Dynamo, NATS, etcd, or NIXL. diff --git a/perf-changelog.yaml b/perf-changelog.yaml index 1876e23e8b..98e40cc7a3 100644 --- a/perf-changelog.yaml +++ b/perf-changelog.yaml @@ -5792,9 +5792,11 @@ - config-keys: - qwen3-0.6b-fp16-mi300x-atom-infera-srt-agg - qwen3-0.6b-fp16-mi300x-atom-infera-srt-disagg + - qwen3-0.6b-fp16-mi355x-atom-infera-srt-agg + - qwen3-0.6b-fp16-mi355x-atom-infera-srt-disagg description: - "Add srt-slurm correctness lanes for the ROCm ATOM engine and the open-source Infera successor to ATOMesh." - "Validate two aggregate ATOM workers behind Infera KV-aware routing and a two-node 1P/1D ATOM deployment using Mooncake RDMA." - - "Use rocm/infera:atom-v0.1.1 with Qwen3-0.6B fixed-sequence requests on MI300X and pin the combined AMD, vLLM Router, ATOM, and Infera srt-slurm branch." + - "Use rocm/infera:atom-v0.1.1 with Qwen3-0.6B fixed-sequence requests on MI300X and MI355X, and pin the combined AMD, vLLM Router, ATOM, and Infera srt-slurm branch." - "Overlay the exact upstream Infera KV-event decoder fix from AMD-AGI/Infera#114 so ATOM's tagged-map events populate the router cache view during pre-release hardware validation." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2558 diff --git a/utils/test_mi355x_atom_srt_contract.py b/utils/test_mi355x_atom_srt_contract.py new file mode 100644 index 0000000000..a81700fee6 --- /dev/null +++ b/utils/test_mi355x_atom_srt_contract.py @@ -0,0 +1,78 @@ +"""High-signal contracts for ATOM/Infera validation on MI355X.""" + +from pathlib import Path + +import yaml + + +REPO_ROOT = Path(__file__).resolve().parents[1] +RECIPE_ROOT = ( + REPO_ROOT + / "benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi355x" +) +AGG_RECIPE = RECIPE_ROOT / "agg-2w-fixed-seq.yaml" +DISAGG_RECIPE = RECIPE_ROOT / "disagg-1p1d-fixed-seq.yaml" +MASTER_CONFIG = REPO_ROOT / "configs/amd-master.yaml" +LAUNCHER = REPO_ROOT / "runners/launch_mi355x-amds-srt.sh" + + +def test_aggregate_exercises_two_atom_workers_and_kv_aware_infera(): + recipe = yaml.safe_load(AGG_RECIPE.read_text()) + + assert recipe["resources"] == { + "gpu_type": "mi355x", + "gpus_per_node": 8, + "agg_nodes": 1, + "agg_workers": 2, + "gpus_per_agg": 1, + } + assert recipe["frontend"]["type"] == "infera" + assert recipe["frontend"]["args"]["router-policy"] == "kv-aware" + assert recipe["backend"]["type"] == "atom" + assert recipe["backend"]["enable_kv_events"] is True + assert recipe["backend"]["aggregated_environment"]["PYTHONPATH"] == ( + "/infera-source" + ) + command = recipe["benchmark"]["command"] + assert "--random-prefix-len 96" in command + assert "--endpoint /v1/completions" in command + + +def test_disaggregate_uses_one_prefill_one_decode_and_mooncake(): + recipe = yaml.safe_load(DISAGG_RECIPE.read_text()) + + assert recipe["resources"]["prefill_workers"] == 1 + assert recipe["resources"]["decode_workers"] == 1 + assert recipe["resources"]["prefill_nodes"] == 1 + assert recipe["resources"]["decode_nodes"] == 1 + assert recipe["backend"]["type"] == "atom" + assert recipe["backend"]["connector"] == "mooncake" + assert recipe["backend"]["enable_kv_events"] is True + assert recipe["backend"]["prefill_environment"]["PYTHONPATH"] == ( + "/infera-source" + ) + + +def test_matrix_rows_route_only_through_the_atom_srt_launcher(): + configs = yaml.safe_load(MASTER_CONFIG.read_text()) + launcher = LAUNCHER.read_text() + + agg = configs["qwen3-0.6b-fp16-mi355x-atom-infera-srt-agg"] + disagg = configs["qwen3-0.6b-fp16-mi355x-atom-infera-srt-disagg"] + assert agg["framework"] == "atom" + assert agg["router"] == {"name": "infera", "version": "0.1.1"} + assert agg["scenarios"]["fixed-seq-len"][0]["search-space"][0][ + "prefill" + ]["additional-settings"] == [ + "CONFIG_FILE=recipes/atom/qwen3-0.6b/mi355x/agg-2w-fixed-seq.yaml" + ] + assert disagg["framework"] == "atom-disagg" + assert disagg["kv-p2p-transfer"] == "mooncake" + assert disagg["scenarios"]["fixed-seq-len"][0]["search-space"][0][ + "prefill" + ]["additional-settings"] == [ + "CONFIG_FILE=recipes/atom/qwen3-0.6b/mi355x/disagg-1p1d-fixed-seq.yaml" + ] + assert "297da661ad058bb1ea4bad06be528ce4a0bbe9e2" in launcher + assert "8ed8f1728c745d4e91ba9eaa09ed81159aa57e41" in launcher + assert "scancel" not in launcher From 85f55d6a6576fb86ef9605882624cdf38655b7c0 Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Tue, 11 Aug 2026 12:22:00 -0500 Subject: [PATCH 59/67] fix(atom): keep Mooncake transfer paths reusable --- .../atom/qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml | 4 ++++ runners/launch_mi300x-amds-srt.sh | 2 +- runners/launch_mi355x-amds-srt.sh | 2 +- utils/test_mi300x_srt_slurm_contract.py | 3 ++- utils/test_mi355x_atom_srt_contract.py | 2 +- 5 files changed, 9 insertions(+), 4 deletions(-) diff --git a/benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml b/benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml index 6a187a6512..1e60fdac05 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml @@ -42,6 +42,10 @@ frontend: backend: type: atom connector: mooncake + # The MI300X private fabric is mlx5_1. ATOM's generic GPU-index heuristic + # selects a nonexistent rdma0 alias on this cluster and silently falls back + # to TCP, so pin the actual HCA for the P/D correctness lane. + mooncake_ib_device: mlx5_1 enable_kv_events: true prefill_environment: &worker_environment HF_HOME: "/hf_hub_cache" diff --git a/runners/launch_mi300x-amds-srt.sh b/runners/launch_mi300x-amds-srt.sh index fe2ebd1de0..ad73cc56de 100644 --- a/runners/launch_mi300x-amds-srt.sh +++ b/runners/launch_mi300x-amds-srt.sh @@ -4,7 +4,7 @@ set -euo pipefail # MI300X srt-slurm validation path. The existing launcher remains the default; # matrix rows opt in by exporting CONFIG_FILE through additional-settings. SRT_SLURM_REPOSITORY="https://github.com/SemiAnalysisAI/srt-slurm.git" -SRT_SLURM_COMMIT="297da661ad058bb1ea4bad06be528ce4a0bbe9e2" +SRT_SLURM_COMMIT="f71cbb1cd7a4247ce50d54d84254c1842258b6e3" INFERA_REPOSITORY="https://github.com/cquil11/Infera.git" INFERA_COMMIT="8ed8f1728c745d4e91ba9eaa09ed81159aa57e41" SLURM_PARTITION="compute" diff --git a/runners/launch_mi355x-amds-srt.sh b/runners/launch_mi355x-amds-srt.sh index fdf66745a4..90161a293d 100755 --- a/runners/launch_mi355x-amds-srt.sh +++ b/runners/launch_mi355x-amds-srt.sh @@ -5,7 +5,7 @@ set -euo pipefail # in explicitly with CONFIG_FILE; all existing MI355X launch behavior remains # unchanged for every other row. SRT_SLURM_REPOSITORY="https://github.com/SemiAnalysisAI/srt-slurm.git" -SRT_SLURM_COMMIT="297da661ad058bb1ea4bad06be528ce4a0bbe9e2" +SRT_SLURM_COMMIT="f71cbb1cd7a4247ce50d54d84254c1842258b6e3" INFERA_REPOSITORY="https://github.com/cquil11/Infera.git" INFERA_COMMIT="8ed8f1728c745d4e91ba9eaa09ed81159aa57e41" SLURM_PARTITION="compute" diff --git a/utils/test_mi300x_srt_slurm_contract.py b/utils/test_mi300x_srt_slurm_contract.py index 55d4ec2889..5b762bcd9a 100644 --- a/utils/test_mi300x_srt_slurm_contract.py +++ b/utils/test_mi300x_srt_slurm_contract.py @@ -97,7 +97,7 @@ def test_official_matrix_routes_disagg_through_the_pinned_srt_launcher(): "CONFIG_FILE=recipes/vllm/qwen3-0.6b/mi300x/" "disagg-1p1d-fixed-seq.yaml" ] - assert "297da661ad058bb1ea4bad06be528ce4a0bbe9e2" in launcher + assert "f71cbb1cd7a4247ce50d54d84254c1842258b6e3" in launcher assert launcher.count("setup ARCH=x86_64") == 2 assert "--no-preflight" in launcher assert 'ENROOT_RUNTIME_PATH="\\${TMPDIR:-/tmp}/enroot-runtime-\\${UID}"' in launcher @@ -225,6 +225,7 @@ def test_atom_recipes_use_infera_and_keep_worker_metrics_honest(): assert "--backend openai" in command assert "--endpoint /v1/completions" in command assert disaggregate["backend"]["connector"] == "mooncake" + assert disaggregate["backend"]["mooncake_ib_device"] == "mlx5_1" assert 'ATOM_IMAGE="rocm/infera:atom-v0.1.1"' in launcher assert 'INFERA_COMMIT="8ed8f1728c745d4e91ba9eaa09ed81159aa57e41"' in launcher assert 'REMOTE_INFERA_RUNTIME="${REMOTE_BASE}/runtime/infera-${INFERA_COMMIT}"' in launcher diff --git a/utils/test_mi355x_atom_srt_contract.py b/utils/test_mi355x_atom_srt_contract.py index a81700fee6..211ba20c0a 100644 --- a/utils/test_mi355x_atom_srt_contract.py +++ b/utils/test_mi355x_atom_srt_contract.py @@ -73,6 +73,6 @@ def test_matrix_rows_route_only_through_the_atom_srt_launcher(): ]["additional-settings"] == [ "CONFIG_FILE=recipes/atom/qwen3-0.6b/mi355x/disagg-1p1d-fixed-seq.yaml" ] - assert "297da661ad058bb1ea4bad06be528ce4a0bbe9e2" in launcher + assert "f71cbb1cd7a4247ce50d54d84254c1842258b6e3" in launcher assert "8ed8f1728c745d4e91ba9eaa09ed81159aa57e41" in launcher assert "scancel" not in launcher From 1a5b36f390d5f8c48e37e0e4df97b0c24e145c5d Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Tue, 11 Aug 2026 12:38:01 -0500 Subject: [PATCH 60/67] fix(atom): validate Mooncake TCP transfer path --- .../atom/qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml | 10 +++++----- .../atom/qwen3-0.6b/mi355x/disagg-1p1d-fixed-seq.yaml | 4 ++++ runners/launch_mi300x-amds-srt.sh | 2 +- runners/launch_mi355x-amds-srt.sh | 2 +- utils/test_mi300x_srt_slurm_contract.py | 4 ++-- utils/test_mi355x_atom_srt_contract.py | 3 ++- 6 files changed, 15 insertions(+), 10 deletions(-) diff --git a/benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml b/benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml index 1e60fdac05..5e27f67171 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml @@ -1,5 +1,5 @@ # Small two-node ATOM P/D correctness lane. Infera discovers both workers, -# routes the completions API, and ATOM transfers KV through Mooncake RDMA. +# routes the completions API, and ATOM transfers KV through Mooncake. name: "mi300x-atom-qwen3-0.6b-disagg-1p1d-fixed-seq" @@ -42,10 +42,10 @@ frontend: backend: type: atom connector: mooncake - # The MI300X private fabric is mlx5_1. ATOM's generic GPU-index heuristic - # selects a nonexistent rdma0 alias on this cluster and silently falls back - # to TCP, so pin the actual HCA for the P/D correctness lane. - mooncake_ib_device: mlx5_1 + # The stable atom-v0.1.1 image bundles Mooncake before ROCm DMA-BUF memory + # registration support. Use its supported TCP transport for this correctness + # lane; connection pooling is supplied by the srt-slurm ATOM adapter. + mooncake_protocol: tcp enable_kv_events: true prefill_environment: &worker_environment HF_HOME: "/hf_hub_cache" diff --git a/benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi355x/disagg-1p1d-fixed-seq.yaml b/benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi355x/disagg-1p1d-fixed-seq.yaml index d2f2f6c4c2..8d4b0f798a 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi355x/disagg-1p1d-fixed-seq.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi355x/disagg-1p1d-fixed-seq.yaml @@ -42,6 +42,10 @@ frontend: backend: type: atom connector: mooncake + # The stable atom-v0.1.1 image predates Mooncake's ROCm DMA-BUF RDMA fix. + # Explicit TCP still exercises the real P/D KV path without false RDMA + # success; srt-slurm enables Mooncake connection pooling for this mode. + mooncake_protocol: tcp enable_kv_events: true prefill_environment: &worker_environment HF_HOME: "/hf_hub_cache" diff --git a/runners/launch_mi300x-amds-srt.sh b/runners/launch_mi300x-amds-srt.sh index ad73cc56de..0d23d88f32 100644 --- a/runners/launch_mi300x-amds-srt.sh +++ b/runners/launch_mi300x-amds-srt.sh @@ -4,7 +4,7 @@ set -euo pipefail # MI300X srt-slurm validation path. The existing launcher remains the default; # matrix rows opt in by exporting CONFIG_FILE through additional-settings. SRT_SLURM_REPOSITORY="https://github.com/SemiAnalysisAI/srt-slurm.git" -SRT_SLURM_COMMIT="f71cbb1cd7a4247ce50d54d84254c1842258b6e3" +SRT_SLURM_COMMIT="83ef50ba6bf0bf4a91341d31fdcded141fdaa1b4" INFERA_REPOSITORY="https://github.com/cquil11/Infera.git" INFERA_COMMIT="8ed8f1728c745d4e91ba9eaa09ed81159aa57e41" SLURM_PARTITION="compute" diff --git a/runners/launch_mi355x-amds-srt.sh b/runners/launch_mi355x-amds-srt.sh index 90161a293d..8eab043b9b 100755 --- a/runners/launch_mi355x-amds-srt.sh +++ b/runners/launch_mi355x-amds-srt.sh @@ -5,7 +5,7 @@ set -euo pipefail # in explicitly with CONFIG_FILE; all existing MI355X launch behavior remains # unchanged for every other row. SRT_SLURM_REPOSITORY="https://github.com/SemiAnalysisAI/srt-slurm.git" -SRT_SLURM_COMMIT="f71cbb1cd7a4247ce50d54d84254c1842258b6e3" +SRT_SLURM_COMMIT="83ef50ba6bf0bf4a91341d31fdcded141fdaa1b4" INFERA_REPOSITORY="https://github.com/cquil11/Infera.git" INFERA_COMMIT="8ed8f1728c745d4e91ba9eaa09ed81159aa57e41" SLURM_PARTITION="compute" diff --git a/utils/test_mi300x_srt_slurm_contract.py b/utils/test_mi300x_srt_slurm_contract.py index 5b762bcd9a..e62db538f2 100644 --- a/utils/test_mi300x_srt_slurm_contract.py +++ b/utils/test_mi300x_srt_slurm_contract.py @@ -97,7 +97,7 @@ def test_official_matrix_routes_disagg_through_the_pinned_srt_launcher(): "CONFIG_FILE=recipes/vllm/qwen3-0.6b/mi300x/" "disagg-1p1d-fixed-seq.yaml" ] - assert "f71cbb1cd7a4247ce50d54d84254c1842258b6e3" in launcher + assert "83ef50ba6bf0bf4a91341d31fdcded141fdaa1b4" in launcher assert launcher.count("setup ARCH=x86_64") == 2 assert "--no-preflight" in launcher assert 'ENROOT_RUNTIME_PATH="\\${TMPDIR:-/tmp}/enroot-runtime-\\${UID}"' in launcher @@ -225,7 +225,7 @@ def test_atom_recipes_use_infera_and_keep_worker_metrics_honest(): assert "--backend openai" in command assert "--endpoint /v1/completions" in command assert disaggregate["backend"]["connector"] == "mooncake" - assert disaggregate["backend"]["mooncake_ib_device"] == "mlx5_1" + assert disaggregate["backend"]["mooncake_protocol"] == "tcp" assert 'ATOM_IMAGE="rocm/infera:atom-v0.1.1"' in launcher assert 'INFERA_COMMIT="8ed8f1728c745d4e91ba9eaa09ed81159aa57e41"' in launcher assert 'REMOTE_INFERA_RUNTIME="${REMOTE_BASE}/runtime/infera-${INFERA_COMMIT}"' in launcher diff --git a/utils/test_mi355x_atom_srt_contract.py b/utils/test_mi355x_atom_srt_contract.py index 211ba20c0a..57e91dfe5b 100644 --- a/utils/test_mi355x_atom_srt_contract.py +++ b/utils/test_mi355x_atom_srt_contract.py @@ -47,6 +47,7 @@ def test_disaggregate_uses_one_prefill_one_decode_and_mooncake(): assert recipe["resources"]["decode_nodes"] == 1 assert recipe["backend"]["type"] == "atom" assert recipe["backend"]["connector"] == "mooncake" + assert recipe["backend"]["mooncake_protocol"] == "tcp" assert recipe["backend"]["enable_kv_events"] is True assert recipe["backend"]["prefill_environment"]["PYTHONPATH"] == ( "/infera-source" @@ -73,6 +74,6 @@ def test_matrix_rows_route_only_through_the_atom_srt_launcher(): ]["additional-settings"] == [ "CONFIG_FILE=recipes/atom/qwen3-0.6b/mi355x/disagg-1p1d-fixed-seq.yaml" ] - assert "f71cbb1cd7a4247ce50d54d84254c1842258b6e3" in launcher + assert "83ef50ba6bf0bf4a91341d31fdcded141fdaa1b4" in launcher assert "8ed8f1728c745d4e91ba9eaa09ed81159aa57e41" in launcher assert "scancel" not in launcher From 6c5311ad52ee0c029cbbc24384b615428b9e0562 Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Tue, 11 Aug 2026 12:57:22 -0500 Subject: [PATCH 61/67] fix(atom): verify installed framework versions --- .../atom/qwen3-0.6b/mi300x/agg-2w-fixed-seq.yaml | 4 ++-- .../atom/qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml | 5 ++--- .../atom/qwen3-0.6b/mi355x/agg-2w-fixed-seq.yaml | 4 ++-- .../atom/qwen3-0.6b/mi355x/disagg-1p1d-fixed-seq.yaml | 5 ++--- runners/launch_mi300x-amds-srt.sh | 2 +- runners/launch_mi355x-amds-srt.sh | 2 +- utils/test_mi300x_srt_slurm_contract.py | 2 +- utils/test_mi355x_atom_srt_contract.py | 2 +- 8 files changed, 12 insertions(+), 14 deletions(-) diff --git a/benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi300x/agg-2w-fixed-seq.yaml b/benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi300x/agg-2w-fixed-seq.yaml index e3f368aa22..6d137bb052 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi300x/agg-2w-fixed-seq.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi300x/agg-2w-fixed-seq.yaml @@ -14,8 +14,8 @@ identity: container: image: "rocm/infera:atom-v0.1.1" frameworks: - atom: "0.1.1" - infera: "0.1.1" + atom: "0.1.4.dev113+g5837907f3" + infera: "0.0.0" slurm: time_limit: "00:45:00" diff --git a/benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml b/benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml index 5e27f67171..732a21bcc0 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml @@ -14,9 +14,8 @@ identity: container: image: "rocm/infera:atom-v0.1.1" frameworks: - atom: "0.1.1" - infera: "0.1.1" - mooncake: "bundled" + atom: "0.1.4.dev113+g5837907f3" + infera: "0.0.0" slurm: time_limit: "00:45:00" diff --git a/benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi355x/agg-2w-fixed-seq.yaml b/benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi355x/agg-2w-fixed-seq.yaml index 340b2071bf..526e783a74 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi355x/agg-2w-fixed-seq.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi355x/agg-2w-fixed-seq.yaml @@ -14,8 +14,8 @@ identity: container: image: "rocm/infera:atom-v0.1.1" frameworks: - atom: "0.1.1" - infera: "0.1.1" + atom: "0.1.4.dev113+g5837907f3" + infera: "0.0.0" slurm: time_limit: "00:30:00" diff --git a/benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi355x/disagg-1p1d-fixed-seq.yaml b/benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi355x/disagg-1p1d-fixed-seq.yaml index 8d4b0f798a..8147ee22f9 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi355x/disagg-1p1d-fixed-seq.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi355x/disagg-1p1d-fixed-seq.yaml @@ -14,9 +14,8 @@ identity: container: image: "rocm/infera:atom-v0.1.1" frameworks: - atom: "0.1.1" - infera: "0.1.1" - mooncake: "bundled" + atom: "0.1.4.dev113+g5837907f3" + infera: "0.0.0" slurm: time_limit: "00:45:00" diff --git a/runners/launch_mi300x-amds-srt.sh b/runners/launch_mi300x-amds-srt.sh index 0d23d88f32..9a97810c86 100644 --- a/runners/launch_mi300x-amds-srt.sh +++ b/runners/launch_mi300x-amds-srt.sh @@ -4,7 +4,7 @@ set -euo pipefail # MI300X srt-slurm validation path. The existing launcher remains the default; # matrix rows opt in by exporting CONFIG_FILE through additional-settings. SRT_SLURM_REPOSITORY="https://github.com/SemiAnalysisAI/srt-slurm.git" -SRT_SLURM_COMMIT="83ef50ba6bf0bf4a91341d31fdcded141fdaa1b4" +SRT_SLURM_COMMIT="141f035b5539fa8bbc1b4018ae4817283093092d" INFERA_REPOSITORY="https://github.com/cquil11/Infera.git" INFERA_COMMIT="8ed8f1728c745d4e91ba9eaa09ed81159aa57e41" SLURM_PARTITION="compute" diff --git a/runners/launch_mi355x-amds-srt.sh b/runners/launch_mi355x-amds-srt.sh index 8eab043b9b..1b54854894 100755 --- a/runners/launch_mi355x-amds-srt.sh +++ b/runners/launch_mi355x-amds-srt.sh @@ -5,7 +5,7 @@ set -euo pipefail # in explicitly with CONFIG_FILE; all existing MI355X launch behavior remains # unchanged for every other row. SRT_SLURM_REPOSITORY="https://github.com/SemiAnalysisAI/srt-slurm.git" -SRT_SLURM_COMMIT="83ef50ba6bf0bf4a91341d31fdcded141fdaa1b4" +SRT_SLURM_COMMIT="141f035b5539fa8bbc1b4018ae4817283093092d" INFERA_REPOSITORY="https://github.com/cquil11/Infera.git" INFERA_COMMIT="8ed8f1728c745d4e91ba9eaa09ed81159aa57e41" SLURM_PARTITION="compute" diff --git a/utils/test_mi300x_srt_slurm_contract.py b/utils/test_mi300x_srt_slurm_contract.py index e62db538f2..7431e41e25 100644 --- a/utils/test_mi300x_srt_slurm_contract.py +++ b/utils/test_mi300x_srt_slurm_contract.py @@ -97,7 +97,7 @@ def test_official_matrix_routes_disagg_through_the_pinned_srt_launcher(): "CONFIG_FILE=recipes/vllm/qwen3-0.6b/mi300x/" "disagg-1p1d-fixed-seq.yaml" ] - assert "83ef50ba6bf0bf4a91341d31fdcded141fdaa1b4" in launcher + assert "141f035b5539fa8bbc1b4018ae4817283093092d" in launcher assert launcher.count("setup ARCH=x86_64") == 2 assert "--no-preflight" in launcher assert 'ENROOT_RUNTIME_PATH="\\${TMPDIR:-/tmp}/enroot-runtime-\\${UID}"' in launcher diff --git a/utils/test_mi355x_atom_srt_contract.py b/utils/test_mi355x_atom_srt_contract.py index 57e91dfe5b..37ff0dbf26 100644 --- a/utils/test_mi355x_atom_srt_contract.py +++ b/utils/test_mi355x_atom_srt_contract.py @@ -74,6 +74,6 @@ def test_matrix_rows_route_only_through_the_atom_srt_launcher(): ]["additional-settings"] == [ "CONFIG_FILE=recipes/atom/qwen3-0.6b/mi355x/disagg-1p1d-fixed-seq.yaml" ] - assert "83ef50ba6bf0bf4a91341d31fdcded141fdaa1b4" in launcher + assert "141f035b5539fa8bbc1b4018ae4817283093092d" in launcher assert "8ed8f1728c745d4e91ba9eaa09ed81159aa57e41" in launcher assert "scancel" not in launcher From d71afb37f872299ab322a0f03a3b03302c78246b Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Tue, 11 Aug 2026 12:59:06 -0500 Subject: [PATCH 62/67] docs(atom): record validated TCP transport --- perf-changelog.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/perf-changelog.yaml b/perf-changelog.yaml index 98e40cc7a3..fe5dbc03f1 100644 --- a/perf-changelog.yaml +++ b/perf-changelog.yaml @@ -5796,7 +5796,7 @@ - qwen3-0.6b-fp16-mi355x-atom-infera-srt-disagg description: - "Add srt-slurm correctness lanes for the ROCm ATOM engine and the open-source Infera successor to ATOMesh." - - "Validate two aggregate ATOM workers behind Infera KV-aware routing and a two-node 1P/1D ATOM deployment using Mooncake RDMA." - - "Use rocm/infera:atom-v0.1.1 with Qwen3-0.6B fixed-sequence requests on MI300X and MI355X, and pin the combined AMD, vLLM Router, ATOM, and Infera srt-slurm branch." + - "Validate two aggregate ATOM workers behind Infera KV-aware routing and a two-node 1P/1D ATOM deployment using Mooncake TCP with connection pooling." + - "Use rocm/infera:atom-v0.1.1 with Qwen3-0.6B fixed-sequence requests on MI300X and MI355X, verify the installed ATOM and Infera distributions, and pin the combined AMD, vLLM Router, ATOM, and Infera srt-slurm branch." - "Overlay the exact upstream Infera KV-event decoder fix from AMD-AGI/Infera#114 so ATOM's tagged-map events populate the router cache view during pre-release hardware validation." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2558 From 2e2f3756fff53499df24bd222233408fab30d12a Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Tue, 11 Aug 2026 13:22:25 -0500 Subject: [PATCH 63/67] fix(atom): pin tcp-only Mooncake source overlay --- .../atom/qwen3-0.6b/mi300x/agg-2w-fixed-seq.yaml | 4 ++-- .../qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml | 4 ++-- .../atom/qwen3-0.6b/mi355x/agg-2w-fixed-seq.yaml | 4 ++-- .../qwen3-0.6b/mi355x/disagg-1p1d-fixed-seq.yaml | 4 ++-- runners/launch_mi300x-amds-srt.sh | 12 +++++++++--- runners/launch_mi355x-amds-srt.sh | 16 +++++++++++++--- utils/test_mi300x_srt_slurm_contract.py | 10 ++++++++-- utils/test_mi355x_atom_srt_contract.py | 5 +++-- 8 files changed, 41 insertions(+), 18 deletions(-) diff --git a/benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi300x/agg-2w-fixed-seq.yaml b/benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi300x/agg-2w-fixed-seq.yaml index 6d137bb052..0f557e4992 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi300x/agg-2w-fixed-seq.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi300x/agg-2w-fixed-seq.yaml @@ -31,7 +31,7 @@ frontend: type: infera enable_multiple_frontends: false env: - PYTHONPATH: "/infera-source" + PYTHONPATH: "/atom-source:/infera-source" args: router-policy: kv-aware @@ -42,7 +42,7 @@ backend: HF_HOME: "/hf_hub_cache" HF_HUB_CACHE: "/hf_hub_cache" HUGGINGFACE_HUB_CACHE: "/hf_hub_cache" - PYTHONPATH: "/infera-source" + PYTHONPATH: "/atom-source:/infera-source" PYTHONUNBUFFERED: "1" OMP_NUM_THREADS: "1" atom_config: diff --git a/benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml b/benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml index 732a21bcc0..87a1e9e528 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi300x/disagg-1p1d-fixed-seq.yaml @@ -34,7 +34,7 @@ frontend: type: infera enable_multiple_frontends: false env: - PYTHONPATH: "/infera-source" + PYTHONPATH: "/atom-source:/infera-source" args: router-policy: kv-aware @@ -50,7 +50,7 @@ backend: HF_HOME: "/hf_hub_cache" HF_HUB_CACHE: "/hf_hub_cache" HUGGINGFACE_HUB_CACHE: "/hf_hub_cache" - PYTHONPATH: "/infera-source" + PYTHONPATH: "/atom-source:/infera-source" PYTHONUNBUFFERED: "1" OMP_NUM_THREADS: "1" decode_environment: *worker_environment diff --git a/benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi355x/agg-2w-fixed-seq.yaml b/benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi355x/agg-2w-fixed-seq.yaml index 526e783a74..54fc657e3c 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi355x/agg-2w-fixed-seq.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi355x/agg-2w-fixed-seq.yaml @@ -31,7 +31,7 @@ frontend: type: infera enable_multiple_frontends: false env: - PYTHONPATH: "/infera-source" + PYTHONPATH: "/atom-source:/infera-source" args: router-policy: kv-aware @@ -42,7 +42,7 @@ backend: HF_HOME: "/hf_hub_cache" HF_HUB_CACHE: "/hf_hub_cache/hub" HUGGINGFACE_HUB_CACHE: "/hf_hub_cache/hub" - PYTHONPATH: "/infera-source" + PYTHONPATH: "/atom-source:/infera-source" PYTHONUNBUFFERED: "1" OMP_NUM_THREADS: "1" atom_config: diff --git a/benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi355x/disagg-1p1d-fixed-seq.yaml b/benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi355x/disagg-1p1d-fixed-seq.yaml index 8147ee22f9..a5cad0d691 100644 --- a/benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi355x/disagg-1p1d-fixed-seq.yaml +++ b/benchmarks/multi_node/srt-slurm-recipes/atom/qwen3-0.6b/mi355x/disagg-1p1d-fixed-seq.yaml @@ -34,7 +34,7 @@ frontend: type: infera enable_multiple_frontends: false env: - PYTHONPATH: "/infera-source" + PYTHONPATH: "/atom-source:/infera-source" args: router-policy: kv-aware @@ -50,7 +50,7 @@ backend: HF_HOME: "/hf_hub_cache" HF_HUB_CACHE: "/hf_hub_cache/hub" HUGGINGFACE_HUB_CACHE: "/hf_hub_cache/hub" - PYTHONPATH: "/infera-source" + PYTHONPATH: "/atom-source:/infera-source" PYTHONUNBUFFERED: "1" OMP_NUM_THREADS: "1" decode_environment: *worker_environment diff --git a/runners/launch_mi300x-amds-srt.sh b/runners/launch_mi300x-amds-srt.sh index 9a97810c86..69c7755f43 100644 --- a/runners/launch_mi300x-amds-srt.sh +++ b/runners/launch_mi300x-amds-srt.sh @@ -7,6 +7,8 @@ SRT_SLURM_REPOSITORY="https://github.com/SemiAnalysisAI/srt-slurm.git" SRT_SLURM_COMMIT="141f035b5539fa8bbc1b4018ae4817283093092d" INFERA_REPOSITORY="https://github.com/cquil11/Infera.git" INFERA_COMMIT="8ed8f1728c745d4e91ba9eaa09ed81159aa57e41" +ATOM_REPOSITORY="https://github.com/cquil11/ATOM.git" +ATOM_COMMIT="2ab42bcd9473095206d2bd2df263c56a0b6430d9" SLURM_PARTITION="compute" EXCLUDED_NODES="chi-mi300x-049,chi-mi300x-121" REMOTE_BASE="/raid/hf-hub-cache/inferencex/srt-slurm" @@ -51,6 +53,7 @@ RUN_KEY="${GITHUB_RUN_ID:-manual}-${GITHUB_RUN_ATTEMPT:-0}-${RUNNER_NAME:-runner REMOTE_RUNTIME="${REMOTE_BASE}/runtime/inferencex-${RUN_KEY}" REMOTE_SRT_RUNTIME="${REMOTE_BASE}/runtime/srt-slurm-${SRT_SLURM_COMMIT}" REMOTE_INFERA_RUNTIME="${REMOTE_BASE}/runtime/infera-${INFERA_COMMIT}" +REMOTE_ATOM_RUNTIME="${REMOTE_BASE}/runtime/atom-${ATOM_COMMIT}" REMOTE_RESULTS="${REMOTE_BASE}/results" WORK_DIR="${GITHUB_WORKSPACE}/.srt-slurm-${RUN_KEY}" SRT_REPO_DIR="${WORK_DIR}/srt-slurm" @@ -84,6 +87,7 @@ srun --ntasks-per-node=1 bash -c ' runtime="${REMOTE_RUNTIME}" srt_runtime="${REMOTE_SRT_RUNTIME}" infera_runtime="${REMOTE_INFERA_RUNTIME}" + atom_runtime="${REMOTE_ATOM_RUNTIME}" export ENROOT_RUNTIME_PATH="\${TMPDIR:-/tmp}/enroot-runtime-\${UID}" mkdir -p "\$ENROOT_RUNTIME_PATH" "\$runtime" "${REMOTE_RESULTS}" "${REMOTE_BASE}/containers" chmod 700 "\$ENROOT_RUNTIME_PATH" @@ -145,6 +149,7 @@ srun --ntasks-per-node=1 bash -c ' ensure_git_checkout "\$srt_runtime" "${SRT_SLURM_REPOSITORY}" "${SRT_SLURM_COMMIT}" make -C "\$srt_runtime" --no-print-directory setup ARCH=x86_64 ensure_git_checkout "\$infera_runtime" "${INFERA_REPOSITORY}" "${INFERA_COMMIT}" + ensure_git_checkout "\$atom_runtime" "${ATOM_REPOSITORY}" "${ATOM_COMMIT}" tar -xzf "/tmp/inferencex-benchmark-\${SLURM_JOB_ID}.tar.gz" -C "\$runtime" printf "%s\\n" "${GITHUB_SHA:-unknown}" > "\$runtime/.inferencex-source-head" ' @@ -163,12 +168,12 @@ ACTUAL_SRT_COMMIT=$(git -C "$SRT_REPO_DIR" rev-parse HEAD) mkdir -p "${SRT_REPO_DIR}/$(dirname "$CONFIG_PATH")" cp "$LOCAL_RECIPE" "${SRT_REPO_DIR}/${CONFIG_PATH}" cp "$CLUSTER_PROFILE" "${WORK_DIR}/srtslurm.yaml" -python3 - "${WORK_DIR}/srtslurm.yaml" "$REMOTE_RUNTIME" "$REMOTE_RESULTS" "$REMOTE_INFERA_RUNTIME" <<'PY' +python3 - "${WORK_DIR}/srtslurm.yaml" "$REMOTE_RUNTIME" "$REMOTE_RESULTS" "$REMOTE_INFERA_RUNTIME" "$REMOTE_ATOM_RUNTIME" <<'PY' import sys from pathlib import Path path = Path(sys.argv[1]) -runtime, results, infera_runtime = sys.argv[2:] +runtime, results, infera_runtime, atom_runtime = sys.argv[2:] needle = " /raid/hf-hub-cache: /hf_hub_cache\n" text = path.read_text() if text.count(needle) != 1: @@ -179,7 +184,8 @@ path.write_text( needle + f" {runtime}: /infmax-workspace\n" + f" {results}: /results\n" - + f" {infera_runtime}: /infera-source\n", + + f" {infera_runtime}: /infera-source\n" + + f" {atom_runtime}: /atom-source\n", ) ) PY diff --git a/runners/launch_mi355x-amds-srt.sh b/runners/launch_mi355x-amds-srt.sh index 1b54854894..23b10a0579 100755 --- a/runners/launch_mi355x-amds-srt.sh +++ b/runners/launch_mi355x-amds-srt.sh @@ -8,6 +8,8 @@ SRT_SLURM_REPOSITORY="https://github.com/SemiAnalysisAI/srt-slurm.git" SRT_SLURM_COMMIT="141f035b5539fa8bbc1b4018ae4817283093092d" INFERA_REPOSITORY="https://github.com/cquil11/Infera.git" INFERA_COMMIT="8ed8f1728c745d4e91ba9eaa09ed81159aa57e41" +ATOM_REPOSITORY="https://github.com/cquil11/ATOM.git" +ATOM_COMMIT="2ab42bcd9473095206d2bd2df263c56a0b6430d9" SLURM_PARTITION="compute" SGLANG_IMAGE="lmsysorg/sglang-rocm:v0.5.17-rocm720-mi35x-20260809" ATOM_IMAGE="rocm/infera:atom-v0.1.1" @@ -15,6 +17,7 @@ SHARED_BASE="/it-share/gharunners2/srt-slurm" SHARED_HF_CACHE="/it-share/hf-hub-cache" SHARED_RESULTS="${SHARED_BASE}/results" SHARED_INFERA_RUNTIME="${SHARED_BASE}/runtime/infera-${INFERA_COMMIT}" +SHARED_ATOM_RUNTIME="${SHARED_BASE}/runtime/atom-${ATOM_COMMIT}" : "${GITHUB_WORKSPACE:?GITHUB_WORKSPACE must be set by Actions}" : "${RESULT_FILENAME:?RESULT_FILENAME must be set by the benchmark workflow}" @@ -72,6 +75,12 @@ fi git -C "${SHARED_INFERA_RUNTIME}" fetch --quiet origin "${INFERA_COMMIT}" git -C "${SHARED_INFERA_RUNTIME}" checkout --quiet --detach "${INFERA_COMMIT}" test "\$(git -C "${SHARED_INFERA_RUNTIME}" rev-parse HEAD)" = "${INFERA_COMMIT}" +if [[ ! -d "${SHARED_ATOM_RUNTIME}/.git" ]]; then + git clone --quiet "${ATOM_REPOSITORY}" "${SHARED_ATOM_RUNTIME}" +fi +git -C "${SHARED_ATOM_RUNTIME}" fetch --quiet origin "${ATOM_COMMIT}" +git -C "${SHARED_ATOM_RUNTIME}" checkout --quiet --detach "${ATOM_COMMIT}" +test "\$(git -C "${SHARED_ATOM_RUNTIME}" rev-parse HEAD)" = "${ATOM_COMMIT}" exec 9>"${SHARED_IMAGE}.lock" flock -w 2400 9 if ! unsquashfs -s "$SHARED_IMAGE" >/dev/null 2>&1; then @@ -119,12 +128,12 @@ ACTUAL_SRT_COMMIT=$(git -C "$SRT_REPO_DIR" rev-parse HEAD) mkdir -p "${SRT_REPO_DIR}/$(dirname "$CONFIG_PATH")" cp "$LOCAL_RECIPE" "${SRT_REPO_DIR}/${CONFIG_PATH}" cp "$CLUSTER_PROFILE" "${WORK_DIR}/srtslurm.yaml" -python3 - "${WORK_DIR}/srtslurm.yaml" "$GITHUB_WORKSPACE" "$SHARED_RESULTS" "$SHARED_INFERA_RUNTIME" <<'PY' +python3 - "${WORK_DIR}/srtslurm.yaml" "$GITHUB_WORKSPACE" "$SHARED_RESULTS" "$SHARED_INFERA_RUNTIME" "$SHARED_ATOM_RUNTIME" <<'PY' import sys from pathlib import Path path = Path(sys.argv[1]) -workspace, results, infera_runtime = sys.argv[2:] +workspace, results, infera_runtime, atom_runtime = sys.argv[2:] needle = " /it-share/hf-hub-cache: /hf_hub_cache\n" text = path.read_text() if text.count(needle) != 1: @@ -135,7 +144,8 @@ path.write_text( needle + f" {workspace}: /infmax-workspace\n" + f" {results}: /results\n" - + f" {infera_runtime}: /infera-source\n", + + f" {infera_runtime}: /infera-source\n" + + f" {atom_runtime}: /atom-source\n", ) ) PY diff --git a/utils/test_mi300x_srt_slurm_contract.py b/utils/test_mi300x_srt_slurm_contract.py index 7431e41e25..bf4fd4a2cc 100644 --- a/utils/test_mi300x_srt_slurm_contract.py +++ b/utils/test_mi300x_srt_slurm_contract.py @@ -228,16 +228,22 @@ def test_atom_recipes_use_infera_and_keep_worker_metrics_honest(): assert disaggregate["backend"]["mooncake_protocol"] == "tcp" assert 'ATOM_IMAGE="rocm/infera:atom-v0.1.1"' in launcher assert 'INFERA_COMMIT="8ed8f1728c745d4e91ba9eaa09ed81159aa57e41"' in launcher + assert 'ATOM_COMMIT="2ab42bcd9473095206d2bd2df263c56a0b6430d9"' in launcher assert 'REMOTE_INFERA_RUNTIME="${REMOTE_BASE}/runtime/infera-${INFERA_COMMIT}"' in launcher for recipe in (aggregate, disaggregate): - assert recipe["frontend"]["env"]["PYTHONPATH"] == "/infera-source" + assert recipe["frontend"]["env"]["PYTHONPATH"] == ( + "/atom-source:/infera-source" + ) role_environments = [ value for key, value in recipe["backend"].items() if key.endswith("_environment") ] assert role_environments - assert all(env["PYTHONPATH"] == "/infera-source" for env in role_environments) + assert all( + env["PYTHONPATH"] == "/atom-source:/infera-source" + for env in role_environments + ) def test_fixed_sequence_commands_keep_all_arguments_attached(tmp_path): diff --git a/utils/test_mi355x_atom_srt_contract.py b/utils/test_mi355x_atom_srt_contract.py index 37ff0dbf26..dd2a8e32b7 100644 --- a/utils/test_mi355x_atom_srt_contract.py +++ b/utils/test_mi355x_atom_srt_contract.py @@ -31,7 +31,7 @@ def test_aggregate_exercises_two_atom_workers_and_kv_aware_infera(): assert recipe["backend"]["type"] == "atom" assert recipe["backend"]["enable_kv_events"] is True assert recipe["backend"]["aggregated_environment"]["PYTHONPATH"] == ( - "/infera-source" + "/atom-source:/infera-source" ) command = recipe["benchmark"]["command"] assert "--random-prefix-len 96" in command @@ -50,7 +50,7 @@ def test_disaggregate_uses_one_prefill_one_decode_and_mooncake(): assert recipe["backend"]["mooncake_protocol"] == "tcp" assert recipe["backend"]["enable_kv_events"] is True assert recipe["backend"]["prefill_environment"]["PYTHONPATH"] == ( - "/infera-source" + "/atom-source:/infera-source" ) @@ -76,4 +76,5 @@ def test_matrix_rows_route_only_through_the_atom_srt_launcher(): ] assert "141f035b5539fa8bbc1b4018ae4817283093092d" in launcher assert "8ed8f1728c745d4e91ba9eaa09ed81159aa57e41" in launcher + assert "2ab42bcd9473095206d2bd2df263c56a0b6430d9" in launcher assert "scancel" not in launcher From bdc0e2043115679ba41fb0a143b3ced0e54594fa Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Tue, 11 Aug 2026 13:28:29 -0500 Subject: [PATCH 64/67] fix: pin fetchable stable ATOM overlay --- runners/launch_mi300x-amds-srt.sh | 2 +- runners/launch_mi355x-amds-srt.sh | 2 +- utils/test_mi300x_srt_slurm_contract.py | 2 +- utils/test_mi355x_atom_srt_contract.py | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/runners/launch_mi300x-amds-srt.sh b/runners/launch_mi300x-amds-srt.sh index 69c7755f43..825395f823 100644 --- a/runners/launch_mi300x-amds-srt.sh +++ b/runners/launch_mi300x-amds-srt.sh @@ -8,7 +8,7 @@ SRT_SLURM_COMMIT="141f035b5539fa8bbc1b4018ae4817283093092d" INFERA_REPOSITORY="https://github.com/cquil11/Infera.git" INFERA_COMMIT="8ed8f1728c745d4e91ba9eaa09ed81159aa57e41" ATOM_REPOSITORY="https://github.com/cquil11/ATOM.git" -ATOM_COMMIT="2ab42bcd9473095206d2bd2df263c56a0b6430d9" +ATOM_COMMIT="2ab42bc2c64d1ad04f698c396da48473e71a6dbb" SLURM_PARTITION="compute" EXCLUDED_NODES="chi-mi300x-049,chi-mi300x-121" REMOTE_BASE="/raid/hf-hub-cache/inferencex/srt-slurm" diff --git a/runners/launch_mi355x-amds-srt.sh b/runners/launch_mi355x-amds-srt.sh index 23b10a0579..4b9ebf03df 100755 --- a/runners/launch_mi355x-amds-srt.sh +++ b/runners/launch_mi355x-amds-srt.sh @@ -9,7 +9,7 @@ SRT_SLURM_COMMIT="141f035b5539fa8bbc1b4018ae4817283093092d" INFERA_REPOSITORY="https://github.com/cquil11/Infera.git" INFERA_COMMIT="8ed8f1728c745d4e91ba9eaa09ed81159aa57e41" ATOM_REPOSITORY="https://github.com/cquil11/ATOM.git" -ATOM_COMMIT="2ab42bcd9473095206d2bd2df263c56a0b6430d9" +ATOM_COMMIT="2ab42bc2c64d1ad04f698c396da48473e71a6dbb" SLURM_PARTITION="compute" SGLANG_IMAGE="lmsysorg/sglang-rocm:v0.5.17-rocm720-mi35x-20260809" ATOM_IMAGE="rocm/infera:atom-v0.1.1" diff --git a/utils/test_mi300x_srt_slurm_contract.py b/utils/test_mi300x_srt_slurm_contract.py index bf4fd4a2cc..793e8151ef 100644 --- a/utils/test_mi300x_srt_slurm_contract.py +++ b/utils/test_mi300x_srt_slurm_contract.py @@ -228,7 +228,7 @@ def test_atom_recipes_use_infera_and_keep_worker_metrics_honest(): assert disaggregate["backend"]["mooncake_protocol"] == "tcp" assert 'ATOM_IMAGE="rocm/infera:atom-v0.1.1"' in launcher assert 'INFERA_COMMIT="8ed8f1728c745d4e91ba9eaa09ed81159aa57e41"' in launcher - assert 'ATOM_COMMIT="2ab42bcd9473095206d2bd2df263c56a0b6430d9"' in launcher + assert 'ATOM_COMMIT="2ab42bc2c64d1ad04f698c396da48473e71a6dbb"' in launcher assert 'REMOTE_INFERA_RUNTIME="${REMOTE_BASE}/runtime/infera-${INFERA_COMMIT}"' in launcher for recipe in (aggregate, disaggregate): assert recipe["frontend"]["env"]["PYTHONPATH"] == ( diff --git a/utils/test_mi355x_atom_srt_contract.py b/utils/test_mi355x_atom_srt_contract.py index dd2a8e32b7..9f50957fc5 100644 --- a/utils/test_mi355x_atom_srt_contract.py +++ b/utils/test_mi355x_atom_srt_contract.py @@ -76,5 +76,5 @@ def test_matrix_rows_route_only_through_the_atom_srt_launcher(): ] assert "141f035b5539fa8bbc1b4018ae4817283093092d" in launcher assert "8ed8f1728c745d4e91ba9eaa09ed81159aa57e41" in launcher - assert "2ab42bcd9473095206d2bd2df263c56a0b6430d9" in launcher + assert "2ab42bc2c64d1ad04f698c396da48473e71a6dbb" in launcher assert "scancel" not in launcher From f9bdbe1c583182be2071c99fe0bb22cdcb68efb7 Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Tue, 11 Aug 2026 13:38:09 -0500 Subject: [PATCH 65/67] fix: pin Mooncake TCP transport guard --- runners/launch_mi300x-amds-srt.sh | 2 +- runners/launch_mi355x-amds-srt.sh | 2 +- utils/test_mi300x_srt_slurm_contract.py | 2 +- utils/test_mi355x_atom_srt_contract.py | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/runners/launch_mi300x-amds-srt.sh b/runners/launch_mi300x-amds-srt.sh index 825395f823..6224e5d553 100644 --- a/runners/launch_mi300x-amds-srt.sh +++ b/runners/launch_mi300x-amds-srt.sh @@ -4,7 +4,7 @@ set -euo pipefail # MI300X srt-slurm validation path. The existing launcher remains the default; # matrix rows opt in by exporting CONFIG_FILE through additional-settings. SRT_SLURM_REPOSITORY="https://github.com/SemiAnalysisAI/srt-slurm.git" -SRT_SLURM_COMMIT="141f035b5539fa8bbc1b4018ae4817283093092d" +SRT_SLURM_COMMIT="5ecfb13d1ba0960045482f1ef006312d8729d37a" INFERA_REPOSITORY="https://github.com/cquil11/Infera.git" INFERA_COMMIT="8ed8f1728c745d4e91ba9eaa09ed81159aa57e41" ATOM_REPOSITORY="https://github.com/cquil11/ATOM.git" diff --git a/runners/launch_mi355x-amds-srt.sh b/runners/launch_mi355x-amds-srt.sh index 4b9ebf03df..61def8ba68 100755 --- a/runners/launch_mi355x-amds-srt.sh +++ b/runners/launch_mi355x-amds-srt.sh @@ -5,7 +5,7 @@ set -euo pipefail # in explicitly with CONFIG_FILE; all existing MI355X launch behavior remains # unchanged for every other row. SRT_SLURM_REPOSITORY="https://github.com/SemiAnalysisAI/srt-slurm.git" -SRT_SLURM_COMMIT="141f035b5539fa8bbc1b4018ae4817283093092d" +SRT_SLURM_COMMIT="5ecfb13d1ba0960045482f1ef006312d8729d37a" INFERA_REPOSITORY="https://github.com/cquil11/Infera.git" INFERA_COMMIT="8ed8f1728c745d4e91ba9eaa09ed81159aa57e41" ATOM_REPOSITORY="https://github.com/cquil11/ATOM.git" diff --git a/utils/test_mi300x_srt_slurm_contract.py b/utils/test_mi300x_srt_slurm_contract.py index 793e8151ef..0c88ec3111 100644 --- a/utils/test_mi300x_srt_slurm_contract.py +++ b/utils/test_mi300x_srt_slurm_contract.py @@ -97,7 +97,7 @@ def test_official_matrix_routes_disagg_through_the_pinned_srt_launcher(): "CONFIG_FILE=recipes/vllm/qwen3-0.6b/mi300x/" "disagg-1p1d-fixed-seq.yaml" ] - assert "141f035b5539fa8bbc1b4018ae4817283093092d" in launcher + assert "5ecfb13d1ba0960045482f1ef006312d8729d37a" in launcher assert launcher.count("setup ARCH=x86_64") == 2 assert "--no-preflight" in launcher assert 'ENROOT_RUNTIME_PATH="\\${TMPDIR:-/tmp}/enroot-runtime-\\${UID}"' in launcher diff --git a/utils/test_mi355x_atom_srt_contract.py b/utils/test_mi355x_atom_srt_contract.py index 9f50957fc5..9624d08bbb 100644 --- a/utils/test_mi355x_atom_srt_contract.py +++ b/utils/test_mi355x_atom_srt_contract.py @@ -74,7 +74,7 @@ def test_matrix_rows_route_only_through_the_atom_srt_launcher(): ]["additional-settings"] == [ "CONFIG_FILE=recipes/atom/qwen3-0.6b/mi355x/disagg-1p1d-fixed-seq.yaml" ] - assert "141f035b5539fa8bbc1b4018ae4817283093092d" in launcher + assert "5ecfb13d1ba0960045482f1ef006312d8729d37a" in launcher assert "8ed8f1728c745d4e91ba9eaa09ed81159aa57e41" in launcher assert "2ab42bc2c64d1ad04f698c396da48473e71a6dbb" in launcher assert "scancel" not in launcher From eb814bbf0894e17132d43b241e4671388e698ba9 Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Tue, 11 Aug 2026 14:01:46 -0500 Subject: [PATCH 66/67] fix(ci): restore e2e eval matrix output --- .github/workflows/e2e-tests.yml | 1 + utils/changelog_gate_tests/test_run_sweep_gating.py | 8 ++++++++ 2 files changed, 9 insertions(+) diff --git a/.github/workflows/e2e-tests.yml b/.github/workflows/e2e-tests.yml index c973df94e6..d90f3fc543 100644 --- a/.github/workflows/e2e-tests.yml +++ b/.github/workflows/e2e-tests.yml @@ -261,6 +261,7 @@ jobs: MULTI_AGENTIC_EVAL=$(echo "$CONFIG_JSON" | python3 -c "import sys,json; d=json.load(sys.stdin); print(json.dumps([x for x in d if x.get('scenario-type') == 'agentic-coding' and 'prefill' in x and x.get('run-eval', False)]))" | score_matrix multi-agentic-eval) SINGLE=$(echo "$CONFIG_JSON" | python3 -c "import sys,json; d=json.load(sys.stdin); print(json.dumps([x for x in d if 'prefill' not in x and x.get('scenario-type') != 'agentic-coding' and not x.get('eval-only', False)]))" | score_matrix single) MULTI=$(echo "$CONFIG_JSON" | python3 -c "import sys,json; d=json.load(sys.stdin); print(json.dumps([x for x in d if 'prefill' in x and x.get('scenario-type') != 'agentic-coding' and not x.get('eval-only', False)]))" | score_matrix multi) + EVALS=$(echo "$CONFIG_JSON" | python3 -c "import sys,json; d=json.load(sys.stdin); print(json.dumps([x for x in d if 'prefill' not in x and x.get('scenario-type') != 'agentic-coding' and x.get('run-eval', False)]))" | score_matrix eval) MULTI_EVAL=$(echo "$CONFIG_JSON" | python3 -c "import sys,json; d=json.load(sys.stdin); print(json.dumps([x for x in d if 'prefill' in x and x.get('scenario-type') != 'agentic-coding' and x.get('run-eval', False)]))" | score_matrix multi-eval) { echo "agentic-config=$AGENTIC" diff --git a/utils/changelog_gate_tests/test_run_sweep_gating.py b/utils/changelog_gate_tests/test_run_sweep_gating.py index ed30ec5ad9..a2ecad93d4 100644 --- a/utils/changelog_gate_tests/test_run_sweep_gating.py +++ b/utils/changelog_gate_tests/test_run_sweep_gating.py @@ -417,6 +417,14 @@ def test_e2e_workflow_cannot_dispatch_database_ingest() -> None: assert "INFX_FRONTEND_PAT" not in workflow +def test_e2e_workflow_defines_every_emitted_matrix_variable() -> None: + workflow = (REPO_ROOT / ".github/workflows/e2e-tests.yml").read_text() + assigned = set(re.findall(r"^\s+([A-Z_]+)=", workflow, re.MULTILINE)) + emitted = set(re.findall(r'echo "[a-z-]+=\$([A-Z_]+)"', workflow)) + + assert emitted <= assigned + + def test_priority_classifier_runs_for_enabled_actions() -> None: scenario = { **_PR, From 4a98aa9c9719e786b5d443af80de5ae6bc6de63b Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Sun, 16 Aug 2026 23:14:08 -0500 Subject: [PATCH 67/67] fix(amd): serialize ATOM source staging --- perf-changelog.yaml | 1 + runners/launch_mi300x-amds-srt.sh | 11 +++++-- runners/launch_mi355x-amds-srt.sh | 42 ++++++++++++++++++------- utils/test_mi300x_srt_slurm_contract.py | 4 +-- utils/test_mi355x_atom_srt_contract.py | 4 +++ 5 files changed, 46 insertions(+), 16 deletions(-) diff --git a/perf-changelog.yaml b/perf-changelog.yaml index dcd617e1be..635670c080 100644 --- a/perf-changelog.yaml +++ b/perf-changelog.yaml @@ -5805,4 +5805,5 @@ - "Validate two aggregate ATOM workers behind Infera KV-aware routing and a two-node 1P/1D ATOM deployment using Mooncake TCP with connection pooling." - "Use rocm/infera:atom-v0.1.1 with Qwen3-0.6B fixed-sequence requests on MI300X and MI355X, verify the installed ATOM and Infera distributions, and pin the combined AMD, vLLM Router, ATOM, and Infera srt-slurm branch." - "Overlay the exact upstream Infera KV-event decoder fix from AMD-AGI/Infera#114 so ATOM's tagged-map events populate the router cache view during pre-release hardware validation." + - "Serialize shared Infera and ATOM source staging per checkout and publish completed clones atomically so concurrent aggregate and disaggregated validation jobs cannot corrupt one another." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2558 diff --git a/runners/launch_mi300x-amds-srt.sh b/runners/launch_mi300x-amds-srt.sh index 6224e5d553..31ab31d0bf 100644 --- a/runners/launch_mi300x-amds-srt.sh +++ b/runners/launch_mi300x-amds-srt.sh @@ -125,22 +125,29 @@ srun --ntasks-per-node=1 bash -c ' local target="\$1" local repository="\$2" local commit="\$3" + local lock_fd local temporary="\${target}.tmp.\${SLURM_JOB_ID}.\${BASHPID}" local quarantine="\${target}.incomplete.\${SLURM_JOB_ID}.\${BASHPID}" + + mkdir -p "\$(dirname "\$target")" + exec {lock_fd}>"\${target}.lock" + flock -w 2400 "\$lock_fd" if [[ ! -d "\$target/.git" ]]; then if [[ -e "\$target" ]]; then - mv "\$target" "\$quarantine" + mv -T "\$target" "\$quarantine" fi git clone --quiet "\$repository" "\$temporary" git -C "\$temporary" fetch --quiet origin "\$commit" git -C "\$temporary" checkout --quiet --detach "\$commit" test "\$(git -C "\$temporary" rev-parse HEAD)" = "\$commit" - mv "\$temporary" "\$target" + mv -T "\$temporary" "\$target" else git -C "\$target" fetch --quiet origin "\$commit" git -C "\$target" checkout --quiet --detach "\$commit" test "\$(git -C "\$target" rev-parse HEAD)" = "\$commit" fi + flock -u "\$lock_fd" + exec {lock_fd}>&- } ensure_container_image "${ENGINE_SQSH}" "${ENGINE_IMAGE}" if [[ -n "${AUX_IMAGE}" ]]; then diff --git a/runners/launch_mi355x-amds-srt.sh b/runners/launch_mi355x-amds-srt.sh index 61def8ba68..72ab623ae9 100755 --- a/runners/launch_mi355x-amds-srt.sh +++ b/runners/launch_mi355x-amds-srt.sh @@ -69,18 +69,36 @@ set -euo pipefail export ENROOT_RUNTIME_PATH="\${TMPDIR:-/tmp}/enroot-runtime-\${UID}" mkdir -p "\$ENROOT_RUNTIME_PATH" "$(dirname "$SHARED_IMAGE")" "$SHARED_HF_CACHE" chmod 700 "\$ENROOT_RUNTIME_PATH" -if [[ ! -d "${SHARED_INFERA_RUNTIME}/.git" ]]; then - git clone --quiet "${INFERA_REPOSITORY}" "${SHARED_INFERA_RUNTIME}" -fi -git -C "${SHARED_INFERA_RUNTIME}" fetch --quiet origin "${INFERA_COMMIT}" -git -C "${SHARED_INFERA_RUNTIME}" checkout --quiet --detach "${INFERA_COMMIT}" -test "\$(git -C "${SHARED_INFERA_RUNTIME}" rev-parse HEAD)" = "${INFERA_COMMIT}" -if [[ ! -d "${SHARED_ATOM_RUNTIME}/.git" ]]; then - git clone --quiet "${ATOM_REPOSITORY}" "${SHARED_ATOM_RUNTIME}" -fi -git -C "${SHARED_ATOM_RUNTIME}" fetch --quiet origin "${ATOM_COMMIT}" -git -C "${SHARED_ATOM_RUNTIME}" checkout --quiet --detach "${ATOM_COMMIT}" -test "\$(git -C "${SHARED_ATOM_RUNTIME}" rev-parse HEAD)" = "${ATOM_COMMIT}" +ensure_git_checkout() { + local target="\$1" + local repository="\$2" + local commit="\$3" + local lock_fd + local temporary="\${target}.tmp.\${SLURM_JOB_ID}.\${BASHPID}" + local quarantine="\${target}.incomplete.\${SLURM_JOB_ID}.\${BASHPID}" + + mkdir -p "\$(dirname "\$target")" + exec {lock_fd}>"\${target}.lock" + flock -w 2400 "\$lock_fd" + if [[ ! -d "\$target/.git" ]]; then + if [[ -e "\$target" ]]; then + mv -T "\$target" "\$quarantine" + fi + git clone --quiet "\$repository" "\$temporary" + git -C "\$temporary" fetch --quiet origin "\$commit" + git -C "\$temporary" checkout --quiet --detach "\$commit" + test "\$(git -C "\$temporary" rev-parse HEAD)" = "\$commit" + mv -T "\$temporary" "\$target" + else + git -C "\$target" fetch --quiet origin "\$commit" + git -C "\$target" checkout --quiet --detach "\$commit" + test "\$(git -C "\$target" rev-parse HEAD)" = "\$commit" + fi + flock -u "\$lock_fd" + exec {lock_fd}>&- +} +ensure_git_checkout "${SHARED_INFERA_RUNTIME}" "${INFERA_REPOSITORY}" "${INFERA_COMMIT}" +ensure_git_checkout "${SHARED_ATOM_RUNTIME}" "${ATOM_REPOSITORY}" "${ATOM_COMMIT}" exec 9>"${SHARED_IMAGE}.lock" flock -w 2400 9 if ! unsquashfs -s "$SHARED_IMAGE" >/dev/null 2>&1; then diff --git a/utils/test_mi300x_srt_slurm_contract.py b/utils/test_mi300x_srt_slurm_contract.py index 0c88ec3111..e27bac84a5 100644 --- a/utils/test_mi300x_srt_slurm_contract.py +++ b/utils/test_mi300x_srt_slurm_contract.py @@ -122,8 +122,8 @@ def test_official_matrix_routes_disagg_through_the_pinned_srt_launcher(): assert 'export SRTCTL_RUNTIME_SOURCE_DIR="$REMOTE_SRT_RUNTIME"' in launcher assert '#SBATCH --nodes=7' in launcher assert 'ensure_git_checkout()' in launcher - assert 'mv "\\$target" "\\$quarantine"' in launcher - assert 'mv "\\$temporary" "\\$target"' in launcher + assert 'mv -T "\\$target" "\\$quarantine"' in launcher + assert 'mv -T "\\$temporary" "\\$target"' in launcher assert "scancel" not in launcher diff --git a/utils/test_mi355x_atom_srt_contract.py b/utils/test_mi355x_atom_srt_contract.py index 9624d08bbb..50a9e7f449 100644 --- a/utils/test_mi355x_atom_srt_contract.py +++ b/utils/test_mi355x_atom_srt_contract.py @@ -77,4 +77,8 @@ def test_matrix_rows_route_only_through_the_atom_srt_launcher(): assert "5ecfb13d1ba0960045482f1ef006312d8729d37a" in launcher assert "8ed8f1728c745d4e91ba9eaa09ed81159aa57e41" in launcher assert "2ab42bc2c64d1ad04f698c396da48473e71a6dbb" in launcher + assert "ensure_git_checkout()" in launcher + assert 'exec {lock_fd}>"\\${target}.lock"' in launcher + assert 'flock -w 2400 "\\$lock_fd"' in launcher + assert 'mv -T "\\$temporary" "\\$target"' in launcher assert "scancel" not in launcher