feat(runtime): add AMD accelerator support

[cquil11]

Summary

Adds clean AMD/ROCm orchestration support to srt-slurm, stacked on the native vLLM Router and SGLang Router work in #2.

• adds a typed accelerator_vendor setting and ROCm-native ROCR_VISIBLE_DEVICES binding
• supports gres-based accelerator allocation and platform-neutral container/source transport
• supports direct vLLM, vLLM Router + MoRIIO, and SGLang Router + AMD MoRI
• propagates runtime srun_options to native router processes
• detects native router and AMD MoRI package identity in runtime provenance
• skips Dynamo-only NATS/etcd setup on native-router/direct-vLLM paths
• adds compute-only setup for mixed login/compute environments
• keeps fixed-sequence benchmarking downstream through InferenceX's unchanged custom benchmark script

Scope boundary

This does not attempt to run Dynamo on ROCm. Dynamo, NIXL, and Dynamo's NVIDIA-specific frontend/runtime assumptions are out of scope. AMD validations use direct vLLM, native vLLM Router, or native SGLang Router. KV transfer uses MoRIIO for vLLM and AMD MoRI for SGLang.

Validation

Repository CI

• Full Linux unit suite and coverage
• Lint and formatting
• Typecheck
• Recipe validation
• Mock/server integration
• CodeQL and copyright checks

InferenceX exact workflow validation

• MI300X aggregate, direct vLLM: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/31360797256
• MI300X 1P1D, vLLM Router + MoRIIO: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/31359534279
• MI355X aggregate, SGLang Router: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/31363011459
• MI355X small 1P1D, SGLang Router + MoRI: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/31372898593
• MI355X production Qwen3.5-397B-A17B-FP8 1P1D TP8: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/31373547673

The production MI355X run used two full nodes, native SGLang Router, one TP8 prefill worker and one TP8 decode worker, AMD MoRI, and fixed ISL 8192 / OSL 1024 / concurrency 8. All 8 warmups and 16 measured requests succeeded. The Slurm job completed 0:0, and the result, runtime logs, aggregation, and success-rate jobs all passed.

The steady MI355X SGLang image was also checked directly with an exact 4,946,616,832-byte two-GPU MoRI RDMA write. That validates the merged automatic dma-buf large-MR path that replaces the older image's failing plain ibv_reg_mr behavior.

Companion PR

• feat(amd): orchestrate AMD clusters with srt-slurm InferenceX#2542

---

Note

Medium Risk
Touches Slurm submission templates, cluster networking for etcd/NATS, and vLLM P/D connector/health behavior—important for job success on AMD and mixed login/compute layouts, but scoped with validation and tests rather than Dynamo-on-ROCm.

Overview
Adds AMD/ROCm orchestration: cluster accelerator_vendor, gpu_sbatch_directive (gres / none), vendor-neutral set_visible_devices (ROCR vs CUDA), and validation for vLLM Router MoRI-IO P/D (topology constraints, dynamic discovery, role-aware MoRIIOConnector JSON from Slurm layout).

Extends native-router / direct-backend paths: skip NATS/etcd in validate_setup for vllm, vllm-router, sglang, trtllm_serve; make setup-compute for arch-matched uv only; optional runtime_config_transport: embedded (base64 recipe + srtslurm.yaml in the batch script, node-local log bootstrap); srtctl wait with optional log tail.

Networking and readiness: control plane uses infra_node_ip on the configured fabric; IP discovery prefers hostname -I before default-route; SGLang/static routers can wait on backend /health before launch; MoRI vLLM Router readiness uses a one-token /v1/completions probe instead of /workers.

Operational fixes: HF pre-download respects HF_HUB_CACHE and keeps the bash wrapper so env reaches the container; Dynamo version installs go to a job-local /logs overlay with --no-deps; worker/infra logs and config dumps use container-mounted paths; failure log tails tolerate non-UTF-8 bytes.

^{Reviewed by Cursor Bugbot for commit c0472a8. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 3 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit d7b483c. Configure here.}

[cursor]

Compute setup still requires Tachometer

High Severity

validate_setup still requires bin/tachometer-scraper on native-router paths, but make setup-compute only installs uv. Submit then tells users to rerun that target, which cannot satisfy the check, so the documented AMD/compute-only setup cannot submit a job.

Additional Locations (2)

• Makefile#L72-L89
• docs/installation.md#L53-L63

 

^{Reviewed by Cursor Bugbot for commit d7b483c. Configure here.}

[cursor]

SGLang DP validator never runs

Low Severity

_validate_sglang_data_parallelism is new dead code: nothing calls it, including SrtConfig.__post_init__. Invalid SGLang tp-size/dp-size layouts are not rejected at load or dry-run and still fail only after workers start.

Additional Locations (1)

• src/srtctl/core/schema.py#L1749-L1761

 

^{Reviewed by Cursor Bugbot for commit d7b483c. Configure here.}

[cquil11]

Cleanup pushed in 93b854c.

• Remove the recipe-level host_setup_script hook and its non-container host srun.
• Add platform-neutral srtctl wait JOB_ID --log-file PATH: stream the allocation log, tolerate queue/accounting lag and transient query timeouts, and propagate the terminal allocation exit status. It never cancels or resubmits jobs.
• InferenceX now calls native apply --json and wait; the adapter does not manage Slurm or repair hosts.

Validation: 249 focused base runtime tests passed. On the rebased ATOM child, 396 runtime/backend/router/config/CLI tests plus 76 benchmark tests passed. Three existing benchmark shell tests fail on a missing profiling.sh path on this macOS environment; the same failures were reproduced on the old runtime. Read-only live Slurm accounting checks correctly returned COMPLETED/0 for job 42158 and FAILED/1 for job 42154.

ATOM PR #7 is rebased on this head. The already-running InferenceX ATOM sweep remains pinned to its original runtime; it is not validation of this cleanup.

[cquil11]

MiniMax-M3 MI355X native-vLLM validation exposed a real orchestration defect in run https://github.com/SemiAnalysisAI/InferenceX/actions/runs/33836289639: all four TP children inherited the same VLLM_PORT scan base, and MoRI listeners collided before serving started. The unmodified upstream vLLM 0.28.0 port allocator reproduces the collision with that environment; without it, the kernel allocates ephemeral ports.

Fixed in c0472a8: MoRI keeps upstream ephemeral-port allocation; non-MoRI rendezvous bases remain unchanged. No engine source patch, custom image, cleanup helper, or model-option change. Two regression cases (backend connector and role override) fail before the fix and pass afterward. PR #7 incorporates the fix at 33a3d81.

112 focused vLLM/router/ATOM/accelerator/dry-run tests pass on the combined runtime. The broad local suite reports 1546 passed, 2 skipped, 6 deselected, and 5 failures; all five reproduce on the exact pre-change 81d4627 baseline (three sa-bench shell-path tests, one local shell-profile test, and one Linux-only CPU-affinity mock on macOS). Previously validated InferenceX recipes remain pinned to 81d4627; only the archived MiniMax compatibility lane selects the new runtime.

[cquil11]

The MoRI native-port fix in c0472a8 is now validated on MI355X with official vLLM ROCm v0.28.0 and the official vLLM Router nightly, through the combined PR #7 runtime 33a3d81.

Complete InferenceX run: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/33837838363 — all jobs green, attempt 1. Port PR: SemiAnalysisAI/InferenceX#2824.

1P1D TP4/TP4: all nine 8k/1k concurrency points, 5,116/5,116 measured requests, plus full 1,319-question GSM8K (94.69% strict; unchanged threshold 90%). Per-point and per-sample artifact audits passed, including zero empty eval responses. An earlier independent full throughput sweep on the same runtime also passed. Both allocations cleaned up normally.

No engine source patches or graph-mode/config workarounds were needed. The source fix only preserves native ephemeral-port allocation for MoRI TP children; existing non-MoRI port behavior remains unchanged and covered by the focused regression tests. Previously validated InferenceX recipe pins were not changed.