feat(atom): add native ATOM and AToMesh support

[cquil11]

Summary

Add native ROCm ATOM workers and the official AToMesh router for aggregate and prefill/decode deployments, using srt-slurm's existing endpoint allocation, Slurm launch, and health/lifecycle machinery.

• Launch atom.entrypoints.openai_server with native CLI spelling, model identity, and role-aware Mooncake transfer configuration.
• Launch atomesh with the ATOM backend, static worker topology, and worker readiness gating. Infera is not part of this layer.
• Restrict managed ATOM logical endpoints to one Slurm node each; P/D deployments can use multiple nodes and endpoints.
• Include integration fixes for native-router infrastructure selection, optional Tachometer requirements, DP routing, and writable job-local evaluation/runtime artifacts.

Stack

GitHub native stack: #1 (shared AMD runtime) → #11 (vLLM MoRI-IO integration) → #7 (ATOM/AToMesh).

This PR is based on #11 so its review diff excludes the newly separated MoRI integration. ATOM uses Mooncake; it has no engine dependency on MoRI. The rebase preserves this PR's complete repository tree exactly relative to pre-split head df0fe39f1db8d4c6022a6d6c93d27a1be9e90200.

Validation

• Existing InferenceX #2627 full native PR sweep: success, attempt 1; all 11 throughput points and 48,600/48,600 measured requests, plus both complete 1,319-question GSM8K evaluations. Strict accuracies were 96.74% and 96.36%, above the unchanged 91% threshold. Results were staged.
• The validated engine was an unmodified official ATOM nightly containing its upstream token-limit fix. See InferenceX #2627 for exact revisions, image identity, and artifacts.
• Six 2P1D evaluation requests recovered through existing client retries after KV-notification stalls. The cause of those stalls remains unresolved. Power telemetry was not validated.
• Repository CI was green at the pre-split head. The updated base passed 344 focused tests and feat(vllm): add native MoRI-IO router integration #11 passed 357; source lint/formatting and tree-equivalence checks passed. CI is rerun automatically for the rebased branches.
• No E2E workflows were dispatched for this split. The hardware link is historical validation of the recorded implementation, not a new run of the rewritten commits.

---

Note

Medium Risk
Touches job orchestration (infra gating, worker env, vLLM Router DP routing) and adds a new backend/frontend pair on the critical serving path; changes are well-tested but affect production cluster launches.

Overview
Adds ROCm ATOM as a first-class backend (backend.type: atom) paired with the AToMesh static router (frontend.type: atomesh). srtctl launches atom.entrypoints.openai_server with managed model/port/TP flags, role-specific atom_config, and Mooncake --kv-transfer-config for P/D; AToMesh is wired like other static routers with --backend atom and per-worker /health gating. Recipes are validated to keep each logical ATOM endpoint on a single Slurm node.

vLLM Router behavior is corrected for multi-node hybrid-LB DP: --intra-node-data-parallel-size is applied only when one base URL represents the full logical endpoint; otherwise each node-local pool is advertised unexpanded so global DP ranks are not remapped to 0..N-1. Health expectations and schema validation follow the same rule.

Orchestration and eval improvements: NATS/etcd start only for frontend.type: dynamo (including atomesh among static frontends); Tachometer binary checks are optional when collection is disabled; SGLang workers get SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK when the SGLang router is DP-aware; post-eval lm-eval preserves benchmark.env and bench.sh uses a job-local writable venv/caches (optional SRTCTL_LM_EVAL_RESULT_DIR).

^{Reviewed by Cursor Bugbot for commit ed45770. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cquil11]

Validated the writable lm-eval runtime fix on MI355X using the exact serving image lmsysorg/sglang-rocm:v0.5.17-rocm720-mi35x-20260817 on healthy node mia1-p01-g16.

The smoke created a job-local Python 3.10 venv with --system-site-packages, verified the venv was the active sys.prefix, installed lm-eval successfully, and imported it from the writable venv path. Repository validation: focused lm-eval tests 7/7 pass; the full benchmark test file has 71 pass / 3 known baseline SA-bench fixture failures, identical on pristine head 10a58d2. Bash syntax and diff checks pass.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit f1537b3. Configure here.}

[cquil11]

ATOM port parity fix pushed in 265dbed359ef13f92c206d214978560738a97589:

• Preserve native ATOM CLI spelling. Its original parser uses --kv_cache_dtype and --no-enable_prefix_caching, not their hyphen-normalized forms. The adapter now passes recipe keys through literally; reserved srt-slurm-owned argument checks still canonicalize aliases.
• Preserve the original worker-local proxy_ip in Mooncake KV-transfer configuration.
• Extend the existing command-contract test rather than add a new test file. All 39 ATOM/dry-run tests pass, and all four MI355X ATOM recipe worker commands parse with the original ATOM parser.

Downstream validation: SemiAnalysisAI/InferenceX#2627. This retains the upstream June image, default RDMA transport and original engine/workload settings. Hardware results are pending.

[cquil11]

Fixed the native ATOMesh startup failure observed in InferenceX run 33665959215 (all four Slurm allocations failed before worker launch because the orchestrator attempted Dynamo-only NATS/etcd startup). Commit 527525d selects that infrastructure only for the Dynamo frontend. No engine arguments, image, workload, or KV transport settings changed.

The new regression test failed before the fix for ATOMesh and passes afterward; it also confirms SGLang still skips the infrastructure and Dynamo still starts it. The relevant suite has 217 passing tests and 2 skipped. Three unrelated sa-bench shell-test failures and 19 existing test lint findings reproduce unchanged in a clean checkout of the pre-fix commit, so they are not claimed as passing.

InferenceX PR #2627 now pins this runtime through shared PR #2542. Previously validated InferenceX PRs #2555 and #2630 retain their validated heads; fresh ATOM hardware validation is next.

[cquil11]

Fixed a confirmed ATOM model-identity mismatch in fa99473. The pinned ATOM server advertises its literal --model argument, but srt-slurm supplied the path basename to lm-eval. InferenceX run 33720270808 therefore requested DeepSeek-V4-Pro while AToMesh registered deepseek-ai/DeepSeek-V4-Pro, producing HTTP 503 before worker generation. The fix preserves the HF ID or container-visible local/staged model path for ATOM only; SGLang/vLLM naming is unchanged. Three new parameterized cases in the existing ATOM test file failed before the fix and pass afterward. All 420 focused configuration, evaluation, ATOM, frontend, Slurm, dry-run, health, and accelerator tests pass. Hardware revalidation remains pending; this is not a claim of a green full sweep.

[cquil11]

Rebased onto PR #1 at 93b854c; new head 81d4627. All existing ATOM/AToMesh fixes are retained, with the host-repair hook removed and native job waiting inherited from the base.

396 focused runtime/backend/router/config/CLI tests and 76 benchmark tests passed. The three known macOS profiling.sh shell-test failures reproduce before this rebase. No serving flags, model identity, KV transport, or benchmark workload were intentionally changed by this cleanup.

Shared InferenceX adapter/entrypoint cleanup: SemiAnalysisAI/InferenceX#2542

Current ATOM hardware run remains untouched on old head 1fb90fea and runtime fa99473:
https://github.com/SemiAnalysisAI/InferenceX/actions/runs/33782782708
A subsequent full native PR sweep is still required for the cleaned-up launcher/runtime.

[cquil11]

Validation update for this stack:

• ATOM/AToMesh: InferenceX PR port MI355X DeepSeek V4 ATOM disagg to srt-slurm InferenceX#2627 has a green native full PR sweep https://github.com/SemiAnalysisAI/InferenceX/actions/runs/33825304569 and successfully staged results. That validated recipe remains pinned to 81d4627; its ATOM implementation was not changed by the later vLLM port fix.
• vLLM/MoRI: InferenceX PR Validate archived MI355X MiniMax M3 vLLM disaggregation with srt-slurm InferenceX#2824 now has a complete green end-to-end run https://github.com/SemiAnalysisAI/InferenceX/actions/runs/33837838363 at this PR head 33a3d81. All nine original 8k/1k points passed (5,116 measured requests), and the full 1,319-question GSM8K eval scored 94.69% strict / 94.62% flexible with zero empty responses and no integration error.

The only runtime delta from 81d4627 to 33a3d81 is the scoped MoRI port-allocation fix from PR #1 and its regression coverage. 112 focused ATOM/vLLM/router/accelerator/dry-run tests passed. No engine patches, custom images, or graph-mode/config workarounds were used. The archived vLLM compatibility PR remains draft because of InferenceX's existing active-key changelog policy; no staging/reuse eligibility is claimed for that manual run.