Repository navigation
Conversation
Add the DeepEP M2N AFD connector: Attention ranks route tokens and dispatch only the routed copies straight to the FFN ranks that own the chosen experts; FFN ranks run their local experts and combine sums each token's top-k contributions back on its Attention rank. - Transport: afdEP's DeepEP-derived ElasticBuffer (optional `afdep` extra, pinned to afdEP main). The former AfdEPConnector is merged into DeepEPM2NAFDConnector: `role_aware` (default true) runs afdEP's AFD role path, false keeps its Masked DeepEP protocol. No kernel sources are vendored in this repository. - Experts: `deepgemm` backend uses the separately installed upstream DeepGEMM package; `fused_experts` (vLLM Triton) is the reference path. - Plugin integration: routed-experts connector interface, FFN routed-experts model runner, connector device placement, Attention-side routing proxy for Qwen3 MoE and DeepSeek V2/V3, Attention TP split, DBO lanes and CUDA graph support. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
Add the DeepEP M2N connector, routed-experts runner and model tests, and move device tests out of tests/unit. - Connector unit tests (config, stage plans, handshake, control plane, DBO lane streams) run against a stand-in afdEP module (tests/unit/fakes/afdep.py) where afdEP is not installed. They cover the merged role_aware option and that each lane builds afdEP's ElasticBuffer with the protocol and FFN rank count. - Network planner, connector device placement, routed FFN runner, worker bootstrap, validation, topology, and Qwen3 / DeepSeek V2 routing-proxy and weight-policy tests. - GPU tests move to tests/gpu/ (gpu marker), with a masked-protocol (role_aware: false) case; the M2N vLLM-config fakes they share with the unit tests live in tests/unit/fakes/vllm_config.py, the recording routed connector in tests/unit/fakes/connector.py. - Qwen3 DeepEP M2N E2E scenarios (2A2F eager, graph, graph+DBO, Attention TP2) in the E2E runner. - validation_paths entries for the new test files. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
The afdep extra pinned ms-llmd/afdEP e763e318, a 2026-10-19 development tree under the old distribution name. It rejects the role path across NVLink domains and has none of the cross-node work: the sparse directional hybrid, FP8 combine replies, role-sized inboxes. The library now lives at afdEP/afdEP-comm as the afdep-comm distribution (import name still afdep); pin its main at 2b74d57 and relock. afdEP-comm main also removed ElasticBuffer's role_aware keyword (afdEP-comm #17): ffn_ranks alone selects the AFD transport. Drop the connector's role_aware option with its check and docstring, and build each lane with ElasticBuffer(**kwargs, ffn_ranks=ffn_size). The two changes go together: on the old pin, role_aware defaulted to false, so dropping it there would silently fall back to the unspecialized protocol. A config that still sets role_aware is rejected as an unknown option. The 2F2A GPU case for the removed protocol goes. The fake ElasticBuffer now rejects keywords the real one lacks: the old fake accepted anything, so the unit tests passed while every real rank would fail at lane creation. The module docstring and README name afdEP-comm and describe the baseline as FastAFD's DeepEP M2N. Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
…rnel fp8_quant.py imported triton directly, so importing the DeepEP M2N connector failed on hosts without Triton and its unit tests could not be collected on a CPU-only host, which AGENTS.md requires of tests/unit. vllm.triton_utils provides Triton when it is installed and placeholders otherwise, as vLLM's own kernels do; the kernel is unchanged on GPUs. Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
Attention ranks allocated a zero buffer sized for DeepEP's worst-case no-sync receive at the top bucket rung (union ranks x bucket x top-k, 96 MiB to 1 GiB per Attention GPU), before vLLM sizes the KV cache. afdEP-comm gives Attention ranks no receive rows on any dispatch, with or without the host sync, so combine only ever read zero rows of it. Allocate a (0, hidden) tensor once instead, keeping one stable address for captured graphs, and fail loudly if the transport ever reports receive rows for an Attention rank. worst_case_receive_rows and its test go with the buffer they sized; the precompile test already checks that every Attention combine gets an empty input. Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
…ments The fake ElasticBuffer rejected unknown keywords but accepted calls that left out the real constructor's required ones, and the lane test built a buffer from two hand-picked arguments. A connector change that stopped passing, say, cpu_timeout_s would still pass while every GPU rank failed at lane creation. Lanes are now built by _new_lane, which assembles the full argument set the init path used to pass inline. The fake requires every required keyword of afdEP-comm's ElasticBuffer and rejects any other, and the lane test goes through _new_lane. Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
fix(gpu): run the DeepEP M2N connector on afdEP-comm
DeepEP M2N needs the Attention side to send topk_ids and topk_weights with the hidden states. Only Qwen3 MoE does: on CUDA, DeepSeek V2/V3 still builds the remote-experts MoE shell, which sends router logits only, so the first MoE forward failed in send_attn_output. DeepSeek would also need its grouped top-k, routed scaling and shared experts on Attention, none of which the M2N path implements yet. Validate the model architecture with the other M2N deployment checks. Both workers run this validation before the connector bootstrap, so every rank fails at startup, naming the supported models, before any collective. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
The handshake did not advertise control_backend. With one role on gloo and the other on nccl it succeeded; the NCCL ranks then created a communicator the gloo ranks never joined, and startup hung until the timeout. Advertise the backend and require every rank to match, so the mismatch fails before any communicator or buffer is created. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
The harness kept the worst error per metric with max(), which ignores NaN, so NaN expert or combine outputs kept the previous worst value and passed the tolerance checks. Reject a non-finite error when it is recorded, and add a negative case for NaN and infinite outputs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
…ectors A routed-experts connector must join the AFD world and set its NCCL environment before vLLM's distributed init, which vLLM runs before it builds the runner. That requirement was spread across a factory method (create_bootstrapped_connector), both workers, and adopt_bootstrapped_connector in both runners, which closed the connector their constructor made and swapped in the worker's. Give the worker layer one owner for it. connector_lifecycle.py creates the connector with the normal factory call and runs its pre-init step (prepare_before_distributed_init, renamed from bootstrap), and checks the connector's device and role rank against vLLM's after init. The workers pass that same instance to the runner through the existing runner seam, as a keyword-only connector argument; runners create the connector themselves only when none is passed, so P2P is unchanged. Runners no longer create a throwaway connector, the factory only creates connectors again, and init_afd_connector after model load stays the common second step. Move worker device resolution to afd_plugin/distributed/device.py as resolve_worker_device_index, with vLLM imported inside the functions so the package stays CPU-safe to import. Registering vLLM's assigned physical GPU ids is now an explicit call (register_assigned_physical_gpu_ids), and resolving without it raises instead of mapping to another rank's GPU. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
…ycle The DeepEP M2N connector planned NICs for RoCE fabrics whose rails are separate, unrouted subnets: it pinned every rank to one shared rail through NCCL_IB_HCA. NCCL reads that variable once per process, when the first communicator is created, so the connector had to join the AFD world before vLLM's distributed init. That requirement was the only reason for the worker lifecycle (connector_lifecycle.py), the pre-init device mirroring (distributed/device.py), the TP rank derived from the worker rank, and the connector injection into both runners. Routed fabrics (InfiniBand, routed RoCE, NVL72, one host) need no NIC planning: NCCL's own per-GPU choice works there. Drop NIC and rail discovery, the shared/explicit NIC policies, rails_routed, nics, nccl_env and the presets; keep NVLink-domain discovery, the GIN decision, hybrid mode and the traffic class. The only variables the connector still sets, EP_DISABLE_GIN and the per-buffer comm stream switch, are read by afdEP when it creates its buffers, so the handshake moves into init_afd_connector and the connector follows the same lifecycle as P2P: the runner creates it and initializes it after the model loads. On unrouted-rail RoCE, set NCCL_IB_HCA to one shared rail in the launch environment. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
…lus-review-fixes Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Run the DeepEP M2N eager, graph and graph+DBO scenarios across two pods, Attention DP2 on one and FFN DP2 on the other (layout 2A0F,0A2F), with GSM8K at 100 samples by default. M2N needs FFN ranks first (afdEP-comm) and each NVLink domain to hold an equal, contiguous block of world ranks (DeepEP), so a two-pod run splits by role with equal rank counts; dispatch and combine then cross pods over NCCL GIN. The multi-pod runner now knows the connector's rendezvous role: world rank 0, which hosts its gloo store, is the first FFN rank. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
M2n connector sm90plus review fixes P1+P2
…evice The runners pass vLLM's world-group local rank, which is a logical index. vLLM places the worker on that index mapped through the assigned physical GPU ids (--device-ids) and CUDA_VISIBLE_DEVICES, so with --device-ids 4,5 the connector chose cuda:0/cuda:1 while the model ran on cuda:4/cuda:5, and init_afd_connector created its transport buffers on another worker's GPU. The pre-init device mirroring that handled this left with the bootstrap lifecycle. Map the rank with current_platform.logical_device_id_to_visible_device_id, as vLLM's Worker.init_device does; vLLM registers the assigned ids before it builds the runner. P2P and the runners are unchanged. Cover the identity, CUDA_VISIBLE_DEVICES, assigned-ids and reordered cases with a CUDA-like platform fake. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
…contract AFDRoutedExpertsConnector is an abstract interface, not an expert-compute implementation, and routed_experts.py was easy to mistake for one next to vLLM's RoutedExperts. Move it into connectors/base.py next to AFDConnectorBase, still as its own subclass so P2P and CAM connectors implement none of its methods. Move LocalExpertWeights, FFNLocalExperts and FP8_DTYPE into connectors/metadata.py with the other shared payload types; base.py already imports metadata.py, so imports stay acyclic. routed_experts.py has no remaining users and is removed. The package-level names in afd_plugin.connectors are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
Address review comments - round 2
## Purpose Add run-vllm-bench-k8s skill for benchmarking AFD recipes on Kubernetes. Code the k8s benchmarking workflow as a skill so that measuring throughput/TTFT/TPOT/ITL for a deployed AFD GPU recipe is repeatable and self-documenting. ## Issue - Related issue(s): vllm-project#464 - Closing keyword, only if fully resolved: Closes vllm-project#464 ## Scope - In scope: GPU colocation recipes only - Out of scope: - baseline recipes - local benchmarking (run request_generator.sh against decode_bench_server.sh directly) - NPU recipes - correctness E2E (run-e2e). ## Implementation Notes .agents/skills/run-vllm-bench-k8s/SKILL.md — the main workflow: - Workload selection (e.g., balanced, max-throughput, prefix-heavy, long-context). - Recipe deployment: delegates to deploy-afd-k8s. - Sweep execution: single Pod spec running the repo's tools/benchmarks/request_generator.sh from the afd-plugin image (generated using the existing docker file). - Result collection (copies from the pod to local storage). - Teardown: cleanup of vllm-bench, vllm-pod, and vllm-service without waiting to be asked; the reports PVC is kept until all runs of interest are copied out. References: - references/comparing-runs.md — comparability checks and per-point deltas for two-run comparisons; recipes to compare must be given explicitly by the user. - references/fetching-past-reports.md — recovering reports from the vllm-bench-reports PVC after the pod is gone. - agents/openai.yaml — agent-facing registration. ## Test Plan - User prompt: "Follow the `run-vllm-bench-k8s` skill to compare two recipes". State, for example, 2a2f_eager_dbo_dp1tp2.sh and 2a2f_graph_dbo_dp1tp2.sh from recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation. - Provide afd-plugin image and model PVC if exists. - Select the workload - e.g., balanced. ## Test Result Expected: a table with runs' comparison and a summary with findings. Example: ``` Reference: 2a2f_eager_dbo_dp1tp2-20261018-111201 (--enforce-eager) Compared: 2a2f_graph_dbo_dp1tp2-20261018-114152 (FULL_DECODE_ONLY CUDA Graph, capture size 64) Model /models/DeepSeek-V2-Lite, 4 GPUs, workload balanced (identical across runs: random ISL 1024 / OSL 256, 512 prompts/point, sweep 2:64 4:64 8:64 16:64) ``` | Rate | Conc | Metric | Eager | Graph | Δ | % | |------|------|--------|-------|-------|-----|---| | 2 | 64 | Output tok/s | 167.9 | 138.5 | −29.4 | −17.5% | | | | TTFT mean / p99 (ms) | 70301 / 76996 | 86034 / 95290 | +15733 / +18294 | +22% | | | | TPOT mean / p99 (ms) | 74.5 / 78.9 | 90.7 / 93.9 | +16.2 / +15.0 | +22% / +19% | | | | E2EL mean (ms) | 89306 | 109173 | +19867 | +22% | | 4 | 64 | Output tok/s | 723.1 | 873.2 | +150.1 | **+20.8%** | | | | TTFT mean / p99 (ms) | 1095 / 1720 | 198 / 323 | −898 / −1397 | **−82% / −81%** | | | | TPOT mean / p99 (ms) | 75.5 / 77.8 | 33.1 / 84.9 | −42.4 / +7.1 | **−56%** / +9% | | | | E2EL mean (ms) | 20345 | 8645 | −11700 | −58% | | 8 | 64 | Output tok/s | 750.7 | 772.7 | +22.0 | +2.9% | | | | TTFT mean / p99 (ms) | 1094 / 1752 | 1354 / 2272 | +260 / +520 | +24% / +30% | | | | TPOT mean / p99 (ms) | 74.7 / 76.1 | 71.1 / 83.1 | −3.6 / +6.9 | −5% / +9% | | | | E2EL mean (ms) | 20141 | 19477 | −664 | −3% | | 16 | 64 | Output tok/s | 794.6 | 972.1 | +177.4 | **+22.3%** | | | | TTFT mean / p99 (ms) | 1065 / 1941 | 1401 / 2560 | +336 / +619 | +32% / +32% | | | | TPOT mean / p99 (ms) | 71.9 / 75.5 | 55.8 / 67.5 | −16.1 / −8.0 | **−22% / −11%** | | | | E2EL mean (ms) | 19389 | 15624 | −3765 | −19% | Completed = 512/512 at every point in both runs; no failures. ### Findings 1. **Saturation**: both runs are saturated across the *entire* sweep — even at 2 req/s, mean TTFT is 70–86 s (pure queue wait), and steady-state output throughput plateaus at ~750 (eager) / ~770–970 (graph) tok/s regardless of offered rate. With `--max-num-batched-tokens 64`, each 1024-token prompt needs 16 chunked-prefill steps, so the offered load exceeds capacity at every point; this sweep measured **queueing behavior under overload, not headroom**. 2. **What moved**: exactly what FULL_DECODE_ONLY graphs should move — **decode**. TPOT improved −56% at 4 req/s and −22% at 16 req/s; TTFT moved the *other* way at 8–16 req/s (+24–32%), consistent with prefill running eager in both cases while graph capture adds memory pressure/interference with chunked prefill. At 4 req/s graph improved everything simultaneously because it drained the queue faster. 3. **Divergence**: the 2 req/s point is inverted (graph −17.5% throughput) — but at that point both runs are fully backlogged, so it reflects deep-queue dynamics rather than steady serving; treat it as noise-of-overload. 4. **Headline**: under sustained load the graph recipe sustains **~22% higher output throughput** (972 vs 795 tok/s at 16 req/s) and **22% lower TPOT**, at the cost of ~32% higher TTFT — i.e., better decode cost and capacity, slightly worse prefill/queue latency. 5. The 8 req/s point is nearly within noise (+2.9% throughput); the meaningful capacity difference shows at 16 req/s. ### Artifacts - `./reports/2a2f_eager_dbo_dp1tp2-20261018-111201/` (one JSON per sweep point + `run.json`) - `./reports/2a2f_graph_dbo_dp1tp2-20261018-114152/` (one JSON per sweep point + `run.json`) ## Docs Impact - Files updated: .agents/skills/run-vllm-bench-k8s/ --- <details> <summary>Essential PR Checklist</summary> - [x] Purpose is clear and linked to public context when possible. - [x] Scope is bounded. - [x] Compatibility with vLLM v0.30.0 is considered. - [x] No changes are made to the vLLM source checkout. - [x] Plugin-owned classes or explicit dotted class paths are preferred over monkey patches. - [x] Any compat shim or monkey patch is isolated, idempotent, version-guarded, documented, and tested. - [x] Imports remain CPU-safe; CUDA-heavy work is delayed or GPU-gated. - [x] Validation evidence is included, including skipped GPU tests when applicable. - [x] Documentation impact is stated. </details> --------- Signed-off-by: Shlomit Koyfman <shlomitk@il.ibm.com>
…pool Add a prompt-pool mode to AFDDecodeBenchConnector (num_prompts > 0 in kv_connector_extra_config). Each Attention DP rank turns its first num_prompts requests into a seeded random prompt pool, identical on every rank, prefills it for real, and pins the finished warmup requests' KV blocks through the connector's delayed-free contract so LRU eviction can never drop a pool entry. Every later request is remapped round-robin onto a pool prompt regardless of its content; the prompt length is rounded to one token past a block boundary, so the prefix cache covers all but the last token and each measured request prefills exactly one token before decoding over real KV. A startup check fails fast when the pool and the running requests cannot both fit the KV cache, and reports the largest pool that fits. num_prompts = 0 keeps the dummy-KV mode unchanged. Add decode_bench_client.py: warms every DP rank with num_prompts requests pinned by the X-data-parallel-rank header, then sends the measured requests through vLLM's load balancer and reports TTFT/TPOT/ITL, throughput, and prefilled tokens per request from the prefix-cache counters (1.00 = decode only). Add a Qwen3-30B-A3B 2A2F single-node recipe with selectable AFD connector (DeepEPM2NAFDConnector or P2pNcclAFDConnector) and KV connector (none, decode_bench, or a raw KV_TRANSFER_CONFIG), usable for plain serving as well; the P2pNcclAFDConnector directory carries a thin entry point. It replaces decode_bench_server.sh. Name the connector logger under the vllm tree: vLLM only configures handlers for "vllm", so INFO lines from other logger names are dropped. Validated on 4xH200 with both connectors: plain serving smoke, and the decode-only benchmark with 1000 x 1025-token prompts per rank, 2000 measured requests, 0 prefix-cache misses, 1.00 prefilled tokens/request. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
decode_bench_client.py now drives every phase with `vllm bench serve` and keeps only the orchestration and the decode-only checks: - Warmup: one vllm bench run per Attention DP rank, in parallel, pinned with the X-data-parallel-rank header, sends --pool-prompts requests of --prompt-len tokens with one output token. --warmup-concurrency and --warmup-bench-args configure these runs. - Pre-check: the server's prefix-cache counters must show --prompt-len prompt tokens per warmup request, so the server's pool prompts are as long as the measured prompts, and every rank must have received a full pool of requests. - Measured: vllm bench serve --dataset-name random --random-input-len <prompt-len> --random-range-ratio 0, plus every argument after `--` (request rate, concurrency, number of requests, output length, ...). Flags the client owns are rejected after `--`. - Post-check: over the measured run the server must have prefilled one token per finished request, with no failed requests and no preemptions; otherwise the client exits non-zero. vllm bench's result JSON for each run and decode_bench_report.json with the checks land in --result-dir. The client's own HTTP load generator, and its aiohttp and numpy dependencies, are gone. --num-prompts is renamed --pool-prompts so it does not collide with vllm bench's --num-prompts. Validated on 4xH200 with both AFD connectors (1000 x 1025-token pool per rank, 2000 measured requests, 256 output tokens, concurrency 64): pool length pre-check 1025.00 tokens/request, 1.0000 prefilled tokens per measured request, 0 misses, 0 preemptions; a mismatched --prompt-len is rejected before the measured run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
One launch script for Qwen3-235B-A22B-FP8 in a 4A2F prefill/decode colocation on a single 6-GPU node: Attention DP4/TP1 on GPUs 0-3 and FFN DP2/TP1 with expert parallelism on GPUs 4-5, FULL_DECODE_ONLY CUDA graphs (8-128). The connector is selectable (DeepEPM2NAFDConnector by default, P2pNcclAFDConnector through its own entry point), and KV_CONNECTOR=decode_bench turns it into the decode-only benchmark over a pinned real-KV prompt pool, driven by decode_bench_client.py with --dp-size 4. Differences from the Qwen3-30B-A3B recipe: - DeepEPM2NAFDConnector dispatches FP8 (M2N_PAYLOAD_FORMAT, default fp8_block128_fp32); M2N_EXTRA_CONFIG adds connector_extra_config members. - Per-role ATTN_EXTRA_ARGS / FFN_EXTRA_ARGS, e.g. --api-server-count 1 --cudagraph-metrics on Attention. - The AFD join timeout and vLLM's CPU-group timeout default to 2 h, since a cold 235B load can finish minutes apart on the two roles. - Defaults sized for the benchmark: 128 sequences per Attention rank, MBT 3072, max-model-len 4096, a 500 x 1025-token pool per rank (about 555K of the ~640K KV tokens an H200 Attention rank has). - The README warns that P2pNcclAFDConnector's FFN hangs with PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True. Validated on 6xH200 with both connectors (M2N + DBO at concurrency 64 and 128): decode-only sweeps at concurrency 64/128/256 and request rates 1-50 (rate x 300 requests), every step one prefilled token per request and no preemptions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
vLLM 0.30 runs a microbatched (DBO) step eager whenever the smallest DP rank holds at most half of the CUDA graph's tokens (`2 * min < N`). A FULL microbatched graph is captured with a fixed request/token split, and native slicing re-derives the split from the real requests, so an underfilled rank would replay shifted metadata. With DP imbalance this fires on most decode steps and an eager microbatched step costs several times a graph step. For FULL one-token decodes, keep the captured split instead: each microbatch takes the request and token range the graph was captured with, and a trailing microbatch without real requests keeps its padded shape (one-token query bound, nonzero host-side sequence bound). The step then replays the microbatched graph on every rank. Configs this is not validated for (speculative decoding, inner-state models, MLA, PCP/DCP) replay the regular graph instead of running eager. Other steps keep native slicing and coordination. - gpu_mrv2_ubatching: instance-bound UBatchRunner.prepare copy with the captured split, a copied sync_cudagraph_and_dp_padding for DP>1 without the eager guard, and the same rule in the DP1 branch. - Unit tests cover the captured split against native vLLM slicing, the padding-only microbatch, and DP1/DP2 graph selection. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
ATTN_TP sets the Attention tensor-parallel size; Attention DP becomes the number of ATTN_DEVICES divided by it (default 1, the DP4/TP1 layout as before). ATTN_TP=4 runs Attention as one DP1/TP4 engine on the same four GPUs; num_attention_ranks stays 4 and the FFN keeps DP2/TP1. The script rejects a TP size that does not divide the Attention device count, and the README notes that one engine then needs MAX_NUM_SEQS and the capture sizes raised to hold every running request. Validated on 6xH200 with both AFD connectors: decode-only sweeps with ATTN_TP=4, MAX_NUM_SEQS=256, capture sizes up to 256, concurrency 256 and request rates 1-50, one prefilled token per request and no preemptions in every step. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
fix(dbo): replay the microbatched CUDA graph for underfilled DP ranks
Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
Take upstream for every file this branch carried as the pre-squash vllm-project#493 commits, keep this branch's parameterized Qwen3 recipes with upstream's defaults, and keep decode_bench_server.sh deleted. - gpu_mrv2_ubatching: cite vLLM 0.32.0 for the copied prepare and sync_cudagraph_and_dp_padding, and match the copy to the 0.32 body. - Qwen3 recipes: upstream ports (6239/18000/18001), FFN first, plain vllm serve; 235B: 256 sequences, 3600 s timeouts, capture sizes 8..256 plus 48, 112 and 192; 30B: capture sizes 8,16,32,64, 4096 batched tokens, max-model-len 4096. - READMEs: merge upstream's layout, configuration notes and deployment check with the decode-only benchmark docs. - decode_bench_client defaults to port 18000; KVCacheLayer is a TypeAlias. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
…onto pinned entries Pin the prompt pool by holding one reference on each entry's full prompt blocks (BlockPool.touch) instead of reporting delay_free_blocks for warmup requests and never completing them. Delayed-free warmup requests stayed in the scheduler for the server's lifetime, so has_requests() never turned False: the engine loop spun on empty steps whenever no request was running, and shutdown and pause could not see the engine as idle. Warmup requests are now freed as usual, and only the full prompt blocks a prefix hit shares are pinned, one block fewer per entry. Map measured requests round-robin over pinned entries only. An entry that is still warming, or whose warmup was aborted, is not in the prefix cache yet, so a request mapped onto it decoded over unwritten blocks and could leave them cached under the entry's hashes. A request that arrives before any entry is pinned falls back to the whole pool; such early requests are counted, logged, and caught by the client's decode-only check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
The M2N connector accepted only placements whose NVLink domains held FFN-first, contiguous blocks of the role world, so pods that each hold FFN and Attention ranks (2 x 4A1F) were rejected. The handshake now plans a transport order from the ranks' NVLink domains: domain by domain, FFN ranks first in each. When the FFN ranks fill the first domains the order is the role world and nothing changes. When every domain holds the same number of FFN ranks, the lanes use a gloo group in transport order and afdEP-comm's per-domain mode (ffn_ranks_per_domain). The role world still drives the handshake, model checks, control plane and expert ownership. - layout.py: RankPlacement, TransportLayout, plan_transport_layout - connector.py: transport layout in HandshakeResult, transport group, ffn_ranks_per_domain on each lane, network plan in transport order - pin afdEP-comm to the ffn-domains-hybrid head (fdff967) - recipe: Qwen3-235B-A22B-FP8 2 x 4A1F multi-pod recipe - E2E: Qwen3 multi-pod 1A1F,1A1F M2N eager/graph/DBO cases - docs and deploy-afd-k8s skill: placement rule, POD=AFD_<index> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
The plugin's INFO lines are not printed under vLLM's logging setup, so the docs should not point users at the transport log line. A placement the connector cannot carry still fails at startup with the pods' shapes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
Combine the decode-only benchmark work with DeepEP M2N flexible Attention/FFN placement across pods. Conflict in the Qwen3-235B-A22B-FP8 M2N recipe README resolved by keeping both sides: the single-node P2P/M2N prerequisites, ATTN_TP and graph notes from decode-bench-kv-pool, plus the multi-pod 2 x 4A1F section from m2n-flexible-placement. Align multipod_4a1f_4a1f_graph.sh with the 4A2F script: the capture size knob is now CUDAGRAPH_CAPTURE_SIZES with the 4A2F default 8,12,16,24,32,48,64,96,128,192,256. Keep the afdep-comm pin fdff967 from m2n-flexible-placement. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
Replace the eleven DeepEPM2NAFDConnector recipe scripts (2a2f_*, 4a2f_*, multipod_4a1f_4a1f_graph.sh) with graph.sh, eager.sh and dbo.sh per model (DeepSeek-V2-Lite, Qwen3-30B-A3B, Qwen3-235B-A22B-FP8), about 95 lines each instead of up to 191. - LAYOUT (default 2A2F) lists one <a>A<f>F entry per pod; POD=AFD_<k> selects the entry. The script derives devices (FFN first), rank totals, DP size/local/start rank, headless and the leader of each role, so one script runs single-host, symmetric multi-pod (1A1F,1A1F, 4A1F,4A1F) and role-split (2A0F,0A2F) placements. ATTN_TP replaces the dp1tp2 scripts. - Run-specific flags move out of the recipe: VLLM_ARGS, ATTN_ARGS, FFN_ARGS and script arguments are appended last, so they override the defaults. DECODE_BENCH takes the decode-only benchmark's kv_connector_extra_config JSON and replaces KV_CONNECTOR and the DECODE_BENCH_* knobs. - Each script stays self-contained (the deploy skill mounts one file); the block below its model settings is identical in every recipe. - The P2P Qwen3 entry points now run graph.sh (235B keeps LAYOUT=4A2F). Docs: the connector recipes README holds the shared layout, settings, multi-pod and benchmark instructions; model READMEs keep only their defaults and notes; tools/benchmarks/README.md holds the decode-only benchmark mechanics. The deploy-afd-k8s skill reads placement and GPU counts from LAYOUT and passes env values with spaces or JSON as quoted YAML. Validated with Qwen3-30B-A3B on H100 (vLLM 0.32.0): 2A2F graph, eager, DBO, ATTN_TP=2, flag overrides and the P2P entry point serve correct completions; decode-only benchmark (500 x 1025-token pool, 2000 requests, OSL 256) passes its checks on 2A2F (13.6 req/s, TPOT p50 18.0 ms) and on two-node 2A0F,0A2F and 1A1F,1A1F (11.0 req/s, TPOT p50 22 ms), plus smoke on both multi-pod layouts. Signed-off-by: ronenkat <ronen.kat@gmail.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # tests/unit/connectors/test_deepep_m2n_connector.py
Brings in vllm-project#533 (deploy-afd-k8s simplify, references/ -> resources/), vllm-project#555 (squashed flexible placement, already on this branch unsquashed), vllm-project#560 (docs consolidation), vllm-project#564 and the afdep-comm v1.1.0 pin. Conflict resolution: - pyproject.toml, uv.lock, tests/unit/fakes/afdep.py: take main's afdep-comm v1.1.0 pin. - DeepEP M2N connector guide, connector recipes README, Qwen3-235B README: keep this branch's LAYOUT-driven graph/eager/dbo recipe docs and the decode-only benchmark section. - recipe/README.md: drop the directory tree as main does; keep the LAYOUT rows and main's new DeepSeek-V2-Lite M2N row. - deploy-afd-k8s skill: keep the LAYOUT placement rules with links moved to resources/; keep main's GPU_COUNT rules for placement and plain recipes, noting that a one-entry LAYOUT has no literal CUDA_VISIBLE_DEVICES; keep main's Ready-condition wait heading; drop main's pointer to the removed decode_bench_server.sh. - Remove multipod_4a1f_4a1f_graph.sh, which vllm-project#555 re-added; the LAYOUT graph.sh with LAYOUT=4A1F,4A1F replaces it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
github-actions
Bot
added
documentation
Drop the prompt-pool mode from AFDDecodeBenchConnector and return
tools/benchmarks/decode_bench.py to main's behavior: every prompt token but
the last is reported as externally computed, so requests skip prefill.
The only difference from main is an explicit TypeAlias on KVCacheLayer so
mypy passes without torch installed.
Remove decode_bench_client.py, whose warmup and prefix-cache checks only
apply to the prompt pool. Decode-only runs now start a recipe with
DECODE_BENCH='{}' and drive it with request_generator.sh, whose default
port follows the recipes' Attention port (18000).
Rewrite tools/benchmarks/README.md for the dummy-KV mode and update the
recipe READMEs and the deploy/bench skills that referenced the pool.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
The GPU prefill_decode_colocation recipes came in two styles: nine LAYOUT-driven M2N scripts that each carried an identical ~75-line launch block, and hand-written P2P scripts that spelled out both vllm serve commands per DP/TP/pod variant. The P2P Qwen3 entry points exec'd the M2N scripts through a script-relative path that breaks when a single recipe file is mounted into a pod. Move the shared block into recipe/gpu/common/afd_colocation.sh (LAYOUT/POD placement, DP flags, the afd config, the decode-bench hook, the override variables and the two servers, plus FFN_TP, SERVE_HOST and an AFD_DRY_RUN print mode). Every recipe now states only its model, connector, default layout, mode and serve flags, and sources the launcher from AFD_RECIPE_LIB or its repository location. The P2P recipes move to the same style: DeepSeek-V2-Lite dbo.sh and eager_dbo.sh replace the seven 2A2F/4A4F/multi-pod variants (selected with LAYOUT, ATTN_TP and FFN_TP), Qwen3.5-122B graph.sh replaces the three-pod script (default LAYOUT=2A0F,2A0F,0A2F), and the Qwen3 entry points become standalone graph.sh recipes. The P2P recipes now use the launcher's ports (18000/18001/6239), POD=AFD_<k>, FFN-first placement and 3600 s timeouts. deploy-afd-k8s ships the launcher in the recipe ConfigMap, sets AFD_RECIPE_LIB and SERVE_HOST in the pod instead of patching the script, and reads placement and ports from LAYOUT and the launcher only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.