FFNaaS for qwen3-30b by ronenkat · Pull Request #53 · ms-llmd/afd-plugin

ronenkat · 2026-11-05T18:57:12Z

No description provided.

## Purpose Fix the missing `pp_size` fixture error in `test_npu_async_attention_allows_model_owned_sp` introduced by vllm-project#539. Its parameterization was removed and pipeline parallel size fixed to 1, but the unused function argument remained. Only the stale argument is removed. The test body and assertions are preserved. ## Issue Follow-up to vllm-project#539. [Buildkite vllm-project#225](https://buildkite.com/vllm/afd-plugin/builds/225#01a1a9f1-f0bd-462f-8651-5dacde7b567f) reported 1430 passed, 47 skipped and this one setup error; the subsequent GPU routing command did not execute. ## Test Plan Run the existing `ready` L4 job, verifying both commands complete: - CPU-safe CI selection, with the CI image's runtime dependencies. - `pytest -sv tests/gpu/test_attention_routing.py`, including both renormalization cases without skips. ## Test Result - Before: reproduced `fixture 'pp_size' not found`. - After: the target and adjacent SP tests passed (4 cases) using CPU Torch and temporary upstream import stubs; real AFD construction code and test bodies executed. - Selected-file pre-commit and `git diff --check` passed. - CI validation is pending. The local adapter results do not establish native CUDA/NPU correctness. ## Docs Impact None: this fixes test setup only. --- <details> <summary>Essential PR Checklist</summary> - [x] Purpose is clear and linked to the failing CI. - [x] Scope is one unused test argument. - [x] Pinned runtime contracts are unchanged. - [x] No vLLM checkout or production code changes. - [x] No new classes, compatibility shims or monkey patches. - [x] Imports remain CPU-safe. - [x] Local validation and its limitations are recorded; native CI is pending. - [x] Documentation impact is stated. </details> Signed-off-by: Zhou ziheng <jiaranran2@gmail.com>

…#547) ## Purpose Bring the `review-pr` skill back in line with the code it reviews. The skill carried pins, runner architecture, ownership, and module inventory from the vLLM 0.26 / dual-runner era; several of its statements were actively wrong for a reviewer following them literally, so it would have produced incorrect findings or missed real blockers. ## Issue - No linked issue. This is documentation maintenance for an agent skill; raise an RFC first if the skill's review workflow itself should change. ## Scope - In scope: `.agents/skills/review-pr/**` only. - Out of scope: production code, tests, design pages, CI configuration, and every other skill under `.agents/skills/`. ## Implementation Notes Corrections against the current tree: | Item | Skill said | Head says | | --- | --- | --- | | vLLM pin | `0.26.0` | `0.32.0` at `5df63a72` | | vLLM-Ascend pin | `80d8c194f` | `bd42eea0` | | CI image tag | `VLLM_BASE_TAG=v0.26.0` | `v0.32.0`, asserted in `docker/Dockerfile.ci` | | Runner variants | V1/V2 maintained in pairs; GPU V2 gated off | no V1 and no fallback; both roles require `VLLM_USE_V2_MODEL_RUNNER=1`, Attention runs MRV2, FFN stays connector-driven | | CODEOWNERS | one global rule, no path owners | global catch-all plus `@ronenkat` on GPU and E2E paths, `@francisleely` on `/csrc/npu/` | | Design-page status | all pages treated as draft | `plugin_boundary.md` and `e2e_testing.md` are `normative`; their `ENTRY-INV-*` / `CFG-INV-*` / `E2E-INV-*` are blockers | Structural changes: - **Pins are read, not remembered.** The skill now instructs the reviewer to read `TARGET_VLLM_VERSION`, the design-page `upstream_refs`, `docker/Dockerfile.ci`, `pyproject.toml`, and the README from the frozen head. The values quoted inside the skill are labelled as its refresh-time `main` values, with the reviewed head winning on disagreement, so the next pin move degrades it to a hint rather than to a false assertion. - **Three FFN execution paths** are separated by their real selection contract — `connector.control_plane is not None`, `issubclass(connector_cls, AFDRoutedExpertsConnector)`, and async CAM on NPU — instead of being described as one runner pair. Routing by connector *name* is now a finding. - **The patch contract now covers patched methods in `afd_plugin/v1/worker/**`**, not only `afd_plugin/compat/`. Those files hold the heaviest patch surface (`attention_model_runner_v2.py` has 9 `# ### PATCH START` blocks, the NPU twin 6), and `tests/unit/compat/patches/test_signatures.py` already owns signature alignment for both. A newly patched function with no table row is a P1. - **New `references/modules/e2e-harness.md`** owns `tests/e2e/**`, which previously had no primary contract even though `e2e_testing.md` is one of the two `normative` pages. It records the current gate sets, the four-device bound, sample thresholds, replay evidence rules, and the no-skip rule. - **New `references/features/dspark-draft.md`** covers the DSpark draft model, its speculator patch, target/draft layered FFN wiring, and its exclusion set (no DBO, no FFN two-stream). - **`skip_ci.py` is recorded as an evidence hazard.** A docs-only or skip-mark-only diff suppresses the whole default pipeline, so a green run on such a PR proves nothing about the code. The skill now says this explicitly instead of treating CI as free validation. Machine-readable metadata for the added references is unchanged: the skill keeps its `name` and `agents/openai.yaml` display name. ## Test Plan Static checks only; this change touches documentation. - Resolve every relative Markdown link inside the skill directory. - Confirm every reference file is reachable directly from `SKILL.md`. - Re-verify each path, symbol, E2E case ID, and test file named in the skill against the current tree. - Confirm no stale pin or runner-variant language remains. ## Test Result Run on `codex/refresh-review-pr-skill` at `6bfc44b5`, based on upstream `main` `e778192d`: | Check | Result | | --- | --- | | Markdown files | 25 files, 1577 lines | | Internal relative links | 0 broken | | Reference reachability from `SKILL.md` | 25 of 25 linked | | Stale `0.26` / `80d8c194` / V1-V2 claims | none remaining | | pre-commit (typos, end-of-file, mixed-line-ending, trailing-whitespace, merge-conflict) | passed | | markdownlint-cli2 | not applicable; `.agents/` is excluded by `.pre-commit-config.yaml` | Two errors were found and fixed during verification, before staging: - a reference to `tests/unit/model_executor/npu/`, which does not exist; - a claim that the DeepEP M2N scenarios run in weekly CI — the Qwen3 MoE module collects them, but CI selects node IDs explicitly and does not run them. No runtime, device, or E2E validation applies: no production code changed. ## Docs Impact - Files updated: `.agents/skills/review-pr/SKILL.md` and its `references/` tree (24 modified, 2 added). - No changes to `docs/` or user-facing documentation. The skill is agent instructions, not user documentation, and markdownlint excludes `.agents/` by design. --- <details> <summary>Essential PR Checklist</summary> - [x] Purpose is clear and linked to public context when possible. - [x] Scope is bounded: one skill directory, documentation only. - [x] Compatibility with vLLM v0.32.0 and the pinned vLLM-Ascend source is considered; the skill now reads the pin instead of hard-coding it. - [x] No changes are made to the vLLM source checkout. - [x] Plugin-owned classes or explicit dotted class paths are preferred over monkey patches (not applicable; no production change). - [x] Any compat shim or monkey patch is isolated, idempotent, version-guarded, documented, and tested (none added). - [x] Imports remain CPU-safe; CUDA-heavy work is delayed or GPU-gated (no code changes). - [x] Validation evidence is included; no GPU tests were skipped because none apply. - [x] Documentation impact is stated. </details> 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

Add the gpu-deepep-m2n backend to the standalone FFN service (python -m afd_plugin.ffn_faas serve): BF16 Qwen3MoeForCausalLM routed experts over DeepEPM2NAFDConnector, eager, FULL_DECODE_ONLY graphs and two-lane DBO, without a vLLM Worker, EngineCore, scheduler, KV cache or complete CausalLM. - Host protocol: FFNBackend.execute_step() returns bool and gains cancel_receive(); the host cancels an idle receive before joining. The NPU backend returns True and keeps a no-op cancel. - GPUFFNModelRunner.execute_connector_driven_step(): the Worker loop body, now shared by AFDFFNWorker and the standalone backend. - GPUFFNRoutedExpertsModelRunner.attach_model() and mark_receiver_exited() for an externally loaded model. - ffn_faas/platforms/gpu: CPU-only adapter, settings and Qwen3 checkpoint description; GPU environment (F-only NCCL world, AFD architecture, Triton MoE pin); FFN-only Qwen3 model shell on vLLM's native loader; DeepEPM2NFFNBackend. - Tests: CPU unit tests, a 4-GPU service smoke in tests/gpu, and four Qwen3 M2N FFN-service E2E scenarios in the E2E runner. - Docs: GPU standalone guide and FFN runtime design update. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>

torchrun sets TORCHELASTIC_USE_AGENT_STORE=True in its workers, so every tcp:// rendezvous joins the agent's store as a client. The F-only world needs that, but the connector's M2N rendezvous must be hosted by F rank 0; with the flag set nobody listened and both roles timed out. Drop the flag once vLLM's F-only groups exist. Also fix the new runner unit test (AFDControlPayload needs explicit flags) and the GPU smoke harness (interleaved per-rank announcements, and kill torchrun's workers, which are outside the launcher's process group). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>

Measured by the GPU smoke test: the NCCL watchdog aborts the F ranks after afd_process_group_timeout_s, so no status file is written. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>

## Purpose Add opt-in Pull-based MegaMoE execution for A3 Async CAM, including per-channel W4A8 and W8A8 expert weights. Attention publishes routed inputs; FFN pulls and executes the expert computation, and Attention combines the results. The existing Push/Layered path remains the default. ## Issue - Related issue(s): Not specified. - Closing keyword, only if fully resolved: Not specified. ## Scope - In scope: the Pull protocol and native MegaMoE executor, W4A8/W8A8 execution plans, eager and FULL graph integration, lifecycle negotiation, tests and documentation. - Out of scope: per-group MegaMoE weights, MegaMoE with FFN two-stream, a general-purpose Pull transport, and other hardware generations. ## Implementation Notes - Enable `connector_extra_config.ffn_mega_moe=true` consistently on Attention and FFN ranks; use dynamic quantization and per-channel weights. With Mega disabled, Async CAM retains Push/Layered execution. - The native executor selects the W4A8 or W8A8 plan from the Layered weight metadata. Mega follows the Layered behavior: apply the configured SwiGLU limit for W4A8 and do not apply it for W8A8. - The connector owns the Pull lifecycle through the generic FFN execution-session contract. Attention publishes `pull_quiet`; FFN drains and publishes `pull_stopped` before resources are released. The current runtime helpers are grouped under `afd_plugin/connectors/npu/async_cam_runtime/`. - Receive capacity is `int(262144 * BATCH_SIZE_FACTOR)`. The Pull layout requires `(max_num_batched_tokens / attn_ranks_per_dp) * topk <= 65535`; the matched performance runs use 16384 maximum batched tokens. - Native operators and the PyTorch binding must be built for the selected Ascend/CANN stack. Both CANN 9.1 and 9.2 header layouts are supported. This branch uses plugin-owned integration. ## Test Plan - Exercise CPU-side connector, execution-session, native registration, weight-selection and graph lifecycle contracts. - Build the native operators for A3 and check the Pull layout and representative W4A8/W8A8 native execution. - Run real-model GSM8K inference and matched Mega ON/OFF end-to-end performance tests. Keep request correctness, actual graph/DBO coverage, service cleanup and timing as separate results. - Validate Mega ON lifecycle and numerical behavior on the target A3 stack. Record baseline behavior as measurement context. ## Test Result **Revision and environment.** The E2E timing runs used production sources at `f47c992234b1a11c7d78f1a250d2f10911f1ce5d` with working-tree E2E shutdown-budget changes; the current Pull worker timeout change was tested separately and was not included in those timings. The CANN 9.2 native build was reused, with source and binary hashes and per-rank loaded libraries checked. Runtime: vLLM 0.32, torch-npu 2.10.0.post7, 16 A3 devices per precision. **CPU checks.** On the current working tree, the branch-related unit-test selection completed with 183 passed and 44 skipped; skips require PyTorch, vLLM, NumPy or the compiled Ascend 910C extension. The build and CANN activation-generation tests contributed 32 of the passed tests. The package and Async CAM startup module import successfully without PyTorch or vLLM. Ruff passed for the branch's changed Python files. GPU E2E tests were not run. **GSM8K inference.** With Mega ON, both models used Attention DP2×TP4 / FFN EP8×TP1, 8-shot GSM8K, 300 samples, concurrency 12 and maximum generation 512: W4A8 DSpark 287/300 strict matches (95.67%); W8A8 MTP 292/300 (97.33%). Inference completed, but the runs reported shutdown failure due to `EngineDeadError`; these are not clean complete-run passes. The W8A8 performance topology below is different and has not received a separate GSM8K run. **End-to-end performance.** Each precision used one ON→OFF pair on a fresh service, with the same frozen inputs, model and settings within the pair. W4A8 DSpark used Attention DP6×TP2 / FFN EP4×TP1; W8A8 MTP used Attention DP4×TP2 / FFN EP8×TP1. W8A8 EP4 expert tensors alone require about 66.17 GiB per FFN rank, exceeding 64 GiB before other allocations. Attention used eager+DBO; FFN used FULL graph; max model length 65537, max batched tokens 16384, memory utilization 0.92 and receive capacity 262144. All 2256 formal requests and 127 warmups per mode completed successfully; actual DP counters, FFN graph replay and Attention decode DBO execution were checked. Six fixed-length prefill cases and the imbalanced case used 96 requests at an offered 6 RPS; the mixed case used 1536 frozen requests at 6 RPS; long and short decode used 24 requests in two 12-concurrent waves. The long/short input/output lengths were 32768/512 and 128/256. W4 and W8 use different checkpoints and topologies, so their numbers do not isolate quantization effects. TTFT includes queueing; offered-rate throughput is not a capacity measurement. Each pair has only one ordering and does not establish stable gains. **Important:** All four timed performance runs logged FFN receive shutdown timeouts, so they are timing results, not clean E2E passes. ON/OFF outputs were not fully identical. Fixed-length cases compare returned text; other cases compare returned token IDs. A matching output count is not an accuracy result. <details> <summary>W4A8 measured performance</summary> TTFT: mean / P50 / P99 in ms; throughput: input tokens/s. Δ is ON relative to OFF. | Scenario | OFF TTFT | ON TTFT | Δ mean | OFF / ON throughput | Matching outputs | |---|---:|---:|---:|---:|---:| | tokens-32768 | 19,898.29 / 19,862.05 / 36,696.68 | 18,204.61 / 18,157.37 / 33,493.12 | -8.51% | 59,937.35 / 63,733.30 | 96/96 | | tokens-16384 | 6,044.64 / 6,040.31 / 10,458.61 | 4,949.81 / 4,959.88 / 8,427.10 | -18.11% | 59,832.09 / 64,590.81 | 96/96 | | tokens-8192 | 777.76 / 779.34 / 796.59 | 720.68 / 720.33 / 728.29 | -7.34% | 47,431.37 / 47,496.66 | 95/96 | | tokens-4096 | 408.36 / 406.32 / 448.01 | 399.38 / 396.99 / 435.00 | -2.20% | 24,210.59 / 24,225.80 | 93/96 | | tokens-2048 | 267.10 / 263.30 / 307.51 | 267.63 / 267.24 / 282.17 | +0.20% | 12,213.55 / 12,200.95 | 96/96 | | tokens-128 | 116.77 / 116.02 / 124.87 | 118.54 / 117.77 / 125.29 | +1.52% | 770.24 / 770.01 | 96/96 | | imbalanced-prefill | 1,812.49 / 261.49 / 8,268.94 | 1,737.14 / 273.46 / 7,835.66 | -4.16% | 23,441.98 / 23,875.72 | 96/96 | | mixed1536 | 5,791.89 / 5,338.73 / 14,933.90 | 3,563.77 / 3,020.74 / 10,996.32 | -38.47% | 56,543.49 / 57,358.96 | 1455/1536 | Decode TPOT: mean / P50 / P99 in ms/token, measured in each wave’s common active window after excluding one second at each edge; throughput: output tokens/s. | Scenario | OFF TPOT | ON TPOT | Δ mean | OFF / ON throughput | Matching outputs | |---|---:|---:|---:|---:|---:| | long-context-decode | 263.65 / 259.21 / 290.51 | 268.34 / 266.29 / 279.74 | +1.78% | 45.62 / 44.73 | 10/24 | | short-context-decode | 264.75 / 260.95 / 292.34 | 270.54 / 268.23 / 280.32 | +2.19% | 45.45 / 44.43 | 21/24 | </details> <details> <summary>W8A8 measured performance</summary> TTFT: mean / P50 / P99 in ms; throughput: input tokens/s. Δ is ON relative to OFF. | Scenario | OFF TTFT | ON TTFT | Δ mean | OFF / ON throughput | Matching outputs | |---|---:|---:|---:|---:|---:| | tokens-32768 | 30,589.92 / 30,560.19 / 58,149.17 | 29,556.24 / 29,581.49 / 55,993.40 | -3.38% | 42,506.98 / 43,709.37 | 96/96 | | tokens-16384 | 10,757.98 / 10,748.58 / 20,025.23 | 10,194.38 / 10,203.84 / 18,919.10 | -5.24% | 43,859.78 / 45,267.12 | 96/96 | | tokens-8192 | 1,656.57 / 1,580.03 / 2,786.48 | 1,267.61 / 1,177.10 / 2,254.43 | -23.48% | 43,642.46 / 45,098.99 | 96/96 | | tokens-4096 | 412.64 / 410.34 / 445.70 | 398.14 / 396.35 / 424.27 | -3.51% | 24,207.19 / 24,226.81 | 95/96 | | tokens-2048 | 268.11 / 264.99 / 287.66 | 271.28 / 269.27 / 297.56 | +1.18% | 12,215.54 / 12,203.56 | 92/96 | | tokens-128 | 117.49 / 116.44 / 126.13 | 120.10 / 120.32 / 127.11 | +2.22% | 770.39 / 770.01 | 96/96 | | imbalanced-prefill | 5,709.20 / 265.18 / 30,758.92 | 5,450.46 / 274.47 / 29,287.33 | -4.53% | 11,806.21 / 12,199.14 | 96/96 | | mixed1536 | 48,893.54 / 48,856.68 / 111,240.40 | 43,534.13 / 40,755.28 / 86,161.97 | -10.96% | 41,786.15 / 44,340.77 | 1452/1536 | Decode TPOT: mean / P50 / P99 in ms/token, measured in each wave’s common active window after excluding one second at each edge; throughput: output tokens/s. | Scenario | OFF TPOT | ON TPOT | Δ mean | OFF / ON throughput | Matching outputs | |---|---:|---:|---:|---:|---:| | long-context-decode | 261.90 / 261.01 / 270.54 | 267.10 / 268.36 / 273.72 | +1.99% | 45.84 / 44.96 | 14/24 | | short-context-decode | 262.94 / 262.84 / 271.58 | 267.92 / 268.83 / 274.27 | +1.89% | 45.70 / 44.87 | 18/24 | </details> **Current shutdown validation.** With the working-tree Pull-session timeout change, 32K prefill and short decode completed and all worker groups exited normally for W4A8 ON (10 groups) and W8A8 ON (12 groups). W8A8 logged `EngineDeadError` after an explicit shutdown signal, worker exit and MPClient shutdown. The worker test file passed 75 cases, included in the CPU result above. ## Docs Impact - Files updated: `csrc/npu/README.md`, `docs/design/module/execution_platforms.md`, `docs/design/module/ffn_runtime.md`, and `docs/npu/ASYNC_FFN_RUNTIME.md`. - If none, reason: Not applicable. --- <details> <summary>Essential PR Checklist</summary> - [x] Purpose is clear and linked to public context when possible. - [x] Scope is bounded. - [x] Compatibility with vLLM v0.32.0 and, for NPU changes, the pinned vLLM-Ascend source is considered, as applicable. - [x] No changes are made to the vLLM source checkout. - [x] Plugin-owned classes or explicit dotted class paths are preferred over monkey patches. - [x] Any compat shim or monkey patch is isolated, idempotent, version-guarded, documented, and tested. - [x] Imports remain CPU-safe; CUDA-heavy work is delayed or GPU-gated. - [x] Validation evidence is included, including skipped GPU tests when applicable. - [x] Documentation impact is stated. </details> --------- Signed-off-by: lirx-pd <616517220@qq.com>

## Purpose Remove the NPU Async CAM FFN two-stream execution path, simplifying FFN startup and execution around the remaining eager and FULL paths. ## Issue - Related issue(s): Not linked. - Closing keyword, only if fully resolved: None. ## Scope - In scope: Remove the two-stream executor, its connector options and startup negotiation, associated tests, and documentation; update the remaining FFN runner and worker paths. - Out of scope: Changes to the native CAM operators, Attention DBO, or the behavior and configuration of the existing layered and MegaMoE executors. ## Implementation Notes - Remove `AsyncFFNTwoStream` and its eager/FULL capture, submission, completion, and shutdown branches. FFN continues through its existing eager or single-graph FULL execution path; Pull retains its session-driven drain. - Remove `ffn_two_stream`, `ffn_compute_aic_num`, `ffn_compute_aiv_num`, and `ffn_combine_aiv_num` from `connector_extra_config`. **This is a configuration compatibility break**: any of these fields, including `ffn_two_stream=false`, is now rejected as unknown. Remove them from existing configurations before upgrading. - Remove two-stream-specific startup signatures, Attention validation, and resource-budget checks. MegaMoE still requires `dynamicQuant=1`. - The FFN startup plan changes; deploy Attention and FFN ranks with the same plugin version. No new compatibility shim or monkey patch is introduced. - Delete two-stream-specific tests and update configuration, startup, graph, worker, and fake coverage for the remaining paths. Add coverage that the removed options are rejected. ## Test Plan - Run CPU unit tests, including connector configuration and startup, graph negotiation, and rejection of the retired options. - Run matched real-model GSM8K evaluations for Layered and Mega ON using Async CAM FULL on A3. - Check repository deployment examples for the four retired options. ## Test Result - Branch: `chore/remove-ffn-two-stream`, commit `28fcfb706cb505cac9550e9e76fbfc909214891d`, based on `upstream/main` at `022d8a7e7a521994f0bbda5d9c278c71e8ceb1dd`. The tested patch has SHA256 `e43d9afb1f1773eea9a6560a247156ea46af243716e36a15c543b2209ab7cad3`; the commit adds one Markdown blank line after testing. - `git diff --check` and applicable pre-commit checks, including Ruff, Markdown lint and mypy: passed. The affected CPU suites reported **127 passed, 5 skipped**; the skips require local `torch` or `vllm`. - CPU unit suite, `pytest -o addopts='' -q tests/unit -m 'not gpu and not vllm_runtime' -ra`, with isolated `setuptools 80.10.2`: **621 passed, 91 skipped, 8 deselected, 10 subtests passed**. Skips include unavailable `torch`, `vllm`, `vllm-ascend`, `numpy`, and `cmake`. CUDA tests were not run. - A3 real-model evaluation used DeepSeek-V4 Flash W4A8, Attention DP2×TP4 eager, FFN EP8×TP1 FULL, DBO off, 8-shot GSM8K, 300 samples, concurrency 12, seed 1024, and up to 512 generated tokens. Both modes used identical question IDs, prompts and inputs. | FFN mode | GSM8K strict match | GSM8K flexible extract | | --- | ---: | ---: | | Layered | 286/300 (95.33%) | 286/300 (95.33%) | | Mega ON | 286/300 (95.33%) | 286/300 (95.33%) | All eight FFN ranks captured and replayed the expected FULL graph in each mode. The 14 unmatched questions in each run include 10 shared cases and four cases unique to each mode. - The A3 image used `vllm 0.32.0+empty`, `vllm-ascend 0.19.1rc2.dev3100+gbd42eea01`, `torch 2.10.0+cpu`, and `torch-npu 2.10.0.post7`; the installed vLLM package did not record its source commit. Existing native binaries were verified against their recorded hashes and reused; the native build inputs match the recorded build except for a removed comment in one header. A fresh native build and FFN eager-mode device validation were not run. - Local raw results and per-sample outputs: `results/remove-ffn-two-stream-022d8a7e/`. ## Docs Impact - Files updated: `csrc/npu/README.md`, the connector, execution platform, FFN runtime and model integration design documents, the NPU Async FFN runtime and Async CAM connector guides, and the DeepSeek-V4 Flash recipe. - If none, reason: Not applicable. --- <details> <summary>Essential PR Checklist</summary> - [x] Purpose is clear and linked to public context when possible. - [x] Scope is bounded. - [x] Compatibility with vLLM v0.32.0 and, for NPU changes, the pinned vLLM-Ascend source is considered, as applicable. - [x] No changes are made to the vLLM source checkout. - [x] Plugin-owned classes or explicit dotted class paths are preferred over monkey patches. - [x] Any compat shim or monkey patch is isolated, idempotent, version-guarded, documented, and tested. - [x] Imports remain CPU-safe; CUDA-heavy work is delayed or GPU-gated. - [x] Validation evidence is included, including skipped GPU tests when applicable. - [x] Documentation impact is stated. </details> Signed-off-by: lirx-pd <616517220@qq.com>

Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>

Keep standalone model attachment with the updated Async CAM diagnostics. Follow vllm-project#550 by removing obsolete FFN two-stream options and configuration keys. Signed-off-by: jiaran-king <jiaranran2@gmail.com>

Signed-off-by: jiaran-king <jiaranran2@gmail.com>

## Purpose Separate Async CAM Push and Pull session ownership so each protocol keeps its transaction state and lifecycle rules together. The connector retains shared payload validation, HCCL ownership and startup coordination. ## Issue - Related issue(s): Not linked. - Closing keyword, only if fully resolved: None. ## Scope - In scope: The common FFN session base, Async CAM connector and sessions, associated tests and fakes, the device harness, and connector/runtime documentation. - Out of scope: Model and expert-weight interfaces, Layered/Mega compute implementations, runner execution orchestration, static validation consolidation, native kernels and ABI, and changes to supported configurations or defaults. ## Implementation Notes - Rename the Pull module to `async_cam_session.py` and keep `PushCAMSession` and `PullCAMSession` together. The connector selects one session at construction. - Push owns a FIFO per Attention stage, routing references and its placeholder. It records routing after a successful dispatch and removes it after a successful combine. FFN retains separate receive, compute and send operations. - Pull owns one in-flight window across stages, native publish/combine, status/idle/source state and the quiet/drain/stopped handshake. It receives Startup, topology and required dimensions explicitly, without retaining the connector. Ownership is recorded before dispatch; failures retain the window and routing references. DBO checks sticky status at the Attention step boundary. - Merge the lifecycle interface and default implementation into one concrete `FFNExecutionSession`. Push and Pull inherit it; remove `DefaultFFNSession` and duplicate default hooks. Custom sessions should inherit `FFNExecutionSession` and override the required hooks. Keep torch and metadata imports in the base module under type checking so session imports remain safe before device bootstrap. - Initialize sessions after negotiation and HCCL setup. Broadcast initialization failures before marking the connector ready. Check session shutdown requirements before destroying HCCL and releasing references. Read-only `comm_args` and `comm_id` expose the live session resources; `AFDAsyncTransferState` retains its existing definition path. - Mega FFN continues to use its fused transaction. The connector rejects separate FFN receive/send calls for Pull. Eager/FULL execution, graph references, service threads and device completion remain with the runner and worker. - Share native argument preparation across transfer methods and use registered schema keyword names. Argument values and ABI remain unchanged. Diagnostic input fields now follow the native schema and continue to report tensor shape, dtype and device without metadata reads. - Move host message-contract coverage into CPU-safe tests and shared recording fakes. Cover stage isolation, Pull exclusivity, combine failure retention, unsupported Pull entry points, closed-connector rejection and startup failure propagation. - Use the public pending-state observation in the device harness. Apply the whole-expert receive-capacity assertion only to Push; baseline and candidate use the same harness correction. ## Test Plan - [x] Full CPU unit suite, recording missing-dependency skips separately. - [x] Changed-file pre-commit checks, including Ruff, mypy 3.11, Markdown lint, SPDX and whitespace checks. - [x] Related units with actual torch, vLLM and vLLM-Ascend dependencies on A3. - [x] Matched W4A8/W8A8, Layered/Mega and eager/FULL device transactions, including TP2, capacity boundaries, empty work, graph replay, Pull idle/drain and resource cleanup. - [x] Matched real-model concurrent requests with active Attention DBO. ## Test Result Results cover base `28fcfb706cb505cac9550e9e76fbfc909214891d` and the current patch. - CPU suite with isolated setuptools 80.10.2: **642 passed, 91 skipped, 8 deselected, 10 subtests passed**. Skips require unavailable torch, vLLM, vLLM-Ascend, numpy or cmake. One environment warning reports Babel's deprecated `cgi` import. - Changed-file pre-commit, Ruff, mypy 3.11, Markdown lint and `git diff --check`: passed. - A3 related units: **209 passed, 17 warnings** from dependency deprecations. - Finite device transactions: **16 matched baseline/candidate runs passed** across W4A8/W8A8, Layered/Mega and eager/FULL. All **80 candidate output tensors exactly matched baseline eager outputs**. Graph capture/replay passed, including eight candidate captures; all ranks completed cleanup, and Mega idle/shutdown passed. Configuration: 2A/2F, Attention TP2, synthetic hidden/intermediate dimensions 256, layers (2, 5), seed 1024, Push receive capacity 32 and Pull capacity 16. Eight baseline runs were reused after verifying unchanged baseline source, native artifacts and fixed inputs; all eight final candidate runs were rerun. - Real model: DeepSeek-V4 Flash W4A8, Attention DP2×TP4, FFN EP8×TP1 Layered eager, token budget/receive capacity 8192 and seed 1024. Baseline and candidate each passed the same **10 concurrent arithmetic requests**, with identical prompts and answers and **E2E exit code 0**. All eight Attention ranks recorded **four nonempty two-stage executions**. DBO prefill/decode thresholds were 2, temperature was 0 and maximum generation length was 256 tokens. The test-only observer enables existing execution evidence without changing production execution or answer assertions. - Cleanup: no NPU processes remained after the model run. The model harness uses process termination; both baseline and candidate emitted two `KeyboardInterrupt` lines and two resource-tracker warnings. Exit code 0 does not establish graceful shutdown of every FFN process. - Environment: CANN 9.2.0, vLLM `0.32.0+empty` at `5df63a7203cdb9328d824a27b682e77c7b9928be`, vLLM-Ascend `0.19.1rc2.dev3100+gbd42eea01` at `bd42eea01d8312c4df469fabaed45f77a12df4d3`, torch `2.10.0+cpu` and torch-npu `2.10.0.post7`. All 47 reused native artifacts matched their recorded hashes. No upstream checkout was modified. CUDA validation, full GSM8K evaluation, performance qualification and a fresh native build were not run. Synthetic transactions and concurrent arithmetic checks do not establish full model accuracy or performance. ## Docs Impact - Files updated: Connector contracts, execution platforms, FFN runtime, Async CAM user/runtime guides. - If none, reason: Not applicable. - Document session ownership, lifecycle defaults, live resource access, shutdown requirements and the validation scope. Register the new host tests in the owning modules' validation paths. --- <details> <summary>Essential PR Checklist</summary> - [x] Purpose is clear and linked to public context when possible. - [x] Scope is bounded. - [x] Compatibility with vLLM v0.32.0 and, for NPU changes, the pinned vLLM-Ascend source is considered, as applicable. - [x] No changes are made to the vLLM source checkout. - [x] Plugin-owned classes or explicit dotted class paths are preferred over monkey patches. - [x] Any compat shim or monkey patch is isolated, idempotent, version-guarded, documented, and tested. No production shim or monkey patch is added. - [x] Imports remain CPU-safe; CUDA-heavy work is delayed or GPU-gated. - [x] Validation evidence is included, including skipped GPU tests when applicable. - [x] Documentation impact is stated. </details> Signed-off-by: lirx-pd <616517220@qq.com>

Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>

GPUFFNRoutedExpertsModelRunner.get_model() returns Any, so assigning it to the Optional self.model attribute left the value typed as possibly None when it was passed to parameter_storage_bytes, and the mypy-3.11 pre-commit hook failed on backend.py. Bind the loaded model to a typed local first, then store it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>

torchrun starts its workers in their own sessions, outside the launcher process group that the E2E harness owns and signals. A SIGTERM to torchrun lets it stop the workers itself (a cancelled FFN service case on 4xH100 left no workers behind), but when torchrun does not exit within the harness grace period, the group SIGKILL reaches only torchrun and its workers survive holding GPUs into the next case (E2E-INV-003/E2E-INV-006). Give the FFN service scenarios an E2E run id so the existing environment-matched FFN sweep also reaps the torchrun workers, as the NPU async path already does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>

validate_external_model_attachment was extracted for the GPU and NPU attach_model methods. The GPU attach_model was removed when the standalone GPU service moved to loading through the routed runner's native load_model, so the helper had a single NPU caller and a new v1/worker module with no owning design page. Restore the guard inline in AsyncNPUFFNModelRunner.attach_model and delete the module. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>

Resolve the model path in the CPU adapter and let vLLM model config plus the AFD registry select the FFN-only shell. Remove the duplicate Qwen3 checkpoint parser and derive expert progress from the loaded runner. Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>

…ing (vllm-project#533) ## Goal Simplify the `deploy-afd-k8s` skill: one readiness check, consistent timeouts, and recipe parsing in a single place. ## Changes 1. **Single readiness signal** (`resources/deploy.md` step 4f): wait for each pod's `Ready` condition, which the existing `readinessProbe` already drives, and stop early when a pod goes `Failed`. This replaces the second pass that grepped pod logs for the `READY`/`ERROR:` lines. 2. **Consistent timeouts** (`templates/pod.yaml`): the per-log 40-minute loops and the separate `/health` loop become one 55-minute startup budget. It stays below the 60-minute `Ready` wait in step 4f, so a slow pod always fails with its own `ERROR:` reason first. The `/health` wait also notices if the recipe process has exited. 3. **One recipe parser** (`resources/resolve-recipe.md`): step 1 records `GPU_COUNT` for each pod (and for a plain recipe), and `deploy.md` step 2 just uses it. This removes the `awk` role-block parser. ## Shutdown instructions `resources/report-and-teardown.md` step 5 now ends by asking the caller (the user, or the task that invoked the skill, e.g. `run-vllm-bench-k8s`) when to delete the running pods. The caller owns that decision and triggers teardown (step 6). The skill never tears down on its own, and the model PVC is still kept. Also renames `references/` to `resources/` and updates the links. ## Validation - The rendered `pod.yaml` parses as YAML, and its startup script passes `bash -n`. - The startup script, run locally against a fake recipe: it reaches `READY`, fails with `ERROR:` when its time budget runs out, and fails when the recipe exits early. - The step 4f wait loop, run against a stubbed `kubectl`: it finishes when every pod is `Ready` and exits non-zero with logs when a pod is `Failed`. - Every relative link resolves, and pre-commit passes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

Retain standalone CLI/configuration reuse while adopting the Async CAM session ownership refactor and current runtime documentation. Signed-off-by: jiaran-king <jiaranran2@gmail.com>

## Summary - Replace the WeChat QR image currently referenced by the README. ## Note - The QR image says it is valid for seven days, until 2026-11-16; refresh it after expiration if needed. Signed-off-by: David Chen <530634352@qq.com>

## Purpose Add path-based labels for pull requests, and make the CI-failure issue form use the existing label name. This change is GitHub configuration only. It does not change Buildkite pipelines, CI trigger semantics, or runtime code. Pull requests currently get no automatic type or platform labels. Issue Forms already set `bug`, `feature request`, `RFC`, `documentation`, and (once this form matches the repo) `CI-failure`. `ready`, `merge-test`, and `weekly-test` stay manual because Buildkite reads them. ## Issue - Related issue(s): none - Closing keyword, only if fully resolved: ## Scope - In scope: - `.github/labeler.yml` maps changed paths to the existing `Ascend`, `NVIDIA`, and `documentation` labels. - `.github/workflows/labeler.yml` runs `actions/labeler@v5` with `contents: read` and `pull-requests: write`. - `.github/ISSUE_TEMPLATE/500-ci-failure.yml` now requests `CI-failure`, which is the label that already exists. The form previously requested `ci-failure`, so form-created issues did not receive the repo label. - Out of scope: - Creating labels, or auto-applying `ready`, `merge-test`, `weekly-test`, `high priority`, `bug`, `feature request`, `RFC`, or `CI-failure`. - Buildkite pipelines and any other CI trigger behavior. ## Implementation Notes `pull_request_target` is used so pull requests from forks can be labeled. A `pull_request` workflow from a fork only gets a read-only token. `pull_request_target` runs the workflow file from the base branch and can write labels. Untrusted pull request content is not checked out and is not executed. The workflow has no checkout step and no `run` step. `actions/labeler` reads the changed-file list and `.github/labeler.yml` from the base repository through the GitHub API. `sync-labels` is `false`, so the workflow only adds configured labels and does not strip labels a person added. Labels that are not in the config, including the Buildkite triggers, are never added or removed. The job does not request `issues: write`, so it cannot create new labels. `dot: true` is set explicitly so globs match `.buildkite` and `.agents`. Because `pull_request_target` uses the workflow on the default branch, this workflow does not label this pull request. It starts labeling after merge. ### Path to label rules Checked against the current tree with minimatch 9, the matcher `actions/labeler` v5 uses. No tracked file receives both `Ascend` and `NVIDIA`. **Ascend** - Any `npu` directory: `afd_plugin/**/npu/**`, `csrc/npu/**`, `docs/npu/**`, `recipe/npu/**`, `tests/npu/**`, and the unit-test mirrors. There is no `.buildkite/npu` directory. - Path segments or filenames containing `npu`, `ascend`, `cam`, or `dspark` (CAM/CAMP2P connectors and NPU DSpark). Uppercase `CAM` recipe and doc names sit under `npu/` and match the directory rule. - NPU-only skill trees: `.agents/skills/upgrade-npu/**` (matched by the `npu` segment), `.agents/skills/upgrade-layered-gmm-ops/**`, `.agents/skills/deploy-dsv4-flash-async/**`. - NPU tests whose filenames do not contain those tokens: async FFN worker tests, the mega-activation package test, the DSV4 Flash async deploy skill test, and the NPU-only DSV4 Flash E2E files `config.py`, `ffn_graph.py`, `ffn_graph_evidence.py`, and `test_deepseek_v4_flash_multi_pod.py`. **NVIDIA** - Any `gpu` directory: `afd_plugin/connectors/gpu/**`, `csrc/gpu/**`, `docs/gpu/**`, `recipe/gpu/**`, `tests/gpu/**`, plus `.agents/skills/upgrade-gpu-version/**`. - Filenames containing `gpu`, `cuda`, or `deepep`, including `cuda_graph.py`, `gpu_*.py`, and DeepEP unit tests. - `.buildkite/cuda/**` and `docker/Dockerfile.ci` (the CUDA `vllm-openai` CI image). `docker/` as a whole is not labeled. - NCCL P2P unit tests `tests/unit/connectors/test_p2p_*.py`. A broad `*p2p*` glob is not used, because it would also match Ascend CAMP2P and `docs/npu/CAM_P2P_CONNECTOR_USER_GUIDE.md`. - GPU-only E2E suites `tests/e2e/models/qwen3_moe/**` and `tests/e2e/models/qwen3_6/**`. - GPU-only skill trees `.agents/skills/deploy-afd-k8s/**` and `.agents/skills/run-vllm-bench-k8s/**`. Shared trees stay unlabeled, including `tests/e2e/models/deepseek_v2_lite/**`, `tests/e2e/runner.py`, and connector factory/base modules. DSV4 Flash GPU files (`test_gpu.py`, `gpu_config.py`) get `NVIDIA` only. **documentation** - `docs/**` (including `docs/npu/**` and `docs/gpu/**`, which also get the platform label) - root `README.md` - `recipe/**/*.md` Module READMEs under code, tests, `csrc`, and `.agents` are not labeled, so a code change does not pick up `documentation`. Recipe shell scripts are not labeled `documentation`. ## Test Plan Local check: every tracked path was matched with minimatch 9 and `dot: true`. Assertions covered the NPU, CUDA, DeepEP, CAM, and docs examples above, and confirmed the forbidden label names are not keys. After merge, maintainers can verify on draft pull requests: 1. Open a draft PR that only touches `docs/npu/TESTING.md`. Expect `Ascend` and `documentation`. Expect `ready`, `merge-test`, `weekly-test`, `high priority`, `bug`, `feature request`, `RFC`, and `CI-failure` to stay absent. 2. Open a draft PR that only touches `tests/gpu/` or `.buildkite/cuda/`. Expect `NVIDIA` only. 3. On that draft, add `ready` by hand. This workflow should not add or remove it. Buildkite remains the reader of that label. 4. Open a CI failure issue from the form and confirm GitHub applies the existing `CI-failure` label. ## Test Result Config-only. No runtime, GPU, or NPU tests apply. `actionlint` 1.7.7 and the repo pre-commit checks for these files passed (YAML, trailing whitespace, typos, actionlint). ## Docs Impact - Files updated: `.github/labeler.yml`, `.github/workflows/labeler.yml`, `.github/ISSUE_TEMPLATE/500-ci-failure.yml` - If none, reason: user-facing docs under `docs/` were not changed. The new behavior is described in this pull request.  <div><a href="https://cursor.com/agents/bc-bbf80cec-2010-5d1c-9e80-fed71c403c9d?cursor_ref=pr_footer&cursor_cta=open_in_web"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/open-in-web-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/open-in-web-light.png"><img alt="Open in Web" width="114" height="28" src="https://cursor.com/assets/images/open-in-web-dark.png"></picture></a> <a href="https://cursor.com/background-agent?bcId=bc-bbf80cec-2010-5d1c-9e80-fed71c403c9d&cursor_ref=pr_footer&cursor_cta=open_in_cursor"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/open-in-cursor-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/open-in-cursor-light.png"><img alt="Open in Cursor" width="131" height="28" src="https://cursor.com/assets/images/open-in-cursor-dark.png"></picture></a> </div> Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com>

…oject#558) ## Purpose The module index omitted two owned runtime files, left `routing_ids.py` without a primary document, and missed five operational guides. Align the index with module front matter, complete the README configuration reference, and replace references to nonexistent multi-pod drivers with the actual deployment instructions. ## Issue Reported during a documentation audit; no linked issue. ## Scope - In scope: documentation ownership, guide links, configuration reference, multi-pod provisioning instructions, and review metadata. - Out of scope: runtime behavior, test implementation, and hardware qualification. ## Implementation Notes - Add `dspark_speculator.py` and `ffn_routed_experts_model_runner.py` to their existing Attention/FFN index rows. - Assign `compat/npu/routing_ids.py` to `compatibility_and_patches`, with its adapter behavior and existing validation references. - Link the DeepEP M2N, Async FFN runtime, layered W13/W2 validation, NPU custom-op build, and E2E guides. - Document `async_dp` and `afd_process_group_timeout_s`, including defaults and constraints. - Replace nonexistent Kubernetes/Docker driver references with the existing multi-pod deployment guide. - Refresh six `last_reviewed` dates to 2026-11-09 and state the current vLLM 0.32.0 (`5df63a72`) / vLLM-Ascend `bd42eea0` targets. The dates describe documentation consistency checks; historical hardware evidence retains its original scope. ## Test Plan - Run all applicable pre-commit hooks on the eight changed Markdown files and `git diff --check`. - Compare the production routing table with module front matter and verify unique ownership of tracked production files. - Resolve local Markdown links and heading targets in changed documents. - Parse the README JSON through `parse_afd_config` under Python 3.11 and compare its keys with `AFDConfig` fields. ## Test Result - All applicable pre-commit hooks passed, including Markdown lint, typos, whitespace, and merge-conflict checks; file-type-specific hooks had no applicable files. - `git diff --check` passed. - All seven production routing rows match front matter; 297 tracked production files each have exactly one primary owner. - All 156 checked local Markdown links and heading targets resolve. - README JSON includes all 10 configuration fields, passes parsing and validation, and matches the documented defaults. - GPU/NPU tests and native builds were not run for this documentation-only change. ## Docs Impact Eight files updated: root `README.md`, `tests/e2e/multi_pod/README.md`, and six documents under `docs/design/module/` (`index`, `attention_runtime`, `compatibility_and_patches`, `e2e_testing`, `execution_platforms`, and `plugin_boundary`). --- <details> <summary>Essential PR Checklist</summary> - [x] Purpose is clear and linked to public context when possible. - [x] Scope is bounded. - [x] Compatibility with vLLM v0.32.0 and, for NPU changes, the pinned vLLM-Ascend source is considered, as applicable. - [x] No changes are made to the vLLM source checkout. - [x] Plugin-owned classes or explicit dotted class paths are preferred over monkey patches. (No code changes.) - [x] Any compat shim or monkey patch is isolated, idempotent, version-guarded, documented, and tested. (No shim or patch changes.) - [x] Imports remain CPU-safe; CUDA-heavy work is delayed or GPU-gated. (No import changes.) - [x] Validation evidence is included, including skipped GPU tests when applicable. - [x] Documentation impact is stated. </details> Signed-off-by: jiangkuaixue123 <jiangxiaozhou111@163.com>

…llm-project#559) ## Purpose Current documentation still describes removed V1 runners, obsolete Async CAM settings, and a Qwen-only DeepEP M2N restriction. Remove stale guidance and reconcile the remaining descriptions with the current implementation. This content-only pass updates 13 Markdown files and removes 108 lines overall. ## Issue Documentation audit follow-up to vllm-project#558; no issue to close. ## Scope - In scope: runtime descriptions, connector/model constraints, configuration reference, native-operator guidance, and recipe discoverability. - Out of scope: code, tests, directory restructuring, CI changes, and new hardware qualification. ## Implementation Notes - Delete V1 execution descriptions and matrix rows, the unsupported DSV2/V3 Async CAM SP description, legacy connector-flag advice, and the outdated recipe tree. Replace repeated limits with links; retain historical evidence links and known correctness/shutdown limitations. - Describe CUDA runner construction through `_make_model_runner()` and NPU's required ModelRunnerV2 path. - Correct DeepEP M2N admission to Qwen3 MoE or DeepSeek-V2 and Qwen's connector-specific gate placement; link the existing DeepSeek-V2-Lite recipe. - Correct `connector_extra_config` storage/hash semantics. Complete the existing Async CAM parameter table with `ffn_mega_moe`, and distinguish default layered execution from Mega/Pull. - Correct native CAM operator selection and eager-Attention versus eager/FULL-FFN descriptions. Checked against `afd_plugin/config.py`, `validation.py`, CUDA/NPU workers, `compat/npu/{feature_validation,forward_context,runtime_config}.py`, the Qwen/DeepSeek wrappers, and Async CAM connector/executor code at base `31652190`. Runtime pins remain vLLM 0.32.0 (`5df63a72`) and vLLM-Ascend `bd42eea0`. Review dates indicate documentation checks, not new device results. ## Test Plan Run applicable pre-commit hooks, check local links/anchors, parse documented common AFD configurations, and compare the Async CAM parameter table with its source schema. ## Test Result - All applicable pre-commit hooks passed on the 13 changed files; `git diff --check` passed. - 335 local Markdown/HTML links and anchors checked across 41 non-agent documents. - 10 configuration examples passed the common AFD parser; README covers all 10 fields. Confirmed extra config is retained and affects the hash. - Async CAM table matches all four accepted source-schema keys; README E2E scenario names match runner definitions. - Checks ran on CPU-only macOS / Python 3.12.13 without torch or vLLM. No GPU/NPU execution, E2E, or native build was run for this documentation-only change. ## Docs Impact Root README, recipe index, NPU native-build README, seven module-design pages, NCCL P2P guide, Async CAM guide, and Async FFN runtime. No new documents or file moves. --- <details> <summary>Essential PR Checklist</summary> - [x] Purpose is clear and linked to public context when possible. - [x] Scope is bounded. - [x] Pinned vLLM / vLLM-Ascend compatibility is considered. - [x] No changes to the vLLM source checkout. - [x] Plugin-owned extension and patch requirements remain unchanged. - [x] Imports remain CPU-safe; no code changes. - [x] Validation evidence and hardware-validation limits are stated. - [x] Documentation impact is stated. </details> Signed-off-by: jiangkuaixue123 <jiangxiaozhou111@163.com>

…project#555) ## Purpose Let the DeepEP M2N connector run pods that each hold Attention and FFN ranks (for example 2 x 4A1F), so a deployment grows by adding identical nodes. The handshake plans the transport order from NVLink domains and uses afdEP-comm v1.1.0's per-domain mode; FFN-prefix placements keep their current path. ## Issue - Related issue(s): vllm-project#554 - Closing keyword, only if fully resolved: Closes #<issue> ## Scope - In scope: placements with equal pods and the same FFN count per pod (`4A1F,4A1F`, `1A1F,1A1F`, `6A2F,6A2F`), plus all FFN-prefix placements accepted today; afdep-comm pin moved to tag `v1.1.0`; a Qwen3-235B-A22B-FP8 2 x 4A1F recipe; multi-pod E2E cases. - Out of scope: unequal pod sizes and a varying FFN count per pod (needs afdEP-comm changes); checking the placement before weight load. ## Implementation Notes - `deepep_m2n/layout.py`: `RankPlacement`, `TransportLayout`, `plan_transport_layout` (pure). They order ranks domain by domain, FFN first, and pick FFN prefix (identity order) or per-domain mode; other placements are rejected with the pods' shapes. - `deepep_m2n/connector.py`: the handshake result carries the transport layout. Per-domain lanes use a separate gloo group (`afd_deepep_m2n_transport`) in transport order and pass `ffn_ranks_per_domain` to `afdep.deepep.ElasticBuffer`. The handshake, model checks, control plane and expert ownership keep the role order, and FFN-prefix layouts take the identical path (same group, same arguments). - `deepep_m2n/network.py`: network plan in transport order. - `pyproject.toml` / `uv.lock`: `afdep-comm @ ...@v1.1.0` (lock resolves to `741b9aa`). - No vLLM extension points, compat shims or monkey patches are added; pods use vLLM's existing `--data-parallel-start-rank` flags. ## Test Plan - CPU gate: `pytest -m "not gpu and not vllm_runtime" tests/unit`. - Full unit on the vLLM v0.32.0 image: `pytest tests/unit` (new `test_deepep_m2n_layout.py`, extended `test_deepep_m2n_connector.py`). - GPU-gated: `tests/e2e/models/qwen3_moe/test_qwen3_moe_multi_pod.py`, M2N `1A1F,1A1F` and `2A0F,0A2F`, eager/graph/DBO; single-host `2A2F`. - Manual: `recipe/gpu/DeepEPM2NAFDConnector/qwen3_235b_a22b_fp8/prefill_decode_colocation/multipod_4a1f_4a1f_graph.sh` and a `4A2F` baseline, full GSM8K and `vllm bench serve`. ## Test Result Revisions: vLLM v0.32.0, torch 2.13.0+cu130, afdEP-comm `fdff967` for the GPU runs (same source tree as `v1.1.0` / `741b9aa`), `v1.1.0` pin for unit. - CPU gate: 1452 tests, 1405 passed, 47 skipped, 0 failed. - Full unit: 1466 tests, 1418 passed, 48 skipped (NPU and cmake only), 0 failed; the 13 layout and 121 M2N connector tests all ran. - Multi-pod E2E, Qwen3-30B-A3B, 2 pods x 2 H200, GSM8K-100: `1A1F,1A1F` and `2A0F,0A2F` x eager/graph/DBO all pass (graph 0.92/0.96). - Single host `2A2F`, Qwen3-30B-A3B: 0.92/0.96. - Qwen3-235B-A22B-FP8, H200, full GSM8K (flexible/strict): `4A2F` one pod 0.9371/0.9325; `4A1F,4A1F` two pods 0.9401/0.9340. - Bench, random 1024/256, 64 requests per Attention rank: `4A2F` (6 GPUs) 3394 tok/s, TPOT 68 ms; `4A1F,4A1F` (10 GPUs) 4624 tok/s, TPOT 98 ms. ## Docs Impact - Files updated: `docs/gpu/DEEPEP_M2N_CONNECTOR_USER_GUIDE.md` (placement rules and table), `docs/design/module/connector_contracts.md`, `recipe/gpu/DeepEPM2NAFDConnector/README.md`, `recipe/gpu/DeepEPM2NAFDConnector/qwen3_235b_a22b_fp8/README.md`, `tests/e2e/README.md`, `.agents/skills/deploy-afd-k8s/` (`POD=AFD_<index>`). - If none, reason: n/a --- <details> <summary>Essential PR Checklist</summary> - [x] Purpose is clear and linked to public context when possible. - [x] Scope is bounded. - [x] Compatibility with vLLM v0.32.0 and, for NPU changes, the pinned vLLM-Ascend source is considered, as applicable. - [x] No changes are made to the vLLM source checkout. - [x] Plugin-owned classes or explicit dotted class paths are preferred over monkey patches. - [x] Any compat shim or monkey patch is isolated, idempotent, version-guarded, documented, and tested. (none added) - [x] Imports remain CPU-safe; CUDA-heavy work is delayed or GPU-gated. - [x] Validation evidence is included, including skipped GPU tests when applicable. - [x] Documentation impact is stated. </details> --------- Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

The base's "reuse native engine CLI and AFD configuration" refactor replaced FFNPlatformAdapter.add_arguments with create_parser (vLLM's EngineArgs parser, external_launcher by default), moved topology and connector options into --additional-config, renamed the receiver join timeout to --receiver-shutdown-timeout and reduced the NPU FFNConfig to launcher facts. Conflicts (npu adapter, backend, environment, settings) take the base behavior. The shared launch module now owns the native parser and the F-only checks both adapters run (positional model, FFN-role AFD config, WORLD_SIZE = --data-parallel-size = afd.num_ffn_ranks, DP rank from RANK, external launcher) plus rank/shutdown validation. The NPU environment keeps the shared base with the base's native device mapping. The GPU backend moves to the same contract: options come from --additional-config, --enable-dbo, and --enforce-eager or --compilation-config FULL_DECODE_ONLY; TP 1 with expert parallel is required; the tokenizer skip and EP weight filter are parser defaults; the F-only VllmConfig is built with EngineArgs.from_cli_args. Removed --host/--port/--attention-ranks/--ffn-ranks/--graph/--dbo/ --connector-extra-config/--afd-timeout. Unit tests, the GPU smoke test, the E2E runner command and the GPU guide follow. Also narrow the loaded FFN-only model to a torch module before counting its parameter storage (mypy). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>

…ect#560) ## Purpose Installation, common configuration, support limits and NPU validation are scattered across the root README, connector guides and design pages. Give readers one documentation entry point and consolidate repeated content. The README shrinks from 372 to 78 lines; this PR removes 272 lines overall. ## Issue Second documentation cleanup PR, following merged vllm-project#559. Based on main at `49aff3bb`; this PR contains documentation structure/navigation changes and reference fixes. ## Scope - In scope: Markdown organization, navigation, deduplication and references used by development skills. - Out of scope: runtime code, tests, CI changes and new hardware qualification. ## Implementation Notes - Add `docs/README.md` as the documentation index, with separate Support matrix and Documentation sections in the root README. Move installation, common AFD configuration and support/evidence into dedicated references; remove duplicate field tables and launch fragments from the landing page and guides. - Move Async FFN and routed CAM design into `docs/design/npu/`, and the historical Layered W13/W2 report into `docs/reports/`. - Merge `docs/npu/TESTING.md` into `tests/npu/README.md`, retaining runtime/DBO and layered precision requirements. Remove duplicate suite descriptions and obsolete eager-only wording. - Keep the module index focused on design ownership; remove its duplicate user-guide list and ASCII dependency diagram. Update incoming links and skill references, including upgrade/review instructions that locate the runtime matrix. - Preserve module status, ownership, validation paths, upstream refs and historical evidence. Runtime pins remain vLLM 0.32.0 (`5df63a72`) and vLLM-Ascend `bd42eea0`. ## Test Plan Check formatting, local navigation, documented configuration against current code, command syntax and preservation of relocated content. ## Test Result - All applicable pre-commit hooks and `git diff --check` passed. - 488 local Markdown/HTML links and anchors checked across 112 documents. - 10 AFD configuration examples parsed; all 10 common fields/defaults and four Async CAM extra keys match the source. - 16 installation/NPU validation shell blocks passed `bash -n`. - Ownership/validation metadata preserved across nine module pages. Three relocated documents and the support matrix retain their content/evidence apart from link rebasing and the validation-limits heading. - Checks ran on CPU-only macOS/Python 3.12.13. No unit suite, device E2E or native build was run for this documentation-only change. ## Docs Impact 44 Markdown files changed, +494/-766 lines. Four references/index pages added, three documents moved and one duplicate testing page merged away. --- <details> <summary>Essential PR Checklist</summary> - [x] Purpose is clear and linked to public context. - [x] Scope is bounded. - [x] Pinned vLLM / vLLM-Ascend compatibility is considered. - [x] No changes to the vLLM source checkout. - [x] Plugin-owned extension and patch requirements remain unchanged. - [x] Imports remain CPU-safe; no code changes. - [x] Validation evidence and hardware-validation limits are stated. - [x] Documentation impact is stated. </details> --------- Signed-off-by: jiangkuaixue123 <jiangxiaozhou111@163.com>

Closes vllm-project#563 ## Summary - Destroy the distinct per-domain `transport_group` during DeepEP M2N connector close. - Clear the transport-group reference so a second close remains a no-op. ## Validation - Added the regression test first and confirmed it failed without the teardown. - With the fix, the focused test passes. - The full DeepEP M2N connector and layout unit selection passes in a remote CUDA vLLM `0.32.0` environment. ## Documentation No impact; this is an internal lifecycle fix. Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>

…m-project#503) ## Background The synchronous CAMP2P path normally runs attention on the Attention role and the router, shared experts, and routed experts on the FFN role. This PR makes router/shared placement an explicit deployment option. With `compute_gate_on_attention=true`, Attention computes top-k and shared experts, while FFN consumes the external routing and runs the routed experts. ## Motivation **Improve eager execution through a different A/F work split.** The measured Attention-side placement delivers **+17.6% throughput** versus the same-commit FFN-side placement in the evaluated eager workload. **Establish the routing contract for quantized activation transport.** Once FFN receives the routing alongside the activation, it can consume an activation quantized on Attention. The MXFP8 pass-through builds on this contract in vllm-project#506. The option is **off by default** and belongs in the common `additional_config["afd"]` block. ## Implementation ```text Default Attention : attention FFN : router + shared experts + routed experts compute_gate_on_attention=true Attention : attention + router + shared experts FFN : transported top-k + routed experts ``` - **Reuse the A2E routing slots.** Send expert IDs as `int32` and top-k weights as `float32` through the existing ids channel. - **Owned prepare and comm subclasses.** `AFDPrepareAndFinalizeWithAllGather` and `AFDPrepareAndFinalizeWithMC2` provide `prepare_external`, which takes the routing for the local input rows and returns the complete `MoEPrepareOutput` together with the aligned routing. AFD comm subclasses override only `_get_prepare_finalize()`, so the native selector still decides which path runs and ordinary execution keeps the upstream classes. There are no global prepare patches left. - **Explicit routing contract.** The routing is an argument and a return value; nothing is stashed on the upstream class, and the module no longer reads or clears an `afd_external_routing` attribute. - **One MoE component.** `AFDExternalRoutingMoE` builds its comm candidates during model initialization, where the config context exists, and each forward only looks up the candidate matching the native selection. It runs prepare, expert execution and finalize, switches the forward-context comm for the duration of the call and restores it on every exit path, and rejects unsupported types such as fused-MC2. - **Consume the external top-k on FFN.** The component hands the aligned routing straight to expert execution. - **Place shared experts on Attention.** Run them between activation send and routed-output receive, then combine. - **Keep protocol selection explicit.** The model configuration selects token-ID or external-routing semantics; the model itself only builds the component for the synchronous FFN role and delegates. The supported opt-in configuration is synchronous CAMP2P, DeepSeek-V4, TP=1, without DBO or ubatching. Both roles must use matching routing settings. ## Performance and accuracy Measured at `828582a` on one A5 4A4F node during the same evening session, with BF16 A2E transport and automatic communication selection: | Configuration | Eager throughput (token/s) | Graph throughput (token/s) | | --- | ---: | ---: | | Attention-side gate (`828582a`) | **192,887** | **326,400** | | FFN-side gate (`828582a`) | 163,997 | 331,455.5 | | Clean baseline (`bcc0126`) | 164,543 | 333,407 | | Comparison | Eager | Graph | | --- | ---: | ---: | | Attention-side vs FFN-side placement | **+17.6%** | **-1.5%** | | FFN-side placement vs clean baseline | -0.3% | -0.6% | The result supports Attention-side placement as an eager optimization and a foundation for MXFP8 pass-through. Placement remains opt-in, allowing deployments to select the appropriate A/F split for their execution mode. Graph-mode Attention-side routing scored **90.60% on the full 1,319-question GSM8K evaluation**, with normal generated output. Measurement setup: concurrency **128**, reported input **8,275** / output **32** tokens per request, `max-num-batched-tokens=3072`, `max-num-seqs=128`, and zero failed requests. Graph uses `FULL_DECODE_ONLY` with capture sizes `[8,16,32,64,128]`; eager uses `--enforce-eager`. Each cell has two runs from the same firmware/session. These are workload-specific measurements with small run counts. ### Re-measured on the target stack at `e2c7075` The refactored implementation was re-measured on the stack this PR declares: vLLM `0.32.0` @ `5df63a7`, vLLM-Ascend `bd42eea0`, BF16 A2E transport, Attention-side gate, automatic communication selection, TP=1, eager (no CUDA/ACL graph), 4A4F on one A5 node. | Run | E2E (ms) | Prefill (token/s) | Output (token/s) | Total (token/s) | Failed | | --- | ---: | ---: | ---: | ---: | ---: | | 13:28 | 5,126.2 | 1,614.2 | 728.59 | 189,136 | 0 / 128 | | 13:34 | 5,146.5 | 1,607.9 | 726.69 | 188,643 | 0 / 128 | The two runs agree within 0.4%, so they are recorded as a repeat baseline rather than a single sample. GSM8K on the same configuration scored **89.92%** on the full 1,319-question evaluation. These numbers describe this configuration only: there is no same-stack reference implementation to compare against, so they are not evidence of a performance change from the refactor. The pre-refactor table above was measured on the previous stack and stays attributed to `828582a`. ### Placement re-measured on the target stack (`e2c7075`) Same node, same stack (vLLM `0.32.0` @ `5df63a7`, vLLM-Ascend `bd42eea0`), same workload (128 concurrency, 8,275 input / 32 output tokens per request), zero failed requests. The clean baseline is upstream main at `b093863`, which keeps the gate on the FFN side. | Mode | Clean baseline `b093863` (total token/s) | Attention-side gate `e2c7075` (total token/s) | Delta | | --- | ---: | ---: | ---: | | Eager | 165,345 | 189,136 / 188,643 | **+14.2%** (mean) | | Graph (`FULL_DECODE_ONLY`) | 364,627 / 362,563 | 354,731 / 353,972 | **-2.5%** (mean) | On this workload, Attention-side placement averages about **+14.2% higher total throughput in eager** and about **2.5% lower in graph**. That is a placement comparison across two commits on one stack; it is not a pure refactor cost, and it does not mean Attention-side placement is unsuitable for graph mode. Graph mode also raises this configuration's own throughput by **87.6%** over eager (189,136 -> 354,731 total token/s). GSM8K on the target stack scored **89.92%** in eager and **89.99%** in graph mode, both on the full 1,319-question evaluation. ## Baseline and scope Rebased onto main at `e58fdec`, which includes the merged vllm-project#500 opaque routing-op fix. The retired `profile_run_eager` implementation and its wiring have been removed. vllm-project#506 builds MXFP8 pass-through on this routing contract. [Incremental diff for this PR: `e58fdec...2c17cce`](Pingzii/afd-plugin@e58fdec...2c17cce) ## Verification CPU, measured at `2c17cce`: **59 passed, 2 skipped**. The focused set covers the routing-id op, the runtime installer wiring, the CAMP2P token-ID path, the routing payload transforms, and the new MoE component. The maintained CPU suite also passes. Two mutation checks were run on the refactored component before the rebase, and both produced the expected failures: - disabling the per-token state cleanup makes the MC2 finalize case fail (it observes state that should already be cleared); - disabling the comm restore makes the prepare-failure case fail (it observes the AFD comm left in place). Device evidence: the head `7430a13` starts on one A5 4A4F node with BF16 transport, `compute_gate_on_attention=true`, the CAMP2P connector, TP=1 and eager mode, and completes a generation request (`POST /v1/completions` returned 200). Accuracy and throughput were measured at `e2c7075`. The pre-refactor table above stays attributed to `828582a`. --------- Signed-off-by: 宋元盟 <songyuanmeng2@huawei.com> Co-authored-by: 宋元盟 <songyuanmeng2@huawei.com>

Add a regression test for the device-selection review finding: with --device-ids 2,3 and LOCAL_RANK=1 the standalone GPU FFN rank binds cuda:1 instead of GPU 3. vLLM only stores the resolved IDs in parallel_config.assigned_physical_gpu_ids; its GPU Worker registers them with set_assigned_physical_gpu_ids before mapping the local rank to a device, and the F path skips that step and selects the device before the config exists. The test drives DeepEPM2NFFNBackend.prepare from the native CLI and stops when the rank binds its GPU. It fails on this branch and passes once the config is built first and the mapping is registered before selection. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>

The standalone GPU FFN rank ignored --device-ids and bound cuda:<LOCAL_RANK>, so FFN ranks could land on the Attention GPUs. vLLM only stores the resolved IDs in parallel_config.assigned_physical_gpu_ids; its GPU Worker registers them with set_assigned_physical_gpu_ids before mapping the local rank to a device. The F path selected the device before the config even existed. Build the F-only config first, then register the assignment and map the local rank through it, as the GPU Worker and the NPU adapter do. Building the config reads no weights, so a bad deployment still fails before the experts are loaded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>

Remove test_device_ids_place_the_rank_on_its_assigned_gpu and the helpers only it used from tests/unit/ffn_faas/test_gpu_adapter.py. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>

…roject#490) DeepSeek-V4 Flash W4A8 routed experts can run in independent F-only processes while retaining native expert-management context and the mature CAM execution path. F does not construct a complete CausalLM, vLLM Worker, upstream ModelRunner, EngineCore, request scheduler, sampling state or KV cache. This is a scoped independent-service delivery under vllm-project#391. The Attention model, runner, connector, routing/shared experts, padding, DBO and shutdown paths keep their existing contracts. ## Implementation - The standalone entry point reuses vLLM's `FlexibleArgumentParser`, `EngineArgs.add_cli_args`, `EngineArgs.from_cli_args` and the existing AFD configuration validators. Graph mode uses `--compilation-config` / `--enforce-eager`; capacities use upstream options, and AFD topology / CAM options use `--additional-config`. Device selection uses native mapping. Only backend selection, management and receiver-shutdown controls add standalone flags. F remains independent of the serving lifecycle. - A CPU-only `FFNBackend`/platform-adapter interface separates the public CLI, Service and host loop from device/model implementation. Ascend/V4 loading, quantization, configuration and CAM composition live in `ffn_faas/platforms/npu/`. - Startup explicitly selects the qualified V4 adapter. `CAMRoutedModel` owns native loading, local placement checks, progress, storage accounting and cached-view lifetime. `V4ModelAdapter` supplies checkpoint names and native owner construction; both V4 description and adapter live in `v4_model.py`, with the native owner lazily loaded from `v4_weights.py`. There is no separate V4 execution runtime. The CAM backend has no V4 type dependency. CAM static placement guards live in platform settings, outside V4 metadata. This is a FaaS-only boundary cleanup, not second-model or quantization support. - `AscendFFNEnvironment` owns the selected device and F-only native compute/EP groups throughout execution. The original routed modules, expert-map managers and quantizers remain resident alongside the separate A/F CAM group. Native post-warmup CPU affinity policy is reused. - The routed-only container reuses the native factory, checkpoint loader and quantizer. Its execution-keyword contract is explicit; the unbound loader adaptation documents its pinned receiver contract and PATCH/delegation exception. Static guards reject EPLB, elastic EP and redundant experts. - Execution reuses `CAMAsyncAFDConnector`, original DS/DR/CS/CR, layered W4A8, paired startup and FULL graph pools. The FFN two-stream path and its core-budget options were removed to follow main (vllm-project#550). There is no new FaaS connector, transport wire ABI or service control operator. Here `FFNBackend` is a lifecycle adapter for a complete runtime, composing the environment, routed owner, CAM connector and mature Async runtime. The public host loop calls `backend.execute_step()`; communication, compute and Graph execution remain in that runtime. This is not RFC vllm-project#391's pure `compute(item)`/facts backend or a shared cross-platform, cross-connector WorkItem/StepPlan/Graph Step Loop. Preserving the mature CAM path is deliberate; a real GPU or synchronous connector integration should drive any future common loop. The environment, backend and host runner provide device/compute ownership, composition and host lifecycle without another Worker wrapper. See [the standalone service guide](https://github.com/vllm-project/afd-plugin/blob/053ffeae23e2a22f9d7cdaaa018a9eb430638067/docs/npu/STANDALONE_FFN_SERVICE.md) and the registered FFN runtime design for supported configuration and ownership. ## Validation Current main synchronization at `053ffea` incorporates main `e58fdec`, including its documentation reorganization. Conflict resolutions update the standalone guide, module ownership and moved links; FaaS implementation files are unchanged. Documentation hooks and local links pass. Targeted standalone and connector/session CPU results at `47c75d8` are reused: **120 passed, 6 skipped**. The skipped cases are the native-parser cases in the lean CPU environment; their earlier **6/6 pass** at `b26bdc3` is reused because the CLI implementation and fixtures are unchanged by this merge and use the same pinned vLLM source. Relevant Ruff, mypy 3.11, Markdown and other pre-commit hooks pass. No NPU, model, GSM8K or performance run was added for this main synchronization. The device and performance results below describe their recorded historical commits and execution conditions; they do not validate the updated runtime. ### Historical validation The 79 CPU cases and relevant checks below passed at `cfb72da`. The device regression below was run at `d3c6198`, using the recorded dependency stack and native binaries. The V4 checkpoint description and adapter remain together in `v4_model.py`. - 79 CPU cases pass, covering model/quantization rejection, local source selection, adapter-supplied routed layers, CPU-only first imports and host lifetime. The import gate precedes framework/fake imports and blocks both ordinary and dynamic torch/torch-npu/vLLM/Ascend imports. Twelve real subprocess isolation cases remain covered. Relevant mypy and pre-commit checks pass; metadata stubs do not qualify a second model. - One finite real 12A/4F regression on the original per-channel W4A8 checkpoint passed: A eager MRV2 TP4/DP3 with two native DBO microbatches on all 12 workers, and four F ranks with EP4, 43-layer FULL/two-stream execution (20 AIC/40 AIV compute, 8 AIV combine). - Three short warmups preceded six frozen 14,399–14,935-token requests at C6 and 17/31/257-token tails, each generating one token with prefix caching explicitly disabled. All nine business token IDs, generated-token log probabilities and top-5 log probabilities matched the existing mature CAM reference at `d136157` exactly. That reference was reused; mature AFD was not redeployed. Functional observations confirmed imports of the new `CAMRoutedModel` resource. - F-only EP4 and separate CAM16 groups remained live with the same native routed owner, expert-map managers and quantizers through the pending/failure boundary, with 64 local experts per layer. All four F ranks failed closed with owners retained. Whole-pair isolation left no owned or NPU process. Attention logged peer errors during F-first teardown; A's final zero exit and F's nonzero exit are not a graceful-drain proof. - The native extension and CANN package were reused without rebuilding. Applicable earlier operator-numerical and pinned-stack checks are retained, not counted as new candidate runs. No GSM8K or performance matrix was rerun. Pinned tested dependencies are vLLM `ced6857afa0ea7b2e3f0846a62e1394e90f15607`, vLLM-Ascend `8d4409d6256d8a6729140ddcc0d1889e3f96cdd6`, CANN 9.1.0 and torch-npu 2.10.0.post4. The finite regression ran on another A3 node after resource migration; its correctness results are separate from the earlier same-node performance comparison. Next-token equality is not a full-model accuracy claim. ## Historical clean prefill comparison These figures predate main's removal of FFN two-stream execution (vllm-project#550). They remain evidence for the versions and conditions below, with no new performance claim for the current merged revision. The candidate at `9151609` was compared with the mature `vllm serve` FFN path on the same source/common clamp fix, native binaries, original W4A8 model and single A3 node. Both used 12A/4F, A TP4/DP3 with native DBO, F FULL/two-stream (20/40 compute and 8 combine), explicit prefix-cache disablement and natural expert routing. Product CLIs loaded no functional observer or profiler. Each measured round replayed the same 36 requests from six six-turn sessions at C6, with one output token. Input lengths were 14,399–58,462 tokens (median 27,951.5), totaling 1,071,401 prompt tokens per round. Frozen priorities, dependencies, domain routing and request payloads were preserved; generated answers did not enter later prompts. One full workload warmup per deployment was excluded. The ordered blocks were A1/A2, B1/B2, B3/B4, A3/A4; both B blocks shared one standalone deployment. | Service / change | Batch (s) | Mean TTFT (s) | Mean of round p95 (s) | | --- | ---: | ---: | ---: | | Mature AFD | 51.301 | 7.839 | 11.814 | | Standalone CAM F | 50.270 | 7.875 | 11.804 | | Standalone vs mature | -2.01% | +0.46% | -0.09% | The mature batch bracket drift was -2.19%; the observed batch advantage is not attributed to a particular change. All three mean metrics and observed round upper comparison bounds stayed below the 2% regression reference. The p95 column averages the per-round p95 values. The comparison measures the whole HTTP prefill path, including Attention and KV work; it is not isolated FFN timing. Four rounds per arm use one standalone deployment bracketed by two mature deployments, so repeats are correlated. The 2% reference is reported against observed variation, not claimed as a universal statistical noninferiority guarantee. Earlier custom-transport and prefix-cache-enabled scores are excluded. A first deployment failed after a long idle wait before any measured round; a later unused-status-port check failed before model launch. Those failures are retained, and each reported timed round is a complete successful replay. ## Lifecycle and remaining scope A pending CAM DR cannot be cancelled by host stop. Failed startup/receive or unfinished batches retain weights, graphs and communication owners until the entire paired deployment is isolated. Diagnostic or HTTP-cleanup failures cannot bypass the CLI's unsafe-session nonzero exit. This does not implement lossless bounded drain, EOS or independent hot restart. Resident expert context does not enable EPLB/EEP in the current static CAM layout. CUDA/other connector backends, synchronous ControlPlane, profiling management, cancellable receive and independent A/F dependency pins remain follow-up scope. Actual CAM pairing still follows native weight loading; a lightweight pre-loading handshake remains unimplemented. The earlier fused implementation and transport work remain preserved separately in vllm-project#491 and the [pre-CAM snapshot](vllm-project@16e25f1). They are not modified here or assumed compatible with this narrowed service. --------- Signed-off-by: jiaran-king <jiaranran2@gmail.com>

Remove the "NPU parallel:" / "GPU parallel:" and "Reuse:" comment blocks that compared each standalone FFN class and method with its counterpart on the other platform, together with the dangling REFACTOR-CANDIDATE(R8) tag. The cross-references named file paths that no longer match main. Where a block carried a platform-local explanation that the docstrings did not already state, move it into the docstring: the adapter's fixed GPU constraints and handshake-adopted options, the backend lifecycle and release order, the per-step synchronize that makes execute_step fence-free, and the base-class versus platform split of both FFN environments. Comments and docstrings only; no behavior change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>

…llm-project#567) ## What this fixes `camp2p` resolves its public names through a package-level `__getattr__` (vllm-project#503). The connector factory still read the class out of the module dictionary: ```python connector_cls = vars(module)[class_name] ``` For a lazily exported class that dictionary is empty, so every CAMP2P start fails before the worker can build the connector: ``` File "afd_plugin/connectors/factory.py", line 40, in loader connector_cls = vars(module)[class_name] KeyError: 'CAMP2pAFDConnector' ``` ## Change Read the class with `getattr`, which follows the `__getattr__` hook and behaves exactly as before for eagerly exported modules. The regression case moved into `tests/unit/connectors/test_p2p_connector.py` with the other factory tests: it registers a temporary connector backed by a module that only answers through `__getattr__`, and registers into a copied `_registry` so the temporary entry does not outlive the test. The missing-name case was dropped; the standalone test file is gone. ## Validation - The moved case fails with the old `vars()` lookup. - `tests/unit/connectors/test_p2p_connector.py` and the maintained CPU suite pass. - DCO, label and pre-commit pass. Signed-off-by: 宋元盟 <songyuanmeng2@huawei.com> Co-authored-by: 宋元盟 <songyuanmeng2@huawei.com>

…oss the session (vllm-project#570) Fixes vllm-project#568. Replaces vllm-project#569. The unit suite runs in one pytest session, so anything a test installs into `sys.modules` stays visible to every later test. Every stand-in below is now limited to the import that needs it and removed on both the success and the failure path. Test-only; no production file changes. ## What changes - **Sync gate proxy (vllm-project#568)** — `pytest.importorskip("vllm_ascend")` next to the existing `vllm.forward_context` guard, so a host with vLLM but without vLLM-Ascend skips instead of failing during collection. The `torch_npu` stand-in stays for the adapter import only. `import afd_plugin` runs before the stand-in is installed, so model registration reflects the actual environment. - **Two `vllm` stubs** (`test_camp2p_token_ids.py`, `test_deepseek_v4_hash_ids.py`) — only the modules that are genuinely missing are installed; a pre-existing `vllm` module is never replaced, so it cannot outlive the context. - **CAMP2P cleanup** — modules the stub block loaded that still bind one of its objects are dropped, together with the parent package attribute, so the lazy package cannot serve a class built against the doubles. Modules that merely contain `None` are left alone. - **Component fixture** (`test_deepseek_v4_external_routing_moe_cpu.py`) — restores only the names it replaced, inside a `finally`; unrelated modules imported meanwhile survive. - **DeepEP** — the `afdep` stand-in lives only around the imports that need it, inside `try/finally`. It has a module spec, so `find_spec("afdep")` reports it correctly while installed and the package as missing afterwards. - **Async FFN runners** — adds explicit `vllm_ascend` import guards to the async FFN graph and runtime tests. ## Verification - A pre-installed `vllm` module survives both stubs; the connector is not left cached, and `camp2p` no longer serves a stub-bound class. - A module that only holds `None` is not evicted; a stand-in removed on a failing import leaves `afdep` and `torch_npu` absent. - `afd_plugin` imports with the stand-in installed and keeps a non-NPU DSV4 registration. - `pytest tests/unit --collect-only` collects with no errors; the affected files run 37 passed, 4 skipped locally. - Full `tests/unit` run on a CPU-only host: only the two `package/` cases fail, and they fail identically on `upstream/main` (checkout-specific, unrelated). --------- Signed-off-by: 宋元盟 <songyuanmeng2@huawei.com> Co-authored-by: 宋元盟 <songyuanmeng2@huawei.com>

## Purpose Unify the GPU `prefill_decode_colocation` recipes on one shared launcher. The existing recipes duplicate most of their launch logic, encode multi-pod placement differently, and vary ports, GPU order, and timeouts without a functional reason. This change reduces each recipe to its model, connector, default layout, DP/TP shape, mode, and `vllm serve` flags, while the shared launcher owns placement, AFD configuration, role startup, and overrides. The new launcher also makes every recipe runnable on one host or across pods from a single `LAYOUT`, directly or through `deploy-afd-k8s`. `AFD_DRY_RUN=1` prints the exact per-pod commands without starting vLLM. ## Issue - Related issue(s): vllm-project#566 - Closing keyword, only if fully resolved: Closes vllm-project#566 ## Scope - In scope: - GPU `prefill_decode_colocation` recipes for `P2pNcclAFDConnector` and `DeepEPM2NAFDConnector`. - The shared launcher and its user documentation. - `deploy-afd-k8s` and `run-vllm-bench-k8s` instructions needed to deploy and benchmark the new recipe shape. - Recipe, connector-guide, benchmark, and review-skill documentation updates. - Out of scope: - NPU recipes and runtime behavior. - `prefill_decode_disaggregation` recipes. - AFD plugin runtime code, connector implementations, and E2E gate definitions. ## Implementation Notes - Recipe scripts change from 19 files / 1,192 lines to 20 files / 374 lines. - Added `recipe/gpu/common/afd_colocation.sh` and `recipe/gpu/common/README.md`. Each personalized recipe now sets its model, connector, default `LAYOUT`, DP/TP defaults, mode, and serve flags before sourcing the shared launcher. - Restored explicit personalized recipes: - DeepSeek-V2-Lite and Qwen3-30B-A3B DeepEP M2N: `2a2f_{eager,graph,graph_dbo}_dp2tp1.sh` plus `2a2f_eager_dp1tp2.sh`. - Qwen3-235B-A22B-FP8 DeepEP M2N: `4a2f_{eager,graph,dbo}_dp4tp1.sh` plus `multipod_4a1f_4a1f_dbo.sh`. - DeepSeek-V2-Lite P2P NCCL: the existing seven personalized 2A2F, 4A4F, and two-pod graph/eager DBO recipes. - Qwen3.5-122B-A10B-FP8 P2P NCCL: retains `multipod_2a_2a_2f_graph.sh`. - The launcher derives device assignment, DP/TP flags, headless state, DP start ranks, AFD rank counts, and both `vllm serve` commands from `LAYOUT` and `POD=AFD_<k>`. - Intentional command changes: - Attention API `18000`, FFN API `18001`, and AFD rendezvous `6239`. - FFN uses each pod's first GPUs and Attention uses the following GPUs. - Both roles use `--cpu-distributed-timeout-seconds 3600` and `afd_process_group_timeout_s: 3600`. - Both roles use `--trust-remote-code` and invoke `vllm serve` directly. - DeepEP M2N recipes use `--enable-ep-weight-filter` on both roles. - Qwen3-235B graph capture sizes add 12/24/48; DBO capture sizes add 384/512. - P2P DeepSeek-V2-Lite and Qwen3.5 tuning flags remain unchanged. - `deploy-afd-k8s` ships both the recipe and shared launcher in the recipe ConfigMap, sets `AFD_RECIPE_LIB` and `SERVE_HOST=0.0.0.0`, derives one pod per `LAYOUT` entry, and resolves Service ports from the launcher. - `tools/benchmarks/request_generator.sh` defaults to the colocation Attention port, `18000`; `run-vllm-bench-k8s` uses the same endpoint. - The common README documents that personalized recipes encode their own DP/TP defaults; `ATTN_TP` and `FFN_TP` should be changed only intentionally. ## Test Plan - CPU-only, no cluster: - Run `bash -n` and `shellcheck --shell=bash` on all 20 recipe wrappers and the shared launcher. - Run each recipe with `AFD_DRY_RUN=1`. - Assert mode flags, layouts, DP/TP flags, device assignments, rank counts, ports, and per-pod headless/start-rank behavior. - Compare old and new generated commands with a fake `vllm` executable. - Verify the single-pod `LAYOUT=2A2F` and two-pod `LAYOUT=4A0F,0A4F` configurations produce distinct Attention and FFN commands. - Static checks: - Run targeted pre-commit checks on changed recipe, documentation, and skill files. - Run `git diff --check`. - GPU smoke: - Deploy the renamed 2A2F graph + DBO DeepSeek P2P recipe through the two-file ConfigMap path used by `deploy-afd-k8s`. - Verify both roles start, the Attention health endpoint answers, and completions are correct. ## Test Result - All 20 wrappers passed `bash -n`, `shellcheck --shell=bash`, and `AFD_DRY_RUN=1`. - Dry-run assertions covered mode flags, layouts, DP/TP flags, device assignments, ports, rank counts, and per-pod headless/start-rank behavior. - A fake-`vllm` old/new comparison found no unintended command drift beyond the documented port, timeout, device-order, mode, and multi-pod changes. - `LAYOUT=2A2F` produced FFN on GPUs 0-1 and Attention on GPUs 2-3. `LAYOUT=4A0F,0A4F` produced Attention on pod `AFD_0` and FFN on pod `AFD_1`, each with DP4 and the expected rank counts. - Targeted pre-commit checks and `git diff --check` passed for the changed recipe and documentation files. - GPU smoke, 4x H100, vLLM `0.32.0`: the 2A2F graph + DBO DeepSeek P2P recipe ran through the `deploy-afd-k8s` ConfigMap path. Both roles became ready, `/health` answered, completion was correct, and 256/256 requests succeeded with ISL 256, OSL 128, and concurrency 64. ## Docs Impact - Files updated: - `recipe/gpu/common/README.md` and the GPU recipe READMEs. - `recipe/README.md`. - `docs/gpu/DEEPEP_M2N_CONNECTOR_USER_GUIDE.md`. - `.agents/skills/deploy-afd-k8s/` and `.agents/skills/run-vllm-bench-k8s/SKILL.md`. - `.agents/skills/review-pr/references/features/disaggregation-topologies.md`. - `tools/benchmarks/request_generator.sh` default-port and usage documentation. - No `docs/design/module/` update is needed because plugin runtime contracts, E2E gate structure, and required E2E cases are unchanged. --- <details> <summary>Essential PR Checklist</summary> - [x] Purpose is clear and linked to public context when possible. - [x] Scope is bounded. - [x] Compatibility with vLLM v0.32.0 and, for NPU changes, the pinned vLLM-Ascend source is considered, as applicable. - [x] No changes are made to the vLLM source checkout. - [x] Plugin-owned classes or explicit dotted class paths are preferred over monkey patches. - [x] Any compat shim or monkey patch is isolated, idempotent, version-guarded, documented, and tested. - [x] Imports remain CPU-safe; CUDA-heavy work is delayed or GPU-gated. - [x] Validation evidence is included, including skipped GPU tests when applicable. - [x] Documentation impact is stated. </details> --------- Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

…ject#574) Three unit tests fail on Windows for reasons unrelated to what they check: - `tests/unit/compat/npu/test_ascend_config_lifecycle.py` reads a source file with `Path.read_text()` and no encoding, so the GBK locale decodes UTF-8 bytes and raises `UnicodeDecodeError`; both reads now state UTF-8. - `tests/unit/package/test_cam_async_build.py` builds its manifest file list with native separators while setuptools works on posix manifest paths, so every required entry misses; the list now uses `as_posix()`. - `tests/unit/package/test_package.py` needs installation metadata to find the `vllm.general_plugins` entry point, which a source tree does not have; it now skips with that reason, and still checks the entry point when the distribution is installed. No production code changes. With these, full `tests/unit` on a Windows CPU host with `torch` and no `vLLM` is 900 passed / 89 skipped / 0 failed (previously 8 failed). Where these come from, for review context: - the six GBK failures: `tests/unit/compat/npu/test_ascend_config_lifecycle.py` arrived in vllm-project#425 (`b7561d7`) with `ast.parse(path.read_text())` and no encoding, so it is latent everywhere except a locale that cannot decode UTF-8; - the manifest failure: vllm-project#539 (`f2c79d8`) replaced the posix-normalised `required` set with `set(files.allfiles)`, whose entries carry native separators, so the comparison only holds where `/` is the separator. The vllm-project#361 version of the test compared normalised paths and was correct; - the entry-point failure: the test first appeared as `tests/test_package.py` in #1 (`dffdb14`, "Add Phase 1 plugin skeleton") and was later moved to `tests/unit/package/test_package.py` by `d1fd84f`; it always needed installation metadata. None of the three involves the transports in vllm-project#506; the tests are older than that branch. On verification scope: this PR's checks are label, pre-commit and DCO -- none of them installs the plugin or runs pytest. The numbers above come from a local run on this branch; the entry-point test exercising an installed build is therefore *not* covered by CI, before or after this change. Signed-off-by: 宋元盟 <songyuanmeng2@huawei.com>

## Purpose Move E2E scenario definitions from the shared runner into model-owned `cases.py` files. For migrated paths, a new combination of existing deployment, task and check capabilities requires no shared-module changes. This implements the existing-case migration from [RFC vllm-project#549](vllm-project#549), preserving current acceptance rather than changing the coverage matrix. ## Issue - Related RFC: vllm-project#549 - This PR does not close the full RFC. ## Scope - Migrate all 48 existing single-deployment definitions: 38 CUDA and 10 NPU. - Migrate ten GPU multi-pod entries: four DeepSeek-V2-Lite and six Qwen3 MoE/DeepEP entries. This includes the three interleaved Qwen combinations already added to main by vllm-project#555. - Preserve the 15 unique CI-selected node IDs, workloads, thresholds, concurrency and required evidence. - Retain the two NPU multi-pod entries and dedicated FFN/DSpark executors on their legacy paths. - No scenario additions/deletions relative to current main, cross-pod execution-evidence enhancement, runtime/compat changes, provisioning framework or CI pipeline changes. ## Implementation Notes - Immutable Case/Deployment/Role/Phase declarations keep model-specific choices in the model directory. Collection reads IDs; checkpoint-dependent settings resolve in the runner child. - Shared immutable eager/decode-graph presets construct common synchronous deployments for V2 on GPU/NPU, Qwen3 MoE and Qwen3.6. Models retain scenario tables, layouts, connector/capture/DBO choices and all workload/acceptance settings. Specialized DSV4 profiles remain model-local; no global scenario registry is introduced. - Deployment owns commands, connector startup order, readiness/liveness and cleanup. GPU pod coordination reuses this local executor while retaining rendezvous, placement, barriers, evaluator selection and verdict/abort propagation. - Tasks own workloads. The runner controls one observation window per phase; graph/DBO/Hash checks consume a shared snapshot. DSV4 GPU declares its two completion rounds and GSM8K as three phases. Existing native baseline evidence requirements are retained. - Structured records distinguish canonical/effective configuration, overrides, execution, coverage, cleanup and acceptance. Weakened GSM8K workloads remain debug runs and cannot qualify a fixed pytest entry; the verdict propagates to every pod. - DBO has one resolution rule for commands, per-role probe requirements and logging: explicit Role configuration overrides the Deployment default. Attention-only native DBO remains supported. HTTP client imports are delayed until task execution. - Align with `main@e58fdec`: NPU async CAM retains worker/service/runner shutdown budgets of 60/90/120 seconds. The worker environment override is preserved and recorded as effective configuration. Other profiles retain their existing budgets. - The zero-shared-diff extension rule applies to all single-deployment Cases and ten GPU multi-pod entries. Retained NPU multi-pod/dedicated paths remain outside that contract; each invocation uses one cleanup path. ## Test Plan Use CPU-safe collection, declaration/command comparisons, host integration with actual processes/sockets/signals, affected-file hooks, and the standard ready CI selection. Reuse earlier targeted device evidence for unchanged GPU paths; do not rerun the entire model/topology matrix. ## Test Result | Check | Result | | --- | --- | | Collection with HTTP client imports blocked | All 48 GPU node IDs and 10 NPU single-deployment IDs collect on a host without torch/vLLM | | Configuration/command comparison | All 48 single-deployment and the original seven GPU multi-pod declarations/argv remain identical to `8388dc3`; the three upstream-added Qwen interleaved combinations are retained | | GPU pod parity with latest main | All ten configurations / 30 role argv exactly match the legacy commands at `main@e58fdec` | | DBO follow-up, CPU job 7039 | Old source reproduces both specified regressions; fixed source passes 25 host checks, zero skips | | Latest main-alignment host run | Job 7048: 25 existing checks pass. Job 7051: the two shutdown cases reproduce old CLI 10 vs expected 90, then both pass after the fix; zero skips. All 27 host behaviors are covered across these runs, with verified owned cleanup and allocation release | | Shared-preset equivalence | Against `14add02`, all 48 single-deployment and seven GPU multi-pod Case declarations and argv match exactly, including workload, checks and cleanup settings | | Shared-preset host integration | CPU job 7175: all 27 existing host checks passed, zero failures/errors/skips; real processes, sockets, signals and TCPStore peers with stub service/evaluator CLIs. Owned process/port/IPC/shared-memory/scratch cleanup verified; allocation completed and released. Frozen test source matches commit `8388dc3` exactly | | Latest-main conflict integration | CPU job 7589: all eight affected multi-pod host checks passed, zero failures/errors/skips, including Qwen role-split and interleaved placement. Owned resource cleanup and allocation release verified; frozen source matches `a055cdc`. Other host checks are unchanged and retain job 7175's evidence. Affected-file pre-commit/mypy and diff checks passed | | Affected-file checks | Original 31-file checks passed; this six-file preset/docs update also passes pre-commit, including mypy/format/docs checks; diff hygiene passes | Host services/evaluators are stubs, and the probe-facing stub explicitly refuses device evidence. The two new host-only Cases use strict group cleanup for non-daemonizing stubs. Their initial run encountered a CPU Python without `os.pidfd_open`; only host-test isolation changed for the targeted recheck. Real NPU Case cleanup remains unchanged, and native CAM orphan/pidfd teardown is not qualified by these host checks. These checks verify harness configuration, process/RPC boundaries, error propagation and cleanup; they do not qualify accelerator execution. Earlier frozen-source hardware evidence includes 13 unique V2 P2P definitions, one DSV4 graph+DBO case (two ten-request rounds, 23/24 GSM8K answers, three four-rank windows and Hash pairing/freshness), a temporary two-GPU composition trial with zero shared-source edits, and one four-H20 interleaved multi-pod smoke on a single host. That smoke runs one seven-sample workload (2/7 correct at the original 0.27 threshold), with its result propagated to the peer. Every completed run has owned-resource cleanup and allocation-release evidence. These results are not a final-head full-matrix run, cross-node/RDMA qualification or additional NPU/DeepEP hardware qualification. Exact argv parity above refers to GPU multi-pod. Single-deployment comparison preserves CLI values/workloads; DSV4 budget flags move with Deployment ownership. The later DBO follow-up compares against `c176029`, and the shutdown follow-up compares with current main's protocol. Target pins: vLLM 0.32.0 (`5df63a7203cdb9328d824a27b682e77c7b9928be`) and vLLM-Ascend `bd42eea01d8312c4df469fabaed45f77a12df4d3`. Host/device evidence records its own frozen source; Linux runtime is Python 3.12.13 / torch 2.13.0+cu130. No model weights or runtime were downloaded/installed for this work. NPU device validation and unexecuted GPU combinations remain explicitly unqualified. ## Docs Impact Update the E2E README/module contract, multi-pod guide, GPU connector case link and run-e2e skill/Kubernetes example. RFC vllm-project#549 and its bilingual design snapshot clarify shared presets versus model-owned scenarios; the coverage matrix is unchanged. Task-local reports, scratch scripts and raw validation output remain outside the repository. --- <details> <summary>Essential PR Checklist</summary> - [x] Purpose is clear and linked to the RFC. - [x] Scope is bounded. - [x] Pinned vLLM/vLLM-Ascend compatibility is considered; NPU hardware evidence is explicitly limited. - [x] No changes to upstream runtime source or plugin runtime/compat code. - [x] Plugin-owned execution paths are retained; no new runtime monkey patches or shims. - [x] Collection stays CPU-safe; device work remains in runner processes. - [x] Host/device results, skips and unexecuted combinations are distinguished. - [x] Documentation impact is stated. </details> --------- Signed-off-by: Zhou ziheng <jiaranran2@gmail.com>

…oject#545) ## Purpose Enable native decode graphs for Async CAM DeepSeek-V4 target Attention and DSpark draft Attention. Reuse the vLLM/vLLM-Ascend graph managers and install AFD metadata at capture/replay boundaries for remote expert calls. ## Issue Related to vllm-project#499; complements the F-side FULL support merged in vllm-project#532. ## Scope - Target native graphs with an eager or native-graph DSpark draft; matching target/draft TP and independent F execution settings. - Preserve Attention-local FlashComm1/SP and native PD. P uses the upstream complete draft model and existing KV transfer; no new PD protocol is added. - Accept native `FULL`/`FULL_DECODE_ONLY` resolution. Remove graph-specific restrictions on compilation mode and model-free speculation; existing model-based draft/MTP checks remain. Device qualification uses mode 0. - Keep DBO excluded and draft graph dependent on target graph, as native target capture owns draft capture. Unequal target/draft TP, native kernels, quantization changes and lifecycle work remain outside this PR. ## Implementation Notes - Stage target metadata after native capture input preparation and before replay, preserving real and padded token counts. Send synchronous DP control only when a connector has a control plane; Async CAM has none. - Scope draft metadata to its own descriptor and query width. Reuse the existing physical KV block-size adapter and restore temporary methods/context on normal and exceptional exits. - Native managers retain ownership of graph buffers, buckets, attention/KV updates, sampling and fallback. No new ModelRunner, F executor or communication protocol is introduced. Preserve main's vllm-project#500 routing fix. - Integrate on main `e58fdec5876782b9c6a03683416611deea42af92`, retaining its MegaMoE path and FFN two-stream removal. No F-runner, connector or native-operator changes are included in the PR diff. ## Test Plan / Test Result The conflict resolution at `fb4095d` merges main `26d8151` and preserves both Async CAM Attention graph validation and the synchronous CAMP2p Attention-side routing constraints from vllm-project#503. Focused CPU checks passed **123 cases**, with **35 dependency-gated skips**. Changed-file pre-commit hooks and `git diff --check` passed. NPU results below are prior evidence; no new device run was performed for this merge. After the earlier main synchronization at `7fca98e`, focused Async CAM connector/session CPU checks pass **40 cases**; **3 graph test modules** are skipped because the host lacks native dependencies. Documentation hooks, local links and diff hygiene pass. Conflict resolutions retain the Graph capability descriptions and main's documentation reorganization. Attention graph implementation files are unchanged. The earlier **112 passed, 33 dependency-gated skips** and Ruff results at `822779f` are reused. The merge retains the graph control-plane test fields and main's `attention_step` connector context. Six pinned-runtime feature-validation checks and the earlier metadata, restoration, signature and device results below are reused pre-merge evidence. No new NPU experiment was run for conflict resolution. Configuration tests continue to admit PD/SP/native options and retain the model-based draft boundary and DBO rejection. Retained NPU comparisons used the pre-merge main-integrated execution code (`5afe9dd89314a36850cbb19e9a4e40d9706c3b83`, with the admission gates removed): | Combination | Fixed functional requests | Greedy output vs same-config eager | Successful target / draft replays per A rank | Padded draft replays per A rank | | --- | ---: | --- | ---: | ---: | | SP / FlashComm1 | 13 | All token-for-token identical | 73 / 86 | 86 | | Native PD + SP / FlashComm1 | 10 | All token-for-token identical | 82 / 83 | 83 | Actual SP flags were checked on both target and draft. PD used `MooncakeHybridConnector` with native full-draft P TP4. Cases cover one-token, 127/128/129-token and chunked 513-token prompts, concurrency and request reuse. PD eager/graph both recorded 405 drafted and 263 accepted tokens. In the SP comparison, the totals were 475/347 and 465/345 respectively; outputs still matched exactly, and concurrent scheduling was not fixed between arms. The pinned native configuration resolver was also executed directly: requested `FULL` resolves to `FULL_DECODE_ONLY` for DSV4's `UNIFORM_BATCH` backend. This was a configuration check, not another model experiment. Compilation mode 3 and model-free speculation admission were checked at the configuration seam only; their device execution is not qualified here. One formal measurement per arm used W4A8, A DP1/TP4, F EP4/TP1 FULL, five speculative tokens, 32 requests, 128 input and 128 output tokens, concurrency 4 and four client warmups. Each arm completed 32/32 requests and 4,096 output tokens. Functional replay probes were removed before timing; their counters remained unchanged afterward. No profiling ran during measurement. | Combination | Mean TPOT, eager → dual graph | Output throughput, eager → dual graph | Mean TTFT, eager → dual graph | | --- | ---: | ---: | ---: | | SP / FlashComm1 | 38.40 → 14.83 ms | 86.89 → 221.53 token/s | 412.96 → 254.95 ms | | Native PD + SP / FlashComm1 | 36.15 → 12.22 ms | 83.85 → 226.11 token/s | 1041.01 → 531.84 ms | Retained earlier non-PD, SP-off evidence: nine finite requests matched the reference token-for-token, with 51 target and 60 draft replays per A rank. Its single performance comparison measured TPOT 32.20 → 14.63 ms, throughput 102.62 → 226.70 token/s and TTFT 393.14 → 234.11 ms. That earlier measurement used `140369b873c34779d0d53a8af82aced3c47484eb` on an older main baseline; it was reused rather than rerun. Do not compare absolute numbers across the separate deployment experiments or attribute the combined gain to one graph. Pinned stack: vLLM `5df63a7203cdb9328d824a27b682e77c7b9928be`, vLLM-Ascend `bd42eea01d8312c4df469fabaed45f77a12df4d3`, CANN 9.2 and torch-npu 2.10.0.post7 on Ascend 910C/A3. Existing native binaries were reused unchanged. MegaMoE with A graphs, W8A8 A graphs, other KV connectors/topologies, GPU execution, DBO and task-level model accuracy are not newly qualified. Successful cleanup of owned processes is not a new graceful-shutdown qualification. ## Docs Impact Update the existing Attention, model/platform, Async CAM/FFN and DSpark recipe contracts. Distinguish actual implementation dependencies from device validation coverage. Experiment scripts, probes, raw logs and private evidence stay outside the patch. --- <details> <summary>Essential PR Checklist</summary> - [x] Purpose is clear and linked to public context. - [x] Scope is bounded. - [x] Compatibility with the pinned vLLM 0.32 / Ascend pair is considered. - [x] No upstream source checkouts are changed. - [x] Existing plugin classes and native graph managers are reused. - [x] Scoped native hooks are restored, documented and signature-checked. - [x] Runtime imports retain NPU boundaries and dependency guards. - [x] Executed, reused and unverified validation paths are distinguished. - [x] Documentation impact is stated. </details> --------- Signed-off-by: jiaran-king <jiaranran2@gmail.com> Signed-off-by: Zhou ziheng <jiaranran2@gmail.com>

The branch's base (codex/ffn-faas-framework) landed on main as the vllm-project#490 squash, so files were resolved against the vllm-project#490 commit this branch had already merged (47c75d8) rather than the older main merge base. That keeps the GPU FFN service changes, takes upstream's later NPU work (including the ffn_model/deepseek_v4/npu regrouping, without resurrecting the old platforms/npu/v4_* files), and leaves every file outside this branch's own changes identical to upstream/main. tests/e2e takes upstream's model-owned case layout (vllm-project#561) as-is; the standalone FFN service E2E cases are ported to it in the next commit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>

vllm-project#561 moved E2E scenarios into model-owned Case/Deployment declarations, which left the branch's legacy runner path for the standalone FFN service unreachable. Port the four afd-deepep-m2n-faas-*-2a2f cases to tests/e2e/models/qwen3_moe/cases.py. Deployment.ffn_service declares that the FFN role runs the standalone FFN service under torchrun instead of `vllm serve`. The deployment builds its command from the same AFD config as the vLLM FFN role, waits for every rank's /ready before the Attention API, and finish() ends a passing case in the service's stop order (/stop on every rank, then Attention) and requires a clean launcher exit and released resources per rank. Cleanup sweeps the torchrun workers by run environment, as they live outside the launcher's process group. Execution evidence is required from Attention only; the service has no vLLM Worker. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>

jiaran-king and others added 23 commits November 5, 2026 14:45

refactor qwen3_moe

6fa7148

Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>

Merge main into PR vllm-project#490 standalone FFN service

bd5dc94

Keep standalone model attachment with the updated Async CAM diagnostics. Follow vllm-project#550 by removing obsolete FFN two-stream options and configuration keys. Signed-off-by: jiaran-king <jiaranran2@gmail.com>

refactor(ffn): reuse native engine CLI and AFD configuration

b26bdc3

Signed-off-by: jiaran-king <jiaranran2@gmail.com>

Share FFN launcher environment checks

717ca64

Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>

Share standalone FFN launch validation

288f323

Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>

Share F-only FFN engine arguments

4b20b9e

Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>

Share FFN model attachment validation

ad89f42

Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>

Share standalone FFN environment invariants

b57c2ab

Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>

Load Qwen3 FFN-only routed experts through vLLM

46772c2

Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>

Fix GPU worker unit-test configuration stubs

a607c3d

Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>

sync with faas branch

25cb087

Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>

ronenkat force-pushed the ffn-faas-qwen3-30b branch from 9680aaf to b772817 Compare November 7, 2026 05:03

ronenkat and others added 6 commits November 8, 2026 14:44

Merge main into PR vllm-project#490 native CLI alignment

47c75d8

Retain standalone CLI/configuration reuse while adopting the Async CAM session ownership refactor and current runtime documentation. Signed-off-by: jiaran-king <jiaranran2@gmail.com>

ronenkat and others added 2 commits November 9, 2026 19:15

github-actions Bot added documentation Improvements or additions to documentation NVIDIA Ascend labels Nov 9, 2026

jiangkuaixue123 and others added 16 commits November 9, 2026 22:32

Navigation Menu

Repository navigation

FFNaaS for qwen3-30b - #53

FFNaaS for qwen3-30b#53
ronenkat wants to merge 47 commits into
codex/ffn-faas-frameworkfrom
ffn-faas-qwen3-30b

ronenkat commented Nov 5, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

ronenkat commented Nov 5, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants