| Documentation | Recipe | Slides | DeepWiki | User Forum | Developer Slack | WeChat |

Overview

afd-plugin is a vLLM external plugin for Attention-FFN Disaggregation (AFD). It provides plugin-owned worker classes, model runners, model wrappers, connectors, configuration validation, compatibility shims, and hardware-gated integration tests for GPU and Ascend NPU deployments.

Note

This project is still experimental and needs more large-scale testing across different hardware backends.

The GPU and Ascend target is vLLM v0.32.0 at commit 5df63a72. The Ascend integration also targets vLLM-Ascend bd42eea0. The plugin does not modify the vLLM source tree. AFD behavior is installed through the vllm.general_plugins entry point, --additional-config, automatically selected role workers, plugin-owned model wrappers, and narrow version-scoped compatibility shims.

Architecture

Current Status

Core runtime support:

• vLLM plugin registration, AFD configuration, and runtime validation.
• Attention/FFN workers, model runners, model wrappers, and connector-driven execution for CUDA and Ascend NPU.
• Eager and FULL_DECODE_ONLY graph execution, plus backend-specific profiling support.
• Native DBO with exactly two ubatches on CUDA V2. Ascend constraints are recorded in the existing NPU runtime matrix.
• DeepSeek MoE handoff at the remote-experts boundary on CUDA, with the gate placed on either Attention or FFN.
• Expert-routed M:N exchange on CUDA (DeepEPM2NAFDConnector) for Qwen3 MoE and DeepSeek-V2: Attention routes each token and dispatches it only to the FFN ranks that own its experts.

Model support:

Model family	Registered architectures	Plugin model wrappers	Notes
DeepSeekV2 / DeepSeekV3 / DeepSeekV3.2	DeepseekForCausalLM, DeepseekV2ForCausalLM, DeepseekV3ForCausalLM, DeepseekV32ForCausalLM	AFDDeepseekForCausalLM, AFDDeepseekV2ForCausalLM, AFDDeepseekV3ForCausalLM	DeepSeekV3.2 uses AFDDeepseekV3ForCausalLM. Each AFD role constructs and loads only its role-required model components, while shared embedding, normalization, and output components remain available where required by the model lifecycle.
DeepSeek V4 Flash	DeepseekV4ForCausalLM	AFDDeepseekV4ForCausalLM	CUDA synchronous and Ascend CAM paths; see the runtime matrix for backend-specific validation limits.
Qwen3 MoE	Qwen3MoeForCausalLM	AFDQwen3MoeForCausalLM	CUDA; DeepEP M2N routes on Attention, P2P routes on FFN.
Qwen3.5 / Qwen3.6 MoE	Qwen3_5MoeForConditionalGeneration	AFDQwen3_5MoeForConditionalGeneration	CUDA text-only Qwen3.6-35B-A3B with --language-model-only, native DP4/TP1/EP4, and synchronous AFD 2A1F. Eager and graph execution are supported; GDN DBO is unsupported; see the E2E limits.

Connector support:

See the recipe index for deployment and benchmark examples.

Connector	Platform	Recommend Stage	Sync or Async	Graph Support	Notes
P2pNcclAFDConnector	CUDA	Decode	Sync	FULL_DECODE_ONLY CUDA graph	FFN ranks are ordered before Attention ranks. num_attention_ranks must be greater than or equal to num_ffn_ranks; the Attention ranks are spread over the FFN ranks in blocks that differ in size by at most one. See the DeepSeek V2 Lite recipe.
DeepEPM2NAFDConnector	CUDA (SM90+)	Decode	Sync	FULL_DECODE_ONLY CUDA graph	Qwen3 MoE or DeepSeek-V2; Attention-side routing, FFN TP1, and matching DBO settings (two microbatches when enabled). Requires afdEP-comm and an expert backend; see the DeepEP M2N guide and recipes.
CAMP2pAFDConnector	Ascend NPU	Decode	Sync	FULL or FULL_DECODE_ONLY; plain MLA graph DBO requires FULL_DECODE_ONLY	Uses HCCL/CAMP2P custom ops and native MRV2. DBO requires two ubatches and Attention DP > 1. See the CAMP2P guide.
CAMAsyncAFDConnector	Ascend NPU	Prefill / decode	Async	FFN-only experimental FULL	DeepSeek-V4 W4A8/W8A8 on A3/910C. Eager Attention supports DP+TP/SP and two native DBO microbatches, including DP1; FFN stays outside native DBO. See the Async CAM guide and FFN runtime contract.

Connector implementations are grouped by backend package: afd_plugin.connectors.gpu for GPU-only connectors, afd_plugin.connectors.npu for NPU-only connectors.

Known gaps:

• NPU hardware evidence covers the specific single-Pod DeepSeek-V2-Lite and DSV4 configurations in the 0.32 validation record. Other models and topologies remain unverified. AFD post-request shutdown failures remain open; successful inference does not imply graceful shutdown.
• Synchronous DSV4 E2E profiles are smoke cases; A5 concurrent answers remain incorrect. See the profile limits.
• Both backends require native ModelRunnerV2. The runtime matrix records graph/DBO constraints and known DeepSeek-V2-Lite shutdown failures.
• GPU and NPU device/E2E tests require real hardware; model E2E also requires weights. DeepEP M2N needs SM90+ and its device/E2E cases run outside CI.
• Qwen3 MoE requires Attention-side gate placement for DeepEP M2N and FFN-side gate placement for P2P. It rejects sequence-parallel MoE, EPLB, pipeline parallelism, speculative decoding, LoRA, and NPU.
• Qwen3.5/Qwen3.6 MoE adapter-family support is limited to the CUDA text-only lane; repository hardware E2E evidence covers Qwen3.6-35B-A3B. GDN DBO is unsupported; upstream hybrid/multimodal DBO validation applies. NPU and multimodal execution are unsupported/unverified; compute_gate_on_attention=true, pipeline parallelism, asynchronous and multi-node execution are unsupported; quantization is unverified; no performance claim is made.

Install

Requires Python 3.11-3.13 and uv.

git clone https://github.com/vllm-project/afd-plugin.git
cd afd-plugin
uv sync --group dev

vllm is an optional runtime extra so CPU-only or macOS development environments can still run import/config tests without a CUDA wheel.

GPU installation

For Linux / CUDA-capable environments only; Ascend users should skip this command:

uv sync --group dev --extra vllm

The optional GPU extra pins vllm==0.32.0.

DeepEPM2NAFDConnector also needs the afdep extra (afdEP-comm) and DeepGEMM, which is built from source; see Installation in the connector guide.

uv sync --group dev --extra vllm --extra afdep

Ascend NPU installation

The Ascend integration uses the source pair below. See the 0.32 NPU validation record for the tested scope and retained shutdown limitations. Earlier 0.30 results retain their historical environment and sample counts. Install a compatible driver and firmware, and confirm the devices with npu-smi info.

Component	Version
Python	Package range 3.11–3.13; tested 3.12.13
vLLM	0.32.0, commit 5df63a7203cdb9328d824a27b682e77c7b9928be
vLLM-Ascend	commit bd42eea0
Tested CANN / torch / torch-npu	9.2 / 2.10.0+cpu / 2.10.0.post7
Tested Triton Ascend / CAM	3.6 / 1.3

Hardware evidence is scoped to the recorded models, configurations, and sample counts; it does not cover every NPU deployment or guarantee clean device shutdown.

Environment

The recorded A3 environment uses the pinned pair and stack above. The repository does not claim a released container tag for the Ascend target. Use the installation guide at that source snapshot to prepare a matching A3/openEuler environment, then install AFD from the repository root.

Install AFD

From the AFD repository root:

AFD_BUILD_ASCEND_OPS=1 \
SOC_VERSION=ascend910_9391 \
python -m pip install -v --no-build-isolation --no-deps -e .

--no-deps preserves the matched NPU runtime, and --no-build-isolation uses its CANN/torch-npu toolchain. The command above forces the Ascend op build for reproducibility. Otherwise, leave AFD_BUILD_ASCEND_OPS unset to use the auto-detection described in Development, set it to 1 if detection misses an Ascend environment, or set it to 0 to skip the ops.

Verify

python - <<'PY'
import torch
import torch_npu
import vllm_ascend

from afd_plugin.compat.npu import ensure_afd_ascend_ops_loaded

assert torch.npu.is_available(), "torch-npu cannot see an Ascend device"
ensure_afd_ascend_ops_loaded()
print("AFD_OPS_OK")
PY

After AFD_OPS_OK, the environment is ready to run the NPU examples and E2E tests. See the NPU testing guide. For implementation details, see the Attention runtime design and FFN runtime design.

Using the Plugin

Install or sync the distribution as vllm-afd-plugin. Python imports use the afd_plugin package name.

AFD is configured through vLLM --additional-config. There is no separate --afd-config flag. When AFD is configured and --worker-cls is omitted, the plugin automatically selects the Attention or FFN worker for the active CUDA or standard Ascend NPU platform. Explicit AFD worker paths remain accepted for compatibility with existing commands, but are not required or stable launch interfaces.

GPU and NPU launches require VLLM_USE_V2_MODEL_RUNNER=1 on both roles. Attention uses native ModelRunnerV2; FFN keeps its connector-driven runner. Neither backend has a V1 fallback. GPU DBO requires two microbatches and accepts Attention DP1 or DP > 1. See the dedicated E2E scenarios.

export VLLM_USE_V2_MODEL_RUNNER=1

GPU Attention-side shape:

vllm serve /path/to/DeepSeek-V2-Lite \
  --served-model-name deepseek-v2-lite-afd-attention \
  --data-parallel-size 1 \
  --tensor-parallel-size 1 \
  --enable-expert-parallel \
  --enforce-eager \
  --host 127.0.0.1 \
  --port 18000 \
  --additional-config '{"afd":{"role":"attention","connector":"P2pNcclAFDConnector","host":"127.0.0.1","port":6239,"num_attention_ranks":1,"num_ffn_ranks":1}}'

GPU FFN-side shape:

vllm serve /path/to/DeepSeek-V2-Lite \
  --served-model-name deepseek-v2-lite-afd-ffn \
  --data-parallel-size 1 \
  --tensor-parallel-size 1 \
  --enable-expert-parallel \
  --enforce-eager \
  --host 127.0.0.1 \
  --port 18001 \
  --additional-config '{"afd":{"role":"ffn","connector":"P2pNcclAFDConnector","host":"127.0.0.1","port":6239,"num_attention_ranks":1,"num_ffn_ranks":1}}'

NPU uses the same config channel with CAMP2pAFDConnector; the plugin selects the NPU worker automatically. Attention uses native Ascend ModelRunnerV2:

VLLM_USE_V2_MODEL_RUNNER=1 vllm serve /path/to/DeepSeek-V2-Lite \
  --served-model-name deepseek-v2-lite-afd-attention \
  --data-parallel-size 1 \
  --tensor-parallel-size 1 \
  --enable-expert-parallel \
  --enforce-eager \
  --host 127.0.0.1 \
  --port 18000 \
  --additional-config '{"afd":{"role":"attention","connector":"CAMP2pAFDConnector","host":"127.0.0.1","port":6239,"num_attention_ranks":1,"num_ffn_ranks":1}}'

Attention and FFN may be started in either order. Send requests only to the Attention API server. FFN workers are connector-driven; scheduler-driven FFN execute_model() calls fail fast.

For repeatable local smoke testing, prefer the bundled runner:

export HF_HOME=/path/to/huggingface
python -c 'from datasets import load_dataset; load_dataset("openai/gsm8k", "main")'

uv run python tests/e2e/runner.py \
  --model /path/to/DeepSeek-V2-Lite \
  --device-backend gpu \
  --scenario afd-eager-2a1f \
  --attention-devices 0,1 \
  --ffn-devices 2 \
  --gsm8k-output-path /tmp/afd-e2e-gsm8k \
  --api-port-base 18000 \
  --afd-port 6239 \
  --common-vllm-arg=--trust-remote-code

For NPU, use --device-backend npu; the runner maps the same device arguments to ASCEND_RT_VISIBLE_DEVICES and selects CAMP2pAFDConnector.

AFD Config

The canonical config shape is:

{
  "afd": {
    "role": "attention",
    "connector": "P2pNcclAFDConnector",
    "async_dp": false,
    "host": "127.0.0.1",
    "port": 1239,
    "afd_process_group_timeout_s": 120,
    "num_attention_ranks": 2,
    "num_ffn_ranks": 1,
    "compute_gate_on_attention": false,
    "connector_extra_config": {}
  }
}

role must be attention or ffn. connector must be P2pNcclAFDConnector, DeepEPM2NAFDConnector, CAMP2pAFDConnector, or CAMAsyncAFDConnector. AFD is active when additional_config["afd"] is present and passes common AFD config validation; omit additional_config["afd"] to disable AFD. Connector-owned connector_extra_config is strictly validated by the selected connector parser when the connector/runtime is constructed. The plugin also accepts selected compatibility aliases such as afd_role, afd_connector, afd_host, and afd_port, plus async for async_dp.

async_dp defaults to false and enables AFD's asynchronous data-parallel compatibility behavior. It is valid only with CAMAsyncAFDConnector, which requires it to be true.

afd_process_group_timeout_s is a positive integer timeout in seconds for the AFD world process-group rendezvous (default: 120). Increase it when rank placement or model loading makes the group join slower; it also bounds the NPU native DBO startup barrier, not device-kernel execution.

Development

Before adding, changing, moving, or removing tests, or changing the behavior they cover, read and follow the test maintenance guide.

Run the default CPU-safe checks:

uv run pytest
uv run ruff check .

The NPU-dependent unit suite must run in an Ascend development environment; see Running NPU unit tests.

Native C/C++ sources are grouped by backend under csrc/: Ascend/CANN sources live in csrc/npu, including the a2e and e2a ACLNN operators, and csrc/gpu is reserved for GPU native sources.

Ascend custom ops are built automatically only when the build environment looks like an Ascend NPU platform, for example when torch_npu, CANN environment variables, or the default Ascend toolkit path are present. GPU builds skip Ascend ops by default. Set AFD_BUILD_ASCEND_OPS=1 or AFD_BUILD_ASCEND_OPS=0 to override the auto-detection.

E2E Test

To run E2E tests, use the run-e2e skill.

License

afd-plugin is licensed under the Apache License 2.0.

Cite

If you find afd-plugin helpful in your research or projects, please consider citing it:

@misc{afdplugin2026,
  title={afd-plugin: Attention-FFN Disaggregation for vLLM},
  author={AFD Plugin Contributors},
  year={2026},
  howpublished={\url{https://github.com/vllm-project/afd-plugin}},
}