Skip to content

test(GPU): Cleanup M2N GPU tests - #585

Open
ronenkat wants to merge 4 commits into
vllm-project:mainfrom
ms-llmd:refactor/cleanup-m2n-tests
Open

ronenkat wants to merge 4 commits into
vllm-project:mainfrom
ms-llmd:refactor/cleanup-m2n-tests

Conversation

@ronenkat

Copy link
Copy Markdown
Collaborator

Purpose

Make tests/gpu/test_deepep_m2n_gpu.py fit a 4-GPU CI budget. A cold run is
dominated by the DeepEP/DeepGEMM JIT compile of each new kernel configuration,
so this removes cases whose only contribution is another compile or coverage
held elsewhere. 35 cases become 23, and none needs more than 4 GPUs.

Issue

Scope

  • In scope: tests/gpu/test_deepep_m2n_gpu.py, tests/gpu/deepep_m2n_harness.py,
    tests/gpu/README.md; comment-only edits in deepep_m2n/connector.py and
    tests/unit/connectors/test_deepep_m2n_connector.py.
  • Out of scope: connector behavior, CI pipeline wiring.

Implementation Notes

Removed cases:

Test Removed Why it is not needed
test_attention_tp_peers_dispatch_disjoint_slices 2F-2DPx2TP-bf16, 2F-2DPx2TP-fp8 (6 GPUs), 2F-1DPx2TP-bf16 Kept 2F-1DPx2TP FP8. The per-DP-rank split over TP peers is CPU logic, covered by test_tokens_per_attention_rank_splits_each_dp_rank_over_its_tp_peers.
test_transport_and_deepgemm_experts 1F1A-bf16, 1F1A-fp8, 1F3A-bf16 Kept 2F2A in BF16 and FP8 plus 1F3A in FP8 (pairwise, as test_two_lanes_dbo_order).
test_transport_with_fused_experts_backend both cases Every FFN check already runs vLLM fused_experts as the reference.
test_fp8_without_host_sync_for_small_buckets whole test The *_nosync standard cases skip the host sync with worst-case shapes; the threshold, ladder and precompile rules are unit-tested.
test_graph_and_eager_plans_mixed_across_ranks whole test, plus mixed_mode_cases and ffn_graph_mode in the harness The FFN eager fallback keeps Attention's replay flag, so ranks plan a step alike; the bucket invariant stays unit-tested.
test_dbo_pipeline_cuda_graph_capture_and_replay both cases test_dbo_graph_capture_reuses_sent_rows runs the same captured DBO scenario with full checks first.

Other changes: test_close_cancels_a_blocked_control_receive runs on 2F2A (reuses
compiled kernels) and checks every FFN rank; the peer-exit wall-clock assertion is
dropped (run_ranks already reports ranks alive at the budget as hung).

Test Plan

  • pytest tests/unit and pytest -m "not gpu and not vllm_runtime" tests/unit
  • pytest -sv -m gpu tests/gpu/test_deepep_m2n_gpu.py on 4 GPUs, cold then warm JIT caches

Test Result

vLLM 0.32.0, torch 2.13.0+cu130, afdEP-comm v1.1.0, DeepGEMM 2.8.2 (5081f85), 4x H200.

  • Unit: 1619 tests, 1565 passed, 52 skipped, 2 failed; CI selection 1605 / 1552 / 51 / 2.
    Both failures (test_npu_feature_validation[gate-on-attention], [async-graph])
    also fail on main 6620336 and are unrelated.
  • GPU (tests/gpu/test_deepep_m2n_gpu.py): 23/23 passed with cold JIT caches
    (17.8 min; the old 35-case suite took 32.1 min cold on the same 4x H200 setup)
    and 23/23 with warm caches (5.4 min).

Docs Impact

  • Files updated: tests/gpu/README.md (GPU requirement, case count, test table).

ronenkat and others added 4 commits November 15, 2026 09:38
…paths

Cut cases that only add a cold JIT compile or repeat coverage held
elsewhere, so the suite fits a 4-GPU CI budget:

- transport: keep 2F2A in BF16 and FP8 plus 1F3A in FP8 (pairwise
  topology x payload, as test_two_lanes_dbo_order already does).
- Attention TP: keep 2F-1DPx2TP in FP8; drop the 6-GPU 2F-2DPx2TP cases.
  The per-DP-rank split over TP peers is covered on CPU by
  test_tokens_per_attention_rank_splits_each_dp_rank_over_its_tp_peers.
- drop test_transport_with_fused_experts_backend: every FFN check
  already runs the fused_experts reference.
- drop test_fp8_without_host_sync_for_small_buckets: the *_nosync
  standard cases skip the host sync with worst-case shapes, and the
  threshold, ladder and precompile rules are unit-tested.
- drop test_graph_and_eager_plans_mixed_across_ranks and its harness
  helpers: the FFN eager fallback keeps Attention's replay flag, so the
  ranks plan a step alike; the bucket invariant stays unit-tested.
- fold test_dbo_pipeline_cuda_graph_capture_and_replay into
  test_dbo_graph_capture_reuses_sent_rows, which runs the same captured
  DBO scenario with full checks before its memory-growth check.
- run test_close_cancels_a_blocked_control_receive on 2F2A so it reuses
  the compiled kernels, and check every FFN rank.
- drop the peer-exit wall-clock assertion; run_ranks already reports
  ranks alive at the budget as hung.

Sync tests/gpu/README.md and the comments that cited removed cases.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
@github-actions github-actions Bot added the NVIDIA NVIDIA GPU, CUDA, and related changes label Nov 15, 2026
@ronenkat

Copy link
Copy Markdown
Collaborator Author

FYI @Etelis

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

NVIDIA NVIDIA GPU, CUDA, and related changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant