Skip to content

fix(compile): capture replayable context parallel collectives - #11

Open
yu-chen-code wants to merge 1 commit into
masterfrom
feat-graph-cp-collectives
Open

yu-chen-code wants to merge 1 commit into
masterfrom
feat-graph-cp-collectives

Conversation

@yu-chen-code

@yu-chen-code yu-chen-code commented Nov 6, 2026 •

Copy link
Copy Markdown
Owner

What type of PR is this?
/kind feature

What does this PR do / why do we need it:

Refreshed on 2026-11-09 against fork/upstream master a6be99d3. PR1421 is now merged upstream, so its duplicate prerequisite has been removed. Current head: a063dd2ab0907f86f8cc5d418d547323ae438390; retained feature/dependency commits: a063dd2a. Git range-diff confirms unchanged feature patches. Focused CPU regression on this exact head: 108 JUnit records, 0 failures, 0 errors. The isolated refresh environments now include master-required PEFT 0.21.2 and accelerate 1.13.0; pip check passes. All numerical runs and older source hashes below are pre-refresh evidence. NPU precision training and formal upstream CI were not rerun for this rebase.

Make asynchronous CP AllGather and Ulysses sequence-to-head exchanges safe to capture and replay in a joint training graph. The existing eager implementation starts a c10d Work object and waits on it in Python. During capture, that Python wait is not a replayable dependency: the baseline captured successfully but produced an incorrect first-step loss when replayed.

During tracing/compilation, use PyTorch native functional all_gather_single / all_to_all_single, retain their output tensor in the collective handle, and materialize it with wait_tensor before restoring the original layout. Autograd follows the native collective's backward. The existing eager Work path and its tensor math stay unchanged.

Allow CP together with graph EP now that these CP exchanges carry explicit graph dependencies. Keep the PP and legacy EP-overlap restrictions. Add focused layout, gradient, wait-lifecycle, and topology tests; update the old CP-rejection expectation.

This change validates the two collective helpers and their composition with FSDP/EP. It does not claim complete DeepSeek-V4.1 DSA attention, compressed-document CP routing, or every CP attention algorithm is graph-ready.

Validation:

  • Baseline reproduction: raw Work capture replay produced a first-step loss relative error of 35.52% on rank 0 and 22.42% on rank 2.
  • Native functional API contract: two-rank CPU/Gloo backward verified both native collectives on Torch 2.15.
  • Ascend 910B2, Torch 2.15.0.dev20261005 / torch-npu 2.15: four ranks, ten AdamW updates, two microbatches. FSDP4+CP2 AllGather and Ulysses both passed. Loss was exact; gradient relative L2 <= 1.03e-7; Adam state relative L2 <= 1.29e-7; parameter relative L2 <= 6.19e-9. The fixed threshold was 1e-3 with denominator floor 1e-8.
  • Actual learned routing/GMM with DP2, CP2, EP2, dense FSDP4 and EDP2 passed ten updates. Dense/expert loss normalization used DP*CP=4, while expert transport used the EDP2 group. Loss, gradients, parameters and Adam tensors were exact against an independent per-backward reduction reference.
  • Before/after eager AllGather and Ulysses comparisons were torch.equal for all recorded gradients, parameters and Adam states across four ranks and ten updates.
  • Pre-refresh full compile and CP UT run: 352 passed and 311 subtests passed; two existing CPU Inductor strict-equality subcases failed by two FP32 ULP on this Torch nightly. The same values reproduce on the unchanged base; no test, tolerance, or suppression was changed to hide them.
  • Both Python 3.9/3.12 CI pylint targets introduced no findings. The collective file retains the same 32 pre-existing custom-autograd signature findings as the exact base. No performance claim is made.

Exact source manifests, commands, before/after tensor receipts and raw failures are retained under T020 runs/20261106-graph-capability-spikes. No formal upstream gate is requested or claimed.

Which issue(s) this PR fixes:

None.

Code review checklist [illustration]:

  • whether to verify the function's return value (It is forbidden to use void to mask the return values of security functions and self-developed functions. C++ STL functions can be masked if there is no problem)
  • Whether to comply with SOLID principle / Demeter's law
  • Whether there is UT test case && the test case is a valid (if there is no test case, please explain the reason)
  • Whether the API change is involved
  • Whether official document modification is involved

@yu-chen-code
yu-chen-code force-pushed the feat-graph-cp-collectives branch from a922f76 to a063dd2 Compare November 9, 2026 11:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant