Skip to content

refactor(fsdp): share eager and graph shard geometry - #18

Open
yu-chen-code wants to merge 1 commit into
master-graph-shared-dtypesfrom
master-graph-shared-geometry
Open

yu-chen-code wants to merge 1 commit into
master-graph-shared-dtypesfrom
master-graph-shared-geometry

Conversation

@yu-chen-code

@yu-chen-code yu-chen-code commented Nov 11, 2026 •

Copy link
Copy Markdown
Owner

What type of PR is this?

/kind feature
/kind refactor

What does this PR do / why do we need it:

Use one allocation-free get_shard_geometry to calculate source-local chunk offsets, valid local shapes and padded communication shapes. Inputs use the actual parameter FSDP group's rank/degree and the shape after source EP/TP partitioning.

Eager HSDP still materializes an unpadded valid Parameter; graph FSDP still keeps equal padded storage. Clone/contiguous/offload order, parameter selection, scalar/frozen/buffer handling and expert checks stay in their existing executors. This shares geometry without introducing a stateful shard-spec layer.

This stacked PR contains one commit over master-graph-shared-dtypes; merge the precision/storage prerequisite first. This exact prefix passed 203 unit tests and 177 subtests. The integrated-series results below were measured on the aggregate head.

The series uses new branches. Existing PR13 and PR15 branches are unchanged. This PR does not associate or close any issue.

Validation uses Python 3.12.13, PyTorch 2.15.0.dev20261005+cpu and torch-npu 2.15.0. The aggregate head has 205 focused unit tests and 181 subtests passing; the separate Inductor suite has 16 tests passing and two existing strict-equality subtest failures. Both failures reproduce on the exact master baseline with the same values (0.701689600944519 versus 0.7016897201538086); assertions are unchanged.

  • A complete small Qwen3-MoE causal LM ran through TextTrainer with FSDP4 + EP2/EDP2, FP32 storage, BF16 compute, FP32 reduction, AdamW, fixed Indexed input and no clipping. Its two-layer model uses H128, 8 experts/top-2, sequence length 32 and vocabulary 512. Ten steps compare independent master eager, candidate eager and candidate joint-FX execution: all 8,000 per-tensor gradient/weight/Adam comparisons are elementwise exact, and relative loss error is zero. This is FX execution with Inductor off.
  • Single-rank FP32 storage/compute with BF16 reduction completed ten steps; old eager, new eager and joint-FX execution match exactly for loss, gradients, weights and Adam states. The reduction cast is captured after compute gradients, before topology-dependent FSDPPass and its early returns.
  • A four-rank FSDP+EP mixed-precision module also completed ten steps, two microbatches per step and gradient clipping; loss, gradients, gradient norm, weights and Adam moments all match exactly.
  • Four-rank Linear(64,65) and Linear(64,2) each completed ten steps. Valid shards, empty tail ranks, zero padding and optimizer states match master eager and graph execution exactly.
  • Actual CPU FX/full/regional execution each completed ten AdamW steps within 1e-3 for loss and every tensor; gradient, weight and Adam-state errors were zero. NPU full/regional lowering was attempted and failed with no scheduling registered for device npu; NPU Inductor execution is not claimed.
  • On the previous aggregate head, full-file raw repository pylint returned 30 on both master and candidate, with the same 21 diagnostics and no new findings. For this update's three changed Python files, raw repository pylint returned 0 with no diagnostics. No suppressions, skips or tolerance changes were added. Formal upstream CI was not triggered.

Which issue(s) this PR fixes:

None

Code review checklist [illustration]:

  • whether to verify the function's return value (It is forbidden to use void to mask the return values of security functions and self-developed functions. C++ STL functions can be masked if there is no problem)
  • Whether to comply with SOLID principle / Demeter's law
  • Whether there is UT test case && the test case is a valid (if there is no test case, please explain the reason)
  • Whether the API change is involved
  • Whether official document modification is involved

@yu-chen-code
yu-chen-code force-pushed the master-graph-shared-geometry branch from 11cf981 to d834569 Compare November 11, 2026 13:11
@yu-chen-code
yu-chen-code force-pushed the master-graph-shared-dtypes branch from 18643c9 to 1099804 Compare November 11, 2026 13:11
@yu-chen-code
yu-chen-code force-pushed the master-graph-shared-geometry branch from d834569 to 4877541 Compare November 12, 2026 08:21
@yu-chen-code
yu-chen-code force-pushed the master-graph-shared-dtypes branch from 1099804 to 265e5b2 Compare November 12, 2026 08:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant