Skip to content

feat(compile): share FSDP precision policy and prepare graph storage - #17

Open
yu-chen-code wants to merge 1 commit into
masterfrom
master-graph-shared-dtypes
Open

yu-chen-code wants to merge 1 commit into
masterfrom
master-graph-shared-dtypes

Conversation

@yu-chen-code

@yu-chen-code yu-chen-code commented Nov 11, 2026 •

Copy link
Copy Markdown
Owner

What type of PR is this?

/kind feature
/kind refactor

What does this PR do / why do we need it:

Prepare graph FSDP storage before optimizer creation and lower the existing eager precision policy into joint forward/backward capture. Logical source-local shapes are retained separately from physical shards; preparation preserves Parameter identity, runs once and exposes final layouts before lazy capture.

FSDP2Manager resolves the same policy, custom source-parameter overrides and module boundaries for both executors. A shared pure resolve_param_dtypes retains eager override/explicit-None/redundant-cast rules. Graph capture records input/output casts and compute parameter views, differentiates those views, then reduces gradients in the selected or actual gradient dtype before storage/FP32-main accumulation. Eager scheduling and gradient communication fallback remain unchanged.

This is the first commit of the series, based on fork master. The necessary storage/mixed-precision foundation from PR13 is included here. This exact prefix passed 200 unit tests and 123 subtests. The integrated-series results below were measured on the aggregate head.

The series uses new branches. Existing PR13 and PR15 branches are unchanged. This PR does not associate or close any issue.

Validation uses Python 3.12.13, PyTorch 2.15.0.dev20261005+cpu and torch-npu 2.15.0. The aggregate head has 205 focused unit tests and 181 subtests passing; the separate Inductor suite has 16 tests passing and two existing strict-equality subtest failures. Both failures reproduce on the exact master baseline with the same values (0.701689600944519 versus 0.7016897201538086); assertions are unchanged.

  • A complete small Qwen3-MoE causal LM ran through TextTrainer with FSDP4 + EP2/EDP2, FP32 storage, BF16 compute, FP32 reduction, AdamW, fixed Indexed input and no clipping. Its two-layer model uses H128, 8 experts/top-2, sequence length 32 and vocabulary 512. Ten steps compare independent master eager, candidate eager and candidate joint-FX execution: all 8,000 per-tensor gradient/weight/Adam comparisons are elementwise exact, and relative loss error is zero. This is FX execution with Inductor off.
  • Single-rank FP32 storage/compute with BF16 reduction completed ten steps; old eager, new eager and joint-FX execution match exactly for loss, gradients, weights and Adam states. The reduction cast is captured after compute gradients, before topology-dependent FSDPPass and its early returns.
  • A four-rank FSDP+EP mixed-precision module also completed ten steps, two microbatches per step and gradient clipping; loss, gradients, gradient norm, weights and Adam moments all match exactly.
  • Four-rank Linear(64,65) and Linear(64,2) each completed ten steps. Valid shards, empty tail ranks, zero padding and optimizer states match master eager and graph execution exactly.
  • Actual CPU FX/full/regional execution each completed ten AdamW steps within 1e-3 for loss and every tensor; gradient, weight and Adam-state errors were zero. NPU full/regional lowering was attempted and failed with no scheduling registered for device npu; NPU Inductor execution is not claimed.
  • On the previous aggregate head, full-file raw repository pylint returned 30 on both master and candidate, with the same 21 diagnostics and no new findings. For this update's three changed Python files, raw repository pylint returned 0 with no diagnostics. No suppressions, skips or tolerance changes were added. Formal upstream CI was not triggered.

Which issue(s) this PR fixes:

None

Code review checklist [illustration]:

  • whether to verify the function's return value (It is forbidden to use void to mask the return values of security functions and self-developed functions. C++ STL functions can be masked if there is no problem)
  • Whether to comply with SOLID principle / Demeter's law
  • Whether there is UT test case && the test case is a valid (if there is no test case, please explain the reason)
  • Whether the API change is involved
  • Whether official document modification is involved

@yu-chen-code
yu-chen-code force-pushed the master-graph-shared-dtypes branch from 18643c9 to 1099804 Compare November 11, 2026 13:11
@yu-chen-code
yu-chen-code force-pushed the master-graph-shared-dtypes branch from 1099804 to 265e5b2 Compare November 12, 2026 08:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant