Skip to content

feat(compile): unify FSDP planning and TextTrainer backend execution - #20

Open
yu-chen-code wants to merge 3 commits into
masterfrom
master-graph-fsdp-unified
Open

yu-chen-code wants to merge 3 commits into
masterfrom
master-graph-fsdp-unified

Conversation

@yu-chen-code

@yu-chen-code yu-chen-code commented Nov 11, 2026 •

Copy link
Copy Markdown
Owner

What type of PR is this?

/kind feature
/kind refactor

What does this PR do / why do we need it:

Share execution-independent FSDP planning between eager execution and TextTrainer joint graphs, and connect the graph's terminal backend without changing eager scheduling. The changes are split into three commits:

  1. Graph storage and shared precision policy. Prepare physical parameter storage before optimizer construction, retain logical shapes for graph capture, and lower the eager FSDP policy into parameter, input/output and gradient casts. Both executors use the same per-parameter dtype resolver; an unspecified reduction dtype preserves the actual gradient dtype. Gradients remain in compute/reduction dtype until storage or FP32-main accumulation.
  2. Shared shard geometry. Reuse one allocation-free planner for chunk offsets, valid local shapes and uniform padded shapes in eager HSDP and graph FSDP. Storage allocation, collective insertion and runtime scheduling remain executor-specific.
  3. TextTrainer joint backend. Project explicit compile.inductor_mode and compile.inductor_configs independently of decoder compilation options. Retain the terminal executable and its training metadata, size backend inputs from physical shards, and preserve the positional state/input calling convention. off executes FX directly; requesting an unavailable backend fails explicitly. This does not add an NPU Inductor scheduler.

The three feature PRs are stacked on separate new branches. This aggregate PR retains exactly three commits over fork master and can be reviewed as the complete change.

The series uses new branches. Existing PR13 and PR15 branches are unchanged. This PR does not associate or close any issue.

Validation uses Python 3.12.13, PyTorch 2.15.0.dev20261005+cpu and torch-npu 2.15.0. The aggregate head has 205 focused unit tests and 181 subtests passing; the separate Inductor suite has 16 tests passing and two existing strict-equality subtest failures. Both failures reproduce on the exact master baseline with the same values (0.701689600944519 versus 0.7016897201538086); assertions are unchanged.

  • A complete small Qwen3-MoE causal LM ran through TextTrainer with FSDP4 + EP2/EDP2, FP32 storage, BF16 compute, FP32 reduction, AdamW, fixed Indexed input and no clipping. Its two-layer model uses H128, 8 experts/top-2, sequence length 32 and vocabulary 512. Ten steps compare independent master eager, candidate eager and candidate joint-FX execution: all 8,000 per-tensor gradient/weight/Adam comparisons are elementwise exact, and relative loss error is zero. This is FX execution with Inductor off.
  • Single-rank FP32 storage/compute with BF16 reduction completed ten steps; old eager, new eager and joint-FX execution match exactly for loss, gradients, weights and Adam states. The reduction cast is captured after compute gradients, before topology-dependent FSDPPass and its early returns.
  • A four-rank FSDP+EP mixed-precision module also completed ten steps, two microbatches per step and gradient clipping; loss, gradients, gradient norm, weights and Adam moments all match exactly.
  • Four-rank Linear(64,65) and Linear(64,2) each completed ten steps. Valid shards, empty tail ranks, zero padding and optimizer states match master eager and graph execution exactly.
  • Actual CPU FX/full/regional execution each completed ten AdamW steps within 1e-3 for loss and every tensor; gradient, weight and Adam-state errors were zero. NPU full/regional lowering was attempted and failed with no scheduling registered for device npu; NPU Inductor execution is not claimed.
  • On the previous aggregate head, full-file raw repository pylint returned 30 on both master and candidate, with the same 21 diagnostics and no new findings. For this update's three changed Python files, raw repository pylint returned 0 with no diagnostics. No suppressions, skips or tolerance changes were added. Formal upstream CI was not triggered.

Which issue(s) this PR fixes:

None

Code review checklist [illustration]:

  • whether to verify the function's return value (It is forbidden to use void to mask the return values of security functions and self-developed functions. C++ STL functions can be masked if there is no problem)
  • Whether to comply with SOLID principle / Demeter's law
  • Whether there is UT test case && the test case is a valid (if there is no test case, please explain the reason)
  • Whether the API change is involved
  • Whether official document modification is involved

@yu-chen-code
yu-chen-code force-pushed the master-graph-fsdp-unified branch from be3c72d to 6209376 Compare November 11, 2026 13:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant