Skip to content

feat(compile): prepare storage and reuse FSDP precision policies - #13

Open
yu-chen-code wants to merge 2 commits into
masterfrom
feat-graph-storage-policy
Open

yu-chen-code wants to merge 2 commits into
masterfrom
feat-graph-storage-policy

Conversation

@yu-chen-code

@yu-chen-code yu-chen-code commented Nov 6, 2026 •

Copy link
Copy Markdown
Owner

What type of PR is this?

/kind feature

What does this PR do / why do we need it:

Refreshed on 2026-11-09 against fork/upstream master a6be99d3. PR1421 is now merged upstream, so its duplicate prerequisite has been removed. Current head: c19a0f0bc37bbdb85db09aa6474f555d28370d14; retained feature/dependency commits: c7d02fea, c19a0f0b. Git range-diff confirms unchanged feature patches. Focused CPU regression on this exact head: 118 JUnit records, 0 failures, 0 errors. The isolated refresh environments now include master-required PEFT 0.21.2 and accelerate 1.13.0; pip check passes. All numerical runs and older source hashes below are pre-refresh evidence. NPU precision training and formal upstream CI were not rerun for this rebase.

Prepare graph FSDP parameter storage before creating optimizer state, and apply the existing FSDP mixed-precision policy to joint forward/backward graphs.

  • TextTrainer creates its compiler and calls prepare_model() before optimizer construction. Preparation retains source-local logical shapes and parameter identities, partitions physical storage once, and exposes final layouts before the first batch. Joint capture remains lazy and uses logical FakeTensor views; graph lowering does not partition live parameters a second time. Failed preparation is terminal, while capture/backend failures retain the prepared storage for a safe retry.
  • Graph precision reuses FSDP2Manager's policy, custom parameter overrides and module selection. Temporary capture hooks record the same floating input/output casts as eager FSDP, including the default cast_forward_inputs=True. The graph casts parameter shards before all-gather, casts gradients before reduction and accumulates FP32 main gradients when requested. Model-owned explicit FP32 regions remain in the model.

Validation on Ascend 910B2 with Torch 2.15.0.dev20261005+cpu / torch-npu 2.15.0:

  • 118 unit tests and 64 subtests passed, including storage idempotence/failure handling, logical static/symbolic capture, optimizer shard shapes, precision overrides and complete AdamW updates.
  • One additional isolated CPU lifecycle contract executed the actual TextTrainer.__init__ with device/distributed setup and data stages mocked. Real graph preparation and the standard AdamW/FP32-main optimizer construction confirmed that the optimizer receives a BF16 3 x 3 local shard and creates a matching FP32 main parameter, while the compiler retains the logical 5 x 3 shape and capture remains lazy.
  • Public CI configuration preflight passed for Python 3.9 and 3.12 with the repository's existing filters; no filter or suppression was added.
  • The pre-refresh head also ran the complete compile suite and related FSDP/precision/gradient unit tests: 421 tests and 382 subtests passed. Two existing standalone Inductor exact-scalar assertions failed in full/regional modes (0.701689600944519 versus 0.7016897201538086); they were not skipped or relaxed, and the complete-suite run therefore exited with failure. The alias-forwarding fixture and other interface tests passed.
  • Four-rank FSDP4 + EP2 + EDP2, real ragged EP/GMM, FP32 storage / BF16 computation / FP32 reduction, default input casts, FP32 main gradients and active clipping at 0.1: 10 optimizer steps with two microbatches each passed. Loss, full gradients, global norms, updated weights and Adam moments all had maximum relative L2 error 0 against eager FSDP; one joint graph was reused.
    Saved pre-clip norms were 0.43545–0.48454, so all 40 rank-steps exceeded 0.1; saved eager pre-gradients and actual first moments confirm scaling, with zero graph/eager gradient/norm/moment error (post-clip gradients were not separately dumped).
  • Independent old/new eager runs compared 980 retrieved tensors exactly across all four ranks, covering initial weights, inputs, IDs, losses, gradients, norms, weights, optimizer moments and step counters. All were equal. Fixed-source hashes, actual editable imports and dependency checks were verified before/after execution.

The numerical runs, constructor contract and public-CI preflight above were pinned to 949c5496. The pre-refresh head 0bf26e68 differs only by one mock in an existing alias-forwarding unit-test fixture; all 1,512 production blobs are identical. Numerical validation was not rerun under the final commit hash.

The distributed acceptance uses GraphCompiler directly and the native eager FSDP path. The separate TextTrainer contract covers its storage/optimizer construction behavior with external stages mocked; end-to-end TextTrainer data loading/training is not established by that test. This PR does not claim DSV4.1 whole-model, CP/TP, checkpoint-resume or backend compilation acceptance.

Which issue(s) this PR fixes:

None

Code review checklist [illustration]:

  • whether to verify the function's return value (It is forbidden to use void to mask the return values of security functions and self-developed functions. C++ STL functions can be masked if there is no problem)
  • Whether to comply with SOLID principle / Demeter's law
  • Whether there is UT test case && the test case is a valid (if there is no test case, please explain the reason)
  • Whether the API change is involved
  • Whether official document modification is involved

@yu-chen-code
yu-chen-code force-pushed the feat-graph-storage-policy branch from 0bf26e6 to c19a0f0 Compare November 9, 2026 11:24
@yu-chen-code
yu-chen-code force-pushed the feat-graph-storage-policy branch from c19a0f0 to 7665551 Compare November 10, 2026 14:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant