Skip to content

feat(compile): bridge graph parameter storage to checkpoint chunks - #14

Open
yu-chen-code wants to merge 3 commits into
masterfrom
feat-graph-checkpoint-layout
Open

yu-chen-code wants to merge 3 commits into
masterfrom
feat-graph-checkpoint-layout

Conversation

@yu-chen-code

@yu-chen-code yu-chen-code commented Nov 6, 2026 •

Copy link
Copy Markdown
Owner

What type of PR is this?
/kind feature

What does this PR do / why do we need it:

Refreshed on 2026-11-09 against fork/upstream master a6be99d3. PR1421 is now merged upstream, so its duplicate prerequisite has been removed. Current head: 813f984f07e466a6549193d5c2b85c539bc4a1bd; retained feature/dependency commits: c7d02fea, bf0eb74c, 813f984f. Git range-diff confirms unchanged feature patches. Focused CPU regression on this exact head: 141 JUnit records, 0 failures, 0 errors. The isolated refresh environments now include master-required PEFT 0.21.2 and accelerate 1.13.0; pip check passes. All numerical runs and older source hashes below are pre-refresh evidence. NPU precision training and formal upstream CI were not rerun for this rebase.

Connect graph-owned FSDP storage to the existing DCP and Hugging Face checkpoint paths using parameter ownership metadata.

  • Build ChunkInfo views from source TP/EP coordinates and the selected FSDP shard. Persist only valid logical rows and restore physical padding before the existing loaders run. Tied aliases share canonical ownership; scalar counters and model buffers keep their original behavior.
  • Map model tensors, FQN-keyed optimizer moments and main-weight entries through the same owner. Preserve state-dict container metadata and leave extra training state untouched. No live Parameter is replaced and no checkpoint planner is rewritten. Cache shape/layout queries for the payload instead of repeating them for every tensor.
  • Prepare graph storage before optimizer/restore through the separate storage prerequisite. CheckpointerCallback adapts graph payloads while its eager path remains unchanged. Existing per-rank extra-state persistence owns scheduler, RNG, data position and global step.
  • Complete the standard strict optimizer load first, then remove only priming-created live optimizer state absent from the actual DCP metadata. Resolve FP32 main Parameters through the public main-to-model identity mapping and canonical FQNs. Persisted zero-valued moments, nonempty moments, main weights and parameter-group steps are preserved; the generic eager/DCP loader is unchanged.
  • Gather FSDP storage, remove its padding, then gather the source TP/EP placement for HF export. CheckpointManager accepts an already-gathered mapping so it does not repeat collectives; omitted mappings retain the existing path.

Validation on deploy32, four Ascend910B2 cards, Python3.12.13 and Torch/torch-npu2.15:

  • Pre-refresh PR head d0cb4ffab1abdb52a71860f1c76cd0a9900a4326: 57 unit tests and 17 subtests passed, covering original checkpoint/DCP/HF regressions, logical padding, aliases, optimizer/main ownership, container metadata, supplied HF mappings and live lazy-state restoration. Both public-CI Pylint configurations report zero unresolved findings for the new three-file restore increment; the earlier wider scope retains its four documented baseline ExportableModel helper warnings. No suppression, filter or skip was added.
  • Actual single-rank Gloo+DCP+callback CPU save/load covers HyperAdamW, its FP32 wrapper, BF16 storage with independently keyed FP32 main Parameters, and native AdamW. All four restore used and legitimate zero-gradient moments exactly, leave never-used state absent, and match the first later real-gradient update exactly. The temporary full load skeleton is retained until the strict loader finishes; no strictness setting is weakened.
  • Combined head f0b60c33704c4507e9d565e444fe69d91c14aadc includes the independent true-unused capture fix from PR16 and the checkpoint increment. The two checkpoint production files have identical blobs to the current PR head; the complete source trees differ. Combined CPU passes 41 unit tests and 23 subtests, plus explicit FP32/BF16 storage, FP32 main-gradient, None and clipping contracts.
  • On that exact combined head, actual DP2+CP2+FSDP4+EP2+EDP2 learned-router/native-GMM training uses FP32 storage, BF16 compute, FP32 reduce, standard HyperAdamW/FP32 optimizer wrapper, StepLR, two microbatches and max norm=1e-4. Three fresh torchrun processes compare continuous ten steps (also against real eager FSDP2Manager) with five steps, real DCP save and a new process restoring the remaining five steps. All 3072 tensor comparisons, including 1600 eager/graph comparisons, are exact. Maximum absolute and relative-L2 errors are 0 under the fixed 1e-3 / floor1e-8 contract. All 92 graph lazy-state checks retain absent unused state and, after training starts, present zero-valued moments for the explicitly used-zero parameter.
  • Historical all-used runs at e74807c7 retain the five-row padding/zero-valid-row rank, tied aliases, scalar, buffer and independently reconstructed export-coordinate coverage (1552 exact tensor comparisons). These remain historical evidence; they are not claimed as a full-tree numerical run of the new PR head. Source/import hashes and pip check were verified before and after each new run.

The graph-only cleanup closes a restore-state contract exposed by true-unused gradients; the underlying DCP priming behavior predates this PR. Complete unused graph capture is supplied by independent PR16, not added by this bridge. NPU BF16-storage resume, full HF file load/export, cross-topology or mid-microbatch recovery, PP with unused gradients and full DSV4.1 training remain outside this acceptance.

Reproduction:

python -m pytest -q tests/ut/compile/test_checkpoint.py \
  tests/ut/compile/test_storage_preparation.py tests/ut/trainer/test_checkpoint_callback.py \
  tests/ut/trainer/test_checkpoint_cadence.py \
  tests/ut/auto_models/_transformers/test_checkpoint_manager_save.py \
  tests/ut/components/checkpoint/test_dcp_checkpointer.py
python -m pip check

T020 runs/20261106-graph-capability-spikes/audit-unused-checkpoint/final-verification.json and checkpoint-blob-equivalence.json record the exact independent CPU and combined NPU heads, raw commands, tensor receipts and blob correspondence. The earlier checkpoint/final-verification.json remains the separate historical all-used geometry record.

Which issue(s) this PR fixes:

None.

Code review checklist [illustration]:

  • whether to verify the function's return value (It is forbidden to use void to mask the return values of security functions and self-developed functions. C++ STL functions can be masked if there is no problem)
  • Whether to comply with SOLID principle / Demeter's law
  • Whether there is UT test case && the test case is a valid (if there is no test case, please explain the reason)
  • Whether the API change is involved
  • Whether official document modification is involved

@yu-chen-code yu-chen-code changed the title feat(compile): preserve graph parameter ownership in checkpoints feat(compile): bridge graph parameter storage to checkpoint chunks Nov 6, 2026
@yu-chen-code
yu-chen-code force-pushed the feat-graph-checkpoint-layout branch from d0cb4ff to 813f984 Compare November 9, 2026 11:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant