Repository navigation
feat(hf weights): load hf weights support engram padding, verify HF loading and unify deepseek crop builders - #1372
Merged
github-actions[bot] merged 1 commit intoNov 2, 2026
Conversation
xiaoyuxing-010818
marked this pull request as ready for review
October 30, 2026 07:21
xiaoyuxing-010818
requested review from
liuchongming74,
suteng,
yangzhenzhang02 and
yaoyifanx
as code owners
October 30, 2026 07:21
xiaoyuxing-010818
marked this pull request as draft
October 30, 2026 07:21
xiaoyuxing-010818
force-pushed
the
feat/hf-weight-verifier-20261030
branch
3 times, most recently
from
October 30, 2026 07:50
3171464 to
118f4e6
Compare
xiaoyuxing-010818
force-pushed
the
feat/hf-weight-verifier-20261030
branch
from
October 30, 2026 07:58
118f4e6 to
ba9833c
Compare
xiaoyuxing-010818
marked this pull request as ready for review
October 30, 2026 08:31
Contributor
Author
|
/retest |
Contributor
Author
|
/check-pr |
|
Gate trigger failed: please fix the /check-pr findings and obtain the |
xiaoyuxing-010818
force-pushed
the
feat/hf-weight-verifier-20261030
branch
from
October 30, 2026 08:43
ba9833c to
d1f7469
Compare
Contributor
Author
|
/retest |
|
|
@xiaoyuxing-010818, your gate pipeline failed at the code check stage. You can configure some code check tools locally following the (pre-push quick guide), to enable automatic checking when you push your code every time. Tools included are: Check_CMakelint, Check_Codespell, Check_Cpplint, Check_Lizard, Check_Pylint, Check_ClangFormat, Check_ShellCheck. |
xiaoyuxing-010818
force-pushed
the
feat/hf-weight-verifier-20261030
branch
from
October 30, 2026 09:15
d1f7469 to
38a7b6c
Compare
Contributor
Author
|
/retest |
|
|
Load native HF checkpoints through the shared DS crop builder and consolidate cropped construction and pretrained loading entry points. Keep Engram logical rows and hashing unchanged. Honor user-defined physical padding without adjusting table sizes to the EP topology. Load each Host owner's overlap with original HF rows and zero-fill its physical padding. Expose model-declared checkpoint logical shapes to both DCP and legacy loaders. Support trailing padding on any tensor dimensions, with bounded rank-local reads and explicit zero filling for DCP loads. Verify live model shards and CPU Host Engram against decoded HF weights before training. Check logical weights and padding separately, and report exact matches, dtype casts, mismatches and incomplete coverage across ranks. Fix indexer inverse naming and keep checkpoint loading and verification within the pylint and complexity gates.
xiaoyuxing-010818
force-pushed
the
feat/hf-weight-verifier-20261030
branch
from
November 2, 2026 07:03
07de6f3 to
aca4c5f
Compare
Contributor
Author
|
/check-pr |
Contributor
Author
|
/retest |
suteng
approved these changes
Nov 2, 2026
|
|
yangzhenzhang02
approved these changes
Nov 2, 2026
yaoyifanx
approved these changes
Nov 2, 2026
suteng
approved these changes
Nov 2, 2026
github-actions
Bot
merged commit Nov 2, 2026
d168d6b
into
mindspore-ai:master
11 of 16 checks passed
5 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Paired: GitHub #1372 ↔ GitCode !1869
What type of PR is this?
/kind feature
/kind refactor
What does this PR do / why do we need it:
本 PR 合并三部分功能:支持加载原生 HF 权重时对 Host Engram 表按 EP 补齐物理行数;在通用加载入口增加可选的逐分片加载结果验证;将 DeepSeek V4.1 的两个裁剪加载入口统一为一个。
1. HF 权重加载支持Engram 的 EP padding
HF 中的 Engram 原始行数不一定能被 EP 整除。若把运行时补齐后的物理 shape 直接用于 HF shape 检查或区域读取,会出现 shape mismatch,或读取超出原始 checkpoint 范围。
logical_rows和物理行数physical_rows。逻辑行数始终保持 HF 原始值,哈希取模和行号映射不变;物理行数向上补齐到 EP 的倍数。padded_num_embeddings,Host 模式也会确保其满足运行时对齐要求。build_cropped_deepseek_v41通过load_base_model控制加载权重或随机初始化。2. 支持加载HF权重后验证结果
HF checkpoint 加载报告成功且训练能输出 loss,仍不能确认最终模型中的每个本地权重分片都与 HF 源权重一致。
HyperAutoModel.from_pretrained(..., verify_hf_weights=...)在模型构建、替换、分片及加载完成后、首次 forward 前执行检查;默认关闭,所有 rank 必须一致启用。检查失败不会继续进入训练。EXACT、DTYPE_CAST、MISMATCH、INCOMPLETE;保存每 rank JSON、全局summary.json和逐张量tensors.csv,提供数量、误差、差异位置、模型值和参考值等信息。wq_b名称逆向还原:将嵌套通配路径明确为indexer,保持正向加载名称不变,避免多捕获组逆向构造后残留\2、导致 HF 权重未被匹配。检验使用零容差;
DTYPE_CAST表示源值转为模型存储 dtype 后精确一致。量化权重的基准是 HF 解码后的值,不是量化前原值。逆向映射和解码复用现有实现,因此本检查验证实际存储符合加载规则,不能独立证明规则的模型语义正确性。3. 统一 DeepSeek 裁剪入口(加载和不加载权重)与示例配置
build_hf_layer_cropped_deepseek_v41,统一使用build_cropped_deepseek_v41;复用同一套模型构建、校验、HF 临时索引与加载流程。load_base_model默认false,保留随机初始化行为;层数仍为可选参数。sequence_alignment: 4改为sample_alignment_size: 4,并显式设置dataset.training_mode: sft,匹配启动脚本生成的messages对话样本及旧转换器的监督语义,修复配置解析和数据初始化阶段的异常。Which issue(s) this PR fixes:
Fixes #1371
Test Plan and Test result:What scenarios were tested, and what were the verification results(Function, performance, reliability, etc.):
load_base_model、移除local_files_only设置后的回归也已通过;不是全量 UT/STtraining_mode=sft、sample_alignment_size=4MISMATCH=0、INCOMPLETE=0;本轮最新接口名的实际脚本已再次通过全量比较,并完成 10-step 训练padding checks=0;不能将其当作 EP128 大表 padding 的 NPU 实测run_deepseek_v41_vlm_online_load_hf.sh;保留 2 层文本、2 层视觉、原始维度、384 个专家及默认训练参数ba9833cc,使用load_base_model=true并开启verify_hf_weights;loss 从15.1224到14.2890,10 个 step 的 loss 和 grad_norm 均为有限值,无 OOM。仅通过启动参数关闭 checkpoint 保存,模型、卡数、精度和训练步数保持配置值。当前结论:权重加载检验、相关回归以及最新配置的默认 8 卡、10-step 训练均已通过。
Self-checklist:(请自检,在[ ]内打上x,我们将检视你的完成情况,否则会导致pr无法合入)