Skip to content

feat(hf weights): load hf weights support engram padding, verify HF loading and unify deepseek crop builders - #1372

Merged
github-actions[bot] merged 1 commit into
mindspore-ai:masterfrom
xiaoyuxing-010818:feat/hf-weight-verifier-20261030
Nov 2, 2026
Merged

github-actions[bot] merged 1 commit into
mindspore-ai:masterfrom
xiaoyuxing-010818:feat/hf-weight-verifier-20261030

Conversation

@xiaoyuxing-010818

@xiaoyuxing-010818 xiaoyuxing-010818 commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Paired: GitHub #1372 ↔ GitCode !1869

What type of PR is this?

/kind feature
/kind refactor


What does this PR do / why do we need it:

本 PR 合并三部分功能:支持加载原生 HF 权重时对 Host Engram 表按 EP 补齐物理行数;在通用加载入口增加可选的逐分片加载结果验证;将 DeepSeek V4.1 的两个裁剪加载入口统一为一个。

1. HF 权重加载支持Engram 的 EP padding

HF 中的 Engram 原始行数不一定能被 EP 整除。若把运行时补齐后的物理 shape 直接用于 HF shape 检查或区域读取,会出现 shape mismatch,或读取超出原始 checkpoint 范围。

  • 区分逻辑行数 logical_rows 和物理行数 physical_rows。逻辑行数始终保持 HF 原始值,哈希取模和行号映射不变;物理行数向上补齐到 EP 的倍数。
  • 对每张保留的 Engram 表分别处理。构建阶段将原有表对齐倍数与 EP 大小取最小公倍数;即使资产已提供 padded_num_embeddings,Host 模式也会确保其满足运行时对齐要求。
  • 加载检查允许 HF 原始逻辑行数与运行时补齐后的物理行数不同,并保留对逻辑行数、表宽度和哈希资产的校验。
  • 各 rank 只读取自身物理行分片与原始 HF 权重范围重叠的行;末尾多出的 padding 行置零,完全落在 padding 区域的 rank 不读取越界源数据。
  • 普通 BF16/FP32 和原生量化 HF 权重均复用现有读取流程,不要求预先生成补齐后的权重文件。build_cropped_deepseek_v41 通过 load_base_model 控制加载权重或随机初始化。

2. 支持加载HF权重后验证结果

HF checkpoint 加载报告成功且训练能输出 loss,仍不能确认最终模型中的每个本地权重分片都与 HF 源权重一致。

  • HyperAutoModel.from_pretrained(..., verify_hf_weights=...) 在模型构建、替换、分片及加载完成后、首次 forward 前执行检查;默认关闭,所有 rank 必须一致启用。检查失败不会继续进入训练。
  • 复用记录的转换规则,以符号张量逆向还原名称、合并、拆分与转置;按各 rank 本地分片的全局坐标分块读取 HF 对应区域,不 gather 完整权重。检查源张量和模型分片的完整覆盖。
  • CPU Host Engram 通过外部参数分片描述参与检查。逻辑行数保持 HF 原始行数,物理行数按 EP 补齐;加载只读取与原始权重重叠的行,额外行置零并单独验证。
  • 输出 EXACT、DTYPE_CAST、MISMATCH、INCOMPLETE;保存每 rank JSON、全局 summary.json 和逐张量 tensors.csv,提供数量、误差、差异位置、模型值和参考值等信息。
  • 修复 indexer wq_b 名称逆向还原:将嵌套通配路径明确为 indexer,保持正向加载名称不变,避免多捕获组逆向构造后残留 \2、导致 HF 权重未被匹配。

检验使用零容差;DTYPE_CAST 表示源值转为模型存储 dtype 后精确一致。量化权重的基准是 HF 解码后的值,不是量化前原值。逆向映射和解码复用现有实现,因此本检查验证实际存储符合加载规则,不能独立证明规则的模型语义正确性。

3. 统一 DeepSeek 裁剪入口(加载和不加载权重)与示例配置

  • 移除重复的 build_hf_layer_cropped_deepseek_v41,统一使用 build_cropped_deepseek_v41;复用同一套模型构建、校验、HF 临时索引与加载流程。
  • 同步迁移 VLM YAML 和启动脚本,显式保留原始宽度、384 个专家和原 router 辅助损失系数。load_base_model 默认 false,保留随机初始化行为;层数仍为可选参数。
  • 适配最新上游数据转换 API:将示例中旧的 sequence_alignment: 4 改为 sample_alignment_size: 4,并显式设置 dataset.training_mode: sft,匹配启动脚本生成的 messages 对话样本及旧转换器的监督语义,修复配置解析和数据初始化阶段的异常。

Which issue(s) this PR fixes:

Fixes #1371


Test Plan and Test result:What scenarios were tested, and what were the verification results(Function, performance, reliability, etc.):

测试项 配置 / 覆盖范围 结果 说明与限制
相关单元测试 独立 zbx checkout,editable 安装并核对导入路径;覆盖普通/量化参考值、逆向转换、精度转换、覆盖检查、异常、Host Engram、padding、indexer 和通用入口 通过:78 项测试、105 个子测试 加载开关更名为 load_base_model、移除 local_files_only 设置后的回归也已通过;不是全量 UT/ST
数据预检查 实际 VLM 数据前 8 条样本;training_mode=sft、sample_alignment_size=4 通过 每条含图像、36 个 token、13 个监督 token;已修复上游数据 API 变更引起的两处配置问题
8 卡 HF 全量权重比较 实际运行 HF 加载脚本;81 个模型张量逆向映射为 2381 个 HF 张量,共 127,395,495,148 个元素 通过:2381 个 EXACT MISMATCH=0、INCOMPLETE=0;本轮最新接口名的实际脚本已再次通过全量比较,并完成 10-step 训练
CPU Host Engram 权重比较 第 1 层表,98,305,579,008 个元素 通过:EXACT 最大绝对误差为 0,已纳入跨 rank 汇总
Engram EP padding 相关 UT 覆盖逻辑/物理行数、原始范围重叠读取、部分 padding、全 padding 分片和补零检查 通过(UT) 本次 NPU 8 卡保留表的逻辑行数可被 EP8 整除,实际报告 padding checks=0;不能将其当作 EP128 大表 padding 的 NPU 实测
默认 8 卡、10-step 训练 直接执行 run_deepseek_v41_vlm_online_load_hf.sh;保留 2 层文本、2 层视觉、原始维度、384 个专家及默认训练参数 通过:10/10 step,退出码 0 最新提交 ba9833cc,使用 load_base_model=true 并开启 verify_hf_weights;loss 从 15.1224 到 14.2890,10 个 step 的 loss 和 grad_norm 均为有限值,无 OOM。仅通过启动参数关闭 checkpoint 保存,模型、卡数、精度和训练步数保持配置值。
精度与数据配置核对 FP32 初始化、BF16 参数计算、FP32 梯度归约、FP32 主参数;已有本地 HF 原生量化权重及含真实图像的合成 VLM 数据 已核对 无隐式权重或数据下载;独立设置 HCCL 通信端口

当前结论:权重加载检验、相关回归以及最新配置的默认 8 卡、10-step 训练均已通过。


Self-checklist:(请自检,在[ ]内打上x,我们将检视你的完成情况,否则会导致pr无法合入)

  • 设计:PR对应的方案是否已经经过Maintainer评审,方案检视意见是否均已答复并完成方案修改
  • 测试:PR中的代码是否已有UT/ST测试用例进行充分的覆盖,新增测试用例是否随本PR一并上库或已经上库
  • 验证:PR描述信息中是否已包含对该PR对应的Feature、Refactor、Bugfix的预期目标达成情况的详细验证结果描述
  • 接口:是否涉及对外接口变更,相应变更已得到接口评审组织的通过,API对应的注释信息已经刷新正确
  • 文档:是否涉及官网文档修改,如果涉及请及时提交资料到Doc仓

@xiaoyuxing-010818
xiaoyuxing-010818 marked this pull request as ready for review October 30, 2026 07:21
@xiaoyuxing-010818
xiaoyuxing-010818 marked this pull request as draft October 30, 2026 07:21
@xiaoyuxing-010818
xiaoyuxing-010818 force-pushed the feat/hf-weight-verifier-20261030 branch 3 times, most recently from 3171464 to 118f4e6 Compare October 30, 2026 07:50
@xiaoyuxing-010818 xiaoyuxing-010818 changed the title feat(deepseek-v41): verify HF weight loading and unify crop builders feat(deepseek-v41): pad Engram, verify HF loading and unify crop builders Oct 30, 2026
@xiaoyuxing-010818
xiaoyuxing-010818 force-pushed the feat/hf-weight-verifier-20261030 branch from 118f4e6 to ba9833c Compare October 30, 2026 07:58
@xiaoyuxing-010818
xiaoyuxing-010818 marked this pull request as ready for review October 30, 2026 08:31
@xiaoyuxing-010818

Copy link
Copy Markdown
Contributor Author

/retest

@xiaoyuxing-010818

Copy link
Copy Markdown
Contributor Author

/check-pr

@mindspore-ai-bot

Copy link
Copy Markdown

Gate trigger failed: please fix the /check-pr findings and obtain the pr-check-pass label before retrying.

@xiaoyuxing-010818

Copy link
Copy Markdown
Contributor Author

/retest

@mindspore-ai-bot

Copy link
Copy Markdown

🔵 The pipeline #1264 is running. Please wait a moment... (Link)

@mindspore-ai-bot

Copy link
Copy Markdown
Project Name Build_Stage Build Result Details
Hyper-parallel_Github_Gate - ❌ FAILURE 1264
Check_Pylint FAILURE 1264
Check_AI_Keyword FAILURE 1264
Check_Lizard FAILURE 1264

@mindspore-ai-bot

Copy link
Copy Markdown

@xiaoyuxing-010818, your gate pipeline failed at the code check stage. You can configure some code check tools locally following the (pre-push quick guide), to enable automatic checking when you push your code every time. Tools included are: Check_CMakelint, Check_Codespell, Check_Cpplint, Check_Lizard, Check_Pylint, Check_ClangFormat, Check_ShellCheck.

@xiaoyuxing-010818
xiaoyuxing-010818 force-pushed the feat/hf-weight-verifier-20261030 branch from d1f7469 to 38a7b6c Compare October 30, 2026 09:15
@xiaoyuxing-010818 xiaoyuxing-010818 changed the title feat(deepseek-v41): pad Engram, verify HF loading and unify crop builders feat(hf weights): pad Engram, verify HF loading and unify crop builders Oct 30, 2026
@xiaoyuxing-010818

Copy link
Copy Markdown
Contributor Author

/retest

@mindspore-ai-bot

Copy link
Copy Markdown

🔵 The pipeline #1267 is running. Please wait a moment... (Link)

@mindspore-ai-bot

Copy link
Copy Markdown

🔵 The pipeline #1329 is running. Please wait a moment... (Link)

@mindspore-ai-bot

Copy link
Copy Markdown
Project Name Build_Stage Build Result Details(Full/Incremental) Detail
Coverage Line Coverage ❌ FAILURE 72%/84% Link

@mindspore-ai-bot

Copy link
Copy Markdown
Platform Type Module Excution Time Time Limit
- pipeline_parallel 689.901s 600.0s
- shard 206.266s 300.0s
- auto_models 102.792s 150.0s
- activation_memory 133.77s 120.0s
- distributed_checkpoint 98.916s 120.0s
- clip_grad 104.142s 120.0s
- optimizer 310.463s 300.0s
- swap_optimizer 363.444s 300.0s
- multicore 123.8195s 120.0s
- fully_shard 362.275s 600.0s
- process_group 84.231s 90.0s
- llama3_examples 93.409s 120.0s
- context_parallel 87.349s 120.0s
- accuracy 104.96s 120.0s
- init_weights 100.619s 118.75s
- tensor_parallel 274.312s 300.0s
- expert_parallel 206.69987s 120.0s
- dtensor 77.165s 120.0s

@mindspore-ai-bot

Copy link
Copy Markdown
Project Name Build_Stage Build Result Details
Hyper-parallel_Github_Gate - ❌ FAILURE 1329
Smoke_Ascend910b_Torch FAILURE 1329

@xiaoyuxing-010818 xiaoyuxing-010818 changed the title feat(hf weights): pad Engram, verify HF loading and unify crop builders feat(hf weights): pad Engram, verify HF loading and unify deepseek crop builders Nov 2, 2026
Load native HF checkpoints through the shared DS crop builder and
consolidate cropped construction and pretrained loading entry points.

Keep Engram logical rows and hashing unchanged. Honor user-defined physical
padding without adjusting table sizes to the EP topology. Load each Host
owner's overlap with original HF rows and zero-fill its physical padding.

Expose model-declared checkpoint logical shapes to both DCP and legacy
loaders. Support trailing padding on any tensor dimensions, with bounded
rank-local reads and explicit zero filling for DCP loads.

Verify live model shards and CPU Host Engram against decoded HF weights
before training. Check logical weights and padding separately, and report
exact matches, dtype casts, mismatches and incomplete coverage across ranks.

Fix indexer inverse naming and keep checkpoint loading and verification
within the pylint and complexity gates.
@xiaoyuxing-010818
xiaoyuxing-010818 force-pushed the feat/hf-weight-verifier-20261030 branch from 07de6f3 to aca4c5f Compare November 2, 2026 07:03
@xiaoyuxing-010818

Copy link
Copy Markdown
Contributor Author

/check-pr

@xiaoyuxing-010818

Copy link
Copy Markdown
Contributor Author

/retest

@mindspore-ai-bot

Copy link
Copy Markdown

🔵 The pipeline #1331 is running. Please wait a moment... (Link)

@mindspore-ai-bot

Copy link
Copy Markdown
Project Name Build_Stage Build Result Details(Full/Incremental) Detail
Coverage Line Coverage ❌ FAILURE 72%/84% Link

@mindspore-ai-bot

Copy link
Copy Markdown
Platform Type Module Excution Time Time Limit
- activation_memory 116.20088s 120.0s
- shard 198.977s 300.0s
- pipeline_parallel 693.683s 600.0s
- accuracy 89.0s 120.0s
- clip_grad 99.293s 120.0s
- fully_shard 338.625s 600.0s
- expert_parallel 184.47213s 120.0s
- optimizer 317.406s 300.0s
- auto_models 104.413s 150.0s
- init_weights 84.54s 118.75s
- distributed_checkpoint 91.027s 120.0s
- tensor_parallel 273.248s 300.0s
- swap_optimizer 369.755s 300.0s
- llama3_examples 78.076s 120.0s
- multicore 134.37463s 120.0s
- context_parallel 97.812s 120.0s
- process_group 79.089s 90.0s
- dtensor 88.8s 120.0s

@mindspore-ai-bot

Copy link
Copy Markdown
Project Name Build_Stage Build Result Details
Hyper-parallel_Github_Gate - ❌ FAILURE 1331
Smoke_Ascend910b_Torch FAILURE 1331

@xiaoyuxing-010818 xiaoyuxing-010818 changed the title feat(hf weights): pad Engram, verify HF loading and unify deepseek crop builders feat(hf weights): load hf weights support engram padding, verify HF loading and unify deepseek crop builders Nov 2, 2026

@liuchongming74 liuchongming74 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@github-actions
github-actions Bot merged commit d168d6b into mindspore-ai:master Nov 2, 2026
11 of 16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature]: 支持 HF 权重加载结果验证并统一 DeepSeek V4.1 裁剪入口

6 participants