Skip to content

fix: correct V4.1 CP position and runtime validation - #1553

Open
xuxl1008 wants to merge 1 commit into
mindspore-ai:masterfrom
xuxl1008:fix/v41-rope-position-ids-cp
Open

xuxl1008 wants to merge 1 commit into
mindspore-ai:masterfrom
xuxl1008:fix/v41-rope-position-ids-cp

Conversation

@xuxl1008

@xuxl1008 xuxl1008 commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Paired: GitHub #1553 ↔ GitCode !1926

问题

本 PR 修复两处对合法 V4.1 训练配置的错误校验。

  1. RoPE 位置与 CP 物理行地址被绑定。 共享压缩 Attention 原先要求 position_ids[..., 0] == rank * local_sequence_length,会拒绝 packed 文档内重新编号。例如 CP2、local Q=64、文档从全局第 61 行开始时,rank1 的第一行对应文档内位置 3,旧校验却要求它为 64。模型已使用 position_ids 生成 position_embeddings;文档隔离、因果范围和通信地址应依据 CP 分片及文档边界。
  2. runtime 验收固定要求裁剪模型中不存在的角色。 关闭 candidate pool 的四层裁剪模型保留 SWA、Full、Reuse,但没有 Reindex 或 candidate 发布/消费。旧验收仍固定要求这些事件,导致合法训练步骤在 runtime 验收阶段报缺项。重计算包装后的 Attention/Engram 也不能直接按外层包装器做模块类型判定。

修改

  • 移除 _validate_cp_inputs 对 RoPE 首位置的等值检查;保留 CP rank/size 及 packed local_start、local_length、global_length 一致性检查。公开 forward 仍保留 position_ids 参数,位置编码由 position_embeddings 传入。
  • 从 feat: optimize V4.1 partial RoPE and fix runtime validation #1405 提取 validation/runtime 修复。增加可选的模型级 shared-state 要求解析入口,在 tracing 前依据最终配置和保留的层确定 producer/consumer keys;原有静态声明仍兼容。
  • 同时核对配置与实际 Attention/Indexer 角色,避免因为模块错误缺失而放宽验收。缺少必要发布/消费、消费早于来源发布、禁止的 producer replay 仍报错。
  • 验收模块类型时解包 HyperParallel checkpoint/exclude/swap 包装器及 Torch checkpoint 包装器。

本 PR 基于 master 独立提交,不改变默认 RoPE 编号、candidate 配置、压缩数学或融合算子行为。#1405 的 partial RoPE 优化不在此次提取范围内。

验证

对照 覆盖 结果
修改前 CP2 vs 修改后 CP2 相同权重、输入和全局 RoPE 位置;六层 Attention,main KV AG/A2AV;输出、输入梯度、参数梯度 260 项逐位一致,最大绝对误差 0
修改后 CP2 vs CP1 六层 r1/r2、Full/Reindex/Reuse、自然文档长度;全局位置与文档内重置位置分别对照;main KV AG/A2AV 520 项通过,最大绝对误差 2.3842e-7,阈值 rtol=1e-5, atol=1e-6
CPU 单测 76 项 model integration / cropped validation;36 项 Attention / packed graph 112 项通过
代码检查 六个修改文件 pylint;git diff --check 通过

2.3842e-7 是不同 CP 分片计算之间的误差,不是删除校验造成的改前/改后误差。旧代码拒绝文档内重置位置,因此该输入通过 CP1 参考验证,而不是声称旧 CP2 路径曾成功运行。

validation 回归覆盖 2/4/24/40 层、candidate 开/关、真实缺失 Indexer、来源/消费/replay 错误及当前重计算包装器。上述新增验证为 CPU/Gloo;本轮未新增 NPU 整模型训练或性能结论。

@xuxl1008

Copy link
Copy Markdown
Contributor Author

/check-pr

@xuxl1008

Copy link
Copy Markdown
Contributor Author

/retest

@mindspore-ai-bot

Copy link
Copy Markdown

🔵 The pipeline #1651 is running. Please wait a moment... (Link)

@mindspore-ai-bot

Copy link
Copy Markdown
Project Name Build_Stage Build Result Details(Full/Incremental) Detail
Coverage Line Coverage ✅ SUCCESS 76%/100% Link

@mindspore-ai-bot

Copy link
Copy Markdown
Platform Type Module Excution Time Time Limit
- distributed_data 326.393s 450.0s
- activation_memory 466.06s 375.0s
- shard 202.802s 300.0s
- auto_models 136.115s 150.0s
- pipeline_parallel 667.749s 675.0s
- optimizer 341.298s 300.0s
- distributed_checkpoint 89.612s 120.0s
- accuracy 114.9s 120.0s
- clip_grad 114.585s 120.0s
- dtensor 98.745s 120.0s
- multicore 123.79062s 120.0s
- fully_shard 404.506s 600.0s
- tensor_parallel 278.348s 300.0s
- init_weights 88.712s 118.75s
- context_parallel 192.287s 240.0s
- swap_optimizer 369.767s 300.0s
- expert_parallel 209.28875s 200.0s
- process_group 76.557s 90.0s
- llama3_examples 89.089s 120.0s

@mindspore-ai-bot

Copy link
Copy Markdown
Project Name Build_Stage Build Result Details
Hyper-parallel_Github_Gate - ✅ SUCCESS 1651

@xuxl1008 xuxl1008 changed the title fix: decouple V4.1 CP offsets from RoPE position IDs fix: correct V4.1 CP position and runtime validation Nov 12, 2026
@xuxl1008
xuxl1008 force-pushed the fix/v41-rope-position-ids-cp branch from e1f5ab8 to ece3075 Compare November 12, 2026 11:15
@xuxl1008

Copy link
Copy Markdown
Contributor Author

/retest

@mindspore-ai-bot

Copy link
Copy Markdown

🔵 The pipeline #1655 is running. Please wait a moment... (Link)

@mindspore-ai-bot

Copy link
Copy Markdown
Platform Type Module Excution Time Time Limit
- distributed_data 331.179s 450.0s
- activation_memory 474.757s 375.0s
- pipeline_parallel 773.64s 675.0s
- clip_grad 99.255s 120.0s
- optimizer 337.986s 300.0s
- accuracy 125.565s 120.0s
- shard 218.074s 300.0s
- auto_models 118.05s 150.0s
- fully_shard 402.93s 600.0s
- swap_optimizer 344.468s 300.0s
- tensor_parallel 269.383s 300.0s
- expert_parallel 208.64187s 200.0s
- context_parallel 172.103s 240.0s
- distributed_checkpoint 84.507s 120.0s
- dtensor 89.706s 120.0s
- init_weights 85.059s 118.75s
- process_group 94.132s 90.0s
- multicore 122.10587s 120.0s
- llama3_examples 89.128s 120.0s

@mindspore-ai-bot

Copy link
Copy Markdown
Project Name Build_Stage Build Result Details
Hyper-parallel_Github_Gate - ❌ FAILURE 1655
coverage FAILURE 1655
Smoke_CPU_X86_CentOS_Level0 FAILURE 1655

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants