Repository navigation
Conversation
xuxl1008
requested review from
kisnwangg,
suteng,
yangzhenzhang02 and
yaoyifanx
as code owners
November 12, 2026 08:03
Contributor
Author
|
/check-pr |
Contributor
Author
|
/retest |
|
🔵 The pipeline #1651 is running. Please wait a moment... (Link) |
|
|
xuxl1008
force-pushed
the
fix/v41-rope-position-ids-cp
branch
from
November 12, 2026 11:15
e1f5ab8 to
ece3075
Compare
Contributor
Author
|
/retest |
|
🔵 The pipeline #1655 is running. Please wait a moment... (Link) |
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Paired: GitHub #1553 ↔ GitCode !1926
问题
本 PR 修复两处对合法 V4.1 训练配置的错误校验。
position_ids[..., 0] == rank * local_sequence_length,会拒绝 packed 文档内重新编号。例如 CP2、local Q=64、文档从全局第 61 行开始时,rank1 的第一行对应文档内位置 3,旧校验却要求它为 64。模型已使用position_ids生成position_embeddings;文档隔离、因果范围和通信地址应依据 CP 分片及文档边界。修改
_validate_cp_inputs对 RoPE 首位置的等值检查;保留 CP rank/size 及 packedlocal_start、local_length、global_length一致性检查。公开 forward 仍保留position_ids参数,位置编码由position_embeddings传入。本 PR 基于 master 独立提交,不改变默认 RoPE 编号、candidate 配置、压缩数学或融合算子行为。#1405 的 partial RoPE 优化不在此次提取范围内。
验证
2.3842e-7,阈值rtol=1e-5, atol=1e-6git diff --check2.3842e-7是不同 CP 分片计算之间的误差,不是删除校验造成的改前/改后误差。旧代码拒绝文档内重置位置,因此该输入通过 CP1 参考验证,而不是声称旧 CP2 路径曾成功运行。validation 回归覆盖 2/4/24/40 层、candidate 开/关、真实缺失 Indexer、来源/消费/replay 错误及当前重计算包装器。上述新增验证为 CPU/Gloo;本轮未新增 NPU 整模型训练或性能结论。