Repository navigation
Conversation
xuxl1008
force-pushed
the
feat/v41-indexer-balance
branch
from
October 22, 2026 01:59
e11c509 to
fa261c3
Compare
xuxl1008
force-pushed
the
feat/v41-index-key-prefix
branch
from
October 22, 2026 01:59
7681257 to
f21ebef
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
本 PR 将 Index K 的全局 all-gather 扩展为按文档因果前缀分发,覆盖原 Q owner 的 KL 与均衡 worker 的 LI 两类需求,并裁剪无梯度选择任务带来的反向通信。
这是 mindspore-ai#1143 的拆分 C,依赖 B 的任务均衡/qr 分发以及 A 的 CP 通信。当前先推个人 fork,前置合入后再提主仓 master。本阶段一个功能提交,不改变 B 的任务分配和 main attention/teacher/KL owner 布局。
主要修改:
index_k_comm: all_gather / all_to_all_v,默认 all_gather;可与 raw/main 的通信选项分别组合。配置示例:
验证:22 项现有 UT;CPU/Gloo CP8 六层共享链,固定任务布局比较 Index K AG/A2AV,四类自然文档、两步 AdamW。输出、输入梯度和 TopK 逐位一致,参数梯度最大绝对差 9.095e-13,更新后参数 4.657e-10;默认 FP32 输出/梯度断言通过。hybrid r1/r2 的最小选择缓存检查 16 条 CP4 记录通过,重计算不重复执行输入通信和 LI。
NPU/HCCL CP4:仅在验证 driver 中把 AG backward 改成 FP32 SUM,对齐 C 的 owner 累加精度;四种任务模式、四类文档、两步 AdamW 共 128 条 rank-case、65024 个指标全部逐位一致。产品公共 AG 保持原 dtype,此验证设置没有进入提交。
产品 BF16 AG 与 A2AV 的 CP8 对照完成了 ragged、owner-empty、hybrid-mix 三类文档全部模式:第一步输出、输入梯度和 TopK 逐位一致,参数梯度最大相对 L2 0.3105%;更新后部分 TopK 改变,第二步输出最大相对 L2 0.2299%、参数梯度 1.1284%。差异首先出现在 Indexer wk 梯度。统一 FP32 SUM 对照支持累加精度差异的判断,不把产品两条路径表述为多步逐位一致,也不把这些观测值设成新的验收阈值。CP8 的最后一组 all-empty 测试因另一任务占卡中止;完整空 bank 检查已由 CPU CP8 与上述 NPU CP4 覆盖。
改动文件 pylint、复杂度及格式检查通过。未新增仓库测试、未修改公共启动器。本轮没有新增完整模型性能结论;实际收益取决于文档分布、均衡后的 worker 前缀并集和通信/pack 成本。
前置草稿:#3
本轮补充检查:独立 review、18 项本地 CPU 边界对照、41 项现有 UT 和两个改动文件 pylint 通过;未新增仓库测试或修改公共启动器。边界包括空 Q/K、不完整 TND 分区、int32/int64 与推理缓存复用。此前本地 NPU 消融已检查对应转换的 TopK/输出/梯度;本轮正式分支 CP8 配对复跑因其它任务占卡被资源监测停止,未作为新精度或性能结果。