Repository navigation
Conversation
This was referenced Oct 21, 2026
Owner
Author
|
/retest |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
本 PR 将 V4.1 的 SWA/main KV 通信从完整 KV all-gather 扩展为按需分发,并将普通 CP 路径的通信发起与消费分开。
这是 mindspore-ai#1143 的拆分 A,前置为自然文档修复 mindspore-ai#1250。当前在个人 fork 上以集成基线分支为 base,便于只评审本阶段;前置合入后再迁移至主仓 master。原 mindspore-ai#1143 暂时保留作完整对照。
主要修改:
swa_comm默认all_to_all_v,仅交换同文档左侧窗口 halo,保留all_gather对照;反向将消费者贡献 SUM 回原 owner。main_kv_comm支持all_gather(默认)与all_to_all_v;后者分发 owner 所需的文档因果前缀,SMLA/teacher/KL 同时适配紧凑 bank 的物理起点。_AsyncAllGatherWait.backward从全量 all-reduce 后取本地片段改为 SUM reduce-scatter,保持原 dtype 和缩放;仍等待归约完成后返回梯度,未引入反向延迟等待。配置沿用同一 replacement:
验证:22 项现有模型/attention UT;CPU/Gloo CP8 六层共享链,两步 AdamW,四类自然文档,raw AG/halo × main AG/prefix 的输出和梯度通过默认 FP32 容差,TopK 无变化,输出最大绝对差 2.384e-7、参数梯度 2.910e-10。公共 async AG 的 4 个 gather 维度和非连续输入共 32 条 CP8 记录与手工 SUM 梯度零容差一致。NPU/HCCL CP4 中,两种 bank 配置的原时序/新时序对照共 64 条 rank-case,输出、TopK、梯度、参数与 AdamW 状态逐位一致。
本轮复用已有测试及本地分析 driver,未新增仓库测试,未修改公共启动器和日志。没有新的整模型性能结论。
git diff --check、复杂度检查通过;其余变更文件 pylint 通过,公共 collectives.py 的 5 条年份/原生 Torch 导入诊断在未修改基线上同样存在,未通过改规则隐藏。后续 B 才加入 Indexer 任务均衡,C 才加入 Index K 前缀分发;Engram、分页 K、融合通信预处理/后处理及 block 交换不在本 PR。