Repository navigation
Conversation
xuxl1008
force-pushed
the
feat/kda-onesided-p2p-20261104
branch
3 times, most recently
from
November 11, 2026 14:35
3c14bf4 to
b33126e
Compare
xuxl1008
force-pushed
the
feat/kda-onesided-p2p-20261104
branch
from
November 11, 2026 15:02
b33126e to
928ef6a
Compare
…pile docs: add code check agent for dynamic-static code unification
…ontributing-ci-gates chore: clarify contribution workflow and remove standalone lint tests
…-peft-dependency fix: remove unused peft dependency
…-numpy2-support fix: enable NumPy 2 compatibility
…-py312-from-docs docs: use a generic Python environment in contributor guides
xuxl1008
force-pushed
the
feat/kda-onesided-p2p-20261104
branch
from
November 15, 2026 13:25
928ef6a to
48900dc
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
背景与目标
KDA 前向沿序列 rank 传播状态 H,反向逆序传播状态梯度 D。大 CP 下,每跳传输、状态 apply 和交接会累积到串行路径。本 PR 将已验证的四包单边流水整理为显式实验接口,作为私仓的稳定工程里程碑;固定包数仍是 O(CP size),默认路径继续使用 HCCL P2P。
实现与本次整理
KDAOneSidedP2P通过p2p_transport接入现有 KDA Core/Layer,共用 prepare、h/dhu 和完整本地 backward。producer_groups=1等完整 S/M/G 就绪再发布四包;producer_groups=2分两组生产,第一组完成即可接力前两包。H96 时对应 H48 生产组和 H24 通信包。两者显式可选,默认仍为两组;更早首包不等于整体更快。StatePullArena持有对称双槽、每方向 epoch/credit、调用独占 H/D、跨流事件和 autograd lease。每包按 ready → SDMA pull → credit → FP32 matmul/add → ready 接力;真实 reset 的独立出口先发布,再接收另一文档的入口。memory_stats()分开报告外部对称内存和待释放 Torch 负载。源摘要保活到最后 relay 读取完成。dba9829a0ef0f02500a2e60728596a8487686341,保留基线已有 Ascend C 功能和 FP32 state apply 修复。单边入口仍只接受 Triton state P2P。配置、数据流与所有权见 KDA 单边状态流水。
验证
最终代码以未经修改的 master functional HCCL 实现为控制,固定真实投影/ShortConv 后的输入和同一份 do:
[0,2,4,6]、[1,3,5,7]三种配置为完整生产、两组生产、第二组 S/G 限 16-worker;均使用四包。每组覆盖 dense、单长文档、混合及完全 TND reset。测试仍只有一个集中 NPU 入口及其 worker。
CP8/H96/8K 首轮因其他作业占用 5 号卡约 50 GiB 而 OOM;四卡复验也遇到 0 号卡新作业占用,最终 H96/8K 使用空闲 1–4 号卡完成。失败记录保留,不计为通过;本次不声称完成新的 CP8/H96 验收或性能重测。
固定依赖:Torch/torch-npu 2.10.0,完整 CANN 9.1.0-beta.3,Triton-Ascend
3.2.2+gitc22984ba;FLA commit6b6f546f9bd10a97be1e8cf2f0120de70cfca482;SHMEM commitd30702bb55d0c9ce028a241ea608ae38dbaebc8e,使用仓库依赖锁对应的私有 SDK/runtime-symbol 适配。原生模块由本次工作树重新构建,框架使用隔离 editable 安装。已有的 CP3/5、H1、两完整 KDA 层/三微批次/AdamW 更新和 CP8 记录保留为历史资格,不写成本次重新运行。当前检查证明的是相同本地算术下的协议与生命周期一致性,不是高精度参考下的全模型误差证明。
显存与能力边界
H96 每 arena 为 24 MiB 状态区加 17 KiB 信号;默认 192 MiB SDK heap 由兼容 arena 共享。调用独占 H/D、需要时保存的 M 及转置临时区另计。Torch allocator 峰值不包含这份外部 heap;不能将通信分包解释为完整 core 的 head 显存分块。
当前验证针对 A2、BF16 B1、K=V128、BT64、等量 Q/K/V heads 和单 host 提交者。需要相邻 peer 的对称内存可达;解除 CP 规模白名单不代表已验证跨机、超节点或 A5。一进程只支持一个兼容 SDK 域;互不相交 CP 子组可以独立初始化。重叠 SDK 域、Ulysses 接入和故障恢复后置。初始化/关闭为 collective;未完成反向会拒绝关闭,中途异常要求重新启动作业。
原生组件显式构建和持有,尚未进入主 wheel 自动安装或 YAML 自动选型。本次收敛不扩大性能实验;producer 内按包通知与 Dual CP 消费者融合分别作为后续工作,不作为此 PR 的合入前置条件。