Skip to content

fix: support natural document lengths in V4.1 packed CP compression - #1250

Merged
liuchongming74 merged 1 commit into
mindspore-ai:masterfrom
xuxl1008:fix/v41-natural-document-compression
Oct 30, 2026
Merged

liuchongming74 merged 1 commit into
mindspore-ai:masterfrom
xuxl1008:fix/v41-natural-document-compression

Conversation

@xuxl1008

@xuxl1008 xuxl1008 commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Paired: GitHub #1250 ↔ GitCode !1811

What type of PR is this?

/kind bug

What does this PR do / why do we need it:

Fix the packed sample boundaries must align with the CSA2 compression ratio error when the standard TopK training path for DeepSeek-V4.1 receives packed/TND data containing odd-length documents. Also disable the post-training candidate pool by default in the validation/cropped builders so that the default validation model uses standard causal TopK selection.

The previous implementation derived compressed K coordinates by dividing global token positions by the compression ratio, so simply removing the alignment check would be incorrect. For example, two documents of lengths 3 and 5 should produce 1 and 2 compressed keys respectively at ratio 2, with compressed boundaries [0, 1, 3]. The last token of the first document must not be grouped with the first token of the second document.

Changes:

  • Form complete compression blocks relative to each document's start, producing floor(document_length / ratio) keys. Trailing tokens that do not form a complete block still participate in Q, SWA, and the training loss.
  • Assign each complete compression block to the CP rank owning its last token. For blocks spanning ranks, exchange the required KV/gate projections and RoPE boundary data through the existing differentiable all-to-all-v interface, returning gradients to their original owners.
  • When ranks produce different numbers of compressed keys, pad communication containers to equal capacity, all-gather them, and remove the container padding. Ranks owning no compressed blocks are supported; container padding does not participate in compression or attention.
  • Correct the Indexer's causal bounds, RoPE positions at compression-block starts, and SMLA mappings between raw and compressed coordinates. Preserve the existing Full/Reindex/Reuse sharing relationships.
  • Support fused SWA when the compressed bank is empty, and retain zero-gradient graph connections on ranks with empty prefixes so that all ranks participate correctly in collective backward.
  • Use the current V4.1 RoPE entry point from master in the natural-document compressor, and retain the merged Indexer/SMLA/KL KV-layout fixes.
  • Keep the existing contiguous compression path for aligned documents. Raw/main/Index KV banks continue to use the all-gather implementation on master. Load balancing, SWA halo exchange, bank-prefix dispatch, and communication overlap are outside this PR.

Natural-document compression requires no additional configuration. It uses the existing document boundaries in packed_seq_params; fused and reference implementations share the same document compression plan. CP still requires the packed total length to be evenly divisible by the CP size and a micro-batch size of 1. Individual documents no longer need even lengths in the standard TopK path. The post-training candidate-pool path retains its compression-alignment requirement.

Changes accompanying the data interface introduced in #1188: the three V4.1 VLM examples use sample_alignment_size: 1, removing per-document padding to even lengths. Two older examples also replace the removed sequence_alignment argument and explicitly set training_mode: sft / sample_format: messages. Optional sample padding, the collator's TP×CP alignment of the complete batch, and the separate ignored padding tail are retained. Image-patch/SP alignment constraints are unchanged. Explicit candidate configurations still require compression-aligned data.

The validation/cropped builders add enable_candidate_pool: false as the default: set the candidate source to -1 and candidate TopK blocks to 0, and disable layer-role remapping used only for candidate smoke tests. The default four-layer crop no longer changes a released Reuse layer into Reindex solely for this test. The released Full/Reindex/Reuse relationships in the 40-layer configuration are preserved. Setting enable_candidate_pool: true explicitly restores candidate validation behavior; exercise_post_training_indexer controls candidate smoke-test remapping only when the candidate pool is enabled. This setting does not modify the original released checkpoint configuration or add candidate-pool support to the fused Indexer.

Algorithm references:

  • DeepSeek-V4 report, §3.4.3: independent compression of packed sequences, exchanging required inputs across CP boundaries, and handling different compressed lengths across ranks.
  • Official DeepSeek-V4.1 Compressor/Indexer: produce only complete, non-overlapping compression groups; use the group's first position for RoPE; expose a group to queries only after it is complete.

Which issue(s) this PR fixes:

Fixes #1261

Related to #1143. This PR independently fixes data ingestion for natural document lengths; it does not close or update the original PR.

Test Plan and Test result:

Rebased onto master 96143af9, which includes dataset PR #1188. The upstream Base attention interface and merged KV-stride fixes are preserved.

  • All 42 relevant CPU unit tests passed, covering candidate-pool defaults and explicit enablement, retained layer roles, comparison against the actual compressor, natural-document packing, CP sharding, and ranks owning no compressed blocks.
  • Complete documents from a real DeepSeek mmap dataset, with lengths 53/137/273/161/47 and no truncation, were passed through SamplePacker → OmniCollator → get_batch → V4.1 runtime → compressor. Under CPU/Gloo CP4, both compressed banks and input gradients matched the per-document compressor reference bitwise; parameter gradients passed PyTorch's default tolerances. The natural total length was 671, with one independent padding token added only at the batch tail. This case exercised compression blocks spanning ranks. It did not include image decoding or the complete VLM Trainer.
  • NPU/HCCL CP4: a six-layer fused chain covering SWA, ratio-2 Full/Reuse, and ratio-1 Full/Reindex/Reuse, with unequal document lengths, empty owner ranks, a completely empty ratio-2 bank, and actual sparse TopK selection. Each of four input cases ran two AdamW steps. Compared with a pre-rebase snapshot using the same CP size, inputs, and initialization, all 10,240 checks matched bitwise, with no non-finite values. Checks covered outputs, TopK, input/parameter gradients, updated parameters, and AdamW state. This establishes that the rebase preserved behavior for these cases; it does not establish bitwise equivalence between per-document and packed BF16 calls for every shape.
  • Changed files passed checks using the CI versions of Pylint 3.3.7 and Lizard 1.17.19 with the existing whitelist. git diff --check also passed.

Previously established independent accuracy references:

  • Five NPU/HCCL CP4 compression-boundary routing cases: pooling, input gradients, and block-start RoPE matched with zero tolerance.
  • Per-document and packed native calls use different BF16 kernel shapes and gradient-accumulation groupings. Outputs matched, but parameter gradients were not bitwise identical, and not all passed the default FP32 tolerances. In the latest CP1 reference comparison, outputs matched bitwise in all five cases; 76 of 315 parameter-gradient checks did not pass the default FP32 tolerances, with a maximum relative L2 error of approximately 0.5403%. This result is retained as observed; no additional percentage threshold is introduced to reclassify it as a pass.
  • The earlier CPU/Gloo CP8 boundary and two-step training regressions, and NPU CP8 two-step fused training checks, were completed. The hardware rerun for this rebase used CP4.

This PR makes no new performance claims.

Code review checklist:

  • Necessary CPU unit tests and existing CP system tests cover the relevant boundaries.
  • Checkpoint parameter names, the fused-operator switch, and the model's compression formula are unchanged.
  • No new third-party dependencies; existing differentiable communication interfaces are reused.
  • The validation candidate-pool default and explicit enablement are documented.

@xuxl1008

Copy link
Copy Markdown
Contributor Author

/check-pr

@xuxl1008

Copy link
Copy Markdown
Contributor Author

/retest

@mindspore-ai-bot

Copy link
Copy Markdown

🔵 The pipeline #986 is running. Please wait a moment... (Link)

@mindspore-ai-bot

Copy link
Copy Markdown
Project Name Build_Stage Build Result Details(Full/Incremental) Detail
Coverage Line Coverage ❌ FAILURE 67%/77% Link

@mindspore-ai-bot

Copy link
Copy Markdown
Platform Type Module Excution Time Time Limit
- optimizer 426.181s 300.0s
- pipeline_parallel 879.568s 600.0s
- shard 248.035s 300.0s
- auto_models 138.287s 150.0s
- clip_grad 128.779s 120.0s
- fully_shard 444.283s 600.0s
- accuracy 110.461s 120.0s
- distributed_checkpoint 118.735s 120.0s
- activation_memory 156.2745s 120.0s
- init_weights 81.201s 118.75s
- context_parallel 108.563s 120.0s
- process_group 93.228s 90.0s
- tensor_parallel 293.848s 300.0s
- multicore 116.7795s 120.0s
- dtensor 109.557s 120.0s
- llama3_examples 108.86s 120.0s
- swap_optimizer 112.628s 120.0s
- expert_parallel 179.898s 120.0s

@mindspore-ai-bot

Copy link
Copy Markdown
Project Name Build_Stage Build Result Details
Hyper-parallel_Github_Gate - ❌ FAILURE 986
Smoke_Ascend910b_Torch FAILURE 986

@xuxl1008

Copy link
Copy Markdown
Contributor Author

/retest

@mindspore-ai-bot

Copy link
Copy Markdown

🔵 The pipeline #1207 is running. Please wait a moment... (Link)

@mindspore-ai-bot

Copy link
Copy Markdown
Project Name Build_Stage Build Result Details
Hyper-parallel_Github_Gate - ❌ FAILURE 1207
Check_Lizard FAILURE 1207

@mindspore-ai-bot

Copy link
Copy Markdown

@xuxl1008, your gate pipeline failed at the code check stage. You can configure some code check tools locally following the (pre-push quick guide), to enable automatic checking when you push your code every time. Tools included are: Check_CMakelint, Check_Codespell, Check_Cpplint, Check_Lizard, Check_Pylint, Check_ClangFormat, Check_ShellCheck.

@xuxl1008

Copy link
Copy Markdown
Contributor Author

已修复本次 Check_Lizard 失败:整理两个 validation builder 的参数换行,保留完整说明、接口和逻辑(AST 不变)。按 CI 的 Lizard 1.17.19、-L 100 和原白名单复验,函数长度为 98/100;本 PR 生产文件检查、Pylint 和 8 项相关 UT 通过。此前本地 pre-commit 使用 NLOC 门限,与 CI 的函数 length 门限不同,现已按实际 CI 命令复核。

@xuxl1008

Copy link
Copy Markdown
Contributor Author

/retest

@mindspore-ai-bot

Copy link
Copy Markdown

🔵 The pipeline #1208 is running. Please wait a moment... (Link)

@mindspore-ai-bot

Copy link
Copy Markdown
Project Name Build_Stage Build Result Details(Full/Incremental) Detail
Coverage Line Coverage ✅ SUCCESS 72%/77% Link

@mindspore-ai-bot

Copy link
Copy Markdown
Platform Type Module Excution Time Time Limit
- activation_memory 114.79075s 120.0s
- auto_models 89.815s 150.0s
- shard 162.137s 300.0s
- pipeline_parallel 510.618s 600.0s
- optimizer 222.763s 300.0s
- accuracy 72.075s 120.0s
- swap_optimizer 273.74s 300.0s
- multicore 109.73487s 120.0s
- fully_shard 279.379s 600.0s
- expert_parallel 138.5405s 120.0s
- tensor_parallel 211.49s 300.0s
- context_parallel 69.593s 120.0s
- clip_grad 68.332s 120.0s
- distributed_checkpoint 71.478s 120.0s
- init_weights 67.702s 118.75s
- dtensor 64.337s 120.0s
- process_group 58.693s 90.0s
- llama3_examples 57.254s 120.0s

@mindspore-ai-bot

Copy link
Copy Markdown
Project Name Build_Stage Build Result Details
Hyper-parallel_Github_Gate - ✅ SUCCESS 1208

@xuxl1008
xuxl1008 force-pushed the fix/v41-natural-document-compression branch from 2b98154 to 51c330f Compare October 29, 2026 15:47
@xuxl1008

Copy link
Copy Markdown
Contributor Author

/retest

@mindspore-ai-bot

Copy link
Copy Markdown

🔵 The pipeline #1239 is running. Please wait a moment... (Link)

@mindspore-ai-bot

Copy link
Copy Markdown
Project Name Build_Stage Build Result Details(Full/Incremental) Detail
Coverage Line Coverage ❌ FAILURE 72%/78% Link

@mindspore-ai-bot

Copy link
Copy Markdown
Platform Type Module Excution Time Time Limit
- accuracy 104.069s 120.0s
- pipeline_parallel 697.932s 600.0s
- shard 215.576s 300.0s
- fully_shard 365.629s 600.0s
- activation_memory 124.92563s 120.0s
- auto_models 105.594s 150.0s
- optimizer 329.623s 300.0s
- distributed_checkpoint 87.475s 120.0s
- clip_grad 109.192s 120.0s
- llama3_examples 83.908s 120.0s
- swap_optimizer 352.603s 300.0s
- expert_parallel 184.19662s 120.0s
- multicore 125.88925s 120.0s
- tensor_parallel 284.929s 300.0s
- context_parallel 90.971s 120.0s
- process_group 89.141s 90.0s
- dtensor 89.222s 120.0s
- init_weights 85.457s 118.75s

@mindspore-ai-bot

Copy link
Copy Markdown
Project Name Build_Stage Build Result Details
Hyper-parallel_Github_Gate - ❌ FAILURE 1239
Smoke_Ascend910b_Torch FAILURE 1239

@liuchongming74 liuchongming74 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@liuchongming74
liuchongming74 merged commit d225bd7 into mindspore-ai:master Oct 30, 2026
9 of 14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support natural document lengths in DeepSeek V4.1 packed context parallelism

5 participants