GLM-4.5 model accuracy alignment

[zhanghonggeng]

PR type

• Bug Fix
• New Feature
• Document Updates
• More Models or Datasets Support

PR information

This PR improves GLM-4.5 accuracy alignment between ms-swift/Megatron-LM and the PaddleFleet implementation.

Main changes:

• Disable gradient clipping when Megatron accuracy-compatible mode is enabled, avoiding numerical differences introduced by gradient clipping.
• Use FP64 accumulation for masked loss reduction in accuracy-compatible mode, then cast the result back to FP32.
• Add batch dump and replay utilities to the existing Megatron utility module:
  • DUMP_DATA_PATH dumps input_ids and labels for each step and distributed rank.
  • LOAD_FIXED_DATA_PATH reloads the corresponding fixed batch for reproducible cross-framework comparisons.
  • Rebuild position_ids and loss_scale after loading a fixed batch.
• Integrate fixed-batch loading and batch dumping into the Megatron trainer data path.
• Keep the default training behavior unchanged when accuracy-compatible mode and the dump/replay environment variables are not enabled.

Dumped files use the following naming convention:

tokens_step{step}_rank{rank}_seq{seq_len}.npy
labels_step{step}_rank{rank}_seq{seq_len}.npy
Example usage:


export DUMP_DATA_PATH=/path/to/dump
Replay the dumped batches with:


export LOAD_FIXED_DATA_PATH=/path/to/dump