diff --git a/dynamic_shape_development.md b/dynamic_shape_development.md new file mode 100644 index 0000000000..255edf2746 --- /dev/null +++ b/dynamic_shape_development.md @@ -0,0 +1,241 @@ +# HyperParallel 静态图动态 Shape 开发说明 + +本文说明当前联合图的动态输入机制、BaseTrainer 接入和 NPU 文本训练入口。文件链接相对于仓库根目录;链接标注当前源码中的函数名与起始行,便于代码变化后重新定位。按尺寸生成专用 FX 图和缓存分派见 [尺寸特化开发说明](size_specialization_development.md)。 + +动态 shape 允许 batch、sequence 等用户输入维度变化,并复用前向与反向联合 FX 图。 “静态图”指算子拓扑被捕获;图中部分尺寸仍是 `SymInt`,执行对象是并行 Pass 改写后的 `GraphModule`。这项能力不等同于算子融合、NPU Graph capture、任意动态控制流或动态输出尺寸支持。 + +## 1. 配置与最小使用方式 + +标准 TextTrainer / BaseTrainer 使用现有配置入口: + +```yaml +compile: + enabled: true + use_joint_graph: true + dynamic: true +``` + +[CompileConfig:114](hyper_parallel/models/build_options.py#L114) 定义配置, [CompileConfig.selects_graph_compiler:137](hyper_parallel/models/build_options.py#L137) 用 `enabled and use_joint_graph` 选择联合图路径。 [BaseTrainer._build_graph_compiler:545](hyper_parallel/trainer/base.py#L545) 传递完整训练配置、模型和现有 mesh。无需另建训练循环;正常的 `scripts/train_lm.py` 入口即可使用这项功能。 + +| 配置 | 语义 | +| --- | --- | +| `dynamic: false`,无映射 | 默认静态捕获 | +| `dynamic: true`,无映射 | 尝试符号化所有用户张量维度 | +| 提供 `dynamic_arg_dims` | 仅符号化指定维度;映射优先于自动选择,即使 `dynamic: false` 也启用符号捕获 | +| `dynamic_arg_dims: {}` | 使用符号捕获机制,但所有用户维度保持静态 | + +参数和 buffer 始终以静态尺寸捕获,开启自动动态模式不会把模型状态全部符号化。自动模式也受算子与 guard 约束;首次样本中的尺寸 0/1 可能被特化。 + +精确选择时,路径从 `forward_backward(**inputs)` 的顶层关键字开始: + +```yaml +compile: + enabled: true + use_joint_graph: true + dynamic_arg_dims: + model_inputs.input_ids: [1] + labels: [-1] +``` + +实际输入若包含随长度变化的 `attention_mask`、`position_ids`,也须标记对应维度。例如 `[B, 1, L, L]` mask 的后两维与序列长度关联;`[B, L]` mask 则标记末维。映射支持字典键、非负列表下标和属性路径;维度允许负索引,列表路径下标不允许负数。路径必须解析为已注册 pytree 中的 Tensor 叶子,校验见 [normalize_dynamic_arg_dims:29](hyper_parallel/compile/tracer/dynamic_shapes.py#L29) 与 [_resolve_input:46](hyper_parallel/compile/tracer/dynamic_shapes.py#L46)。 + +## 2. 独立编译器与 UT 对照 + +以下小代码说明接口;持续数值回归由 UT 承担: + +```python +import torch +from hyper_parallel.compile import GraphCompiler, PassConfig + + +def train_fn(model, x, y): + return (model(x) - y).square().mean() + + +model = torch.nn.Linear(4, 3) +compiler = GraphCompiler( + model, train_fn, + pass_config=PassConfig(fsdp_enabled=False), + dynamic_arg_dims={"x": [0, 1], "y": [0, 1]}, + device=torch.device("cpu"), +) +optimizer = torch.optim.SGD(model.parameters(), lr=0.01) +for batch, sequence in [(2, 5), (3, 7), (4, 3)]: + loss, loss_dict = compiler.forward_backward( + x=torch.randn(batch, sequence, 4), + y=torch.randn(batch, sequence, 3), + ) + optimizer.step() + optimizer.zero_grad() +``` + +[GraphCompiler.forward_backward:218](hyper_parallel/compile/compiler.py#L218) 首次调用懒捕获,后续执行已有图并累积 `param.grad`。它返回 `(loss, loss_dict)`;梯度已经由联合图生成,调用方负责 optimizer step 和清梯度。 [GraphTrainer.train_step:135](hyper_parallel/compile/trainer.py#L135) 复用相同编译器,首次捕获后初始化其优化器。 + +[test_batch_and_sequence_reuse_graph:65](tests/ut/compile/test_dynamic_shapes.py#L65) 比较多种 shape 的 eager loss 和参数梯度; [test_grad_accumulation_across_shapes:112](tests/ut/compile/test_dynamic_shapes.py#L112) 检查跨 shape 累积; [test_optimizer_updates_match_eager:125](tests/ut/compile/test_dynamic_shapes.py#L125) 检查优化器更新与 eager 对齐。这些断言覆盖了小模型演示所需的数值契约。 + +## 3. 捕获、并行改写与执行调用链 + +```text +CompileConfig → BaseTrainer._build_graph_compiler → GraphCompiler +micro batch → forward_backward_step → model_inputs / labels / token_counts +首次调用 → 参数静态化 + 用户输入符号化 → make_fx 捕获 forward + autograd.grad +PassPipeline → 刷新 InputGuards → 执行联合 FX 图 → 累积 param.grad +后续调用 → 校验输入 → 执行同一图 → BaseTrainer 按训练节奏更新优化器 +``` + +[GraphCompiler.compile:168](hyper_parallel/compile/compiler.py#L168) 组织以下步骤: + +```python +joint_graph = trace_model_graph(self.model, self.train_fn, inputs, **trace_kwargs) +pipeline = PassPipeline.from_config(self.pass_config, self.parallel_plan) +pipeline.run(joint_graph.graph_module, **self._build_pass_kwargs()) +if joint_graph.input_guards is not None: + joint_graph.input_guards.refresh() +self._joint_graph = joint_graph +``` + +节选省略可选尺寸分派初始化;完整实现先校验分派配置,再执行可能修改真实参数的并行 Pass。 [trace_model_graph:508](hyper_parallel/compile/tracer/graph_tracer.py#L508) 分离模型状态和用户输入: + +```text +输入:[参数与 buffer 的扁平序列] + [用户输入 pytree 的扁平序列] +输出:[loss] + [可训练参数梯度] + [loss_dict 的值] +``` + +[_make_fwd_bwd_fn:437](hyper_parallel/compile/tracer/graph_tracer.py#L437) 还原输入、临时替换模型状态,并在训练函数内部调用 `autograd.grad`。 [extract_module_state:183](hyper_parallel/compile/tracer/graph_tracer.py#L183) 对 tied state 去重; [_reparametrize_train_state:201](hyper_parallel/compile/tracer/graph_tracer.py#L201) 在捕获期间恢复共享权重的别名关系。 [run_traced_graph:651](hyper_parallel/compile/tracer/graph_tracer.py#L651) 每次读取当前模型状态,在 `torch.no_grad()` 下执行显式反向算子。 [GraphCompiler._accumulate_grads:526](hyper_parallel/compile/compiler.py#L526) 按状态 FQN 顺序映射梯度,并检查数量,避免错配或截断。 + +## 4. 符号输入与 helper 分工 + +[build_symbolic_inputs:66](hyper_parallel/compile/tracer/dynamic_shapes.py#L66) 建立共享的 FakeTensorMode / ShapeEnv: + +```python +fake_mode = FakeTensorMode(allow_non_fake_inputs=True, shape_env=ShapeEnv(duck_shape=False)) +fake_args = [fake_mode.from_tensor(value, static_shapes=True) for value in state_flat] +context = StatelessSymbolicContext( + dynamic_sizes=[DimDynamic.DYNAMIC if dim in dims else DimDynamic.STATIC + for dim in range(value.ndim)] +) +``` + +用户每一维明确选为动态或静态,例如 `(2, 5, 4)` 可捕获为 `(s0, s1, 4)`。 `duck_shape=False` 避免不相关维度仅因首个样本值相同而产生相等约束;真正的关联由算子产生。同一 Tensor 经多个路径被选中时按对象身份合并动态维度,不修改用户真实张量上的 Dynamo 标记。对应回归见 [test_same_sample_sizes_are_independent:101](tests/ut/compile/test_dynamic_shapes.py#L101) 与 [test_alias_marks_merge_and_alias_changes_fail:209](tests/ut/compile/test_dynamic_shapes.py#L209)。 + +当前 helper 的职责已分开:维度选择和 guard 在 `dynamic_shapes.py`。状态重绑定、输入叶子检查、联合 callable 构造、前后向元数据整理仍是 `graph_tracer.py` 内的独立函数。这种拆分区分捕获基础能力与具体算子兼容,未引入另一套 tracer 或外部 MagiCompiler 依赖。 MagiCompiler 的动态维度选择思路可参考,但本实现入口是 `make_fx + autograd.grad`,单独调用 `mark_dynamic()` 不足以接入它。 + +## 5. 执行前的 shape guards + +直接调用 FX GraphModule 时,Dynamo 不会替它检查 guards。 [InputGuards:115](hyper_parallel/compile/tracer/dynamic_shapes.py#L115) 保存用户输入的元数据、FakeTensor 和 ShapeEnv: + +| 检查 | 保护的契约 | +| --- | --- | +| pytree 结构 | 输入键、嵌套结构和叶子类型保持一致 | +| 元数据与常量 | rank、dtype、device、layout、requires_grad 和 Python 常量保持一致 | +| Tensor 对象复用关系 | 原来同一对象的输入不能变成两个不同对象 | +| ShapeEnv 表达式 | 未标记的静态维度、关联尺寸、stride 和形状分支约束 | + +[InputGuards.refresh:145](hyper_parallel/compile/tracer/dynamic_shapes.py#L145) 使用 `ignore_static=False`,并在 Pass 后重新生成约束。 +[InputGuards.validate:149](hyper_parallel/compile/tracer/dynamic_shapes.py#L149) 在执行图和图内通信前检查输入。 +对象复用检查不是任意 storage 重叠分析;guard 不持有首个样本的真实张量存储。 +模型状态不参与用户 guard,FSDP 可合法地在 Pass 中把完整参数改成分片。 + +例如 `if x.shape[1] > 8` 只捕获首次走过的分支,后续跨边界会报 `ValueError`。 +见 [test_python_shape_branch_is_guarded:145](tests/ut/compile/test_dynamic_shapes.py#L145)。 +guard 失败不会自动重新捕获;尤其在模型已分片后,不能直接拿该状态重新建立完整模型图。 +多卡输入仍需遵守各通信组的共同约束;本地 guard 不提供跨 rank 的失败协商。 + +## 6. token_counts 必须保持运行时输入 + +packing 改变有效 token 数;即使张量 shape 相同,忽略标签分布也会改变 loss 权重。 +若训练 callable 从闭包读取首轮 Python 属性,后续属性更新不能自动变成新的图输入。 +[BaseTrainer.forward_backward_step:714](hyper_parallel/trainer/base.py#L714) 显式传递计数: + +```python +graph_kwargs["token_counts"] = { + "current": self.current_token_counts, + "step": self.step_token_counts, +} +loss, loss_dict = self.graph_compiler.forward_backward( + model_inputs=micro_batch, labels=labels, **graph_kwargs, +) +``` + +[BaseTrainer.train_step:922](hyper_parallel/trainer/base.py#L922) 按 optimizer step 和 micro step 计算计数; +[_graph_train_fn:557](hyper_parallel/trainer/base.py#L557) 将其传给 [postforward:687](hyper_parallel/trainer/base.py#L687)。 +[mean_global_loss:53](hyper_parallel/trainer/runtime/metrics.py#L53) 的捕获路径使用 Tensor collective 和 Tensor 加权公式: + +```python +step_len = differentiable_all_reduce(step_token_counts[token_key], "sum", dp_cp_group) +backward_loss = cur_loss * cur_token_len / step_len * device_mesh.dp_size * device_mesh.cp_size +``` + +这里省略日志值的全局归约;捕获路径不通过 `.item()` 做数据依赖的计数分支。 +计数是数值变化的标量 Tensor,不需要新增动态维度,也不能在尺寸特化时把数值固化。 +[test_basetrainer_token_weights_are_runtime_inputs:354](tests/ut/compile/test_dynamic_shapes.py#L354) 检查计数改变时 loss 和梯度改变且继续复用图。 + +## 7. causal padding decomposition + +常见 causal loss 对 labels 做 `pad(..., (0, 1), value=-100)`,再切片实现标签 shift。 +部分 NPU 原生 `constant_pad_nd` 捕获路径曾将符号长度变成首轮长度,再经 CE 的尺寸关系约束 logits。 +[trace_model_graph:508](hyper_parallel/compile/tracer/graph_tracer.py#L508) 仅在符号捕获时启用 PyTorch 自带 decomposition: + +```python +decompositions = get_decompositions([torch.ops.aten.constant_pad_nd.default]) if symbolic else None +traced_graph = make_fx( + _fwd_bwd_fn, decomposition_table=decompositions, **_MAKE_FX_KWARGS +)(*fake_args) +``` + +展开后继续传播符号长度,保持 pad、shift、交叉熵语义,未放宽 guard。 +[test_causal_label_padding_preserves_dynamic_length:398](tests/ut/compile/test_dynamic_shapes.py#L398) 对照 eager loss 和梯度,并检查变长图复用。 + +## 8. 标准 FP32 随机初始化文本训练 + +保留的真实训练示例使用在线文本数据、TokenBatchLoader 和 dense packed attention。 +[prepare_dynamic_text.main:22](hyper_parallel/compile/examples/automodel_text_graph/prepare_dynamic_text.py#L22) 生成确定性的变长 JSONL; +[动态训练 YAML:1](hyper_parallel/compile/examples/automodel_text_graph/train_online_lm_graph_dynamic_tp2_fsdp2.yaml#L1) 配置 TP2、FSDP2 和六次 optimizer step。 +从仓库根目录执行,模型路径提供结构配置和 tokenizer 资产: + +```bash +python -m hyper_parallel.compile.examples.automodel_text_graph.prepare_dynamic_text +ASCEND_RT_VISIBLE_DEVICES=0,1,2,3 bash \ + hyper_parallel/compile/examples/automodel_text_graph/run.sh \ + hyper_parallel/compile/examples/automodel_text_graph/train_online_lm_graph_dynamic_tp2_fsdp2.yaml \ + --model.pretrained_model_name_or_path=/path/to/Qwen3-0.6B +``` + +[run.sh 使用说明:19](hyper_parallel/compile/examples/automodel_text_graph/run.sh#L19) 接受配置和 CLI 覆盖项,标准入口为 `scripts/train_lm.py`。 +可设置 `PYTHON`、`MASTER_PORT`、`NPROC_PER_NODE`、`OUTPUT_DIR` 等常规 launcher 参数;动态能力本身不需要额外诊断入口或魔法配置。 +YAML 明确使用 `model_init_dtype: float32`、`torch_dtype: float32`、`load_base_model: false`,运行的是随机初始化模型。 +packing 将可变数量的原始文本拼成 `[1, packed_length]`,物理 batch 为 1,变化的是序列维度。 +更换 tokenizer 或数据后,实际长度及其重复频率可能变化;动态 batch 维由 UT 另行覆盖。 + +## 9. 安装差异、并行与精度边界 + +已核对的安装差异集中在 SDPA:`whh_titan` 有 `torch_npu/_init/patches/sdpa_symbolic_patches.py`,SDPA 入口是 Python wrapper,直接调用 `npu_fusion_attention_v3`;`whh` 无该文件,SDPA 入口是 builtin。 +builtin 经 C++ SDPA bridge 时可能把符号尺寸具体化;该安装 wrapper 则直接进入 `npu_fusion_attention_v3` 的符号 fake 路径。 +`symbolic_npu_sdpa` 保留已有 Python wrapper,因此两种安装的捕获路径可能不同,升级后应按实际入口验证。 +这里记录的是已确认的 SDPA 文件和入口差异,环境名称本身不是接口保证。 + +[_patch_engine_backward:114](hyper_parallel/compile/tracer/graph_tracer.py#L114) 与 +[_non_strict_tracing_context:139](hyper_parallel/compile/tracer/graph_tracer.py#L139) 独立探测当前安装的内部 API;缺失 engine patch 时会告警,不能据此保证所有联合图捕获可靠。 +这项探测不构成两个环境全部 tracer / autograd engine 补丁的差异判断。 +升级或替换 torch / torch_npu 后,应先运行下面的 UT,再验证目标模型训练。 + +参数的静态尺寸使 FSDP 参数 all-gather 和梯度 reduce-scatter 可以与动态 activation 协作。 +同一 TP 组需提供匹配的输入和通信形状;不同 DP 分片可在满足各自图约束时处理不同长度。 +[BaseTrainer.step_optimizers_and_schedulers:877](hyper_parallel/trainer/base.py#L877) 仍承担训练期优化器更新。 +当前 BF16 模型配套的独立 FP32 主参数优化器副本可能在首次图 FSDP 分片前按完整尺寸建立,分片同步问题尚未修复。 +完整训练示例因此使用 FP32 参数;小算子的 BF16 能力验证不能推导为该优化器组合已经可用。 +MoE 内部按 token 路由产生的数据依赖 shape 不会因输入维度符号化而自动支持。 +动态 PP 在 [GraphCompiler.compile:168](hyper_parallel/compile/compiler.py#L168) 捕获前显式拒绝;CP、EP、sequence/loss parallel 的其他组合需各自验证。 + +## 10. 当前回归入口与证据范围 + +```bash +python -m pytest tests/ut/compile/test_dynamic_shapes.py \ + tests/ut/compile/test_size_specialization.py -q +``` + +当前 UT 以 CPU 数值对照和 mock distributed 覆盖输入符号化、guards、FSDP 改写、BaseTrainer 计数与尺寸分派。 +历史开发运行还验证过多进程 FSDP、NPU causal loss / attention,以及 TP2 + FSDP2 的随机初始化模型变长训练。 +这些记录是指定模型和安装组合的能力证据,不能当作仍维护的持续 ST 入口,也不表示收敛或吞吐验证。 +完整模型尚不能从有限 loss 和完成 optimizer step 推导出逐参数 eager 数值对齐。 +后续扩展优先检查:变化是否已成为显式图输入、guard 是否覆盖其约束,以及后端算子是否保持符号尺寸。 diff --git a/hyper_parallel/compile/README.md b/hyper_parallel/compile/README.md index 096722889f..3a0fe02349 100644 --- a/hyper_parallel/compile/README.md +++ b/hyper_parallel/compile/README.md @@ -122,6 +122,129 @@ trainer = GraphTrainer( trainer.train(dataloader, max_steps=1000) ``` +## Dynamic input shapes + +Static tracing remains the default. Enable symbolic user inputs with one option: + +```python +compiler = GraphCompiler(model=model, train_fn=train_fn, dynamic=True) +``` + +For precise control, pass `dynamic_arg_dims` to `GraphCompiler` or `GraphTrainer`: + +```python +compiler = GraphCompiler( + model=model, + train_fn=lambda m, x, y: ((m(x) - y) ** 2).mean(), + dynamic_arg_dims={"x": [0, 1], "y": [0, 1]}, +) +# x: [batch, sequence, input_features]; y: [batch, sequence, output_features] +loss, loss_dict = compiler.forward_backward(x=x, y=y) +``` + +The first call captures one symbolic forward/backward graph. Later batches with +compatible batch/sequence lengths reuse it; gradients still accumulate into the +live parameters. `GraphTrainer` exposes the same options and owns optimizer steps. +Numerical and graph-reuse regressions live in +[the CPU unit tests](../../tests/ut/compile/test_dynamic_shapes.py). + +For the existing **BaseTrainer + GraphCompiler** path, configure: + +```yaml +compile: + enabled: true + use_joint_graph: true + dynamic_arg_dims: + model_inputs.input_ids: [0, 1] + labels: [0, 1] +``` + +Add paths for other tensors whose sizes vary, such as +`model_inputs.attention_mask: [0, 1]`. Paths refer to the keyword arguments of +`train_fn`; BaseTrainer supplies `model_inputs` and `labels`. Use `dynamic: true` +instead of the mapping to automatically symbolize all user tensor dimensions. + +- A mapping takes precedence over `dynamic`; unspecified dimensions stay static. + An empty mapping selects no dynamic dimensions. Negative dimension indices and + nested dictionary/list paths (`batch.items.0`) are supported. Registered pytree + objects can use attribute paths. Invalid paths or dimensions raise `ValueError`. +- Parameters and buffers always keep concrete shapes, including with automatic + input symbolization. No Dynamo marks are added to the caller's tensors. +- The tracer uses `ShapeEnv` and per-tensor symbolic contexts directly, because + this path captures through `make_fx`, not Dynamo. Runtime checks enforce traced + size/stride relationships, input metadata, tensor identity aliases and Python + constants before executing the joint graph. Shape-dependent Python branches + are guarded; they do not become arbitrary runtime control flow. +- Use a representative first batch with varying dimensions **greater than 1**. + PyTorch specializes dimensions of size 0/1 and may constrain dimensions used + by operators or Python control flow. Inputs outside these constraints raise + an explanatory error. They are not automatically recompiled: FSDP compilation + mutates live parameter shards, so blindly retracing would be unsafe. +- First-stage scope: symbolic FX execution, loss and parameter gradients, + gradient accumulation and optimizer updates. CPU unit tests cover the + BaseTrainer entry and FSDP graph rewriting with mocked communication. + The [NPU text example](examples/automodel_text_graph/README.md) + exercises dynamic token packing through TP2 + FSDP2 with Qwen3-0.6B. Python scalar arguments remain + constants. Pipeline parallel with dynamic shapes is rejected before tracing. + Compiled-kernel backends, CUDA/NPU graph capture, serialized graph caches and + data-dependent output shapes are not added here. + +Run the focused checks with: + +```bash +python -m pytest tests/ut/compile/test_dynamic_shapes.py -q +``` + +The approach follows MagiCompiler's separation of dynamic and static dimensions +and reuse of a general symbolic graph, adapted to HyperParallel's joint-graph +tracer. It adds no runtime dependency on the separate MagiCompiler checkout. + +## Lazy size specialization + +Optionally generate concrete-size FX variants of the general joint graph: + +```yaml +compile: + enabled: true + use_joint_graph: true + dynamic: true + compile_sizes: [116, 120, 127] + compile_size_input: model_inputs.input_ids + compile_size_dim: 1 + max_specializations: 8 +``` + +`GraphCompiler` and `GraphTrainer` also accept these four options directly. +`compile_sizes` is opt-in and requires dynamic mode. If `compile_size_input` is +omitted, selection uses the first symbolic axis among flattened user inputs; +`compile_size_dim` only applies to an explicit path and supports negative axes. + +The first execution uses the general graph, matching MagiCompiler's warmup +policy. Subsequent configured sizes lazily generate a variant and execute it. +Repeated signatures hit the cache; unconfigured sizes use the general graph. +The cache key includes all user tensor shapes, strides, offsets and metadata, +so another batch dimension at the same sequence length creates a separate +variant. New signatures at `max_specializations` capacity use the general graph. + +General guards run before every dispatch, including cache hits. Guard failures +still raise errors; specialization does not expand the valid input domain. +Variants bind pure symbolic shape expressions and regenerate FX code from the +post-pass graph, without recapturing the model or executing tensor/communication +operations during generation. Parameters, token counts and RNG remain live. +This is FX specialization, not an Inductor or NPU kernel compilation backend; +unresolved expressions remain dynamic and no speedup is promised. + +Inspect `compiler.specialization_stats` for compilations, cache hits, general and +specialized calls, folded nodes and capacity fallbacks. The snapshot counts the +current compiler instance; variants are not serialized across processes. + +```bash +python -m pytest tests/ut/compile/test_size_specialization.py -q +``` + +See [the implementation walkthrough](../../size_specialization_development.md) +and the [NPU training example](examples/automodel_text_graph/README.md). + ## Key Design Decisions 1. **Static Inputs**: Parameters are graph inputs, not `get_attr`. This allows passes to split the graph by reshaping placeholders. diff --git a/hyper_parallel/compile/compiler.py b/hyper_parallel/compile/compiler.py index 485ebc086a..8a11f37b9e 100644 --- a/hyper_parallel/compile/compiler.py +++ b/hyper_parallel/compile/compiler.py @@ -42,6 +42,8 @@ class and owns them. from .pass_config import PassConfig, normalize_dp_mode, build_pass_config_from_trainer_config from .graph_parallel_plan import GraphParallelPlan from .passes.pipeline import PassPipeline +from .size_specialization import SizeSpecializationDispatcher, validate_compile_sizes +from .tracer.dynamic_shapes import DynamicArgDims, normalize_dynamic_arg_dims from .tracer.graph_tracer import run_traced_graph, trace_model_graph _LOG = logging.getLogger(__name__) @@ -65,6 +67,12 @@ def __init__( trainer_config: Optional[Any] = None, device: Optional[torch.device] = None, mesh_context: Optional[Any] = None, + dynamic: Optional[bool] = None, + dynamic_arg_dims: DynamicArgDims | None = None, + compile_sizes: Optional[List[int]] = None, + compile_size_input: Optional[str] = None, + compile_size_dim: Optional[int] = None, + max_specializations: Optional[int] = None, ) -> None: """ Args: @@ -85,7 +93,35 @@ def __init__( only the FSDP shard sub-mesh is registered under ``"fsdp"``. Use this to feed an automodel TP-sharded model into the graph-mode FSDP pass. + dynamic: Symbolize user input dimensions; inherits trainer_config.compile.dynamic when omitted. + dynamic_arg_dims: Optional dotted input paths to dynamic dimensions, e.g. {"x": [0, 1]}. + Overrides automatic dimension selection and inherits trainer configuration when omitted. + compile_sizes: Opt-in sizes for lazy FX specialization; inherits trainer configuration. + compile_size_input: Dotted tensor path for dispatch; defaults to the first symbolic input axis. + compile_size_dim: Dispatch axis for an explicit path (default -1). + max_specializations: Maximum cached full input signatures (default 8). """ + compile_config = getattr(trainer_config, "compile", None) + self.dynamic = getattr(compile_config, "dynamic", False) if dynamic is None else dynamic + if not isinstance(self.dynamic, bool): + raise ValueError("dynamic must be a bool") + if dynamic_arg_dims is None: + dynamic_arg_dims = getattr(compile_config, "dynamic_arg_dims", None) + self.dynamic_arg_dims = normalize_dynamic_arg_dims(dynamic_arg_dims) + if compile_sizes is None: + compile_sizes = getattr(compile_config, "compile_sizes", None) + self.compile_size_input = ( + getattr(compile_config, "compile_size_input", None) if compile_size_input is None else compile_size_input + ) + self.compile_size_dim = ( + getattr(compile_config, "compile_size_dim", -1) if compile_size_dim is None else compile_size_dim + ) + self.max_specializations = ( + getattr(compile_config, "max_specializations", 8) if max_specializations is None else max_specializations + ) + self.compile_sizes = validate_compile_sizes(compile_sizes, self.max_specializations) + if self.compile_sizes and not (self.dynamic or self.dynamic_arg_dims is not None): + raise ValueError("compile_sizes requires dynamic=True or dynamic_arg_dims") self.model = model self.train_fn = train_fn self.parallel_plan = parallel_plan @@ -103,6 +139,7 @@ def __init__( self.pass_config.validate() self._joint_graph = None + self._size_dispatcher = None self._pytree_pre_hook: Optional[Callable[[], None]] = None @staticmethod @@ -123,6 +160,11 @@ def is_compiled(self) -> bool: """Whether a joint graph has already been compiled.""" return self._joint_graph is not None + @property + def specialization_stats(self) -> dict[str, Any]: + """Return lazy size-specialization counters, or an empty mapping when disabled.""" + return self._size_dispatcher.stats if self._size_dispatcher is not None else {} + def compile(self, **inputs: Any) -> None: """ Compile model into parallel graph @@ -134,6 +176,10 @@ def compile(self, **inputs: Any) -> None: **inputs: Model inputs, forwarded to ``train_fn`` as keyword arguments and used to trace the joint graph """ + if (self.dynamic or self.dynamic_arg_dims is not None) and self.pass_config.pp_enabled: + raise ValueError( + "Dynamic shapes with pipeline parallel are not supported yet; disable PP or dynamic shapes" + ) if self._pytree_pre_hook is not None: self._pytree_pre_hook() @@ -144,7 +190,17 @@ def compile(self, **inputs: Any) -> None: # runs as plain graph mode without sharding. self._init_device_mesh(self._mesh_context) - joint_graph = trace_model_graph(self.model, self.train_fn, inputs) + trace_kwargs = {} + if self.dynamic or self.dynamic_arg_dims is not None: + trace_kwargs = {"dynamic": self.dynamic, "dynamic_arg_dims": self.dynamic_arg_dims} + joint_graph = trace_model_graph(self.model, self.train_fn, inputs, **trace_kwargs) + # Validate selection before a parallel pass mutates the live model. + size_dispatcher = None + if self.compile_sizes: + size_dispatcher = SizeSpecializationDispatcher( + joint_graph, inputs, self.compile_sizes, self.compile_size_input, + self.compile_size_dim, self.max_specializations, + ) pipeline = PassPipeline.from_config(self.pass_config, self.parallel_plan) @@ -154,7 +210,10 @@ def compile(self, **inputs: Any) -> None: # transformed graph lives on ``joint_graph`` for ``forward_backward``. pipeline.run(joint_graph.graph_module, **pass_kwargs) + if joint_graph.input_guards is not None: + joint_graph.input_guards.refresh() self._joint_graph = joint_graph + self._size_dispatcher = size_dispatcher def forward_backward(self, **inputs: Any) -> Any: """ @@ -461,6 +520,7 @@ def _run_graph(self, **inputs): self._joint_graph, self.model, inputs, + **({"graph_dispatcher": self._size_dispatcher} if self._size_dispatcher is not None else {}), ) def _accumulate_grads(self, grads: List[torch.Tensor]) -> None: diff --git a/hyper_parallel/compile/examples/automodel_text_graph/README.md b/hyper_parallel/compile/examples/automodel_text_graph/README.md new file mode 100644 index 0000000000..ab1f182701 --- /dev/null +++ b/hyper_parallel/compile/examples/automodel_text_graph/README.md @@ -0,0 +1,71 @@ +# Dynamic token batching with graph-mode text training + +This example runs the standard TextTrainer + BaseTrainer + GraphCompiler path +with TP2 + FSDP2 on four NPUs. It uses an online iterable dataset, +TokenBatchLoader and dense packed attention. The model directory provides the +model configuration and tokenizer; the text generator needs no downloaded dataset. + +## Run ordinary training + +From the repository root in a torch + torch_npu environment: + +```bash +python -m hyper_parallel.compile.examples.automodel_text_graph.prepare_dynamic_text + +ASCEND_RT_VISIBLE_DEVICES=0,1,2,3 bash \ + hyper_parallel/compile/examples/automodel_text_graph/run.sh \ + hyper_parallel/compile/examples/automodel_text_graph/train_online_lm_graph_dynamic_tp2_fsdp2.yaml \ + --model.pretrained_model_name_or_path=/path/to/Qwen3-0.6B +``` + +The YAML enables `compile.dynamic: true`; manual dimension annotations are +optional. Packing combines variable numbers of records into `[1, packed_length]`, +so the sequence dimension changes while the physical batch dimension remains one. +The iterable dataset avoids limiting raw records before token packing. +Replace `dataset.data_path` to use another plaintext source. + +This functional smoke run uses random initialization (`model.load_base_model: false`) +and FP32 model parameters. Graph FSDP shards parameters on first capture; +separate optimizer master copies constructed earlier can retain the full shape. +That optimizer combination is not exercised by this example. + +`run.sh` defaults to `scripts/train_lm.py`. It accepts a YAML path followed by +existing CLI overrides. Set `PYTHON=/path/to/venv/bin/python` to select the +interpreter, and use `MASTER_PORT`, `NPROC_PER_NODE` and `OUTPUT_DIR` for launcher +settings. The default YAML remains the indexed/static example. The output is a +normal training log; this entry does not export a separate per-rank audit JSON. + +## Optional size specialization + +Add these overrides to the dynamic training command: + +```bash + '--compile.compile_sizes=[116,120,127]' \ + --compile.compile_size_input=model_inputs.input_ids \ + --compile.compile_size_dim=1 +``` + +The first execution uses the general graph. Later configured sizes generate an +FX variant once per complete input metadata signature and reuse it. Other sizes, +or new signatures at the cache limit, continue using the general graph. +`compile.max_specializations` defaults to 8. Omit `compile_sizes` to disable variants. +A different tokenizer or dataset can change which packed lengths recur. + +Generation binds shape expressions in the post-pass FX graph. It does not +recapture the model, shard parameters again, execute collectives during generation +or invoke an Inductor/NPU kernel compiler. Tensor values, token counts and model +weights remain runtime inputs. This is a functionality example, not a throughput +or convergence benchmark. + +## Regression tests and implementation notes + +Small-model shape/gradient and dispatch checks live in the existing CPU UT: + +```bash +python -m pytest tests/ut/compile/test_dynamic_shapes.py \ + tests/ut/compile/test_size_specialization.py -q +``` + +These tests check eager loss/gradient parity, graph reuse, guards and multiple hot +sizes. CPU UT and mocked communication do not substitute for real NPU training. +Dynamic PP and arbitrary data-dependent shapes remain outside the supported path. diff --git a/hyper_parallel/compile/examples/automodel_text_graph/prepare_dynamic_text.py b/hyper_parallel/compile/examples/automodel_text_graph/prepare_dynamic_text.py new file mode 100644 index 0000000000..c2446c6d4e --- /dev/null +++ b/hyper_parallel/compile/examples/automodel_text_graph/prepare_dynamic_text.py @@ -0,0 +1,42 @@ +# Copyright 2026 Huawei Technologies Co., Ltd +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# ============================================================================ +"""Prepare deterministic plaintext records for dynamic token-budget batching.""" + +import argparse +import json +from pathlib import Path + + +def main() -> None: + """Write locally generated text of several lengths; no download is required.""" + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--output", default="output/automodel_text_graph/dynamic_text.jsonl") + args = parser.parse_args() + output = Path(args.output) + output.parent.mkdir(parents=True, exist_ok=True) + lengths = (3, 7, 11, 17, 23, 31, 43, 59, 73, 89, 101, 113) + records = [ + {"text": "Training example " + str(index) + ": " + " ".join( + ["language", "models", "learn", "from", "text"][token % 5] + for token in range(lengths[index % len(lengths)]) + )} + for index in range(384) + ] + output.write_text("".join(json.dumps(record) + "\n" for record in records), encoding="utf-8") + print(f"Wrote {len(records)} variable-length records to {output}") + + +if __name__ == "__main__": + main() diff --git a/hyper_parallel/compile/examples/automodel_text_graph/run.sh b/hyper_parallel/compile/examples/automodel_text_graph/run.sh index bfbf622298..c93c786373 100755 --- a/hyper_parallel/compile/examples/automodel_text_graph/run.sh +++ b/hyper_parallel/compile/examples/automodel_text_graph/run.sh @@ -14,10 +14,10 @@ # limitations under the License. # ============================================================================ -# GraphTextTrainer text demo launcher — default fixed-shape TP2 + FSDP2 case. +# TextTrainer joint-graph launcher (TP2 + FSDP2 by default). # # Usage: -# bash run.sh +# bash run.sh [config.yaml] [--config.key=value ...] # set -euo pipefail @@ -25,15 +25,16 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" HYPER_PARALLEL_ROOT="$(cd "${SCRIPT_DIR}/../../.." && pwd)" REPO_ROOT="$(cd "${HYPER_PARALLEL_ROOT}/.." && pwd)" -CONFIG="${SCRIPT_DIR}/train_lm_graph_fixed_tp2_fsdp2.yaml" -NPROC_PER_NODE=4 -NNODES=1 -NODE_RANK=0 -MASTER_ADDR="127.0.0.1" -MASTER_PORT=29683 -LABEL="train_lm_graph_fixed_tp2_fsdp2" -OUTPUT_DIR="${SCRIPT_DIR}/output" -TRAIN_ENTRY="${REPO_ROOT}/scripts/train_lm.py" +CONFIG="${1:-${CONFIG:-${SCRIPT_DIR}/train_lm_graph_tp2_fsdp2.yaml}}" +if [[ $# -gt 0 ]]; then shift; fi +NPROC_PER_NODE="${NPROC_PER_NODE:-4}" +NNODES="${NNODES:-1}" +NODE_RANK="${NODE_RANK:-0}" +MASTER_ADDR="${MASTER_ADDR:-127.0.0.1}" +MASTER_PORT="${MASTER_PORT:-29683}" +LABEL="$(basename "${CONFIG}" .yaml)" +OUTPUT_DIR="${OUTPUT_DIR:-${SCRIPT_DIR}/output}" +TRAIN_ENTRY="${TRAIN_ENTRY:-${REPO_ROOT}/scripts/train_lm.py}" mkdir -p "${OUTPUT_DIR}" @@ -48,7 +49,7 @@ echo "master : ${MASTER_ADDR}:${MASTER_PORT}" echo "output_dir : ${OUTPUT_DIR}" echo "==========================================================" -torchrun \ +"${PYTHON:-python}" -m torch.distributed.run \ --nproc_per_node="${NPROC_PER_NODE}" \ --nnodes="${NNODES}" \ --node_rank="${NODE_RANK}" \ @@ -56,7 +57,7 @@ torchrun \ --master_port="${MASTER_PORT}" \ --tee=3 \ --local-ranks-filter=0 \ - "${TRAIN_ENTRY}" "${CONFIG}" \ + "${TRAIN_ENTRY}" "${CONFIG}" "$@" \ 2>&1 | tee "${OUTPUT_DIR}/run_${LABEL}.log" echo "==========================================================" diff --git a/hyper_parallel/compile/examples/automodel_text_graph/train_online_lm_graph_dynamic_tp2_fsdp2.yaml b/hyper_parallel/compile/examples/automodel_text_graph/train_online_lm_graph_dynamic_tp2_fsdp2.yaml new file mode 100644 index 0000000000..5a80fae279 --- /dev/null +++ b/hyper_parallel/compile/examples/automodel_text_graph/train_online_lm_graph_dynamic_tp2_fsdp2.yaml @@ -0,0 +1,80 @@ +# Online dynamic packing: generate data with prepare_dynamic_text.py, then pass the model path to run.sh. +# Graph FSDP shards on first capture; use fp32 parameters without separate optimizer master copies. +model_init_dtype: float32 +model: + _target_: hyper_parallel.models.HyperAutoModelForCausalLM.from_pretrained + # Random initialization keeps this functional smoke test independent of weight loading. + load_base_model: false + torch_dtype: float32 + attn_implementation: sdpa + force_hf: true +optimizer: + _target_: hyper_parallel.components.optim.builders.AdamW + adamw_config: + lr: 0.0001 + adamw_weight_decay: 0.0 +lr_scheduler: + _target_: hyper_parallel.components.optim.lr_scheduler.MultiLRScheduler + lr_decay_style: constant + lr_config: + lr: 0.0001 + lr_warmup_steps: 0 +training: + train_iters: 6 + global_batch_size: 4 + micro_batch_size: 1 + backend: hccl + init_device: meta + logging_steps: 1 + seed: 42 + enable_full_determinism: false +accelerator: + tp_size: 2 + cp_size: 1 + ep_size: 1 + pp_size: 1 + sequence_parallel: false + loss_parallel: false +fsdp_config: + dp_shard_size: 2 +compile: + enabled: true + use_joint_graph: true + dynamic: true + # Optional lazy FX specialization (uncomment all three to enable): + # compile_sizes: [116, 120, 127] + # compile_size_input: model_inputs.input_ids + # compile_size_dim: 1 +dataset: + # Streaming avoids truncating raw records before dynamic token packing. + _target_: hyper_parallel.data.text.build_dataset.build_online_iterable_dataset + data_path: ./output/automodel_text_graph/dynamic_text.jsonl + model_assets: + tokenizer: + _target_: hyper_parallel.data.text.build_tokenizer.AutoTokenizer.from_pretrained + use_fast: true + trust_remote_code: true + data_transform: + _target_: hyper_parallel.data.text.text_transform.build_text_transform + data_type: plaintext + text_keys: text + max_seq_len: 128 + data_config: + labels_are_shifted: true + create_attention_mask_in_dataloader: true +dataloader: + _target_: hyper_parallel.data.batching.TokenBatchLoader + min_buffered_samples: 8 + drop_last: true + use_background_prefetcher: false + num_workers: 0 + pin_memory: false + prefetch_factor: null + collate_fn: + _target_: hyper_parallel.data.batching.build_online_text_collate_fn + get_batch: + _target_: hyper_parallel.data.batching.TextParallelBatch + source_type: online + attention_mode: dense +checkpoint: + save_ckpt: false diff --git a/hyper_parallel/compile/size_specialization.py b/hyper_parallel/compile/size_specialization.py new file mode 100644 index 0000000000..30dc913cb6 --- /dev/null +++ b/hyper_parallel/compile/size_specialization.py @@ -0,0 +1,184 @@ +# Copyright 2026 Huawei Technologies Co., Ltd +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# ============================================================================ +"""Lazy FX shape specialization of an already transformed joint graph.""" + +import logging +import operator +from dataclasses import dataclass +from typing import Any + +import torch +from torch import fx +from torch.utils import _pytree as pytree + +from .tracer.dynamic_shapes import _resolve_input +from .tracer.graph_tracer import JointGraph + + +_LOG = logging.getLogger(__name__) +_SHAPE_OPS = { + torch.ops.aten.sym_size.int, torch.ops.aten.sym_stride.int, + torch.ops.aten.sym_numel.default, torch.ops.aten.sym_storage_offset.default, +} +_SCALAR_OPS = {operator.add, operator.sub, operator.mul, operator.floordiv, operator.mod, operator.neg} + + +def validate_compile_sizes(sizes: list[int] | None, max_specializations: int) -> tuple[int, ...]: + """Validate the bounded, opt-in specialization policy.""" + if sizes is not None and ( + not isinstance(sizes, (list, tuple)) or any(type(size) is not int or size < 0 for size in sizes) + ): + raise ValueError("compile_sizes must be a list of non-negative integers or None") + if type(max_specializations) is not int or max_specializations < 1: + raise ValueError("max_specializations must be a positive integer") + return tuple(dict.fromkeys(sizes or ())) + + +def specialize_graph(joint_graph: JointGraph, user_inputs: list[Any]) -> tuple[fx.GraphModule, int]: + """Generate FX code with bound shape expressions, without executing tensor ops. + + The caller must validate the general graph guards first. Only pure shape + queries and integer arithmetic are folded; weights, token counts, RNG and + collectives remain runtime operations. No model capture or parallel pass is + repeated, so an already sharded live model is never sharded again. + """ + guards = joint_graph.input_guards + tensors = [value for value in user_inputs if isinstance(value, torch.Tensor)] + bindings = guards.shape_env.bind_symbols(guards.fake_tensors, tensors) + graph = fx.Graph() + replacements = {} + folded = 0 + for node in joint_graph.graph_module.graph.nodes: + value = node.meta.get("val") + if node.op == "call_function" and node.target in _SHAPE_OPS | _SCALAR_OPS: + if isinstance(value, torch.SymInt): + concrete = value.node.expr.xreplace(bindings) + if isinstance(concrete, int) or (not concrete.free_symbols and concrete.is_Integer): + replacements[node] = int(concrete) + folded += 1 + continue + copied = graph.node_copy(node, lambda arg: replacements[arg]) + # Symbolic tensor metadata would misdescribe this concrete variant. + copied.meta = {key: val for key, val in node.meta.items() if key not in ("val", "tensor_meta", "example_value")} + replacements[node] = copied + graph.lint() + return fx.GraphModule(joint_graph.graph_module, graph), folded + + +@dataclass +class ConcreteSizeEntry: + """One generated variant for a full input metadata signature.""" + + runtime_size: int + graph_module: fx.GraphModule + folded_nodes: int + + +class SizeSpecializationDispatcher: + """Use a general graph once, then lazily generate configured size variants. + + Selection uses one size, while caching includes every input tensor's shape, + stride and storage offset. This prevents reuse across incompatible secondary + dimensions. Capacity is bounded; new signatures at capacity use the general + graph. All dispatch decisions occur after the general guards have passed. + """ + + def __init__( + self, joint_graph: JointGraph, inputs: dict[str, Any], compile_sizes: tuple[int, ...], + compile_size_input: str | None, compile_size_dim: int, max_specializations: int, + ) -> None: + self.joint_graph = joint_graph + self.compile_sizes = frozenset(compile_sizes) + self.max_specializations = max_specializations + self.entries: dict[tuple, ConcreteSizeEntry] = {} + self.general_calls = 0 + self.specialized_calls = 0 + self.cache_hits = 0 + self.capacity_fallbacks = 0 + self.last_dispatch = "uninitialized" + self.num_state_inputs = len(joint_graph.state_fqns) + user_flat, _ = pytree.tree_flatten(inputs) + self.selector = self._resolve_selector(user_flat, inputs, compile_size_input, compile_size_dim) + + def _resolve_selector(self, user_flat, inputs, path, dim): + if type(dim) is not int: + raise ValueError("compile_size_dim must be an integer") + if path is not None: + if not isinstance(path, str) or not path or any(not part for part in path.split(".")): + raise ValueError("compile_size_input must be a non-empty dotted input path") + tensor = _resolve_input(inputs, path) + if not -tensor.ndim <= dim < tensor.ndim: + raise ValueError("compile_size_dim is out of range for compile_size_input") + for index, value in enumerate(user_flat): + if value is tensor: + return index, dim % tensor.ndim + raise ValueError("compile_size_input must resolve to a registered pytree tensor leaf") + fake_users = self.joint_graph.example_inputs[self.num_state_inputs:] + for index, value in enumerate(fake_users): + if isinstance(value, torch.Tensor): + for axis, size in enumerate(value.shape): + if isinstance(size, torch.SymInt) and size.node.expr.free_symbols: + return index, axis + raise ValueError("compile_sizes needs a symbolic input dimension or an explicit compile_size_input") + + @staticmethod + def _signature(user_inputs: list[Any]) -> tuple: + return tuple( + (tuple(value.shape), value.stride(), value.storage_offset(), value.dtype, value.device, + value.layout, value.requires_grad) + if isinstance(value, torch.Tensor) else (type(value), value) + for value in user_inputs + ) + + @property + def stats(self) -> dict[str, Any]: + """Return a snapshot suitable for logging and JSON audit files.""" + return { + "general_calls": self.general_calls, "specialized_calls": self.specialized_calls, + "cache_hits": self.cache_hits, "compilations": len(self.entries), + "cached_variants": len(self.entries), "capacity_fallbacks": self.capacity_fallbacks, + "compiled_sizes": sorted({entry.runtime_size for entry in self.entries.values()}), + "folded_nodes": sum(entry.folded_nodes for entry in self.entries.values()), + "last_dispatch": self.last_dispatch, + } + + def __call__(self, *flat_inputs: Any) -> Any: + """Execute the selected graph with live state and live tensor values.""" + user_inputs = list(flat_inputs[self.num_state_inputs:]) + index, dim = self.selector + size = user_inputs[index].shape[dim] + if self.general_calls == 0 or size not in self.compile_sizes: + return self._run_general(flat_inputs) + key = self._signature(user_inputs) + entry = self.entries.get(key) + if entry is None: + if len(self.entries) >= self.max_specializations: + self.capacity_fallbacks += 1 + return self._run_general(flat_inputs) + graph_module, folded = specialize_graph(self.joint_graph, user_inputs) + entry = ConcreteSizeEntry(size, graph_module, folded) + self.entries[key] = entry + _LOG.info("Generated size-specialized FX graph: size=%d, folded_nodes=%d, variants=%d", + size, folded, len(self.entries)) + else: + self.cache_hits += 1 + self.specialized_calls += 1 + self.last_dispatch = "specialized" + return entry.graph_module(*flat_inputs) + + def _run_general(self, flat_inputs): + self.general_calls += 1 + self.last_dispatch = "general" + return self.joint_graph.graph_module(*flat_inputs) diff --git a/hyper_parallel/compile/tracer/dynamic_shapes.py b/hyper_parallel/compile/tracer/dynamic_shapes.py new file mode 100644 index 0000000000..aead00dbec --- /dev/null +++ b/hyper_parallel/compile/tracer/dynamic_shapes.py @@ -0,0 +1,161 @@ +# Copyright 2026 Huawei Technologies Co., Ltd +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# ============================================================================ +"""Selective symbolic inputs and runtime guards for make_fx joint graphs.""" + +from typing import Any + +import torch +from torch._dynamo.source import LocalSource +from torch._subclasses import FakeTensorMode +from torch.fx.experimental.symbolic_shapes import DimDynamic, ShapeEnv, StatelessSymbolicContext +from torch.utils import _pytree as pytree + + +DynamicArgDims = dict[str, int | list[int]] + + +def normalize_dynamic_arg_dims(dynamic_arg_dims: DynamicArgDims | None) -> dict[str, list[int]] | None: + """Validate and copy the public path-to-dimensions mapping.""" + if dynamic_arg_dims is None: + return None + if not isinstance(dynamic_arg_dims, dict): + raise ValueError("dynamic_arg_dims must be a mapping of input paths to dimensions") + normalized = {} + for path, dims in dynamic_arg_dims.items(): + if not isinstance(path, str) or not path or any(not part for part in path.split(".")): + raise ValueError("dynamic_arg_dims keys must be non-empty dotted input paths") + dims = [dims] if isinstance(dims, int) else dims + if not isinstance(dims, (list, tuple)) or any(type(dim) is not int for dim in dims): + raise ValueError(f"dynamic_arg_dims[{path!r}] must be an int or a list of ints") + normalized[path] = list(dims) + return normalized + + +def _resolve_input(inputs: dict[str, Any], path: str) -> torch.Tensor: + value = inputs + try: + for part in path.split("."): + if isinstance(value, dict): + value = value[part] + elif isinstance(value, (list, tuple)): + index = int(part) + if index < 0: + raise IndexError(index) + value = value[index] + else: + value = getattr(value, part) + except (KeyError, IndexError, ValueError, AttributeError) as exc: + raise ValueError(f"dynamic_arg_dims input path {path!r} does not exist") from exc + if not isinstance(value, torch.Tensor): + raise ValueError(f"dynamic_arg_dims input {path!r} must be a tensor, got {type(value).__name__}") + return value + + +def build_symbolic_inputs( + state_flat: list[Any], + inputs: dict[str, Any], + dynamic_arg_dims: DynamicArgDims | None, +) -> tuple[FakeTensorMode, tuple[Any, ...]]: + """Fakeify state statically and selected user dimensions symbolically. + + Args: + state_flat: Flat model parameters and buffers. + inputs: Keyword inputs, including registered pytree containers. + dynamic_arg_dims: Explicit paths/dimensions; None symbolizes all user tensor dimensions. + + Returns: + Fake mode and flattened state/user inputs sharing its ShapeEnv. + """ + mapping = normalize_dynamic_arg_dims(dynamic_arg_dims) + user_flat, _ = pytree.tree_flatten(inputs) + tensor_ids = {id(value) for value in user_flat if isinstance(value, torch.Tensor)} + dynamic_dims = {} + if mapping is None: + dynamic_dims = {id(value): set(range(value.ndim)) for value in user_flat if isinstance(value, torch.Tensor)} + else: + for path, dims in mapping.items(): + tensor = _resolve_input(inputs, path) + if id(tensor) not in tensor_ids: + raise ValueError(f"dynamic_arg_dims input {path!r} must be a registered pytree tensor leaf") + selected = dynamic_dims.setdefault(id(tensor), set()) + for dim in dims: + if not -tensor.ndim <= dim < tensor.ndim: + raise ValueError(f"dynamic_arg_dims[{path!r}] dimension {dim} is invalid for rank {tensor.ndim}") + selected.add(dim % tensor.ndim) + + # Equal sample sizes do not imply equality across future batches. + fake_mode = FakeTensorMode(allow_non_fake_inputs=True, shape_env=ShapeEnv(duck_shape=False)) + fake_args = [fake_mode.from_tensor(value, static_shapes=True) for value in state_flat] + for index, value in enumerate(user_flat): + if not isinstance(value, torch.Tensor): + fake_args.append(value) + continue + if value.layout != torch.strided: + raise ValueError("Dynamic graph inputs must use torch.strided layout") + dims = dynamic_dims.get(id(value), set()) + context = StatelessSymbolicContext( + dynamic_sizes=[DimDynamic.DYNAMIC if dim in dims else DimDynamic.STATIC for dim in range(value.ndim)] + ) + fake_args.append(fake_mode.from_tensor(value, source=LocalSource(f"input_{index}"), symbolic_context=context)) + return fake_mode, tuple(fake_args) + + +class InputGuards: + """Check input metadata, constants and traced shape branches before execution. + + The graph is called directly, so Dynamo will not check its guards for us. + Only user tensors participate: FSDP may legitimately shard live model state. + No real input tensors or storage are retained here. + """ + + def __init__(self, inputs: list[Any], fake_inputs: tuple[Any, ...], shape_env: ShapeEnv) -> None: + self.shape_env = shape_env + self.fake_tensors = [value for value in fake_inputs if isinstance(value, torch.Tensor)] + self.leaves = [self._metadata(value) for value in inputs] + self.aliases = self._aliases(inputs) + self.expression = None + self.refresh() + + @staticmethod + def _metadata(value: Any) -> tuple: + if isinstance(value, torch.Tensor): + return (torch.Tensor, value.ndim, value.dtype, value.device, value.layout, value.requires_grad) + return (type(value), value) + + @staticmethod + def _aliases(inputs: list[Any]) -> list[int]: + seen = {} + return [ + seen.setdefault(id(value), index) + for index, value in enumerate(inputs) if isinstance(value, torch.Tensor) + ] + + def refresh(self) -> None: + """Include constraints introduced by tracing and graph passes.""" + self.expression = self.shape_env.produce_guards_expression(self.fake_tensors, ignore_static=False) + + def validate(self, inputs: list[Any]) -> None: + """Reject incompatible inputs before any graph or collective is executed.""" + if [self._metadata(value) for value in inputs] != self.leaves: + raise ValueError("Graph input rank, dtype, device, layout, requires_grad or Python constant changed") + if self._aliases(inputs) != self.aliases: + raise ValueError("Graph input tensor aliasing changed since compilation") + tensors = [value for value in inputs if isinstance(value, torch.Tensor)] + if self.expression and not self.shape_env.evaluate_guards_expression(self.expression, tensors): + raise ValueError( + "Graph input shape/stride violates the traced shape guards. Check unmarked static dimensions, " + "related input sizes and shape-dependent branches. Compile with representative dynamic sizes > 1; " + "sizes 0/1 and Python shape branches may specialize the graph." + ) diff --git a/hyper_parallel/compile/tracer/graph_tracer.py b/hyper_parallel/compile/tracer/graph_tracer.py index 2b82de578d..544859b4e1 100644 --- a/hyper_parallel/compile/tracer/graph_tracer.py +++ b/hyper_parallel/compile/tracer/graph_tracer.py @@ -44,6 +44,8 @@ would keep every forward intermediate alive via ``grad_fn`` references. """ +from __future__ import annotations + __all__ = [ "JointGraph", "trace_model_graph", @@ -63,12 +65,15 @@ import torch from torch import nn +from torch._decomp import get_decompositions from torch._guards import tracing, TracingContext from torch._subclasses import FakeTensorMode from torch.fx.experimental.proxy_tensor import make_fx from torch.fx.traceback import preserve_node_meta from torch.nn.utils import stateless +from .dynamic_shapes import DynamicArgDims, InputGuards, build_symbolic_inputs + # Older torch's make_fx predates record_stack_traces / record_module_stack # and rejects unknown kwargs; forward only the kwargs this torch supports. # The joint graph still captures without record_stack_traces -- the fwd->bw @@ -172,6 +177,7 @@ class JointGraph: # runtime parameter state into the graph. state_fqns: List[str] example_inputs: tuple + input_guards: InputGuards | None = None def extract_module_state(mod: nn.Module) -> Dict[str, torch.Tensor]: @@ -412,50 +418,9 @@ def _input_meta(x: Any) -> Any: return x -def trace_model_graph( # pylint: disable=too-many-locals - model: torch.nn.Module, - train_fn: Callable, - inputs: Dict[str, Any], -) -> JointGraph: - """ - Trace model to generate complete forward + backward graph - - Args: - model: Model (no parallel wrapping) - train_fn: Training function signature: - ``train_fn(model, **inputs) -> loss`` - inputs: Model inputs, forwarded to ``train_fn`` as keyword arguments - - Returns: - JointGraph: Joint forward-backward computation graph - - Note: - Tracing is STATIC-SHAPE: no ``ShapeEnv`` is installed, so the graph - bakes the sample tensors' concrete shapes (symbolic ``sym_size`` - nodes would make per-stage placement ambiguous in ``PpPass``). - This contract is shared by ALL graph-mode users, FSDP included — a - batch whose shape differs from the compile sample must be - re-compiled. - """ - # Extract module state (parameters/buffers) into flat tensors threaded - # through the graph as static inputs (leading placeholders). - model_state = extract_module_state(model) - state_fqns = list(model_state.keys()) - # Which leading state inputs are parameters (vs buffers). FSDP shards - # parameters only: buffers -- e.g. the non-persistent RoPE ``cache`` -- - # are full-rank by nature and must flow through as plain inputs, never - # all-gathered. ``state_is_param`` is indexed by state position, aligned - # with ``state_fqns``. - param_fqns = set(dict(model.named_parameters(remove_duplicate=False)).keys()) - state_is_param = [fqn in param_fqns for fqn in state_fqns] - state_flat, _ = torch.utils._pytree.tree_flatten({"model": model_state}) - - # user_inputs is a plain dict so the traced closure unpacks it back into - # train_fn's keyword arguments. - user_inputs = inputs - user_inputs_flat, user_inputs_spec = torch.utils._pytree.tree_flatten(user_inputs) - - for leaf in [*state_flat, *user_inputs_flat]: +def _validate_trace_leaves(leaves: list[Any]) -> None: + """Reject state or input objects that cannot become explicit FX inputs.""" + for leaf in leaves: if isinstance(leaf, nn.Module): raise ValueError( "trace_model_graph requires explicit tensor state, not " @@ -468,19 +433,17 @@ def trace_model_graph( # pylint: disable=too-many-locals f"got {type(leaf).__name__}." ) - full_args = list(state_flat) + list(user_inputs_flat) - - # Static shapes: no ShapeEnv -> no symbolic sizes -> the traced graph - # carries no aten.sym_size scalar nodes (their CSE-across-phases aliasing - # makes per-stage placement ambiguous). The trainer compiles per fixed - # batch shape, so static tracing loses nothing here. - fake_mode = FakeTensorMode(allow_non_fake_inputs=True) - fake_args = tuple( - _fakeify_input(fake_mode, a) if isinstance(a, torch.Tensor) else a - for a in full_args - ) - num_state_inputs = len(state_flat) +def _make_fwd_bwd_fn( + model: nn.Module, + train_fn: Callable, + state_fqns: list[str], + state_is_param: list[bool], + state_spec: Any, + user_inputs_spec: Any, +) -> tuple[Callable, list[str]]: + """Build the joint callable and its named-loss output metadata.""" + num_state_inputs = len(state_fqns) # ``_fwd_bwd_fn`` communicates the traced ``loss_dict`` key order to the # enclosing scope through this box (plain closure writes cannot escape # the nested ``def``); the keys are attached to the GraphModule below. @@ -539,6 +502,73 @@ def _fwd_bwd_fn(*plain_args): return [loss] + processed_grads + loss_dict_values + return _fwd_bwd_fn, loss_dict_keys_box + + +def trace_model_graph( # pylint: disable=too-many-locals + model: torch.nn.Module, + train_fn: Callable, + inputs: Dict[str, Any], + *, + dynamic: bool = False, + dynamic_arg_dims: DynamicArgDims | None = None, +) -> JointGraph: + """ + Trace model to generate complete forward + backward graph + + Args: + model: Model (no parallel wrapping) + train_fn: Training function signature: + ``train_fn(model, **inputs) -> loss`` + inputs: Model inputs, forwarded to ``train_fn`` as keyword arguments + dynamic: Symbolize all user tensor dimensions when no explicit mapping is supplied. + dynamic_arg_dims: Dotted input paths to dynamic dimensions (supports negative indices). + Providing a mapping enables selective symbolic tracing, even when dynamic is False. + Parameters and buffers always remain static. + + Returns: + JointGraph: Joint forward-backward computation graph + + Note: + Static tracing remains the default. Dynamic traces retain shape guards; + incompatible shapes or shape-dependent branches raise before execution. + """ + # Extract module state (parameters/buffers) into flat tensors threaded + # through the graph as static inputs (leading placeholders). + model_state = extract_module_state(model) + state_fqns = list(model_state.keys()) + # Which leading state inputs are parameters (vs buffers). FSDP shards + # parameters only: buffers -- e.g. the non-persistent RoPE ``cache`` -- + # are full-rank by nature and must flow through as plain inputs, never + # all-gathered. ``state_is_param`` is indexed by state position, aligned + # with ``state_fqns``. + param_fqns = set(dict(model.named_parameters(remove_duplicate=False)).keys()) + state_is_param = [fqn in param_fqns for fqn in state_fqns] + state_flat, state_spec = torch.utils._pytree.tree_flatten({"model": model_state}) + + # user_inputs is a plain dict so the traced closure unpacks it back into + # train_fn's keyword arguments. + user_inputs = inputs + user_inputs_flat, user_inputs_spec = torch.utils._pytree.tree_flatten(user_inputs) + + _validate_trace_leaves([*state_flat, *user_inputs_flat]) + + full_args = list(state_flat) + list(user_inputs_flat) + + if not isinstance(dynamic, bool): + raise ValueError("dynamic must be a bool") + symbolic = dynamic or dynamic_arg_dims is not None + if symbolic: + fake_mode, fake_args = build_symbolic_inputs(state_flat, inputs, dynamic_arg_dims) + else: + fake_mode = FakeTensorMode(allow_non_fake_inputs=True) + fake_args = tuple(_fakeify_input(fake_mode, value) for value in full_args) + num_state_inputs = len(state_flat) + + _fwd_bwd_fn, loss_dict_keys_box = _make_fwd_bwd_fn( + model, train_fn, state_fqns, state_is_param, state_spec, user_inputs_spec + ) + # make_fx only records ``nn_module_stack`` when the traced callable # carries ``_orig_mod``: its ``_init_modes_from_inputs`` then installs a # ``_ModuleStackTracer`` that captures which module's ``forward`` was @@ -548,10 +578,6 @@ def _fwd_bwd_fn(*plain_args): # ``_annotate_autograd_backward``), after tracing. _fwd_bwd_fn._orig_mod = model # type: ignore[attr-defined] - # The pytree spec of the combined state tree is captured here so the - # traced closure can unflatten the leading state placeholders. - _, state_spec = torch.utils._pytree.tree_flatten({"model": model_state}) - ctx = TracingContext(fake_mode) with ( fake_mode, @@ -561,7 +587,12 @@ def _fwd_bwd_fn(*plain_args): torch.autograd.set_multithreading_enabled(False), _non_strict_tracing_context(), ): - traced_graph = make_fx(_fwd_bwd_fn, **_MAKE_FX_KWARGS)(*fake_args) + # Native constant padding can specialize SymInt sizes (notably on + # NPU). Its upstream decomposition preserves causal-LM label lengths. + decompositions = get_decompositions([torch.ops.aten.constant_pad_nd.default]) if symbolic else None + traced_graph = make_fx( + _fwd_bwd_fn, decomposition_table=decompositions, **_MAKE_FX_KWARGS + )(*fake_args) # Stock torch's make_fx does not tag backward nodes (no torchtitan # ``_patch_engine_backward`` hook), so the joint graph splits into @@ -610,6 +641,10 @@ def _fwd_bwd_fn(*plain_args): num_layers=num_layers, state_fqns=state_fqns, example_inputs=fake_args, + input_guards=( + InputGuards(user_inputs_flat, fake_args[num_state_inputs:], fake_mode.shape_env) + if symbolic else None + ), ) @@ -617,6 +652,8 @@ def run_traced_graph( joint_graph: JointGraph, model: torch.nn.Module, inputs: Dict[str, Any], + *, + graph_dispatcher: Callable | None = None, ) -> tuple: """ Execute a traced joint graph against the live model state. @@ -628,6 +665,9 @@ def run_traced_graph( ``torch.no_grad()`` because the graph already contains the explicit backward ops traced by ``torch.autograd.grad``. + An optional graph_dispatcher selects a size variant after all input guards + pass. It must preserve the general graph's flat input/output contract. + FSDP is invisible here: FSDPPass shards ``model``'s parameters in place, so ``model.parameters()`` yields the shards the graph expects as its leading static inputs (the graph re-gathers them via AllGather each step). @@ -658,10 +698,14 @@ def run_traced_graph( state_flat, _ = torch.utils._pytree.tree_flatten({"model": model_state}) + if joint_graph.input_guards is not None: + joint_graph.input_guards.validate(user_flat) + flat_inputs = list(state_flat) + list(user_flat) with torch.no_grad(): - outputs = joint_graph.graph_module(*flat_inputs) + runnable = joint_graph.graph_module if graph_dispatcher is None else graph_dispatcher + outputs = runnable(*flat_inputs) loss_dict_keys = getattr(joint_graph.graph_module, "loss_dict_keys", []) num_loss_outputs = len(loss_dict_keys) diff --git a/hyper_parallel/compile/trainer.py b/hyper_parallel/compile/trainer.py index a971cf9f11..7612ab001e 100644 --- a/hyper_parallel/compile/trainer.py +++ b/hyper_parallel/compile/trainer.py @@ -34,6 +34,7 @@ import torch.distributed as dist from .compiler import GraphCompiler +from .tracer.dynamic_shapes import DynamicArgDims from .pass_config import PassConfig from .graph_parallel_plan import GraphParallelPlan @@ -61,6 +62,12 @@ def __init__( optimizer_config: Optional[dict] = None, device: Optional[torch.device] = None, mesh_context: Optional[Any] = None, + dynamic: Optional[bool] = None, + dynamic_arg_dims: DynamicArgDims | None = None, + compile_sizes: Optional[list[int]] = None, + compile_size_input: Optional[str] = None, + compile_size_dim: Optional[int] = None, + max_specializations: Optional[int] = None, ) -> None: """ Args: @@ -84,6 +91,12 @@ def __init__( already hold the group object) and only the FSDP shard sub-mesh is registered under ``"fsdp"``. Use this to feed an automodel TP-sharded model into the graph-mode FSDP pass. + dynamic: Enable symbolic user input dimensions (forwarded to GraphCompiler). + dynamic_arg_dims: Optional input paths to dynamic dimensions (forwarded to GraphCompiler). + compile_sizes: Sizes to lazily specialize (forwarded to GraphCompiler). + compile_size_input: Optional dispatch tensor path (forwarded to GraphCompiler). + compile_size_dim: Dispatch axis for an explicit path (forwarded to GraphCompiler). + max_specializations: Cache capacity (forwarded to GraphCompiler). """ # The compiler owns the model, device, config, and the compiled # graph; the trainer keeps only the optimizer / loop policy. @@ -95,6 +108,12 @@ def __init__( trainer_config=trainer_config, device=device, mesh_context=mesh_context, + dynamic=dynamic, + dynamic_arg_dims=dynamic_arg_dims, + compile_sizes=compile_sizes, + compile_size_input=compile_size_input, + compile_size_dim=compile_size_dim, + max_specializations=max_specializations, ) self.optimizer_config = optimizer_config or {} self.optimizer = None diff --git a/hyper_parallel/models/build_options.py b/hyper_parallel/models/build_options.py index 8f13713e1a..a1a5357c58 100644 --- a/hyper_parallel/models/build_options.py +++ b/hyper_parallel/models/build_options.py @@ -112,7 +112,12 @@ def get_device_id() -> int: @dataclass class CompileConfig: - """Decoder-layer ``torch.compile`` options exposed by the Trainer.""" + """Decoder-layer and joint-graph compile options exposed by the Trainer. + + In joint-graph mode, ``dynamic`` symbolizes user tensor dimensions. + ``dynamic_arg_dims`` optionally selects dimensions by dotted input path; + a mapping enables dynamic tracing and takes precedence over ``dynamic``. + """ enabled: bool = False use_joint_graph: bool = False @@ -122,6 +127,12 @@ class CompileConfig: backend: Optional[str] = None options: Optional[dict[str, Any]] = None dynamo_cache_size_limit: int = 256 + # Joint-graph input paths, e.g. model_inputs.input_ids: [0, 1]. + dynamic_arg_dims: Optional[dict[str, int | list[int]]] = None + compile_sizes: Optional[list[int]] = None + compile_size_input: Optional[str] = None + compile_size_dim: int = -1 + max_specializations: int = 8 def selects_graph_compiler(self) -> bool: """Return whether this config selects the graph-compiler trainer path.""" @@ -132,6 +143,8 @@ def __post_init__(self) -> None: for name in ("enabled", "use_joint_graph", "fullgraph", "dynamic"): if not isinstance(getattr(self, name), bool): raise TypeError(f"compile.{name} must be a bool") + if self.dynamic_arg_dims is not None and not isinstance(self.dynamic_arg_dims, dict): + raise ValueError("compile.dynamic_arg_dims must be a mapping or None") if not isinstance(self.mode, str) or not self.mode.strip(): raise ValueError("compile.mode must be a non-empty string") if self.backend is not None and ( diff --git a/hyper_parallel/trainer/base.py b/hyper_parallel/trainer/base.py index f36fbfab46..24f5f9aa4a 100644 --- a/hyper_parallel/trainer/base.py +++ b/hyper_parallel/trainer/base.py @@ -559,8 +559,9 @@ def _graph_train_fn( model: torch.nn.Module, model_inputs: Dict[str, Any], labels: Optional[torch.Tensor], + token_counts: Optional[Dict[str, Dict[str, torch.Tensor]]] = None, ) -> tuple[torch.Tensor, Dict[str, torch.Tensor]]: - """Wrap the BaseTrainer forward/loss logic for joint-graph capture.""" + """Capture forward/loss with explicit runtime token counts when provided.""" model_fwd_context = ( self.model_fwd_context() if callable(self.model_fwd_context) @@ -568,13 +569,12 @@ def _graph_train_fn( ) with model_fwd_context: outputs = model(**model_inputs, use_cache=False) - # postforward adds the token-weighted global aggregation used by the - # eager path; it embeds .item()/all_reduce, which the tracer must be - # able to capture for this to compile. Both the backward loss and the - # named per-key losses are returned: the tracer emits the loss_dict - # values as extra graph outputs (after the gradients), so - # multi-key losses keep per-key logging parity with the eager path. - loss, loss_dict = self.postforward(outputs, labels) + # Counts must be graph inputs: packed lengths and ignored-label counts + # can vary even when tensor shapes happen to match the first batch. + if token_counts is None: + loss, loss_dict = self.postforward(outputs, labels) + else: + loss, loss_dict = self.postforward(outputs, labels, token_counts=token_counts) return loss, loss_dict def _init_callbacks(self): @@ -685,22 +685,27 @@ def _to_device(v: Any) -> Any: return micro_batch def postforward( - self, outputs: ModelOutput, labels: Optional[torch.Tensor] + self, outputs: ModelOutput, labels: Optional[torch.Tensor], + *, token_counts: Optional[Dict[str, Dict[str, torch.Tensor]]] = None, ) -> tuple[torch.Tensor, dict[str, torch.Tensor]]: """Postprocess model outputs after forward pass. Args: outputs: Model outputs consumed by the configured loss. labels: Labels associated with the current micro-batch. + token_counts: Explicit current/step token counts for joint-graph + execution. Eager callers use the trainer's live counts. Returns: Backward loss and named globally aggregated loss values. """ local_loss = self.loss_fn(model_output=outputs, labels=labels) + current_counts = self.current_token_counts if token_counts is None else token_counts["current"] + step_counts = self.step_token_counts if token_counts is None else token_counts["step"] loss_dict: Dict[str, torch.Tensor] = mean_global_loss( local_loss, - self.current_token_counts, - self.step_token_counts, + current_counts, + step_counts, device_mesh=self.mesh, ) loss = torch.stack(list(loss_dict.values())).sum() @@ -763,9 +768,16 @@ def forward_backward_step( # The traced ``loss_dict`` values flow back as extra graph # outputs (emitted after the gradients), so the logging # metrics keep per-key parity with the eager path. + graph_kwargs = {} + if hasattr(self, "current_token_counts") and hasattr(self, "step_token_counts"): + graph_kwargs["token_counts"] = { + "current": self.current_token_counts, + "step": self.step_token_counts, + } loss, loss_dict = self.graph_compiler.forward_backward( model_inputs=micro_batch, labels=labels, + **graph_kwargs, ) else: model_fwd_context = ( diff --git a/hyper_parallel/trainer/runtime/metrics.py b/hyper_parallel/trainer/runtime/metrics.py index b7d63ef0f7..10e9d67c57 100644 --- a/hyper_parallel/trainer/runtime/metrics.py +++ b/hyper_parallel/trainer/runtime/metrics.py @@ -107,8 +107,8 @@ def mean_global_loss( # expressed with tensor collectives so the static-shape tracer can # capture it (no ``.item()``, no data-dependent zero-token guard). # ``differentiable_all_reduce`` dispatches to functional collectives - # under tracing; the token-count tensors are baked as trace-time - # constants, matching the tracer's fixed-shape contract. + # under tracing. BaseTrainer passes token-count tensors as explicit + # graph inputs so packed/ignored-token counts remain live. step_len = differentiable_all_reduce( step_token_counts[f"{loss_name}_tokens"], "sum", dp_cp_group ) diff --git a/size_specialization_development.md b/size_specialization_development.md new file mode 100644 index 0000000000..c8f22c1ef8 --- /dev/null +++ b/size_specialization_development.md @@ -0,0 +1,176 @@ +# 通用符号图与按尺寸懒生成 FX 专用图 + +尺寸特化采用 “one general-shape graph + lazy per-size FX specialization”:先捕获一份通用符号联合图,完成既有并行 Pass;运行时遇到配置的热点尺寸,再从该图生成并缓存绑定具体 shape 的 FX 图。其他尺寸继续运行通用图。 + +这里的特化生成独立 `GraphModule` 和 Python 执行代码,不调用 Inductor、NPU kernel compiler 或 NPU Graph capture。未绑定的表达式保留动态形式,因此专用图并不保证完全静态,也不承诺吞吐提升。以下链接相对于仓库根目录,行号按当前实现核对。 + +## 1. 与 MagiCompiler 的对应关系 + +MagiCompiler 的 `PiecewiseBackend` 保留 general-shape callable,首次执行通用图,之后对 `compile_sizes` 中的尺寸懒调用后端编译并缓存。这里沿用其调度思路,实现范围为 FX shape 特化。 + +| MagiCompiler 中的机制 | HyperParallel 中的对应实现 | +| --- | --- | +| general-shape callable | `JointGraph.graph_module` 保存通用符号联合图 | +| 首次通用执行 | 首次热点命中也先执行通用图 | +| `compile_sizes` | YAML / Python 参数指定热点尺寸 | +| `sym_shape_indices[0]` | 自动选择首个符号输入轴,或显式指定输入路径与轴 | +| `ConcreteSizeEntry` | 保存尺寸、专用 `GraphModule` 和折叠节点数 | +| 按 runtime size 查询产物 | 先筛选尺寸,再按完整用户输入元数据查询缓存 | +| `compiler_manager.compile()` | `specialize_graph()` 绑定符号并生成 FX 代码 | +| 后端编译与持久缓存 | 当前未接入 | + +HyperParallel 不导入或依赖 MagiCompiler;没有引入其 piecewise partition、后端管理或磁盘缓存体系。 + +## 2. 配置与边界校验 + +真实文本训练 YAML 的 `compile` 段可以使用: + +```yaml +compile: + enabled: true + use_joint_graph: true + dynamic: true + compile_sizes: [116, 120, 127] + compile_size_input: model_inputs.input_ids + compile_size_dim: 1 + max_specializations: 8 +``` + +字段定义见 [CompileConfig](hyper_parallel/models/build_options.py#L113),参数继承和动态模式检查见 [GraphCompiler.__init__](hyper_parallel/compile/compiler.py#L104)。显式 Python 参数优先;省略时继承 `trainer_config.compile`,未提供配置时使用默认值。 + +| 参数 | 默认值与校验 | +| --- | --- | +| `compile_sizes` | 默认 `None`,空列表也关闭特化;非空时必须启用 `dynamic=True` 或提供 `dynamic_arg_dims` | +| `compile_size_input` | 默认自动选择扁平用户输入中的首个符号轴;显式值必须是可解析到 Tensor 叶子的非空点分路径 | +| `compile_size_dim` | 显式路径默认使用 `-1`;支持负轴,必须是整数且不越界;自动选择时轴来自符号输入 | +| `max_specializations` | 默认 `8`,必须是正整数;限制完整输入签名数量 | + +[validate_compile_sizes()](hyper_parallel/compile/size_specialization.py#L38) 接受整数列表或元组,拒绝负数、布尔值及其他元素,并去重。容量校验即使在特化关闭时也执行。 + +[尺寸选择器](hyper_parallel/compile/size_specialization.py#L115) 在首次捕获后、并行 Pass 改写模型前校验。不存在的路径、非 Tensor 叶子、非法轴都会报错;自动模式找不到符号维度也会报错。显式路径从 `forward_backward(**inputs)` 的顶层关键字开始,例如 `x`、`batch.x` 或文本训练中的 `model_inputs.input_ids`。 + +`compile_sizes` 接受 `0/1` 并不表示通用图能覆盖这些尺寸;是否可执行仍由原有 shape guards 决定。文本训练建议显式选择序列轴,避免自动模式选到 batch 轴。 + +## 3. 首次通用执行与缓存分派 + +[GraphCompiler.compile()](hyper_parallel/compile/compiler.py#L168) 捕获一次联合图,建立分派器,再运行既有并行 Pass。后续特化从完成 Pass 的图派生,不重新捕获模型,也不再次分片真实参数。 + +```mermaid +flowchart TD + A[用户输入与 live 模型状态] --> B[结构和通用图 guards 检查] + B --> C{首次执行或非热点尺寸?} + C -- 是 --> G[通用图] + C -- 否 --> D{完整输入签名已缓存?} + D -- 是 --> S[缓存专用图] + D -- 否 --> E{缓存达到上限?} + E -- 是 --> G + E -- 否 --> F[绑定 shape 并生成 FX 图] + F --> S + G --> H[返回 loss 与梯度并沿用累积流程] + S --> H +``` + +[run_traced_graph()](hyper_parallel/compile/tracer/graph_tracer.py#L651) 先检查模型状态名称、输入 pytree 结构和 guards,再进入 `torch.no_grad()` 执行通用图或分派器。缓存命中同样经过 guards;失败时直接报错,不生成变体、不执行图或通信,也不自动 retrace。 + +[分派器主流程](hyper_parallel/compile/size_specialization.py#L157) 的首次执行始终使用通用图。这是首次图执行的 warmup 策略:即使用户预先调用了 `compiler.compile()`,下一次首次 `forward_backward()` 仍走通用图。 + +配置热点 `[5, 7]`、其他元数据不变时,长度 `5 → 7 → 7 → 9 → 5 → 5` 分别对应:通用执行、生成 7 的图、命中 7、通用执行、生成 5 的图、命中 5。预期通用执行 2 次、专用执行 4 次、缓存命中 2 次、生成 2 份专用图。 + +下面展示直接使用公共入口的方式,等价行为在现有 UT 中验证: + +```python +import torch +from hyper_parallel.compile import GraphCompiler, PassConfig + +def train_fn(model, x): + return model(x).square().mean() + +model = torch.nn.Linear(4, 3) +compiler = GraphCompiler( + model, train_fn, + device=torch.device("cpu"), + pass_config=PassConfig(fsdp_enabled=False), + dynamic_arg_dims={"x": [0, 1]}, + compile_sizes=[5, 7], compile_size_input="x", compile_size_dim=1, +) +for length in (5, 7, 7, 9, 5, 5): + model.zero_grad() + loss, loss_dict = compiler.forward_backward(x=torch.randn(2, length, 4)) +print(compiler.specialization_stats) +``` + +## 4. 特化只绑定 shape,数据与通信保持运行期执行 + +[specialize_graph()](hyper_parallel/compile/size_specialization.py#L49) 使用当前用户 Tensor 的元数据绑定通用图符号: + +```python +guards = joint_graph.input_guards +tensors = [value for value in user_inputs if isinstance(value, torch.Tensor)] +bindings = guards.shape_env.bind_symbols(guards.fake_tensors, tensors) +``` + +它只折叠白名单内的 shape 查询和整数算术,且节点的 `meta["val"]` 必须是 `SymInt`,替换符号后能够得到整数。例如 `[2, 7, 4]` 下的 `batch * length` 可绑定为 `14`,相应 `view(x, [batch * length, 4])` 变为 `view(x, [14, 4])`。 + +其余节点通过 `graph.node_copy()` 复制,最后生成独立 `GraphModule`;通用图不被改写。复制时移除旧的符号 Tensor 元数据,避免把它误当作变体的具体元数据。 + +生成过程读取 shape 元数据,不执行 Tensor 算子、forward/backward、collectives 或 RNG。已存在的 FSDP all-gather、reduce-scatter 与等待节点保留在生成图中,随后正常执行;生成变体不会额外插入或提前执行通信。 + +[完整输入签名](hyper_parallel/compile/size_specialization.py#L137) 包括所有用户 Tensor 的 shape、stride、storage offset、dtype、device、layout、requires_grad,以及非 Tensor 叶子的类型和值。分派尺寸只是热点筛选条件,并非完整缓存键。 + +因此 `[2, 7, 4]` 和 `[3, 7, 4]` 即使都命中长度 `7`,也需要两份图。部分元数据由 guards 固定;缓存键仍完整记录它们。改变 dtype、别名关系或关联尺寸时,先按通用图契约检查,不会靠缓存键绕过 guards。 + +容量按签名计数,不按热点长度计数。达到 `max_specializations` 后,新签名回退到通用图;已经缓存的签名继续命中。当前没有淘汰、磁盘持久化或跨 rank 共享;每个 compiler 独立维护缓存。 + +缓存不保存真实用户 Tensor 或其存储,也不将权重、Tensor 标量值或随机数据折叠为捕获值。[执行入口](hyper_parallel/compile/tracer/graph_tracer.py#L681) 每次读取 live 模型参数和缓冲;优化器更新与梯度累积沿用原有流程。 + +文本训练中的 `token_counts.current` 与 `token_counts.step` 由 [BaseTrainer.forward_backward_step()](hyper_parallel/trainer/base.py#L771) 显式传入图。[图训练函数](hyper_parallel/trainer/base.py#L557) 和 [loss 聚合](hyper_parallel/trainer/runtime/metrics.py#L105) 消费当前 Tensor 值;相同 shape 下 token 数变化仍会改变 loss 权重与梯度,不需要重新生成图。 + +## 5. 统计接口与验证入口 + +[GraphCompiler.specialization_stats](hyper_parallel/compile/compiler.py#L164) 返回只读统计快照;未建立分派器或关闭特化时为 `{}`。[GraphTrainer](hyper_parallel/compile/trainer.py#L103) 将配置和执行委托给 compiler,目前没有另设公开统计属性;BaseTrainer 可通过 `trainer.graph_compiler.specialization_stats` 读取。 + +| 字段 | 含义 | +| --- | --- | +| `general_calls` | warmup、非热点和容量回退的通用图执行次数 | +| `specialized_calls` / `cache_hits` | 专用执行总次数 / 复用已有条目的次数 | +| `compilations` / `cached_variants` | 已生成且缓存的 FX 图数量,不是模型捕获或底层 kernel 编译次数 | +| `compiled_sizes` | 已缓存条目的尺寸去重列表;同尺寸可对应多个签名 | +| `folded_nodes` | 缓存图中折叠的 shape 节点总数,不按执行次数累加 | +| `capacity_fallbacks` / `last_dispatch` | 容量导致的回退次数 / 最近一次分派类型 | + +当前可复现的逻辑验证集中在 [test_size_specialization.py](tests/ut/compile/test_size_specialization.py#L35) 和 [test_dynamic_shapes.py](tests/ut/compile/test_dynamic_shapes.py#L40): + +- 热点 warmup、懒生成、命中、非热点回退、不同次级维度与缓存上限。 +- 真实 FX 代码变化、shape 节点折叠、loss/梯度对照、live 权重与梯度累积。 +- 配置继承、选择器校验、guards 优先、输入存储释放和 RNG 状态。 +- BaseTrainer 的动态配置与运行期 token 权重,以及符号 padding 的 causal loss。 + +```bash +python -m pytest tests/ut/compile/test_dynamic_shapes.py \ + tests/ut/compile/test_size_specialization.py -q +``` + +UT 使用 CPU,分布式逻辑通过 mock 验证。普通 NPU smoke 使用现有 TextTrainer、真实模型结构和 TP2 + FSDP2 的数据与训练流程,补充集成路径验证;它不替代逐参数精度对照或吞吐基准。 + +## 6. 普通四卡文本训练 smoke + +从仓库根目录运行,模型路径须包含配置和 tokenizer。默认 YAML 随机初始化模型结构,运行 6 个 optimizer step;`prepare_dynamic_text.py` 生成多种长度的本地文本,不需要下载数据: + +```bash +python -m hyper_parallel.compile.examples.automodel_text_graph.prepare_dynamic_text + +ASCEND_RT_VISIBLE_DEVICES=0,1,2,3 \ +OUTPUT_DIR=./output/automodel_text_graph/specialized_npu \ +bash hyper_parallel/compile/examples/automodel_text_graph/run.sh \ + hyper_parallel/compile/examples/automodel_text_graph/train_online_lm_graph_dynamic_tp2_fsdp2.yaml \ + --model.pretrained_model_name_or_path=/path/to/Qwen3-0.6B \ + '--compile.compile_sizes=[116,120,127]' \ + --compile.compile_size_input=model_inputs.input_ids \ + --compile.compile_size_dim=1 +``` + +[run.sh](hyper_parallel/compile/examples/automodel_text_graph/run.sh#L52) 默认启动 `scripts/train_lm.py`,接受已有 CLI 覆盖,输出普通训练日志。[动态 YAML](hyper_parallel/compile/examples/automodel_text_graph/train_online_lm_graph_dynamic_tp2_fsdp2.yaml#L40) 中的三项特化配置可选;省略后运行通用符号图。 + +检查训练完成、loss 有限和热点反复出现时的 `Generated size-specialized FX graph` 日志。实际 packing 长度依赖 tokenizer 和数据;`[116, 120, 127]` 是该示例的可选热点配置,其他输入应按重复出现的尺寸调整。普通入口不自动导出专门的逐 rank 审计 JSON。 + +当前验证入口为现有 UT 与普通训练命令。动态 PP、guard 失败后自动 retrace、缓存淘汰、持久缓存、后端 kernel 编译和跨进程缓存共享仍在本实现范围之外。 diff --git a/tests/ut/auto_models/test_build_options.py b/tests/ut/auto_models/test_build_options.py index 4765b629d5..a807366d10 100644 --- a/tests/ut/auto_models/test_build_options.py +++ b/tests/ut/auto_models/test_build_options.py @@ -60,6 +60,11 @@ def test_compile_config_fields(self): ("backend", "None"), ("options", "None"), ("dynamo_cache_size_limit", "256"), + ("dynamic_arg_dims", "None"), + ("compile_sizes", "None"), + ("compile_size_input", "None"), + ("compile_size_dim", "-1"), + ("max_specializations", "8"), ], ) diff --git a/tests/ut/auto_models/test_public_api.py b/tests/ut/auto_models/test_public_api.py index 040756b00d..ddaf9cabcb 100644 --- a/tests/ut/auto_models/test_public_api.py +++ b/tests/ut/auto_models/test_public_api.py @@ -177,7 +177,11 @@ def test_accelerator_config_fields(self): @arg_mark(plat_marks=["cpu_linux", "cpu_macos"], level_mark="level0", card_mark="allcards", essential_mark="essential") def test_compile_config_fields(self): - """CompileConfig field order and defaults stay unchanged.""" + """ + Feature: Dynamic joint-graph training. + Description: CompileConfig field order and defaults stay unchanged. + Expectation: Losses, gradients and public contracts remain correct. + """ from hyper_parallel.trainer.config import CompileConfig self.assertEqual( @@ -191,6 +195,11 @@ def test_compile_config_fields(self): ("backend", "None"), ("options", "None"), ("dynamo_cache_size_limit", "256"), + ("dynamic_arg_dims", "None"), + ("compile_sizes", "None"), + ("compile_size_input", "None"), + ("compile_size_dim", "-1"), + ("max_specializations", "8"), ], ) diff --git a/tests/ut/compile/test_dynamic_shapes.py b/tests/ut/compile/test_dynamic_shapes.py new file mode 100644 index 0000000000..3f305b5ad3 --- /dev/null +++ b/tests/ut/compile/test_dynamic_shapes.py @@ -0,0 +1,422 @@ +# Copyright 2026 Huawei Technologies Co., Ltd +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# ============================================================================ +"""End-to-end symbolic joint graphs, configuration and runtime shape guards.""" + +import copy +import gc +import unittest +import weakref +from contextlib import nullcontext +from types import SimpleNamespace +from unittest.mock import Mock, patch + +import torch + +from hyper_parallel.compile import GraphCompiler, GraphTrainer, PassConfig +from hyper_parallel.compile.passes.parallel.fsdp_pass import FSDPPass +from hyper_parallel.compile.tracer.dynamic_shapes import normalize_dynamic_arg_dims +from hyper_parallel.compile.tracer.graph_tracer import run_traced_graph, trace_model_graph +from hyper_parallel.models.build_options import CompileConfig +from hyper_parallel.trainer.base import BaseTrainer +from tests.common.mark_utils import arg_mark + + +def _loss(model, x, y): + return ((model(x) - y) ** 2).mean() + + +class TestDynamicShapes(unittest.TestCase): + """Real forward/backward capture with changing user tensor sizes.""" + + def setUp(self) -> None: + """Create a CPU-only compiler configuration.""" + torch.manual_seed(42) + self.model = torch.nn.Linear(4, 3) + self.config = PassConfig(fsdp_enabled=False) + + def _compiler(self, **kwargs): + return GraphCompiler(self.model, _loss, pass_config=self.config, **kwargs) + + def _assert_step(self, compiler, batch, sequence): + x = torch.randn(batch, sequence, 4) + y = torch.randn(batch, sequence, 3) + self.model.zero_grad() + actual, _ = compiler.forward_backward(x=x, y=y) + expected = _loss(self.model, x, y) + gradients = torch.autograd.grad(expected, tuple(self.model.parameters())) + torch.testing.assert_close(actual, expected) + for parameter, gradient in zip(self.model.parameters(), gradients): + torch.testing.assert_close(parameter.grad, gradient) + + @arg_mark(plat_marks=["cpu_linux", "cpu_macos"], level_mark="level0", + card_mark="allcards", essential_mark="essential") + def test_batch_and_sequence_reuse_graph(self): + """ + Feature: Symbolic graph shape contracts. + Description: One symbolic graph computes loss and gradients for several shapes. + Expectation: Dynamic dimensions reuse graphs and static dimensions retain guards. + """ + compiler = self._compiler(dynamic_arg_dims={"x": [0, 1], "y": [0, 1]}) + with patch.object(compiler, "compile", wraps=compiler.compile) as compile_spy: + for batch, sequence in [(2, 5), (3, 7), (4, 3), (2, 9)]: + self._assert_step(compiler, batch, sequence) + self.assertEqual(compile_spy.call_count, 1) + joint = compiler._joint_graph + for value in joint.example_inputs[:2]: + for dim in value.shape: + self.assertIs(type(dim), int) + self.assertIsInstance(joint.example_inputs[2].shape[0], torch.SymInt) + self.assertEqual(joint.example_inputs[2].shape[2], 4) + + def test_automatic_dynamic_mode(self): + """dynamic=True needs no manual input paths.""" + compiler = self._compiler(dynamic=True) + self._assert_step(compiler, 2, 5) + self._assert_step(compiler, 3, 7) + + def test_negative_dims_and_nested_inputs(self): + """Dictionary/list paths and negative dimension indices are supported.""" + compiler = GraphCompiler( + self.model, lambda model, batch: _loss(model, batch["items"][0], batch["target"]), + pass_config=self.config, + dynamic_arg_dims={"batch.items.0": [-3, -2], "batch.target": [0, 1]}, + ) + for batch, sequence in [(2, 5), (3, 7)]: + inputs = {"items": [torch.randn(batch, sequence, 4)], "target": torch.randn(batch, sequence, 3)} + actual, _ = compiler.forward_backward(batch=inputs) + torch.testing.assert_close(actual, _loss(self.model, inputs["items"][0], inputs["target"])) + + def test_same_sample_sizes_are_independent(self): + """Unrelated dimensions with equal hints do not acquire equality guards.""" + compiler = GraphCompiler( + self.model, lambda model, x, z: model(x).mean() + model(z).mean(), + pass_config=self.config, dynamic_arg_dims={"x": 0, "z": 0}, + ) + compiler.forward_backward(x=torch.randn(3, 4), z=torch.randn(3, 4)) + x, z = torch.randn(5, 4), torch.randn(7, 4) + actual, _ = compiler.forward_backward(x=x, z=z) + torch.testing.assert_close(actual, self.model(x).mean() + self.model(z).mean()) + + def test_grad_accumulation_across_shapes(self): + """Microbatches of different sizes still accumulate gradients.""" + compiler = self._compiler(dynamic=True) + expected_grads = [torch.zeros_like(parameter) for parameter in self.model.parameters()] + for batch in (2, 5, 3): + x, y = torch.randn(batch, 4), torch.randn(batch, 3) + compiler.forward_backward(x=x, y=y) + grads = torch.autograd.grad(_loss(self.model, x, y), tuple(self.model.parameters())) + for expected, grad in zip(expected_grads, grads): + expected.add_(grad) + for parameter, expected in zip(self.model.parameters(), expected_grads): + torch.testing.assert_close(parameter.grad, expected) + + def test_optimizer_updates_match_eager(self): + """GraphTrainer preserves live state across variable-length optimizer steps.""" + reference = copy.deepcopy(self.model) + optimizer = torch.optim.Adam(reference.parameters(), lr=1e-3, foreach=False) + trainer = GraphTrainer(self.model, _loss, pass_config=self.config, dynamic=True, + optimizer_config={"lr": 1e-3}, device=torch.device("cpu")) + for batch, sequence in [(2, 5), (3, 7), (4, 3)]: + x, y = torch.randn(batch, sequence, 4), torch.randn(batch, sequence, 3) + actual, _ = trainer.train_step(x=x, y=y) + expected = _loss(reference, x, y) + expected.backward() + torch.testing.assert_close(actual, expected) + trainer.optimizer_step() + optimizer.step() + optimizer.zero_grad() + for parameter, ref in zip(self.model.parameters(), reference.parameters()): + torch.testing.assert_close(parameter, ref) + + @arg_mark(plat_marks=["cpu_linux", "cpu_macos"], level_mark="level0", + card_mark="allcards", essential_mark="essential") + def test_python_shape_branch_is_guarded(self): + """ + Feature: Symbolic graph shape contracts. + Description: A branch chosen while tracing must not silently run for incompatible sizes. + Expectation: Dynamic dimensions reuse graphs and static dimensions retain guards. + """ + def train(model: torch.nn.Module, x: torch.Tensor) -> torch.Tensor: + """Evaluate the parameterized loss for the dynamic input.""" + loss = model(x).sum() + return loss * 2 if x.shape[0] > 3 else loss * 3 + + compiler = GraphCompiler(self.model, train, pass_config=self.config, dynamic_arg_dims={"x": 0}) + compiler.forward_backward(x=torch.randn(5, 4)) + x = torch.randn(7, 4) + actual, _ = compiler.forward_backward(x=x) + torch.testing.assert_close(actual, train(self.model, x)) + before = self.model.weight.grad.clone() + with self.assertRaisesRegex(ValueError, "shape guards"): + compiler.forward_backward(x=torch.randn(2, 4)) + torch.testing.assert_close(self.model.weight.grad, before) + + def test_unmarked_dimension_is_static(self): + """Explicit mappings take precedence over automatic dimension selection.""" + compiler = self._compiler(dynamic=True, dynamic_arg_dims={"x": 0, "y": 0}) + self._assert_step(compiler, 2, 5) + self._assert_step(compiler, 3, 5) + with self.assertRaisesRegex(ValueError, "shape guards"): + self._assert_step(compiler, 3, 7) + + def test_related_sizes_are_guarded(self): + """Broadcast-incompatible label sizes fail before graph execution.""" + compiler = self._compiler(dynamic=True) + self._assert_step(compiler, 2, 5) + with self.assertRaisesRegex(ValueError, "shape guards"): + compiler.forward_backward(x=torch.randn(3, 7, 4), y=torch.randn(2, 7, 3)) + + def test_zero_one_specialization_is_guarded(self): + """Torch's size-one specialization is reported instead of silently reused.""" + compiler = self._compiler(dynamic=True) + self._assert_step(compiler, 1, 5) + with self.assertRaisesRegex(ValueError, "sizes 0/1"): + self._assert_step(compiler, 2, 5) + + def test_metadata_and_constants_are_guarded(self): + """Dtype, rank and captured Python constants cannot change underneath a graph.""" + compiler = GraphCompiler(self.model, lambda model, x, scale: model(x).mean() * scale, + pass_config=self.config, dynamic=True) + compiler.forward_backward(x=torch.randn(2, 4), scale=2) + for inputs in [ + {"x": torch.randn(3, 4), "scale": 3}, + {"x": torch.randn(3, 4, dtype=torch.float64), "scale": 2}, + {"x": torch.randn(3, 2, 4), "scale": 2}, + ]: + with self.subTest(inputs=inputs): + with self.assertRaisesRegex(ValueError, "rank, dtype"): + compiler.forward_backward(**inputs) + + def test_stride_is_guarded(self): + """Changing layout cannot invalidate traced view/reshape assumptions.""" + compiler = self._compiler(dynamic=True) + compiler.forward_backward(x=torch.randn(2, 4), y=torch.randn(2, 3)) + with self.assertRaisesRegex(ValueError, "shape guards"): + compiler.forward_backward(x=torch.randn(4, 3).t(), y=torch.randn(3, 3)) + + def test_alias_marks_merge_and_alias_changes_fail(self): + """Multiple paths to one tensor merge dimensions and retain identity guards.""" + compiler = GraphCompiler(self.model, lambda model, x, z: (model(x) + model(z)).mean(), + pass_config=self.config, dynamic_arg_dims={"x": 0, "z": 1}) + for shape in [(2, 5, 4), (3, 7, 4)]: + value = torch.randn(*shape) + actual, _ = compiler.forward_backward(x=value, z=value) + torch.testing.assert_close(actual, (2 * self.model(value)).mean()) + with self.assertRaisesRegex(ValueError, "aliasing"): + compiler.forward_backward(x=value, z=value.clone()) + + def test_sample_storage_is_not_retained(self): + """Runtime guards retain only metadata and fake tensors.""" + compiler = self._compiler(dynamic=True) + x, y = torch.randn(2, 4), torch.randn(2, 3) + references = weakref.ref(x), weakref.ref(y) + compiler.compile(x=x, y=y) + del x, y + gc.collect() + self.assertTrue(all(reference() is None for reference in references)) + + @arg_mark(plat_marks=["cpu_linux", "cpu_macos"], level_mark="level0", + card_mark="allcards", essential_mark="essential") + def test_static_default_unchanged(self): + """ + Feature: Symbolic graph shape contracts. + Description: Opting out retains concrete tensor metadata and no symbolic guards. + Expectation: Dynamic dimensions reuse graphs and static dimensions retain guards. + """ + compiler = self._compiler() + self._assert_step(compiler, 2, 5) + self.assertIsNone(compiler._joint_graph.input_guards) + for dim in compiler._joint_graph.example_inputs[2].shape: + self.assertIs(type(dim), int) + + def test_invalid_mapping(self): + """Bad paths, non-tensors and invalid dimensions report user-facing errors.""" + for mapping in [{"missing": 0}, {"x": 3}, {"x": -4}, {"scale": 0}]: + with self.subTest(mapping=mapping): + compiler = GraphCompiler(self.model, lambda model, x, scale: model(x).mean() * scale, + pass_config=self.config, dynamic_arg_dims=mapping) + with self.assertRaisesRegex(ValueError, "dynamic_arg_dims"): + compiler.compile(x=torch.randn(2, 4), scale=2) + for mapping in [[], {"": 0}, {"x..a": 0}, {"x": True}, {"x": [1.5]}, {"x": "0"}]: + with self.subTest(mapping=mapping): + with self.assertRaisesRegex(ValueError, "dynamic_arg_dims"): + normalize_dynamic_arg_dims(mapping) + + def test_pp_rejected_before_tracing(self): + """Unsupported pipeline mode fails without mutating model state.""" + compiler = GraphCompiler(self.model, _loss, pass_config=PassConfig(pp_enabled=True), dynamic=True) + with patch("hyper_parallel.compile.compiler.trace_model_graph") as tracer: + with self.assertRaisesRegex(ValueError, "pipeline parallel"): + compiler.compile(x=torch.randn(2, 4), y=torch.randn(2, 3)) + tracer.assert_not_called() + + @arg_mark(plat_marks=["cpu_linux", "cpu_macos"], level_mark="level0", + card_mark="allcards", essential_mark="essential") + def test_buffer_shapes_stay_static(self): + """ + Feature: Symbolic graph shape contracts. + Description: Model buffers keep fixed dimensions even when all user dims are symbolic. + Expectation: Dynamic dimensions reuse graphs and static dimensions retain guards. + """ + model = torch.nn.Sequential(torch.nn.Linear(4, 3)) + model.register_buffer("offset", torch.ones(3)) + joint = trace_model_graph(model, lambda m, x: (m(x) + m.offset).mean(), + {"x": torch.randn(2, 4)}, dynamic=True) + for tensor in joint.example_inputs[:joint.graph_module.num_state_inputs]: + for dim in tensor.shape: + self.assertIs(type(dim), int) + x = torch.randn(5, 4) + actual, _, _ = run_traced_graph(joint, model, {"x": x}) + torch.testing.assert_close(actual, (model(x) + model.offset).mean()) + + def test_fsdp_pass_preserves_symbolic_inputs(self): + """Parameter sharding leaves dynamic user metadata and guards intact.""" + model = torch.nn.Linear(4, 4) + joint = trace_model_graph(model, lambda m, x: m(x).square().mean(), + {"x": torch.randn(3, 4)}, dynamic_arg_dims={"x": 0}) + with patch("hyper_parallel.compile.passes.parallel.fsdp_pass.dist") as distributed, \ + patch("hyper_parallel.compile.passes.parallel.fsdp_pass._resolve_process_group"): + distributed.is_initialized.return_value = True + distributed.get_world_size.return_value = 2 + distributed.get_rank.return_value = 0 + FSDPPass().run(joint.graph_module, PassConfig(fsdp_degree=2), model=model, fsdp_group_name="fsdp") + joint.graph_module.graph.lint() + self.assertEqual(tuple(model.weight.shape), (2, 4)) + joint.input_guards.refresh() + joint.input_guards.validate([torch.randn(7, 4)]) + self.assertTrue(any("all_gather" in str(node.target) for node in joint.graph_module.graph.nodes)) + + @arg_mark(plat_marks=["cpu_linux", "cpu_macos"], level_mark="level0", + card_mark="allcards", essential_mark="essential") + def test_basetrainer_inherits_dynamic_config(self): + """ + Feature: Dynamic joint-graph training. + Description: Existing BaseTrainer construction and step paths pass the new configuration through. + Expectation: Losses, gradients and public contracts remain correct. + """ + class KeywordModel(torch.nn.Module): + def __init__(self) -> None: + """Create a keyword-compatible linear model.""" + super().__init__() + self.linear = torch.nn.Linear(4, 3) + + def forward(self, input_ids: torch.Tensor, use_cache: bool = False) -> torch.Tensor: + """Apply the model to a variable-shape input.""" + return self.linear(input_ids) + + base = BaseTrainer.__new__(BaseTrainer) + base.model_integration = Mock(spec=["record_batch"]) + base.config = SimpleNamespace( + compile=CompileConfig(enabled=True, use_joint_graph=True, + dynamic_arg_dims={"model_inputs.input_ids": [0, 1], "labels": [0, 1]}), + accelerator=SimpleNamespace(tp_size=1, sequence_parallel=False, loss_parallel=False), + fsdp_config=SimpleNamespace(dp_shard_size=1, edp_shard_size=1), + training=SimpleNamespace(empty_cache_before_backward=False), + ) + base.model, base.device, base.mesh, base.state = KeywordModel(), torch.device("cpu"), None, None + base.model_fwd_context = nullcontext() + + def postforward(outputs: torch.Tensor, labels: torch.Tensor) -> tuple: + """Build the scalar loss and named diagnostics.""" + loss = ((outputs - labels) ** 2).mean() + return loss, {"main": loss.detach()} + + base.postforward = postforward + base._build_graph_compiler() + with patch.object(base.graph_compiler, "compile", wraps=base.graph_compiler.compile) as compile_spy: + for batch, sequence in [(2, 5), (3, 7)]: + x, y = torch.randn(batch, sequence, 4), torch.randn(batch, sequence, 3) + base.model.zero_grad() + actual, losses = base.forward_backward_step({"input_ids": x}, loss_inputs={"labels": y}) + expected, _ = postforward(base.model(x), y) + torch.testing.assert_close(actual, expected) + torch.testing.assert_close(losses["main"], expected.detach()) + gradients = torch.autograd.grad(expected, tuple(base.model.parameters())) + for parameter, gradient in zip(base.model.parameters(), gradients): + torch.testing.assert_close(parameter.grad, gradient) + self.assertEqual(compile_spy.call_count, 1) + + + @arg_mark(plat_marks=["cpu_linux", "cpu_macos"], level_mark="level0", + card_mark="allcards", essential_mark="essential") + def test_basetrainer_token_weights_are_runtime_inputs(self): + """ + Feature: Dynamic joint-graph training. + Description: Changing token counts updates the loss and gradients without retracing. + Expectation: Losses, gradients and public contracts remain correct. + """ + class KeywordModel(torch.nn.Linear): + def forward(self, input_ids: torch.Tensor, use_cache: bool = False) -> torch.Tensor: + """Apply the model to a variable-shape input.""" + return super().forward(input_ids) + + base = BaseTrainer.__new__(BaseTrainer) + base.model_integration = Mock(spec=["record_batch"]) + base.config = SimpleNamespace(training=SimpleNamespace(empty_cache_before_backward=False)) + base.model = KeywordModel(4, 3) + base.device, base.state, base.mesh = torch.device("cpu"), None, None + base.model_fwd_context = nullcontext() + base.loss_fn = lambda model_output, labels: (model_output - labels).square().mean() + base.graph_compiler = GraphCompiler(base.model, base._graph_train_fn, + pass_config=self.config, dynamic=True) + + def aggregate(loss: torch.Tensor, current: dict, step: dict, device_mesh: object) -> dict: + """Weight the current loss by live token counts.""" + return {"foundation_loss": loss * current["foundation_tokens"] / step["foundation_tokens"]} + + with patch("hyper_parallel.trainer.base.mean_global_loss", side_effect=aggregate): + with patch.object(base.graph_compiler, "compile", wraps=base.graph_compiler.compile) as compile_spy: + for batch, count in [(2, 2), (2, 6), (3, 3)]: + x, y = torch.randn(batch, 4), torch.randn(batch, 3) + base.current_token_counts = {"foundation_tokens": torch.tensor(count)} + base.step_token_counts = {"foundation_tokens": torch.tensor(8)} + base.model.zero_grad() + actual, losses = base.forward_backward_step({"input_ids": x}, loss_inputs={"labels": y}) + expected = (base.model(x) - y).square().mean() * count / 8 + torch.testing.assert_close(actual, expected) + torch.testing.assert_close(losses["foundation_loss"], expected) + gradients = torch.autograd.grad(expected, tuple(base.model.parameters())) + for parameter, gradient in zip(base.model.parameters(), gradients): + torch.testing.assert_close(parameter.grad, gradient) + self.assertEqual(compile_spy.call_count, 1) + + + @arg_mark(plat_marks=["cpu_linux", "cpu_macos"], level_mark="level0", + card_mark="allcards", essential_mark="essential") + def test_causal_label_padding_preserves_dynamic_length(self): + """ + Feature: Symbolic graph shape contracts. + Description: The standard shifted-label CE path must not specialize token length. + Expectation: Dynamic dimensions reuse graphs and static dimensions retain guards. + """ + model = torch.nn.Sequential(torch.nn.Embedding(32, 8), torch.nn.Linear(8, 32)) + + def causal_loss(module: torch.nn.Module, x: torch.Tensor, y: torch.Tensor) -> torch.Tensor: + """Compute cross entropy with dynamically shifted labels.""" + shifted = torch.nn.functional.pad(y, (0, 1), value=-100)[..., 1:].contiguous() + return torch.nn.functional.cross_entropy(module(x).flatten(0, 1), shifted.flatten()) + + compiler = GraphCompiler(model, causal_loss, pass_config=self.config, dynamic=True) + for length in (5, 7, 11): + x, y = (torch.randint(0, 32, (1, length)) for _ in range(2)) + model.zero_grad() + actual, _ = compiler.forward_backward(x=x, y=y) + expected = causal_loss(model, x, y) + torch.testing.assert_close(actual, expected) + gradients = torch.autograd.grad(expected, tuple(model.parameters())) + for parameter, gradient in zip(model.parameters(), gradients): + torch.testing.assert_close(parameter.grad, gradient) + self.assertFalse(any(node.target == torch.ops.aten.constant_pad_nd.default + for node in compiler._joint_graph.graph_module.graph.nodes)) diff --git a/tests/ut/compile/test_size_specialization.py b/tests/ut/compile/test_size_specialization.py new file mode 100644 index 0000000000..298278e87f --- /dev/null +++ b/tests/ut/compile/test_size_specialization.py @@ -0,0 +1,214 @@ +# Copyright 2026 Huawei Technologies Co., Ltd +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# ============================================================================ +"""Numerical and dispatch regressions for lazy concrete-size FX variants.""" + +import gc +import unittest +import weakref +from types import SimpleNamespace +from unittest.mock import patch + +import torch + +from hyper_parallel.compile import GraphCompiler, PassConfig +from hyper_parallel.models.build_options import CompileConfig +from tests.common.mark_utils import arg_mark + + +def _loss(model, x, y, weight): + loss = (model(x) - y).square().mean() * weight + return loss, {"mse": loss.detach()} + + +class TestSizeSpecialization(unittest.TestCase): + """Exercise real generated variants and general fallback on CPU.""" + + def setUp(self) -> None: + """Create a CPU model for each independent dispatch test.""" + torch.manual_seed(42) + self.model = torch.nn.Linear(4, 3) + + def _compiler(self, **kwargs): + options = {"dynamic_arg_dims": {"x": [0, 1], "y": [0, 1]}, "compile_sizes": [7], + "compile_size_input": "x", "compile_size_dim": 1} + options.update(kwargs) + return GraphCompiler(self.model, _loss, pass_config=PassConfig(fsdp_enabled=False), + device=torch.device("cpu"), **options) + + def _step(self, compiler, batch, length, weight=1.0): + x, y = torch.randn(batch, length, 4), torch.randn(batch, length, 3) + scale = torch.tensor(weight) + self.model.zero_grad() + actual, losses = compiler.forward_backward(x=x, y=y, weight=scale) + expected, _ = _loss(self.model, x, y, scale) + torch.testing.assert_close(actual, expected) + torch.testing.assert_close(losses["mse"], expected.detach()) + grads = torch.autograd.grad(expected, tuple(self.model.parameters())) + for parameter, gradient in zip(self.model.parameters(), grads): + torch.testing.assert_close(parameter.grad, gradient) + + @arg_mark(plat_marks=["cpu_linux", "cpu_macos"], level_mark="level0", + card_mark="allcards", essential_mark="essential") + def test_lazy_compile_hit_and_general_fallback(self) -> None: + """ + Feature: Lazy FX specialization across multiple configured sizes. + Description: Exercise warmup, two hot sizes, repeated hits and cold-size fallback. + Expectation: One model capture yields two variants with eager loss and gradient parity. + """ + compiler = self._compiler(compile_sizes=[5, 7]) + with patch.object(compiler, "compile", wraps=compiler.compile) as capture: + self._step(compiler, 2, 5) + general_code = compiler._joint_graph.graph_module.code + self.assertEqual(compiler.specialization_stats["compilations"], 0) + self._step(compiler, 2, 7) + entry = next(iter(compiler._size_dispatcher.entries.values())) + self.assertGreater(entry.folded_nodes, 0) + self.assertNotIn("sym_size", entry.graph_module.code) + self.assertNotEqual(entry.graph_module.code, general_code) + self._step(compiler, 2, 7, weight=0.25) + self._step(compiler, 2, 9) + self._step(compiler, 2, 5) + self._step(compiler, 2, 5, weight=0.5) + self.assertEqual(capture.call_count, 1) + stats = compiler.specialization_stats + self.assertEqual(stats["compilations"], 2) + self.assertEqual(stats["compiled_sizes"], [5, 7]) + self.assertEqual((stats["general_calls"], stats["specialized_calls"], stats["cache_hits"]), (2, 4, 2)) + self.assertEqual(compiler._joint_graph.graph_module.code, general_code) + + def test_first_hot_call_runs_general(self): + """The initial execution follows MagiCompiler's general-graph warmup policy.""" + compiler = self._compiler() + self._step(compiler, 2, 7) + self.assertEqual(compiler.specialization_stats["compilations"], 0) + self._step(compiler, 2, 7) + self.assertEqual(compiler.specialization_stats["compilations"], 1) + + def test_secondary_dimensions_and_capacity(self): + """Equal dispatch sizes with different batch dimensions require distinct variants.""" + compiler = self._compiler(max_specializations=2) + for batch, length in [(2, 5), (2, 7), (3, 7), (4, 7), (2, 7)]: + self._step(compiler, batch, length) + stats = compiler.specialization_stats + self.assertEqual(stats["compilations"], 2) + self.assertEqual(stats["capacity_fallbacks"], 1) + self.assertEqual(stats["cache_hits"], 1) + + def test_live_weights_optimizer_and_accumulation(self): + """Cached variants consume updated weights and accumulate gradients normally.""" + compiler = self._compiler() + optimizer = torch.optim.SGD(self.model.parameters(), lr=0.1) + for length in (5, 7, 7, 9, 7): + self._step(compiler, 2, length) + optimizer.step() + x, y, weight = torch.randn(2, 7, 4), torch.randn(2, 7, 3), torch.tensor(0.5) + self.model.zero_grad() + expected, _ = _loss(self.model, x, y, weight) + gradients = torch.autograd.grad(expected, tuple(self.model.parameters())) + for _ in range(2): + compiler.forward_backward(x=x, y=y, weight=weight) + for parameter, gradient in zip(self.model.parameters(), gradients): + torch.testing.assert_close(parameter.grad, gradient * 2) + + def test_guards_run_before_cached_dispatch(self): + """Neither a cache hit nor a configured size may bypass general guards.""" + compiler = self._compiler() + self._step(compiler, 2, 5) + self._step(compiler, 2, 7) + stats = compiler.specialization_stats + with self.assertRaisesRegex(ValueError, "rank, dtype"): + compiler.forward_backward(x=torch.randn(2, 7, 4).double(), y=torch.randn(2, 7, 3), + weight=torch.tensor(1.0)) + with self.assertRaisesRegex(ValueError, "shape/stride"): + compiler.forward_backward(x=torch.randn(2, 7, 4), y=torch.randn(2, 8, 3), weight=torch.tensor(1.0)) + self.assertEqual(compiler.specialization_stats, stats) + + def test_auto_selection_and_nested_explicit_path(self): + """Automatic selection uses a symbolic axis; explicit selection supports pytree paths.""" + compiler = self._compiler(compile_size_input=None, compile_sizes=[3]) + self._step(compiler, 2, 5) + self._step(compiler, 3, 5) + self.assertEqual(compiler.specialization_stats["compiled_sizes"], [3]) + nested = GraphCompiler(self.model, lambda model, batch: model(batch["x"]).square().mean(), + pass_config=PassConfig(fsdp_enabled=False), dynamic=True, + compile_sizes=[7], compile_size_input="batch.x", compile_size_dim=-2) + nested.forward_backward(batch={"x": torch.randn(2, 5, 4)}) + nested.forward_backward(batch={"x": torch.randn(2, 7, 4)}) + self.assertEqual(nested.specialization_stats["compiled_sizes"], [7]) + + def test_storage_not_retained(self): + """The variant cache contains metadata and graphs, never real user tensor storage.""" + compiler = self._compiler() + self._step(compiler, 2, 5) + x, y = torch.randn(2, 7, 4), torch.randn(2, 7, 3) + refs = weakref.ref(x), weakref.ref(y) + compiler.forward_backward(x=x, y=y, weight=torch.tensor(1.0)) + del x, y + gc.collect() + self.assertTrue(all(ref() is None for ref in refs)) + + @arg_mark(plat_marks=["cpu_linux", "cpu_macos"], level_mark="level0", + card_mark="allcards", essential_mark="essential") + def test_specialization_does_not_execute_rng(self) -> None: + """ + Feature: Runtime RNG semantics during size specialization. + Description: Compare random tensor losses and RNG states with eager execution. + Expectation: Variant generation consumes no RNG and tensor values remain live. + """ + def noisy_loss(model: torch.nn.Module, x: torch.Tensor) -> torch.Tensor: + """Compute a loss with runtime random tensor values.""" + return model(x + torch.randn_like(x)).square().mean() + + compiler = GraphCompiler(self.model, noisy_loss, pass_config=PassConfig(fsdp_enabled=False), + dynamic=True, compile_sizes=[7], compile_size_input="x", compile_size_dim=1) + compiler.forward_backward(x=torch.randn(2, 5, 4)) + for seed in (123, 456): + x = torch.randn(2, 7, 4) + self.model.zero_grad() + torch.manual_seed(seed) + actual, _ = compiler.forward_backward(x=x) + state_after_graph = torch.get_rng_state() + torch.manual_seed(seed) + expected = noisy_loss(self.model, x) + torch.testing.assert_close(actual, expected) + torch.testing.assert_close(torch.get_rng_state(), state_after_graph) + gradients = torch.autograd.grad(expected, tuple(self.model.parameters())) + for parameter, gradient in zip(self.model.parameters(), gradients): + torch.testing.assert_close(parameter.grad, gradient) + + def test_configuration_inheritance_and_disabled_default(self): + """Trainer-level configuration reaches the compiler without changing defaults.""" + config = CompileConfig(dynamic=True, compile_sizes=[7], compile_size_input="x", compile_size_dim=1) + compiler = GraphCompiler(self.model, _loss, trainer_config=SimpleNamespace(compile=config), + pass_config=PassConfig(fsdp_enabled=False)) + self._step(compiler, 2, 5) + self._step(compiler, 2, 7) + self.assertEqual(compiler.specialization_stats["compilations"], 1) + disabled = self._compiler(compile_sizes=[]) + self._step(disabled, 2, 5) + self.assertEqual(disabled.specialization_stats, {}) + + def test_invalid_policy_and_selector(self): + """Reject invalid policy or input selection before parallel transformation.""" + for options in ({"compile_sizes": [True]}, {"compile_sizes": [-1]}, {"compile_sizes": "7"}, + {"max_specializations": 0}, {"max_specializations": True}): + with self.subTest(options=options), self.assertRaises(ValueError): + self._compiler(**options) + with self.assertRaisesRegex(ValueError, "requires dynamic"): + self._compiler(dynamic=False, dynamic_arg_dims=None) + for options in ({"compile_size_input": "missing"}, {"compile_size_dim": 5}, + {"compile_size_dim": True}): + with self.subTest(options=options), self.assertRaises(ValueError): + self._step(self._compiler(**options), 2, 5) diff --git a/tests/ut/compile/test_tracer.py b/tests/ut/compile/test_tracer.py index fc3598b4ce..d82a56bfe6 100644 --- a/tests/ut/compile/test_tracer.py +++ b/tests/ut/compile/test_tracer.py @@ -46,6 +46,7 @@ run_traced_graph, trace_model_graph, ) +from tests.common.mark_utils import arg_mark class _LinearWithBuffer(nn.Module): @@ -217,22 +218,24 @@ def forward(self, x: torch.Tensor) -> torch.Tensor: class TestTiedWeights(unittest.TestCase): """Tied parameters trace to one placeholder and one undoubled gradient.""" + @arg_mark(plat_marks=["cpu_linux", "cpu_macos"], level_mark="level0", + card_mark="allcards", essential_mark="essential") def test_tied_weight_one_placeholder_and_full_gradient(self): - """Test the deduped alias is one static input with the full gradient. - - Without dedup the same tensor is flattened once per FQN, so the graph - carries two placeholders and FSDP later inserts two collectives; and a - naive ``remove_duplicate=True`` (without ``tie_weights``) would bake the - second use in as a ``get_attr`` constant instead. The tracer must land - in the middle: one placeholder, no ``get_attr``, and the shared weight's - gradient equal to autograd's full (not doubled) gradient. + """ + Feature: Shared parameters in joint graphs. + Description: Trace tied embedding/projection weights as one static input. + Expectation: The shared gradient matches eager without get_attr constants. + + Flattening aliases twice inserts duplicate placeholders and collectives. + Deduplication with tie_weights preserves both uses of the same input and + yields one gradient containing the full contribution of both modules. """ torch.manual_seed(0) model = _TiedEmbedding() x = torch.randint(0, 10, (2, 3)) y = torch.randn(2, 3, 10) - joint, loss, grads = _trace_and_run( + joint, loss, grads, _ = _trace_and_run( model, lambda m, x, y: ((m(x) - y) ** 2).mean(), x, y ) gm = joint.graph_module diff --git a/tests/ut/trainer/test_config_overrides.py b/tests/ut/trainer/test_config_overrides.py index ecf77c145b..9cc9342eba 100644 --- a/tests/ut/trainer/test_config_overrides.py +++ b/tests/ut/trainer/test_config_overrides.py @@ -183,6 +183,16 @@ def test_invalid_yaml_value(self) -> None: ): self._parse("--model.width=[unclosed") + def test_dynamic_joint_graph_mapping(self) -> None: + """Typed configuration accepts nested tensor paths and both dimension forms.""" + config = self._parse( + "--compile.enabled=true", + "--compile.use_joint_graph=true", + "--compile.dynamic_arg_dims={model_inputs.input_ids: [0, 1], labels: 0}", + ) + self.assertTrue(config.compile.selects_graph_compiler()) + self.assertEqual(config.compile.dynamic_arg_dims, {"model_inputs.input_ids": [0, 1], "labels": 0}) + if __name__ == "__main__": unittest.main()