Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
241 changes: 241 additions & 0 deletions dynamic_shape_development.md

Large diffs are not rendered by default.

123 changes: 123 additions & 0 deletions hyper_parallel/compile/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -122,6 +122,129 @@ trainer = GraphTrainer(
trainer.train(dataloader, max_steps=1000)
```

## Dynamic input shapes

Static tracing remains the default. Enable symbolic user inputs with one option:

```python
compiler = GraphCompiler(model=model, train_fn=train_fn, dynamic=True)
```

For precise control, pass `dynamic_arg_dims` to `GraphCompiler` or `GraphTrainer`:

```python
compiler = GraphCompiler(
model=model,
train_fn=lambda m, x, y: ((m(x) - y) ** 2).mean(),
dynamic_arg_dims={"x": [0, 1], "y": [0, 1]},
)
# x: [batch, sequence, input_features]; y: [batch, sequence, output_features]
loss, loss_dict = compiler.forward_backward(x=x, y=y)
```

The first call captures one symbolic forward/backward graph. Later batches with
compatible batch/sequence lengths reuse it; gradients still accumulate into the
live parameters. `GraphTrainer` exposes the same options and owns optimizer steps.
Numerical and graph-reuse regressions live in
[the CPU unit tests](../../tests/ut/compile/test_dynamic_shapes.py).

For the existing **BaseTrainer + GraphCompiler** path, configure:

```yaml
compile:
enabled: true
use_joint_graph: true
dynamic_arg_dims:
model_inputs.input_ids: [0, 1]
labels: [0, 1]
```

Add paths for other tensors whose sizes vary, such as
`model_inputs.attention_mask: [0, 1]`. Paths refer to the keyword arguments of
`train_fn`; BaseTrainer supplies `model_inputs` and `labels`. Use `dynamic: true`
instead of the mapping to automatically symbolize all user tensor dimensions.

- A mapping takes precedence over `dynamic`; unspecified dimensions stay static.
An empty mapping selects no dynamic dimensions. Negative dimension indices and
nested dictionary/list paths (`batch.items.0`) are supported. Registered pytree
objects can use attribute paths. Invalid paths or dimensions raise `ValueError`.
- Parameters and buffers always keep concrete shapes, including with automatic
input symbolization. No Dynamo marks are added to the caller's tensors.
- The tracer uses `ShapeEnv` and per-tensor symbolic contexts directly, because
this path captures through `make_fx`, not Dynamo. Runtime checks enforce traced
size/stride relationships, input metadata, tensor identity aliases and Python
constants before executing the joint graph. Shape-dependent Python branches
are guarded; they do not become arbitrary runtime control flow.
- Use a representative first batch with varying dimensions **greater than 1**.
PyTorch specializes dimensions of size 0/1 and may constrain dimensions used
by operators or Python control flow. Inputs outside these constraints raise
an explanatory error. They are not automatically recompiled: FSDP compilation
mutates live parameter shards, so blindly retracing would be unsafe.
- First-stage scope: symbolic FX execution, loss and parameter gradients,
gradient accumulation and optimizer updates. CPU unit tests cover the
BaseTrainer entry and FSDP graph rewriting with mocked communication.
The [NPU text example](examples/automodel_text_graph/README.md)
exercises dynamic token packing through TP2 + FSDP2 with Qwen3-0.6B. Python scalar arguments remain
constants. Pipeline parallel with dynamic shapes is rejected before tracing.
Compiled-kernel backends, CUDA/NPU graph capture, serialized graph caches and
data-dependent output shapes are not added here.

Run the focused checks with:

```bash
python -m pytest tests/ut/compile/test_dynamic_shapes.py -q
```

The approach follows MagiCompiler's separation of dynamic and static dimensions
and reuse of a general symbolic graph, adapted to HyperParallel's joint-graph
tracer. It adds no runtime dependency on the separate MagiCompiler checkout.

## Lazy size specialization

Optionally generate concrete-size FX variants of the general joint graph:

```yaml
compile:
enabled: true
use_joint_graph: true
dynamic: true
compile_sizes: [116, 120, 127]
compile_size_input: model_inputs.input_ids
compile_size_dim: 1
max_specializations: 8
```

`GraphCompiler` and `GraphTrainer` also accept these four options directly.
`compile_sizes` is opt-in and requires dynamic mode. If `compile_size_input` is
omitted, selection uses the first symbolic axis among flattened user inputs;
`compile_size_dim` only applies to an explicit path and supports negative axes.

The first execution uses the general graph, matching MagiCompiler's warmup
policy. Subsequent configured sizes lazily generate a variant and execute it.
Repeated signatures hit the cache; unconfigured sizes use the general graph.
The cache key includes all user tensor shapes, strides, offsets and metadata,
so another batch dimension at the same sequence length creates a separate
variant. New signatures at `max_specializations` capacity use the general graph.

General guards run before every dispatch, including cache hits. Guard failures
still raise errors; specialization does not expand the valid input domain.
Variants bind pure symbolic shape expressions and regenerate FX code from the
post-pass graph, without recapturing the model or executing tensor/communication
operations during generation. Parameters, token counts and RNG remain live.
This is FX specialization, not an Inductor or NPU kernel compilation backend;
unresolved expressions remain dynamic and no speedup is promised.

Inspect `compiler.specialization_stats` for compilations, cache hits, general and
specialized calls, folded nodes and capacity fallbacks. The snapshot counts the
current compiler instance; variants are not serialized across processes.

```bash
python -m pytest tests/ut/compile/test_size_specialization.py -q
```

See [the implementation walkthrough](../../size_specialization_development.md)
and the [NPU training example](examples/automodel_text_graph/README.md).

## Key Design Decisions

1. **Static Inputs**: Parameters are graph inputs, not `get_attr`. This allows passes to split the graph by reshaping placeholders.
Expand Down
62 changes: 61 additions & 1 deletion hyper_parallel/compile/compiler.py
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,8 @@ class and owns them.
from .pass_config import PassConfig, normalize_dp_mode, build_pass_config_from_trainer_config
from .graph_parallel_plan import GraphParallelPlan
from .passes.pipeline import PassPipeline
from .size_specialization import SizeSpecializationDispatcher, validate_compile_sizes
from .tracer.dynamic_shapes import DynamicArgDims, normalize_dynamic_arg_dims
from .tracer.graph_tracer import run_traced_graph, trace_model_graph

_LOG = logging.getLogger(__name__)
Expand All @@ -65,6 +67,12 @@ def __init__(
trainer_config: Optional[Any] = None,
device: Optional[torch.device] = None,
mesh_context: Optional[Any] = None,
dynamic: Optional[bool] = None,
dynamic_arg_dims: DynamicArgDims | None = None,
compile_sizes: Optional[List[int]] = None,
compile_size_input: Optional[str] = None,
compile_size_dim: Optional[int] = None,
max_specializations: Optional[int] = None,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

换个名

) -> None:
"""
Args:
Expand All @@ -85,7 +93,35 @@ def __init__(
only the FSDP shard sub-mesh is registered under ``"fsdp"``.
Use this to feed an automodel TP-sharded model into the
graph-mode FSDP pass.
dynamic: Symbolize user input dimensions; inherits trainer_config.compile.dynamic when omitted.
dynamic_arg_dims: Optional dotted input paths to dynamic dimensions, e.g. {"x": [0, 1]}.
Overrides automatic dimension selection and inherits trainer configuration when omitted.
compile_sizes: Opt-in sizes for lazy FX specialization; inherits trainer configuration.
compile_size_input: Dotted tensor path for dispatch; defaults to the first symbolic input axis.
compile_size_dim: Dispatch axis for an explicit path (default -1).
max_specializations: Maximum cached full input signatures (default 8).
"""
compile_config = getattr(trainer_config, "compile", None)
self.dynamic = getattr(compile_config, "dynamic", False) if dynamic is None else dynamic
if not isinstance(self.dynamic, bool):
raise ValueError("dynamic must be a bool")
if dynamic_arg_dims is None:
dynamic_arg_dims = getattr(compile_config, "dynamic_arg_dims", None)
self.dynamic_arg_dims = normalize_dynamic_arg_dims(dynamic_arg_dims)
if compile_sizes is None:
compile_sizes = getattr(compile_config, "compile_sizes", None)
self.compile_size_input = (
getattr(compile_config, "compile_size_input", None) if compile_size_input is None else compile_size_input
)
self.compile_size_dim = (
getattr(compile_config, "compile_size_dim", -1) if compile_size_dim is None else compile_size_dim
)
self.max_specializations = (
getattr(compile_config, "max_specializations", 8) if max_specializations is None else max_specializations
)
self.compile_sizes = validate_compile_sizes(compile_sizes, self.max_specializations)
if self.compile_sizes and not (self.dynamic or self.dynamic_arg_dims is not None):
raise ValueError("compile_sizes requires dynamic=True or dynamic_arg_dims")
self.model = model
self.train_fn = train_fn
self.parallel_plan = parallel_plan
Expand All @@ -103,6 +139,7 @@ def __init__(
self.pass_config.validate()

self._joint_graph = None
self._size_dispatcher = None
self._pytree_pre_hook: Optional[Callable[[], None]] = None

@staticmethod
Expand All @@ -123,6 +160,11 @@ def is_compiled(self) -> bool:
"""Whether a joint graph has already been compiled."""
return self._joint_graph is not None

@property
def specialization_stats(self) -> dict[str, Any]:
"""Return lazy size-specialization counters, or an empty mapping when disabled."""
return self._size_dispatcher.stats if self._size_dispatcher is not None else {}

def compile(self, **inputs: Any) -> None:
"""
Compile model into parallel graph
Expand All @@ -134,6 +176,10 @@ def compile(self, **inputs: Any) -> None:
**inputs: Model inputs, forwarded to ``train_fn`` as keyword
arguments and used to trace the joint graph
"""
if (self.dynamic or self.dynamic_arg_dims is not None) and self.pass_config.pp_enabled:
raise ValueError(
"Dynamic shapes with pipeline parallel are not supported yet; disable PP or dynamic shapes"
)
if self._pytree_pre_hook is not None:
self._pytree_pre_hook()

Expand All @@ -144,7 +190,17 @@ def compile(self, **inputs: Any) -> None:
# runs as plain graph mode without sharding.
self._init_device_mesh(self._mesh_context)

joint_graph = trace_model_graph(self.model, self.train_fn, inputs)
trace_kwargs = {}
if self.dynamic or self.dynamic_arg_dims is not None:
trace_kwargs = {"dynamic": self.dynamic, "dynamic_arg_dims": self.dynamic_arg_dims}
joint_graph = trace_model_graph(self.model, self.train_fn, inputs, **trace_kwargs)
# Validate selection before a parallel pass mutates the live model.
size_dispatcher = None
if self.compile_sizes:
size_dispatcher = SizeSpecializationDispatcher(
joint_graph, inputs, self.compile_sizes, self.compile_size_input,
self.compile_size_dim, self.max_specializations,
)

pipeline = PassPipeline.from_config(self.pass_config, self.parallel_plan)

Expand All @@ -154,7 +210,10 @@ def compile(self, **inputs: Any) -> None:
# transformed graph lives on ``joint_graph`` for ``forward_backward``.
pipeline.run(joint_graph.graph_module, **pass_kwargs)

if joint_graph.input_guards is not None:
joint_graph.input_guards.refresh()
self._joint_graph = joint_graph
self._size_dispatcher = size_dispatcher

def forward_backward(self, **inputs: Any) -> Any:
"""
Expand Down Expand Up @@ -461,6 +520,7 @@ def _run_graph(self, **inputs):
self._joint_graph,
self.model,
inputs,
**({"graph_dispatcher": self._size_dispatcher} if self._size_dispatcher is not None else {}),
)

def _accumulate_grads(self, grads: List[torch.Tensor]) -> None:
Expand Down
71 changes: 71 additions & 0 deletions hyper_parallel/compile/examples/automodel_text_graph/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
# Dynamic token batching with graph-mode text training

This example runs the standard TextTrainer + BaseTrainer + GraphCompiler path
with TP2 + FSDP2 on four NPUs. It uses an online iterable dataset,
TokenBatchLoader and dense packed attention. The model directory provides the
model configuration and tokenizer; the text generator needs no downloaded dataset.

## Run ordinary training

From the repository root in a torch + torch_npu environment:

```bash
python -m hyper_parallel.compile.examples.automodel_text_graph.prepare_dynamic_text

ASCEND_RT_VISIBLE_DEVICES=0,1,2,3 bash \
hyper_parallel/compile/examples/automodel_text_graph/run.sh \
hyper_parallel/compile/examples/automodel_text_graph/train_online_lm_graph_dynamic_tp2_fsdp2.yaml \
--model.pretrained_model_name_or_path=/path/to/Qwen3-0.6B
```

The YAML enables `compile.dynamic: true`; manual dimension annotations are
optional. Packing combines variable numbers of records into `[1, packed_length]`,
so the sequence dimension changes while the physical batch dimension remains one.
The iterable dataset avoids limiting raw records before token packing.
Replace `dataset.data_path` to use another plaintext source.

This functional smoke run uses random initialization (`model.load_base_model: false`)
and FP32 model parameters. Graph FSDP shards parameters on first capture;
separate optimizer master copies constructed earlier can retain the full shape.
That optimizer combination is not exercised by this example.

`run.sh` defaults to `scripts/train_lm.py`. It accepts a YAML path followed by
existing CLI overrides. Set `PYTHON=/path/to/venv/bin/python` to select the
interpreter, and use `MASTER_PORT`, `NPROC_PER_NODE` and `OUTPUT_DIR` for launcher
settings. The default YAML remains the indexed/static example. The output is a
normal training log; this entry does not export a separate per-rank audit JSON.

## Optional size specialization

Add these overrides to the dynamic training command:

```bash
'--compile.compile_sizes=[116,120,127]' \
--compile.compile_size_input=model_inputs.input_ids \
--compile.compile_size_dim=1
```

The first execution uses the general graph. Later configured sizes generate an
FX variant once per complete input metadata signature and reuse it. Other sizes,
or new signatures at the cache limit, continue using the general graph.
`compile.max_specializations` defaults to 8. Omit `compile_sizes` to disable variants.
A different tokenizer or dataset can change which packed lengths recur.

Generation binds shape expressions in the post-pass FX graph. It does not
recapture the model, shard parameters again, execute collectives during generation
or invoke an Inductor/NPU kernel compiler. Tensor values, token counts and model
weights remain runtime inputs. This is a functionality example, not a throughput
or convergence benchmark.

## Regression tests and implementation notes

Small-model shape/gradient and dispatch checks live in the existing CPU UT:

```bash
python -m pytest tests/ut/compile/test_dynamic_shapes.py \
tests/ut/compile/test_size_specialization.py -q
```

These tests check eager loss/gradient parity, graph reuse, guards and multiple hot
sizes. CPU UT and mocked communication do not substitute for real NPU training.
Dynamic PP and arbitrary data-dependent shapes remain outside the supported path.
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
# Copyright 2026 Huawei Technologies Co., Ltd
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ============================================================================
"""Prepare deterministic plaintext records for dynamic token-budget batching."""

import argparse
import json
from pathlib import Path


def main() -> None:
"""Write locally generated text of several lengths; no download is required."""
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--output", default="output/automodel_text_graph/dynamic_text.jsonl")
args = parser.parse_args()
output = Path(args.output)
output.parent.mkdir(parents=True, exist_ok=True)
lengths = (3, 7, 11, 17, 23, 31, 43, 59, 73, 89, 101, 113)
records = [
{"text": "Training example " + str(index) + ": " + " ".join(
["language", "models", "learn", "from", "text"][token % 5]
for token in range(lengths[index % len(lengths)])
)}
for index in range(384)
]
output.write_text("".join(json.dumps(record) + "\n" for record in records), encoding="utf-8")
print(f"Wrote {len(records)} variable-length records to {output}")


if __name__ == "__main__":
main()
Loading
Loading