English | 简体中文
HyperParallel model training parses YAML into TrainerConfig, then a task Trainer constructs runtime objects. See YAML Trainer Configuration for Target and CLI dotted override parsing rules.
| Requirement | Recommended entry point | Modify Trainer? |
|---|---|---|
| Adjust existing component parameters | Edit YAML fields or use CLI overrides | No |
| Replace model submodules or add parallel plans | Configure matching rules, replacement functions, or sharding contracts in plan_overrides |
No |
| Add model construction, data processing, losses, optimizers, or learning rate policies | Implement a Python class or function and reference it through _target_ |
No |
| Support new modality assets, model input protocols, or training loops | Add a task Trainer reusing shared BaseTrainer phases |
Yes |
A _target_ in plan_overrides modifies model internals or computation; a component node's _target_ constructs a model, data, loss, optimizer, or scheduler object used directly by the Trainer.
Edit the training YAML or launch command.
training:
train_iters: 10The same field can be overridden through the CLI:
torchrun --standalone --nproc_per_node=8 \
scripts/train_lm.py hyper_parallel/models/qwen3_moe/recipes/train.yaml \
--training.train_iters=10Field paths start at the TrainerConfig root. Parsing fails if a field does not exist or its value cannot be converted to the declared type.
Configure module replacements through plan_overrides.replace; see the PlanOverride definition for all fields.
plan_overrides:
- match: "*.language_model.layers.*.input_layernorm"
replace:
kind: module
implementation: my_project.modules.replace_rms_normYAML parsing produces list[PlanOverride]. The Trainer normalizes this into replacement rules (§15.2.6), then _apply_module_replacement_actions() applies them. Integration requires:
- Each path pattern in
matchmust match at least one model submodule; the framework obtains the actual type from matched objects. Setreplace.source_typeonly when an additional expected-type assertion is needed, optionally withexact_type: true. replace.implementationpoints to a replacement factory importable in the training environment. Supply static arguments with a{_target_: path, ...}mapping.- Decorate the factory with
@module_replacement, declaremodule,module_fqn, andcontextparameters, and return atorch.nn.Module.
See RMSNorm for a complete implementation.
Model-family adapters are organized by responsibility: module/weight conversion in adapter/conversion/; data encoding, processors,
transforms, and runtime inputs in adapter/data/; model forward contracts in adapter/runtime/; CP/EP
implementations in adapter/distributed/; parallel and recomputation policies in adapter/policies/; and acceptance declarations in
adapter/validation/. Do not create empty directories for responsibilities a model does not have. YAML references concrete implementations directly through _target_.
The sole training entry point is models/<family>/recipes/train.yaml.
Developers can implement components externally and reference them through _target_ in training YAML without modifying framework source code.
TrainerConfig.loss_fn is an optional Target. BaseTrainer._build_loss() selects the loss module based on whether this field is configured:
- When
loss_fnis configured,config.loss_fn.build()constructs the specified implementation. - Otherwise, the default
ModelOutputLossreads the loss frommodel_output.loss.
In either case, the constructed object must be a torch.nn.Module and is stored as self.loss_fn. During training, BaseTrainer.postforward() invokes it as follows:
self.loss_fn(model_output=outputs, labels=labels)An ordinary custom post-forward loss_fn has three requirements:
_target_constructs atorch.nn.Module.forward()accepts the keyword argumentsmodel_outputandlabels.forward()returnstorch.Tensorordict[str, torch.Tensor].
Losses that intervene before complete model outputs are generated, such as Chunk Loss, which avoids full logits, may implement two optional lifecycle interfaces:
bind_model(model, distributed_setup): Called once after loss construction to bind the model-family adapter after FSDP has been applied;prepare_model_inputs(model_inputs, loss_inputs): Called before each micro-batch to convert the get-batch runtime's separate loss-only fields into the model-integrated loss protocol.
Trainer behavior is unchanged for ordinary losses that do not implement these methods.
For example, in my_project/losses.py:
from typing import Any
import torch
class ScaledModelOutputLoss(torch.nn.Module):
def __init__(self, scale: float = 1.0) -> None:
super().__init__()
self.scale = scale
def forward(
self,
*,
model_output: Any,
labels: torch.Tensor | None,
) -> torch.Tensor:
del labels
return model_output.loss * self.scaleConfigure the implementation in training YAML:
loss_fn:
_target_: my_project.losses.ScaledModelOutputLoss
scale: 0.5TrainerConfig.optimizer is a required Target. BaseTrainer._build_optimizer() calls config.optimizer.build(model=self.model) to construct the configured optimizer component, supplying the already-built model.
After target construction, the Trainer calls get_optimizer() and stores its result as self.optimizer.
A custom optimizer target therefore has two requirements:
- Accept the
modelkeyword argument. - Provide
get_optimizer()on the constructed object, returning the optimizer used by the training loop.
For example, in my_project/optimizers.py:
import torch
class SGDOptimizer:
def __init__(self, lr: float, *, model: torch.nn.Module) -> None:
self.optimizer = torch.optim.SGD(model.parameters(), lr=lr)
def get_optimizer(self) -> torch.optim.Optimizer:
return self.optimizerConfigure the implementation in training YAML:
optimizer:
_target_: my_project.optimizers.SGDOptimizer
lr: 1.0e-3See AdamW for a complete implementation.
| Configuration field | Construction site | Runtime arguments supplied by the Trainer |
|---|---|---|
model |
BaseTrainer._build_model() |
distributed_setup, peft_config, activation_checkpoint, activation_swap, compile_config |
dataset |
BaseTrainer._build_dataset() |
transform, tokenizer, mesh_context, training_config |
dataloader |
BaseTrainer._build_dataloader() → build_dataloader() |
dataset, constructed collate_fn, batch_sampler, batch_size, dp_world_size, max_seq_len, seed |
optimizer |
BaseTrainer._build_optimizer() |
model |
lr_scheduler |
BaseTrainer._build_lr_scheduler() |
optimizer, train_iters |
loss_fn |
BaseTrainer._build_loss() |
None during construction; optional lifecycle hooks receive the model, parallel configuration, and loss inputs |
See HyperParallel Model Training for other component configurations and tutorials.
A new training task typically requires changes in two places:
- Configuration: Add a root field and its dataclass to
TrainerConfigonly when the task needs an independent configuration section. No new root field is needed if the settings fit existingmodel,dataset,dataloader, ortrainingsections. - Task Trainer: Add
<task>_trainer.pyunderhyper_parallel/trainer/, reuse shared construction phases fromBaseTrainer, and implement the task's model assets, sample transformation, collator,get_batch, and training steps.
For example, VLMTrainer reuses TrainerConfig and implements these operations directly in the task Trainer:
_build_model_assets(): Build the processor from the model path and obtain its tokenizer._build_data_transform()and_build_collate_fn(): Pass the processor to the multimodal sample transformation and build the multimodal collator._build_get_batch(): Createparallel_contextfrom the parallel mesh, then build the get-batch runtime that converts DataLoader output into model inputs.train_step(): Obtainmodel_inputsandloss_inputsfromget_batchfor model forward/backward and loss token statistics respectively.