Skip to content

feat(data): add PackedTensor preprocessing modes - #4106

Open
rohitrango wants to merge 1 commit into
rohit/sft_v2_stage2from
rohit/packedtensor_padding
Open

rohitrango wants to merge 1 commit into
rohit/sft_v2_stage2from
rohit/packedtensor_padding

Conversation

@rohitrango

Copy link
Copy Markdown
Contributor

What does this PR do ?

Adds configurable PackedTensor preprocessing and patchifies Nemotron Omni pixels before materialization.

  • Replaces the top-level pad_to_max_shape boolean with preprocess_mode and preprocess_kwargs.
  • Adds native-resolution patchification into per-image (C_i, P²) blocks, packs them on dimension 0, and returns (1, total_C, P²).
  • Carries preprocessing settings through slicing, concatenation, wire transport, and materialization.
  • Enables patchification with patch size 16 for supported Nemotron Omni processors.
  • Adds NemotronH_Omni_Reasoning_V3Processor to the placeholder-style processor set.

Issues

None.

Usage

PackedTensor(
    pixel_values,
    dim_to_pack=0,
    preprocess_mode="patchify",
    preprocess_kwargs={"patch_dim": 16},
)

Before your PR is "Ready for review"

  • Read and followed the contributor guidelines.
  • Added or updated unit coverage for preprocessing and patchification.
  • Full pre-commit lint suite passed on Slurm job 18281365 (Ruff lint, import sorting, Ruff format, Pyrefly, and config checks).
  • Affected unit suite passed on Slurm job 18281365: 90 passed, 1 skipped (megatron.bridge unavailable).
  • GRPO and SFT GPU recipe validation passed on Slurm job 18281365.
  • Updated relevant inline documentation.

Additional Information

Scripts validated

uv run examples/run_vlm_grpo.py --config examples/configs/recipes/vlm/vlm_grpo-nemotron-omni-30ba3b-clevr-2n8g-megatron-tp8ep8.v1-tq_mooncake.yaml logger.wandb.name=grpo-nemotron-with-patches logger.wandb_enabled=true logger.wandb.project=sft-dev cluster.num_nodes=1

image
uv run examples/run_sft_v2.py --config examples/configs/recipes/vlm/vlm_sft-nemotron-omni-30ba3b-clevr-1n8g-megatron-tp8ep8-energon.v1.packing.yaml logger.wandb.name=sft-nemotron-with-patches logger.wandb_enabled=true logger.wandb.project=sft-dev sft.max_steps=50

(brown: baseline, green: with patches)
image

Previous PR and self-review

Previous PR: #4082. A self-review was performed on the previous PR, and the resulting feedback was addressed in this PR.

@copy-pr-bot

copy-pr-bot Bot commented Sep 12, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@rohitrango
rohitrango added this pull request to stack #4109 September 12, 2026 00:15
@rohitrango
rohitrango marked this pull request as ready for review September 12, 2026 00:15
@rohitrango
rohitrango requested review from a team as code owners September 12, 2026 00:15
@copy-pr-bot

copy-pr-bot Bot commented Sep 12, 2026

Copy link
Copy Markdown

Auto-sync is disabled for ready for review pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

Signed-off-by: rohitrango <rohit.rango@gmail.com>
@rohitrango
rohitrango force-pushed the rohit/packedtensor_padding branch from 977f0f3 to 3b6df1a Compare September 15, 2026 17:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants