Add composable world-model pipelines and Cosmos3 export - #466
Draft
xiaoyu-work wants to merge 10 commits into
Draft
Add composable world-model pipelines and Cosmos3 export#466xiaoyu-work wants to merge 10 commits into
xiaoyu-work wants to merge 10 commits into
Conversation
Introduce executable pipeline schema 1.1 with typed generated-input programs, recurrent state lifecycle, scheduler and autoregressive controls, parameterized transforms, runtime assets, and dtype/EP hints. Implement complete Cosmos3 Omni and Edge exports with the shared MoT generator, Wan video VAE, optional AVAE audio tokenizer, domain-aware action heads, real checkpoint weight routing, CLI support, documentation, and numerical tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 997ab62c-69cb-4802-a5c6-fddd51bfd583 Signed-off-by: Xiaoyu Zhang <xiaoyuzhang@microsoft.com>
|
|
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
Focus the world-model guides on export commands, package contents, supported checkpoints, runtime usage, and practical options. Remove internal architecture and manifest implementation details from the user guide. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 997ab62c-69cb-4802-a5c6-fddd51bfd583 Signed-off-by: Xiaoyu Zhang <xiaoyuzhang@microsoft.com>
Performance Comparison
|
🏗️ Architecture Diff
No architecture changes detected. ✅ Legend: ⚪ No change · 🔵 Minor (attrs/inits) · 🟡 Moderate (nodes added/removed) · 🔴 Major (interface changed) |
Add model-agnostic build, generation, and pipeline configuration for world-model exporters, plus safe shared helpers for diffusers-style checkpoint inspection and asset resolution. Keep LLM ArchitectureConfig and every model/component implementation unchanged; Cosmos3 Omni and Edge only consume the new outer pipeline configuration. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 997ab62c-69cb-4802-a5c6-fddd51bfd583 Signed-off-by: Xiaoyu Zhang <xiaoyuzhang@microsoft.com>
Match upstream chunk-zero temporal resampling so Cosmos3 can decode a single latent frame without running Conv on an empty temporal extent. Preserve bit-identical multi-frame behavior and add ORT/PyTorch parity coverage. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 997ab62c-69cb-4802-a5c6-fddd51bfd583 Signed-off-by: Xiaoyu Zhang <xiaoyuzhang@microsoft.com>
Replace the fixed SigLIP1-style vision graph with the published packed variable-resolution SigLIP2 contract, correct block-major projection ordering, and use interleaved multimodal RoPE. Add image/video feature routing, processor metadata, real-checkpoint numerical parity tests, and image/video understanding coverage while preserving text-only behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 997ab62c-69cb-4802-a5c6-fddd51bfd583 Signed-off-by: Xiaoyu Zhang <xiaoyuzhang@microsoft.com>
Describe conditioned video encoding, classifier-free guidance, prompt packing, and official Cosmos3 Edge generation recipes in pipeline.json. Ship the recommended negative prompt when available and preserve checkpoint-specific action behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 997ab62c-69cb-4802-a5c6-fddd51bfd583 Signed-off-by: Xiaoyu Zhang <xiaoyuzhang@microsoft.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Apply the repository Ruff formatter to resolve CI formatting findings. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 997ab62c-69cb-4802-a5c6-fddd51bfd583 Signed-off-by: Xiaoyu Zhang <xiaoyuzhang@microsoft.com>
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com> Signed-off-by: Xiaoyu <85524621+xiaoyu-work@users.noreply.github.com>
Fix the Ruff unused-import failure after the latest world-model cleanup. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 997ab62c-69cb-4802-a5c6-fddd51bfd583 Signed-off-by: Xiaoyu Zhang <xiaoyuzhang@microsoft.com>
Contributor
Author
|
@copilot resolve the merge conflicts in this pull request |
Resolve conflicts between the world-model/Cosmos3 export work and main: - --features: keep world-model alongside main's prune-lm-head -> prune-prefill-prefix rename (CLI, README, CHANGELOG, cli_reference). - _diffusers_builder: keep both the component allowlist and pipeline component task overrides from main and the EP/trace-optimization plumbing from this branch, plus the corresponding tests. - components/tasks __init__: export main's new components/tasks together with the Cosmos3 Edge vision and Cosmos3 audio/generator additions; drop the projector removed by this branch. - _weight_loading: thread main's revision pin through the new iter_weight_shards generator. - _sub_configs: drop duplicated Cosmos3-Edge vision token fields that main added independently. - wan_vae_test: expect the fused Swish op introduced by main. Signed-off-by: GitHub <noreply@github.com> Co-authored-by: xiaoyu-work <85524621+xiaoyu-work@users.noreply.github.com>
Contributor
Merged
Also fixed two silent (non-conflicting) merge breakages:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Introduce executable pipeline schema 1.1 with typed generated-input programs, recurrent state lifecycle, scheduler and autoregressive controls, parameterized transforms, runtime assets, and dtype/EP hints.
Implement complete Cosmos3 Omni and Edge exports with the shared MoT generator, Wan video VAE, optional AVAE audio tokenizer, domain-aware action heads, real checkpoint weight routing, CLI support, documentation, and numerical tests.