Add LFM2.5-VL multimodal support - #507
Open
justinchuby wants to merge 1 commit into
Open
Conversation
Implement the SigLIP2 NaFlex vision tower, projector, embedding fusion, and hybrid LFM2 decoder package for LiquidAI/LFM2.5-VL-3B. Pin Hugging Face revisions through export and golden workflows, emit ORT GenAI metadata, and add architecture, alignment, parity, CUDA golden, and generation coverage. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
Performance Comparison
|
Contributor
There was a problem hiding this comment.
Pull request overview
Adds first-class Mobius support for LiquidAI’s lfm2_vl multimodal family by introducing a dedicated 3-model split (vision encoder + embedding fusion + hybrid decoder), along with revision pinning across CLI/export/golden generation and ORT GenAI metadata updates.
Changes:
- Introduces
Lfm2VlForConditionalGeneration+Lfm2VlTaskto export a standardizeddecoder/vision_encoder/embeddingModelPackage forlfm2_vl. - Adds SigLIP2 NaFlex tower + LFM2-VL pixel-unshuffle projector implementation and parity tests vs Transformers.
- Threads immutable
--revisionthrough CLI, ORT GenAI auto-export/artifacts, torch reference loading, and adds L4/L5 goldens for the real checkpoint.
Reviewed changes
Copilot reviewed 28 out of 28 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| tests/weight_alignment_test.py | Adds HF key-alignment coverage for the lfm2_vl three-model initializer layout. |
| tests/e2e_golden_test.py | Extends hybrid-cache handling for conv layer types and tightens L5 length mismatch behavior when exact match is required. |
| tests/cli_test.py | Verifies --revision is propagated through CLI build plumbing (including diffusers detection). |
| tests/_test_configs.py | Adds a tiny lfm2_vl config fixture for graph construction/unit tests. |
| testdata/golden/vision-language/lfm2_5-vl-3b.json | Adds pinned L4 golden output for real-image vision-language prefill. |
| testdata/golden/vision-language/lfm2_5-vl-3b_generation.json | Adds pinned L5 deterministic generation golden (exact token sequence + text). |
| testdata/cases/vision-language/lfm2_5-vl-3b.yaml | Adds an end-to-end L4+L5 golden test case pinned to an immutable HF revision. |
| src/mobius/tasks/_vision_language_3model.py | Introduces Lfm2VlTask implementing NaFlex vision I/O + hybrid decoder cache export. |
| src/mobius/tasks/init.py | Exports and registers the new lfm2-vl task name → Lfm2VlTask. |
| src/mobius/models/lfm2.py | Refactors LFM2 config defaults + projection-key rename helpers; adds inputs_embeds path to the LFM2 backbone. |
| src/mobius/models/lfm2_vl.py | New LFM2-VL 3-model split implementation (NaFlex tower + projector + embedding fusion + hybrid decoder) and weight routing. |
| src/mobius/models/lfm2_vl_test.py | Adds config extraction, graph contract checks, HF parity tests, and weight-routing tests for LFM2-VL. |
| src/mobius/models/init.py | Exposes Lfm2VlForConditionalGeneration from the public models API. |
| src/mobius/integrations/ort_genai/genai_config.py | Enables LFM2 hybrid-cache metadata generation for lfm2_vl. |
| src/mobius/integrations/ort_genai/genai_config_test.py | Adds a targeted test asserting hybrid-cache metadata for lfm2_vl. |
| src/mobius/integrations/ort_genai/auto_export.py | Adds lfm2_vl model type support, preserves processor_config.json for LFM2-VL, and threads revision through remote fetches. |
| src/mobius/integrations/ort_genai/auto_export_test.py | Tests that auto-export pins revision for both build and artifact generation, including tokenizer/config fetching. |
| src/mobius/components/_siglip2_naflex.py | New SigLIP2 NaFlex tower (pre-patchified input, dynamic antialiased position interpolation, mask handling). |
| src/mobius/components/_siglip2_naflex_test.py | Adds parity and parameter-name tests for the SigLIP2 NaFlex tower. |
| src/mobius/components/init.py | Exports NaFlex tower symbols from the public components API. |
| src/mobius/_testing/torch_reference.py | Adds revision plumbing to reference multimodal HF loading (tokenizer/processor/config/weights). |
| src/mobius/_registry.py | Registers lfm2_vl model_type → module/task/config class mapping and updates default model IDs/categorization. |
| src/mobius/_configs/per_model/_lfm2_vl_vision.py | Adds NaFlex-specific vision extraction hook for composite LFM2-VL configs. |
| src/mobius/_configs/per_model/init.py | Imports the new per-model LFM2-VL vision hook for registration. |
| src/mobius/_configs/_base.py | Introduces Lfm2VlConfig (projector + composite lifting) in the config system. |
| src/mobius/_configs/init.py | Exports Lfm2VlConfig from the public configs API. |
| src/mobius/main.py | Adds --revision CLI flag and propagates it through build/export paths. |
| scripts/generate_golden.py | Threads pinned revision through golden generation for both text-only and multimodal cases. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds production Mobius support for
LiquidAI/LFM2.5-VL-3B(model_type=lfm2_vl) at immutable revision5a414ead75d45db003906d06fb62bd5b6846cec0.The implementation exports a standardized three-model package:
vision_encoder: SigLIP2 NaFlex patch embeddings, dynamic antialiased position interpolation, masked vision attention, pixel-unshuffle, and the LFM2-VL projectorembedding: token embedding plus image-feature substitution, including zero-image cached decodedecoder: existing LFM2 hybrid short-convolution/full-attention decoder with convolution and KV cache reuseIt also pins revisions through configuration, weights, tokenizer/processor assets, CLI/runtime export, and golden generation; adds ORT GenAI metadata for hybrid cache names/layer types; and preserves the upstream adaptive-tiling processor contract.
Architecture and integration checklist
lfm2_vlwith thelfm2-vlthree-model taskonnx_ironly (no protobuf APIs)decoder,vision_encoder, andembeddingkeyspixel_values,pixel_attention_mask, andspatial_shapesprocessor inputslayer_types, convolution cache names/size, image token ID, andpast_present_share_buffer=false--revisionCLI plumbing across config, weights, assets, processor, and golden callspipeline-cat-chonk.jpegValidation evidence
Environment: Windows, NVIDIA RTX A1000 8 GB (driver 573.44), ONNX Runtime 1.26.0, ONNX Runtime GenAI 0.15.2, Transformers 5.14.1.
L1 / weight alignment / L3 synthetic parity
Result: 18 passed. This includes ONNX graph/checker coverage, complete three-package Hugging Face weight alignment, NaFlex vision/projector full-tensor parity (
rtol=atol=1e-4), embedding fusion, and LFM2 prefill plus cached-decode full-logit parity (rtol=atol=1e-4).L2 / YAML schema
Results: 247 passed schema tests; 2 passed targeted schema/full pinned-config graph tests.
L4 CUDA prefill (bf16)
Result: 1 passed in 244.47s. Executes real processor → ONNX NaFlex vision/projector → ONNX embedding fusion → ONNX hybrid decoder on CUDAExecutionProvider and matches the pinned upstream prefill golden.
L5 CUDA cached deterministic generation (bf16)
Result: 1 passed in 264.92s. Exact 15-token cached decode:
Decoded:
A brown cat is walking on the snow with a fence behind it.Float16 CUDA parity
The same L4 and L5 commands with the case dtype temporarily set to
float16both passed (244.35s and 244.15s) with the same exact 15-token sequence. The committed canonical golden case remains bf16.CLI and runtime metadata
Result: successful export of
decoder/model.onnx,vision_encoder/model.onnx,embedding/model.onnx,genai_config.json, pinned tokenizer/chat-template assets, and upstreamprocessor_config.json. Intrinsic checks verified all referenced files, 30 hybrid layer types, convolution cache size 2, NaFlex input mapping, token IDs, and non-shared cache buffers.Formatting, lint, and broad regression
Result: passed with no findings.
The broad non-integration suite completed with 3967 passed, 58 skipped. Five unrelated existing numerical tests failed: one GGUF Q4_0 dequantization parity case, two decomposed softcap-attention parity cases, and two Qwen-Image CUDA low-precision parity cases. No LFM2-VL or revision/runtime-metadata test failed.
Explicit waivers / runtime notes
pixel_attention_maskvision input as an unknown schema key. Current ORT GenAI cannot represent LFM2-VL adaptive patch masking; direct three-stage ONNX execution is fully covered by CUDA L4/L5. The upstreamprocessor_config.jsonis intentionally preserved rather than emitting incorrect simplified transforms.MOBIUS_TEST_BUILD_EP=cudatriggers an ONNX Runtime 1.26.0 optimizer failure becauseCausalConvWithStatehas no provider assignment. The portableonnx-standardgraph runs all three stages through CUDAExecutionProvider and passes L4/L5; this is the validated deployment path for the current runtime.Review