Add production Qwen3.8-27B support - #498
Open
justinchuby wants to merge 12 commits into
Open
Conversation
🏗️ Architecture Diff
qwen3_5_vl (hybrid-qwen-vl) / embedding — 18 change(s)Op summary: 32 → 47 nodes --- base
+++ head
@@ -1,14 +1,29 @@
Gather
Constant
Equal
+Equal
+Not
+Or
Unsqueeze
+Reshape
+Cast
+Reshape
Cast
Constant
CumSum
Constant
Sub
Constant
+CumSum
+Constant
+Sub
+ReduceSum
+Add
+Where
+Constant
Clip
+Shape
+Reshape
CastLike
Constant
ShapeAdded nodes:
Modified attributes:
Connectivity changes:
qwen3_5_vl (hybrid-qwen-vl) / vision_encoder — 435 change(s)Op summary: 247 → 271 nodes --- base
+++ head
@@ -1,134 +1,233 @@
+Cast
Reshape
Conv
Reshape
+Constant
+Gather
+Constant
+Gather
+Constant
+Gather
+Mul
+Mul
+Constant
+CumSum
+Constant
+Constant
+Pad
+ReduceSum
+Constant
+Constant
+Range
Slice
-Squeeze
-Slice
-Squeeze
-Slice
-Squeeze
-Mul
-Mul
-ReduceMax
-Scan
Shape
-Constant
-Gather
-Constant
-Squeeze
+ConstantOfShape
+Constant
+Mul
+ScatterElements
+Constant
+CumSum
+Gather
+Sub
+Gather
+Gather
+Mul
+Mod
+Constant
+Div
+Mod
+Constant
+Div
+Div
+Mod
+Constant
+Div
+Constant
+Mod
+Constant
+Mul
+Add
+Constant
+Mul
+Add
+Cast
+Cast
+Cast
+Cast
+Mul
+Sub
+Div
+Mul
+Sub
+Div
+Floor
+Floor
+Cast
+Cast
+Constant
+Constant
+Add
+Min
+Constant
+Add
+Min
+Sub
+Unsqueeze
+Sub
+Unsqueeze
+Constant
+Mul
+Constant
+Mul
+Add
+Add
+Add
+Add
+Cast
+Gather
+Gather
+Gather
+Gather
+Sub
+Mul
+Add
+Sub
+Mul
+Add
+Sub
+Mul
+Add
+CastLike
+Add
+Unsqueeze
+Unsqueeze
+Concat
+Gather
+Gather
+Squeeze
+Squeeze
+Gather
+Gather
+Gather
+Gather
+Concat
+Concat
+Constant
+Equal
+Compress
+Unsqueeze
+Concat
+LayerNormalization
+Transpose
+MatMul
+Add
+Split
+Reshape
+Reshape
+CastLike
+CastLike
+Unsqueeze
+Unsqueeze
+Split
+Split
+Mul
+Mul
+Sub
+Mul
+Mul
+Add
+Concat
+Mul
+Mul
+Sub
+Mul
+Mul
+Add
+Concat
+Reshape
+Reshape
+Shape
+Squeeze
+Constant
Constant
Range
Unsqueeze
Unsqueeze
-Less
+GreaterOrEqual
+Cast
+ReduceSum
+Constant
+Sub
+Unsqueeze
+Unsqueeze
+Equal
+CastLike
+CastLike
+Where
+Unsqueeze
+Unsqueeze
+Unsqueeze
+Unsqueeze
+Attention
+Squeeze
+Transpose
+MatMul
+Add
+Add
+LayerNormalization
+Transpose
+MatMul
+Add
+Gelu
+Transpose
+MatMul
+Add
+Add
+Reshape
+LayerNormalization
+Transpose
+MatMul
+Add
+Gelu
+Transpose
+MatMul
+Add
+LayerNormalization
+Transpose
+MatMul
+Add
+Split
+Reshape
+Reshape
+CastLike
+CastLike
+Unsqueeze
+Unsqueeze
+Split
+Split
+Mul
+Mul
+Sub
+Mul
+Mul
+Add
+Concat
+Mul
+Mul
+Sub
+Mul
+Mul
+Add
+Concat
+Reshape
Reshape
Shape
-Slice
-Constant
-Concat
-Reshape
-Compress
-CastLike
-Add
-Slice
-Squeeze
-Slice
-Squeeze
-Slice
-Squeeze
-Mul
-Mul
-ReduceMax
-Scan
-Shape
-Constant
-Gather
-Constant
-Squeeze
+Squeeze
+Constant
Constant
Range
Unsqueeze
Unsqueeze
-Less
-Reshape
-Shape
-Slice
-Constant
-Concat
-Reshape
-Compress
-Gather
-Gather
-Squeeze
-Squeeze
-Gather
-Gather
-Gather
-Gather
-Concat
-Concat
-Slice
-Squeeze
-ReduceMax
-Scan
-Shape
-Constant
-Gather
-Constant
-Squeeze
-Constant
-Range
-Unsqueeze
-Unsqueeze
-Less
-Reshape
-Shape
-Slice
-Constant
-Concat
-Reshape
-Compress
-Constant
-CumSum
-Constant
-Constant
-Pad
-LayerNormalization
-Transpose
-MatMul
-Add
-Split
-Reshape
-Reshape
-CastLike
-CastLike
-Unsqueeze
-Unsqueeze
-Split
-Split
-Mul
-Mul
-Sub
-Mul
-Mul
-Add
-Concat
-Mul
-Mul
-Sub
-Mul
-Mul
-Add
-Concat
-Reshape
-Reshape
-Shape
-Squeeze
-Constant
-Constant
-Range
-Unsqueeze
-Unsqueeze
GreaterOrEqual
Cast
ReduceSum
@@ -159,89 +258,14 @@
MatMul
Add
Add
-Reshape
-LayerNormalization
-Transpose
-MatMul
-Add
-Gelu
-Transpose
-MatMul
-Add
-LayerNormalization
-Transpose
-MatMul
-Add
-Split
-Reshape
-Reshape
-CastLike
-CastLike
-Unsqueeze
-Unsqueeze
-Split
-Split
-Mul
-Mul
-Sub
-Mul
-Mul
-Add
-Concat
-Mul
-Mul
-Sub
-Mul
-Mul
-Add
-Concat
-Reshape
-Reshape
-Shape
-Squeeze
-Constant
-Constant
-Range
-Unsqueeze
-Unsqueeze
-GreaterOrEqual
-Cast
-ReduceSum
-Constant
-Sub
-Unsqueeze
-Unsqueeze
-Equal
-CastLike
-CastLike
-Where
-Unsqueeze
-Unsqueeze
-Unsqueeze
-Unsqueeze
-Attention
-Squeeze
-Transpose
-MatMul
-Add
-Add
-LayerNormalization
-Transpose
-MatMul
-Add
-Gelu
-Transpose
-MatMul
-Add
-Add
-LayerNormalization
-Reshape
-Transpose
-MatMul
-Add
-Gelu
-Transpose
-MatMul
-Add
-Unsqueeze
-Concat
+LayerNormalization
+Reshape
+Transpose
+MatMul
+Add
+Gelu
+Transpose
+MatMul
+Add
+Unsqueeze
+ConcatAdded nodes:
Removed nodes:
Modified attributes:
Connectivity changes:
Initializer changes:
Legend: ⚪ No change · 🔵 Minor (attrs/inits) · 🟡 Moderate (nodes added/removed) · 🔴 Major (interface changed) |
Contributor
There was a problem hiding this comment.
Pull request overview
Adds production support for pinned Qwen/Qwen3.8-27B by reusing the existing Qwen3.5 hybrid 3-model VL architecture (decoder + embedding + vision encoder), expanding the runtime/metadata integration surface and adding reduced-real + Olive Q4 validation assets.
Changes:
- Extend Qwen VL embedding + vision pipelines to support packed image-then-video feature ordering and keep the processor boundary float32 (cast at the graph boundary).
- Replace Scan-based Qwen3-VL vision position embedding / rotary / cu_seqlens logic with a packed-stream coordinate approach.
- Add reduced-real pinned fixture + goldens + Olive Q4_K_M recipe and wire in additional ORT GenAI auto-export mappings/tests for
qwen3_5*.
Reviewed changes
Copilot reviewed 19 out of 19 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/qwen38_real_weight_test.py | Adds unit + integration coverage for the reduced-real fixture and Olive Q4 package validation. |
| tests/integration_test.py | Pins Qwen3.8 config/processor and adds mixed image/video pipeline parity coverage. |
| testdata/golden/vision-language/qwen3_8-27b-reduced.json | New L4 golden for reduced-real logits top-k validation. |
| testdata/golden/vision-language/qwen3_8-27b-reduced_generation.json | New L5 golden for reduced-real cached generation validation. |
| testdata/cases/vision-language/qwen3_8-27b.yaml | Adds a (CI-skipped) case descriptor for Qwen3.8-27B VL. |
| src/mobius/tasks/_vision_language_3model.py | Forces pixel_values input dtype to float32 and casts to model dtype inside the vision graph. |
| src/mobius/models/qwen35.py | Updates MTP-related documentation/comments to reflect separate optional drafter packaging. |
| src/mobius/models/qwen35_test.py | Adds Qwen3.8 alias/contract tests (registry IDs, MTP classification, scatter semantics, float32 boundary). |
| src/mobius/models/qwen_vl.py | Updates Qwen3-VL embedding scatter to handle packed image+video ordering and token IDs. |
| src/mobius/integrations/ort_genai/auto_export.py | Maps qwen3_5* model types to ORT GenAI qwen3_5 and includes qwen3_5_text in VL model-type sets. |
| src/mobius/integrations/ort_genai/auto_export_test.py | Adds coverage for qwen3_5 model-type resolution, packed vision processor emission, and metadata emission without runtime gates. |
| src/mobius/components/_qwen3_vl_vision.py | Reworks Qwen3-VL vision pos-embed interpolation/rotary/cu_seqlens to avoid control-flow subgraphs. |
| src/mobius/_registry.py | Updates registry test model ID for qwen3_5 to Qwen3.8-27B (and retains qwen3_5_vl test IDs). |
| src/mobius/_configs/_vision_defaults.py | Plumbs video + vision boundary token IDs from HF configs into vision defaults. |
| examples/olive/qwen3_8-27b/validate_reduced_checkpoint.py | New pinned reduced-real range-fetch validator for parity, save/load, and Q4 package audit. |
| examples/olive/qwen3_8-27b/requirements.txt | Adds minimal dependencies for the Olive validation recipe. |
| examples/olive/qwen3_8-27b/README.md | Documents the reduced-real + Olive validation workflow and runtime waivers. |
| examples/olive/qwen3_8-27b/optimize.py | Adds decoder-only Q4_K_M quantization workflow and package assembly logic. |
| examples/olive/qwen3_8-27b/inference.py | Adds direct ORT generation helper for reduced Qwen3.8 hybrid-VL packages. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
justinchuby
marked this pull request as draft
August 15, 2026 05:22
justinchuby
force-pushed
the
justinchuby-add-nemotron-35-lightning
branch
from
August 15, 2026 05:38
cfbbc7a to
ae4fd70
Compare
justinchuby
force-pushed
the
justinchuby-add-qwen3-8-27b
branch
from
August 15, 2026 05:50
54433fd to
05a478a
Compare
6 tasks
Treat the official checkpoint as a pinned Qwen3.5 hybrid VL alias, preserve image/video processor semantics, classify MTP as a separate optional drafter, and make packed vision coordinates CUDA-safe without Scan subgraphs. Add deterministic real-processor parity, reduced-real FP32/FP16/BF16 validation, cached generation goldens, and an Olive Q4_K_M package recipe with recurrent-gate stability safeguards. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
Isolate Olive caches so recurrent-gate exclusions are always honored, use a portable CUDA vision graph around the ORT 1.26 PackedMHA defect, and validate the final Q4 package with CUDA reload plus deterministic cached CPU generation. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
Emit the faithful qwen3_5 GenAI model type without consulting downstream runtime support, retain processor metadata in Q4 packages, validate local processor reload, and remove the static Olive recipe that could not encode required graph-derived exclusions. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
Document that repeated identical ORT 1.26 CUDA MatMulNBits runs on the exact assembled package can diverge or produce non-finite logits, while CPU cached generation remains the semantic acceptance path. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
Map the extracted qwen3_5_text subtype back to faithful qwen3_5 multimodal metadata and select the packed Qwen PatchImage processor pipeline for normal config and local exports. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
Omit the incompatible production tokenizer from the 256-token reduced fixture while retaining and independently reloading the pinned image and video processor metadata. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
Omit out-of-range production generation IDs from the token-ID-only fixture and require empty variant directories so stale tokenizer assets cannot survive reruns. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
Merge source provenance and the token-ID-only input contract into the Q4 manifest, and carry processor waivers through package assembly. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
Load the example's sibling inference module under a Qwen-specific name so prior Nemotron tests cannot poison Python's module cache in combined CI runs. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
Construct the absent-video mask entirely in the ONNX graph and preserve the original boolean image mask for media feature selection. Add runtime coverage for image-only embedding scatter when video_token_id is unset. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
Compute packed vision coordinates once, replace quadratic media lookup with boundary scatter and prefix sum, and reuse patch-local indices for frame boundaries. Simplify equivalent bilinear interpolation so Qwen3.5-VL stays below the deterministic graph-node regression threshold while preserving real image/video parity and CUDA semantics. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
justinchuby
force-pushed
the
justinchuby-add-qwen3-8-27b
branch
from
August 15, 2026 06:38
05a478a to
3758328
Compare
Performance Comparison
|
Preserve qwen3_5_text for decoder-only exports while selecting qwen3_5 for multimodal vision and embedding packages, including local-config exports whose composite config was unwrapped. Cover both generated package topologies without adding a runtime capability gate. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
justinchuby
marked this pull request as ready for review
August 15, 2026 07:05
justinchuby
marked this pull request as draft
August 15, 2026 12:40
justinchuby
marked this pull request as ready for review
August 15, 2026 12:42
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Qwen/Qwen3.8-27B@1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0as a checkpoint alias of the existing dense Qwen3.5 hybrid vision-language architectureqwen3_5_text, while multimodal Qwen3.5/Qwen3.8 packages emitqwen3_5Stacked on #487 final skill-split head
a8cd77570bca980861eeadab9e6fa077464f0138. Exact PR head:067cb489599687b74f96574de20133cb96e016a7. Tracks #483.Final validation
1d4bf0f...: image max abs0.00697723, cosine0.999995530; video0.00411959,0.999997377; mixed two-row/opposite placeholder order0.00677243,0.999998629; all below the automated0.01max-error gate with exact final argmax0.00077166, cosine1.0; FP16 CUDA max abs0.00341797, cosine0.999998391; deterministic 20-token cached generation and provider profile (5129CUDA /1449CPU node executions)com.microsoft::MatMulNBitsnodes; decoder, embedding, and vision sessions load with CUDA; local image/video processor metadata reloads; the exact package produces deterministic finite 20-token cached generation on CPUScatterElements+CumSumin O(patches + media), and frame starts reuse patch-local IDs428 -> 467top-level nodes (+9.1%, model size unchanged), below the unchanged 10% blocker threshold; remote Benchmark base/head/Compare jobs passedqwen3_5_textand multimodalqwen3_5; HF-ID and local-config selection share_select_ort_model_type113 passed; focused Qwen:45 passed, 21 skipped; combined Nemotron->Qwen module-isolation order:3 passed1 passed; FP16 CUDA and assembled Q4 acceptance:2 passed; reduced L4/L5 and BF16 reload remain green3962 passed, 61 skippedlintrunner -a -m a8cd775...reports no issues,git diff --checkclean067cb48...: no actionable findings; all2/2live review threads resolvedRuntime and resource waivers
CausalConvWithState/Softplusplacement); newer available GPU wheels require CUDA 13 in this environment. BF16 export, full population, save, and package reload pass. This downstream limitation is not an export gate.MatMulNBitscan return nondeterministic/non-finite logits for repeated identical inputs on this fixture. CUDA package/session/provider loading is proven, while semantic acceptance is the deterministic 20-token cached CPU run from the exact same assembled package. CPU generation is not claimed as GPU L5.PackedMultiHeadAttentionis nondeterministic across changing packed vision batches; stable CUDA image/video/mixed evidence uses the equivalent portable standard-attention vision graph while Mobius continues to emit both variants.CI
Exact-head run
31870831200completed. Architecture, benchmark, lint/lintrunner, build, L1, L3, and the Linux/Windows Python 3.11-3.13 matrices pass. Three jobs are red but reproduce unchanged on the exact #487 base run31869640946:Integration (fast): the same two unrelated DeepSeek assertions fail on base and head (test_deepseek_v2_lite_prefill_logits_matchandtest_deepseek_non_mla_decoder_prefill_logits_match).minicpm-v-4_6,qwen2-vl-2b,qwen2_5-vl-3b,qwen3-vl-2b,qwen3_5-2b). The Qwen3.8 reduced-real tests skip in this generic runner because their opt-in fixture flag is not set; their explicit local CUDA evidence is reported above.These are inherited base/infrastructure failures, not Qwen3.8 regressions. PR #498 remains draft pending parent confirmation.