Add complete GLM-OCR vision-language support by justinchuby · Pull Request #506 · onnxruntime/mobius

justinchuby · 2026-08-17T17:30:48Z

Summary

Adds production Mobius support for zai-org/GLM-OCR (model_type: glm_ocr) at immutable checkpoint revision ca5d8b3e287e52589e37c28385d9655ee4372f9d.

Implements the dedicated biased Conv3d vision tower, per-head Q/K RMSNorm, packed bidirectional attention, dynamic float32 2D RoPE, post-norm/downsample, and gated merger.
Implements the four-norm GLM text decoder, interleaved M-RoPE, embedding/image-feature mixer, KV-cache decode, and standardized decoder / vision_encoder / embedding package split.
Adds architecture/config extraction, registry and CLI wiring, exact checkpoint weight routing, and immutable revision propagation through Transformers, Diffusers, NeMo, ORT GenAI, and onnx-genai runtime assets.
Adds pinned L1-L5 coverage using a real nonzero 1024x1280 document and the real processor contract.

Upstream Transformers implementation was inspected at immutable revision 858d4a26b0239eb840670ee3416a69deb2afdd88.

Validation

Post-rebase results on head 529913d138d5de352e9b5876c01c468b859b5bb1 over base cf77d0f7fb28adebbeb3aaecb25fe2e51ee0ddd0:

10 passed, 2552 deselected       # GLM-OCR graph, alignment, schema, synthetic parity
261 passed, 10 skipped           # CLI, ORT GenAI, and onnx-genai metadata suites
115 passed                       # rebased ORT GenAI conflict-resolution smoke
L4 CUDA: 1 passed                # 104.78s
L5 CUDA: 1 passed                # 99.59s
lintrunner -a: passed
affected-model detection: ["glm_ocr"]

GPU commands:

$env:MOBIUS_TEST_DEVICE='cuda'
python -m pytest tests\e2e_golden_test.py -m golden --models glm_ocr -v --timeout=300 --tb=short
python -m pytest tests\e2e_golden_test.py -m generation --models glm_ocr -v --timeout=300 --tb=short

python -m mobius build --model zai-org/GLM-OCR `
  --revision ca5d8b3e287e52589e37c28385d9655ee4372f9d `
  <output> --dtype bf16 --ep cuda --runtime ort-genai

The saved CLI artifact loaded all three models with CUDAExecutionProvider first, executed the complete vision -> embedding -> decoder pipeline, exercised cached decode, and exactly reproduced 24 tokens and:

MOBIUS OCR VALIDATION

Nemotron Parse 2.0

This document verifies deterministic

Processor contract from the pinned checkpoint:

pixel_values: float32 [6808, 1176]
image_grid_thw: int64 [1, 92, 74]
prompt: 1714 tokens
smart resize: nominal 336x336, min_pixels=12544, max_pixels=9633792

Waivers / environment notes

onnxruntime-genai==0.15.2 parses the generated genai_config.json, but constructing og.Model exits with Windows access violation -1073741819 before emitting a runtime diagnostic. The same saved models pass the full CUDA pipeline and exact cached generation directly with onnxruntime-gpu==1.26.0; this is treated as a downstream ORT GenAI runtime compatibility limitation, not an export waiver.
FP16 semantic parity was validated with the portable standard-attention graph on CUDA. The ORT 1.26 CUDA-specialized FP16 GQA session path exits during decoder session creation; BF16 CUDA-specialized GQA is the checkpoint-native path and passes L4/L5 plus the saved-artifact probe.
ORT 1.28 on this host requires CUDA 13 (cublasLt64_13.dll); the installed CUDA 12.8 environment therefore uses onnxruntime-gpu==1.26.0.

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Implement the dedicated packed vision tower, four-norm GLM decoder, embedding mixer, registry and ORT GenAI integration for zai-org/GLM-OCR. Pin upstream artifacts and add L1-L5 graph, alignment, schema, synthetic parity, real-document prefill, and cached generation coverage. Signed-off-by: Justin Chu <justinchu@microsoft.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Keep GLM-OCR rotary trigonometry dynamic and float32 without changing shared Qwen behavior, including extreme-aspect-ratio coverage. Propagate immutable revisions through Diffusers, NeMo, and onnx-genai runtime assets. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>

Preserve the blank-line separation required after retaining both upstream generation-config coverage and GLM immutable-revision coverage during the semantic rebase conflict resolution. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>

github-actions · 2026-08-20T00:40:14Z

Performance Comparison

Comparing 9068a05 → 9e33359

Model	Metric	Baseline	Current	Delta
bert (feature-extraction)	model_size_bytes	359 KB	359 KB	+0.0%	⚪
bert (feature-extraction)	num_nodes	68	68	+0.0%	⚪
falcon	model_size_bytes	364 KB	364 KB	+0.0%	⚪
falcon	num_nodes	66	66	+0.0%	⚪
gemma2	model_size_bytes	428 KB	428 KB	+0.0%	⚪
gemma2	num_nodes	105	105	+0.0%	⚪
gpt2	model_size_bytes	388 KB	388 KB	+0.0%	⚪
gpt2	num_nodes	54	54	+0.0%	⚪
llama	model_size_bytes	425 KB	425 KB	+0.0%	⚪
llama	num_nodes	60	60	+0.0%	⚪
llama (static-cache)	model_size_bytes	425 KB	425 KB	+0.0%	⚪
llama (static-cache)	num_nodes	56	56	+0.0%	⚪
mamba (ssm-text-generation)	model_size_bytes	296 KB	296 KB	+0.0%	⚪
mamba (ssm-text-generation)	num_nodes	94	94	+0.0%	⚪
phi3	model_size_bytes	421 KB	421 KB	+0.0%	⚪
phi3	num_nodes	58	58	+0.0%	⚪
phi3 (static-cache)	model_size_bytes	421 KB	421 KB	+0.0%	⚪
phi3 (static-cache)	num_nodes	54	54	+0.0%	⚪
qwen2	model_size_bytes	425 KB	425 KB	+0.0%	⚪
qwen2	num_nodes	60	60	+0.0%	⚪
qwen2 (static-cache)	model_size_bytes	425 KB	425 KB	+0.0%	⚪
qwen2 (static-cache)	num_nodes	56	56	+0.0%	⚪
qwen3_5_moe (hybrid-text-generation)	model_size_bytes	506 KB	506 KB	+0.0%	⚪
qwen3_5_moe (hybrid-text-generation)	num_nodes	264	264	+0.0%	⚪
qwen3_5_text (hybrid-text-generation)	model_size_bytes	458 KB	458 KB	+0.0%	⚪
qwen3_5_text (hybrid-text-generation)	num_nodes	126	126	+0.0%	⚪
qwen3_5_vl (hybrid-qwen-vl)	model_size_bytes	977 KB	977 KB	+0.0%	⚪
qwen3_5_vl (hybrid-qwen-vl)	num_nodes	428	428	+0.0%	⚪
t5 (seq2seq)	model_size_bytes	836 KB	836 KB	+0.0%	⚪
t5 (seq2seq)	num_nodes	176	176	+0.0%	⚪
whisper (speech-to-text)	model_size_bytes	1008 KB	1008 KB	+0.0%	⚪
whisper (speech-to-text)	num_nodes	128	128	+0.0%	⚪

No performance regressions.

github-actions · 2026-08-20T00:42:25Z

🏗️ Architecture Diff

Comparing 9068a05 → 9e33359

Model	Sub-model	Status
bert (feature-extraction)	model	⚪
falcon	model	⚪
gemma2	model	⚪
gemma4 (gemma4)	decoder	⚪
gemma4 (gemma4)	embedding	⚪
gemma4 (gemma4)	vision_encoder	⚪
gemma4_text	model	⚪
gpt2	model	⚪
llama	model	⚪
llama (static-cache)	model	⚪
mamba (ssm-text-generation)	model	⚪
phi3	model	⚪
phi3 (static-cache)	model	⚪
qwen	model	⚪
qwen (static-cache)	model	⚪
qwen2	model	⚪
qwen2 (static-cache)	model	⚪
qwen2_moe	model	⚪
qwen2_moe (static-cache)	model	⚪
qwen3	model	⚪
qwen3 (static-cache)	model	⚪
qwen3_5_moe (hybrid-text-generation)	model	⚪
qwen3_5_text (hybrid-text-generation)	model	⚪
qwen3_5_vl (hybrid-qwen-vl)	decoder	⚪
qwen3_5_vl (hybrid-qwen-vl)	embedding	⚪
qwen3_5_vl (hybrid-qwen-vl)	vision_encoder	⚪
qwen3_moe	model	⚪
qwen3_moe (static-cache)	model	⚪
qwen3_next (hybrid-text-generation)	model	⚪
t5 (seq2seq)	decoder	⚪
t5 (seq2seq)	encoder	⚪
whisper (speech-to-text)	decoder	⚪
whisper (speech-to-text)	encoder	⚪

No architecture changes detected. ✅

Legend: ⚪ No change · 🔵 Minor (attrs/inits) · 🟡 Moderate (nodes added/removed) · 🔴 Major (interface changed)

justinchuby · 2026-08-21T04:25:24Z

@copilot please update from main

…cr-support # Conflicts: # src/mobius/__main__.py # src/mobius/_testing/torch_reference.py # src/mobius/integrations/ort_genai/auto_export.py Co-authored-by: justinchuby <11205048+justinchuby@users.noreply.github.com>

Co-authored-by: justinchuby <11205048+justinchuby@users.noreply.github.com>

CLAassistant · 2026-08-21T04:36:39Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ justinchuby
❌ Copilot
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

Copilot · 2026-08-21T04:39:49Z

@copilot please update from main

Merged main in commit d904343, resolving conflicts in __main__.py, _testing/torch_reference.py, and integrations/ort_genai/auto_export.py (including a few duplicate-kwarg merge artifacts). Full non-integration suite (4203 tests), CLI/ORT-GenAI suites, lintrunner, and CodeQL all pass.

justinchuby · 2026-08-23T17:58:18Z

@copilot update from main

…cr-support Signed-off-by: GitHub <noreply@github.com> # Conflicts: # src/mobius/__main__.py # src/mobius/_configs/per_model/__init__.py # src/mobius/_registry.py # src/mobius/integrations/onnx_genai/auto_export.py # src/mobius/integrations/onnx_genai/inference_metadata.py # src/mobius/integrations/ort_genai/auto_export.py # src/mobius/integrations/ort_genai/auto_export_test.py # src/mobius/models/__init__.py # src/mobius/tasks/__init__.py # tests/arch_validation_test.py # tests/cli_test.py Co-authored-by: justinchuby <11205048+justinchuby@users.noreply.github.com>

Copilot · 2026-08-23T18:24:08Z

@copilot update from main

Merged main in commit 9e33359, resolving conflicts in __main__.py, _configs/per_model/__init__.py, _registry.py, onnx_genai/ort_genai integrations, models/__init__.py, tasks/__init__.py, arch_validation_test.py, and cli_test.py. Full non-integration suite (4775 tests) and lintrunner pass.

justinchuby · 2026-08-23T18:27:47Z

@copilot make sure fusible ops are using onnx attention / gqa fused operators, rotary embedding, and related latest onnx / contrib ops in the model

Copilot · 2026-08-23T18:32:17Z

@copilot make sure fusible ops are using onnx attention / gqa fused operators, rotary embedding, and related latest onnx / contrib ops in the model

This is already the case, reusing the same shared components as the other vision-language models rather than any custom unfused path:

Text decoder (Glm4TextModel/DecoderLayer/Attention): emits fused RotaryEmbedding and Attention (opset 24) for prefill/dynamic-cache decode, and com.microsoft.GroupQueryAttention for the static-cache decode path — same shared _attention.py/_rotary_embedding.py components used by every other decoder in the repo.
Vision tower (GlmOcrVisionAttention extends Qwen25VLVisionAttention): emits com.microsoft.PackedMultiHeadAttention when the target EP supports it, falling back to fused Attention otherwise — the identical ep_capabilities().supports_packed_multi_head_attention pattern used by Qwen2.5-VL/Qwen3-VL/MAGE-VL vision towers.

I built the unit-test graph and confirmed the emitted op types: decoder graph contains RotaryEmbedding/Attention nodes, vision_encoder graph contains Attention nodes (no manual softmax/QK^T is emitted). The 2D vision RoPE cos/sin computation itself (arbitrary per-patch H/W position pairs, not a sequential position_ids table) is computed manually before being fed into the fused attention op, matching the pre-existing Qwen2.5-VL vision RoPE approach — this is inherited, not new, and the fused RotaryEmbedding op's cache-lookup contract doesn't fit that non-sequential 2D case. No code changes were needed.

justinchuby requested review from a team and a lite review from Copilot August 17, 2026 17:30

justinchuby added the automated label Aug 17, 2026

github-advanced-security Bot found potential problems Aug 17, 2026

View reviewed changes

Comment thread src/mobius/components/_glm_ocr_vision.py Fixed

Comment thread src/mobius/models/glm_ocr.py Fixed

Copilot started reviewing on behalf of justinchuby August 17, 2026 17:31 View session

Copilot AI reviewed Aug 17, 2026

justinchuby and others added 2 commits August 19, 2026 17:35

justinchuby force-pushed the justinchuby-add-glm-ocr-support branch from 4a55082 to 235c0fa Compare August 20, 2026 00:37

Copilot started work on behalf of justinchuby August 21, 2026 04:25 View session

Copilot AI and others added 2 commits August 21, 2026 04:29

Merge main into GLM-OCR branch, resolving merge conflicts

8f78554

Co-authored-by: justinchuby <11205048+justinchuby@users.noreply.github.com>

Copilot finished work on behalf of justinchuby August 21, 2026 04:40

Copilot started work on behalf of justinchuby August 23, 2026 17:58 View session

Copilot finished work on behalf of justinchuby August 23, 2026 18:24

Copilot started work on behalf of justinchuby August 23, 2026 18:28 View session

Copilot finished work on behalf of justinchuby August 23, 2026 18:32

justinchuby merged commit 560ace7 into main Aug 23, 2026
20 of 24 checks passed

justinchuby deleted the justinchuby-add-glm-ocr-support branch August 23, 2026 18:54

Navigation Menu

Uh oh!

Add complete GLM-OCR vision-language support - #506

Add complete GLM-OCR vision-language support#506
justinchuby merged 6 commits into
mainfrom
justinchuby-add-glm-ocr-support

justinchuby commented Aug 17, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

github-actions Bot commented Aug 20, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 20, 2026 •

edited

Loading

Uh oh!

justinchuby commented Aug 21, 2026

Uh oh!

CLAassistant commented Aug 21, 2026

Uh oh!

Copilot AI commented Aug 21, 2026

Uh oh!

justinchuby commented Aug 23, 2026

Uh oh!

Copilot AI commented Aug 23, 2026

Uh oh!

justinchuby commented Aug 23, 2026

Uh oh!

Copilot AI commented Aug 23, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

justinchuby commented Aug 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Validation

Waivers / environment notes

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Aug 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Performance Comparison

Uh oh!

github-actions Bot commented Aug 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🏗️ Architecture Diff

Uh oh!

justinchuby commented Aug 21, 2026

Uh oh!

CLAassistant commented Aug 21, 2026

Uh oh!

Copilot AI commented Aug 21, 2026

Uh oh!

justinchuby commented Aug 23, 2026

Uh oh!

Copilot AI commented Aug 23, 2026

Uh oh!

justinchuby commented Aug 23, 2026

Uh oh!

Copilot AI commented Aug 23, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

justinchuby commented Aug 17, 2026 •

edited

Loading

github-actions Bot commented Aug 20, 2026 •

edited

Loading

github-actions Bot commented Aug 20, 2026 •

edited

Loading