Feature/add glm ocr - #1177
Draft
shagsood wants to merge 6 commits into
Draft
Conversation
Adds QEfficient modeling support for GLM-OCR, a Vision-Language Model using M-RoPE (mrope_section) with a ViT vision encoder and GLM-style text decoder. Transformers 5.1.0+ required; import guarded with U6 try/except in pytorch_transforms.py. Key design decisions: - Path A (KVCacheTransform, native HF classes, no trust_remote_code) - QEffGlmOcrVisionModel.forward() fully vectorised — avoids Python for-loop in HF rot_pos_emb() that produces ONNX SplitToSequence / SequenceEmpty / ConcatFromSequence, rejected by QAIC (wiki quic#37) - masked_scatter → torch.where cumsum-gather (V4) for image injection - FP16 clamp after patch-merger output (V1) - position_ids shape (4,B,S): [0]=1D text, [1:]=M-RoPE T/H/W - Single-QPC default; dual-QPC available via kv_offload=True Also fixes three transformers 5.x compatibility issues (wiki quic#38): - quantizer_awq.py: AwqBackendPackingMethod→AwqBackend shim - cache_utils.py: HybridCache/HybridChunkedCache graceful fallback - __init__.py: diffusers+peft import guards for env mismatches Validated (2026-05-20): - S1 HF CPU forward: PASS (model runs, tokens produced) - S3 ONNX export: PASS (no unsupported ops) - S4 QPC compile: PASS (both single-QPC and dual-QPC) - S4 hardware run: pending qaic group membership on this host Signed-off-by: shagsood <shagsood@qti.qualcomm.com>
Two compile fixes found during full-weight run (203s compile, mxfp6+mxint8): 1. Remove pixel_values_RetainedState from single-QPC output names. Root cause: the compiler DCEs the vision encoder in the decode spec (its output is unused in V10 no-op path), making pixel_values usage inconsistent between prefill and decode specs. Fix: don't use retained state for pixel_values; runtime re-uploads on every call. 2. Remove pixel_values and image_grid_thw from single-QPC dynamic axes. Both are baked as ONNX constants (via .item() in vision forward); declaring them dynamic caused a secondary shape-inconsistency warning. Validated: S3 ONNX export: PASS (no unsupported ops) S4 QPC compile: PASS (203s, TS4, mxfp6+mxint8) S4 hardware run: pending qaic group membership Signed-off-by: shagsood <shagsood@qti.qualcomm.com>
The fork pins transformers via pyproject.toml:22, so glm_ocr is always present in the installed transformers. The try/except ImportError fallback is dead code in this single-pin workflow. Removes: - try/except guard around the HF import in pytorch_transforms.py - the `if _HAS_GLM_OCR:` guard around the QEff-side import - the `if _HAS_GLM_OCR:` guard around the KVCacheTransform / CustomOpsTransform registrations Behaviour unchanged. AST parse passes. Local-only commit; not pushed. Signed-off-by: shagsood <shagsood@qti.qualcomm.com>
…apes
Fixes two correctness bugs and eight ONNX-export tracing bugs in the GLM-OCR
QEff modeling file, all pre-existing since the original 2026-05-20 onboarding.
Before these fixes, real-weight hardware output was `" twice twice twice..."`
repetition garbage; after, it produces coherent OCR ("The image features a
vintage green Volkswagen Beetle parked on a paved street. The" against a
turquoise Beetle image, matching HF-CPU's "A vintage turquoise Volkswagen
Beetle is parked on a paved street in front", NLL=0.24, PPL=1.27).
Correctness fixes (HF-PyTorch first-token parity now exact, diff=0.0):
1. GLM interleaved-pair RoPE (not Qwen split-half). qeff_apply_rotary_pos_emb
was copy-pasted from qwen3_vl but Qwen and GLM use mathematically different
rotation conventions. GLM's rotate_half_llm swaps adjacent pairs
(x0,x1,x2,x3,...) -> (-x1,x0,-x3,x2,...); Qwen's rotate_half swaps halves.
Additionally, GLM cos/sin need repeat_interleave(2, dim=-1) after the
M-RoPE section select (HF's own apply_rotary_pos_emb does this).
Implemented pair-swap as a static gather (swap_idx + signs vector) that
is numerically identical to HF (verified diff=0.0) and trace-friendly.
2. GLM sandwich-norm decoder layer (4 layernorms, not 2). HF's
GlmOcrTextDecoderLayer applies input_layernorm, post_self_attn_layernorm
(before residual add), post_attention_layernorm, and post_mlp_layernorm
(before residual add). The original QEff override only invoked
input_layernorm and post_attention_layernorm; the other two modules were
on `self` with loaded weights but never called. Missing them caused ~40-
unit hidden-state magnitude divergence per layer.
ONNX-export tracing fixes (needed to compile the correct math on qaic):
3. Replace `-1` wildcards in .reshape()/.view() with explicit dims across
the modeling file. torch.jit.trace in this fork's export path bakes wrong
constants (usually 2) for `-1` when the surrounding shape depends on a
dynamic axis. Affected: text Q/K/V projections, text attention output,
vision QKV projection, vision attention output, patch-merger view,
patch-merger downsample, image-features gather (two sites).
4. Replace repeat_interleave(2, dim=-1) on cos/sin with static gather
`x[..., arange(head_dim) // 2]`. repeat_interleave is trace-hostile
and additionally mis-shapes the downstream unsqueeze.
5. Replace stack+flatten-based rotate_half_llm with a single static-index
gather + sign vector. The stack+flatten pattern (even with strided-slice
halves) produced a rank-7 downstream tensor after trace due to the
stack dim propagating incorrectly.
Adapting to upstream API drift (all mainline changes since 2026-05):
6. past_key_value_update() now returns 4 values (key, value, attention_mask,
cache_kwargs). Updated the QEffGlmOcrTextAttention call site to unpack
the new attention_mask and pass it into eager_attention_forward
(matching the llama/qwen3_vl pattern).
Also removes a dead import of QEfficient.transformers.models.aya_vision that
referenced a module never present in this fork's git history (only a stray
__pycache__ residue) — this had broken `import QEfficient` from any fresh
checkout of this branch since the original commit.
Verified on transformers==5.5.4 (upstream/main pin), zai-org/GLM-OCR real
weights, single-QPC TS4 (prefill=320, ctx=1024, mxfp6 + mxint8_kv,
num_devices=4). Compile: 272s. Decode: 222.8 tok/s. Quality gates 3/3 PASS.
Signed-off-by: shagsood <shagsood@qti.qualcomm.com>
…r_generation + example Strips the U1-V12 numbered invariant catalog, header architecture summary, and debugging-narrative comments from the modeling file so it reads like sibling files (qwen3_vl, llama4) - functional one-liners only where genuinely non-obvious, no cross-references to the invariant catalog. Also removes dead constants (_BS, _FBS, _VISION_SIZE, _GRID_HEIGHT, _GRID_WIDTH) and two unused locals left over from get_dummy_inputs. No logic changes; re-verified HF==QEff first-token parity is unchanged (diff=0.0) after the cleanup. Adds QEffGlmOcrForConditionalGeneration.prepare_inputs_for_generation, mirroring qwen3_vl's pattern: builds the 4D M-RoPE position_ids via get_rope_index() plus a 1D text-index row, defaults mm_token_type_ids when absent, and pads to prefill_seq_len. Verified its output matches the runner's hand-rolled compute_4d_position_ids() exactly on real tokens. Adds examples/image_text_to_text/models/glm_ocr/glm_ocr.py. The generic single-QPC generate() path in modeling_auto.py unconditionally recomputes 2D position_ids from attention_mask, so it can't consume the new prepare_inputs_for_generation output directly (this is why qwen3_vl's own example only ever compiles kv_offload=True/dual-QPC). GLM-OCR's only validated compile is single-QPC, so the example drives inference via QAICInferenceSession directly - using prepare_inputs_for_generation to build the position_ids feed - rather than switching to an unvalidated dual-QPC compile. Ran end-to-end on hardware this session: coherent output matching the HF baseline. Signed-off-by: shagsood <shagsood@qti.qualcomm.com>
…rapper QEffGlmOcrDecoderWrapper.forward() used vision_embeds.shape[0] as the flatten target when merging vision tokens into text embeddings, copied from the single-QPC forward() where image_embeds already arrives 2D (total_tokens, C). In the dual-QPC decoder, vision_embeds crosses the ONNX graph boundary from the separately-compiled vision QPC and arrives 3D (vision_batch_size, vision_size, C) - shape[0] is the batch dim (1), not the token count, so the reshape target must be shape[0] * shape[1]. Verified end-to-end on real weights + hardware: both compile modes now work. Single-QPC (already validated) unaffected - re-ran the CPU parity loop to confirm zero change (HF==QEff first token still exact, diff=0.0). Dual-QPC compiles both QPCs (97s total, TS4) and produces coherent, non-repetitive output on AI 100 hardware (377 tok/s decode). Signed-off-by: shagsood <shagsood@qti.qualcomm.com>
quic-hemagnih
marked this pull request as draft
July 13, 2026 05:07
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
GLM-OCR (zai-org/GLM-OCR)
Adds QEfficient support for zai-org/GLM-OCR, a ~0.9B VLM: GLM text decoder (16L, h=1536, 16Q/8KV heads, head_dim=128) + ViT-24L vision encoder + patch-merger
projector, using M-RoPE for joint text/vision positions.
Compile: single-QPC, prefill=320, ctx=1024, mxfp6 + mxint8_kv, TS4 (16 cores / 4 devices)
Compile time: 272s
Decode: ~223–237 tok/s
Sample output (336×336 image, greedy)
A vintage turquoise Volkswagen Beetle is parked on a paved street in frontA vintage turquoise Volkswagen Beetle parked on a paved street, with aKey deviations from HF
masked_scatter→torch.where+ cumsum-gather for image-token injection (ONNX-unfriendly op)rot_pos_embvectorized (avoids ONNX sequence ops QAIC rejects)prepare_inputs_for_generationfor 4D M-RoPEposition_ids