Skip to content

feat(weight-free): weight-free ONNX export for causal LMs (dynamo path) - #1181

Closed
amarquic wants to merge 54 commits into
quic:quic-enable-dynamo-for-causallmfrom
amarquic:enable_weightfree_for_causallm
Closed

feat(weight-free): weight-free ONNX export for causal LMs (dynamo path)#1181
amarquic wants to merge 54 commits into
quic:quic-enable-dynamo-for-causallmfrom
amarquic:enable_weightfree_for_causallm

Conversation

@amarquic

@amarquic amarquic commented Jul 13, 2026 •
edited
Loading

Copy link
Copy Markdown

Summary

Adds weight-free ONNX export for causal LM models on top of the dynamo export infrastructure in #1178.

Weight-free export builds the model on meta device (no weights in RAM), exports ONNX graph structure only, and lets the QAIC compiler load weights directly from the original safetensors checkpoint at compile time via weight_spec.json.

New files

  • QEfficient/exporter/weight_free.py — core weight-free export: promotes ONNX initializers to external data, builds weight_spec.json (5 checkpoint key lookup strategies including Mixtral mlp→block_sparse_moe rename)
  • QEfficient/exporter/weight_spec.py — weight spec format and I/O
  • QEfficient/exporter/checkpoint_transforms.py — checkpoint prep pipeline: MoE expert stacking, dtype conversion, .bin→safetensors auto-convert
  • examples/text_generation/weight_free_export_verify.py — single-model smoke test
  • examples/text_generation/continuous_batching_weightfree.py — CB weight-free smoke test
  • examples/performance/compute_context_length/weight_free_ccl_verify.py — CCL verification

Key changes

  • QEfficient/base/modeling_qeff.py — use_weight_free_export flag; meta-device guard for _model_offloaded_check()
  • QEfficient/transformers/models/modeling_auto.py — use_weight_free_export wired through export() and compile(); CCL comp_ctx_lengths Dim fix
  • QEfficient/base/onnx_transforms.py — disable RewriteUnsupportedOpsTransform and SplitMultiInputMaxMinTransform for dynamo path
  • QEfficient/utils/export_utils.py — dynamo guard on apply_torch_patches(); separate dynamo/TorchScript subfunction setup paths
  • QEfficient/utils/torch_patches.py — temporarily_enable/disable_nested_compile_regions()
  • QEfficient/customop/ — backward stubs for CtxScatter/Gather; GemmaRMSNorm weight offset at runtime
  • Model fixes: Mixtral meta guard + attention_scaling, GraniteMoe optimizeSoftmax (router + attention), Falcon meta guard, GPT-OSS meta guard + device fixes, Qwen3-MoE, GLM4-MoE

Test plan

  • Weight-free export + compile + inference verified on: GPT2, Llama-3.2-1B, Mistral-7B, Qwen2-1.5B, Qwen3-30B-A3B, Phi-3-mini, Gemma-2-2b, OLMo2-1B, Falcon-7B, GPT-J-6B, Starcoder2-3B, GraniteForCausalLM, GraniteMoE, GLM4-MoE, GPT-OSS-20B, Mixtral-8x7B, Qwen3-MoE
  • CCL verified on Qwen3-30B-A3B
  • Continuous batching weight-free verified on Qwen2-1.5B, Llama-3.2-1B

vbaddi and others added 30 commits July 10, 2026 11:50
@vbaddi @smedhe
feat: dynamo-inline compile + subfunction export (squashed)
d685906 
Squash of prior dynamo work (16 WIP commits) into a single commit
to simplify rebase onto mainline. Original history preserved at
tag backup/before-rebase-20260501-212703.

Contents:
- Enable torch.dynamo-based export with inline compiler across causal
  LMs, VLMs, embedding, ASR models.
- Subfunction extraction / nested compile region utilities.
- Torch patches and custom-op utils consolidated under QEfficient/utils.
- Dynamo-specific tests and e2e example.

(cherry picked from commit 70f0cac)
Signed-off-by: Sharvari Medhe <smedhe@qti.qualcomm.com>
@vbaddi @smedhe
nit(0501): improve the @torch.compiler.nested_compile_region for othe…
2845cbe 
…r models

Signed-off-by: vbaddi <vbaddi@qti.qualcomm.com>
(cherry picked from commit 087f8d4)
Signed-off-by: Sharvari Medhe <smedhe@qti.qualcomm.com>
@vbaddi @smedhe
nit(0502): Add support for vision and other missing category models
3ebf86c 
Signed-off-by: vbaddi <vbaddi@qti.qualcomm.com>
(cherry picked from commit 5fa0d57)
Signed-off-by: Sharvari Medhe <smedhe@qti.qualcomm.com>
@vbaddi @smedhe
nit(0504): Align w/built pytorch from the changes (whl), WIP, move ba…
8274c3b 
…ck previous commit, if this fails

Signed-off-by: vbaddi <vbaddi@qti.qualcomm.com>
(cherry picked from commit 2ca7b45)
Signed-off-by: Sharvari Medhe <smedhe@qti.qualcomm.com>
@vbaddi @smedhe
nit(0504): update and align torch_patches, WIP, move back previous co…
022e951 
…mmit (nit(0502), if this fails

Signed-off-by: vbaddi <vbaddi@qti.qualcomm.com>
(cherry picked from commit 22d6385)
Signed-off-by: Sharvari Medhe <smedhe@qti.qualcomm.com>
@smedhe
adding fake ops and cache utils related changes for dynamo
53e75e3 
Signed-off-by: Sharvari Medhe <smedhe@qti.qualcomm.com>
(cherry picked from commit 575545a)
Signed-off-by: Sharvari Medhe <smedhe@qti.qualcomm.com>
@smedhe
changing torch.tensor to torch.full_like for MIN_MASKED_ATTENTION_VALUE
4b4cc7d 
Signed-off-by: Sharvari Medhe <smedhe@qti.qualcomm.com>
(cherry picked from commit 806f2cf)
Signed-off-by: Sharvari Medhe <smedhe@qti.qualcomm.com>
@smedhe
refactoring dynamic axes to dynamic shapes converter func
b6ef376 
Signed-off-by: Sharvari Medhe <smedhe@qti.qualcomm.com>
(cherry picked from commit 6dd6362)
Signed-off-by: Sharvari Medhe <smedhe@qti.qualcomm.com>
@smedhe
commenting out torch installation temporarily
08f5e6d 
Signed-off-by: Sharvari Medhe <smedhe@qti.qualcomm.com>
(cherry picked from commit 3266f7c)
Signed-off-by: Sharvari Medhe <smedhe@qti.qualcomm.com>
@smedhe
adding changes after rebasing on 0626
a0c43a5 
Signed-off-by: Sharvari Medhe <smedhe@qti.qualcomm.com>
@smedhe
adding glm4moe,gptoss,granitemoe and cache utils related changes
e30530f 
Signed-off-by: Sharvari Medhe <smedhe@qti.qualcomm.com>
@smedhe
adding gemma3 model related changes
8dd9412 
Signed-off-by: Sharvari Medhe <smedhe@qti.qualcomm.com>
@smedhe
adding further gemma3 changes
51a7a5d 
Signed-off-by: Sharvari Medhe <smedhe@qti.qualcomm.com>
@smedhe
adding changes for qwen2.5 vl and qwen3 vl
80f9498 
Signed-off-by: Sharvari Medhe <smedhe@qti.qualcomm.com>
@smedhe
adding dynamic shapes to get_sampling_inputs_and_outputs
376c516 
Signed-off-by: Sharvari Medhe <smedhe@qti.qualcomm.com>
@smedhe
adding changes for qwen3vl model
d6cdde4 
Signed-off-by: Sharvari Medhe <smedhe@qti.qualcomm.com>
@smedhe
removing export reports, disabling rewriteunsupportedops
bb68035 
Signed-off-by: Sharvari Medhe <smedhe@qti.qualcomm.com>
@smedhe
adding fbs in dynamic shapes for causal lm, other minor fixes
3ba8ea5 
Signed-off-by: Sharvari Medhe <smedhe@qti.qualcomm.com>
@smedhe
fixing export utils
3c9d499 
Signed-off-by: Sharvari Medhe <smedhe@qti.qualcomm.com>
@smedhe
correction in shapes in custom ops utils for CB
3399d17 
Signed-off-by: Sharvari Medhe <smedhe@qti.qualcomm.com>
@smedhe
reordering the if statements
7f8f62d 
Signed-off-by: Sharvari Medhe <smedhe@qti.qualcomm.com>
@smedhe
reverting the modeling changes for VLMs
198cce0 
Signed-off-by: Sharvari Medhe <smedhe@qti.qualcomm.com>
@smedhe
reverting dynamo and glm moe dsa changes in VLM related infra
218db64 
Signed-off-by: Sharvari Medhe <smedhe@qti.qualcomm.com>
@smedhe
making changes in onnx transforms and pyproject.toml
0bbcfe0 
Signed-off-by: Sharvari Medhe <smedhe@qti.qualcomm.com>
@smedhe
adding license header in custom_op_utils
6e94271 
Signed-off-by: Sharvari Medhe <smedhe@qti.qualcomm.com>
@smedhe
removing the existing dynamo test cases
73f646c 
Signed-off-by: Sharvari Medhe <smedhe@qti.qualcomm.com>
@smedhe
adding use_dynamo=False in layerwise
519d552 
Signed-off-by: Sharvari Medhe <smedhe@qti.qualcomm.com>
@smedhe
adding changes to solve the unit tests failures
d6f55bf 
Signed-off-by: Sharvari Medhe <smedhe@qti.qualcomm.com>
@amarquic
feat(weight-free): add weight-free ONNX export infrastructure for cau…
ffbe11a 
…sal LMs

New files:
- exporter/weight_free.py: core weight-free dynamo export — builds meta model,
  traces with FakeTensors, promotes ONNX initializers to external data,
  builds weight_spec.json mapping ONNX names to safetensors checkpoint keys
- exporter/weight_spec.py: WeightSpec format — describes external weight files
  and tensor→checkpoint-key mappings consumed by the QAIC compiler
- exporter/checkpoint_transforms.py: checkpoint preparation pipeline —
  DtypeConversionCheckpointTransform, MoEExpertStackingCheckpointTransform
  (stacks per-expert tensors into batched layout), and
  GptOssMxfp4ExpertDequantSplitCheckpointTransform (dequant MXFP4 experts)
- exporter/__init__.py: export new checkpoint transform classes
- scripts/preprocess_moe_checkpoint.py: standalone MoE checkpoint prep script
- examples/text_generation/weight_free_export_verify.py: e2e weight-free
  export verification (export → ORT → parity check)
- examples/text_generation/weight_free_export_simple.py: minimal weight-free
  export example for causal LMs

Signed-off-by: amarshar <amarshar@qti.qualcomm.com>
Signed-off-by: Amar <amarshar@qti.qualcomm.com>
@amarquic
feat(weight-free): model-level fixes for dynamo weight-free export
c262013 
- gemma.py, gemma2.py: align cos/sin device for dynamo tracing
- glm4_moe.py: __qeff_init__ uses meta-device guard for weight-free;
  register gate_proj/up_proj/down_proj_t directly on self.experts;
  remove get_all_projections() helper; fix rotary_dim computation
- granite.py: full_like for causal mask; device alignment
- granitemoe.py: weight-free subfunction export support
- grok1.py: replace bool-mask expert loop with gather-BMM pattern;
  add __qeff_init__ stacking per-expert weights; meta-device guard
- llama.py: fix decoder layer for dynamo — buffers, nested compile regions,
  rotary signature alignment
- phi.py, phi3.py: QEffPhi3MLP to avoid SplitToSequence from chunk(2);
  device alignment fixes
- qwen2.py: cos/sin device alignment
- qwen3.py: extend qeff_apply_rotary_pos_emb signature and align cos/sin
- qwen3_moe.py: weight-free dynamo export; expert weight stacking with
  meta-device guard; RAM reduction via view aliases (no clone)

Signed-off-by: amarshar <amarshar@qti.qualcomm.com>
Signed-off-by: Amar <amarshar@qti.qualcomm.com>
amarquic added 11 commits July 13, 2026 13:13
@amarquic
chore(utils): export generate_mdp_partition_config from utils package
a849132 
Signed-off-by: amarshar <amarshar@qti.qualcomm.com>
Signed-off-by: Amar <amarshar@qti.qualcomm.com>
@amarquic
feat(sampler_utils): add dynamic_shapes as 4th return value for dynam…
9c1f68d 
…o path

get_sampling_inputs_and_outputs now accepts and populates dynamic_shapes
alongside dynamic_axes, building Dim entries for each sampler input so
the dynamo export path gets consistent symbolic shapes for samplers.

Signed-off-by: amarshar <amarshar@qti.qualcomm.com>
Signed-off-by: Amar <amarshar@qti.qualcomm.com>
@amarquic
fix(pytorch_transforms): wire Phi3MLP and Grok1MoeBlock for weight-fr…
69dea12 
…ee export

- Add Phi3MLP → QEffPhi3MLP to KVCacheTransform: QEffPhi3MLP wraps the MLP
  to avoid SplitToSequence from chunk(2) in dynamo ONNX, which QAIC does
  not support
- Wire QEffGrok1MoeBlock.__qeff_init__ into KVCacheExternalModuleMapperTransform:
  stacks per-expert weights during init so the gather-BMM forward works
  correctly for weight-free export

Signed-off-by: amarshar <amarshar@qti.qualcomm.com>
Signed-off-by: Amar <amarshar@qti.qualcomm.com>
@amarquic
fix(utils): remove generate_mdp_partition_config from utils exports
59b0fdc 
In this branch the function lives in compile/mdp_generator.py not
utils/_utils.py — importing it via utils/__init__.py caused ImportError.

Signed-off-by: amarshar <amarshar@qti.qualcomm.com>
Signed-off-by: Amar <amarshar@qti.qualcomm.com>
@amarquic
fix(attn_mask_utils): add device=position_ids.device to torch.arange …
c250a3f 
…calls

Prevents meta/cpu device mismatch during weight-free export tracing.

Signed-off-by: amarshar <amarshar@qti.qualcomm.com>
Signed-off-by: Amar <amarshar@qti.qualcomm.com>
@amarquic
fix(modeling_auto): restore missing def prefill( header lost in edit
f818dd9 
Signed-off-by: amarshar <amarshar@qti.qualcomm.com>
Signed-off-by: Amar <amarshar@qti.qualcomm.com>
@amarquic
feat(weight-free): add .bin→safetensors auto-convert and mlp/block_sp…
1ed8c64 
…arse_moe key mapping

checkpoint_transforms.py:
- Add _convert_bin_to_safetensors(): auto-converts .bin checkpoints to safetensors
  on first use via save_pretrained(safe_serialization=True); idempotent
- CheckpointTransformPipeline.apply(): detect .bin files and convert before transform
- MoEExpertStackingCheckpointTransform: extend EXPERT_RE and _LayerStacker.add() to
  handle Mixtral w1/w3/w2 naming (w1→gate, w3→up, w2→down)

weight_free.py:
- _resolve_checkpoint_dir(): fall back to downloading .bin files when no safetensors
  exist on HuggingFace Hub (e.g. GPT-J-6B); CheckpointTransformPipeline auto-converts
- _find_checkpoint_key(): add lookup 5 to bridge mlp↔block_sparse_moe rename —
  transformers 5.x renamed the MoE block attribute, ONNX uses 'mlp' but Mixtral
  checkpoint has 'block_sparse_moe'

Signed-off-by: amarshar <amarshar@qti.qualcomm.com>
Signed-off-by: Amar <amarshar@qti.qualcomm.com>
@amarquic
fix(models): dynamo/weight-free fixes for Falcon, GPT-OSS, GraniteMoe…
91a108b 
…, Mistral, Mixtral, OLMo2

falcon/modeling_falcon.py:
- QEffFalconModel.__qeff_init__: meta device guard for sin/cos — skip storing
  as nn.Parameter on meta device to avoid meta tensor ONNX initializer failure

gpt_oss/modeling_gpt_oss.py:
- qeff_apply_rotary_pos_emb: add device/dtype alignment for weight-free tracing
- ctx_indices/short_read_idx: add device=position_ids.device to torch.arange

granitemoe/modeling_granitemoe.py:
- QEffGraniteMoeAttention.__qeff_init__: set num_heads/num_key_value_heads as
  instance attributes — required by @nested_compile_region inside decoder layer
- past_key_value_update: fix 3-tuple unpack to 4-tuple
- eager_attention_forward: use full_like for causal mask to avoid config.torch_dtype
- QEffGraniteMoeTopKGating: drop .to(dtype) after softmax (SoftMax+Cast → QAIC crash)
- QEffGraniteMoeDecoderLayer: return hidden_states directly instead of tuple

mistral/modeling_mistral.py:
- qeff_apply_rotary_pos_emb: add device/dtype alignment

mixtral_moe/modeling_mixtral.py:
- QEffMixtralExperts.__qeff_init__: meta device guard for gate/up/down projections
- QEffMixtralModel.__qeff_init__: drop attention_scaling multiply (creates new
  tensor that becomes top-level ONNX initializer; direct reference avoids it)
- QeffMixtralDecoderLayer: add @torch.compiler.nested_compile_region
- QEffMixtralExperts: add to KVCacheTransform so __qeff_init__ is called
- eager_attention_forward: use full_like for causal mask

olmo2/modeling_olmo2.py:
- qeff_apply_rotary_pos_emb: add device/dtype alignment

pytorch_transforms.py:
- Wire QEffMixtralExperts.__qeff_init__ (MixtralExperts→QEffMixtralExperts mapping)
- Wire QEffGraniteMoeAttention.__qeff_init__ (already mapped, __qeff_init__ added)

Signed-off-by: amarshar <amarshar@qti.qualcomm.com>
Signed-off-by: Amar <amarshar@qti.qualcomm.com>
@amarquic
feat(examples): add weight-free export and CCL verification scripts
03ce067 
examples/text_generation/weight_free_export_verify.py:
- Single-model smoke-test: weight-free dynamo export → compile → QPC inference
- --no_weight_free: regular from_pretrained + dynamo path
- --layers: override num_hidden_layers for fast testing
- Config loaded without trust_remote_code first (fixes Falcon legacy config)

examples/text_generation/continuous_batching_weightfree.py:
- Single-model weight-free export with continuous batching (full_batch_size)
- Same config fix for legacy models like Falcon

examples/performance/compute_context_length/weight_free_ccl_verify.py:
- Single-model CCL verification: compile with comp_ctx_lengths_prefill/decode
- Prints specializations.json to confirm CCL slots were compiled
- Default: ctx_len=4096, CCL=[1024,2048,4096], weight-free + dynamo

Signed-off-by: amarshar <amarshar@qti.qualcomm.com>
Signed-off-by: Amar <amarshar@qti.qualcomm.com>
@amarquic
fix(export_utils): restore dynamo guard on apply/undo_torch_patches a…
b956256 
…fter rebase

The rebase auto-merge dropped the if not use_dynamo: guard introduced in
Sharvari's d6f55bf commit. Restore it so TorchScript patches are not
applied on the dynamo export path, and restore use_dynamo in the state
dict so _cleanup_onnx_subfunctions can check it.

Signed-off-by: Amar <amarshar@qti.qualcomm.com>
@amarquic
fix(granitemoe): fix two optimizeSoftmax SIGABRT crashes in subfunction
9952784 
- Attention softmax: remove .to(query.dtype) — emits SoftMax->Cast in the
  subfunction ONNX which triggers optimizeSoftmax assertion in QAIC compiler
- Router softmax: replace torch.softmax with manual Exp/Sum/Div — with
  num_local_experts=8, top_k_gates has 8 Einsum consumers; optimizeSoftmax
  asserts when a SoftMax node has more than one consumer

Signed-off-by: Amar <amarshar@qti.qualcomm.com>
@vbaddi vbaddi added enhancement New feature or request 1.23 Release 1.23 Features qeff.weightfree WeightFree Onnx Export based on Dyanmo and Meta state labels Jul 13, 2026
vbaddi
vbaddi requested changes Jul 13, 2026
View reviewed changes
Comment thread QEfficient/transformers/models/qwen3_moe/modeling_qwen3_moe.py Outdated

# Normal export: compute derived params — values embedded in ONNX.
self.gate_proj = nn.Parameter(self.gate_up_proj[:, : self.expert_dim, :].detach().clone().transpose(1, 2))
self.up_proj = nn.Parameter(self.gate_up_proj[:, self.expert_dim :, :].detach().clone().transpose(1, 2))

@vbaddi vbaddi Jul 13, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: why did we do clone for normal export, when original was just .detach() and without .clone()?
same question for all moe related changes.

@Cs23m011 Cs23m011 Jul 13, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are seeing the aliasing issue in dynamo without clone () method. It will not export in dynamo without clone as it will get nodes with same name. And in weight free export we are not using this the else block anyway.

@vbaddi vbaddi Jul 13, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re: And in weight free export we are not using this the else block anyway. If not then we should keep it same as before right?

@amarquic amarquic Jul 13, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But if user uses Weight_free_export=false then it will go via dynamo path and changing that will fail dynamo export because of aliasing issue.

Comment thread examples/performance/compute_context_length/weight_free_ccl_verify.py
@@ -0,0 +1,193 @@
#!/usr/bin/env python3

@vbaddi vbaddi Jul 13, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Can you add a readme (.md under /docs) on how this weightfree approach works? details on how to enable it? How download/conversion happens? and conversion being onetime thing, How we can reuse the .safetensors across machines and runs?

@amarquic amarquic Jul 13, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay I will add that.

@amarquic amarquic Jul 14, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

Comment thread QEfficient/exporter/weight_free/spec.py
@@ -0,0 +1,96 @@
#

@vbaddi vbaddi Jul 13, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's create a 'weightfree/' directory and push these weightfree related things under it?

@amarquic amarquic Jul 14, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Please review.

Comment thread QEfficient/customop/ctx_scatter_gather_cb.py Outdated
pass

@staticmethod
def backward(ctx, grad_output):

@vbaddi vbaddi Jul 13, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: if we call model.eval(), is backward expected? Are these changes required?

@amarquic amarquic Jul 14, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed. Please review.

Comment thread scripts/preprocess_moe_checkpoint.py
@@ -0,0 +1,318 @@
# -----------------------------------------------------------------------------

@vbaddi vbaddi Jul 13, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unit test infra is missing, pls work @smedhe and @quic-amitraj to get this added to avoid manual runs.

@vbaddi

vbaddi commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

lint and format are missing. Pls follow the workflows to enable and repush once its fixed.

amarquic added 4 commits July 13, 2026 16:39
@amarquic
chore(lint): fix ruff E741/F401/F811/F841 and format all changed files
d165fdc 
- cache_utils.py: restore missing return k_out, v_out in QEffHybridCache.update()
  and QEffSlidingWindowCache.update(); remove orphan method blocks that were
  left floating inside wrong classes after rebase
- exporter/__init__.py: use explicit re-export syntax (X as X) for F401
- checkpoint_transforms.py, glm4_moe, grok1, preprocess_moe_checkpoint.py:
  rename ambiguous variable I to ffn_dim / hidden_dim (E741)
- gemma, gemma2, qwen3: remove unused mask_value assignment (F841)
- Run ruff format across all 39 changed files

Signed-off-by: Amar <amarshar@qti.qualcomm.com>
@amarquic
test(dynamo): add weight-free export test lane + dynamo test infra
2395871 
Creates tests/dynamo/ following the same structure as PR quic#1174:
  - conftest.py: markers, dynamo_workdir fixture, xdist-safe result capture,
    HTML/JSON report generation
  - utils/hash_manager.py: blake2b-keyed workdirs (collision-safe under -n auto)
  - utils/hardware_manager.py: file-lock QAIC device pool for HW lanes
  - utils/report_generator.py: HTML + JSON coverage matrix output

Extends DynamoModelSpec in model_registry.py with two new flags:
  weight_free_supported     - True for all validated causal LM architectures
  weight_free_cb_supported  - True for all architectures where CB is also verified
  Both flags False for VLM text-side models (out of scope for this PR)

Adds tests/dynamo/causal_lm/test_weight_free.py with 4 CPU-only lanes:
  1. test_weight_free_export        - meta-device export produces ONNX + weight_spec.json
  2. test_weight_free_no_embedded_weights - ONNX has no large embedded initializers
  3. test_weight_free_spec_coverage - all spec keys resolve in prepared_checkpoint/
  4. test_weight_free_cb_export     - continuous_batching=True export produces artifacts

All lanes are @pytest.mark.dynamo_export (CPU, no QAIC hardware) and
@pytest.mark.regular. Run with:
  pytest tests/dynamo/causal_lm/test_weight_free.py -m 'dynamo_export' -n auto

Signed-off-by: Amar <amarshar@qti.qualcomm.com>
@amarquic
feat(weight-free): add MoEFusedExpertSplitCheckpointTransform + docs …
281baaf 
…+ cleanup

- exporter/weight_free/transforms.py: add MoEFusedExpertSplitCheckpointTransform
  for MoE checkpoints that already store experts in fused/stacked form
  (gate_up_proj [E,2I,H] + down_proj [E,H,I]). Splits them into the derived
  layout that QEff ONNX initializers expect (gate_proj/up_proj/down_proj_t).
  Registered in pipeline between MoEExpertStackingCheckpointTransform (old
  per-expert format) and DtypeConversionCheckpointTransform (dense fallback).

- modeling_auto.py: register MoEFusedExpertSplitCheckpointTransform in
  QEFFAutoModelForCausalLM._checkpoint_transforms pipeline

- customop: remove backward() stubs from CtxScatter/Gather — not needed
  for dynamo weight-free export (verified by test)

- exporter/weight_free/core.py: fix sys.path for scripts/memory_profiling
  (extra .parent for new subdirectory depth)

- docs/source/weight_free_export.md: add weight-free export documentation

- examples/text_generation/weight_free/: update docstring paths and add
  __init__.py

Signed-off-by: Amar <amarshar@qti.qualcomm.com>
@amarquic
fix(weight-free): add gate↔router lookup for Qwen3-MoE router weight
3614992 
Qwen3-MoE stores the router weight as mlp.router.weight in the checkpoint
but the QEff wrapper accesses it as mlp.gate.weight. Add lookup strategy 6
to _find_checkpoint_key to handle this rename in both directions so the
weight is correctly promoted to an external reference instead of remaining
as an embedded meta tensor that causes SerdeError.

Signed-off-by: Amar <amarshar@qti.qualcomm.com>
@quic-amitraj
quic-amitraj force-pushed the quic-enable-dynamo-for-causallm branch 2 times, most recently from a5ebf6f to 7082a82 Compare July 21, 2026 07:49
@amarquic
fix(weight-free,dtype): respect model config dtype throughout export …
4f5f49c 
…pipeline

- core.py: use config.dtype (not hardcoded float32) for meta model cast and
  checkpoint conversion; getattr fallback handles dtype=None configs
- core.py: propagate enable_proxy flag through _build_meta_qeff_model
- cache_utils.py: use v_out.dtype instead of torch.float32 for invalid-mask
  zeros_like in 5 KV cache update paths — avoids fp32↔bf16 dtype mismatch
- modeling_llama.py: use attn_weights.dtype for masked_fill and softmax
  upcast instead of hardcoded float32
- export_compile_infer.py: add --dtype flag (default float16) to control
  model weight dtype; add enable_proxy=False
- modeling_qeff.py: skip SplitTensorsTransform for weight-free export

Signed-off-by: Amar <amarshar@qti.qualcomm.com>
@quic-amitraj
quic-amitraj force-pushed the quic-enable-dynamo-for-causallm branch 3 times, most recently from 7e7959d to 1f893d5 Compare July 23, 2026 15:03
@vbaddi

vbaddi commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

duplicate #1220

@vbaddi vbaddi closed this Jul 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vbaddi vbaddi vbaddi requested changes
+1 more reviewer
@Cs23m011 Cs23m011 Cs23m011 left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

@quic-amitraj quic-amitraj

@amarquic amarquic

Labels

1.23 Release 1.23 Features enhancement New feature or request qeff.weightfree WeightFree Onnx Export based on Dyanmo and Meta state

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@amarquic @vbaddi @Cs23m011 @quic-amitraj @smedhe