feat(mlx): support Qwen3.8 Flash Next (qwen4_exp) - #2283
Open
tharalds wants to merge 1 commit into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Qwen3.8 Flash Next uses the
qwen4_exparchitecture, whose QSA/GatedDeltaNet cache, hyper-connections, n-gram PLE storage, and weight sanitizer
currently live in MLX-VLM rather than MLX-LM. EXO could discover the model but
could not load or generate from a compatible checkpoint through its MLX engine.
Changes
continuous-batching, MTP, and external-PLE support.
qwen4_expthrough MLX-VLM while preserving MLX-LM loading for everyexisting architecture.
snapshots, QSA trimming, prefill rollback, and prefix-cache reuse.
parallel placement remains disabled until QSA/GDN/MoE/PLE sharding has a
dedicated bit-exact implementation.
low,medium, andxhightemplate values and recognize its EOS tokens.sh0wie/Qwen3.8-Flash-Next-REAP-288-MLX-4bitcheckpoint. Vision is notadvertised because the checkpoint publisher has only evaluated text after
pruning.
smoke harness, and operator documentation.
The disaggregated-prefill wire path now rejects MLX-VLM caches explicitly
instead of attempting to serialize incomplete recurrent/QSA state.
Why It Works
MLX-VLM owns the current
qwen4_expimplementation and sanitizes its officialconditional-generation weights. EXO retains only that model's language module,
wraps its structured output as logits, and continues to use EXO's existing chat,
generation, placement, and prefix-cache orchestration.
The recommended REAP-288 checkpoint corrects two source-conversion defects:
RMSNorm tensors use the zero-centered convention expected by MLX-VLM, and PLE
tensors plus per-tensor quantization overrides use the runtime's module names.
Its external PLE manifest also keeps active memory substantially below the
checkpoint's on-disk size.
Test Plan
Manual Testing
that no indexed file was missing or empty.
41.7 GB active memory.
coherent response, peaking at 44.5 GB active memory.
section.
from 2.8 seconds to 0.7 seconds while returning identical correct output.
produced the same incoherent output; it is deliberately not cataloged.
Automated Testing
deselected), covering Qwen4-Exp loading, real Metal prefill/decode, QSA and
recurrent cache restore, pipeline handling, reasoning, API tool use, parsers,
and disaggregated-prefill behavior.
ruff check .ruff format --check .basedpyright— 0 errors, 0 warnings.uv lock --check