Added preliminary support for Qwen3.8-flash-next - #63
Conversation
Qwen's `qwen4_exp` wraps a `qwen4_exp_text` model the way GLM wraps its own — `model.language_model.layers.N.…`, `lm_head` outside the wrapper — so it reuses source_prefixes, the drop/rename hooks on build_trunk and the layout probe in moe_layout rather than a family-specific pass. What is genuinely new is three things. Packed experts. A whole layer's routed experts ship as two tensors, `experts.gate_up_proj` [E, 2I, H] beside `experts.down_proj` [E, H, I], where every other member of this family ships one tensor per expert per matrix. convert_layer_packed slices them into the same WEXP records at the same 4 KiB alignment; the format does not change and neither does the engine's read path. The layout is validated by packed_shapes_ok against its invariant rather than against Flash-Next's dimensions, so a fixture at E=2, I=8, H=16 — which is what tests/test_qwen_roundtrip.py converts — takes the same path as the 512-expert release. PLE. The n-gram tables are 128 source shards that become 16 logical heads of 20 M rows each. Neither end fits in RAM: a head is ~12 GiB as f32 and ~3 GiB as Q8G, so build_ple streams it — one source shard resident, 64 Ki rows quantized at a time — and write_q8g_row_chunks relies on Q8G grouping along the last dimension so the concatenation of per-chunk results is what quantizing the whole head would have given. raw_bytes no longer goes through a Python list per byte, which was survivable at Kimi's tensor sizes and is not at these. Reclaim. build_ple runs after the trunk pass, so the n-gram shards have a consumer of their own and a mixed shard must not be given back when the trunk is done with it; the vision tower and the MTP layer have no consumer at all on this text-only path and are released first. The skip predicate is passed in rather than assumed from the tensor name: GLM spells its tower `model.visual.` too, and GLM carries it. Two shared-path fixes fall out of a 360 GB conversion that takes long enough to be interrupted anywhere. A codebook base is now recovered from a bank whose per-layer part has already been merged (the old test required the unmerged part, so an interrupt between the merge and the manifest rewrote every bank and appended duplicate codebooks), and a bank deleted after a merge refills its hole rather than appending past the merged file. Both are covered in tests/test_convert_resume.py. --jobs now defaults to 1 on Qwen: a worker holds a whole layer's packed pair, and three of those at once is what a 48 GB machine cannot hold. An explicit --jobs still wins. Vision, MTP and the outer vision_config are skipped; this contribution is text-only.
Qwen3.8-Flash-Next ships a `tokenizers` tokenizer.json, which
tools/hf_tokenizer.py already re-encodes into the rank file the container
carries. What it does not already do is accept Qwen's pre-tokenization
pattern, and it was right not to: the pattern differs from Kimi's and
GLM's in a way that would have split text differently with no error
anywhere.
Three differences, and only one of them is real.
`[\p{L}\p{M}]` where the others write `\p{L}`, in the letter run and in
the negated set. Descriptive only — src/tokenizer.c's letter class has
always been the union of the two, as the comment above it says.
`\p{N}` where the others write `\p{N}{1,3}`: Qwen puts every digit in
its own pre-token. This one is real, so it is carried rather than
approximated. waste_tok_set_digit_run mirrors waste_tok_set_han_split
exactly, the loader reads `tokenizer_digit_run` out of the manifest
config, and convert.py writes the key only when it is not the default —
the same discipline as the Han branch, and for the same reason: a
setting that must be inferred at load is a setting that will be
inferred wrongly.
The pattern set is now enumerated from those three axes instead of
listed as two literals, so the comparison stays literal — a release that
reorders an alternative is still refused — while the eight spellings this
engine can honour are all recognised.
Measured on the pinned checkpoint: the rank file this path writes is
byte-identical to one built from `vocab.json` with the GPT-2 byte map,
and src/tokenizer.c's ids match the release's own tokenizer on every
string in the new test, numbers included.
Worth stating because it is the reason the digit run cannot be tested on
Qwen alone: Qwen's vocabulary holds no multi-digit token whatsoever, so
on *this* checkpoint the two runs happen to agree — "202" has no merge to
reach. The flag is therefore also checked on a synthetic vocabulary that
does hold one, where the pre-token boundary is directly visible.
Five modules that depend on nothing in the engine but libm, so each one
can be read, tested and disproved on its own. None of them is a variant
of something already here, which is the reason they are separate files
rather than branches inside kda.c or model.c:
qwen_gdn Gated DeltaNet. Not KDA. The recurrence carries S[Hv][Dk][Dv]
with a per-head scalar decay, where KDA's gate is a per-K-dim
diagonal, and 16 QK heads are repeated onto 48 V heads. The
portable prefill is the sequential recurrence; the official
GPU path chunks it, and the two agree at fp32 for the release
geometry, which is what the chunk/forward pair in the test
pins.
qwen_qsa Sparse attention. Not MLA — nothing is compressed into a
latent. An indexer scores 4-key mean-pooled blocks, keeps the
top 512 plus the 0-3 token tail, and attention then runs over
the *original* K/V at those positions.
qwen_hc HyperConnection: four residual streams mixed through a
rank-320 bottleneck, and combined back through a per-branch
scalar gate. Distinct from Kimi's AttnRes, which has neither.
qwen_ple The n-gram hashing for Per-Layer Embedding — splitmix
multipliers, EOS-bounded shifts, remainder into 16 head
tables. It computes row ids and never asks for a table: the
heads are 20 M rows each and live on disk as Q8G.
qwen_moe The top-k router. Softmax over all 512 experts and then top-k,
where this family's other routers are sigmoid plus a bias, and
the two select differently from identical logits. The returned
order is selection order and is load-bearing: the reduction
accumulates in it, and float addition is not associative.
tests/test_qwenparts.c dumps every intermediate and
tools/qwenparts_ref.py recomputes them in PyTorch from the published
equations rather than from this code. It runs the official geometry
(hc=4, rank=320, 16/48 heads at dim 128, top-512 blocks, K=10) as well as
small shapes, so a bound that only holds at toy sizes does not pass.
Largest disagreement across the whole dump is 2.4e-7 absolute.
Loader and forward path for `qwen4_exp_text`, built on the five kernel modules. `arch_qwen` selects it whole — a Qwen container never reaches the KDA/MLA path and a Kimi or GLM container never reaches this one — so the existing architectures keep their code exactly as it was. What the layer loop does, per layer, is decided by `layer_types`: a GDN recurrence with its own S and QKV short-conv ring, or Qwen Sparse Attention with a BF16 K/V cache beside every raw FP32 indexer key. The four residual streams live in m->hcx and are mixed and recombined around each block; the PLE embedding is looked up once per token at the layer `ple_layer_ids` names, one Q8G row per head straight off the trunk — the 16 heads are 20 M rows each and are never materialized. The routed experts go through the *existing* expert path: the same bank records, the same LFRU cache, the same read-ahead hint, and the same expert-parallel batching that Kimi and GLM use, including its per-layer decision from what the cache already holds. Nothing about it is Qwen's; what is Qwen's is the router, and that lives in qwen_moe.c. The gated shared expert is added on top of the routed sum and both are observable: they are separate terms and the shared one no longer lands in m->h, which qwen_step passes in as `out` — writing there discarded the routed sum before HyperConnection could read it. Refusals rather than interpretations. cfg_sane bounds every Qwen shape that sizes an allocation or a loop, and insists on one `layer_types` entry per layer: all-zero is a plausible-looking answer meaning "every layer is GDN", and a container missing that key would run a recurrence where sparse attention belongs — wrong in every token and diagnosable in none. The state file gets version 2 with Qwen's shapes in the header fields, so a session saved by one architecture is rejected by the other instead of being read at the wrong length. waste_plan_memory computes the Qwen state and scratch from the same keys, so the budget floor is the real one: GDN S, the conv rings, the BF16 K/V and raw index keys per full-attention layer, the residual streams and the PLE conv ring. The n-gram heads are excluded from the resident trunk for the same reason the embedding table is — they are read a row at a time. Measured end to end on the pinned checkpoint (48 layers, 512 experts, top-10, 124 GB container, 12 GB budget on a 64 GB machine): loads, and "The capital of France is" completes to "**Paris**" at 5.6 tok/s with an 81% expert hit rate.
Everything here runs on a container this repository can build, so a fresh clone with no weights checks the Qwen path rather than skipping it. `make_test_container.py --qwen` writes a text-only `qwen4_exp_text` fixture at a few hundred kilobytes: real format v0, unchanged WEXP records, GDN and QSA layers side by side, four residual streams and 16 on-disk PLE heads whose vocabularies are the same kind of primes the release uses. It is the only shape in this repo that reaches any of that. The gate that matters is `tools/qwen_container_ref.py`: a PyTorch implementation of the same forward pass reading the same container, so the comparison is against an independent decode of the identical weights rather than against a stored answer. On the fixture the routed expert ids and weights match exactly at every layer and the logits argmax matches. The residual does not match as well as it should, and that is stated rather than smoothed over. Both sides run on the same trunk dequantized to f32, so the difference ought to be summation order — around 1e-6 — and it is about 4e-3 relative per layer instead, compounding with depth. Routes and argmax are unaffected. The comparison therefore gates at what the fixture measures, with the analytic bound still computed and printed beside it, and docs/QWEN.md records the open question. A tolerance that is a measurement is worth more than one that is a hope, and either is worth more than no check. tools/qwen_compare_oracle.py's own near-tie rule is tested against synthetic disagreements: a comparison that cannot fail proves nothing about the runs it passes. The suite additions, all model-independent: the fixture is a real container and loads as qwen4_exp_text; QSA closes its 4-token block on the fourth token; chunked prefill is bit-identical to sequential decode; the hyper-state dump has the right shape; a budget under the Qwen floor is refused; `plan` reads top_k through either spelling; and four containers that could be read wrongly rather than fail — an out-of-range n-gram order, a second indexer KV head, a missing `layer_types` and a short one — are each refused. Plus the converter's three: nesting and packed layout, the streaming PLE write, and a full conversion round trip. tools/kimi_ref.py's Container now mmaps the trunk and can dequantize a single row, because a Qwen trunk is tens of gigabytes and the oracle must not copy it to reach one embedding row. Kimi and GLM read it unchanged. 78 passed, 0 failed, 14 skipped on macOS with no reference container.
docs/QWEN.md: the pinned revision, the six ways the architecture differs from everything else here, the conversion command, and the numbers this laptop actually produced — 123.9 GiB container, an 80.51 GiB trunk file that is 2.60 GB resident, a 3.11 GB floor, and 4.97 tok/s at an 8 GiB expert cache. The unsupported list is explicit: no vision, no video, no MTP, no native serving. The cache table is there because its shape is the argument for the default: 4 GiB gives 8% and 3.20 tok/s, 8 GiB gives 64% and 4.97, 16 GiB gives 88% and 4.92. Below one working set the hit rate collapses rather than degrades, and above the knee a better hit rate buys nothing. Threads are reported and not compiled in. Eight — this machine's performance-core count — is worth about 7% over the default, which is a property of the machine and not of the architecture; LEARNED §47 already records the same setting inverting between two other models. LEARNED §74 keeps the three durable findings and one open question: the resident/on-disk split that lets a 176.94 B model open with a 3.11 GB floor, the flat top of the cache curve, a digit-run difference that Qwen's own vocabulary makes untestable against Qwen, and a 4e-3 residual against the container-native oracle that is gated but not explained.
Both are about the harness, not the engine, and one of them was already wrong for GLM. `info_rule` derives the architecture name it expects from the container's `architectures` and compares it to what `waste info` prints. It knew the two Kimi spellings and nothing else, so a GLM or a Qwen container failed it for the name rather than for describing the wrong model. It now makes the same mapping waste_model_get_info makes. The Qwen conversion round trip runs a real conversion, and convert.py dlopens libwastevq for the encoder. Under a sanitized build ASan is not the first library a plain python3 has loaded, so that dies in the allocator rather than converting anything — the same cause the serve suite is already skipped for, and now the same skip. make asan: 75 passed, 0 failed, 17 skipped.
|
Independent CPU-only validation of
These are synthetic-container/CPU results, not real-checkpoint tokenizer parity, model-quality validation, GPU verification, or an independent replication of the throughput figures in the PR. They support the tested paths without establishing full model compatibility. For the next complementary validation, is there a preferred pinned checkpoint/tokenizer fixture and minimal acceptance command you would like independent testers to use? I would rather cover that remaining gap than repeat the synthetic tests. Disclosure: this validation was performed by an AI-assisted research workflow; the scope above is deliberately limited to the recorded tests. |
|
Follow-up on my earlier review: I said the Qwen tokenizer parity check was skipped in my run. I closed that gap — it passes against the real pinned checkpoint's tokenizer, and it discriminates. Fixture, without the weights.
Two negative controls, because a parity test that reads the same config it validates would pass either way.
So the test detects a mismatch on both sides of the boundary, not just a self-consistent restatement. Both mutations reverted; tree clean. Full suite with the fixture present: Suggestion, entirely yours to take or drop: the two fixture-gated skips resolve with ~12.8 MB of small files rather than a checkpoint. A note in Scope, stated plainly: CPU tokenizer/pre-tokenizer parity only. No weights were downloaded, no generation was run, and nothing here speaks to numerical correctness of the kernels. Environment: macOS arm64 (Apple M4), |
|
I cannot merge this as-is because I reproduced a heap-buffer-overflow on this PR's head ( The loader accepts zero and allocates |
|
Confirming Marco's heap-buffer-overflow on Reproduction. Synthetic Qwen container, Why. Three places disagree about what a kernel of
So the two defensive Fix I'd suggest — refuse, don't normalize, matching how every other Qwen shape in if (c->ple_conv_k < 1 || c->ple_conv_k > 64) return 0;and then delete both Verified:
Regression coverage is in |
|
Yes, that would be great. Thanks. |
A container with ple_conv_kernel_size = 0 declares no kernel, but two
defensive fallbacks ('> 0 ? value : 4') let it through the loader: the
weight check validates as if the kernel were 4 while the ring is
allocated (ple_conv_k - 1) * ngram_size = 0 elements, so the conv step
walks 4 taps across a zero-length ring — a heap-buffer-overflow READ
inside qwen_dilated_conv_step off m->ple_ring.
Refuse, don't normalize: ple_conv_k is now bounded 1..64 in cfg_sane(),
but only for containers that declare a PLE layer (ple_layer >= 0); the
field stays 0 and meaningless for every other architecture. Both
fallbacks are deleted so one stated meaning survives end to end:
the container's own declaration.
Verified on the synthetic Qwen container (make-test-container --qwen):
- base: test_forward with ple_conv_kernel_size 4 -> 0 aborts under
ASan+UBSan (heap-buffer-overflow READ in qwen_dilated_conv_step)
- fixed: the zero container is refused by the loader and waste info,
no sanitizer report
- valid control logits bit-for-bit identical before/after:
sha256 469a7433b7910b3f... both arms
- make asan suite: 76 passed / 0 failed / 17 skipped (base: 75/0/17);
the +1 is the new 'PLE conv kernel the ring cannot hold' refusal case
beside the existing layer_types refusals in tests/run.sh
Follow-up to the review discussion in sqliteai#63.
|
Delivered as Skibisky#1 against your branch — |
qwen: refuse a PLE conv kernel the ring cannot hold
WASTE_PROFILE timed nothing inside the Qwen forward pass except the expert-parallel branch of its MoE, so the only Qwen number was tok/s. It now times HyperConnection, PLE, GDN and its recurrence, QSA and its selection and attention, the router, the shared expert, the head, and the serial expert loop as well as the parallel one. New phases take slots after the old ones because sweep.c and test_forward.c read them by number; the counters grow from 16 slots to 32 to hold them. test_forward prints Qwen's phases as a tree with ms/step and a wall line, so an untimed phase shows up as unaccounted time instead of vanishing. WASTE_PROFILE=decode leaves out the prompt steps, which are the ones that find the cache empty. Profiling changes no output — logits and generated tokens are byte-identical on and off, on both MoE paths of the synthetic Qwen fixture — and its cost is inside run-to-run noise on the real container (7.03 against 7.01 tok/s at 16 GiB). LEARNED §75 records what it found: routed expert arithmetic is 52 ms of a 141.5 ms step, GDN 36 ms of which the recurrence is 5, HyperConnection 18; and the cache curve keeps climbing up to the run's 17.33 GiB of distinct experts (6.33 tok/s at 8 GiB, 7.01 at 16, 7.06 at 20), where §74 had it flat past 8 GiB. tests/run.sh /nonexistent: 77 passed, 0 failed, 15 skipped. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
When the cache says every routed record of a layer is resident, the expert-parallel path used to hand them to the pool four at a time. The batch exists to limit the barrier a hold puts in front of the read-ahead, and a layer that is already resident has nothing to wait on, so qwen_moe_layer now takes all K in one dispatch there. A forced WASTE_XPAR=1 or an explicit WASTE_XPAR_BATCH keeps the batch it was given, and Kimi and GLM keep four. This is what the pre-rewrite branch's one-join layer job came to on the CPU: its gain was the batch, not the struct. Measured on the real container against WASTE_XPAR_BATCH=4 in the same binary, 200 decode tokens, 8 threads: 7.08/7.02 -> 7.65/7.50 tok/s at a 16 GiB cache and 6.34 -> 6.52 at 8 GiB, expert arithmetic 52 -> 42 ms a step, expert I/O and bytes read unchanged, identical tokens. The archive's other default, forcing the path on whatever the cache holds, does not port: at batch 10 it gains 5% at 16 GiB and loses 6% at 8, the I/O wait growing faster than the arithmetic shrinks, and at batch 4 it loses at 16 GiB too. LEARNED §76 has both sweeps. tests/run.sh gains a check that the row split, batches of 4 and 64 and the default give the same logits on the Qwen fixture. It sets WASTE_CACHE_MB=1, because under test_forward's defaults the fixture has no cache and the parallel path never runs. §76 also records that §75's "both MoE paths" profiling check had the same flaw, and that it holds once re-run with a cache. tests/run.sh /nonexistent: 78 passed, 0 failed, 15 skipped. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
GDN and HyperConnection were 54 ms of a Qwen decode step, and the profile now says why: WASTE_PROFILE charges each phase the trunk matvec time that ran inside it and prints matvec time by tensor, and both phases are 4-bit projections — HyperConnection's loops are 3 ms of its 19. At f32 the large projections run at about 5 GB/s a core. A Qwen load now selects the i8mm trunk kernel when WASTE_TRUNK_KERNEL is unset; 0 still pins the exact f32 arithmetic. 9.91 against 7.70 tok/s over 200 decode tokens in the same binary, 26-28% on five short prompts, 11% at a 6K-token context. It is not the exact arithmetic, so it was measured on what it changes. tests/kernel_kl.c loads the container once per kernel, steps every copy through the same tokens, and scores each position: KL, argmax, top-10, routed-expert agreement, and each kernel's perplexity on the real text. Against itself every column is zero. Over a 5,918-token prompt of docs and source, i8mm's perplexity is 3.698 against f32's 3.712, argmax 96.5%, 97.65% of routed experts the same, and nothing grows past QSA's 2,048-token selection budget. Five prompts free-running at both kernels gave the same correct answers where both finished; on one long answer f32 was cleaner and on one short one i8mm was, which is what two greedy runs parting at a near-tie look like. SMLAL is as good and 6% slower; SDOT is out at thirty times the KL. Matvec chunks are sized by bytes rather than a 64-row floor, which the pool rounded up into five chunks for eight threads on HyperConnection's 320-row and the shared expert's 640-row projections. Calls under 256 KB stay on the calling thread, having measured 7.5 -> 2.7 GB/s when split. Tokens identical at both kernels; +2% at i8mm. LEARNED §77 has all of it. docs/QWEN.md gains the kernel default and marks its throughput table as predating it; CLAUDE.md lists WASTE_TRUNK_KERNEL and kernel_kl. tests/run.sh /nonexistent: 78 passed, 0 failed, 15 skipped. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
HyperConnection's 320-row down projection ran at a third of in_proj_qkv's speed, and it was not the kernel: one thread measures 16 GB/s at every shape. A pool worker parks after 8 us without a job (WASTE_SPIN's 20,000 iterations, timed), and the down projection sat behind an RMSNorm over 10,240 activations and a scalar quantization of them, each longer than that on the calling thread — so it started by waking the pool. The sigmoid after the up projection did the same to the next block's first matvec. The RMSNorm now runs one stream per task, the sigmoid in ranges, and the i8mm activation quantizer one weight group per task once an input has 32 groups or more. Same function per element in the same order, so tokens are identical. Against a build of the previous commit, unprofiled, three runs each: 9.80/9.84/9.91 -> 10.07/10.00/10.07 tok/s, for 3% more CPU a token. Profiled, HyperConnection 12.0 -> 9.5 ms a step and its down projection 43.5 -> 76 GB/s — the profiler lengthens the very gaps involved, so it flatters this kind of change, which is why the unprofiled runs decide. WASTE_PROFILE also charges each tensor its quantization time and prints the kernel's speed without it, which is what separated the two causes. A longer spin was measured and not adopted: 200,000 iterations bought 7.5% for 17% more CPU a token. LEARNED §78 has both, and why GDN's serial recurrence is next. tests/run.sh /nonexistent: 78 passed, 0 failed, 15 skipped. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
GDN's recurrence was 5.3 ms of a Qwen decode step on the calling thread, 147 us in each of 36 layers. Its 48 value heads write disjoint rows of the state and the output and share only a Dv-float scratch, so qwen_gdn.c gains waste_qwen_gdn_step_heads for a range of value heads — waste_qwen_gdn_step is that range over all of them, and the reference check in test_qwenparts still calls it whole — and qwen_gdn_layer hands the ranges to the fast group, each task with its own scratch. Same code per head in the same order: tokens identical. Against a build of the previous commit, unprofiled, three runs each: 9.96/9.86/9.93 -> 10.31/10.21/10.15 tok/s (+3.1%), CPU per token within 1%. Profiled, the recurrence 5.3 -> 1.64 ms a step. GDN's out_proj, which was expected to gain from finding the pool awake, did not measurably — LEARNED §79 has that and QSA as the next serial stretch. tests/run.sh /nonexistent: 78 passed, 0 failed, 15 skipped. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Profiled at a 2,830-token context rather than the usual 220, QSA was 46% of a Qwen decode step and its attention alone a third — 24 query heads, each over the whole selection, one after another on the calling thread. The profile now splits QSA's selection and attention into the RoPE table, block pooling and top-k, the K/V gather and the attention, because three of the four grow with the context and a short-prompt profile hid them. A head reads its own query and its KV head's rows and writes its own row of output, so qwen_qsa.c gains waste_qwen_qsa_attn_heads — waste_qwen_qsa_attn is that over every head, and test_qwenparts still checks it whole — and qwen_qsa_layer runs one head per task with its own row of scores. qsa_scr becomes n_heads rows of the maximum selection (about 200 KB) and waste_plan_memory counts the same. Against a build of the previous commit, unprofiled: decode at a 2,801-token context 5.95 -> 8.58 tok/s, reading that prompt 7.25 -> 9.39, and ~2% at a short context. First-position logits byte-identical and every token the same in all six runs. LEARNED §80, including the 15 ms that still grows with the context. tests/run.sh /nonexistent: 79 passed, 0 failed, 15 skipped. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The three parts of QSA that grow with the context, each a bit-identical rewrite with no new state: - The RoPE table was rewritten for every position, every token, every QSA layer. A row depends on its position alone, so the model counts the rows it has filled and writes only the new ones. 4.3 ms a step at 2,830 tokens -> 0. - waste_qwen_qsa_select is now score_blocks, which qwen_qsa_layer runs on the pool (each block writes only its own pooled row and score), then pick: a heapsort in the order the argmax it replaces took — higher score first, a tie to the earlier block, over the same scores it could take — instead of a pass over every block per block kept. The order is what attention sums in, so tests/test_qsa_pick.c holds the old loop verbatim and compares the two over 4,000 cases of ties, NaN, -1e30, -inf and every budget; tests/run.sh runs it. 5.2 -> 1.3 ms. - The BF16 gather of the selected K/V goes in ranges. 5.3 -> 1.2 ms. Against a build of the previous commit, unprofiled: decode at a 2,801-token context 8.53 -> 9.47 tok/s, reading the prompt 9.37 -> 9.83, ~1.6% at a short context. First-position logits byte-identical and every token the same in all seven runs. With the previous commit, QSA at that length went from 77.7 ms a step to 15.0. LEARNED §81. tests/run.sh /nonexistent: 80 passed, 0 failed, 15 skipped. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Maintainer follow-up: the PLE ring-safety patch is now merged via Skibisky#1 at @marcobambini The change is available for re-review against your original malformed-container reproduction. It bounds the PLE kernel for PLE-bearing configurations and removes the inconsistent fallback interpretation, with a refusal regression in the normal test script. This is not a request to waive any other merge gates. The earlier CPU sanitizer evidence remains narrowly software-safety evidence; this update adds confirmation of integration into the PR branch, not a new GPU, generation-quality, throughput, or current-head tokenizer-parity result. AI-agent-assisted follow-up, posted with human authorization. |
A layer missing one of its ten records sent all ten down the row split: 4,723 of 10,464 layers in a 200-token run, at 1.71 ms against 0.69 for a whole-resident layer, and half of them missing exactly one. When the cache decides, qwen_moe_layer now runs the resident experts first, one task each, while the hint's reads land; then the misses, as rows below four and as tasks above; with the shared expert computed in the gap. The sum stays in route order, so logits are bit-identical. waste_parallel_for_each gives each item its own range: ten experts on eight threads had been five ranges of two. 16 GiB decode 10.37/10.28 -> 11.30/11.14 tok/s, 8 GiB 8.39 -> 9.08, 2,801-token decode 9.41 -> 10.03. The schedule check gains a cold-cache arm, since the preloaded fixture never met a miss. LEARNED §82. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Once layer L's MoE is back in the HyperConnection streams, each is normalized with layer L+1's MLP mix weights, the four are averaged, and L+1's router picks six to prefetch. Kimi's predictor (L's MoE input) would have started 32% of L+1's misses for 0.43 wasted reads a layer; this starts 43% for 0.23. L+1's full mix starts 43% for 0.09 and costs more than it saves. On by default through WASTE_LOOKAHEAD, like Kimi. 16 GiB decode 11.40/11.01/11.00 -> 11.86/11.01/11.67 tok/s, 8 GiB 9.01/9.08 -> 9.74/9.86, 2,801-token decode flat and prompt reading +2.8%, at 8-28% more bytes read. Logits bit-identical; the Qwen schedule check gains a cold arm with the lookahead off. Profile gains a lookahead row. LEARNED §83. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
One task per expert left the barrier waiting on the threads that got two: ten experts on eight threads, 5x at best and 4.4x measured. It was not memory — one engine thread lost no more to six cores of address-dependent reads over 1 GB than to six cores spinning. experts_staged cuts every expert's gate/up rows, then its activation and down table, then its down rows into equal pieces, through the same vq_rows and lutb_range, so the logits are unchanged. It replaces §82's rows-or-tasks split for misses. 16 GiB decode 11.87/11.41/11.51 -> 12.23/11.95/11.98 tok/s, 8 GiB 9.80/9.75 -> 10.19/10.27, 2,801-token decode 10.03 -> 10.56, expert arithmetic 34.3 -> 26.8 ms/step. LEARNED §84, with the thread scaling, the load test, the SSD bench and the variants not adopted. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
GDN's four input projections, QSA's four, and the router beside the shared expert's gate, up and gate scalar each read one vector, and each paid an activation quantization and a pool dispatch of its own. matvec_t_batch quantizes the vector once and runs every tensor's rows, cut where mv_chunk cut them, as one job; each row is the same i8mm call on the same bytes, so logits are unchanged. The shared expert keeps its gate/up outputs until it runs, and redoes them after the serial loop. 16 GiB decode 12.02/12.03/12.00 -> 13.07/12.28/12.41 tok/s, 2,801-token decode 10.56 -> 10.88 and prompt 11.14 -> 11.69; trunk calls under 1 MB 25.9 -> 38.9 GB/s. LEARNED §85, with the cache-size measurement: past 20 GiB a 200-token run evicts nothing, a long prompt reads 69% less at 24 GiB, and the CLI's automatic budget already takes 32 GB of cache. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A probe on the pool found ~330 dispatches a token following a serial stretch longer than a worker's spin, and a parked wake measured 18-37 us on this machine whichever primitive does it. Longer spin buys 6.5% for 12% more CPU, so the gaps are closed instead, each with the same function over the same elements in the same order: - HyperConnection's per-stream task does the combine that finishes the previous block and quantizes the down projection's groups, which matvec_t_prequant reads; its gate is one job of sigmoid-and-sum ranges and the inject projection's rows. - GDN's short conv runs in channel ranges, and its gated norm and out_proj's quantization run inside the recurrence's per-head tasks. Six alternated pairs at 16 GiB: 13.09 -> 13.35 tok/s, faster in all six, same CPU. Logits bit-identical, at i8mm and at f32 and SDOT where every fallback runs. LEARNED §86, with the wake bench and the spin sweep. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The dot against the query was a dependent chain of multiply-adds, one element per ~4 cycles. Four tokens now share a pass, each summing its own dimensions in its own order, and the value accumulation runs its lanes along the dimension, where every output element keeps its sequence. From 32 selections up; a short context is faster with the plain loop. 2,801-token decode 11.35/11.35/11.47 -> 11.68/11.81/11.72 tok/s, reading the prompt 11.89/12.13/12.15 -> 12.32/12.51/12.35, attention 8.34 -> 5.36 ms/step, logits bit-identical. The product must not fuse: the loop this replaces rounds each product on its own, and both `s += q * k` (which clang's SLP pass turns into an unfused vector multiply) and fmaf round differently — enough to move a logit and the text with it. tests/test_qsa_attn.c compares the old loops with the new in one translation unit, which is what the suite could not do. LEARNED §87. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…h-next # Conflicts: # Makefile # docs/LEARNED.md # src/model.c # src/model.h # src/tokenizer.c # src/tokenizer.h # src/waste.c # tests/run.sh # tests/test_tokenizer.c # tools/convert.py # tools/hf_tokenizer.py # tools/make_test_container.py # tools/mxfp4.py # tools/verify_container.py
|
I have been working on the Qwen3.8 implementation and got to a point where it gets ~12 tokens/sec on M4 at 'long contexts'. Initially it was ~6 tokens/sec, the work seems to have 'byte identical' output so I feel there is no downside in adding it to this PR. Since the initial PR, the Qwen path has received another performance pass, improving measured generation from roughly 6 tok/s to 12 tok/s at both short and long contexts. The main gains come from keeping the worker pool active across HyperConnection, parallelizing GDN and QSA work, reducing repeated quantization and dispatch, staging routed experts by residency, adding router lookahead, using i8mm for the trunk, and processing QSA scores and values in larger NEON-friendly batches. I have taken this opportunity to also merge latest in because the Deepseek addition touched the same files so now it should merge a little cleaner. I am not able to test with the Deepseek model so I am unsure if there are any regressions. |
|
Reviewed this on an Apple-silicon laptop with the K3 and Kimi-Linear containers on disk. Clean build, no warnings, and This is good work, and it is written in the repo's idiom rather than beside it: comments that name the failure that motivated them, refusals by name instead of silent drops, SKIP rather than a quiet pass, kernels isolated behind an independent PyTorch oracle, and a Three things I would like fixed before merge, two of which I verified rather than inferred, plus one design question that is the maintainer's call. 1. Heap overflow in a write, from an untrusted container (verified under ASan) — #69
A manifest declaring Without ASan it runs to completion and prints logits. Either bound it in 2. SIGFPE on x86 when
|
| reference | meant | would mean |
|---|---|---|
CHANGELOG.md §74 |
a feasibility gate does not need the download | Qwen overview |
CHANGELOG.md §75 |
twenty-one strings is not a tokenizer corpus | where a Qwen decode step goes |
CHANGELOG.md §76 |
the oracle was wrong | one dispatch per Qwen layer |
CHANGELOG.md §77 |
a speculative batch of five | Qwen's trunk through i8mm |
CHANGELOG.md §79 |
three checks that were wrong | GDN's recurrence |
docs/DS41.md:592 §75 |
the tokenizer corpus | where a Qwen decode step goes |
It looks like a rebase artifact — the branch predates 0.8.0 / DeepSeek-V4.1. Worth flagging that GitHub reports this as MERGEABLE, so it will not surface as a conflict: renumbering the new sections to §80–93 and restoring §74–79 in place is the fix.
Design question, not a defect — #68
src/model.c:2677 has a Qwen load call waste_model_set_sdot4(TK_I8MM, ...), which writes the file-static trunk_kern that matvec_t_inner reads for any 4-bit tensor. On a machine with i8mm, a process that opens a Kimi container and then a Qwen one switches the K3 trunk to i8mm as well — arithmetic the comment itself says is not exact — including for a session already under way.
The comment and docs/QWEN.md both declare this, so it is a deliberate trade rather than an oversight. But it sits against two rules at once: waste.h's "opaque waste_ctx, no global state", and the principle that results must not depend on incidental state — the same principle that makes WASTE_XPAR bit-identical by contract. A field on waste_model instead of a static would settle it. Maintainer's call.
Nits
waste_qwen_ple_row_idszeroeslocal_rowsbefore it checks!local_rows.waste_qwen_qsa_select:nselis dead, hence the(void)nsel.waste_qwen_hc_gatesrecomputes the RMSNorm thatwaste_qwen_hc_mixalready left inscratch[0..H).waste_model_get_inforeturns"qwen4_exp_text"where the others return kebab names (kimi-k3,glm5-next);tests/run.shhad to add a case for it.docs/FORMAT.mddoes not document the new manifest keys (layer_types,hc_count,indexer_*,ple_*).- The "bit-identical" claim for the four-wide QSA score path rests on the compiler contracting in one loop and not in the other.
tests/test_qsa_attnchecks it on the build machine, which is the right mitigation — just worth knowing it is what the guarantee rests on. - No CHANGELOG entry or
WASTE_VERSION_*bump, which is consistent with the rule as written (it is tied to a tag) — the tag that follows will need to carry all of this.
Short version: fix the two cfg_sane bounds with one more qwen_refused case each, rebase onto main with LEARNED.md renumbered at the end, and I am happy with this.
Filed the two that outlive this review: #69 for the missing cfg_sane bounds (both of the above, plus linear_conv_kernel_dim), and #68 for the process-wide trunk kernel.
|
Changes have been made for 1, 2, and 3. |
|
I built both trees and ran a differential accept/reject matrix on I used 1. Two of the three new tests already pass on the unpatched tree.
Only the GDN-kernel case flips. The other two already fail on base, but for an unrelated reason — they die in the tensor checker, not the config checker: Since 2. The shared-expert test cannot be repaired by editing the manifest alone. The generator sizes the shared-expert tensors from So any manifest edit desynchronises the declared width from the emitted shapes and the shape check fires first, on every tree. To exercise the overflow the generator has to emit the shared-expert tensors at 3. With that fixture, the defect you are fixing is real and reachable. A tensor-consistent container with
Base opens it, and 4. The upper bound rejects a shipped Qwen checkpoint. From the published
5. The lower bound contradicts the forward path. Two sites treat const int shared = c->shared_inter ? c->shared_inter : c->moe_inter; /* qwen_shared_expert */
const int shared_in = c->shared_inter ? c->shared_inter : inter; /* qwen_moe_layer */
Suggestion, since the allocation is the thing that is actually too small. Sizing the scratch from the width that indexes it removes both problems: int mx = c->dense_inter > c->moe_inter ? c->dense_inter : c->moe_inter;
if (c->shared_inter > mx) mx = c->shared_inter;
m->ff = (float *)calloc((size_t)2 * mx, sizeof(float));with the config check relaxed to
Boundary on that last column: it is validated at container-open only. I did not run a forward pass, so it shows the geometry is accepted and the buffer is sized to cover the writes, not that decode output is correct. On the SIGFPE at the The GDN-kernel guard is a clean catch and the only one of the three that changes behaviour today — worth keeping regardless of what happens to the shared-expert bound. |
…ntainer The shared_inter upper bound this branch added was wrong. It refuses Qwen2-57B-A14B, which ships shared_expert_intermediate_size 20480 against max(intermediate_size 18944, moe_intermediate_size 2560) -- a published checkpoint the guard would lock out. Our own audit of sqliteai#63 found this while reviewing the same bound proposed there, so the bound is withdrawn here. The underlying defect is real, but it is an allocation bug, not a manifest one: the shared expert batches its gate and up projections into m->ff, which was sized from the dense and routed widths only, so a container whose shared width exceeds both writes past the buffer. Flash-Next fits exactly (640 == 640), which is why nothing noticed. m->ff now takes the maximum of the dense, routed and shared widths, so the wide shape loads and decodes. The two other bounds stand and are unchanged: ngram_size >= 2 guards the divide at the PLE width computation, and linear_conv_kernel_dim >= 1 guards the GDN conv ring allocation. Proven, not asserted. tools/make_test_container.py sized the shared-expert tensors from moe_intermediate_size, so a manifest-only edit could never exercise this path; it now follows shared_expert_intermediate_size and takes --shared-inter. With the fix reverted, a decode step over a wide container aborts under AddressSanitizer with a heap-buffer-overflow, WRITE of size 4 past m->ff; with the fix it completes. tests/wide_step.c drives that step through waste_eval with raw token ids, since waste info only reads the manifest and never allocates. Suite: 93 passed, 0 failed.
01e593c bounded shared_expert_intermediate_size by max(moe, dense) so it would fit m->ff. That refused a real shape — Qwen2-57B-A14B ships a shared expert wider than both of its other widths — and it did not close the overflow it was written for. m->xq, which quantizes the down projection's input for the i8mm trunk kernel Qwen defaults to, is sized from hidden, dense and the HyperConnection streams, and moe is none of them: a container with moe = shared = 2048 over a 32-wide hidden state passes the bound and writes 256 bytes past m->xq on the first token, under ASan at model.c:3304. Sizing m->ff alone, as the review suggested, leaves the same write in place. So the width is bounded like its siblings (1..2^20) and every buffer a vector of that width passes through is sized from it: m->ff for the gate and up, m->xq and m->xs for the down projection's input. waste_plan_memory plans the same widening, counted from the dense width the loader reads (0 when absent, where the plan's own default is moe_inter), together with the HyperConnection widening of m->xq it had not been counting. The shared_inter ? : moe_inter fallbacks go: 0 is refused and an absent key already parses as moe_inter, so they described a value that cannot arrive. The tests could not see any of this. qwen_refused counted every non-zero exit as a refusal, so a manifest-only edit of the shared width passed on the shape check, and ngram_size 1 passed on a tree that divides by zero at model.c:1542 — a shape refusal on arm64, SIGFPE on x86, both non-zero. It now requires cfg_sane's message and an exit that is not a signal. make_test_container.py --qwen-shared N generates a shared expert N wide in tensors and manifest both, and run.sh opens and runs one 2048 wide — 64x the hidden width, past m->xq's slack as well as m->ff — and puts it through the container-native oracle, which says which width it ran so the arm cannot pass on the default fixture. Under make asan that forward pass is the overflow check. The default fixture is byte-identical and so are its logits. Each new check fails on a tree without its fix: 8d5405d 68c4a40 this shared width 0 refused accepted PASS PASS ngram_size 1, by cfg_sane not sane PASS PASS conv kernel 0, by cfg_sane accepted PASS PASS shared 2048 opens and runs overflow refused PASS Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
01e593c moved the Qwen entries of docs/LEARNED.md from §74-87 to §80-93, so that main's own §74-79 stay where they were, and renumbered the references between those entries. The fifteen outside the file kept the old numbers, and every one of them then named a DeepSeek-V4.1 entry: the i8mm comment in waste_model_load pointed at "a speculative batch of five", qwen_moe_layer's at "the oracle was wrong". Each is the same entry six further on. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
test_qsa_attn passed on a release build and failed under make asan on arm64 — "differing output 0: -0.0781467855 vs -0.0781467929" — against a kernel that had not changed. The reference it compares with is the old score loop, `s += q * k`, and the rounding of that expression is the compiler's rather than the language's: clang at -O2 vectorizes the products and rounds each on its own, at -O1 it emits one fused multiply-add per element. The four-wide loop was written with product and sum as separate statements precisely so it would not fuse, which matched the -O2 reference and not the -O1 one. The engine had the same split in it. waste_qwen_qsa_attn_heads scores a selection four tokens at a time and the remainder in that same scalar loop, so at -O1 the two halves of one selection were rounded differently, and an -O1 build's QSA logits differed from its -O2 build's. Both scalar score loops — the engine's tail and the test's reference — now write product and sum as separate statements, which -ffp-contract=on never contracts at any level and which is what -O2 compiled the old form to. The release build's numbers do not move: the Qwen fixture scores every token through this loop and its logits are byte-identical, sha256 edea1752... before and after. test_qsa_attn passes at -O1 and -O2, 0 of 40 cases. The value loop is left as it was. It is contracted at every level, which is what the NEON path's vfmaq_f32 matches. CI's asan job runs on x86_64 without -mfma, where nothing can fuse, so it would not have seen this; a Mac running make asan, as CLAUDE.md lists it, does. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Thanks both — I've pushed three commits on top of 1. The shared-expert bound did not close the overflow —
|
8d5405d |
68c4a40 |
now | |
|---|---|---|---|
| shared width 0 refused | accepted | PASS | PASS |
ngram_size 1 refused by cfg_sane |
refused by a later check | PASS | PASS |
conv kernel 0 refused by cfg_sane |
accepted | PASS | PASS |
| shared 2048 opens and runs (ASan) | overflow | refused | PASS |
On the SIGFPE: it is reachable. With UBSan's integer-divide-by-zero on 8d5405d and ngram_size: 1, the synthetic fixture stops at src/model.c:1542:36: runtime error: division by zero. The ngram_head.0.weight refusal in your matrix comes after that division — arm64 returns 0, which becomes a shape refusal; x86's idiv traps.
3. Fifteen LEARNED references pointed at DeepSeek entries — acc98c2
The renumbering moved the Qwen sections by six and updated the references between them, but the fifteen in CLAUDE.md, docs/QWEN.md, src/model.c and src/qwen_qsa.c kept the old numbers. The i8mm comment in waste_model_load pointed at "a speculative batch of five", for example. Each one is now shifted by six to where its section is.
4. make asan failed on arm64 — 84369e3
test_qsa_attn passed at -O2 and failed at -O1 (-0.0781467855 vs -0.0781467929). The reference is the old s += q * k, and the compiler decides how that rounds: separately at -O2, as one fused multiply-add at -O1. The engine's scalar tail had the same split, so at -O1 a selection's tail tokens were scored differently from the ones done four at a time. Both scalar score loops now keep the product and the sum as separate statements, which is what -O2 already compiled them to — release logits are byte-identical. CI would not have caught this: its asan job runs on x86 without -mfma, where nothing can fuse.
Results: tests/run.sh 103 passed / 0 failed / 11 skipped; make asan 96 / 0 / 17; the default fixture and its logits byte-identical to before. #68 (the process-wide trunk_kern) is still open and tracked on its own; it does not block this.
The fixture check built its paths inside the Python program text —
open('$QWENC/manifest.json') — and MSYS2 rewrites a POSIX path in a native
program's argv but not one inside a -c string, so on Windows it opened
/tmp/... and reported the fixture as not loading. The path is now an
argument, which is how the other checks here already pass theirs.
test_convert_qwen.py printed "→", which the Windows console's code page
cannot encode, so the test died in print() before reporting anything.
It prints "->" now.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
linux-arm64 failed test_qsa_attn in CI, 40 cases of 40, and it failed on this branch's head before 84369e3 as well. gcc's vectorizer turns the reference's one-accumulator dot product into in-order vector products, each rounded on its own, and leaves the kernel's four-accumulator loop scalar and fused. No contraction flag closes that: -ffp-contract off, on and fast all fail it, and only disabling the vectorizer passes. 84369e3's separate statements fixed clang at -O1 and could not fix this, because gcc contracts across statements and vectorizes the two loops differently whatever they are written as. So the rounding is written into the code instead of left to the compiler. waste_qwen_qsa_mac in qwen_qsa.h is fmaf where the target can fuse (__ARM_FEATURE_FMA, __FMA__) — one rounding by definition, one instruction — and two statements where it cannot, where no compiler can fuse either. The four-wide scores, the one-token scores, the scalar value tail and the test's reference all go through it, and the NEON value path is taken only where it is fused too. test_qsa_attn now passes on clang at -O1 and -O2 and on gcc on arm64 at -O1, -O2 and -O3, with contraction off and under ASan — 0 cases of 40 everywhere. This changes arm64 numbers. Scores of a selection of 32 tokens or more, the four-wide path, now round once per element where clang rounded twice; x86 without -mfma is unchanged, having nothing to fuse with. The Qwen fixture selects fewer than 32 tokens and its logits are byte-identical, so it cannot show the change, and the container-native oracle agrees at 1.9e-06 on the default fixture and 2.4e-06 on the wide one. The real checkpoint is not on this machine to measure. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
CI's first run on this branch was red, on three checks that had never run on it before. None of them came from the changes above, and all three reproduce on
|
PR #63 takes Qwen3.8-Flash-Next from 7.01 tok/s to about 12.3 over eleven changes. Ten are parallelism and one is arithmetic, and only the arithmetic one ports: DeepSeek-V4.1 gains 14.8% from the i8mm trunk kernel, which needs no code because WASTE_TRUNK_KERNEL already exists and the branch only changes its default. The other two measured here — the batched trunk matvec and the staged expert schedule — are refuted with the numbers that killed them, and for one reason: both target work units too small to be worth a dispatch. Qwen's GDN projects through 48-row tensors and its experts are 640 rows; DeepSeek's smallest projection is 512 rows and its experts are 2,304, so the fork-join it would amortize is already amortized. That is the useful half, and it predicts the same nothing for §84, §85, §90 and §92. §94 also records the reading mistake that chose the first of the two: the trunk matvec table's rate column says how badly a call runs and its bytes column says whether it matters, and only the second one picks the work. Both entries name the branch their code sits on — perf/mvb-ds41 and perf/xpar-staged-ds41 — and neither is merged. Renumbered before it was ever pushed: it was written as §80–81 on a local main, and PR #63 took §80-93 for its Qwen entries, as its review asked, without being able to see it. Only the numbers and the note that predicted them changed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…s forty Following §94's finding that the i8mm trunk kernel is the one thing of the Qwen branch's speed work that ports to DeepSeek-V4.1, this is what checking it properly turned up. i8mm is +15% and moves no routing decision that the reference could resolve — one tie in 2,080 — with a top-10 identical to tools/ds41_ref.py. WASTE_VQ8=1, the register-resident int8 VQ3R table EXP1 built, is a further +21% and changes 574 of 880 routing decisions, the first of them a real disagreement at ten times the tie threshold. It is not a speed switch, it is a different model. The entry's own number is the gate. On the real container i8mm measures 0.056% relative L2 against a suite threshold of 0.01%; on the fixture tests/run.sh actually runs it measures 0.001%. Six layers against forty and 128 hidden against 5,120: the fixture is 53x quieter than the model on the same change, and cannot see a per-matvec error that compounds with depth. That comparison establishes something about the gate and not about the kernel. 0.01% is a bug detector, not a numerics budget — the oracle reads the same quantized container, and the container is 3-bit experts and a 4-bit trunk. On a MoE what bears on "is this the same model" is the routing and the top-k, and i8mm passes both where VQ8 does not. Also recorded: gate 9's speculative-decoding break-even is counted in bytes, and at this operating point expert I/O is 8.8% of a step against 65.5% for the expert arithmetic, so the deciding number wants recomputing on this profile rather than inheriting. And what measured nothing, so the next person need not: WASTE_METAL_MOE on DeepSeek, the thread count from 6 to 18, and externally T-MAC and Vec-LUT, whose in-register table lookup this repo already implements. Renumbered before it was ever pushed: it was written as §82 on a local main, and PR #63 took §80-93 for its Qwen entries, as its review asked, without being able to see it. Only the numbers and the note that predicted them changed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
main gained sqliteai#63 after this branch was cut, and sqliteai#63 restructured the default trunk path: the `if ds41:` block this branch edits is now `if ds41 and not engram:`, one level out and after the Qwen PLE pass. It still called build_engram() without build_engram_meta(), so on main the bug this branch fixes was present in a place the branch did not touch — the branch's own guard fails on main at tools/convert.py:2445. Resolved by taking main's structure and adding the call there. The other half, the duplicate call on the --reclaim path, merged cleanly. Each of the two build_engram() calls is now followed by exactly one build_engram_meta(), and the guard passes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A fifth architecture, text only, contributed by @Skibisky in #63: Gated DeltaNet, Qwen Sparse Attention over the original K/V, four HyperConnection residual streams, a softmax router with a gated shared expert, and a per-layer n-gram embedding whose 16 tables stay on the trunk and are read a row per head per token — which is how 176.94 B parameters open with a 3.11 GB floor. The throughput in docs/QWEN.md is the contributor's; the checkpoint is not on this machine. Measured on this commit: routes exact against the container-native oracle at every layer, logits within 1.9e-06, and within 2.4e-06 with a shared expert 64x the hidden width. For 0.8.0 users: a DeepSeek-V4.1 conversion without --reclaim wrote no Engram index, so the container could not be opened and nothing said why (#71, fixed by @helenkwok in #72). Also the server's listen backlog, which reset connections under load. On arm64, QSA's four-wide scores now round once per element where the branch measured with two roundings; kernel_kl on the real checkpoint is owed. i8mm as DeepSeek-V4.1's default is measured, +15%, and waits for #68. No ABI move: src/waste.h changes only its version macros. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
I measured the The change is not confined to selections of 32 or more
Isolating
(head geometry For contrast, The synthetic fixture does select 32 or more, and does show the changeIt selects fewer than 32 only at the default 16-token id list. I instrumented
So the four-wide path runs, just not on 16 tokens. Extending the id list is enough. Then, checksumming the attention output itself at both revisions (
The mandatory control for a byte-identical result: the two So I would put it slightly differently than "its logits stay byte-identical and cannot show the change": with more than 16 tokens the fixture does exercise the change, and byte-identical logits under a measurably perturbed attention output is a real (if small) piece of evidence that the perturbation is absorbed before the logits on this model — 4-bit grouped activation quantization plus the bf16 K/V round-trip are both downstream of it. What I could not checkNo gcc on this machine ( |
3 participants
Qwen3.8-Flash-Next is a 125B model with 6B active experts plus 51B n-gram embedding and 4B MTP.
Unable to hold it all in memory at once, I decided to attempt to fork WARP to make use of the streaming, cache, and similar technologies for both the MoE and the n-gram embeddings.
An attempt was made to incorporate the MTP for token prediction but on the hardware available the accuracy and time taken never was able to deliver a gain due to contention of shared resources (this work will be provided on a future branch on my own fork as a reference).
This adds text-only support for Qwen3.8-Flash-Next’s qwen4_exp architecture. The converter handles its nested Hugging Face configuration, packed expert tensors, and 128 PLE embedding shards while reusing WARP’s existing prefix, rename, quantization, streaming, and resume machinery. Tokenizer support also gains Qwen’s single-digit pre-tokenization behavior without introducing an architecture-specific tokenizer path.
The CPU runtime implements Qwen’s Gated DeltaNet and sparse-attention layers, HyperConnection residual streams, PLE retrieval, and routed MoE with a gated shared expert. These components integrate with WARP’s existing expert cache, dynamic expert-parallel execution, memory planning, state persistence, and container format. Unsupported capabilities—including vision, video, MTP, and native Qwen serving—are explicitly refused or left outside this PR’s scope.
Validation includes standalone C kernels checked against independent PyTorch references, a synthetic qwen4_exp container, converter and round-trip coverage, tokenizer tests, and a container-native forward-pass oracle. Routes match exactly, the worst hidden-state difference is 2.4e-7, and final logits differ by 1.9e-6 with the same argmax. On the converted 124 GB checkpoint, the runtime produces coherent output and sustains approximately 7 tok/s using eight threads and an 8 GiB expert cache
The heavy lifting was performed by Codex Sol 5.6, occasionally using Luna sub-agents, and the occasional Claude Opus review.
Additional on-going investigations were trying to 'outsource' non-critical calculations with low memory contentions to Metal/GPU, which isn't looking so great.
I am posting up this work and submitting this PR because Qwen3.8-Max is out and might share similar architecture.