WASTE_TRUNK_KERNEL=2 is worth +15% on DeepSeek-V4.1 and is off unless the environment asks for it. Nothing needs building: the kernel is src/simd_i8mm.c, EXP1 lever 3, and the switch has been in src/model.c since that branch. What is missing is a default.
Measured on the M5 Pro this repo develops on, ds41.waste, WASTE_CACHE_MB=17000, 40 decode steps after a 12-token prompt, mean of the last 30, arms rotated across three rounds because this machine drifts:
| trunk kernel |
tok/s |
| 0, f32 — today's default |
3.87–3.97 |
| 2, i8mm |
4.52–4.56 |
LEARNED §94 and §96 have the protocol and the rest of the search that found it; §96 has why the evidence below is the evidence that counts.
Why this is safe to default, on the standard this repo uses for a MoE
Routing. WASTE_DUMP_ROUTE over 2,080 decisions, i8mm against the f32 reference, through tests/route_diff.py:
202 of 2080 decisions differ, first at token 24 layer 38: [125] -> [118], at relative margin 1.663e-06 — a tie the reference itself could not resolve; the other 201 are downstream of it
One flipped tie, below the resolution of the reference that produced it, and everything after is that tie's consequence. §71's case exactly. Over a shorter 880-decision run the routes are identical.
Top-k. Against tools/ds41_ref.py on the real container, 12 tokens: the top-10 is identical to the oracle's, and so is f32's.
The Qwen branch decided the same question the right way. docs/QWEN.md on #63: next-token perplexity 3.698 for i8mm against f32's 3.712 over 5,918 tokens of real text, 96.5% top-1 agreement, 97.65% of routed experts the same. i8mm came out ahead on that sample, which says the error is noise-shaped rather than biased.
What the L2 says, and why it is not the gate
On the real container i8mm is 0.056% relative L2 against f32's 0.0014%, and tests/run.sh gates the DS41 oracle at 0.01%. That looks disqualifying and is not:
- the check is a bug detector — it fires when CSA2, mHC, Engram or the router is wrong, not when a kernel is deliberately approximate;
- the oracle reads the same quantized container, so f32's 0.0014% is summation order alone, and the container around it is 3-bit experts and a 4-bit trunk — approximations orders of magnitude coarser than what is being weighed;
- it cannot see this anyway. On the fixture
run.sh actually runs, i8mm measures 0.001066% and passes comfortably: 6 layers and hidden 128 against the model's 40 and 5,120. The fixture is 53x quieter than the model on the same change. That is its own problem and §96 records it, but it means the threshold was never carrying the weight this decision needs.
What it should not look like
Not #63's mechanism. That branch defaults i8mm for Qwen by having waste_model_load write the file-static trunk_kern, which is #68: a process that opens a Kimi container and then a Qwen one moves the first one's arithmetic too. A per-container default needs the per-model field #68 asks for, and then Qwen's default and this one are the same one-line change in the same place.
So: #68 first, then a trunk_kern of TK_I8MM for ds41 containers, with WASTE_TRUNK_KERNEL still pinning it either way.
Owed afterwards, not before
tests/kernel_kl.c arrives with #63 and is the instrument that measured Qwen's case — perplexity, top-1 and route agreement over a long real prompt, at every position. Run it on DeepSeek once #63 lands and put the number in docs/DS41.md beside the throughput table. If it disagrees with the routing and top-k evidence above, that is worth knowing and the default comes back out; nothing here expects it to.
Two things this does not propose, for the record. WASTE_VQ8=1 is a further +21% and changes 574 of 880 routing decisions with the first a real disagreement — §96 has it, and it stays a switch. And docs/DS41.md's throughput table is measured at the f32 default; it will need re-measuring with the tokenizer-driven CLI harness it used, not with test_forward.
WASTE_TRUNK_KERNEL=2is worth +15% on DeepSeek-V4.1 and is off unless the environment asks for it. Nothing needs building: the kernel issrc/simd_i8mm.c, EXP1 lever 3, and the switch has been insrc/model.csince that branch. What is missing is a default.Measured on the M5 Pro this repo develops on,
ds41.waste,WASTE_CACHE_MB=17000, 40 decode steps after a 12-token prompt, mean of the last 30, arms rotated across three rounds because this machine drifts:LEARNED §94 and §96 have the protocol and the rest of the search that found it; §96 has why the evidence below is the evidence that counts.
Why this is safe to default, on the standard this repo uses for a MoE
Routing.
WASTE_DUMP_ROUTEover 2,080 decisions, i8mm against the f32 reference, throughtests/route_diff.py:One flipped tie, below the resolution of the reference that produced it, and everything after is that tie's consequence. §71's case exactly. Over a shorter 880-decision run the routes are identical.
Top-k. Against
tools/ds41_ref.pyon the real container, 12 tokens: the top-10 is identical to the oracle's, and so is f32's.The Qwen branch decided the same question the right way.
docs/QWEN.mdon #63: next-token perplexity 3.698 for i8mm against f32's 3.712 over 5,918 tokens of real text, 96.5% top-1 agreement, 97.65% of routed experts the same. i8mm came out ahead on that sample, which says the error is noise-shaped rather than biased.What the L2 says, and why it is not the gate
On the real container i8mm is 0.056% relative L2 against f32's 0.0014%, and
tests/run.shgates the DS41 oracle at 0.01%. That looks disqualifying and is not:run.shactually runs, i8mm measures 0.001066% and passes comfortably: 6 layers and hidden 128 against the model's 40 and 5,120. The fixture is 53x quieter than the model on the same change. That is its own problem and §96 records it, but it means the threshold was never carrying the weight this decision needs.What it should not look like
Not #63's mechanism. That branch defaults i8mm for Qwen by having
waste_model_loadwrite the file-statictrunk_kern, which is #68: a process that opens a Kimi container and then a Qwen one moves the first one's arithmetic too. A per-container default needs the per-model field #68 asks for, and then Qwen's default and this one are the same one-line change in the same place.So: #68 first, then a
trunk_kernofTK_I8MMfords41containers, withWASTE_TRUNK_KERNELstill pinning it either way.Owed afterwards, not before
tests/kernel_kl.carrives with #63 and is the instrument that measured Qwen's case — perplexity, top-1 and route agreement over a long real prompt, at every position. Run it on DeepSeek once #63 lands and put the number indocs/DS41.mdbeside the throughput table. If it disagrees with the routing and top-k evidence above, that is worth knowing and the default comes back out; nothing here expects it to.Two things this does not propose, for the record.
WASTE_VQ8=1is a further +21% and changes 574 of 880 routing decisions with the first a real disagreement — §96 has it, and it stays a switch. Anddocs/DS41.md's throughput table is measured at the f32 default; it will need re-measuring with the tokenizer-driven CLI harness it used, not withtest_forward.