model: make the trunk kernel per-model, not per-process (fixes #68)

[mfethe1]

A Qwen load sets the trunk kernel for the whole process, so a Kimi context already open switches to i8mm (#68).

Reproduced

Two contexts in one process on an M4 (hw.optional.arm.FEAT_I8MM: 1), upstream 09fcff3: load a non-Qwen container, prefill 16 tokens, keep it open, load a Qwen container, then waste_model_reset the first model and replay the identical tokens.

build: WASTE 0.8.1 (container v0, backend NEON, crc32 armv8, arm64)
WASTE_TRUNK_KERNEL env: (unset)
A loaded: kimi.waste, 4 layers, 8 experts, vocab 256
A prefill ok, vocab 256
B loaded: qwen.waste, arch_qwen=1
logits differing: 256 / 256
max |delta|: 0.423375845
first differing index: 0
argmax before: 209 (13.046713)   after: 209 (13.053581)
VERDICT: DIVERGED - opening B changed A's logits

Controls, so the diff means something

A bare diff would not distinguish this from prefill nondeterminism. Three controls, same binary, same fixtures:

control	result
B is a second non-Qwen container instead of Qwen	0 / 256 differ, IDENTICAL
B is Qwen, but WASTE_TRUNK_KERNEL pinned	0 / 256 differ, IDENTICAL
pinned WASTE_TRUNK_KERNEL=2 (TK_I8MM) from the start	0 / 256, and A's before logit is 13.053581

The third one is the identification: 13.053581 is exactly the after value from the uncontrolled run. The post-Qwen non-Qwen model is computing i8mm arithmetic.

Cause

src/model.c:2698

if (m->cfg.arch_qwen && !trunk_kern_env) waste_model_set_sdot4(TK_I8MM, sdot4_sg);

trunk_kern and sdot4_sg are file-static and read by matvec_t_inner(), so this is a process-wide write. The comment just above it already notes the behaviour; #68 is the case where it becomes wrong.

Fix

Carry trunk_kern / sdot4_sg on waste_model. A fresh load still inherits the process default, so single-model processes and the env override behave exactly as before, and Qwen containers still select i8mm — verified, not assumed:

qwen.waste  arch_qwen=1 trunk_kern=2 sdot4_sg=32     (TK_I8MM, perf preserved)
kimi.waste  arch_qwen=0 trunk_kern=0 sdot4_sg=32
after loading qwen.waste:  A.trunk_kern=0   B.trunk_kern=2

Also adds waste_model_set_kernel(m, mode, sg) so kernel_kl.c / sweep.c can retune one model without touching others. waste_model_set_sdot4() is unchanged and still sets the process default.

Verification

• Bug case after the fix: 0 / 256, max |delta|: 0. Reverse load order also identical.
• tests/run.sh: 97 passed, 0 failed, 16 skipped, including PASS a Qwen load leaves an open non-Qwen model's kernel and logits alone.
• Those 16 skips are not from this patch. I built a pristine worktree at 09fcff3 and ran it: also 96 passed, 0 failed, 16 skipped (96 being 97 minus the new test). The delta against your 103/0/11 is K3_DIR / DS41_DIR releases I don't have locally.
• make asan on both trees: 94 passed, 0 failed, 18 skipped each, and a diff of every PASS|FAIL|SKIP line between patched and pristine is empty.
• The new test was checked for the failure it claims to detect: compiled against the unpatched library it exits 3 with FAIL 256/256 logits changed after loading qwen.waste.

Test fixtures come from tools/make_test_container.py (plain and --qwen), so the test needs nothing outside the repo.

[marcobambini]

Thanks for this. The diagnosis is right and the writeup is excellent: the three controls, and especially pinning TK_I8MM from the start to reproduce the post-Qwen value exactly, are what turn a logit diff into an identification. Moving trunk_kern / sdot4_sg onto waste_model is the right shape for the fix, and I reproduced the isolation here (M5 Pro, FEAT_I8MM=1): 256/256 logits bit-identical.

It can't go in as is, though. Two things are broken, one of them silently.

1. kernel_kl and sweep now compare a kernel against itself

Both load their models once and flip the kernel between steps with waste_model_set_sdot4() (tests/kernel_kl.c:244,250, tests/sweep.c:192). After this change that call only moves the default for future loads, so the open models never switch. The PR adds waste_model_set_kernel() for exactly this purpose but doesn't move either tool onto it.

Same synthetic container, kernel_kl <tiny> ids 16 0 2,3 32:

	k2 KL max	k2 relL2
main @ 09fcff3	1.18e-08	1.09e-05
this PR	0.00e+00	0.00e+00

That's the worst way for this to fail: it reads as "the kernels agree perfectly". kernel_kl is how LEARNED §83 was measured, and it's the documented way to compare two trunk kernels.

Fix: after loading, waste_model_set_kernel(ma, ka, sg) and waste_model_set_kernel(mb[i], kb[i], sg) (once each; the per-step calls go away), and waste_model_set_kernel(&m, arm[a], sdot4_sg_env) in sweep's is_sdot4 arm. A check in run.sh that kernel_kl ... 0 2 is non-zero on a CPU that has a second kernel would keep this from coming back.

2. The new test assumes an i8mm CPU

CI fails the new check on linux-x86_64, windows-x86_64, asan (x86) and macos-arm64, and passes only on linux-arm64. The test requires qwen.trunk_kern == 2, but kern_clamp() degrades TK_I8MM:

• x86: WASTE_CPU_DOTPROD is an ARM flag, so the result is TK_F32 (0);
• the macOS runner has no i8mm, so the result is TK_SMLAL (3).

So "Qwen containers still select i8mm" holds on an M4, not in general. The expected kernel should come from waste_cpu_features(), the same way kern_clamp computes it. If that equals the plain model's kernel (always on x86), there's nothing to isolate and the check should report SKIP, per the repo rule that a missing prerequisite is never a pass. Exiting with a distinct code such as 2 for "nothing to test" lets run.sh call sk. The same applies when WASTE_TRUNK_KERNEL is set, since Qwen then doesn't pick i8mm by design. Please use TK_I8MM rather than the literal 2.

3. Smaller things

• SPDX header is missing on tests/test_kernel_isolation.c, which is why build guards fails.
• tests/run.sh: the new block sits between the "Chunked prefill against sequential decode…" comment and the code it describes. Please move it above that comment. On failure, print $TMP/kiso.log (e.g. tail -5) like the neighbouring checks do: CI currently gives no clue which assertion fired.
• $MODEL can be a real container. It's loaded with cache_bytes = 0 (automatic budget, so a fully resident preload on Kimi-Linear) just to run 10 tokens; honouring WASTE_CACHE_MB as test_forward does, or passing a small explicit cache, would keep the check cheap. If $MODEL is itself a Qwen container the test exits 1 and shows as FAIL; that should be a SKIP.
• Stale comment: the one above the load site (src/model.c, "The kernel is one setting for the whole process, so a process that loads Qwen and then another architecture keeps i8mm for both…") now describes the behaviour this PR removes.

With 1 and 2 addressed this is a clear improvement. Happy to re-review.

[mfethe1]

Guard added in 77d624e. Thanks for catching the self-comparison — you were right that it was the worst possible shape for that bug, and I want to be precise about why the fix is the shape it is.

What the guard asserts. run.sh now reads the KL mean back from a k0 vs k2 run and fails on a non-positive value. Only the sign is asserted. The magnitude is fixture-dependent and I deliberately did not pin it — a threshold tuned to the synthetic container would be a second thing that passes for the wrong reason.

Verified against the bug, not just against green. A guard that has never failed is an assumption. I rebuilt kernel_kl with the pre-fix process-global setter (waste_model_set_sdot4 in place of waste_model_set_kernel at both call sites) and ran the guard against both binaries:

• pre-fix binary → KL mean 0.00e+00 → guard fails, exit 1
• fixed binary → KL mean 3.07e-09 → guard passes, exit 0

So the check demonstrably discriminates; it is not green-by-construction.

Why it is gated on the isolation test rather than a CPU probe. kiso_rc == 0 is already exactly the condition "this CPU selected TK_I8MM", so there is a second kernel to compare against. Where i8mm is absent, kern_clamp() (src/model.c:6313) folds k2 onto another arm and a zero would be honest rather than broken — asserting there would produce a false failure. That is borne out across CI on this run:

• linux-arm64 — guard ran: PASS kernel_kl compares two distinct kernels (KL mean 3.08e-09)
• linux-x86_64 — skipped, features=0x700, no i8mm
• macos-arm64 — skipped, features=0x3, no i8mm

Your M5 Pro reproduction and the CI arm64 runner independently land on the same order of magnitude (3.07e-09 local, 3.08e-09 on the runner), which is a better cross-check on the measurement than anything I could assert alone.

One pair I considered and rejected. k0 vs k3 (TK_F32 vs TK_SMLAL) survives kern_clamp on every CPU and would have needed no skip at all — attractive, but mvq4_rows_smlal is inside #if defined(__ARM_NEON) || defined(__aarch64__) and compiles to a no-op on x86, so that pair would silently compare a kernel against itself on exactly the platforms the skip was meant to serve. Same bug, new home. The i8mm gate is the narrower but truthful option.

CI is 11/11 SUCCESS on 77d624e. Full local suite: 98 passed, 0 failed.

Not claimed: this is still the 0.4 MB synthetic container, so it establishes that the two arms are distinct, not that either is numerically right on a real checkpoint. Ready for another look when you have time.

[mfethe1]

Items 2 and 3 are addressed as well — all of them are in the branch now, and CI is green on all 11 checks including the four that were failing.

2. The test no longer assumes an i8mm CPU. The expected kernel now comes from waste_cpu_features(), the same path kern_clamp() uses, and the literal 2 is gone in favour of TK_I8MM. There are three distinct SKIP routes, all exiting 2 so run.sh calls sk:

• WASTE_TRUNK_KERNEL set — Qwen's load-time selection is overridden by design
• no i8mm on this CPU — a Qwen load would not select TK_I8MM here, so there is nothing to isolate
• $MODEL is itself a Qwen container — the fixture can't serve as the non-Qwen arm

That last one was a FAIL before, which was wrong: a missing prerequisite is never a pass, and it isn't a failure either.

Observed across this run: linux-arm64 runs the check, linux-x86_64 skips at features=0x700, macos-arm64 skips at features=0x3. The claim is now "Qwen containers select i8mm where the CPU has it", which is the claim the code actually supports.

3. The smaller items.

• SPDX header added to tests/test_kernel_isolation.c — build guards passes.
• run.sh block moved above the chunked-prefill comment, and failures now tail -5 "$TMP/kiso.log" like the neighbouring checks. The SKIP path prints tail -1 so the reason is visible rather than silent.
• $MODEL now honours WASTE_CACHE_MB (run.sh passes 512), so the check no longer triggers a fully resident preload on a real container just to run ten tokens.
• Stale comment in src/model.c removed — it described the process-global behaviour this PR deletes.

On the run.sh guard you suggested. I took it, and verified it against the bug rather than against green: rebuilding kernel_kl with the pre-fix process-global setter makes the guard fail (KL mean 0.00e+00, exit 1), and the fixed binary passes (3.07e-09, exit 0). A guard that has never failed is an assumption, so I wanted it to have failed at least once on purpose.

Full CI: 11/11 green, including linux-x86_64, windows-x86_64, asan + fuzz and macos-arm64, which were the four failing before. Ready for re-review whenever you have time — and thanks for the level of detail in the first pass, item 1 especially was a real bug and the table made it obvious.