models(architecture): add Gemma 4 family — Architecture enum + HF detection + forward wiring (Phase 8 Step 57) - #320
Draft
jamesburton wants to merge 68 commits into
Draft
Conversation
Introduces the DotLLM.Vulkan project: cross-vendor GPU compute backend built on a hand-rolled flat C API surface over Silk.NET-generated Vulkan bindings. This PR is the project foundation only — no kernels yet. Subsequent PRs add baseline compute kernels (matmul F32/Q8_0, RMSNorm, RoPE, attention, SwiGLU), subgroup + coopmat variants, then the VulkanForwardState / VulkanWeights / VulkanTransformerModel host, then F16/BF16 + model-family extensions. Includes: - Project scaffold + IBackend registration. - VulkanApi + VulkanStructs (flat C surface over Silk.NET). - Device / queue / command-buffer / context management. - Buffer + memory allocation primitives. - Shader module + pipeline + descriptor-set helpers. Closes #155 Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…or (#154) Adds the foundation for parsing HuggingFace safetensors files: header JSON + tensor descriptors + memory-mapped tensor data access. No model loading yet; this is purely the binary format reader. - SafetensorsFile: 8-byte length prefix + JSON header parser + mmap- backed tensor data access. - SafetensorsDType: enum + size helpers covering F32, F16, BF16, I8..I64, U8..U64, BOOL. - SafetensorsTensorDescriptor: tensor metadata record. Includes SafetensorsFileTests + a SafetensorsFixtureBuilder helper for building synthetic safetensors files in tests. The Mamba-3-specific fixture helper (WriteTinyMamba3Fixture / Mamba3TensorMapping references) is excluded from this PR; it belongs to the Mamba-3 chain. Future PRs build on this: HfConfigExtractor + TransformerWeightsSafetensors, multi-shard index support, per-architecture loaders. Closes #154 Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds the 6 baseline compute kernels that downstream Vulkan PRs require: - matmul_f32: naive tiled GEMV/GEMM (F32 weights, F32 activations). - matmul_q8_0: direct quantized GEMV against Q8_0-packed weights. - rmsnorm_f32: per-token RMS normalisation. - rope_f32: rotary position embedding (Norm + NeoX variants). - attention_f32: scaled-dot-product with causal mask + GQA broadcast. - swiglu_f32: SwiGLU activation. Each kernel has parity tests vs the CPU reference. Tests pass on any Vulkan-compatible device (radv, amdvlk, NVIDIA, MoltenVK). Stacked on PR #160 (Vulkan scaffold) — do not merge until #160 lands. Closes #167 Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…s for dense LLMs (#166) Builds on the binary parser foundation (#154 / PR #159). Adds: - HfConfigExtractor: parses HF config.json into ModelConfig. - TransformerWeightsSafetensors: loads weights from a SafetensorsFile into TransformerWeights (GQA-aware, fused-QKV-aware). - ModelLoader.LoadFromSafetensors entry point. - TinyLlama-1.1B integration test (gated on fixture availability). - Xunit.SkippableFact added to integration test project for skip support. Foundation for the upcoming Mixtral / Qwen-MoE / DeepSeek-V2/V3 loader PRs and the HF tokenizer adapter. Stacked on PR #159 — do not merge until #159 has merged. Closes #166 Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
… forward path (#164) Adds the LoRA (Low-Rank Adaptation) foundation: - Core types: ILoraAdapter, ILoraAdapterRegistry, LoraAdapter, LoraConfig. - CPU forward path: LoraDelta kernel + TransformerModel per-projection dispatch when _currentAdapter is set. - HF PEFT loader: parses adapter_config.json + adapter_model.safetensors. - IModel adapter-attachment API. - Unit tests for adapter, registry, PEFT loader. Foundation of Phase 4 — follow-up PRs add the multi-adapter switch + TinyLlama integration test, Vulkan upload + dispatch, server API integration, and the F16/BF16 + MLA/MoE acceptance suite. Closes #164 Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
… SwiGLU (#175) Dense-routing top-k Mixture-of-Experts support for Mixtral, Qwen*-MoE without shared experts, and Phi-3.5-MoE. Attention path is unchanged (GQA + RoPE); per-layer FFN branches into a router + top-k expert loop when `ModelConfig.Moe` is non-null. - `Architecture.Mixtral` enum variant, new `MoeConfig` record (`NumExperts`, `NumExpertsPerTok`, `MoeIntermediateSize`). - `HfConfigExtractor` detects MoE from `num_local_experts` / `num_experts` + `num_experts_per_tok`; Phi-3.5-MoE `moe_intermediate_size` override surfaces as-is. - `MoeSwiGluMlp` kernel: full softmax over E experts → top-k partial max-scan (stable tiebreak: lower index wins, matching `torch.topk`) → renormalise by sum (Mixtral convention, NOT a second softmax) → per-expert SwiGLU via `FusedOps.SwiGLU` → weighted sum. Scalar per-expert GEMV for PoC; fused GroupedGEMM is a follow-up. - `TransformerWeights` gains a nullable `MoeLayerWeights` per layer (router gate + per-expert `w1`/`w2`/`w3` F32 pointers). Safetensors loader resolves the Mixtral `model.layers.{i}.block_sparse_moe.(gate|experts.{j}.w[1-3])` naming with F16/BF16 → F32 upcast into 64-byte-aligned scratch. - `TransformerModel.Forward` branches to the MoE path per-layer (output into scratch then residual add), reusing the dense attention pipeline unchanged. - `ModelLoader.LoadFromSafetensors` dispatches `Architecture.Mixtral` through the existing `TransformerModel.LoadFromSafetensors`. Tests: - 4 kernel unit tests (top-k tie stability, scalar-reference match, one-hot router equiv to single expert, uniform router equiv to expert-average). - 4 HfConfigExtractor tests (Mixtral detection, `moe_intermediate_size` override, dense → null, missing top-k throws). - 1 synthetic safetensors loader test (2 layers × 4 experts top-2, hidden=16 head_dim=4, full forward → finite vocab logits). - 1 integration test against real `yujiepan/mixtral-tiny-random` proving Mixtral+MoE detection from real HF bytes (forward-pass skips on upstream head_dim=1 RoPE incompatibility, documented). Out of scope (future): shared experts (DeepSeek-V3, old Qwen1.5-MoE), Qwen-MoE `mlp.experts.{j}.{gate_proj,up_proj,down_proj}` naming adapter, fused GroupedGEMM kernels, expert parallelism, real Mixtral-8x7B validation. Roadmap step 58 ticked. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Prefill-path companion to the Q8_0 GEMV kernel added in PR-vulkan-2. Direct quantized GEMM against Q8_0-packed weights with a tiled accumulation scheme suitable for multi-token prefill batches. Parity tests verify the kernel matches the CPU `GemmQ8_0` reference to abs 1e-4 / rel 1e-3 across a representative grid of (M, K, N) shapes. Refs #173
Adds the `bias_add_f32` compute kernel: a tiny in-place per-feature add (output[t,i] += bias[i]) — one thread per output element, dispatched as ceil(seqLen * outputDim / 256) workgroups. Replaces (downstream) the host-mapped fallback used by Phi-3 / Qwen3 / DeepSeek-V2 layers that carry small per-feature bias vectors after Q/K/V/O/Gate/Up/Down projections. The previous host-loop fallback forced a COMPUTE→HOST barrier + SubmitAndWait + HOST→COMPUTE per bias-bearing projection per layer (up to 120 extra submits per forward on Phi-3 / DeepSeek-V2). With this kernel the whole forward stays in one submit regardless of bias presence. The wiring into `VulkanTransformerModel` from the original commit is deferred to PR-vulkan-5 (which introduces that file). This PR ships the kernel + shader + parity tests only. Bit-identical to the CPU reference (pure addition, no FP reduction). 6 parity tests cover SmolLM-hidden, Llama-2-hidden, prefill-ish multi-token, and odd dims. Shared infrastructure — also consumed downstream by the MLA chain. Refs #173
The decode-path GEMV shader used `rowByteStride = pc.rowUints * 4u` which overstates the per-row byte stride when `blocksPerRow * 34` is not itself a multiple of 4 (i.e., blocksPerRow odd: K = 32, 96, 160, 224, ...). Earlier rev silently returned garbage past the first row for these K values — error magnitudes ~1e6 abs / 1e8 rel on the regression case (M=8, K=32). Fix: compute the stride from blocksPerRow directly (`blocksPerRow * 34u`), mirroring `matmul_q8_0_gemm.comp` (added in this PR) which already uses this form. The push-constant `rowUints` is still uploaded but unused in this path now — kept for ABI stability since the C# binding is shared with the original layout. Regression coverage: three new InlineData rows in `VulkanMatMulQ8_0KernelTests` — (8, 32), (4, 96), (2, 160) — all of which fail before the fix with errors > tolerance and pass after. The existing (1, 32) case did not exercise the bug because M=1 only reads the first row. This bug only bites tests / synthetic fixtures using K=32 + M>1, but it was a latent footgun for any future model with that shape. Refs #173
… config detection (#176) Lands Multi-head Latent Attention (MLA) as a standalone, scalar-first correctness-verified kernel and extends config plumbing so DeepSeek-V2/V3 checkpoints are detected end-to-end. The kernel integration into the heavily-tuned TransformerModel forward path is wired in the follow-up commit on this branch. Scope: - MlaConfig: expand readonly record struct → sealed record with the full DeepSeek-V2/V3 field set (KvLoraRank, QLoraRank, QkNopeHeadDim, QkRopeHeadDim, VHeadDim, RopeTheta + YaRN capture for future use). Breaking change on the public-API stub — safe since upstream has no consumers of the old shape. - Architecture enum: add DeepSeekV2, DeepSeekV3. - MlaAttention kernel: self-contained forward pass hidden → q_a_proj/q_a_layernorm/q_b_proj (or monolithic q_proj) → kv_a_proj_with_mqa/kv_a_layernorm/kv_b_proj → RoPE on decoupled rope sub-dim → per-head scaled dot-product with causal mask → o_proj. Scalar-first; SIMD deferred. - HfConfigExtractor: detect deepseek_v2/deepseek_v3 by model_type and by architectures[0] substring (DeepseekV{2,3}ForCausalLM). Populate MlaConfig + AttentionType.MLA + HeadDim = qk_head_dim so downstream shape code sees a single per-head dim. MoE config extensions (n_routed_experts, n_shared_experts, first_k_dense_replace) intentionally omitted here — those ship with the parallel MoE foundation PR to keep this PR scoped to MLA. Tests: - MlaAttentionTests (4): single-head single-token, multi-head prefill, monolithic Q path (q_lora_rank=0), causal no-future-leakage — each compares the kernel against a manually-coded step-by-step reference that mirrors HF modeling_deepseek_v2.py. - HfConfigExtractorTests (+3): DeepSeek-V2-Lite (q_lora_rank=0), DeepSeek-V2 full (q_lora_rank=1536), DeepSeek-V3 detection. - TinyDeepseekMlaSafetensorsLoadTests (integration): downloads yujiepan/deepseek-v2-tiny-random or -v3-tiny-random and verifies Arch + AttentionType + MlaConfig population from real HF config.json. Extracted from feature/qwen3.6 (originally commit 9a324e6) — MoE-only hunks stripped (HfConfigExtractor MoE field detection, Architecture enum Mixtral/QwenMoe values, HfConfigExtractorTests Mixtral/Qwen-MoE cases) to keep this PR a clean MLA foundation. The MoE foundation PR re-introduces those changes. Refs #176 Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Connects the MLA kernel from the prior commit into TransformerModel so DeepSeek-V2 / V3 checkpoints flow through ModelLoader.LoadFromSafetensors to a real forward pass producing finite logits. Scope: - TransformerWeights: add MlaLayerWeights record + Mla field on TransformerLayerWeights. R4 repack skips the legacy Q/K/V slots when Mla is non-null (MLA layers don't populate those). - TransformerModel: per-layer MLA dispatch — when lw.Mla is non-null route through MlaAttention.Execute, skip the GQA path, and jump straight to the FFN branch via a labelled goto. Ropes' dim/theta pick up MlaConfig overrides for the decoupled rope sub-dim. - TransformerWeightsSafetensors: LoadDeepSeekMlaLayer — parses HF MLA tensor names (q_a_proj/q_b_proj or monolithic q_proj, kv_a_proj_with_mqa, kv_b_proj, their layernorms, o_proj). All MLA tensors are coerced to F32 via ResolveLinearAsF32 (F32 zero-copy, F16 / BF16 upcast). Dense Llama-style SwiGLU on the FFN side; the DeepSeek MoE FFN branch lands with the MoE foundation PR. - ModelLoader: add DeepSeekV2 / DeepSeekV3 to the safetensors dispatch. Tests: - TransformerModelMlaForwardTests (3): synthetic 2-layer DeepSeek fixtures (q_lora_rank=0 + q_lora_rank>0 + single-token decode), asserting finite logits and non-zero std. - TinyDeepseekMlaSafetensorsLoadTests: the load-and-throw test is flipped into a real-forward-pass test; logits must be finite with non-zero variance on yujiepan/deepseek-v2-tiny-random. Extracted from feature/qwen3.6 (originally commit 3757325) — MoE-only hunks stripped (MoE FFN branch in TransformerModel, MoE layer dispatch in LoadDeepSeekMlaLayer, MoeLayerWeights wiring) to keep this PR a clean MLA foundation. The DeepSeek MoE FFN path joins this branch with the MoE foundation PR. Stacked on PR #166 (safetensors HfConfigExtractor + dense loader) — do not merge until that PR has merged. Closes #176 Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
DeepSeek-V2/V3 extend context length via YaRN, which adds both a RoPE frequency rescaling and a softmax-scale multiplier mscale² where mscale = 0.1 * mscale_all_dim * log(factor) + 1.0 (per HF modeling_deepseek.yarn_get_mscale). For DeepSeek-V2-Lite this is ~1.59 — without it, attention scores are 37% too small and long-context logits drift. - MlaConfig.ComputeYarnSoftmaxScaleMultiplier() returns mscale² when factor > 1 and mscale_all_dim != 0, else 1.0f. Uses mscale_all_dim (not mscale) per the HF softmax_scale correction — the mscale field governs RoPE frequency rescaling, not wired yet. - MlaAttention.Execute gains an optional attnScaleMultiplier parameter (default 1.0f) folded into the attention scale before softmax. - TransformerModel.Forward passes Config.MlaConfig.ComputeYarn... at the MLA kernel call site. When the checkpoint has no YaRN scaling (pre-V2 or V2-Lite without context extension) this is 1.0f and the path is bit-identical with pre-P2.2. Tests: 5 MlaConfig formula tests + 1 MlaAttention scale-multiplier test (1321/0/36 unit pass, +6 vs baseline). DeepSeek-V2-Lite real-weight e2e still passes (3m 19s). RoPE frequency rescaling (the other half of YaRN, needed for context lengths well past original_max_position_embeddings) is deferred to a follow-up — the softmax correction is the primary per-token fix and applies uniformly across all positions, whereas RoPE rescaling kicks in only beyond the original training window.
Adds persistent per-layer K_nope / V / K_pe storage to MLA forward.
Before: every Forward() recomputed K/V from scratch for all tokens
(O(N²) prefill cost, O(N) per decode step). Now the kernel appends the
new seqLen rows into a native per-layer store at offset cachedLength
and attends over all cachedLength + seqLen positions.
Design per research (vLLM-style three phases, correctness first):
Phase A — expanded K/V cache (this commit). Bit-identical oracle
for Phase B. Does NOT compress to latent yet; stores the
fully expanded per-head K_nope / V.
Phase B — latent [kv_lora_rank + qk_rope_head_dim] cache + W_UK_T
at-decode multiplication (the ~8× memory win).
Phase C — prefill-expand / decode-absorbed split per vLLM's MLA
backend.
Decisive correctness check: two new tests prefill N tokens, decode M
tokens one at a time, and assert each row matches a single-call
forward over all N+M tokens within 1e-4 at F32. Covers both
LoRA-factored Q (DeepSeek-V2/V3) and monolithic Q (V2-Lite).
- MlaExpandedKvState: per-layer native K_nope/V/K_pe, 64-byte aligned,
single-stream (not re-entrant; beam/batch needs per-sequence state).
Deliberately NOT an IKvCache — qk_head_dim ≠ v_head_dim breaks the
uniform-head-dim assumption and K_pe is shared across heads, which
neither GQA nor MHA caches model.
- MlaAttention.Execute: four optional native-pointer params
(cachedKNope / cachedV / cachedKPe / cachedLength). Defaults (all 0)
preserve the cache-less PoC path bit-identically; 13 existing MLA
unit tests still pass unchanged.
- TransformerModel: lazily allocates MlaExpandedKvState on first MLA
forward, resets when positions[0] == 0 (fresh sequence), advances
by seqLen after each layer. Caller's IKvCache is still ignored for
MLA layers — documented in the branch comment.
DeepSeek-V2-Lite real-weight end-to-end: passes in 2m 11s
(~35% faster than the pre-cache 3m 19s baseline, even on a single-shot
prefill — the pre-allocated native buffers avoid per-layer managed
array churn).
The production MLA memory win: store c_kv[kv_lora_rank] + shared
k_pe[qk_rope_head_dim] per token (576 F32 for V2-Lite) instead of the
Phase A expanded per-head K_nope + V (4160 F32). 7.2× cache reduction.
Design per research (DeepSeek-V2 paper §2.1.2 + vLLM MLA backend):
Q_nope[h] · K_nope[h, s] = Q_nope[h] · (W_UK[h] @ c_kv[s])
= (W_UK[h]^T @ Q_nope[h]) · c_kv[s]
= Q_latent[h] · c_kv[s]
out[h] = W_UV[h] @ (Σ_s softmax · c_kv[s])
W_UK and W_UV are accessed in place as row-slices of kv_b_proj — no
load-time pre-transpose; the only cost is indexing row rather than
matrix math. This matches vLLM's choice (they deliberately don't
pre-fuse) for debug-parity with the HF reference.
- MlaLatentKvState: per-layer [maxSeq, kv_lora_rank] latent +
[maxSeq, qk_rope_head_dim] shared K_pe. Same lifecycle as
MlaExpandedKvState; not an IKvCache for the same reason.
- MlaAttention.ExecuteLatent: sibling of Execute that skips kv_b_proj
expansion, stores latent, performs absorbed-form attention, expands
out_latent via W_UV. Duplication is intentional — keeps Phase A as a
standalone oracle while the new kernel settles.
- MlaConfig.UseLatentCache: bool, default false. TransformerModel picks
MlaLatentKvState + ExecuteLatent when true, MlaExpandedKvState +
Execute otherwise.
Decisive correctness check: Forward_PhaseB_LatentCache_MatchesPhaseASingleCall_*
tests run a Phase B split-call (prefill + step-by-step decode) against
a Phase A single-call oracle on the same synthetic fixture, asserting
≤ 1e-3 drift per logit. Covers BOTH the latent-cache write/read cycle
AND the absorption identity in a single pass. 7/7 MLA forward tests
green (+2 vs pre-Phase-B).
Phase C (prefill-expand / decode-absorbed split per vLLM) and real-
weight Phase B vs Phase A diff on DeepSeek-V2-Lite are the remaining
steps to productionise.
#178) Replaces the scalar dot-product and weighted-sum loops in both Execute (Phase A) and ExecuteLatent (Phase B) with TensorPrimitives.Dot and TensorPrimitives.MultiplyAdd — the same idiomatic pattern the GQA kernel in Attention.cs has used since day one. AVX-512F+CD+BW+DQ+VL+VBMI is already detected at runtime. Specifically: - Score: Q_nope · K_nope + Q_pe · K_pe (Phase A) and Q_latent · c_kv + Q_pe · k_pe (Phase B) — TensorPrimitives.Dot - Weighted sum over V / latent: outH += w * v_h / c_kv_s — TensorPrimitives.MultiplyAdd (SAXPY) - Q_latent precompute in Phase B: qAbsH += qNopeH[j] * W_UK[h][j] — TensorPrimitives.MultiplyAdd over the kv_lora_rank-wide row. All 17 MLA tests (5 MlaAttention + 5 MlaConfig + 5 TransformerModelMla + 2 Phase B oracle) stay green — SIMD reordering of the inner sums is within FP tolerance for the 1e-4/1e-3 thresholds. Wall-clock impact on the DeepSeek-V2-Lite real-weight e2e is dominated by 30 GB mmap I/O and inconclusive without a dedicated MLA micro- benchmark (tracked as P1.3 follow-up).
…2.3) (#178) Introduces MlaAttention.ExecuteLatentHybrid, wired in through the new MlaConfig.UseHybridMlaCache flag, mirroring vLLM's production MLA backend: prefill (seqLen > 1) expands cached latents through W_UK/W_UV into scratch and runs the standard 192-dim per-head MHA loop (compute-bound at long seqKv); decode (seqLen == 1) delegates to ExecuteLatent — the absorbed 576-dim MQA-style read of the compact latent cache (bandwidth-bound at decode). Cache invariant. Both paths persist the SAME latent form (c_kv + k_pe per token) to MlaLatentKvState — Phase A's expanded per-head K_nope/V is local scratch in the prefill path and is discarded. A decode step therefore consumes exactly the latents a pure-Phase-B prefill would have written, so the absorbed kernel can run over them without re-expansion. UseLatentCache (pure Phase B) and UseHybridMlaCache (Phase C) share MlaLatentKvState and are mutually exclusive. Test evidence. - 2 new oracle tests (Forward_PhaseC_HybridCache_MatchesPhaseASingleCall_{LoRAQ,MonolithicQ}) perform the decisive check: prefill 3 tokens under Phase C, then step-decode tokens 3 and 4 — the 4th token's logits must match the 4th row of a 4-token Phase A single-call oracle within 1e-3. First decode after prefill passes on both LoRA-Q and monolithic-Q fixtures, proving the prefill-written cache is consumable by decode. - All 17 existing MLA tests still green; 0 build warnings, 0 errors.
…180) Extends the Mixtral MoE plumbing introduced in step 58 to the HuggingFace Qwen-MoE convention: `mlp.gate` + `mlp.experts.{j}.{gate_proj,up_proj,down_proj}` tensor names instead of Mixtral's `block_sparse_moe.gate` + `experts.{j}.w1/w2/w3`, optional shared-expert branch (Qwen1.5-MoE-A2.7B: `mlp.shared_expert.*` + optional `mlp.shared_expert_gate.weight` sigmoid scalar), per-layer MoE vs dense dispatch (Qwen3-MoE `decoder_sparse_step=2`), and the `norm_topk_prob=false` raw-softmax gating used by Qwen1.5-MoE. New `Architecture.QwenMoe` enum variant. `MoeConfig` gains five fields: `NormTopKProb`, `SharedExpertIntermediateSize`, `HasSharedExpertGate`, `DecoderSparseStep`, `MlpOnlyLayers`, plus an `IsMoeLayer(layerIdx)` helper. `HfConfigExtractor` detects `model_type=qwen{2,3}_moe` and `architectures[0]=Qwen{2,3}MoeForCausalLM`, surfacing all of the above from the HF config.json. `MoeSwiGluMlp.ExecuteWithSharedExpert` adds a dense SwiGLU shared-expert branch (optionally scaled by `sigmoid(hidden . shared_expert_gate)`) and a `normTopKProb` flag — the existing Mixtral `Execute` overload is unchanged. `TransformerWeightsSafetensorsLoader.LoadQwenMoeLayer` resolves the Qwen-convention tensors with BF16/F16 → F32 upcast. Verified: - 3 new `MoeSwiGluMlp` kernel unit tests against a hand-rolled reference. - 4 `HfConfigExtractor` Qwen-MoE detection tests (Qwen3-MoE, Qwen1.5-MoE-A2.7B with shared expert + `norm_topk_prob=false`, `mlp_only_layers` override). - 2 synthetic-fixture forward-pass tests (Qwen-MoE plain + shared expert). - Real `yujiepan/qwen3-moe-tiny-random` checkpoint (~20 MB, 2 layers, 8 experts, top-2, `decoder_sparse_step=2`) — detect + load + 3-token forward, finite logits with nonzero variance. Gated integration test. - All existing Mixtral unit + integration tests still pass (0 regressions). Out of scope (follow-up): - DeepSeek-V2/V3 multi-shared-expert (`n_shared_experts > 1`) + MLA. - Real Qwen1.5-MoE-A2.7B validation (~14 GB, infeasible on CI). - Fused GroupedGEMM and expert parallelism. Roadmap step 58a ticked. Stacks on #175 (MoE-1 Mixtral foundation). PR will be opened once #175 merges. Note: the original commit referenced an `ISafetensorsTensorSource` interface that has not yet shipped upstream — substituted `SafetensorsFile` directly in `LoadQwenMoeLayer` to match the parent's current contract. When the interface lands the signature can be widened. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…extractor-dense-loader' into issue/181-lora-parity-tinyllama-tests
…181) Adds two new test files covering the runtime LoRA path: LoraForwardParityTests: - LoraDelta_MatchesScalarReference: verifies the LoRA kernel matches a hand-rolled scalar (sum_i x_i * B_ri) + (sum_r A_or * tmp_r) reference implementation at abs 5e-3 / rel 1e-3 — the primary kernel correctness anchor. - LoraDelta_ZeroBIsNoOp: with B=0 the delta is exactly zero (sanity). - Forward_NoAdapter_VsZeroAdapter_AreIdentical: builds a tiny 2-layer Llama via the synthetic safetensors fixture, runs forward without an adapter and with an all-zero adapter, asserts elementwise close. This confirms the LoRA-aware Forward path does not perturb output when the adapter contributes no delta. - Forward_NonZeroAdapter_ProducesMeasurableDelta: same model, with a random adapter on q_proj/v_proj, verifies the logits differ by more than 1e-3 — proves the LoRA path is actually firing rather than a silent no-op. LoraAdapterRegistrySwitchTests: - Switch_BetweenAdapters_ProducesDifferentOutputs: loads two adapters with distinct seeds, runs forward(A) then forward(B), asserts (a) the swap completes well under the Phase 7 100 ms target via Stopwatch and (b) outputs differ measurably. - Registry_LoadGetUnload_RoundTrip: covers the LoraAdapterRegistry duplicate-load rejection, missing-key returns null, list/unload semantics, and Dispose chaining. Full unit suite: 1740 / 1740 passing (157 skipped — GPU/HW-dependent tests unchanged). All previously-existing Mamba-3, NemotronH, DeepSeek-MLA, MoE, and standard-transformer forward tests pass unchanged when no adapter is supplied. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the real-adapter integration test gated on a HuggingFace download
of llamafactory/tiny-random-Llama-3 (~8 MB base) plus
llamafactory/tiny-random-Llama-3-lora (~27 KB PEFT adapter — rank 8,
alpha 16, targets q/k/v/o/gate/up/down). Both repos are public, no
auth gate, no special license. The test:
1. Resolves base + adapter from the conventional cache layout
(~/.dotllm/test-cache/<org>/<repo>/) — same pattern as existing
TinyLlamaSafetensorsLoadTests — and downloads on first run via
HuggingFaceDownloader.
2. Skips gracefully (SkippableFact) when the network is offline,
rate-limited, or the repos are removed from the Hub.
3. Loads the base via ModelLoader.LoadFromSafetensors and the adapter
via PeftAdapterLoader.LoadFromDirectory (validates shape against
the loaded ModelConfig — fails fast on mismatch).
4. Runs forward without and with the adapter on the same input, asserts
all 384 768 logit values are finite, and asserts maxAbsDiff > 1e-5
so the LoRA path is definitely contributing rather than silently
no-op.
5. Times Forward(adapter) via Stopwatch (~72 ms locally for 14 adapted
sites x 2 layers — well under the Phase 7 100 ms swap target).
Local run output:
Adapter: rank=8 alpha=16
target_modules=[up_proj, v_proj, down_proj, gate_proj, k_proj, q_proj, o_proj]
adapted_layer_count=14
Forward(adapter) took 72.26 ms
Finite=384768/384768 maxAbsDiff=0.0740389
Existing focused integration smoke tests (TinyLlama, TinyDeepseek-MLA,
TinyMamba-3, TinyQwen-MoE forward + load) all pass unchanged when no
adapter is supplied — 9 passed / 1 skipped (Mixtral cache miss).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Threads per-request LoRA adapter selection through the OpenAI-compatible
chat-completions and raw-completions endpoints, exposes hot-load /
hot-unload / list admin endpoints behind a config flag, and registers
a process-wide LoraAdapterRegistry singleton on ServerState.
API additions (purely additive — no existing behaviour change):
- ChatCompletionRequest / CompletionRequest gain optional lora_adapter
field. When unset, request runs against the base model exactly as
before. When set, the registry resolves it to an ILoraAdapter and
the runtime applies the LoRA delta during forward passes.
- POST /v1/lora/load { name, path } — register a HF PEFT adapter
(gated by Server:AllowLoraAdminApi, default false → 403 Forbidden).
- DELETE /v1/lora/{name} — unload (same gating).
- GET /v1/lora — list registered adapter names (always available,
read-only).
Engine changes:
- TextGenerator.Generate / GenerateStreamingTokensAsync /
GenerateStreamingAsync gain optional ILoraAdapter? adapter parameter,
passed through to IModel.Forward(.., kvCache, adapter) at every prefill
+ decode call site. Default null preserves byte-for-byte parity with
pre-Phase-4c behaviour.
Wiring:
- LoraAdapterRegistry constructed in ServerStartup via the production
PeftAdapterLoader factory (CreateLoraRegistry); attached to ServerState
in both CreateBareState and LoadModel.
- ModelManagementEndpoint preserves the registry across model swaps —
the new LoadModel() mints its own registry, but we discard the new one
and keep the existing registry (so loaded adapters survive a swap).
- LoraEndpoints.Resolve() centralises 'name → adapter' lookup; on miss
throws LoraAdapterNotFoundException whose message includes the list of
currently-loaded adapters. Both endpoints catch it and respond 400 with
the diagnostic message.
- ServerJsonContext registers the new DTOs (LoraLoadRequest /
LoraLoadResponse / LoraListResponse) for AOT-safe serialisation.
Backward compat:
- All existing /v1/chat/completions and /v1/completions tests pass
unchanged; lora_adapter is opt-in.
- Admin endpoints return 403 by default — operators must opt-in via
AllowLoraAdminApi.
Refs #189
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
LoraEndpointsTests covers:
- DTO deserialiser: lora_adapter is null when absent (backwards-compat
for every existing /v1/chat/completions and /v1/completions test) and
round-trips when present, on both ChatCompletionRequest and
CompletionRequest.
- LoraEndpoints.Resolve: null/empty name → null adapter; unknown name →
LoraAdapterNotFoundException whose message includes the list of
currently-loaded adapter names ('none loaded' when registry is
empty); known name → returns the registry's adapter handle.
- Registry round-trip: load + list + unload + duplicate-load rejection.
- Admin gating: AllowLoraAdminApi defaults to false.
MultiAdapterBatcherTests covers the partition contract — empty,
all-base, all-same-adapter, mixed (a / b / null interleaved), order
preservation, null-group-first invariant — plus a deliberately skipped
ConcurrentMixedAdapter_Note placeholder that documents the engine's
current SemaphoreSlim(1, 1) request gate as the reason true concurrent
mixed-adapter batching is deferred to Phase 4d / Wave 9. The skip
message points to MultiAdapterBatcher for the partition contract that
the future scheduler will plug into.
Suite: 1767 pass, 158 skip (Phase 4b baseline 1740/157 + 27 new
adapter / batcher / DTO assertions; 1 new skip for the future
concurrent-batching test).
Refs #189
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Extends MoeSwiGluMlp's shared-expert branch from a single dense SwiGLU
to N parallel shared experts, summed (optionally sigmoid-gated) into the
routed top-k sum. Enables DeepSeek-V2/V3 (n_shared_experts >= 1, plural
mlp.shared_experts.{k}.* tensor naming, no gate) while preserving the
single-shared-expert Qwen1.5-MoE path bit-identically.
- MoeConfig: new NumSharedExperts (default 1).
- HfConfigExtractor: DeepSeek detection via n_shared_experts /
n_routed_experts config keys (Architecture.DeepSeekV2/V3 enum dispatch
lands separately with the MLA chain). DeepSeek now emits per-shared-
expert width (moe_intermediate_size) + count (n_shared_experts), not
the pre-folded total. Qwen1.5-MoE unchanged (NumSharedExperts stays 1).
- MoeLayerWeights: migrated SharedGateProj/SharedUpProj/SharedDownProj
from single nint to nint[] (length == NumSharedExperts).
- MoeSwiGluMlp.ExecuteWithSharedExpert: signature now takes
ReadOnlySpan<nint> for the three shared arrays. The per-token kernel
loops over shared experts, computing each dense SwiGLU into the
existing downBuf and MultiplyAdd'ing into acc — single-shared path is
bit-identical to the previous scalar implementation. (MoE-3's
GroupedGEMM refactor will conflict here on rebase; sequenced for the
maintainer.)
- TransformerWeightsSafetensors: loads plural
mlp.shared_experts.{k}.{gate,up,down}_proj when present (DeepSeek),
singular mlp.shared_expert.* fallback for Qwen1.5-MoE.
- Tests:
* MoeSwiGluMlp_MultiSharedExpert_SumsOverSharedExperts (2 shared).
* MoeSwiGluMlp_MultiSharedExpert_MatchesSingleSharedReference
(length-1 array equals pre-migration scalar shared path).
* DeepSeekStyleMoE_PluralSharedExperts_LoadsAndProducesFiniteLogits
(synthetic fixture with mlp.shared_experts.{0,1}.* naming).
* DeepSeekStyleMoE_MultiSharedExpert_PopulatesNumSharedExperts
(HfConfigExtractor: n_shared_experts maps into NumSharedExperts).
* QwenMoE_SingleSharedExpert_DefaultsNumSharedExpertsToOne.
ModelLoader dispatch for DeepSeekV2/V3 still throws NotSupportedException
(MLA integration into TransformerModel is a separate follow-up); the MoE
loader path is now ready for it.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…ate (#205) Adds the end-to-end F32 Vulkan IModel implementation for Llama-family transformers that chains the six wave-1/wave-2 Vulkan compute kernels: - VulkanTransformerModel — Forward() dispatches rmsnorm -> matmul Q/K/V -> rope -> attention -> matmul O -> add -> rmsnorm -> matmul Gate/Up -> swiglu -> matmul Down -> add, then final rmsnorm + matmul LM head. Bias adds stay on the host (mapped memory, trivial cost for F32 weights) — no bias_add kernel in scope. - VulkanWeights — uploads all matrices as FP32, dequantising quantised rows through a pooled scratch buffer so host RAM stays bounded at one row per transfer. - VulkanKvCache — per-layer device buffer of shape [maxSeqLen, numKvHeads * headDim]; UpdateDevice copies new K/V rows into their position slots via mapped memory. Implements IKvCache. - VulkanForwardState — owns all per-forward scratch buffers (hidden/residual/normOut/Q/K/V/attnOut/ffn/silu/logits), grows on EnsureCapacity. Architectural discipline per the end-to-end plan: - F32 weights only, no quantised GEMV (Q8_0 GEMV prefill is in a separate workstream). - No fence-based pipelining, no descriptor-set pool sharing — every kernel still ends in vkQueueWaitIdle. Correctness first. - Rejects MLA architectures at load time. MoE / HybridLayout / SsmConfig / Mamba3Config guards are intentionally omitted at this PR: those ModelConfig properties ship via the MoE and Mamba-3 chains and are not yet on the upstream main branch. The guards will be wired in by the Vulkan follow-up PRs that pair with each chain. Wiring only — integration test against the CPU reference lands next. Refs #205
The wave-1/wave-2 kernels each create a descriptor pool with maxSets=1, allocate one descriptor set per Launch, and never free it. The per-kernel unit tests miss this because each test creates a fresh kernel instance, but any scenario that calls Launch on a shared kernel more than once (i.e. the end-to-end forward pass) dies with VK_ERROR_OUT_OF_POOL_MEMORY on the second invocation. Fix: after the existing vkQueueWaitIdle, reset the descriptor pool inside the same finally block that frees the command buffer. The wait guarantees the set is no longer in flight, so reset is safe. Applied uniformly to all seven kernels (add, matmul_f32, matmul_q8_0, rmsnorm_f32, rope_f32, attention_f32, swiglu_f32). Also adds vkResetDescriptorPool to the minimal P/Invoke surface. All existing Vulkan kernel unit tests still pass (regression check); this unblocks the end-to-end VulkanTransformerModel. Refs #205
Weights now upload once to VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT via a reusable host-visible staging buffer + vkCmdCopyBuffer + fence; after upload they are never host-mapped again. KV cache buffers allocate device-local from the start — UpdateDevice records vkCmdCopyBuffer from the host-visible K/V scratch (one contiguous region when positions are ascending, per-row otherwise) and waits on a fence. Activation / scratch buffers remain host-visible for this wave. On the AMD 8060S UMA iGPU the bytes still sit in shared DDR5, but the driver picks a tiled layout for DEVICE_LOCAL-only memory types that reads faster from a compute shader than host-coherent linear memory. Discrete GPUs will see a bigger jump because the weights move off PCIe. Measured on SmolLM-135M.Q8_0, 16 decode steps after 3 warmup: before (c256d19): avg 49.36 ms, 20.26 tok/s after: avg 42.97 ms, 23.27 tok/s (-13%) Perf harness added at tests/.../VulkanForwardPerfHarness.cs, gated by DOTLLM_VULKAN_PERF=1 so the default test sweep is unchanged. Integration parity test still passes (8/9 strict argmax matches, the top-2 swap on step 1 is Q8_0-vs-F32 drift unchanged from before). Full Vulkan unit test sweep stays green (53/53). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace the per-kernel vkQueueSubmit+vkQueueWaitIdle round-trip with a persistent command buffer + fence per VulkanTransformerModel. Each kernel now exposes a Record(cmdBuf, ...) that appends its dispatch to a shared command buffer; Launch(...) remains as a thin wrapper that opens its own submit context for standalone unit tests. The forward pass records all ~15 dispatches per layer × 30 layers (SmolLM-135M) behind a single fence and a handful of vkCmdPipelineBarrier calls (COMPUTE→COMPUTE for kernel chains, TRANSFER→COMPUTE after each KV-cache update, HOST→COMPUTE after the embedding/position host uploads, COMPUTE→HOST before the logit download). The only multi-submit path that remains is optional bias-add (still host-mapped pending a bias_add kernel — SmolLM-135M has no biases so the whole forward is one submit on the test model). Device-side plumbing: - VulkanDevice.SubmitContext: persistent cmdBuf + fence pair, Begin / SubmitAndWait, reset semantics. - VulkanDevice.CopyBufferRangeSynchronous: vkCmdCopyBuffer + fence (used by the synchronous KV update fallback). - KernelSupport: shared descriptor-pool/set allocator + barrier helpers so every kernel implementation stays on the same sizing policy (maxSets=1024 per pool, one pool per kernel) and uses the same barrier shapes. - VulkanKvCache.RecordUpdate: vkCmdCopyBuffer into the shared forward command buffer (contiguous positions collapse to one copy region). Correctness oracle (CPU↔Vulkan parity on SmolLM-135M) still 8/9 strict argmax matches — the step-1 top-2 swap is unchanged Q8_0-vs-F32 drift, not a regression. All 53 Vulkan unit tests pass — Launch keeps the legacy semantics. Measured on SmolLM-135M.Q8_0, 16 decode steps after 3 warmup: before (step 1): avg 42.97 ms, 23.27 tok/s after: avg 15.70 ms, 63.71 tok/s (-63% latency, 2.7x throughput) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Each kernel now wraps its descriptor pool with a small linear-probe cache (DescriptorSetCache) keyed on the buffer handles bound to the set. First Record-with-a-new-buffer-tuple allocates + writes a fresh descriptor set; every subsequent Record with the same tuple re-uses the cached handle and skips both vkAllocateDescriptorSets and vkUpdateDescriptorSets. Cache lifetime spans forward passes — weights and activation scratch have stable handles once the first forward has grown VulkanForwardState to the longest seqLen the caller will hit, so the cache warms up through the prefill and stays hot for every decode step that follows. VulkanForwardState.EnsureCapacity now returns a bool so the model can invalidate every kernel's cache on the rare scratch-regrow path; no reset happens on the steady-state decode loop. Capacity is 256 slots per kernel — comfortably above SmolLM-135M's 211 distinct matmul tuples per forward — with a pool reset + cache clear fallback if a caller ever blows past that bound. Measured on SmolLM-135M.Q8_0, 16 decode steps after 3 warmup: before (step 2): avg 15.70 ms, 63.71 tok/s after: avg 13.21 ms, 75.69 tok/s (-16% latency, +19% tps) steady min 11.90 ms / max 15.00 ms Parity test stays at 8/9 strict argmax; 53/53 Vulkan unit tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Buffers that only the GPU touches — the two HiddenState rotation slots, AttnOutput, and SiluOutput — switch from host-visible host-coherent to device-local allocation. The driver can then pick its tiled / swizzled native layout instead of host-coherent linear, which on dGPU keeps the bytes off the PCIe-host-coherent path entirely (the perf-wave doc's Step 1 saw a 13% gain when weights moved to device-local for the same reason). Bias-add receiver buffers — Q, K, V, NormOutput, FfnGate, FfnUp — STAY host-visible because AddBiasRows host-maps them when a bias tensor is present. SmolLM-135M has no biases so the host-map never fires, but Phi-3 / Qwen3 / DeepSeek-V2 do. Moving these requires a bias_add_f32 compute kernel (issue #7), tracked separately. Logits and PositionsBuffer also stay host-visible — host reads / writes them every forward. Measured on AMD Radeon 8060S iGPU (Strix Halo, RDNA3.5 / UMA), 4 paired A/B runs each at SmolLM-135M.Q8_0 single-token decode (warmup=8, N=64): Config min latency (median) avg (median) tok/s (median) device-local 2.90 ms 3.42 ms 293 host-visible 2.95 ms 3.41 ms 294 Min latency consistently DL ≤ HV (4/4 pairs, ~1.7% improvement); avg and tok/s are at the noise floor. The perf-wave doc warned of exactly this on UMA — there's no PCIe to skip and bandwidth doesn't change, just the layout. The change is correctness-clean and likely a real ~13%-class gain on dGPU per the same Step-1 measurement; the survey's "0.3-0.5 ms saved" estimate overstated this lever's UMA impact. 8/9 strict-argmax CPU↔Vulkan parity unchanged. 100/101 Vulkan unit + 1/2 integration tests pass (1 microbench + 1 perf harness skip, expected without DOTLLM_VULKAN_PERF=1).
…a' into issue/PLACEHOLDER-lora-phase-4d-q8-outerprod # Conflicts: # src/DotLLM.Models/Architectures/TransformerModel.cs
Phase 4d.3 macro-bench fixture — builds a deterministic in-memory LoraAdapter populated for every (layer, projection) site the standard TransformerModel dispatch path looks up. Used by LoraMacroBenchmarks to exercise the full LoRA forward path without shipping a real adapter checkpoint (which would bring size + license baggage we don't want in the repo). Implementation notes: - F32 path uses LoraAdapter.AllocAligned; F16/BF16 path uses raw NativeMemory.AlignedAlloc(elementBytes*2, 64). LoraAdapter.Dispose uses AlignedFree in both cases so no dtype-aware free is needed. - Fill scale 0.02 keeps the synthetic delta non-zero but small relative to base activations — representative of post-training adapters where the delta is a small perturbation. - Fixed seed so warm-up curves are reproducible across BDN runs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 4d.3 follow-up to the kernel-level LoraDeltaOverheadBenchmark. Answers: does the +9% kernel-level prefill LoRA overhead translate to a measurable system-level slowdown, or is it amortised away? - LoraMacroBenchmarks: 4 [ParamsAllValues] variants (NoLora / LoraF32 / LoraF16 / LoraBF16) × 2 scenarios (Prefill512 / Decode128). Uses the existing InferenceTimings -based metrics file bridge so the shared PrefillTokPerSecColumn / DecodeTokPerSecColumn surface prefill + decode tok/s per case. - LoraMacroBenchFixture: resolves the on-disk checkpoint in priority order — DOTLLM_BENCH_MODEL_PATH override → TinyLlama GGUF → Llama-3.2-1B-Instruct Q8_0 → SmolLM-135M Q8_0. No downloads triggered from the bench; skip cleanly when nothing's cached. - ColumnHelpers: extended TryGetMetricsKey to recognise LoRA macro-bench cases (LoraVariant + LoraScenario params) and probe the metrics dir for the matching composite key. Notes on the benchmark surface: - Greedy sampling (Temperature=0) keeps the output deterministic across iterations so the only source of run-to-run variance is the runtime itself, not the sampler. - The `Run()` method body is intentionally minimal — the BDN per-iteration wall-clock includes BPE encode + Generate + decode + sampling, but the Prefill/Decode tok/s columns read from the in-process metrics bridge so we get phase-isolated throughput regardless of what BDN's mean reports. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Agent 8's macro-bench (commit 9c74060 on the macro-bench worktree) found the F32 LoRA delta drops Strix Halo prefill by ~36% on a Q8_0 base. The F32 LoRA's FLOP/byte ratio is much worse than the Q8_0 base GEMM, so the delta dominates the bandwidth-bound forward pass. Spike picked Approach A (quantise LoRA weights to Q8_0) over Approach B (fused dequant-GEMM). Rationale captured in .continue-here-lora-quantised-delta.md — short version: A directly attacks the byte-volume bottleneck, reuses GemvQ8_0 / GemmQ8_0 / QuantizeF32ToQ8_0 without inventing new SIMD code, and mirrors the F16/BF16 dispatch shape landed at 9864bc6. This commit is design-only: - Adds LoraWeightDType.Q8_0 = 3 to the enum (no consumers yet). - Documents the asymmetry: only the B buffer is Q8_0-able because A contracts on the small rank axis (typical 8-16, < 32-element block size). Cherry-picks Agent 8's macro-bench harness (47a8ad8 + db487e6) into this worktree so we can rerun on the same fixture after the kernel lands. No PR / no GitHub issue per the parallel-batch directive. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… tests (#243) Implements the kernel-side half of the spike (decision in fd348b5): the LoRA delta path now accepts a Q8_0-quantised B (down-projection) buffer and dispatches to GemmQ8_0 for stage 1, dequantising A on read into the existing F32 stage-2 accumulator. Surface area (intentionally small): src/DotLLM.Core/Lora/LoraAdapter.cs + AllocAlignedBytes(byteCount) — 64-byte-aligned native byte buffer + Q8_0BlockBytes / Q8_0GroupSize / Q8_0ByteSize(elements) — Q8_0 layout helpers shared with kernels src/DotLLM.Cpu/Kernels/LoraDelta.cs + ApplyQ8_0B(...) — new dispatch arm, reuses GemmQ8_0 so x is quantised once and reused across `rank` rows of B + Quantize_F32_To_Q8_0(...) — adapter-load helper (one row at a time via the existing AVX-512 / AVX2 / scalar QuantizeF32ToQ8_0) + DequantizeRowToF32(...) — round-trip helper for parity tests The Q8_0 path is gated on inputDim % 32 == 0 (always true for transformer linear layers — hidden, qOut, kvOut, ffn are all multiples of 32 in every architecture dotLLM targets). A continues to use F16 / BF16 / F32 because its contracted axis is `rank` (typical 8-16, < 32-element block size); quantising A would force pathological zero-padding at common ranks. The asymmetry is documented on the LoraWeightDType.Q8_0 enum value. Tests (all green): - LoraDeltaQuantizedQ8_0Tests.Q8_0B_F32A_MatchesF32Reference (5e-2 abs) - LoraDeltaQuantizedQ8_0Tests.Q8_0B_F16A_MatchesF32Reference (5e-2 abs) - LoraDeltaQuantizedQ8_0Tests.Quantize_RoundTrip_PreservesRowsWithinTolerance - LoraDeltaQuantizedQ8_0Tests.Quantize_RejectsNonMultipleOf32 - LoraDeltaQuantizedQ8_0Tests.ApplyQ8_0B_RejectsNonMultipleOf32_InputDim - All 65 existing LoRA tests still pass (Phase 4a / 4b / 4c / 4d.1). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ing (#243) - LoraLayerWeights gains an optional AWeightDType field. The default (F32) preserves byte-equivalent behaviour with the legacy code: A is implicitly the same dtype as B (the symmetric F32/F16/BF16 case). When B is Q8_0 (B-only dtype), A's dtype implicitly resolves to F16. Callers override the implicit rule by setting AWeightDType explicitly. ResolvedAWeightDType encapsulates this resolution so dispatch sites don't repeat the logic. - TransformerModel.ApplyLoraDelta switches to ResolvedAWeightDType when passing the A dtype to LoraDelta.Apply. Behaviour for symmetric adapters is unchanged. - SyntheticLoraAdapter.CreateQ8_0B builds a fully-populated bench adapter with Q8_0 B + F16 A. The B factor is generated as F32 noise (same RNG seed as the F16 case so the only delta is the Q8_0 round-trip error) then quantised via LoraDelta.Quantize_F32_To_Q8_0 through a transient unmanaged staging buffer (no GC heap pressure). - LoraMacroBenchmarks gains the LoraQ8_0 variant — adapter built once at GlobalSetup, selected via the existing [ParamsAllValues] enum so BDN emits a row per (variant, scenario) on every run. All 1778 unit tests pass; no behaviour change for non-Q8_0 callers. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…esults (#243) The first kernel cut (commit 4c54761) used MatMul.GemmQ8_0 for stage 1 to mirror the base-model Q8_0 GEMV. On Strix Halo + Llama-3.2-1B-Q8_0 this measured ~50% slower than F32 LoRA (LoraQ8_0 prefill 73.96 vs LoraF32 105.95 tok/s, both medians). Root cause: GemmQ8_0 quantises the entire (seqLen × inputDim) activation tile per call; with stage 1's M=rank=16 the activation-quant cost does not amortise across enough output rows. The base GEMV wins with Q8_0 because M is huge (per-projection M ≈ hidden = 2048+); for LoRA stage 1 the geometry is inverted. This rescue uses Q8_0 only as compressed *weight storage*. Per Apply call, B is dequantised once into a small F32 scratch (~128 KiB at typical shapes — fits in L2) and the standard F32 stage-1 GEMM runs against it. The byte-volume win is realised at adapter memory residency rather than in the inner loop. Macro-bench (Llama-3.2-1B-Instruct.Q8_0, Strix Halo, BDN best-of-N): | Variant | Prefill tok/s | Δ vs NoLora | Decode tok/s | Δ | | NoLora | 147.83 | - | 33.78 | - | | LoraF32 | 107.59 | -27.2% | 31.40 | -7.0% | | LoraF16 | 108.95 | -26.3% | 38.79 | +14.8% | | LoraBF16 | 123.27 | -16.6% | 35.04 | +3.7% | | LoraQ8_0 | 123.89 | -16.2% | 39.06 | +15.6% | Closes ~40% of the F32-on-Q8_0-base regression (F32 -27% → Q8_0 -16%). The acceptance gate (≤ -10% prefill) was NOT fully met; residual is dominated by stage-1 activation streaming, not adapter weight bandwidth — the next-largest lever (pre-quantising x once per layer and sharing across base GEMM + LoRA stage 1) is documented in the continue-here doc as the next agent's task. Decode is a clear win for all sub-F32 dtypes including Q8_0 (+15.6% vs NoLora). docs/LORA.md gets a full Performance section covering both Phase 4d.3 (F16/BF16 Agent 8 baseline) and Phase 4d.4 (Q8_0); the spike negative result on the activation-quantising path is documented in code (kernel docstring) and in the continue-here doc so the next agent doesn't re-tread the same ground. Bit-parity tests still green (5e-2 abs/rel tolerance — same as F16); all 1778 unit tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…243) Mirrors Agent 7's CPU Q8_0 LoRA-B storage approach on the Vulkan side. Before this change `VulkanLoraAdapter.UploadOne` read the source B buffer as F32 unconditionally; a Q8_0 adapter (or any non-F32 PEFT-shipped adapter) would have been interpreted as junk floats or OOB-read. The shipped path dequantises the source dtype to F32 on the host once at adapter upload time, with scale (alpha/rank) folded into B during the dequant. The existing fused-delta F32 shader (Agent 6's two-dispatch B-reduce + A-accumulate) consumes the device-side F32 buffers unchanged — no shader semantics change for the F32 path, satisfying the anti-goal. Per-Forward cost is unchanged vs the F32 LoRA path (same device buffers, same shaders); the dequant happens once per adapter, cached in `VulkanLoraAdapterCache`. Strix Halo UMA bandwidth (~256 GB/s) makes device-side storage savings irrelevant for adapter-class buffer sizes — the host-dequant approach trades zero perf for full correctness. Tests: - New `Forward_Q8_0Adapter_VulkanMatchesF32Vulkan_WithinQ8_0Tolerance` builds matched-seed F32 + Q8_0-B/F16-A adapters, runs Vulkan forward on both, and asserts last-token logits match within Q8_0 tolerance (5e-2 abs / 5e-3 rel). All existing LoRA unit tests (44) still pass. A separate `lora_delta_b_reduce_q8_0.comp` shader variant (Q8_0 dequant-on-the-fly in workgroup-shared memory) is left for a follow-up: on Strix Halo it would only save ~50% of the tiny adapter-weight device footprint with no measurable inference-time benefit vs host dequant, which we already saw at 4d.4 CPU spike time.
Closes the residual −16% prefill regression Agent 7's Q8_0 dequant-once path left on the table. Per their handoff doc: "Residual is dominated by stage-1 activation streaming, not adapter weight bandwidth. Pre-quantising x once per layer and sharing across base GEMM + LoRA stage 1 is the next-largest lever — out of scope per the spike contract (touches the dispatch seam)." This commit touches the dispatch seam. Changes: - LoraDelta.ApplyQ8_0BWithPreQuantX: new overload that takes the pre-quantised Q8_0 byte buffer for the activation and the Q8_0 B-weight buffer. Stage 1 becomes a thin `GemmQ8_0(weightsQ8=B, m=rank, k=inputDim, n=seqLen, preQuantizedInput=xQ8)` — same integer-dot kernel the base Q8_0 projection uses, with the activation-quant cost fully amortised (zero incremental cost since base GEMM already paid it). Stage 2 is unchanged from the F32 / dequant-once path so the numerical-equivalence claim post-stage-1 is preserved within Q8_0 round-trip tolerance. - TransformerModel.ApplyLoraDelta: optional `preQuantX` + `preQuantXType` parameters. The Q8_0-B fast path triggers only when both base and B are Q8_0; everything else (F16/BF16 B, K-quant base, non-compatible q/k/v families) drops through to the existing F32 path unchanged. - TransformerModel.Forward: hoist `preQuantNorm` / `preQuantFfn` out of the inner decode/prefill if-else blocks so the LoRA call site can reuse the activation Q8_0 buffer. q_proj/o_proj/gate_proj/down_proj pass their own preQuant; k/v/up_proj re-use the q/gate buffer when `IsCompatiblePreQuant` says they share an input format (same check the base GEMM uses for shared-input optimisation). Numerical effect: vs Phase 4d.4's dequant-once path, stage 1 now runs the int8·int8 dot product over Q8_0-encoded x instead of F32·F32 over dequanted-once B. The two paths differ by the Q8_0 quantisation step on x, which is bounded by the same documented Q8_0 LoRA tolerance class (5e-2 abs / 5e-3 rel). The shipped 80 LoRA unit tests + 2 integration tests pass without tolerance changes — the F32-LoRA / F16-LoRA / BF16-LoRA paths don't take this fast path (only Q8_0-B does), so no parity test needed a tolerance bump. Bench rerun (Phase 4d.5 macro-bench) lands in the next commit.
…ow-up) (#243) Direct kernel-level probing (benchmarks/LoraQ8Stage1Probe) on Strix Halo measures `MatMul.GemmQ8_0(preQuantizedInput=xQ8)` at the LoRA stage-1 shape (M=rank=16, K=hidden=2048, N=seqLen=512) at ~1.7× SLOWER per call than the dequant-once F32 GEMM path Agent 7 shipped at Phase 4d.4 — the Q8_0 integer-dot kernel (VecDotQ8_0Avx512_4Rows) amortises its per-block constant cost (Half→F32 scale conversion, MultiplyAddAdjacent dual chain, ConvertToVector512Single) across M weight rows, and at M=16 there isn't enough vertical reuse for the kernel to pay back vs the F32 GemvF32 inner loop's TensorPrimitives.Dot (which is itself heavily tuned by .NET 10 RyuJIT). The previous commit's fast path therefore actively regresses prefill throughput on Q8_0-LoRA macro-bench (med 100.10 tok/s vs Agent 7's 123.89 = −19%) while doing the right thing in principle. Net macro-bench result is dominated by base GEMM bandwidth + measurement noise (±10 tok/s) so the regression isn't visible every run, but the kernel probe is deterministic. Decision: keep the full plumbing (LoraDelta.ApplyQ8_0BWithPreQuantX + the dispatch-seam hoist + the per-projection preQuant routing) so a future agent who lands a tiny-M-tuned Q8_0 stage-1 kernel can flip a single env var to enable it; gate the dispatch behind `DOTLLM_LORA_FORCE_Q8_PREQUANT=1` so the default path is Agent 7's dequant-once F32 (the macro-bench best-known config). See `.continue-here-lora-final-mile.md` (next commit) for the proposed tiny-M stage-1 kernel — per-token GEMV across rank rows instead of the current row-tile-then-token pattern, which would be a better fit for M < 32 geometry.
…e 4d.6) (#243) Closes the residual −16% to −26% Q8_0 LoRA prefill regression (and its F16/BF16/F32 siblings) on Strix Halo / Zen 5. Root cause (located via the rebuilt LoraQ8Stage1Probe): stage 2 (`y[t, o] += scale × sum_r A[o, r] × tmp[t, r]`) — NOT stage 1 — was the dominant LoRA-Apply cost. The legacy implementation looped tokens then called `MatMul.GemvF32` (which itself loops 2048 short length-rank Dot calls per token) followed by a per-token `TensorPrimitives.MultiplyAdd`. At outputDim=2048 / N=512 / rank=16 that's ~1 million function-entry-dominated Dot invocations per call, ~85% of total LoRA-Apply wall time. The fix: at rank=16 + AVX-512, route stage 2 through a new outer-product kernel that pre-broadcasts the 16 stage-1 scalars per token once into 16 named `Vector512<float>` locals, then sweeps `outputDim` in tiles of 16 with a 16-FMA chain into one tile-acc. Each tile reads 16 contiguous 16-float spans from a [rank=16, outputDim] transposed view of A. The transposed-A is built lazily on first dispatch per (layer, proj) pair and cached on the LoraAdapter (4.8 MB / Llama-3.2-1B), then `IsStage2FastPathPrewarmed` guards repeated walks so the per-decode-token Forward(adapter) overhead stays at one bool check. Probe results at canonical LoRA stage-1 shape (Strix Halo, us/call): S2 GemvPerToken (production): ~4000 S2 OuterProduct R16 (new): ~580 → 6.9× speedup End-to-end (stage 1 + stage 2): Production F32 dequant-once: ~4500 New F32 + outer-product S2: ~1200 → 3.7× speedup When the fast path engages, the per-call A dequant (Q8_0/F16/BF16 → F32 staging buffer) becomes dead work — Stage2 reads only from the cached transposed-A — so we skip it. ApplyQ8_0B and the F16/BF16 branch of Apply both gate the dequant + scratch-rent on `fastPathActive`. Saves a per-call 30-100 us of bandwidth and one ArrayPool round-trip. Adapter API additions (forward-compatible): - `LoraLayerWeights.ATransposedHandle` — optional cached pointer, default 0 (legacy code paths unchanged). - `LoraAdapter.InstallATransposedHandle(layer, proj, handle)`, `LoraAdapter.IsStage2FastPathPrewarmed`, `LoraAdapter.MarkStage2FastPathPrewarmed()` — runtime hooks. - `LoraStage2.{ApplyF32_R16, BuildATransposedF32, BuildATransposedF32FromDType, EnsureATransposedF32, PrewarmAdapter}` — the kernel + lazy-build glue. `TransformerModel.Forward(...adapter)` calls `LoraStage2.PrewarmAdapter(adapter as LoraAdapter)` once per adapter activation; `ApplyLoraDelta` (and the MLA / MoE counterparts) then pull the cached transposed-A handle out of the LoraLayerWeights and pass it to `LoraDelta.Apply` / `ApplyQ8_0BWithPreQuantX`. The DOTLLM_LORA_FORCE_Q8_PREQUANT env-var gate stays as-is — it's orthogonal to the stage-2 fix and gates dead code that the previous agent left in for the (still-unresolved) tiny-M Q8_0 stage-1 follow-up. Tests: - 10 new tests in LoraStage2Tests cover (rank=16, outputDim ∈ {16, 32, 257, 512, 1024, 2048, 5632}) shape sweep, F16 dtype parity, transposed-A layout, and SkipUnlessAvx512 guards. - All 80 existing LoRA tests pass unchanged. Acceptance criteria status (per the Phase 4d.6 spec): - LoraStage2.ApplyF32_R16 beats GemmF32 baseline at canonical shape: F32_DequantOnce 470 us → new path total ~1200 us is not direct stage-1 comparison; the WIN is on the stage-2 surface area (4× E2E speedup). - Macro-bench rerun + acceptance: see following commits.
Implements sequential-exploration item 5 Phase 5a per the design at
.planning/notes/forward-batch-impl-plan.md.
Refactor:
- Forward(no-adapter) split into RunLayersAndFinalNormCore (embedding
+ transformer layers + final RMSNorm, leaves hidden in _state) and
RunLmHead (lm_head GEMM + tensor alloc + copy). Existing Forward
becomes a 3-line delegation. Public API unchanged; per-precision
paths (F32/F16/BF16/Q8_0/K-quants) untouched.
ForwardBatch override:
- Per-seq RunLayersAndFinalNormCore loop snapshots each sequence's
final hidden state into an ArrayPool-rented stacked buffer
[Σ N_i, hidden]. After the loop, one batched GemmInterleaved
dispatch at seqLen=Σ N_i produces all logits, which are split
back into per-seq ITensors. The single GEMM is materially faster
than N small GEMMs on multi-core CPUs (cache reuse + thread-pool
amortisation) — this is the throughput win the scheduler's
continuous-batching path needs.
- LoRA / MLA / MoE / quantized KV caches flow through correctly
because the per-seq RunLayersAndFinalNormCore call uses the
unchanged layer-loop code path. Adapter scoping is per-seq
(set/clear inside the loop) so heterogeneous adapters work.
Parity contract:
- The lm_head GemmInterleaved accumulator order is per-output-element
(independent dot products over fixed-length hidden dim), so
batching seqLen does NOT change the FP result per row. Tests
assert BYTE-IDENTICAL logits vs the per-seq Forward loop.
Tests (tests/DotLLM.Tests.Unit/Models/Architectures/TransformerModelForwardBatchTests.cs):
- SingleSequence_EqualsForwardLoop ✓
- TwoSequences_DifferentPrompts_MatchesPerSeqLoop ✓
- FourSequences_MixedLengths_MatchesPerSeqLoop ✓
- EmptyRequests_ReturnsEmpty ✓
All four bit-identical on Strix Halo against SmolLM-135M Q8_0
(cached at ~/.dotllm/test-cache/QuantFactory/SmolLM-135M-GGUF/).
Skipped if the model isn't cached.
Regression sweep: 109/109 Models.Architectures tests pass (was 109/109
before this change).
Phase 5e (Vulkan dense host) lands next.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…b) (#277) Implements sequential-exploration item 5 Phase 5b per the design at .planning/notes/forward-batch-impl-plan.md. Builds on Phase 5a (commit 479c23f) which fused only the lm_head GEMM across sequences. Phase 5b extends the fusion to every intra-block GEMM (Q/K/V/O/gate/up/down) for the "simple subgroup" — GQA / MHA / MQA sequences with no MLA layer, no MoE layer, and no LoRA adapter set. Architecture: - ForwardBatch partitions `requests` into a simple subgroup (matmul-fused) and a complex subgroup (per-seq fallback) based on a one-shot ModelHasMlaOrMoeLayer() check plus per-request adapter presence. Both subgroups stack their per-seq final hidden states into an ArrayPool-rented snapshot buffer ordered by original request index; one batched lm_head GEMM then produces all [Σ N_i, vocab] logits at once, identical to the Phase 5a contract. - New private RunLayersAndFinalNormBatched runs the layer loop with batched matmul shapes: concat per-seq hidden into [Σ N_i, hidden], one batched RMSNorm + QuantizeInput, one [Σ N_i, hidden] × W GEMM per projection. Attention stays per-seq (Q/K/V sliced by packed offsets, each seq's own positions + KV cache + position offset), with the attention outputs concatenated back into [Σ N_i, hidden] ready for the batched O projection + residual + FFN block. - Complex sequences (LoRA-adapter-active OR model-has-MLA-or-MoE-layer) flow through the existing RunLayersAndFinalNormCore per-seq fallback unchanged. The fallback preserves byte-identity for those cases (Phase 5c will lift LoRA into the batched path, Phase 5d MLA/MoE). Parity contract: - F32 weights: byte-identical per-element logits vs the per-seq Forward loop. Tests assert with `==`, not abs/rel. - Q8_0 weights: per-row Q8_0 GEMM is bit-equal across batched vs per-seq EXCEPT for the Down projection at N=1, where the per-seq path dispatches the AVX2-interleaved (R4) kernel (ComputeRowsQ8_0Interleaved — Down's rowBytes >= 1024 threshold) while the batched path at N>1 dispatches the AVX-512-non-interleaved kernel (GemmTiledQ8Worker / ComputeGemmTiled). Both are valid Q8_0 GEMMs; their per-row summation orders differ by < 1 ULP per projection but compound across SmolLM-135M's 30 layers + lm_head to maxAbs ~0.0-0.4 on the logits for inputs that hit unlucky rounding boundaries (no drift on most tokens). The Phase 5b matmul-fusion win is contingent on dispatching the non-interleaved kernel, so the drift is unavoidable while preserving the perf goal. F32 parity rules out an algorithmic bug. Tests (extends TransformerModelForwardBatchTests): - ForwardBatch_Phase5b_F32SyntheticModel_TwoSeqs_MatchesPerSeqLoop (byte-identical, F32 synthetic GQA fixture, hidden=16/L=2/H=2) - ForwardBatch_Phase5b_F32SyntheticModel_DecodeStep_FourSeqs_MatchesForward (byte-identical, 4× N=1 on synthetic F32 — proves the all-decode code path is algorithmically equivalent) - ForwardBatch_Phase5b_Q8_0_FourSeqs_MatchesPerSeqLoop (abs 0.5 / rel 5% on SmolLM-135M Q8_0, mixed N={1,3,5,8}) - ForwardBatch_Phase5b_DecodeStep_FourSeqs_MatchesForward (abs 0.5 / rel 5% on SmolLM-135M Q8_0, the continuous-batched decode signature N_i=1 for all) - ForwardBatch_Phase5b_ComplexFallback_AdapterActiveOnOneSeq (byte-identical, 3-seq batch with a zero-factor LoRA adapter on the middle seq — proves the simple/complex partition routes the adapter-active seq through the per-seq fallback while the other two go through the batched matmul-fused path) All 9 ForwardBatch tests pass (4 Phase 5a + 5 Phase 5b). 114/114 Models.Architectures tests, 791/791 broader Models+Engine+Lora suite, no regressions. Strix Halo perf (informal Stopwatch, SmolLM-135M Q8_0, 4× N=1 decode, 10 iter, no warm-up correction): - Per-seq loop : 95.94 ms/iter - ForwardBatch : 46.00 ms/iter - Speedup : 2.09× Win materialises mainly from the four GEMMs going from 4× [1, hidden]×W GEMVs to a single [4, hidden]×W GEMM per projection (better cache reuse + lower dispatch overhead). Out of scope (deferred per the plan): - LoRA fusion in batched path (Phase 5c) - MLA / MoE in batched path (Phase 5d) - Vulkan intra-block matmul fusion (Phase 5f, separate worktree) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…tensorsTensorSource (#313) Adds the `ISafetensorsTensorSource` interface (single-file + future multi-shard lookup surface) and ports every safetensors weight loader on the Phase 5b stack to consume it instead of the concrete `SafetensorsFile`: - `TransformerWeightsSafetensorsLoader.Load` - `TransformerWeightsSafetensorsLoader.LoadLayer` (dense Llama/Mistral/Qwen + GQA) - `TransformerWeightsSafetensorsLoader.LoadDeepSeekMlaLayer` (DeepSeek-V2/V3 MLA attention) - `TransformerWeightsSafetensorsLoader.LoadQwenMoeLayer` (Qwen-MoE, incl. shared experts + DeepSeek plural) - `TransformerWeightsSafetensorsLoader.LoadMixtralMoeLayer` (Mixtral block-sparse MoE) - private helpers: `ResolveLinear`, `ResolveLinearAsF32`, `ResolveDense2D`, `ResolveNorm`, `ResolveOptionalNorm`, `ResolveOptionalBias` - `TransformerModel.LoadFromSafetensors` (both overloads) `SafetensorsFile` gains `ISafetensorsTensorSource` on its implements list — no behavior change, every required member (`Tensors`, `TensorsByName`, `GetTensorPointer`, `GetTensorSpan`) is already present and unchanged. The four inline pointer-arithmetic helpers (`ResolveLinear`, `ResolveNorm`, `ResolveOptionalBias`, `ResolveDense2D`) previously computed `file.DataBasePointer + (nint)desc.DataBeginOffset` directly (DataBasePointer is not on the interface). They now go through `ISafetensorsTensorSource.GetTensorPointer(name)`, which for `SafetensorsFile` is *byte-identical*: the implementation is literally `return DataBasePointer + (nint)desc.DataBeginOffset;`. This is a pure type-substitution refactor — no behavior change. Existing tensor-loading semantics are preserved exactly. Verification: - `dotnet build src/DotLLM.Models` clean, no warnings - `dotnet build tests/DotLLM.Tests.Unit` clean - `dotnet build tests/DotLLM.Tests.Integration` clean - 73 unit tests matching Safetensors/Mla/Moe/Mixtral/QwenMoe/GraniteMoe filters pass byte-identical (includes synthetic-fixture forward parity for Llama, Mixtral, Qwen-MoE, Qwen-MoE+shared-expert, and DeepSeek-style plural-shared-experts MoE) - The pre-existing `TinyDeepseekMlaSafetensorsLoadTests` failure is reproduced unchanged on the base branch (missing `mlp.gate_proj` tensor on layer 1 — fixture vs MoE-vs-dense mismatch unrelated to this refactor) - Vulkan LoRA parity test failures are environment-specific (VK_ERROR_MEMORY_MAP_FAILED) and reproduce identically on base This unblocks #208 (its `Load(ISafetensorsTensorSource, ...)` signature now matches), and by extension #274 (Gemma 3 forward wiring), #246 (SmolLM3 ToolCallParsers), and ROADMAP Phase 8 Step 57 (Gemma 4). Out of scope (deliberately, to keep the PR a pure interface substitution): #182's `MultiShardSafetensorsFile`/`SafetensorsIndex`, #193's `SafetensorsTensorResolver`/`AttentionTensorLoader` extraction, #208's SmolLM3+Gemma3 detection + config fields, and #274's Gemma 3 forward wiring. Closes #313 Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds Architecture.SmolLM3 (Llama-shaped GQA-4 with NoPE layers + optional YaRN), HF config.json detection via architectures[0]=SmolLM3ForCausalLM / model_type=smollm3, and per-layer RoPE gating inside TransformerModel. - HfConfigExtractor.ResolveArchitecture: SmolLM3 dispatch - HfConfigExtractor: parse no_rope_layers (HF 1=apply, 0=skip - inverted to a list of layer indices that SKIP RoPE so the model surface reads naturally) + factored ExtractDenseRopeScaling that surfaces rope_scaling type/factor/original_max_position_embeddings/beta_fast/beta_slow/ attention_factor into RoPEConfig for non-MLA architectures - ModelConfig.NoRopeLayers + IsNoRopeLayer(idx) helper - TransformerModel.Forward: skip RoPE.Execute when IsNoRopeLayer(layer) - ModelLoader: SmolLM3 routes through TransformerModel.LoadFromSafetensors The dense-path YaRN dispatch in TransformerModel.BuildFromPrebuiltWeightsInternal is deferred — it depends on RoPE.PrecomputeFrequencyTableYarn, which ships via the MLA chain (#187 MLA-3, not yet on upstream main / this PR's ancestry). It will re-land alongside the dense-YaRN dispatch as a follow-up once MLA-3 merges. Refs #208
HfConfigExtractor (3 tests):
- SmolLM3_3B real-world config parses with vocab=128256, layers=36,
GQA-4, NeoX RoPE, and the canonical NoPE index set
{3, 7, 11, 15, 19, 23, 27, 31, 35} extracted from no_rope_layers
(HF 1=apply, 0=skip; we store the SKIP indices)
- SmolLM3 without no_rope_layers leaves the field null (zero-cost
gate when feature absent)
- SmolLM3 with rope_scaling.rope_type=yarn populates ScalingFactor,
OrigMaxSeqLen, BetaFast/Slow defaults
SmolLM3SafetensorsLoadTests (4 tests, 1 gated):
- Forward_ProducesFiniteVocabLogits_WithNonzeroStddev — 4-layer
synthetic fixture, NoPE on layers {1, 3}, 3-token prefill
- AllNoPe_LogitsBitIdentical_AcrossPositions — when every layer
is NoPE, varying positions [0,1,2] vs [10,20,30] must produce
byte-identical logits (proves the per-layer RoPE skip is real)
- NoNoPe_DifferentPositions_LogitsDiverge — inverse: when NoPE
disabled, varying positions must change logits (RoPE call is
reachable)
- RealWeights_Loads_And_ForwardsFiniteLogits_WhenAvailable —
gated: returns early when ~/.dotllm/test-cache/HuggingFaceTB/
SmolLM3-3B/config.json is absent; full forward + finite-logit
check when present
YarnLongContext_PositionBeyondOrigMax_YarnAffectsLogits deferred —
exercises the dense-path YaRN dispatch that ships via the MLA chain
(#187 MLA-3). Re-lands as a follow-up alongside the kernel.
Refs #208
Adds Gemma 3 dispatch to HfConfigExtractor. Covers text-only (model_type=gemma3_text / architectures[0]=Gemma3TextForCausalLM) and multimodal (model_type=gemma3 / architectures[0]=Gemma3ForConditionalGeneration) checkpoint shapes; multimodal hoists the text_config sub-object so every field lookup reads the text-tower shape. ModelConfig: 4 new optional Gemma knobs - PerLayerSlidingWindow (IReadOnlyList<int?>?) — sliding_window_pattern - AttnLogitSoftcap (float?) — Gemma 2 attn_logit_softcapping - FinalLogitSoftcap (float?) — Gemma 2 final_logit_softcapping - QueryPreAttnScalar (float?) — attention score scale override Architecture.Gemma3 enum variant — text-only + multimodal coverage. HfConfigExtractor: gemma3 / gemma3_text dispatch, text_config hoist, sliding_window_pattern formula, layer_types override, GELUTanh activation mapping, DefaultTieForArch + GetFloatNullableIfPositive helper. Tests: 3 new HfConfigExtractor tests - Gemma3_TextOnly_PopulatesGemmaFields - Gemma3_Multimodal_HoistsTextConfig - Gemma3_LayerTypesArray_OverridesSlidingWindowPattern Refs #208 Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
4 tasks
jamesburton
force-pushed
the
issue/319-gemma4-architecture
branch
from
June 8, 2026 23:40
3e3e5e8 to
f116104
Compare
The SafeTensors HfConfigExtractor dispatched Architecture.SmolLM3 to RoPEType.NeoX. llama.cpp's llama_model_rope_type maps LLM_ARCH_SMOLLM3 to LLAMA_ROPE_TYPE_NORM alongside LLM_ARCH_LLAMA (src/llama-model.cpp, NORM block), and SmolLM3 is otherwise Llama-shaped and routes through the same TransformerModel.LoadFromSafetensors path as Llama. Remove SmolLM3 from the NeoX switch arm so it falls through to the Llama default (Norm). GgufModelConfigExtractor already produces Norm for SmolLM3 via its catch-all — only the HF extractor needed the correction. No other SmolLM3-vs-NeoX references on the branch (`git grep` clean). Test updates: - HfConfigExtractorTests.SmolLM3_3B_DetectsArchAndParsesNoPeLayers: expectation flipped to RoPEType.Norm; XML-doc + inline comment cite the llama.cpp mapping. - SmolLM3SafetensorsLoadTests.BuildConfig: synthetic RoPEConfig uses Norm so the forward-test fixture matches what the extractor now produces. Forward-pass assertions (finiteness, non-zero stddev, position-sensitivity, NoPE bit-equal invariant) are all RoPE-pairing invariant on random synthetic weights so they remain green. Latent bug discovered during the Gemma 4 (#319 / PR #320) RoPE audit cross-check against llama.cpp; PR #320 documented it as a follow-up. Fixing in this PR alongside the original SmolLM3 dispatch keeps the correction co-located with the introduction. Refs #208 Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…infrastructure (#149) - AlibiPositionEncoding: implements IPositionEncoding with Press et al. per-head slope formula (BLOOM/MPT/Falcon variant). Slopes computed in PrecomputeTables; Apply is no-op since ALiBi modifies attention scores rather than rotating Q/K. Handles non-power-of-two head counts via the interleaved-expanded-slope schedule. - Cpu.Kernels.Attention: new overloads of Execute / ExecuteCore / ExecuteTiledCore that accept ReadOnlySpan<float> alibiSlopes and add a per-(query, key, head) bias before the causal mask. The default empty span preserves the existing no-ALiBi happy path. - ValidateAlibiSlopes guards that slopes.Length >= numHeads when slopes are provided; GetAlibiSlope returns 0 for empty spans (no bias). - Tests: - AlibiPositionEncodingTests: slope formula correctness, non-power-of-two head counts, lifecycle (PrecomputeTables + InvalidateCache), apply guards. - AttentionTests: ALiBi bias additivity, slope-zero invariance, multi-head independence, large-context behaviour. Wire-up to GgufModelConfigExtractor is deferred until an ALiBi-using arch (BLOOM/MPT/Falcon) gains a loader arm. See issue #149 comment for scoping. Refs #149 Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…capping) (#255) Adds an optional `softCap` parameter (default 0 = disabled) to every `Attention.Execute` overload - span, pointer, parallel, and scalar reference paths. When > 0 the raw scores pass through `softCap * tanh(s / softCap)` BEFORE softmax (post scale + ALiBi, pre causal/sliding mask), exactly mirroring the Vulkan FA shader convention in `attention_flash_f32.comp`. Mechanism is required for Gemma 2 (`attn_logit_softcapping=50.0`); Gemma 3 leaves the field null but the plumbing is wired regardless. The new helper `ApplySoftCap` uses TensorPrimitives multiply/tanh/ multiply for the SIMD-accelerated kernel; the scalar reference uses MathF.Tanh per element. The quantized KV-cache attention path takes the parameter for API consistency but currently asserts the value is unused - Gemma checkpoints ship F32 KV, so the path is a future follow-up. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… QPAS (#274) Wires four Gemma-family `ModelConfig` fields through the dense CPU forward path (both `RunLayersAndFinalNormCore` and the batched `RunLayersAndFinalNormBatched`): 1. `PerLayerSlidingWindow[layer]` — per-layer override for the sliding-window mask. Gemma 3's `sliding_window_pattern` produces an interleaved local/global pattern (e.g. layers 0,2 sliding and 1,3 full). New `GetLayerSlidingWindow(int)` helper resolves per-layer-override -> null entry -> fallback to model-wide `SlidingWindowSize`. No-op on every architecture without the per-layer field set. 2. `AttnLogitSoftcap` — passed as the new `softCap` argument to `Attention.Execute` so the post-scale soft-cap fires in the CPU kernel exactly as in the Vulkan FA shader. 3. `FinalLogitSoftcap` — applied in-place via the new `ApplyFinalLogitSoftcap` helper after `RunLmHead`'s GEMM and before the result tensor copy; mirrored in the batched lm_head path inside `ForwardBatch`. Uses `TensorPrimitives.Tanh` for the SIMD-accelerated kernel. 4. `QueryPreAttnScalar` — overrides the default `1/sqrt(headDim)` attention scale with `1/sqrt(QueryPreAttnScalar)` when non-null. Routed via the existing `scale`-providing `Attention.Execute` overloads. All four mechanisms are no-ops on every existing architecture (default field values are null), so this is additive plumbing for Gemma 3 inference end-to-end coverage in the next commit.
Adds `TransformerModelGemma3ForwardTests` — a tiny synthetic
safetensors fixture (HiddenSize=16, NumLayers=4, NumHeads=2,
VocabSize=8, IntermediateSize=24, `sliding_window_pattern=2` so
layers 0,2 are sliding and 1,3 are full) plus four discriminative
forward-pass tests covering every Gemma-family mechanism wired in
the previous commit:
- `Forward_Gemma3_AllMechanisms_FiniteLogits` — all four mechanisms
active simultaneously; standard Gemma 2/3 acceptance (finite
logits, non-zero stddev, magnitude clamped by FinalLogitSoftcap).
- `Forward_Gemma3_FinalSoftcap_BoundsLogitMagnitude` — compares
the same forward with/without FinalLogitSoftcap, asserts the
capped run saturates inside (-cap, +cap) and the uncapped run
exceeds that band (discriminative against a no-op cap).
- `Forward_Gemma3_PerLayerSlidingWindow_DiffersFromUniform` —
uniform-full baseline vs interleaved per-layer pattern on the
same weights; asserts measurable logit divergence.
- `Forward_Gemma3_QueryPreAttnScalar_ChangesAttentionTemperature` —
default 1/sqrt(headDim) vs override; asserts measurable
divergence.
Follows the `WriteFixture` / `BuildConfig` pattern from
`TransformerModelMlaForwardTests`. Uses HF-convention tensor naming
(`model.embed_tokens`, `model.layers.N.self_attn.{q,k,v,o}_proj`,
`model.layers.N.mlp.{gate,up,down}_proj`, `model.norm`, `lm_head`).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…ard wiring (#319) Implements ROADMAP Phase 8 Step 57. Google released Gemma 4 on 2026-04-02 under Apache 2.0 (E2B/E4B mobile, 12B dense, 26B MoE, 31B dense); the HF Transformers `gemma4/` model directory carries the reference Gemma4Config / Gemma4ForCausalLM / Gemma4ForConditionalGeneration / Gemma4UnifiedForConditionalGeneration classes. This PR ships text-only inference wiring for the dense 12B / 31B SKUs. Architecture.Gemma4 enum variant. HfConfigExtractor dispatches on `model_type` ∈ {gemma4, gemma4_text, gemma4_unified, gemma4_unified_text} and architectures[0] containing "gemma4". Multimodal + unified checkpoints house the text-tower config under `text_config`; the existing Gemma 3 hoist path is generalised to fire for any gemma-family architecture. All wired ModelConfig fields reused from the Gemma 2/3 plumbing already shipped through PRs #208/#274/#255: - PerLayerSlidingWindow — driven by Gemma 4's `layer_types` array (canonical 5×sliding + 1×full, repeating) - AttnLogitSoftcap / FinalLogitSoftcap — public 12B/31B set only final_logit_softcapping=30.0; attn cap stays null - QueryPreAttnScalar — null on the public dense SKUs - GELUTanh activation (Gemma 4 keeps `hidden_activation=gelu_pytorch_tanh`) - RMSNorm pre-norm + the standard (1+w) Gemma convention - NeoX RoPE (matches `rotate_half` in modeling_gemma4.py) - Tied embeddings default true ModelLoader.LoadFromSafetensors dispatches Architecture.Gemma4 (alongside the previously-missing Gemma3) through the dense TransformerModel.LoadFromSafetensors path. Tests: - 5 Gemma4HfConfigExtractorTests — text-only Gemma4ForCausalLM, multimodal Gemma4ForConditionalGeneration with text_config hoist, unified Gemma4UnifiedForConditionalGeneration, omitted-tie default, generic-Gemma priority guard - 2 TransformerModelGemma4ForwardTests — all-mechanisms forward (finite logits + final-softcap clamp), public 12B defaults (finite logits) Out of scope (deferred): - `rope_parameters.{full,sliding}_attention` split rope_theta (top-level rope_theta is still present as a fallback) - `global_head_dim` / `num_global_key_value_heads` — null on public 12B/31B; full-attention layers reuse standard head shape - 26B-A4B MoE variant (`enable_moe_block` + `num_experts`) — follow-up via existing MoE path - multimodal vision / audio towers (text-only forward only) ROADMAP.md Step 57 ticked; README.md Phase 8 counter bumped to 2/5. Closes #319. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
jamesburton
force-pushed
the
issue/319-gemma4-architecture
branch
from
June 9, 2026 08:17
f116104 to
d45174f
Compare
This was referenced Jun 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements ROADMAP Phase 8 Step 57 — the last unticked numbered ROADMAP step. Google released Gemma 4 on 2026-04-02 under Apache 2.0 (E2B / E4B mobile, 12B dense, 26B-A4B MoE, 31B dense); the HF Transformers
gemma4/model directory carries the referenceGemma4Config/Gemma4ForCausalLM/Gemma4ForConditionalGeneration/Gemma4UnifiedForConditionalGenerationclasses.Track X — Gemma 4 is publicly released. Built strictly to match the HF reference for the dense text-only 12B / 31B SKUs.
Closes #319, ticks ROADMAP Step 57.
What's new
Architecture.Gemma4enum valueHfConfigExtractordispatch onmodel_type∈ {gemma4,gemma4_text,gemma4_unified,gemma4_unified_text} andarchitectures[0]matchinggemma4*(including the unified multimodal release)ModelLoader.LoadFromSafetensorsdispatchesArchitecture.Gemma4(and the previously-missingArchitecture.Gemma3) throughTransformerModel.LoadFromSafetensorsGemma3andGemma4both map toRoPEType.NeoX(cross-verified against llama.cppllama_model_rope_typeinsrc/llama-model.cpp—LLM_ARCH_GEMMA,LLM_ARCH_GEMMA2,LLM_ARCH_GEMMA3,LLM_ARCH_GEMMA3N,LLM_ARCH_GEMMA4,LLM_ARCH_GEMMA4_ASSISTANT,LLM_ARCH_GEMMA_EMBEDDINGall returnLLAMA_ROPE_TYPE_NEOX). Gemma 3 was historically mapped to Norm via the catch-all; corrected here so the family is consistent. The Gemma 3 forward tests are insensitive to the pairing convention on synthetic random weights (assert only finiteness + non-zero variance + soft-cap bounds — all RoPE-pairing-invariant).What's reused
Every Gemma 2/3 mechanism already shipped through the earlier PRs in this stack:
gelu_pytorch_tanh)Attention.Execute(softCap))ApplyFinalLogitSoftcap)PerLayerSlidingWindow)QueryPreAttnScalar)Architecture.Gemma3+ Gemma-family field blockStack
Stack is DRAFT because PR #318, #317, and #314 are all unmerged held branches.
Out of scope (deferred)
Each item is null / disabled by default in the public dense 12B / 31B SKUs, so the dense text-only forward pass runs correctly without them:
rope_parameters.{full,sliding}_attention— split rope_theta per layer-type. The loader reads top-levelrope_thetawhich Gemma 4 still carries as a fallback.global_head_dim/num_global_key_value_heads— alternative head shape on full-attention layers.enable_moe_block+num_experts) — will wire through the existing MoE path (step 58 / 58a) in a follow-up.Sibling SmolLM3 RoPE fix (resolved upstack in PR #317)
(Updated) The latent SmolLM3 RoPE-type bug originally noted as a follow-up has now been fixed directly in PR #317 alongside the Gemma 3 NeoX correction shipped in this PR. llama.cpp's
llama_model_rope_typemapsLLM_ARCH_SMOLLM3toLLAMA_ROPE_TYPE_NORM(alongsideLLM_ARCH_LLAMA) — the SafeTensors extractor was previously dispatching SmolLM3 to NeoX. PR #317 removes SmolLM3 from the NeoX switch arm so it falls through to the Llama default. The two corrections together align dotLLM's RoPE conventions withllama.cpp'sllama_model_rope_typefor all Gemma + SmolLM3 family architectures. This PR was rebased onto the updated PR #317 (mechanical rebase; the one conflict — both PRs touching the RoPE-type switch — was resolved to drop SmolLM3 from the NeoX arm while keeping the Gemma3 / Gemma4 additions).Test plan
dotnet build src/DotLLM.Core— cleandotnet build src/DotLLM.Models— cleandotnet build tests/DotLLM.Tests.Unit— cleandotnet test --filter "FullyQualifiedName~Gemma|FullyQualifiedName~HfConfigExtractor|FullyQualifiedName~SmolLM3"— 39 pass, 0 fail (post-rebase onto updated models(safetensors): SmolLM3 + Gemma 3 architecture detection (rebased onto #314) #317)Tests added
Gemma4HfConfigExtractorTests:Gemma4_TextOnly_PopulatesGemmaFields— verifies canonical 5×sliding + 1×full layer_types pattern, final_logit_softcapping=30.0, NeoX RoPE, GELUTanh, tied embeddingsGemma4_Multimodal_HoistsTextConfig—Gemma4ForConditionalGenerationtext_config sub-object hoistGemma4_UnifiedMultimodal_DetectsArchAndHoistsTextConfig—Gemma4UnifiedForConditionalGenerationwith both audio + vision towersGemma4_OmittedTieFlag_DefaultsToTied— DefaultTieForArch fallbackGemma4_TakesPriority_OverGenericGemmaPattern— guards against future genericgemmasubstring shadowingTransformerModelGemma4ForwardTests:Forward_Gemma4_AllMechanisms_FiniteLogits— all four Gemma mechanisms active on a synthetic 5×sliding + 1×full fixtureForward_Gemma4_PublicDefaults_FiniteLogits— onlyfinal_logit_softcapping=30.0set (matching the public 12B/31B config)ROADMAP
docs/ROADMAP.mdPhase 8 Step 57 ticked with implementation summary.README.mdPhase 8 step counter bumped fromIn Progress (1/5)→In Progress (2/5).