Nightly 2026-07-22/23: template byte gate (prompts byte-identical to transformers), dtype-pollution fix, config-driven EOS/BOS salvage, dep refresh

Cargo.toml

@@ -70,7 +70,14 @@ ureq = "3"
 sha2 = "0.11"
 
 serde = { version = "1", features = ["derive"] }
-serde_json = "1"
+# `preserve_order` (insertion-order maps) is deliberate and load-bearing for
+# prompt bytes: Python/transformers renders tool JSON in dict insertion order,
+# so our tool-bearing prompts only byte-match the reference stack's rendering
+# with it on. It is also what the hf-chat-template dev-dependency (the
+# template byte gate) requires — without it here, test builds would unify the
+# feature on while production consumer builds leave it off, and the two would
+# render DIFFERENT prompt bytes (found 2026-07-23 when the gate landed).
+serde_json = { version = "1", features = ["preserve_order"] }
 thiserror = "2"
 once_cell = "1"
 

README.md

@@ -48,7 +48,9 @@ It exists because two local-first apps — **[laurelane](https://github.com/phys
   family's renderer, or — when a `tokenizer.json` sits beside it — whose declared added-token ids don't
   match that real tokenizer, is a loud `ImportError::Inconsistent` instead of a model that silently
   mis-stops, mis-templates, or mis-tokenizes. Both sibling files are optional (a GGUF-derived dir has
-  neither → no behavior change).
+  neither → no behavior change). On a successful safetensors load the parsed `TokenizerConfig` is surfaced
+  on the returned `Loaded{Qwen2,Qwen3,Lfm2}` struct (`tokenizer_config`), so a consumer reads config-driven
+  EOS/BOS/PAD ids straight off the model; a GGUF load surfaces `None` (self-contained).
 - **Import validation** — a two-stage error taxonomy: `ImportError` for the file→module stage (missing
   file, parse, load, and an `Incomplete` per-tensor missing/errored diff) and `SanityError` for the
   runtime liveness a checked load can't see — NaN/Inf logits, a vocab-width mismatch, or a
@@ -86,6 +88,12 @@ It exists because two local-first apps — **[laurelane](https://github.com/phys
   emitted a parseable Hermes call and LFM2.5-1.2B emitted exactly
   `<|tool_call_start|>[get_weather(city="Paris")]<|tool_call_end|>` (`tests/real_toolcall.rs`,
   `tests/real_toolcall_lfm.rs`).
+- **Template byte gate** — the hardcoded renderers are proven **byte-identical to
+  `transformers.apply_chat_template`** rendering the checkpoint's own imported `chat_template`
+  (via the `hf-chat-template` dev-dependency): plain, multi-turn, the full Hermes `# Tools` block,
+  and function-call history all match byte-for-byte on Qwen3-0.6B (`tests/template_gate.rs`).
+  Prompt JSON deliberately serializes with Python `json.dumps` spacing and insertion-order keys
+  (serde_json `preserve_order`) — the exact bytes the reference stack renders and models emit back.
 - **f16 inference, validated** — Qwen2.5-1.5B runs coherently on `GpuF16` (weights + KV in f16, the
   q·kᵀ attention scores + softmax computed in an f32 island to stop f16 overflow): **~3.6 GiB runner
   VRAM vs ~7.9 GiB f32, at identical speed**; the parity gate re-passes unchanged on f32, where the