feat(tokenizers): Jinja loop.previtem / loop.nextitem

[jamesburton]

Summary

• Adds loop.previtem / loop.nextitem to the Jinja evaluator's per-iteration loop context (src/DotLLM.Tokenizers/ChatTemplates/JinjaEvaluator.cs). Both are simply omitted from the loop dict at the first/last iteration respectively, so member access on them falls through the evaluator's existing Undefined convention (EvalMemberAccess already returns Undefined for an absent dict key) instead of a new sentinel or a throw — matches Jinja2 semantics, where previtem/nextitem are falsy-undefined at the boundaries, never an error.
• Also adds the "undefined" test to EvalIsTest (x is undefined / x is not undefined), mirroring the existing IsDefinedExpr path. Required, not optional scope creep: Qwen3.8-27B's real template checks preserve_thinking is undefined in its assistant-message branch, which sits on the path to every tool message — without this the acceptance-criterion test can't execute at all.

Important caveat

The verbatim template excerpt used in the new integration test deliberately drops the template's unconditional reasoning-effort prelude (lines 42-87 of the real file), which uses a parenthesized tuple literal (not in ('xhigh', 'medium', 'low')) that the Jinja parser doesn't support yet — a pre-existing, unrelated gap, filed separately as #409.

The full, unmodified Qwen3.8-27B template still fails to parse until #409 lands — so this PR fixes a necessary condition for tool calling on that model, not a sufficient one. Don't read this as "tool calling now works end-to-end for Qwen3.8-27B."

Tests

• JinjaEvaluatorTests: discriminating previtem/nextitem cases (actual neighbour values — not "always current item" or "always undefined"), explicit first/last boundary checks via is defined/is not defined, and new is undefined/is not undefined cases.
• Qwen38ChatTemplateTests (new, tests/DotLLM.Tests.Integration): renders the real Qwen/Qwen3.8-27B chat_template.jinja tool-response branch verbatim (the render_content macro + the multi-step-tool loop + the per-message render loop + generation-prompt tail — lines 1-41 and 88-170 of the file fetched from HF 2026-08-14, unmodified) and asserts on actual merge behaviour: consecutive role == "tool" messages merge into a single <|im_start|>user ... <|im_end|> turn, and a lone tool message opens+closes in the same loop iteration.

Verified

• dotnet test (Release) on DotLLM.Tests.Unit filtered to JinjaEvaluatorTests/JinjaChatTemplateTests: 114 passed, 0 failed.
• dotnet test (Release) on DotLLM.Tests.Integration filtered to ChatTemplate: 6 passed, 3 skipped (the pre-existing Mach-1 tests skip because that fixture isn't staged locally — unrelated to this change), 0 failed.
• Both test projects build clean in Release (transitively builds every src/ project referenced by them — no separate top-level solution file in this repo).

Not verified

• Did not attempt to fix the tuple-literal parser gap (fix(tokenizers): Jinja parser rejects parenthesized tuple literals (blocks Qwen3.8-27B template parse) #409) — out of scope per the issue's scope-discipline note.
• Did not run the skipped Mach-1 fixture-backed tests (fixture not staged locally); they were already skip-gated before this change and are unaffected by it.

Closes #399

[Copilot]

Pull request overview

Adds missing Jinja loop-context and test semantics needed to render real-world chat templates (notably Qwen3.8’s tool-response branch) within DotLLM.Tokenizers’ Jinja2-subset interpreter.

Changes:

• Extend loop context in for evaluation with loop.previtem / loop.nextitem, omitted at boundaries to reuse the evaluator’s existing Undefined behavior.
• Add support for the is undefined / is not undefined test in EvalIsTest (aligned with existing Undefined sentinel semantics).
• Add unit + integration coverage, including a verbatim excerpt test of Qwen3.8-27B’s tool-response branch behavior.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File	Description
src/DotLLM.Tokenizers/ChatTemplates/JinjaEvaluator.cs	Implements loop.previtem/loop.nextitem via boundary key omission and adds is undefined test handling using the existing Undefined sentinel.
tests/DotLLM.Tests.Unit/Tokenizers/ChatTemplates/JinjaEvaluatorTests.cs	Adds focused unit tests verifying neighbor correctness and boundary undefined/defined behavior, plus is undefined/is not undefined cases.
tests/DotLLM.Tests.Integration/Tokenizers/ChatTemplates/Qwen38ChatTemplateTests.cs	Adds an integration test rendering Qwen3.8’s real tool-response branch excerpt and asserting correct tool-message merge behavior.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[jamesburton]

Update: mutation-testing follow-up

Independent review mutation-tested the loop.previtem/loop.nextitem suite (7 deliberate mutants) against real Jinja2 3.1.6, byte-diffed the excised template excerpt against the live HF file (identical), and confirmed 6/7 mutants died. Added two tests to close the gaps the review found (pushed in 4388dfe6):

• IsUndefined_False_WhenVariableIsNull — kills the surviving mutant (is undefined returning true for a legitimately null value). The shipped code was already correct here, but nothing asserted it; this is the property that keeps a legitimately-null previtem distinguishable from an omitted (undefined) one.
• ForLoop_NestedLoop_DoesNotLeakIntoOuterLoopPrevNextItem — confirms an inner for-loop's own loop variable doesn't clobber the outer loop's previtem/nextitem once the inner loop finishes.

Both additive; 116/116 unit tests pass (Release).

Two other findings from that review are not addressed here — filed/tracked separately, out of this issue's scope:

• Attribute access on Undefined returning Undefined where Jinja2 raises (real silent-failure risk).
• Tuple stringification.

CI note

CI currently shows Integration Tests failing on this branch — not caused by this change. 9/12 failures are Unknown tokenizer.ggml.pre type: 'qwen2' (the #373 gate firing; fixed by gap PR #410), the other 3 are the fixture-skip issue fixed by PR #412. This branch goes green once those land — no rebase needed here beyond a CI rerun.