feat(eval): shared perplexity harness with llama.cpp-comparable sliding-window mode

[jamesburton]

Tracked by jamesburton#231 (fork issue tracker) — deliberately not using a Closes keyword, since #231 here is an unrelated issue in this repo. Addresses the blocker recorded in #416: "Every remaining lever changes numerics and none of them can currently be evaluated."

Summary

A shared perplexity harness with a llama.cpp-comparable sliding-window mode, so numerics-changing kernel work has an evaluation gate. Self-contained: abstractions in DotLLM.Core.Evaluation, evaluator in DotLLM.Engine, a TransformerModel adapter, and a dotllm perplexity CLI verb.

What "llama.cpp-comparable" actually means

This turned out to be the substance of the work, so it is pinned down explicitly rather than asserted.

llama.cpp walks the corpus in non-overlapping chunks of n_ctx and scores only the second half of each — the first half is context that is never scored, so the scored ranges have gaps. Generalised to (ContextLength L, Stride S, UnscoredPrefix P): window w covers [w*S, w*S + L) and scores absolute targets [w*S + P, w*S + L). PerplexityOptions.LlamaCppDefault(L) gives S = L, P = L/2.

Advance and scored span are independent, and a single "stride" cannot express this. An earlier revision collapsed them, which produced the same scored-token count over a different token set — a figure that looks comparable and is not. It read 24.88 where llama.cpp read 24.01. Positions also restart at 0 per window, because each chunk is an independent sequence; absolute positions additionally throw on any corpus longer than the model's context.

Design notes

One axis, not two implementations. The ten existing per-test perplexity helpers on our fork diverged for exactly one reason: whether a backend's Forward returns all rows or only the last. CPU returns [seqLen, vocab], so one pass scores every target (O(n)); CUDA returns the final row only, so each target needs a growing-prefix re-prefill (O(n²)). Modelling that as IPerplexityModel.ReturnsAllRows lets one evaluator serve both, and a test asserts the two paths produce an identical number on the same tokens.

The evaluator never loads weights, and the corpus is streamed and tokenized in chunks. On unified-memory parts a large VRAM carve-out leaves host RAM scarce, and perplexity — a long run of full-context prefills rather than a single load — is the workload most punished by holding a second host-side copy.

TeacherForced is preserved verbatim as the existing in-tree methodology. It is ratio-oriented (the signal is an OFF/ON ratio under a <1% gate), not comparable to published figures, and is documented as such.

Verification

Geometry is exact against llama-perplexity build 8683 (d0a6dfeb2) in every configuration tested — window and scored-token counts match: 4/512, 15/1920, 18/2304.

Absolute figures, SmolLM-135M Q8_0 on wikitext-2 at -c 256, with byte-exact tokenization (requires the fix in #417):

slice	this harness	llama.cpp	Δ
4 chunks	25.0548	25.8113	−2.93%
15 chunks	23.7864	24.0137	−0.95%
18 chunks	25.0649	25.9366	−3.36%

Unit tests: 18 covering both modes, both backend shapes, window tiling, per-window position reset, row/target mapping, and a test asserting the llama.cpp-default and contiguous-tiling schemes score different token sets — they produce the same count, so a count check cannot distinguish them.

Known limitation, stated plainly

The residual above is not closed. It is consistently negative across slices (this harness scores slightly better than llama.cpp), which is the signature of a systematic forward-pass difference rather than a harness defect: geometry is exact, and the token streams are now byte-identical.

Ruled out by measurement, not argument: window geometry; token-stream content (0 mismatches over 3904 tokens after #417); the --max-tokens cap; and per-chunk BOS substitution. The remaining untested hypothesis is batching — llama.cpp evaluates chunks with n_seq=8, batch_size=2048, while this harness runs one window per forward.

So: trustworthy as a relative gate today (identical geometry, identical tokens, deterministic), and not yet something to quote as an absolute figure against published numbers. I would rather ship that distinction explicitly than imply a comparability I have not demonstrated.

Diagnostics included

--per-window, --dump-tokens and --tokens-file separate tokenization from scoring from geometry. They are not decoration — feeding llama.cpp's exact ids back through the scorer is what localized the original discrepancy, and diffing token streams is what found #417.

Follow-up

The ten duplicated per-test helpers live only on our fork's dev, so migrating them onto this harness — gated on producing numerically identical results — is a separate change and not part of this PR.

[Copilot]

Pull request overview

Adds a shared perplexity evaluation harness to dotLLM, including a llama.cpp-comparable sliding-window mode, so numerics-changing kernel work can be gated by a repeatable evaluator rather than ad-hoc per-test helpers.

Changes:

• Introduces DotLLM.Core.Evaluation contracts (IPerplexityModel, PerplexityOptions/Result, modes) and an engine-side PerplexityEvaluator with both teacher-forced and sliding-window scoring.
• Adds streaming corpus tokenization (CorpusReader) and a TransformerModel adapter (TransformerPerplexityModel) so evaluation can run without re-loading weights.
• Exposes the harness via a new dotllm perplexity CLI verb and adds unit tests covering evaluator geometry and stable log-prob scoring.

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tests/DotLLM.Tests.Unit/Evaluation/PerplexityEvaluatorTests.cs	Unit coverage for teacher-forced vs sliding-window behavior, window tiling, and row/target mapping.
tests/DotLLM.Tests.Unit/Evaluation/LogProbTests.cs	Verifies numerical stability and invariants of log-softmax target log-probability.
tests/DotLLM.Tests.Unit/Evaluation/FakePerplexityModel.cs	Deterministic IPerplexityModel test double recording forward calls/windows.
tests/DotLLM.Tests.Unit/Evaluation/CorpusReaderTests.cs	Validates streamed tokenization ordering, max-tokens cap, and chunk-boundary handling.
src/DotLLM.Models/Evaluation/TransformerPerplexityModel.cs	Adapter from TransformerModel to IPerplexityModel for evaluation.
src/DotLLM.Engine/Evaluation/PerplexityEvaluator.cs	Core evaluator implementing teacher-forced and sliding-window perplexity computation.
src/DotLLM.Engine/Evaluation/LogProb.cs	Numerically stable log-softmax (log P(target)) helper used by the evaluator.
src/DotLLM.Engine/Evaluation/CorpusReader.cs	Streaming corpus reader that tokenizes in chunks without loading whole corpus text.
src/DotLLM.Core/Evaluation/PerplexityResult.cs	Defines modes/options/result data structures and llama.cpp-default preset.
src/DotLLM.Core/Evaluation/IPerplexityModel.cs	Defines the minimal model surface required for perplexity scoring.
src/DotLLM.Cli/Program.cs	Registers the new perplexity CLI command.
src/DotLLM.Cli/Commands/PerplexityCommand.cs	Implements dotllm perplexity: load → stream/tokenize or read ids → score → report.
docs/superpowers/specs/2026-07-30-perplexity-harness-design.md	Design spec documenting geometry/constraints and rationale.
docs/superpowers/plans/2026-07-30-perplexity-harness.md	Detailed implementation plan and acceptance/verification steps.

Comments suppressed due to low confidence (1)

src/DotLLM.Cli/Commands/PerplexityCommand.cs:95

• The command currently requires --corpus even when --tokens-file is provided, which makes the documented "read pre-tokenized ids instead of tokenizing --corpus" path unusable. Validation should require either --corpus or --tokens-file, and check existence of the chosen input.

        if (string.IsNullOrWhiteSpace(settings.Corpus))
        {
            AnsiConsole.MarkupLine("[red]--corpus is required.[/]");
            return 1;
        }

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[jamesburton]

Validated against llama.cpp on wikitext-2 — and three defects fixed in the process

The harness now reproduces llama.cpp to well within its own error bar. Getting there
turned up three real defects, all in the part that claims comparability.

Result

SmolLM-135M Q8_0, wikitext-2 test, n_ctx=512, 589 chunks, 150,195 scored tokens on both sides:

	perplexity	vs llama.cpp
llama.cpp b8683 (d0a6dfeb2)	18.8440 ± 0.14515	—
ours, fed llama.cpp's exact token ids	18.8911 ± 0.14543	+0.25% (0.23σ)
ours, end-to-end (own tokenizer)	18.8950	+0.27%

The error bars agree to 0.2% — an independent check of the new statistic against
llama.cpp's own on identical data. Tokenizer agreement over the corpus is exact:
0 mismatches in 301,948 ids.

What was wrong

1. LlamaCppDefault scored one target too many. llama.cpp sets first = n_ctx/2,
then accumulates count += n_ctx - first - 1 (perplexity.cpp:629),
scoring targets (n_ctx/2, n_ctx) — the token at n_ctx/2 is context, never a target.
We scored it as well, giving n_ctx/2 targets per chunk against llama.cpp's n_ctx/2 - 1:
the same name over a different token set, which is precisely what this mode exists to prevent.

2. No error bar. This one matters more than it looks. Before the fix I measured a
consistent ~3% gap on a 20k-char corpus and spent a long time hunting a forward-pass bug.
There was none — on that corpus llama.cpp's own figure is 25.9366 ± 1.67994, i.e. ±6.5%,
so a 3% difference was never evidence of anything. A harness that reports a bare number
invites exactly that mistake. It now prints PPL +/- err in llama.cpp's format.

Computed by Welford rather than the textbook E[x²] - E[x]²: the per-token NLL of a
converged model clusters tightly about a few nats, so those two terms agree to most of
their significant digits and their difference is largely rounding. On a constant NLL the
closed form returns ~1e-6 where the true variance is 0. Welford is exact there and agrees
with llama.cpp everywhere else.

3. --tokens-file still demanded --corpus, so the one flag that separates a tokenizer
difference from a scoring difference could not be used at all. It now replaces --corpus
and accepts the JSON-array form reference tools print. This flag did the load-bearing work
above: feeding llama.cpp's exact ids is what proved the +0.25% is not tokenization.

What the harness does and does not guarantee

• Relative comparisons are sound. Identical geometry, identical tokens, deterministic
  (three identical runs → bit-identical per-window output). This is what gates a kernel change.
• Absolute figures are comparable to published llama.cpp numbers in sliding-window mode,
  to within the error bar, provided model, corpus, context and unscored prefix all match.
• The residual +0.25% is sub-σ and not separately attributable; it is consistent with ordinary
  numerical differences (accumulation order, SIMD kernels, activation-quantization detail).

Note on ordering

End-to-end agreement depends on #417 — without that pre-tokenizer fix the same corpus
gives 44 token mismatches against llama.cpp. The scoring path itself is independent of it,
as the --tokens-file row shows.

Reproducing

llama-perplexity -m SmolLM-135M.Q8_0.gguf -f wiki.test.raw -c 512 -ngl 0
dotllm perplexity SmolLM-135M.Q8_0.gguf -f wiki.test.raw -c 512

[jamesburton]

All six addressed in c00068a — five were documentation that had drifted from the implementation,
and one was behavioural.

CorpusReader whitespace (behavioural, fixed). Good catch, and it was worse than the docstring
mismatch it looked like. Cutting at the last whitespace character can land inside a run of them,
and a GPT-2-style pre-tokenizer treats a whitespace run as one unit — so splitting one yields a
different stream than tokenizing the file in one pass. Cuts now land at the run's start, so the
whole run moves into the carry.

I checked whether it actually bit before changing anything. It did not, but my prior evidence was
thinner than I had realised: the wikitext-2 validation I reported crosses only ~20 chunk boundaries
(1.29 MB at 64 KiB). Re-run against whole-file tokenization at 997 / 4096 / 8192 / 65536 — the
smallest exercising ~1,290 boundaries — all four produce identical streams, before and after.
So the fix is a robustness one, now covered by tests over runs of spaces, mixed space/tab/newline,
newlines-only and CRLF.

The unbounded-carry half is real and I have documented it rather than papered over it: with no
whitespace there is no safe cut, so the corpus is read whole. Flushing at an arbitrary character
would bound the memory but change the token stream, which is the one thing this harness must never
do silently. Asserted in a test so it is a stated trade.

IPerplexityModel.positions (fixed). You are right and this was actively misleading — the doc
invited implementers to rely on absolute positions, and sliding-window has passed window-relative
ones restarting at 0 since e303be4. Beyond matching llama.cpp's chunk semantics, that is what lets
a corpus longer than MaxContextLength be scored at all; absolute positions overflow the model's
limit on window two. Contract now says so explicitly.

PerplexityMode.TeacherForced (fixed, docs). Correct — it ignores Stride and
UnscoredPrefix. I documented rather than implemented, deliberately: that mode exists to reproduce
the pre-existing CUDA prefill harnesses bit for bit, and giving it window geometry would move the
numbers their quality gates were calibrated against. Now stated as a deliberate constraint with the
reason, and it points at SlidingWindow for walking a corpus.

CLI defaults (fixed, both sites). The remarks said stride = context/2; the default is
stride = context with unscored prefix = context/2 + 1. The --unscored-prefix help text was
also stale — it still said context/2 after ca8a92b corrected the off-by-one against llama.cpp's
count += n_ctx - first - 1. Both now match the code.

Evaluation tests 26/26.

[jamesburton]

Apologies for the close/reopen noise — that was accidental on our side, not a withdrawal.

The branch was deleted while tidying up after porting this work into our fork's integration branch,
and GitHub closes a PR when its head branch goes. Restored to the same commit (c00068a7); nothing
was rewritten and the diff is unchanged.

Still open for review, and no action needed from you.