feat(eval): perplexity layer-select/offload plus a layer-cycling mode (#395)

[jamesburton]

Closes #395

What this adds

dotllm perplexity could only score a model wholly on one device, so a model larger than a
single device's memory could not be measured at all — which is exactly the situation for
Nemotron-3.5-Lightning-30B-A3B (18.9 GB) and Qwen3.8-27B (~17 GB) against a 12 GB card.

Three placements now, in increasing order of what they buy:

Flags	Placement
--gpu-layers N	GPU [0..N), CPU [N..L) — ordinary prefix offload, same dispatch and semantics as run/chat/serve
--gpu-layers N --first-layer K	CPU [0..K), GPU [K..K+N), CPU [K+N..L) — an arbitrary contiguous GPU window
--gpu-layers N --cycle	a GPU window of N layers slid across the whole trunk within one corpus pass

Cycling checkpoints the hidden state at each layer cut and replays the next window from those saved
activations rather than from token embeddings, so every layer is GPU-executed exactly once instead of
paying for ceil(L/N) CPU-bottlenecked passes.

Design

• ILayerWindowModel / ILayerWindowExecutor (DotLLM.Core.Evaluation) — a backend-neutral
  hidden-in/hidden-out layer-window contract. A factory shape, not a selector, because the whole
  point is that only one window's weights are device-resident at a time.
• CyclingPerplexityEvaluator (DotLLM.Engine.Evaluation) — the pass driver and boundary
  checkpoints. Two boundaries alive at a time (read + write), windows x context x hidden x 4 B
  each.
• PerplexityWindowPlan — the corpus-window enumeration, now shared by PerplexityEvaluator and
  the cycling driver. Boundary activations are indexed by window position, so a second copy of that
  loop is the one thing that could silently desynchronise the passes; the replay adapter also
  cross-checks each window's token ids and throws rather than scoring window i's activations
  against window j's targets.
• The final scoring pass is the ordinary PerplexityEvaluator, running against the saved
  boundaries. That is what keeps --per-window, --tokens-file, --bos, --stride,
  --unscored-prefix, the error bar and the whole methodology identical to a whole-device run for
  free.
• CompositeLayerWindowModel — per-window device assignment, so an arbitrary GPU window is a
  three-entry assignment over one contract rather than a fourth hand-written CPU/GPU splitter.
• CpuLayerWindowModel / CudaLayerWindowModel — the backend implementations.
• The result table now always prints a Layer placement row, whole-device runs included, so a
  scraped table cannot report a split or cycled run as a single-device one.

Why the output head runs on the host

CudaPipelineStage.FinishLogits() returns only the last row; sliding-window scoring needs logits
for every row. So device windows are always built isFinalStage: false and the final norm + LM
head run host-side. Every transformer layer still runs on the GPU; the head is one GEMM against a
weight negligible next to a layer window.

Recurrent state — a deliberate reconciliation with the acceptance criterion

The criterion asks for recurrent state to be "checkpointed alongside the hidden state". The
implementation instead makes recurrent state phase-local by construction, and I want that to be
explicit rather than look like an oversight:

Recurrent state is per layer, and each layer belongs to exactly one layer window which replays the
entire corpus in window order. So the state a layer must start corpus window w with is zero — in
a cycled run and in a whole-device run alike. Nothing recurrent needs to cross a layer cut; only the
hidden state does. Serialising a recurrent snapshot into the boundary record would be dead weight
that still would not protect against the failure that actually occurs.

The failure that actually occurs is a missing per-corpus-window reset inside a pass — issue #261's
bug, multiplied, because each pass replays the whole corpus. ILayerWindowExecutor.ResetState() is
therefore called before every corpus window in every pass, and the tests discriminate on exactly
that.

What was verified

CPU layer windowing — bit-identical. Split-then-resume vs whole-model Forward:

• Hybrid (SyntheticQwen35MoeGguf, 4 layers, GDN at 0 and 2 so GDN state slots sit on opposite
  sides of the cut), cut at layer 2: maxAbs = 0.0, maxRel = 0.0.
• Dense TransformerModel (4-layer Llama-shaped fixture), cut at layer 1: maxAbs = 0.0,
  maxRel = 0.0.

LayerCyclingPerplexityTests (new, 8 facts, all passing). On the recurrent 4-layer fixture:

• cycled == whole-model mean NLL to 12 decimal places, with a 2-layer window and with a 1-layer
  window;
• the discriminating fact: dropping the per-corpus-window reset in a pass changes the number
  (CycledScoring_WithoutPerWindowReset_ProducesADifferentNumber). Without this assertion the two
  equality facts would pass just as happily against a state-leaking implementation;
• the degeneracy is pinned: over a single corpus window the leaked and reset forms coincide, so
  a one-window corpus cannot discriminate them. Recorded as a test so a later edit cannot shorten the
  corpus and silently disarm the discriminator;
• --per-window diagnostics fire in cycling mode with the same per-window figures a whole-model run
  reports;
• partitions that do not cover the trunk are rejected rather than scored.

CUDA layer windows — measured on a real RTX 3060 12 GB (Llama-3.2-1B-Q8_0, L=16):

• cycled [0..8)+[8..16) boundary hidden vs a single [0..16) window: maxAbs = 0.125 at a
  reference magnitude of 141.6 — exactly one FP16 ULP — meanAbs = 0.00225. That is the
  boundary's separate add + RMSNorm replacing an in-window fused add-RMSNorm, i.e. rounding, not
  logic.
• cycled trunk + host head vs CudaTransformerModel.Forward last row (vocab 128256):
  maxAbs = 0.150, meanAbs = 0.021, top-1 identical.
• per-window VRAM release, which is the point of the issue: nvidia-smi memory.used 672 MiB idle
  → 2481 MiB whole trunk → 672 MiB after dispose → 1685 MiB with [0..8) → 672 MiB after dispose.
  VRAM does not accumulate across windows.

CudaLayerCyclingPerplexityTests (new, 3 facts, all passing on the local RTX 3060) against
Llama-3.2-1B-Q8_0, which fits the card whole. Scoring real English through the model's own
tokenizer (mean NLL ~3.0), not random ids:

• the acceptance criterion: cycled (4-layer windows) vs whole-device CUDA, teacher-forced —
  mean NLL 2.999327 vs 3.000402, 0.0358% (bound: 0.3%). Teacher-forced because the
  whole-device CUDA model returns only the last logit row and therefore cannot run sliding-window
  mode at all, so that is the only like-for-like comparison available.
• the sharper instrument: cycling with 16 one-layer windows vs a single 16-layer window changes
  only the number of boundary round trips, same head, same kernels — mean NLL 3.130479 vs
  3.132169, 0.0540% (bound: 0.2%).
• proof the bounds can fail: a 5% element-wise corruption of the boundary checkpoint moves the
  figure 0.632%, well outside the bound.

CLI, all three placements run end-to-end on Llama-3.2-1B-Q8_0, 400 tokens, context 128:

Run	Perplexity	Mean NLL	Elapsed
--device cpu (reference)	108.1703	4.683706	53.5 s
--gpu-layers 8 --cycle --per-window	107.1435	4.674169	19.8 s
--gpu-layers 4 --first-layer 4	106.8207	4.671151	40.8 s

Both device paths sit within 0.3% of the whole-CPU mean NLL, which is the expected size of FP16
device arithmetic against FP32 CPU. --per-window lines printed, the phase progress printed, and the
Layer placement row rendered in every case.

Build: solution-wide dotnet build -c Release clean, no new warnings.

One real bug the CLI smoke run caught that no library test could: a placement renders as
[0..8), and Spectre reads the leading [ as a markup tag, so every windowed/cycled run aborted
with "malformed markup tag" instead of printing its table. Fixed in 9310a14.

Review round (D1-D4)

D1 (blocking) — scaled RoPE now rejected. CudaPipelineStage hands LaunchRoPE only
theta/dim/type, and no CUDA overload takes YaRN's scaling factor, original context length, attn factor
or beta fast/slow — while the CPU reference does apply them. Scaled RoPE changes the frequency table
rather than the layer graph, so a YaRN GGUF cleared every structural rejection and would then have been
scored with the unscaled table.
Mitigating context, so this reads at its true size: the gap is pre-existing and backend-wide
(whole-device --device cuda is equally unscaled) and this does not fix it — it keeps the new
guard's promise honest. llama3 scaling maps to RoPEScalingType.None, so the Llama-3.x family
including the test fixture is unaffected; SmolLM3-128k is already caught by the NoRopeLayers
rejection.
Verified: ValidateSupported became internal (a rejection that silently stopped firing would look
exactly like a passing build), and CudaLayerWindowScopeGuardTests covers all five scaling types
plus a positive control and an explicit "unscaled RoPE still accepted" case, so the guard cannot
pass by rejecting everything — 9/9 passing.

D3 — hybrid + --cycle no longer stack-traces. Model construction moved inside the try.
Verified end-to-end: a synthetic Qwen3MoeHybrid GGUF now exits 1 with the one-line
mixture-of-experts FFN routing is not implemented ... message, no stack trace.

D2 — dead LM-head VRAM removed. New skipOutputHead on CudaWeights.LoadFromGguf, driven from
isFinalStage rather than the layer range. Two measurements, and they answer different questions:

• Per-window, cuMemGetInfo_v2 around the same window built isFinalStage: true vs false:
  268 MiB, matching the vocab x hidden arithmetic for the raw quantized copy. This is the figure
  that matters for "can the last window fit", and it scales with vocab x hidden.
• Peak across the whole cycle: 140 MiB (my earlier controlled A/B on a 757 MiB idle baseline,
  2465 -> 2325 MiB). Lower because peak is a max over windows: [0..8) carrying the embedding table
  lands at about the same figure as [8..16) carrying the head.

My original explanation for the gap — tied embeddings and Q8_0 dedup — was wrong and has been
dropped from the code comments. The last window sets skipTokenEmbed: firstLayer != 0, so there is no
embedding table there to dedup against; and a loaded Q8_0 GEMV kernel means only the raw quantized
copy is uploaded, never an extra FP16 one. Existing CudaPipelineTransformerModel behaviour is
unchanged (stage 1 is isFinalStage), confirmed by its split-parity tests.

D4 — taken, and it found two things. Random ids scored 12.18 against a uniform floor of
ln(128256) ~ 11.76, so the figure was pinned near chance; switched to real English and tightened the
bounds from 1%/0.5% to 0.3%/0.2%. The new fault-injection test then failed twice before it
passed, which is the useful part:

1. A uniform scale of the boundary is nearly invisible — 0.043%, below plain rounding — because
   the next layer begins with RMSNorm, which is scale-invariant. The perturbation is now
   element-wise with alternating sign, which rotates the checkpoint's direction instead.

2. Calibration is documented on the test and in docs/GPU.md. Corrected in review round 2 after an
   independent full-curve measurement:

   • The demonstrated-reliable detection level is 5%, not 2%. A 2% element-wise corruption
     measures 0.165%, below the 0.2% bound — so a 2% corruption cannot be relied on to be
     caught. Measured curve: 0.5% → 0.138%, 1% → 0.123–0.131%, 2% → 0.165%, 5% → 0.585–0.632%.
   • The response is non-monotonic below ~2%: 0.5% moves the figure more than 1% does, because
     at that scale the effect is dominated by which tokens flip rather than by magnitude. Dose-response
     reasoning does not apply there.
   • Sub-0.1% figures are not constants. Two independently written, individually byte-reproducible
     harnesses disagree 5.7x on the 16-cuts-vs-1 figure (0.0540% vs 0.0094%) on identical inputs,
     with the disagreement scaling with boundary count. Those rows are now published as ranges marked
     sequence-dependent, not scalars. Filed as eval: sub-0.1% perplexity deltas are sequence artifacts — two deterministic harnesses disagree 5.7x on identical inputs #429, since it affects how thresholds are derived
     repo-wide.

   The bounds themselves (0.3% / 0.2%) are validated and unchanged — only the claims about what they
   resolve were corrected. The bit-identical CPU tests carry the logic load.

D5 — acknowledged, not fixed. RunLayerWindowed holds two host TransformerWeights (the CPU
model's and the CUDA window model's), each repacking. Irrelevant at 1B, relevant at 19 GB. Follow-up.

Round-2 nits, all fixed: the acceptance checklist's stale "1% bound"; a doubled <remarks> block
on ValidateSupported; the "~0.5 GB on a 1B model" code comments, now the measured 268 MiB; and
CudaDevice.GetDevice still being evaluated above the try in RunLayerWindowed — enumerating the
device is itself a CUDA call, so a missing device reproduced the unhandled-stack-trace class of
problem D3 fixed, in a different failure mode. The catch now also covers CudaException and
InvalidOperationException.

No automated test covers --first-layer's mixed CPU+CUDA composition. The unit tests are all-CPU,
the integration tests all-CUDA; only the CLI smoke run above exercised the three-way assignment.
Stating it rather than leaving it to be discovered.

What was NOT verified — please read before merging

• The headline 30B case is not verifiable locally and was not run. The two motivating models are
  MoE/hybrid, and CudaPipelineStage — which the CUDA layer-window executor is built on — is
  documented dense/GQA-causal only. CudaLayerWindowModel therefore rejects MoE / MLA / hybrid /
  SSM / Gemma variants at construction with a named NotSupportedException, rather than scoring them
  wrongly. For those models this PR delivers --gpu-layers prefix offload; --cycle /
  --first-layer on CUDA covers dense models. Hybrid cycling is verified on CPU. GPU cycling for
  MoE/hybrid is follow-up work.
• --gpu-layers prefix offload is teacher-forced only. I assumed the hybrid model's CPU tail
  would return all rows and therefore support sliding-window scoring; the CLI smoke run showed it
  does not — HybridTransformerModel returns the last row only, so --gpu-layers without --cycle
  falls to the O(n²) growing-prefix path (235 s for 64 tokens on the 1B model, against 19.8 s for 400
  tokens cycled). Correcting my own earlier assumption here rather than leaving it implied. Cycling
  does not have this problem, because its host-side output head produces all rows — which turns out
  to be a second, unplanned reason to prefer it. Making the hybrid path return all rows is out of
  scope for this PR.
• The GDN-ordinal mapping is verified by reading the code, not by the parity test. The ordinal
  table is built once in the constructor keyed by absolute layer index, so a window indexes the
  slot a full run would. But because state is zeroed before every sequence and each GDN layer touches
  only its own slot within a run, a naive restart-the-count-at-the-window implementation would still
  be injective over zeroed state and produce identical output. The parity run corroborates; it does
  not prove.
• Cycling is CUDA-only. --cycle / --first-layer reject Vulkan and CPU-only devices explicitly
  rather than silently degrading.
• Checkpoints are held in host RAM; spilling to disk is not implemented. A wikitext-2 sweep at
  hidden = 2688 is ~3.6 GB per boundary, two alive at a time.
• End-perplexity remains a weak backend discriminator — each window blends host and device
  arithmetic and a uniform bias can partially cancel. docs/GPU.md says so and points at the saved
  boundary activations as the sharper instrument.
• Pre-existing unrelated failures in the unit suite. Final run (excluding Vulkan): 3149 passed,
  4 failed — all four CudaMoeFfnBitNetI2SBatchedGemmTests. A fifth,
  CudaGraphCaptureEquivalenceTest.EagerVsGraphDecode_BitNet_CrossesGraphDepthThreshold_Match, failed
  on an earlier run and passed on this one, so it appears intermittent. None references any type this
  branch changed — the BitNet GEMM test names none of them at all, and the graph-capture test's
  TransformerModel references are all to CudaTransformerModel, not the CPU TransformerModel that
  was refactored. I did not re-run them against a clean dev build, so I am flagging rather than
  asserting that they pre-date this branch.

Behaviour note

--gpu-layers matches run/chat's semantics including the nuance that an explicit count decides
the backend: --device cpu --gpu-layers 8 offloads 8 layers to CUDA, as it does in run.

Docs

docs/GPU.md gains a "Measuring a model larger than the device" section: the three placements, why
cycling exists, the prefix-offload teacher-forced caveat, the checkpoint arithmetic, the host-head
rationale, the recurrent-state invariant, the discriminator caveat, and the dense-GQA-only scope.

Acceptance criteria

• --gpu-layers on perplexity, semantics matching run/chat/serve.
• Arbitrary contiguous GPU layer window (--first-layer + --gpu-layers) selectable from the CLI.
• A layer-cycling mode covering all layers on the GPU across one corpus pass via
  boundary-activation checkpointing.
• Recurrent state handled across the cut, with a test on a hybrid architecture that
  discriminates the with/without forms — implemented as a phase-local invariant plus a
  per-corpus-window reset rather than as a serialised snapshot; see the reconciliation section
  above, which explains why a snapshot would be dead weight.
• Equivalence test: cycled PPL == whole-device PPL on a model that fits entirely on the GPU,
  within a documented tolerance (0.0358% of mean NLL measured, 0.3% bound; plus an exact CPU
  cycled-vs-whole test to 12 decimal places).
• The split appears in the output on every run, cycled and whole-device alike.
• --per-window and --tokens-file continue to work in cycling mode. --per-window is covered
  by a unit test and by the CLI run above. --tokens-file was run: --dump-tokens from the
  whole-CPU run, then fed back with --gpu-layers 8 --cycle, reproducing perplexity
  107.1435 +/- 19.74914 — identical to the corpus-file cycled run.
• Docs note on when to use cycling mode and its discriminator limits.

🤖 Generated with Claude Code

https://claude.ai/code/session_01DzekWWxE4d52Hpa31WBYfX

[Copilot]

Pull request overview

Adds layer-window execution and a layer-cycling perplexity mode so dotllm perplexity can evaluate models larger than GPU memory by offloading/partitioning the transformer trunk (including arbitrary contiguous GPU windows) and replaying from checkpointed boundary activations.

Changes:

• Introduces ILayerWindowModel/ILayerWindowExecutor plus CPU/CUDA implementations and a CompositeLayerWindowModel for per-window device assignment.
• Implements CyclingPerplexityEvaluator and PerplexityWindowPlan, and refactors PerplexityEvaluator to share the single window-enumeration plan.
• Extends dotllm perplexity CLI with --gpu-layers, --first-layer, and --cycle, plus unit/integration test coverage and updated GPU docs.

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/DotLLM.Tests.Unit/Evaluation/LayerCyclingPerplexityTests.cs	Adds unit coverage for cycling correctness, recurrent reset discrimination, and per-window diagnostics parity.
tests/DotLLM.Tests.Integration/Cuda/CudaLayerCyclingPerplexityTests.cs	Adds CUDA integration acceptance tests comparing cycled vs whole-device runs within tolerances.
src/DotLLM.Models/Gguf/SyntheticQwen35MoeGguf.cs	Extends synthetic hybrid fixture generation to support variable even layer counts for window/cycle coverage.
src/DotLLM.Models/Evaluation/CpuLayerWindowModel.cs	Exposes CPU models via the layer-window contract for cycling and parity validation.
src/DotLLM.Models/Architectures/TransformerModel.cs	Adds trunk window execution and output-head application helpers to support cycling/windowing.
src/DotLLM.Models/Architectures/Qwen3MoeHybridTransformerModel.cs	Adds windowed trunk execution and output-head application for hybrid (recurrent) fixture/cycling.
src/DotLLM.Engine/Evaluation/PerplexityWindowPlan.cs	Centralizes and validates corpus window enumeration shared by whole-device and cycling evaluation.
src/DotLLM.Engine/Evaluation/PerplexityEvaluator.cs	Refactors sliding-window evaluation to use PerplexityWindowPlan for shared enumeration/BOS handling.
src/DotLLM.Engine/Evaluation/CyclingPerplexityEvaluator.cs	Implements cycling driver: boundary checkpointing, replay adapter, and partition validation.
src/DotLLM.Engine/Evaluation/CompositeLayerWindowModel.cs	Composes multiple ILayerWindowModel backings to express arbitrary GPU windows and CPU/GPU splits.
src/DotLLM.Cuda/Evaluation/CudaLayerWindowModel.cs	Implements CUDA per-window upload/free execution via CudaPipelineStage, with host-side output head.
src/DotLLM.Core/Evaluation/ILayerWindowModel.cs	Adds backend-neutral window execution and reset contract for cycling/offload evaluation.
src/DotLLM.Cli/Commands/PerplexityCommand.cs	Adds CLI flags and execution paths for prefix offload, arbitrary window placement, and cycling.
docs/GPU.md	Documents new perplexity placements, cycling rationale, constraints, and scope limits.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.