fix(tokenizers): pre-tokenize raw text before byte-encoding; add StarCoder/SmolLM pre-type family

[jamesburton]

Tracked by jamesburton#237 (fork issue tracker) — deliberately not using a Closes keyword, since #237 here is an unrelated issue in this repo.

Summary

BPE pre-tokenization was applied to byte-encoded text rather than raw text, which silently disabled every whitespace-dependent alternative in every pre-tokenizer pattern. Alongside that, smollm was unmapped (and an unmapped type means no pre-tokenization at all), and the pre-type table could not express a multi-stage pipeline.

Found while validating a perplexity harness (jamesburton#231) against llama-perplexity on wikitext-2.

The three defects

1. Split-then-encode ordering. Gpt2TiktokenEncoding.Encode converted input to GPT-2 byte-level Unicode first, then ran the pre-tokenization regex over the result. In that encoding byte 0x20 maps to U+0120, not to a literal space — so \s, a leading ' ?', and \s+(?!\S) could never match. llama.cpp splits the raw text and byte-encodes each segment afterwards; Encode now does the same. This affected every model using pre-tokenization, not just the unmapped ones.

2. smollm unmapped, and null means "no pre-tokenization". SmolLM-135M.Q8_0.gguf declares tokenizer.ggml.pre = smollm, which the table did not handle, so BPE merges ran across the entire input with no boundaries. Adds the StarCoder/SmolLM block covering the eight pre-types llama.cpp falls through a single case: starcoder, refact, command-r, smollm, codeshell, exaone, minerva, mellum2. command-r was previously mapped to the Llama-3 regex, which is wrong.

3. A pre-type is a pipeline, not one expression. llama.cpp's regex_exprs is an ordered list, each expression further splitting the previous stage's segments. This family genuinely needs two stages:

"\p{N}"
"'s|'t|'re|'ve|'m|'ll|'d| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)"

GetRegex → GetRegexes accordingly. PreTokenize also preserves unmatched spans: the old code encoded only regex matches, silently dropping anything unmatched. That is harmless for the GPT-2 expression, which ends in |\s+ and matches everything, but not for this family's pattern, which omits that alternative by design — dropping characters there corrupts the stream rather than merely re-splitting it.

Verification

Full token-stream equality against llama.cpp build 8683 (d0a6dfeb2) — not equal counts. That distinction is the point: counts matched exactly (3904 both) while 30 ids differed, so any count-based assertion would have passed throughout and the bug would have stayed invisible.

Model	pre-type	tokens	mismatches
SmolLM-135M Q8_0	smollm	3904	0
Llama-3.2-3B-Instruct Q8_0	llama3	945	0

over a 16.4 KB slice of wikitext-2 wiki.test.raw. llama.cpp prepends BOS for Llama-3 (add_bos_token = true); that is a caller-level concern, not a pre-tokenizer one, and is excluded from the comparison.

Both families are covered because defect #1 meant the previously-"working" ones were equally suspect.

Unit tests: 1167 passed, 0 failed, 36 skipped.

Why this matters beyond perplexity

Any text with a spaced apostrophe — Teddy 's Story, ubiquitous in wikitext-raw-style corpora — tokenized differently from the reference. That affects generation quality, prompt fidelity, and every comparison against llama.cpp or HF. It degrades quietly: the output is almost right, which is the hardest kind of defect to notice.

Measured on our side, it moved a 15-chunk perplexity from 24.8439 to 23.7864 against llama.cpp's 24.0137.

Note for reviewers

A silent null for an unrecognized pre-type remains a poor default — it disables pre-tokenization rather than failing loudly. Left as-is here to keep the change focused, but worth revisiting.

[Copilot]

Pull request overview

Fixes tiktoken-style BPE pre-tokenization to mirror llama.cpp semantics by running regex pre-tokenization on raw text (before GPT-2 byte-level Unicode mapping), and by modeling tokenizer.ggml.pre as an ordered regex pipeline (including the StarCoder/SmolLM family).

Changes:

• Apply pre-tokenization on raw text, then byte-encode + BPE each segment to prevent whitespace-dependent patterns from being silently disabled.
• Extend tokenizer.ggml.pre mapping to include StarCoder/SmolLM-family pre-types and represent them as a two-stage pipeline.
• Update unit tests to validate pipeline mapping and to guard the raw-text whitespace matching invariant.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
tests/DotLLM.Tests.Unit/Tokenizers/BpeTokenizerTests.cs	Updates tests for pipeline-based pre-tokenization and adds a regression test for the raw-text ordering bug.
src/DotLLM.Tokenizers/Bpe/TiktokenPreTokenizer.cs	Changes pre-type mapping from a single regex to a regex pipeline; adds StarCoder/SmolLM-family mapping.
src/DotLLM.Tokenizers/Bpe/Gpt2TiktokenEncoding.cs	Reworks encoding to pre-tokenize raw text first, then byte-encode + BPE per segment; preserves unmatched spans in pipeline splitting.
src/DotLLM.Tokenizers/Bpe/BpeTokenizer.cs	Wires the new GetRegexes pipeline into tiktoken tokenizer construction.

Comments suppressed due to low confidence (1)

src/DotLLM.Tokenizers/Bpe/Gpt2TiktokenEncoding.cs:212

• PreTokenize initializes the spans list with capacity text.Length, which can allocate a very large backing array for long prompts even though the number of spans is usually far smaller. Using a capped initial capacity avoids unnecessary memory pressure.

        var spans = new List<(int Start, int Length)>(text.Length) { (0, text.Length) };

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[jamesburton]

Both addressed in 579c79f.

_preRegexes doc (fixed). Agreed, and it mattered more than doc-rot usually does. The comment
still described a single regex, and said nothing about null. Both are now explicit — in
particular that null means no pre-tokenization at all, not a sensible default: merges then run
across boundaries the model was trained to respect, producing a stream that mostly matches the
reference and diverges at a handful of sites. That is exactly the bug this PR fixes, so leaving it
undocumented invited its reintroduction. Also recorded why it is an array: a pre-type maps to an
ordered pipeline (the StarCoder/SmolLM family isolates digits before its main pattern), and
collapsing that to one expression silently mis-tokenizes.

new List<int>(text.Length) over-allocating (valid, deliberately not here). Real, and it is the
subject of #413 — I opened #419 for it and measured this exact question. Two findings:

• Sizing by an estimated token count instead saves a further 2.66x on prose.
• But it costs more on a vocabulary emitting ~1 token per character, where growth by doubling
  exceeds the over-reserve it avoids (40.0 -> 43.0 bytes/char measured).

So it is a heuristic trade rather than a strict improvement, which is why it is not being smuggled
into a correctness fix. #419 carries the unambiguous part of the allocation work (hoisting the
per-segment merge queue: 16.0x on a llama-bpe vocabulary).

Worth noting the two PRs interact, since it is not obvious from either alone: once this lands, the
segmented path allocates a queue per pre-token, so TokenizerAllocationBenchmarks goes 32.73 ->
83.13 bytes/char. #419 on top brings it to 16.76 — below where it stands today. Detail on #413.

Tokenizer tests 228/228.