glm: fix [gMASK]<sop> tokenization without BOS metadata

[mvid]

Problem

GLM chat templates start with [gMASK]<sop>. Some converted GGUFs omit tokenizer.ggml.bos_token_id, and the current fallback assigns <sop> to bos_id. That makes both direct chat construction and rendered server prompts tokenize the preamble as <sop><sop>.

The rendered-token table already maps [gMASK] to bos_id, so the fallback conflicts with the tokenizer's own intended mapping.

Change

• Prefer the [gMASK] token when GLM BOS metadata is absent, retaining <sop> as a compatibility fallback.
• Append <sop> only when it differs from bos_id.
• Add a model-independent tokenizer test covering direct chat construction, rendered [gMASK]<sop> tokenization, and direct-path BOS/SOP deduplication.

Compatibility boundary: a tokenizer that lacks [gMASK] entirely still has different direct and rendered behavior. The direct path emits one <sop> after deduplication, while rendered [gMASK]<sop> still maps both literals to bos_id == sop_id. Official GLM Flash vocabularies contain [gMASK]; this PR retains the old <sop> fallback without redefining that no-[gMASK] rendered-template edge.

Testing

Machine: Apple M5 Max, 128 GB, Metal.

Model used for the full suite: DeepSeek V4 Flash Layers37-42Q4KExperts-OtherExpertLayersIQ2XXSGateUp-Q2KDown-AProjQ8-SExpQ8-OutQ8.

make clean
make -j4
DS4_TEST_MODEL=/path/to/DeepSeek-V4-Flash-Layers37-42Q4KExperts-OtherExpertLayersIQ2XXSGateUp-Q2KDown-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix-fixed-0731.gguf make test

All tests passed, including the new tests/test_tokenizer target and the existing server, agent, Metal tensor-equivalence, sampling, and CLI suites.

The metadata-less GLM GGUF used during diagnosis predates the current upstream GLM-5.3 metadata format, so current main rejects it during shape validation before tokenizer setup (expected block_count=79 for GLM 5.2, got 46). The permanent regression therefore uses a synthetic GLM vocabulary and runs without a model file.

Requirements

• I have read and agree with the contributing guidelines.
• AI usage disclosure: YES.