Adopt official DeepSeek V4 DSML tool calling format and preserve Unicode in GPT-2 BPE decoder

[dpavlin]

Summary

DeepSeek-V4 was trained natively on the DSML (DeepSeek Modeling Language) tool calling format, as specified in tokenizer.chat_template in official model weights and GGUF metadata.

Currently, Lucebox's ChatFormat::DEEPSEEK4 in chat_template.cpp provides an ad-hoc JSON instruction (<function_call>{"name": ..., "arguments": ...}</function_call>). Over longer multi-turn agent sessions, the model naturally reverts to its native DSML training distribution, outputting <｜DSML｜tool_calls> envelopes.

Adopting the native DSML chat template and parser improves model adherence, removes formatting drift, and aligns with DeepSeek's official inference guidelines.

---

Canonical DSML Format (from GGUF metadata)

1. System Prompt & Tool Schemas

## Tools

You have access to a set of tools to help answer the user question. You can invoke tools by writing a "<｜DSML｜tool_calls>" block like the following:

<｜DSML｜tool_calls>
<｜DSML｜invoke name="$TOOL_NAME">
<｜DSML｜parameter name="$PARAMETER_NAME" string="true|false">$PARAMETER_VALUE</｜DSML｜parameter>
...
</｜DSML｜invoke>
<｜DSML｜invoke name="$TOOL_NAME2">
...
</｜DSML｜invoke>
</｜DSML｜tool_calls>

String parameters should be specified as is and set `string="true"`. For all other types (numbers, booleans, arrays, objects), pass the value in JSON format and set `string="false"`.

If thinking_mode is enabled (triggered by <think>), you MUST output your complete reasoning inside <think>...</think> BEFORE any tool calls or final response.

Otherwise, output directly after </think> with tool calls or final response.

### Available Tool Schemas

{"name": "read", "description": "...", "parameters": {...}}

You MUST strictly follow the above defined tool name and parameter schemas to invoke tool calls.

2. Multi-turn Serialization

• Tool Result Input:

  <｜User｜><tool_result>{"result": "content"}</tool_result>
  

• Assistant Invocation:

  <｜Assistant｜><think>Reasoning...</think>
  
  <｜DSML｜tool_calls>
  <｜DSML｜invoke name="read">
  <｜DSML｜parameter name="path" string="true">/path/to/file</｜DSML｜parameter>
  </｜DSML｜invoke>
  </｜DSML｜tool_calls><｜end of sentence｜>
  

---

Key Considerations & Dependencies

1. Tokenizer Decoding of Special UTF-8 Glyphs:
   ｜ is Unicode U+FF5C (Fullwidth Vertical Line). In tokenizer.cpp, decode_gpt2_bpe() currently maps any codepoint $cp \ge 324$ to '?' via gpt2_unicode_to_byte(). To correctly decode <｜DSML｜...>, decode_gpt2_bpe() should preserve unmapped Unicode codepoints as raw UTF-8 bytes instead of returning '?'.

2. Stream Holdback Window:
   <｜DSML｜tool_calls> is 22 UTF-8 bytes. tool_syntax_holdback() in tool_parser.cpp needs to be sized ($\ge 21$ bytes) to prevent premature emission of tag fragments into the content stream.

3. Tool Parser Support:
   tool_parser.cpp should accept <｜DSML｜tool_calls> alongside standard <tool_calls> and <function_call> envelopes, parsing both string="true" raw string values and string="false" JSON-encoded primitives/objects.