diff --git a/TECHNICAL_REPORTS/2165-typed-chat-text-20261013.en.md b/TECHNICAL_REPORTS/2165-typed-chat-text-20261013.en.md new file mode 100644 index 000000000..24afb44d3 --- /dev/null +++ b/TECHNICAL_REPORTS/2165-typed-chat-text-20261013.en.md @@ -0,0 +1,59 @@ +# Technical Report: PR #2165 - Typed Chat Text + +**Date**: 2026-10-13 +**Status**: Ready for review +**Languages**: Rust +**Risk Level**: Medium + +## Executive Summary + +Checkpoint-selected chat templates that accept typed text parts but silently discard string content are now detected once during model loading. Typed `ChatMessage` renders use a borrowed text-part view only for those templates, while raw JSON rendering and string-compatible templates retain their existing contracts. + +## 1. Problem Statement + +The Granite Vision 3.2 checkpoint ships a standalone multimodal template that is correctly preferred for image requests, but that template reads text from typed content parts. CLI and server text-only paths supplied strings, so the question was omitted from the prompt: the real checkpoint produced a 47-token prompt and an unrelated greeting instead of answering the question. + +Falling back to the tokenizer configuration's string template would split image and text behavior across templates and would make CLI, server, and prompt-cache history rendering disagree. + +## 3. Technical Decisions + +### Detect compatibility once at selection time + +After selecting the checkpoint template, the loader renders one string probe and one typed-text probe through the unchanged raw renderer. It stores `wrap_string_content = true` only when the typed probe preserves the marker and the string probe does not; either render error leaves adaptation disabled. + +### Adapt only typed messages without copying text + +`apply_inner` serializes a small borrowed wrapper containing the original role and a single typed text part. The wrapper holds `&str` references and is passed directly to MiniJinja, avoiding an intermediate JSON tree and duplicate string allocation. Exhaustive `ChatMessage` destructuring makes a future field addition fail compilation until the rendering view preserves it. + +### Preserve existing contracts + +`apply_raw*`, explicit `with_template` processors, the default processor, string-compatible templates, typed image ordering, and probe-error behavior remain unchanged. The fix does not alter CLI or server call sites and adds no dependency. + +## 7. Change Summary + +| Category | Files | Summary | +|---|---:|---| +| Production | 1 | Selection-time compatibility flag and borrowed `apply_inner` rendering view | +| Regression tests | 1 | Eight tests for exact Granite prompts, roles, empty text, raw behavior, compatible templates, constructors, and probe errors | +| Pinned fixtures | 5 | Granite Vision 3.2 and Granite 4 templates with source provenance | +| Reports | 2 | English and Korean implementation records | + +## Appendix + +### Test Results + +- `cargo test --locked -p mlxcel -p mlxcel-surgery --lib --profile test-fast --features cuda -j4 text_wrap_tests -- --test-threads=1`: 8 passed, 0 failed; 9,105 unrelated mlxcel tests and 160 surgery tests were filtered out. +- Repository contract gate passed: workspace versions, kernel dtype keys, kernel port dispatch, llama compatibility, ROCm overlay checks, 123 Python tooling tests, and 49 WebUI contract fixtures. +- On NVIDIA GB10/CUDA with `mlx-community/granite-vision-3.2-2b-4bit` revision `351831ab5218b1f66bbfdfd57bd8f20d81ce8baf`, CLI answered “The capital of France is Paris.” in 8 tokens. Server string and typed-text requests returned the same answer with 55 prompt tokens and 8 completion tokens. +- The frozen baseline returned an unrelated 20-token greeting for both server shapes with 47 prompt tokens, reproducing the defect. +- Implementation review, security rereview, and static finalization found no remaining issues. + +### Provenance + +- Frozen source diff SHA-256: `1d088cbdc926b50a1af5fc8810435dd2cccb3b846bf5477a00b363ea6d9ceaa4`. +- Candidate binary SHA-256: CLI `706fcf0864c6adbd251e2725db1f623044b49e6a33436617925dc9b1da4d09f8`; server `496534a54f6ea92ceaea5e3cfaec20237b11f753bd5fef114927208607a2998c`. +- Baseline binary SHA-256: CLI `592c33a141ccd7a497732e132ac6da302661e22400e27fbcdb2cb3ca858ae8ff`; server `bc076e4d0b2fde769eb15ababe241e72828c5f9c0b3ec2142137cef688c5a656`. + +### Validation Limits + +The issue unit ran focused CUDA tests and one real checkpoint on Linux/GB10. It did not run the full workspace test or Clippy suites, ROCm inference, macOS inference, or additional model families; the parent integration gate and hosted CI own the broad checks. diff --git a/TECHNICAL_REPORTS/2165-typed-chat-text-20261013.ko.md b/TECHNICAL_REPORTS/2165-typed-chat-text-20261013.ko.md new file mode 100644 index 000000000..a312e08b3 --- /dev/null +++ b/TECHNICAL_REPORTS/2165-typed-chat-text-20261013.ko.md @@ -0,0 +1,59 @@ +# 기술 보고서: PR #2165 - Typed Chat Text + +**작성일**: 2026-10-13 +**상태**: 리뷰 준비 완료 +**언어**: Rust +**위험도**: 중간 + +## 요약 + +typed text part는 처리하지만 문자열 content를 조용히 버리는 checkpoint 선택 템플릿을 모델 로드 시 한 번 감지하도록 수정했습니다. 해당 템플릿의 typed `ChatMessage` 렌더만 빌려온 text-part view를 사용하며, raw JSON 렌더와 문자열 호환 템플릿의 기존 계약은 그대로 유지합니다. + +## 1. 문제 정의 + +Granite Vision 3.2 checkpoint는 이미지 요청에 필요하여 우선 선택되는 standalone multimodal 템플릿을 제공하지만, 이 템플릿은 typed content part에서만 텍스트를 읽습니다. CLI와 서버의 text-only 경로는 문자열을 전달했으므로 질문이 prompt에서 사라졌고, 실제 checkpoint는 47-token prompt와 질문에 무관한 인사말을 생성했습니다. + +tokenizer configuration의 문자열 템플릿으로 fallback하면 이미지와 텍스트 요청이 서로 다른 템플릿을 사용하게 되고 CLI, 서버, prompt-cache history 렌더가 달라집니다. + +## 3. 기술적 선택과 그 이유 + +### 선택 시점에 호환성을 한 번 감지 + +checkpoint 템플릿을 선택한 뒤 변경하지 않은 raw renderer를 통해 문자열 probe와 typed-text probe를 각각 한 번 렌더합니다. typed probe만 marker를 보존할 때 `wrap_string_content = true`를 저장하며, 어느 probe에서든 렌더 오류가 발생하면 변환을 활성화하지 않습니다. + +### 텍스트 복사 없이 typed message만 변환 + +`apply_inner`는 원래 role과 하나의 typed text part를 담은 작은 borrowed wrapper를 직렬화합니다. wrapper는 `&str` 참조를 유지한 채 MiniJinja에 직접 전달되므로 중간 JSON tree와 문자열 중복 할당이 없습니다. `ChatMessage`를 빠짐없이 destructuring하여 향후 필드가 추가되면 렌더 view가 해당 필드를 보존하도록 갱신하기 전까지 컴파일이 실패합니다. + +### 기존 계약 유지 + +`apply_raw*`, 명시적인 `with_template` processor, default processor, 문자열 호환 템플릿, typed image 순서, probe 오류 동작은 바뀌지 않습니다. CLI와 서버 호출부를 수정하지 않았고 새 의존성도 없습니다. + +## 7. 변경 요약 + +| 분류 | 파일 수 | 내용 | +|---|---:|---| +| Production | 1 | 선택 시점 호환성 flag와 borrowed `apply_inner` 렌더 view | +| 회귀 테스트 | 1 | 정확한 Granite prompt, role, 빈 문자열, raw 동작, 호환 템플릿, constructor, probe 오류를 검증하는 8개 테스트 | +| 고정 fixture | 5 | 출처가 기록된 Granite Vision 3.2 및 Granite 4 템플릿 | +| 보고서 | 2 | 영문 및 국문 구현 기록 | + +## 부록 + +### 테스트 결과 + +- `cargo test --locked -p mlxcel -p mlxcel-surgery --lib --profile test-fast --features cuda -j4 text_wrap_tests -- --test-threads=1`: 8개 통과, 실패 0개이며 관련 없는 mlxcel 테스트 9,105개와 surgery 테스트 160개는 filter되었습니다. +- 저장소 contract gate를 통과했습니다: workspace version, kernel dtype key, kernel port dispatch, llama compatibility, ROCm overlay 검사, Python tooling 테스트 123개, WebUI contract fixture 49개. +- NVIDIA GB10/CUDA에서 `mlx-community/granite-vision-3.2-2b-4bit` revision `351831ab5218b1f66bbfdfd57bd8f20d81ce8baf`을 실행한 결과, CLI는 8 token으로 “The capital of France is Paris.”라고 답했습니다. 서버의 문자열 및 typed-text 요청도 모두 55 prompt token과 8 completion token으로 같은 답을 반환했습니다. +- 동결된 baseline은 두 서버 요청 형식 모두에서 47 prompt token과 질문에 무관한 20-token 인사말을 반환하여 결함을 재현했습니다. +- 구현 리뷰, 보안 재검토, 정적 최종화에서 남은 문제가 발견되지 않았습니다. + +### Provenance + +- 동결 source diff SHA-256: `1d088cbdc926b50a1af5fc8810435dd2cccb3b846bf5477a00b363ea6d9ceaa4`. +- Candidate binary SHA-256: CLI `706fcf0864c6adbd251e2725db1f623044b49e6a33436617925dc9b1da4d09f8`; server `496534a54f6ea92ceaea5e3cfaec20237b11f753bd5fef114927208607a2998c`. +- Baseline binary SHA-256: CLI `592c33a141ccd7a497732e132ac6da302661e22400e27fbcdb2cb3ca858ae8ff`; server `bc076e4d0b2fde769eb15ababe241e72828c5f9c0b3ec2142137cef688c5a656`. + +### 검증 한계 + +이 이슈 단위에서는 focused CUDA 테스트와 Linux/GB10의 실제 checkpoint 하나를 검증했습니다. 전체 workspace test 및 Clippy suite, ROCm inference, macOS inference, 추가 모델 family는 실행하지 않았으며 broad check는 상위 integration gate와 hosted CI가 담당합니다. diff --git a/src/server/chat_template.rs b/src/server/chat_template.rs index 236331fde..64c2f76e0 100644 --- a/src/server/chat_template.rs +++ b/src/server/chat_template.rs @@ -59,6 +59,37 @@ pub struct ChatMessage { pub content: String, } +/// Borrowed serialization shape for templates that require typed text parts. +/// +/// The explicit, exhaustive destructuring in [`From`] makes adding a field to +/// [`ChatMessage`] a compile error here until that field is preserved in this +/// rendering shape as well. +#[derive(Serialize)] +struct TypedTextChatMessage<'a> { + role: &'a str, + content: [TypedTextContentPart<'a>; 1], +} + +#[derive(Serialize)] +struct TypedTextContentPart<'a> { + #[serde(rename = "type")] + kind: &'static str, + text: &'a str, +} + +impl<'a> From<&'a ChatMessage> for TypedTextChatMessage<'a> { + fn from(message: &'a ChatMessage) -> Self { + let ChatMessage { role, content } = message; + Self { + role, + content: [TypedTextContentPart { + kind: "text", + text: content, + }], + } + } +} + /// Sentinel attached as the [`std::error::Error::source`] of every error a /// chat template raises through the `raise_exception(...)` callable. /// @@ -178,6 +209,12 @@ pub struct ChatTemplateProcessor { bos_token: String, eos_token: String, add_generation_prompt: bool, + /// Whether typed [`ChatMessage`] renders must express text as content parts. + /// + /// Some multimodal checkpoint templates silently drop a plain string even + /// though they render an equivalent typed text part. This is detected once + /// while loading the checkpoint and never changes the raw JSON contract. + wrap_string_content: bool, /// Cached result of `supports_tools()` introspection. /// `None` means not yet computed. supports_tools_cached: Option, @@ -406,12 +443,13 @@ impl ChatTemplateProcessor { let template = preprocess_template(template); let forced_tool_call_format = detect_forced_tool_call_format(&template); - Ok(Some(Self { + let mut processor = Self { template, compiled_template: Arc::new(OnceLock::new()), bos_token, eos_token, add_generation_prompt: true, + wrap_string_content: false, supports_tools_cached: None, forced_tool_call_format, default_enable_thinking: false, @@ -420,7 +458,9 @@ impl ChatTemplateProcessor { kimi_k3_family_unrenderable: false, #[cfg(test)] template_compile_count: Arc::new(std::sync::atomic::AtomicUsize::new(0)), - })) + }; + processor.wrap_string_content = template_requires_typed_text_content(&processor); + Ok(Some(processor)) } /// Create a processor with a custom template string @@ -433,6 +473,7 @@ impl ChatTemplateProcessor { bos_token: String::new(), eos_token: String::new(), add_generation_prompt: true, + wrap_string_content: false, supports_tools_cached: None, forced_tool_call_format, default_enable_thinking: false, @@ -1091,7 +1132,16 @@ impl ChatTemplateProcessor { .filter(|t| !t.is_empty()) .map(minijinja::Value::from_serialize); - let messages_val = minijinja::Value::from_serialize(messages); + let wrapped_messages = self.wrap_string_content.then(|| { + messages + .iter() + .map(TypedTextChatMessage::from) + .collect::>() + }); + let messages_val = match wrapped_messages.as_ref() { + Some(messages) => minijinja::Value::from_serialize(messages), + None => minijinja::Value::from_serialize(messages), + }; let context = build_template_context( messages_val, &self.bos_token, @@ -1476,6 +1526,34 @@ fn template_renders_typed_image_content(template: &str) -> bool { } } +const TEXT_CONTENT_PROBE_MARKER: &str = "__mlxcel_text_probe__"; + +/// Detect a checkpoint template that accepts typed text parts but drops strings. +/// +/// Both probes go through the raw renderer so this decision cannot observe or +/// recursively trigger the typed-message adaptation it controls. Any render +/// error is treated conservatively as no adaptation. +fn template_requires_typed_text_content(processor: &ChatTemplateProcessor) -> bool { + let string_content = serde_json::json!([{ + "role": "user", + "content": TEXT_CONTENT_PROBE_MARKER, + }]); + let typed_content = serde_json::json!([{ + "role": "user", + "content": [{"type": "text", "text": TEXT_CONTENT_PROBE_MARKER}], + }]); + match ( + processor.apply_raw(&string_content, None), + processor.apply_raw(&typed_content, None), + ) { + (Ok(string_render), Ok(typed_render)) => { + !string_render.contains(TEXT_CONTENT_PROBE_MARKER) + && typed_render.contains(TEXT_CONTENT_PROBE_MARKER) + } + _ => false, + } +} + fn preprocess_template(template: String) -> String { // Handle all whitespace-control variants of the two delimiters. Using // explicit replaces keeps this allocation-light and regex-free. @@ -5036,3 +5114,7 @@ TOOL assert!(out.starts_with("")); } } + +#[cfg(test)] +#[path = "chat_template_text_wrap_tests.rs"] +mod text_wrap_tests; diff --git a/src/server/chat_template_text_wrap_tests.rs b/src/server/chat_template_text_wrap_tests.rs new file mode 100644 index 000000000..ec1435924 --- /dev/null +++ b/src/server/chat_template_text_wrap_tests.rs @@ -0,0 +1,196 @@ +// Copyright 2025-2026 Lablup Inc. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Checkpoint-selected templates that require typed text content (#2135). + +use std::path::PathBuf; + +use serde_json::json; + +use super::{ChatMessage, ChatTemplateProcessor}; + +const QUESTION: &str = "What is the capital of France?"; +const GRANITE_SYSTEM: &str = "<|system|>\nA chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions.\n"; +const GRANITE_3_CONFIG: &str = + include_str!("../../tests/fixtures/granite_vision_3_2/tokenizer_config.json"); + +fn message(role: &str, content: &str) -> ChatMessage { + ChatMessage { + role: role.into(), + content: content.into(), + } +} + +fn granite_3() -> ChatTemplateProcessor { + let path = PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("tests/fixtures/granite_vision_3_2"); + ChatTemplateProcessor::from_model_path(&path) + .expect("fixture must load") + .expect("fixture must contain a template") +} + +fn checkpoint(template: &str) -> ChatTemplateProcessor { + let directory = tempfile::tempdir().expect("tempdir"); + std::fs::write( + directory.path().join("tokenizer_config.json"), + json!({"chat_template": template}).to_string(), + ) + .expect("write tokenizer config"); + ChatTemplateProcessor::from_model_path(directory.path()) + .expect("checkpoint must load") + .expect("checkpoint must contain a template") +} + +#[test] +fn granite_3_from_model_path_preserves_the_exact_user_prompt() { + let processor = granite_3(); + assert!(processor.wrap_string_content); + assert_eq!( + processor + .apply(&[message("user", QUESTION)], None) + .expect("render"), + format!("{GRANITE_SYSTEM}<|user|>\n{QUESTION}\n<|assistant|>\n") + ); +} + +#[test] +fn granite_3_wraps_system_assistant_and_empty_content() { + let processor = granite_3(); + let history = [ + message("system", "Answer concisely."), + message("user", QUESTION), + message("assistant", "Paris."), + message("user", ""), + ]; + let expected = concat!( + "<|system|>\nAnswer concisely.\n", + "<|user|>\nWhat is the capital of France?\n", + "<|assistant|>\nParis.<|end_of_text|>", + "<|user|>\n\n" + ); + assert_eq!( + processor + .apply_history_with_kwargs(&history, None, &Default::default()) + .expect("history render"), + expected + ); + assert_eq!( + processor.apply(&history, None).expect("generation render"), + format!("{expected}<|assistant|>\n") + ); +} + +#[test] +fn granite_3_raw_rendering_keeps_its_original_contract() { + let processor = granite_3(); + let typed = json!([{ + "role": "user", + "content": [{"type": "image"}, {"type": "text", "text": QUESTION}], + }]); + assert_eq!( + processor.apply_raw(&typed, None).expect("typed render"), + format!("{GRANITE_SYSTEM}<|user|>\n\n{QUESTION}\n<|assistant|>\n") + ); + + let raw_string = json!([{"role": "user", "content": QUESTION}]); + let rendered = processor + .apply_raw(&raw_string, None) + .expect("raw string render"); + assert!(!rendered.contains(QUESTION)); +} + +#[test] +fn string_compatible_templates_keep_string_content() { + let processor = checkpoint( + "{% for m in messages %}{% if m.content is string %}string:{{ m.content }}{% else %}typed:{{ m.content[0].text }}{% endif %}{% endfor %}", + ); + assert!(!processor.wrap_string_content); + assert_eq!( + processor + .apply(&[message("user", QUESTION)], None) + .expect("render"), + format!("string:{QUESTION}") + ); +} + +#[test] +fn granite_3_config_only_string_template_is_not_wrapped() { + let directory = tempfile::tempdir().expect("tempdir"); + std::fs::write( + directory.path().join("tokenizer_config.json"), + GRANITE_3_CONFIG, + ) + .expect("write tokenizer config"); + let processor = ChatTemplateProcessor::from_model_path(directory.path()) + .expect("checkpoint must load") + .expect("checkpoint must contain a template"); + assert!(!processor.wrap_string_content); + assert!( + processor + .apply(&[message("user", QUESTION)], None) + .expect("render") + .contains(QUESTION) + ); +} + +#[test] +fn granite_4_accepts_both_shapes_without_wrapping() { + let path = PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("tests/fixtures/granite_4_vision"); + let processor = ChatTemplateProcessor::from_model_path(&path) + .expect("fixture must load") + .expect("fixture must contain a template"); + assert!(!processor.wrap_string_content); + assert!( + processor + .apply(&[message("user", QUESTION)], None) + .expect("string render") + .contains(QUESTION) + ); + assert!( + processor + .apply_raw( + &json!([{ + "role": "user", + "content": [{"type": "text", "text": QUESTION}], + }]), + None, + ) + .expect("typed render") + .contains(QUESTION) + ); +} + +#[test] +fn explicit_template_constructors_do_not_enable_wrapping() { + let template = include_str!("../../tests/fixtures/granite_vision_3_2/chat_template.jinja"); + let processor = ChatTemplateProcessor::with_template(template.into()); + assert!(!processor.wrap_string_content); + assert!( + !processor + .apply(&[message("user", QUESTION)], None) + .expect("render") + .contains(QUESTION) + ); + assert!(!ChatTemplateProcessor::default().wrap_string_content); +} + +#[test] +fn probe_render_errors_leave_wrapping_disabled() { + let processor = checkpoint(concat!( + "{% if messages[0].content is string %}", + "{{ raise_exception('string rejected') }}", + "{% else %}{{ messages[0].content[0].text }}{% endif %}" + )); + assert!(!processor.wrap_string_content); + assert!(processor.apply(&[message("user", QUESTION)], None).is_err()); +} diff --git a/tests/fixtures/granite_4_vision/SOURCE.md b/tests/fixtures/granite_4_vision/SOURCE.md new file mode 100644 index 000000000..5f750b586 --- /dev/null +++ b/tests/fixtures/granite_4_vision/SOURCE.md @@ -0,0 +1,10 @@ +# granite_4_vision chat-template fixture + +Source: [mlx-community/granite-4.0-3b-vision-4bit](https://huggingface.co/mlx-community/granite-4.0-3b-vision-4bit/tree/70fe1d89f42c71b5635cd945ed304bed438fa1e3) + +Pinned revision: 70fe1d89f42c71b5635cd945ed304bed438fa1e3 + +License: apache-2.0 (declared by the source model card). +The template is copied verbatim; no checkpoint weights are included. + +- Source [chat_template.jinja](https://huggingface.co/mlx-community/granite-4.0-3b-vision-4bit/resolve/70fe1d89f42c71b5635cd945ed304bed438fa1e3/chat_template.jinja), SHA-256: 29825b7fcc5ce44c7fd3f9977d8f2cc1b9ccef713ea2fe4046ab0d57b72c84a1. diff --git a/tests/fixtures/granite_4_vision/chat_template.jinja b/tests/fixtures/granite_4_vision/chat_template.jinja new file mode 100644 index 000000000..55d26aa26 --- /dev/null +++ b/tests/fixtures/granite_4_vision/chat_template.jinja @@ -0,0 +1,180 @@ +{#- ===== Task tag prompt constants ===== -#} +{%- set chart2code_prompt = "Generate code that recreates the chart as best as possible." -%} +{%- set chart2csv_prompt = "Please examine this chart image. Consider you are a data visualization expert, and extract the data into a CSV table.\n\nYour CSV should:\n- Include a header row with clear column names\n- Represent all data series/categories shown in the chart\n- Use numeric values that match the chart as closely as possible\n\nOutput only the CSV data, nothing else." -%} +{%- set chart2summary_prompt = "Can you describe this chart image?" -%} +{%- set tables_json_prompt = "Identify and extract the tabls schema\n Extruct the schema of all the tables in the image sorted according to the reading order.\nThe output must be a valid JSON object containing a list of dictionaries with the following structure:\n\n {\n \"dimensions\": {\n \"rows\": ,\n \"columns\": ,\n \"header_rows\": ,\n \"total_rows\": \n },\n \"cells\": [\n {\n \"row\": ,\n \"col\": ,\n \"colspan\": ,\n \"rowspan\": ,\n \"type\": \"<'header' or 'data'>\",\n \"header_level\":
,\n \"content\": \"\"\n },\n ...\n ]\n }" -%} +{%- set tables_html_prompt = "Identify and extract the tabls schema\n Extruct the schema of all the tables in the image sorted according to the reading order.\nThe output must be a list of valid HTML tables" -%} +{%- set tables_otsl_prompt = "Identify and extract the tabls schema\n Extruct the schema of all the tables in the image sorted according to the reading order.\nThe output must be a list of valid OTSL objects, each consists of the following fields: \n - a cell with content in it\n - an empty cell\n - a cell that is merged with the cell to its left\n - a cell that is merged with the cell above it\n - a cell that is merged with both the cell above it and the cell to its left\n - a new line\n - a clumn header\n - the beginning of the OTSL table\n - the end of the OTSL table\n\n An example for an output:\n [\n first table header1first table header2data1data2data with horizontal spandata with vertical spandata3,\n second table header1second table header2data1data2data with horizontal spandata with vertical spandata3\n ]" -%} + + +{#- ===== Tag expansion dispatcher ===== -#} +{%- macro expand_tags(text) -%} +{%- set has_image = "" in text -%} +{#- Determine image position: prefix if appears before the tag, suffix if after -#} +{%- if has_image -%} + {%- set img_idx = text.index("") -%} + {%- if "" in text -%}{%- set tag_idx = text.index("") -%} + {%- elif "" in text -%}{%- set tag_idx = text.index("") -%} + {%- elif "" in text -%}{%- set tag_idx = text.index("") -%} + {%- elif "" in text -%}{%- set tag_idx = text.index("") -%} + {%- elif "" in text -%}{%- set tag_idx = text.index("") -%} + {%- elif "" in text -%}{%- set tag_idx = text.index("") -%} + {%- else -%}{%- set tag_idx = 999999 -%} + {%- endif -%} + {%- set img_prefix = "\n" if img_idx < tag_idx else "" -%} + {%- set img_suffix = "\n" if img_idx >= tag_idx else "" -%} +{%- else -%} + {%- set img_prefix = "" -%} + {%- set img_suffix = "" -%} +{%- endif -%} +{%- if "" in text -%} + {{- img_prefix + chart2code_prompt + img_suffix -}} +{%- elif "" in text -%} + {{- img_prefix + chart2csv_prompt + img_suffix -}} +{%- elif "" in text -%} + {{- img_prefix + chart2summary_prompt + img_suffix -}} +{%- elif "" in text -%} + {{- img_prefix + tables_json_prompt + img_suffix -}} +{%- elif "" in text -%} + {{- img_prefix + tables_html_prompt + img_suffix -}} +{%- elif "" in text -%} + {{- img_prefix + tables_otsl_prompt + img_suffix -}} +{%- else -%} + {{- text -}} +{%- endif -%} +{%- endmacro -%} + +{#- ===== Original chat template ===== -#} +{% macro render_content(x) %} + {%- if x is string %} + {{ x }} + {%- else %} + {%- for chunk in x %} + {%- if chunk['type'] == 'text' -%} + {{ chunk['text']}} + {%- elif chunk['type'] == 'image' -%} + {{- " +" }} + {%- endif -%} + {%- endfor -%} + {%- endif -%} + {% endmacro %} + + {%- set tools_system_message_prefix = 'You are a helpful assistant with access to the following tools. You may call one or more tools to assist with the user query.\n\nYou are provided with function signatures within XML tags:\n' %} +{%- set tools_system_message_suffix = '\n\n\nFor each tool call, return a json object with function name and arguments within XML tags:\n\n{\"name\": , \"arguments\": }\n. If a tool does not exist in the provided list of tools, notify the user that you do not have the ability to fulfill the request.' %} +{%- set documents_system_message_prefix = 'You are a helpful assistant with access to the following documents. You may use one or more documents to assist with the user query.\n\nYou are given a list of documents within XML tags:\n' %} +{%- set documents_system_message_suffix = '\n\n\nWrite the response to the user\'s input by strictly aligning with the facts in the provided documents. If the information needed to answer the question is not available in the documents, inform the user that the question cannot be answered based on the available data.' %} +{%- set g4_default_system_message = 'You are a helpful assistant. Please ensure responses are professional, accurate, and safe.' %} +{%- if available_tools is defined and available_tools %} + {%- set tools = available_tools %} +{%- endif %} +{%- set ns = namespace(tools_system_message=tools_system_message_prefix, + documents_system_message=documents_system_message_prefix, + default_system_message=g4_default_system_message, + system_message='' + ) %} +{%- if tools %} + {%- for tool in tools %} + {%- set ns.tools_system_message = ns.tools_system_message + '\n' + (tool | tojson) %} + {%- endfor %} + {%- set ns.tools_system_message = ns.tools_system_message + tools_system_message_suffix %} +{%- else %} + {%- set ns.tools_system_message = '' %} +{%- endif %} +{%- if documents %} + {%- for document in documents %} + {%- set ns.documents_system_message = ns.documents_system_message + '\n' + (document | tojson) %} + {%- endfor %} + {%- set ns.documents_system_message = ns.documents_system_message + documents_system_message_suffix %} +{%- else %} + {%- set ns.documents_system_message = '' %} +{%- endif %} +{%- if messages[0].role == 'system' %} + {%- if messages[0].content is string %} + {%- set ns.system_message = messages[0].content %} + {%- elif messages[0].content is iterable %} + {%- for entry in messages[0].content %} + {%- if entry.type== 'text' %} + {%- if ns.system_message != '' %} + {%- set ns.system_message = ns.system_message + '\n' %} + {%- endif %} + {%- set ns.system_message = ns.system_message + entry.text %} + {%- endif %} + {%- endfor %} + {%- endif %} + {%- if tools and documents %} + {%- set ns.system_message = ns.system_message + '\n\n' + ns.tools_system_message + '\n\n' + ns.documents_system_message %} + {%- elif tools %} + {%- set ns.system_message = ns.system_message + '\n\n' + ns.tools_system_message %} + {%- elif documents %} + {%- set ns.system_message = ns.system_message + '\n\n' + ns.documents_system_message %} + {%- endif %} +{%- else %} + {%- if tools and documents %} + {%- set ns.system_message = ns.tools_system_message + '\n\n' + ns.documents_system_message %} + {%- elif tools %} + {%- set ns.system_message = ns.tools_system_message %} + {%- elif documents %} + {%- set ns.system_message = ns.documents_system_message %} + {%- endif %} +{%- endif %} +{%- if ns.system_message %} + {{- '<|start_of_role|>system<|end_of_role|>' + ns.system_message + '<|end_of_text|>\n' }} +{%- else %} + {{- '<|start_of_role|>system<|end_of_role|>' + ns.default_system_message + '<|end_of_text|>\n' }} +{%- endif %} +{%- for message in messages %} + {%- set content = namespace(val='') %} + {%- if render_content(message['content']) is string %} + {%- set content.val = render_content(message['content']) %} + {%- else %} + {%- if render_content(message['content']) is iterable %} + {%- for entry in render_content(message['content']) %} + {%- if entry.type== 'text' %} + {%- if content.val != '' %} + {%- set content.val = content.val + '\n' %} + {%- endif %} + {%- set content.val = content.val + entry.text %} + {%- endif %} + {%- endfor %} + {%- endif %} + {%- endif %} + {%- if (message.role == 'user') or (message.role == 'system' and not loop.first) %} + {{- '<|start_of_role|>' + message.role + '<|end_of_role|>' + expand_tags(content.val) + '<|end_of_text|>\n' }} + {%- elif message.role == 'assistant' %} + {{- '<|start_of_role|>' + message.role + '<|end_of_role|>' + content.val }} + {%- if message.tool_calls %} + {%- for tool_call in message.tool_calls %} + {%- if (loop.first and content.val) or (not loop.first) %} + {{- '\n' }} + {%- endif %} + {%- if tool_call.function %} + {%- set tool_call = tool_call.function %} + {%- endif %} + {{- '\n{"name": "' }} + {{- tool_call.name }} + {{- '", "arguments": ' }} + {%- if tool_call.arguments is string %} + {{- tool_call.arguments }} + {%- else %} + {{- tool_call.arguments | tojson }} + {%- endif %} + {{- '}\n' }} + {%- endfor %} + {%- endif %} + {{- '<|end_of_text|>\n' }} + {%- elif message.role == 'tool' %} + {%- if loop.first or (messages[loop.index0 - 1].role != 'tool') %} + {{- '<|start_of_role|>user<|end_of_role|>' }} + {%- endif %} + {{- '\n\n' }} + {{- content.val }} + {{- '\n' }} + {%- if loop.last or (messages[loop.index0 + 1].role != 'tool') %} + {{- '<|end_of_text|>\n' }} + {%- endif %} + {%- endif %} +{%- endfor %} +{%- if add_generation_prompt %} + {{- '<|start_of_role|>assistant<|end_of_role|>' }} +{%- endif %} \ No newline at end of file diff --git a/tests/fixtures/granite_vision_3_2/SOURCE.md b/tests/fixtures/granite_vision_3_2/SOURCE.md new file mode 100644 index 000000000..b5873cc94 --- /dev/null +++ b/tests/fixtures/granite_vision_3_2/SOURCE.md @@ -0,0 +1,13 @@ +# granite_vision_3_2 chat-template fixture + +Source: [mlx-community/granite-vision-3.2-2b-4bit](https://huggingface.co/mlx-community/granite-vision-3.2-2b-4bit/tree/351831ab5218b1f66bbfdfd57bd8f20d81ce8baf) + +Pinned revision: 351831ab5218b1f66bbfdfd57bd8f20d81ce8baf + +License: apache-2.0 (declared by the source model card). +The template is copied verbatim; no checkpoint weights are included. + +- Source [chat_template.jinja](https://huggingface.co/mlx-community/granite-vision-3.2-2b-4bit/resolve/351831ab5218b1f66bbfdfd57bd8f20d81ce8baf/chat_template.jinja), SHA-256: 80693db11342b45ab8393957fcfb29531ce7f61a0ba102a6fb6758fb29fea3fc. +- Source [tokenizer_config.json](https://huggingface.co/mlx-community/granite-vision-3.2-2b-4bit/resolve/351831ab5218b1f66bbfdfd57bd8f20d81ce8baf/tokenizer_config.json), SHA-256: 88a1891e9bc06ab0dc259a2c99ca3aee603c09ca214714f0f686a1cceee2b7aa. + +The checked-in tokenizer_config.json contains only the source file's chat_template value, preserving the competing string template used by the loading regression. All other metadata is omitted. diff --git a/tests/fixtures/granite_vision_3_2/chat_template.jinja b/tests/fixtures/granite_vision_3_2/chat_template.jinja new file mode 100644 index 000000000..b50f63e71 --- /dev/null +++ b/tests/fixtures/granite_vision_3_2/chat_template.jinja @@ -0,0 +1,40 @@ +{%- if tools %} + {{- '<|start_of_role|>available_tools<|end_of_role|> +' }} + {%- for tool in tools %} + {{- tool | tojson(indent=4) }} + {%- if not loop.last %} + {{- ' + +' }} + {%- endif %} + {%- endfor %} + {{- '<|end_of_text|> +' }} +{%- endif %} +{%- for message in messages if message['role'] == 'system'%}{% else %}<|system|> +A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. +{% endfor %}{%- for message in messages %} + {%- if message['role'] == 'system' %} + {{- '<|system|> +' + message['content'][0]['text'] + ' +' }} + {%- elif message['role'] == 'user' %}<|user|> + {# Render all images first #}{% for content in message['content'] | selectattr('type', 'equalto', 'image') %}{{ ' +' }}{% endfor %}{# Render all text next #}{% for content in message['content'] | selectattr('type', 'equalto', 'text') %}{{ content['text'] + ' +' }}{% endfor %} +{%- elif message['role'] == 'assistant' %} + {{- '<|assistant|> +' + message['content'][0]['text'] + '<|end_of_text|>' }} + {%- elif message['role'] == 'assistant_tool_call' %} + {{- '<|start_of_role|>assistant<|end_of_role|><|tool_call|>' + message['content'][0]['text'] + '<|end_of_text|> +' }} + {%- elif message['role'] == 'tool_response' %} + {{- '<|start_of_role|>tool_response<|end_of_role|>' + message['content'][0]['text'] + '<|end_of_text|> +' }} + {%- endif %} + {%- if loop.last and add_generation_prompt %} + {{- '<|assistant|> +' }} + {%- endif %} +{%- endfor %} \ No newline at end of file diff --git a/tests/fixtures/granite_vision_3_2/tokenizer_config.json b/tests/fixtures/granite_vision_3_2/tokenizer_config.json new file mode 100644 index 000000000..6aa6cffdf --- /dev/null +++ b/tests/fixtures/granite_vision_3_2/tokenizer_config.json @@ -0,0 +1,3 @@ +{ + "chat_template": "{%- if tools %}\n {{- '<|start_of_role|>available_tools<|end_of_role|>\n' }}\n {%- for tool in tools %}\n {{- tool | tojson(indent=4) }}\n {%- if not loop.last %}\n {{- '\n\n' }}\n {%- endif %}\n {%- endfor %}\n {{- '<|end_of_text|>\n' }}\n{%- endif %}\n{%- for message in messages if message['role'] == 'system'%}{% else %}<|system|>\nA chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions.\n{% endfor %}{%- for message in messages %}\n {%- if message['role'] == 'system' %}\n {{- '<|system|>\n' + message['content'] + '\n' }}\n {%- elif message['role'] == 'user' %}\n {{- '<|user|>\n' + message['content'] + '\n' }}\n {%- elif message['role'] == 'assistant' %}\n {{- '<|assistant|>\n' + message['content'] + '<|end_of_text|>' }}\n {%- elif message['role'] == 'assistant_tool_call' %}\n {{- '<|start_of_role|>assistant<|end_of_role|><|tool_call|>' + message['content'] + '<|end_of_text|>\n' }}\n {%- elif message['role'] == 'tool_response' %}\n {{- '<|start_of_role|>tool_response<|end_of_role|>' + message['content'] + '<|end_of_text|>\n' }}\n {%- endif %}\n {%- if loop.last and add_generation_prompt %}\n {{- '<|assistant|>\n' }}\n {%- endif %}\n{%- endfor %}" +}