Skip to content

fix: preserve text in typed-content checkpoint templates - #2165

Merged
inureyes merged 3 commits into
lablup:mainfrom
Jaeyeong-CHOI:fix/2135-preserve-typed-chat-text
Oct 12, 2026
Merged

inureyes merged 3 commits into
lablup:mainfrom
Jaeyeong-CHOI:fix/2135-preserve-typed-chat-text

Conversation

@Jaeyeong-CHOI

@Jaeyeong-CHOI Jaeyeong-CHOI commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Granite Vision 3.2 selects its standalone multimodal template, but that template silently drops plain-string message content. The loader now compares one string probe with one typed-text probe and stores a compatibility flag only when the typed form preserves the marker.

Typed ChatMessage renders for affected checkpoints use a borrowed typed-text view in apply_inner. Raw JSON requests, explicit template overrides, default processors, and string-compatible templates keep their existing content contracts.

What changed

  • Add selection-time detection with conservative error handling and no per-request probing.
  • Serialize affected messages directly from borrowed role/text fields without an intermediate JSON tree or copied strings.
  • Add eight focused regressions with pinned Granite Vision 3.2 and Granite 4 fixtures, plus English and Korean technical reports.

Validation

  • Combined CUDA graph targeted tests: 8 passed, 0 failed (text_wrap_tests); 9,105 unrelated mlxcel tests and 160 surgery tests were filtered out.
  • Repository contract gate: version, kernel, llama compatibility, ROCm overlay, 123 Python tooling tests, and 49 WebUI fixtures passed.
  • Real mlx-community/granite-vision-3.2-2b-4bit revision 351831ab5218b1f66bbfdfd57bd8f20d81ce8baf on NVIDIA GB10/CUDA: CLI answered Paris in 8 tokens; server string and typed-text requests both answered Paris with 55 prompt tokens and 8 completion tokens. The frozen baseline reproduced the 47-token prompt and unrelated greeting.
  • Implementation review, security rereview, and static finalization found no remaining issues.
  • Full workspace test and Clippy suites, hosted CI, ROCm inference, and macOS inference are outside this issue unit; the parent integration gate owns broad validation.

Reports

  • TECHNICAL_REPORTS/2165-typed-chat-text-20261013.en.md
  • TECHNICAL_REPORTS/2165-typed-chat-text-20261013.ko.md

Closes #2135

Fixes #2135

Jaeyeong-CHOI and others added 3 commits October 7, 2026 00:27
Normalize eligible string messages at the checkpoint template boundary.
Preserve raw conversation fields and explicit template input contracts,
with conservative probes and focused regression fixtures.

Fixes lablup#2135
Detect selected checkpoint templates that render typed text but drop strings, then adapt typed ChatMessage renders through a borrowed text-part view. Keep raw JSON and string-compatible template contracts unchanged, and cover Granite Vision 3.2 and Granite 4 with pinned fixtures.

Validated with eight focused CUDA-profile tests, repository contract checks, and real Granite CLI and server requests on GB10.

Fixes lablup#2135
@inureyes inureyes changed the title fix: preserve text in typed-content chat templates fix: preserve text in typed-content checkpoint templates Oct 12, 2026
@inureyes inureyes added status:review Under review type:bug Bug fixes, error corrections, or issue resolutions priority:high High priority area:models Model architectures, weights, loading, metadata area:inference Generation, sampling, decoding (incl. speculative, DRY) area:cli Command-line interface / CLI flags modeltype:vlm Vision-language model status:done Completed and removed status:review Under review labels Oct 12, 2026
@inureyes
inureyes merged commit 1e8c702 into lablup:main Oct 12, 2026
27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:cli Command-line interface / CLI flags area:inference Generation, sampling, decoding (incl. speculative, DRY) area:models Model architectures, weights, loading, metadata modeltype:vlm Vision-language model priority:high High priority status:done Completed type:bug Bug fixes, error corrections, or issue resolutions

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix: granite-vision text-only requests drop the user message

2 participants