Skip to content

dots3-note: the image processor refuses instead of resizing, so no non-conformant image is servable #2537

Description

@localai-org-maint-bot

Row: MODEL-MM-dots3-note-dots3-note-for-causal-lm

Dots3NoteImageProcessor::ProcessImage refuses any image whose resized size
differs from the size it was handed, instead of resizing it. Upstream always
resizes, so on the released geometry this refuses essentially every real user
image.

What the port does

src/vllm/multimodal/dots3_note_processor.cpp, in ProcessImage:

const std::array<int64_t, 2> rs =
    Dots3NoteResizedSize(height, width, f, cfg_.min_pixels, cfg_.max_pixels);
const int64_t rh = rs[0], rw = rs[1];
if (rh != height || rw != width) {
  throw std::runtime_error(
      "Dots3NoteImageProcessor: image requires resize (...); the bicubic "
      "resize path is not ported (W6a uses conformant images).");
}

Dots3NoteResizedSize is ported and correct: it mirrors resized_size
(common/processor.py:97 @ 9035151d6) including the min_pixels and
max_pixels rebalance. What is missing is the step after it, the actual
Image.Resampling.BICUBIC resample at processor.py:174. W6a is served only
conformant fixture images, so nothing on the row's gate reaches it.

Why it matters beyond bookkeeping

factor = patch_size * merge_size, and on dots-studio/dots3-note-prev that is
14 * 2 = 28. An image is servable today only when both of its dimensions are
already multiples of 28 AND its pixel count already sits inside
[min_pixels, max_pixels]. Photographs, screenshots and every common
resolution fail that. Once W6b lifts the MoE ViT refusal and the released
checkpoint becomes servable, this refusal is what a user hits instead.

Patchifying at the wrong grid would be worse than refusing: it changes the
placeholder count and serves a well-shaped wrong prompt, on a row that
.agents/specs/dots3-note.md §6.4 records as having no oracle to catch it. The
refusal is the right interim behaviour. It is the missing capability that is
the debt.

What closing this needs

  • The bicubic resample itself, matched to PIL's Image.Resampling.BICUBIC
    the same convolution kernel (a = -0.5), the same support radius, the same
    per-axis two-pass order and the same clamping — because a "close enough"
    resampler moves every patch and this row has no token-exact denominator to
    catch the difference.
  • A gate that measures the resampler against the reference on a non-conformant
    image, not merely that the placeholder count comes out right: a shape-valid
    gate passes a wrong artifact.
  • The refusal deleted in the same change, so the message and the behaviour
    cannot drift apart.

Record

Owner: this row, W8 (the MM front end brick that owns the processor;
.agents/specs/dots3-note.md §7). Recorded under ## Owed in that spec.
Found by the fresh review of #2523 (W6a), whose finding was that the code
comment and the runtime message both claimed the debt was already recorded
under ## Owed when it was recorded nowhere.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions