feat(webv2): surface Ref2VA image-reference cost, tier the detail defaults - #209
Open
lstein wants to merge 1 commit into
Open
feat(webv2): surface Ref2VA image-reference cost, tier the detail defaults#209lstein wants to merge 1 commit into
lstein wants to merge 1 commit into
Conversation
…48px defaults An image reference is patchified like any frame -- width * height / 1024 rows at H3's 16x spatial compression and 2x2 patch -- and those rows are re-attended at EVERY denoising step, with attention quadratic in the sequence length. "Max detail" pins the short edge to 2048 whatever the generation size is, so a 1920x1080 reference normalizes to 3648x2048 and adds 7,296 rows: +19% on a 1344x768 x 124-frame request (~1.8 GiB by the denoise node's 0.25 MiB/row model, ~1.4x the attention work), and +34% on a 768x768 one. "Match generation size" costs ~1,000 rows for the same image. Nothing in the panel said any of this before the queue. Two changes: - Each image reference card now shows the size the graph will encode and the rows it adds, live as the detail or the canvas changes. `resolveMiniMaxH3ReferenceImage` mirrors the backend's `resolve_reference_image_short_edge` + `normalize_reference_image`, banker's rounding included, so the estimate is the real size; its table test is cross-checked against the Python implementation. - The FIRST image reference still defaults to "max" -- upstream's rule, and the primary subject is where the detail earns its cost -- while later ones default to "match". Three 2048px references are what turn a modest surcharge into a doubling, and the marginal value of a supporting reference at seven times the output's pixel density is small. The help text says so, and the per-card row count makes it visible. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S4B5exWsbC2z2Uu2tA167A
lstein
requested review from
JPPhoto,
Pfannkuchensack and
blessedcoolant
as code owners
September 4, 2026 13:10
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The image-reference detail choice ("Max detail (2048px)" vs "Match generation size") is a much bigger performance lever than it looks, and the panel gave no hint of that.
An image reference is patchified like any frame —
width × height / 1024rows at H3's 16× spatial compression and 2×2 patch — and those rows are re-attended at every denoising step, with attention quadratic in the sequence length.maxpins the short edge to 2048 regardless of the generation size, so the cost is worst exactly when the output is smallest.Measured with the real code paths (124 frames, the UI default; VRAM from the denoise node's own 0.25 MiB/row model):
maxmatchThree
maxreferences add ~16k rows — roughly 2× the attention work and ~+4 GiB.What
1. Each image reference card shows its cost. The size the graph will encode plus the rows it adds, under the detail selector, updating live as the detail or the canvas changes.
resolveMiniMaxH3ReferenceImage(infeatures/video/core/dimensions.ts, beside the existing canvas ports) mirrors the backend'sresolve_reference_image_short_edge+normalize_reference_image— banker's rounding included, via the module's existingroundHalfToEven— so this is the real size, not an approximation. Null (and nothing rendered) whenmatchhas no canvas to match.2. Tiered detail defaults. The first image reference still starts at
max— upstream's rule, and the primary subject is where the extra detail earns its cost — while later ones start atmatch. The backend node's own default is unchanged atmax; this is a panel convenience for the multi-reference case, and both the help text and the new per-card row count make it visible rather than surprising.Testing
resolveMiniMaxH3ReferenceImage: an 8-row table cross-checked against the Python implementation's output for the same inputs, plus the 2048 clamp, the no-target-area cases, and degenerate input.getDefaultReferenceImageDetail: first-image / later-image / video-only-list cases.pnpm run check:architecturegreen locally (509 unit + 158 browser files, perf and project-file gates included).Independent of #206; both touch
VideoReferenceListField.tsx, so whichever lands second will want a trivial merge.🤖 Generated with Claude Code
https://claude.ai/code/session_01S4B5exWsbC2z2Uu2tA167A