feat(LTX25-UPSAMPLER-ARMS): port the upsampler's dims=2 arm, and mirror model.py:47's else - #2582
Open
localai-org-maint-bot wants to merge 6 commits into
Open
feat(LTX25-UPSAMPLER-ARMS): port the upsampler's dims=2 arm, and mirror model.py:47's else#2582localai-org-maint-bot wants to merge 6 commits into
localai-org-maint-bot wants to merge 6 commits into
Conversation
…ral one is a decision (#2577) `ltx25-completion-scope.md` sized A8 and A9 together at order 6, S each, and called A8 the cheapest of the two. That sizing is right about the operator and does not survive contact with the consumer, which is what this spec records. A9, the `dims == 2` arm, is a port. Upstream chooses `Conv2d` over `Conv3d` at model.py:47 and folds the frame axis into the batch at :86, which has exactly two consequences: four parameter groups become 4-D kernels, and GroupNorm's statistics become per frame. Its output keeps the frame count, which is what both spatial call sites require, so it is reachable AND consumable and lands here. A8 is returned undecided. Its operator is small, but it returns [c, 2f-1, 2h, 2w] against two consumers that require the frame count back unchanged, so porting it would replace a named refusal with a generic shape complaint for every clip longer than eight frames. That is a product decision rather than a porting question, and it is listed under `## Owed` instead of guessed. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…or model.py:47's else (#2577) `Ltx2LatentUpsample` refused every `dims != 3` config. `dims` is read off the checkpoint's own metadata, mirroring `model_configurator.py:17`, so a 2-D upsampler was an ordinary input this engine turned away. Upstream's `conv = torch.nn.Conv2d if dims == 2 else torch.nn.Conv3d` (model.py:47) reaches four parameter groups, and its forward folds the frame axis into the batch (:86) and unfolds it at :100. That fold has exactly two consequences and both are the port: the four groups become 4-D kernels, and GroupNorm normalises PER FRAME rather than over the clip. This runs one frame at a time, which reproduces the fold rather than approximating it -- a one-frame volume hands the existing reduction over `frames * height * width` precisely the per-frame statistic. The stack is now one lambda for both ranks, because upstream's two branches run the identical module sequence and differ only in the fold around them. `model.py:47`'s `else` is mirrored as written, so any `dims` that is not 2 builds Conv3d. The old `Require(config.dims == 3, ...)` refused checkpoints upstream runs, which was a refusal this port invented. What survives is the one 2-D combination upstream cannot run either: `temporal_upsample` builds a Conv3d (:68-71) that the 2-D forward would hand a 4-D tensor. Reached and consumable, not merely computed. The arm returns the frame count unchanged with H and W doubled, which is what both spatial call sites require -- `vshape.frames` at ltx2_video.cpp:3509-3516 and `slot_positions.size()` at :3536-3548 -- so the new end-to-end case renders a full-size clip through `LoadVideoEngine`/`Generate` instead of asserting a changed error message. Red first: the dims=2 golden failed on six parameter counts (5184 vs 1728, 27648 vs 9216) and then threw `dims=2 is not ported`. Two mutations, both restored byte-for-byte. Forcing `two_d = false` reds the golden AND the render, which is the reachability proof. Dropping the fold while keeping the 2-D convolutions reds the golden at max|diff| 0.679842 and leaves the render GREEN, because every shape is identical -- the defect this file's header warns about, and the reason the value gate is the one that holds it. A8, the spatiotemporal arm, is NOT ported here. Its operator is small, but it returns `[c, 2f-1, 2h, 2w]` against two consumers that need the frame count back, so porting it would trade a named refusal for a generic shape complaint on every clip over eight frames. That is a product decision, returned under `## Owed` rather than guessed. Also corrects two claims in `ltx2_upsampler.h` that the tree falsifies (#2580): there are three upsampler call sites, not one, and DFR's rounds loop drives the temporal arm the header called unreachable. Closes #2577 Closes #2580 FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…URES rows by hand `dd2123f69` landed the DFlash2 EXL3 draft row. The only file both sides touch is `docs/FEATURES.md`, which is a keyed record, so its automatic three-way merge was discarded: the target-branch version was taken whole and this row's two scoped edits reapplied on top. Against `origin/main` the file is now 2 added lines and 1 rewritten one, which is exactly this row's own pair and no unrelated key. Nothing else overlaps, and no claim in this branch's prose reads on the DFlash2 draft loader. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…ck the conv2d bias, and refuse dims=2 at load (#2577) Five review findings, of which three were defects in this row's own change. The RATIONAL contradiction is the one that mattered. `dims=2` with `rational_resampler` is a configuration upstream cannot run -- `SpatialRationalResampler.forward` unpacks five dimensions at spatial_rational_resampler.py:41 and the dims=2 forward reaches it at model.py:94 having folded the frame axis away at :86 -- but every operator in that branch is already per-frame, so a folded one-frame volume satisfies all of them and this port computed a complete, finite, plausible latent instead. No shape check anywhere could fault it. That is the same failure mode this row's own mutation B demonstrated, left open one branch over. Refused by name. `Conv2dPad1PerFrame` checked its weight and not its bias, where the sibling `Conv3dPad1` has checked both since #644. A checkpoint with a correct kernel and a short bias indexed past the end of a std::vector. The dims=2 arm routes four parameter groups through that helper, so this change is what made the gap reachable, and nothing upstream of it validates shapes: `Ltx2LoadVaeWeights` loads by NAME. The dims=2 temporal checkpoint was refused, but from inside the rounds loop, which is reached only after two full denoise stages. The `temporal_upsampler_path` load block exists so a caller learns at LOAD, says so in its own comment, and checked both flags without checking `dims`. Guard added beside its two siblings. Both contradictions are now EXECUTED against the real upstream module by `scripts/gen-ltx2-pipeline-goldens.py`, which asserts each one raises and what it says, and emits nothing -- the golden is byte-identical. That gate is what stops either refusal from drifting into one upstream would happily serve. It also corrects a mechanism this row asserted and had not measured. `Conv3d` does NOT raise on the rank: it accepts a 4-D input as an unbatched 5-D one, so the CHANNEL count is what disagrees ("expected input[1, 3, 32, 4, 6] to have 32 channels, but got 3"). At `frames == mid_channels` the conv passes and PixelShuffleND(1) fails instead. The conclusion held and the reasoning did not. The reader anchors in ltx2_video.cpp moved by the 16 lines the load guard adds, and are repasted from the gate's own derived list. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…pairs 35 commits, the Qwen3.5 EXL3 head row and the ROCm managed-alloc fix among them. No incoming commit touches this row's code: `git log HEAD..origin/main --` over the upsampler, its header, the golden generator and the goldens returns nothing. `docs/FEATURES.md` is the one shared file and is a keyed record, so its automatic three-way merge was discarded again: the target-branch version was taken whole and this row's two scoped edits reapplied on top, with both anchors re-asserted as unique first. Against `origin/main` the file is 2 added lines and 1 rewritten one, which is this row's own pair and no unrelated key. Re-verified after the merge rather than assumed: the goldens regenerate BYTE-IDENTICALLY from the pinned upstream, so nothing this row measured moved under it. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
… row's own guard moved The load-time guard adds 16 lines near the top of `ltx2_video.cpp`, so every anchor below it moved by exactly 16 — including the ones this spec, the header and two issues use to argue A8's case. They were correct when written and wrong when pushed, with no edit in between. `test_ltx2_video`'s recorded-reader-anchor gate catches its own list and nothing catches prose, so these were re-derived by grepping for the statements themselves rather than by adding 16 to each: the three call sites are :3521, :3548 and :5058, their shape checks :3525-3531 and :3552-3563, `vshape.frames` :3411, and the temporal-only guard :3473-3502 with its D11b sentence at :3499-3501. Recorded in the spec because a line anchor is a claim about a file that a later hunk in the SAME change can falsify, and this row falsified its own. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
mudler
force-pushed
the
row/LTX25-UPSAMPLER-ARMS
branch
from
September 2, 2026 16:06
bd7d5b5 to
34edb89
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Port the latent upsampler's
dims == 2arm, which the second phase can already consume.Ltx2LatentUpsamplerefused everydims != 3config.dimsis read off thecheckpoint's own metadata, mirroring
model_configurator.py:17, so a 2-Dupsampler was an ordinary input this engine turned away.
What upstream builds, and what that costs
conv = torch.nn.Conv2d if dims == 2 else torch.nn.Conv3d(model.py:47)reaches four parameter groups --
initial_conv(:49), both ResBlock stacks(
:53,:76-78, viares_block.py:21) andfinal_conv(:80). The forwardfolds the frame axis into the batch (
:86) and unfolds it at:100.That fold has exactly two consequences and both are the port:
EnumerateLtx2UpsamplerTensorsemits different shapes;
(b f)makes every frame its ownsample. Our reduction runs over
frames * height * width.This runs one frame at a time, which reproduces the fold rather than
approximating it: a one-frame volume hands the existing reduction precisely the
per-frame statistic, so no second normaliser exists to drift.
The stack became one lambda for both ranks, because upstream's two branches run
the identical module sequence and differ only in the fold around them.
model.py:47'selseis mirrored as written, so anydimsthat is not 2 buildsConv3d. The old
Require(config.dims == 3, ...)refused checkpoints upstreamruns -- a refusal this port had invented.
The two contradictions that stay refused, and why one of them mattered
dims=2withtemporal_upsampleanddims=2withrational_resamplerareconfigurations upstream cannot run. Both are now executed against the real
module by
scripts/gen-ltx2-pipeline-goldens.py, which asserts each raises andwhat it says, and emits nothing -- so the golden stays byte-identical while the
refusals are pinned to upstream behaviour rather than to reasoning about it.
The rational one is the finding. Every operator in that branch is already
per-frame, so a folded one-frame volume satisfies all of them and this port
computed a complete, finite, plausible latent where upstream raises
not enough values to unpack (expected 5, got 4)(
spatial_rational_resampler.py:41). No shape check anywhere could fault it.That is the same failure mode this row's own mutation B demonstrated, left open
one branch over, and it took a review to find.
It also corrects a mechanism this row asserted without measuring.
Conv3ddoesnot raise on the rank: it accepts a 4-D input as an unbatched 5-D one, so the
channel count is what disagrees --
expected input[1, 3, 32, 4, 6] to have 32 channels, but got 3. Atframes == mid_channelsthe conv passes andPixelShuffleND(1)fails instead. The conclusion held; the reasoning did not.A third repair moves the
dims=2temporal refusal to load time. It wasrefused, but from inside the rounds loop, reached only after two full denoise
stages -- while the
temporal_upsampler_pathblock exists precisely so a callerlearns at load, and says so in its own comment.
A fourth gives
Conv2dPad1PerFramethe bias check its siblingConv3dPad1hashad since #644. A correct kernel with a short bias indexed past the end of a
std::vector; thedims=2arm routes four parameter groups through that helper,so this change is what made the gap reachable.
Reached, and consumable rather than merely computed
Ltx2UpsampleVideoLatenthas three product call sites. Both spatial onesrequire the frame count back unchanged --
vshape.framesatltx2_video.cpp:3525-3531andslot_positions.size()at:3552-3563. Thedims=2arm returns exactly that, with H and W doubled, so the end-to-end caserenders a full-size clip through
LoadVideoEngine/Generateinstead ofasserting a changed error message.
Evidence
Red first: the
dims=2golden failed on six parameter counts (5184vs1728for
initial_conv,27648vs9216for each ResBlock conv) and then threwltx2 upsampler: dims=2 is not ported.Two mutations, each restored byte-for-byte and sha256-verified:
const bool two_d = falsetest_ltx2_video.cpp:690max|diff| = 0.679842The generator's new assertions are armed, not decorative: replacing the expected
substring makes it raise and quote the real upstream exception.
Tests are ported by executing the real upstream module at the
fd4ded7fpin.403 golden lines added, no existing golden moved, oracle revision constants
unchanged, and the goldens regenerate byte-identically after the merge.
A8 is deliberately not here, and now has its own issue
The spatiotemporal arm returns
[c, 2f-1, 2h, 2w]against the two consumersabove that need the frame count back. They agree only when
vshape.frames == 1,i.e.
frames <= 8, where the doubling is cancelled by the mandatory first-framedrop. Porting it would trade a named refusal for a generic shape complaint on
every real clip -- the substitution the guard at
ltx2_video.cpp:3473-3502exists to prevent.
That is a product decision, so it is tracked by #2584 and listed under
## Owed. It was originally recorded against #2577, which this PR closes, so theowed item would have pointed at a closed issue the moment it landed;
check-agent-record.pycould not have caught that, because the## Owedentrysatisfies the gate on its own.
This reverses the scope document's sizing, which called A8 the cheaper of the
two. It is right about the operator and had not counted the second consumer.
That document is operator-owned and is not edited from here; the contradiction
is recorded in the row spec.
Also
Corrects two claims in
ltx2_upsampler.hthat the tree falsifies (#2580): thereare three upsampler call sites, not one, and DFR's rounds loop drives the
temporal arm the header called unreachable. The same false statement survives in
a user-facing
Fail()atltx2_video.cpp:3499-3501, already tracked as D11b andleft to its owner.
Every
ltx2_video.cppanchor in this branch's prose moved by the 16 lines theload guard adds and was re-derived by grepping for the statements themselves.
docs/FEATURES.mdis a keyed record, so its automatic three-way merge wasdiscarded on both merges and the two scoped rows reapplied by hand.
Closes #2577
Closes #2580
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]