Skip to content

feat(LTX25-UPSAMPLER-ARMS): port the upsampler's dims=2 arm, and mirror model.py:47's else - #2582

Open
localai-org-maint-bot wants to merge 6 commits into
mainfrom
row/LTX25-UPSAMPLER-ARMS
Open

feat(LTX25-UPSAMPLER-ARMS): port the upsampler's dims=2 arm, and mirror model.py:47's else#2582
localai-org-maint-bot wants to merge 6 commits into
mainfrom
row/LTX25-UPSAMPLER-ARMS

Conversation

@localai-org-maint-bot

@localai-org-maint-bot localai-org-maint-bot commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator

Port the latent upsampler's dims == 2 arm, which the second phase can already consume.

Ltx2LatentUpsample refused every dims != 3 config. dims is read off the
checkpoint's own metadata, mirroring model_configurator.py:17, so a 2-D
upsampler was an ordinary input this engine turned away.

What upstream builds, and what that costs

conv = torch.nn.Conv2d if dims == 2 else torch.nn.Conv3d (model.py:47)
reaches four parameter groups -- initial_conv (:49), both ResBlock stacks
(:53, :76-78, via res_block.py:21) and final_conv (:80). The forward
folds the frame axis into the batch (:86) and unfolds it at :100.

That fold has exactly two consequences and both are the port:

  • the four groups become 4-D kernels, so EnumerateLtx2UpsamplerTensors
    emits different shapes;
  • GroupNorm normalises per frame, because (b f) makes every frame its own
    sample. Our reduction runs over frames * height * width.

This runs one frame at a time, which reproduces the fold rather than
approximating it: a one-frame volume hands the existing reduction precisely the
per-frame statistic, so no second normaliser exists to drift.

The stack became one lambda for both ranks, because upstream's two branches run
the identical module sequence and differ only in the fold around them.

model.py:47's else is mirrored as written, so any dims that is not 2 builds
Conv3d. The old Require(config.dims == 3, ...) refused checkpoints upstream
runs -- a refusal this port had invented.

The two contradictions that stay refused, and why one of them mattered

dims=2 with temporal_upsample and dims=2 with rational_resampler are
configurations upstream cannot run. Both are now executed against the real
module
by scripts/gen-ltx2-pipeline-goldens.py, which asserts each raises and
what it says, and emits nothing -- so the golden stays byte-identical while the
refusals are pinned to upstream behaviour rather than to reasoning about it.

The rational one is the finding. Every operator in that branch is already
per-frame, so a folded one-frame volume satisfies all of them and this port
computed a complete, finite, plausible latent where upstream raises
not enough values to unpack (expected 5, got 4)
(spatial_rational_resampler.py:41). No shape check anywhere could fault it.
That is the same failure mode this row's own mutation B demonstrated, left open
one branch over, and it took a review to find.

It also corrects a mechanism this row asserted without measuring. Conv3d does
not raise on the rank: it accepts a 4-D input as an unbatched 5-D one, so the
channel count is what disagrees -- expected input[1, 3, 32, 4, 6] to have 32 channels, but got 3. At frames == mid_channels the conv passes and
PixelShuffleND(1) fails instead. The conclusion held; the reasoning did not.

A third repair moves the dims=2 temporal refusal to load time. It was
refused, but from inside the rounds loop, reached only after two full denoise
stages -- while the temporal_upsampler_path block exists precisely so a caller
learns at load, and says so in its own comment.

A fourth gives Conv2dPad1PerFrame the bias check its sibling Conv3dPad1 has
had since #644. A correct kernel with a short bias indexed past the end of a
std::vector; the dims=2 arm routes four parameter groups through that helper,
so this change is what made the gap reachable.

Reached, and consumable rather than merely computed

Ltx2UpsampleVideoLatent has three product call sites. Both spatial ones
require the frame count back unchanged -- vshape.frames at
ltx2_video.cpp:3525-3531 and slot_positions.size() at :3552-3563. The
dims=2 arm returns exactly that, with H and W doubled, so the end-to-end case
renders a full-size clip through LoadVideoEngine/Generate instead of
asserting a changed error message.

Evidence

Red first: the dims=2 golden failed on six parameter counts (5184 vs 1728
for initial_conv, 27648 vs 9216 for each ResBlock conv) and then threw
ltx2 upsampler: dims=2 is not ported.

Two mutations, each restored byte-for-byte and sha256-verified:

Mutation Golden End-to-end render What it proves
const bool two_d = false RED RED at test_ltx2_video.cpp:690 The production path really enters the new branch. A render that fails, not a unit test
keep the 2-D convolutions, drop the fold RED, max|diff| = 0.679842 GREEN The fold's only consequence is numerical. Every shape is identical, so no shape check can see it and the value golden is the sole gate

The generator's new assertions are armed, not decorative: replacing the expected
substring makes it raise and quote the real upstream exception.

Tests are ported by executing the real upstream module at the fd4ded7f pin.
403 golden lines added, no existing golden moved, oracle revision constants
unchanged, and the goldens regenerate byte-identically after the merge.

A8 is deliberately not here, and now has its own issue

The spatiotemporal arm returns [c, 2f-1, 2h, 2w] against the two consumers
above that need the frame count back. They agree only when vshape.frames == 1,
i.e. frames <= 8, where the doubling is cancelled by the mandatory first-frame
drop. Porting it would trade a named refusal for a generic shape complaint on
every real clip -- the substitution the guard at ltx2_video.cpp:3473-3502
exists to prevent.

That is a product decision, so it is tracked by #2584 and listed under
## Owed. It was originally recorded against #2577, which this PR closes, so the
owed item would have pointed at a closed issue the moment it landed;
check-agent-record.py could not have caught that, because the ## Owed entry
satisfies the gate on its own.

This reverses the scope document's sizing, which called A8 the cheaper of the
two. It is right about the operator and had not counted the second consumer.
That document is operator-owned and is not edited from here; the contradiction
is recorded in the row spec.

Also

Corrects two claims in ltx2_upsampler.h that the tree falsifies (#2580): there
are three upsampler call sites, not one, and DFR's rounds loop drives the
temporal arm the header called unreachable. The same false statement survives in
a user-facing Fail() at ltx2_video.cpp:3499-3501, already tracked as D11b and
left to its owner.

Every ltx2_video.cpp anchor in this branch's prose moved by the 16 lines the
load guard adds and was re-derived by grepping for the statements themselves.
docs/FEATURES.md is a keyed record, so its automatic three-way merge was
discarded on both merges and the two scoped rows reapplied by hand.

Closes #2577
Closes #2580

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]

…ral one is a decision (#2577)

`ltx25-completion-scope.md` sized A8 and A9 together at order 6, S each, and
called A8 the cheapest of the two. That sizing is right about the operator and
does not survive contact with the consumer, which is what this spec records.

A9, the `dims == 2` arm, is a port. Upstream chooses `Conv2d` over `Conv3d` at
model.py:47 and folds the frame axis into the batch at :86, which has exactly
two consequences: four parameter groups become 4-D kernels, and GroupNorm's
statistics become per frame. Its output keeps the frame count, which is what
both spatial call sites require, so it is reachable AND consumable and lands
here.

A8 is returned undecided. Its operator is small, but it returns [c, 2f-1, 2h,
2w] against two consumers that require the frame count back unchanged, so
porting it would replace a named refusal with a generic shape complaint for
every clip longer than eight frames. That is a product decision rather than a
porting question, and it is listed under `## Owed` instead of guessed.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]
…or model.py:47's else (#2577)

`Ltx2LatentUpsample` refused every `dims != 3` config. `dims` is read off the
checkpoint's own metadata, mirroring `model_configurator.py:17`, so a 2-D
upsampler was an ordinary input this engine turned away.

Upstream's `conv = torch.nn.Conv2d if dims == 2 else torch.nn.Conv3d`
(model.py:47) reaches four parameter groups, and its forward folds the frame
axis into the batch (:86) and unfolds it at :100. That fold has exactly two
consequences and both are the port: the four groups become 4-D kernels, and
GroupNorm normalises PER FRAME rather than over the clip. This runs one frame at
a time, which reproduces the fold rather than approximating it -- a one-frame
volume hands the existing reduction over `frames * height * width` precisely the
per-frame statistic.

The stack is now one lambda for both ranks, because upstream's two branches run
the identical module sequence and differ only in the fold around them.

`model.py:47`'s `else` is mirrored as written, so any `dims` that is not 2 builds
Conv3d. The old `Require(config.dims == 3, ...)` refused checkpoints upstream
runs, which was a refusal this port invented. What survives is the one 2-D
combination upstream cannot run either: `temporal_upsample` builds a Conv3d
(:68-71) that the 2-D forward would hand a 4-D tensor.

Reached and consumable, not merely computed. The arm returns the frame count
unchanged with H and W doubled, which is what both spatial call sites require --
`vshape.frames` at ltx2_video.cpp:3509-3516 and `slot_positions.size()` at
:3536-3548 -- so the new end-to-end case renders a full-size clip through
`LoadVideoEngine`/`Generate` instead of asserting a changed error message.

Red first: the dims=2 golden failed on six parameter counts (5184 vs 1728, 27648
vs 9216) and then threw `dims=2 is not ported`. Two mutations, both restored
byte-for-byte. Forcing `two_d = false` reds the golden AND the render, which is
the reachability proof. Dropping the fold while keeping the 2-D convolutions reds
the golden at max|diff| 0.679842 and leaves the render GREEN, because every shape
is identical -- the defect this file's header warns about, and the reason the
value gate is the one that holds it.

A8, the spatiotemporal arm, is NOT ported here. Its operator is small, but it
returns `[c, 2f-1, 2h, 2w]` against two consumers that need the frame count back,
so porting it would trade a named refusal for a generic shape complaint on every
clip over eight frames. That is a product decision, returned under `## Owed`
rather than guessed.

Also corrects two claims in `ltx2_upsampler.h` that the tree falsifies (#2580):
there are three upsampler call sites, not one, and DFR's rounds loop drives the
temporal arm the header called unreachable.

Closes #2577
Closes #2580

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]
…URES rows by hand

`dd2123f69` landed the DFlash2 EXL3 draft row. The only file both sides touch is
`docs/FEATURES.md`, which is a keyed record, so its automatic three-way merge was
discarded: the target-branch version was taken whole and this row's two scoped
edits reapplied on top. Against `origin/main` the file is now 2 added lines and 1
rewritten one, which is exactly this row's own pair and no unrelated key.

Nothing else overlaps, and no claim in this branch's prose reads on the DFlash2
draft loader.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]
…ck the conv2d bias, and refuse dims=2 at load (#2577)

Five review findings, of which three were defects in this row's own change.

The RATIONAL contradiction is the one that mattered. `dims=2` with
`rational_resampler` is a configuration upstream cannot run --
`SpatialRationalResampler.forward` unpacks five dimensions at
spatial_rational_resampler.py:41 and the dims=2 forward reaches it at
model.py:94 having folded the frame axis away at :86 -- but every operator in
that branch is already per-frame, so a folded one-frame volume satisfies all of
them and this port computed a complete, finite, plausible latent instead. No
shape check anywhere could fault it. That is the same failure mode this row's
own mutation B demonstrated, left open one branch over. Refused by name.

`Conv2dPad1PerFrame` checked its weight and not its bias, where the sibling
`Conv3dPad1` has checked both since #644. A checkpoint with a correct kernel and
a short bias indexed past the end of a std::vector. The dims=2 arm routes four
parameter groups through that helper, so this change is what made the gap
reachable, and nothing upstream of it validates shapes: `Ltx2LoadVaeWeights`
loads by NAME.

The dims=2 temporal checkpoint was refused, but from inside the rounds loop,
which is reached only after two full denoise stages. The `temporal_upsampler_path`
load block exists so a caller learns at LOAD, says so in its own comment, and
checked both flags without checking `dims`. Guard added beside its two siblings.

Both contradictions are now EXECUTED against the real upstream module by
`scripts/gen-ltx2-pipeline-goldens.py`, which asserts each one raises and what it
says, and emits nothing -- the golden is byte-identical. That gate is what stops
either refusal from drifting into one upstream would happily serve.

It also corrects a mechanism this row asserted and had not measured. `Conv3d`
does NOT raise on the rank: it accepts a 4-D input as an unbatched 5-D one, so
the CHANNEL count is what disagrees ("expected input[1, 3, 32, 4, 6] to have 32
channels, but got 3"). At `frames == mid_channels` the conv passes and
PixelShuffleND(1) fails instead. The conclusion held and the reasoning did not.

The reader anchors in ltx2_video.cpp moved by the 16 lines the load guard adds,
and are repasted from the gate's own derived list.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]
…pairs

35 commits, the Qwen3.5 EXL3 head row and the ROCm managed-alloc fix among them.
No incoming commit touches this row's code: `git log HEAD..origin/main --` over
the upsampler, its header, the golden generator and the goldens returns nothing.

`docs/FEATURES.md` is the one shared file and is a keyed record, so its automatic
three-way merge was discarded again: the target-branch version was taken whole
and this row's two scoped edits reapplied on top, with both anchors re-asserted
as unique first. Against `origin/main` the file is 2 added lines and 1 rewritten
one, which is this row's own pair and no unrelated key.

Re-verified after the merge rather than assumed: the goldens regenerate
BYTE-IDENTICALLY from the pinned upstream, so nothing this row measured moved
under it.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]
… row's own guard moved

The load-time guard adds 16 lines near the top of `ltx2_video.cpp`, so every
anchor below it moved by exactly 16 — including the ones this spec, the header
and two issues use to argue A8's case. They were correct when written and wrong
when pushed, with no edit in between.

`test_ltx2_video`'s recorded-reader-anchor gate catches its own list and nothing
catches prose, so these were re-derived by grepping for the statements themselves
rather than by adding 16 to each: the three call sites are :3521, :3548 and
:5058, their shape checks :3525-3531 and :3552-3563, `vshape.frames` :3411, and
the temporal-only guard :3473-3502 with its D11b sentence at :3499-3501.

Recorded in the spec because a line anchor is a claim about a file that a later
hunk in the SAME change can falsify, and this row falsified its own.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]
@mudler
mudler force-pushed the row/LTX25-UPSAMPLER-ARMS branch from bd7d5b5 to 34edb89 Compare September 2, 2026 16:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants