diff --git a/.agents/specs/ltx25-upsampler-arms.md b/.agents/specs/ltx25-upsampler-arms.md new file mode 100644 index 000000000..0fb9dbdd0 --- /dev/null +++ b/.agents/specs/ltx25-upsampler-arms.md @@ -0,0 +1,372 @@ +# LTX-2.5 — the latent upsampler's `dims == 2` arm + +Row: `LTX25-UPSAMPLER-ARMS`. Campaign: [`ltx-2-5.md`](ltx-2-5.md) +(operator-owned; **not edited by this row**). Issue: +[#2577](https://github.com/mudler/vllm.cpp/issues/2577). Scope source: +[`ltx25-completion-scope.md`](ltx25-completion-scope.md) §8 order 6, which pairs +**A8** and **A9** as "the cheapest real gaps", size S each, no dependencies. + +Base: `origin/main` at `6974557b0`. + +Upstream pin: + +| Reference | Revision | +|---|---| +| Lightricks/LTX-2 (`packages/ltx-core`) | `fd4ded7f2d88d3da713abcdd4ad41ecc4a9314ca` | +| vLLM-Omni | `a4ea67a` | + +vLLM-Omni registers `ltx2` and is the primary oracle for this architecture. It +carries no `LatentUpsampler`, so the operator's definition is read from the +model author's own runtime, which is the row `ltx-2` occupies in AGENTS.md's +secondary-oracle table. + +## 0. Honesty statement — what this row does and does not claim + +This row lands **A9 only**. A8 was investigated to the same depth and is +returned undecided under `## Owed`, with the measurement that makes it a +decision rather than a port. The scope document sized the two together and +called A8 "the single cheapest real gap"; that sizing is correct about the +*operator* and does not survive contact with the *consumer*, which is the +finding §5 records. + +Nothing below is quoted from another document. Every local anchor was read from +the tree at `6974557b0` and every upstream anchor with +`git show fd4ded7f:`, never from a working tree, because a fork's working +tree is not the pin. + +## 1. The gap, on this tree + +`Ltx2LatentUpsample` refuses every non-3-D config. The anchor below reads +`6974557b0`, the base this spec was written against, because the row's own +implementation DELETES those lines -- a §1 anchor is the only kind here that a +successful row is supposed to falsify: + +``` +src/vllm/model_executor/models/ltx2_upsampler.cpp:467-472 @ 6974557b0 + Require(config.dims == 3, "ltx2 upsampler: dims=... is not ported. ..."); +``` + +`config.dims` is not a constant this port chose. It is read from the +checkpoint's own metadata at `src/vllm/model_executor/models/ltx2_loader.cpp` +(`Ltx2ParseUpsamplerConfig`), mirroring +`model/upsampler/model_configurator.py:17` (`dims = config.get("dims", 3)`). +A `dims=2` checkpoint is therefore an ordinary input, not a hypothetical. + +## 2. What upstream builds + +`model/upsampler/model.py:47`: + +```python +conv = torch.nn.Conv2d if dims == 2 else torch.nn.Conv3d +``` + +That one line reaches four parameter groups — `initial_conv` (`:49`), both +`ResBlock` stacks (`:53`, `:76-78`, whose own `conv` is chosen the same way at +`res_block.py:21`) and `final_conv` (`:80`). The `upsampler` module itself is +**not** chosen by `dims`: its branch (`:55-72`) tests the two flags only. + +`model.py:85-100` is the forward: + +```python +if self.dims == 2: + x = rearrange(latent, "b c f h w -> (b f) c h w") # :86 + x = self.initial_conv(x); x = self.initial_norm(x); x = self.initial_activation(x) + for block in self.res_blocks: x = block(x) + x = self.upsampler(x) # :94 + for block in self.post_upsample_res_blocks: x = block(x) + x = self.final_conv(x) + x = rearrange(x, "(b f) c h w -> b c f h w", b=b, f=f) # :100 +``` + +### 2.1 The two consequences of the fold, which ARE the port + +**Kernel rank.** Every convolution above is a 4-D `Conv2d` weight, where the +`dims == 3` arm's identically-named tensor is 5-D. This is the same rank trap +the temporal arm already documents at `ltx2_upsampler.cpp:453-456` for +`upsampler.0.weight`, now applying to four more groups. + +**GroupNorm statistics.** `(b f)` makes every frame its own sample, so +`GroupNorm(32, ...)` reduces over `(channels_per_group, H, W)`. Our `GroupNorm` +reduces over `frames * height * width` (`ltx2_upsampler.cpp:178`), which is the +`dims == 3` statistic. A port that reused it would produce a correctly shaped, +finite, plausible latent that is wrong everywhere — the failure mode this file's +header already names twice. + +There is no third consequence. `Silu` is pointwise, `PixelShuffle2d` and the +rational resampler are already per-frame operators, and the `upsampler` branch +does not read `dims`. + +## 3. Design + +Mirror the fold rather than re-deriving its effect. Inside the existing +per-batch loop, the `dims == 2` arm iterates frames and runs each as its own +one-frame `Volume`. That reproduces `(b f)` exactly and, because it does, the +existing `GroupNorm` gives the per-frame statistic with no change to it: a +one-frame volume's `frames * height * width` **is** `height * width`. + +The convolutions become `Conv2dPad1PerFrame`, which already exists +(`ltx2_upsampler.cpp:120`) and already takes a 4-D weight and already asserts +its element count at `:130-131`. `ResBlockForward` gains a `dims` parameter and +selects between the two conv helpers, mirroring `res_block.py:21` rather than +duplicating the block. + +`EnumerateLtx2UpsamplerTensors` emits 4-D shapes for the four groups when +`dims == 2`. Nothing else in that function moves. + +**Why not a separate 2-D code path.** AGENTS.md forbids a parallel hand-written +path where a seam can carry the behaviour, and upstream itself expresses the +difference as one conv-constructor choice plus a reshape. Two functions here +would be two things to keep in agreement for no gain. + +## 4. Reachability + +The dims=2 arm is reached from a production entry point on its default +configuration, and its output is consumable rather than merely computed. + +Entry point: the `upsampler_path` load extra +(`include/vllm/multimodal/ltx2_video.h`), read by `LoadVideoEngine` and applied +by the second phase's input transform at +`src/vllm/multimodal/ltx2_video.cpp:3521-3524`. + +Consumability is the part worth stating, because it is what separates A9 from +A8. With the default flags a `dims=2` upsampler returns `[c, f, 2h, 2w]`: +frames preserved by the fold, `h` and `w` doubled by `PixelShuffleND(2)`. + +`Ltx2UpsampleVideoLatent` has **three** product call sites, all in +`ltx2_video.cpp`, and each pins the frame axis: + +| # | Call site | What it requires of the frame axis | +|---|---|---| +| 1 | `:3521`, the video latent | `up.frames == vshape.frames`, checked at `:3525-3531` | +| 2 | `:3548`, the generated keyframe slots (`dfr_pipeline.py:348`) | `up_slots.frames == slot_positions.size()`, checked at `:3552-3563` | +| 3 | `:5058`, DFR's temporal rounds | the **temporal** arm, not this one | + +Sites 1 and 2 are the spatial arm's, and both require the frame count to come +back unchanged. A `dims=2` upsampler does exactly that, so the shapes agree at +**every** frame count and a caller who supplies a `dims=2` checkpoint renders at +the full requested size. + +The smallest failing test therefore enters through `LoadVideoEngine` and +`Generate`, not through `Ltx2LatentUpsample` directly, and asserts a completed +render rather than a changed error message. + +## 5. Why A8 is NOT in this row + +The operator is genuinely small: `model.py:55-59` is `Conv3d(mid, 8*mid)` + +`PixelShuffleND(3)`, and `:109-113` routes it through the same first-frame drop +the temporal arm already uses. The generator already constructs the module +(`scripts/gen-ltx2-pipeline-goldens.py`, the `spatiotemporal` block), so an +upstream golden is available for the asking. + +The consumers are the problem, and there are two of them rather than one. The +arm returns `[c, 2f-1, 2h, 2w]`, while both spatial call sites in §4's table +require the frame count back unchanged — `vshape.frames` at `:3525-3531` and +`slot_positions.size()` at `:3552-3563`. Those agree only when `f == 1`, that is +when the temporal doubling is exactly undone by the mandatory drop. + +`vshape.frames` is `(frames - 1) / factors.time + 1` (`:3411`) and +`factors.time` is the default-constructed 8 +(`ltx2_video.cpp:2871`, `ltx2_pipeline.h:460-462`), so `f == 1` needs +`frames <= 8`. That is reachable — `num_frames` in 2..8 passes through verbatim +at `:2852-2853`, and the only lower bound anywhere on the path is `frames < 1` at +`:2869` — but it is the one configuration in which the capability's whole point, +the doubled frame axis, is cancelled before anything can observe it. + +For every other clip, porting the operator would replace the named refusal at +`ltx2_upsampler.cpp:497` with the generic shape complaint at `:3527`. The guard +immediately above it (`ltx2_video.cpp:3473-3502`) exists to prevent precisely +that substitution for the temporal-only arm, and its comment says so. + +So the choice is between mirroring upstream's compute and keeping a diagnosis +this tree deliberately built. That is a product decision. AGENTS.md's stop +condition covers it: return `NEEDS_DECISION` rather than invent behaviour, and +rather than land an arm no production path consumes. Recorded under `## Owed`. + +## 6. Tests, ported in this change + +| Upstream | Ported as | +|---|---| +| `model.py:47,85-100` executed at reduced dimensions | a `Dims2` arm in `scripts/gen-ltx2-pipeline-goldens.py` §8, emitting out-shape, parameter manifest and value golden from the real module | +| the parameter contract implied by `conv = Conv2d` | `CheckManifest` over the `Dims2` manifest, which fails on a 5-D shape before any value is compared | +| — | the end-to-end reachability case in `tests/vllm/multimodal/test_ltx2_video.cpp`, which renders through `LoadVideoEngine`/`Generate` with a `dims=2` checkpoint | + +Both sides rebuild every weight from the shared name-keyed stream, so no weight +byte is checked in and a shape disagreement changes the values rather than +hiding. + +The refusal case in `test_ltx2_pipeline.cpp` ("the upsampler refuses the arms it +does not implement") loses its two `dims` arms and keeps the rest. + +It also gains the EIGHTH CELL, where both 2-D contradictions are set at once. +Only one of them is upstream's: `__init__` is an if/elif chain, so with +`spatial_upsample` false `elif temporal_upsample` (`model.py:68`) builds the +Conv3d and never reads `rational_resampler`. The generator drives that cell +through the real module as a third contradiction and asserts the CHANNEL-count +exception, so which refusal the port fires there is measured rather than +reasoned. The C++ case asserts the MESSAGE and asserts the other flag is not +named, because the polarity cannot see the defect: the cell is refused whichever +`Require` runs first. + +## 7. Risks + +- **The fold is mirrored but the golden is the only thing that proves it.** + A per-frame loop and a folded batch differ in no shape, only in the GroupNorm + statistic. The `Dims2` golden is the gate; `mid_channels = 32` with + `GroupNorm(32, ...)` gives one channel per group, so the per-frame and + per-clip statistics differ and the golden separates them. +- **A 2-D checkpoint is not one this campaign has in hand.** The arm is gated + against the executed upstream module, not against a shipped file, and §8 says + so rather than implying a checkpoint gate it does not have. + +## 8. Gates + +```sh +python3 scripts/gen-ltx2-pipeline-goldens.py --ltx2 ~/_git/LTX-2 \ + --vllm-omni ~/_git/vllm-omni --out tests/vllm/models/ltx2_pipeline_goldens.inc +ctest --test-dir build -R 'ltx2_pipeline|ltx2_video' --output-on-failure +scripts/agent-preflight.sh --staged +``` + +## Now + +`DONE` — A9 landed against issue #2577. A8 stays open under `## Owed`. + +## Outcome + +### What was built + +`dims == 2` is no longer refused. `EnumerateLtx2UpsamplerTensors` emits 4-D +kernels for `initial_conv`, both ResBlock stacks and `final_conv` when the +checkpoint sets it; `ResBlockForward` takes the rank as a parameter, mirroring +`res_block.py:21`; and the forward runs one frame at a time, which is how +`model.py:86`'s fold is reproduced rather than approximated. + +The stack itself became one lambda used by both ranks. Upstream's two forward +branches run the identical module sequence and differ only in the fold around +them, so a second copy here would have been the parallel path AGENTS.md forbids. + +`model.py:47`'s `else` is now mirrored as written: any `dims` that is not 2 +builds Conv3d. The old `Require(config.dims == 3, ...)` refused checkpoints +upstream runs, which was a refusal this port invented. + +### Red before green + +The dims=2 golden was written first and failed for both intended reasons on the +unmodified tree: six parameter-count mismatches (`5184 == 1728` for +`initial_conv`, `27648 == 9216` for each ResBlock conv), then the forward threw +`ltx2 upsampler: dims=2 is not ported`. Reproduce it by reverting +`src/vllm/model_executor/models/ltx2_upsampler.cpp` to its parent and rerunning +`test_ltx2_pipeline -tc='*dims=2 arm*'`; the goldens are already in the tree, so +the red is a checkout away and is not carried here as a log file. + +After the change, on the restored tree: `test_ltx2_pipeline` 62/62 cases and +3625 assertions green, `test_ltx2_video` 109/109 cases and 4866 assertions green +including the new end-to-end render. The generator regenerates +`ltx2_pipeline_goldens.inc` byte-identically with its third contradiction case +added, because a contradiction case emits nothing. + +### Mutations — four RED, each restored byte-for-byte + +| Mutation | Golden | End-to-end render | What it proves | +|---|---|---|---| +| **A**: `const bool two_d = false` | RED, `conv3d weight has the wrong element count` | **RED**, same throw at `test_ltx2_video.cpp:690` | The production path — `LoadVideoEngine` + `Generate` through the `upsampler_path` extra — really enters the new branch. This is the reachability proof, and it is a render that fails, not a unit test | +| **B**: keep the 2-D convolutions, drop the per-frame fold | **RED**, `max|diff| = 0.679842`, and the per-frame independence property RED at `0.439442` | **GREEN** | The fold's only consequence is numerical. Every shape is identical, so no shape check anywhere can see it and the value golden is the sole gate. This is the claim the code comment makes, now measured | +| **C**: the eighth cell's expected substring, `channels, but got` -> `not enough values to unpack` | n/a | n/a | **RED**: the generator raised and quoted what upstream really said -- `expected input[1, 3, 32, 4, 6] to have 32 channels, but got 3 channels instead`. The eighth cell fails by the TEMPORAL mechanism, measured against the module rather than argued from the if/elif chain | +| **D**: swap the two `dims=2` `Require`s back, rational first | n/a | n/a | **RED at exactly two assertions**, `test_ltx2_pipeline.cpp:2282-2283`, 61/62 cases and 3623/3625. Nothing else moves, because the polarity is identical either way -- which is the whole reason the case reads the message. Restored byte-for-byte, sha256 verified | + +Mutation B is the one worth keeping in mind: it is the shape of defect this file's +header warns about twice, and it passes an end-to-end render. + +Mutations C and D are the pair that gates the ORDER of the two refusals. C says +what upstream does; D says the port would stop saying it if the order changed. + +### A sibling defect this row did NOT fix, and why + +`src/vllm/multimodal/ltx2_video.cpp:3499-3501` tells a caller who supplies the +temporal-only checkpoint that "no phase of any recipe this engine serves +consumes it, because its only upstream consumer is DFRPipeline's rounds loop, +which is not ported". DFR **is** ported and drives that arm at `:5058`. It is +the same false statement this row corrected in `ltx2_upsampler.h` under +[#2580](https://github.com/mudler/vllm.cpp/issues/2580), one file over, and it +is user-facing where the header's was not. + +It is already tracked as **D11b** in +`.agents/specs/ltx25-completion-scope.md` §6.1, so it is recorded here rather +than re-filed — intake without an exit is what that document says produced 701 +open issues. Left to its owner because repairing a user-facing refusal is its +own red-first change, not a comment fix ridden along on this one. + +### The anchors this row cited went stale inside this row + +The change moves lines on THREE axes, and the first pass at this section saw +only one of them. The load-time guard adds 16 lines near the top of +`ltx2_video.cpp`, so every anchor below it moved by 16. The port also takes +`ltx2_upsampler.cpp` from 582 lines to 714 across nine separate hunks -- 723 +once the repair below swaps the two refusals -- so +anchors into that file moved by an amount that depends on where they point -- +`:465` became `:497` and `:166` became `:178`, and no single offset produces +both. That second axis is the one the first pass at this section missed +entirely, and "moved by exactly 16" is why: an offset that holds for one file +reads like a rule. +`test_ltx2_video`'s recorded-reader-anchor gate caught its own list and nothing +catches prose. + +Every anchor into those three files that appears in a file this change edits was +re-derived by grepping for the statement, and each is listed with what was +grepped for in the pull request body. That is the claim this section makes, and +it is deliberately narrower than "every anchor in this branch's prose": the +first pass asserted the wider claim, swept the spec, the header and the issues, +and missed both test files and one half-updated range. The narrow claim is what +was measured. + +What the narrow claim leaves out, stated rather than implied: 4 anchors to +`ltx2_upsampler.cpp:465` and `:467` survive in `.agents/specs/` +(`ltx25-completion-scope.md:75, :341, :342` and `ltx25-retire-dead-arms.md:433`), +and dozens of anchors into `ltx2_video.cpp` past line 1525 survive across other +rows' specs and `.agents/completed/`. Those are other rows' records of their own +trees, and `ltx25-completion-scope.md` is an operator-owned shared file this row +does not edit for the reason §"The scope document contradicts §5" already gives. +Editing them from here would make this branch write a file every concurrent pull +request writes, which AGENTS.md §Records names as the failure to avoid. + +Worth stating because the numbers were correct when written and wrong when +pushed, with no edit in between: a line anchor is a claim about a file that a +later hunk in the SAME change -- or a merge that carries no hunk of its own -- +can falsify. The repair commit demonstrated it twice more: swapping the two +`dims=2` refusals moved `BlurDownsample` from `:590` to `:599` inside this same +section's edit, and that anchor had already been stale since #641. + +### What was NOT done + +No `dims=2` checkpoint exists on the NAS, so this arm is gated against the +executed upstream module at reduced dimensions and has no real-weight result. +That is stated rather than implied, and it is why `docs/USAGE.md` gains no +checkpoint row: there is no artifact to pin. + +## Owed + +- **A8, the spatiotemporal upsampler arm** (`model/upsampler/model.py:55-59`, + refused through `Ltx2RefuseUnportedPipelineFeature(kSpatiotemporalUpsampler)`). + Returned as a decision, not a port: §5 has the measurement. Tracked by + [#2584](https://github.com/mudler/vllm.cpp/issues/2584), which carries the + derivation and the three answers a decision has to choose between. + + **It has its own issue for a reason.** It was first recorded against + [#2577](https://github.com/mudler/vllm.cpp/issues/2577), the issue this row + CLOSES, so the owed item would have pointed at a closed issue the moment the + work landed. `scripts/check-agent-record.py` cannot catch that, because this + `## Owed` entry satisfies the gate on its own -- which is precisely why the + entry has to name a live issue rather than lean on the gate. + +### The scope document contradicts §5, and this is the record of it + +`.agents/specs/ltx25-completion-scope.md:371` calls A8 "the single cheapest real +gap and the only reachable unported-feature refusal", and its §8 sizes it S +beside A9. §5 above falsifies the sizing: the operator is cheap and the landing +is not, because the second consumer at `ltx2_video.cpp:3548` was not counted. + +That document is operator-owned and a shared-file lock under AGENTS.md +§Records, so it is **not edited from this row**. The contradiction is recorded +here and in [#2584](https://github.com/mudler/vllm.cpp/issues/2584) instead, for +the operator to reconcile at their own cadence. Anyone reading the sequence +should treat order 6's "S each" as holding for A9 alone. diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 9e4c76807..091fb30c2 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -195,6 +195,7 @@ in `ltx2_text_encoder.cpp` is the call that would have to change. | LTX-2.5 T2A guidance space | LTX-2.5 DiT (T2A arm) | `test_ltx2_video` "the guider is handed x0 predictions" through `Generate`, on all 3 arms plus the guider output and the Euler input; a seam case puts the two spaces 1.5e-07 apart at rescale 0 and 0.352 at 0.7 | Combines **denoised (x0)**, mirroring `X0Model` (`model.py:590-604`). Was velocity space, which agrees only at rescale 0 ([#1039](https://github.com/mudler/vllm.cpp/issues/1039)) | | LTX-2.5 VIDEO guidance | LTX-2.5 DiT, joint video+audio | `test_ltx2_video`'s `ltx2 one_stage:` cases through `Generate`; all FOUR arms carry the x0 invariant and the guider output replays EXACTLY | `--pipeline-kind one_stage` runs `_guided_denoise`: 4 forwards/step, combined per modality in **x0**. Was ONE unguided forward, every `video_guidance` field dead ([#1092](https://github.com/mudler/vllm.cpp/issues/1092)) | | LTX-2.5 cross-attention perturbations | LTX-2.5 DiT | `test_ltx2_video` and `test_ltx2_device` each gate one direction ALONE, on a forward where the other stream is PRESENT but DISABLED. Swapping the two flags is RED on both arms | `SKIP_A2V_CROSS_ATTN` / `SKIP_V2A_CROSS_ATTN`, which `modality_scale = 3.0` selects on every video row. On the DEVICE forward too since 2026-08-19 ([#1092](https://github.com/mudler/vllm.cpp/issues/1092)) | +| LTX-2.5 latent upsampler `dims=2` arm | LTX-2.5 spatial upsampler checkpoint whose config sets `dims: 2` | `ltx2 the latent upsampler reproduces upstream on the dims=2 arm` (`test_ltx2_pipeline`) against the EXECUTED upstream module, plus `a dims=2 upsampler checkpoint RENDERS` (`test_ltx2_video`) through `LoadVideoEngine`/`Generate`; removing the fold reds the golden at max|diff| 0.68 while the render stays green, so the value gate and not a shape check is what holds it | No flag: `dims` is read off the checkpoint's own config, so a 2-D upsampler at `--upsampler` just works. Conv2d for `initial_conv`, both ResBlock stacks and `final_conv` (model.py:47); the frame axis folds into the batch (:86), which makes GroupNorm PER FRAME. Frame count preserved, H and W doubled, which is what both spatial call sites require. Two 2-D contradictions upstream cannot run are refused by name and EXECUTED against the real module by the golden generator: `temporal_upsample` (a Conv3d handed a folded 4-D tensor) and `rational_resampler` (five-value unpack at spatial_rational_resampler.py:41) | | LTX-2.5 audio-to-video (`A2VidPipelineTwoStage`) | LTX-2.5 DiT + audio VAE encoder + spatial upsampler | `test_ltx2_pipeline` and `test_ltx2_video`'s `ltx2 a2vid:` cases through `LoadVideoEngine`+`Generate`; the take's latent is bit-identical across SEEDS and moves with the WINDOW | `--pipeline-kind a2vid_two_stage`. Guided half-res stage 1, DERIVED schedule, plain Euler; distilled 3-sigma stage 2. `--audio-path` and `--lora` REQUIRED; the distilled adapter rides stage 2 ALONE (#1118) | | LTX-2.5 keyframe interpolation (`KeyframeInterpolationPipeline`) | LTX-2.5 DiT + spatial upsampler | `ltx2 keyframe:` cases in `test_ltx2_pipeline` / `test_ltx2_video` via `LoadVideoEngine`+`Generate`: frame 0 APPENDS against a `ti2vid_two_stage` control, the x0 invariant on four arms, the 4096 anchor | `--pipeline-kind keyframe_interpolation`. No frame-0 special case, so `--first-frame` is guidance to interpolate FROM; stage 2's audio leaves. `--lora` REQUIRED. `--last-frame` new (#1191). CPU fixtures | | LTX-2.5 two-stage text/image-to-video (`TI2VidTwoStagesPipeline`) | LTX-2.5 DiT + spatial upsampler | `test_ltx2_pipeline` and `test_ltx2_video` `ltx2 ti2vid:` cases through `LoadVideoEngine`+`Generate`; the x0 invariant on all FOUR arms, and the 4096 anchor read at two geometries against a res_2s control that moves | `--pipeline-kind ti2vid_two_stage`. Guided half-res stage 1 on the UNADAPTED model, plain Euler; distilled 3-sigma stage 2. `--lora` REQUIRED, no `--audio-path`; stage 1's audio leaves. CPU fixtures, Full-model run owed | @@ -391,7 +392,7 @@ CPU elementwise GEMM (f32/f16/bf16) runs AVX2 and AVX-512 tiers on x86 where the | Kimi-Linear-48B-A3B (KDA + NoPE-MLA + MoE hybrid) | **Runner fold LANDS (ROW 7 §21, #122): the ENGINE/SERVER surface serves Kimi at the 122/128 golden profile (engine==CLI 128/128); STRICT stays closed (intrinsic p7 near-tie)** | server 19.0 tok/s wall / CLI 18.9 vs vLLM ~21 (~0.90×), speed residual named (§21) | | Muse Glimmer 30B (Meta) | Text gated at **reduced depth 4/52** only; vision wired but never reference-checked | [spec](../.agents/specs/muse-glimmer.md) / [#268](https://github.com/mudler/vllm.cpp/issues/268). Full depth, multi-step decode, image/video, server path and parser scoping open. vLLM speed OPEN GAP; llama.cpp bar #333 | | LTX-2.5 AUTO duration (the duration head) | Brick ported, never constructed | `duration_head_path` is REFUSED by name rather than accepted-and-ignored ([#611](https://github.com/mudler/vllm.cpp/issues/611)); supplying a head cannot load one. Give `num_frames` or `duration` | -| LTX-2.5 arms a request CAN reach | Refused by name at the call site | The spatiotemporal latent upsampler (both flags set). Supplying that checkpoint names that arm, not the temporal one. The temporal-only x2 arm is ported, not refused | +| LTX-2.5 arms a request CAN reach | Refused by name at the call site | The spatiotemporal latent upsampler (both flags set). Supplying that checkpoint names that arm, not the temporal one. Its operator is small; what keeps it out is that it returns `[c, 2f-1, 2h, 2w]` while both spatial call sites require the frame count back unchanged, so porting it would trade a named refusal for a shape complaint ([#2584](https://github.com/mudler/vllm.cpp/issues/2584)). The temporal-only x2 arm is ported, not refused, and so is `dims=2`; what stays refused in 2-D are its two contradictions, `temporal_upsample` and `rational_resampler`, neither of which upstream can run either | | LTX-2.5 resolution | Off-grid sizes refused, naming the offending axis and a size you can actually pass; frames still round | `--width`/`--height` must divide 64 (two-stage) or 32 (one-stage), from the VAE factor times the phase downscale ([#919](https://github.com/mudler/vllm.cpp/issues/919)). `--frames` rounds to `8k + 1`. No size cap | | LTX-2.5 arms nothing can request | Declared, not requestable | `int8-convrot` (ComfyUI-only), single-node multi-GPU, `BetaScheduler` (upstream selects no scheduler either). No flag or extra asks for these. `multishot` was RETIRED (absent upstream) and `kLoraFusion` too (now served) | | Qwen3.8-27B quantized arms (Q4_K_M GGUF, its `clip` mmproj, the `unsloth` "NVFP4" = `mixed-precision`) | All three files ACCOUNTED against committed manifests (866 / 334 / 1968 names). **Q4_K_M RUNS on CPU and its token gate vs llama.cpp `b10451` FAILED** (2026-08-23): tokenizer exact 6/6, generation divergent 5/6, every divergence a rank-2 loss under 0.18 logits over 288 steps, so a precision difference and not a wiring defect ([evidence](bench-evidence/qwen38-27b-q4km-token-gate-20260823.md)). No speed or memory number is admissible. NVFP4 still un-run (#1632) | BF16 gated (#915). `--mmproj` (ABI v22) loads a `clip` GGUF; a load refuses a file with a tensor no loader reads. "NVFP4" @`7d6f8d4d`: W4A4 loads, FP8 refused ([spec](../.agents/specs/qwen38-27b-quant-arms.md)) | diff --git a/include/vllm/model_executor/models/ltx2_pipeline.h b/include/vllm/model_executor/models/ltx2_pipeline.h index 273160406..8ab792f1b 100644 --- a/include/vllm/model_executor/models/ltx2_pipeline.h +++ b/include/vllm/model_executor/models/ltx2_pipeline.h @@ -1176,7 +1176,7 @@ Ltx2PipelineRecipe ResolveLtx2PipelineRecipe(const std::string& pipeline_kind, // TWO KINDS live here, and conflating them overstated what this port refuses: // // REACHABLE REFUSAL — a product path constructs the condition and throws, so a -// caller CAN trip it. `kSpatiotemporalUpsampler` (ltx2_upsampler.cpp:465) is +// caller CAN trip it. `kSpatiotemporalUpsampler` (ltx2_upsampler.cpp:497) is // the ONE. `ltx2_video.cpp` reaches it through `Ltx2UpsampleVideoLatent` when a // phase asks for the spatial-upsample transform. The TEMPORAL-ONLY x2 // upsampler is NOT among them: it is ported (`2e9d95e74`, spec diff --git a/include/vllm/model_executor/models/ltx2_upsampler.h b/include/vllm/model_executor/models/ltx2_upsampler.h index e31d4e6dd..ee2f4c259 100644 --- a/include/vllm/model_executor/models/ltx2_upsampler.h +++ b/include/vllm/model_executor/models/ltx2_upsampler.h @@ -19,19 +19,28 @@ // Ltx2RationalForScale <- spatial_rational_resampler.py:10-14 // Ltx2UpsampleVideoLatent <- model.py:129-143 (upsample_video) // -// ─── WHAT THE TEMPORAL ARM IS REACHABLE FROM ───────────────────────────────── -// NOTHING, today, and that is stated here rather than only in the spec because a -// header is what the next reader opens. It is ported and gated against executed -// upstream at reduced dimensions, and `Ltx2ParseUpsamplerConfig` -// (ltx2_loader.cpp:1431-1444) reads `temporal_upsample` off a checkpoint. But the -// engine's ONE upsampler call site is the `kSpatialUpsample` phase input -// transform (multimodal/ltx2_video.cpp:1408-1466), which shape-checks the result -// against a SPATIALLY doubled latent and fails otherwise; and upstream's only -// consumer is `DFRPipeline`'s rounds loop (ltx-pipelines/dfr_pipeline.py:235-245, -// 402-407), which is not ported. The shipped temporal checkpoint +// ─── WHAT EACH ARM IS REACHABLE FROM ───────────────────────────────────────── +// Stated here rather than only in the spec because a header is what the next +// reader opens. `Ltx2UpsampleVideoLatent` has THREE product call sites, all in +// multimodal/ltx2_video.cpp, and each one pins the frame axis: +// +// :3521 the video latent, the `kSpatialUpsample` phase input transform. +// Requires `up.frames == vshape.frames` at :3525-3531. +// :3548 the generated keyframe slots, which take the SAME spatial upsampler +// (dfr_pipeline.py:348). Requires `slot_positions.size()` at :3552-3563. +// :5058 DFR's temporal-refinement rounds, the TEMPORAL arm. Reached through +// the `temporal_upsample_rounds` load extra, and instrumented by +// `trace.temporal_upsample_calls` (multimodal/ltx2_video.h:741). +// +// This paragraph said "NOTHING, today" of the temporal arm and "the engine's ONE +// upsampler call site" of the spatial one. Both were false by the time they were +// read: site :5058 drives the temporal arm and is not a phase input transform at +// all. Corrected under issue #2580; the count is what the dims=2 port's +// reachability argument rests on, so it is derived here rather than remembered. +// The shipped temporal checkpoint // (`ltx-2.5-latent-temporal-upscaler-x2-bf16-1.0.safetensors`, -// ltx-pipelines/docs/pipelines.md:176) is not on the NAS either, so no -// real-weight result exists. +// ltx-pipelines/docs/pipelines.md:176) is not on the NAS, so no real-weight +// result exists for that arm. // // ─── WHAT SEPARATES THIS FROM THE VAE'S CONVOLUTIONS ───────────────────────── // These are plain `torch.nn.Conv3d`/`Conv2d` with `padding=1` — ZERO padding on @@ -55,13 +64,46 @@ // * The temporal arm DROPS THE FIRST FRAME after the shuffle (model.py:109-113), // so `f` frames in produce `2f - 1` out and not `2f`. // +// ─── THE dims=2 ARM (model.py:47, :85-100) ─────────────────────────────────── +// PORTED. `conv = torch.nn.Conv2d if dims == 2 else torch.nn.Conv3d` (:47) +// reaches four parameter groups — `initial_conv`, both ResBlock stacks and +// `final_conv` — so each is a 4-D kernel where the 3-D arms build a 5-D one. The +// `upsampler` branch (:55-72) never reads `dims` and keeps its rank. +// +// The forward folds the frame axis into the BATCH (:86) and unfolds at :100, +// which has one consequence a shape check cannot see: GroupNorm normalises PER +// FRAME. This port reproduces the fold by running one frame at a time, so the +// existing reduction over `frames * height * width` gives the per-frame +// statistic without a second normaliser. Gated by "reproduces upstream on the +// dims=2 arm" (test_ltx2_pipeline) against the executed module, and reached +// end-to-end by "a dims=2 upsampler checkpoint RENDERS" (test_ltx2_video). +// +// Any `dims` that is not 2 builds Conv3d, which is upstream's own `else` at :47 +// and is mirrored rather than narrowed to a refusal upstream does not raise. +// // ─── NOT PORTED, refused by name ───────────────────────────────────────────── // * `spatial_upsample AND temporal_upsample` (model.py:55-59) — a DIFFERENT // operator from the temporal-only arm: `Conv3d(mid, 8*mid)` + PixelShuffleND(3). -// Asking for it throws. -// * `dims == 2` (model.py:85-100) — a checkpoint that sets it wants Conv2d -// everywhere, i.e. no temporal convolution at all. LTX-2.5's upsampler is -// dims=3; the 2-D arm is refused rather than approximated by the 3-D one. +// Asking for it throws. Its operator is small; what keeps it out is that it +// returns `[c, 2f-1, 2h, 2w]` and BOTH spatial call sites above require the +// frame count back unchanged. Owed, with the measurement, in +// .agents/specs/ltx25-upsampler-arms.md. +// * `dims == 2` WITH `temporal_upsample` — not an arm but a contradiction: +// upstream builds that upsampler as a Conv3d (:68-71) and the 2-D forward +// hands it a 4-D tensor. It raises on the CHANNEL COUNT and not on the rank: +// `Conv3d` reads a 4-D input as an unbatched 5-D one, so at 3 frames against +// `mid_channels` 32 it reports "to have 32 channels, but got 3". At +// `frames == mid_channels` the conv passes and `PixelShuffleND(1)` fails +// instead. Two mechanisms, one contradiction — refused by name once. +// * `dims == 2` WITH `rational_resampler` — the sibling, and the dangerous one: +// every operator in that branch is per-frame, so this port would compute a +// finite, plausible latent no shape check could fault. Upstream raises +// `not enough values to unpack (expected 5, got 4)` at +// spatial_rational_resampler.py:41. Refused by name. +// +// Both are EXECUTED against the real module by +// scripts/gen-ltx2-pipeline-goldens.py, which asserts each raises and what it +// says, so neither refusal can drift into one upstream would serve. // // ─── DTYPE ─────────────────────────────────────────────────────────────────── // f32, because this is the CPU REFERENCE arm and the gate compares the ALGORITHM diff --git a/scripts/gen-ltx2-pipeline-goldens.py b/scripts/gen-ltx2-pipeline-goldens.py index 5966d2c1e..51d785755 100644 --- a/scripts/gen-ltx2-pipeline-goldens.py +++ b/scripts/gen-ltx2-pipeline-goldens.py @@ -1168,6 +1168,120 @@ def section_upsampler(out) -> None: emit_manifest(out, "kLtx2UpsTemporalParam", tmanifest) emit_f32(out, "kLtx2UpsTemporalGolden", tresult.numpy()) + # ---- the dims=2 arm (model.py:47, 85-100) ----------------------------- + # + # `conv = torch.nn.Conv2d if dims == 2 else torch.nn.Conv3d` (:47) reaches + # FOUR parameter groups — `initial_conv` (:49), both ResBlock stacks (:53, + # :76-78, via res_block.py:21) and `final_conv` (:80) — so every one of them + # is a 4-D kernel here where the dims=3 arms above build a 5-D one. The + # manifest is what gates that on the C++ side; a port that reused the 3-D + # enumeration fills different values and fails on the golden as well. + # + # The forward folds the frame axis into the BATCH (`rearrange(latent, + # "b c f h w -> (b f) c h w"`, :86) and unfolds at :100. That fold is not + # cosmetic: it makes every frame its own GroupNorm sample, so the statistics + # run over (channels_per_group, H, W) rather than over the whole clip. With + # `_UPS_MID = 32` and GroupNorm(32, ...) there is ONE channel per group, so + # the per-frame and per-clip statistics genuinely differ and this golden + # separates them. A port that kept the 3-D reduction returns a correctly + # shaped, finite, plausible latent that is wrong in every element. + # + # `_UPS_TEMPORAL_F = 3` frames, so the frame axis differs from H (4) and W + # (6) and a fold that lost or transposed it cannot pass by coincidence. + emit_scalar(out, "kLtx2UpsDims2Frames", _UPS_TEMPORAL_F) + d2count = _UPS_IN * _UPS_TEMPORAL_F * _UPS_H * _UPS_W + d2latent = torch.from_numpy(make("ltx2.ups.dims2.latent", d2count, 1.0)).reshape( + 1, _UPS_IN, _UPS_TEMPORAL_F, _UPS_H, _UPS_W + ) + emit_f32(out, "kLtx2UpsDims2Latent", d2latent.numpy()) + + dims2 = LatentUpsampler( + in_channels=_UPS_IN, + mid_channels=_UPS_MID, + num_blocks_per_stage=_UPS_BLOCKS, + dims=2, + spatial_upsample=True, + temporal_upsample=False, + spatial_scale=2.0, + rational_resampler=False, + ) + dims2.eval() + d2manifest = fill_module(dims2, "ltx2.ups.Dims2.") + d2result = dims2(d2latent) + emit_i64(out, "kLtx2UpsDims2OutShape", list(d2result.shape)) + emit_manifest(out, "kLtx2UpsDims2Param", d2manifest) + emit_f32(out, "kLtx2UpsDims2Golden", d2result.numpy()) + + # THE TWO CONTRADICTIONS the dims=2 arm cannot reach, asserted against the + # EXECUTED module and deliberately emitting nothing. These are FAILURE cases, + # so what the two `Require`s in ltx2_upsampler.cpp are gated by is the + # exception upstream raises, not a value — and running them here is what stops + # either refusal from drifting into one upstream would happily serve, which is + # the polarity AGENTS.md names as this port inventing a refusal. + # + # Both are the same event: the dims=2 forward folds the frame axis into the + # batch (model.py:86) and hands `self.upsampler` a 4-D tensor at :94, while + # both of these modules were built for a 5-D one. The MESSAGES are asserted + # because they differ, and because the obvious reading of the temporal one is + # wrong — torch does not raise on the rank there. `Conv3d` accepts a 4-D input + # as an UNBATCHED 5-D one, so `(b f, c, h, w)` is read as `(C, D, H, W)` and + # the CHANNEL count is what fails. With `_UPS_MID = 32` and 3 frames those two + # differ, which is the only reason the check fires at all. + for tag, flags, want in ( + ( + "temporal", + dict(spatial_upsample=False, temporal_upsample=True, rational_resampler=False), + "channels, but got", + ), + ( + "rational", + dict(spatial_upsample=True, temporal_upsample=False, rational_resampler=True), + "not enough values to unpack", + ), + # THE CELL WHERE BOTH CONTRADICTIONS ARE SET, and the reason the C++ side + # checks them in the order it does. `__init__` is an if/elif chain: with + # `spatial_upsample` False, `elif temporal_upsample` (model.py:68) builds + # the Conv3d and NEVER reads `rational_resampler`, so this config owns no + # SpatialRationalResampler and cannot fail by its unpack. It fails by the + # temporal mechanism, and this case is what says so with the module rather + # than with reasoning about it. `ltx2_upsampler.cpp` therefore checks the + # temporal contradiction FIRST; swapping the two gives this cell an + # explanation naming a module upstream did not construct. + # + # Measured, not asserted: replacing the expected substring below with the + # rational one makes the generator raise and quote what upstream really + # said -- "expected input[1, 3, 32, 4, 6] to have 32 channels, but got 3". + ( + "temporal-and-rational", + dict(spatial_upsample=False, temporal_upsample=True, rational_resampler=True), + "channels, but got", + ), + ): + contradiction = LatentUpsampler( + in_channels=_UPS_IN, + mid_channels=_UPS_MID, + num_blocks_per_stage=_UPS_BLOCKS, + dims=2, + spatial_scale=2.0, + **flags, + ) + contradiction.eval() + fill_module(contradiction, f"ltx2.ups.Dims2{tag}.") + try: + contradiction(d2latent) + except Exception as exc: # noqa: BLE001 — the raise IS the assertion + if want not in str(exc): + raise AssertionError( + f"dims=2 + {tag} raised {str(exc)!r}, which does not carry {want!r}. " + "ltx2_upsampler.cpp names that mechanism in its refusal, so either " + "the pin moved or the comment is wrong" + ) from exc + else: + raise AssertionError( + f"dims=2 + {tag} RAN on upstream at this pin. ltx2_upsampler.cpp refuses " + "it, and a refusal upstream does not raise is this port's invention" + ) + # The spatial+temporal arm stays REFUSED (model.py:55-59: `8 * mid_channels` # and `PixelShuffleND(3)`, a different operator). Its parameter shape is # emitted anyway so the C++ refusal is gated against what upstream would diff --git a/src/vllm/model_executor/models/ltx2_upsampler.cpp b/src/vllm/model_executor/models/ltx2_upsampler.cpp index 564d0da4c..a0a171aea 100644 --- a/src/vllm/model_executor/models/ltx2_upsampler.cpp +++ b/src/vllm/model_executor/models/ltx2_upsampler.cpp @@ -129,6 +129,15 @@ Volume Conv2dPad1PerFrame(const Volume& in, int64_t out_channels, out.data.assign(static_cast(out.elems()), 0.0f); Require(weight.size() == static_cast(out_channels * in.channels * k * k), "ltx2 upsampler: conv2d weight has the wrong element count"); + // The bias half of the contract, which `Conv3dPad1` above has always had and + // this helper did not. Indexing `bias[oc]` unchecked reads past the end of a + // std::vector when a checkpoint carries a correct kernel and a short bias -- + // UB and silent garbage, where the identical 3-D defect gets a named refusal. + // Nothing upstream of here validates it: `Ltx2LoadVaeWeights` loads by NAME and + // checks no shape. The dims=2 arm routes four parameter groups through this + // helper, so it is the path that made the gap reachable. + Require(bias.size() == static_cast(out_channels), + "ltx2 upsampler: conv2d bias has the wrong element count"); for (int64_t oc = 0; oc < out_channels; ++oc) { for (int64_t f = 0; f < out.frames; ++f) { @@ -210,14 +219,22 @@ void Silu(std::vector& x) { // ResBlock.forward (res_block.py:29-37). The residual is added BEFORE the // activation — `activation(x + residual)`, not `activation(x) + residual`. -Volume ResBlockForward(const Ltx2VaeWeights& weights, const std::string& prefix, +// `two_d` selects the same way `res_block.py:21` does. It is a parameter rather +// than two functions because upstream expresses the difference as one +// constructor choice over an otherwise identical block, and a second copy here +// would be a second thing to keep in agreement for no gain. +Volume ResBlockForward(const Ltx2VaeWeights& weights, const std::string& prefix, bool two_d, const Volume& in) { - Volume x = Conv3dPad1(in, in.channels, weights.Get(prefix + "conv1.weight"), - weights.Get(prefix + "conv1.bias")); + const auto conv = [&](const Volume& v, int64_t out_ch, const std::vector& w, + const std::vector& b) { + return two_d ? Conv2dPad1PerFrame(v, out_ch, w, b) : Conv3dPad1(v, out_ch, w, b); + }; + Volume x = conv(in, in.channels, weights.Get(prefix + "conv1.weight"), + weights.Get(prefix + "conv1.bias")); GroupNorm(x, weights.Get(prefix + "norm1.weight"), weights.Get(prefix + "norm1.bias")); Silu(x.data); - Volume y = Conv3dPad1(x, in.channels, weights.Get(prefix + "conv2.weight"), - weights.Get(prefix + "conv2.bias")); + Volume y = conv(x, in.channels, weights.Get(prefix + "conv2.weight"), + weights.Get(prefix + "conv2.bias")); GroupNorm(y, weights.Get(prefix + "norm2.weight"), weights.Get(prefix + "norm2.bias")); for (size_t i = 0; i < y.data.size(); ++i) y.data[i] += in.data[i]; Silu(y.data); @@ -390,24 +407,39 @@ std::vector Ltx2BlurKernel(int64_t kernel_size) { std::vector EnumerateLtx2UpsamplerTensors( const Ltx2UpsamplerConfig& config) { - // `named_parameters()` order for LatentUpsampler(dims=3): initial_conv, + // `named_parameters()` order, the same for both ranks: initial_conv, // initial_norm, res_blocks, upsampler, post_upsample_res_blocks, final_conv. const std::string p = config.prefix; const int64_t in_c = config.in_channels; const int64_t mid = config.mid_channels; std::vector specs; - specs.push_back({p + "initial_conv.weight", {mid, in_c, 3, 3, 3}}); + // model.py:47 — `conv = torch.nn.Conv2d if dims == 2 else torch.nn.Conv3d`. + // That ONE line reaches four parameter groups: `initial_conv` (:49), both + // ResBlock stacks (:53 and :76-78, whose own conv is chosen the same way at + // res_block.py:21) and `final_conv` (:80). The `upsampler` branch (:55-72) + // never reads `dims`, so it is deliberately NOT built through this helper — + // its rank is decided by the two FLAGS instead, below. + // + // The `else` is upstream's and is mirrored as written: any `dims` that is not + // 2 builds Conv3d there, so 3 is the shipped value rather than the only legal + // one, and this port does not invent a refusal upstream does not raise. + const auto conv_w = [&](int64_t out_ch, int64_t in_ch) { + return config.dims == 2 ? std::vector{out_ch, in_ch, 3, 3} + : std::vector{out_ch, in_ch, 3, 3, 3}; + }; + + specs.push_back({p + "initial_conv.weight", conv_w(mid, in_c)}); specs.push_back({p + "initial_conv.bias", {mid}}); specs.push_back({p + "initial_norm.weight", {mid}}); specs.push_back({p + "initial_norm.bias", {mid}}); auto res_block = [&](const std::string& prefix) { - specs.push_back({prefix + "conv1.weight", {mid, mid, 3, 3, 3}}); + specs.push_back({prefix + "conv1.weight", conv_w(mid, mid)}); specs.push_back({prefix + "conv1.bias", {mid}}); specs.push_back({prefix + "norm1.weight", {mid}}); specs.push_back({prefix + "norm1.bias", {mid}}); - specs.push_back({prefix + "conv2.weight", {mid, mid, 3, 3, 3}}); + specs.push_back({prefix + "conv2.weight", conv_w(mid, mid)}); specs.push_back({prefix + "conv2.bias", {mid}}); specs.push_back({prefix + "norm2.weight", {mid}}); specs.push_back({prefix + "norm2.bias", {mid}}); @@ -444,7 +476,7 @@ std::vector EnumerateLtx2UpsamplerTensors( for (int64_t i = 0; i < config.num_blocks_per_stage; ++i) { res_block(p + "post_upsample_res_blocks." + std::to_string(i) + "."); } - specs.push_back({p + "final_conv.weight", {in_c, mid, 3, 3, 3}}); + specs.push_back({p + "final_conv.weight", conv_w(in_c, mid)}); specs.push_back({p + "final_conv.bias", {in_c}}); return specs; } @@ -464,12 +496,58 @@ Ltx2LatentVolume Ltx2LatentUpsample(const Ltx2UpsamplerConfig& config, if (config.temporal_upsample && config.spatial_upsample) { Ltx2RefuseUnportedPipelineFeature(Ltx2UnportedPipelineFeature::kSpatiotemporalUpsampler); } - Require(config.dims == 3, - "ltx2 upsampler: dims=" + std::to_string(config.dims) + - " is not ported. The dims=2 arm (model/upsampler/model.py:85-100) builds Conv2d " - "everywhere, i.e. NO temporal convolution at all, so the 3-D path cannot serve " - "it. LTX-2.5's upsampler is dims=3. Owed and recorded in " - ".agents/specs/ltx-2-5.md phase L5."); + // model.py:47's `else` is mirrored as written: `dims == 2` is Conv2d and + // EVERYTHING else is Conv3d, so there is no refusal here — upstream raises + // none, and inventing one would refuse a checkpoint upstream runs. + const bool two_d = config.dims == 2; + // The one combination upstream cannot run. `dims=2` takes the forward at :85, + // which hands `self.upsampler` a 4-D tensor, while `temporal_upsample` built + // that module as a Conv3d (:57 for both flags, :70 for temporal alone). + // + // WHAT ACTUALLY FIRES UPSTREAM IS NOT THE RANK, and this comment said it was. + // `Conv3d` accepts a 4-D input as an UNBATCHED 5-D one, so the folded + // `(b*f, c, h, w)` is read as `(C, D, H, W)` and the CHANNEL count is what + // disagrees. Measured at the pin with `_UPS_MID = 32` and 3 frames: + // RuntimeError: Given groups=1, weight of size [64, 32, 3, 3, 3], expected + // input[1, 3, 32, 4, 6] to have 32 channels, but got 3 channels instead + // The conclusion survives the correction and the reasoning does not: at + // `frames == mid_channels` that Conv3d would PASS, and `PixelShuffleND(1)`'s + // einops pattern (pixel_shuffle.py:47-52) is what fails instead. So upstream + // refuses the configuration by two different mechanisms depending on a shape + // coincidence, which is exactly why this refuses it by NAME once and up front. + // `scripts/gen-ltx2-pipeline-goldens.py` runs both contradictions through the + // real module and asserts the message, so a pin that changed either mechanism + // fails the generator rather than leaving this paragraph wrong again. + // + // ORDER IS PART OF THE MESSAGE. This `Require` comes BEFORE the rational one + // because upstream's `__init__` is an if/elif chain: `elif temporal_upsample` + // (model.py:68) builds the Conv3d and never consults `rational_resampler`, so + // a config with BOTH flags at dims=2 owns no SpatialRationalResampler and must + // not be told about one. Refuse/accept polarity is the same either way -- the + // eighth cell is refused whichever fires -- which is exactly why only reading + // the message catches a swap back. + Require(!(two_d && config.temporal_upsample), + "ltx2 upsampler: dims=2 with temporal_upsample=true is not a configuration upstream " + "can run. model/upsampler/model.py:68-71 builds the temporal upsampler as a Conv3d, " + "and the dims=2 forward (:85-100) folds the frame axis into the batch and so feeds " + "it a 4-D tensor. The frame axis is gone by then, which is why no 2-D arm can " + "upsample it."); + + // The SIBLING contradiction, and it fails differently from the temporal one: + // every operator inside the rational branch is already per-frame, so a folded + // one-frame volume satisfies all of them and this config computes a complete, + // finite, plausible latent. Upstream cannot: `SpatialRationalResampler.forward` + // opens with `b, _, f, _, _ = x.shape` + // (model/upsampler/spatial_rational_resampler.py:41) and the dims=2 forward + // reaches it at model.py:94 having already folded the frame axis away at :86, + // so torch raises `not enough values to unpack (expected 5, got 4)`. No shape + // check here can see this one, which is why it has to be a refusal. + Require(!(two_d && config.rational_resampler), + "ltx2 upsampler: dims=2 with rational_resampler=true is not a configuration upstream " + "can run. SpatialRationalResampler.forward unpacks five dimensions at " + "model/upsampler/spatial_rational_resampler.py:41, and the dims=2 forward " + "(model/upsampler/model.py:85-100) folds the frame axis into the batch at :86 before " + "reaching it at :94, so it is handed a 4-D tensor and raises."); Require(latent.channels == config.in_channels, "ltx2 upsampler: latent has " + std::to_string(latent.channels) + " channels, config declares " + std::to_string(config.in_channels)); @@ -479,23 +557,25 @@ Ltx2LatentVolume Ltx2LatentUpsample(const Ltx2UpsamplerConfig& config, result.batch = latent.batch; result.channels = config.in_channels; - for (int64_t b = 0; b < latent.batch; ++b) { - Volume x; - x.channels = latent.channels; - x.frames = latent.frames; - x.height = latent.height; - x.width = latent.width; - const int64_t stride = latent.channels * latent.frames * latent.height * latent.width; - x.data.assign(latent.data.begin() + b * stride, latent.data.begin() + (b + 1) * stride); + // model.py:47 again, at the four call sites rather than at the shapes. + const auto conv = [&](const Volume& v, int64_t out_ch, const std::vector& w, + const std::vector& b_) { + return two_d ? Conv2dPad1PerFrame(v, out_ch, w, b_) : Conv3dPad1(v, out_ch, w, b_); + }; - // model.py:102-104. - x = Conv3dPad1(x, config.mid_channels, weights.Get(p + "initial_conv.weight"), - weights.Get(p + "initial_conv.bias")); + // ONE stack for both ranks, because upstream's two forward branches + // (model.py:87-99 and :102-124) run the identical module sequence and differ + // only in the fold around them and in the rank of the four convolutions + // inside. Writing it twice would be the parallel path AGENTS.md forbids. + const auto run_stack = [&](Volume x) { + // model.py:87-89 / :102-104. + x = conv(x, config.mid_channels, weights.Get(p + "initial_conv.weight"), + weights.Get(p + "initial_conv.bias")); GroupNorm(x, weights.Get(p + "initial_norm.weight"), weights.Get(p + "initial_norm.bias")); Silu(x.data); for (int64_t i = 0; i < config.num_blocks_per_stage; ++i) { - x = ResBlockForward(weights, p + "res_blocks." + std::to_string(i) + ".", x); + x = ResBlockForward(weights, p + "res_blocks." + std::to_string(i) + ".", two_d, x); } if (config.temporal_upsample) { @@ -503,6 +583,7 @@ Ltx2LatentVolume Ltx2LatentUpsample(const Ltx2UpsamplerConfig& config, // self.temporal_upsample` (:109) is tested BEFORE the resampler check // (:114). A full 3-D conv (not per-frame): the temporal arm's // `upsampler.0` is a Conv3d (model.py:70), unlike the spatial arm's Conv2d. + // Unreachable when `two_d`, which is refused above. x = Conv3dPad1(x, kLtx2UpsamplerTemporalFactor * config.mid_channels, weights.Get(p + "upsampler.0.weight"), weights.Get(p + "upsampler.0.bias")); x = PixelShuffle1d(x, kLtx2UpsamplerTemporalFactor); @@ -517,17 +598,77 @@ Ltx2LatentVolume Ltx2LatentUpsample(const Ltx2UpsamplerConfig& config, x = PixelShuffle2d(x, rational.num, rational.num); x = BlurDownsample(x, rational.den, kLtx2BlurKernelSize); } else { - // model.py:117-119 — per-frame Conv2d then PixelShuffleND(2). + // model.py:117-119 — per-frame Conv2d then PixelShuffleND(2). Already + // per-frame for BOTH ranks, because the `upsampler` branch never reads + // `dims`: at dims=2 the fold has already made the volume one frame, and + // the operator is the same either way. x = Conv2dPad1PerFrame(x, 4 * config.mid_channels, weights.Get(p + "upsampler.0.weight"), weights.Get(p + "upsampler.0.bias")); x = PixelShuffle2d(x, 2, 2); } for (int64_t i = 0; i < config.num_blocks_per_stage; ++i) { - x = ResBlockForward(weights, p + "post_upsample_res_blocks." + std::to_string(i) + ".", x); + x = ResBlockForward(weights, p + "post_upsample_res_blocks." + std::to_string(i) + ".", + two_d, x); + } + x = conv(x, config.in_channels, weights.Get(p + "final_conv.weight"), + weights.Get(p + "final_conv.bias")); + return x; + }; + + for (int64_t b = 0; b < latent.batch; ++b) { + Volume x; + x.channels = latent.channels; + x.frames = latent.frames; + x.height = latent.height; + x.width = latent.width; + const int64_t stride = latent.channels * latent.frames * latent.height * latent.width; + x.data.assign(latent.data.begin() + b * stride, latent.data.begin() + (b + 1) * stride); + + if (two_d) { + // model.py:86 — `rearrange(latent, "b c f h w -> (b f) c h w")`, undone at + // :100. Running one frame at a time IS that fold: `(b f)` makes every + // frame its own sample, and GroupNorm here reduces over + // `frames * height * width`, so a ONE-frame volume hands it exactly the + // per-frame statistic upstream computes. Passing the whole clip to the + // 2-D convolutions instead would agree on every shape and be wrong in + // every element, which is why the dims=2 golden — not a shape check — is + // what holds this down. + const int64_t in_plane = x.height * x.width; + Volume folded; + for (int64_t f = 0; f < x.frames; ++f) { + Volume plane; + plane.channels = x.channels; + plane.frames = 1; + plane.height = x.height; + plane.width = x.width; + plane.data.assign(static_cast(x.channels * in_plane), 0.0f); + for (int64_t c = 0; c < x.channels; ++c) { + for (int64_t i = 0; i < in_plane; ++i) { + plane.data[static_cast(c * in_plane + i)] = + x.data[static_cast((c * x.frames + f) * in_plane + i)]; + } + } + plane = run_stack(plane); + if (f == 0) { + folded.channels = plane.channels; + folded.frames = x.frames; + folded.height = plane.height; + folded.width = plane.width; + folded.data.assign(static_cast(folded.elems()), 0.0f); + } + const int64_t out_plane = plane.height * plane.width; + for (int64_t c = 0; c < plane.channels; ++c) { + for (int64_t i = 0; i < out_plane; ++i) { + folded.data[static_cast((c * folded.frames + f) * out_plane + i)] = + plane.data[static_cast(c * out_plane + i)]; + } + } + } + x = folded; + } else { + x = run_stack(x); } - x = Conv3dPad1(x, config.in_channels, weights.Get(p + "final_conv.weight"), - weights.Get(p + "final_conv.bias")); result.frames = x.frames; result.height = x.height; diff --git a/src/vllm/multimodal/ltx2_video.cpp b/src/vllm/multimodal/ltx2_video.cpp index 3aa77ae2d..22bf514d7 100644 --- a/src/vllm/multimodal/ltx2_video.cpp +++ b/src/vllm/multimodal/ltx2_video.cpp @@ -414,7 +414,7 @@ constexpr char kLtx2DurationHeadPathExtra[] = "duration_head_path"; // they are no longer trusted: the list below is derived from this file on every // run and compared, and the failure prints the replacement to paste in. // READER ANCHORS (derived and gated by test_ltx2_video): -// 1079 1089 1090 1176 1272 1288 1377 1381 1484 1546 1654 1696 1738 1740 +// 1079 1089 1090 1176 1272 1288 1377 1381 1484 1562 1670 1712 1754 1756 const char* const kKnownLoadExtras[] = { kLtx2AudioPromptEmbedsExtra, kLtx2PipelineKindExtra, kLtx2ModelVersionExtra, @@ -1523,6 +1523,22 @@ std::unique_ptr Ltx2VideoEngine::Load(const VideoModelParams& p "width untouched), so a checkpoint that also doubles the spatial axes would return " "a latent no tile range can index."); } + // THE THIRD SIBLING, and it belongs beside the other two rather than in the + // rounds loop. `Ltx2LatentUpsample` refuses dims=2 with temporal_upsample — + // it is a contradiction upstream cannot run — but that refusal fires at the + // first round, which is reached only after two full denoise stages. This + // block exists so a caller who supplied the wrong file learns at LOAD, which + // is the reason its own comment above gives, and it checked both flags here + // without checking `dims`, so this one checkpoint cleared both siblings and + // failed minutes in anyway. + if (im.temporal_upsampler_cfg.dims == 2) { + Fail("the checkpoint at '" + std::string(kLtx2TemporalUpsamplerPathExtra) + + "' declares dims=2 with temporal_upsample=true, which upstream cannot run: " + "model/upsampler/model.py:68-71 builds the temporal upsampler as a Conv3d, and the " + "dims=2 forward (:85-100) folds the frame axis into the batch before reaching it, so " + "it is handed a 4-D tensor. The frame axis is gone by then, which is why no 2-D arm " + "can upsample it."); + } im.temporal_upsampler_weights = Ltx2LoadVaeWeights(f); im.has_temporal_upsampler = true; } @@ -3471,7 +3487,7 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { // supplied a genuine SPATIOTEMPORAL checkpoint that they had handed over the // temporal one — wrong on both counts, and pointing them at the arm they // already had. It also shadowed the ledger refusal at - // `ltx2_upsampler.cpp:465`, which names the spatiotemporal arm and was + // `ltx2_upsampler.cpp:497`, which names the spatiotemporal arm and was // therefore unreachable from any request. Narrowed here so a both-flags // config falls THROUGH to that refusal. Gated by test_ltx2_video's // "a SPATIOTEMPORAL upsampler checkpoint is refused as SPATIOTEMPORAL". diff --git a/tests/vllm/models/ltx2_pipeline_goldens.inc b/tests/vllm/models/ltx2_pipeline_goldens.inc index d3a2814e2..bd1844af2 100644 --- a/tests/vllm/models/ltx2_pipeline_goldens.inc +++ b/tests/vllm/models/ltx2_pipeline_goldens.inc @@ -2207,6 +2207,409 @@ inline constexpr float kLtx2UpsTemporalGolden[] = { 0.485517055f, 0.0335913487f, 0.55690515f, -0.0289731808f, 0.885259986f, -0.610681891f, }; +inline constexpr int64_t kLtx2UpsDims2Frames = 3; +inline constexpr float kLtx2UpsDims2Latent[] = { + 0.181088224f, -0.0120419953f, 0.795654356f, 0.125761703f, 0.334531128f, 0.159080729f, + 0.734152257f, -0.916255772f, -0.405625641f, -0.857130706f, -0.0135827949f, -0.95731622f, + 0.218598366f, -0.738133371f, -0.93304944f, -0.540049732f, -0.595831811f, 0.406834334f, + 0.479516178f, -0.450047523f, 0.13826327f, 0.212332621f, 0.473556548f, -0.0629030913f, + -0.929635048f, -0.760962784f, 0.171503201f, -0.0701538846f, 0.909344077f, -0.0439069271f, + -0.897934556f, 0.968792498f, 0.0636623949f, 0.477928549f, -0.97080487f, 0.422710538f, + 0.249427438f, 0.663199067f, -0.354463995f, 0.965093732f, -0.45280695f, 0.000621905609f, + 0.26203236f, 0.772810817f, 0.134890392f, 0.599602878f, -0.132872984f, -0.14700821f, + -0.186663374f, 0.393290848f, 0.348897517f, 0.160501942f, -0.79594177f, -0.782469332f, + -0.570157886f, -0.178150967f, 0.58774966f, -0.31347844f, 0.488238871f, 0.880881786f, + -0.0242416058f, 0.761379838f, 0.961615682f, 0.887404084f, -0.916659176f, -0.0510472655f, + -0.470835447f, 0.689592183f, -0.690772891f, 0.410303295f, -0.422379225f, -0.8544119f, + 0.561585665f, -0.942586243f, -0.163454786f, -0.141976655f, 0.320403814f, 0.817669749f, + -0.998528898f, -0.108177818f, 0.826877296f, -0.633625448f, -0.448734879f, -0.246771231f, + -0.69879669f, 0.265871793f, 0.457926452f, 0.147060946f, -0.73108995f, 0.450917244f, + -0.387186855f, -0.366325825f, 0.215132117f, 0.558014989f, -0.997626603f, -0.282812238f, + 0.833082139f, 0.835577667f, -0.0217689443f, 0.0616079867f, -0.692652345f, 0.642827868f, + 0.512572885f, -0.0403843448f, -0.841926038f, -0.407603651f, -0.356280863f, 0.577077031f, + 0.606764257f, 0.191563174f, 0.49638924f, -0.374608815f, -0.725797117f, -0.322677642f, + 0.0715746507f, 0.580258012f, 0.640728951f, -0.569805145f, 0.184176162f, -0.596500754f, + -0.49173215f, -0.327561408f, 0.963382244f, -0.197386175f, 0.597814083f, -0.44003281f, + -0.326859206f, 0.634899139f, 0.561330557f, 0.325108409f, -0.692298889f, 0.0560987815f, + 0.702719808f, -0.491130829f, -0.590169787f, 0.170375422f, 0.977761269f, 0.49000448f, + 0.801072955f, 0.46383822f, 0.768134475f, 0.447700053f, 0.697136343f, 0.00571768638f, + -0.399555415f, 0.868342042f, -0.285770535f, -0.550402164f, 0.150562778f, -0.0849747956f, + 0.148855552f, 0.669201076f, 0.34502995f, 0.78261292f, 0.556730628f, 0.900611937f, + 0.319647163f, 0.702498794f, -0.558810472f, 0.661898017f, 0.176946059f, 0.104238532f, + -0.967356443f, -0.439534217f, -0.138429314f, 0.419104934f, -0.926414073f, -0.454348624f, + -0.121448852f, -0.711981714f, -0.379177928f, -0.114980228f, -0.851304591f, -0.296427786f, + -0.0575137734f, 0.373382062f, -0.622730553f, 0.386007309f, 0.996865571f, -0.895684004f, + 0.677476645f, 0.95827347f, -0.93209976f, -0.370653927f, -0.703423679f, 0.1060348f, + 0.576770842f, -0.547168374f, 0.0923096165f, -0.961260676f, -0.7623173f, -0.485876173f, + -0.798021913f, 0.981109262f, -0.806658089f, -0.912253141f, 0.714325845f, -0.701749861f, + -0.472889066f, 0.394762099f, -0.617031753f, 0.0977048799f, -0.864341021f, 0.472547412f, + 0.608497679f, 0.97790581f, -0.750700772f, 0.790310681f, 0.157977283f, 0.405681044f, + -0.418317378f, 0.0427558534f, 0.0898217261f, -0.790034473f, 0.493627876f, -0.46424666f, + 0.689468384f, 0.455211133f, -0.333523065f, -0.765941501f, 0.233043909f, -0.419253588f, + 0.146960735f, 0.352034956f, -0.518569112f, 0.0101857334f, 0.132130325f, 0.607055187f, + -0.0742827728f, 0.407203197f, -0.334842086f, -0.0538440645f, -0.172338501f, -0.53731817f, + -0.562205315f, 0.842431009f, 0.301437646f, 0.570695996f, 0.481191039f, -0.421348214f, + 0.20796068f, 0.352692664f, -0.873777568f, 0.462710828f, 0.130206123f, 0.233702928f, + -0.21180144f, -0.735891759f, -0.0984804258f, -0.400337577f, 0.880786121f, 0.119920053f, + -0.307294279f, 0.856267691f, -0.816491842f, 0.464350671f, -0.269940495f, 0.443350911f, + 0.265402824f, -0.155549571f, -0.472514004f, 0.988085628f, -0.0647370666f, -0.826126158f, + 0.485464215f, 0.0540195443f, 0.145360678f, 0.169680238f, 0.636446059f, -0.0486890748f, + 0.0898618326f, 0.621376216f, -0.134600118f, -0.0620668754f, 0.342687428f, -0.918920159f, + 0.639202118f, -0.598030031f, -0.666994274f, 0.257202178f, 0.806539834f, 0.463929743f, + -0.46360299f, -0.95618993f, -0.855308354f, 0.226559997f, 0.556455612f, -0.622835159f, + -0.948379874f, -0.262026846f, 0.154110909f, 0.0107685169f, 0.181660697f, -0.507305801f, + -0.974269927f, 0.990030408f, -0.0336277299f, 0.520975649f, -0.392803133f, -0.564396262f, + -0.0444994718f, 0.206658274f, -0.809571683f, -0.349311739f, 0.488761038f, -0.9374488f, + -0.056082882f, 0.750094414f, -0.14624925f, -0.412210673f, 0.810554981f, 0.836163878f, + 0.686124682f, 0.46389991f, 0.717097402f, -0.503414333f, 0.363496155f, 0.242195949f, + -0.445271701f, -0.757464945f, 0.96809274f, -0.439795166f, 0.562856913f, 0.96900636f, + -0.602451026f, -0.0595050789f, -0.912559867f, 0.209086701f, -0.821123004f, 0.628860533f, + -0.424943149f, 0.0910921246f, -0.839263141f, -0.0404860713f, 0.467127711f, 0.118507475f, + 0.335788369f, -0.0555304773f, -0.822860479f, -0.25254342f, 0.595494449f, 0.206198201f, + 0.668388486f, -0.394215435f, -0.42488277f, -0.0281695649f, 0.944485664f, -0.429031193f, + -0.0434612408f, 0.815118432f, -0.183611661f, -0.851008475f, -0.242832243f, 0.467449218f, + -0.328020275f, 0.04309671f, 0.741672218f, 0.370259464f, 0.341159523f, -0.588085949f, + 0.136303276f, 0.924581945f, -0.999050617f, 0.579241753f, 0.846957684f, -0.988440752f, + 0.481850684f, 0.166262612f, -0.402408212f, -0.0776432306f, -0.171219543f, -0.600280166f, + -0.892585337f, -0.578511f, -0.63526696f, -0.738455951f, -0.363838822f, 0.0778452381f, + -0.488764495f, -0.923031986f, -0.46038568f, -0.665779352f, 0.518688977f, 0.187142059f, + 0.44007656f, 0.580231845f, -0.510262072f, 0.740038931f, -0.00540134497f, -0.132687137f, + -0.486506462f, 0.791014791f, 0.348093361f, -0.83424747f, 0.644981861f, 0.171034947f, + 0.258129895f, 0.234581366f, -0.195583478f, 0.0200211871f, -0.836870551f, 0.902238071f, + 0.177294716f, 0.826213837f, 0.313137591f, 0.0814262852f, 0.0039157928f, -0.4860183f, + 0.650961339f, 0.787454963f, -0.463933945f, 0.685996532f, 0.974322677f, -0.553694665f, + -0.361299753f, -0.240121633f, 0.734097004f, 0.274333596f, -0.273647875f, 0.338060737f, + 0.938043058f, -0.422803104f, 0.766340435f, -0.962049127f, 0.267524004f, 0.921048224f, + -0.839280844f, 0.0103274118f, -0.725435138f, -0.281520009f, -0.866512716f, 0.0514708795f, +}; + +inline constexpr int64_t kLtx2UpsDims2OutShape[] = { + 1, 6, 3, 8, 12, +}; + +inline constexpr const char* kLtx2UpsDims2ParamNames[] = { + "ltx2.ups.Dims2.initial_conv.weight", + "ltx2.ups.Dims2.initial_conv.bias", + "ltx2.ups.Dims2.initial_norm.weight", + "ltx2.ups.Dims2.initial_norm.bias", + "ltx2.ups.Dims2.res_blocks.0.conv1.weight", + "ltx2.ups.Dims2.res_blocks.0.conv1.bias", + "ltx2.ups.Dims2.res_blocks.0.norm1.weight", + "ltx2.ups.Dims2.res_blocks.0.norm1.bias", + "ltx2.ups.Dims2.res_blocks.0.conv2.weight", + "ltx2.ups.Dims2.res_blocks.0.conv2.bias", + "ltx2.ups.Dims2.res_blocks.0.norm2.weight", + "ltx2.ups.Dims2.res_blocks.0.norm2.bias", + "ltx2.ups.Dims2.upsampler.0.weight", + "ltx2.ups.Dims2.upsampler.0.bias", + "ltx2.ups.Dims2.post_upsample_res_blocks.0.conv1.weight", + "ltx2.ups.Dims2.post_upsample_res_blocks.0.conv1.bias", + "ltx2.ups.Dims2.post_upsample_res_blocks.0.norm1.weight", + "ltx2.ups.Dims2.post_upsample_res_blocks.0.norm1.bias", + "ltx2.ups.Dims2.post_upsample_res_blocks.0.conv2.weight", + "ltx2.ups.Dims2.post_upsample_res_blocks.0.conv2.bias", + "ltx2.ups.Dims2.post_upsample_res_blocks.0.norm2.weight", + "ltx2.ups.Dims2.post_upsample_res_blocks.0.norm2.bias", + "ltx2.ups.Dims2.final_conv.weight", + "ltx2.ups.Dims2.final_conv.bias", +}; +inline constexpr int64_t kLtx2UpsDims2ParamCounts[] = { + 1728, 32, 32, 32, 9216, 32, 32, 32, 9216, 32, + 32, 32, 36864, 128, 9216, 32, 32, 32, 9216, 32, + 32, 32, 1728, 6, +}; + +inline constexpr float kLtx2UpsDims2Golden[] = { + -0.174188003f, -0.19056657f, 0.0674376339f, -0.324625552f, 0.379167795f, -0.497133315f, + 0.476703435f, -0.0960824937f, -0.164228439f, -0.0728208274f, 0.310943574f, -0.165813327f, + -0.158851951f, 0.477083504f, 0.120832056f, 0.0885711759f, 0.606073141f, -0.852977931f, + 0.231014937f, 0.278964043f, 0.160233676f, -0.00428466499f, 0.026458092f, -0.0843308717f, + -0.411712289f, -0.0570101105f, 0.0985209271f, -0.228669003f, -0.116717815f, -0.486097187f, + -0.67689842f, -0.15679203f, 0.166315615f, -0.243245721f, 0.191971987f, -0.506874263f, + -0.151411936f, 0.053874597f, 0.41609621f, 0.1869919f, -0.111943692f, 1.1880939f, + 0.385846466f, 0.434614837f, 0.551352978f, 0.0241067484f, -0.230742335f, -0.286054015f, + -0.176392466f, 0.369276613f, 0.564570904f, -0.379709572f, -0.445709586f, 0.51557976f, + 0.589401543f, -0.843937635f, -0.147408843f, -0.474536479f, 0.141456142f, -0.0248531736f, + 0.223881423f, -0.0526425987f, 0.12343435f, 0.20411253f, 0.848881602f, -0.318271279f, + 0.525307596f, -0.671241879f, -0.173208714f, 0.772630334f, 0.273529738f, -0.203770041f, + -0.093548432f, 0.0635024011f, -0.292020947f, -0.42276749f, 0.296650946f, -0.183545917f, + 0.707106352f, -0.32376793f, -0.149354666f, 0.0407520086f, -0.281864345f, 0.0408807099f, + 0.0846381336f, -0.156930298f, -0.266610831f, 0.0865510404f, -0.0175738931f, -0.0963223428f, + -0.24679029f, 0.203335583f, -0.282579213f, -0.269701719f, -0.0140439477f, 0.11923226f, + 0.0404552445f, -0.100669257f, 0.26880151f, -0.0592160001f, -0.300730318f, -0.482423216f, + 0.11162626f, -0.401732445f, 0.240106687f, 0.000170469284f, 0.135924876f, -0.174968913f, + -0.210822612f, -0.151006743f, -0.0091766715f, 0.795213163f, -0.601685405f, -0.560323238f, + 0.512979984f, -0.022885561f, 0.449440122f, 0.253897727f, -0.114784636f, 0.153080434f, + -0.560768247f, -0.11377833f, -0.568962872f, 0.136600405f, -0.410836726f, -0.352175593f, + 0.396457165f, -0.764952958f, 0.297196388f, -0.550908089f, -0.39912647f, -0.231846154f, + -0.10240414f, 0.50171876f, -0.259371161f, -0.0989836007f, 1.13193107f, 0.146968156f, + 0.530135334f, -0.116997205f, 0.30090642f, 0.406279594f, 0.225569561f, 0.0494384281f, + -0.279779255f, -0.211114466f, -0.581426144f, -0.965393066f, 0.416291535f, 0.425104469f, + -0.164309382f, -0.184698403f, 0.374219954f, -0.332437128f, 0.473272324f, -0.0703949556f, + 0.0522284508f, 0.189426571f, 0.487077653f, -0.195648521f, 0.606073558f, 0.24777323f, + 0.161744267f, -0.0731854513f, 0.697260618f, -0.0590750426f, 0.0200545415f, -0.102297761f, + 0.302042931f, -0.126213342f, -0.0823383778f, 0.352176487f, 0.289914697f, 0.0354605466f, + 0.389330715f, -0.289072335f, -0.425107598f, 0.0746738911f, -0.653491735f, -0.310157478f, + -0.124902025f, 0.0759057105f, -0.160330147f, 0.0102494061f, 0.000458524097f, -0.0411158949f, + -0.323645949f, 0.055761639f, -0.289503932f, 0.0990386382f, 0.08707349f, -0.0465130396f, + -0.0376012288f, -0.0396953076f, 0.0503035039f, -0.220177606f, 0.209226087f, -0.102195941f, + 0.277406812f, -0.21933265f, 0.0510927625f, 0.025872007f, 0.145196259f, -0.171402216f, + -0.058267355f, -0.0805187747f, -0.0090585798f, 0.213781655f, -0.204092503f, 0.240091711f, + -0.00825726986f, 0.469817996f, -0.0222599152f, -0.277051806f, 0.242125452f, 0.21371451f, + -0.163020819f, 0.00309440494f, -0.0473117642f, -0.270024478f, -0.942484975f, -0.485522717f, + -0.306049883f, -0.0649293214f, 0.0630688667f, -0.305936098f, -0.194858447f, -0.105517842f, + -0.266118377f, 0.0618316829f, 0.127231613f, 0.0755237788f, -0.587517142f, 0.0297779236f, + 0.575198472f, 0.150575086f, 0.24365814f, 0.648334265f, -0.104864314f, 0.0506706089f, + -0.25750342f, 0.0528887212f, 0.347831935f, 0.304933369f, 0.268673241f, -0.240385637f, + 0.464707553f, -0.123323329f, 0.585772872f, -0.257270426f, -0.401115239f, -0.233988613f, + 0.12462914f, 0.108220696f, -0.194274053f, -0.146172225f, 0.841643572f, -0.144756973f, + -0.0716652349f, -0.0693205744f, 0.420592308f, -0.0992206708f, 0.139777631f, 0.0737165362f, + -0.199594855f, -0.285042346f, 0.18103905f, -0.716760218f, 0.231604472f, -0.00273196399f, + -0.0263817683f, 0.179990917f, 0.27927807f, -0.666725397f, -0.310269415f, -0.0798714459f, + 0.0155901611f, -0.0457675457f, 0.349891186f, -0.587074757f, 0.209341198f, 0.00894902647f, + 0.0101971477f, -0.0274719745f, -0.258497059f, -0.127556205f, -0.11292436f, -0.0936131924f, + -0.27605015f, -0.271669388f, -0.142207369f, 0.0393095873f, -0.409242213f, -0.14702417f, + -0.500193477f, 0.0253907852f, -0.116600655f, -0.362053007f, -0.0448269844f, -0.0702195615f, + -0.108686596f, -0.432862639f, -0.64817667f, 0.294467628f, -0.241657734f, -0.741521955f, + -0.483769953f, -0.456901997f, -0.328767419f, -0.25869137f, -0.121877581f, -0.26355046f, + -0.137378186f, 0.18713218f, -0.215106457f, -0.644768715f, -0.263201714f, 0.0902615637f, + -1.11715019f, 0.158314705f, 0.325921565f, -0.28438279f, 0.0699375495f, 0.214683115f, + 0.293959558f, -0.951837659f, -0.368970037f, -0.642544389f, -1.16186607f, -0.0462333262f, + -0.0400216579f, -0.970490575f, -0.696104884f, -0.412809193f, -0.127705097f, 0.201611444f, + -0.157822654f, 0.18961367f, -0.247199103f, 0.361305356f, -0.0282814354f, -0.520739675f, + -0.201827079f, 0.370734334f, -0.458566546f, 0.190316975f, -0.198256522f, 0.0252460688f, + 0.186364606f, -0.503503799f, 0.00675155967f, -0.761315465f, 0.0217808187f, -0.986500025f, + 0.232764542f, -0.314198971f, -0.769356847f, -0.93380785f, -0.227795869f, -0.141494185f, + 0.0985975936f, -0.0375447422f, -0.0517667308f, 0.00409392267f, 0.129133373f, 0.201754406f, + -0.273640692f, -0.0854163617f, 0.24318859f, -0.081100218f, -0.00516046584f, 0.101659022f, + 0.257008702f, -0.349522263f, 0.481587589f, -0.107755363f, -0.00337427855f, -0.238903761f, + -0.10696698f, 0.135213852f, 0.0978900865f, -0.216973662f, 0.22595337f, 0.156452015f, + -0.131653324f, -0.0352797955f, -0.164765865f, -0.206746161f, -0.324864805f, -0.0750838295f, + 0.0106421188f, -0.290402651f, 0.00362143666f, -0.501441598f, 0.0319297165f, -0.0261260886f, + -0.0011170879f, -0.0666110665f, -0.759366989f, -0.424796939f, -0.383632243f, -0.248578727f, + -1.06661236f, -0.532866478f, -0.106295481f, 0.0236105192f, 0.0172626004f, -0.155730516f, + -0.373221695f, 0.1635371f, -0.0407338664f, -0.116581157f, -0.845293105f, 0.565847576f, + -0.442639917f, -0.515133381f, -0.533615172f, 0.236676455f, 0.299115419f, -0.124146901f, + 0.124964237f, -0.292885363f, -0.970242262f, -0.0500882156f, 0.0670315325f, -1.07895494f, + -0.767366648f, -0.983762681f, -0.904300749f, -0.00325004756f, -0.293683261f, -0.147575811f, + -0.0959611088f, 0.638485193f, 0.221430063f, -0.385823935f, 0.115972772f, 0.139215112f, + 0.257407635f, 0.123169556f, -1.06472254f, -0.324829161f, 0.287888408f, 0.0518415645f, + 0.216901898f, -0.490948856f, -0.268682957f, -0.241587996f, -0.577601075f, -0.619583309f, + -0.274089485f, -0.169404015f, -0.24234578f, -0.613620818f, -0.283509642f, -0.0877766609f, + 0.264492363f, 0.0258878097f, -0.288474679f, 0.14130488f, -0.393508404f, 0.179828182f, + 0.138505638f, -0.566798449f, 0.625209689f, 0.25661099f, -0.282390952f, 0.132131979f, + 0.185555875f, 0.066414535f, 0.203220725f, -0.322863042f, 0.14493674f, -0.197400481f, + 0.0844270289f, -0.0310366526f, -0.00416995957f, 0.244300872f, 0.136735991f, -0.0103929807f, + 0.113290764f, -0.266795814f, -0.0738740042f, -0.277734727f, -0.224043205f, -0.1987032f, + -0.148748159f, -0.303484797f, -0.00671050698f, -0.114812061f, 0.0921058133f, 0.0678730309f, + -0.156062081f, -0.168800116f, -0.176072299f, 0.322793007f, -0.182530954f, -0.941022277f, + -0.756128669f, -0.457079768f, -0.299289286f, 0.0132683963f, -0.557795048f, -0.0207253098f, + -0.0502150804f, -0.019246459f, -0.397828043f, 0.0172357857f, 0.279028624f, -0.371406615f, + -0.744699895f, -0.387808442f, -0.0496607721f, 0.0226008743f, -0.151594728f, 0.0838017538f, + 0.157449454f, -0.72301805f, -0.54917407f, -0.372718215f, -0.375807494f, -0.662977636f, + -0.314363599f, -0.337717175f, -0.335553229f, 0.103869259f, -0.193515852f, -0.271663427f, + -0.461314619f, -0.00688380003f, -0.297023118f, -0.167437524f, -0.232199937f, 0.0634251162f, + -1.4166646f, -0.790597022f, -0.175200999f, 0.0474257544f, 0.21910432f, -0.0417810977f, + 0.063192755f, -0.416039348f, -0.51031214f, -0.0693283379f, -0.300075471f, -0.591353178f, + -0.392669976f, -0.702080786f, -0.189393461f, -0.0436507016f, -0.494403273f, -0.213343039f, + -0.0956978202f, 0.283381134f, 0.197630912f, -0.455120265f, 0.324013144f, 0.0842273235f, + 0.22386843f, -0.280572861f, -0.399609596f, -0.0301891565f, 0.0722124875f, -0.167019337f, + 0.435171783f, -0.120845973f, 0.609409809f, 0.570730567f, -0.668394327f, -0.0842728689f, + 0.348645091f, -0.219201148f, 0.241116315f, 0.0478282273f, 0.194491893f, 0.226213142f, + 0.103526115f, 0.0695117712f, 0.0272003338f, 0.451404095f, -0.000448621809f, 0.0686458349f, + -0.138840735f, 0.235500216f, -0.0178735293f, 0.263875306f, 0.161568999f, -0.0700050369f, + -0.093469359f, -0.00517371297f, -0.0421015769f, 0.110727072f, -0.285820365f, -0.149659693f, + -0.733068407f, 0.130311459f, -0.279981256f, 0.107047357f, 0.209676638f, 0.0142957419f, + 0.299762726f, -0.263884664f, -0.834542632f, 0.108980134f, -0.0409950167f, -0.542257905f, + -0.332090199f, -0.531186223f, -0.109518863f, -0.4028835f, -0.0990753323f, -0.0543839708f, + -0.254162908f, 0.223360047f, -0.351595283f, 0.00597658753f, -0.924431324f, 0.142797053f, + -0.378358662f, -0.00698576868f, -0.524465501f, 0.34058255f, 0.0779817104f, 0.1425208f, + 0.12394686f, 0.332214773f, -0.178609058f, 0.0194249749f, -0.185431898f, -0.0430174768f, + 0.420395434f, -0.0886029899f, -0.243834913f, 0.274149776f, -0.0608870536f, -0.0188300833f, + -0.290267646f, 0.431677759f, -0.324621022f, 0.252026916f, -0.520924389f, 0.813670158f, + -0.476995707f, 0.47789824f, 0.394927442f, -0.301170826f, 0.057179898f, -0.0345475227f, + -0.0792018324f, -0.0380874723f, 0.0848468766f, -0.202589393f, -0.189626664f, -0.18296285f, + -0.40740037f, 0.0205977634f, 0.223917261f, 0.133288056f, 0.200421274f, -0.336890638f, + -0.166087747f, 0.141862586f, 0.328541875f, 0.101939902f, 0.204850122f, -0.140600204f, + -0.0356298126f, 0.188759044f, 0.0830793604f, 0.0504732281f, -0.390204251f, -0.130579218f, + 0.00914085284f, -0.0325093828f, 0.0946474448f, 0.208101138f, -0.279373556f, -0.301461101f, + -0.0250428785f, 0.0666632205f, 0.235876739f, -0.0871038139f, -0.229082704f, 0.135999888f, + 0.0179442838f, 0.374537617f, -0.29257983f, -0.0411619619f, -0.625857949f, 0.272999823f, + -0.399761796f, -0.0636012182f, -0.185430378f, 0.345329225f, 0.0983758718f, -0.276695251f, + 0.261641532f, 0.215027869f, -0.11779283f, 0.0601113364f, -0.476812959f, 0.447129726f, + 0.438205421f, -0.737962484f, -0.27129963f, 0.141761452f, -0.0114507526f, -0.156988636f, + -0.304740846f, -0.062183544f, -0.417303503f, 0.222160816f, -0.292296231f, 0.515023351f, + -0.304025054f, 0.0007199049f, -0.645695627f, 0.163706884f, -0.0142798424f, -0.1060417f, + 0.247226059f, -0.323973447f, 0.196322218f, 0.541672945f, -0.0132012665f, -0.758261681f, + -0.505240738f, 0.0626065806f, -0.0647694394f, -0.115146324f, -0.159103647f, 0.0601017475f, + -0.249742568f, 0.515628934f, 0.0929552913f, 0.0453907326f, -0.296602398f, 0.394620717f, + -0.371505022f, 0.24597168f, -0.0618017912f, 0.145207763f, -0.310844898f, 0.165483326f, + 0.179680794f, 0.115456834f, 0.00299241394f, 0.0094692409f, 0.357498676f, -0.175308391f, + -0.45826298f, 0.0565670207f, 0.0529623032f, -0.280635208f, 0.207495108f, -0.103173383f, + -0.0118117668f, 0.0777033567f, 0.145031542f, -0.185621396f, -0.386643797f, -0.0935347527f, + -0.0240512714f, 0.053439416f, 0.00768299401f, 0.146285623f, -0.280498952f, -0.0820012987f, + 0.314474583f, -0.0844391435f, -0.115575582f, 0.403500199f, -0.101030864f, 0.0118706971f, + -0.172192588f, -0.595462799f, -0.358914137f, -0.206985325f, 0.281515449f, 0.123655163f, + -0.138578564f, -0.0451957583f, -0.471632183f, -0.163167223f, 0.060606271f, -0.430965066f, + -0.428130627f, 0.0239109546f, 0.0690359175f, -0.19524245f, -0.1857512f, -0.0294173062f, + -0.153938353f, -0.312311798f, 0.027086854f, -0.0453756899f, -0.1420919f, -0.277209759f, + 0.0955872834f, -0.198139906f, -0.226966679f, -0.170816153f, -0.0605729148f, 0.112789989f, + -0.127119079f, 0.119506091f, -0.736781836f, 0.572638452f, -0.615827262f, 0.182346925f, + -0.330283135f, 0.243072763f, -0.249934003f, -0.00375910103f, 0.274342537f, 0.136966899f, + -0.144159302f, 0.663296223f, -0.199919015f, -0.853017449f, -0.12073119f, 0.480327338f, + 0.0714192018f, -0.612998366f, 0.13612482f, 0.376488328f, 0.310026228f, -0.000280916691f, + -0.144830897f, 0.0123759732f, -0.486645609f, 0.142163098f, 0.086750567f, 1.10744262f, + -0.445039511f, 0.239563972f, 0.336355954f, 0.16845879f, -0.161429211f, 0.0598516762f, + 0.340604275f, 0.182850361f, -0.272716224f, 0.166407064f, -0.238973707f, 0.339752376f, + -0.37842226f, -0.00167253055f, 0.268601686f, -0.202768818f, -0.0797307044f, 0.0059452951f, + 0.00645658979f, 0.394459218f, 0.500231564f, -0.454792351f, -0.33398813f, 0.302670211f, + -0.329709649f, 0.00646245014f, 0.31128639f, 0.0660012364f, -0.0958865657f, -0.271848053f, + 0.0276435092f, -0.103072695f, 0.269349873f, 0.0707909614f, -0.109695688f, 0.114203319f, + -0.254692256f, 0.33112967f, -0.145400524f, 0.231654108f, 0.00433222577f, 0.00127281994f, + -0.185765922f, -0.0456727147f, -0.0773763955f, 0.16544342f, -0.203457922f, -0.224551603f, + 0.0594727919f, 0.439424425f, -0.592717826f, 0.197676063f, -0.0561501235f, -0.325547755f, + -0.182620868f, 0.236791447f, -0.512256384f, 0.410075456f, 0.292133659f, 0.737094343f, + 0.359535217f, 0.225151896f, -0.487968683f, -0.0240837187f, 0.271048009f, 0.18644388f, + 0.132456541f, 0.148454666f, -0.313128173f, 0.392565399f, 0.227742881f, 0.973639548f, + -0.241943926f, 0.3428967f, -0.121946461f, 0.369866163f, -0.191075787f, -0.10631939f, + -0.12958926f, 0.569521964f, -0.144529611f, 0.29896462f, -0.249582887f, -0.163345829f, + 0.547636688f, 0.369260311f, 0.0605326965f, 0.131736606f, -0.0821396858f, 0.472731709f, + -0.16583769f, -0.186108872f, 0.0568353832f, 0.741481662f, -0.753222764f, 0.478188455f, + -0.315870285f, 0.480199873f, 0.0133511722f, -0.207949445f, 0.254877657f, -0.0889365748f, + -0.134125173f, 0.295174092f, 0.128026575f, -0.046454832f, -0.453294605f, -0.0951894671f, + 0.451915145f, 0.531752825f, 0.268047482f, 0.305936903f, 0.00758378208f, -0.013047684f, + 0.0111742094f, -0.0653666407f, -0.145532221f, -0.346878946f, -0.328903705f, -0.0824568272f, + -0.309459239f, -0.295349836f, 0.315883875f, 0.211758271f, -0.00685027242f, -0.102043323f, + 0.071591951f, 0.17344594f, 0.0328767113f, 0.0509981513f, 0.102618426f, 0.274998724f, + -0.110857069f, 0.113918923f, -0.1603425f, 0.000202134252f, -0.121055678f, 0.0361575857f, + -0.0650050938f, 0.100662895f, -0.59185946f, 0.404326439f, 0.324480206f, 0.155593082f, + -0.336397678f, 0.0879899263f, -0.105308339f, 0.11096856f, 0.1515356f, -0.126025423f, + 0.252498239f, 0.0277675204f, 0.304712981f, 0.500546873f, 0.248947456f, 0.404363126f, + -0.39211008f, 0.204110011f, -0.286089003f, 0.485109419f, -0.021686025f, -0.0216803327f, + 0.0924726874f, -0.0602575541f, 0.101176925f, 0.648777366f, 0.0789405257f, 0.259479791f, + 0.0623849183f, 0.173424095f, -0.837495804f, 0.139569104f, 0.193046302f, 0.0165695753f, + -0.21198757f, -0.0612483099f, 0.210769653f, 0.208085924f, 0.47590062f, 0.512484193f, + 0.664914966f, 0.520836771f, 0.0987238884f, 0.289289266f, -0.0791569352f, -0.0134095028f, + 0.0544062033f, 0.362598121f, -0.66686064f, 0.311635792f, -0.394515336f, 0.254220426f, + 0.118883401f, 0.532184482f, -0.204117924f, 0.032267645f, -0.130263358f, 0.0479741693f, + -0.086835131f, -0.245476991f, -0.109186962f, -0.0847499445f, 0.0924869031f, 0.10439387f, + 0.333466202f, 0.400474668f, -0.0594672784f, 0.0905969143f, 0.0242707431f, -0.0106039047f, + -0.0519451797f, 0.0123407189f, 0.013446385f, 0.207491651f, 0.269147217f, 0.110793382f, + -0.189392298f, 0.445392787f, -0.0256277993f, 0.149685442f, -0.222952396f, -0.129785642f, + 0.000632800162f, 0.229996443f, 0.0596598834f, 0.0538125634f, -0.196528748f, -0.0261976719f, + -0.0966887772f, 0.372852892f, 0.105312377f, 0.155139834f, 0.0890397057f, 0.0703629255f, + -0.160411507f, -0.0330938548f, -0.556431353f, 0.0236040056f, 0.014960289f, 0.866781473f, + -0.0171927065f, 0.361750245f, -0.145273805f, -0.0864485949f, -0.217848539f, -0.233475417f, + -0.0921870023f, -0.00911326706f, -0.214374557f, 0.406815886f, -0.0190701708f, 0.311112791f, + -0.0951131731f, 0.199975625f, 0.301987469f, 0.0555290543f, 0.178921416f, 0.0242535025f, + -0.074621208f, -0.349460065f, -0.485584348f, 0.968082607f, 0.140515685f, 0.20685181f, + -0.377713621f, 0.267317057f, -0.214633435f, 0.185337588f, 0.152142003f, 0.193830177f, + -0.016866073f, 0.503222644f, 0.208358034f, 0.508968353f, 0.0793749988f, 0.540182948f, + 0.329143584f, 0.493190557f, 0.192696914f, 0.0467872322f, 0.223433003f, 0.156217754f, + 0.171620965f, 0.0284073725f, 0.198744267f, 0.45991683f, -0.0440253243f, 0.144220829f, + 0.094504118f, 0.383900493f, -0.244637564f, 0.0807483941f, 0.0627801418f, 0.190469876f, + -0.168619454f, 0.0306364149f, -0.163394779f, 0.167155132f, -0.508452952f, 0.246902704f, + 0.214514986f, 0.137167394f, 0.350961745f, 0.0778658316f, -0.0817017853f, -0.0197854862f, + -0.0187255293f, 0.39063549f, -0.442286074f, 0.310407162f, -0.160236225f, 0.221253723f, + -0.00331787765f, 0.271175623f, -0.169235781f, 0.0517389774f, -0.397500247f, -0.0214155167f, + 0.242373765f, 0.141345844f, 0.30392012f, -0.0593039989f, 0.227418318f, -0.132553205f, + 0.309975982f, -0.153068125f, 0.0724881738f, -0.027438812f, 0.113973878f, -0.238781571f, + 0.00154996966f, 0.365385264f, 0.0115920678f, 0.178200483f, -0.0829066634f, -0.393446207f, + -0.395860165f, -0.376394331f, 0.0314556733f, -0.405982882f, -0.000776132569f, -0.0699861795f, + 0.127171159f, -0.308018446f, -0.0542895719f, -0.17884554f, -0.348225653f, -1.00327587f, + 0.19604668f, 0.414156377f, -0.345914155f, -0.240799412f, -0.123780861f, 0.0896770582f, + -0.0316753387f, -0.0100131631f, -0.103553563f, 0.0392953604f, 0.267718852f, -0.238640875f, + 0.34261474f, -0.159957469f, -0.21596384f, -0.164301336f, 0.179002255f, -0.244498625f, + 0.518552899f, -0.302212507f, 0.133251935f, 0.278559178f, 0.357559025f, 0.391127437f, + 0.845105767f, 0.416706204f, -0.345570117f, -0.318146229f, 0.227020249f, -0.0358332694f, + -0.00586987473f, -0.0607267618f, -0.00866837054f, -0.379764616f, -0.419965178f, 0.00106362253f, + 0.691878915f, -0.218807712f, 0.0208936781f, 0.173440486f, -0.300604165f, -0.127732918f, + -0.0659972727f, -0.270701528f, 0.232182771f, 0.579617143f, 0.48161453f, -0.49274525f, + 0.168616027f, -0.00146871805f, 0.504892349f, 0.00273478776f, 0.175735623f, -0.0473229624f, + 0.183422416f, 0.115669459f, -0.336215794f, -0.0430468805f, -0.0864433199f, -0.15743652f, + -0.26993227f, 0.0947152078f, -0.0227583051f, 0.0183102246f, -0.288359761f, -0.12401703f, + 0.111069247f, -0.110739306f, -0.0410218388f, -0.25274381f, -0.0202127919f, -0.385935694f, + 0.370125145f, 0.231324434f, 0.167616129f, -0.274653733f, -0.0204962529f, 0.0763727799f, + 0.0200875923f, 0.160913348f, 0.183165699f, -0.53161782f, -0.31489563f, 0.259803802f, + 0.20113337f, -0.0943305045f, 0.182105809f, 0.0520472229f, 0.257046759f, -0.163054585f, + 0.237876207f, 0.432153523f, -0.140405208f, 0.683302283f, 0.142594263f, 0.58571887f, + -0.289439023f, -0.560717881f, -0.23339662f, -0.502087593f, 0.311300159f, -0.0986192822f, + 0.00357695855f, -0.0588946193f, 0.157236904f, 0.303535879f, 0.61058712f, 0.347811699f, + 0.0651569963f, -0.465369999f, 0.537363052f, -0.277234018f, 0.0619690269f, -0.213457838f, + -0.112374485f, -0.537795544f, 0.234381258f, 0.653604746f, 0.619360805f, -0.138211578f, + -0.12679489f, -0.175606161f, -0.20790939f, 0.41214782f, 0.158491641f, -0.111430943f, + 0.307760864f, 0.37233761f, 0.247063905f, 0.0522629619f, 0.528377056f, 0.0966698602f, + -0.281323671f, -0.355138004f, 0.64858979f, -0.252954006f, -0.186683819f, -0.376601934f, + 0.237004116f, -0.0381030813f, 0.207139239f, -0.134570405f, -0.517904997f, -0.308564216f, + 0.617320836f, -0.214224577f, 0.15062663f, 0.104215316f, -0.15971911f, -0.0251858234f, + -0.00963952765f, -0.170584917f, -0.307276964f, 0.0300887842f, -0.321906418f, -0.179011226f, + -0.0434426516f, -0.0465113595f, 0.195556641f, -0.0484886244f, -0.0411519706f, -0.029596135f, + -0.110946529f, -0.173654541f, 0.335389167f, -0.232126981f, 0.147764742f, -0.298581243f, + 0.122278661f, -0.211879119f, 0.323815525f, 0.0866600573f, 0.106657967f, -0.0856715143f, + -0.199217483f, 0.262240618f, 0.162046775f, -0.573071718f, 0.0697588772f, -0.116072863f, + 0.570300817f, -0.113942429f, -0.00842121243f, -0.276192218f, 0.236994267f, -0.106252551f, + 0.0965183824f, -0.00386106968f, -0.0278082192f, 0.435864568f, -0.424450785f, -0.279332876f, + 0.417287081f, -0.167044684f, 0.162960321f, 0.097316131f, -0.232201159f, -0.0304639246f, + -0.0169148948f, 0.0736370087f, 0.0584136993f, -0.680971742f, 0.117280275f, -0.49190405f, + 0.277138174f, -0.101141423f, 0.504014552f, 0.0267170109f, 0.233307421f, 0.0688534379f, + 0.0362077765f, -0.505575895f, 0.081911929f, 0.507611811f, 0.107323602f, -0.136304751f, + -0.320526868f, 0.0643033534f, 0.146599904f, -0.237069756f, 0.0012720339f, -0.0288555324f, + 0.245336547f, -0.248126239f, 0.454427391f, 0.257643938f, 0.190806478f, -0.0228414461f, + 0.514825523f, -0.270663142f, 0.4313187f, 0.212902129f, -0.43399626f, -0.104305051f, + 0.387887895f, -0.306578249f, 0.376728147f, 0.0662077069f, 0.219945669f, -0.0845528841f, + -0.153984949f, 0.129026055f, -0.296641409f, -0.62199086f, 0.216035232f, -0.131470174f, + -0.0387005582f, 0.0832562968f, 0.204449818f, -0.0658617243f, -0.0439203121f, -0.253425151f, + -0.344955534f, 0.0648814738f, 0.230917022f, 0.0381323397f, -0.194979742f, -0.0063324105f, + 0.148553088f, -0.131036937f, 0.331846207f, -0.160936028f, 0.18105489f, -0.142283484f, + 0.114826366f, -0.0430562235f, 0.419674575f, -0.0313794464f, 0.381467879f, -0.0972297564f, + -0.374319315f, 0.175825894f, -0.157610044f, -0.0551765636f, 0.203287661f, -0.0698809326f, + 0.236918658f, 0.175944418f, 0.365592539f, -0.175932199f, 0.0635617375f, 0.188517973f, + 0.437832773f, -0.336933762f, -0.0754605681f, -0.0201271139f, 0.321256161f, -0.0203632414f, + 0.377470315f, 0.220971793f, 0.255336553f, -0.112160996f, 0.551944077f, -0.0964897126f, + -0.279742599f, 0.0784681737f, 0.051167354f, -0.285445631f, 0.416058689f, -0.78205353f, + 0.124964744f, -0.500108361f, -0.573876143f, 0.0622367449f, 0.320155501f, -0.116039671f, + 0.035899464f, 0.0277688801f, -0.000809153542f, -0.2218775f, 0.652082801f, 0.930035353f, + 0.43815437f, -0.5008865f, 1.38336611f, 0.0526852831f, 0.0250353664f, -0.148763418f, + 0.230214983f, -0.405604899f, -0.112599663f, -0.356437892f, -0.228097722f, -0.692100048f, + 0.497451484f, -0.130678043f, -0.101596184f, -0.165828377f, 0.00548521429f, -0.0280849785f, + 0.100196078f, -0.127078503f, 0.438180864f, 0.457230657f, -0.0560073256f, -0.044703193f, + -0.0020352304f, 0.302042484f, 0.288549185f, 0.06829454f, 0.258589059f, 0.272551775f, + 0.0636158511f, -0.0284813698f, -0.11237976f, -0.257405549f, -0.320597708f, -0.38174057f, + 0.172765985f, -0.192915887f, -0.220388234f, -0.172601372f, 0.183805779f, -0.138337359f, + 0.19039464f, 0.370598942f, -0.191058144f, -0.018585816f, 0.489928335f, 0.0218410939f, + 0.0886444598f, -0.119595751f, 0.207078189f, -0.0853661299f, 0.254612535f, 0.00164549798f, + 0.0386651382f, -0.217443049f, -0.00408920646f, 0.0465721786f, -0.18976678f, 0.216396064f, + 0.0501514226f, -0.215909943f, 0.52984333f, -0.525925219f, 0.0781071931f, 0.157250911f, + -0.139597073f, 0.269092292f, 0.288446844f, 0.153175414f, 0.361954629f, -0.102880195f, + -0.139300689f, -0.071541816f, 0.975820839f, 0.207019195f, 0.371370703f, -0.31070438f, + -0.194333017f, 0.174113676f, -0.232878238f, -0.463331729f, -0.0474871397f, -0.197537243f, + -0.122307979f, -0.349582314f, 0.265481532f, -0.256766051f, -0.137520447f, -0.0571631491f, + 0.157311559f, 0.0791166276f, 0.518022597f, 0.0480866432f, 0.367400885f, 0.0211001337f, + 0.425725609f, 0.400145203f, -0.0921460688f, 0.171678901f, 0.254100382f, 0.441641688f, + 0.0763057396f, -0.0935568064f, -0.248465627f, -0.967218816f, 0.282862306f, -0.469038665f, + 0.207153663f, -0.355587453f, -0.145890653f, -0.0986361653f, 0.334949553f, -0.166654363f, + 0.403231859f, 0.254161984f, -0.0345563889f, 0.50133872f, 0.211890697f, 0.0853061825f, + -0.195352733f, 0.032662265f, 0.416701108f, -0.0420162082f, 0.428934067f, 0.141047224f, + -0.05249051f, 0.0197423249f, 0.0204004422f, -0.270726085f, 0.173498213f, -0.330816567f, + 0.0719912946f, -0.244619906f, 0.00626999885f, 0.198540732f, -0.258849382f, -0.291533619f, + -0.00963344797f, 0.0649380162f, 0.325994372f, 0.418272436f, 0.0126235634f, -0.537509859f, + -0.0341592468f, -0.00896826386f, -0.104769781f, 0.0290428102f, 0.219839364f, -0.126393348f, + -0.184354991f, -0.0739680827f, -0.0242563188f, 0.203575894f, -0.635400176f, -0.126711935f, + 0.0806170031f, 0.0986588299f, 0.179583639f, -0.0104112774f, 0.0495617613f, 0.482835263f, + 0.00256212056f, -0.200713813f, -0.580702186f, -0.252850711f, 1.02561522f, -0.0621974431f, + 0.829668522f, 0.327130497f, -0.0344709232f, -0.128673345f, 0.729137957f, 0.243879229f, + -0.14195928f, -0.0353268534f, -0.173741013f, -0.102253087f, -0.295482546f, -0.607205033f, + -0.0549504608f, -0.533897996f, 0.507747233f, -0.330410331f, 0.151903629f, 0.215051949f, + 0.0896668881f, -0.0622655153f, -0.0911758989f, 0.170324564f, 0.248491511f, -0.0517638177f, + 0.500895262f, 0.846594095f, 0.6860286f, 0.175998434f, 0.0213961899f, -0.183933347f, + 0.161302209f, -0.251403809f, 0.308368683f, 0.14680016f, 0.0307062715f, -0.453711003f, + -0.0287549198f, -0.0887027383f, 0.262331665f, -0.301454961f, 0.00580979884f, -0.20417656f, + 0.195075229f, -0.28542152f, 0.454269707f, 0.292104036f, 0.125653207f, -0.151984975f, + 0.0914702863f, -0.112546951f, 0.348899871f, 0.0926545709f, 0.0765267611f, 0.455139965f, + 0.203707024f, 0.0542512983f, -0.242122591f, -0.754379988f, 0.0632058233f, -0.0417160168f, + 0.311085939f, -0.0555022471f, 0.574783564f, -0.029963918f, -0.282187313f, -0.313570559f, +}; + inline constexpr int64_t kLtx2UpsSpatiotemporalUpsamplerShape[] = { 256, 32, 3, 3, 3, }; diff --git a/tests/vllm/models/test_ltx2_pipeline.cpp b/tests/vllm/models/test_ltx2_pipeline.cpp index 8ed22f9d4..3c0ce3d14 100644 --- a/tests/vllm/models/test_ltx2_pipeline.cpp +++ b/tests/vllm/models/test_ltx2_pipeline.cpp @@ -1299,7 +1299,7 @@ TEST_CASE("ltx2 every out-of-scope feature is refused BY NAME") { // // REACHABLE REFUSALS: a product path constructs the condition, so a caller can // trip this. ONE of the five is, not two. The engine reaches - // `ltx2_upsampler.cpp:465` through `Ltx2UpsampleVideoLatent`, which + // `ltx2_upsampler.cpp:497` through `Ltx2UpsampleVideoLatent`, which // `ltx2_video.cpp` calls when a phase asks for the spatial-upsample transform. // // `kBetaScheduler` USED TO BE LISTED HERE and is not reachable. Its call site @@ -1432,8 +1432,8 @@ TEST_CASE("ltx2 every out-of-scope feature is refused BY NAME") { // enumerator that appears in `src/` proves only that the compiler can see it. // So the check is on the ENTRY FUNCTION of each arm's chain: // -// kSpatiotemporalUpsampler `Ltx2LatentUpsample` (ltx2_upsampler.cpp:465) -// <- `Ltx2UpsampleVideoLatent` (:566) +// kSpatiotemporalUpsampler `Ltx2LatentUpsample` (ltx2_upsampler.cpp:497) +// <- `Ltx2UpsampleVideoLatent` (:681) // <- `ltx2_video.cpp`, the phase that upsamples // kBetaScheduler `Ltx2Schedule` (ltx2_pipeline.cpp:199) // <- NOTHING @@ -1573,7 +1573,7 @@ TEST_CASE("ltx2 docs/FEATURES.md never calls a REACHABLE refusal unrequestable") const std::string doc = buf.str(); REQUIRE(doc.size() > 1000); - // The ONE arm with a product call site: `ltx2_upsampler.cpp:465` constructs the + // The ONE arm with a product call site: `ltx2_upsampler.cpp:497` constructs the // SPATIOTEMPORAL upsampler condition, and `ltx2_video.cpp` reaches it through // `Ltx2UpsampleVideoLatent`. Named by the word the doc uses for it, since that // is what a reader sees. @@ -1950,6 +1950,39 @@ vllm::Ltx2LatentVolume TemporalUpsamplerLatent() { return latent; } + +// `dims=2` (model.py:47) — `conv = torch.nn.Conv2d if dims == 2 else Conv3d`, +// which reaches `initial_conv`, both ResBlock stacks and `final_conv`. The +// `upsampler` branch (:55-72) does NOT read `dims`, so the flags stay at the +// shipped default and this arm differs from `PixelShuffle` above in kernel RANK +// and in the GroupNorm reduction, nothing else. +vllm::Ltx2UpsamplerConfig Dims2UpsamplerConfig(const std::string& prefix) { + vllm::Ltx2UpsamplerConfig config; + config.in_channels = vllm_test::kLtx2UpsInChannels; + config.mid_channels = vllm_test::kLtx2UpsMidChannels; + config.num_blocks_per_stage = vllm_test::kLtx2UpsBlocksPerStage; + config.dims = 2; + config.spatial_upsample = true; + config.temporal_upsample = false; + config.spatial_scale = 2.0; + config.rational_resampler = false; + config.prefix = prefix; + return config; +} + +// 3 frames, so the folded axis differs from H (4) and W (6) and a fold that +// dropped or transposed it cannot pass by coincidence. +vllm::Ltx2LatentVolume Dims2UpsamplerLatent() { + vllm::Ltx2LatentVolume latent; + latent.batch = 1; + latent.channels = vllm_test::kLtx2UpsInChannels; + latent.frames = vllm_test::kLtx2UpsDims2Frames; + latent.height = vllm_test::kLtx2UpsHeight; + latent.width = vllm_test::kLtx2UpsWidth; + latent.data = Make("ltx2.ups.dims2.latent", latent.elems(), 1.0); + return latent; +} + } // namespace TEST_CASE("ltx2 the constants the headers call pinned are actually pinned") { @@ -1968,7 +2001,7 @@ TEST_CASE("ltx2 the constants the headers call pinned are actually pinned") { // the argument does not make the default unreachable — the default IS the // shipped width, which ltx2_upsampler.h has said all along. Only the // Rational1p5 arm reaches it, because `BlurDownsample` runs on the - // rational `den` (ltx2_upsampler.cpp:439) and 1.5 -> {3, 2} is the only + // rational `den` (ltx2_upsampler.cpp:599) and 1.5 -> {3, 2} is the only // one of the THREE ARMS with den != 1. (0.75 -> {3, 4} would reach it too // and no arm covers it, so this is arm coverage, not a property of the // supported-scale map.) @@ -2106,6 +2139,78 @@ TEST_CASE("ltx2 the latent temporal upsampler reproduces upstream") { CHECK(vllm::kLtx2UpsamplerTemporalFactor == vllm_test::kLtx2UpsTemporalFactor); } +TEST_CASE("ltx2 the latent upsampler reproduces upstream on the dims=2 arm") { + // model.py:47 picks Conv2d over Conv3d, and :85-100 folds the frame axis into + // the BATCH before running the whole stack and unfolding at :100. Two things + // follow and this case is aimed at exactly those two. + const vllm::Ltx2UpsamplerConfig config = Dims2UpsamplerConfig("ltx2.ups.Dims2."); + const ParamBag bag = BuildUpsamplerParams(config); + + // THE PARAMETER CONTRACT FIRST, and it is the stronger half of this gate. + // Four groups are 4-D here where every dims=3 arm builds them 5-D: + // `initial_conv` is [mid, in, 3, 3] = 1728 against [mid, in, 3, 3, 3] = 5184, + // and each ResBlock conv is [mid, mid, 3, 3] = 9216 against 27648. A port that + // reused the 3-D enumeration dies HERE, before a single value is compared. + CheckManifest(bag, vllm_test::kLtx2UpsDims2ParamNames, vllm_test::kLtx2UpsDims2ParamCounts, + std::size(vllm_test::kLtx2UpsDims2ParamNames)); + // `upsampler.0.weight` is the ONE conv that does not move: the upsampler + // branch never reads `dims`, so it is a Conv2d in the dims=3 spatial arm too. + CHECK(bag.weights.Get("ltx2.ups.Dims2.upsampler.0.weight").size() == + static_cast(4 * vllm_test::kLtx2UpsMidChannels * vllm_test::kLtx2UpsMidChannels * + 3 * 3)); + + const vllm::Ltx2LatentVolume latent = Dims2UpsamplerLatent(); + const vllm::Ltx2LatentVolume got = vllm::Ltx2LatentUpsample(config, bag.weights, latent); + CHECK(got.batch == vllm_test::kLtx2UpsDims2OutShape[0]); + CHECK(got.channels == vllm_test::kLtx2UpsDims2OutShape[1]); + // THE FRAME COUNT COMES BACK UNCHANGED. This is the property that makes the + // arm consumable rather than merely computed: both spatial call sites in + // ltx2_video.cpp require it (`vshape.frames` at :3525-3531 and + // `slot_positions.size()` at :3552-3563). + CHECK(got.frames == vllm_test::kLtx2UpsDims2OutShape[2]); + CHECK(got.frames == latent.frames); + CHECK(got.height == vllm_test::kLtx2UpsDims2OutShape[3]); + CHECK(got.width == vllm_test::kLtx2UpsDims2OutShape[4]); + + const double worst = + MaxAbsDiff(got.data, vllm_test::kLtx2UpsDims2Golden, std::size(vllm_test::kLtx2UpsDims2Golden)); + INFO("LatentUpsampler arm = Dims2 max|diff| = ", worst); + CHECK(worst <= kRoundOff); + + // THE GROUPNORM REDUCTION, isolated. `(b f)` makes every frame its own sample, + // so the statistics run over (channels_per_group, H, W) and NOT over the clip. + // With mid_channels = 32 and GroupNorm(32, ...) there is one channel per group, + // so the two reductions genuinely differ; running the dims=3 statistic here + // returns a correctly shaped, finite, plausible latent that is wrong in every + // element, which is why the value golden above — not a shape check — is what + // holds this down. Asserted as a PROPERTY too: a per-frame normaliser cannot + // make the arm's output depend on frames it was not shown, so upsampling the + // middle frame ALONE must reproduce that frame's slice of the full result. + vllm::Ltx2LatentVolume one = latent; + one.frames = 1; + const int64_t plane = latent.height * latent.width; + one.data.assign(static_cast(latent.channels) * static_cast(plane), 0.0f); + for (int64_t c = 0; c < latent.channels; ++c) { + for (int64_t i = 0; i < plane; ++i) { + one.data[static_cast(c * plane + i)] = + latent.data[static_cast((c * latent.frames + 1) * plane + i)]; + } + } + const vllm::Ltx2LatentVolume got_one = vllm::Ltx2LatentUpsample(config, bag.weights, one); + REQUIRE(got_one.frames == 1); + const int64_t out_plane = got.height * got.width; + double worst_slice = 0.0; + for (int64_t c = 0; c < got.channels; ++c) { + for (int64_t i = 0; i < out_plane; ++i) { + const double a = got.data[static_cast((c * got.frames + 1) * out_plane + i)]; + const double b = got_one.data[static_cast(c * out_plane + i)]; + worst_slice = std::max(worst_slice, std::fabs(a - b)); + } + } + INFO("dims=2 per-frame independence max|diff| = ", worst_slice); + CHECK(worst_slice <= kRoundOff); +} + TEST_CASE("ltx2 the upsampler refuses the arms it does not implement") { const vllm::Ltx2LatentVolume latent = ReducedUpsamplerLatent(); const ParamBag bag = BuildUpsamplerParams(ReducedUpsamplerConfig(false, 2.0, "ltx2.ups.x.")); @@ -2131,16 +2236,95 @@ TEST_CASE("ltx2 the upsampler refuses the arms it does not implement") { CHECK(Mentions(spatiotemporal_message, "temporal")); CHECK(Mentions(spatiotemporal_message, "PixelShuffleND(3)")); - vllm::Ltx2UpsamplerConfig two_d = ReducedUpsamplerConfig(false, 2.0, ""); - two_d.dims = 2; - CHECK(Mentions(refuse("dims=2", two_d), "dims")); - - // dims=2 is refused for the TEMPORAL arm too, and it has to be checked - // separately: the two arms take different branches, so a `dims` guard placed - // inside the spatial branch would let a 2-D temporal config through. + // `dims=2` is NO LONGER refused — it is ported, and "reproduces upstream on + // the dims=2 arm" above gates it against the executed module. What survives + // here is the one 2-D combination upstream itself cannot run: the temporal + // upsampler is a Conv3d (model.py:68-71) and the dims=2 forward (:85-100) + // folds the frame axis into the batch before reaching it. + // + // NOT "torch raises on the rank", which is what this paragraph used to say and + // what `ltx2_upsampler.cpp` and `ltx2_upsampler.h` were corrected to stop + // saying. `Conv3d` accepts a 4-D input as an UNBATCHED 5-D one, so the folded + // `(b*f, c, h, w)` is read as `(C, D, H, W)` and the CHANNEL count is what + // disagrees; at `frames == mid_channels` that Conv3d would pass and + // `PixelShuffleND(1)`'s einops pattern fails instead. Two mechanisms behind + // one shape coincidence, which is why the port refuses the configuration by + // NAME rather than by either of them. `scripts/gen-ltx2-pipeline-goldens.py` + // runs both contradictions through the real module and asserts the message. + // + // Checked through the TEMPORAL config specifically, because the two arms take + // different branches and a guard placed inside one of them would let the other + // through. vllm::Ltx2UpsamplerConfig temporal_two_d = TemporalUpsamplerConfig(""); temporal_two_d.dims = 2; - CHECK(Mentions(refuse("temporal dims=2", temporal_two_d), "dims")); + const std::string two_d_message = refuse("temporal dims=2", temporal_two_d); + CHECK(Mentions(two_d_message, "dims=2 with temporal_upsample=true")); + // NOT the retired "is not ported" refusal, which is the whole point of the + // change: a caller who supplies an ordinary 2-D spatial checkpoint must now be + // served, and only this contradiction refused. + CHECK(two_d_message.find("is not ported") == std::string::npos); + + // THE EIGHTH CELL, where the two dims=2 contradictions are BOTH set and only + // one of them is upstream's. `__init__` is an if/elif chain: with + // `spatial_upsample` false, `elif temporal_upsample` (model.py:68) builds the + // Conv3d and never reads `rational_resampler`, so this config owns no + // SpatialRationalResampler at all. Refusing it while naming a module upstream + // did not construct sends the caller to the wrong flag. + // + // This asserts the MESSAGE, not the polarity, because the polarity cannot see + // the defect: the config is refused whichever `Require` fires first. Swapping + // the two back in `ltx2_upsampler.cpp` leaves every other case green and reds + // exactly these two lines. + vllm::Ltx2UpsamplerConfig temporal_and_rational = TemporalUpsamplerConfig(""); + temporal_and_rational.dims = 2; + temporal_and_rational.rational_resampler = true; + const std::string both_message = refuse("temporal+rational dims=2", temporal_and_rational); + CHECK(Mentions(both_message, "dims=2 with temporal_upsample=true")); + CHECK(both_message.find("rational_resampler=true") == std::string::npos); + + // THE SIBLING OF THE GUARD ABOVE, and the one this row first shipped without. + // `dims=2` with the RATIONAL resampler is upstream's OTHER 2-D contradiction: + // `SpatialRationalResampler.forward` opens with `b, _, f, _, _ = x.shape` + // (model/upsampler/spatial_rational_resampler.py:41), and the dims=2 forward + // reaches it at model.py:94 having ALREADY folded the frame axis into the + // batch at :86. Upstream raises + // `ValueError: not enough values to unpack (expected 5, got 4)` — the same + // mechanism as the temporal case, one line further in. + // + // Driven with its OWN correctly enumerated weights, because that is the whole + // point: every operator inside the rational branch is already per-frame, so a + // one-frame volume satisfies all of them and this config computes a complete, + // finite, plausible latent that upstream cannot produce. No shape check + // anywhere can see it. Only a refusal can. + vllm::Ltx2UpsamplerConfig rational_two_d = ReducedUpsamplerConfig(true, 2.0, "ltx2.ups.r2."); + rational_two_d.dims = 2; + const ParamBag rational_bag = BuildUpsamplerParams(rational_two_d); + const std::string rational_two_d_message = RefusalMessage( + [&] { (void)vllm::Ltx2LatentUpsample(rational_two_d, rational_bag.weights, latent); }); + INFO("arm = rational dims=2 refusal = ", rational_two_d_message); + CHECK(Mentions(rational_two_d_message, "dims=2 with rational_resampler=true")); + CHECK(Mentions(rational_two_d_message, "spatial_rational_resampler.py:41")); + + // THE CONV2D BIAS CONTRACT, which the 3-D helper has had since #644 + // (`Conv3dPad1`, ltx2_upsampler.cpp:82) and the 2-D one did not. Before this + // repair `Conv2dPad1PerFrame` checked only the weight, so a dims=2 checkpoint + // carrying a correct kernel and a bias one element short read past the end of + // a std::vector — UB and silent garbage — where the identical dims=3 defect + // gets a named refusal. Nothing upstream of the helper validates it: + // `Ltx2LoadVaeWeights` (ltx2_video.cpp:1490) loads by NAME and checks no shape. + // + // `initial_conv` is the first of the four groups the dims=2 arm routes through + // this helper, so it is the earliest place the read would happen. + vllm::Ltx2UpsamplerConfig short_bias_cfg = ReducedUpsamplerConfig(false, 2.0, "ltx2.ups.b2."); + short_bias_cfg.dims = 2; + ParamBag short_bias = BuildUpsamplerParams(short_bias_cfg); + std::vector& initial_bias = short_bias.weights.tensors["ltx2.ups.b2.initial_conv.bias"]; + REQUIRE(initial_bias.size() == static_cast(vllm_test::kLtx2UpsMidChannels)); + initial_bias.pop_back(); + const std::string short_bias_message = RefusalMessage( + [&] { (void)vllm::Ltx2LatentUpsample(short_bias_cfg, short_bias.weights, latent); }); + INFO("short conv2d bias refusal = ", short_bias_message); + CHECK(Mentions(short_bias_message, "conv2d bias has the wrong element count")); // Upstream's own ValueError when neither flag is set (model.py:73-74). vllm::Ltx2UpsamplerConfig neither = ReducedUpsamplerConfig(false, 2.0, ""); diff --git a/tests/vllm/multimodal/test_ltx2_video.cpp b/tests/vllm/multimodal/test_ltx2_video.cpp index 4736ee63a..1f8ce6b07 100644 --- a/tests/vllm/multimodal/test_ltx2_video.cpp +++ b/tests/vllm/multimodal/test_ltx2_video.cpp @@ -740,7 +740,7 @@ TEST_CASE("ltx2 video: the second phase upsamples, and refuses when it cannot") // checkpoint as well as a temporal-only one, so a genuine SPATIOTEMPORAL // checkpoint was told it is the temporal x2 upsampler and pointed at the spatial // one. Wrong on both counts: it is neither, it is the third arm, and the ledger - // refusal that names it (`ltx2_upsampler.cpp:465`) sat behind a guard that could + // refusal that names it (`ltx2_upsampler.cpp:497`) sat behind a guard that could // not be reached from a request. // // The defect is an IMPLICATION between two guards over one variable, which no @@ -779,6 +779,40 @@ TEST_CASE("ltx2 video: the second phase upsamples, and refuses when it cannot") CHECK(msg.find("the upsampled latent is") == std::string::npos); } } + // THE REACHABILITY CASE FOR THE dims=2 ARM, and the reason it asserts a + // COMPLETED RENDER rather than a changed message. `dims` is read off the + // checkpoint's own config (`Ltx2ParseUpsamplerConfig`, mirroring + // model_configurator.py:17), so a 2-D upsampler is an ordinary input that a + // caller supplies through the `upsampler_path` load extra — the same entry + // point the three subcases above use, on its default configuration. + // + // It is CONSUMABLE and not merely computed, which is what separates this arm + // from the spatiotemporal one that stays refused. The fold at model.py:86/:100 + // returns the frame count unchanged and PixelShuffleND(2) doubles H and W, so + // the result is exactly the `[c, f, 2h, 2w]` the phase requires at + // ltx2_video.cpp:3525-3531. A port that got the fold wrong would still satisfy + // that check, which is why the VALUE gate lives in test_ltx2_pipeline and this + // case is about reach. + // + // The fixture writes 4-D kernels here without being told to: it enumerates + // through `EnumerateLtx2UpsamplerTensors(cfg)`, so the rank follows `dims` on + // both sides and a 5-D enumeration would fail to load rather than mis-render. + SUBCASE("a dims=2 upsampler checkpoint RENDERS, at the full requested size") { + vllm::Ltx2UpsamplerConfig dims2 = + ltx2_fixture::ReducedUpsamplerConfig(ltx2_fixture::ReducedDitParams().in_channels); + dims2.dims = 2; + const std::string path = ws.root + "/dims2_upsampler.safetensors"; + ltx2_fixture::WriteReducedUpsampler(dims2, path); + + vllm::multimodal::VideoModelParams mp = FixtureParams(ws.paths); + mp.extras["upsampler_path"] = path; + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(mp); + const vllm::multimodal::VideoResult result = + engine->Generate(FixtureGen(ws.root + "/dims2_ups")); + CHECK(result.width == 64); + CHECK(result.height == 64); + } SUBCASE("with one, the render lands at the FULL requested size") { vllm::multimodal::VideoModelParams mp = FixtureParams(ws.paths); mp.extras["upsampler_path"] = ws.paths.upsampler; @@ -2779,6 +2813,48 @@ TEST_CASE("ltx2 video: DFR's temporal rounds DRIVE the temporal x2 latent upsamp CHECK(trace.round_merged_slot_tiles[2] == 1); } + SUBCASE("a dims=2 TEMPORAL upsampler checkpoint is refused AT LOAD, not minutes in") { + // THE PLACEMENT IS THE POINT. `dims=2` with `temporal_upsample` is a + // contradiction upstream cannot run, and `Ltx2LatentUpsample` refuses it — + // but that refusal fires inside the rounds loop (ltx2_video.cpp:5058), which + // is reached only after two full denoise stages. The load block that owns + // `temporal_upsampler_path` exists precisely so a caller who supplied the + // wrong file learns at load rather than several minutes in, and it says so + // in its own comment. It checked `temporal_upsample` and `spatial_upsample` + // there and did not check `dims`, so this one checkpoint cleared both + // siblings and failed downstream anyway. + // + // Asserted at `LoadVideoEngine`, not at `Generate`: a case that only checks + // the message would stay green with the guard back in the rounds loop, which + // is the exact defect. + vllm::Ltx2UpsamplerConfig temporal_two_d = + ltx2_fixture::ReducedUpsamplerConfig(ltx2_fixture::ReducedDitParams().in_channels); + temporal_two_d.spatial_upsample = false; + temporal_two_d.temporal_upsample = true; + temporal_two_d.dims = 2; + const std::string two_d_path = ws.root + "/dfr_temporal_dims2.safetensors"; + ltx2_fixture::WriteReducedUpsampler(temporal_two_d, two_d_path); + + vllm::multimodal::VideoModelParams mp = FixtureParams(ws.paths); + mp.extras[vllm::multimodal::kLtx2PipelineKindExtra] = "dfr"; + mp.extras[vllm::multimodal::kLtx2CheckpointClassExtra] = FixtureCheckpointClass("dfr"); + SupplyRequiredAdapter(&mp, "dfr", ws.root + "/dfr_dims2_lora.safetensors"); + mp.extras["upsampler_path"] = ws.paths.upsampler; + mp.extras[vllm::multimodal::kLtx2TemporalUpsamplerPathExtra] = two_d_path; + try { + (void)vllm::multimodal::LoadVideoEngine(mp); + FAIL_CHECK("a dims=2 temporal upsampler must be refused at LOAD, beside its two siblings"); + } catch (const std::exception& e) { + const std::string msg = e.what(); + INFO(msg); + CHECK(msg.find("dims=2") != std::string::npos); + CHECK(msg.find(std::string(vllm::multimodal::kLtx2TemporalUpsamplerPathExtra)) != + std::string::npos); + // The upstream site that makes it a contradiction rather than a taste. + CHECK(msg.find("model.py:68-71") != std::string::npos); + } + } + SUBCASE("rounds without a temporal upsampler are refused, not silently skipped") { // Upstream's second refusal, raised at the top of `__call__` beside the // malformed-value one (dfr_pipeline.py:286-287). Refused rather than run at