Repository navigation
Improve the performance of dissolve transitions - #1355
Merged
Merged
Conversation
Previously, a dissolve between two opaque 4K60 H.264 clips at 100 Mbit/s ran at 8.3 fps on a 16-core desktop, when played on one thread as the Kdenlive monitor does. Most of this time went to color conversions that changed nothing. Kdenlive sets fix_background_alpha on every dissolve, and with this property, the luma transition requested both frames as rgba and converted the result back to yuv422. It did this only to make the padding that filter_resize adds transparent, so for opaque video, the four conversions had no effect. The round trip also slowed down the work after the transition, because it left the result with an alpha mask that was fully opaque. Kdenlive composites its tracks with a qtblend transition, which saw this mask and so composited the frame in rgba too. The transition now requests yuv422 and marks both frames with fix_background_alpha. The padding stays transparent, because when filter_resize pads a marked frame that has no alpha mask, it first adds an opaque one. The alpha of the result also stays correct: when one frame is translucent, the transition gives the other frame an opaque mask if it has none, as the round trip did, so that alpha_over can still compute it. With this change, the same dissolve runs at 73 fps. Its luma is also closer to the source, because the rgba round trip clipped colors that rgb cannot represent. In the first frame of the dissolve, luma is now identical to the decoded source, and before, the two differed by a PSNR of 38 dB. Apart from this, the output does not change. These cases give the same result as before: - A 16:9 clip in a 4:3 profile gets transparent padding. - With alpha_over, a dissolve into a clip with transparency shows the track below through it, with the same colors in every frame, also with a luma map and with invert. - With fix_background_alpha off, the output is bit-identical.
Previously, the luma transition split the dissolve of translucent frames into slices, but it mixed opaque frames line by line on the calling thread. For 4K frames, this loop took a large part of the time of each frame. Opaque frames now use slices too, as translucent frames do. The output does not change, and on a 16-core desktop, a dissolve between two 4K60 H.264 clips at 100 Mbit/s goes from 73 to 77 fps.
Previously, during forward playback, the producer made a deep copy of each decoded frame in last_good_frame. It needs the last good frame to conceal decoding errors: when a frame fails to decode, the producer shows the last good frame in its place. But the image cache already held a copy of the same frame, so for a 4K clip, each frame cost a second 16 MB copy that was useful only after an error. When the image cache is on, the producer now keeps only the position of the last good frame, and gets the frame from the cache when decoding fails. The cache can lose that frame in two cases, so the producer copies the frame before they happen: - when frames before it go into the cache, which happens during backward playback or after a seek back; - when the cache is reset. Without the image cache, the producer still copies every frame. The producer conceals errors with the same frames as before. This was checked with clips with corrupted parts, played forward, played backward, after a seek back, after a cache reset, and with a small image cache or none. With this change, forward playback is faster. On a 16-core desktop, a dissolve between two 4K60 H.264 clips at 100 Mbit/s goes from 61 to 69 fps, and in a Kdenlive project with such clips, from 73 to 86 fps.
filter_resize pads with the resize_alpha frame property, and until now a missing property and an explicit 0 meant the same thing because every reader and writer used get_int and set_int. To pad an opaque yuv422 image with transparency without converting it to rgba and back, filter_resize needs to tell an explicit request from the default. Add a getter that reports whether resize_alpha is set and a setter that can clear it, so that code that saves and restores the property keeps a missing value missing. copy_consumer_image_hints() now uses them: it previously wrote 0 onto every frame it copied hints to, which would have made every padded frame look like an explicit request.
ddennedy
changed the title
The luma transition marked both frames with a fix_background_alpha frame property so that filter_resize would give an opaque yuv422 image a mask and pad it with transparency. That coupled filter_resize to one transition through a property that means nothing to it. filter_resize now creates the mask when resize_alpha is explicitly set to a value other than 255, and pads with that value, which is what padding an rgba image did. The luma transition requests it by setting resize_alpha to 0 when fix_background_alpha is on and the frame has no request yet. The composite transition saves and restores resize_alpha around its b frame with mlt_frame_get/set_resize_alpha, so that a frame without the property does not leave with an explicit 0. Frames that never set resize_alpha are padded exactly as before, without a mask.
dissolve_yuv422() and luma_composite_yuv422() each requested transparent padding, fetched both images as yuv422, checked both alpha masks, and added an opaque mask to one side when the other was translucent. The order matters: padding must be requested before the images are fetched, and the opaque mask must be added only when the other side is translucent, or the result keeps a fully opaque mask and qtblend goes back to rgba. Both now call prepare_yuv422_pair(). The dissolve also uses one slice function instead of two. An alpha_over flag picks composite_line_yuv_float() or composite_line_yuv() per line, and each image advances by its own width. This changes the output when the two frames differ in size, which contradicts "The output does not change" in "luma: dissolve opaque yuv422 frames in slices". Before, the opaque dissolve stepped the source by the width clamped to the destination and ran over the full destination height, and the alpha-over dissolve used that clamped width as the stride of both images. A wider source was then read at the wrong rows, and a shorter one was read past its end. The luma wipe had the same source stride problem. Now both blend only the overlapping area, each image with its own stride, and the alpha masks skip the rest of each row. When the sizes are equal, the output does not change.
The last good frame for error concealment was kept in three fields of the producer and updated in several places in producer_get_image(): before a cache put of an earlier frame and before a cache reset, the cached frame had to be copied out, or a later decoding failure would find nothing to show. Forgetting one of those copies would silently break concealment. Move the state and its transitions into a small avformat_conceal type with record, detach, fetch, and close functions, which are now the only code that changes it. The picture is either an owned frame, once it has been copied out of the image cache, or otherwise the image cache key. The behavior does not change.
Covers filter_resize padding with a missing, an explicit 0, and an explicit 255 resize_alpha; that copying consumer hints and the composite transition keep a missing resize_alpha missing; the luma dissolve and wipe with equal and mismatched frame sizes; and the avformat error concealment state for forward playback, a cache eviction after detach, and backward playback.
ddennedy
force-pushed
the
dissolve-performance
branch
from
October 28, 2026 00:34
0a39760 to
46ab3b2
Compare
Contributor
|
Awesome! I wonder what other low hanging fruit performance improvements are just waiting to be discovered :) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Supersedes #1343.
This adopts the useful parts of #1343 by @iniw and revises them in response to the review there. Patches 1, 2, and 7 of #1343 are cherry-picked with their original authorship. The follow-up commits address the review comments. Patches 3 (custom yuv420p to yuyv422 conversion in avformat), 4 and 5 (copying probed stream info), and 6 (threaded
memcpyinmlt_frame_clone) are not included.Commits
From #1343 (author @iniw):
Follow-ups:
mlt_frame_get/set_resize_alpha. A missingresize_alphaand an explicit 0 now mean different things. The pair is the only way to read, write, or clear the property, so that code that saves and restores it keeps a missing value missing.copy_consumer_image_hints()uses it; before, it wrote 0 onto every frame. New exports are listed underMLT_7.44.0inmlt.vers.resize_alpha. Replaces thefix_background_alphaframe property that coupledfilter_resizeto the luma transition.filter_resizecreates an opaque mask for a yuv422 image only whenresize_alphais explicitly set to something other than 255. Frames that never set it are padded as before.conceal.c. One owner for the last good frame (an owned copy once detached, otherwise the image cache key). No behavior change.test_dissolve. Coversresize_alphapadding and preservation through hints and composite, dissolve and wipe with equal and mismatched sizes, and the concealment state for forward playback, eviction after detach, and backward playback.Performance
Measured with the test from #1343 unchanged:
gen-clips.sh(two 4K60 H.264 clips at about 100 Mbit/s),dissolve.mlt(a Kdenlive-style timeline with a 60-frame luma dissolve withfix_background_alpha, composited withqtblend), andbench.c(one thread pulls yuv422 and s16 audio per frame, like the Kdenlive monitor). Instead of the PR'smeasure.sh, which rebuilds MLT, master and the branch were run from two run directories that differ only inlibmlt,libmltcore, andlibmltavformat(pluslibmltqt6forqtblend). Each was run 3 times, alternating.7b6ba22d)0a39760c)That is about 5.1× on first play and 9.2× on replay. #1343 reported 4.8× and 9.0× for its patches 1 and 2 on a faster machine, so the ratios agree. Its further gains came from the patches left out here.
On a real Shotcut project (two 8-bit 4:2:0 HEVC 4K60 GoPro clips with a luma dissolve, rendered offline with an avformat null-muxer consumer, 148 frames), the steady rate went from 8.5 to 34.3 fps (about 4×).
Test system
-O2 -g -DNDEBUG) for both master and the branch. Improve the performance of dissolve transitions #1343's script used Release (-O3).Not measured
test_dissolvechecks pixel values for small synthetic frames.