Skip to content

[Copilot] Add FlatBuffer runtime interpreter for Lottie animations - #583

Open
jonwis wants to merge 42 commits into
CommunityToolkit:mainfrom
jonwis:jonwis-build-windows-interpreter-sample-tool
Open

jonwis wants to merge 42 commits into
CommunityToolkit:mainfrom
jonwis:jonwis-build-windows-interpreter-sample-tool

Conversation

@jonwis

@jonwis jonwis commented Sep 1, 2026 •

Copy link
Copy Markdown

Summary

Adds a data-driven alternative to per-animation generated native code:

  1. LottieGen -Language flatbuffer serializes the translated WinCompData graph into .lcomp files.
  2. A native C++/WinRT reader realizes that graph using the caller's compositor.
  3. Applications compile the reader into their existing binaries and embed or otherwise supply the data.

There is no separate runtime DLL, COM activation layer, or exported loader ABI. The earlier
registration-free COM design has been removed. Each consuming binary owns its reader implementation.
The existing C#, C++/CX, and C++/WinRT generation paths remain available.

Format and integration

The format stores the translated Composition graph, not the original Lottie JSON. Translation stays
in the generator; the application needs only the native reader and compiled data.

  • Indexed graph references and interned strings preserve object sharing.
  • The schema, generated C#/C++ bindings, serializer, and managed round-trip reader are included.
  • The native reader verifies the FlatBuffer, schema version, required API contract, and graph references.
  • Contract-specific fallback translations produce separate .lcomp variants; the host selects one.
  • Direct2D supplies native path geometry without a Win2D dependency.

The same serialized format serves two separately compiled C++/WinRT projections:

Composition API Reader definition
Windows.UI.Composition LOTTIE_RUNTIME_WINUI2
Microsoft.UI.Composition LOTTIE_RUNTIME_WINUI3

Consumers build as C++20 with exactly one definition per reader compilation. The projection-specific
namespaces keep the two native implementations distinct; this is source integration, not an ABI bridge.

LottieGen.nupkg includes the reader source/header, generated schema binding, pinned FlatBuffers
headers, licenses, and integration instructions. It does not distribute a reader DLL or COM manifest.

Native loading APIs

  • LoadComposition retains the Visual-only loading contract.
  • LoadCompositionWithMetadata returns the root visual, duration, dimensions, frame rate, markers,
    live theme properties, and realized image-surface descriptors.
  • Theme properties are the actual property set referenced by the graph's expressions. Updating them
    changes the existing visual tree.
  • Hosts can observe image completion and reuse loaded image surfaces across new trees from the same
    buffer and XAML host compositor.
  • Metadata validation rejects malformed values and indices. Marker order, duplicate names, and finite
    out-of-range progress values are preserved; projection and playback policy belong to the host.

The host owns XAML integration, animated-visual adapters, progress animation, image-completion policy,
and object lifetime. Image-bearing animations use the XAML host's compositor.

Anonymized native-client size example

Representative migrations from generated native animation code to embedded .lcomp resources:

DLL Animations migrated Final DLL size change
foo.dll 33 -42.26%
bar.dll 9 -25.56%
baz.dll 2 -9.33%
qux.dll 2 -1.45%
quux.dll 1 +1.54%
Combined 47 -23.50%

Counts are distinct animations, not generated API-contract variants. Negative percentages mean a
smaller DLL. The combined percentage is weighted by the original DLL sizes, not averaged across rows.

These are matched standalone Release/x64 measurements of complete DLLs, including each DLL's reader,
host adapters, and embedded payloads. They exclude PDBs and unchanged dependencies, and do not measure
whole-package size or working set. One pair used identical analysis-disabled settings on both sides.
The two-animation result in baz.dll includes removing fallback variants outside that host's supported
OS range.

The single-animation DLL grows because reader/adapter overhead outweighs the generated code removed.
The reader cost is amortized within each binary, not shared through a new runtime DLL.

Validation and remaining work

  • Managed tests cover graph round trips, corpus coverage, and malformed-buffer rejection.
  • The focused native regression executable builds with /W4 /WX and passes on WinUI 2/x64. It covers
    metadata validation, the legacy no-metadata API, live theme/cache identity, owned-property
    initialization ordering, image reuse, and invalid image descriptors.
  • The standalone inspection-tool source provides load, --dump, and --show paths. Its build still
    needs removal of a stale loader cleanup reference left by the deleted COM implementation; the
    native regression result above does not claim that the inspection tool or the full matrix is green.

(via Copilot)

Copilot AI and others added 26 commits August 31, 2026 17:49
Co-authored-by: jonwis <18537118+jonwis@users.noreply.github.com>
…/writer

Co-authored-by: jonwis <18537118+jonwis@users.noreply.github.com>
…pData graph

Co-authored-by: jonwis <18537118+jonwis@users.noreply.github.com>
Co-authored-by: jonwis <18537118+jonwis@users.noreply.github.com>
…++ verifier

Metadata is serialized last but its strings must be present in the string
table, which is written first. Intern them up front, then freeze the table
so a missed interning site throws instead of handing out an out of range
index. Also store marker durations as a proportion of the animation and
carry the composition width and height through to the metadata.

Co-authored-by: jonwis <18537118+jonwis@users.noreply.github.com>
The managed counterpart of the native interpreter. It exists so that the
wire format can be round tripped and compared with the original graph,
which is a much stronger check than testing the serializer alone. Every
index is bounds checked and the buffer is verified before it is read,
because buffers are untrusted input.

Co-authored-by: jonwis <18537118+jonwis@users.noreply.github.com>
…d graphs

Co-authored-by: jonwis <18537118+jonwis@users.noreply.github.com>
Co-authored-by: jonwis <18537118+jonwis@users.noreply.github.com>
…y kind

Co-authored-by: jonwis <18537118+jonwis@users.noreply.github.com>
Co-authored-by: jonwis <18537118+jonwis@users.noreply.github.com>
Co-authored-by: jonwis <18537118+jonwis@users.noreply.github.com>
Co-authored-by: jonwis <18537118+jonwis@users.noreply.github.com>
- Move #include <unknwn.h> before LottieRuntime.h (which pulls in
  C++/WinRT headers), satisfying the classic-COM interop requirement.
- Use winrt::implements<> with the projected interface listed first
  for GeometrySource and Effect, so winrt::make<> returns a com_ptr
  convertible to CompositionPath / IGraphicsEffect directly.
- Fix StrokeMiterLimit cast (was uint32_t, should be float).
- Default-construct WinRT projection type caches with nullptr fill
  value; these types have no default constructor.
- Link dxguid.lib for D2D1 effect CLSIDs (Composite, GaussianBlur).
A console tool that links LottieRuntime.vcxproj, loads a serialized
Lottie composition (.lcomp) given as its command-line argument, and
prints the composition object hierarchy of the resulting visual tree:
visuals, their shapes (recursively through container shapes), fill and
stroke brushes, and gradient stops. Each line shows the runtime class
name and, where set, the Comment property, so the printed tree can be
matched back to the source Lottie layers.

Builds via project reference to LottieRuntime.vcxproj and copies
LottieRuntime.dll next to the exe as a post-build step, since C++
project references do not do this automatically.
Defines ILottieCompositionLoader (IID 3874B71B-05E6-42A9-B838-1AE7F0BA4E1C),
a classic COM interface accepting (UINT32 length, BYTE const* buffer, REFIID,
void**) that wraps LoadComposition for callers without a C++/WinRT toolchain.
Implements it as LottieCompositionLoader (CLSID 9F7B7089-F512-4BBC-81FE-4FF12B4DA159)
via winrt::implements<>, plus DllGetClassObject/DllCanUnloadNow exports and a
LottieRuntime.def to keep those names undecorated.

The class is never registered with the system: LottieRuntime.manifest is a
private-assembly manifest declaring the comClass, copied next to a consuming
exe and named directly by an ACTCTX, so CoCreateInstance resolves the CLSID
without any HKEY_CLASSES_ROOT entry.

Adds LottieRuntimeComTest, a console harness that builds an activation
context from that manifest, activates it, CoCreateInstances the loader, and
calls LoadComposition on a file argument.

Known issue: CoCreateInstance currently returns E_ACCESSDENIED (0x80070005)
even though CreateActCtx/ActivateActCtx succeed and the DLL exports
DllGetClassObject/DllCanUnloadNow correctly - the private-assembly activation
path needs further debugging before this is fully working end-to-end.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 58fac82f-0489-4c57-a18d-2b7ba273faa6
Create the desktop DispatcherQueue lazily before activating the loader's Compositor. This keeps CoCreateInstance side-effect-free and gives Windows.UI.Composition the thread infrastructure it requires.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 58fac82f-0489-4c57-a18d-2b7ba273faa6
Use C++/WinRT's module lock for IClassFactory::LockServer and DllCanUnloadNow so COM cannot unload the DLL while its objects remain alive.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 58fac82f-0489-4c57-a18d-2b7ba273faa6
Build the COM test, record it under Time Travel Debugging, and collect optional SideBySide and COM event logs into one directory.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 58fac82f-0489-4c57-a18d-2b7ba273faa6
Remove the one-off investigation helper now that the activation failure is diagnosed.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 58fac82f-0489-4c57-a18d-2b7ba273faa6
Handle omitted property-set state, expose the required effect-source interface, cache and cycle-check canvas geometries and strings, preserve sprite shadows, and load URI and stream image surfaces.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 58fac82f-0489-4c57-a18d-2b7ba273faa6
Use the FlatBuffers headers and flatc from pinned upstream default-branch commit 5761d6e for C++. Keep C# generation on the 25.2.10 runtime available through Microsoft's configured NuGet feed.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 58fac82f-0489-4c57-a18d-2b7ba273faa6
Create custom controllers explicitly, pass them to StartAnimation, and initialize root property sets before descending into children so controller expressions can bind to root progress.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 58fac82f-0489-4c57-a18d-2b7ba273faa6
Remove exported C++ symbols, require callers to own the DispatcherQueue, support architecture-neutral registration-free activation, load embedded image streams synchronously, and make the native project portable from its packaged source layout.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 58fac82f-0489-4c57-a18d-2b7ba273faa6
Replace separate validation and dump executables with LottieRuntime.exe supporting validate, --dump, and --show modes. Document JSON conversion and package the complete native runtime source set, COM project, generated binding, and FlatBuffers headers in LottieGen.nupkg.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 58fac82f-0489-4c57-a18d-2b7ba273faa6
Disable incremental linking and enable /O1, function-level linking, string pooling, LTCG, reference elimination, and COMDAT folding for Release builds.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 58fac82f-0489-4c57-a18d-2b7ba273faa6
Keep the runtime loadable on Windows 10 1809 while rejecting compositions whose declared UniversalApiContract version is unavailable before graph hydration.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 58fac82f-0489-4c57-a18d-2b7ba273faa6
@jonwis

jonwis commented Sep 2, 2026

Copy link
Copy Markdown
Author

@dotnet-policy-service agree company="Microsoft"

Copilot AI and others added 2 commits September 2, 2026 16:17
Co-authored-by: jonwis <18537118+jonwis@users.noreply.github.com>
Co-authored-by: jonwis <18537118+jonwis@users.noreply.github.com>
Split runtime schema dependencies from tool-side serialization, avoid copying byte-array buffers, and process keyframes in one pass.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 408cb9b9-d2b1-4a7a-bc3e-f6ab25b8793f
…preter

Add a managed FlatBuffer interpreter alongside the native one
@aborziak-ms

Copy link
Copy Markdown
Collaborator

@jonwis I left a few inline comments, but wanted to share some higher-level thoughts as well:

  • I'd suggest scoping this PR to the .lcomp format + LottieGen output + a small native C++ runtime. For native, a simple .h/.cpp drop-in that takes the app's Compositor and works with AnimatedVisualPlayer should be enough. I don't think we necessarily need the COM DLL here, since C++ apps can just compile the drop-in. The C# interpreter would mostly duplicate Instantiator, which the managed side (Lottie-Windows package) can reuse.
  • I can pick up the managed side separately and integrate lcomp into Lottie-Windows UWP and WinUI packages. The .lcomp graph is basically what Loader already builds from JSON, so we can load it through the existing LottieVisualSource path and get AnimatedVisualPlayer, images, WinUI 3 etc. for free.
  • Since the format would be shared by both, it'd be good to settle a few things now: include all UAP versions (like codegen does), have a versioning for schema changes, and keep the generated types internal.
  • For FlatBuffers, maybe just check in include/flatbuffers + LICENSE inside the native runtime folder instead of a full submodule? It's the only thing that needs the headers, it keeps the drop-in self-contained.

Comment thread source/LottieGen/LottieJsonFileProcessor.cs Outdated
Comment thread dlls/LottieRuntime/LottieRuntimeCom.cpp Outdated
Comment thread dlls/LottieRuntime/LottieRuntime.h Outdated
Comment thread source/LottieRuntime/README.md Outdated
@@ -0,0 +1,108 @@
# LottieRuntime (managed)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the managed side should just live in Lottie-Windows. Then there's one C# library that handles both JSON and .lcomp, and .lcomp can reuse the existing pipeline (CompositionDeserializer → Instantiator → AnimatedVisualPlayer) instead of a separate C# interpreter. I'd keep LottieRuntime as the small native runtime for C++ only. Otherwise, I'm worried we end up with too many pieces doing the same thing but a little differently.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK. I'll extend the existing C# library with this support instead and remove this one.

Comment thread source/LottieRuntime/LottieRuntime.cpp
@jonwis

jonwis commented Oct 2, 2026

Copy link
Copy Markdown
Author
  • I don't think we necessarily need the COM DLL here, since C++ apps can just compile the drop-in. The C# interpreter would mostly duplicate Instantiator, which the managed side (Lottie-Windows package) can reuse.

OK, will narrow to just the producer and C++ consumer code, and update the C# instantiator. In Windows, we'll want to have a shared copy of the instantiator. What do you think about "a pretty OK sample" that shows using this in a common way? Or do we just assume the bots will figure out how to have one copy per process?

  • pick up the managed side separately and integrate lcomp into Lottie-Windows UWP and WinUI packages.

That'd be great! Is there anything needed to get this ready for a C++/WinRT consumer of WASDK XAML/Composition? Does Lottie-Windows have a C++/WinRT package?

  • it'd be good to settle a few things now: include all UAP versions (like codegen does), have a versioning for schema changes, and keep the generated types internal.

See note on supported UAP versions. At some point we should include "supported WASDK versions" and migrate people to using the Microsoft.UI types. I'll make the changes you suggest.

include/flatbuffers + LICENSE inside the native runtime

I'll have to see what that looks like; IIRC I had to tinker a bit to get flatc.exe working correctly.

Thanks for the comments!

Remove the managed and COM loader implementations, add projection-selectable native source, emit multi-UAP FlatBuffer variants, and consolidate the validation tool under source/LottieRuntime.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: adcabe6f-aa5c-4e06-948d-b49e5adae80c
Replace the FlatBuffers submodule with the pinned public headers and license, and document a validated regeneration workflow for matching flatc binaries and headers.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: adcabe6f-aa5c-4e06-948d-b49e5adae80c
@jonwis

jonwis commented Oct 13, 2026 •

Copy link
Copy Markdown
Author

Addressed the FlatBuffers vendoring suggestion from your review: the submodule is removed, and the pinned C++ headers plus upstream LICENSE are now checked into external/flatbuffers. UPSTREAM.md records the exact commit/version, and build/RegenerateFlatbuffers.ps1 now documents where to get the matching flatc.exe and source archive, validates both compiler/header versions, and can refresh the vendored copy. The checked-in files were verified byte-for-byte against the formerly pinned submodule revision.

Jon Wiswall and others added 9 commits October 16, 2026 13:40
Add native metadata loading with live theme properties, cached loaded-image descriptors and image reuse. Preserve the existing visual-only API, fix owned property-set realization ordering, and document the APIs with focused synthetic native regression tests.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: ee78c03d-680b-409f-8f5b-264e3e7f9317
Hosts can implement IAnimatedVisual2 by calling StopAnimations from
DestroyAnimations and StartAnimations from CreateAnimations, as LottieGen's
generated code did. Without it AnimatedVisualPlayer's Resources optimization
cannot release idle animations.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: cfdf7544-9ef9-4586-8d82-56d89206cc1d
Add LottieVisualSource, an IAnimatedVisualSource2 implementation over serialized
compositions for hosts that use AnimatedVisualPlayer. Visuals implement
IAnimatedVisual2 so AnimationOptimization="Resources" can stop animations while
idle, and IClosable. Sources with images also implement
IDynamicAnimatedVisualSource and report image loading progress.

The source selects the embedded variant for the running UAP contract, forwards
theme colors (and scalars through SetScalarProperty) to live and future visuals,
and builds markers using the options each host needs.

Add VisualSourceTests, which hosts XAML and checks animation stop and restart,
theme bindings, marker ranges, closing, and error reporting. Undefine
BUILD_WINDOWS in LottieRuntime.cpp so the vendored copy builds unchanged.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: cfdf7544-9ef9-4586-8d82-56d89206cc1d
…-sample-tool' into jonwis-build-windows-interpreter-sample-tool
The task rejected -Language flatbuffer even though LottieGen.exe supports it.
Accept it in validation and report the generated .lcomp files (one per UAP
version variant) as OutputFiles, using the paths LottieGen prints.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: cfdf7544-9ef9-4586-8d82-56d89206cc1d
Restore pinned FlatBuffers dependencies for MSBuild, bootstrap vcpkg when needed, and replace vendored headers. Package the schema, native sources, headers, and license under the documented consumer layout.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d33f7269-9a0c-4437-aadd-0da22dcee956
Remove package-content validation, restore vcpkg before invoking its compiler for both bindings, and support historical ports in repo-local vcpkg checkouts.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d33f7269-9a0c-4437-aadd-0da22dcee956
Scope managed analyzers and GitVersioning to C# projects and remove stale loader cleanup from the native validation tool.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d33f7269-9a0c-4437-aadd-0da22dcee956

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants