[MONZikWasTaken]

Description

Reworks the native Qt frame generation so it can be tested, controlled and trusted on real hardware, and restores the attribution to the Framegen project the earlier Electron integration came from (#718 → #763). The feature is called FG-Legacy in the app and docs; it is a stepping stone until the FlyHigh model replaces the engine.

Experimental. The current engine (block matching on a ≤320×180 flow grid) was never designed for games: expect strong artifacts on thin edges and fast motion. The UI says so with an Experimental badge and notice. The FlyHigh model will fix this.

What changes

Levels and modes

• 2×, 3× and 4× end to end: settings → pacer → presentation. The engine computes the pair's flow once and synthesises N − 1 frames at t = k / N (each with its own render target and uniform buffer); the presenter keeps five external texture slots so every frame of a 4× pair is selectable.
• One mode selector: Off / Multiplier / Target FPS / Display (Display picks the largest level the detected refresh can show). Display refresh is detected from the window's screen, with a manual override.

Capability test (gates the feature)

• FrameGenerationBenchmark measures the engine offscreen on a private QRhi at the stream resolution, timing 2×, 3× and 4× separately (p95Ms, p95Ms3, p95Ms4). Results are stored per engine|gpu|resolution in settings (Rust-normalised, bounded).
• Frame generation stays unavailable until a measurement exists. Which levels fit is decided live from the selected FPS and display: cost(N) ≤ 0.6 × interval, refresh ≥ N × fps × 0.975.
• Run test opens a dialog with the full table: one row per stream FPS, one cell per level, each with ✓ or the reason (display / GPU / engine) and numbers on hover. Apply writes FPS, mode and level together.

In-stream control

• Ctrl+G stream menu gets a Frame generation tile while the feature is in use; new rebindable shortcut Ctrl+Shift+G toggles it and reports through the stream control message.
• Frame generation choices now persist through the core (they were only applied to the local settings map).

Pacer, fixed against a live GFN session (CS2, 1080p "60" FPS, 240 Hz)

• The "60 FPS" stream actually ran at 61.7 FPS, so 4× needed ~247 Hz and the 240 Hz panel flipped to display-refresh constantly. The interval is now the median of the last eight per-frame deltas; the gate tolerates 5 % and lowers the factor for a pair instead of switching generation off.
• Bursty delivery (two frames ~2 ms apart) was treated as overload with a 2 s cooldown. A burst now just drops the stale schedule; the cooldown is reserved for a presenter that stopped swapping (renamed Paused in the overlay). A single skipped source frame no longer re-warms.
• The stats overlay names the reason behind Discontinuity; every status change is logged to qt-native.log with the pacer's numbers, and a GUI-thread stall probe logs gui-stall ms=… (frames are requested from the GUI thread).

Attribution

• third_party/framegen/README.md, WEIGHTS_LICENSE.md and a THIRD_PARTY_NOTICES entry restored; GitHub icon in the capability-test row opens the Framegen repository. Design notes in docs/fg-legacy.md.

Diagnostics (opt-in, no effect unless set)

• OPENNOW_FG_CAPTURE_DIR=<dir> saves the source pair and every generated frame as PNG a few times per session.
• OPENNOW_FG_DUMP=<prev.png>;<cur.png>;<outdir>;<factor> runs the engine on two images in tst_frameinterpolator.

Testing

Windows 11, RTX 4060 Ti, D3D11, Qt 6.8.3:

• tst_framepacer 24/24, tst_frameinterpolator 32/32 plus the opt-in dump case skipped (new 3×/4× step test: generated frames land exactly at t = k/N on translated texture), tst_framegenerationbenchmark 5/5, tst_nativeframegeneration 13/13 (new 4× presenter test), cargo test for opennow-core, cargo fmt --check, translation and keyboard-layout checks.
• QML smokes with a visible window: frame generation settings, stats (desktop/console), frame-rate settings, stream controllers menu, shortcuts, guide.
• Live GFN session: generation runs at 4× on a 240 Hz panel (local output 170–214 FPS). Capability test on this GPU at 640×360: 0.23 / 0.27 / 0.23 ms per source frame for 2× / 3× / 4×.

Known limitations

• Visual quality is bounded by the block-matching engine: where flow is unreliable it copies the current frame (motion still reads as 60) and thin edges sparkle. This PR does not change the synthesis quality heuristics.
• Stream-route QML smokes fail under QT_QPA_PLATFORM=offscreen on Windows ("Could not select the graphics adapter") with or without this change; they pass with a visible window.

🤖 Generated with Claude Code

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

• Source mode: Auto
• Repository: Unavailable
• Branch: Unavailable
• File: No configuration file from this repository applied
• Commit: Unavailable
• Layers: defaults
• Review profile: CHILL
• Plan: Advanced
• Run ID: 10c2d36c-7a35-4e6d-bd46-c48108d9d225

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• Trigger review

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

• X
• Mastodon
• Reddit
• LinkedIn

CodeRabbit Verify

• Run CodeRabbit Verify to test the changes in this pull request.

[zortos293]

@capyai give me screenshots how the settings looks

[capy-ai]

Screenshots of the frame generation settings at this PR's head, from the desktop settings smoke harness (Linux, Xvfb, 1440×900, 1440p/120 FPS profile). The "tested" states use the PR's --smoke-frame-generation-dialog fixture, which seeds a measurement on a 240 Hz display override. The GPU name reads llvmpipe because this machine has no real GPU.

Before the capability test (frame generation locked)

Capability test dialog

After the test: Multiplier mode

Target FPS mode

Display mode

Advanced → Display refresh override

---

Crash on Linux: on this branch the Stream settings page segfaults as soon as it opens, on every Vulkan (non-Windows, non-Metal) build. ShellStore.ensureFrameGenerationGpu() calls FrameGenerationBenchmark::gpuName(), and the temporary PrivateRhi it creates is destroyed in the wrong order. Members are destroyed in reverse declaration order, so instance (the QVulkanInstance) goes first and ~QRhi then calls vkDeviceWaitIdle through a dead instance:

#0 QVulkanDeviceFunctions::vkDeviceWaitIdle
#2 QRhi::~QRhi()
#5 PrivateRhi::~PrivateRhi  FrameGenerationBenchmark.cpp:80
#6 FrameGenerationBenchmark::gpuName  FrameGenerationBenchmark.cpp:179

Declaring instance before rhi fixes it. With that change applied locally, every page above renders:

 struct PrivateRhi {
-    std::unique_ptr<QRhi> rhi;
 #if !defined(Q_OS_WIN) && !QT_CONFIG(metal) && QT_CONFIG(vulkan) && __has_include(<vulkan/vulkan.h>)
     std::unique_ptr<QVulkanInstance> instance;
 #endif
+    std::unique_ptr<QRhi> rhi;
     QString error;
 };

Windows (D3D11) and macOS (Metal) don't take this path, which is why it didn't show up in the Windows testing.

[capy-ai]

Thanks for this. The design holds up: generation stays inside the existing presenter, it's off by default, storage is bounded, and sender timestamps are kept. Review on 17d01c04 (merges cleanly into dev) found a few bugs to fix before merge. All of them were reproduced on Linux/Qt 6.8.3; paths are relative to opennow-qt/.

Crash (P1)

• src/streaming/rendering/FrameGenerationBenchmark.cpp:81-83 destroys the Vulkan instance before its QRhi, which then segfaults in QRhi::~QRhi → vkDeviceWaitIdle. Opening desktop Stream settings reaches this through gpuName(), so it crashes even with frame generation off.

Capability gate (P2)

• qml/state/settings/SettingsState.qml:807 saves a finished measurement under the currently selected resolution instead of result.width/height. If you change resolution while the test runs, the result lands under the wrong key. A 640×360 run got stored as 3840×2160 and unlocked 4×.
• SettingsState.qml:568 and FrameGenerationBenchmark.cpp:297-315 accept incomplete profiles. Rust turns missing timings into 0, QML passes ok:true, and evaluation never checks the sample count, so frames:0, p95Ms:0 enables 4×.
• FrameGenerationBenchmark.cpp:92-93 benchmarks the default D3D11 adapter, not the stream adapter chosen in GraphicsDeviceSelection.cpp:266. On multi-GPU Windows systems it measures a different GPU than the one presenting.
• SettingsState.qml:468,545-556 only fills in the GPU identity when Stream settings opens or a test runs. After a restart that goes straight into a game, a saved profile resolves to factor 0.

Pacer (P2)

• src/streaming/rendering/StreamFramePacer.h:91,160: a skipped-frame pair (33.33 ms apart) gets scheduled at 16.67 ms cadence, so its motion plays at 2× speed. The arrival gap is also divided before it's checked against the 50 ms limit, so a 90 ms stall still gets interpolated.

Test

• tests/tst_nativeframegeneration.cpp:531 lowers the fixture delay to 8 ms while timestamps still advance by 20 ms. The recovery check at :533 then gets display-refresh instead of active; putting the delay back to 20 ms makes it pass.

Full suite (ctest -LE interactive-desktop): 549/551 pass, and the two failures are the crash and that recovery test. Core Rust tests (845) and locales:check pass. Regression tests for the gate and pacer cases would be great.

[capy-ai]

@MONZikWasTaken GitHub currently reports merge conflicts with dev, so this PR cannot be merged as it stands. Please update your branch from the latest dev, resolve the conflicts, and push the result here. Once updated, please rerun the relevant tests and make sure the PR's CI passes so we can reassess it for merge.

[capy-ai]

Pushed two separately reviewable repairs without merging this PR:

• f75716b fixes the reproduced Vulkan teardown crash, rejects incomplete capability measurements, preserves measured GPU/resolution keys, hydrates saved-profile identity before direct launch, and prevents a result for another resolution from switching a valid current selection off. Both identity lookup and measurement now use the selected Windows adapter LUID and reject fallback to a different adapter.
• 1d986da restores the recovery fixture's original 20 ms cadence to match its 20 ms sender timestamps. Its 8 ms sleep selected arrival cadence and exceeded the configured display gate. All recovery, ownership, release and active-state assertions remain unchanged; production pacing is untouched.

After the final edits, the application builds and all 551 noninteractive Qt CTests pass. The parent independently reran the benchmark, native frame-generation and orchestration targets successfully. Windows-only adapter tests still need Windows CI; Linux uses llvmpipe and does not establish physical hybrid-GPU acceptance.

This screenshot shows the real Qt dialog with seeded smoke timings, not measured GPU performance:

The seven existing dev-integration conflicts remain, so these passing checks do not make the feature merge-ready.

[capy-ai]

Published current-dev integration as merge commit 0cd994c6, retaining the separate benchmark/profile repairs f75716b2 and recovery-fixture correction 1d986da7. This updates the feature branch only; the PR has not been merged.

All seven real conflicts are resolved. The integration retains dev's sender-authored timing, sender-paced admission/deadlines, exact borrowed-handle release, and scene-graph/device teardown. Native streamer and NativeStreamRuntime remain byte-identical to dev's ABI18 implementations. The callback now combines those contracts with factor-based generation and drains pending generated frames on NO_FRAME/STALE_FRAME paths. Both the FG tile and pointer-lock row remain in the menu with coherent focus traversal.

New combined-contract tests cover sender-paced bypass at factors 0/2/4, borrowed NO_FRAME/STALE_FRAME handles while draining a 4× pair, and optional-control navigation in windowed/fullscreen modes. The benchmark is now included in dev's CI unit targets and Windows runtime/deployment lists; its registration regression failed before the fix.

Validation on the final integrated tree:

• Complete application build passes.
• All 591/591 noninteractive Qt CTests pass, including 41 CI unit targets, in 584.24 seconds.
• Seven focused contract suites and 21 overlay/navigation checks pass. Parent independently reran six sender/benchmark/native/orchestration/menu targets successfully.
• All 24 CI-workflow checks and staged/unstaged whitespace checks pass.

Fresh exact-head cross-platform CI/review is still required. Physical Windows D3D11/hybrid-GPU adapter selection and presentation feedback, macOS pointer confinement, and real GFN gameplay are not validated by this Linux/llvmpipe run.

The real Qt menu below uses synthetic acceptance telemetry and optional-control visibility, not gameplay or physical pointer-lock evidence.

[capy-ai]

Published two separately scoped follow-ups:

• 81841299 fixes the two original pacing findings. The discontinuity gate now checks the raw arrival gap; generated slots use the accepted source pair's full span, while nominal cadence and display gates remain unchanged. Ten new regression rows failed before the fix: six skipped-pair timing cases across factors 2/3/4 and sender/arrival clocks, plus four over-limit raw-gap cases. The exact 50 ms boundary remains accepted, and the next consecutive pair returns to normal scheduling. All existing assertions remain intact.
• 291fc6c4 corrects the Windows compile failure at the two new std::min calls by using (std::min)(...), without global header or build configuration changes. Macro-exposure syntax checks fail before and pass afterward; this host check is not an MSVC build.

The behavioral pacing patch passes all 591 Qt CTests and the 36-case pacer suite. After the final parentheses-only normalization, the app rebuild and all five affected native/pacer/sender/benchmark/stream-video targets pass; parent independently reran those same five targets. Fresh exact-head Windows CI is required to clear the reported failure. The prior Windows job did compile both the selected-adapter benchmark source and its tests, but stopped before executing them at the callback compile error.

[capy-ai]

All seven checks passed at 291fc6c4, including the unchanged-head macOS rerun. The SDP failure is independently reproducible as a pending MID collision, now isolated in #1406 rather than mixed into frame generation.

Published 92521905 to persist the benchmark's Qt JUnit result file in the existing CI diagnostics artifact. The prior Windows run executed the benchmark successfully but captured no per-case output, so it could not prove whether selected-adapter acceptance passed or skipped. No workload or assertion changed. Benchmark and orchestration checks pass; the parent independently reran both and inspected all 18 result cases. Linux correctly skips Windows D3D11 selection; software Vulkan timing is not physical-GPU acceptance. Fresh exact-head Windows CI must supply the missing case-level evidence. Hybrid-GPU selection/presentation feedback, Cocoa confinement and live gameplay remain unverified.

[capy-ai]

Exact-head verification at 92521905 is complete: all seven checks pass. The new Windows benchmark JUnit artifact contains 18 cases, zero failures and zero skips. selectedAdapterIsUsedAndMismatchesFailClosed, GPU identity lifecycle and measuresTheEngineOnThisGpu all passed, rather than being silently skipped.

The recorded adapter is Microsoft Basic Render Driver / D3D11, with gpuTimed: false. This confirms the selected-adapter and mismatch checks execute under Windows, not physical hybrid-GPU or production presentation-feedback acceptance. Those hardware checks, Cocoa confinement and live gameplay remain outstanding. No PR was merged.