graphics-hook: Synchronize Vulkan captures with semaphores

[tbgiles]

Description

When capturing a Vulkan application, specifically when the graphics and present queues are not identical, it is insufficient to rely on memory barriers alone.

When working across queues, as some applications may, semaphores are required to handle cross-queue synchronization, and ensure that capture happens strictly after rendering has completed.

An example of what I was able to see without change is shown in the attached image.

When the graphics and present queues are not identical, capturing has a race. A barrier in the presentation queue won't synchronize the work done in the graphics queue, since it only asserts an ordering on work in the same queue, so commands may still be executing in the graphics queue at the moment we take the capture (which happens on the present queue).

Before this change, capture submission did not wait on a semaphore. The changes here assert an ordering from graphics -> capture -> present using OBS-owned semaphores to prevent mid-draw artifacts, and are indexed on each swapchain image rather than logical frame to ensure that each is protected accordingly. Capture submission now waits for rendering to complete.

Motivation and Context

Vulkan applications with separate graphics/present queues were being captured with artifacts like those I have shown above. To be clear, this is not an visible when graphics/present happen in the same queue. While I didn't see any open tickets for this specifically, I was able to unearth this ticket that seemed to exhibit some of the same issues I was having. I made sure I was using an up-to-date driver first, then I enabled verbose validation and saw a host of messages when capturing/previewing regarding sync hazards (write racing write, and present after write), like in the tagged issue. That same issue mentions "...improper synchronization, resulting in OBS grabbing the image sometimes at the wrong interval (mid program write)", which is ultimately what I observed happening, and is addressed in this PR.

How Has This Been Tested?

Hardware: NVIDIA GeForce RTX 2070 / Ryzen 7 5800x
OS Version: Microsoft Windows 11 Home / Version 10.0.26200
Graphics Driver Version: GeForce Game Ready Driver 616.92

I've tested it over the last week by running (each step against both altered/unaltered hook dll):

1. Own renderer using same graphics/present queue
2. Own renderer using different graphics/present queue
3. Vulkan Cube (vkcube)

The changes are isolated to the Vulkan graphics-hook dll. I've tested in both x64 Debug and x64 Release, and left the applications running for several minutes.

For my renderer, I forced both same and separate graphics/present queues. Artifacts were no longer showing, both through the preview window, as well as in the output file after recording a small clip. I also did not see any SYNC-HAZARD-WRITE-RACING-WRITE or SYNC-HAZARD-PRESENT-AFTER-WRITE after this change. I forced a number of swapchain recreations (to make sure that the recreation works with the new locks) and continued the capture.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• Tweak (non-breaking change to improve existing functionality)

Checklist:

• I have read the contributing document.
• My code has been run through clang-format.
• My code follows the project's style guidelines
• My code is not on the master branch.
• My code has been tested.
• All commit messages are properly formatted and commits squashed where appropriate.
• I have included updates to all appropriate documentation.