diff --git a/AGENTS.md b/AGENTS.md index c69eb37c..44c312ee 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -110,6 +110,13 @@ rebuilt, and getting that wrong is silent rather than loud. Build the reference resolution-changing stream to its first size and a whole-stream reference will not compare. +Pixelforge can also run on a device the caller created, for decode and for +encode. `tests/adopted.rs` builds one from a Vulkan 1.1 instance, as an +application that never asked for 1.3 would, hands pixelforge spare queues, and +checks the output against a context pixelforge created itself. It needs no +ffmpeg. Anything that reaches a core 1.2 or 1.3 entry point panics there, so +run it after touching submission or barrier code. + Make sure there are no Vulkan validation layer errors during execution. Enable them with `PIXELFORGE_VALIDATION=1`; the layer's messages are routed through `tracing`, so pair it with `RUST_LOG=warn` (or `debug` for the layer's own diff --git a/CHANGELOG.md b/CHANGELOG.md index 46d67387..e2aa1610 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -102,6 +102,23 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Validation layer messages are now routed into `tracing` through a `VK_EXT_debug_utils` messenger. Previously the layer was loaded with nowhere to report, so enabling validation verified nothing. +- Encoding on a caller-created device: `encode_device_requirements` and + `build_from_existing_encode`, the counterparts of the decode pair. An + application that renders on its own device can encode its images there + directly, with no external memory and no second device. +- `DeviceRequirements::features` names the device features pixelforge needs as + booleans, so a caller can set them in whichever feature structs it already + chains. +- `VideoContextBuilder::with_encode_queue`, `with_decode_queue`, + `with_transfer_queue` and `with_compute_queue` choose exactly which queue an + adopted context submits to, so the caller can keep its own queues to itself. + `DeviceRequirements::queues` says which family serves which role, and + `DeviceRequirements::internally_synchronized_queues` whether the device can + share a queue safely instead. +- `ColorConverter::convert_async` and `Encoder::encode_after` wait for, and in + the converter's case signal, a `TimelinePoint`, so a frame can go from the + caller's rendering through conversion to encode ordered on the GPU, with no + CPU wait in between. ### Changed @@ -112,6 +129,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 replaced by `VideoContext::has_push_descriptor`. - Adopting a caller-created device for decode (`build_from_existing_decode`) now requires the `timelineSemaphore` feature in addition to `synchronization2`. +- Adopted devices may come from an instance asking for any Vulkan version from + 1.1 up. Pixelforge reaches synchronization2 through its extension rather than + core 1.3, and the requirements list `VK_KHR_timeline_semaphore`. +- `DeviceRequirements` is `#[non_exhaustive]`, like the new `DeviceFeatures` + and `QueueRoles`, so later fields are not breaking changes. It can no longer + be built with a struct literal outside pixelforge. ### Fixed diff --git a/README.md b/README.md index c4b2e23a..448b44f0 100644 --- a/README.md +++ b/README.md @@ -15,6 +15,9 @@ H.265 and AV1 encode, and H.264 decode. Encoding hands back an [`EncodeFuture`]; decoding delivers frames through a [`DecodeSource`] as the GPU finishes with them. - **GPU color conversion**: RGB/BGR → YUV via Vulkan compute shaders (BT.709, BT.2020, sRGB→BT.2020+PQ, scRGB-linear→BT.2020+PQ). +- **Shared devices**: encode and decode on a Vulkan device your application + created, from Vulkan 1.1 up, on queues you choose, ordered against your own + work with timeline semaphores. - **HDR support**: 10-bit encoding (P010, YUV444P10), PQ transfer function, BT.2020 color space. - **GPU-native API**: Encode directly from Vulkan images (`vk::Image`). - **Flexible configuration**: Rate control (CBR, VBR, CQP), quality levels, GOP settings. @@ -247,6 +250,38 @@ let mut encoder = Encoder::new(context, encode_config)?; // converter.convert(input_image, layout, encoder.input_image())?; ``` +### Encoding on your own device + +An application that already renders with Vulkan can give pixelforge its own +device instead of letting it create a second one. Frames then never leave +the device: no external memory, no import, and no CPU wait between the +application's work and pixelforge's. + +[`VideoContextBuilder::encode_device_requirements`] lists the queue +families, extensions and features to create the device with, and +[`VideoContextBuilder::build_from_existing_encode`] adopts it. A `VkQueue` +may not be submitted to from two threads at once, so give pixelforge queues +the application does not use, with +[`VideoContextBuilder::with_encode_queue`] and friends. + +[`ColorConverter::convert_async`] and [`Encoder::encode_after`] then order +each frame on the GPU: the application's submission signals a +[`TimelinePoint`], the conversion waits for it and signals its own, and the +encode waits for that. This stays off the CPU as long as the encoder's own +[`Encoder::input_image`] is what gets encoded, as below: any other image is +copied into it first, and that copy is waited for on the CPU. + +```rust +// `rendered` reaches `frame_number` once the application has drawn `image`. +let converted = converter.convert_async( + image, + vk::ImageLayout::GENERAL, + encoder.input_image(), + &[TimelinePoint::new(rendered, frame_number)], +)?; +let packet = encoder.encode_after(encoder.input_image(), &[converted])?; +``` + ## Benchmarking Run the encode latency benchmark with: @@ -266,9 +301,6 @@ cargo run --example query_capabilities # Decode H.264 to raw YUV cargo run --example decode -- input.264 output.yuv -# Decode on a caller-created Vulkan device -cargo run --example decode_adopted -- input.264 output.yuv - # Encode, choosing the codec (h264, h265 or av1) cargo run --example encode -- h265 diff --git a/examples/decode_adopted.rs b/examples/decode_adopted.rs deleted file mode 100644 index 6e6d077f..00000000 --- a/examples/decode_adopted.rs +++ /dev/null @@ -1,148 +0,0 @@ -//! Decode on a caller-created device. -//! -//! Mirrors what an application with its own Vulkan device does: create the -//! instance and device itself, then hand them to pixelforge via -//! `build_from_existing_decode` so decoded images live on the app's device. -//! -//! Usage: -//! decode_adopted [output.yuv] -//! -//! Output is NV12 in display order, comparable to: -//! ffmpeg -i input.264 -pix_fmt nv12 reference.yuv - -use std::fs::File; - -use ash::vk; -use ash::vk::TaggedStructure; -#[allow(dead_code)] -mod common; -use common::{Readback, decode_stream, write_nv12}; - -use pixelforge::decoder::{DecodeConfig, Decoder}; -use pixelforge::encoder::Codec; -use pixelforge::vulkan::VideoContextBuilder; - -fn main() -> Result<(), Box> { - let mut args = std::env::args().skip(1); - let input_path = args - .next() - .expect("usage: decode_adopted [output.yuv]"); - let output_path = args.next(); - - // --- The application creates its own instance and device --- - let entry = unsafe { ash::Entry::load()? }; - let app_name = c"decode-adopted"; - let app_info = vk::ApplicationInfo::default() - .application_name(app_name) - .api_version(vk::API_VERSION_1_3); - let instance_info = vk::InstanceCreateInfo::default().application_info(&app_info); - let instance = unsafe { entry.create_instance(&instance_info, None)? }; - - // Ask pixelforge what a decode device needs, and pick a physical device that - // satisfies it — exactly the negotiation an app would do. - let builder = VideoContextBuilder::new().require_decode(Codec::H264); - let physical_devices = unsafe { instance.enumerate_physical_devices()? }; - let (physical_device, reqs) = physical_devices - .iter() - .find_map(|&pd| { - builder - .decode_device_requirements(&entry, &instance, pd) - .ok() - .map(|r| (pd, r)) - }) - .expect("no physical device can decode H.264"); - - let name = unsafe { - std::ffi::CStr::from_ptr( - instance - .get_physical_device_properties(physical_device) - .device_name - .as_ptr(), - ) - }; - println!("Device: {}", name.to_string_lossy()); - println!( - "pixelforge needs queue families {:?} and {} extensions", - reqs.queue_families, - reqs.extensions.len() - ); - - // The app merges pixelforge's requirements with its own. Here there is - // nothing else, so we use them directly. - let priorities = [1.0f32]; - let queue_infos: Vec = reqs - .queue_families - .iter() - .map(|&f| { - vk::DeviceQueueCreateInfo::default() - .queue_family_index(f) - .queue_priorities(&priorities) - }) - .collect(); - let ext_ptrs: Vec<*const std::os::raw::c_char> = - reqs.extensions.iter().map(|e| e.as_ptr()).collect(); - - let mut sync2 = vk::PhysicalDeviceSynchronization2Features::default().synchronization2(true); - // The decoder orders its pipelined submissions with timeline semaphores. - let mut timeline = - vk::PhysicalDeviceTimelineSemaphoreFeatures::default().timeline_semaphore(true); - // Unified image layouts are what let decoded frames be used in place, with - // no copy and no layout transition. `reqs` says whether this device can, - // and puts the extension in `reqs.extensions` when it can; both feature - // bits have to be enabled here, and pixelforge told with - // `declare_unified_image_layouts` below, because Vulkan cannot be asked - // afterwards which features a device was created with. - let mut unified = vk::PhysicalDeviceUnifiedImageLayoutsFeaturesKHR::default() - .unified_image_layouts(true) - .unified_image_layouts_video(true); - let mut device_info = vk::DeviceCreateInfo::default() - .queue_create_infos(&queue_infos) - .enabled_extension_names(&ext_ptrs) - .push(&mut sync2) - .push(&mut timeline); - if reqs.unified_image_layouts { - device_info = device_info.push(&mut unified); - } - let device = unsafe { instance.create_device(physical_device, &device_info, None)? }; - - // --- Hand the app's device to pixelforge --- - let builder = if reqs.unified_image_layouts { - builder.declare_unified_image_layouts() - } else { - println!("device has no unified image layouts; frames will be copied out"); - builder - }; - let context = builder.build_from_existing_decode( - entry.clone(), - instance.clone(), - physical_device, - device.clone(), - )?; - - let stream = std::fs::read(&input_path)?; - let mut output = output_path.as_ref().map(File::create).transpose()?; - let mut readback = output - .is_some() - .then(|| Readback::new(&context)) - .transpose()?; - let mut decoder = Decoder::new(context, DecodeConfig::h264().with_byte_stream())?; - let mut count = 0usize; - - decode_stream(&mut decoder, &stream, |frame| { - write_nv12(&frame, &mut readback, &mut output)?; - count += 1; - Ok(()) - })?; - println!("Decoded {} frames on the adopted device", count); - - // The context borrowed the device, so everything holding Vulkan objects has - // to go before the device does. That includes the readback helper, which - // owns a command pool, a fence and a staging buffer of its own. - drop(decoder); - drop(readback); - unsafe { - device.destroy_device(None); - instance.destroy_instance(None); - } - Ok(()) -} diff --git a/src/converter/mod.rs b/src/converter/mod.rs index 07f6f20f..15eadaa7 100644 --- a/src/converter/mod.rs +++ b/src/converter/mod.rs @@ -10,6 +10,8 @@ mod pipeline; use crate::encoder::ColorDescription; use crate::error::{PixelForgeError, Result}; +use crate::sync::TimelinePoint; +use crate::video::TimelineChain; use crate::vulkan::VideoContext; use ash::vk; use tracing::debug; @@ -387,6 +389,12 @@ pub struct ColorConverter { command_pool: vk::CommandPool, command_buffer: vk::CommandBuffer, fence: vk::Fence, + /// Whether a conversion was submitted and `fence` not yet waited on. The + /// next conversion reuses the command buffer and output buffer, and + /// destroys this one's source view, so it waits for this one first. + in_flight: bool, + /// Signalled by each conversion, for [`Self::convert_async`] to hand out. + timeline: TimelineChain, } impl ColorConverter { @@ -760,9 +768,41 @@ impl ColorConverter { target_image: vk::Image, ) -> Result<()> { let start = std::time::Instant::now(); + self.convert_async(src_image, src_layout, target_image, &[])?; + self.wait_idle()?; + debug!("ColorConverter::convert() took {:?}", start.elapsed()); + Ok(()) + } - // A conversion that failed part way may have left its view behind. - self.destroy_src_view(); + /// Convert like [`Self::convert`], but without waiting on the CPU. + /// + /// The conversion waits on the GPU for every point in `wait` before + /// reading `src_image`, and returns the point it signals when done. Pass + /// that to [`Encoder::encode_after`](crate::Encoder::encode_after) to + /// encode the result, and wait on it before writing `src_image` again. + /// + /// On a device shared with the caller, where the caller renders or copies + /// into `src_image` itself, this is the whole handover: the caller's + /// submission signals a [`TimelinePoint`], the conversion waits on it, the + /// encode waits on the conversion. If the caller's queue is in another + /// family than [`VideoContext::compute_queue_family`], create `src_image` + /// with `VK_SHARING_MODE_CONCURRENT` across both, and pass the layout it + /// is actually in as `src_layout`: `UNDEFINED` is taken to mean a + /// first-time external-memory import and acquires ownership from + /// `VK_QUEUE_FAMILY_EXTERNAL`, which an image that never left the device + /// does not have to give. + /// + /// The next call to either method waits for this conversion to finish + /// before recording, since both reuse the same command buffer. + pub fn convert_async( + &mut self, + src_image: vk::Image, + src_layout: vk::ImageLayout, + target_image: vk::Image, + wait: &[TimelinePoint], + ) -> Result { + // Before anything is reused, and before the previous view goes. + self.wait_idle()?; let src_view = self.create_src_view(src_image)?; let device = self.context.device(); @@ -1024,33 +1064,54 @@ impl ColorConverter { .map_err(|e| PixelForgeError::CommandBuffer(e.to_string()))?; } - // Submit and wait. + let waits: Vec = + wait.iter().map(TimelinePoint::wait_info).collect(); + let (semaphore, value) = self.timeline.pending_signal(); + let signals = [vk::SemaphoreSubmitInfo::default() + .semaphore(semaphore) + .value(value) + .stage_mask(vk::PipelineStageFlags2::ALL_COMMANDS)]; + let command_buffers = + [vk::CommandBufferSubmitInfo::default().command_buffer(self.command_buffer)]; + let submit_info = vk::SubmitInfo2::default() + .wait_semaphore_infos(&waits) + .command_buffer_infos(&command_buffers) + .signal_semaphore_infos(&signals); + unsafe { device .reset_fences(&[self.fence]) .map_err(|e| PixelForgeError::CommandBuffer(e.to_string()))?; - - let command_buffers = [self.command_buffer]; - let submit_info = vk::SubmitInfo::default().command_buffers(&command_buffers); - - device - .queue_submit(self.context.compute_queue(), &[submit_info], self.fence) - .map_err(|e| PixelForgeError::CommandBuffer(e.to_string()))?; - - device - .wait_for_fences(&[self.fence], true, u64::MAX) + self.context + .sync2() + .queue_submit2(self.context.compute_queue(), &[submit_info], self.fence) .map_err(|e| PixelForgeError::CommandBuffer(e.to_string()))?; } - self.destroy_src_view(); + self.timeline.commit(); + self.in_flight = true; - let elapsed = start.elapsed(); - debug!("ColorConverter::convert() took {:?}", elapsed); + Ok(TimelinePoint::new(semaphore, value)) + } + /// Wait on the CPU for the last conversion submitted, if it is still + /// running, and destroy the view it read its source through. A conversion + /// that failed before submitting leaves its view for this to destroy too. + fn wait_idle(&mut self) -> Result<()> { + if self.in_flight { + unsafe { + self.context + .device() + .wait_for_fences(&[self.fence], true, u64::MAX) + } + .map_err(|e| PixelForgeError::CommandBuffer(e.to_string()))?; + self.in_flight = false; + } + self.destroy_src_view(); Ok(()) } /// Create the view this conversion reads `src_image` through. It lives - /// until the conversion is done, see [`Self::destroy_src_view`]. + /// until the conversion is done, see [`Self::wait_idle`]. fn create_src_view(&mut self, src_image: vk::Image) -> Result { let view_info = vk::ImageViewCreateInfo::default() .image(src_image) @@ -1081,9 +1142,14 @@ impl ColorConverter { impl Drop for ColorConverter { fn drop(&mut self) { - self.destroy_src_view(); + // Nothing below may be destroyed while a conversion still uses it. + let _ = self.wait_idle(); unsafe { let device = self.context.device(); + // Not destroyed here: an encode may still be waiting on a point + // this converter handed out, and that wait outlives the + // conversion it is waiting for. + self.context.retire_semaphore(self.timeline.semaphore()); // Destroy sampler. device.destroy_sampler(self.sampler, None); diff --git a/src/converter/pipeline.rs b/src/converter/pipeline.rs index 65c9aa2e..3eb332f9 100644 --- a/src/converter/pipeline.rs +++ b/src/converter/pipeline.rs @@ -140,6 +140,7 @@ pub fn create_converter( let push_descriptor = ash::khr::push_descriptor::Device::load(context.instance(), context.device()); + let timeline = crate::video::TimelineChain::new(&context)?; // Create command pool for compute queue. let pool_info = vk::CommandPoolCreateInfo::default() @@ -177,6 +178,8 @@ pub fn create_converter( command_pool, command_buffer, fence, + in_flight: false, + timeline, push_descriptor, }) } diff --git a/src/decoder/common.rs b/src/decoder/common.rs index d7379520..e55db5c5 100644 --- a/src/decoder/common.rs +++ b/src/decoder/common.rs @@ -581,7 +581,7 @@ impl DecoderCommon { /// (if distinct), into the layouts `vkCmdDecodeVideo` requires. pub fn record_barriers(&self, slot: u8) { let session = self.session.as_ref().expect("active session"); - let device = self.context.device(); + let sync2 = self.context.sync2(); let mut barriers: Vec = Vec::new(); let subresource = |layer: u32| vk::ImageSubresourceRange { @@ -633,7 +633,7 @@ impl DecoderCommon { } let dependency = vk::DependencyInfo::default().image_memory_barriers(&barriers); - unsafe { device.cmd_pipeline_barrier2(self.decode_command_buffer(), &dependency) }; + unsafe { sync2.cmd_pipeline_barrier2(self.decode_command_buffer(), &dependency) }; } /// The decode command buffer of the picture currently being recorded. @@ -674,7 +674,8 @@ impl DecoderCommon { pub fn submit_decode(&mut self) -> Result<()> { let device = self.context.device().clone(); let queue = self.decode_queue; - self.pipeline.submit_decode(&device, queue) + self.pipeline + .submit_decode(&device, self.context.sync2(), queue) } /// Close off the current picture and hand `frames` to the completion thread. diff --git a/src/decoder/pipeline.rs b/src/decoder/pipeline.rs index 62b965fc..ef22383e 100644 --- a/src/decoder/pipeline.rs +++ b/src/decoder/pipeline.rs @@ -241,6 +241,7 @@ impl DecodePipeline { pub(crate) fn submit_decode( &mut self, device: &ash::Device, + sync2: &ash::khr::synchronization2::Device, decode_queue: vk::Queue, ) -> Result<()> { let slot = &self.slots[self.current_slot]; @@ -285,7 +286,7 @@ impl DecodePipeline { .signal_semaphore_infos(&signals); unsafe { - device + sync2 .queue_submit2(decode_queue, &[submit], slot.decode_fence) .map_err(|e| PixelForgeError::Synchronization(e.to_string()))?; } @@ -302,6 +303,7 @@ impl DecodePipeline { pub(crate) fn submit_copy( &mut self, device: &ash::Device, + sync2: &ash::khr::synchronization2::Device, transfer_queue: vk::Queue, ) -> Result<()> { let slot = &self.slots[self.current_slot]; @@ -340,7 +342,7 @@ impl DecodePipeline { .signal_semaphore_infos(&signals); unsafe { - device + sync2 .queue_submit2(transfer_queue, &[submit], slot.transfer_fence) .map_err(|e| PixelForgeError::Synchronization(e.to_string()))?; } diff --git a/src/decoder/transfer.rs b/src/decoder/transfer.rs index 03f5f53b..86752bd2 100644 --- a/src/decoder/transfer.rs +++ b/src/decoder/transfer.rs @@ -36,6 +36,7 @@ impl DecoderCommon { let command_buffer = self.pipeline.current().transfer_command_buffer; let base_layer = frame.array_layer; let device = self.context.device().clone(); + let sync2 = self.context.sync2(); let aspects = vk::ImageAspectFlags::PLANE_0 | vk::ImageAspectFlags::PLANE_1; let range = |image: vk::Image, layer: u32| vk::ImageMemoryBarrier2 { image, @@ -85,7 +86,7 @@ impl DecoderCommon { }, ]; let dep = vk::DependencyInfo::default().image_memory_barriers(&to_transfer); - unsafe { device.cmd_pipeline_barrier2(command_buffer, &dep) }; + unsafe { sync2.cmd_pipeline_barrier2(command_buffer, &dep) }; let plane = |aspect: vk::ImageAspectFlags| vk::ImageSubresourceLayers { aspect_mask: aspect, @@ -95,7 +96,7 @@ impl DecoderCommon { }; let (cdiv_hor, cdiv_vert) = frame.pixel_format.chroma_div(); let regions = [ - vk::ImageCopy2::default() + vk::ImageCopy::default() .src_subresource(vk::ImageSubresourceLayers { base_array_layer: base_layer, ..plane(vk::ImageAspectFlags::PLANE_0) @@ -106,7 +107,7 @@ impl DecoderCommon { height: frame.coded_height, depth: 1, }), - vk::ImageCopy2::default() + vk::ImageCopy::default() .src_subresource(vk::ImageSubresourceLayers { base_array_layer: base_layer, ..plane(vk::ImageAspectFlags::PLANE_1) @@ -118,13 +119,19 @@ impl DecoderCommon { depth: 1, }), ]; - let copy = vk::CopyImageInfo2::default() - .src_image(frame.image) - .src_image_layout(vk::ImageLayout::TRANSFER_SRC_OPTIMAL) - .dst_image(dst_image) - .dst_image_layout(vk::ImageLayout::TRANSFER_DST_OPTIMAL) - .regions(®ions); - unsafe { device.cmd_copy_image2(command_buffer, ©) }; + // `vkCmdCopyImage` rather than its `2` form: the copy needs nothing the + // newer entry point adds, and this one exists on every device, where + // `vkCmdCopyImage2` needs Vulkan 1.3 or `VK_KHR_copy_commands2`. + unsafe { + device.cmd_copy_image( + command_buffer, + frame.image, + vk::ImageLayout::TRANSFER_SRC_OPTIMAL, + dst_image, + vk::ImageLayout::TRANSFER_DST_OPTIMAL, + ®ions, + ) + }; // Restore the source layout so a DPB image stays a valid reference. let restore = [vk::ImageMemoryBarrier2 { @@ -137,7 +144,7 @@ impl DecoderCommon { ..range(frame.image, base_layer) }]; let dep = vk::DependencyInfo::default().image_memory_barriers(&restore); - unsafe { device.cmd_pipeline_barrier2(command_buffer, &dep) }; + unsafe { sync2.cmd_pipeline_barrier2(command_buffer, &dep) }; Ok(()) } @@ -146,6 +153,7 @@ impl DecoderCommon { pub fn submit_copy(&mut self) -> Result<()> { let device = self.context.device().clone(); let queue = self.transfer_queue; - self.pipeline.submit_copy(&device, queue) + self.pipeline + .submit_copy(&device, self.context.sync2(), queue) } } diff --git a/src/encoder/av1/record.rs b/src/encoder/av1/record.rs index 89fd4035..62611ec1 100644 --- a/src/encoder/av1/record.rs +++ b/src/encoder/av1/record.rs @@ -50,7 +50,7 @@ impl Av1 { let ref_dpb_slots: Vec = self.references.iter().map(|r| r.dpb_slot).collect(); unsafe { record_dpb_barriers( - common.device(), + common.context.sync2(), command_buffer, &common.dpb_images, common.use_layered_dpb, diff --git a/src/encoder/codec.rs b/src/encoder/codec.rs index b0f69b01..74edee7e 100644 --- a/src/encoder/codec.rs +++ b/src/encoder/codec.rs @@ -24,6 +24,7 @@ use crate::encoder::resources::{ }; use crate::encoder::{ColorDescription, EncodeConfig, FrameType, RateControlMode}; use crate::error::{PixelForgeError, Result}; +use crate::sync::TimelinePoint; use crate::vulkan::VideoContext; /// Per-encoder state shared by every codec. @@ -72,6 +73,9 @@ pub(crate) struct EncoderCommon { pub upload_command_pool: vk::CommandPool, pub upload_command_buffer: vk::CommandBuffer, pub upload_fence: vk::Fence, + /// Caller work this frame's first submission must wait for. The upload + /// takes them when it copies, otherwise the encode submission does. + pub pending_waits: Vec, } impl EncoderCommon { @@ -100,6 +104,7 @@ impl EncoderCommon { if src_image == dst_image { return Ok(()); } + let waits = std::mem::take(&mut self.pending_waits); let params = UploadParams { upload_command_buffer: self.upload_command_buffer, @@ -111,6 +116,7 @@ impl EncoderCommon { pixel_format: self.config.pixel_format, input_image_layout, upload_queue: self.context.transfer_queue(), + waits: &waits, }; upload_image_to_input(&self.context, ¶ms)?; self.pipeline.current_mut().input_image_layout = vk::ImageLayout::VIDEO_ENCODE_SRC_KHR; @@ -128,9 +134,13 @@ impl EncoderCommon { let encode_queue = self.context.video_encode_queue().ok_or_else(|| { PixelForgeError::NoSuitableDevice("No video encode queue available".to_string()) })?; - let future = self - .pipeline - .submit_current(self.context.device(), encode_queue)?; + let waits = std::mem::take(&mut self.pending_waits); + let future = self.pipeline.submit_current( + self.context.device(), + self.context.sync2(), + encode_queue, + &waits, + )?; self.dpb_slot_active[self.current_dpb_slot as usize] = true; Ok(future) } @@ -312,9 +322,16 @@ impl CodecEncoder { self.common.pipeline.input_image() } - /// Encode one frame, returning a future for its packet. See [`crate::Encoder::encode`]. - pub fn encode(&mut self, src_image: vk::Image) -> Result { + /// Encode one frame once `wait` is reached, returning a future for its + /// packet. See [`crate::Encoder::encode_after`]. + pub fn encode_after( + &mut self, + src_image: vk::Image, + wait: &[TimelinePoint], + ) -> Result { let plan = self.common.begin_frame(); + self.common.pending_waits.clear(); + self.common.pending_waits.extend_from_slice(wait); self.common.upload(src_image)?; let setup = self.codec.begin_picture(&mut self.common, &plan)?; @@ -713,6 +730,7 @@ pub(crate) fn build_encoder_common(req: &CommonInitRequest) -> Result = self.l0_references.iter().map(|r| r.dpb_slot).collect(); unsafe { record_dpb_barriers( - common.device(), + common.context.sync2(), command_buffer, &common.dpb_images, common.use_layered_dpb, diff --git a/src/encoder/h265/record.rs b/src/encoder/h265/record.rs index 70d50c71..10ce65b6 100644 --- a/src/encoder/h265/record.rs +++ b/src/encoder/h265/record.rs @@ -39,7 +39,7 @@ impl H265 { let ref_dpb_slots: Vec = self.l0_references.iter().map(|r| r.dpb_slot).collect(); unsafe { record_dpb_barriers( - common.device(), + common.context.sync2(), command_buffer, &common.dpb_images, common.use_layered_dpb, diff --git a/src/encoder/mod.rs b/src/encoder/mod.rs index b225bc5c..ed3a02bf 100644 --- a/src/encoder/mod.rs +++ b/src/encoder/mod.rs @@ -42,6 +42,7 @@ pub const DEFAULT_H265_QP: u32 = 28; pub const DEFAULT_MAX_REFERENCE_FRAMES: u32 = 4; use crate::error::Result; +use crate::sync::TimelinePoint; use crate::vulkan::VideoContext; /// Video codec types. @@ -571,7 +572,11 @@ pub struct EncodedPacket { /// behind a single boxed pointer instead of an enum that re-dispatches by hand. trait EncoderApi: Send { fn input_image(&self) -> vk::Image; - fn encode(&mut self, src_image: vk::Image) -> Result; + fn encode_after( + &mut self, + src_image: vk::Image, + wait: &[TimelinePoint], + ) -> Result; fn flush(&mut self) -> Result<()>; fn request_idr(&mut self); fn invalidate_reference_frames(&mut self, first_lost_display_order: u64); @@ -582,8 +587,12 @@ impl EncoderApi for codec::CodecEncoder { fn input_image(&self) -> vk::Image { codec::CodecEncoder::input_image(self) } - fn encode(&mut self, src_image: vk::Image) -> Result { - codec::CodecEncoder::encode(self, src_image) + fn encode_after( + &mut self, + src_image: vk::Image, + wait: &[TimelinePoint], + ) -> Result { + codec::CodecEncoder::encode_after(self, src_image, wait) } fn flush(&mut self) -> Result<()> { codec::CodecEncoder::flush(self) @@ -663,7 +672,29 @@ impl Encoder { /// # } /// ``` pub fn encode(&mut self, src_image: vk::Image) -> Result { - self.0.encode(src_image) + self.0.encode_after(src_image, &[]) + } + + /// Encode a frame once GPU work the caller submitted has finished. + /// + /// Like [`Self::encode`], but the frame's first GPU submission waits for + /// every point in `wait` before touching `src_image`. Pass the point + /// returned by [`ColorConverter::convert_async`](crate::ColorConverter::convert_async) + /// to encode what it converted without waiting for the conversion on the + /// CPU, or a point the caller's own rendering signals. + /// + /// Only a `src_image` that is [`Self::input_image`] avoids the CPU + /// entirely. Any other image is first copied into the input image, and + /// that copy is waited for on the CPU before the encode is recorded, so + /// this call then blocks until the points in `wait` are reached and the + /// copy is done. Render or convert into [`Self::input_image`] to keep the + /// whole handover on the GPU. + pub fn encode_after( + &mut self, + src_image: vk::Image, + wait: &[TimelinePoint], + ) -> Result { + self.0.encode_after(src_image, wait) } /// Wait for all in-flight frames to finish encoding (end-of-stream barrier). diff --git a/src/encoder/pipeline.rs b/src/encoder/pipeline.rs index 9cb58b6e..d46ef585 100644 --- a/src/encoder/pipeline.rs +++ b/src/encoder/pipeline.rs @@ -47,6 +47,7 @@ use crate::encoder::resources::{ }; use crate::encoder::{BitDepth, EncodedPacket, FrameType, PixelFormat}; use crate::error::{PixelForgeError, Result}; +use crate::sync::TimelinePoint; use crate::video::{SlotSync, TimelineChain}; use crate::vulkan::VideoContext; @@ -357,7 +358,9 @@ impl EncodePipeline { pub(crate) fn submit_current( &mut self, device: &ash::Device, + sync2: &ash::khr::synchronization2::Device, encode_queue: vk::Queue, + external_waits: &[TimelinePoint], ) -> Result { let wait = self.timeline.wait(); let signal = self.timeline.pending_signal(); @@ -393,13 +396,26 @@ impl EncodePipeline { ) }; + // The previous encode, which shares the DPB, then whatever the caller + // asked this frame to wait for. + let waits: Vec = wait + .into_iter() + .map(|(semaphore, value)| { + vk::SemaphoreSubmitInfo::default() + .semaphore(semaphore) + .value(value) + .stage_mask(vk::PipelineStageFlags2::VIDEO_ENCODE_KHR) + }) + .chain(external_waits.iter().map(TimelinePoint::wait_info)) + .collect(); unsafe { submit_encode_only( device, + sync2, command_buffer, fence, encode_queue, - wait, + &waits, Some(signal), )?; } diff --git a/src/encoder/resources.rs b/src/encoder/resources.rs index 31278a60..ccf17d55 100644 --- a/src/encoder/resources.rs +++ b/src/encoder/resources.rs @@ -1,5 +1,6 @@ use crate::encoder::{BitDepth, PixelFormat}; use crate::error::{PixelForgeError, Result}; +use crate::sync::TimelinePoint; use crate::vulkan::VideoContext; use ash::vk::TaggedStructure; use ash::vk::{self, Handle}; @@ -390,7 +391,7 @@ pub(crate) fn clear_input_image(context: &VideoContext, params: &ClearImageParam } /// Parameters for uploading an image to the encoder's input image. -pub(crate) struct UploadParams { +pub(crate) struct UploadParams<'a> { /// The command buffer to use for the upload. pub upload_command_buffer: vk::CommandBuffer, /// The fence to use for synchronization. @@ -409,6 +410,8 @@ pub(crate) struct UploadParams { pub input_image_layout: vk::ImageLayout, /// The queue to submit transfer operations to. pub upload_queue: vk::Queue, + /// Caller work the copy must wait for. + pub waits: &'a [TimelinePoint], } /// Upload an image to the encoder's input image via GPU-to-GPU copy. @@ -425,7 +428,7 @@ pub(crate) struct UploadParams { /// Returns Ok(()) on success, or an error if any Vulkan operation fails. pub(crate) fn upload_image_to_input( context: &crate::vulkan::VideoContext, - params: &UploadParams, + params: &UploadParams<'_>, ) -> Result<()> { let device = context.device(); @@ -601,11 +604,20 @@ pub(crate) fn upload_image_to_input( unsafe { device.end_command_buffer(params.upload_command_buffer) } .map_err(|e| PixelForgeError::CommandBuffer(e.to_string()))?; - let submit_info = vk::SubmitInfo::default() - .command_buffers(std::slice::from_ref(¶ms.upload_command_buffer)); + let command_buffers = + [vk::CommandBufferSubmitInfo::default().command_buffer(params.upload_command_buffer)]; + let waits: Vec = + params.waits.iter().map(TimelinePoint::wait_info).collect(); + let submit_info = vk::SubmitInfo2::default() + .wait_semaphore_infos(&waits) + .command_buffer_infos(&command_buffers); - unsafe { device.queue_submit(params.upload_queue, &[submit_info], params.upload_fence) } - .map_err(|e| PixelForgeError::CommandBuffer(e.to_string()))?; + unsafe { + context + .sync2() + .queue_submit2(params.upload_queue, &[submit_info], params.upload_fence) + } + .map_err(|e| PixelForgeError::CommandBuffer(e.to_string()))?; unsafe { device.wait_for_fences(&[params.upload_fence], true, u64::MAX) } .map_err(|e| PixelForgeError::CommandBuffer(e.to_string()))?; @@ -626,7 +638,7 @@ pub(crate) fn upload_image_to_input( /// /// The command buffer must be in recording state. pub(crate) unsafe fn record_dpb_barriers( - device: &ash::Device, + sync2: &ash::khr::synchronization2::Device, command_buffer: vk::CommandBuffer, dpb_images: &[vk::Image], use_layered_dpb: bool, @@ -710,7 +722,7 @@ pub(crate) unsafe fn record_dpb_barriers( } let dependency = vk::DependencyInfo::default().image_memory_barriers(&all_barriers); - unsafe { device.cmd_pipeline_barrier2(command_buffer, &dependency) }; + unsafe { sync2.cmd_pipeline_barrier2(command_buffer, &dependency) }; } /// Prepare an encode command buffer for recording. @@ -810,24 +822,16 @@ pub(crate) unsafe fn record_post_encode_dpb_barrier( /// The command buffer must have been ended. pub(crate) unsafe fn submit_encode_only( device: &ash::Device, + sync2: &ash::khr::synchronization2::Device, command_buffer: vk::CommandBuffer, fence: vk::Fence, encode_queue: vk::Queue, - wait_timeline: Option<(vk::Semaphore, u64)>, + wait_infos: &[vk::SemaphoreSubmitInfo<'_>], signal_timeline: Option<(vk::Semaphore, u64)>, ) -> Result<()> { let command_buffer_info = vk::CommandBufferSubmitInfo::default().command_buffer(command_buffer); let command_buffer_infos = [command_buffer_info]; - let wait_infos: Vec = wait_timeline - .into_iter() - .map(|(semaphore, value)| { - vk::SemaphoreSubmitInfo::default() - .semaphore(semaphore) - .value(value) - .stage_mask(vk::PipelineStageFlags2::VIDEO_ENCODE_KHR) - }) - .collect(); let signal_infos: Vec = signal_timeline .into_iter() .map(|(semaphore, value)| { @@ -839,7 +843,7 @@ pub(crate) unsafe fn submit_encode_only( .collect(); let submit_info = vk::SubmitInfo2::default() - .wait_semaphore_infos(&wait_infos) + .wait_semaphore_infos(wait_infos) .command_buffer_infos(&command_buffer_infos) .signal_semaphore_infos(&signal_infos); @@ -850,7 +854,7 @@ pub(crate) unsafe fn submit_encode_only( } unsafe { - device + sync2 .queue_submit2(encode_queue, &[submit_info], fence) .map_err(|e| PixelForgeError::CommandBuffer(e.to_string()))?; } diff --git a/src/lib.rs b/src/lib.rs index c6598ee0..5eb9f1c0 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -9,6 +9,9 @@ //! Encoding hands back an [`EncodeFuture`]; decoding delivers frames through a //! [`DecodeSource`] as the GPU finishes with them. //! - **GPU color conversion**: RGB/BGR → YUV via Vulkan compute shaders (BT.709, BT.2020, sRGB→BT.2020+PQ, scRGB-linear→BT.2020+PQ). +//! - **Shared devices**: encode and decode on a Vulkan device your application +//! created, from Vulkan 1.1 up, on queues you choose, ordered against your own +//! work with timeline semaphores. //! - **HDR support**: 10-bit encoding (P010, YUV444P10), PQ transfer function, BT.2020 color space. //! - **GPU-native API**: Encode directly from Vulkan images (`vk::Image`). //! - **Flexible configuration**: Rate control (CBR, VBR, CQP), quality levels, GOP settings. @@ -244,6 +247,49 @@ //! # } //! ``` //! +//! ## Encoding on your own device +//! +//! An application that already renders with Vulkan can give pixelforge its own +//! device instead of letting it create a second one. Frames then never leave +//! the device: no external memory, no import, and no CPU wait between the +//! application's work and pixelforge's. +//! +//! [`VideoContextBuilder::encode_device_requirements`] lists the queue +//! families, extensions and features to create the device with, and +//! [`VideoContextBuilder::build_from_existing_encode`] adopts it. A `VkQueue` +//! may not be submitted to from two threads at once, so give pixelforge queues +//! the application does not use, with +//! [`VideoContextBuilder::with_encode_queue`] and friends. +//! +//! [`ColorConverter::convert_async`] and [`Encoder::encode_after`] then order +//! each frame on the GPU: the application's submission signals a +//! [`TimelinePoint`], the conversion waits for it and signals its own, and the +//! encode waits for that. This stays off the CPU as long as the encoder's own +//! [`Encoder::input_image`] is what gets encoded, as below: any other image is +//! copied into it first, and that copy is waited for on the CPU. +//! +//! ```rust,no_run +//! # use ash::vk; +//! # use pixelforge::{ColorConverter, Encoder, TimelinePoint}; +//! # fn frame( +//! # converter: &mut ColorConverter, +//! # encoder: &mut Encoder, +//! # rendered: vk::Semaphore, +//! # frame_number: u64, +//! # image: vk::Image, +//! # ) -> Result<(), Box> { +//! // `rendered` reaches `frame_number` once the application has drawn `image`. +//! let converted = converter.convert_async( +//! image, +//! vk::ImageLayout::GENERAL, +//! encoder.input_image(), +//! &[TimelinePoint::new(rendered, frame_number)], +//! )?; +//! let packet = encoder.encode_after(encoder.input_image(), &[converted])?; +//! # Ok(()) +//! # } +//! ``` +//! //! # Benchmarking //! //! Run the encode latency benchmark with: @@ -263,9 +309,6 @@ //! # Decode H.264 to raw YUV //! cargo run --example decode -- input.264 output.yuv //! -//! # Decode on a caller-created Vulkan device -//! cargo run --example decode_adopted -- input.264 output.yuv -//! //! # Encode, choosing the codec (h264, h265 or av1) //! cargo run --example encode -- h265 //! @@ -307,6 +350,7 @@ pub mod decoder; pub mod encoder; pub mod error; pub mod image; +pub mod sync; pub(crate) mod video; pub mod vulkan; @@ -332,6 +376,7 @@ pub use encoder::{ }; pub use error::PixelForgeError; pub use image::InputImage; +pub use sync::TimelinePoint; pub use vulkan::VideoContextBuilder; /// Re-export VideoContext for convenience. diff --git a/src/sync.rs b/src/sync.rs new file mode 100644 index 00000000..66ca372f --- /dev/null +++ b/src/sync.rs @@ -0,0 +1,45 @@ +//! Ordering pixelforge's GPU work against the caller's, without the CPU. + +use ash::vk; + +/// A point on a timeline semaphore: reached once the semaphore's value is at +/// least `value`. +/// +/// Returned by [`ColorConverter::convert_async`](crate::ColorConverter::convert_async) +/// for the work it submitted, and accepted by it and by +/// [`Encoder::encode_after`](crate::Encoder::encode_after) as work to wait +/// for. On a device shared with the caller this is how a frame moves from the +/// caller's rendering to the converter to the encoder with every step ordered +/// on the GPU: each submission waits on the point the previous one signals, +/// and nothing waits on the CPU. +/// +/// The semaphore must be a timeline semaphore on the same device as the +/// context. A point pixelforge hands out stays valid for as long as the +/// context does, so a submission of the caller's own that waits on one must +/// finish before the last clone of the context is dropped. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct TimelinePoint { + /// A timeline semaphore. + pub semaphore: vk::Semaphore, + /// The value to wait for. + pub value: u64, +} + +impl TimelinePoint { + /// The point where `semaphore` reaches `value`. + pub fn new(semaphore: vk::Semaphore, value: u64) -> Self { + Self { semaphore, value } + } + + /// A wait on this point that holds back every stage of the submission. + /// + /// Pixelforge's submissions start with layout transitions whose stages the + /// caller cannot know, so a narrower stage mask could let one of them run + /// before the caller's work is done. + pub(crate) fn wait_info(&self) -> vk::SemaphoreSubmitInfo<'static> { + vk::SemaphoreSubmitInfo::default() + .semaphore(self.semaphore) + .value(self.value) + .stage_mask(vk::PipelineStageFlags2::ALL_COMMANDS) + } +} diff --git a/src/vulkan.rs b/src/vulkan.rs index 8564a767..7fc3d936 100644 --- a/src/vulkan.rs +++ b/src/vulkan.rs @@ -7,6 +7,11 @@ use std::ffi::{CStr, CString}; use std::os::raw::c_char; use tracing::{debug, info, warn}; +mod encode_device; +mod queues; +use queues::QueueOverrides; +pub use queues::{DeviceQueue, QueueRoles}; + /// Route validation-layer messages into `tracing`. /// /// Without a messenger the validation layer has nowhere to report to and its @@ -60,6 +65,7 @@ pub struct VideoContextBuilder { /// enables unified image layouts when the device supports them, and one /// that adopts a caller's device assumes they were not enabled. unified_image_layouts: Option, + queues: QueueOverrides, } impl Default for VideoContextBuilder { @@ -78,6 +84,7 @@ impl VideoContextBuilder { required_encode_codecs: Vec::new(), required_decode_codecs: Vec::new(), unified_image_layouts: None, + queues: QueueOverrides::default(), } } @@ -111,6 +118,52 @@ impl VideoContextBuilder { self } + /// Submit video encode work to `queue` on an adopted device. + /// + /// See [`Self::with_transfer_queue`] for why this matters. Ignored by + /// [`Self::build`], which creates its own device and owns every queue on + /// it. + pub fn with_encode_queue(mut self, queue: DeviceQueue) -> Self { + self.queues.encode = Some(queue); + self + } + + /// Submit video decode work to `queue` on an adopted device. + /// + /// See [`Self::with_transfer_queue`] for why this matters. Ignored by + /// [`Self::build`], which creates its own device and owns every queue on + /// it. + pub fn with_decode_queue(mut self, queue: DeviceQueue) -> Self { + self.queues.decode = Some(queue); + self + } + + /// Submit uploads and readback copies to `queue` on an adopted device. + /// + /// Without this, an adopted context takes queue 0 of each family it needs, + /// and on most hardware queue 0 of the transfer and compute families is the + /// caller's own graphics queue. A `VkQueue` must not be submitted to from + /// two threads at once, so a caller that keeps submitting to its queue + /// while pixelforge works on another thread has to either create a spare + /// queue and give it here, or create its queue with + /// `VK_DEVICE_QUEUE_CREATE_INTERNALLY_SYNCHRONIZED_BIT_KHR` (see + /// [`DeviceRequirements::internally_synchronized_queues`]). + /// + /// The queue's family must support transfer, and `queue.index` must be + /// below the number of queues the caller created in that family. The + /// second part cannot be checked. Ignored by [`Self::build`]. + pub fn with_transfer_queue(mut self, queue: DeviceQueue) -> Self { + self.queues.transfer = Some(queue); + self + } + + /// Submit the colour converter's dispatches to `queue` on an adopted + /// device. See [`Self::with_transfer_queue`]. Ignored by [`Self::build`]. + pub fn with_compute_queue(mut self, queue: DeviceQueue) -> Self { + self.queues.compute = Some(queue); + self + } + /// Build the VideoContext. pub fn build(self) -> Result { VideoContext::new(self) @@ -138,10 +191,27 @@ impl VideoContextBuilder { if unified_image_layouts { extensions.push(ash::khr::unified_image_layouts::NAME); } + let queues = QueueRoles { + encode: None, + decode: Some(families.decode), + transfer: families.transfer, + compute: families.compute, + }; Ok(DeviceRequirements { - queue_families: families.unique(), + queue_families: queues.unique_families(), + queues, extensions, + features: DeviceFeatures { + synchronization2: true, + timeline_semaphore: true, + ..DeviceFeatures::default() + }, unified_image_layouts, + internally_synchronized_queues: supports_internally_synchronized_queues( + instance, + physical_device, + &DeviceExtensions::of(instance, physical_device), + ), }) } @@ -200,6 +270,7 @@ impl VideoContextBuilder { VideoContext::from_existing_decode( self.required_decode_codecs, self.unified_image_layouts.unwrap_or(false), + self.queues, entry, instance, physical_device, @@ -208,19 +279,34 @@ impl VideoContextBuilder { } } -/// Queue families and extensions a caller's device must provide to decode. +/// Queue families, extensions and features a caller's device must provide for +/// pixelforge to work on it. +/// +/// Returned by [`VideoContextBuilder::decode_device_requirements`] and +/// [`VideoContextBuilder::encode_device_requirements`]. Enable every extension +/// in [`Self::extensions`] and every feature set in [`Self::features`]: +/// pixelforge assumes all of them are on, since Vulkan cannot be asked +/// afterwards which were. /// -/// Returned by [`VideoContextBuilder::decode_device_requirements`]. Two device -/// features must be enabled as well, and are not listed here because they are -/// feature-struct fields rather than extensions: `synchronization2` and -/// `timelineSemaphore`. +/// The instance may ask for any Vulkan version from 1.1 up. Pixelforge reaches +/// everything newer through the extensions listed here, so an application that +/// created its instance for 1.1 or 1.2 does not have to raise it. #[derive(Debug, Clone)] +#[non_exhaustive] pub struct DeviceRequirements { /// Queue families pixelforge needs a queue created for. Merge these with /// your own (deduplicated) when building the device. pub queue_families: Vec, + /// The family behind each role in [`Self::queue_families`]. An adopted + /// context takes queue 0 of each unless told otherwise with + /// [`VideoContextBuilder::with_transfer_queue`] and friends, which is + /// where a caller that needs its own queue to itself points pixelforge at + /// a spare one. + pub queues: QueueRoles, /// Device extensions pixelforge needs enabled. Merge with your own. pub extensions: Vec<&'static std::ffi::CStr>, + /// Device features pixelforge needs enabled. Merge with your own. + pub features: DeviceFeatures, /// Whether this device can support unified image layouts for video, which /// is what lets decoded frames be sampled with no copy and no layout /// transition. When true, `VK_KHR_unified_image_layouts` is included in @@ -234,6 +320,42 @@ pub struct DeviceRequirements { /// into a private image, which is correct but costs a full-frame GPU copy /// per frame. pub unified_image_layouts: bool, + /// Whether this device supports `VK_KHR_internally_synchronized_queues`. + /// + /// A queue created with `VK_DEVICE_QUEUE_CREATE_INTERNALLY_SYNCHRONIZED_BIT_KHR` + /// (and the `internallySynchronizedQueues` feature enabled) may be + /// submitted to from several threads at once, the driver doing the + /// locking. That makes it safe to give pixelforge the same queue the + /// caller uses, which is the way out on hardware with too few queues to + /// spare one. Pixelforge needs to be told nothing: it is the caller's + /// queue that has to be created this way. The extension is not in + /// [`Self::extensions`], since only a caller sharing its queue needs it. + pub internally_synchronized_queues: bool, +} + +/// Device features, as booleans rather than a `pNext` chain. +/// +/// A caller merging these into its own device creation may already chain +/// `VkPhysicalDeviceVulkan12Features` or `VkPhysicalDeviceVulkan13Features`, +/// and Vulkan forbids chaining those alongside the per-feature structs they +/// contain. So pixelforge says which bits it needs and leaves it to the caller +/// to set them in whichever structs it uses. The field names match the Vulkan +/// feature names. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] +#[non_exhaustive] +pub struct DeviceFeatures { + /// `synchronization2`, from `VkPhysicalDeviceSynchronization2Features`. + pub synchronization2: bool, + /// `timelineSemaphore`, from `VkPhysicalDeviceTimelineSemaphoreFeatures`. + pub timeline_semaphore: bool, + /// `samplerYcbcrConversion`, from + /// `VkPhysicalDeviceSamplerYcbcrConversionFeatures`. + pub sampler_ycbcr_conversion: bool, + /// `ycbcr2plane444Formats`, from + /// `VkPhysicalDeviceYcbcr2Plane444FormatsFeaturesEXT`. + pub ycbcr_2plane_444_formats: bool, + /// `videoEncodeAV1`, from `VkPhysicalDeviceVideoEncodeAV1FeaturesKHR`. + pub video_encode_av1: bool, } /// The queue families pixelforge selects for decoding on a given device. @@ -243,19 +365,6 @@ struct DecodeQueueFamilies { compute: u32, } -impl DecodeQueueFamilies { - /// The distinct families, in a stable order. - fn unique(&self) -> Vec { - let mut out = vec![self.decode]; - for f in [self.transfer, self.compute] { - if !out.contains(&f) { - out.push(f); - } - } - out - } -} - /// The queue family for the colour conversion's compute work, if there is one. /// /// A family that does not also do graphics is preferred. A driver may back @@ -272,23 +381,34 @@ fn find_compute_queue_family(queue_families: &[vk::QueueFamilyProperties]) -> Op .map(|(idx, _)| idx as u32) } -/// Select the decode / transfer / compute queue families on `physical_device`, -/// failing if it cannot decode the required codecs. +/// The families [`scan_queue_families`] found. +struct ScannedFamilies { + /// The last family with the requested video flag, if any. + video: Option, + /// `timestampValidBits` of [`Self::video`]'s family. + video_timestamp_valid_bits: u32, + transfer: u32, + compute: u32, +} + +/// Find a family with the `video` flag, a transfer family and a compute family +/// on `physical_device`, the same way [`VideoContext::new`] does. /// -/// Mirrors the selection [`VideoContext::new`] does inline, but scoped to the -/// decode path: a video-decode family, a transfer family (preferring a -/// dedicated engine over one that also does video — see the scoring), and a -/// compute family (see [`find_compute_queue_family`]). -fn find_decode_queue_families( - entry: &ash::Entry, +/// The transfer family is picked by preference rather than last-wins: a family +/// that also does video is the worst choice, since it contends with the +/// encode or decode work itself, then the graphics family, which the caller +/// most likely renders on, then compute, and best of all a dedicated transfer +/// engine. The compute family is [`find_compute_queue_family`]'s. +fn scan_queue_families( instance: &ash::Instance, physical_device: vk::PhysicalDevice, - decode_codecs: &[Codec], -) -> Result { + video_flag: vk::QueueFlags, +) -> Result { let queue_families = unsafe { instance.get_physical_device_queue_family_properties(physical_device) }; - let mut decode = None; + let mut video = None; + let mut video_timestamp_valid_bits = 0; let mut transfer = u32::MAX; let mut transfer_score = -1i32; @@ -296,8 +416,9 @@ fn find_decode_queue_families( let idx = idx as u32; let flags = props.queue_flags; - if flags.contains(vk::QueueFlags::VIDEO_DECODE_KHR) { - decode = Some(idx); + if flags.contains(video_flag) { + video = Some(idx); + video_timestamp_valid_bits = props.timestamp_valid_bits; } if flags.contains(vk::QueueFlags::TRANSFER) { let is_video = flags @@ -318,11 +439,6 @@ fn find_decode_queue_families( } } - let decode = decode.ok_or_else(|| { - PixelForgeError::NoSuitableDevice( - "Physical device has no video decode queue family".to_string(), - ) - })?; if transfer == u32::MAX { return Err(PixelForgeError::NoSuitableDevice( "Physical device has no transfer queue family".to_string(), @@ -331,6 +447,39 @@ fn find_decode_queue_families( let compute = find_compute_queue_family(&queue_families).ok_or_else(|| { PixelForgeError::NoSuitableDevice("Physical device has no compute queue family".to_string()) })?; + Ok(ScannedFamilies { + video, + video_timestamp_valid_bits, + transfer, + compute, + }) +} + +/// Select the decode / transfer / compute queue families on `physical_device`, +/// failing if it cannot decode the required codecs. +/// +/// Mirrors the selection [`VideoContext::new`] does inline, but scoped to the +/// decode path: a video-decode family, a transfer family (preferring a +/// dedicated engine over one that also does video — see the scoring), and a +/// compute family (see [`find_compute_queue_family`]). +fn find_decode_queue_families( + entry: &ash::Entry, + instance: &ash::Instance, + physical_device: vk::PhysicalDevice, + decode_codecs: &[Codec], +) -> Result { + let ScannedFamilies { + video: decode, + transfer, + compute, + .. + } = scan_queue_families(instance, physical_device, vk::QueueFlags::VIDEO_DECODE_KHR)?; + + let decode = decode.ok_or_else(|| { + PixelForgeError::NoSuitableDevice( + "Physical device has no video decode queue family".to_string(), + ) + })?; // Confirm the device actually decodes the codecs asked for. let available = query_decode_codecs(entry, instance, physical_device); @@ -350,7 +499,28 @@ fn find_decode_queue_families( }) } -/// Device extensions required to decode the given codecs. +/// The extensions a physical device offers, for asking which it has. +/// +/// Enumerated once, so a caller checking several extensions does not ask the +/// driver for the list again each time. A device whose list cannot be read is +/// treated as offering nothing. +pub(super) struct DeviceExtensions(Vec); + +impl DeviceExtensions { + pub(super) fn of(instance: &ash::Instance, physical_device: vk::PhysicalDevice) -> Self { + Self( + unsafe { instance.enumerate_device_extension_properties(physical_device) } + .unwrap_or_default(), + ) + } + + pub(super) fn has(&self, name: &CStr) -> bool { + self.0 + .iter() + .any(|ext| unsafe { CStr::from_ptr(ext.extension_name.as_ptr()) } == name) + } +} + /// Whether `physical_device` can support unified image layouts *for video*. /// /// Both the extension and both of its feature bits are needed: @@ -360,15 +530,7 @@ fn supports_unified_image_layouts( instance: &ash::Instance, physical_device: vk::PhysicalDevice, ) -> bool { - let exts = match unsafe { instance.enumerate_device_extension_properties(physical_device) } { - Ok(exts) => exts, - Err(_) => return false, - }; - let present = exts.iter().any(|ext| { - let name = unsafe { std::ffi::CStr::from_ptr(ext.extension_name.as_ptr()) }; - name == ash::khr::unified_image_layouts::NAME - }); - if !present { + if !DeviceExtensions::of(instance, physical_device).has(ash::khr::unified_image_layouts::NAME) { return false; } let mut features = vk::PhysicalDeviceUnifiedImageLayoutsFeaturesKHR::default(); @@ -377,11 +539,33 @@ fn supports_unified_image_layouts( features.unified_image_layouts != 0 && features.unified_image_layouts_video != 0 } +/// Whether `physical_device` supports `VK_KHR_internally_synchronized_queues`, +/// extension and feature both. +fn supports_internally_synchronized_queues( + instance: &ash::Instance, + physical_device: vk::PhysicalDevice, + extensions: &DeviceExtensions, +) -> bool { + if !extensions.has(ash::khr::internally_synchronized_queues::NAME) { + return false; + } + let mut features = vk::PhysicalDeviceInternallySynchronizedQueuesFeaturesKHR::default(); + let mut query = vk::PhysicalDeviceFeatures2::default().push(&mut features); + unsafe { instance.get_physical_device_features2(physical_device, &mut query) }; + features.internally_synchronized_queues != 0 +} + +/// Device extensions required to decode the given codecs. fn decode_extension_names(decode_codecs: &[Codec]) -> Vec<&'static std::ffi::CStr> { + // Synchronization2 and timeline semaphores are core in 1.3 and 1.2, but + // are listed anyway: enabling a promoted extension is harmless, and on a + // device created from an instance asking for less it is what makes them + // available at all. let mut names = vec![ ash::khr::video_queue::NAME, ash::khr::video_decode_queue::NAME, ash::khr::synchronization2::NAME, + ash::khr::timeline_semaphore::NAME, ]; if decode_codecs.contains(&Codec::H264) { names.push(ash::khr::video_decode_h264::NAME); @@ -408,6 +592,14 @@ struct VideoContextInner { instance: ash::Instance, physical_device: vk::PhysicalDevice, device: ash::Device, + /// `vkQueueSubmit2` and `vkCmdPipelineBarrier2` through their KHR names. + /// + /// The core entry points exist only when the device was created from an + /// instance asking for Vulkan 1.3. An adopted device may come from an + /// application that asked for less, where the core pointers are null and + /// calling them crashes. The KHR names are valid on any device with + /// `VK_KHR_synchronization2` enabled, which pixelforge requires either way. + sync2: ash::khr::synchronization2::Device, video_encode_queue_family: Option, video_encode_timestamp_valid_bits: u32, video_encode_queue: Option, @@ -424,9 +616,13 @@ struct VideoContextInner { has_push_descriptor: bool, has_unified_image_layouts: bool, /// Whether this context created (and therefore must destroy) the device and - /// instance. A context adopted from a caller's device via - /// [`VideoContext::from_existing_decode`] borrows them and destroys neither. + /// instance. A context adopted from a caller's device borrows them and + /// destroys neither. owns_device: bool, + /// Semaphores whose owners are gone but which a submission may still be + /// waiting on, destroyed with the context. See + /// [`VideoContext::retire_semaphore`]. + retired_semaphores: std::sync::Mutex>, /// Validation message sink, present only when validation is enabled and the /// instance is ours. Destroyed before the instance it belongs to. debug_messenger: Option<(ash::ext::debug_utils::Instance, vk::DebugUtilsMessengerEXT)>, @@ -434,6 +630,15 @@ struct VideoContextInner { impl Drop for VideoContextInner { fn drop(&mut self) { + // Ours whoever owns the device, and gone before the device can be. + let retired = std::mem::take( + self.retired_semaphores + .get_mut() + .unwrap_or_else(|poisoned| poisoned.into_inner()), + ); + for semaphore in retired { + unsafe { self.device.destroy_semaphore(semaphore, None) }; + } // Only tear down the device/instance we created ourselves. An adopted // device is owned by the caller and outlives this context. if self.owns_device { @@ -468,6 +673,26 @@ impl VideoContext { &self.inner.device } + pub(crate) fn sync2(&self) -> &ash::khr::synchronization2::Device { + &self.inner.sync2 + } + + /// Hand over a semaphore to be destroyed with the context rather than now. + /// + /// For a semaphore whose owner is being dropped while a submission from + /// elsewhere may still wait on it: the colour converter's timeline, which + /// an encode submitted with [`crate::Encoder::encode_after`] can still be + /// waiting on when the converter goes. Every encoder holds the context and + /// waits for its queues when dropped, so by the time the context goes, + /// nothing of pixelforge's waits on these any more. + pub(crate) fn retire_semaphore(&self, semaphore: vk::Semaphore) { + self.inner + .retired_semaphores + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()) + .push(semaphore); + } + pub(crate) fn video_encode_queue_family(&self) -> Option { self.inner.video_encode_queue_family } @@ -1110,6 +1335,7 @@ impl VideoContext { Ok(Self { inner: std::sync::Arc::new(VideoContextInner { entry, + sync2: ash::khr::synchronization2::Device::load(&instance, &device), instance, physical_device, device, @@ -1129,6 +1355,7 @@ impl VideoContext { has_push_descriptor, has_unified_image_layouts: has_unified_layouts, owns_device: true, + retired_semaphores: Default::default(), debug_messenger, }), }) @@ -1142,6 +1369,7 @@ impl VideoContext { fn from_existing_decode( required_decode_codecs: Vec, declared_unified_image_layouts: bool, + queue_overrides: QueueOverrides, entry: ash::Entry, instance: ash::Instance, physical_device: vk::PhysicalDevice, @@ -1158,10 +1386,34 @@ impl VideoContext { let memory_properties = unsafe { instance.get_physical_device_memory_properties(physical_device) }; - // The caller created a queue for each family (from device_requirements). - let video_decode_queue = unsafe { device.get_device_queue(families.decode, 0) }; - let transfer_queue = unsafe { device.get_device_queue(families.transfer, 0) }; - let compute_queue = unsafe { device.get_device_queue(families.compute, 0) }; + let decode = queues::resolve( + &instance, + physical_device, + "decode", + queue_overrides.decode, + families.decode, + vk::QueueFlags::VIDEO_DECODE_KHR, + )?; + let transfer = queues::resolve( + &instance, + physical_device, + "transfer", + queue_overrides.transfer, + families.transfer, + vk::QueueFlags::TRANSFER, + )?; + let compute = queues::resolve( + &instance, + physical_device, + "compute", + queue_overrides.compute, + families.compute, + vk::QueueFlags::COMPUTE, + )?; + + let video_decode_queue = unsafe { device.get_device_queue(decode.family, decode.index) }; + let transfer_queue = unsafe { device.get_device_queue(transfer.family, transfer.index) }; + let compute_queue = unsafe { device.get_device_queue(compute.family, compute.index) }; let supported_decode_codecs = query_decode_codecs(&entry, &instance, physical_device); @@ -1180,24 +1432,25 @@ impl VideoContext { } info!( - "Adopted caller device for decode: decode family {}, transfer family {}, compute family {}", - families.decode, families.transfer, families.compute + "Adopted caller device for decode: decode queue {:?}, transfer queue {:?}, compute queue {:?}", + decode, transfer, compute ); Ok(VideoContext { inner: std::sync::Arc::new(VideoContextInner { entry, + sync2: ash::khr::synchronization2::Device::load(&instance, &device), instance, physical_device, device, video_encode_queue_family: None, video_encode_timestamp_valid_bits: 0u32, video_encode_queue: None, - video_decode_queue_family: Some(families.decode), + video_decode_queue_family: Some(decode.family), video_decode_queue: Some(video_decode_queue), - transfer_queue_family: families.transfer, + transfer_queue_family: transfer.family, transfer_queue, - compute_queue_family: families.compute, + compute_queue_family: compute.family, compute_queue, memory_properties, device_properties, @@ -1206,6 +1459,7 @@ impl VideoContext { has_push_descriptor: false, has_unified_image_layouts: declared_unified_image_layouts, owns_device: false, + retired_semaphores: Default::default(), // The caller owns the instance; reporting is theirs to set up. debug_messenger: None, }), diff --git a/src/vulkan/encode_device.rs b/src/vulkan/encode_device.rs new file mode 100644 index 00000000..3534820d --- /dev/null +++ b/src/vulkan/encode_device.rs @@ -0,0 +1,351 @@ +//! Encoding on a device the caller created. +//! +//! The encode counterpart of [`VideoContextBuilder::build_from_existing_decode`]. +//! An application that already renders on its own device, and wants to encode +//! what it renders, can hand that device to pixelforge. The encoder and the +//! colour converter then work on the application's images directly: no +//! external memory, no export and import, and a semaphore instead of a CPU +//! wait where one side hands a frame to the other. + +use super::queues::{self, QueueOverrides}; +use super::{ + DeviceExtensions, DeviceFeatures, DeviceQueue, DeviceRequirements, QueueRoles, VideoContext, + VideoContextBuilder, VideoContextInner, scan_queue_families, + supports_internally_synchronized_queues, +}; +use crate::encoder::Codec; +use crate::error::{PixelForgeError, Result}; +use ash::vk; +use ash::vk::TaggedStructure; +use tracing::info; + +/// The queue families pixelforge selects for encoding on a given device. +struct EncodeQueueFamilies { + /// The encode queue: the caller's choice, or queue 0 of the family the + /// scan found. Settled before anything else, since codec support and + /// timestamp support both belong to a family. + encode: DeviceQueue, + encode_timestamp_valid_bits: u32, + transfer: u32, + compute: u32, +} + +/// What pixelforge will use on `physical_device` to encode `required`. +/// +/// When `required` is empty every codec the device encodes is included, so a +/// caller that has not decided yet gets a device that can do any of them. +struct EncodePlan { + families: EncodeQueueFamilies, + codecs: Vec, + push_descriptor: bool, + ycbcr_2plane_444: bool, + sampler_ycbcr_conversion: bool, + internally_synchronized_queues: bool, +} + +impl EncodePlan { + /// `encode_queue` is the caller's choice of encode queue, if it made one. + /// Codecs are checked on that queue's family, since that is the one the + /// encoder will submit to. + fn new( + entry: &ash::Entry, + instance: &ash::Instance, + physical_device: vk::PhysicalDevice, + required: &[Codec], + encode_queue: Option, + ) -> Result { + let scanned = + scan_queue_families(instance, physical_device, vk::QueueFlags::VIDEO_ENCODE_KHR)?; + let scanned_encode = scanned.video.ok_or_else(|| { + PixelForgeError::NoSuitableDevice( + "Physical device has no video encode queue family".to_string(), + ) + })?; + let encode = queues::resolve( + instance, + physical_device, + "encode", + encode_queue, + scanned_encode, + vk::QueueFlags::VIDEO_ENCODE_KHR, + )?; + let encode_timestamp_valid_bits = if encode.family == scanned_encode { + scanned.video_timestamp_valid_bits + } else { + let props = + unsafe { instance.get_physical_device_queue_family_properties(physical_device) }; + props[encode.family as usize].timestamp_valid_bits + }; + + let extensions = DeviceExtensions::of(instance, physical_device); + let has = |name: &std::ffi::CStr| extensions.has(name); + + // The feature bits behind two of the extensions. A driver may offer an + // extension and still report its feature off, and a caller told to + // enable a feature the device lacks gets VK_ERROR_FEATURE_NOT_PRESENT + // from vkCreateDevice. A struct is chained only when its extension is + // offered, as Vulkan requires. + let mut ycbcr = vk::PhysicalDeviceSamplerYcbcrConversionFeatures::default(); + let mut ycbcr_444 = vk::PhysicalDeviceYcbcr2Plane444FormatsFeaturesEXT::default(); + let mut av1 = vk::PhysicalDeviceVideoEncodeAV1FeaturesKHR::default(); + let mut query = vk::PhysicalDeviceFeatures2::default().push(&mut ycbcr); + if has(ash::ext::ycbcr_2plane_444_formats::NAME) { + query = query.push(&mut ycbcr_444); + } + if has(ash::khr::video_encode_av1::NAME) { + query = query.push(&mut av1); + } + unsafe { instance.get_physical_device_features2(physical_device, &mut query) }; + + let family = encode.family; + let mut available = Vec::new(); + if has(ash::khr::video_encode_h264::NAME) + && VideoContext::check_h264_encode_support(entry, instance, physical_device, family) + { + available.push(Codec::H264); + } + if has(ash::khr::video_encode_h265::NAME) + && VideoContext::check_h265_encode_support(entry, instance, physical_device, family) + { + available.push(Codec::H265); + } + if has(ash::khr::video_encode_av1::NAME) + && av1.video_encode_av1 != 0 + && VideoContext::check_av1_encode_support(entry, instance, physical_device, family) + { + available.push(Codec::AV1); + } + let codecs = if required.is_empty() { + available + } else { + for codec in required { + if !available.contains(codec) { + return Err(PixelForgeError::CodecNotSupported(format!( + "Queue family {} does not support encoding {:?}", + family, codec + ))); + } + } + required.to_vec() + }; + if codecs.is_empty() { + return Err(PixelForgeError::NoSuitableDevice( + "Physical device encodes none of H.264, H.265 or AV1".to_string(), + )); + } + + Ok(Self { + families: EncodeQueueFamilies { + encode, + encode_timestamp_valid_bits, + transfer: scanned.transfer, + compute: scanned.compute, + }, + codecs, + push_descriptor: has(ash::khr::push_descriptor::NAME), + ycbcr_2plane_444: has(ash::ext::ycbcr_2plane_444_formats::NAME) + && ycbcr_444.ycbcr2plane444_formats != 0, + sampler_ycbcr_conversion: ycbcr.sampler_ycbcr_conversion != 0, + internally_synchronized_queues: supports_internally_synchronized_queues( + instance, + physical_device, + &extensions, + ), + }) + } + + fn extensions(&self) -> Vec<&'static std::ffi::CStr> { + // Synchronization2 and timeline semaphores are core in 1.3 and 1.2, and + // listed anyway so a device created from an instance asking for less + // still has them. + let mut names = vec![ + ash::khr::video_queue::NAME, + ash::khr::video_encode_queue::NAME, + ash::khr::synchronization2::NAME, + ash::khr::timeline_semaphore::NAME, + ]; + for codec in &self.codecs { + names.push(match codec { + Codec::H264 => ash::khr::video_encode_h264::NAME, + Codec::H265 => ash::khr::video_encode_h265::NAME, + Codec::AV1 => ash::khr::video_encode_av1::NAME, + }); + } + if self.push_descriptor { + names.push(ash::khr::push_descriptor::NAME); + } + if self.ycbcr_2plane_444 { + names.push(ash::ext::ycbcr_2plane_444_formats::NAME); + } + names + } + + fn features(&self) -> DeviceFeatures { + DeviceFeatures { + synchronization2: true, + timeline_semaphore: true, + sampler_ycbcr_conversion: self.sampler_ycbcr_conversion, + ycbcr_2plane_444_formats: self.ycbcr_2plane_444, + video_encode_av1: self.codecs.contains(&Codec::AV1), + } + } + + fn roles(&self) -> QueueRoles { + QueueRoles { + encode: Some(self.families.encode.family), + decode: None, + transfer: self.families.transfer, + compute: self.families.compute, + } + } +} + +impl VideoContextBuilder { + /// What a caller-created device must provide for pixelforge to encode on + /// it, and to run the colour converter. + /// + /// Use this when you already have your own Vulkan device (a renderer's, or + /// one a game created) and want to encode images that live on it, without + /// exporting them to a second device. The codecs asked for with + /// [`Self::require_encode`] must all be supported; with none asked for, + /// every codec the device encodes is included. + /// + /// The colour converter needs `VK_KHR_push_descriptor`, which is included + /// whenever the device has it. On a device without it, encoding from YUV + /// input still works and only the converter is unavailable. + pub fn encode_device_requirements( + &self, + entry: &ash::Entry, + instance: &ash::Instance, + physical_device: vk::PhysicalDevice, + ) -> Result { + let plan = EncodePlan::new( + entry, + instance, + physical_device, + &self.required_encode_codecs, + self.queues.encode, + )?; + let queues = plan.roles(); + Ok(DeviceRequirements { + queue_families: queues.unique_families(), + queues, + extensions: plan.extensions(), + features: plan.features(), + unified_image_layouts: false, + internally_synchronized_queues: plan.internally_synchronized_queues, + }) + } + + /// Adopt a caller-created device for encoding, rather than creating one. + /// + /// The device must have been created with the queue families, extensions + /// and features reported by [`Self::encode_device_requirements`], called + /// with the same codecs for the same `physical_device`. The instance may + /// ask for any Vulkan version from 1.1 up. + /// + /// The resulting context **borrows** `instance` and `device`: dropping it + /// frees neither, so the caller must keep both alive for at least as long + /// as the context and every encoder and converter made from it. + /// + /// Pixelforge submits to one queue per role, queue 0 of each family unless + /// told otherwise. A caller that keeps submitting to its own queues while + /// pixelforge works on another thread should give it spare ones with + /// [`Self::with_encode_queue`], [`Self::with_transfer_queue`] and + /// [`Self::with_compute_queue`]. + pub fn build_from_existing_encode( + self, + entry: ash::Entry, + instance: ash::Instance, + physical_device: vk::PhysicalDevice, + device: ash::Device, + ) -> Result { + VideoContext::from_existing_encode( + &self.required_encode_codecs, + self.queues, + entry, + instance, + physical_device, + device, + ) + } +} + +impl VideoContext { + fn from_existing_encode( + required_encode_codecs: &[Codec], + queue_overrides: QueueOverrides, + entry: ash::Entry, + instance: ash::Instance, + physical_device: vk::PhysicalDevice, + device: ash::Device, + ) -> Result { + let plan = EncodePlan::new( + &entry, + &instance, + physical_device, + required_encode_codecs, + queue_overrides.encode, + )?; + let families = &plan.families; + let encode = families.encode; + let transfer = queues::resolve( + &instance, + physical_device, + "transfer", + queue_overrides.transfer, + families.transfer, + vk::QueueFlags::TRANSFER, + )?; + let compute = queues::resolve( + &instance, + physical_device, + "compute", + queue_overrides.compute, + families.compute, + vk::QueueFlags::COMPUTE, + )?; + + let device_properties = unsafe { instance.get_physical_device_properties(physical_device) }; + let memory_properties = + unsafe { instance.get_physical_device_memory_properties(physical_device) }; + + let video_encode_queue = unsafe { device.get_device_queue(encode.family, encode.index) }; + let transfer_queue = unsafe { device.get_device_queue(transfer.family, transfer.index) }; + let compute_queue = unsafe { device.get_device_queue(compute.family, compute.index) }; + + info!( + "Adopted caller device for encode: {:?}, encode queue {:?}, transfer queue {:?}, compute queue {:?}", + plan.codecs, encode, transfer, compute + ); + + Ok(VideoContext { + inner: std::sync::Arc::new(VideoContextInner { + sync2: ash::khr::synchronization2::Device::load(&instance, &device), + entry, + instance, + physical_device, + device, + video_encode_queue_family: Some(encode.family), + video_encode_timestamp_valid_bits: families.encode_timestamp_valid_bits, + video_encode_queue: Some(video_encode_queue), + video_decode_queue_family: None, + video_decode_queue: None, + transfer_queue_family: transfer.family, + transfer_queue, + compute_queue_family: compute.family, + compute_queue, + memory_properties, + device_properties, + supported_encode_codecs: plan.codecs.clone(), + supported_decode_codecs: Vec::new(), + has_push_descriptor: plan.push_descriptor, + has_unified_image_layouts: false, + owns_device: false, + retired_semaphores: Default::default(), + // The caller owns the instance; reporting is theirs to set up. + debug_messenger: None, + }), + }) + } +} diff --git a/src/vulkan/queues.rs b/src/vulkan/queues.rs new file mode 100644 index 00000000..d4078639 --- /dev/null +++ b/src/vulkan/queues.rs @@ -0,0 +1,166 @@ +//! Which queues pixelforge submits to on a device it did not create. +//! +//! A `VkQueue` must be externally synchronized: two threads may not submit to +//! it at once. A context that creates its own device owns every queue on it, +//! so the question never comes up. A context adopted from a caller's device +//! shares that device with the caller, and by default takes queue 0 of each +//! family it needs, which is very often the queue the caller renders and +//! presents on. Submitting there from another thread is a data race. +//! +//! [`DeviceQueue`] lets the caller say exactly which queue each role uses, so it +//! can create spare queues for pixelforge and keep its own to itself. + +use crate::error::{PixelForgeError, Result}; +use ash::vk; + +/// One queue on a device: a queue family and an index within it, as passed to +/// `vkGetDeviceQueue`. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] +pub struct DeviceQueue { + /// Queue family index. + pub family: u32, + /// Index of the queue within its family. The caller must have created at + /// least `index + 1` queues in `family`. + pub index: u32, +} + +impl DeviceQueue { + /// Queue `index` of `family`. + pub fn new(family: u32, index: u32) -> Self { + Self { family, index } + } +} + +/// The queue family pixelforge would pick for each role it may use. +/// +/// Returned as part of [`DeviceRequirements`](super::DeviceRequirements). A +/// role that the requested work does not need is `None`. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +#[non_exhaustive] +pub struct QueueRoles { + /// Video encode submissions. + pub encode: Option, + /// Video decode submissions. + pub decode: Option, + /// Uploads and readback copies. + pub transfer: u32, + /// The colour converter's compute dispatches. + pub compute: u32, +} + +impl QueueRoles { + /// The distinct families, in a stable order. + pub fn unique_families(&self) -> Vec { + let mut out = Vec::new(); + for f in [ + self.encode, + self.decode, + Some(self.transfer), + Some(self.compute), + ] + .into_iter() + .flatten() + { + if !out.contains(&f) { + out.push(f); + } + } + out + } +} + +/// Queues the caller chose for an adopted context, one per role. `None` means +/// queue 0 of the family pixelforge selects. +#[derive(Debug, Clone, Copy, Default)] +pub(crate) struct QueueOverrides { + pub encode: Option, + pub decode: Option, + pub transfer: Option, + pub compute: Option, +} + +/// Resolve the queue for one role: the caller's choice if there is one, +/// checked against what the role needs, otherwise queue 0 of `default_family`. +/// +/// The index cannot be checked against how many queues the caller actually +/// created, since Vulkan does not report that; it is checked against how many +/// the family offers, which catches the mistakes that can be caught. +pub(crate) fn resolve( + instance: &ash::Instance, + physical_device: vk::PhysicalDevice, + role: &str, + chosen: Option, + default_family: u32, + needs: vk::QueueFlags, +) -> Result { + let Some(queue) = chosen else { + return Ok(DeviceQueue::new(default_family, 0)); + }; + let families = unsafe { instance.get_physical_device_queue_family_properties(physical_device) }; + let Some(props) = families.get(queue.family as usize) else { + return Err(PixelForgeError::InvalidInput(format!( + "{role} queue: family {} does not exist (the device has {})", + queue.family, + families.len() + ))); + }; + if !family_can(props.queue_flags, needs) { + return Err(PixelForgeError::InvalidInput(format!( + "{role} queue: family {} has {:?}, which does not include {:?}", + queue.family, props.queue_flags, needs + ))); + } + if queue.index >= props.queue_count { + return Err(PixelForgeError::InvalidInput(format!( + "{role} queue: index {} is out of range, family {} has {} queue(s)", + queue.index, queue.family, props.queue_count + ))); + } + Ok(queue) +} + +/// Whether a family with `flags` can do `needs`. Graphics and compute families +/// support transfer operations whether or not they advertise it. +fn family_can(flags: vk::QueueFlags, needs: vk::QueueFlags) -> bool { + let mut effective = flags; + if flags.intersects(vk::QueueFlags::GRAPHICS | vk::QueueFlags::COMPUTE) { + effective |= vk::QueueFlags::TRANSFER; + } + effective.contains(needs) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn graphics_and_compute_imply_transfer() { + assert!(family_can( + vk::QueueFlags::GRAPHICS, + vk::QueueFlags::TRANSFER + )); + assert!(family_can( + vk::QueueFlags::COMPUTE, + vk::QueueFlags::TRANSFER + )); + assert!(!family_can( + vk::QueueFlags::VIDEO_ENCODE_KHR, + vk::QueueFlags::TRANSFER + )); + assert!(!family_can( + vk::QueueFlags::TRANSFER, + vk::QueueFlags::COMPUTE + )); + } + + #[test] + fn unique_families_keeps_first_occurrence_order() { + let roles = QueueRoles { + encode: Some(4), + decode: None, + transfer: 1, + compute: 4, + }; + assert_eq!(roles.unique_families(), vec![4, 1]); + } +} diff --git a/tests/adopted.rs b/tests/adopted.rs new file mode 100644 index 00000000..84937065 --- /dev/null +++ b/tests/adopted.rs @@ -0,0 +1,430 @@ +//! Pixelforge on a device the caller created, from an instance that asked for +//! Vulkan 1.1. +//! +//! An application that shares its device with pixelforge chose that device's +//! API version for its own reasons, and plenty choose less than 1.3. On such a +//! device the core 1.2 and 1.3 entry points are null, so anything pixelforge +//! reaches that way crashes. These tests build exactly that device, hand +//! pixelforge a spare queue where the hardware has one, and check the output +//! against a context pixelforge created itself on the same GPU. + +use ash::vk; +use ash::vk::TaggedStructure; +#[allow(dead_code)] +mod common; +use common::source::create_src_image; +use common::{Readback, decode_stream}; + +use pixelforge::decoder::{DecodeConfig, Decoder}; +use pixelforge::encoder::Codec; +use pixelforge::vulkan::{DeviceQueue, DeviceRequirements, VideoContext, VideoContextBuilder}; +use pixelforge::{ + ColorConverter, ColorConverterConfig, ColorRange, ColorSpec, EncodeConfig, Encoder, + InputFormat, OutputFormat, RateControlMode, TimelinePoint, +}; + +/// An instance and device the test owns, destroyed on drop after everything +/// built on them. +struct AppDevice { + entry: ash::Entry, + instance: ash::Instance, + physical_device: vk::PhysicalDevice, + device: ash::Device, +} + +impl Drop for AppDevice { + fn drop(&mut self) { + unsafe { + self.device.destroy_device(None); + self.instance.destroy_instance(None); + } + } +} + +fn validation_requested() -> bool { + std::env::var("PIXELFORGE_VALIDATION").is_ok() +} + +/// Create a Vulkan 1.1 instance, with validation when asked for. +fn instance_1_1() -> Result<(ash::Entry, ash::Instance), Box> { + let entry = unsafe { ash::Entry::load()? }; + let app_info = vk::ApplicationInfo::default() + .application_name(c"pixelforge-adopted-test") + .api_version(vk::API_VERSION_1_1); + let validation = c"VK_LAYER_KHRONOS_validation"; + let has_validation = unsafe { entry.enumerate_instance_layer_properties()? } + .iter() + .any(|l| unsafe { std::ffi::CStr::from_ptr(l.layer_name.as_ptr()) } == validation); + let layers = if validation_requested() && has_validation { + vec![validation.as_ptr()] + } else { + Vec::new() + }; + let info = vk::InstanceCreateInfo::default() + .application_info(&app_info) + .enabled_layer_names(&layers); + let instance = unsafe { entry.create_instance(&info, None)? }; + Ok((entry, instance)) +} + +/// Create the application's device from `reqs`, keeping queue 0 of every +/// family for itself wherever the family has a second queue. +fn create_app_device( + entry: ash::Entry, + instance: ash::Instance, + physical_device: vk::PhysicalDevice, + reqs: &DeviceRequirements, +) -> Result> { + let family_props = + unsafe { instance.get_physical_device_queue_family_properties(physical_device) }; + let priorities = [1.0f32; 2]; + let queue_infos: Vec = reqs + .queue_families + .iter() + .map(|&f| { + let count = family_props[f as usize].queue_count.min(2) as usize; + vk::DeviceQueueCreateInfo::default() + .queue_family_index(f) + .queue_priorities(&priorities[..count]) + }) + .collect(); + let ext_ptrs: Vec<_> = reqs.extensions.iter().map(|e| e.as_ptr()).collect(); + + // The per-feature structs, since this device is not 1.3: the promoted + // struct types are the same types under their extension names. + let features = reqs.features; + let mut sync2 = vk::PhysicalDeviceSynchronization2Features::default() + .synchronization2(features.synchronization2); + let mut timeline = vk::PhysicalDeviceTimelineSemaphoreFeatures::default() + .timeline_semaphore(features.timeline_semaphore); + let mut ycbcr = vk::PhysicalDeviceSamplerYcbcrConversionFeatures::default() + .sampler_ycbcr_conversion(features.sampler_ycbcr_conversion); + let mut ycbcr_444 = + vk::PhysicalDeviceYcbcr2Plane444FormatsFeaturesEXT::default().ycbcr2plane444_formats(true); + let mut av1 = vk::PhysicalDeviceVideoEncodeAV1FeaturesKHR::default().video_encode_av1(true); + let mut unified = vk::PhysicalDeviceUnifiedImageLayoutsFeaturesKHR::default() + .unified_image_layouts(true) + .unified_image_layouts_video(true); + let mut info = vk::DeviceCreateInfo::default() + .queue_create_infos(&queue_infos) + .enabled_extension_names(&ext_ptrs) + .push(&mut sync2) + .push(&mut timeline) + .push(&mut ycbcr); + if features.ycbcr_2plane_444_formats { + info = info.push(&mut ycbcr_444); + } + if features.video_encode_av1 { + info = info.push(&mut av1); + } + if reqs.unified_image_layouts { + info = info.push(&mut unified); + } + let device = unsafe { instance.create_device(physical_device, &info, None)? }; + Ok(AppDevice { + entry, + instance, + physical_device, + device, + }) +} + +/// The last queue this test created in `family`: a spare where there are two. +fn spare(app: &AppDevice, family: u32) -> DeviceQueue { + let props = unsafe { + app.instance + .get_physical_device_queue_family_properties(app.physical_device) + }; + DeviceQueue::new(family, props[family as usize].queue_count.min(2) - 1) +} + +/// Decode `stream` on `context` and read every frame back. +fn decode_all( + context: &VideoContext, + stream: &[u8], +) -> Result>, Box> { + let mut readback = Readback::new(context)?; + let mut decoder = Decoder::new(context.clone(), DecodeConfig::h264().with_byte_stream())?; + let mut frames = Vec::new(); + decode_stream(&mut decoder, stream, |frame| { + let data = readback.read(&frame)?; + frames.push([data.y, data.uv].concat()); + Ok(()) + })?; + drop(decoder); + drop(readback); + Ok(frames) +} + +/// The GPU a context pixelforge created for itself would pick, found on +/// `instance` by name so an adopted context can be compared against it. +fn same_gpu(instance: &ash::Instance, own: &VideoContext) -> vk::PhysicalDevice { + let name = own.device_properties().device_name; + unsafe { instance.enumerate_physical_devices() } + .expect("physical devices") + .into_iter() + .find(|&pd| unsafe { instance.get_physical_device_properties(pd) }.device_name == name) + .expect("the GPU the own context used") +} + +#[test] +#[ignore = "requires a Vulkan Video device"] +fn decode_on_a_vulkan_1_1_device() -> Result<(), Box> { + common::init_logging(); + let stream = std::fs::read("tests/data/bframes.264")?; + + let own = VideoContextBuilder::new() + .enable_validation(validation_requested()) + .require_decode(Codec::H264) + .build()?; + let expected = decode_all(&own, &stream)?; + + let (entry, instance) = instance_1_1()?; + let builder = VideoContextBuilder::new().require_decode(Codec::H264); + let physical_device = same_gpu(&instance, &own); + drop(own); + let reqs = builder.decode_device_requirements(&entry, &instance, physical_device)?; + let app = create_app_device(entry, instance, physical_device, &reqs)?; + + let mut builder = builder + .with_decode_queue(spare(&app, reqs.queues.decode.unwrap())) + .with_transfer_queue(spare(&app, reqs.queues.transfer)) + .with_compute_queue(spare(&app, reqs.queues.compute)); + if reqs.unified_image_layouts { + builder = builder.declare_unified_image_layouts(); + } + let context = builder.build_from_existing_decode( + app.entry.clone(), + app.instance.clone(), + app.physical_device, + app.device.clone(), + )?; + let actual = decode_all(&context, &stream)?; + drop(context); + + assert_eq!(actual.len(), expected.len(), "frame count"); + for (i, (a, e)) in actual.iter().zip(&expected).enumerate() { + assert!(a == e, "frame {i} differs from the own-context decode"); + } + Ok(()) +} + +const ENCODE_WIDTH: u32 = 320; +const ENCODE_HEIGHT: u32 = 240; +const ENCODE_FRAMES: u32 = 8; + +/// A BGRA frame that moves with `n`, so the encoder has motion to code. +fn moving_frame(n: u32) -> Vec { + let mut data = Vec::with_capacity((ENCODE_WIDTH * ENCODE_HEIGHT * 4) as usize); + for y in 0..ENCODE_HEIGHT { + for x in 0..ENCODE_WIDTH { + let (sx, sy) = (x + n * 3, y + n); + let r = (sx ^ sy) as u8; + let g = sx.wrapping_mul(2).wrapping_add(sy) as u8; + let b = (sy * 3) as u8; + data.extend_from_slice(&[b, g, r, 255]); + } + } + data +} + +/// How [`encode_all`] hands each frame from the converter to the encoder. +#[derive(Clone, Copy, PartialEq)] +enum Handoff { + /// `convert`, which waits on the CPU, then `encode`. + Cpu, + /// `convert_async` waiting on a point the caller signals, then + /// `encode_after` waiting on the conversion: ordered on the GPU, with + /// nothing waited for on the CPU until the packets are collected. + Gpu, +} + +/// Convert and encode `ENCODE_FRAMES` frames as H.264 on `context`, the way a +/// capture pipeline would: an RGB image goes through the colour converter +/// straight into the encoder's input. +fn encode_all( + context: &VideoContext, + handoff: Handoff, +) -> Result, Box> { + let device = context.device(); + let mut encoder = Encoder::new( + context.clone(), + EncodeConfig::h264(ENCODE_WIDTH, ENCODE_HEIGHT) + .with_rate_control(RateControlMode::Cqp) + .with_quality_level(22) + .with_frame_rate(30, 1) + .with_b_frames(0), + )?; + let mut converter = ColorConverter::new( + context.clone(), + ColorConverterConfig::new( + ENCODE_WIDTH, + ENCODE_HEIGHT, + InputFormat::BGRA, + OutputFormat::NV12, + ColorSpec::Srgb, + ColorSpec::Srgb, + ColorRange::Limited, + ), + )?; + + // Stands in for the caller's own rendering: each frame's conversion waits + // for a value the "renderer" signals once the frame is ready. The KHR + // entry points, since the adopted device is Vulkan 1.1. + let timeline_fns = ash::khr::timeline_semaphore::Device::load(context.instance(), device); + let mut type_info = vk::SemaphoreTypeCreateInfo::default() + .semaphore_type(vk::SemaphoreType::TIMELINE) + .initial_value(0); + let rendered = unsafe { + device.create_semaphore( + &vk::SemaphoreCreateInfo::default().push(&mut type_info), + None, + )? + }; + + // Signalled after the converter is dropped, see the last encode below. + let after_drop = u64::from(ENCODE_FRAMES) + 1; + + // Every source stays alive until the end: with a GPU handoff nothing says + // a conversion has finished reading one until its packet is back. + let mut sources = Vec::new(); + let mut pending = Vec::new(); + for n in 0..ENCODE_FRAMES { + let src = + unsafe { create_src_image(context, ENCODE_WIDTH, ENCODE_HEIGHT, &moving_frame(n))? }; + let target = encoder.input_image(); + let future = match handoff { + Handoff::Cpu => { + converter.convert(src.image, vk::ImageLayout::GENERAL, target)?; + encoder.encode(target)? + } + Handoff::Gpu => { + let ready = TimelinePoint::new(rendered, u64::from(n) + 1); + let converted = converter.convert_async( + src.image, + vk::ImageLayout::GENERAL, + target, + &[ready], + )?; + // Signalled only after the conversion was submitted, so it + // really has to wait. + unsafe { + timeline_fns.signal_semaphore( + &vk::SemaphoreSignalInfo::default() + .semaphore(rendered) + .value(ready.value), + )? + }; + // The last encode also waits for a point signalled only once + // the converter is gone, so it is certainly still queued, + // waiting on the converter's semaphore, when that happens. + let mut waits = vec![converted]; + if n + 1 == ENCODE_FRAMES { + waits.push(TimelinePoint::new(rendered, after_drop)); + } + encoder.encode_after(target, &waits)? + } + }; + sources.push(src); + pending.push(future); + } + // Dropped while an encode that waits on its points is still queued, as a + // caller rebuilding its converter might: its semaphore has to outlive + // that wait. + drop(converter); + if matches!(handoff, Handoff::Gpu) { + unsafe { + timeline_fns.signal_semaphore( + &vk::SemaphoreSignalInfo::default() + .semaphore(rendered) + .value(after_drop), + )? + }; + } + let mut stream = Vec::new(); + for future in pending { + stream.extend_from_slice(&pollster::block_on(future)?.data); + } + drop(encoder); + drop(sources); + unsafe { device.destroy_semaphore(rendered, None) }; + Ok(stream) +} + +/// Luma PSNR between two NV12 frames of the encode size. +fn luma_psnr(a: &[u8], b: &[u8]) -> f64 { + let n = (ENCODE_WIDTH * ENCODE_HEIGHT) as usize; + let mse = a[..n] + .iter() + .zip(&b[..n]) + .map(|(&x, &y)| (x as f64 - y as f64).powi(2)) + .sum::() + / n as f64; + if mse == 0.0 { + f64::INFINITY + } else { + 10.0 * (255.0f64 * 255.0 / mse).log10() + } +} + +#[test] +#[ignore = "requires a Vulkan Video device"] +fn encode_on_a_vulkan_1_1_device() -> Result<(), Box> { + common::init_logging(); + let own = VideoContextBuilder::new() + .enable_validation(validation_requested()) + .require_encode(Codec::H264) + .require_decode(Codec::H264) + .build()?; + let expected = encode_all(&own, Handoff::Cpu)?; + + let (entry, instance) = instance_1_1()?; + let builder = VideoContextBuilder::new().require_encode(Codec::H264); + let physical_device = same_gpu(&instance, &own); + let reqs = builder.encode_device_requirements(&entry, &instance, physical_device)?; + assert!( + reqs.extensions.contains(&ash::khr::push_descriptor::NAME), + "the converter needs push descriptors, which every driver this runs on has" + ); + let app = create_app_device(entry, instance, physical_device, &reqs)?; + let context = builder + .with_encode_queue(spare(&app, reqs.queues.encode.unwrap())) + .with_transfer_queue(spare(&app, reqs.queues.transfer)) + .with_compute_queue(spare(&app, reqs.queues.compute)) + .build_from_existing_encode( + app.entry.clone(), + app.instance.clone(), + app.physical_device, + app.device.clone(), + )?; + let actual = encode_all(&context, Handoff::Gpu)?; + drop(context); + + // Both streams decoded by the same decoder, so any difference is the + // encode's, or the handoff's: an encode that did not wait for its + // conversion would pick up the previous frame or a half-written one, far + // below the floor here. The floor is not equality because not every + // encoder is reproducible: RADV gives bit-identical streams, while ANV has + // been seen as low as 47 dB between two runs of the same synchronous path. + let expected_frames = decode_all(&own, &expected)?; + let actual_frames = decode_all(&own, &actual)?; + assert_eq!(actual_frames.len(), ENCODE_FRAMES as usize, "frame count"); + assert_eq!(expected_frames.len(), actual_frames.len(), "frame count"); + let psnrs: Vec = actual_frames + .iter() + .zip(&expected_frames) + .map(|(a, e)| luma_psnr(a, e)) + .collect(); + println!( + "adopted vs own encode, luma PSNR per frame: {:?}; streams bit-identical: {}", + psnrs, + actual == expected + ); + for (i, psnr) in psnrs.iter().enumerate() { + assert!( + *psnr >= 40.0, + "frame {i}: {psnr:.2} dB against the own-context encode" + ); + } + Ok(()) +} diff --git a/tests/common/mod.rs b/tests/common/mod.rs index 4aabcff2..62dbbc98 100644 --- a/tests/common/mod.rs +++ b/tests/common/mod.rs @@ -131,6 +131,9 @@ pub struct Readback { /// the caller has to get right. _context: VideoContext, device: ash::Device, + /// Barriers through the KHR entry point, which a device created below + /// Vulkan 1.3 still has; the adopted-device tests create one. + sync2: ash::khr::synchronization2::Device, queue: vk::Queue, pool: vk::CommandPool, command_buffer: vk::CommandBuffer, @@ -168,6 +171,7 @@ impl Readback { }; Ok(Self { _context: context.clone(), + sync2: ash::khr::synchronization2::Device::load(context.instance(), &device), device, queue: context.transfer_queue(), pool, @@ -200,7 +204,7 @@ impl Readback { layer_count: 1, }; let regions = [ - vk::BufferImageCopy2::default() + vk::BufferImageCopy::default() .buffer_offset(0) .buffer_row_length(frame.coded_width) .buffer_image_height(frame.coded_height) @@ -210,7 +214,7 @@ impl Readback { height: frame.coded_height, depth: 1, }), - vk::BufferImageCopy2::default() + vk::BufferImageCopy::default() .buffer_offset(y_size as u64) .buffer_row_length(frame.coded_width / 2) .buffer_image_height(frame.coded_height / 2) @@ -256,23 +260,22 @@ impl Readback { .dst_queue_family_index(vk::QUEUE_FAMILY_IGNORED) .image(frame.image) .subresource_range(range)]; - self.device.cmd_pipeline_barrier2( + self.sync2.cmd_pipeline_barrier2( self.command_buffer, &vk::DependencyInfo::default().image_memory_barriers(&to_src), ); } - self.device.cmd_copy_image_to_buffer2( + self.device.cmd_copy_image_to_buffer( self.command_buffer, - &vk::CopyImageToBufferInfo2::default() - .src_image(frame.image) - .src_image_layout(if needs_transition { - vk::ImageLayout::TRANSFER_SRC_OPTIMAL - } else { - frame.layout - }) - .dst_buffer(buffer) - .regions(®ions), + frame.image, + if needs_transition { + vk::ImageLayout::TRANSFER_SRC_OPTIMAL + } else { + frame.layout + }, + buffer, + ®ions, ); if needs_transition { @@ -287,7 +290,7 @@ impl Readback { .dst_queue_family_index(vk::QUEUE_FAMILY_IGNORED) .image(frame.image) .subresource_range(range)]; - self.device.cmd_pipeline_barrier2( + self.sync2.cmd_pipeline_barrier2( self.command_buffer, &vk::DependencyInfo::default().image_memory_barriers(&restore), ); diff --git a/tests/common/source.rs b/tests/common/source.rs index 58aec1f2..3c3292da 100644 --- a/tests/common/source.rs +++ b/tests/common/source.rs @@ -22,13 +22,21 @@ impl Drop for SrcImage { } } +/// Record and run one command buffer on the colour converter's queue, and +/// wait for it. +/// +/// The converter's queue because every image this touches is the converter's +/// too: the source it reads and the output it writes. Those images belong to +/// one queue family, and on a device with a separate transfer family (NVIDIA +/// has one) work there would need an ownership handover the tests do not do, +/// and could not make the source readable by shaders at all. pub unsafe fn one_shot( context: &VideoContext, record: F, ) -> Result<(), Box> { let device = context.device(); let pool_info = vk::CommandPoolCreateInfo::default() - .queue_family_index(context.transfer_queue_family()) + .queue_family_index(context.compute_queue_family()) .flags(vk::CommandPoolCreateFlags::TRANSIENT); let pool = unsafe { device.create_command_pool(&pool_info, None) }?; let cb_info = vk::CommandBufferAllocateInfo::default() @@ -46,8 +54,8 @@ pub unsafe fn one_shot( device.end_command_buffer(cb)?; let cbs = [cb]; let submit = vk::SubmitInfo::default().command_buffers(&cbs); - device.queue_submit(context.transfer_queue(), &[submit], vk::Fence::null())?; - device.queue_wait_idle(context.transfer_queue())?; + device.queue_submit(context.compute_queue(), &[submit], vk::Fence::null())?; + device.queue_wait_idle(context.compute_queue())?; device.destroy_command_pool(pool, None); } Ok(())