Skip to content

Fix Vulkan fallback crash with empty optional inputs - #6870

Open
futz12 wants to merge 1 commit into
Tencent:masterfrom
futz12:fix-vulkan-empty-optional-inputs
Open

Fix Vulkan fallback crash with empty optional inputs#6870
futz12 wants to merge 1 commit into
Tencent:masterfrom
futz12:fix-vulkan-empty-optional-inputs

Conversation

@futz12

@futz12 futz12 commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Summary

Fix a crash in mixed Vulkan/CPU execution when a CPU fallback layer has an optional input that is empty on both the
host and Vulkan sides.

Problem

When a layer does not support Vulkan, ncnn downloads its Vulkan inputs before executing the layer on CPU.

For optional inputs, such as an empty K/V cache during the first decoder step of MultiHeadAttention, both the host
Mat and GPU VkMat may be empty.

The existing fallback code only checked whether the destination was empty and then attempted to transfer from an empty
source:

if (blob_mats[bottom_blob_index].dims == 0)
{
    cmd.record_download(blob_mats_gpu[bottom_blob_index], ...);
}

This caused an access violation on Windows (0xC0000005) when record_download() received an empty VkMat.

The same issue could occur in the opposite host-to-Vulkan direction.

Fix

Only record a transfer when the destination is empty and the source actually exists:

  if (blob_mats_gpu[bottom_blob_index].dims == 0 &&
      blob_mats[bottom_blob_index].dims != 0)
  {
      cmd.record_upload(...);
  }

  if (blob_mats[bottom_blob_index].dims == 0 &&
      blob_mats_gpu[bottom_blob_index].dims != 0)
  {
      cmd.record_download(...);
  }

If both sides are empty, the input remains empty and is passed to the layer as an optional input.

Validation

Tested with a translation encoder/decoder graph where int8 MultiHeadAttention falls back to CPU while other layers run
on Vulkan.

Verified:

  • First decoder step with empty K/V cache no longer crashes.
  • Subsequent decoder steps with non-empty K/V cache work correctly.
  • Full int8 mixed Vulkan/CPU inference completes successfully.
  • fp32 Vulkan zero-copy inference remains unchanged.
  • NVIDIA RTX 4060 and Intel Raptor Lake Vulkan devices both pass.
  • CPU-only inference remains functional.

@github-actions github-actions Bot added the core label Jul 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant