Optimize Arm bf16 qb4w fully connected conversion path - #11112
Open
JakeStevens wants to merge 2 commits into
Open
Optimize Arm bf16 qb4w fully connected conversion path#11112JakeStevens wants to merge 2 commits into
JakeStevens wants to merge 2 commits into
Conversation
Keep qd8_bf16_qb4w fully connected native on Arm only when an optimized i8mm or NEON dot-product kernel is available and the input, output, and weight scales use the supported BF16 types. Otherwise, retain the fp32 rewrite fallback.
Add NEON v8 BF16-to-QS8 conversion microkernels and select the u32 kernel for Arm packed-LHS conversion. This replaces the scalar-imagic conversion path on Arm while preserving scalar and x86 fallback behavior.
Test plan:
cmake --build build-test --target bf16-qs8-vcvt-test -j8
./build-test/test/bf16-qs8-vcvt-test # 64/64 pass
cmake --build build-android-arm64 --target bf16-qs8-vcvt-test -j8
Benchmark (Pixel 10, Android 17/API 37, arm64-v8a):
Configured a Release build for Android API 24 with NDK r29/Clang 21.
cmake --build build-android-arm64 --target vunary-bench -j8
adb push build-android-arm64/bench/vunary-bench /data/local/tmp/
adb shell /data/local/tmp/vunary-bench --benchmark_filter='bf16_qs8_vcvt.*(neonv8_u32|scalar_imagic_u4)'
Results (real time, CPU scaling and ASLR enabled):
N=8192: neonv8_u32 1110 ns, 22.1348 GB/s; scalar_imagic_u4 1182 ns, 20.7856 GB/s
N=65536: neonv8_u32 8886 ns, 22.1246 GB/s; scalar_imagic_u4 9455 ns, 20.7941 GB/s
neonv8_u32 was 1.064-1.065x faster than scalar_imagic_u4.
qukhan
reviewed
Sep 3, 2026
| std::vector<uint16_t> weights_scale = {0x3F80, 0x3F80}; | ||
| std::vector<uint8_t> weights_data = std::vector<uint8_t>(32, 0x88); | ||
|
|
||
| void Build(enum xnn_datatype input_datatype, |
Collaborator
There was a problem hiding this comment.
I'm wondering how difficult it would be to use the Tensor API to build this test graph like was done in
XNNPACK/test/subgraph/fp16-to-fp32-fallback.cc
Lines 128 to 159 in a5acbbe
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two related changes:
Keep qd8_bf16_qb4w fully connected native on Arm only when an optimized i8mm or NEON dot-product kernel is available and the input, output, and weight scales use the supported BF16 types. Otherwise, retain the fp32 rewrite fallback.
Add NEON v8 BF16-to-QS8 conversion microkernels and select the u32 kernel for Arm packed-LHS conversion. This replaces the scalar-imagic conversion path on Arm while preserving scalar and x86 fallback behavior.
Test plan:
Benchmark (Pixel 10, Android 17/API 37, arm64-v8a):
Configured a Release build for Android API 24 with NDK r29/Clang 21.
Results (real time, CPU scaling and ASLR enabled):