Fix ARM64 ASIMD dot kernel operand constraints under LTO - #6009
Merged
martin-frbg merged 2 commits intoSep 3, 2026
Conversation
The inline assembly modifies its pointers, strides, and loop counter. Declare them as early-clobber read/write operands so LTO cannot assign them to registers that still contain live inputs.
Collaborator
|
Ugh, that haunted ThunderX2 kernel strikes again... Thanks for the fix. |
Collaborator
|
As an aside, having a full-LTO build in CI doesn't sound bad at all, as it would also serve to verify that LTO is possible with the codebase (I assume this would have to exclude the linktest done on shared library builds, as that one ignores actual function signatures) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fix incorrect operand constraints in the ARM64 ASIMD dot kernel that can produce wrong results with GCC full LTO. A small
cblas_strmvregression test is included.Cause and fix
An inline-assembly block has the instructions themselves and an operand list describing their effects to the compiler. GCC does not derive those effects from the instructions. Here the operand list says that the array pointers (
x,y), their strides (inc_x,inc_y), and the loop counter (j) are only read, although the instructions overwrite all five.This becomes a real failure under full LTO. GCC sees that both strides are
1and, based on the incorrect operand list, may keep them in the same register. The kernel converts each stride from elements to bytes by multiplying it by four because onefloatoccupies four bytes:The shared register is therefore multiplied twice and becomes
16. The pointers advance 16 bytes (four floats) instead of 4 bytes (one float), so the dot product reads the wrong elements.Declaring the five values as
+&rfixes the contract:+tells GCC that the value is both read and overwritten, while&prevents its register from being reused for another still-live input (rselects a general-purpose register). This keeps the two strides separate.n, which is not modified, remains input-only. The assembly instructions and numerical operations are unchanged.This is separate from #5917 and #5918, which fixed accumulator initialization and vector-register clobbers in the same kernel.
Validation
Tested on Linux AArch64 with static, single-threaded VORTEX builds and full LTO:
developwith the new test: fails withexpected 85, got 52.The reproducer multiplies a 3x3 lower-triangular matrix by
[11, 13, 17]; the correct result is[22, 85, 252].CI coverage
Permanently covering the original failure would require an additional static VORTEX full-LTO ARM64 job. That is possible, but the ongoing runner cost and maintenance seem disproportionate for this narrow case, so this PR only adds the regression test. I can add targeted CI coverage if maintainers prefer.